SlowGuess/NAACL_2022_OCR huggingface dataset:
Multi $^2$ WOZ: A Robust Multilingual Dataset and Conversational Pretraining for Task-Oriented Dialog
Chia-Chien Hung<sup>1</sup>, Anne Lauscher<sup>2</sup>, Ivan Vulić<sup>3</sup>, Simone Paolo Ponzetto<sup>1</sup> and Goran Glavaš<sup>4</sup>
<sup>1</sup>Data and Web Science Group, University of Mannheim, Germany
2MilaNLP, Bocconi University, Italy
3 LTL, University of Cambridge, UK CAIDAS, University of Wurzburg, Germany
{chia-chien, simone}@informatik.uni-mannheim.de
anne.lauscher@unibocconi.it iv250@cam.ac.uk
goran.glavas@uni-wuerzburg.de
Research on (multi-domain) task-oriented dialog (TOD) has predominantly focused on the English language, primarily due to the shortage of robust TOD datasets in other languages, preventing the systematic investigation of cross-lingual transfer for this crucial NLP application area. In this work, we introduce  $\text{MULTI}^2 \text{WOZ}$ , a new multilingual multi-domain TOD dataset, derived from the well-established English dataset  $\text{MULTIWOZ}$ , that spans four typologically diverse languages: Chinese, German, Arabic, and Russian. In contrast to concurrent efforts (Ding et al., 2021; Zuo et al., 2021),  $\text{MULTI}^2 \text{WOZ}$  contains gold-standard dialogs in target languages that are directly comparable with development and test portions of the English dataset, enabling reliable and comparative estimates of cross-lingual transfer performance for TOD. We then introduce a new framework for multilingual conversational specialization of pretrained language models (PrLMs) that aims to facilitate cross-lingual transfer for arbitrary downstream TOD tasks. Using such conversational PrLMs specialized for concrete target languages, we systematically benchmark a number of zero-shot and few-shot cross-lingual transfer approaches on two standard TOD tasks: Dialog State Tracking and Response Retrieval. Our experiments show that, in most setups, the best performance entails the combination of (i) conversational specialization in the target language and (ii) few-shot transfer for the concrete TOD task. Most importantly, we show that our conversational specialization in the target language allows for an exceptionally sample-efficient few-shot transfer for downstream TOD tasks.
Task-oriented dialog (TOD) is arguably one of the most popular natural language processing (NLP) application areas (Yan et al., 2017; Henderson et al.,
2019, inter alia), with more importance recently given to more realistic, and thus, multi-domain conversations (Budzianowski et al., 2018; Ramadan et al., 2018), in which users may handle more than one task during the conversation, e.g., booking a taxi and making a reservation at a restaurant. Unlike many other NLP tasks (e.g., Hu et al., 2020; Liang et al., 2020; Ponti et al., 2020, inter alia), the progress towards multilingual multi-domain TOD has been hindered by the lack of sufficiently large and high-quality datasets in languages other than English (Budzianowski et al., 2018; Zang et al., 2020) and more recently, Chinese (Zhu et al., 2020). This lack can be attributed to the fact that creating TOD datasets for new languages from scratch or via translation of English datasets is significantly more expensive and time-consuming than for most other NLP tasks. However, the absence of multilingual datasets that are comparable (i.e., aligned) across languages prevents a reliable estimate of effectiveness of cross-lingual transfer techniques in multi-domain TOD (Razumovskaia et al., 2021).
In order to address these research gaps, in this work we introduce MULTI $^2$ WOZ, a reliable and large multilingual evaluation benchmark for multi-domain task-oriented dialog, derived by translating the monolingual English-only MultiWOZ data (Budzianowski et al., 2018; Eric et al., 2020) to four linguistically diverse major world languages, each with a different script: Arabic (AR), Chinese (ZH), German (DE), and Russian (RU).
Compared to the products of concurrent efforts that derive multilingual datasets from English MultiWOZ (Ding et al., 2021; Zuo et al., 2021), our MULTI $^2$ WOZ is: (1) much larger – we translate all dialogs from development and test portions of the English MultiWOZ (in total 2,000 dialogs containing the total of 29.5K utterances); (2) much more reliable – complete dialogs, i.e., utterances
Human Language Technologies, pages 3687 - 3703
as well as slot-values, have been manually translated (without resorting to error-prone heuristics), and the quality of translations has been validated through quality control steps; and (3) parallel – the same set of dialogs has been translated to all target languages, enabling the direct comparison of the performance of multilingual models and cross-lingual transfer approaches across languages.
We then use MULTI $^2$ WOZ to benchmark a range of state-of-the-art zero-shot and few-shot methods for cross-lingual transfer in two standard TOD tasks: Dialog State Tracking (DST) and Response Retrieval (RR). As the second main contribution of our work, we propose a general framework for improving performance and sample-efficiency of cross-lingual transfer for TOD tasks. We first leverage the parallel conversational OpenSubtitles corpus (Lison and Tiedemann, 2016) to carry out a conversational specialization of a PrLM for a given target language, irrespective of the downstream TOD task of interest. We then show that this intermediate conversational specialization in the target language (i) consistently improves the DST and RR performance in both zero-shot and few-shot transfer, and (ii) drastically improves sample-efficiency of few-shot transfer.
2 Multi²WOZ
In this section we describe the construction of the MULTI $^2$ WOZ dataset, providing also details on inter-translator reliability. We then discuss two concurrent efforts in creating multilingual TOD datasets from MultiWOZ and their properties, and emphasize the aspects that make our MULTI $^2$ WOZ a more reliable and useful benchmark for evaluating cross-lingual transfer for TOD.
2.1 Dataset Creation
Language Selection. We translate all 2,000 dialogs from the development and test portions of the English MultiWOZ 2.1 (Eric et al., 2020) dataset to Arabic (AR), Chinese (ZH), German (DE), and Russian (RU). We selected the target languages based on the following criteria: (1) linguistic diversity (DE and RU belong to different Indo-European subfamilies - Germanic and Slavic, respectively; ZH is a Sino-Tibetan language and AR Semitic), (2) diversity of scripts (DE and RU use Latin and Cyrillic scripts, respectively, both alphabet scripts; AR script represents the Abjad script type, whereas the ZH Hanzi script belongs to logographic scripts),
(3) number of native speakers (all four are in the top 20 most-spoken world languages), and (4) our access to native and fluent speakers of those languages who are proficient in English.
Two-Step Translation. Following the well-established practice, we carried out a two-phase translation of the English data: (1) we started with an automatic translation of the dialogs – utterances as well as the annotated slot values – followed by (2) the manual post-editing of the translations. We first automatically translated all utterances and slot values from the development and test dialogs from the MultiWOZ 2.1 (Eric et al., 2020) (1,000 dialogs in each portion; 14,748 and 14,744 utterances, respectively) to our four target languages, using Google Translate.<sup>1</sup> We then hired two native speakers of each target language,<sup>2</sup> all with a University degree and fluent in English, to post-edit the (non-overlapping sets of) automatic translations, i.e., fix the errors in automatic translations of utterances as well as slot values.
Since we carried out the automatic translation of the utterances independently of the automatic translation of the slot values, the translators were instructed to pay special attention to the alignment between each translated utterance and translations of slot value annotations for that utterance. We show an example utterance with associated slot values after the automatic translation and manual post-editing in Table 1.
Quality Control. In order to reduce the translation costs, our human post-editors worked on disjoint sets of dialogs. Because of this, our annotation process contained an additional quality assurance step. Two new annotators for each target language judged the correctness of the translations on the random sample of 200 dialogs (10% of all translated dialogs, 100 from the development and test portion each), containing 2,962 utterances in total. The annotators had to independently answer the following questions for each translated utterance from the sample: (1) Is the utterance translation acceptable? and (2) Do the translated slot values match the translated utterance? On average, across all target languages, both quality annotators
$^{1}$ Relying on its Python API: https://pypi.org/project/googletrans
In order to reduce the translation costs, we initially attempted to post-edit the translations via crowdsourcing. We tried this for Russian using the popular platform Toloka (toloka.yandex.com); however, the translation quality remained unsatisfactory even after several post-editing rounds.
Utterance	Value for “attraction-name”
Original	No hold off on booking for now.	cineworld cinema
Can you help me find an attraction called cineworld cinema?
Automatic Trans.	目前暂无预订。您能帮我找到一个名为cineworld Cinema的景点吗?	Cineworld电影
Manual Correc.	目前暂无预订。您能帮我找到一个名为电影世界电影院的景点吗?	电影世界电影院
Table 1: Example utterance (from the dialog MUL0484) with a value for a slot ("attraction-name"). We show the original English text, the automatic translation to Chinese and the final translation after manual post-editing.
for the respective language answered affirmatively to both questions for  $99\%$  of all utterances. Adjusting for chance agreement, we measured the Inter-Annotator Agreement (IAA) in terms of Cohen's  $\kappa$  (Cohen, 1960), observing the almost perfect agreement $^3$  of  $\kappa = 0.824$  for the development set and  $\kappa = 0.838$  for test set.
Annotation Duration and Cost. In total, we hired 16 annotators, four for each of our four target languages: two for post-editing and two for quality assessment. The overall effort spanned almost full 5 months (from July to November 2021), and amounted to 1,083 person-hours. With the remuneration rate of  $16 \, \text{$/h}$ , creating MULTI $^2$ WOZ cost us $17,328.
2.2 Comparison with Concurrent Work
Two concurrent works also derive multilingual datasets from MultiWOZ (Ding et al., 2021; Zuo et al., 2021), with different strategies and properties, discussed in what follows.
GlobalWOZ (Ding et al., 2021) encompasses Chinese, Indonesian, and Spanish datasets. The authors first create templates from dialog utterances by replacing slot-value strings in the utterances with the slot type and value index (e.g., "...and the post code is cb238el" becomes the template "...and the post code is [attraction-postcode-1]". They then automatically translate all templates to the target languages. Next, they select a subset of 500 test set dialogs for human post-editing with the following heuristic: dialogs for which the sum of corpus-level frequencies of their constitutive 4-grams (normalized with the dialog length) is the largest. Since this selection step is independent for each language, each GlobalWOZ portion contains translations of
a different subset of English dialogs: this prevents any direct comparison of downstream TOD performance across languages. Even more problematically, the selection heuristic directly reduces linguistic diversity of dialogs chosen for the test set of each language, as it favors the dialogs that contain the same globally most frequent 4-grams. Due to this artificial homogeneity of its test sets, Global-WOZ is very likely to overestimate downstream TOD performance for target languages.
Unlike GlobalWOZ, AllWOZ (Zuo et al., 2021) does automatic translation of a fixed small subset of MultiWOZ plus post-editing in seven target languages. However, it encompasses only 100 dialogs and 1,476 turns; as such, it is arguably too small to draw strong conclusions about the performance of cross-lingual transfer methods. Its usefulness in joint domain and language transfer evaluations is especially doubtful, since it covers individual MultiWOZ domains with an extremely small number of dialogs (e.g., only 13 for the Taxi domain). Finally, neither Ding et al. (2021) nor Zuo et al. (2021) provide any estimates of the quality of their final datasets nor do they report their annotation costs.
In contrast to GlobalWOZ, MULTI $^2$ WOZ is a parallel corpus – with the exact same set of dialogs translated to all four target languages; as such it directly enables performance comparisons across the target languages. Further, containing translations of all dev and test dialogs from MultiWOZ (i.e., avoiding sampling heuristics), MULTI $^2$ WOZ does not introduce any confounding factors that would distort estimates of cross-lingual transfer performance in downstream TOD tasks. Finally, MULTI $^2$ WOZ is 20 times larger (per language) than AllWOZ: experiments on MULTI $^2$ WOZ are thus much more likely to yield conclusive findings.
3 Cross-lingual Transfer for TOD
The parallel nature and sufficient size of  $\mathrm{MULTI}^2\mathrm{WOZ}$  allow us to benchmark and compare a number of established and novel cross-lingual
3 According to Landis and Koch (1977), if  $\kappa \geq 0.81$
4 Interestingly, the authors do not provide any motivation or intuition for this heuristic. It is also worth noting that they count the 4-gram frequencies, upon which the selection of the dialogs for post-editing depends, on the noisy automatic translations.
transfer methods for TOD. In particular, (1) we first inject general conversational TOD knowledge into XLM-RoBERTa (XLM-R; Conneau et al., 2020), yielding TOD-XLMR ( $\S 3.1$ ); (2) we then propose several variants for conversational specialization of TOD-XLMR for target languages, better suited for transfer in downstream TOD tasks ( $\S 3.2$ ); (3) we investigate zero-shot and few-shot transfer for two TOD tasks: DST and RR ( $\S 3.3$ ).
3.1 TOD-XLMR: A Multilingual TOD Model
Recently, Wu et al. (2020) demonstrated that specializing BERT (Devlin et al., 2019) on conversational data by means of additional pretraining via a combination of masked language modeling (MLM) and response selection (RS) objectives yields improvements in downstream TOD tasks. Following these findings, we first (propose to) conversationally specialize XLM-R (Conneau et al., 2020), a state-of-the-art multilingual PrLM covering 100 languages, in the same manner: applying the RS and MLM objectives on the same English conversational corpus consisting of nine human-human multi-turn TOD datasets (see Wu et al. (2020) for more details). As a result, we obtain TOD-XLMR – a massively multilingual PrLM specialized for task-oriented conversations. Note that TOD-XLMR is not yet specialized (i.e., fine-tuned) for any concrete TOD task (e.g., DST or Response Generation). Rather, it is enriched with general task-oriented conversational knowledge (in English), presumed to be beneficial for a wide variety of TOD tasks.
3.2 Target-Language Specialization
TOD-XLMR has been conversationally specialized only in English data. We next hypothesize that a further conversational specialization for a concrete target language X can improve the transfer EN  $\rightarrow$  X for all downstream TOD tasks. Accordingly, similar to Moghe et al. (2021), we investigate several intermediate training procedures that further conversationally specialize TOD-XLMR for the target language X (or jointly for EN and X). For this purpose, we (i) compile target-language-specific as well as cross-lingual corpora from the CCNet (Wenzek et al., 2020) and OpenSubtitles (Lison and Tiedemann, 2016) datasets and (ii) experiment with different monolingual, bilingual, and cross-lingual training procedures. Here, we propose a novel cross-lingual response selection (RS) objective and demonstrate its effectiveness in cross-lingual transfer for downstream TOD tasks.
Training Corpora. We collect two types of data for language specialization: (i) "flat" corpora (i.e., without any conversational structure): we simply randomly sample 100K sentences for each language from the respective monolingual portion of CCNet (we denote with Mono-CC the individual 100K-sentence portions of each language; with Bi-CC the concatenation of the English and each of target language Mono-CCs, and with Multi-CC the concatenation of all five Mono-CC portions); (ii) parallel dialogs (in EN and target language X) from OpenSubtitles (OS), a parallel conversational corpus spanning 60 languages, compiled from subtitles of movies and TV series. We leverage the parallel OS dialogs to create two different crosslingual specialization objectives, as described next.
Training Objectives. We directly use the CC portions (Mono-CC, Bi-CC, and Multi-CC) for standard MLM training. We then leverage the parallel OS dialogs for two training objectives. First, we carry out translation language modeling (TLM) (Conneau and Lample, 2019) on the synthetic dialogs which we obtain by interleaving  $K$  randomly selected English utterances with their respective target language translations; we then (as with MLM), dynamically mask  $15\%$  of tokens of such interleaved dialogs; we vary the size of the context the model can see when predicting missing tokens by randomly selecting  $K$  (between 2 and 15) for each instance. Second, we use OS to create instances for both monolingual and cross-lingual Response Selection (RS) training. RS is a simple binary classification task in which for a given pair of a context (one or more consecutive utterances) and response (a single utterance), the model has to predict whether the response utterance immediately follows the context (i.e., it is a true response) or not (i.e., it is a false response). RS pretraining has been proven beneficial for downstream TOD in monolingual English setups (Mehri et al., 2019; Henderson et al., 2019, 2020; Hung et al., 2022).
In this work, we leverage the parallel OS data to introduce the cross-lingual RS objective, where the context and the response utterance are not in the same language. In our experiments, we carry out both (i) monolingual RS training in the target language (i.e., both the context and response utterance are, e.g., in Chinese), denoted RS-Mono, and (ii) cross-lingual RS between English (as the source language in downstream TOD tasks) and the target language, denoted RS-X. We create hard RS neg
EN Subtitle	ZH Subtitle
- Professor Hall. - Yes. - I think your theory may be correct. - Walk with me. Just a few weeks ago, I monitored the strongest hurricane on record. The hail, the tornadoes, it all fits. Can your model factor in storm scenarios?	-霍尔教授 -是的 -我认为你的理论正确 -跟我来 上周我观测到史上最大的飓风 霹暴和龙卷风也符合你的理论 你能预测暴风雨的形成吗?
Translation LM (TLM)	- Professor Hall. - Yes. - I think your theory may be [MASK]. - Walk with...-霍尔教授 -是的 -我认为你的[MASK]正确...
Response Selection (RS)	Context: 上周我观测到史上最大的飓风 霹暴和龙卷风也符合你的理论	Monolingual (RS-Mono) True Response: 你能预测暴风雨的形成吗? False Response: 你有彼得的电脑断层扫描吗?	Cross-lingual (RS-X) True Response: Can your model factor in storm scenarios? False Response: Do you have Peter's CT scan results?
Table 2: Examples of training instances for conversational specialization for the target language created from OpenSubtitles (OS). Top row: an example of a dialog created from OS, parallel in English and Chinese. Below are training examples for different training objectives: (1) Translation Language Modelling (TLM) on the interleaved English-Chinese parallel utterances; (2) two variants of Response Selection (RS) – (a) monolingual in the target language (RS-Mono) and (b) cross-lingual (RS-X).
atives, by coupling contexts with non-immediate responses from the same movie or episode (same imdbID), as well as easy negatives by randomly sampling  $m \in \{1,2,3\}$  responses from a different movie of series episode (i.e., different imdbID). Hard negatives encourage the model to reason beyond simple lexical cues. Examples of training instances for OS-based training (for EN-ZH) are shown in Table 2.
3.3 Downstream Cross-lingual Transfer
Finally, we fine-tune the various variants of TOD-XLMR, obtained through the above-described specialization (i.e., intermediate training) procedures, for two downstream TOD tasks (DST and RR) and examine their cross-lingual transfer performance. We cover two cross-lingual transfer scenarios: (1) zero-shot transfer in which we only fine-tune the models on the English training portion of MultiWOZ and evaluate their performance on the MULTI $^2$ WOZ test data of our four target languages; and (2) few-shot transfer in which we sequentially first fine-tune the models on the English training data and then on the small number of dialogs from the development set of MULTI $^2$ WOZ, in similar vein to (Lauscher et al., 2020). In order to determine the effect of our conversational target language specialization (§3.2) on the downstream sample efficiency, we run few-shot experiments with different numbers of target language training dialogs, ranging from  $1\%$  to  $100\%$  of the size of MULTI $^2$ WOZ development portions.
Evaluation Tasks and Measures. We evaluate different multilingual conversational PrLMs in cross-lingual transfer (zero-shot and few-shot) for
two prominent TOD tasks: dialog state tracking (DST) and response retrieval (RR).
DST is commonly cast as a multi-class classification task, where given a predefined ontology and dialog history (a sequence of utterances), the model has to predict the output state, i.e., (domain, slot, value) tuples (Wu et al., 2020). We adopt the standard joint goal accuracy as the evaluation measure: at each dialog turn, it compares the predicted dialog states against the manually annotated ground truth which contains slot values for all the (domain, slot) candidate pairs. A prediction is considered correct if and only if all predicted slot values exactly match the ground truth.
RR is a ranking task that is well-aligned with the RS objective and relevant for retrieval-based TOD systems (Wu et al., 2017; Henderson et al., 2019): given the dialog context, the model ranks  $N$  dataset utterances, including the true response to the context (i.e., the candidate set includes the one true response and  $N - 1$  false responses). We follow Henderson et al. (2020) and report the results for  $N = 100$ , i.e., the evaluation measure is recall at the top 1 rank given 99 randomly sampled false responses, denoted as  $\mathbb{R}_{100}@\mathbb{1}$ .
Models and Baselines. We briefly summarize the models that we compare in zero-shot and few-shot cross-lingual transfer for DST and RR. As baselines, we report the performance of the vanilla multilingual PrLM XLM-R (Conneau et al., 2020) $^6$  and its variant further trained on the English TOD data from (Wu et al., 2020): TOD-XLMR ( $\S 3.1$ ). Comparison between XLM-R and TOD-XLMR
5The model is required to predict slot values for each (domain, slot) pair at each dialog turn.
$^{6}$ We use xlm-roberta-base from HuggingFace.
quantifies the effect of conversational English pretraining on downstream TOD performance, much like the comparison between BERT and TOD-BERT done by Wu et al. (2020); however, here we extend the comparison to cross-lingual transfer setups. We then compare the baselines against a series of our target language-specialized variants, obtained via intermediate training on CC (Mono-CC, Bi-CC, and Multi-CC) by means of MLM, and on OS jointly via TLM and RS (RS-X or RS-Mono) objectives (see §3.2 again).