A Systematic Study of Performance Disparities in Multilingual Task-Oriented Dialogue Systems
Authors: Songbo Hu
Han Zhou
Moy Yuan
Milan Gritta
Guchun Zhang
Ignacio Iacobacci
Anna Korhonen
Ivan Vulić
Venue: EMNLP 2023 Main (2023)
Keywords: multilingual task-oriented dialogue systems
analysis of performance disparities

Abstract: Achieving robust language technologies that can perform well across the world's many languages is a central goal of multilingual NLP. In this work, we take stock of and empirically analyse task performance disparities that exist between multilingual task-oriented dialogue (ToD) systems. We first define new quantitative measures of absolute and relative equivalence in system performance, capturing disparities across languages and within individual languages. Through a series of controlled experiments, we demonstrate that performance disparities depend on a number of factors: the nature of the ToD task at hand, the underlying pretrained language model, the target language, and the amount of ToD annotated data. We empirically prove the existence of the adaptation and intrinsic biases in current ToD systems: e.g., ToD systems trained for Arabic or Turkish using annotated ToD data fully parallel to English ToD data still exhibit diminished ToD task performance. Beyond providing a series of insights into the performance disparities of ToD systems in different languages, our analyses offer practical tips on how to approach ToD data collection and system development for new languages.

A Systematic Study of Performance Disparities in Multilingual Task-Oriented Dialogue Systems
Songbo Hu!
Han Zhou' Zhangdie Yuan² Milan Gritta²
Guchun Zhang Ignacio lacobacci" Anna Korhonen' Ivan Vulić'
'Language Technology Lab, University of Cambridge, UK
*Department of Computer Science and Technology, University of Cambridge, UK Huawei Noah's s Ark Lab, London, UK
iv2.lkk22.ii2200aaaaa 1.2 h241,, zy317, alk23, iv250)@cam. ac. uk
3 imilanngrittn guchun. zhang, iin.iiginnioo.o wal.laaccoobcco Jecuaa Jehuawei com
Abstract
The development of ToD systems requires large- scale, in-domain datasets to exploit the potential of deep icarning-basee components to effectively han- dle complex dialogue patterns (Budzianowski et al., 2018; Lin et al.. 2021b; Hu et al., 2023). The cre- ation of datasets for new domains and languages is challenging (Shah et al., 2018; Larson and Leach, 2022): it requires expertise and substantial time and financial investment that typically exceeds the requirements of other NLP tasks (Casanueva al., 2022). Therefore, the progress in ToD is still largely confined to a small number of high-resouree languages (Razumovskaia et al., 2022).
Achieving robust language technologies that can perform well across the world's many lan- guages is central goal of multilingual NLP. In this work, we take stock of and empirically analyse task performance disparities that ex- ist between multilingual task-oriented dialogue (ToD) systems. We first define new quantita- measures of absolute and relative equiv- alence in system performance, capturing dis- parities across languages and within individual languages. Through series of controlled ex- periments, we demonstrate that performance disparities depend on a number factors: nature of the ToD task at hand, the underlying pretrained language model, the target language, and the amount of ToD annotated data. We em- pirically prove the existence of the adaptation bias and intrinsic biases in current ToD sys- tems: e.g., ToD systems trained for Arabic or Turkish using annotated ToD data fully paral- lel to English ToD data still exhibit diminished ToD task performance. Beyond providing series of insights into the performance dispari- ties ToD systems different languages, our analyses offer practical tips how approach ToD data collection and system development for new languages.
On the other hand, recent advances in multilin- gual pretrained language models (mPLMs) (De- vlin et al., 2019; Conneau et al., 2020a; Xue et al., 2021a) have conceptually enabled cross-lingual transfer between any or more languages seen at pretraining (Wu and Dredze, 2019; Conneau et al., 2020b; Hu et al., 2020), or even to unseen lan- guages (Ansell et al., 2021). These models offer a promising basis for developing ToD systems for previously unsupported languages, obviating the need for expensive, language-specinee data acqui- sition. However, mPLMs do not equally benefit all languages: previous studies focused on stan- dard NLP tasks (Hu al., 2020; Lauscher et al., 2020) have revealed that they exhibit unequal per- formance across languages, with particularly ad- verse effects observed for low-resource languages.
1 Introduction
The aim of task-oriented dialogue (ToD) (Gupta al., 2006; Tür al., 2010; Young, 2010) to model the interaction between a human user and a system agent with the goal of accomplishing specific, well-defined tasks. To date, ToD tech- nology has proven useful for many sectors, rang- ing from the hospitality industry (Henderson et al., 2014, 2019) to healthcare (Laranjo al., 2018), online shopping (Yan et al., 2017), banking (Alti- nok, 2018), and travel (Raux et al., 2005; El Asri et al., 2017), among others. These systems provide users with access to state-of-the the-art services and they catalyse technological expansion.
The scarcity or complete absence of high-quality in-domain ToD data (i.e., adaptation bias; Bom- masani et al. 2021) and the compounding under- representation of target languages inherent to mPLMs (i.e., intrinsic bias; Bommasani et al. 2021) are likely collectively contribute to the exacer- bation of performance disparities in multilingual ToD systems. These disparities, specifically error rate disparities (Barocas al., 2019), may result in extrinsic harms for downstream applications (Gal- liers and Spärck Jones, 1993), affecting the utility,

---

experience, and satisfaction of system users. They may also deprive non-English speakers of equal opportunities, hampering their ability to benefit from the rapid and ever-growing advancements in language technology.
disparities between English and other languages even when the development approaches are simi- lar. Such disparities can be alleviated through in- language data collection, with the necessary data volume increases as the task complexity rises. Ad- ditionally, we demonstrate the feasibility and bene- fits of strategic annotation budget allocation in mul- tilingual dialogue collection, showing how model performance can be enhanced without additional annotation costs.
Aligned with the broader goal (Council of Eu- ropean Union, 2018) of developing multilingually inclusive NLP, this work offers a first systematic analysis of the performance disparities exhibited by multilingual ToD systems.' In $3, we first propose two measures for studying performance disparity, namely the absolute and relative equiv- alence in system performance. The former evalu- ates the disparities in task performance across lan- guages, while the latter measures the achieved per- formance compared to its potential 'upper bound' performance within the same language.
We hope that the key findings of this study, beyond empirically validating the current perfor- mance disparities in multilingual ToD systems, will also serve as practical guidelines for ToD sys- tem developers, contributing to the long-term goal of constructing robust and inclusive ToD systems for much larger number of languages, aligning with the recent initiatives in MT (Siddhant et al., 2022; NLLB Team al., 2022) and other NLP tasks (Søgaard, 2022; Ruder et al., 2023). Our data and code are available at: https: thg.tub......tgggg/ggg/gggg... thub. chmb Fambridgell t1 //ml tonlliiiwwn 3woz analysis.
We apply these measures to study performance disparities in a recently developed MULTI3WoZ resource (Hu et al., 2023), large-scale multilin- gual multi-domain multi-parallel ToD dataset.
In particular, relying on MULTI3 Woz, oZ, we in- vestigate the following research questions:
Related Work
(RQ1) Given the recent progress in mPLMs, Ma- chine Translation, and cross-lingual transfer, is language-specify data still necessary for the devel- opment of a ToD system for new language?
Multilingual ToD Systems and Datasets. Previ- ous work has shown that fine-tuning methods are competitive or even outperform in-context learn- ing with large language models (LLMs) for cross- lingual transfer (Asai et al., 2023) and ToD sys- tems (Hudešek and Dušek, 2023; Heck et al., 2023), This applics even with smaller models and when the number of training examples limited. There- fore, state-of-the-un multilingual ToD systems are typically developed by fine-tuning mPLMs with -specfic datascts.
(RQ2) Given access to the same mPLMs, equivalent amounts of high high-quality in in-language training data, and a similar development approach as that used to create an English ToD dataset, is it possible to develop ToD system for a new language that achieves near-Enve Eagr-Engiish performance?
(RQ3) How much training data is required in a new language to achieve performance comparable to a ToD system trained with an equivalent amount of in-domain, in-language data as in English?
Developing ToD systems without the disadvan- tage of data scarcity² necessitates a dataset that has the following properties (Hu et al., 2023): (i) large-scale data, containing sufficient volume of (both training and test) data for in-depth compara- tive cross-language analyses, (ii) comprehensive task-specific annotations, supporting the training of multiple ToD tasks, (iii) natural-sounding con- versations, ensuring natural flow within the dia- logues and avoiding artificial performance infla- tion (Majewska et al., 2023), and (iv) wide lan- guage coverage, enabling controlled experiments
(RQ4) Which data collection strategy maximises system performance across metrics while minimis- ing the amount of annotation required? Such strategy could optimise the cost-efficiency of anno- tation for new language.
Our empirical findings highlight performance
'In this study, we use the terms performance disparity and error rate disparity interchangeably. We acknowledge that real-work production settings, there are various evaluation dimensions to assess system performance, with error rate, while critical, being only one of them.
"The term 'disadvantage of data scarcity' in this context specifically refers to the limited quantity and/or quality of in-domain in-language training data for multilingual ToD systems. This data scarcity issue is a major source of the adaptive bias mentioned carlier.
'A related experimental study conducted by Debnath et al. (2021) on multilingual QA systems emphasises taking the most advantage of existing resources rather than expanding the system to languages that have not been previously supported.

---

and comparison across representative of lan- guages and in different linguistic contexts.
community it involves addressing long-standing challenges in dialogue system evaluation (Mchri et al., 2022). Our study focuses on more realis- tic task: measurement of performance disparities using established automatic evaluation metrics as proxies to measure users' judgement of the system. This approach aligns with similar methodologies adopted in related works that assess system per- formance on other NLP tasks (Blasi et al., 2022; Khanuja et al., 2023, inter alia).
The majority of the existing ToD datasets fail to simultaneously satisfy all the aforementioned prop- erties. For example, most existing multilingual ToD datasets (Upadhyay et al., 2018; Schuster et al., 2019; Dao et al., 2021; Moghe et al., 2022; Majewska et al., 2023, inter alia) have been de- signed support single component within ToD system, typically Natural Language Understanding (NLU), failing to fulfil property (ii). Recently, sev- eral datasets have emerged that support multiple dialogue tasks, but again none of these satisfy all the properties mentioned above. For instance, some datasets such as Mult22WOZ (Hung et al., 2022) do not provide any training data at all, while others like AlIWOZ (Zuo al., 2021) only provide min- imal training set of post-edited dialogee.. Datasets such as GlobalWOZ. (Ding et al., 2022) rely solely on automatically created machine-translated training data, failing to fulfil properties (i) and (iii). BiToD (Lin et al., 2021b) is reasonably large and features coherent dialogues, but it spans only two, highest-resourced languages, failing (iv).
3.2 Notion of Equivalence in Performance
Preliminaries and Notation. The development of a ToD system involves training a dialogue model, denoted as P(-). using a task-specific dialogue dataset ID. The system's performance is then eval- uated using automatic evaluation metric M(.). In multilingual setup, the dialogue model P(-) is commonly implemented using mPLM. The training procedure involves cross-lingual transfer training, utilising a dataset ID SRC a typically high- resource source language, and a dataset D JGT in a low-resource target language. The following dis- cussion is made under an ideal assumption that the datasets IDSCC and IT) are of equal size and quality, and cover exactly the same conversational flows and information types (which is provided by MULTI3 W We assume that the system de- veloped the target language is not affected by in-domain data scarcity, domain shifts, or different in-domain distributions.
Our study is enabled by new multilingual ToD dataset that overcomes the limitations of earlier datasets: MuLriWWoZ- the first lcage-scale, mul- tilingual multi-domain multi-parallel ToD dataset that has all of the properties discussed above and can, therefore, serve as comprehensive bench- mark and departure point for our analysis. For completeness, we provide summary description of MULTI3WoZ in $3.3.
A bsolute 0- O-Equivalence. We train system, de- noted as pssc (.), on the source language dataset DSSCC and evaluate its performance using the met- ric M(PSRC()) (-)). Similarly, we train a system for the target language, denoted as (.), using the target language dataset DTT Subsequently, we assess the performance of the target language sys- tem using the metric M(-). To quantify perfor- mance disparities in systems development across different languages, we introduce the concept of absolute f-equivalence. This concept represents a performance threshold of system developed in the target language, which is compared against the fully supervised system in the source language. We define that two systems achieve absolute 0- equivalence iff M(PIGT(.)) > 0 M(PSRC(.)) where [0, 1j.4 For instance, the ToD system
3 Methodology
3.1 Measuring Performance of ToD Systems This study investigates the performance dispar- ities observed in ToD systems across different languages. Ideally, to be inclusive and widely useful, multilingual ToD systems should attain comparable performance for all the languages for which NLP can be developed. The development of strongly multilingual systems that can deal with di- verse linguistic patterns is also important for robust- ness of ToD systems. However, measuring the full generality of system performance is challenging: it requires a comprehensive procedure involving e.g.. a large-scale user study and the apparatus of mea- surement theory (Barocas et al., 2019). This is only attainable via collaborative efforts of the research
"The relationship between evaluation metrics and the ac- tual henefits experienced by users can he complex and non- linear (Blasi et al., 2022). For instance, in the domain of

---

developed for French may demonstrate a perfor- mance level of 0.95-equivalence, indicating a sub- stantial degree of equivalence in comparison to the system in the source language. However, it may not achieve the performance levels of 0.99-equivalence or perfect I- I-equivalence. Unless noted otherwise, we use specific example 0 value of 0.95 to facili- tate the comparison in this study.
3.3 MULTI3WOZ: A Quick Recap
MULTIBWOZ (Hu et al., 2023) is a new large-scale multilingual, multi-domain, and multi-parallel ToD dataset that, unlike previous datasets, fully meets the criteria mentioned in $2 for our inves- tigation: it provides training, development, and test sets in both the source and target languages (i.e., IDSRC and DTOT that are not only of equal in size and created using the same protocols by native speakers of the respective languages, but also pos- sess comparable quality through covering the same conversation flows via multi-parallel dialogues.
To ensure an equal performance in each lan- guage, the ideal scenario would involve two sys- tems achieving absolute l-equivalence. However, our experimental results in $4 (RQ2) will reveal a significant disparity between systems developed for English and other languages, even when trained on an equal amount of dialogue data. This discrepancy can be attributed the inherent bias of the underly- ing mPLM, which cannot be easily mitigated, espe- cially when it comes to low-resource languages. To gain deeper insights into performance disparities, we also define relative d-equivalence.
This dataset D comprises a total of 36,640 (4x9, 160) parallel yet linguistically and cultur- ally adapted dialogues in four languages: Ara- bic (IDARA; Afro-Asiatic), English (I) Indo- European), French (ID Indo-European), and Turkish TUR; Turkic), constructed by leveraging the well-established multi-domain English Multi- WoZ. dataset (Budzianowski et al., 2018), partic- ularly its cleaned version 2.3 (Han et al., 2021). In this study, we consider the English dialogues DENG the source dataset DDRCc, Additionally, among other languages, each dataset DARA,, D and TUR is treated as target dataset DOT Each target dataset is generated by adapting a recent bottom-up outline-based approach introduced by Majewska et al. (2023) to address the limitations of translation- based design. For further technical de- tails concerning MULTI3WOZE we refer the reader to the original work (Hu et al., 2023).
Relative 0-Equivalence. It is challenging col- lect large in-domain, in-language training dataset for ToD. Consequently, an in-language training dataset yeu is often considerably smaller in size compared to DGT and DDscc, Assuming that the full D is available, we train a system, denoted (.) for the target language. Addition- ally, we train another system, (-), using ei- ther the limited target language dataset DTTT fow or a combination of DSRC and Djew few through cross- lingual training. To quantify the performance dis- parities arising from the scarcity of in-language training data for a target language, we employ the term relative 0-equivalence. This concept repre- sents performance threshold for a system devel- oped in the target language, which is compared against the system trained with full dataset in the same target language. We define that the two sys- tems achieve relative d-equivalence iff the metric M(P]ar few (.)) 2 M(PreT (-)), where 0 E [0,1]. 1]. Relative 0-equivalence threshold for higher 0 values might be achievable by enlarging the size few without changing the underlying mPLM. By focusing on strategically expanding the target language dataset, we can analyse the impact of adaptive bias in ToD data without explicitly mod- elling the compounding intrinsic bias of the model.
few
Experimental Setup
Our experiments involve three standard ToD tasks: Natural Language Understanding (NLU), Dialogue State Tracking (DST), and Natural Language Gen- cration (NLG). We briefly recap each task along with its experimental setup.
NLU, This task is commonly decomposed into two well-established subtasks: intent detection (ID) and slot labeling (SL). In ID, the objective is to classify the user's utterance and to determine the presence of a specific domain-intent pair from a predefined set of intents in the ontology. It is treated as a multi- class classification task. SL is a sequence tagging task that identifies the presence of value and its corresponding slot within the utterance.
of DGT
machine-assisted human translation, the correlation between machine translation accuracy and productivity gain can ex- hibit intricate patterns (Sanchez-Torron and Kochn, 2016). In this study, we simplify the relationship between the chosen evaluation metric and users' judgement of the system.
"We elaborate on all the other details in Appendix A. "For example, Restaurant-Infon Inform is the domain-intent pair for the utterance There will be of us and 19:45 would be great. The value 5 corresponds to the slot number of people,

---

We evaluate ID and SL methods implemented atop XLM-Rbase and XLM-RLarge (Conneau et al., 2020a). The evaluation conducted by measuring the accuracy of correctly identifying the presence of all domain-intent pairs, as well as its F1 score. For SL, we employ the commonly used BIO la- belling scheme to annotate each token in the user's utterance. The evaluation of SL is based on Fl score, precision, and recall in accurately identify- ing each slot value within the utterance.
ties across different domains and multilingual NLP tasks, thereby promoting the development of inclu- sive and linguistically robust systems.
5 Results and Discussion
Centred around the definitions of d-equivalence, our experiments are focused on answering research questions from $1. Our primary objective is to identify, quantify, and mitigate performance dispar- ities within multilingual ToD systems. To solidify our empirical findings, we present supplementary experimental results in Appendix B; they offer ad- ditional support to our main claims as well as aux- iliary findings.
DST. Our DST models are based on TSDST (Lin et al., 2021a), strong DST baseline which trans- forms DST into QA task by incorporating slot descriptions. For implementation, we utilise mul- tiple multilingual sequence-to-sequees including mT5small. mT51arge (Xue et al., 2021b), Flanl and Flan-T51arge (Chung et al., 2022). To evaluate our approach, we follow the standard MultiWOZ preprocessing and evaluation setups (Wu et al., 2019), excluding the 'hospital' and 'police' domains due to the absence of test. dialogues in these domains. We report Joint Goal Accuracy (IGA), Turn Accuracy, and Joint F1.7
models,
To answer RQI, we conduct evaluations of mPLMs in three standard scenarios: fully super- vised, zero-shot cross-lingure transfer, and translate train. Our empirical results highlight the necessi- ties of language-speciff data in the development of a ToD system for a new language. In response to RQ2, our subsequent analysis highlights per- formance disparities among task- dialogue models, even when trained with equivalent amounts of data. We further examine model performances in both few-shot and 'many'-sho settings, shedding light on the correlations between task complexity and the required amount of data, thus addressing RQ3. Finally, we investigate the advantages of col- lecting a dataset that maximises lexical coverage. achieving improved cost-efficiency of annotation, answering RQ4.
NLG. We approach the NLG task sequence-to-sequence problem, again sup- ported by mT5small, mT51arge, Flan-T5small, and Flan-T51arge- Our approach involves taking the concatenation of the two preceding his- torical utterances and the linearised 'oracle' dialogue act (e.g., [antoomm//rsstturrnn///(rrr input to generate a system response. Following MultiWOZ conventions, evaluate with the corpus BLEU score (Papineni et al., 2002). However, we evaluate lexicalised utterances without performing delexicalisation. We also report ROUGE-L (Lin, 2004) and METEOR (Banerjee and Lavie, 2005).
(RQI) Supervised, Translation, and Zero-shot. To answer RQI, we examine the performance of individual modules multilingual ToD sys- tems under three scenarios: (1) the ideal situa- tion, where a multi multi-parallel dataset in the target language D ाоा is available in equal size and qual- ity English, (2) the real life situation for most languages, where there is complete absence of in-domain in-language training data, and (3) the standard translate-tran situation for languages that have a reliable machine translation (MT) system but lack ver, In the latter case, we employ an MT system to generate a dataset D in the target language from ENG
One More Thing... We emphasise the wide appli- cability of our proposed notions of f-equivalence, as they are not restricted to any specific metric, model, or dataset. They are task-agnostic notions applicable to a diverse range of models beyond those mentioned in ToD tasks. We propose using them tool to measure performance dispari-
and the value 19:45 corresponds to slot time_of_ boookigng. "The JGA measure represents the proportion of dialogue turns in the dataset where all slots have been correctly filled with their ground truth values. For assessing turn accuracy, each (domain, slot, value) triplet is compared against its corre- sponding ground truth label. Furthermore, the joint Fl score is computed as the macro-averaged Fl score across all these slots, considering all the turns present in the datasct.
In scenario (1), we adopt a fully supervised ap- proach for training and evaluating individual lan- guages separately. For example, we train and eval- uate systems P-RA(-) exclusively on the Arabic dataset DARA In scenario (2), we perform zero- shot cross- lingual transfer evaluation for each tar-

---

Intent Detection
Stot Labelling
Dialogue State Tracking
Natural Language Generation
Language
Accuracy
Fl
Prectsion
Recall
JGA
Turn Acc.
F1
BLEU
ROUGE
METTOR
Fully Supervised
ENG
93.2o20
96. lug.3
6
95.7
1843
57.2
97.7
92.5
20. 20.
47.345,4
42.944.1
ARA
92.7G2.1
95.0g4.
42.4422
48.5
45.2a50
42.0p
96.4663
88.0
6.8179
0.815.0
19.436.
FRA
89.2s88
93.0g2
76.922
m9.2
78.0
47.6
96.891
89. 89.4m
12.9
39.6ax9
33.8
TUR
92.201.5
95.0g44
76.9
87.6
87.1
50.5s29
97.1
90.5v1
5.5242
24.7
22.548.
Zero-shot Cross-lingual Transfer
ARA
82.1687
88.2
27.4
31.2
29.2a..
1915
82.5
17.0 ss
0.2ga
2.5
2.42.
FRA
83,9y20
89. 89.8
58.5
61.2
59.8 59.8449
5.5k7
86.6gs.
40.1
0.5
4.24
6.159
TUR
87.0rL
91 A41.7
68. 1ax.5
74.7ma
71 21.2
3,5
85.2
34.4152
0.3u4
3.744
6.1
Translate Train
ARA
72.0gms
81. $1.915.9
Oo
0g
0o
9.232.
89 lou,
52.7
1.112
6.36.
7,475
FRA
66.243.
77.4449
10.4 98
90.6w04
60.0
2.6522
20.433.
15.
TUR
71.2
82.2
0g
0g 06
10.5129
90.5v4.
60.4
1.01
16.9174
12.7isn
Table 1: (1) Fully supervised, (2) zero-shot cross- cross-lingual transfer from English DING, and (3) translate train from GlobalWOZ. T&E' dataset tar for ID, SI., DST, and NLG tasks MULTI3Wo7. Performance evaluations were conducted for two distinct model categories: large models, namely XLM- and mT5garge, and small models, specifically XLM-Roase and mT5snall. The reported results follow the format "large modelamal model" For example, the entry 93. 93.292.0 denotes XLM-Riarge achieves 93.2 accuracy in English ID whereas XLM- Rease achieves 92.0.
get language. To achieve this, we train an English system (-) using the EYENG dataset, and sub- sequently assess its performance on the target lan- guage datasets, namely ID DARA,, D and ITTRR
Absolute 6 B-Euuvvllence
1.2
ENG
ARA
FRA
TUR
1.0
Lastly, in scenario (3), we utilise the 'F&E' propor- tion of the GlozalWOZ dataset (Ding et al., 2022)
0.6
0.4
DOT This dataset leverages Google Translate convert English utterances into the target lan- guage while maintaining the slot values associated with entities in English.
0.2
ID. AOCORACCY
SL-1
DET-IGA DSF-TVAN ACc.
NIC-BLRU
NEG-ROUGE
Figure I: A cross-language comparison for absolute equivalence in performance for each supervised system prot with respect to the fully supervised English system (.). The ID and SL models are based on XLM-RIarga while the DST and NLG models are based on mT551al1- The performance is reported based task- metric pairs, such as evaluating ID with Accuracy. The 0. 0.95-equivalence line requires a system to achieve 95% of the pino performance (see $3.2).
Table 1 presents the experimental results for the three scenarios. Despite employing state-of-the art mPLMs, we observe a substantial cross-lingual transfer gap (Hu et al., 2020) across all tasks, under- scoring the significance of in-domain in-language data for the development of ToD systems. More- over, in zero-shot setups, larger models exhibit better cross-lingual transfcrability for SL, ID, and DST. However, this advantage diminishes when full-sized training data becomes available.
In sum, our empirical results highlight the cru- cial role of acquiring high-quality in-domain in- language training data when developing ToD sys- tems for a new language.
For translate-train, we observe that the system trained using the MT-ed dataset DJOT underper- forms the fully supervised systems (.). Inter- estingly, the translate-train system performs worse than the English system (.), even in the zero- shot cross- lingual transfer setup, for ID and SL. 9
(RQ2) Intrinsic Bias in mPLMs. In RQ2, we quantitatively evaluate the observed performance disparity across languages by measuring the ab- solute f-cquivalence. Figure illustrates the per- formance ratio of each system in relation to the English systems, This figure also highlights the ab- solute 0.95-equivalence threshold, which signifies the point at which a system achieves 95% of the
IIsttad employing an MT system translate 1) ENO mt. we utilise the GlobalWOZ dataset our translate-traie dataset. This dataset fulfils our objective by offering translated dialogues using Google Translate and provides character-level spans for slot values. We acknowledge that the quality of D not are naturally influenced by the choice of MT system. "The poor performance of SI. models trained on Gloh- aIWOZ can be attributed to the presence of erroneous span
annotations within the GOOZ dataset, demonstrating the error-prone nature of MTT-ased multilingual ToD generation.

---

Arabic SL.
Aboolute (-Equivalene
1.2
TUR
Arabic SL serves as the bottleneck across the benchmark, with significant decrease in per- formance compared to other language. In the MULTI3 WoZ dataset, the slot-value spans an- notated the character level, and an exact match is required for a span to be considered correctly identified. On the other hand, Rust et al. (2021) observed that the tokenisers used in multilingual models may lead to sub-optimal performance for downstream tasks. To delve further into the analy- sis, aligned the slot boundaries with the token boundaries by defining the slot span the min- imal token span that covers the entire slot in the utterance. Using this approach, the identical model achieved an improved Fl score of 81.1 (135.9) for Arabic SI., confirming that the sub-optimal tokeni- sation of XLM-R was the primary factor contributing to the performance degradation.
1.0
0.95
0.8
0.6
0.4
0,2
0.0
NLU ATALAnCE
Figure 2: Absolute equivalence in performance dif- ferent mPLMs for DST (JGA) and NLG (BLEU).
performance of the fully supervised English system. For instance, the English ID system achieves an ac- curacy of 93.2. Thus, target system must attain at least 88.5 accuracy to achieve 0.95-equivalence.
is observed that the ID models in the target languages meet the 0.95-equivalence performance threshold. However, for the more complex tasks of SI., DST, and NLG, the models generally fall short of reaching the threshold, when evaluating the models using the widely accepted metrics (e.g., JGA). An exception is the Turkish NLG model, which surpasses the performance of the English model by achieving a higher BLEU score. It is important to note that the BLEU score is sensitive to the specific linguistic properties of the language being evaluated and may vary depending on the language under consideration.
In sum, despite having access to the same models and ToD data, ToD systems in largel languages have not yet reached performance levels on par with English and have failed our goal of absolute 0.95eqqivaleenee in performance. This limitation can primarily be attributed to the intrinsic bias inherent in state-of-the-ar mPLMs.
(RQ3) Adaptive Bias in Few-shot Learning. Sub- sequently, we investigate the impact of the searcity of in-domain, in-language training data on per- formance disparities, recognising it as a primary source of aforementioned adaptive bias. Here, our objective is to collect small-sized dataset D few in the target language that enables the trained system to achieve relative 0.95 0.95-qquivalence in performance (M (MPGTT (-)) > 0.95 (-)); see $3.2).
In addition, DST models for all languages achieve the 0.95eequvvalence performance thresh- old when the metric M(-) is Turn Accuracy. How- ever, when evaluating the models using JGA, the target languages fail to reach the threshold. Addi- tionally, we note that the Arabic NLG model out- performs the French but falls behind in terms of ROUGE score. These findings highlight the dependence of the equivalent performance on the choice of evaluation metrics and emphasise the importance of aligning evalua- tion metrics with users' judgement of the system.
in terms of BLEU score,
Figure 3(a) shows the relative equivalence, that is, the performance ratio for a system trained on increasing percentage of Arabic-only data (without cross-lingual transfer) compared fully supervised systems (-RA(-) trained on the complete ARA The subsets of Arabic-only data are randomly sam- pled from the full Arabic training dataset. As an- ticipated, we find that simpler tasks like ID and SL reach relative equivalence with less than 20% of the training dataset. Beyond this point, increasing the training data size leads only marginal perfor- mance gains. More complex tasks such as DST and NLG require more data, with relative equivalence being achieved approximately 50% and 80% of the training data, respectively. Similar trends are observed for other languages, including English,
The attainment of absolute equivalence in perfor- mance across languages can also differ when em- ploying different mPLMs. Figure 2 indicates that each mPLM can have varying impacts on down- stream applications. Morcover, it is worth noting that the performance of mPLMs still lags behind their monolingual counterparts. For instance, by substituting the multilingual model with the mono- lingual
model (Safaya et al., 2020), we achieved a Fl score of 54.9 (1 9.7) for

---

Relative B-Eigiivalnnc
1.0
1.0
0.8
0.8
0.6
0.6
0.4
0.4
0.2
ID AccunAc
De1.JCA NLG-BLBU
0.2
ATARI
ENG
0.0
110%
20% 30% 40% 50% 60% 70% 80% 90%100%
0.0
100% 20% 30% 40% 50% 60% 70% 80% 90% 100%
10%
(a) Different Tasks Arabic
(b) NLG on Different Languages
Figure Relative equivalence performance for (a) ID, SL, DST, and NLG for system trained on increasing percentage of randomly sampled Arabic-only training data with respect systems PARA(.) (.) trained with full data and (b) NLG for systems trained on increasing percentage of randomly sampled monolingual data, in relation to systems trained on the full dataset. The ID and SI. models based XaM-Riarge and evaluated using Accuracy and Fl scores. The DST and NLG models are based on mT5 and evaluated on JGA and BLEU scores.
shown in Figure (appendix). Interestingly, the English systems surpass the equivalence threshold faster compared to other target languages.
lection. This research question considers the real life challenge faced by developers who need create ToD systems in a new language without any in-domain, in-language data available. The ob- jective is to maximise system performance across multiple metrics while minimising annotation cost by annotating the minimum amount of data. Specif- ically, assume the availability of large-scale source dataset TDCCC Our objective is effectively leverage ID to create a target language dataset DD Der few that covers a subset of DGGr, aiming to train a system that outperforms other possible subset selections of the same size.
Figure 3(b) shows the relative equivalence in performance of NLG for systems trained on pro- gressively larger proportions of monolingual train- ing data across different languages. Notably, even when employing the same mPLM, English out- performs other target languages by achieving the 0.95-equivalence threshold with 40% the train- ing data. On the other hand, the remaining target languages require more than 80% of the training data to reach the threshold. Similar trends are ob- served for SL and DST, illustrated in Figure 5, with the exception of ID models which all achieve the threshold with less than 5% of the training data, Furthermore, we note the following auxiliary findings regarding the introduction of cross-lingual transfer training: 1) We observe that cross-lingual transfer learning results in obvious gains when the amount of target language training data is lim- ited (less than 30%). However, when evaluating systems based on the relative 0.95-equivalence threshold, this transfer learning strategy only leads to marginal or negligible improvements. 2) The mixed cross-lingual training strategy outperforms its sequential training counterpart, which is consis- tent with prior research (Schmidt et al., 2022).
For simplicity, we assume that the annotation costs for dialogues in all languages are identical. We compare different strategies for creating D fe from ID SRC and use to create multi-parallel tar- get datasct D Ter few thus simulating the described scenario. Specifically, we evaluate following strate- gies. (i) Random Sampling: This is a baseline strategy, where we randomly sample a subset of DSRC few and create the corresponding D DIGT in the tar- get language. This strategy has been consistently applied throughout our previous analysis. (ii) Max N-gram: This strategy is heuristically select dia- logues that maximise the trigram diversity of SRC The objective is to enhance the diversity and cov- erage of language patterns in Djer
few
few
few'
few" (iii) Equal Domain: In this strategy, we sample an equivalent number of dialogues for each domain. The goal is to ensure balanced representation of all domains in DD IGT (iv) Equal Slot: This strategy focuses on sampling a of dialogues while maintaining a balanced distribution of slot occurrences. (v) Max Length: This strategy involves selecting a list of dialogues with the longest length, i.e., the highest
In sum, the proportion of data required to achieve relative 0. 0.95-equivalence in performance varies depending on the complexity of the task. Sim- pler tasks can reach this threshold with less than 20% of the in-language data. However, more com- plex tasks such as DST and NLG often require larger proportion, typically exceeding 50%.
(RQ4) Cost-efficiency in Multilingual ToD Col-

---

ID
SL.
DST
NLG
Conclusion and Future Work
Strategy
Accuracy
We presented a systematic analysis of performance disparities in modern multilingual ToD systems, utilising proposed quantitative measures of abso- lute and relative equivalence in performance. By addressing four key research questions, our empiri- cal investigation revealed the presence of adapta- tion and intrinsic biases current ToD systems and provided insights on how to best carry out data collection and system development for new languages. This study opens up new avenues for future research, including mitigating biases, im- proving system robustness, and expanding ToD systems across languages and domains.
Random Sampling
87.9
65.2
20.7
10.4
Max N-gram
88.8
66.5
23.6
12.2
Equal Domain
87.2
65.3
21.1
10.1
Equal Slot
87.9
65.0
26.2
11.3
Max Length
88.3
66.4
26.7
11.5
Table 2: The average performance of the models for each task across three target languages: Arabic, French, and Turkish. Each model trained using 5% of the target language data, which was generated using each data creation strategy introduced in RQ4. The sum- marised results presented in this table are derived from Table 10, Table 12, and Table 13 in the Appendix. The ID and SL models are built upon XLM XLM-Riarge- Rarge. while the DST and NLG models are based on mno11
Limitations
The limitations of this work are primarily related to the assumptions made during our analysis. Firstly, in $3 we rely on established automatic evalua- tion metrics and assume a strong correlation be- tween these metrics and human judgement of sys- tem performance. While this assumption has been widely adopted in previous studies (Blasi et al., 2022: Khanuja et al., 2023, inter alia), it im- portant to acknowledge the ongoing challenge dialogue system evaluation: Recent research (Yeh et al., 2021; Mehri et al., 2022) has highlighted that automatic evaluation metrics demonstrate only moderate correlation with human judgement. This indicates the need for further development of di- alogue evaluation. On the other hand, it is worth noting that our proposed notions of f-equivalence are not restricted to any specific metric but are gen- erally applicable, including to further metrics.
number utterances.
Table 2 presents the averaged task performance for the three target languages, considering few- shot setup where each model is trained on 5% of the full dataset T) TOT The results indicate that the Max N-gram and Max Length strategies consis- tently outperform the baseline Random Sampling strategy across all tasks. Notably, the Max N-gram strategy achieves the highest performance for ID, SL, and NLG tasks. These findings emphasise the potential of strategic annotation budget allocation in multilingual dialogue collection, which can lead to enhanced model performance without incurring additional annotation costs.
Furthermore, we present the following auxiliary findings. 1) We found that there is a negligible per- formance gain when collecting a full-sized (100%) validation set compared to smaller (e.g.. 10%) val- idation set. 2) When developing ToD systems for observed
Another limitation is the potential misinterpre- tation of --equivalence as implying that pToT(.) is as 0-good as (.). It is important to note that achieving f-equivalence does not indicate that the system in the target language, prst (-), possesses '0 ppropoortioon of capability compared to the system in the source language. (.) The concept of equivalence in performance focuses on providing an indicator of the performance disparities across languages or within a target language, as well as aims to minimise these disparities. Similar to eval- uation metrics, an example, generation system achieving 20 BLEU score does not necessarily imply that it is twice good system obtaining 10 BLEU score. Our proposed measures should be interpreted cautiously, considering the nuanced nature of dialogue evaluation and the potential non-
multiple languages simultancously, that collecting distinct (mutually exclusive) sets of dialogues in each language, without overlapping dialogue patterns, led to marginal enhancements in NLU in a cross-lingual transfer setup. However, this observed pattern did not extend to NLG.
In sum, our findings show the benefits of strategic budget allocation in multilingual dialogue collec- tion, which can enhance model performance with- out additional costs. Particularly, the Max N-gram strategy, which involves creating a target dataset frw few from a subset SRc that maximises both tri- gram and lexical coverage, leads to improved per- formance compared to the random baseline.
TGT

---

linear relationship between metric scores and user experience (Blasi et al., 2022).
quently, the cumulative computational budget of these experiments exceeded 5,000 GPU hours on state-of-the-art GPUs. Considering the importance of conserving valuable computing resources and minimising energy consumption, we did not to run parallel experiments with different random seeds, However, we would like to emphasise that, despite based on a single run, all our main claims remain consistent across all languages and tasks, thereby affirming their reliability and generalisability.
Besides, in $5 (RQ4), we abstract away many fine-grained details in a real-word data collection project. For example, we assume that annotating a dialogue for all languages incurs uniform costs. While this assumption simplifies the analysis, may not accurately reflect the real-world scenario. The actual cost of annotating dialogues can vary across languages due to factors such as the avail- ability of native or bilingual annotators. Further- more, some strategies tested in our study, such as the Max Length strategy, may result in increased data collection time and subsequently higher costs. We hope that our findings could inspire future work to delve into more sophisticated annotation strate- gies and conduct more fine-grate mee-graaned analysis, con- sidering these finer details of real-world data collec- tion. Given the scope of this paper, we were unable to delve into the intricacies of cost variations in multilingual dialogue annotation.
Ethics Statement
The experimental study obtained full Ethics Ap- proval from the University of Cambridge ad- vance. No human participants or personal data were directly involved in this study. How- ever, our models leverage two data sources: the MUI.TI3 Woz dataset and the pre-training data of each mPLM employed in this study. Hu et al. (2023) highlights that the creation and publication of MULTI3WOZ comply with the EU General Data Protection Regulation (GDPR). Particularly, this dataset consists solely of hypothetical dialogues in which the domains and content have been restricted and predefined, minimising the risk of personal data being present. On the other hand, it is im- portant to acknowledge that although these PLMs are publicly available, there exists potential risk of privacy violations (Brown et al., 2022; Carlini et al., 2021).
The scope of languages analysed in this study is limited by the availability of training data. As mentioned earlier, the MULT13WOZ dataset pro- vides us with the opportunity to conduct this type of study for the first time. Therefore, our analysis is based on this single dataset, which includes four languages: Arabic, English, French, and Turkish. While these languages are representative of differ- ent language families and feature diverse range of linguistic properties, recognise that our findings are derived from small sample of world languages. Future research might (and should) contribute more resources, particularly for under-represented lan- guages, and expand the analysis to a broader lin- guistic landscape. Furthermore, a fully inclusive dialogue system should consider not only text input but also other modalities, such as spoken and sign languages. We also acknowledge that the limita- tions of training data have restricted the analysis text input only, thus discarding potential dispar- ities stemming from other relevant tools such as automatic speech recognition (Babu et al., 2022).
Acknowledgements
Songbo Hu is supported by the Cambridge Inter- national Scholarship. Han Zhou is supported by the UK Research and Innovation (UKRI) Fron- tier Research Grant EP/Y031350/1 (the UK gov- ernment's funding guarantee for ERC Advanced Grants) at the University of Cambridge. Anna Ko- rhonen acknowledges the support of the UK EP- SRC grant EP/T02450X/1 and the UKRI Frontier grant PPP/Y33135/11 Ivan Vulić acknowledges the support of personal Royal Society University Re- search Fellowship 'nncuuive and Sustainable Lan- guage Technology for a Truly Multilingual World' (no 22 137; 2022-). The team recognises support from Huawei Technologies.
Finally, we acknowledge that our experimental results are derived from single run, rather than multiple runs with varying random seeds. In this study, we have conducted comprehensive set of experiments including four different languages and spanning across range of tasks. These tasks in- clude simpler tasks such as ID and SL, as well as more complex tasks like DST and NLG. Conse-
References
Duygu Altinok. 2018. An ontology-based dialogue management system for banking and finance dialogue systems. CoRR, abs/ abs//88440444 804.04838.

---

Alan Ansell, Edoardo Maria Ponti, Jonas Pfeiffer, Se- bastian Ruder, Goran Glavaš, Ivan Vulié, and Anna Korhonen. 2021. MAD-G: Multilingual adapter gen- cration for efficient cross-lingual transfer. In Find- ings of the Association for Computational Linguis- fics: EMNLP 2021, pages 4762-475 17711 Punta Cana, Dominican Republic. Association for Computational Linguistics.
Pawet Budzianowski, Tsung-Hsiee Wen, Bo-Hsiang Tseng, lñigo Casanueva, Stefan Ultes, Osman Ra- madan, and Mílica Gašić. 2018, MultiWO7. a large- scale multi-domain Wizard-of-Oz dataset for task- oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5016-5026, Brussels, Belgium. Association for Computational Linguistics.
Akari Asai, Sneha Kudugunta, Xinyan Velocity Yu, Terra Blevins, Hila Gonen, Machel Reid, Yulia Tsvetkov, Sebastian Ruder, and Hannaneh Hajishirzi. 2023. Buffet: Benchmarking large language models for few-shot cross-lingual transfer. arXiv preprint arXiv:2305.1453 14857.
Nicholas Carlini, Florian Tramer, Eric Wallace. Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom B Brown, Dawn Song, UI- far Erlingsson, et al. 2021. Extracting training data from large language models. In USENIX Security Symposium, volume 6.
Inigo Casanueva, Ivan Vulić, Georgios Spithourakis. and Pawet Budzianowski. 2022. NLU++: A multi- label, slot-rich, generalisable dataset for natural lan- guage understanding in task-oriented dialogue. In Findings of the Association for Computational Lin- guisties: NAACL 2022, pages 1998-2015 Seattle, United States. Association for Computational Lin- guistics.
Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf. Juan Pino, Alexei Baevski, Alexis Conneau, and Michael Auli. 2022.
XLS5R: sepe-supe cross-lingual speech repre- sentation learning scale. In Interspeech 2022, 23rd Annual Conference of the International Speech Communication Association, Incheon, Korea, 18-22 September 2022, pages
ISCA.
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- ret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghami, Siddhartha Brahma, et al. 2022. Scaling instruction finetuned rnstuuctionfiieeuueed language models. arXiv preprint arXiv:22 1416.
Satanjeev Banerjee and Alon Lavie. 2005. METEOR:
An automatic metric for MT evaluation with im- proved correlation with human judgments. In Pro- ceedings of the ACL Workshop trinsic Evaluation Measures for Machine Transla- tion and/or Summarization, pages 65-72, Ann Arbor, Michigan. Association for Computational Linguis- tics.
Intrinsic and Ex-
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020a. Unsupervised cross-lingual representation learning at scale. In Pro- ceedings of the 58th Anmual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics.
Solon Barocas, Moritz Hardt, and Arvind Narayanan. 2019. Fairness and Machine Learning: Limitations and Opportunities. fairmlbook.org http://www. fairmlbook. org.
Damian Blasi, Antonios Anastasopoulos. and Gra- ham Neubig. 2022. Systematic inequalities in lan- guage technology performance across the world's languages. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5486-5505, Dublin, Ireland. Association for Computational Linguistics.
Alexis Conneau, Shijic Wu, Haoran Li, Luke Zettle- moyer, and Veselin Stoyanov. 2020b. Emerging cross-lingual structure in pretrained language mod- els. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6022 6034, Online. Association for Computational Linguisties.
Council of European Union. 2018. Language equality in the digital age. European Parliament resolution of 11 September 2018 on language equality in the digital age (2018/2028(NNI).
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx. Michael S Bernstein, Jeannette Bohg, Antoine Bosse- lut, Emma Brunskill, et al. 2021. On the opportuni- ties and risks of foundation models. arXiv preprint 108.2880...07258.
Mai Hoang Dao. Thinh Hung Truong, and Dat Quoc Nguyen. 2021. Intent detection and slot filling for vietnamese. In Interspeech 2021, 22nd Annual Con- ference of the International Speech Communication Association, Brno, Czechia, 30 August 3 September 2021, pages 4698-4702. ISCA.
Hannah Brown, Katherine Lee, Fatemehsadat Mireshghallah, Reza Shokri, and Florian Tramèr. 2022. What does it mean for language model to preserve privacy? In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, FAceT '22, page 2280-2292 New York, NY, USA. Association for Computing Machinery.
Arnab Dehnath, Navid Rajabi, Fardina Fathmiul Alam, and Antonios Anastasopoulos. 2021. Towards more equitable question answering systems: How much more data do you need? In Proceedings of the 59th

---

Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 2: Short Papers), pages 621-629, Online. Association for Computational Linguistics.
Matthew Henderson, Blaise Thomson, and Jason D. Williams. 2014. The third dialog state tracking chal- lenge. In 2014 IEFE Spoken Language Technology Workshop. SLT 2014, South Lake Tahoe, NV, USA, December 7-10, 2014, pages 324-329. IEEE
Jacob Devlin, Ming-Woi Wei Chang, Kenton Lee, and
Matthew Henderson, Ivan Vulić, Iñigo Casanueva, Pawel Budzianowski, Daniela Gerz. Sam Coope. Georgios Spithourakis, Tsung-Hsien Hsien Wen, Nikola Mrkšić, and Pei-Hao Su. 2019. Polyresponse: A rank-based approach to task-oriented dialogue with application in restaurant search and booking. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019 System Demon- strations, pages 181-186. Association for Computa- tional I Linguistics.
Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume 1 (Long and Short Pupers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Bosheng Ding, Junjie Hu, Lidong Bing, Mahami Alju- nied, Shafiq Joty. Luo Si, and Chunyan Miao. 2022. GlobalWoZ: Globalizing MultiWoZ to develop mul- tilingual task-oriented dialogue systems. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Pa- pers), pages 1639-1657, Dublin, Ireland. Association for Computational Linguistics.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Gra- ham Neubig, Orhan Firat, and Melvin Johnson. 2020. XTREME: A massively multilingual multi- task benehmark for evaluating cross-lingual gener- alisation. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 4411 4421. PMLR.
Layla El Asri, Hannes Schulz, Shikhar Sharma, Jeremie Zumer, Justin Harris, Emery Fine, Rahul Mchrotra, and Kaheer Suleman. 2017. Frames: a corpus for adding memory to goal-oriented dialogue systems. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 207-219, Saar- brücken, Germany. Association for Computational Linguistics.
Songbo Hu, Han Zhou, Mete Hergul, Milan Gritta, Guchun Zhang, Ignacio lacobacci, Ivan Vulic, and Anna Korhonen. 2023. Multi3woz: A multilingual, multi-domain, multi-parallel dataset for training and evaluating culturally adapted task oriented dialog sys- tems. CoRR, abs/2307. 14031.
Julia Rose Galliers and K Spärck Jones. 1993. Evaluat- ing natural language processing systems. Technical report, University of Cambridge, Computer Labora- tory.
Vojtëch Hudeček and Ondicj Dušck. 2023. Are llms all you need for task-oriented dialogue? arXiv preprint arXiv:2304.06556
Narendra K. Gupta, Gökhan Tür, Dilek Hakkani-Tite Srinivas Bangalore, Giuseppe Riccardi, and Mazin Gilbert. 2006. The at&t spoken language understand- ing system. IEEF Transactions on Speech and Audio Processing, 14(1):213-222.
Chia-CChin Hung, Anne Lauscher, Ivan Vulié, Simone Ponzetto, and Goran Glavaš. 2022. Multi2WOZ: A robust multilingual dataset and conversational pre- training for task-oriented dialog. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 3687-3703. Seattle, United States. Association for Computational Linguistics.
Ting Han, Ximing Liu, Ryuichi Takanobu, Yixin Lian, Chongxuan Huang. Dazhen Wan, Wei Peng, and Min- lie Huang. 2021. Multiwoz 2.3: A multi-domain task- oriented dialogue dataset enhanced with annotation corrections and co-reference annotation. In Natu- ral Language Processing and Chinese Computing- 10th CCF International Conference, NLPCC 2021, Qingdao, China, October 13-17, 2021, Proceedings. Part I1, volume 13029 of Lecture Notes in Computer Science, pages 206-218. 18. Springer.
Simran Khanuja. Sebastian Ruder, and Partha Talukdar. 2023. Evaluating the diversity, equity, and inclu- sion of NLP technology: A case study for Indian languages. In Findings of the Association for Compu- tational Linguistics: FACI 2023, pages 1763-1777, Dubrovnik, Croatia. Association for Computational Linguistics.
Michael Heck, Nurul Lubis, Benjamin Ruppik, Renato Vukovic, Shutong Feng. Christian Geishauser, Hsien- chin Lin, Carel van Nickerk, and Milica Gasic, 2023. ChatGPT for zero-shot dialogue state tracking: A solution or an opportunity? In Proceedings of the 61st Annual Meeting of the Association for Compu- tational Linguistics ( /Volume 2: Short Papers), pages 936-950, Toronto, Canada. Association for Compu- tational Linguistics.
Liliana Laramjo, Adam G. Dunn, Huong Ly Tong, Ah- met Baki Kocaballi, Jessica A. Chen, Rabia Bashir, Didi Surian, Blanca Gallego, Farah Magrabi, Annie Y. S. Lau, and Enrico W. Coiera. 2018 Conversa- tional agents in healthcare: a systematic review. Jour- nal of the American Medical Informatics Association. 25(9): 10:::2240 1-448-1555

---

Stefan Larson and Kevin Leach. 2022. survey of intent classification and slot-filling datasets for task- oriented dialog. CoRR, abs/2207.1322 13211.
Bapi Akula, Loie Barrault, Gabriel Mejia Gonza- lez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shan- non Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No language left behind: Scal- ing human-centered machine translation. CORR, abs/2207 046722.
Anne Lauscher, Vinit Ravishankar, Ivan Vulié, and Goran Glavaš. 2020. From zero to hero: On the limitations of zero-shot language transfer with mul- tilingual Transformers. In Proceedings of the 2020 Conference on Empirical Methods Natural Lan- guage Processing (EMNLP), pages 4483 0483,4499, On- line. Association for Computational Linguistics.
Chin-Yers Yew Lin. 2004. ROUGE: package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81. Barcelona, Spain. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 31 1-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Zhaojiang Lin, Bing Liu, Seungwhan Moon, Paul Crook, Zhenpeng Zhou, Zhiguang Wang, Zhou Yu. Andrea Madotto. Eunjoon Cho, and Rajen Subba. 2021a. Leveraging slot descriptions for zero-shot cross-domain dialogue State Trackir. In Proceed- ings of the 2021 Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5640-5648, Online. Association for Computational Linguistics.
Antoine Raux, Brian Langner, Dan Bohus, Alan W. Black, and Maxine Eskénazi. 2005. Let's go public! Taking a spoken dialog system to the real world. In INTERSPEECH 2005 Eurospeech, 9th European Conference on Speech Communication and Technol- ogy, Lishon, Porugaal, September 4-8, 2005, pages 885-888. ISCA.
Zhaojiang Lin, Andrea Madotto, Genta Indra Winata, Peng Xu, Feijun Jiang, Yuxiang Hu, Chen Shi, and Pascale Fung, 2021b. BiToD: A bilingual multi- domain dataset for task-oriented dialogue modeling. In Proceedings of the Neural Information Process- ing Systems Track on Datasets and Benchmarks 1, NeurlPS Datasets and Benchmarks 2021, December 2021, virtual.
Evgeniia Razumovskaia, Goran Glavaš. Olga Majew- ska, Edoardo Maria Ponti, Anna Korhonen, and Ivan Vulić. 2022. Crossing the conversational chasm: A primer on natural language processing for multilin- gual task-oriented dialogue systems. Journal of Arti- ficial Intelligence Research, 74:1351-1402.
Sebastian Ruder, Jonathan H. Clark, Alexander Gutkin, Mihir Kale, Min Ma. Massimo Nicosia, Shruti Rijh- wami, Parker Riley, Jean Michel A. Sarr, Xinyi Wang, John Wieting, Nitish Gupta, Anna Katanova, Christo Kirov, Dana L. Dickinson, Brian Roark, Bidisha Samanta, Connie Tao, David I. Adelani, Vera Ax- elrod, Isaac Caswell, Colin Cherry, Dan Garrette, R. Reeve Ingle, Melvin Johnson, Dmitry Panteleev. and Partha Talukdar, 2023. XTREME-UP: user- centric scarce-date benchmark for under-represented languages. CoRR, abs/2305. 11938.
Olga Majewska, Evgeniia Razumovskaia, Edoardo Maria Ponti, Ivan Vulić, and Anna Korbonen. 2023. Cross-lingual dialogue dataset creation via outline-based generation. Transactions of the Association for Computational Linguistics, 11:139-156.
Shikib Mchri, Jinho Choi, Luis Fernando D'Haro, Jan Deriu, Maxine Eskenazi, Milica Gasic, Kallirroi Georgila, Dilek Hakkani-Tur, Zekang Li, Verena Rieser, Samira Shaikh, David Traum, Yi-Ting Yeh, Zhou Yu. Yizhe Zhang, and Chen Zhang. 2022. Re- port from the nsf future directions workshop on auto- matic evaluation of dialog: Research directions and challenges.
Phillip Rust, Jonas Pfeiffer, Ivan Vulié, Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tok- enizer? on the monolingual performance of multilin- gual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the 1 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 3118-3135, Online. Association for Computational Linguistics.
Nikita Moghe, Evgeniia Razumovskaia, Liane Guil- Jou, Ivan Vulic. Anna Korhonen, and Alexandra Birch. 2022. MULTINNLU+:: A multilingual, multi- intent, multi-domain dataset for natural language understanding in task-oriented dialogue. CoRR, abs/2212.10455.
Ali Safaya, Moutasem Abdullatif, and Deniz Yuret. 2020. KUISAIL at SemEval-2020 task 12: BERT- CNN for offensive speech identification in social me- dia. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 2054-2059, Barcelona (online). International Committee for Computational Linguistics.
Meta AI NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Jan- ice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood,

---

Marina Sanchez-Tomee and Philipp Koehn. 2016. Ma- chine translation quality and post-editor productiv- ity. In Conferences of the Association for Machine Translation in the Americas: MT Researchers" Track, pages 16-26, Austin, TX, USA. The Association for Machine Translation in the Americas.
Speech and Signal Processing, ICASSP 2018. Cal- gary, AB, Canada, April 15-20, 2018, pages 6034- 6038. IEEE.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer. Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-ar natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 45, Online, Association for Computational Linguistics.
Fabian David Schmidt, Ivan Vulié, and Goran Glavaš. 2022. Don't 't stop fine-tuning: On training regimes for few-shot cross-lingual transfer with multilingual language models. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 10725-10742, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Sehastian Schuster, Sonal Gupta, Rushin Shah, and Mike Lewis. 2019. Cross-lingual transfer learning for multilingual task oriented dialog. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Pupers), pages 3795-3805, Minneapolis, Min- nesota. Association for Computational Linguistics.
Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini-Asi. Caiming Xiong, Richard Socher, and Pascale Fung- 2019. Transfcrable multi-domain state generator for task- soisetrrented dialogue systems. In Proceedings of the 57th Annual Meeting of the Association for Compu- rational Linguistics, pages 808-819, Florence, Italy. Association for Computational Linguistics.
Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas:
Pararth Shah, Dilek Hakkani- Tür,, Bing Liu, and Gökhan Tür. 2018. Bootstrapping a neural conversational agent with dialogue self-play, crowdsourcing and Proceedings of the 2018 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Vol- ume 3 (Industry Papers), pages 41-51, Association for Computational Linguistics.
The surprising cross- pooss-lingual effectiveness of BERT, In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing ( (EMNLP-IJCNLL pages 533-844, Hong Kong, China. Association for Computational Linguis- tics.
on-line reinforcement learning.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami A1-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 202 m55: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 483 983-498. On- line. Association for Computational Linguistics.
Aditya Siddhant, Ankur Bapna, Orhan Firat, Yuan Cao, Mia Xu Chen, Isaac Caswell, and Xavier Garcia. 2022. Towards the next 1000 languages in mul- tilingual machine translation: Exploring the syn- ergy between supervised and self-super learning. CoRR, Abs/2201.031111
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale. Rami AI-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021b. mT5: A massively multilingual mre-trineed text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 483-498, On- line. Association for Computational Linguistics.
Anders Segaard. 2022. Should we ban English NLP for a year? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. pages 5254-5260, Abu Dhabi, United Arah Emirates. Association for Computational Linguistics.
Gökhan Tür, Andreas Stolcke, L. Lynn Voss, Stan- ley Peters, Dilek Hakkani-Tür, John Dowding, Benoît Favre, Raquel Fernández, Matthew Frampton. Michael W. Frandsen, Clint Frederickson. Martin Graciarena, Donald Kintzing, Kyle Leveque, Shane Mason, John Niekrasz, Matthew Purver, Korbinian Riedhammer, Elizabeth Shriberg, Jing Tien, Dimitra Vergyri, and Fan Yang. 2010. The CAL.O meeting as- sistant system. /EEE Transactions on Speech Audio Processing, 1866)1601-1611
Zhao Yan, Nan Duan, Peng Chen, Ming Zhou, Jianshe Zhou, and Zhoujun Li. 2017. Building task-orienter dialogue systems for online shopping. In Proceed- ings of the Thirry-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, Cal- iflornu,, USA, pages 4618 4626. AAAI Press.
Yi-Ting Yeh, Maxine Eskenazi, and Shikib Mehri. 2021. A comprehensive assessment of dialog evaluation metrics. In The First Workshop on Evaluations and Assessments of Neural Conversation Systems, pages 15-33, Online. Association for Computational Lin- guistics.
Shyam Upadhyay, Manaal Faruqui, Gökhan Tür, Dilek Hakkani-Tinn and Larry P. Heck. 2018 (almost) zero- shot cross-lingual spoken language understanding. In 2018 IEEE International Conference on Acoustics,

---

Steve Young. 2010. Cognitive user interfaces, IEEE Signal Processing Magazine, 27(3):128-144
A Experimental Details
Experiments on ID. Our ID model is implemented as a multi-class classifier. At each dialogue turn t, an mPLM model encodes the concatenation of the previous two historical utterances (u,-2 and up-1) along with the current utterance (u,) to pro- vide embedding vectors the sequence. For each domain-intent pair d-i defined by the ontology, the representation of the *
' token is subsequently projected down to two logits and passed through a softmax layer to form Bernoulli distribution indicating d-i appears in the u₁. Unless oth- erwise specified, all of our models, including ID models, in this paper are implemented using the HuggingFace repository (Wolf et al., 2020). All ID experiments were rum single A100 80 GiB GPU and 32-core vCPU.
Lei Zuo, Kun Qian, Bowen Yang, and Zhou Yu. 2021. AIIWOZ Towards multilingual task-oriented dialog systems for all. CoRR, abs/2112.08333.
Table 3 presents the selected hyper-parameter parameters for the conducted experimental study, including the ID models.
Table lists all the PLMs we used in this work, along with their respective checkpoints in the Hug- gingface repository.
Table 5 shows the time consumption of the mod- els for all tasks in the experimental study, in- cluding the ID models, The time consumption is measured based on five independent runs on the MULTI3WoZ dataset.
SL. Our SL model is implemented as a sequence tagger. Precisely, at each dialog turn f, the model encodes the concatenation of the previous two ut- terances (u,-2 and u,-1) along with the current utterance (u,) to provide embedding vectors ken levels. We adopt the widely-used BIO labelling scheme to annotate cach token in the user's utter- ance. Specifically, each token labelled with either B-d-i-s (e.g., S--RestaurnntInforre Food), denot- ing the beginning of a slot-value pair with the cor- responding slot name, I-d-i-s indicating it is inside the slot-value, O indicating that the token is not associated with any slot-value pair. The predic- tion is performed through token classification. We implemented our SL model using the Hugging- Face repository and conducted the experiments on a single A 100 80 GiB GPU and 32-core vCPU. The selected the time consumption of our SL. experiments are
mPLMs used, and
"https://hugginge co/tasks/ token ciseen-llasifiicatino

---

Hyper-paramettr
Value
Model
HuggingFace Checkpoint
ID: XLM Russ, XLM-Rtarg Riarge
XLM-Roase
xim-roberta-base
batch size
128
Rarge
xlm-roberta-larg large
lcarning rate
2c-5
weight decay
0.1
5m5
google/mt5-sman
evaluation per steps max training steps max context char length
500
mT 1aarg
googlc/mt5-
20000
Flan-T55mal1
google/fan-t5-small
150
early stopping patience
2
Flan-T51arge
google/ftan-t5large
Arabic-BERT
auafayalbert-lage-aaabb
SI.: Raaee XLM-R
batch size
128
Table 4: The employed mPLMs in experimental
learning
2c-5
weight decay
0.1
study and their Huggingface Checkpoints.
evaluation per steps
500
max training steps max context char length
20000
early stopping patience
Sctup
Time Consumption
DST: mT5 Flan Flan-T5peal
ID: XLM-Roase
batch size learning rate
8
Training per steps Inference on full test
3:26
le-5
0:06
max training epochs
5
ID): XLM-Riarge
early stopping patience
2
Training per 500 steps Inference on fill test
10:29
DST: iaSga Flanne
0:20
batch size
SL: XLM-Roase
learning rate
le.5
number of training epochs
Training per 500 steps Inference full test
3:21
0:45
NLG: mT Flan-T5
SL: XLM- Riarge
batch size
32
learning rate
Te-3
Training per 500 steps Inference on full test
10:36
weight decay evaluation per steps max training steps max context char length
0.01
1:24
2000
DST: mT
40000
200
Training per epoch Inference full test
3:56 6:53
early stopping patience
2
NLG: mT Flan-T5targe
DST: m
batch size
Training per epoch" Inference on full test"
30:54:23
learning
Ic-3
17:17
weight decay
0.01
NLG: mT5
evaluation per steps
2000
max training steps max context char length
40000
Training per 2000 steps Inference on full test
7:12 0:56
200
early stopping patience
2
NI.G: mT5
Table 3: Model hyper-parameters for each task-model pair. For example, ID: XLM-Roase. XLM-Riarge section in this table shows the hyper-parameters for ID models boned with XLM-Rpasa and XLM-Riarge- To select the optimal model checkpoint, we employ early stop- ping and select the one with the best validation perfor- mance, measured by FI, FI, validation loss, and BLEU for ID. SL, DST, and NLG, respectively. Unless explic- itly specified, all other hyper-paramette are set to their default values as defined the HuggingFace Transform- T5DST repository.
Training per 2000 steps Inference on full test
37:02
3:22
Table 5: Time consumption of models for each dialogue task. It computed
average of 5 runs machine with a single A 100 80 GiB GPU and 32-core vCPU. (+) For the DST experiments utilising mT5 models, we conducted the experiments equipped with two A 100 80 GiB GPUs. We report this run time based on single run account for its significant time consumption.
machine
summarised in Table 3, Table 4, and Table 5, re- spectively.
model design and training methodology, we refer readers to the original work by Lin et al. (2021a), Our implementation based on the official T5DST
DST. For detailed information about the DST

---

GitHub Repository." 11 For the DST experiments utilising smal models, we run them on a single A100 80 GiB GPU and 32-core vCPU. Due to the lack of support in the official code for evaluating and saving models per training steps, we adopt a specific training protocol. In the fully supervised setup, we train DST models for fixed number of 5 epochs. On the other hand, in the few-shot setup, we train the models for approximately the same number of epochs that corresponds to a similar number of training steps as epochs. For instance, if we train models with 10% of the full training data, we train them for a maximum of 50 epochs. For the DST experiments utilising m151arge mod- els, we conducted the experiments on machine equipped with two A100 80 GiB GPUs. However, due to the significant time consumption, the models were only trained for 1 epoch. We report the se- lected Hlper-parametten mPLMs used, and the time consumption of our DST experiments in Table 3, Table 4, and Table 5, respectively.
stated in $5, as also supported by Figure 3(b). They demonstrate that our claim extends beyond single NLG task and are applicable to other tasks as well.
Figure 6 demonstrates the benefits of incorporat- ing cross-lingual transfer training in dialogue tasks. We observe that the coossllingual transfer learning from I yields substantial improvements when the amount of target language training data is lim- ited (less than 30%). However, when assessing sys- tems based on the relative 0.95-equivaeence thresh- old, this transfer learning strategy only results in marginal or negligible improvements. While sup- port our supporting for our auxiliary finding 1), these results once again emphasise the significance of in-domain, in-language training data. They also present an open research challenge to explore the optimal utilisation of cross-lingual transfer learning in conjunction with a larger quantity of in-language training data.
Figure 7 compares the benefits of incorporating two types of cross-lingual transfer training strate- gies in dialogue tasks. In the figure, the term 10% of training data denotes the mixed training strat- egy using the complete DENG dataset combined with 10% of the ARA dataset in the same training epoch. This mixed training strategy outperforms its sequential training counterpart, where the model is initialised based on pENG for ID and NLG tasks. In addition, Figure 8 confirms that findings holds across languages. These findings align with pre- vious results (Schmidt et al., 2022). These results provide support for our auxiliary finding 2).
NLG. The NLG model is implemented using the HuggingFace repository. All NLG experiments were run on a single 1100 80 GiB GPU and a 32- core vCPU. We report the selected hyper-parameter, mPLMs used, and the time consumption of our DST experiments in Table 3, Table 4, and Table 5, respectively.
B Additional Results on MULTI3WOZ
To solidify the empirical findings of our paper, we present supplementary experimental results that provide additional support for our main claims, as well as auxiliary findings discussed in $5.
Table 6, Table 7, Table 8, and Table 9 present the performance of ID, SL, DST, and NLG tasks across different languages: Arabic, English, French, and Turkish, respectively. These tables specifically demonstrate the model's performance a few-shot setup, where the proportion of available training data increases progressively. The discussion and analyses regarding the research question are de- rived from the performance presented in these ta- bles.
(RQ3) Adaptive Bias in Few-shot Learning.
Figure 4 visually presents the amount of in- language training data necessary to attain relative 0.95-qquivalence in performance for four tasks: ID, SL, DST, and NLG across three languages: En- glish, French, and Turkish. These results validate our claim stated in $5, as also supported by Fig- ure 3(a). They demonstrate that our claim extends beyond the Arabic language and are applicable to other languages as well.
(RQ4) Cost-efficiency in Multilingual ToD Col- lection.
Figure 5 shows the amount of in-language training data necessary to attain relative -95-cquivalenne in performance for each languages across three tasks: ID, SL, and DST. These results validate our claim
Figure 9 illustrates the comparative performance of SL models trained on different proportions of data in Arabic, French, and Turkish, using data created with two strategies: Random Sampling and Max N-gram.
I1 /https://gitthhe wtgggttbb.cmmaaaebookreeeerrrrd -Sot-SSo

---

ID
SL
DST
NLG
% of Training Data
Accurscy
Precision
Recall
Turn Acc.
FI
ROUGE
METEOR
Arabic
1%
84.7
89.8
33.4
37.5
35.3
84.9
46.9
5.5
3.6
16.5
2%
85.9
90.6
34.0
40.4
36.9
8.7
89.3
62.7
7.5
3.9
20.4
3%
88.9
92.4
38.0
42.8
40.2
11.0
90.8
67.0
8.6
7.8
22.1
4%
89.1
92.7
38.1
43.7
40.7
13.9
91.7
69.9
9.9
6.2
24,4
5%
89.5
92.8
38.0
44.2
40.9
14.1
91.9
70.3
9.5
5.9
23.8
6%
88.3
92.0
38.1
44.5
26.1
94.3
78.9
25.2
93.0
45.2
27.2
94.4
80.2
7,4
25.4
8%
89.6
93.0
40.5
44.8
42.5
28.3
94.6
80.5
11.8
9.2
27.4
9%
89.6
92.7
39.2
45.2
42.0
29.2
94.9
81.3
12.1
8.1
27.9
10%
89.2
92.8
39.8
45.0
42.2
32.0
95.2
82.5
12.4
8,4
28.1
20%
91.0
93.9
40.7
45.9
43.1
36.9
95.8
95.8
14.0
10.3
30.3
30%
91.4
94.1
46.4
96.3
87.2
31.6
40%
91.5
94.3
47.2
44.0
96.5
87.8
15.7
12.7
32,8
50%
92.3
94.7
41.5
47.4
44.3
45.6
96.6
88.2
16.0
13.4
33.4
60%
92.4
94.7
41.8
48.2
44.8
46.0
96.7
88.5
16.4
13.6
33.9
70%
92.6
95.0
41.9
47.6
44.6
46.8
96.8
89.0
16.9
13.8
34.7
80%
92.5
94.9
42.0
48.1
44.9
46.5
96.7
88.9
17.4
14.7
35.3
90%
94.9
48
45.8
47.7
96.9
89.4
15.4
35.4
100%
92.7
95.0
42.4
48.5
45.2
47.9
96.9
89.4
17.9
15.0
36.0
Table 6: The performance of ID, SL, DST, and NLU models trained on an increasing percentage of Arabic data. The ID and SL models are built upon XLM-Riarge> and the DST and NLG models based mT mmal1
DST
NLG
% of Training Data
Accuracy
Fl
Precision
Recull
Fl
JGA
Turn Acc.
FI
BLEU
ROUGE
METEOR
English
1%
81.6
88.3
86.3
88.3
87.3
17.0
91.1
71.5
8.4
29.7
26.9
2%
85.4
91.2
89.2
91.6
90.4
24.5
93.5
78.0
10.9
34.7
31.5
3%
85.7
89.5
93.2
30.5
94.6
80.9
36.3
33.0
4%
91.3
93.9
95.0
82.5
38.2
34.7
5%
88.8
93.4
91.2
94.0
92.6
39.8
96.1
86.1
14.8
41.2
37.5
6%
90.0
94.0
92.3
93.5
92.9
35.8
95.7
84.8
15.0
41.9
37.9
7%
90.0
94.0
91.9
93.9
92.9
40.4
96.2
86.7
14.9
40.9
36,4
8%
89.5
93.9
91.8
94.7
93.2
44.3
96.5
88.2
17.3
44.1
39,9
9%
88.7
93.5
92.1
95.3
96.4
87.8
15.7
38.0
10%
91.0
94.9
92.1
95.1
93.5
43.2
96.5
87.5
17.0
43.7
39.7
20%
91.1
94.9
93.1
95.2
94.1
47.0
96.8
89.5
18.7
46.2
42.3
30%
91.3
95.0
93.3
95.1
94.5
54.2
97.4
91.8
19.5
46.5
42.5
40%
92.4
95.7
93.6
94.8
94.2
55.2
97.5
92.1
20.3
47.4
43.1
50%
92.7
95.8
97.7
92.8
20.4
47.3
43.0
92.4
95.6
95.7
56.6
97.7
92.9
20.2
47.4
43.6
70%
92.8
95.9
94.2
95.8
95.0
58.4
97.8
93.2
20.5
47.6
43.6
80%
93.0
96.0
94.6
95.9
95.2
58.7
97.8
93.5
20.5
47.7
43.2
90%
92.5
95.7
95.0
95.5
95.3
59.3
97.8
93.6
20.8
48.1
43,9
100%
93.2
96.1
94.6
95.7
95.1
59.8
97.9
93.5
20.8
48.4
44.1
Table The performance of ID, SL., DST, and NLU models trained increasing percentage of English data. The ID and SL models are built upon XLM-R and the DST and NLG models are based on mT5

---

ID Accurscy
SL
DST
NLG ROUGE
% of
Training Data
Precision
Recall
JGA
Turn Acc.
FI
METEOR
French
1%
79.5
86.5
64.4
64.8
64.6
3.0
83.9
46.3
4.2
21.4
16.9
2%
80.6
87.3
70.1
71.1
70.6
9.8
89.8
63.9
6.3
27.1
22.2
3%
85.8
90.2
72.4
71.9
72.1
14.9
91.9
70.0
7.3
29.7
24.6
4%
85.9
90.7
72.0
72.4
72.2
23.9
93.8
78.4
8.3
31.6
26.2
5%
85.7
90.6
71.9
72.4
72.1
20.7
93.3
76.0
8.1
30.5
25.7
6%
85.8
90.7
72.4
74.1
25.4
94.3
79.6
7.9
30.6
25.3
90.6
73.4
75.2
74.3
30.4
94.8
81.3
9.4
33.3
27.8
8%
86.5
91.0
75.0
74.4
74.7
27.1
94.5
79.6
9.6
34.1
28.7
9%
86.0
90.8
73.3
74.1
73.7
33.0
95.3
83.3
9.1
33.6
28.1
10%
86.3
90.9
74.0
72.9
73.5
31.4
95.1
82.5
9.2
33,5
28.0
20%
87.2
91.7
75.5
77.3
76.4
38.2
95.9
85.9
11.1
36,9
31.3
30%
88.2
92.2
76.3
78.0
87.8
37.8
32.2
40%
89.0
92.8
77.1
77.0
77.1
44.7
96.6
88.4
12.2
37.6
32.2
50%
89.0
92.8
78.1
78.3
78.2
44.4
96.6
88.5
12.5
38.5
33.0
60%
88.5
92.6
76.3
78.6
77.4
46.4
96.7
88.9
12.9
39.1
33.4
70%
89.0
92.9
78.6
78.3
78.5
47.8
96,9
89.5
13.3
39.5
34.0
80%
89.2
93.0
77.8
78.8
78.3
46.4
96.8
89.3
13.4
39.9
34.4
90%
89.2
92.9
78.4
78.4
97.0
89.9
13.7
40.4
34.9
100%
89.2
93.0
76.9
79.2
78.0
49.7
97.0
90.1
13.9
40.9
35.2
Table 8: The performance of ID. SL, DST, and NLU models trained on an increasing percentage of French data. The ID and SL models are built upon XLM-Riarge> and the DST and NLG G models based mT mmal1-
DST
NLG ROUGE
% of Training Data
Accuracy
Fl
Precision
Recull
Fl
JGA
Turn Acc.
FI
BLEU
METEOR
Turkish
1%
79.9
86.8
74.8
76.5
75.7
3.6
85.7
48.1
6.9
27.1
24.5
2%
84.9
90.2
81.1
82.0
81.6
11.7
90.4
68.4
10.2
32.9
29.6
3%
86.8
81.0
82.2
81.6
20.2
92.8
76.2
10.8
34.8
2.30.9
91.7
83.7
27.3
94.5
81.0
38.5
34.4
5%
88.4
92.4
81.8
83.4
82.6
27.3
94.4
81.2
13.6
39.1
34.6
6%
88.7
92.9
82.9
83.7
83.3
32.9
95.3
83.5
14.9
40.9
36.7
7%
88.1
92.1
82.9
84.2
83.5
33.1
95.2
83.6
15.6
42.1
37.6
8%
88.7
92.7
83.7
85.0
84.3
32.1
95.2
84.1
16.3
43.0
38.6
9%
88.7
83.9
85.0
34.8
95.5
84.6
43.8
39.1
10%
88.3
92.2
84.6
84.9
84.7
35.9
95.6
85.1
17.2
44.6
39.9
20%
89.6
93.2
84.5
85.8
85,1
44.1
96.5
87.9
19.3
47.4
42.5
30%
90.7
94.1
84.3
84.6
84.4
46.3
96.7
89.0
20.3
48.9
43,9
40%
91.5
94.5
85.2
86.0
85.6
48.9
96.8
89.6
21.3
49,9
44,9
50%
91.4
94.5
87.1
85.6
48.2
96.9
89.7
21.8
50.6
45.7
91.8
94.7
86.3
50.1
97.1
90.1
22.7
51.9
47.0
70%
91.7
94.6
85.7
87.1
86.4
50.8
97.1
90.5
23.4
52.5
47.4
80%
91.6
94.7
86.8
87.1
87.0
51.3
97.1
90.5
23.7
53.2
48,1
90%
92.0
94.9
86.6
86.8
86.7
52.1
98.1
90.5
23.7
53.2
48.2
100%
92.2
95.0
86.6
87.6
87.1
52.9
97.3
91.2
24.2
53.7
48.6
Table The performance of ID, SL., DST, and NLU models trained increasing percentage of Turkish data. The ID and SL models are built upon XLM-R and the DST and NLG models are based on mT5 5

---

Relative é-Equivalence
1.1
1.0
0.8
1%
2%
3%
4%
5%
6%
7%
8%
9%
10%
(a)
Relative S-ative--Equie Equivalence
1.1
1.0
Relative A-Euivaleece
1.0
0.8
0.6
0.8
Teatc
04
1%
2%
3%
4%
5%
6%
7%
8%
9%
10%
02
(b)
0.0
10%
20%
40%
50%
60%
80%
100%
Relative d-Equivalence
1.0
English
0.8
Relative A-quivallne
0.6
044
0.6
02
0.4
0.0
10%
20%
40% 50% 80% 100%
0.2
(c) DST
0.0
10%
20% 30%
40%
50%
60% 70% 80% 90% 100%
Figure 5: The relative equivalence in performance of different tasks: (a) ID, (b) SL, and (c) DST, is evalu- ated for systems trained an increasing percentage of monolingual training data, in relation to fully supervised systems trained on the full monolingual dataset. The and SL models are based XLM XLM-Riarge. while the NLG models are based on mT gmall-
(b) French
Relative S-Equiivalenee
1.0
0.8
0.6
0.4
0.2
Di BGA
NUG-ELEU
0.0
10%
20% 30% 40% 50% 60% 70% 80% 100%
(c) Turkish
Figure 4: Relative equivalence in performance for ID, SL, DST, and NLG for system trained increasing percentage of (a) English-only, (b) French-only, and (c)
Relative A-Equivalenct
1.0
Turkish-only training data with respect to fully super- vised systems, namely (IN)(.), poRA(.), and pTuR(.), (-), trained on full sized monolingual in-language data. The and SL models are based on XLM-R Margge while the DST and NLG models are based on mT5
0.8-
0.6
0.4
51 SITERISTTD
TRARRRETER NUG TRANSTE
10% 20% 30% 40% 50% 60% 70% 80% 90%
Figure 6: The comparative performance of Arabic SL and NLG systems evaluated by training them on in- creasing percentage of Arabic-only training data, with (transfer) and without (supervised) cross-lingual trans- training from ENNG The evaluation is conducted with respect to fully supervised systems PARA(. (-) trained on the complete DARA dataset.

---

Figure 10 illustrates the comparative performance of SL and DST models trained on different propor- tions of Arabic data, using data created with two strategies: Random Sampling and Max N-gram.
Relative (1 O-Bquivaleng
1.0
0.6
Table 10. Table 11, Table 12, and Table 13 dis- play the relative performance of all tasks in Arabic, English, French, and Turkish, respectively, when considering the aforementioned strategies. These results support for our main claim $5.
0.2
1D ID-SCCURNTTAL NEG SroUThTI
NEG Mroon
1%
37
6%
8%
9%
10%
Figure The performance of Arabic SL and NLG systems is compared by training them on increasing percentages Arabic-only training data and full IDNNNG Two loosslliggual transfer training strategies, namely sequential and mixed, are emplayed and compared. The evaluation is conducted by comparing the systems' per- formance to that of fully supervised systems pARA() trained on the complete Arabic training dataset ID ARA ID is evaluated using Accuracy, while NLG is evaluated using BLEU, The ID model is based on XLM Resse, and the NL.G model is based on m mnall
Figure 11 illustrates a compression between two strategies for creating the validation set: propor- tional and full. In the few-shot setup, the propor- tional strategy constructs a validation that is proportionate to the training set (compared to the full training set). For example, we train a model using 10% the original training set and an equiv- alent 10% of the original validation set. Conversely, the full strategy involves consistently training the system with complete validation set, such as 10% of the training set and the entire validation set. The experimental results demonstrate that the inclusion of a full validation set yields only marginal neg- ligible improvements system performance. Con- sequently, is recommended that future system de- velopers allocate their efforts towards the collection of high-quality in-language datasets, as opposed to focusing on acquiring full validation set, which has traditionally been done in existing literature. These results support our auxiliary finding 1) in $5.
Relative &-Equivalensy
1.1
1.0
0.9
0.8
NEPESONT Micm TRANPPR
0.7
1%
2%
3%
4%
5%
6%
7%
8%
9%
10%
(a) Arabic
Relative ( E-Eqivallenee
1.1
1.0
Figure 12 illustrates a comparison between two strategies for creating the training set in hypothet- ical setup aimed at developing dialogue systems in multiple languages simultaneously. The Same strategy involves training system using the same set of dialogue patterns in all four languages, utilis- ing multi-parallel dataset. On the other hand, the Different strategy employs different training sets with mutually exclusive dialogue patterns for train- ing the system. For instance, a system trained with 20% of the training data using the Different strat- egy will encounter 80% of the dialogue patterns. The results demonstrate that the Different strate- gies outperform the Same strategies in terms of the NLU task. Specifically, covering diverse dialogue patterns yields marginally better performance for NLU. However, this pattern does not hold true for NLG. These results support our auxiliary finding 2) in $5.
0.9
0.8
SUTERORDD
1%
2%
3%
4%
5%
6%
7%
8%
9%
10%
(b) French
Relative Relative&--guiiee D-Equivalens
1.0
0.9
0.8
0.7
1%
2%
3%
4%
5%
6%
7%
8%
9%
10%
(c) Turkish
Figure& 8: Relative equivalence in performance for ID for system trained on increasing percentage of in-language data for (a) Arabic, (b) French, and (c) Turkish, with respect to fully supervised systems, namely PENG (.). (-), and piuR trained on full sized monolingual in- in-lanuaaee data. Two cross- cons-ll transfer training strategies, namely sequential and mixed, are employed and compared. The models are based on XLM-Raus and evaluated with Accuracy.

---

ID
SL
DST
NLG
% of
Training Data
Accuracy
Precision
Recall
JGA
Turn
FI
BLEU
ROUGE
METEOR
Random Sampling
1%
84.7
89.8
33.4
37.5
35.3
84.9
46.9
5.5
3.6
16.5
44.2
14.1
91.9
70.3
23.8
10%
89.2
92.8
39.8
45.0
42.3
32.0
95.2
82.5
12.4
28.1
50%
92.4
94.7
41.5
47.4
44.3
45.6
96.6
88.2
16.0
13.4
33.4
100%
92.7
95.0
42.4
48.5
45.2
47.9
96.9
89.4
17.9
15.0
36.0
N-gram
1%
86.5
91.0
33.5
38.1
35,6
6,6
87.7
58.0
6.7
5.0
18.8
5%
90.4
93.6
39.9
45.8
42.6
23.9
94.1
78.8
10.5
10.0
24.7
10%
94.0
40.4
46.4
33.8
95.4
83.8
11.9
11.9
27.7
43.1
96.7
88.7
14.1
34.4
100%
92.7
95.0
42.4
48.5
45.2
47.9
96.9
89.4
17.9
15.0
36.0
Equal Domain
1%
90.3
32.8
36.7
34.7
2.9
82.7
40.5
4.9
17.3
5%
89.0
92.5
37.0
43.3
39.9
16.3
92.6
73.4
9.1
7.1
24.9
10%
89.9
93.1
39.2
44.5
41.7
26.4
94.5
80.1
11.5
7.1
27.7
50%
92.1
94.6
41.5
47.2
44.2
44.9
96.6
88.4
16.0
13.3
35.2
100%
92.7
95.0
42.4
48.5
47.9
96.9
89.4
15.0
36.0
Equal Slot
19
85.1
89.6
34.1
38.9
89.7
62.3
6.0
17.4
88.6
93.9
78.1
24.7
10%
90.0
93.3
36.0
43.5
31.6
95.2
82.5
12.3
10.4
27.2
50%
92.4
94.9
41.3
45.9
43.6
45.2
96.6
88.1
16.2
13.6
33.3
100%
92.7
95.0
42.4
48.5
45.2
47.9
96.9
89.4
17.9
15.0
36,0
Max Length
1%
86.9
91.2
32.9
37.3
35.0
6.3
88.0
57.1
5.8
4.7
17.3
5%
89.6
93.2
38.7
43.8
41.1
23.7
94.1
79.4
10.2
9.0
24.9
10%
90.7
93.8
40.2
45.8
43.8
34.0
95.6
84.1
12.0
11.0
27.7
96.6
88.4
15.0
35.2
100%
92.7
95.0
42.4
48.5
45.2
47.9
96.9
89.4
17.9
15.0
36.0
Table 10: The performance of ID, SL., DST, and NLU models trained on an increasing percentage of Arabic data, generated using data creation strategies $5 (RQ4). The ID and SL models are built upon KLM-R1arge and the DST and NLG models are based on nT55mall-

---

ID
SL
DST
NLG
% of
Training Data
Accuracy
Precision
Recall
JGA
Turn
FI
BLEU
ROUGE
METEOR
Random Sampling
1%
81.6
88.3
86.3
88.3
87.3
17.0
91.1
71.5
8.4
29.7
26.9
91.2
94.2
96.1
86.1
41.2
37.5
10%
91.0
94.9
92.1
95.1
93.5
43.2
96.5
87.5
17.0
43.7
39.7
50%
92.7
95.8
94.3
94.9
94.6
56.4
97.7
92.8
20.4
47.3
43.0
100%
93.2
96.1
94.6
95.7
95.1
59.8
97.9
93.5
20.8
48.4
44.1
N-gram
1%
84.8
90.6
87.6
89.8
88.7
17.3
92.4
73.2
11.2
35.1
31.8
5%
89.2
93.5
92.1
93.5
92.8
41.2
96.3
87.0
16.8
43.4
39.6
10%
90.4
94.0
92.8
94.6
93.7
47.5
96.9
89.6
18.6
45.4
41.5
94.8
97.8
93.0
47.7
43.4
100%
93.2
96.1
94.6
95.7
95.1
59.8
97.9
93.5
20.8
48.4
44.1
Equal Domain
1%
82.4
89.2
85.1
88.1
86.6
12.7
91.0
68.0
9.2
31.6
31.2
5%
88.5
93.1
91.9
94.0
92.9
34.5
95.6
84.2
15.0
41.1
39.7
10%
89.4
93.9
92.2
94.2
93.2
45.7
96.7
88.9
16.1
42.5
42.4
50%
92.3
95.7
94.0
95.4
94.7
57.1
97.7
93.1
20.2
47.3
42.9
100%
93.2
96.1
94.6
95.7
95.1
59.8
97.9
93.5
20.8
48.4
44.1
Equal Slot
198
83,2
89.3
95.8
94.8
19.2
92.3
73.3
32.8
29.4
85.3
10%
90.3
94.4
94.6
95.7
95.2
44.0
96.5
88.2
17.2
44.2
39.8
50%
92.4
95.7
94.6
95.7
95.1
56.5
97.7
93.0
20.1
47.6
43.4
100%
93.2
96.1
94.6
95.7
95.1
59.8
97.9
93.5
20.8
48.4
44.1
Max Length
1%
84.5
90.0
88.0
91.7
89.8
15.4
91.6
71.0
10.8
34.2
31.2
5%
89.5
93.8
92.2
93.8
93.0
38.6
96.0
86.0
16.9
43.9
39.7
10%
90.4
93.8
92.6
93.6
93.1
49.3
97.1
90.3
18.5
46.5
42.4
95.2
97.7
100%
93.2
96.1
94.6
95.7
95.1
59.8
97.9
93.5
20.8
48.4
44.1
Table 11: The performance of ID, SL. DST, and NLU models trained on an increasing percentage English data, generated using data creation strategies $5 (RQ4). The ID and SL models are built upon MLM-Raargo- and the DST and NLG models are based on mT5

---

ID
SL
DST
NLG
% of
Training Data
Accuracy
Precision
Recall
JGA
Turn
FI
BLEU
ROUGE
METEOR
Random Sampling
1%
79.5
86.5
64.4
64.8
64.6
3.0
83.9
46.3
4.2
21.4
16.9
71.9
72.4
94.4
25.7
10%
86.3
90.9
74.0
72.9
73.5
31.4
95.1
82.5
9.2
33.5
28.0
50%
89.0
92.8
78.1
78.3
78.2
44.4
96.6
88.5
12.5
38.5
33.0
100%
89.2
93.0
76.9
79.2
78.0
49.7
97.0
91.2
13.9
40.9
35.2
Max N-gram
1%
82.3
88.4
66.7
69.1
67.9
9.7
90.3
63.6
6.0
27.2
22.0
5%
86.8
91.2
71.4
73.6
72.5
24.2
94.4
79.5
9.5
34.5
28.7
10%
87.1
91.6
73.6
76.7
75.1
33.5
95.5
83.6
35.6
30.0
77.6
78.5
96.7
34.4
100%
89.2
93.0
76.9
79.2
78.0
49.7
97.0
91.2
13.9
40.9
35.2
Equal Domain
1%
78.9
86.5
63.3
67.2
65.2
86.9
52.7
4.8
24.2
22.4
5%
85.7
90.6
72.8
72.9
72.8
21.4
93.4
76.3
6.6
28.4
27.2
10%
86.8
91.5
75.0
74.8
74.9
31.9
95.1
82.5
8.6
32.1
30.0
50%
88.5
92.7
77.7
78.9
78.3
45.7
96.6
88.7
12.8
38.8
34.2
100%
89.2
93.0
76.9
79.2
97.0
91.2
13.9
35.2
Equal Slot
198
83.2
88.0
68.8
69.5
7.2
88.0
58.2
5.3
24.8
20.1
93.9
77.9
28.0
10%
86.5
91.1
74.1
74.6
74.3
36.1
95.6
84.1
9.3
33.7
28.2
50%
88.6
92.5
77.7
78.1
77.9
44.8
96.6
88.3
12.8
39.4
33.7
100%
89.2
93.0
76.9
79.2
78.0
49.7
97.0
91.2
13.9
40.9
35.2
Max Length
1%
82.4
88.0
69.3
70.0
89.8
6.9
89.1
60.2
6.1
27.5
22.4
5%
86.2
90.8
73.7
74.5
93.0
26.0
94.2
79.7
8.6
32.4
27.2
10%
87.5
91.8
75.8
76.3
93.1
37.6
95.7
84.6
35.6
30.0
77.1
96.7
34.2
100%
89.2
93.0
76.9
79.2
78.0
49.7
97.0
91.2
13.9
40.9
35.2
Table 12: The performance of ID, SL. DST, and NLU models trained on an increasing percentage of French data, generated using data creation strategies $5 (RQ4). The ID and SL models are built upon KLM-Riargo- and the DST and NLG models are based on mT5

---

ID
SL
DST
NLG
% of
Training Data
Accuracy
Precision
Recall
JGA
Turn
FI
BLEU
ROUGE
METEOR
Random Sampling
1%
79.9
86.8
74.8
76.5
3.6
85.7
48.1
7.0
27.1
24.5
94.4
81.2
34.6
10%
88.3
92.2
84.6
84.9
84.7
35.9
95.6
85.1
17.2
44.6
39.9
50%
91.4
94,5
87.2
85.6
86.4
48.2
96.9
89.7
21.8
50.6
45.7
100%
92.2
95.0
86.6
87.6
87.1
52.9
97.3
91.2
24.2
53.7
48.6
N-gram
1%
85.0
89.9
77.9
79.4
78.6
9.5
90.2
66.1
10.6
33.8
30.2
5%
89.1
92.9
83.4
85.1
84.3
22.8
94.2
79.1
16.4
43.6
38.5
10%
89.8
93.4
84.9
85.3
85.1
37.4
95.9
85.9
46.7
41.9
97.1
90.2
100%
92.2
95.0
86.6
87.6
87.1
52.9
97.3
91.2
24.2
53.7
48.6
Equal Domain
1%
82.7
89.0
77.9
77.3
77.6
86.4
53.6
7.8
27.7
30.5
5%
86.9
91.6
83.5
82.6
83.0
25.6
94.0
79.8
14.6
40.5
37.8
10%
89.8
93.4
83.5
85.0
84.2
37.5
95.7
84.9
16.0
42.8
42.2
50%
91.5
94.5
86.0
86.4
86.2
47.7
96.9
89.7
22.5
51.5
47.4
100%
92.2
95.0
86.6
87.6
87.1
52.9
97.3
91.2
24.2
53.7
48.6
Equal Slot
19
83.4
89.2
80.5
80.9
80.7
89.8
62.9
8.4
29.7
26.6
95.0
82.7
36.9
10%
89.6
93.4
83.5
84.4
84.0
36.2
95.6
85.2
17.4
45.2
40.5
50%
91.6
94.6
86.0
86.8
86.4
49.1
97.0
89.8
22.6
51.6
46.5
100%
92.2
95.0
86.6
87.6
87.1
52.9
97.3
91.2
24.2
53.7
48.6
Max Length
1%
85.8
90.5
76.5
77.0
76.8
10.5
89.8
64.2
10.2
34.2
30.5
5%
89.0
92.7
83.6
84.2
83,9
30.4
95.1
83.1
15.8
42.5
37.8
10%
89,9
93.6
84.8
84.9
84.9
39.7
96.2
86.6
18.8
46.9
42.2
91.9
85.1
86.7
97.0
100%
92.2
95.0
86.6
87.6
87.1
52.9
97.3
91.2
24.2
53.7
48.6
Table 13: The performance of ID, SL, DST, and NLU models trained on an increasing percentage of Turkish data, generated using data creation strategies $5 (RQ4). The ID and SL models are built upon RLa-Riargo- and the DST and NLG models are based on mT5

---

Relative A-Equivalence
1.0
0.8
0.6
0.4
Relative d-Eqqivalent squivalence
1.0
0.2
0.0
30% 40%
70% 80% 90%%
0.9
(a) Arabic
Relative E-Eguivalencs
1.0
0.8
0.8
1%
5%
10%
50%
100%
00.6
0.4
Figure 9: The relative equivalence in performance of SL models trained on an increasing percentage of data
0.2
Si-Poooeranns St.
0.0
20%
30%
in Arabic, French, and Turkish, generated using both the Random Sampling and Max N-gram strategies. The SL models are built upon aargR and evaluated using the F1 metric.
(b) French
Relative Sntdiive--Eqiiiee 8-Equivalenee
1.0
04
0.
N MEG-PROOOEeN NICIFIX
0.0
10%
20% 30% 40% 50% 60% 70% 80% 10%%
(c) Turkish
Relative E-quivalee
1.0
0.8
0.6
ID Pausummeet ID-bua
Relative A-Equivalent alence
1.0
0.0
100 20%
30% 40% 50% 60% 70% 80% 90%100%
(d) Turkish
0.8
and NLG for system trained increasing percentage of in- language data for (a) Arabic, (b) English, (c) French, and (d) Turkish, with respect fully supervised sys- tems, namely pARA(.), pENO(.), paRA() (.). and (T)R(.), trained on sized monolingual in-language data. We compared strategies for creating the validation set: proportional and full. In the few-shot setup, the propor- tional strategy constructs a validation set that propor- tionate the training set (compared to the full training set). For example, we train model using 10% of the original training and an equivalent 10% of the origi- nal validation set. Conversely, the full strategy involves consistently training the system with a complete valida- tion set, such 10% of the training validation set. The ID and SL models are built upon RLM-R and evaluated with Accuracy and FI. The NLG models are based on mT5 and evaluated with BLEU. Due to the significant computational demand, we did not measure the performance of the DST models in these experiments.
Figure Relative performance for
0.6
0.4
SC SU-HANOCH
3L-MAN <RAM
0.2
NEG-HANNIO
NLG-Mas Max NoaIAA
0.0
100%
Figure 10: The relative performance SL and DST models trained on an increasing percentage of Ara- bic data, generated using both Random Sampling and Max N-gram strategies, in comparison the per- formance of (.). The SL models are built upon XLM-Rjarge and evaluated using the Fl metric, while DST models are based mT5 and evaluated using the JGA metric. In addition, we observed performance gains for ID, although the improvements were marginal and not visually represented in the figure.
and the entire

---

Relative (-Equivalence
1.0
0.8-
0.6
(a) Arabic
Relative f-Equivalence
0.8
Stas
52.00n
NiG No1 EXmmaNc
0.4
5%
20%
(b) French
Reletive Sestiiee--aaiie 8-Equi valence
1.0
0.8
0.6
maver remrrre
0.4
SL-dxza
SL- D--Dnaast
N1G commeee
1%
5%
10%
20%
(c) Turkish
Figure The relative performance of ID, SI,, and NLG for systems trained on increasing percentages of multilingual data. Specifically, a system trained on 10% training data implies that this system is trained using 10% of the training data for all four languages, si- multaneously. We evaluated the trained systems on three languages: (a) Arabic, (b) French, and (c) Turkish, with respect to fully supervised systems, namely (ARA(.), (-) and (-)*(-), (-), trained on full sized monolingual in-language data. We compare ating the training set. Firstly, the Same strategy refers to training system with the same set dialogue pat- terns in the four languages. In other words, use the multi-parallel dataset to train the system. Secondly, the Different strategy utilises different training sets with mutually exclusive dialogue patterns to train the system. For example, a system trained with 20% of the training data using the Different strategy will encounter 80% of the dialogue patterns. The ID and SL models are built upon XLM-Riarge and evaluated with Accuracy and Fl. The NLG models are based mT5 and evaluated with BLEU, Due to the significant computational de- mand, we not measure performance the DST models in these experiments.
strategies for


Reviews:
[{"review_id": "YsZyVokL9G", "reviewer": "Reviewer_Pe2h", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your encouraging review. Here, our clarifications to your questions and concerns:\n\n***\n\n**Reason To Reject 1: Limited language coverage.**\n\nWhile we agree that the present study only covers a small sample of world languages, these languages are from different language families and already exhibit distinct linguistic properties. These properties contain aspects such as word order, grammatical gender, inflectional systems, and verb conjugation. For example, Arabic has complex morphology and a right-to-left script. French has gender and number markers for nouns and adjectives, with agreement extending to determiners. Turkish features a complex system of agglutination and a strict subject-object-verb word order. Our experimental results also confirm that these varied linguistic properties influence the system performance.\n\nAlthough these languages are not extremely low-resourced, they do belong to the category of under-served languages (Song et al., 2023) in the context of ToD systems because of their high demand (Blasi et al., ACL 2022) and the absence of training data.\n\nDeveloping ToD systems in other languages requires new data collection efforts. Our study is timely to provide **practical advice** on increasing the cost-efficiency of multilingual ToD data collection and thereby increasing the language coverage for future system development. For example, instead of collecting an exhaustive dataset, we might consider gathering only 50% of the dataset to build a DST model achieving a 95% of the JGA. With more available resources for more (lower-resource) languages in the future, we could expect to expand the analysis to a broader linguistic landscape in the future.\n\nReferences:\n\nBlasi et al., ACL 2022. Systematic Inequalities in Language Technology Performance across the World’s Languages\n\nSong et al., 2023. GlobalBench: A Benchmark for Global Progress in Natural Language Processing\n\n***\n\n**Presentation Improvement 1: Including RQs in Section 5.**\n\nWe will expand the introductory paragraph of Section 5 and clarify how each research question is related to the subsequent analysis following your suggestion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Systematic Study of Multilingual ToD systems: This work provides a good systematic and experimentally sound study for evaluating the disparities across ToD models within languages and across languages. The experimentation is well-backed with results and the paper yields various insightful conclusions. \n\n- Well-written work: This paper is pretty well-written with technically defining difficult concepts. It’s easy to read and follow. Tables and figures have elaborate captions which make them easy to understand.", "reasons_to_reject": "- Less languages studied: Although this is an issue of the underlying dataset, but the insights may not be generalizable to all languages since it’s been studied only for 3 languages overall, none of which are low-resource in nature", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper conducts an exhaustive and systematic study for empirically evaluating task performance disparities between multilingual task-oriented dialogue (ToD) systems. They define new quantitative measures of absolute and relative equivalence to quantity the disparities within and across languages. Overall, they answer four research questions basing their conclusions on a number of factors like the nature of the task, the underlying pre-trained model, the source and target languages, and the amount of ToD annotated data. Overall, their work aims at aiding in better collection of ToD data which in turn can boost better development of multilingual ToD models.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Section 5: It would be good to include the research questions included in the first section here for better readability."}, "extra_text": {}}, {"review_id": "0xyVxAq9tR", "reviewer": "Reviewer_Hf6r", "summary": "", "questions": "I kept wondering throughout how exactly the smaller datasets were compiled. The answers are provided towards the end of the paper in a satisfactory manner, but I still wondered whether it was too late to introduce them.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your insightful review. Here are our clarifications to your questions:\n\n***\n \n**Reason To Reject 1: Some details are in the Appendix.**\n\nDue to the limited space, the Appendix describes:\n1) our experimental setup (e.g., hyperparameter choices and GPU runtime),\n2) additional experiential results to show that our findings hold across each target language and each ToD task, where we aimed to summarise all the key findings in the main paper,\n3) results to support our auxiliary findings, which will also be further expanded in the main paper given one extra page,\n4) an anonymised version for the Multi3WOZ paper following the Double-Blind Instructions. This will be removed after its formal publication as the Multi3WOZ work will also become publicly available — the actual EMNLP submission is a self-contained submission, and we enclosed the version of the Multi3WOZ paper for interested readers who want to know more about the collection process and the data (as at the time of the paper submission, the Multi3WOZ paper was not publicly available).\n\n***\n\n**Reason To Reject 2: Confusion in “modelling the annotation process with more fine-grained details”.**\n\nSorry for the confusion. In Section 5 (RQ 4), our findings have shown, as a proof of concept, that strategic budget allocation in a multilingual data collection project can enhance model performance without additional costs. In our analysis, we abstract away many “fine-grained details” in a real-word data collection project. For example, we assume that the annotation cost for all data entries in all languages are identical. However, human annotators are usually paid based on the amount of time they devote. Consequently, annotating each data entry may cost differently. For example, translating a long dialogue may cost more than a short one simply due to the increased time consumption. We will make these ‘fine lines’ clearer in the paper.\n \nWe hope that our findings could inspire future work to delve into more sophisticated annotation strategies and conduct a “more fine-grained” analysis. In other words, we encourage future analysis to consider these finer details of real-world data collection. It does not necessarily mean that a multilingual data collection project needs to collect annotations at a finer level of granularity to be successful. We agree that this sentence (Lines 713 – 714 in the Limitation Section) is confusing. We will clarify it in the final version.\n\n***\n\n**Reason To Reject 3: Why report zero-shot performance relative to English for all target languages?**\n\nIn the zero-shot cross-lingual transfer scenario, where task-specific data for target languages is absent, we exploit abundant task-specific data from resource-rich source languages (in our study, English) to make predictions in target languages (Arabic, French, and Turkish). To elaborate, we finetune **multilingual PLMs** with English in-domain dialogue data and evaluate these models with testing data in each target language. This well-established paradigm has been widely used in previous research, as exemplified by the work of Lauscher et al. (EMNLP 2020) and the work by Ansell et al. (ACL 2023), among others.\n\nReferences:\n \nAnsell et al., ACL 2023. Distilling Efficient Language-Specific Models for Cross-Lingual Transfer\n\nLauscher et al., EMNLP 2020. From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers\n\n***\n\n**Question 1: Clarification on the procedure of sampling few-shot data for RQ3.**\n\nIn line 535 and the caption of Figure 2, we will add a sentence clarifying that we randomly select a subset (e.g., 20%) from the complete training dataset for few-shot training, and we’ll also release the subsets for reproducibility.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The overall theme of the paper is that not all languges benefit equally well from multilingual PLMs, possibly with adverse effects on resource-poor languages. For example, the paper finds that TOD systems trained for Arabic and Turkish with data fully parallel to English TOD display low performance. This is a nice (and experimentaly-supported) finding, though the paper does not fully clarify whether this result is obtained due to a mismatch between data evaluation metrics (BLEU, etc.) and human judgement, the typological difference of these languages, or other reasons. The reason probably lies somewhere in between. In fact, the authors acknowledge the recent research in the field highlighting the fact that automatic evaluation metrics only moderately correspond to human judgement. I agree with the authors that there's need for further research  regarding data evaluation metrics versus human annotations, and it's good that they mention this point in the paper.", "reasons_to_reject": "I do not have strong reasons to reject this paper, though I felt that the experiments required a full understanding the attachments, which I had expected to quickly skim. It turned out that much of the detail skipped in the current paper is provided in the attachment, which is a paper submitted to a different platform. \n\nIt's not clear to me what the authors mean in lines 713-714 concerning the need for finer details at the annotation stage. I understand the authors' point that the authors couldn't delve into the intricacies of cost variations in ML data annotation, but why should we need details at a finer level of granularity to achieve success in different languages? This point probably relates to the data collection methodology explained in the attachment but not in the current paper.\n\nIn lines 440 and the following paragraph, zero-shot cross-lingual evaluations are carried out by training an English system using an English dataset. The performance of the system is tested in target languages. It is not clear to me why an English system was trained rather than a multilingual model.", "questions_to_authors": "I kept wondering throughout how exactly the smaller datasets were compiled. The answers are provided towards the end of the paper in a satisfactory manner, but I still wondered whether it was too late to introduce them.", "paper_topic_and_main_contributions": "Given the importance of multilingual resources for NLP and the need for diaologue systems, this paper introduces a large-scale parallel multilingual task-oriented dialogue, referred to as TOD covering dialog turns over 4 languages (French, Arabic, Turkish, English). The data creation is described and various experiments over this dataset are explained focusing on the performance of the models over each language.The authors describe the notions of \"absolute\" and \"relative equivalence\" used to capture performance disparities across languages as well as within each language. The paper claims to emprically prove the existence of the adaptation and intrinsic biases in the ToD systems.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hXG9DoMCzY", "reviewer": "Reviewer_5jxt", "summary": "", "questions": "A. Considering that MULTI3WoZ includes only four languages, should the diversity of languages also be considered one of the required data properties? How to ensure that the findings are not biased by this single dataset and address potential implications on the generalizability of the results (e.g., language family, resource level)?\n\nB. The focus of the paper is on the analysis of task performance disparities, yet RQ 3 and 4 seem to be somewhat off-topic (i.e., data requirements and collection strategy). Please explain the detailed rationale behind the choice of these research questions and how they contribute to the overall research goal of this study.\n\nC. The paper emphasizes the performance disparities between English and other languages, \"even with\" similar development approaches. However, it is noteworthy that the shared mPLMs themselves already show potential biases in favor of English across languages. Considering this, it is not surprising to assume that the shared mPLMs in similar development lead to the observed performance disparities. Thus, what is the valuable insight from this observation?\n\nD. The paper mentions that the reliability and generalisability of findings are affirmed even with only a single run in the experiments. Could the authors provide more detailed explanations or evidence to support this claim?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your helpful review and insightful suggestions. Here are our clarifications to your questions: \n\n***\n\n**Reason To Reject 1 and Question A: Limited language coverage.**\n \nWhile this study only covers a small sample of world languages, these languages are from different language families and already exhibit distinct linguistic properties. These properties contain aspects such as word order, grammatical gender, inflectional systems, and verb conjugation. For example, Arabic has complex morphology and a right-to-left script. French has gender and number markers for nouns and adjectives, with agreement extending to determiners. Turkish features a complex system of agglutination and a strict subject-object-verb word order. Our experimental results also confirm that these varied linguistic properties influence the system performance.\n \nAlthough these languages are not extremely low-resourced, they do belong to the category of under-served languages (Song et al., 2023) in the context of ToD systems because of their high demand (Blasi et al., ACL 2022) and the absence of training data.\n\nCovering a typologically diverse set of languages is a desirable and important property for system development. On the other hand, developing ToD systems in other languages requires new data collection efforts. Our study is timely to provide practical advice on increasing the cost-efficiency of multilingual ToD data collection and thereby increasing the language coverage for future system development. For example, instead of collecting an exhaustive dataset, we might consider gathering only 50% of the dataset to build a DST model achieving a 95% of the JGA. With more available resources for more (lower-resource) languages, we could expand the analysis to a broader linguistic landscape in the future.\n\nReferences:\n \nBlasi et al., ACL 2022. Systematic Inequalities in Language Technology Performance across the World’s Languages\n \nSong et al., 2023. GlobalBench: A Benchmark for Global Progress in Natural Language Processing\n\n***\n\n**Reason To Reject 2 and Question C: What is the distinct novelty that this work offers compared to previous research in the field?**\n\nIn addition to offering a “pioneering systematic analysis” of performance disparities, focused on the high-impact downstream application of task-oriented dialogue, and empirically confirming the existence of adaptation bias and intrinsic bias, in comparison to prior research, our paper introduces several novel contributions:\n\n1) We propose measures to quantify the performance disparities (Section 3.2).\n2) Based on these quantitative measures, our analysis provides practical advice to multilingual ToD system development. For example, Table 1 shows that smaller models outperform their larger counterparts for the NLG tasks. It's worth noting that earlier ToD systems typically employ smaller PLMs for response generation, often without explaining the rationale behind (Section 5 RQ1 and RQ2).\n3) To the best of our knowledge, we perform the first quantitative analysis of the impact of the scarcity of in-domain, in-language training data on performance disparities (Section 5 RQ3).\n4) In addition to demonstrating the existence of inherent bias (model bias), our proposed measures enable us to quantify the extent of intrinsic bias. Table 1 and Figure 1 show the extent to which performance drops for each language due to inherent bias in individual PLMs. Furthermore, the experimental results presented in Figure 2 (b) indicate that mPLMs not only perform better in English but also have a faster learning curve in the few-shot scenario, yielding superior relative performance with the same volume of training data compared to other languages.\n5) Our investigation in response to RQ3 is motivated by challenges encountered in real-world data collection projects, thereby providing practical advice to future multilingual ToD dataset collection. Particularly, we show that the amount of data required depends on the task complexity, with 20% of the in-language data for simpler tasks and more than 50% of the data for more complex tasks such as DST and NLG. Depending on the performance goal of developed systems, future data collection efforts could collect a subset of the full dataset and ultimately increase the language coverage with a fixed budget.\n6) As a proof of concept, our findings In Section 5 (RQ 4) shows that strategic budget allocation in a multilingual data collection project can enhance model performance without additional costs. We hope that our findings will inspire future research in improving the cost-efficiency in multilingual data collection, thereby broadening language coverage under budget constraints.\n\n***\n\n**Reason To Reject 3 and Question B: What is the central theme for this paper?**\n\nOur paper focuses on conducting a **quantitative** analysis of performance disparities in the context of multilingual ToD systems, addressing research questions RQ1, RQ2, and RQ3. Following this analysis, we aim to provide **practical advice** on future data collection efforts (RQ3, RQ4), which is a crucial step to minimise these disparities for under-served languages (Song et al., 2023). Our analysis answers an important real-world question: How to balance the amount of data per language and the language coverage under budget constraints? In addition, we show the possibility of enhancing model performance by strategic budget allocation in multilingual data collection, thereby increasing the utility of the language technology with no additional cost (Blasi et al., ACL 2022). As outlined above, the central theme of this paper is: identifying, quantifying, and minimising performance disparity in multilingual ToD systems.\n\nWe include a short description of MULTI3WOZ dataset in Section 3.3 because the paper introducing this dataset is under review and not publicly available. In the final version, we will shorten this section.\n\nReferences:\n \nBlasi et al., ACL 2022. Systematic Inequalities in Language Technology Performance across the World’s Languages\n\nSong et al., 2023. GlobalBench: A Benchmark for Global Progress in Natural Language Processing\n\n***\n\n**Question D: Can the conclusions be affirmed adequately based on a single run with one random seed?**\n\nIn the Limitation section, we acknowledge that our experimental results are derived from a single run due to the high cumulative computational budget of these experiments (> 5,000 GPU hours). Three random runs with varying random seeds will increase the computational budget by more than 10k GPU hours.\n\nIndeed, we did try runs with multiple seeds for less computationally intensive tasks: intent detection and slot labelling. We run parallel experiments for fully supervised models, similar to those reported in Table 1, using three different random seeds. As shown in the following table, the results do not show any real volatility that would impact the main empirical findings and trends in the results.\n\n| Language | ID-Accuracy | ID-F1 | SL-Precision | SL-Recall | SL-F1 |\n|----------|-------------|-------|--------------|-----------|-------|\n| Arabic   | 0.2         | 0.2   | 0.3          | 0.3       | 0.3   |\n| English  | 0.1         | 0.1   | 0.1          | 0.0       | 0.0   |\n| French   | 0.2         | 0.2   | 0.3          | 0.4       | 0.3   |\n| Turkish  | 0.1         | 0.1   | 0.1          | 0.2       | 0.1   |\n\nTable: Standard deviation computed from three experimental runs using varied random seeds. The presented values correspond to each task-metric pair. For instance, the \"ID-Accuracy\" column represents the standard deviation for accuracy in the intent detection experiment.\n\n\nNonetheless, we also highlight the consistency of our findings across **all target languages and tasks**. In other words, our findings hold when evaluated through four independent runs for the four target languages. This consistency in outcomes increases our confidence in the reliability and generalisability of our empirical findings.\n\n\n\n***\n\n**Missing Reference 1: A more in-depth analysis of related research.**\n \nIn Section 1, we provide an overview of the background and relevant literature about performance disparities in language technology across languages. Previous studies (e.g., Hu et al., ICML 2020; Lauscher et al., EMNLP 2020; Debnath et al., ACL 2021) have recognised the presence of the performance disparities. However, these studies have not undertaken a comprehensive systematic analysis that covers the ToD domain.\n\nYour suggested stance paper by Søgaard (EMNLP 2022) advocates a seemingly radical but thought-provoking proposal on implementing policies that discourage a predominant focus on English NLP for a period of time. This approach could potentially contribute to the reduction of performance disparities across various languages and make a broader impact with collaborative efforts from the research community \n\nCompared to this work, our paper presents an experimental study on analysing the performance disparities in ToD and offers **practical advice** on multilingual ToD data collection. We will incorporate this reference into the final version of our paper and add a discussion emphasising our contribution compared to related research.\n\nReferences:\n\nDebnath et al., ACL 2021. Towards more equitable question answering systems: How much more data do you need? \n\nHu et al., ICML 2020, XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalisation\n\nLauscher et al., EMNLP 2020. From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers\n\nSøgaard, EMNLP 2022. Should We Ban English NLP for a Year?\n\n***\n\n**Presentation Improvement 1.**\n\nFollowing your suggestions, we will incorporate the following modifications into the final version of our paper:\n\n1) In Section 5, we will provide a succinct introduction that clarifies the connections between each of our research questions and the central theme of our paper.\n2) We will condense Section 3.3 by retaining only the essential explanations required to establish the notations used in this paper.\n3) With the additional space, we will add a discussion highlighting our contribution compared to related research.\n4) Regarding the dataset, Multi3WOZ is the only dataset suitable for supporting our analysis. In fact, we have reported testing results on a second dataset. Please see the results on GlobalWOZ in Table 1. However, GlobalWOZ does not provide human-created data for system training, which is why we decided to base our empirical study on high-quality human-curated data from Multi3WOZ.\n5) We leave performing a similar analysis on other NLP applications as a promising future work, and we’ll expand the discussion as suggested.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper stands as the pioneering systematic analysis of performance disparities in multilingual ToD systems, introducing two fresh quantitative equivalence measures. By conducting experiments on the latest ToD benchmark, MULTI3WoZ, the paper investigates several valuable research questions. The empirical findings bring insights into the challenges and opportunities associated with building ToD systems for different languages.\n- The paper conducts comprehensive investigations across multiple types of tasks under controlled settings. Experimental results provide meaningful evidence to support its findings for each research question. The commitment to releasing the data and code fosters transparency and reproducibility.\n- The paper provides a good example of utilizing quantitative measures to analyze the performance disparities across languages in ToD systems. By introducing new evaluation metrics, adopting a comprehensive experimental design, and focusing on research-question-oriented paradigms, the study establishes a benchmark for future research, contributing to the advancement of evaluation methodologies, not only in this field but also in multilingual NLP.", "reasons_to_reject": "+ The paper presents a comprehensive analysis of multiple tasks and models, but its primary reliance on the MULTI3WoZ dataset, encompassing only four languages (Arabic, English, French, and Turkish), raises concerns about the generalizability of the findings to a broader linguistic landscape. While the dataset has its own advantages for the ease of study, the limited scope could potentially bias the conclusions and limit their applicability.\n+ The findings of this paper, while insightful, may not be entirely groundbreaking, as similar observations regarding adaptation bias and intrinsic bias have been reported previously. The existence of performance disparities between English and other languages and the need for training data in new languages are widely acknowledged in the NLP community. While the study provides empirical contributions, it does not seem to present entirely novel findings.\n+ The clarity of focus in this paper is a concern as it seems to lack a well-defined central theme. Some research questions appear questionable and possibly unrelated to the main objective of analyzing task performance disparities. Additionally, a significant portion of the paper is devoted to non-priority aspects, such as dataset introduction, which might diminish its status as a rigorous research study and instead give the impression of a follow-up analysis blog of the MULTI3WoZ dataset.", "questions_to_authors": "A. Considering that MULTI3WoZ includes only four languages, should the diversity of languages also be considered one of the required data properties? How to ensure that the findings are not biased by this single dataset and address potential implications on the generalizability of the results (e.g., language family, resource level)?\n\nB. The focus of the paper is on the analysis of task performance disparities, yet RQ 3 and 4 seem to be somewhat off-topic (i.e., data requirements and collection strategy). Please explain the detailed rationale behind the choice of these research questions and how they contribute to the overall research goal of this study.\n\nC. The paper emphasizes the performance disparities between English and other languages, \"even with\" similar development approaches. However, it is noteworthy that the shared mPLMs themselves already show potential biases in favor of English across languages. Considering this, it is not surprising to assume that the shared mPLMs in similar development lead to the observed performance disparities. Thus, what is the valuable insight from this observation?\n\nD. The paper mentions that the reliability and generalisability of findings are affirmed even with only a single run in the experiments. Could the authors provide more detailed explanations or evidence to support this claim?", "paper_topic_and_main_contributions": "It is a long-standing goal of the NLP community to establish a general system capable of performing effectively across multiple languages. For this goal, the paper focuses on task-oriented dialogue (ToD) systems and empirically analyzes task performance disparities. In addition to standard task performance metrics, the authors propose two primary quantitative measures: absolute and relative equivalence in system performance. Leveraging the latest ToD dataset, MULTI3WoZ, the paper conducts a series of carefully designed experiments under controlled settings, centered around four predefined research questions. The empirical evidence, involving multiple tasks across NLU, DST, and NLG, confirms the presence of adaptation and intrinsic biases in current ToD systems. The role of acquiring high-quality data in a new language is particularly emphasized. The further study provides practical insights for data collection and system development to maximize cost efficiency. Overall, the findings offer valuable guidelines for future research to effectively expand ToD systems across languages.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "While there might not be a specific missing reference, the paper may benefit from offering a more in-depth analysis of related research to distinguish its contributions from others, even though common beliefs surrounding English hegemony already exist in NLP. The following can be one example.\n+ Should We Ban English NLP for a Year? (Søgaard, EMNLP 2022)", "Typos_Grammar_Style_and_Presentation_Improvements": "To enhance the presentation, it is suggested to frontload and clarify the main focus, aligning research questions with the goal of analyzing task performance disparities. Condensing certain parts, such as the dataset introduction, will create space for core analysis and results, emphasizing the primary contributions. Introducing diverse datasets could provide a broader perspective on language performance disparities, enhancing the value of this work."}}]

Author rebuttal:

Meta-review: This paper provides an empirical analysis of task performance disparities between multilingual task-oriented dialogue (ToD) systems, demonstrating that these disparities depend on the nature of the task at hand, the underlying pre-trained language model, the target language, and the amount of annotated data.  The findings offer valuable guidelines for future research to effectively expand ToD systems across languages.  Two of the reviewers express a concern that only four languages were covered, and that therefore the findings may not generalise to other languages.  The authors addressed this to some extent in their rebuttal by pointing out that the four languages were from different families, emphasising their linguistic variation.
Decision: Accept-Main