PRESTO: A Multilingual Dataset for Parsing Realistic Task-Oriented Dialogs
Authors: Rahul Goel
Waleed Ammar
Aditya Gupta
Siddharth Vashishtha
Motoki Sano
Faiz Surani
Max Chang
HyunJeong Choe
David Greene
Chuan He
Rattima Nitisaroj
Anna Trukhina
Shachi Paul
Pararth Shah
Rushin Shah
Zhou Yu
Venue: EMNLP 2023 Main (2023)
Keywords: task oriented dialogs
semantic parsing
nlp

Abstract: Research interest in task-oriented dialogs has increased as systems such as Google Assistant, Alexa and Siri have become ubiquitous in everyday life. However, the impact of academic research in this area has been limited by the lack of datasets that realistically capture the wide array of user pain points. To enable research on some of the more challenging aspects of parsing realistic conversations, we introduce PRESTO, a public dataset of over 550K contextual multilingual conversations between humans and virtual assistants. PRESTO contains a diverse array of challenges that occur in real-world NLU tasks such as disfluencies, code-switching, and revisions. It is the only large scale human generated conversational parsing dataset that provides structured context such as a user's contacts and lists for each example. Our mT5 model based baselines demonstrate that the conversational phenomenon present in PRESTO are challenging to model, which is further pronounced in a low-resource setup.

PRESTO: A Multilingual Dataset for Parsing Realistic Task-Oriented Dialogs
Rahul Goel", Waleed Ammar* Aditya Gupta', Siddharth Vashishtha
Motoki Sano', Faiz Surani Max Chang, HyunJeong Choe, David Greene, Kyle He, Rattima Nitisaroj, Anna Trukhina, Shachi Paul, Pararth Shah, Rushin Shah, Zhou Yu Google Inc. (presto.. dataset@@oogle com)
University of Rochester 9 University of California, Santa Barbara
Columbia University
Holistic Intelligence for Global Good
Abstract
such MultiWoz (Budzianowski al., 2018; Eric et al., 2020), TOP (Gupta et al., 2018), MTOP (Li et al., 2021), and SMCalFlow (Andreas et al., 2020) have enabled researchers to study numerous developments in conversational semantic parsing, e.g., Budzianowski and Vulić (2019), Pasupat et al. (2021). However, many challenging aspects of pars- ing, such as disfluencies, code-switching, and the use of the structured context which grounds conver- sations, have been missing in these datasets, stifling meaningful research progress those areas.
Research interest task-oriented dialogs has increased systems such Google Assistant, Alexa and Siri have become ubiquitous in ev- impact academic research in this area has lack of datasets that realistically capture the wide array of user pain points. To enable re- search on some of the more challenging aspects parsing realistic conversations, we introduce PRESTO!, public dataset of over 550K con- textual multilingual conversations between hu- mans and virtual assistants. PRESTO contains diverse array of challenges that occur in real- world NLU tasks such as disfluencies, code- switching, and revisions. scale human generated conversational parsing dataset that provides structured context such user's contacts and lists for each example. Our mT5 model-based baselines demonstrate that the conversational phenomena present in PRESTO challenging model, which is further pronounced in a low-resource setup.
eryday life. However, been limited by the
With that in mind, we present PRESTO, a dataset that better represents real conversations users have with their virtual assistants. PRESTO consists of:
the only large-
Challenging conversational phenomena such as disfluencies, code-switching and user revi- sions, addressing some the aforementioned limitations of current virtual assistants and enabling researchers to test new methods of addressing these challenges.
1 Introduction
Realistic multilingual dialogues. Unlike other datasets, conversations in each language were not obtained by translating English dia- log, but instead contributed by native speakers of each language.
Virtual dialog agents (a.k.a. "assistants"" are in- creasingly becoming part of our everyday life. From setting up alarms to controlling one's home environment, users now converse with their devices to accomplish many tasks previously done manu- ally. Parsing ooiintree dialogs is the problem of understanding the user's intent and any associ- ated arguments so that the assistant can fulfill the requested task.
Human and synthetic structured context with the conversations. The data contributors were instructed look and optionally reference relevant context in the user's virtual environ- ment. For example, contributors could ref- erence the user's contact list while initiating a phone call or sending a message using the virtual assistant.
Given the prominence of data fataenriven methods NLP, public availability of relevant datasets dictates which problems can be effectively stud- ied by the research community at large. Datasets
When users speak naturally to their virtual assis- tants, variety of conversational phenomena are observed: disfluent turns (Gupta et al., 2021), revi- sions (also known as conversational repair (Cassell et al., 2000)) and code-switching (also known as
Equal contribution. Work done while Google,
Work done during an internship at Google.
'tttps: //tg thub. com/ google e-researcch ch- ch-datasets/
presto

---

Correct-argument (en)
Cancel-action (hi)
Correct-action (es)
Context:
Context:
Lists
Contacts
Notes Meeting lomorrow Wedding
Lists <ffee yprir
Contacts at ater sliiww
Notes Wiich attin
Lists
Contacts
Notes
School
Charlotte
Pendentes
Gliva
Recager
suppyy
Taylor
Deberce
Karla
pedio
Cleaning
Henry Moore
Vacaciones
Ramón Ontz
Santer io beuuss
Dialog:
Dialog:
Place call to Henry Moore
#i8 ydit filee smreew पोिााा yeah
Agreger aiquiler de coche la lista vacaciones
Calling Herry
(Henry
veा whear wirgh
De acuerdo, he añiadido alquiler de coche
Moore Studios Gardens)
No, Henry Moore my contacts
No, ver lista vacaciones
Initiato coll
S
Cuncel
Parson
Istt Jabel
Persoral contact
gpur m8!
ver listas
Henry
my
Figure I: Examples of revision dialog sessions from PRESTO. PRESTO includes annotated dialogs in 6 languages (de, en, es, fr. hi, ja) with various characteristics such as corrections pooreec-argument correct-action), cancellations (cancel-action) etc. Each example consists Input: a user's virtual state (context), one more user utterances and the corresponding virtual assistant responses (dialog). Output: the semantic parse of the last user utterance in the dialog (parse).
code-mixing (Agarwal al., 2022)), to name few. Our dataset highlights several such linguis- tic phenomena, and provides thousands (Table I) of examples of each phenomenon per language, each produced by a native speaker. Overall, our dataset contains more than 550k annotated utter- ances across six languages. Each user utterance is further verified by three native speakers for flu- ency and correctness and then annotated separately twice. Section $3 discusses how the data col- lected.
the number of examples in each language.
Prior large multilingual datasets for conversa- tional semantic parsing such as MTOP (Li et al., 2021) and MASSIVE (FitzGerald al., 2022) con- tain non-English conversations obtained by translat- ing English conversations other languages, ? sulting unnatural and synthetic utterances which are unlikely be spoken by native speakers of the non non- Non-English language. For example, an Arabic example in MASSIVE:
Section $2 discusses context as well as other im- portant phenomena which are well-represented in this dataset such as user revisions and disfluent ut- terances. In section $4, (Xue et al., 2021) present some baseline model performance metrics for this dataset, c.g., exact match accuracy on the test sets, the relative perfor- mance of monolingually- vs. multilingualll-trrinee models, and data scaling effects with respect to var- ious linguistic phenomena.
حل الوفع خدمتت عن المملل إل أحاح
التحقز عكاك
mT5-based models
does not resemble native Arabic speech of any di- alect. The corresponding English utterance the MASSIVE dataset is: need to have location ser- vices on can you check.
Code-Switched Utterances. Multilingual users often mix words from two languages in the same utterance, shown in Fig. 2. Recognizing the difficulty of parsing such utterances, we asked bilingual data contributors to provide mixed utter- ances, and marked these examples with a special tag 'code-see swie tching' to enable isolated research focused on addressing this phenomenon. Table shows the number of code-switched examples in
2 Dataset Characteristics
In this section, we discuss the characteristics of PRESTO dataset that set it apart from existing datasets: native conversations in multiple lan- guages, code-switched utterances, user revisions, disfluencies, and structured context.
Native Speakers. The PRESTO dataset only in- cludes utterances provided by native speakers of the language, with no translation. Table shows
'The translation was constrained in order to ensure the English source and the translation have the same semantic parse with argument values in English translated to the corre- sponding phrases in the other language.

---

Language
# Intents Slots
Avg. /lots/
Avg. Tokens/ Utterance
Avg. Prev. Utterances
Total Examples
Code Switching
User Revisions
Spoken Disfluencies
Utterance
German
34/ 285
1.60
9.17
2.56
83,584
12.357
22.129
16,781
English
34/ 303
1.57
9.03
2.49
95,671
5,918
27,741
17,588
Spanish
34/ 299
1.72
10.70
2.57
16,164
12,570
27,713
21,510
French
34/303
1.63
10.73
2.63
95,870
12,939
25,157
20,137
Hindi
385
1.47
9.32
2.55
72,107
16,517
15,833
10,193
Japanese
34/292
1.73
11.23
2.58
109,528
15,200
29,474
23,838
Total
1.63
10.11
2.57
552,924
75,501
148,047
110,047
Table 1: Corpus and sentence level statisties of PRESTO, slices per language, including various linguistic phenomenon.
Send digital object
with user revisions in the last utterance for each language PRESTO.
Medium
Attachment
SMS
è सास
Screenshot
it qu))
Disfluencies. Due to the spoken nature of most conversations with virtual assistants, user utter- ances frequently have disfluencies such as repeated phrases and filler words, as shown in Fig. Table shows the number of examples with disfluencies for each language in PRESTO.
Log nutrition +B
Provider
?
Time 8
Sparkpeople record my meal from
esta mañana
Resume exercise
Device 2
Ich will mein Workout auf dem iPhone
weitermachen
Send. digital object
Figure 2: Examples of Hindi-English (Send a screen- shot along with the SMS.). Enngish (Sparkpeo- ple, record my meal from this morning.), ), and German- English (I want continue my workout on the IPhone.) code-switched utterances from PRESTO.
Medium E3
Person
Write an email uh yeah write an
emaii
Carla Harris
Pay bill
Service type
カスl..
カス
each language.
Order_nie
Provider
Dropoft location
The example in Fig. highlights another important aspect of realistic conversations: users often revise their request to the assistant. We have such tags in our dataset which are all grouped under the broader user-revision linguis- tic phenomenon. The tags are: correct-action, correct aargumet, within-turn-coreectin and cancel- action. Sometimes the revision is necessary due to a mistake made by the virtual as- sistant, as Fig. (correct-argumenn). At other times, the user may simply change their mind about what they want the assistant to do which may hap- pen in the same utterance, e.g., "Add bread no no wait add wheat bread to my shopping list", in a later utterance, c.g., "Sorry add wheat bread in- stead." Another common revision users make is to cancel their requests, e.g., "Sorry don't add any- thing". Fig. shows examples of some of these revisions. Table reflects the number of examples
User Revisions.
Commandes un Ub un un Uber pour lo cinéma
Figure Examples of utterances in English, Japanese (Gas bill, I want to pay the gas bill.,, and French (Order an Ub an Uber for the cinema) with disfluencies (filler words or repetitions) from PRESTO.
Structured Context. Users interact with virtual assistants in a virtual world (henceforth context) that may include structured objects like contacts, shopping lists, to-do lists, alarms, smart devices, etc. Context may or may not be needed to inter- pret user utterances. Models should be able to model and ignore structured information based on the query. In Fig. ecorrect-arguuee argmeent), the vir- tual assistant fails to relate 'Henry Moore' to the user's contacts and incorrectly interprets the name as a reference the "[enny Moore Foundation". The gold parse provided in the dataset shows the correct interpretation for the last utterance, leverag- ing the structured context.
'We use a language ID classifier to determine which of the two languages is dominant in given utterance.

---

Dataset
Languages
Multi-tum
Explicit Context
Labeled Conversational
Phenomena
PRESTO
6
Sode-Switching Disfluencies,
User-R Revisions, Coreferences"
TreeDST (Cheng al., 2020)
1
X
Coreferences
SMCalFlow (Andreas al.. 2020)
X
Coreferences, User- -evr-ievsss
MultiWO (Budzianowski et al., 2018)
X
Coreferences"
DSTC10 (Kim et al., 2022)
V
X
Distluencies*
TOP (Gupta et al., 2018)
X
X
MTOP (Li et al., 2021)
X
MASSIVE (FitzGerald et al., 2022)
51
X
SNIPS (Coucke et ul., 2018)
X
X
MultiATIS+ (Xu et al., 2020)
9
X
X
Table 2: Comparison of PRESTO with other recent datasets on or related semantic parsing of task-oriented dialog. denotes phenomena which is implicitly present in the dataset but not explicitly tagged.
Other Statisties. PRESTO covers 34 intents, in- cluding the intent 'Other' which a catch-all in- tent used for all out-of-scope utterances and con- stitutes 11.3% of all examples in the dataset. The average number of the previous user turns is around 2.6, which is representative of the relatively short conversations users tend to have with virtual assis- tants today. Table provides summary statistics for the number of tokens labeled user utterances in each language, demonstrating the large variety in utterance lengths.
droid Developers, which cover a wide spectrum of domains including finance, health, fitness, food, transportation, social, communications, shopping among others.* 4 Our dataset has over 34 unique intents and 370 arguments across 6 languages. Table 6 talks about the intent distribution in the dataset. We use two complementary approaches to collect contextual and non-contextual examples.
Contextual Examples. Data contributors were asked to have conversations with a virtual assistant simulator based on diffcrent sets of instructions for each language and for each type of targeted linguis- tic phenomenon (e.g., sode-wwithhing data collection request targeted single intent and was initialized with potentially relevant structured context, e.g., contacts, notes lists, which optionally be referenced by data contributors in the utterances they produce. The initial seed contacts, notes and lists are authored by native speakers and are shared for all examples one request. For ex- ample, when initiating a call, data contributors may use one of the names in the contacts (e.g., "call mom"), but they may also use the name of a busi- not in the contact list (e.g., "call medonalds"). Each user utterance from the collected conver- sations is exported as a candidate example for se- mantic annotation by trained expert linguists who produce two types of annotations." First, linguists decide whether the example is in scope or not. Out of scope examples include incoherent or nonsen-
There have been multiple re- lated datasets which are listed in Table 2. The existing task-triented dialog datasets can be split by capabilities and representations. The salient ca- pabilities single turn conversational, mono- lingual multilingual, presence of conversational phenomenon, and single-domain vs multi-domain. terms of representations, there are 2 popular paradigms, a direct semantic parse typically con- sisting of intents and slots vs maintaining an ex- plicit dialog state. With the popularity of pretrained text-to-text models, the natural modeling choice for all of these has converged to similar models. In fact, be seen
Related Datasets
Each
context representation stant dialog state for that conversation. While other datasets have focussed on multi-intent and compo- sitional utterances (Cheng et al.. 2020), PRESTO emphasizes on labelled conversational phenomena and explicit context, which are important for mod- cling real-world task-oriented dialogues.
"nttps://developer android. cow/referencee/ app app-actionss actaaiiiinbuullll in-intenss
3 Data Collection
'Due to limited annotation capacity, we were not able to annotate all candidate examples and we prioritized candidate examples which have at least one previous turn. Candidates which were not included were discarded and are not included in this data release.
The scope for data collection is based on the App Actions built-in intents available for 3rd-party An-

---

As can be seen, the overall query generation and annotation process is similar for contextual and non-contextual examples. However, contextual ex- amples require more care in annotation therefore additionally ask trained in-house linguists who also speak the language verify the plausibility of each example and annotations and exclude ones that are out of scope or unclear as discussed earlier. sical utterances, unsupported arguments and out- of-scope intents. Then, they choose the intent and nested arguments as shown in Fig. Finally, they added semantic tags which indicate which linguistic phenomena of interest are expressed in this example, such as, correct-argument cancel-action code-switching, etc.
2. [Phase 1.2] Annotators 3, 4 indepen-
dently validate the query (scope judgment). all 2, 3, 4 agree that the query is valid, then the query goes to phase 2. Upon any disagree- ment, the example discarded.
Annotators and annotate the query independently. their annotations match, the query is considered resolved along with the annotation.
[Phase 2.1]
Non-Contextual Examples. key challenge in representing context real world semantic parsing is that most examples can be parsed without using it, for example utterances like "play music" are typ- ically non-contextual but an utterance like "play workout music" might be contextual if the user has playlist called "workout". This makes it chal- lenging modeling task; context ideally should only be used when relevant. On the one hand, ex- clusively using relevant contextual examples when training a conversational semantic parser results in models which are over-sensitive to irrelevant con- textual features. On the other hand, exclusively using non-contextual training examples where the user utterance is not related to previous turns or structured context teaches the model that the con- text does not matter, and may result in emphasizing arbitrary contextual features which happen to cor- relate with some prediction in the training data. balanced approach is to include both contex- tual and non-contcxtual examples in training and use targeted evaluation sets to understand the im- pact of different ways of representing context. We obtain non-contextual examples by asking linguists produce single-turn utterances along with their gold parses. Then, we pair them with samples of the structured context and previous turns from the contextual conversations (described later in the post-processing section). We denote such synthetic generated context in the data by the tag context: synthetic.
4. [Phase 2.2] 5, 6 disagree, the query is sent to annotator 7 who annotates the query. If 7 matches any of 5 or 6, then the query is resolved to the matching annotation else the example is discarded.
We manually analyzed 6K examples to ver- ify the efficacy of our approach. We as- sessed the correctness of the parse and the plausibility of the user utterance. We used labels to assess the semantic parse qual- ity: :ouuurayyGGODD 'Accuracy: : INTENT", "Accuracy: BAD: SLOT'. We used labels to as- sess the utterance: Scccepaaiilty:HHCG "Acceptability:LOnn
and
"Accuracy: is used when the top- level intent and all slot values are accurate. "Accuracy:BAl INTENT' is used when an intent is not accurate. 'Accuracy: securaacy:BDD BAD: SLOT* is used when the intent is accurate but one or more slot values are not accurate. 'Acceptabilis li is used when the utterance is likely to be spo- ken typed by Assistant users, and is grammat- ically and contextually correct, otherwise use acceptability
Data Quality. We adopt several mechanisms for boosting the data quality of collected data. All our data collection is done in phases. Phase 1 is data collection, where annotators come up with queries pertaining to scenario. Phase 2 is the actual semantic parse annotation for the query. For non-contextual examples, we use crowd compute like platform with native speakers to gen- erate and annotate the data. The process is de- scribed below:
Before post-processing, annotators agreed on utterance plausibility (acceptability) for 92.9% of examples, and agreed on full semantic parse anno- tation (accuracy) for 90.7% of examples.
The breakdown of intent and argument emror in contextual examples can be found Table 3 for five the six languages (Spanish was not included in this analysis due to capacity constraints).
1. [Phase 1.1] Annotator authors a query (contextual or non-contextual).

---

Language
Bad Intent
Bad Argument
4 Experiments
German 2.8%
3.7%
In this section, we demonstrate a few experimental setups which are enabled by PRESTO, provide baseline results for future work, and summarize our findings.
English 2.9%
8.5%
French 0.0%
2.5%
Hindi 1.9%
4.8%
Japanese 4.3%
2.1%
Table 3: Type of accuracy errors in examples by lan- guage from human evaluation.
4.1 Setup
Each example in the dataset designated as train, development or test with respective probabilities 0.50, 0.15 and 0.35, which enables us to have large test sets even when doing focused evaluation on a particular phenomenon of interest as in $4.3. All data splits are provided as part of the data release. We use the t5x library (Roberts et al., 2022) to fine- tune mT5's public checkpoint (Xue al., 2021) the train portion of the data (unless otherwise stated in k-shot experiments).
Post-processing. One of the common annotation errors we found is that code-switched examples often had the wrong language ID (LangID) associ- ated with them. To address this, ID classifier to determine the dominant language code-switched examples.
language
Some arguments (due to fine-grained domain modeling) were often confused for other argu- ments that have similar (but distinct) semantics, e.g., ListItem vs. ExistingListIten resulting in inconsistent annotations and contributing to the error rate discussed earlier. To address this, we merged pairs of argument names where the distinc- tion rarely matters in practice. To make our data uniform, we add synthetic context and previous turns to non-contextual examples. Examples that are missing previous turns are augmented reuse previous turns from other examples by first sam- pling the number of previous turns using a negative binomial distribution with r 5,p = 0.8, then sampling one of the collected conversations that has this many turns. Examples that are missing given type of structured context, e.g., contacts, are augmented by randomly sampling contact names from related argument values specified by our data contributors, which results in realistic and diverse contexts.
For all experiments, we report exact match ac- curacy of the predicted semantic parse, which gives the model one point for each example where the intent and all the arguments are correctly pre- dicted and zero points otherwise. All experiments are based on the mT5-Base model (580M param- eters), except the scaling experiments in $A.1 which demonstrate the effect of model scaling mT5-Larre (1.2B parameters), mT5-XL (3.7B pa- rameters), and mT5-XXL (13B parameters). All models (except the monolingual models discussed in $4.5) are fine-tuned on the union of training ex- amples from all languages in PRESTO. Few shot experiments for code switching, user revisions and disfluencies all share the same training set, e.g., the 5-shot model across all three sections consist of 5 code-switching examples, cancellations, within- turn corrections, intent cross-turn corrections, 5 argument cross-turn corrections and disfluency examples, in addition to all examples which do not represent any of these phenomena.
We do another quality check measure Accu- racy and Acceptability after procepssgsssing by ran- domly sampling 500 queries from each language (except Spanish due to capacity constraints). We ask 3 independent annotators (same as the ear- lier quality check) to rate the queries. After post- processing, we find that linguists agree, defined by 2 out of annotators independently agrecing, utterance plausibility (acceptability) for 98.8% 8 of examples and agree on full semantic parse anno- tation (accuracy) for 93.24% of examples. This shows our post protessocsssing decreased the data noise significantly.
Features. The input sequence fed to the model consists of the last user utterance which needs be parsed, followed by previous turns (both user and assistant turns) in reverse chronological order, followed by some representation of the structured context (more on this in $4.4), with separator token between consecutive fields. An example is shown in Figure 4.
We do a rough hyperparam- eter tuning and fine-tune the model for 20K steps.
Hyper-parameters
'https: rhree//ooooooooeeeeeeraatttttt main/tt5/eerrmmlle///////tt/////

---

Input:
examples with the relevant phenomena. We trained several mT5-base models on different training sets with varying numbers of examples with each lin- guistic phenomenon: 0, 5, 25, 125, 625, 3125, and 15,625 examples We then evaluated the models on a test set with only examples of the relevant phenomena.
No, Henry Moore in contacts | Calling Henry
Noore Foundation (Henry Moore Studios Gardens)
Place call to Henry Moore [SEP] Lists: School
Supply, Cleaning, [SEP] Contacts: Charlotte
Taylor, Henry Moore, [SEP] Notes: Meeting
tomorrow,
Output: Initiate_call( callee Personal_contact( person Henry Moore )
Code-Switching As seen in Fig, 5, when no code-switching examples were used in training, the model's exact match accuracy ranged between 56-72%, performing best on Japanese and worst on Hindi. Even at 0 shot, we saw a relatively high performance for this phenomenon. We hypothe- size that this is due to using LangID to identify code-switching utterances, which may cause code- switched utterances to leak into the training set.
Figure 4: Example serialization and output of features. The top box shows the encoding for the first input in Fig. 1. The output sequence depth-fire firt-firstt traversal of the semantic parse.
We use the Adafactor optimizer with 0.8 decay rate and 0 step offset. We have a max input length of 512 for training and 1024 for inference and a batch size of 128.
User revisions Similar to code-switching, this setting has the training dominated by examples without user revisions, while the test set only in- cluded examples with user revisions. The zero-shot exact match accuracy ranged between 18-28%. Un- like code switching, adding only 25 examples of each type of user revisions notably improved the performance on user revisions. After the first few examples, the performance improved linearly as the number of examples with relevant phenomena grew exponentially from 25 to 125 to 625. The performance slowed down when the number of ex- amples reached 15K.
4.2 Overall Results
Before diving into focused evaluations on various phenomena of interest, it is instructive to examine the overall model performance on this dataset when trained on 100, 1K, 10K examples, as well as the full the training set. The results are shown in Fig. 5, and they demonstrate linear increase in model performance the number of training examples grows exponentially.
We next assess how difficult it is for mT55based models to parse examples with different linguis- tic phenomena. Table 4 shows the exact match accuracy for a multilingual model trained on ex- amples without these phenomena. Each column corresponds to different test set with examples that have user revisions, disfluent utterances. code- switching, none of the above. Unsurprisingly, the models perform much worse on examples with these phenomena when they are not exposed to them in training. In contrast, when the model is exposed to lot of training examples with these phenomena, as in Table 7, the performance gap across the different test sets is smaller, except for code-switching.
Disfluencies Here, we turn to disfluent utterances and try to estimate how well our models fare on them with various number of disfluent examples in the training set. Fig. 5 shows the results for the focused test set with disfluent utterances. In the zero-shot setup, the performance starts at higher accuracy range for all languages, compared to use revisions. As the number of training examples grows exponentially, we observe a linear increase in exact match accuracy for all languages.
4.4 Structured Context
PRESTO provides an opportunity examine how structured context can be used improve conver- sational parsing models. We use a focused test set that highlights context by only including contextual examples (first approach in $3)).
4.3 Results on Linguistic Phenomena
We evaluated how well an LLM-based model parses examples with different linguistic phenom- ena when fine-tuned on a large number of mul- tilingual examples but only a limited number of
For user revisions, 5 examples would mean 5 examples with intent corrections + 5 with argument corrections + + with within-turm corrections + 5 with cancellations

---

dc-DE
hi-IN
es-ES
en-US
fr-FR
ja-JP
100
100
Exact Match Accuracy (%)
80
Exact Match Accuracy (%)
80
70
703
60
60
40
40
30
30
20
20
5
25
125
625
3125
15625
0
5
25
125
625
3125
15625
Disfluencies Training Examples
Code-Switching Training Examples
(a)
(b)
100
100
90
90
Exact Match Accuracy (56
Exact Match Accuracy (%)
70
60
60
50
50
40
20
20
10
10
900
5
25
125
625
3125
15625
1000
10000
274743
User Revisions Training Examples
Overall Training Examples
(c)
(d)
Figure 5: Scaling plots the baseline multilingual mTT5 model on test set PRESTO for: (a) disfluencies, (b) code-mixing, (c) user revisions, and (d) all inclusive (i.e. sampling uniformly from the overall PRESTO training data mixture).
We linearize and prepend the context tokens to the T5 parser input with [SEP] tokens as shown in the example presented in 4.1. Initially, we simply linearized the entire context and added that as a feature. We noticed that naively prepending all the context had too much noise and was not resulting in performance improvements. Hence to represent context in the input sequence, first identified context notes, contact names and list names which had trigram similarity greater than 0.6 with the last user utterance and only used the last two turns. We call this filtered context representation" since it filters out parts of the context which are unlikely to provide useful information for the parser.
contextual examples to the training set. In particu- lar, we fine-tune the mT5- Base model with filtered context representation on all the non-contextual training examples addition to 10, 100, 1K or 10K contextual training examples. Table 5 provides average performance for 5 languages with and with- out context. We notice very little improvement with addition of the context features. Our interpretation for this result is that, even though the data contribu- tors were encouraged to use the structured context when conversing with the virtual assistant, most utterances can be understood without referencing the structured context. For example, it is easy for the model to parse the utterance "add apples to my shopping list", with no knowledge about what lists the user (See $3 for more details on contextual
First, we examine how the performance in- creases on the focused test set as we include more

---

Language
No Phenomenon
User Revisions
Disflency
Code-Switching
German
81.76
23.56
64.91
56.81
English
85.16
22.45
63.78
67.43
Spanish
81.26
19.94
61.31
62.60
Japanese
83.41
22.95
54.31
71.71
Hindi
76.60
26.94
54.89
58.10
French 84.27
15.25
67.42
64,21
Overall
82.338
21.31
61.26
63.16
Table 4: Exact match accuracy results (%) on the test set for the zero-shot multilingual mT5 model (i.e. trained on all examples with no marked phenomena).
vs. non-contextual examples.)
100
Number of Examples
Non Con-Contextual Model
Contextual Model
Exxet Match Accuracy (5)
80
70
21.23
23.94
60
100
40.84
41.90
50
1000
64.29
64.93
10000
80.28
80.50
30
Table 5: Average performance (exact match accuracy (%)) on the contextual test set across all the languages for contextual vs non contextual models. It can be seen that in low resource setup contextual models outperform non-contextual model, i.c. modeling context helps.
20
Monolingual
10
Multilingual
100
1000
10000
274743
Per Language Training Examples
Figure Average performance of multilingual and monolingual models. In low resource setting, we can see that multilingual model performs better than a monolingual one.
4.5 Monolingual vs. Multilingual Models
previous sections, we train single parser on training examples from all languages in PRESTO, as proposed in Ammar et al. (2016). A more tra- ditional approach is to train several monolingual models. Figure 6 shows how the monolingual mod- els compare to multilingual models with varying amounts of training data. Our results show that in lower data regimes there is clear gap between monolingual and multilingual models, but when using all training instances in the dataset, mono- lingual and multilingual models converge towards similar performance, in terms of overall accuracy.
does not yield large improvements contextual examples. We also notice that multilingual models perform better than monolingual models in lower data regimes.
Open Questions. By introducing new dataset with unique characteristics, this work deliberately asks more questions than it answers. Some of the obvious modeling questions which this dataset help answer are: How do we train models which perform well on conversational phenomenon like code-switching, user revisions and disfluencies. with a relatively small number of training examples that exhibit these phenomena? How do we identify examples for which context is critical to under- standing the user's utterance? How can structured context be represented in our models most opti- mally? How can we increase cross-lingual super- vision in multilingual models make them more data efficient? We hope that PRESTO useful in answering these and other questions about conver- sational semantic parsing.
5 Conclusion
We introduce PRESTO, a over 550K-examples, six-language dataset for parsing realistic task- oriented dialogs. PRESTO is enriched with a high representation of contextual examples, code switch- ing, user revisions and disfluencies. Our initial re- sults demonstrate that it is possible for our models perform well various linguistic phenomena, but only when the model is exposed to very large number of training examples with the given phe- nomenon. We observe that feeding a simple rep- resentation of the structured context to the model

---

Limitations
Yu Su, Zachary Tellman, Sam Thomson, Andrei Vorobev, Izabela Witoszko, Jason Wolfe, Abby Wray. Yuchen Zhang, and Alexander Zotov. 2020. Task- Oriented Dialogue as Dataflow Synthesis. Transac- lions of the Association for Computational Linguis- tics, 8:556-571.
The PRESTO dataset was collected before the re- lease of the ChatGPT APIS, resulting in the absence of any performance testing on the newly collected phenomena during data collection. Furthermore, the extensive size of the dataset poses challenges in testing models such ChatGPT on PRESTO with- out incurring supplementary expenses. Pan al. (2023) employed a multi-turn prompt framework to evaluate ChatGPT's performance on comparable datasets and reported a Joint Goal Accuracy (JGA) of 64.23 on MultiWOZ2.44 compared to the state- of-the-art (SOTA) finetuned score of 75.90. Table 2 demonstrates that MultiWOZ is less intricate than PRESTO. Thus, consider the PRESTO dataset to remain challenging benchmark for such mod- cls. Subsequently, future research can focus on conducting additional modeling experiments.
Pawel Budzianowski and Ivan Vulić. 2019. Hello, It's GPT-2 - How Can I Help You? Towards the Use of Pretrained Language Models for Task-Oriented Dialogue Systems. In Proceedings of the 3rd Work- shop on Neural Generation and Translation, pages 12- Hong Kong. Association for Computational Linguistics.
Pawet Budzianowski, Tsung- Hsien Wen, bo-Hsiang Tseng, lñigo Casanueva, Stefan Ultes, Osman Ra- madan, and Milica Gašić. 2018. MultiWOZ - A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Yask-Oriented Dialogue Modelling. In Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 501 6-5026, Brussels, Belgium. Association for Computational Linguistics.
Ethical Considerations
Justine Cassell, Tim Bickmore, Lee Camphell, Hannes Vilhjalmsson, and Hao Yan. 2000. Human Conversa- tion as System Framework: Designing Embodied Conversational Agents. Embodied Conversational Agents, pages 29 63.
The PRESTO dataset does not incorporate copy- righted material or otherwise violate intellectual property of third parties since all the raw text was directly collected from trained data contributors who received fair market compensation for their work. All data contributors were vetted to be native speakers for the contributed data, although their na- tionalities may vary. See Section 3 for more details on data collection, and Section 2 for more details on data characteristics.
Jianpeng Cheng, Devang Agrawal, Héctor Martinez Alonso, Shruti Bhargava, Joris Driesen, Federico Flego, Dain Kaplan, Dimitri Kartsaklis, Lin Li, Dhivya Piraviperumal, Jason D. Williams, Hong Yu. Diarmuid Ó Séaghdha, and Anders Johannsen. 2020. Conversational Semantic Parsing for Dialog State Tracking. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8107-8117, Online. Association for Computational Linguistics.
References
Anmol Agarwal, Jigar Gupta, Rahul Goel, Shyam Upad- hyay, Pankaj Joshi, and Rengarajan Aravamudhan. 2022. CST5: Data Augmentation for Code-Switched Semantic Parsing. arXiv preprint arX1v:2211.07514.
Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Calta- girone, Thibaut Lavril, et al. 2018. Snips Voice Plat- form: an embedded Spoken Language Understanding system for private-by by-design voice interfaces. arXiv preprint arXiv: 1805. 10190.
Waleed Ammar, George Mulcaire, Miguel Ballesteros, Chris Dyer, and Noah A. Smith. 2016. Many Lan- guages, One Parser. Transactions of the Association for Compulational Linguistics, 4:431
Mihail Eric, Rahul Goel, Shachi Paul, Abhishek Sethi, Sanchit Agarwal, Shuyang Gao, Adarsh Kumar, Anuj Goyal, Peter Ku, and Dilek Hakkani-Tur. 2020. Mul- tiWOZ 2.1: A Consolidated Multi-Domai Domain Dia- logue Dataset with State Corrections and State Track- ing Baselines. In Proceedings of the Twelfih Lan- guage Resources and Evaluation Conference, pages $22-428, Marseille, France. European Language Re- sources Association.
Jacob Andreas, John Bufe, David Burkett, Charles Chen, Josh Clausman, Jean Crawford, Kate Crim, Jordan DeLoach, Leah Dorner, Jason Eisner, Hao Fang, Alan Guo, David Hall, Kristin Hayes, Kellie Hill, Diana Ho, Wendy Iwaszuk, Smriti Jha, Dan Klein, Jayant Krishnamurthy, Theo Lanman, Percy Liang, Christo- pher H. Lin, Ilya Lintsbakh, Andy McGovem, Alek- sandr Nisnevich. Adam Pauls, Dmitrij Petters, Brent Read, Dan Roth, Subhro Roy, Jesse Rusak, Beth Short, Div Slomin, Ben Snyder, Stephon Striplin,
Jack FitzGerald, Christopher Hench, Charith Peris, Scott Mackie, Kay Rottmann, Ana Sanchez, Aaron Nash, Liam Urbach, Vishesh Kakarala, Richa Singh, Swetha Ranganath, Laurie Crist, Misha Britan,
hhtps///oooma..ccm///////nonoooo and whisper-apis

---

Wouter Leeuwis. Gokhan Tur, and Prem Natara- jan. 2022. MASSIVE: A IM-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Divess Languages.
Joshua Newlan, and Andrea Gesmundo, 2022. Scal- ing Up Models and Data with t5x and seqio. arXiv preprint arXiv:2203. 17189.
Weijia Xu, Batool Haider, and Saab Mansour. 2020. End-to-End Slot Alignment and Recognition for Cross-Lingual NLU. In Proceedings of the 2020 Conference Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 5052-50000000 5052-5063, On- line. Association for Computational Linguistics.
Aditya Gupta, Jiacheng Xu, Shyam Upadhyay. Diyi Yang, and Manaal Faruqui. 2021. Disf-QQ:: A Benchmark Dataset for Understanding Disfluencies in Question Answering. In Findings of the Associa- tion for Compulational Linguistics: ACL-IJCNLP 2021, pages 3309-3319, Online. Association for Computational Linguistics.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A Massively Multilingual Pre-trained Text-to-Texl Transformer. In Proceed- ings of the 2021 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, pages 483- 498, Online. Association for Computational Linguis- tics.
Sonal Gupta, Rushin Shah, Mrinal Mohit, Anuj Ku- mar, and Mike Lewis. 2018. Semantic Parsing for Task Oriented Dialog using Hierarchical Represen- tations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2787-2792, Brussels, Belgium. Association for Computational Linguistics.
Seokhwan Kim, Yang Liu, Di Jin, Alexandros Papan- gelis, Behnam Hedayatnia, Karthik Gopalakrishnan, and Dilek Hakkani- -ür.. 2022. mhowledge-groundee task-oriented dialogue modeling spoken conversa- tions track at DSTC10. In AAAI 2022 Workshop on Dialog System Technology Challenge.
Haoran Li, Abhinav Arora, Shuohui Chen, An- chit Gupta, Sonal Gupta, and Yashar Mehdad. 2021. MTOP: A Comprehensive Multilingual Task- Oriented Semantic Parsing Benchmark. In Proceed- ings of the 16th Conference of the European Chap- ler of the Association for Compulational Linguistics: Main Volume, pages 2950-2962, Online. Association for Computational Linguistics.
Wenbo Pan, Qiguang Chen, Xiao Xu, Wanxiang Che, and Libo Qin. 2023, A preliminary evaluation of chatgpt for zero-shot dialogue understanding. arXiv preprint arXhv:2304.04255. 044266.
Panupong Pasupat, Yuan Zhang, and Kelvin Guu, 2021. Controllable Semantic Parsing via Retrieval Augmen- tation. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7683-7698, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sha- Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, Curtis Hawthorne, Aitor Lewkowyez, Alex Salcianu, Mare van Zee, Jacob Austin, Se- bastian Goodman, Livio Baldini Soares, Haitang Hu. Sasha Tsvyashchenko, Aakanksha Chowdh- cry, Jasmijn Bastings, Jannis Bulian, Xavier Gar- cia. Jianmo Ni, Andrew Chen, Kathleen Kenealy, Jonathan H. Clark, Stephan Lee, Dan Garrette, James Lee-Thorp. Colin Raffel, Noam Shazeer, Marvin Ritter, Maarten Bosma, Alexandre Passos, Jeremy Maitin-Sheppe Shepard, Noah Fiedel, Mark Omernick, Bren- nan Saeta, Ryan Sepassi, Alexander Spiridonov,

---

A
Appendix
A.1 Model Scaling
We present the results on the overall dataset as well the model scaling results in this section. Table 7 presents results on the test sets if we use all the available training data.
The experiments presented so far use mT5-Base since it incurs lower cost than larger models and provides reasonable performance on many NLP tasks. In this section, we examine larger model sizes impact the results on this dataset. As a case study, we focus on the 125-shot models for code switching, disfluencies and user revisions. Fig. 7 demonstrates the impact of increased model size on the overall test results, As expected, the larger models consistently give better results but are still lower than the best results we were able to achieve by using the full training set with mT5-Base (dis- cussed carlier in Fig. 5). Fig. 7 demonstrates similar pattern for the focused evaluation on user revisions, and we phenomena. These results confirm that m strikes good balance between model size and per- formance, and we recommend using mT5--Base for extensions of this work.
similar pattern for other

---

Intent
German
English
Spanish
French
Hindi
Japanese
2779
2931
3382
3148
2122
3527
Add item to list
944
2959
2913
986
1958
2503
BuyEventTickets
2550
2463
2883
2842
1697
3049
Cancel
1479
3029
2061
1775
2318
2525
Cancel ride
2252
2353
3239
3047
2053
3098
Check Bores_ordr___e staauus
2700
2877
3448
3455
2136
3934
Create _ist
844
594
745
1063
778
1023
Create note
Create_note
1566
1700
1903
1819
1568
1998
Find_ Ladd_parking
4562
1210
4387
2345
3117
2692
GetGenericBassnessSSyyy
3130
2613
2782
2904
1676
3139
Get bill
2249
2466
2922
2582
1857
3275
Get health_stae
2477
2512
2827
2847
1982
2879
Get list
701
705
698
676
856
761
Get message tenttet
2102
2142
2371
2343
1320
2879
Get_note note
1737
1905
2022
2277
1978
2138
Get-product
2341
2076
2483
2438
1574
3344
Get peetsseurrtty_pee _prcc
2175
2333
2707
2425
1752
3245
Initiate_call
1822
4292
2239
1965
1807
2250
Log_exercise exercise
2602
2708
2050
2663
1962
3049
Log_nutrittne
2219
2459
2667
2605
1786
3204
Open_ape app
2488
2497
3367
3076
2069
3643
Order menu item
224
860
659
736
155
740
Order ride
2175
2342
2857
2992
1809
2993
Other
8457
17499
5650
10937
9163
13737
Pause exercise
2821
2995
3109
3149
1830
3495
Pay_bill
2280
2304
2837
2823
1834
3048
Play plme
2640
2667
2889
3048
1941
3074
Post sost_message
2597
2224
3065
3278
2049
3529
Record_ vvdeo
2609
2295
3317
3369
1934
3864
Resume_eeercise
2367
2060
3136
2929
1991
2896
gond_ddgitalovl
2307
3127
2804
2652
2016
2905
Start_exercis
2719
3030
3272
3182
2031
3311
Stop_exercise
2719
2814
3207
3127
2078
3432
Take photo
3950
2630
5266
4367
4910
4349
Overall
83584
95671
96164
95870
72107
109528
Table 6: Intent distribution by language for PRESTO.
Language
No Phenomenon
User Revisions
Disfluency
Code-Switching
German
82.41
81.33
84.19
72.50
English
86.
85.36
86.20
75.99
Spanish
82.95
82.26
82.81
72.22
Japanese
84.91
82,22
80.46
84.41
Hindi
78.30
73.24
75.49
73.24
French
85.42
82.99
85.82
75.98
Overall
83.66
81.85
82.92
75.88
Table 7: Exact match accuracy results (%) for the multilingual mT5 model trained on the full training set of PRESTO.

---

de-DE
hi-IN
es-ES
en-US
fr-FR
ja-JP
70
35
Exact Match Accuracy (%)
Exact Match Accuracy (%)
65
25
20
60
15
55
10
0.58
1.3
3.7
0.58
1.3
3.7
Overall data
User revisions
Exact Match Accuracy (%)
Exaet Match Accuracy (%)
70
65
60
60
55
53
50
50
0.58
1.3
3.7
13
0.58
1.3
3.7
13
Code-mixing
Disfluencies
Figure 7: Scaling plots for increasing the model capacity. This scaling study was done using 125 shot training set. As expected larger models generally perform better.


Reviews:
[{"review_id": "cnLiYzFSjd", "reviewer": "Reviewer_WWoB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review the paper draft. To your point, it’s been hard to keep up with the latest SOTA as we iterate on our experimental design, in the post-ChatGPT world we live in. We’d like to emphasize that the core contribution of this work is the largest multilingual dataset for task-oriented dialog systems with original (i.e. not translated) conversations, which also includes a high representation of disfluencies, user revisions, and code-switching. We argue that data contributions are at least as significant as modeling contributions and that the modeling and experiments part of the paper are only intended to demonstrate the utility of the data resources we carefully planned, collected, annotated, and reviewed in a systematic and ethical way as described in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1)\tThe paper's main contribution is its inclusion of diverse challenges found in real-world TOD tasks, such as disfluencies, revisions, and code-switching for multilingual tasks. This enhances the realism of the TOD dataset, which is often lacking in existing datasets.\n2)\tThe experiment setup is commendable, incorporating exact match accuracy evaluation, diverse training sets with varying example numbers for each linguistic phenomenon, and comparisons between contextual and non-contextual models, as well as monolingual and multilingual models. These choices lead to several interesting findings discussed in the conclusion.\n", "reasons_to_reject": "The baseline model used in this paper appears to be weak, as all experiments solely rely on fine-tuning the mT5 model without any comparisons to other SOTA models in the TOD domain.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces a public dataset PRESTO, with over 550K contextual multilingual conversations between humans and virtual assistants. A series of experiments are performed with mT5-based baseline models.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dH3TkvYy76", "reviewer": "Reviewer_ePrg", "summary": "", "questions": "A: For the code-switching examples, is English always one of the languages? Or can there be e.g. Spanish-French code-switching?\n\nB:  On p. 5 the \"Post-processing\" step is found to improve acceptability (92.9%->98.8%) and accuracy (90.7%->93.24%). It would be helpful to understand which actions contributed to the improvements. The two actions listed in the main paper are relabelling for language ID and adding past turns for non-contextual examples; the appendix also mentions that confusable labels were merged (this should be in the main section too) - it's not immediately clear why any of these should improve acceptability of the user utterance since they do not change the query text?\n\nC: On p.8 the authors state \"most utterances can be understood without referencing the structured context\". This is likely less true for more ambiguous utterances, e.g. \"show Bob\" rather than \"show Bob in my contacts\" or \"show my note called Bob\". Are such ambiguities captured in the dataset?\n\nD: As shown in Figure 4, a turn contains the conversational history of user and system utterances. What is the space of possible system utterances? Does the system sometimes ask questions to prompt, confirm, disambiguate? Does this influence the interpretation of the target turn?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and questions. Below are the answers to your questions:\n\n - A) Yes. English is always one of the languages used in code-switched utterances. For this task, we selected annotators who were fluent in both English and the local language with which code-switching occurs.\n\n\n - B) We will add a discussion on this in the revised version and will also include the discussion on merging the labels in the main section when given the extra page in the camera-ready version. \n\n\n - C) In the annotation process, annotators were instructed to create queries based on the structured context. As a result, we have queries that refer to the context. However, it is difficult to determine how often ambiguity arises in parsing the query due to the context alone by simply examining the data. Our results suggest that most of the time, the utterance can be parsed without referencing the context.\n\n\n - D) The system does ask clarifying questions to confirm and disambiguate user queries (an example is Figure 1 (cancel-action)). Our annotators were asked to continue the conversation with the assistant until they had reached a targeted intent in each of the data collection requests. We consider the entire context to influence the interpretation of the target turn and hence all ‘previous_turns’ are used in the input to predict the output.\n\nOther comments:\n - Average utterances: Thank you for pointing that out. We will add the variances in the revised version to make that point clear.\n\n - p4 contextual examples: Yes. We provide the entire sequence of utterances in a metadata field called ‘previous_turns’, so it is indeed possible to recover the original conversation sequence. We use the entire conversation in the input as seen in Figure 4.\n\n - p4, Non-contextual examples: We computed the accuracy metric on a sample after post-processing to see if the system response still remains valid. As mentioned in the the post-processing section, our numbers indicate that this is not an issue.\n\n - Indeed, PRESTO does not contain multi-intent conversations. We will add the relevant suggested information in the revised version of the paper.\n\nWe thank you for your important questions and suggestions. We will take these points into consideration when writing our revised version of the paper with an extra page.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- As in many areas, progress in conversational system research is constrained by the availability of large realistic datasets. Since PRESTO contains some phenomena not present in other datasets (disfluencies, code-switching) and is larger than most, I expect it to become a widely-used benchmark dataset.\n\n- The design of the data, including the focus on phenomena like disfluencies and structured context, is clearly informed by the experience of delivering language understanding systems to real users of virtual assistants. This perspective is of great value to the broader research community, including academic researchers who may have less exposure to the practical aspects of building high-quality assistants at scale.\n\n- The paper is very clearly written.\n\n- The experiments make sense and show meaningful insights into the task. E.g. the authors show that models trained on data without disfluencies find it difficult to understand data containing them.\n", "reasons_to_reject": "No significant weaknesses. ", "questions_to_authors": "A: For the code-switching examples, is English always one of the languages? Or can there be e.g. Spanish-French code-switching?\n\nB:  On p. 5 the \"Post-processing\" step is found to improve acceptability (92.9%->98.8%) and accuracy (90.7%->93.24%). It would be helpful to understand which actions contributed to the improvements. The two actions listed in the main paper are relabelling for language ID and adding past turns for non-contextual examples; the appendix also mentions that confusable labels were merged (this should be in the main section too) - it's not immediately clear why any of these should improve acceptability of the user utterance since they do not change the query text?\n\nC: On p.8 the authors state \"most utterances can be understood without referencing the structured context\". This is likely less true for more ambiguous utterances, e.g. \"show Bob\" rather than \"show Bob in my contacts\" or \"show my note called Bob\". Are such ambiguities captured in the dataset?\n\nD: As shown in Figure 4, a turn contains the conversational history of user and system utterances. What is the space of possible system utterances? Does the system sometimes ask questions to prompt, confirm, disambiguate? Does this influence the interpretation of the target turn?", "paper_topic_and_main_contributions": "This paper presents PRESTO, a large (550k example) dataset for conversational semantic parsing. The dataset covers multiple languages and conversational phenomena, inspired by the kind of requests made by users of real conversational systems. The first half of the paper describes the design of the dataset and the annotation process; the second half demonstrates how the dataset can be used in experiments to shed light on what makes it challenging to build accurate natural language understanding systems.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- line 168: I don't follow how Table 1 (which reports averages and totals rather than ranges and variances) demonstrates \"the large variety in utterance lengths\"\n\n- p.4, \"Contextual Examples\": in the first step data contributors were told which targeted phenomenon (phenomena?) to include, e.g. code switching. Then linguists were asked to label annotations with \"semantic tags which indicate which linguistic phenomena of interest are expressed in this example\". If we knew already which phenomena are included, why do we need to check again - is it to verify that the data  contributors followed the instruction, or to label each turn inside a conversation?\n\n- p.4, \"Contextual Examples\": these examples originate in full conversations and are then split up into utterances. Is it possible to group utterances and recover the original conversation sequence?\n\n- p.4, \"Non-Contextual Examples\": to create conversations the authors concatenate a single-turn utterance with a set of previous utterances and structured context. It is possible that the presence of such context could change the interpretation of the single-turn utterance. Was any checking done to prevent this?\n\n- Table 2 shows that PRESTO has features that other datasets do not. For transparency it would be good to mention features that other datasets have but PRESTO does not. For example, TreeDST contains multi-intent conversations and compositional utterances - if I understand correctly (p. 4, \"each data collection request targeted a single intent\") such phenomena are not included in PRESTO. I'd also suggest including the size of the various datasets in turns and/or conversations.\n\n"}}, {"review_id": "eFAPQ1KAcx", "reviewer": "Reviewer_8Nj3", "summary": "", "questions": "A. Does the dataset include audio? (I'm not completely sure whether the conversations were typed or transcribed)", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments on our work. Our dataset does not include audio. The conversations were typed by the annotators. We will clarify this in the revised version. Due to limited space, the section on the influence of disfluencies in 4.3 was omitted. We will add it in the camera-ready version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Large annotated resources are always relevant for research. Furthermore, in this case, the dataset focuses on several linguistic phenomena that are not typically considered in previously existing datasets with conversations between humans and virtual assistants. Thus, it is particularly relevant, especially considering the multiple languages covered in the dataset. Finally, considering that data collection and annotation included multiple filtering steps to ensure quality, it is expected that the dataset is actually representative of the phenomena that occur in interactions between humans and virtual assistants.", "reasons_to_reject": "I don't see any major reason to reject the paper.", "questions_to_authors": "A. Does the dataset include audio? (I'm not completely sure whether the conversations were typed or transcribed)", "paper_topic_and_main_contributions": "This paper presents a new multilingual dataset of conversations between humans and virtual assistants. It includes more than 550k annotated user utterances. The utterances are all by native speakers and cover six different languages of multiple families. Each conversation is also paired with context information (e.g. contacts, notes, lists), which can be referenced by the user. In addition to intents and slots, each utterance is also annotated for disfluencies, code switching, and user revisions. The data collection and annotation included several annotation and filtering steps to improve the quality of the dataset and its annotations. Preliminary experiments using fine-tuned versions of the multilingual T5 model show that these models can be robust to the multiple linguistic phenomena that occur in the dataset, but only when exposed to a large amount of training examples featuring those phenomena.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Section 4.3 is missing a discussion regarding the influence of disfluencies. "}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a new multilingual dataset of task-oriented dialogues with structured contextual information (user contacts, lists).

**Pros**: All reviewers agree the dataset will be of significant value to the community, with most reviewers emphasizing the dataset's focus on phenomena which is absent from most current benchmarks; e.g., disfluencies, code-switching, and revisions. Experiments are considered "commendable" and "show real world insights into the task". The dataset is considered to be "clearly informed by [real-world] experience" and one which "enhances the realism... which is often lacking in existing datasets."

**Cons**: One reviewer raises concerns about the caliber of the model used for baselines.
Decision: Accept-Main