GrounDialog: A Dataset for Repair and Grounding in Task-oriented Spoken Dialogues for Language Learning
Year: 2023

Abstract:
Improving conversational proficiency is a key target for students learning a new language. While acquiring conversational proficiency, students must learn the linguistic mechanisms of Repair and Grounding (R&G) to negotiate meaning and find common ground with their interlocutor so conversational breakdowns can be resolved. Task-oriented Spoken Dialogue Systems (SDS) have long been sought as a tool to hone conversational proficiency. However, the R&G patterns for language learners interacting with a task-oriented spoken dialogue system are not reflected explicitly in any existing datasets. Therefore, to move the needle in Spoken Dialogue Systems for language learning we present GrounDialog: an annotated dataset of spoken conversations where we elicit a rich set of R&G patterns.

heading: 1 Introduction and Motivation
text: Many conversations are impromptu back-and-forth interactions that often have no prior preparation or review. As a result, conversational breakdowns (Benner et al., 2021; Li et al., 2020) may occur due to minor misinterpretation, mishearing, misspeaking, or a general lack of common ground (Traum, 1994). Interlocutors use Repair mechanisms (Albert and de Ruiter, 2018) to detect and resolve communicative problems during conversations; and Grounding mechanisms to establish common ground. For example, we often ask our interlocutors to repeat what they said, explain themselves, request clarifications, etc. Such processes arise proactively or when the initial communication attempt has failed, during which modification and revision to the previous utterances are needed to proceed the conversations naturally.
According to Long (1983), R&G is meaningful in the following perspectives: 1) repair the dis-
*This work was done while first author was an intern at ETS.
course when breakdown occurs and 2) avoid conversational breakdowns. Table 1 shows an example dialogue between low-proficiency (LPS) and high-proficiency (HPS) English speakers, where LPS paraphrases themselves to repair the discourse when trouble occurs. Besides, speakers usually try their best to avoid breakdowns in conversations. Based on Long (1983), there are plenty of strategies they can adopt to prevent the breakdowns during communications: 1) relinquish topic control; 2) simplify topic by asking "yes-no" questions; 3) confirm comprehensions of speakers before proceeding, etc.
From the perspective of a language learner, dialogues serve as important media in language acquisition and learning (Eszenyi and van der Wijst, 2006). When language learners chat with highproficiency speakers, language learners make considerable efforts to ground what they have to say (Eszenyi and van der Wijst, 2006). More specifically, the low-proficiency speakers (LPS) attempt to negotiate the meanings of conversations with high-proficiency speakers (HPS). According to Foster and Ohta (2005) and Cook (2015), interactional processes including negotiation for meaning and various kinds of repair and grounding are among the many ways learners gain access to the second language acquisition. Besides, LPS can also en-
300
hance their language skills, general communication skills and cultural knowledge during the conversations with HPS (Eszenyi and van der Wijst, 2006).
While R&G is common in nearly all conversations, it is particularly important for language learners as learners are still building up the full understanding of the language. They may also bring R&G influences of their primary language into the language they are learning. It is also possible that low-proficiency speakers (or language learners) employ additional or different R&G mechanisms than high-proficiency speakers of a language. Therefore, there is a lot to know about R&G mechanisms from low-proficiency speakers.
In this paper, we present a dataset that can help linguists and other researchers with several novel linguistic tasks such as identifying R&G patterns. Further, while repair and grounding is an important linguistic mechanism, it is rarely reflected explicitly in the design of spoken dialogue systems that aim to help people learn a new language. Our dataset can fill this gap by allowing researchers to model dialogue state tracking with R&G, generating responses with R&G turns, etc.
We collected this dataset by connecting a high-proficiency speaker and a low-proficiency speaker on a crowd sourcing platform. The highproficiency speaker played the role of a human resources (HR) assistant in a wizard-of-oz style and was tasked to convey information about an interview. The low-proficiency speaker played the role of an interviewee and was tasked with finding specific information about the same interview through their conversation with the highproficiency speaker. While R&G may occur as a course of natural conversation, we further induced it by giving the interlocutors some conflicting and incomplete pieces of information. We collected the voice of the low-proficiency speaker and the text responses of the wizard.
To the best of our knowledge, GrounDialog dataset is the first task-oriented dialogue dataset specifically tailored for repair and grounding in spoken conversations between high-proficiency and low-proficiency speakers. Each dialogue in the dataset is transcribed by human experts and contains vocal markers and disfluencies, such as "uh" and "um". It is annotated with R&G types, intents, and slots that are relevant to dialogue state mapping tasks. Hence, GrounDialog can be used to develop a task-oriented conversational agent, equipped with
the R&G ability to detect communicative trouble, and adopt certain strategies to repair the discourse when trouble occurs.
The rest of the paper presents related work, details of the data collection process, the data annotation scheme, analyses of the data, and initial model benchmarks.

heading: 2 Related Work
text: As indicated in Dorathy and Mahalakshmi (2011), task-based language teaching (TBLT) puts emphasis on the utilization of tasks as the critical element in the language classroom given that tasks can offer better contexts for active language acquisition and second language promotion. From the perspective of dialogue systems, it is the task-oriented dialogue (ToD) that can help language learners achieve their proficiency goals through task completion. Previous dialogue systems have shown great promise in increasing second language acquisition proficiency. Bibauw et al. (2019) provide an overview of all spoken dialogue systems for language learning. TimpeLaughlin et al. (2022) have compared learning language via role-play with a spoken dialogue system versus human, and found that spoken dialogue systems are a feasible alternative to human interaction in the role-playing context. Divekar et al. (2021) have found that interaction with spoken dialogue systems in immersive contexts improved students proficiency and decreased their anxiousness while using a foreign language thereby indicating there may be increased willingness to communicate with automated humanoid interlocutors. All this points to evidence that spoken dialogue systems are an effective tool for language acquisition.
Many spoken dialogue systems for the use of language learning have been built using off-the-shelf intent and slot detectors, and dialogue state managers (Bibauw et al., 2019). Divekar et al. (2018) have found some repair and grounding mechanisms in their dialogue system for language learning such as systems being able to respond to learners’ questions like "what do you mean" or "what can I say next" in a rule-based system. However, quick scaling up for such systems can only come with datasets.
Several datasets exist to help build task-oriented dialogues such as Schema-Guided-Dialogue (SGD) (Rastogi et al., 2020), MultiWoZ (Budzianowski et al., 2018), Dialogue State Tracking Challenges (DSTC) 1-3 (Williams et al., 2013; Henderson et al.,
2014a,b) and DSTC 4-5 (Kim et al., 2017). Besides, there are other frequently used speech-based ToD data, including Fluent Speech Commands (FSC) 1, Audio-Snips (Coucke et al., 2018), Carnegie Mellon Communicator Corpus (CMCC)(Bennett and Rudnicky, 2002) and Let’s Go Dataset 2.
However, existing task-oriented dialogue datasets do not reflect the language learning perspective as there are no constraints in their collection process that one interlocutor must be a low-proficiency speaker. Moreover, most datasets are also a result of a text-based interaction (Wang et al., 2019; Chen et al., 2021; Liang et al., 2021). This also means that the existing datasets will not contain R&G patterns specific for language learners interacting with a task-oriented spoken dialogue systems.
Therefore, we present a new dataset, namely GrounDialog, which will be the first dedicated ToD dataset specifically tailored for R&G in HPS-LPS conversations. Besides, the dataset can address the need for R&G in spoken form in specific scenarios that do not exist in the text-based exchange.

heading: 3 Data Collection Set-up
text: Our goal was to collect conversations between highproficiency (HPS) and low-proficiency speakers (LPS). To accomplish this, we use Amazon Mechanical Turk (AMTurk) to recruit and connect pairs of HPS and LPS for our study. To identify whether a participant is HPS or LPS, we provided the participants descriptions of CEFR levels (Council of Europe, 2001) and asked them to self-identify their proficiency level 3. For the purposes of this study, turkers who identify themselves as Beginner, Elementary, Intermediate, and Upper Intermediate i.e., A1-B2 levels were regarded as LPS; whereas those selecting Advanced and Proficient i.e., C1-C2 are considered as HPS. An assumption of our study is that we draw the line between HPS and LPS arbitrarily at B2 and trust the turker’s self-reported proficiency to be accurate. With this setup, we can end up with nearly equal size of HPS and LPS, which can ease the turker-pairing process for our data collection. A detailed explanation of the data collection process and conversational task for both HPS and LPS is shown below. Subsequently, we
1https://fluent.ai/fluent-speech-commands-a-dataset-forspoken-language-understanding-research/
2https://dialrc.github.io/LetsGoDataset/ 3The complete pre-chat survey form is shown in appendix
A
will present the general statistics of the collected dialogues and users. The study was approved by the IRB of the institute conducting this research. All participants were adults and provided consent before starting data collection. All collected data released with the paper is anonymized to our best abilities.

heading: 3.1 Conversational Task
text: In order to collect the conversational data that fits our purpose of having a conversation between an automated interlocutor and human, we follow the Wizard-of-Oz set-up (Kelley, 1984). The set-up has also been validated by many previous studies (Wen et al., 2016; Asri et al., 2017; Budzianowski et al., 2018). In general, two turkers (i.e. one HPS and the other LPS) were paired to communicate with each other. We contextualize their task into a pre-interview setting, where an HR hiring manager talks to an interviewee. Specifically, we set LPS to be the interviewee and HPS to be the HR hiring manager. We assign different goals for each role: the interviewee needs to find out the answers to a set of interview-related questions (e.g. interview time, duration, location, etc.), whereas the HR manager is given the information LPS will need and asked to be in charge of scheduling an appointment with the connected interviewee. To induce more repair and grounding turns in the conversation, we provided overlapping but inconsistent information to the interlocutors. For example, the interviewee is instructed that the interview is going to be 30 minutes, whereas the HR manager has 45 minutes in their task specification. We assumed that the difference in information will lead to the interlocutors being confused, asking clarification from each other, and resolving the situation by picking a time (Foster and Ohta, 2005).

heading: 3.2 Dialogue Interface
text: To establish a stable live connection between two turkers, we adapted VisDial AMT Chat (Das et al., 2017) to connect two humans, enable voice input/output, and connect to an off-the-shelf textto-speech service.
To simulate a Wizard-of-Oz like setting, we enable the LPS to directly record their speech, whereas the HPS input texts into a chat box and their responses are converted into speech using an off-the-shelf Text-To-Speech. The synthesized speech is played on the LPS side. In this way, the LPS could get a feeling of being connected to a
"chatbot", even though the responses are actually written by a human. The instructions for the LPS said that they will be connected to a human or a chatbot. In this way, we left it ambiguous for the LPS to decide for themselves whether they are talking to a chatbot or not. The HPS were told that they would appear as a bot so as to elicit bot-like communication from them. The example dialogue interfaces together with the instructions for both HPS and LPS are shown in Figure 8.

heading: 3.3 Data Statistics
text: In total, we collected 42 dialogues, including 1, 569 turns, from 55 unique turkers, where there are 29 high-proficiency speakers (HPS) and 26 lowproficiency speakers (LPS). Dialogues collected in our dataset are fairly long, with an average number of 37.4 turns per dialogue. Figure 2 presents a distribution over the sentence lengths for both HPS and LPS. The average sentence lengths are 10.02 and 8.55 for HPS and LPS respectively. We collected a total of 793 spoken utterances from LPS, and 777 textual responses from HPS.

heading: 3.4 User Statistics
text: After completing the conversational task, we asked each turker to input their demographic information through a post-chat survey form 4.
Specifically, for the turkers who did fill in our survey after the chat, there are 35 males and 16 females, with the age spanning from 22 to 63. The majority of the turkers are from India (45%) and the Unitied States (37%). Also, the self-identified English proficiency levels based on CEFR (Council of Europe, 2001) for the collected users are shown in Figure 1. As mentioned before, we take C1-C2 as high-proficiency speaker, and A1-B2 as low-proficiency speaker.

heading: 3.5 Speech data and transcriptions
text: There are 793 audio recordings collected from the accepted LPS5, of which 586 audio files are transcribed by SpeechPad 6, a reliable third-party transcription service, and the remaining 207 files are manually transcribed by the lead authors to inspect the quality of the data. The details of the concrete quality inspection process can be found in appendix
4Out of 55 unique turkers, four of them did not fill in the post-chat survey.
5LPS is accepted based on the speech quality and conversation completeness with HPS.
6https://www.speechpad.com/
C. The minimum, maximum and mean duration for the audio files collected from LPS are 1.38s, 38.82s and 6.8s, respectively.

heading: 4 GrounDialog Corpus
text: The primary goal of the data collection was to gather free-form conversations with repair and grounding (R&G) patterns, between highproficiency (HPS) and low-proficiency (LPS) English speakers. For this work, we constrain ourselves to the domain of job interviews, where an HR hiring manager attempts to schedule an upcoming interview with an interviewee candidate and answers any related questions. We leave the conversations in other domains to our future work.
To analyse the R&G patterns in the collected data from MTurk, we inherit R&G types from previous studies (Dobao and Martínez, 2007; Eszenyi and van der Wijst, 2006; Long, 1983; Foster and Ohta, 2005; Schegloff, 1997; Clark, 1996). The complete list of R&G types is shown in table 2. A detailed explanation of R&G annotation scheme is described below. In addition, similar to other taskoriented dialogue datasets (Budzianowski et al.,
2018; Rastogi et al., 2020), we also annotated the intents and slots for our GrounDialog corpus. To ease our annotation process, we adopted Inception (Klie et al., 2018), which is an open-source annotation software platform.

heading: 4.1 Annotation Scheme

heading: 4.1.1 Repair and Grounding
text: R&G can occur over several dialogue turns. It contains the context of the initial communication attempt, questions, and finally a resolution. We tagged these in our dataset as: Context, Question, R&G type and R&G complete. The definition for each item type is defined as follows:
• Context: the initial utterance as the context of the R&G.
• Question: the utterance that triggers the disfluency of the conversation between the two speakers.
• R&G type: the R&G type as defined in table 2.
• Complete: the utterance that signals the completion of the R&G process.
Note that R&G type is the required item for each R&G annotation, whereas Context, Question and Complete are optional. This is due to the fact that 1) some R&G types can be initiated without the Context and Question and 2) R&G process maybe not always completed as the conversation moves on.

heading: 4.1.2 Intent and Slot
text: Based on the unified dialog acts ontology defined in He et al. (2022), we proposed ontologies for both intent and slot for our GrounDialog corpus. The full ontology is shown in table 3. The more detailed descriptions for each intent and slot are shown in appendix D.

heading: 4.2 Annotation Statistics and Analysis

heading: 4.2.1 Repair and Grounding Annotations
text: The annotations for R&G, Intent and Slot are completed by the lead author. To ensure the quality of the annotations, the lead author and the second author manually inspected each item through comprehensive discussions. The questionable annotation items were corrected if the lead author agreed with the second author.
Figure 3 shows the distribution of different R&G types (a) and R&G related annotations (b) in GrounDialog corpus. There are 269 annotations for R&G types, among which 155 are from HPS and 114 are from LPS. As you can see in figure 3 (a), approximately 30% of the R&G types annotated in HPS utterances are Proactive Grounding (PG). This is due to the fact that the HR manager tends to ask questions that proactively fill in the communication gap and encourage the interviewee candidate to engage in the conversations. For example, in cases when the interviewee candidate forgot to ask questions related to the location of the interview, the HR manager would ask Do you know how to get to the company?. On the other hand, as expected, LPS used more Clarification Request (CR) in their speech in order to negotiate and confirm critical information for the interview. The example CR is shown in table 2.
After including Context, Question and R&G complete, we gathered 604 R&G related annotations, which is nearly 40% of all the dialogue 7. It can be observed in figure 3 (b) that both HPS and LPS leverage R&G for smoother communication, indicating the potential usefulness of our task setup in terms of negotiation of meaning in natural HPS-LPS conversations.

heading: 4.2.2 Intent and Slot Annotation
text: As for the intent annotations in GrounDialog, there are 1, 884 in total, with the number of intents in HPS and LPS being 878 and 1, 006, respectively. Figure 4 (left) demonstrates the distribution of intents annotated in the corpus for both HPS and LPS. As you can see, the top two intents are inform and request, which is similar to larger dialogue datasets like Budzianowski et al. (2018). In our dataset, almost 90% of dialogue utterances have one or two intents indicating the potential of training a language understanding module with our corpus.
Figure 4 (right) presents the distribution of slots annotated in both HPS and LPS responses. There are in total 612 slot annotations, within which 497 slots are annotated from HPS and 115 slots are from LPS. In our GrounDialog corpus, the HPS (i.e. HR managers) tend to give out information in multiple sentences. An example HPS utterance providing concrete location details of the interview is shown below:
7Each R&G related annotation is associated with a single utterance. Therefore, the R&G ratio of our dataset is approximately calculated as: 604 / 1569 ≈ 40%.

heading: ID R&G type Description Dialogue Example from GrounDialog
text: Ways to commute to our company: from Penn Station; exit via southwest corner of the station, walk along the Broadway for 3 minutes. The company is on the right side of the road.
In the example, four values for the Location slot are in bold. This is also the reason why nearly 45% of the slots in HPS responses are Location. In general, HPS produced much more slots compared
to LPS, which corresponds to the difference in the number of inform intent produced in HPS and LPS responses.

heading: 4.3 GrounDialog for Language Learning
text: As the major focus of this work, it is beneficial to take a deeper look at the R&G related annotations in GrounDialog, and discuss the potential utilities of the dataset for language learning.
As we have analyzed in the previous section, nearly 40% of the utterances are related to R&G. Figure 5 also presents the distribution of number of R&G annotations per dialogue. Almost 80% of the dialogues have at least four R&G related annotations, showing the richness of R&G patterns in GrounDialog. In general, GrounDialog encapsulates 12 R&G types in the natural HPS-LPS conversations under our task set-up. According to Figure 3(a), the top three R&G strategies for HPS are proactive grounding (PG), self-clarification (SCL) and check understanding (CU), whereas LPS mostly uses clarification request (CR), selfparaphrase (SP) and self-repetition (SR). This indicates that GrounDialog explicitly encourages LPS
to request clarification, rephrase or repeat previous utterances in cases when the initial communication with HPS failed.
Besides, we specifically annotated R&G complete to mark the sentences that signals the completion of a R&G process. Based on Figure 3(b), among all 269 R&G annotated in GrounDialog, 174 of them are actually completed, leading to a 65% completion rate. Figure 6 shows the distribution of number of R&G complete per dialogue. Nearly 80% of dialogues have at least three R&G complete, again suggesting the richness of R&G patterns. Also, given the high frequency of R&G related annotations in figure 3(b), we can imply that HPS tends to initiate the R&G much more often compared to LPS in GrounDialog.
From the language learning perspective, learners need R&G patterns to deepen their understanding of the language. For this purpose, GrounDialog can be used to train a chatbot that can generate responses conditioned on our R&G ontology to initiate R&G process, repair the communication gaps, and ground the meanings of conversations for
the language learners.

heading: 5 GrounDialog as a Benchmark for R&G in Task-oriented Dialogue
text: GrounDialog is designed as the first dedicated taskoriented dialogue dataset incorporating R&G patterns in HPS-LPS conversations. To show the potential usefulness of the corpus, we break down the dialogue modelling task into two sub-tasks and report a benchmark result for each of them: R&G detection and dialogue state tracking. Specifically, we performed few-shot learning following recent
advances in large language models (Brown et al., 2020; Wei et al., 2022), by prompting two most popular large language models, namely ChatGPT and GPT-4 8, with our carefully engineered prompts for both tasks. The details for each prompt are shown in appendix E.

heading: 5.1 R&G Detection
text: We show that by using the R&G annotations in GrounDialog, an R&G detection model can be trained to determine 1) if communication disfluencies occur; and 2) which type of R&G strategy (as defined in table 2) to choose in order to fix the potential disfluencies incurred in conversations.
Similar to previous section, we prompted GPT-4 for this experiment with the specific prompt defined in appendix E. Note that we tested on 40 out of 42 dialogues, excluding the two we used to design the prompt. For the utterances that do not need R&G, we ask the model to predict "None". The overall detection accuracy is shown in table 4 on the rightmost column 9. As we can see, prompting GPT-4 can achieve over 62% accuracy on the test dialogues, showing the potential of GrounDialog in training neural models in detecting R&G patterns in natural human-human conversations.
8We used gpt-3.5-turbo for ChatGPT and gpt-4 (default 8k version) for GPT-4.
9We do not report the results for ChatGPT since it failed to follow the prompt instructions.

heading: 5.2 Dialogue State Tracking
text: A good conversational system requires robust natural language understanding (NLU) and dialogue state tracking (DST) modules. For our benchmark results, we specifically prompted ChatGPT and GPT-4, both of which are popular ground-breaking large language models (LLMs) these days, with our domain-specific prompts. We follow the evaluation metrics for slot extraction in MultiWoz 1.0 (Budzianowski et al., 2018), where overall slot accuracy and joint goal accuracy are reported. For intent classification, we report the general classification accuracy. Table 4 demonstrates the performance of both models in terms of both sub-tasks. As we have only eight slot types in GrounDialog, both models achieved fairly high scores in slot accuracy and joint goal accuracy, with GPT-4 slightly outperforming ChatGPT. With regard to classifying intents, both models achieved over 60% accuracy, even though we have a larger group of intents to classify. These results demonstrate the potential utility of GrounDialog in building a good taskoriented conversational agent with solid NLU and DST modules.

heading: 6 Conclusion and Future Work
text: In this paper, we collected and annotated a new dataset GrounDialog, which is the first dedicated task-oriented dialogue dataset specifically designed for studying repair and grounding in spoken conversations between high-proficiency and lowproficiency speakers. We described the data collection procedure, annotation schemes, and presented a series analysis over the data. In addition, we demonstrated the potential and utility of GrounDialog by performing two tasks: R&G detection and dialogue state tracking. The results showed that GrounDialog can be used to train a conversational agent with the R&G capability. It could be further used to detect communicative gaps, which can be addressed in dialogue design.
In future, we plan to extend GrounDialog to a much larger dataset potentially covering multiple domains other than job interviews. Besides, we will use GrounDialog as a benchmark for a shared task to build task-oriented dialog agent with R&G ability. We will also conduct comprehensive user studies to determine the R&G patterns that are most useful in improving learner’s conversational proficiency during language learning. Further, we plan to present findings from the speech data so
researchers can use speech signals along with text to identify repair and grounding related turns.

heading: A Pre-chat English proficiency self-identification survey
text: See Figure 7 below.

heading: B Dialogue interface and instructions for High-proficiency and Low-proficiency speakers
text: See Figure 8 below.

heading: C Audio data quality inspection
text: This section details the process to inspect the quality of collected audio data. First of all, due to the fact that some collected audio contains long pauses (usually more than 10 seconds without any valid speech), we listened to each audio that is longer than 15 seconds carefully. Then we used ffmpeg10 to truncate the inspected audio which indeed contains long pause to the extend where the audio is natural and continuous. Next, for each audio data, we applied an internal automatic speech recognition tool to detect if the audio is silent all the time. As a result, we discarded all silent audio, and submit the remaining data to SpeechPad 11 for transcriptions.

heading: D Descriptions of Intent and Slot
text: In this section, we explain different types of intent and slots, and show some examples for better understanding. Specifically, we followed the conventions defined in (He et al., 2022). The descriptions for each intent and slot are shown in Table 5 and 6, respectively.
10https://ffmpeg.org/ 11https://www.speechpad.com/

heading: E Large Language Models prompts for Dialogue State Tracking and R&G Detection
text: The prompts we used for experiments in section 5 are shown in Table 7, 8 and 9, respectively. The intent classification and slot extraction task are conducted on a single utterance, whereas R&G detection is conducted on a complete dialogue.

heading: Future Work
text: In future, we plan to extend GrounDialog to a much larger dataset potentially covering multiple domains other than job interviews. Besides, we will use GrounDialog as a benchmark for a shared task to build task-oriented dialog agent with R&G ability. We will also conduct comprehensive user studies to determine the R&G patterns that are most useful in improving learner’s conversational proficiency during language learning. Further, we plan to present findings from the speech data so
researchers can use speech signals along with text to identify repair and grounding related turns