# Conversational Grounding in Multimodal Dialog Systems

Biswesh Mohapatra

<figure><img src="image_1.png" /></figure>

## ► To cite this version:

Biswesh Mohapatra. Conversational Grounding in Multimodal Dialog Systems. ICMI '23: International conference on multimodal interaction, Oct 2023, Paris, France. pp.706-710, 10.1145/3577190.3614226. hal-04268148

HAL Id: hal-04268148

https://hal.science/hal-04268148v1

Submitted on 2 Nov 2023

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_3.png" /></figure>

# Conversational Grounding in Multimodal Dialog Systems

Biswesh Mohapatra

biswesh.mohapatra@inria.fr

Inria

Paris, Île-de-France, France

## ABSTRACT

The process of “conversational grounding” is an interactive process that has been studied extensively in cognitive science, whereby participants in a conversation check to make sure their interlocutors understand what is being referred to. This interactive process uses multiple modes of communication to establish the information between the participants. This could include information provided through eye-gaze, head movements, intonation in speech, along with the content of the speech. While the process is essential to successful communication between humans and between humans and machines, work needs to be done on testing and building the capabilities of the current dialogue system in managing conversational grounding, especially in multimodal medium of communication. Recent work such as Benotti and Blackburn [3] have shown the importance of conversational grounding in dialog systems and how current systems fail in them which is essential for the advancement of Embodied Conversational Agents and Social Robots. Thus my Ph.D. project aims to test, understand and improve the functioning of current dialog models with respect to Conversational Grounding.

## CCS CONCEPTS

* Human-centered computing; * Computing methodologies
→ Discourse, dialogue and pragmatics;

## KEYWORDS

Multimodal Dialog Systems; Large Language Models; Conversational Grounding; Reference Resolution

### ACM Reference Format:

Biswesh Mohapatra. 2023. Conversational Grounding in Multimodal Dialog Systems. In *INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION* (ICMI ’23), *October 09–13, 2023, Paris, France*. ACM, New York, NY, USA, 5 pages. https://doi.org/10.1145/3577190.3614226

## 1 INTRODUCTION

In linguistics, Clark and Brennan. [4] propose the concept of "common ground" - the mutual knowledge and mutual assumptions that are essential for successful communication. This common ground is accumulated over the course of a conversation and is built via words, of course, but also through the use of other modalities: pointing to objects in the environment, nodding to indicate that one has understood, and staring at the speaker to indicate that one needs

more information, as pointed out by Nakano et al. [13]. This interactive process of building a common ground during a conversation by making sure that the interlocutors have understood the information that is being exchanged is called conversational grounding. One way of thinking about this is that these units have an underlying uncertainty which is negotiated and removed by the participants before getting added to the shared information. The uncertainty comes from ambiguities that could be in the form of spatial references like "that car" or event references like "that was funny". These uncertainties are solved through negotiations by providing additional information such as providing further context like "the big one next to the Ferrari" or by asking for clarifications by the listener such as "You mean the blue one?". The speakers also make sure that the listener has understood by looking at the behavior of the listener, such as if the listener is staring at them confused, and providing them with additional information to remove the ambiguity. A grounding mechanism in a dialog system deals with removing the ambiguity between the user and the agent while creating a local common understanding among them. This ambiguity could be cleared both when the model is acting as the speaker by sensing the lack of understanding from the listener and providing them with additional information or when the model is acting as the listener by asking for clarifications when necessary. Without a good grounding mechanism, conversations would not be robust and would often lead to misunderstandings. In fact, this is evident in the state-of-the-art dialogue systems that are increasingly using Large Language Models (LLM) as the Natural Language Understanding and Natural Language Generation modules. These LLMs are incapable of retaining and understanding all the information exchanged with the interlocutor during a session of conversation, as shown by Benotti and Blackburn [3]. They are also shown to be not very effective at making sure that the listener has grounded the information. MindCraft [1] further emphasized the importance of common ground in a situated dialog i.e. a dialog that takes place in a physical environment either real or simulated. Hence building a good grounding module is an essential part of any dialogue system. However, most of the prior research in the area has been conducted using rule-based models and has not been integrated into today's LLMs that were trained using alignment [16] and trained to perform zero-shot learning like Llama [22], ChatGPT [14] and GPT4 [15]. Integrating the two types of models could help us overcome the issues of generalization in rule-based models and issues of robustness in LLMs.

Benotti and Blackburn [3] further emphasize that grounding can be divided into two categories - (i) Symbol Grounding: the set of capabilities that link symbols with perceptions. Here one tries to accurately link the system's linguistic representation with its sensor data in the real world. (ii) Conversational Grounding: the process of seeking and providing incremental evidence of mutual

<footnote>Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s).

ICMI '23, *October 09–13, 2023, Paris, France*

© 2023 Copyright held by the owner/author(s).

ACM ISBN 979-8-4007-0055-2/23/10.

https://doi.org/10.1145/3577190.3614226</footnote>
understanding through dialogue, and repairing them in cases where there is no mutual understanding. While much research is being undertaken on symbol grounding, especially by roboticists, conversational grounding remains a major challenge for the advancement of truly intelligent dialogue systems.

Moreover, today most dialog systems use LLMs for processing the conversation which are text-only models and do not take into consideration the multi-modal aspect of grounding in situated environments. These situated features include non-verbal behaviors such as head nods and stares, para-verbal behaviors such as intonation, volume, and pauses, and interaction with the environment such as pointing and describing the locations in the spatial context. Removing the information present in intonation can lead to the introduction of ambiguities in the models. For example, an utterance that repeats the previous utterance can either be a confirmation of the previous utterance or a question for clarification depending upon the intonation. Grounding in such a context becomes even more challenging as compared to just text-based models. While text-based large language models are able to take advantage of the vast resources of data available publicly for their training, such multimodal data of daily human interactions are scarce and thus need models with the ability to ground the conversations in such low-resource settings. These capabilities would be essential for creating social robots that can interact with humans in the physical environment.

The goal of this PhD project is thus to enable robust conversational grounding for contemporary dialog systems. The work will build on prior research, in modular dialogue systems, that did deal with conversational grounding. However, since the majority of the previous work has been done using symbolic models that are hard to generalize, the work will take advantage of recent developments in artificial neural-network-based LLMs that are trained on a vast amount of textual data that was scraped from the internet. These models have shown the ability to generalize to new scenarios, however, their effectiveness in dialog systems is yet to be completely tested. The project not only aims to look at multimodal inputs but also eventually will look into generating multimodal outputs with respect to conversational grounding which would be essential for Embodied Conversational Agents and Social Robots.

## 2 BACKGROUND AND RELATED WORK

An immense amount of work has been done in addressing the issues of grounding in conversational systems, especially in rule-based modular dialog systems. The proposal of Common Ground in conversations [4, 5] became the basis for further work which looked into the various aspects of grounding. Grounding is difficult because overlapping utterances might be pointing to different information. In other words, interlocutors might start exchanging a piece of new information before going back to the previous information in a short span of time making it tough for our Language Models to process them. The information is grounded only when both parties involved agree to it. However, this agreement is not always shown explicitly. The interlocutors have to look for signs of understanding from the other person. Reaching an agreement could thus require solving some uncertainties present in the information. [4] described 4 states of uncertainty:

(1) B didn't notice that A uttered any u.

(2) B noticed that A uttered some u.

(3) B correctly heard u.

(4) B understood what A meant by u.

Paek and Horvitz [17] later proposed similar work which stated that uncertainties work at different hierarchical levels of mutual understanding: Channel Level, Signal Level, Intention Level and Conversation Level. Both [17] and [4] show the importance of creating a hierarchy of understanding to truly solve the issue of conversational grounding in the spoken dialog system.

Clark [4] also emphasized the difference in grounding for the different mediums of conversation. While text-based conversations such as messages have all the information provided through a single modality, the information gets divided into different modalities as they increase in number. Dialogs happening over the phone provide new information through acoustic features. Face-to-Face interactions bring additional visual information that plays an important role in grounding [13]. Kontogiorgos et. al. [11] have demonstrated that uncertainty in face-face interaction is generally represented through non-verbal signals such as eye-gaze. Dimosthenis and Joakim [7] found that listeners' gaze was observed to be a dominant indicator of speakers' perception of grounding, and led to increased joint-attention with every new speaker utterance. At the same time, even if the fragmented construction of utterances causes additional cognitive load for speakers, instructing entire units of information in a single utterance was not observed to be necessarily effective.

In order to solve these problems of mutual understanding, Traum [24] proposed the concepts of Grounding Acts. Grounding Acts paved the way to discretize the phenomenon of grounding into basic units. They also provided a way to create the hierarchical level of understanding as discovered in other mentioned work. The seven categories of Grounding Acts help us understand when a piece of information is considered grounded in the common ground. Each unit of grounding information starts with an initiation and is grounded after an implicit or explicit acknowledgment. Later, Roque and Traum [21] emphasized that the Grounding Acts could also help us in understanding the degree of uncertainty in a dialog.

However, the work by Traum and Allen [23] was done on a simple task-oriented dialog which could be narrowed down to a list of key-value pairs that needed to be extracted from the dialog. Not much work was done on open-domain dialogues that have a final task as an objective i.e. the number of topics on which one can converse is not fixed. Such task-based dialogues which require back-and-forth information exchange on unanticipated domains or referents are very common in day-to-day life and require a different approach to solve them.

Moreover, Grounding Acts only help us understand when something might be grounded. They do not provide us with much information on how to represent and work with grounded information. The grounded information must be devoid of ambiguities wherever needed, especially regarding the references being made. Grosz et al. [2, 9] proposed centering as a model of the complexity of the inferences required to integrate the meaning of an utterance into the meaning of the preceding discourse. Centering Theory was among the first theories to provide a rule-based structure to determine the
ambiguities in a discourse. However, this theory was intended for
discourse and not particularly for dialogues.

Domain Reference Theory [6] tried to resolve the issues present
in theories made for discourse. It looked into the major problems
with other theories which made them difficult to implement in dia-
logues including the three major issues - i) most of the theories look
into references from the point of linking operations to previously
mentioned referents which is not always the case as can be seen
for bridges. "I am cooking. Come here" has "here" referring to the
kitchen which has not been mentioned previously. ii) Reference
resolution relies not only on discursive information but equally
on conceptual and visual information. Thus, using a model based
on text-based dialogues doesn't provide the entire context. iii) Re-
ferring is not only considered as identifying a referent but also
imposing a particular point of view on the referent and the manner
it has been isolated within a set of potential referents. "the big ball"
which was once used to describe a particular ball can be used to
refer to it in the future. The Domain Reference Theory considers
reference domains that consist of attribute-value pairs minimally
including a unique identifier and a type. Most of the above work
has been done in a symbolic setting which generally performs great
under closed-world tasks but shows its weaknesses when asked to
generalize to other tasks without any additional effort.

Some recent works have tried to focus on Reference Centric
Multimodal Models by leveraging the success of artificial neural
networks in recent times. [8] tries to use an end-to-end neural
network-based model for solving multimodal referencing problems.
Recent work on generative agents has shown the potential of LLM
in interaction environments. In the work by Park et al. [18] Multiple
agents were created and provided an identity initially. One of the
agents was provided with the task of organizing a valentines party
in a sandbox environment. The agents had a memory module and
used LLMs to decide the importance of different memories. The
agents demonstrated capabilities of planning relevant events and ex-
ecuting them through human-like interactions. However, the work
is in a virtual environment with artificial agents in action which
does not mimic all the behaviors of human dialogues, especially,
conversational grounding since they do not use utterances in real-
time leading to much simpler conversational exchanges between
the agents. While this work provides an insight into the capabilities
of LLMs, more work needs to be done for conversational grounding.

3 RESEARCH QUESTIONS

In line with the previous motivation and related work, my Ph.D.
research aims to study and develop a framework for incorporat-
ing multimodal conversational grounding capabilities into dialog
systems, which have increasingly used Large Language Models for
processing conversations. We ask the following questions -

(1) How good are current Large Language Models with respect to multimodal conversational grounding and where could they be improved?

(2) How can multimodal context (for example, a scene that in-
terlocutors are viewing) be inserted into the LLMs to help in
conversational grounding?

(3) How can the grounded information be represented and stored efficiently in order to use them with Large Language Models?

(4) How can the models better negotiate a piece of information
during the conversation before grounding them?

(5) How can we generate multimodal output to reduce ambiguity during the negotiation phase of grounding and make sure that the listener has grounded the information?

4 RESEARCH PLAN

In order to carry out the work on multimodal conversational ground-
ing, we needed a dataset that had the following qualities -

* a corpus containing multimodal data of human-human spo-
ken conversations with ego-centric view of both the partici-
pants to help model grounding in real life.

* The data should contain various examples of references to objects established in common ground, which requires tasks that engage participants for longer periods of time and where they perform the act of conversational grounding more often.

Recently, the Meetup [10] dataset was released where two par-
ticipants were put on a 2D grid. Each vertex contains a room and
the participants are tasked to reach the same room. They can only
see their own rooms. When participants move east, west, north, or
south, they go to a new room and the image of that room pops up.
In order to meet in the same room, they have to describe the rooms
to the other participant, discuss a strategy to meet at the same point,
remember the rooms described by the other participants and create
a mental model of the rooms of the other participant. Thus it is an
excellent dataset to help us study and build models for grounding.

4.1 Testing the Capabilities of Dialog Systems

Current language models do not have architectures to take into con-
sideration the possibility of negotiations, clarifications, or requests
for clarification during the conversation. They treat the entire dia-
log history as one unit where utterances are arranged according to
the time. However, many of these dialogues contain overlapping
utterances, and multiple independent pieces of information might
be exchanged at the same time. This property of natural spoken
dialog makes them unique. Since, the advent of LLMs, dialog models
have been able to take advantage of their capabilities to generate
grammatically and semantically correct utterances. However, their
performance in phenomena that are specific to dialogs such as
conversational grounding has not been studied. Hence, we are cur-
rently doing a thorough study of their performance with respect to
grounding phenomena. We have therefore taken our annotations of
the Meetup dataset and given them to a number of state-of-the-art
LLMs in order to test the various grounding phenomenons such
as Clarification, Cancellation, Acknowledgment, Reference Resolu-
tion etc. For now, we are incorporating the image information by
providing their descriptions to the LLMs in the form of text.

4.2 Representing and Storing the grounded
information

In order to negotiate and ground the information exchanged during
a conversation, we need a good and effective way to represent the
grounded information. In order to represent the grounded infor-
mation, we might need to remove the ambiguities that come in the
form of references wherever necessary. While humans remove the
ambiguities only when it is necessary to establish certain information, models do not have an inherent understanding of when to remove such ambiguities during a conversation. Thus building models to help the agent in taking the decision of when to negotiate the ambiguity of the information with the interlocutor becomes essential for conversational grounding. In order to build such models, there exists a lot of scope for incorporating reasoning and inferring capabilities into conversational grounding by inducing worldly prior knowledge into them to reduce the ambiguities, especially since the advancement in pre-trained LLMs. Additionally, the multimodal information from the conversations also helps us to figure out the intents of the utterances which in turn helps us remove ambiguities before storing the grounded information. Finally, once we decide to ground a piece of information, the task of exploring ways to store the common ground effectively in order to use them during the dialog generation with less inference time becomes an important topic that the project will look into.

# 5 METHODOLOGY

In order to execute our research plans as discussed above we are currently working on the following topics :

(1) Our first work is to annotate two publicly-available dialog datasets with Grounding Acts(GA) and Common Grounding Units(CGU) that are inspired from Traum [23]. This includes a spoken dialog corpus named Spot the Difference [12] and Meetup [10]. Annotation of the utterances into Grounding Acts and grouping them into different units containing one single independent unit would help us figure out the real challenges of grounding for our models and will also provide us with more data to solve the issues.

(2) We are using the annotated utterances to study the large language models and understand if they are able to figure out the intention behind an utterance which could be to repair one of the previous information being provided, cancel previous information, add new information, ask for clarifying new information etc. Creating a way to look into the performance of these models with respect to Grounding Acts will thus provide us with an important peek into the strengths and weaknesses of these models when it comes to using them for dialog systems.

# 6 PRELIMINARY RESULTS

As of now, we have already annotated the Meetup Dataset with Grounding Acts(GAs) and Common Grounding Units(CGUs). The annotations were done by two individuals with a background in computational linguistics. The Inter-Rater Reliability on the annotation was 0.74. The annotation came with its own challenges but also helped us provide great insights into how conversational grounding works in such natural day-to-day conversations.

During the annotation of CGUs, we noticed 27 instances where a particular CGU was re-opened after being grounded. While 11 of the instances had a time difference of more than 10 seconds, the maximum time between when a CGU was grounded and then re-opened was 1 minute 9 seconds(keeping in mind the overall conversation was never more than 3 minutes in the dataset). While most of the CGUs were grounded one utterance later(1599 such

instances), the longest CGU was grounded after 13 utterances from its initiation. This shows the need for long-term memories and the ability to edit them in our dialog systems. There were 32 instances where the dialogs were not conclusive enough for the annotators to decide if the information should be grounded or not. These generally contained instances where one participant provides multiple information and asks a few questions together while the response from the other participant is ambiguous regarding the specific question that it was answering. This provokes the need for our models to have the capability to discuss and remove ambiguities whenever necessary. Further, in 23 instances, the annotators had to change their annotation for a particular utterance after looking at future utterances. The utterances were ambiguous and didn't belong to any particular category until we looked at the next few utterances further down.

The work on testing the Large Language Models is currently undergoing. We have created various test cases using the Meetup annotated dataset to test different LLMs. We are testing T5[20], Godel[19], Llama[22], GPT3.5 and GPT4[15]. Since, we have the models of T5, Godel and Llama, we finetuned them on the meetup dataset and tested both the vanilla and finetuned versions. For each phenomenon, we took a real instance from the Meetup dataset and made slight changes like the addition of a new question at the end and then checked the perplexity of the model for the correct and wrong responses. All our inputs contain the prior information providing instructions about the situation of the two participants, the rules of the game and the description of the images that the participants are looking at. The image descriptions can be also found in between utterances when participants change a room.

As can be seen in Table 1, the preliminary results show most of the models struggling with different phenomena. GPT 3.5 by far performs the best but still struggles in Request Acknowledgment and Cancellation showing their tendencies to answer the obvious answers like a 'yes' when a yes-no question is asked. Similarly, they lack the ability to remove a piece of information from the grounded information when they are canceled. While the finetuned models have a lower perplexity as compared to the vanilla models, the perplexity is low for both the correct and the wrong utterances. Hence, while finetuning does help the model and the model has higher probability of generating utterances that are like the ones spoken in the task, it doesn't make the model understand the pragmatics of the dialogs which are shown through the Grounding Acts.

# 7 FUTURE WORK AND CONCLUSION

Our tests have shown that current LLMs still lack in abilities to ground conversations. In order to solve this issue, I am planning to build modular dialog systems that break this complex problem into smaller units. The first module would categorize the utterances into different grounding acts and group them into grounding units. This would assist our generator models in focusing on the important utterances. Soon, we will start with the modeling of Common Grounding Units and Grounding Acts using the annotated data. We also plan to test the models on other phenomena like uncertainty and anaphora references. The study will help us understand the current issues of the models and also help us focus our attention on solving them. During the fine-tuning of our models with the
<table><caption>Table 1: Results of ratio of test cases where correct utterance had lower perplexity</caption><thead><tr><td>Model</td><td>Repair</td><td>Cancel</td><td>Request-Acknowledgment</td><td>Request-Repair</td></tr></thead><tbody><tr><td>T5</td><td>9/20</td><td>11/20</td><td>13/20</td><td>10/20</td></tr><tr><td>Godel</td><td>8/20</td><td>13/20</td><td>9/20</td><td>10/20</td></tr><tr><td>T5 - Finetuned</td><td>9/20</td><td>11/20</td><td>8/20</td><td>10/20</td></tr><tr><td>Godel - Finetuned</td><td>7/20</td><td>10/20</td><td>9/20</td><td>9/20</td></tr><tr><td>GPT 3.5</td><td>16/20</td><td>11/20</td><td>11/20</td><td>17/20</td></tr></tbody></table>

meetup dataset, we soon realized that memory is an important issue in today's Language Models that need to be resolved as well. Thus we will look into representing and storing the grounded information over a longer period of time which is crucial for a robust dialog model. We will also look into working directly with images instead of converting the images to descriptions and then using them with LLMs.

## ACKNOWLEDGMENTS

This research was funded by CORDI-S to which I am grateful for the opportunity. I would also like to thank my doctoral advisors Prof. Justine Cassell and Dr. laurent Romary for their constant support and feedback. I would also like to express my sincere gratitude to the members of the ArticuLab at Inria Paris for their invaluable assistance in my PhD journey.

## REFERENCES

[1] Cristian-Paul Bara, Sky CH-Wang, and Joyce Chai. 2021. MindCraft: Theory of Mind Modeling for Situated Dialogue in Collaborative Tasks. In *Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing*. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 1112–1125. https://doi.org/10.18653/v1/2021.emnlp-main.85

[2] Aravind Joshi Barbara Grosz and Scott Weinstein. 1986. Towards a computational theory of discourse interpretation.

[3] Luciana Benotti and Patrick Blackburn. 2021. Grounding as a Collaborative Process. In *Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume*. Association for Computational Linguistics, Online, 515–531. https://doi.org/10.18653/v1/2021.eacl-main.41

[4] Herbert H. Clark and Susan E. Brennan. 1991. Grounding in Communication. In *Perspectives on Socially Shared Cognition*, Lauren Resnick, Levine B., M. John, Stephanie Teasley, and D. (Eds.). American Psychological Association, 13–1991.

[5] Herbert H. Clark and Ed Schaefer. 1989. Contributing to Discourse. *Cogn. Sci.* 13 (1989), 259–294.

[6] Alexandre Denis. 2010. Generating Referring Expressions with Reference Domain Theory. In *Proceedings of the 6th International Natural Language Generation Conference*. Association for Computational Linguistics. https://aclanthology.org/W10-4203

[7] Kontogiorgos Dimosthenis and Gustafson Joakim. 2021. Measuring Collaboration Load With Pupillary Responses - Implications for the Design of Instructions in Task-Oriented HRI. Front Psychol. https://doi.org/10.3389/fpsyg.2021.623657

[8] Daniel Fried, Justin Chiu, and Dan Klein. 2021. Reference-Centric Models for Grounded Collaborative Dialogue. In *Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing*. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 2130–2147. https://doi.org/10.18653/v1/2021.emnlp-main.163

[9] Barbara J. Grosz, Aravind K. Joshi, and Scott Weinstein. 1983. Providing a Unified Account of Definite Noun Phrases in Discourse. In *21st Annual Meeting of the Association for Computational Linguistics*. Association for Computational Linguistics, Cambridge, Massachusetts, USA, 44–50. https://doi.org/10.3115/981311.981320

[10] Nikolai Ilinykh, Sina Zarrieß, and David Schlangen. 2019. Meet Up! A Corpus of Joint Activity Dialogues in a Visual Environment. In *Proceedings of the 23rd Workshop on the Semantics and Pragmatics of Dialogue - Full Papers*. SEMDIAL, London, United Kingdom. http://semdial.org/anthology/Z19-Ilinykh_semdial_0006.pdf

[11] Dimosthenis Kontogiorgos, Andre Pereira, and Joakim Gustafson. 2019. Estimating Uncertainty in Task-Oriented Dialogue. In *2019 International Conference on*

Multimodal Interaction (Suzhou, China) (ICMI '19). Association for Computing Machinery, New York, NY, USA, 414–418. https://doi.org/10.1145/3340555.3353722

[12] José Lopes, Nils Hemmingsson, and Oliver Astrand. 2018. The Spot the Difference corpus: a multi-modal corpus of spontaneous task oriented spoken interactions. In *Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)*. European Language Resources Association (ELRA), Miyazaki, Japan. https://aclanthology.org/L18-1305

[13] Yukiko Nakano, Gabe Reinstein, Tom Stocky, and Justine Cassell. 2003. Towards a Model of Face-to-Face Grounding. In *Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics*. Association for Computational Linguistics, Sapporo, Japan, 553–561. https://doi.org/10.3115/1075096.1075166

[14] OpenAI. 2022. Introducing ChatGPT. https://openai.com/blog/chatgpt.

[15] OpenAI. 2023. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL]

[16] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. arXiv:2203.02155 [cs.CL]

[17] Tim Paek and Eric Horvitz. 2000. Conversation as Action under Uncertainty. In *Proceedings of the Sixteenth Conference on Uncertainty in Artificial Intelligence (Stanford, California) (UAI'00)*. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 455–464.

[18] Joon Sung Park, Joseph C. O'Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023. Generative Agents: Interactive Simulacra of Human Behavior. arXiv:2304.03442 [cs.HC]

[19] Baolin Peng, Michel Galley, Pengcheng He, Chris Brockett, Lars Liden, Elnaz Nouri, Zhou Yu, Bill Dolan, and Jianfeng Gao. 2022. GODEL: Large-Scale Pre-training for Goal-Directed Dialog. arXiv. https://www.microsoft.com/en-us/research/publication/godel-large-scale-pre-training-for-goal-directed-dialog/

[20] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. *J. Mach. Learn. Res.* 21, 1, Article 140 (jan 2020), 67 pages.

[21] Antonio Roque and David R. Traum. 2008. Degrees of Grounding Based on Evidence of Understanding. In *SIGDIAL Workshop*.

[22] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lam-ple. 2023. LLaMA: Open and Efficient Foundation Language Models. *ArXiv abs/2302.13971* (2023).

[23] David Traum and James Allen. 1994. A "Speech Acts" Approach To Grounding In Conversation. (08 1994).

[24] David Rood Traum. 1995. *A Computational Theory of Grounding in Natural Language Conversation*. Ph. D. Dissertation. USA. UMI Order No. GAX95-23171.