# Analysing topic shifts in task-oriented dialogues

Amandine Decker, Maxime Amblard, Ellen Breitholtz

► To cite this version:

Amandine Decker, Maxime Amblard, Ellen Breitholtz. Analysing topic shifts in task-oriented dialogues. Journées scientifiques du GDR Lift - LIFT 2023, Nov 2023, Nancy, France. hal-04356426

HAL Id: hal-04356426

https://inria.hal.science/hal-04356426v1

Submitted on 20 Dec 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# Analysing topic shifts in task-oriented dialogues

Amandine Decker¹,² Maxime Amblard¹ Ellen Breitholtz²

(1) Universite de Lorraine, CNRS, Inria, LORIA, F-54000 Nancy, France

(2) CLASP, University Göteborg

`{amandine.decker, maxime.amblard}@univ-lorraine.fr, ellen.breitholtz@ling.gu.se`

## RÉSUMÉ

**Analyser les changements de sujet dans les conversations ayant un but.**

Le sujet d'une conversation joue un rôle crucial dans sa cohérence globale car ce qui est discuté à un moment donné limite les contributions possibles des participants. Lancer un nouveau sujet de conversation alors que le précédent est encore en cours de discussion peut être une source de confusion en l'absence de signaux appropriés. Cependant, la notion de sujet de conversation est débattue en linguistique et n'est pas suffisamment abordée en modélisation du dialogue. Une description précise de cette notion ainsi que de celle de changement de sujet permettrait de comprendre ce que nous percevons lorsque nous jugeons comme cohérente une séquence d'interventions. Afin d'analyser les différents types de changements de sujet, nous proposons de créer un corpus de conversations ayant un but (discussion du dilemme éthique de la *montgolfière*), où les conversations sont des échanges de messages écrits. Ces discussions limitées thématiquement vont nous permettre de comprendre les changements de sujet dans les dialogues.

## ABSTRACT

Topics play an important role in dialogue coherence, as what is currently discussed constrains the possible contributions of the participants, and initiating a topic while the previous one is still under discussion may be confusing without appropriate signals. However, how to actually define the notion of topic is debated in linguistics and not sufficiently discussed in dialogue modelling. A precise description of topics and topic shifts in conversation would contribute to understanding what it is we perceive when we judge a sequence of utterances to be coherent. In order to analyse different types of topic shifts, we propose to create a corpus of written task-oriented conversations (discussion of the ethical dilemma of the *balloon task*), where the dialogues happen by message exchanges. Such a controlled setting where the main topic is fixed, and subtopics are more easily identifiable, could be very helpful when it *comes to understanding how people change the topic and react to topic shifts in dialogues*.

**MOTS-CLÉS** : Modélisation de dialogue, changements de sujet, sémantique.

**KEYWORDS:** Topic modelling, Topic shifts, Semantics.
# 1 Motivation and context

Dialogue is at the core of human interactions, but it is also the place for many misunderstandings. It is thus interesting to build models to represent dialogue and try to better understand the features of a coherent interaction.

Interaction is however complex to characterise and model. In conversations, the participants exchange on different topics and the understanding of the current state of the dialogue by each participant relies on their interpretation of the topic under discussion. Even though topics are usually quite stable, shifts can happen more or less abruptly for several reasons such as a more pressing matter to discuss or the current topic being exhausted. Identifying topic shifts is crucial to understand and take part in the conversation. Conversely, creating confusion by changing the topic untimely and/or in an unclear manner could be seen as uncooperative (Grice, 1975). Moreover, the topic can sometimes gradually change throughout the conversation which can make it complicated to keep track of the most relevant information.

Having a better understanding of topic shifts in dialogue would help us include topics and topic shifts in current dialogue modelling theories (Amblard et al., 2011; Hunter et al., 2015; Maraev et al., 2018; Breitholtz, 2020). It could also enable us to analyse different types of dialogue-like conversations such as message threads on social media and identify off-topic contributions. Moreover, a symbolic approach of dialogue modelling could participate in improving language models’ handling of abrupt or unconventional topic shifts.

Topic shifts mechanisms are not all straightforward to investigate as participants usually implicitly agree on the topic and negotiate its boundaries through interaction. Thus, a more controlled setting with a fixed main topic, and more easily identifiable subtopics, could be very helpful to understand how people change the topic and react to topic shifts in dialogues.

Topic shifts mostly occur for three reasons: the dialogue content gives an idea to a participant (e.g., remembering an anecdote related to the current topic), the context changes and/or provides something to speak about (e.g., someone is coming who should not hear the current conversation, noises interrupt the dialogue, ...), or a participant has a specific agenda and shifts the topic, potentially abruptly, to fulfil it. In all these cases, the other participants must understand that the topic is changing to prevent disruptions in the dialogue. Specific tasks given to the participants could help us gather different types of topic shifts in our corpus.

Our objective is to construct a corpus that encompasses standard interactions, disrupted interactions, and interactions with multiple goals. The dialogues would be collected with a Chat Tool (Healey & Mills, 2009) that enables the experimenter to intervene in the conversations by modifying or delaying the participants’ messages for example. This will facilitate the comparison of how participants handle topic shifts. Our goal is to gather data first in English and later in French and possibly Swedish. Based on these analyses, we will define a proximity measure for determining topic change. Using it, we could segment the dialogues into thematic units and propose a more general theory of interaction.
## 2 ChatTool

DiET (Dialogue Experiment Toolkit) is an experiment platform developed by Healey & Mills (2009) that provides ways to gather text-messages conversations in a controlled environment. This ChatTool allows experimenter to link participants either with the Telegram app or with a custom computer interface and to store their conversation. All the messages sent by the participants are passed to a server which relays the messages to the addressees. The transition through the server enables the experimenter to control the conversation for they have the possibility to interfere with the conversations. Different manipulations are possible, and both the Telegram app and the custom computer interface have specificities when it comes to the display functionalities.

The experimenter can modify the content of messages, the identity of the sender of the messages and the timing of the messages. They can also insert fake messages and make it look like one of the participants sent them. These manipulations, illustrated by Figure 1, are meant to make it possible to investigate different dialogue phenomenon such as the organisation of a conversation, miscommunication or dialogue coherence.

<figure><img src="image_2.png" /><figcaption>Figure 1: Possible manipulations of the messages with the ChatTool</figcaption></figure>

## 3 Phenomena to Observe

Our goal is to investigate speakers' understanding of topic shifts. In this context, we would like to analyse the following phenomena.

Topic shifts are not always completely clear, especially when the new topic is not obviously different from the previous one. On the other hand, introducing a new topic that is completely different from the one at hand can be confusing and/or considered rude by the other participants. We would like to see how people react to more or less striking topic shifts.
There exist different ways to mark a topic shift and in particular some linguistic markers such as *by the way* can indicate to the addressees that the speaker is changing the topic. Removing these markers would enable us to assess their influence on mutual understanding.

A topic can change gradually in a conversation or more abruptly. In the first case, each intervention slightly changes the topic and thus helps understanding what is discussed. In the second case, the last interventions can help one notice that the previous topic has been wrapped up which licenses moving to a new one. We would like to see how people cope with missing some parts of the conversation, for instance by preventing one of the participants to receive some of the messages.

The meaning of some contributions can completely change if a piece of them is altered. For example, if someone is trying to describe an image with a cat but every time they use the word 'cat' it is replaced by 'dog', the final message will be completely different. The addressee would picture a different scene in their head and have a different perception of the at-hand topic from what the speaker expected. A message could also become complicated to understand if it contains an argument where one of the premise has been modified. If one receives the message "I need to buy gloves because it is getting so *warm* outside" instead of "I need to buy gloves because it is getting so *cold* outside", their interpretation can be unpredictable. They could believe the sender was being ironic, that they made a mistake, or potentially have an interpretation according to which buying gloves when it gets warm makes sense. We would like to observe the way people interpret messages where a core information has been modified, and in particular if they realise that they have a different understanding of their current discussion or if both speakers try to accommodate the other messages by coming up with a fitting interpretation.

In parallel to these phenomena, we would be interested in understanding how, after how long and with what motivation people shift the topic. This can only be done if the participants create topic shifts on their own as pushing them to change the topic would create a different phenomenon where one has specific reasons to change the current topic.

# 4 Tasks

This project aims to collect dialogues where participants perform a task involving various subtopics and to analyse the topic shifts and the different phenomena we discussed above.

One such task could be to discuss the ethical dilemma of the *balloon task* (Breitholtz *et al.*, 2021) where participants are asked to choose which of the four passengers of a hot air balloon to throw away, given that they would crash and all die if no one is sacrificed. The subtopics can there be defined by the different arguments in favour or against a passenger. We selected this task because its focus provides a manageable scope for conversation. We could create more or less obvious topic shifts by inserting a fake message discussing one
of the passengers when another one was discussed, or a message with a different argument from the one currently discussed. We could also create disruption by changing the passenger discussed in certain messages without notification.

A second task where the participants are supposed to cooperate to reach a common goal but where one of them could have hidden intentions would create interactions as both speakers would try to convince the other of their good intentions. Such a task could be realised through a game where the players both own a set of cards representing the wires of a bomb they are trying to disarm. The players know their own cards but not the ones of the other player. Disarming the bomb means cutting the wires in a certain order and thus returning the cards in a certain order. Discussing which card to return every turn would represent a topic. Inserting messages on a specific card or changing the colours that are discussed could create the phenomena discussed in Section 3.

Our last task is a description task inspired from the PhotoBook task (Haber et al., 2019). In this task both participants receive a set of images and they have to determine which one of them they have in common. Describing one image could be considered a sub-topic here. Again, this task provides a manageable scope and should limit the amount of personal information shared by the participants. The modifications of the messages could consist in swapping one entity for another, and we could create more or less clear topic shifts by adding messages related to a different image from the one currently discussed.

All three tasks are designed to prevent the participants from sharing personal information and engaging on personal topics (Amblard et al., 2014). Our goal is to observe the different phenomena described in Section 3 while minimising the impact of our manipulations on the content of the interventions.

# 5 Experimental Setup

Our goal is to gather conversations for the three tasks described in Section 4 where we manipulate some of the messages to observe the phenomenon discussed in Section 3. Table 1 describes the different manipulations we imagined for all the tasks and phenomena.

In total we have three tasks for four phenomena to observe. We also need one control setting for each task which makes fifteen different experiments. Our goal is to collect about 300 conversations in total (about 20 per experiment setting) where there would be three groups:

* about 30 conversations with first-time participants;

* about 90 conversations with participants joining in for the second, third or fourth time;

* about 180 conversations with ‘trained’ participants who have already taken part in the experiments more than 4 times.
<table><caption>Table 1: Manipulations of the conversations in the different tasks</caption><thead><tr><th>Phenomenon</th><th>Ethical Task</th><th>Dilemma</th><th>Negotiation Task</th><th>Image Task</th><th>Description</th></tr></thead><tbody><tr><td>Reaction to more or less clear TS</td><td>Add message on another passenger / with another argument</td><td></td><td>Add message on another card</td><td>Add message on a very different image from the one discussed / on a similar image</td><td></td></tr><tr><td>Influence of TS markers</td><td></td><td colspan="3">Gather a list of markers and delete them from messages</td><td></td></tr><tr><td>Missing information</td><td>Delete some arguments</td><td></td><td>Delete some arguments</td><td>Delete some messages with a '?'</td><td></td></tr><tr><td>Modified messages</td><td>Change the passenger discussed</td><td></td><td>Change the colour of the discussed wire</td><td>Modify some entities</td><td></td></tr><tr><td>Control</td><td></td><td></td><td>No outside interventions</td><td></td><td></td></tr></tbody></table>

We would then annotate the dialogues with the topics but also the type of topic shifts. A topic shift can be more or less brutal and one of our goal is to define a scale where the most abrupt changes can be annotated with e.g. “explicitly marked” and “due to external factors”. We also want to annotate the Questions Under Discussion (QUDs) (Kuppevelt, 1995; Ginzburg et al., 1996) in order to compare the granularity that topics and QUDs encode.

# 6 Conclusion

This paper describes a corpus collection experiment aiming at creating a resource to analyse how people understand each other in more chaotic environments. Our assumption is that speakers are able to maintain interaction in spite of partial or distorted information.

We would like to gather data in three languages: English, French, and Swedish. On top of the contribution in terms of language resources, this would be beneficial to compare how people handle topic shifts in different languages.

The analyses of this corpus would enable us to identify topic shift mechanisms and determine which ones are more easily understandable. This would participate in incorporating topics to formal dialogue models but also in improving language models when it comes to understanding less obvious topic shifts. Our goal in the future is to create a proximity measure to identify topic shifts, which could for instance help find off-topic and potentially harmful contributions in dialogue-like interactions such as social media ones.
References

AMBLARD M., FORT K., MUSIOL M. & REBUSCHI M. (2014). L'impossibilité de l'anonymat dans le cadre de l'analyse du discours. In *Journée ATALA éthique et TAL*, Paris, France. HAL: hal-01079308.

AMBLARD M., MUSIOL M. & REBUSCHI M. (2011). Une analyse basée sur la S-DRT pour la modélisation de dialogues pathologiques. In M. LAFOURCADE & V. PRINCE, Éds., *Actes de la 18e conférence sur le Traitement Automatique des Langues Naturelles - TALN 2011*, p. 93-98, Montpellier, France: Laboratoire d'Informatique de Robotique et de Microélectronique. HAL: hal-00601622.

BREITHOLTZ E. (2020). *Enthymemes and Topoi in Dialogue: The Use of Common Sense Reasoning in Conversation*. Leiden, The Netherlands: Brill. DOI: https://doi.org/10.1163/9789004436794.

BREITHOLTZ E., COOPER R., HOWES C. & LAVELLE M. (2021). *Reasoning in Multiparty Dialogue Involving Patients with Schizophrenia*, In (In)coherence of Discourse: Formal and Conceptual Issues of Language, p. 43-63. Springer International Publishing: Cham. DOI: 10.1007/978-3-030-71434-5_3.

GINZBURG J. et al. (1996). Dynamics and the semantics of dialogue. *Logic, language and computation*, **1**, 221-237.

GRICE H. P. (1975). Logic and conversation. In P. COLE & J. L. MORGAN, Éds., *Syntax and Semantics*, volume 3, p. 45-47: New York: Academic Press.

HABER J., BAUMGÄRTNER T., TAKMAZ E., GELDERLOOS L., BRUNI E. & FERNÁNDEZ R. (2019). The PhotoBook dataset: Building common ground through visually-grounded dialogue. In A. KORHONEN, D. TRAUM & L. MÀRQUEZ, Éds., *Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics*, p. 1895-1910, Florence, Italy: Association for Computational Linguistics. DOI: 10.18653/v1/P19-1184.

HEALEY P. G. & MILLS G. J. (2009). A dialogue experimentation toolkit. In *Proceedings of the Annual Meeting of the Cognitive Science Society*, volume 31.

HUNTER J., ASHER N., KOW E., PERRET J. & AFANTENOS S. (2015). Defining the Right Frontier in Multi-Party Dialogue. In *19th Workshop on the semantics and pragmatics of dialogue (SemDial 2015 - goDIAL)*, p. pp. 95-103, Göteborg, Sweden. HAL: hal-01535950.

KUPPEVELT J. V. (1995). Discourse structure, topicality and questioning. *Journal of Linguistics*, **31**(1), 109-147. DOI: 10.1017/S002222670000058X.

MARAEV V., GINZBURG J., LARSSON S., TIAN Y. & BERNARDY J.-P. (2018). Towards kos/ttr-based proof-theoretic dialogue management. In *Proceedings of the 22nd Workshop on the Semantics and Pragmatics of Dialogue*, Aix-en-Provence, France: SEMDIAL.