# Conversational Feedback in Scripted versus Spontaneous Dialogues: A Comparative Analysis

Ildikó Pilán, Laurent Prévot, Hendrik Buschmeier, Pierre Lison

<figure><img src="image_1.png" /></figure>

## To cite this version:

Ildikó Pilán, Laurent Prévot, Hendrik Buschmeier, Pierre Lison. Conversational Feedback in Scripted versus Spontaneous Dialogues: A Comparative Analysis. 2024. hal-04535316

HAL Id: hal-04535316

https://hal.science/hal-04535316

Preprint submitted on 6 Apr 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Conversational Feedback in Scripted versus Spontaneous Dialogues: A Comparative Analysis

Ildikó Pilán, Laurent Prévot, Hendrik Buschmeier, Pierre Lison

► To cite this version:

Ildikó Pilán, Laurent Prévot, Hendrik Buschmeier, Pierre Lison. Conversational Feedback in Scripted versus Spontaneous Dialogues: A Comparative Analysis. 2024. hal-04535316

HAL Id: hal-04535316

https://hal.science/hal-04535316v1

Preprint submitted on 6 Apr 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Conversational Feedback in Scripted versus Spontaneous Dialogues:
A Comparative Analysis

**Ildikó Pilán**

Norwegian Computing Center, Oslo, Norway

PILAN@NR.NO

**Laurent Prévot**

CEFC, CNRS, MEAE, Taipei, Taiwan

Aix Marseille Université & CNRS, LPL, Aix-en-Provence, France

LAURENT.PREVOT@UNIV-AMU.FR

**Hendrik Buschmeier**

Bielefeld University, Bielefeld, Germany

HBUSCHME@UNI-BIELEFELD.DE

**Pierre Lison**

Norwegian Computing Center, Oslo, Norway

PLISON@NR.NO

**Editor:** Name Surname

Submitted 09/2023; Accepted MM/YYYY; Published online MM/YYYY

## Abstract

Scripted dialogues such as movie and TV subtitles constitute a widespread source of training data for conversational NLP models. However, the linguistic characteristics of those dialogues are notably different from those observed in corpora of spontaneous interactions. This difference is particularly marked for communicative feedback and grounding phenomena such as backchannels, acknowledgments, or clarification requests. Such signals are known to constitute a key part of the conversation flow and are used by the dialogue participants to provide feedback to one another on their perception of the ongoing interaction. This paper presents a quantitative analysis of such communicative feedback phenomena in both subtitles and spontaneous conversations. Based on dialogue data in English, French, German, Hungarian, Italian, Japanese, Norwegian and Chinese, we extract both lexical statistics and classification outputs obtained with a neural dialogue act tagger. Two main findings of this empirical study are that (1) conversational feedback is markedly less frequent in subtitles than in spontaneous dialogues and (2) subtitles contain a higher proportion of negative feedback. Furthermore, we show that dialogue responses generated by large language models also follow the same underlying trends and include comparatively few occurrences of communicative feedback, except when those models are explicitly fine-tuned on spontaneous dialogues.

**Keywords:** conversational grounding, conversational feedback, dialogue, spontaneous speech, subtitles

## 1. Introduction

While the amount of text data available online for training NLP models is large and growing steadily, the situation is more contrasted for conversational speech data. Indeed, while corpora of spontaneous spoken interactions have been collected for various languages (Dingemanse and Liesenfeld, 2022),
# Conversational Feedback in Scripted versus Spontaneous Dialogues:
A Comparative Analysis

**Ildikó Pilán**

Norwegian Computing Center, Oslo, Norway

PILAN@NR.NO

**Laurent Prévot**

CEFC, CNRS, MEAE, Taipei, Taiwan

Aix Marseille Université & CNRS, LPL, Aix-en-Provence, France

LAURENT.PREVOT@UNIV-AMU.FR

**Hendrik Buschmeier**

Bielefeld University, Bielefeld, Germany

HBUSCHME@UNI-BIELEFELD.DE

**Pierre Lison**

Norwegian Computing Center, Oslo, Norway

PLISON@NR.NO

**Editor:** Name Surname

Submitted 09/2023; Accepted MM/YYYY; Published online MM/YYYY

## Abstract

Scripted dialogues such as movie and TV subtitles constitute a widespread source of training data for conversational NLP models. However, the linguistic characteristics of those dialogues are notably different from those observed in corpora of spontaneous interactions. This difference is particularly marked for communicative feedback and grounding phenomena such as backchannels, acknowledgments, or clarification requests. Such signals are known to constitute a key part of the conversation flow and are used by the dialogue participants to provide feedback to one another on their perception of the ongoing interaction. This paper presents a quantitative analysis of such communicative feedback phenomena in both subtitles and spontaneous conversations. Based on dialogue data in English, French, German, Hungarian, Italian, Japanese, Norwegian and Chinese, we extract both lexical statistics and classification outputs obtained with a neural dialogue act tagger. Two main findings of this empirical study are that (1) conversational feedback is markedly less frequent in subtitles than in spontaneous dialogues and (2) subtitles contain a higher proportion of negative feedback. Furthermore, we show that dialogue responses generated by large language models also follow the same underlying trends and include comparatively few occurrences of communicative feedback, except when those models are explicitly fine-tuned on spontaneous dialogues.

**Keywords:** conversational grounding, conversational feedback, dialogue, spontaneous speech, subtitles

## 1. Introduction

While the amount of text data available online for training NLP models is large and growing steadily, the situation is more contrasted for conversational speech data. Indeed, while corpora of spontaneous spoken interactions have been collected for various languages (Dingemanse and Liesenfeld, 2022),
those are generally of a limited size and often focus on particular topics or tasks. To address this scarcity of available data, a common approach is to instead rely on corpora of authored dialogues extracted from movie scripts (Danescu-Niculescu-Mizil and Lee, 2011) or movie and TV subtitles (Lison et al., 2018; Davies, 2021) when modeling spontaneous conversation.

However, those dialogues have different properties than spontaneous interactions. Most importantly, movie scripts are written for the purpose of narrating a story. Many key conversational phenomena such as disfluencies (Shriberg, 1996), overlapping talk (Schegloff, 2000), and backchannels (Yngve, 1970) are therefore either absent or uncommon, unless they explicitly contribute to the storyline (Berliner, 1999; Chepinchikj and Thompson, 2016). Subtitles, in addition, are also subject to strict length constraints, with a maximum of two lines with at most 40–50 characters per line and an on-screen display time between one and six seconds. As a consequence, subtitles tend to only transcribe the most salient information conveyed in each utterance.

This paper provides a quantitative analysis of how subtitles differ from spontaneous dialogues, focusing more specifically on *conversational feedback* (Allwood et al., 1992) and *grounding* (Clark and Schaefer, 1989) phenomena. Even though feedback cues are often short, non-lexical words, they are essential for moving forward smoothly within a dialogue as they indicate, among others, speakers’ common understanding and agreement (Allwood et al., 1992). To highlight the relevant linguistic properties respectively seen in subtitles and in corpora of spontaneous conversations, we first compile a range of lexical statistics. To go beyond lexical distributions, we subsequently train a neural dialogue act tagger to estimate the relative frequencies of various feedback signals in each corpus. We also apply this tagger on synthetic dialogues generated by different language models and show that only when fine-tuning on unscripted dialogues is the proportion of feedback phenomena comparable to that found in spontaneous conversations. Our experiments are conducted using corpora for eight languages, namely English, Chinese, French, German, Hungarian, Italian, Japanese and Norwegian. Spontaneous dialogue sources for these languages are more readily available and this subset also allows for investigations across different language families. Since there has been a significant increase in the popularity of large language models fine-tuned for conversational interaction, but little is known about the ability of these state-of-the-art systems to handle feedback cues, we also offer a small-scale qualitative analysis of how they deal with such communicative signals.

The paper is structured as follows: Section 2 reviews related work on conversational feedback and the linguistic analysis of subtitles. Section 3 then presents the corpora of both spontaneous and scripted dialogues employed in the experiments. Section 4 describes and discusses the lexical distributions of grounding phenomena observed in these corpora. Building on those results, Section 5 compares them to the frequency of feedback-related linguistic phenomena estimated using a neural dialogue act tagger trained on the Switchboard corpus (Stolcke et al., 1998; Godfrey et al., 1992). We then describe our experiments carried out on synthetic dialogues in Section 6 which is followed by a short analysis of multimodal feedback cues in Section 7. In Section 8, we summarize our observations about how the latest conversational language models deal with feedback signals. Finally, Section 9 concludes the paper and outlines future directions of research.

## 2. Related Work

### 2.1 Conversational Feedback and Grounding

A key aspect of any communicative activity is management of the information flow, a process often called *conversational grounding* (Clark and Schaefer, 1989). The study of grounding and related
those are generally of a limited size and often focus on particular topics or tasks. To address this scarcity of available data, a common approach is to instead rely on corpora of authored dialogues extracted from movie scripts (Danescu-Niculescu-Mizil and Lee, 2011) or movie and TV subtitles (Lison et al., 2018; Davies, 2021) when modeling spontaneous conversation.

However, those dialogues have different properties than spontaneous interactions. Most importantly, movie scripts are written for the purpose of narrating a story. Many key conversational phenomena such as disfluencies (Shriberg, 1996), overlapping talk (Schegloff, 2000), and backchannels (Yngve, 1970) are therefore either absent or uncommon, unless they explicitly contribute to the storyline (Berliner, 1999; Chepinchikj and Thompson, 2016). Subtitles, in addition, are also subject to strict length constraints, with a maximum of two lines with at most 40–50 characters per line and an on-screen display time between one and six seconds. As a consequence, subtitles tend to only transcribe the most salient information conveyed in each utterance.

This paper provides a quantitative analysis of how subtitles differ from spontaneous dialogues, focusing more specifically on *conversational feedback* (Allwood et al., 1992) and *grounding* (Clark and Schaefer, 1989) phenomena. Even though feedback cues are often short, non-lexical words, they are essential for moving forward smoothly within a dialogue as they indicate, among others, speakers’ common understanding and agreement (Allwood et al., 1992). To highlight the relevant linguistic properties respectively seen in subtitles and in corpora of spontaneous conversations, we first compile a range of lexical statistics. To go beyond lexical distributions, we subsequently train a neural dialogue act tagger to estimate the relative frequencies of various feedback signals in each corpus. We also apply this tagger on synthetic dialogues generated by different language models and show that only when fine-tuning on unscripted dialogues is the proportion of feedback phenomena comparable to that found in spontaneous conversations. Our experiments are conducted using corpora for eight languages, namely English, Chinese, French, German, Hungarian, Italian, Japanese and Norwegian. Spontaneous dialogue sources for these languages are more readily available and this subset also allows for investigations across different language families. Since there has been a significant increase in the popularity of large language models fine-tuned for conversational interaction, but little is known about the ability of these state-of-the-art systems to handle feedback cues, we also offer a small-scale qualitative analysis of how they deal with such communicative signals.

The paper is structured as follows: Section 2 reviews related work on conversational feedback and the linguistic analysis of subtitles. Section 3 then presents the corpora of both spontaneous and scripted dialogues employed in the experiments. Section 4 describes and discusses the lexical distributions of grounding phenomena observed in these corpora. Building on those results, Section 5 compares them to the frequency of feedback-related linguistic phenomena estimated using a neural dialogue act tagger trained on the Switchboard corpus (Stolcke et al., 1998; Godfrey et al., 1992). We then describe our experiments carried out on synthetic dialogues in Section 6 which is followed by a short analysis of multimodal feedback cues in Section 7. In Section 8, we summarize our observations about how the latest conversational language models deal with feedback signals. Finally, Section 9 concludes the paper and outlines future directions of research.

## 2. Related Work

### 2.1 Conversational Feedback and Grounding

A key aspect of any communicative activity is management of the information flow, a process often called *conversational grounding* (Clark and Schaefer, 1989). The study of grounding and related
phenomena, such as conversational feedback (Allwood et al., 1992), has been instrumental to cognitive approaches to communication (Clark, 1996), and to dialogue system development (Traum, 1994; Paek and Horvitz, 2000; Yaghoubzadeh et al., 2015). Typical feedback signals such as acknowledgments, showing understanding and acceptance were, in fact, also among the criteria suggested for the Empathy Scale for Human-Computer Communication suggested by Concannon and Tomalin (2023).

Conversational feedback may appear at different positions in a dialogue. However, a number of corpus studies found that they have a tendency to occur at specific places, mostly where they cause little interference (Kjellmer, 2009). These places of occurrence have also been referred to as *Feedback Relevant Spaces* (Heldner et al., 2013; Howes and Eshghi, 2021). Even though often considered to be semantically of little relevance, Howes and Eshghi (2021) offer a semantic processing model for conversational feedback phenomena which underlines the crucial role feedback plays in moving forward within a dialogue.

Although, arguably, any utterance relates directly or indirectly to grounding (through implicit and high level pragmatic inference, Clark and Schaefer 1989), *acknowledgments* and other positive feedback signals (see Ex. (1)), along with *repair* (see Ex. (2)), have been identified as the most prominent grounding mechanisms (Jefferson, 1972; Bunt, 1994). Their frequency in human-human dialogue is known to be very high (e.g., Stolcke et al., 2000a) and universal across languages (Liesenfeld and Dingemanse, 2022; Dingemanse et al., 2015). These conversational signals, while they do not cover all grounding phenomena, can therefore be seen as a useful proxy to quantify feedback in a dialogue.

(1) A: and uh it really does irk me to see those guys out there uh you know making that ///much money///
B: ///yeah///<fnref n="1" />

Recent works have emphasized the role of feedback and grounding signals in their study of human-human conversations (Fusaroli et al., 2017; Dideriksen et al., 2022; Dingemanse and Liesenfeld, 2022) as well as human-agent interaction (Visser et al., 2014; Hough and Schlangen, 2016; Buschmeier and Kopp, 2018; Axelsson et al., 2022).

The literature tends to merge the two closely related concepts of *backchannels* and *acknowledgments*. Backchannels (Yngve, 1970), or *continuers* (Schegloff, 1982), are not positioned on the main channel, but uttered by the “listener”, often as low intensity unobtrusive overlapping speech (Heldner et al., 2010) or non-verbally (Allwood et al., 2007; Truong et al., 2011). Acknowledgments, on the other hand, have a slightly broader, functional definition of minimal positive feedback (Jefferson, 1984; Allwood et al., 1992).

There has been a large body of work on lexical markers, also called *cue phrases* or *discourse markers* (Jefferson, 1984; Allwood et al., 1992; Muller and Prévot, 2003), since they present interesting linguistic features and constitute convenient explicit cues for detecting feedback utterances automatically (Jurafsky et al., 1998; Gravano et al., 2012; Prévot et al., 2015). Gravano et al. (2012) developed a list of affirmative cue words made of *alright*, *mm-hm*, *okay*, *right*, *uh-huh*, *yeah*. Form-Function studies of similar lists have been made at least for Swedish (Allwood, 1988), U.S. English (Ward, 2006), and French (Prévot et al., 2015).

There is less emphasis in the literature on direct negative feedback associated with rejection and corrective dialogue acts. Although Allwood et al. (1992) suggests a polarity dimension for

<footnote n="1">1. Notation: ///text/// produced in overlap with the speech of the other speaker. From Switchboard (Godfrey et al., 1992)</footnote>
phenomena, such as conversational feedback (Allwood et al., 1992), has been instrumental to cognitive approaches to communication (Clark, 1996), and to dialogue system development (Traum, 1994; Paek and Horvitz, 2000; Yaghoubzadeh et al., 2015). Typical feedback signals such as acknowledgments, showing understanding and acceptance were, in fact, also among the criteria suggested for the Empathy Scale for Human-Computer Communication suggested by Concannon and Tomalin (2023).

Conversational feedback may appear at different positions in a dialogue. However, a number of corpus studies found that they have a tendency to occur at specific places, mostly where they cause little interference (Kjellmer, 2009). These places of occurrence have also been referred to as *Feedback Relevant Spaces* (Heldner et al., 2013; Howes and Eshghi, 2021). Even though often considered to be semantically of little relevance, Howes and Eshghi (2021) offer a semantic processing model for conversational feedback phenomena which underlines the crucial role feedback plays in moving forward within a dialogue.

Although, arguably, any utterance relates directly or indirectly to grounding (through implicit and high level pragmatic inference, Clark and Schaefer 1989), *acknowledgments* and other positive feedback signals (see Ex. (1)), along with *repair* (see Ex. (2)), have been identified as the most prominent grounding mechanisms (Jefferson, 1972; Bunt, 1994). Their frequency in human-human dialogue is known to be very high (e.g., Stolcke et al., 2000a) and universal across languages (Liesenfeld and Dingemanse, 2022; Dingemanse et al., 2015). These conversational signals, while they do not cover all grounding phenomena, can therefore be seen as a useful proxy to quantify feedback in a dialogue.

(1) A: and uh it really does irk me to see those guys out there uh you know making that ///much money///

B: ///yeah///<fnref n="1" />

Recent works have emphasized the role of feedback and grounding signals in their study of human-human conversations (Fusaroli et al., 2017; Dideriksen et al., 2022; Dingemanse and Liesenfeld, 2022) as well as human-agent interaction (Visser et al., 2014; Hough and Schlangen, 2016; Buschmeier and Kopp, 2018; Axelsson et al., 2022).

The literature tends to merge the two closely related concepts of *backchannels* and *acknowledgments*. Backchannels (Yngve, 1970), or *continuers* (Schegloff, 1982), are not positioned on the main channel, but uttered by the “listener”, often as low intensity unobtrusive overlapping speech (Heldner et al., 2010) or non-verbally (Allwood et al., 2007; Truong et al., 2011). Acknowledgments, on the other hand, have a slightly broader, functional definition of minimal positive feedback (Jefferson, 1984; Allwood et al., 1992).

There has been a large body of work on lexical markers, also called *cue phrases* or *discourse markers* (Jefferson, 1984; Allwood et al., 1992; Muller and Prévot, 2003), since they present interesting linguistic features and constitute convenient explicit cues for detecting feedback utterances automatically (Jurafsky et al., 1998; Gravano et al., 2012; Prévot et al., 2015). Gravano et al. (2012) developed a list of affirmative cue words made of *alright, mm-hm, okay, right, uh-huh, yeah*. Form-Function studies of similar lists have been made at least for Swedish (Allwood, 1988), U.S. English (Ward, 2006), and French (Prévot et al., 2015).

There is less emphasis in the literature on direct negative feedback associated with rejection and corrective dialogue acts. Although Allwood et al. (1992) suggests a polarity dimension for

<footnote n="1">1. Notation: ///text/// produced in overlap with the speech of the other speaker. From Switchboard (Godfrey et al., 1992)</footnote>
characterizing feedback, most recent studies have focused on positive feedback. Indeed, in collaborative dialogue and everyday conversations, which are the two genres dominating available datasets, positive feedback constitutes the large majority of explicit feedback (e.g., Malisz et al., 2016).

Negative feedback is instead often expressed constructively, using repair mechanisms, specifically *clarification requests* (Purver, 2004). These may rely on simple lexical cues (e.g., for English, *pardon?, huh?*), sluices (such as *what?, who?*), or on clarification ellipsis, as in the following example (Fernández et al., 2007):

(2) **A:** and then we're going to turn east

**B:** mmhmm

**A:** not straight east slightly sort of northeast

**B:** slightly northeast?<fnref n="2" />

Conversational feedback forms a central part of dialogue annotation frameworks (Core and Allen, 1997; Bunt, 2011). In Switchboard-DAMSL (Stolcke et al., 1998; Godfrey et al., 1992), for example, the following acts relate to feedback: *Acknowledge (Backchannel)*, *Agree/Accept, Response acknowledgment, Repeat-phrase, Backchannel in question form, Tag-question, Signal-non-understanding*. Many acknowledgments and repair-initiators also fit under the concept of *Non Sentential Utterances* (NSU) analyzed for example in Fernández et al. (2007); Ginzburg (2012); Dragone and Lison (2016).

The occurrence of feedback signals in dialogue transcriptions can be detected using various types of sequence labeling models, including classical hidden Markov models (Stolcke et al., 2000b) as well as more recent neural architectures (Liu et al., 2017; Noble and Maraev, 2021).

Conversational feedback can also be delivered in the form of non-verbal signals such as headgestures, nods and laughter (Schegloff, 1982) which, however, are not necessarily transcribed or thoroughly annotated in the available corpora. Maraev et al. (2021) incorporated information about laughter to improve the performance of a neural dialogue act tagger and found that for 76% of utterances annotated as *non-verbal* in the Switchboard corpus (Godfrey et al., 1992), their system predicted the *Acknowledge (Backchannel)* label.

## 2.2 Analysis of Subtitles

Subtitles are typically short written text snippets and they accompany audiovisual content on the screen. Since they encode spoken language in a written form, they are often subject to condensation and normalization, where non-standard verbal elements (repetitions, signs of hesitation etc.) are omitted or replaced by more standard alternatives (Gottlieb, 2012). This is due to constraints on the length and the readability of the subtitles as well as adherence to writing conventions, especially for professional subtitle writers. As subtitles are meant to be displayed alongside audiovisual content, this is rarely a problem as viewers can typically recover omitted dialogue-relevant cues from the accompanying images and sounds.

We can distinguish between *interlingual subtitling*, where the original language of the audio is different from the subtitling language; and *intralingual subtitling*, where the audio and the subtitle language are the same with non-verbal elements also recorded in writing, typically for the benefit of hearing impaired audiences or non-native speakers (Gottlieb, 2012).

Corpora of movie and TV subtitles have been studied in both linguistics and language teaching, where it serves as an easy-to-access approximation of spoken language use (Werner, 2020). Quaglio

<footnote n="2">2. From HCRC Map Task (Anderson et al., 1991).</footnote>
characterizing feedback, most recent studies have focused on positive feedback. Indeed, in collaborative dialogue and everyday conversations, which are the two genres dominating available datasets, positive feedback constitutes the large majority of explicit feedback (e.g., Malisz et al., 2016).

Negative feedback is instead often expressed constructively, using repair mechanisms, specifically *clarification requests* (Purver, 2004). These may rely on simple lexical cues (e.g., for English, *pardon?, huh?*), sluices (such as *what?, who?*), or on clarification ellipsis, as in the following example (Fernández et al., 2007):

(2) A: and then we're going to turn east

B: mmhmm

A: not straight east slightly sort of northeast

B: slightly northeast?<fnref n="2" />

Conversational feedback forms a central part of dialogue annotation frameworks (Core and Allen, 1997; Bunt, 2011). In Switchboard-DAMSL (Stolcke et al., 1998; Godfrey et al., 1992), for example, the following acts relate to feedback: *Acknowledge (Backchannel)*, *Agree/Accept, Response acknowledgment, Repeat-phrase, Backchannel in question form, Tag-question, Signal-non-understanding*. Many acknowledgments and repair-initiators also fit under the concept of *Non Sentential Utterances* (NSU) analyzed for example in Fernández et al. (2007); Ginzburg (2012); Dragone and Lison (2016).

The occurrence of feedback signals in dialogue transcriptions can be detected using various types of sequence labeling models, including classical hidden Markov models (Stolcke et al., 2000b) as well as more recent neural architectures (Liu et al., 2017; Noble and Maraev, 2021).

Conversational feedback can also be delivered in the form of non-verbal signals such as headgestures, nods and laughter (Schegloff, 1982) which, however, are not necessarily transcribed or thoroughly annotated in the available corpora. Maraev et al. (2021) incorporated information about laughter to improve the performance of a neural dialogue act tagger and found that for 76% of utterances annotated as *non-verbal* in the Switchboard corpus (Godfrey et al., 1992), their system predicted the *Acknowledge (Backchannel)* label.

## 2.2 Analysis of Subtitles

Subtitles are typically short written text snippets and they accompany audiovisual content on the screen. Since they encode spoken language in a written form, they are often subject to condensation and normalization, where non-standard verbal elements (repetitions, signs of hesitation etc.) are omitted or replaced by more standard alternatives (Gottlieb, 2012). This is due to constraints on the length and the readability of the subtitles as well as adherence to writing conventions, especially for professional subtitle writers. As subtitles are meant to be displayed alongside audiovisual content, this is rarely a problem as viewers can typically recover omitted dialogue-relevant cues from the accompanying images and sounds.

We can distinguish between *interlingual subtitling*, where the original language of the audio is different from the subtitling language; and *intralingual subtitling*, where the audio and the subtitle language are the same with non-verbal elements also recorded in writing, typically for the benefit of hearing impaired audiences or non-native speakers (Gottlieb, 2012).

Corpora of movie and TV subtitles have been studied in both linguistics and language teaching, where it serves as an easy-to-access approximation of spoken language use (Werner, 2020). Quaglio

<footnote n="2">2. From HCRC Map Task (Anderson et al., 1991).</footnote>
(2008) note that English subtitles of the TV series Friends (1994–2004) share some features with natural conversation, while a more detailed analysis reveals a number of differences, notably in terms of expressive language. Pavesi (2020) also found that proximal demonstrative pronouns (e.g., *this*) are more frequent in cinematic discourse than in other spoken corpora. The author argued that this phenomenon is related to these aiding the narrative by directing viewers' attention towards specific characters or elements in the story.

Rühlemann (2020) compared real conversations and scripted ones and observed that continuers were absent from the latter. Prevot et al. (2019) compared data from the *Open Subtitles* corpus (Lison and Tiedemann, 2016; Lison et al., 2018) in English, French and Mandarin with both written and conversational corpora and found that OpenSubtitles occupied an intermediate position between written and conversational data in terms of lexical and syntactic features.

The present paper differs from previous works in three ways. First, our analysis rests upon a larger number of corpora distributed across eight languages. Second, we rely on both statistics extracted from superficial features (lexical occurrences, utterance lengths) as well as estimates derived from a neural dialogue act tagging model. Finally, we focus specifically on communicative feedback, ignoring other conversational phenomena.

## 3. Corpora

In our experiments, we make use of data from both OpenSubtitles and existing corpora of real conversations in eight different languages. Out of these, five are Indo-European (German, English, French, Italian and Norwegian) and three are from other language families (Hungarian, Japanese and Mandarin Chinese).

### 3.1 Spontaneous Dialogues

**German (de)** For German, we use the Hamburg MapTask corpus (HZSK, 2010), in which twelve dyads of (L2) speakers of German engage in dyadic task-oriented short dialogues.

**English (en)** Data sources for English dialogues are easily available and cover various genres. We use Switchboard, consisting of dyadic topic oriented phone conversation (Godfrey et al., 1992) as well as Fisher (Cieri et al., 2004) for some experiments; AMI, with multi-party multimodal task-oriented dialogues (Carletta, 2007); HCRC MapTask comprising dyadic task-oriented short dialogues (Anderson et al., 1991); and STAC, a multi-party negotiation chat corpus (Asher et al., 2016).

**French (fr)** The French corpora include CID consisting of dyadic, 1-hour long, loosely topic-oriented face-to-face conversations with 16 participants (Blache et al., 2017); French MapTask with 16 participants (Gorisch et al., 2014); and Aix-DVD, dyadic face-to-face conversations about movie preferences of 16 participants (Prévot et al., 2016).

**Hungarian (hu)** The source for this language is the BUSZI-2 corpus (Budapesti Szociolingvisztikai Interjú, "Budapest Sociolinguistic Interview", Váradi, 2003). The data was recorded between 1987–1989, where 50 participants with different educational levels were part of both a 30-minute directed conversation and then performed a set of language tasks. These included, for example, inserting a verb into a sentence in the correct form and grammaticality judgments.
(2008) note that English subtitles of the TV series Friends (1994–2004) share some features with natural conversation, while a more detailed analysis reveals a number of differences, notably in terms of expressive language. Pavesi (2020) also found that proximal demonstrative pronouns (e.g., *this*) are more frequent in cinematic discourse than in other spoken corpora. The author argued that this phenomenon is related to these aiding the narrative by directing viewers' attention towards specific characters or elements in the story.

Rühlemann (2020) compared real conversations and scripted ones and observed that continuers were absent from the latter. Prevot et al. (2019) compared data from the *Open Subtitles* corpus (Lison and Tiedemann, 2016; Lison et al., 2018) in English, French and Mandarin with both written and conversational corpora and found that OpenSubtitles occupied an intermediate position between written and conversational data in terms of lexical and syntactic features.

The present paper differs from previous works in three ways. First, our analysis rests upon a larger number of corpora distributed across eight languages. Second, we rely on both statistics extracted from superficial features (lexical occurrences, utterance lengths) as well as estimates derived from a neural dialogue act tagging model. Finally, we focus specifically on communicative feedback, ignoring other conversational phenomena.

## 3. Corpora

In our experiments, we make use of data from both OpenSubtitles and existing corpora of real conversations in eight different languages. Out of these, five are Indo-European (German, English, French, Italian and Norwegian) and three are from other language families (Hungarian, Japanese and Mandarin Chinese).

### 3.1 Spontaneous Dialogues

**German (de)** For German, we use the Hamburg MapTask corpus (HZSK, 2010), in which twelve dyads of (L2) speakers of German engage in dyadic task-oriented short dialogues.

**English (en)** Data sources for English dialogues are easily available and cover various genres. We use Switchboard, consisting of dyadic topic oriented phone conversation (Godfrey et al., 1992) as well as Fisher (Cieri et al., 2004) for some experiments; AMI, with multi-party multimodal task-oriented dialogues (Carletta, 2007); HCRC MapTask comprising dyadic task-oriented short dialogues (Anderson et al., 1991); and STAC, a multi-party negotiation chat corpus (Asher et al., 2016).

**French (fr)** The French corpora include CID consisting of dyadic, 1-hour long, loosely topic-oriented face-to-face conversations with 16 participants (Blache et al., 2017); French MapTask with 16 participants (Gorisch et al., 2014); and Aix-DVD, dyadic face-to-face conversations about movie preferences of 16 participants (Prévot et al., 2016).

**Hungarian (hu)** The source for this language is the BUSZI-2 corpus (Budapesti Szociolingvisztikai Interjú, "Budapest Sociolinguistic Interview", Váradi, 2003). The data was recorded between 1987–1989, where 50 participants with different educational levels were part of both a 30-minute directed conversation and then performed a set of language tasks. These included, for example, inserting a verb into a sentence in the correct form and grammaticality judgments.
<table><caption>Table 1: Overview of the spontaneous dialogue data.</caption><thead><tr><th>Language</th><th>de</th><th>en</th><th>fr</th><th>hu</th><th>it</th><th>ja</th><th>no</th><th>zh</th><th><strong>total</strong></th></tr></thead><tbody><tr><th><strong># conversations</strong></th><td>24</td><td>2766</td><td>48</td><td>50</td><td>88</td><td>120</td><td>259</td><td>120</td><td><strong>3475</strong></td></tr><tr><th><strong># utterances</strong></th><td>4K</td><td>373K</td><td>27K</td><td>31K</td><td>24K</td><td>39K</td><td>86K</td><td>18K</td><td><strong>602K</strong></td></tr></tbody></table>

<table><caption>Table 2: Overview of the subtitle data.</caption><thead><tr><th>Language</th><th>de</th><th>en</th><th>fr</th><th>hu</th><th>it</th><th>ja</th><th>no</th><th>zh</th><th><strong>total</strong></th></tr></thead><tbody><tr><th><strong># movies</strong></th><td>98</td><td>100</td><td>100</td><td>68</td><td>95</td><td>74</td><td>87</td><td>93</td><td><strong>715</strong></td></tr><tr><th><strong># utterances</strong></th><td>131K</td><td>140K</td><td>126K</td><td>93K</td><td>138K</td><td>106K</td><td>98K</td><td>114K</td><td><strong>946K</strong></td></tr></tbody></table>

**Italian (it)** We use the spontaneous dialogues from the CLIPS corpus (Savy and Cutugno, 2009), consisting of two subsets: one based on a map task, and another elicited via spotting differences between two slightly different images. Dialogues with a high proportion (> 10%) of utterances with dialectal words were excluded. All markups indicating comments, unfilled pauses etc. were removed, but markups containing filled pauses (e.g., <mh>) were kept and reduced to plain text in the utterances since many were connected to feedback.

**Japanese (ja)** We employ the transcripts of the CallHome Japanese corpus (Den and Fry, 2000) consisting of 120 unscripted telephone conversations between native speakers, mostly family members or close friends.

**Norwegian (no)** The Norwegian data consists of the NoTa-Oslo corpus (Johannessen et al., 2007), containing interviews and conversations from 2004–2006 with 166 informants from the Oslo area. The informants were selected to be representative in terms of age, gender and education. The dialogues consist of 10-minute semi-formal interviews and 30-min informal dialogues with other informants.

**Mandarin Chinese (zh)** The source of our Mandarin Chinese data was CALLHOME (Wheatley, 1996) consisting of unscripted telephone conversations between native speakers.

## 3.2 Subtitles

The scripted dialogues are extracted from OpenSubtitles 2018 (Lison et al., 2018), a large collection of subtitles of over 3.7 million subtitles (amounting to ≈ 22.1 billion words) extracted from the OpenSubtitles.org database and covering 60 languages. We include both (1) subtitles for the hearing impaired, where the subtitle language and the language employed in the movie/episode are identical and (2) subtitles for foreign audiences as two separate subtitle subsets. The subtitles are then filtered according to the following criteria:

1. We only consider recent movies (year ≥ 1990) to reflect contemporary language use, in line with the corpora of spontaneous conversations.

2. We ignore subtitles with less than 100 utterances and exclude genres that are less relevant for the present study (Documentary, Reality-TV, Biography, Sport, Musical, Music, Adult, Animation, Short and Game-Show).
<table><caption>Table 1: Overview of the spontaneous dialogue data.</caption><thead><tr><th>Language</th><th>de</th><th>en</th><th>fr</th><th>hu</th><th>it</th><th>ja</th><th>no</th><th>zh</th><th><strong>total</strong></th></tr></thead><tbody><tr><th># conversations</th><td>24</td><td>2766</td><td>48</td><td>50</td><td>88</td><td>120</td><td>259</td><td>120</td><td><strong>3475</strong></td></tr><tr><th># utterances</th><td>4K</td><td>373K</td><td>27K</td><td>31K</td><td>24K</td><td>39K</td><td>86K</td><td>18K</td><td><strong>602K</strong></td></tr></tbody></table>

<table><caption>Table 2: Overview of the subtitle data.</caption><thead><tr><th>Language</th><th>de</th><th>en</th><th>fr</th><th>hu</th><th>it</th><th>ja</th><th>no</th><th>zh</th><th><strong>total</strong></th></tr></thead><tbody><tr><th># movies</th><td>98</td><td>100</td><td>100</td><td>68</td><td>95</td><td>74</td><td>87</td><td>93</td><td><strong>715</strong></td></tr><tr><th># utterances</th><td>131K</td><td>140K</td><td>126K</td><td>93K</td><td>138K</td><td>106K</td><td>98K</td><td>114K</td><td><strong>946K</strong></td></tr></tbody></table>

**Italian (it)** We use the spontaneous dialogues from the CLIPS corpus (Savy and Cutugno, 2009), consisting of two subsets: one based on a map task, and another elicited via spotting differences between two slightly different images. Dialogues with a high proportion (> 10%) of utterances with dialectal words were excluded. All markups indicating comments, unfilled pauses etc. were removed, but markups containing filled pauses (e.g., <mh>) were kept and reduced to plain text in the utterances since many were connected to feedback.

**Japanese (ja)** We employ the transcripts of the CallHome Japanese corpus (Den and Fry, 2000) consisting of 120 unscripted telephone conversations between native speakers, mostly family members or close friends.

**Norwegian (no)** The Norwegian data consists of the NoTa-Oslo corpus (Johannessen et al., 2007), containing interviews and conversations from 2004–2006 with 166 informants from the Oslo area. The informants were selected to be representative in terms of age, gender and education. The dialogues consist of 10-minute semi-formal interviews and 30-min informal dialogues with other informants.

**Mandarin Chinese (zh)** The source of our Mandarin Chinese data was CALLHOME (Wheatley, 1996) consisting of unscripted telephone conversations between native speakers.

## 3.2 Subtitles

The scripted dialogues are extracted from OpenSubtitles 2018 (Lison et al., 2018), a large collection of subtitles of over 3.7 million subtitles (amounting to ≈ 22.1 billion words) extracted from the OpenSubtitles.org database and covering 60 languages. We include both (1) subtitles for the hearing impaired, where the subtitle language and the language employed in the movie/episode are identical and (2) subtitles for foreign audiences as two separate subtitle subsets. The subtitles are then filtered according to the following criteria:

1. We only consider recent movies (year ≥ 1990) to reflect contemporary language use, in line with the corpora of spontaneous conversations.

2. We ignore subtitles with less than 100 utterances and exclude genres that are less relevant for the present study (Documentary, Reality-TV, Biography, Sport, Musical, Music, Adult, Animation, Short and Game-Show).
We select ten movies per audience type (hearing impaired vs. foreign viewers) from the five
genres with the largest number of subtitles available in our English data, which were, in decreasing
order of frequency: drama, comedy, crime, action, and romance.<fnref n="3" /> A movie could involve more than
one genres thus we also check for this. Table 2 shows the amount of movies and utterances per
language for the selected subtitles.<fnref n="4" />

4. Lexical Statistics

We start by computing lexical distributions to compare the datasets presented in the previous section with regard to communicative feedback patterns.

4.1 Lexical Overview of Short Utterances

The lexical statistics of acknowledgment cues can give a first picture of the feedback content in subtitles. In spoken dialogue, acknowledgments tend to be produced by the addressee (not the speaker) and therefore they are short productions uttered in overlap and potentially with a lower voice. Out of those three properties (brevity, overlap, lower volume), only the first one is accessible in our dataset, as the subtitles (and the STAC corpus for English) are purely text-based. Given the correlation between acknowledgments and utterance length, we start by analyzing “very short utterances” (Edlund et al., 2009), defined here as three tokens or less.

We rely on the *scatter text* visualization (Kessler, 2017, see Dingemanse and Liesenfeld 2022 for a
similar use) to get a first overview of lexical differences between subtitles and spontaneous dialogues.
Figure 1 compares English-language romance subtitles with Switchboard corpus by plotting the
scaled *f*-score for each term against both corpora. This scaled *f*-score is computed as the harmonic
mean of *term precision*, that is, the number of times a word occurs in a category divided by the
number of times the word occurs in all categories; and *term frequency*, i.e., the number of times a
word occurs in a category divided by the number of words in that category. In this figure, the more a
token is located in the bottom right corner (marked with dark red dots), the more it is over-represented
in spontaneous dialogues; while items in the upper-left corner (dark blue dots) are frequent words in
subtitles. Tokens close to the diagonal, represented with yellow and orange dots, are associated with
both datasets.

In Figure 1, besides some preprocessing differences (e.g., *that's*, *bye-bye*, @, (laughter)), we can
observe a number of acknowledgment cues that occur more often in spontaneous speech, such as
gosh, yep, etc. and to a lesser extent wow, hum, uh, etc., which also occur in subtitles but relatively
less frequently. On the other hand, the tokens over-represented in subtitles often relate to action
verbs (*come*, *listen*), vocative expressions (*dad*, *mom*), politeness forms (*thanks*, *please*), although
some tokens can also be related to feedback (e.g., *hey*, *alright*, *damn*). See Appendix A for the other
languages and genres.

<footnote n="3">3. The number of available movies for the hearing impaired subset is < 10 for the following languages and genres: de-action (8), it-action (5), no-action (3), no-romance (4). This is mostly due to missing information about the original language of the movie in OpenSubtitles. To increase the number of available subtitles to work with for Japanese, we also included Animation among the listed genres for this language.</footnote>

<footnote n="4">4. In subtitles, and for several other corpora, we did not have a segmentation into properly defined utterances. OpenSubtitles “utterances” included subtitling constraints. As a consequence, for simplicity reasons, we use the term “utterance” in the paper, but it should be understood broadly.</footnote>
We select ten movies per audience type (hearing impaired vs. foreign viewers) from the five
genres with the largest number of subtitles available in our English data, which were, in decreasing
order of frequency: drama, comedy, crime, action, and romance.<fnref n="3" /> A movie could involve more than
one genres thus we also check for this. Table 2 shows the amount of movies and utterances per
language for the selected subtitles.<fnref n="4" />

4. Lexical Statistics

We start by computing lexical distributions to compare the datasets presented in the previous section with regard to communicative feedback patterns.

## 4.1 Lexical Overview of Short Utterances

The lexical statistics of acknowledgment cues can give a first picture of the feedback content in subtitles. In spoken dialogue, acknowledgments tend to be produced by the addressee (not the speaker) and therefore they are short productions uttered in overlap and potentially with a lower voice. Out of those three properties (brevity, overlap, lower volume), only the first one is accessible in our dataset, as the subtitles (and the STAC corpus for English) are purely text-based. Given the correlation between acknowledgments and utterance length, we start by analyzing “very short utterances” (Edlund et al., 2009), defined here as three tokens or less.

We rely on the *scatter text* visualization (Kessler, 2017, see Dingemanse and Liesenfeld 2022 for a similar use) to get a first overview of lexical differences between subtitles and spontaneous dialogues. Figure 1 compares English-language *romance* subtitles with Switchboard corpus by plotting the scaled *f*-score for each term against both corpora. This scaled *f*-score is computed as the harmonic mean of *term precision*, that is, the number of times a word occurs in a category divided by the number of times the word occurs in all categories; and *term frequency*, i.e., the number of times a word occurs in a category divided by the number of words in that category. In this figure, the more a token is located in the bottom right corner (marked with dark red dots), the more it is over-represented in spontaneous dialogues; while items in the upper-left corner (dark blue dots) are frequent words in subtitles. Tokens close to the diagonal, represented with yellow and orange dots, are associated with both datasets.

In Figure 1, besides some preprocessing differences (e.g., *that's*, *bye-bye*, @, (laughter)), we can
observe a number of acknowledgment cues that occur more often in spontaneous speech, such as
gosh, yep, etc. and to a lesser extent wow, hum, uh, etc., which also occur in subtitles but relatively
less frequently. On the other hand, the tokens over-represented in subtitles often relate to action
verbs (*come*, *listen*), vocative expressions (*dad*, *mom*), politeness forms (*thanks*, *please*), although
some tokens can also be related to feedback (e.g., *hey*, *alright*, *damn*). See Appendix A for the other
languages and genres.

<footnote n="3">3. The number of available movies for the hearing impaired subset is < 10 for the following languages and genres: de-action (8), it-action (5), no-action (3), no-romance (4). This is mostly due to missing information about the original language of the movie in OpenSubtitles. To increase the number of available subtitles to work with for Japanese, we also included Animation among the listed genres for this language.</footnote>

<footnote n="4">4. In subtitles, and for several other corpora, we did not have a segmentation into properly defined utterances. OpenSubtitles “utterances” included subtitling constraints. As a consequence, for simplicity reasons, we use the term “utterance” in the paper, but it should be understood broadly.</footnote>
<figure><img src="image_3.png" /><figcaption>Figure 1: Scatter text visualization for short utterances (comprising three tokens or less) in English-language subtitles (*romance* genre) vs. spontaneous dialogues from Switchboard.</figcaption></figure>

## 4.2 Feedback Utterances

To target more precisely lexical items related to communicative feedback, we extract the uni-, bi- and trigrams of very short utterances from both subtitles and spontaneous dialogues. Moreover, feedback can also be added in initial position of longer turns/contributions. When extracting feedback-related lexical material, we therefore extract two locations: ‘very short utterances’ (less than three tokens) and initial positions (one token) of all other utterances. Comparing term frequencies between these locations and the overall corpus allowed us to design language-specific cue word lists. Those lists, provided in Table 9 in Appendix A, distinguish between four core classes of feedback:

* positive feedback/acknowledgment (+)

* neutral/continuer (=)

* negative feedback (-)

* clarification request (?).

We plot the proportion of utterances corresponding to these definitions in each corpus. In Figures 2, 3 and 4, plots (a) illustrate the proportion of utterances mapped to the above labels across all utterances in the corpus and plots (b) focus on the utterances that have received a label other than OTH. Figure 5 shows the lexical distribution of the most frequent lexical items observed in the utterances of plots (b) for English and French.

As can be seen in the figures, the cue-based statistics differ radically between subtitle datasets and spontaneous dialogue. This difference is observed across all languages and sub-genres, (see
<figure><img src="image_2.png" /><figcaption>Figure 1: Scatter text visualization for short utterances (comprising three tokens or less) in English-language subtitles (*romance* genre) vs. spontaneous dialogues from Switchboard.</figcaption></figure>

## 4.2 Feedback Utterances

To target more precisely lexical items related to communicative feedback, we extract the uni-, bi- and trigrams of very short utterances from both subtitles and spontaneous dialogues. Moreover, feedback can also be added in initial position of longer turns/contributions. When extracting feedback-related lexical material, we therefore extract two locations: ‘very short utterances’ (less than three tokens) and initial positions (one token) of all other utterances. Comparing term frequencies between these locations and the overall corpus allowed us to design language-specific cue word lists. Those lists, provided in Table 9 in Appendix A, distinguish between four core classes of feedback:

* positive feedback/acknowledgment (+)

* neutral/continuer (=)

* negative feedback (-)

* clarification request (?).

We plot the proportion of utterances corresponding to these definitions in each corpus. In Figures 2, 3 and 4, plots (a) illustrate the proportion of utterances mapped to the above labels across all utterances in the corpus and plots (b) focus on the utterances that have received a label other than OTH. Figure 5 shows the lexical distribution of the most frequent lexical items observed in the utterances of plots (b) for English and French.

As can be seen in the figures, the cue-based statistics differ radically between subtitle datasets and spontaneous dialogue. This difference is observed across all languages and sub-genres, (see
<figure><img src="image_4.png" /><figcaption>Figure 2: English across genres (rule-based, based on cue word lists).</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Figure 3: French across genres (rule-based, based on cue word lists).</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>Figure 4: Mandarin Chinese across genres (rule-based, based on cue word lists).</figcaption></figure>
<figure><img src="image_3.png" /><figcaption>Figure 2: English across genres (rule-based, based on cue word lists).</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>Figure 3: French across genres (rule-based, based on cue word lists).</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Figure 4: Mandarin Chinese across genres (rule-based, based on cue word lists).</figcaption></figure>
<figure><img src="image_7.png" /><figcaption>Figure 5: Most common lexical items found in feedback utterances in English and French.</figcaption></figure>

Appendix A for other languages). Given the noisy nature of our data, we worked to identify and reduce other sources of divergence between our sub-corpora. STAC, as a chat corpus, present a different picture than other dialogue corpora, notably due to the extensive use of emojis. Similarly, for English and French, we explored the impact of expressions of politeness (highly frequent in OpenSubtitles) on the provided statistics. In both cases those particularities contribute significantly to the distributions but did not change the overall picture of our analysis (see Figure 13 in Appendix A).

One key difference between the corpora relates to the overall frequency of feedback cues (labeled as positive, neutral, negative, or expressing a clarification request), which is much higher in spontaneous dialogues (40–50%) than in subtitles (10–20%), as observed in the plots (a). Furthermore, as shown in the plots (b), feedback in spontaneous dialogues consists mostly of positive or neutral (continuers) feedback, while subtitles have very few neutral signals but exhibit a much higher proportion of negative feedback and clarification request. This may be explained by the prevalence of “confrontational” dialogues in movie and TV subtitles, which are more useful from a narrative perspective.

Feedback utterances are not only expressed through a restricted set of lexical items. Repeated material is another important clue to spot such utterances. In the case of continuers and assessments, lexical items are dominating the phenomena. Clarification requests are more difficult to assess, as those are often formulated as repeated material requiring clarification. We tried to extract those as well, but faced the problem of turn alternations. Most of our spontaneous dialogue corpora are well segmented in speaker turns, a new turn indicating a change of speakers. Speaker information is unfortunately absent from subtitles (Lison and Meena, 2016), where segments merely indicate a continuous sequence displayed on screen, without indication of turn boundaries. We also observed that compared to spontaneous dialogues, subtitles host a large number of verbatim repeats produced by the movie situations (e.g someone not answering being called) and emphasis for more lively discussion (e.g question repeated several time because lack of answers). As a consequence, heuristics to detect clarification request by repetitions are difficult to establish for Open Subtitles.

## 4.3 Length Distributions

The differences in lexical statistics between OpenSubtitles and corpora of spontaneous dialogues are substantial. To shed light on those differences, Figure 6 shows the length (in tokens) of utterances for three English datasets. We compared the lengths distribution for utterances classified as feedback/grounding (gr, in blue) or not (oth, in red) based on our cue-word extraction. While in
<figure><img src="image_6.png" /><figcaption>Figure 5: Most common lexical items found in feedback utterances in English and French.</figcaption></figure>

Appendix A for other languages). Given the noisy nature of our data, we worked to identify and reduce other sources of divergence between our sub-corpora. STAC, as a chat corpus, present a different picture than other dialogue corpora, notably due to the extensive use of emojis. Similarly, for English and French, we explored the impact of expressions of politeness (highly frequent in OpenSubtitles) on the provided statistics. In both cases those particularities contribute significantly to the distributions but did not change the overall picture of our analysis (see Figure 13 in Appendix A).

One key difference between the corpora relates to the overall frequency of feedback cues (labeled as positive, neutral, negative, or expressing a clarification request), which is much higher in spontaneous dialogues (40–50%) than in subtitles (10–20%), as observed in the plots (a). Furthermore, as shown in the plots (b), feedback in spontaneous dialogues consists mostly of positive or neutral (continuers) feedback, while subtitles have very few neutral signals but exhibit a much higher proportion of negative feedback and clarification request. This may be explained by the prevalence of “confrontational” dialogues in movie and TV subtitles, which are more useful from a narrative perspective.

Feedback utterances are not only expressed through a restricted set of lexical items. Repeated material is another important clue to spot such utterances. In the case of continuers and assessments, lexical items are dominating the phenomena. Clarification requests are more difficult to assess, as those are often formulated as repeated material requiring clarification. We tried to extract those as well, but faced the problem of turn alternations. Most of our spontaneous dialogue corpora are well segmented in speaker turns, a new turn indicating a change of speakers. Speaker information is unfortunately absent from subtitles (Lison and Meena, 2016), where segments merely indicate a continuous sequence displayed on screen, without indication of turn boundaries. We also observed that compared to spontaneous dialogues, subtitles host a large number of verbatim repeats produced by the movie situations (e.g someone not answering being called) and emphasis for more lively discussion (e.g question repeated several time because lack of answers). As a consequence, heuristics to detect clarification request by repetitions are difficult to establish for Open Subtitles.

## 4.3 Length Distributions

The differences in lexical statistics between OpenSubtitles and corpora of spontaneous dialogues are substantial. To shed light on those differences, Figure 6 shows the length (in tokens) of utterances for three English datasets. We compared the lengths distribution for utterances classified as feedback/grounding (gr, in blue) or not (oth, in red) based on our cue-word extraction. While in
<figure><img src="image_8.png" /><figcaption>Figure 6: Utterance lengths distribution (in tokens) across different English datasets.</figcaption></figure>

spontaneous dialogue, utterances of 1 or 2 tokens are overwhelmingly feedback-related signals, this
is not the case for subtitles, where many very short utterances relate to verbs, vocative expressions or
politeness forms (see Section 4.1). Subtitles are indeed designed to be displayed under strict time
constraints, thereby favoring short, action-oriented utterances.

## 4.4 Evaluation of the Cue Word Lists

To gain insights into the reliability of the above estimates, we evaluate the quality of our English cue word lists on a dataset annotated with dialogue acts, namely the Switchboard corpus (Stolcke et al., 1998). We make a binary distinction and group, on the one hand, all feedback-related labels into a *Feedback* category, where we included the same subset of Switchboard (SWBD) tags as for the *Backchannel* and the *Assessment* classes used for our neural tagger (see Table 4 for details). On the other hand, all labels that are not connected to communicative feedback are merged into a single *Other* category. We find that the cue word lists have an accuracy of 85%, an F1 of 0.76 and a recall of 0.89 for feedback cues with respect to the annotated SWBD labels (Table 3). They can thus serve as a useful basis for frequency estimates for comparing spontaneous and scripted dialogues.<fnref n="5" />

<footnote n="5">5. It is worth noting again in connection to these performance scores that the cue word lists were compiled, in part, by inspecting token frequencies in our corpora (see Section 4.2), including Switchboard, a higher recall is therefore expected, but not necessarily a high precision.</footnote>
<figure><img src="image_7.png" /><figcaption>Figure 6: Utterance lengths distribution (in tokens) across different English datasets.</figcaption></figure>

spontaneous dialogue, utterances of 1 or 2 tokens are overwhelmingly feedback-related signals, this
is not the case for subtitles, where many very short utterances relate to verbs, vocative expressions or
politeness forms (see Section 4.1). Subtitles are indeed designed to be displayed under strict time
constraints, thereby favoring short, action-oriented utterances.

## 4.4 Evaluation of the Cue Word Lists

To gain insights into the reliability of the above estimates, we evaluate the quality of our English cue word lists on a dataset annotated with dialogue acts, namely the Switchboard corpus (Stolcke et al., 1998). We make a binary distinction and group, on the one hand, all feedback-related labels into a *Feedback* category, where we included the same subset of Switchboard (SWBD) tags as for the *Backchannel* and the *Assessment* classes used for our neural tagger (see Table 4 for details). On the other hand, all labels that are not connected to communicative feedback are merged into a single *Other* category. We find that the cue word lists have an accuracy of 85%, an F1 of 0.76 and a recall of 0.89 for feedback cues with respect to the annotated SWBD labels (Table 3). They can thus serve as a useful basis for frequency estimates for comparing spontaneous and scripted dialogues.<fnref n="5" />

<footnote n="5">5. It is worth noting again in connection to these performance scores that the cue word lists were compiled, in part, by inspecting token frequencies in our corpora (see Section 4.2), including Switchboard, a higher recall is therefore expected, but not necessarily a high precision.</footnote>
<table><caption>Table 3: Results for the English cue word lists applied on the Switchboard corpus.</caption><thead><tr><th>Utterance type</th><th>P</th><th>R</th><th>F<sub>1</sub></th><th># instances</th></tr></thead><tbody><tr><td>Feedback</td><td>0.67</td><td>0.89</td><td>0.76</td><td>56,744</td></tr><tr><td>Other</td><td>0.95</td><td>0.84</td><td>0.89</td><td>153,835</td></tr><tr><td>Macro avg</td><td>0.81</td><td>0.86</td><td>0.83</td><td>210,579</td></tr><tr><td>Weighted avg</td><td>0.88</td><td>0.85</td><td>0.86</td><td>210,579</td></tr></tbody></table>

The cue word lists overestimate somewhat the proportion of feedback (26.9% vs. the annotated 35.5%). False positive feedback include a few cases with answers to yes-no questions and *well* in utterance initial position. We find that the DAMSL Switchboard label *Abandoned or Turn-Exit* includes some feedback-related phenomena e.g., *Uh d* together with other types of utterances unrelated to feedback (e.g., *but she was*). Given its miscellaneous nature, we have excluded this label from our feedback category and thus, when cue words identify these as feedback, they count towards incorrect matches even if they indeed contain feedback cues. Sources of false negatives include demonstratives (*That*) and other words and phrases with multiple communicative functions (*Do you?*, *Pardon me?*) which are not covered by our lists.

# 5. Neural Dialogue Act Tagging

Although the above statistics allowed us to uncover substantial differences in subtitles and spontaneous dialogues, they remain somewhat noisy estimates, as many feedback-related lexical items tend to be ambiguous. In this section, we refine our analysis by employing a neural dialogue act tagging model to quantify more precisely the frequency of a more varied set of feedback signals in subtitles and spontaneous dialogues.

## 5.1 Data

As training data for our dialogue act model, we use the DAMSL-Switchboard corpus (Stolcke et al., 1998; Godfrey et al., 1992), where we map the original (finer-grained) set of Switchboard (SWBD) tags, and their clustered DAMSL-SWBD equivalents, into five coarse dialogue act (DA) classes: *Forward looking*, *Yes/no answers*, *Assessment*, *Backchannel* and *Other*. The two classes most directly relevant for feedback, namely *Backchannel* and *Assessment*, are inspired, in part, by Mezza et al. (2018). Distinguishing between these two feedback-related classes is also motivated by Goodwin (1986), who outline a number of positional and functional differences between these. The *Backchannel* category consists of the SWBD-DAMSL labels *Acknowledge* (*Backchannel*), (SWBD tag b), *Backchannel in question form* (bh), *Response Acknowledgment* (bk), *Summarize/reformulate* (bf) and *Signal-non-understanding* (br). As this latter tag shows, negative feedback signals are also part of the *Backchannel* category, which is motivated by the small amount of relevant instances that would likely not be sufficient for reliably learning a separate class for these. The *Assessment* category comprises not only the labels *Agree/Accept* (aa), but also *Appreciation* (ba) and *Exclamation* (fe). The forward looking category contains utterances expressing explanations, instructions and suggestions
<table><caption>Table 3: Results for the English cue word lists applied on the Switchboard corpus.</caption><thead><tr><th>Utterance type</th><th>P</th><th>R</th><th>F<sub>1</sub></th><th># instances</th></tr></thead><tbody><tr><td>Feedback</td><td>0.67</td><td>0.89</td><td>0.76</td><td>56,744</td></tr><tr><td>Other</td><td>0.95</td><td>0.84</td><td>0.89</td><td>153,835</td></tr><tr><td>Macro avg</td><td>0.81</td><td>0.86</td><td>0.83</td><td>210,579</td></tr><tr><td>Weighted avg</td><td>0.88</td><td>0.85</td><td>0.86</td><td>210,579</td></tr></tbody></table>

The cue word lists overestimate somewhat the proportion of feedback (26.9% vs. the annotated 35.5%). False positive feedback include a few cases with answers to yes-no questions and *well* in utterance initial position. We find that the DAMSL Switchboard label *Abandoned or Turn-Exit* includes some feedback-related phenomena e.g., *Uh d* together with other types of utterances unrelated to feedback (e.g., *but she was*). Given its miscellaneous nature, we have excluded this label from our feedback category and thus, when cue words identify these as feedback, they count towards incorrect matches even if they indeed contain feedback cues. Sources of false negatives include demonstratives (*That*) and other words and phrases with multiple communicative functions (*Do you?*, *Pardon me?*) which are not covered by our lists.

# 5. Neural Dialogue Act Tagging

Although the above statistics allowed us to uncover substantial differences in subtitles and spontaneous dialogues, they remain somewhat noisy estimates, as many feedback-related lexical items tend to be ambiguous. In this section, we refine our analysis by employing a neural dialogue act tagging model to quantify more precisely the frequency of a more varied set of feedback signals in subtitles and spontaneous dialogues.

## 5.1 Data

As training data for our dialogue act model, we use the DAMSL-Switchboard corpus (Stolcke et al., 1998; Godfrey et al., 1992), where we map the original (finer-grained) set of Switchboard (SWBD) tags, and their clustered DAMSL-SWBD equivalents, into five coarse dialogue act (DA) classes: *Forward looking*, *Yes/no answers*, *Assessment*, *Backchannel* and *Other*. The two classes most directly relevant for feedback, namely *Backchannel* and *Assessment*, are inspired, in part, by Mezza et al. (2018). Distinguishing between these two feedback-related classes is also motivated by Goodwin (1986), who outline a number of positional and functional differences between these. The *Backchannel* category consists of the SWBD-DAMSL labels *Acknowledge* (*Backchannel*), (SWBD tag b), *Backchannel in question form* (bh), *Response Acknowledgment* (bk), *Summarize/reformulate* (bf) and *Signal-non-understanding* (br). As this latter tag shows, negative feedback signals are also part of the *Backchannel* category, which is motivated by the small amount of relevant instances that would likely not be sufficient for reliably learning a separate class for these. The *Assessment* category comprises not only the labels *Agree/Accept* (aa), but also *Appreciation* (ba) and *Exclamation* (fe). The forward looking category contains utterances expressing explanations, instructions and suggestions
<table><caption>Table 4: SWBD labels mapped to coarse-grained dialogue act groups.</caption><thead><tr><th>Mapped DA group</th><th># instances</th><th>SWBD labels</th></tr></thead><tbody><tr><td>Forward looking</td><td>109,382</td><td>sd, fx/sv, bf, na, ny'e, arp, nd, no, cc, co, oo, ad, qr/qy, qw, qw'd, qh, qo</td></tr><tr><td>Backchannel</td><td>41,017</td><td>b, bk, bh, bf, br</td></tr><tr><td>Assessment</td><td>15,727</td><td>aa, fe/ba</td></tr><tr><td>Yes/no answer</td><td>4,324</td><td>ny, nn</td></tr><tr><td>Other</td><td>40,124</td><td>all other categories</td></tr><tr><td>Total</td><td>210,574</td><td></td></tr></tbody></table>

as well as questions. Table 4 shows the distribution of instances per label and their corresponding SWBD tag.<fnref n="6" />

## 5.2 Model Training

We opt for BERT as our neural architecture (Devlin et al., 2019) for the sequence labelling. We fine-tune the monolingual bert-base-cased pre-trained model using 80% of the (cased) Switchboard data as training data. The remaining 20% is divided equally between a development and a test set. We set up the task as a sequence classification problem, where the preceding utterance is included as context. Since the dataset is highly unbalanced, with some of the classes of interest having considerably fewer instances than the majority classes, we apply class-weighted training.<fnref n="7" /> We train the model for five epochs with default early stopping parameters, except for patience, which we set to 5. The batch size is set to 8 and the learning rate to 4e-5. We run and compare three different random seeds, which achieve comparable performance.<fnref n="8" /> We adopt the best performing model out of the three tested seeds, which is trained for three epochs before exiting with early stopping.

To further improve the recall for feedback phenomena, we also adjust the probability threshold for assigning an output label, and raise the threshold to 0.8 and 0.6 for the *Forward looking* and *Other* categories respectively and we lower the threshold to 0.25 for the *Yes/no answer* and *Assessment* classes. In case none of the predicted labels are above these thresholds, we pick the label with the highest probability excluding *Forward looking* to account for a potential majority class bias.

Table 5 provides details on the tagging performance of the neural dialogue act tagger on the test set of the Switchboard corpus. The model performs relatively well overall, with an accuracy of 0.81. The detection of the feedback-related classes varies, with *Backchannel* being detected rather well (0.84 precision, 0.83 recall) while *Assessment* achieving somewhat lower scores (0.53 precision, 0.67 recall). See Appendix B for more detailed results reported for each individual corpus.

<footnote n="6">6. web.stanford.edu/jurafsky/ws97/manual.august1.html</footnote>

<footnote n="7">7. The class weights are set to 0.7 for forward-looking utterances and to 0.9 for the other classes.</footnote>

<footnote n="8">8. The average performance for the three runs is 0.73(±0.03) macro F1 and 0.83(±0.01) weighted F1 before modifying the standard probability thresholds for predictions.</footnote>
<table><caption>Table 4: SWBD labels mapped to coarse-grained dialogue act groups.</caption><thead><tr><th>Mapped DA group</th><th># instances</th><th>SWBD labels</th></tr></thead><tbody><tr><td>Forward looking</td><td>109,382</td><td>sd, fx/sv, bf, na, ny'e, arp, nd, no, cc, co, oo, ad, qr/qy, qw, qw'd, qh, qo</td></tr><tr><td>Backchannel</td><td>41,017</td><td>b, bk, bh, bf, br</td></tr><tr><td>Assessment</td><td>15,727</td><td>aa, fe/ba</td></tr><tr><td>Yes/no answer</td><td>4,324</td><td>ny, nn</td></tr><tr><td>Other</td><td>40,124</td><td>all other categories</td></tr><tr><td>Total</td><td>210,574</td><td></td></tr></tbody></table>

as well as questions. Table 4 shows the distribution of instances per label and their corresponding SWBD tag.<fnref n="6" />

## 5.2 Model Training

We opt for BERT as our neural architecture (Devlin et al., 2019) for the sequence labelling. We fine-tune the monolingual bert-base-cased pre-trained model using 80% of the (cased) Switchboard data as training data. The remaining 20% is divided equally between a development and a test set. We set up the task as a sequence classification problem, where the preceding utterance is included as context. Since the dataset is highly unbalanced, with some of the classes of interest having considerably fewer instances than the majority classes, we apply class-weighted training.<fnref n="7" /> We train the model for five epochs with default early stopping parameters, except for patience, which we set to 5. The batch size is set to 8 and the learning rate to 4e-5. We run and compare three different random seeds, which achieve comparable performance.<fnref n="8" /> We adopt the best performing model out of the three tested seeds, which is trained for three epochs before exiting with early stopping.

To further improve the recall for feedback phenomena, we also adjust the probability threshold for assigning an output label, and raise the threshold to 0.8 and 0.6 for the *Forward looking* and *Other* categories respectively and we lower the threshold to 0.25 for the *Yes/no answer* and *Assessment* classes. In case none of the predicted labels are above these thresholds, we pick the label with the highest probability excluding *Forward looking* to account for a potential majority class bias.

Table 5 provides details on the tagging performance of the neural dialogue act tagger on the test set of the Switchboard corpus. The model performs relatively well overall, with an accuracy of 0.81. The detection of the feedback-related classes varies, with *Backchannel* being detected rather well (0.84 precision, 0.83 recall) while *Assessment* achieving somewhat lower scores (0.53 precision, 0.67 recall). See Appendix B for more detailed results reported for each individual corpus.

<footnote n="6">6. web.stanford.edu/jurafsky/ws97/manual.august1.html</footnote>

<footnote n="7">7. The class weights are set to 0.7 for forward-looking utterances and to 0.9 for the other classes.</footnote>

<footnote n="8">8. The average performance for the three runs is 0.73(±0.03) macro F1 and 0.83(±0.01) weighted F1 before modifying the standard probability thresholds for predictions.</footnote>
<table><caption>Table 5: Results for the best performing neural tagger on the Switchboard test set.</caption><thead><tr><th>Dialogue act group</th><th>P</th><th>R</th><th>F<sub>1</sub></th><th># instances</th></tr></thead><tbody><tr><td>Forward looking</td><td>0.93</td><td>0.83</td><td>0.88</td><td>10,938</td></tr><tr><td>Other</td><td>0.66</td><td>0.78</td><td>0.71</td><td>4,012</td></tr><tr><td>Assessment</td><td>0.53</td><td>0.67</td><td>0.59</td><td>1,573</td></tr><tr><td>Backchannel</td><td>0.84</td><td>0.83</td><td>0.83</td><td>4,102</td></tr><tr><td>Yes/no answer</td><td>0.67</td><td>0.76</td><td>0.71</td><td>432</td></tr><tr><td>Macro avg</td><td>0.73</td><td>0.77</td><td>0.75</td><td>21,057</td></tr><tr><td>Weighted avg</td><td>0.83</td><td>0.81</td><td>0.81</td><td>21,057</td></tr></tbody></table>

## 5.3 Detection Results for Dialogue Act Groups

We then apply the trained dialogue act tagger on all spontaneous dialogue corpora (except Switchboard) and the subtitle data described in Section 3.2 to automatically detect linguistic signals of conversational feedback.

Before applying the tagger on non-English corpora, we machine translate the data using the Google Translate API.<fnref n="9" /> This choice was preferable to training dialogue act tagging models for each of the languages separately as the quality of current machine translation systems is generally considered high enough for being a viable alternative (Isbister et al., 2021) and publicly available conversational corpora with dialogue act annotations is often scarce or nonexistent for languages other than English. Furthermore, preliminary results with bert-base-multilingual-cased have also indicated poorer performance compared to the English model applied on translated data.

To make our data sources more consistent and less noisy, we filter out utterances containing a single character as well as markers representing meta-linguistic information in some spontaneous dialogue corpora (e.g., double brackets in Fischer and smiley faces in STAC). Those markers are also removed from the Switchboard data before both training and testing.

Table 6 presents the results obtained after applying the dialogue act tagger on our spontaneous dialogues and subtitle corpora for all eight languages included in this study. From this we can observe that *Backchannel* instances are considerably more frequent (by a factor three) in spontaneous dialogues than in subtitles for half of the languages – which is in line with the results of our lexical analysis in Section 4. For Hungarian, Italian and Chinese, the difference between the two types of corpora in instances categorized as *Backchannel* is somewhat smaller than for the other languages, but subtitles still contain fewer backchannels. Moreover, *Backchannel* instances are slightly more frequent in subtitles than in spontaneous conversations for Norwegian based on the tagger output. The number of utterances labeled as *Assessment* differs less, but subtitles still seem to contain fewer utterances of this feedback type for almost all genres and languages, except French (see Appendix B for detailed results).

## 5.4 Error Analysis

To have a better understanding of tagger performance and the frequency estimates of feedback cues relying on it, we conduct a more in-depth examination of the tagger's output.

<footnote n="9">9. https://cloud.google.com/translate</footnote>
<table><caption>Table 5: Results for the best performing neural tagger on the Switchboard test set.</caption><thead><tr><th>Dialogue act group</th><th>P</th><th>R</th><th>F<sub>1</sub></th><th># instances</th></tr></thead><tbody><tr><td>Forward looking</td><td>0.93</td><td>0.83</td><td>0.88</td><td>10,938</td></tr><tr><td>Other</td><td>0.66</td><td>0.78</td><td>0.71</td><td>4,012</td></tr><tr><td>Assessment</td><td>0.53</td><td>0.67</td><td>0.59</td><td>1,573</td></tr><tr><td>Backchannel</td><td>0.84</td><td>0.83</td><td>0.83</td><td>4,102</td></tr><tr><td>Yes/no answer</td><td>0.67</td><td>0.76</td><td>0.71</td><td>432</td></tr><tr><td>Macro avg</td><td>0.73</td><td>0.77</td><td>0.75</td><td>21,057</td></tr><tr><td>Weighted avg</td><td>0.83</td><td>0.81</td><td>0.81</td><td>21,057</td></tr></tbody></table>

## 5.3 Detection Results for Dialogue Act Groups

We then apply the trained dialogue act tagger on all spontaneous dialogue corpora (except Switchboard) and the subtitle data described in Section 3.2 to automatically detect linguistic signals of conversational feedback.

Before applying the tagger on non-English corpora, we machine translate the data using the Google Translate API.<fnref n="9" /> This choice was preferable to training dialogue act tagging models for each of the languages separately as the quality of current machine translation systems is generally considered high enough for being a viable alternative (Isbister et al., 2021) and publicly available conversational corpora with dialogue act annotations is often scarce or nonexistent for languages other than English. Furthermore, preliminary results with bert-base-multilingual-cased have also indicated poorer performance compared to the English model applied on translated data.

To make our data sources more consistent and less noisy, we filter out utterances containing a single character as well as markers representing meta-linguistic information in some spontaneous dialogue corpora (e.g., double brackets in Fischer and smiley faces in STAC). Those markers are also removed from the Switchboard data before both training and testing.

Table 6 presents the results obtained after applying the dialogue act tagger on our spontaneous dialogues and subtitle corpora for all eight languages included in this study. From this we can observe that *Backchannel* instances are considerably more frequent (by a factor three) in spontaneous dialogues than in subtitles for half of the languages – which is in line with the results of our lexical analysis in Section 4. For Hungarian, Italian and Chinese, the difference between the two types of corpora in instances categorized as *Backchannel* is somewhat smaller than for the other languages, but subtitles still contain fewer backchannels. Moreover, *Backchannel* instances are slightly more frequent in subtitles than in spontaneous conversations for Norwegian based on the tagger output. The number of utterances labeled as *Assessment* differs less, but subtitles still seem to contain fewer utterances of this feedback type for almost all genres and languages, except French (see Appendix B for detailed results).

## 5.4 Error Analysis

To have a better understanding of tagger performance and the frequency estimates of feedback cues relying on it, we conduct a more in-depth examination of the tagger's output.

<footnote n="9">9. https://cloud.google.com/translate</footnote>
<table><caption>Table 6: Proportions (%) of dialogue act groups detected by our neural tagger in the spontaneous conversation (SPCONV) and in the subtitle (SUBS) corpora.</caption><thead><tr><th>Language</th><th>Data</th><th>de</th><th>en</th><th>fr</th><th>hu</th><th>it</th><th>ja</th><th>no</th><th>zh</th></tr></thead><tbody><tr><td rowspan="2">Forward look.</td><td>SPCONV</td><td>16.60</td><td>41.71</td><td>43.18</td><td>58.37</td><td>33.55</td><td>42.34</td><td>52.26</td><td>53.95</td></tr><tr><td>SUBS</td><td>59.11</td><td>63.06</td><td>61.87</td><td>58.81</td><td>59.94</td><td>51.15</td><td>64.83</td><td>62.50</td></tr><tr><td rowspan="2">Other</td><td>SPCONV</td><td>53.12</td><td>37.24</td><td>39.01</td><td>17.86</td><td>41.52</td><td>20.11</td><td>22.93</td><td>32.44</td></tr><tr><td>SUBS</td><td>26.35</td><td>24.78</td><td>25.74</td><td>26.95</td><td>26.89</td><td>37.49</td><td>22.85</td><td>28.74</td></tr><tr><td rowspan="2">Assessment</td><td>SPCONV</td><td>16.50</td><td>9.11</td><td>4.62</td><td>15.49</td><td>12.64</td><td>15.74</td><td>17.05</td><td>6.96</td></tr><tr><td>SUBS</td><td>9.08</td><td>7.07</td><td>7.72</td><td>9.29</td><td>8.34</td><td>6.48</td><td>6.53</td><td>5.00</td></tr><tr><td rowspan="2">Backchannel</td><td>SPCONV</td><td>11.57</td><td>10.79</td><td>11.96</td><td>4.28</td><td>5.73</td><td>18.96</td><td>2.67</td><td>5.65</td></tr><tr><td>SUBS</td><td>3.49</td><td>3.72</td><td>3.44</td><td>3.48</td><td>3.45</td><td>3.74</td><td>3.47</td><td>3.00</td></tr><tr><td rowspan="2">Yes/no answer</td><td>SPCONV</td><td>2.22</td><td>1.15</td><td>1.24</td><td>4.00</td><td>6.55</td><td>2.84</td><td>5.09</td><td>1.00</td></tr><tr><td>SUBS</td><td>1.97</td><td>1.37</td><td>1.68</td><td>1.47</td><td>1.38</td><td>1.15</td><td>2.32</td><td>0.76</td></tr></tbody></table>

In general, the proportion of the *Backchannel* category for SPCONV is considerably lower for Hungarian, Italian, Norwegian and Mandarin than for the other languages. The reason behind this is the taggers inability to generalize to spelling variants of backchannel signals such as *hmm*, *mh*, which are missing from or are underrepresented in its training data and are hence often misclassified.

As we have seen in Table 6, our tagger labels slightly more utterances as *Backchannel* in subtitles than in spontaneous conversations for Norwegian. At a closer inspection, however, Norwegian subtitles seem to contain more false positives in the *Backchannel* category compared to other languages. These include, for example, questions, e.g., (context: *Det er så tragisk*. “It’s so tragic.”) *Hva er tragisk?* “What is tragic?” and (context: *Dette er ikke ditt oppgjør*. “This is not your settlement.”) *Overrasket?* “Surprised?”.

Moreover, fewer assessments are automatically detected in the spontaneous dialogues than in the subtitle corpora for French. We found that the tagger had difficulty detecting feedback clues when they are part of longer utterances, independent of whether they appear in an utterance-initial position or not. Assessments occurring in longer utterances are more common in the CID corpus, where, for example, *on a été voir euh Rois et Reine* “we went to see uh kings and queen” was followed by the utterance *ah oui je voulais le voir ce film* “oh yes I wanted to see this film”, which was labeled as forward looking.

We also inspect the most frequent utterances tagged as *Backchannel* and as *Assessment* to gain additional insight into the types of feedback occurring in these data sources. While results for English are mostly correct, results for other languages contain some misclassifications, partly due to actual or apparent translation issues since, as mentioned before, those labels are obtained via translations into English. A translation challenge for short utterances that we can notice is related to polysemy, e.g., *Cosa*? “Thing?”, also translatable as “What?”, in Italian. Furthermore, the most frequent utterances classified as one of the two feedback categories end considerably less often in a punctuation in spontaneous dialogues than in subtitles (see Tables 13-15 and Tables 16–18 in Appendix B for a list of the top ten predicted backchannel utterances in spontaneous conversations and dialogues respectively). This might be, in part, due to transcription conventions for speech data where such
<table><caption>Table 6: Proportions (%) of dialogue act groups detected by our neural tagger in the spontaneous conversation (SPCONV) and in the subtitle (SUBS) corpora.</caption><thead><tr><th>Language</th><th>Data</th><th>de</th><th>en</th><th>fr</th><th>hu</th><th>it</th><th>ja</th><th>no</th><th>zh</th></tr></thead><tbody><tr><td rowspan="2">Forward look.</td><td>SPCONV</td><td>16.60</td><td>41.71</td><td>43.18</td><td>58.37</td><td>33.55</td><td>42.34</td><td>52.26</td><td>53.95</td></tr><tr><td>SUBS</td><td>59.11</td><td>63.06</td><td>61.87</td><td>58.81</td><td>59.94</td><td>51.15</td><td>64.83</td><td>62.50</td></tr><tr><td rowspan="2">Other</td><td>SPCONV</td><td>53.12</td><td>37.24</td><td>39.01</td><td>17.86</td><td>41.52</td><td>20.11</td><td>22.93</td><td>32.44</td></tr><tr><td>SUBS</td><td>26.35</td><td>24.78</td><td>25.74</td><td>26.95</td><td>26.89</td><td>37.49</td><td>22.85</td><td>28.74</td></tr><tr><td rowspan="2">Assessment</td><td>SPCONV</td><td>16.50</td><td>9.11</td><td>4.62</td><td>15.49</td><td>12.64</td><td>15.74</td><td>17.05</td><td>6.96</td></tr><tr><td>SUBS</td><td>9.08</td><td>7.07</td><td>7.72</td><td>9.29</td><td>8.34</td><td>6.48</td><td>6.53</td><td>5.00</td></tr><tr><td rowspan="2">Backchannel</td><td>SPCONV</td><td>11.57</td><td>10.79</td><td>11.96</td><td>4.28</td><td>5.73</td><td>18.96</td><td>2.67</td><td>5.65</td></tr><tr><td>SUBS</td><td>3.49</td><td>3.72</td><td>3.44</td><td>3.48</td><td>3.45</td><td>3.74</td><td>3.47</td><td>3.00</td></tr><tr><td rowspan="2">Yes/no answer</td><td>SPCONV</td><td>2.22</td><td>1.15</td><td>1.24</td><td>4.00</td><td>6.55</td><td>2.84</td><td>5.09</td><td>1.00</td></tr><tr><td>SUBS</td><td>1.97</td><td>1.37</td><td>1.68</td><td>1.47</td><td>1.38</td><td>1.15</td><td>2.32</td><td>0.76</td></tr></tbody></table>

In general, the proportion of the *Backchannel* category for SPCONV is considerably lower for Hungarian, Italian, Norwegian and Mandarin than for the other languages. The reason behind this is the taggers inability to generalize to spelling variants of backchannel signals such as *hmm*, *mh*, which are missing from or are underrepresented in its training data and are hence often misclassified.

As we have seen in Table 6, our tagger labels slightly more utterances as *Backchannel* in subtitles than in spontaneous conversations for Norwegian. At a closer inspection, however, Norwegian subtitles seem to contain more false positives in the *Backchannel* category compared to other languages. These include, for example, questions, e.g., (context: *Det er så tragisk*. “It’s so tragic.”) *Hva er tragisk?* “What is tragic?” and (context: *Dette er ikke ditt oppgjør*. “This is not your settlement.”) *Overrasket?* “Surprised?”.

Moreover, fewer assessments are automatically detected in the spontaneous dialogues than in the subtitle corpora for French. We found that the tagger had difficulty detecting feedback clues when they are part of longer utterances, independent of whether they appear in an utterance-initial position or not. Assessments occurring in longer utterances are more common in the CID corpus, where, for example, *on a été voir euh Rois et Reine* “we went to see uh kings and queen” was followed by the utterance *ah oui je voulais le voir ce film* “oh yes I wanted to see this film”, which was labeled as forward looking.

We also inspect the most frequent utterances tagged as *Backchannel* and as *Assessment* to gain additional insight into the types of feedback occurring in these data sources. While results for English are mostly correct, results for other languages contain some misclassifications, partly due to actual or apparent translation issues since, as mentioned before, those labels are obtained via translations into English. A translation challenge for short utterances that we can notice is related to polysemy, e.g., *Cosa*? “Thing?”, also translatable as “What?”, in Italian. Furthermore, the most frequent utterances classified as one of the two feedback categories end considerably less often in a punctuation in spontaneous dialogues than in subtitles (see Tables 13-15 and Tables 16–18 in Appendix B for a list of the top ten predicted backchannel utterances in spontaneous conversations and dialogues respectively). This might be, in part, due to transcription conventions for speech data where such
marks are sometimes omitted. In general, the tagger seemed to display some tendency to associate sentence final question marks to feedback cues when applied to subtitles.

## 6. Conversational Feedback in Synthetic Dialogues

The previous sections have investigated the differences between scripted and spontaneous dialogues when it comes to the presence of feedback phenomena, on the basis of available corpora. This analysis was to a large extent motivated by the fact that scripted dialogues, given their much broader availability as a source of training material, may have a substantial influence on the ability of large language models to process and generate communicative feedback.

In this section, we investigate this question more thoroughly. More precisely, we further train existing language models on the various conversational corpora presented in Section 3 and use those to generate synthetic dialogues. The goal of such experiments is to assess whether language models trained on scripted vs. spontaneous dialogues do indeed exhibit the same differences in communicative feedback as the ones observed in the previous section.

We start the training process with a pretrained GPT-2 model (Radford et al., 2019) for the language in question<fnref n="10" />, and further train this model using a language modeling objective separately on each corpora. To account for the corpus size differences, the number of epochs is adjusted to ensure that the total number of gradient updates is similar across all corpora.

The GPT-2 models trained on each corpora are then employed to produce a set of synthetic dialogues (100 dialogues of about 50 turns per model<fnref n="11" />). For the fine-tuned models, all turns are automatically generated, while for the base models, the following dialogue beginning is used as context: *Hi! – Hi, how are you? – Fine, and you?* to improve the quality of the output. Finally, the neural dialogue tagger from Section 5 is applied on those dialogues (translated into English) to assess the relative frequency of various dialogue act groups, as shown in Table 7. Here, we include results for the base models without further fine-tuning (*BASE*<sub>syn</sub>) as well as scores for these base models separately fine-tuned on subtitles (*SUBS*<sub>syn</sub>) and spontaneous conversations (*SPCONV*<sub>syn</sub>). We also repeat the results from Table 6 for our original, non-synthetic subtitle (*SUBS*) and spontaneous dialogue (*SPCONV*) data for an easier comparison.

We can observe from Table 7 that the difference in proportion of communicative feedback (*Assessment* and *Backchannel* classes) in the synthetic data is similar to that in the original datasets used for training. The synthetic data produced by the models fine-tuned on subtitles contain a slightly higher proportion of feedback than what can be found in the data generated with the *BASE* models for most languages. The difference is much more striking, however, between the dialogues generated when training on spontaneous vs. the scripted conversations. Exceptions to these trends are: (1) the *Assessment* class in French, due to the particular nature of the CID and STAC corpora discussed in the previous sections; and (2) the *Backchannel* class for Hungarian. Although 11% of all Hungarian *SPCONV*<sub>syn</sub> utterances contained (or consisted of) *hmm*, most of these have been classified by the tagger as *Other* or *Forward looking*. A qualitative analysis of the dialogues also shows that the initial turns of the synthetic dialogues has a large influence on the rest of the dialogue. In particular,

<footnote n="10">10. The following pretrained models available from the HuggingFace repository (https://huggingface.co/models) are employed: gpt2-base (English), gpt-fr-cased-small (French), german-gpt2 (German), gpt2-small-italian (Italian), PULI-GPT-2 (Hungarian), norwegian-gpt2 (Norwegian), gpt2-chinese-cluecorpussmall (Mandarin Chinese), and japanese-gpt2-medium (Japanese).</footnote>

<footnote n="11">11. We exclude 8 synthetic Japanese dialogues from the results, as they contain a large amount of noise (utterances consisting of almost only non-alphabetical characters).</footnote>
marks are sometimes omitted. In general, the tagger seemed to display some tendency to associate sentence final question marks to feedback cues when applied to subtitles.

## 6. Conversational Feedback in Synthetic Dialogues

The previous sections have investigated the differences between scripted and spontaneous dialogues when it comes to the presence of feedback phenomena, on the basis of available corpora. This analysis was to a large extent motivated by the fact that scripted dialogues, given their much broader availability as a source of training material, may have a substantial influence on the ability of large language models to process and generate communicative feedback.

In this section, we investigate this question more thoroughly. More precisely, we further train existing language models on the various conversational corpora presented in Section 3 and use those to generate synthetic dialogues. The goal of such experiments is to assess whether language models trained on scripted vs. spontaneous dialogues do indeed exhibit the same differences in communicative feedback as the ones observed in the previous section.

We start the training process with a pretrained GPT-2 model (Radford et al., 2019) for the language in question<fnref n="10" />, and further train this model using a language modeling objective separately on each corpora. To account for the corpus size differences, the number of epochs is adjusted to ensure that the total number of gradient updates is similar across all corpora.

The GPT-2 models trained on each corpora are then employed to produce a set of synthetic dialogues (100 dialogues of about 50 turns per model<fnref n="11" />). For the fine-tuned models, all turns are automatically generated, while for the base models, the following dialogue beginning is used as context: *Hi! – Hi, how are you? – Fine, and you?* to improve the quality of the output. Finally, the neural dialogue tagger from Section 5 is applied on those dialogues (translated into English) to assess the relative frequency of various dialogue act groups, as shown in Table 7. Here, we include results for the base models without further fine-tuning (*BASE*<sub>syn</sub>) as well as scores for these base models separately fine-tuned on subtitles (*SUBS*<sub>syn</sub>) and spontaneous conversations (*SPCONV*<sub>syn</sub>). We also repeat the results from Table 6 for our original, non-synthetic subtitle (*SUBS*) and spontaneous dialogue (*SPCONV*) data for an easier comparison.

We can observe from Table 7 that the difference in proportion of communicative feedback (*Assessment* and *Backchannel* classes) in the synthetic data is similar to that in the original datasets used for training. The synthetic data produced by the models fine-tuned on subtitles contain a slightly higher proportion of feedback than what can be found in the data generated with the *BASE* models for most languages. The difference is much more striking, however, between the dialogues generated when training on spontaneous vs. the scripted conversations. Exceptions to these trends are: (1) the *Assessment* class in French, due to the particular nature of the CID and STAC corpora discussed in the previous sections; and (2) the *Backchannel* class for Hungarian. Although 11% of all Hungarian *SPCONV*<sub>syn</sub> utterances contained (or consisted of) *hmm*, most of these have been classified by the tagger as *Other* or *Forward looking*. A qualitative analysis of the dialogues also shows that the initial turns of the synthetic dialogues has a large influence on the rest of the dialogue. In particular,

<footnote n="10">10. The following pretrained models available from the HuggingFace repository (https://huggingface.co/models) are employed: gpt2-base (English), gpt-fr-cased-small (French), german-gpt2 (German), gpt2-small-italian (Italian), PULI-GPT-2 (Hungarian), norwegian-gpt2 (Norwegian), gpt2-chinese-cluecorpussmall (Mandarin Chinese), and japanese-gpt2-medium (Japanese).</footnote>

<footnote n="11">11. We exclude 8 synthetic Japanese dialogues from the results, as they contain a large amount of noise (utterances consisting of almost only non-alphabetical characters).</footnote>
<table><caption>Table 7: Proportion (%) of dialogue act groups detected by our neural tagger in the synthetic dialogues with no fine-tuning (BASE<sub>syn</sub>), as well as fine-tuned on spontaneous conversations (SPCONV<sub>syn</sub>) and subtitles (SUBS<sub>syn</sub>). For easier reference, we also include in this table the results on the original subtitle and spontaneous dialogue data (SUBS and SPCONV), initially shown in Table 6.</caption><thead><tr><th>Language</th><th>Data</th><th>de</th><th>en</th><th>fr</th><th>hu</th><th>it</th><th>ja</th><th>no</th><th>zh</th></tr></thead><tbody><tr><td rowspan="5">Forward l.</td><td><b>BASE</b><sub>syn</sub></td><td>67.88</td><td>48.52</td><td>74.55</td><td>63.57</td><td>71.81</td><td>60.62</td><td>87.02</td><td>90.92</td></tr><tr><td><b>SUBS</b><sub>syn</sub></td><td>54.0</td><td>42.0</td><td>55.6</td><td>33.0</td><td>52.1</td><td>50.76</td><td>51.7</td><td>43.33</td></tr><tr><td><b>SPCONV</b><sub>syn</sub></td><td>18.5</td><td>36.5</td><td>44.6</td><td>48.3</td><td>36.2</td><td>53.11</td><td>53.0</td><td>46.14</td></tr><tr><td><b>SUBS</b></td><td>59.11</td><td>63.06</td><td>61.87</td><td>58.81</td><td>59.94</td><td>51.15</td><td>64.83</td><td>62.50</td></tr><tr><td><b>SPCONV</b></td><td>16.60</td><td>41.71</td><td>43.18</td><td>58.37</td><td>33.55</td><td>42.34</td><td>52.26</td><td>53.95</td></tr><tr><td rowspan="5">Other</td><td><b>BASE</b><sub>syn</sub></td><td>23.84</td><td>40.96</td><td>16.96</td><td>28.16</td><td>22.59</td><td>33.59</td><td>10.48</td><td>7.38</td></tr><tr><td><b>SUBS</b><sub>syn</sub></td><td>36.1</td><td>47.9</td><td>36.2</td><td>52.3</td><td>38.0</td><td>36.09</td><td>40.6</td><td>44.29</td></tr><tr><td><b>SPCONV</b><sub>syn</sub></td><td>51.2</td><td>39.7</td><td>36.1</td><td>31.4</td><td>34.0</td><td>14.27</td><td>20.9</td><td>32.62</td></tr><tr><td><b>SUBS</b></td><td>26.35</td><td>24.78</td><td>25.74</td><td>26.95</td><td>26.89</td><td>37.49</td><td>22.85</td><td>28.74</td></tr><tr><td><b>SPCONV</b></td><td>53.12</td><td>37.24</td><td>39.01</td><td>17.86</td><td>41.52</td><td>20.11</td><td>22.93</td><td>32.44</td></tr><tr><td rowspan="5">Assessment</td><td><b>BASE</b><sub>syn</sub></td><td>5.06</td><td>5.55</td><td>5.36</td><td>5.31</td><td>2.47</td><td>3.39</td><td>1.86</td><td>1.08</td></tr><tr><td><b>SUBS</b><sub>syn</sub></td><td>5.5</td><td>4.4</td><td>3.9</td><td>9.3</td><td>3.6</td><td>7.52</td><td>3.3</td><td>7.42</td></tr><tr><td><b>SPCONV</b><sub>syn</sub></td><td>16.8</td><td>7.5</td><td>4.3</td><td>13.8</td><td>15.5</td><td>19.78</td><td>18.7</td><td>9.62</td></tr><tr><td><b>SUBS</b></td><td>9.08</td><td>7.07</td><td>7.72</td><td>9.29</td><td>8.34</td><td>6.48</td><td>6.53</td><td>5.00</td></tr><tr><td><b>SPCONV</b></td><td>16.50</td><td>9.11</td><td>4.62</td><td>15.49</td><td>12.64</td><td>15.74</td><td>17.05</td><td>6.96</td></tr><tr><td rowspan="5">Backchannel</td><td><b>BASE</b><sub>syn</sub></td><td>2.05</td><td>3.05</td><td>1.69</td><td>2.17</td><td>1.11</td><td>2.04</td><td>0.52</td><td>0.44</td></tr><tr><td><b>SUBS</b><sub>syn</sub></td><td>2.5</td><td>2.8</td><td>1.9</td><td>2.8</td><td>2.8</td><td>4.18</td><td>1.6</td><td>3.80</td></tr><tr><td><b>SPCONV</b><sub>syn</sub></td><td>16.8</td><td>13.0</td><td>13.5</td><td>2.4</td><td>6.8</td><td>9.89</td><td>2.2</td><td>9.69</td></tr><tr><td><b>SUBS</b></td><td>3.49</td><td>3.72</td><td>3.44</td><td>3.48</td><td>3.45</td><td>3.74</td><td>3.47</td><td>3.00</td></tr><tr><td><b>SPCONV</b></td><td>11.57</td><td>10.79</td><td>11.96</td><td>4.28</td><td>5.73</td><td>18.96</td><td>2.67</td><td>5.65</td></tr><tr><td rowspan="5">Yes/no ans.</td><td><b>BASE</b><sub>syn</sub></td><td>1.16</td><td>1.93</td><td>1.44</td><td>0.80</td><td>2.01</td><td>0.36</td><td>0.11</td><td>0.18</td></tr><tr><td><b>SUBS</b><sub>syn</sub></td><td>2.0</td><td>2.9</td><td>2.4</td><td>2.6</td><td>3.5</td><td>1.45</td><td>2.9</td><td>1.16</td></tr><tr><td><b>SPCONV</b><sub>syn</sub></td><td>1.4</td><td>3.2</td><td>1.5</td><td>4.1</td><td>7.5</td><td>2.94</td><td>5.2</td><td>1.92</td></tr><tr><td><b>SUBS</b></td><td>1.97</td><td>1.37</td><td>1.68</td><td>1.47</td><td>1.38</td><td>1.15</td><td>2.32</td><td>0.76</td></tr><tr><td><b>SPCONV</b></td><td>2.22</td><td>1.15</td><td>1.24</td><td>4.00</td><td>6.55</td><td>2.84</td><td>5.09</td><td>1.00</td></tr></tbody></table>

dialogues in which the first turns included fewer feedback phenomena tended to reproduce this pattern in the subsequent turns.

## 7. Multimodal Feedback Cues in Movies

Conversational feedback in human face-to-face interaction makes extensive use of nonverbal multi-modal behaviors such as head gestures (nods, tilts, shakes; e.g., Allwood et al. 2007; Malisz et al. 2016), facial expressions (smiles, frowns; e.g., Bavelas and Chovil 2018), and gaze (Bavelas et al., 2002). Because film is an inherently visual medium, it could be argued that conversational feedback
<table><caption>Table 7: Proportion (%) of dialogue act groups detected by our neural tagger in the synthetic dialogues with no fine-tuning (BASE<sub>syn</sub>), as well as fine-tuned on spontaneous conversations (SPCONV<sub>syn</sub>) and subtitles (SUBS<sub>syn</sub>). For easier reference, we also include in this table the results on the original subtitle and spontaneous dialogue data (SUBS and SPCONV), initially shown in Table 6.</caption><thead><tr><th>Language</th><th>Data</th><th>de</th><th>en</th><th>fr</th><th>hu</th><th>it</th><th>ja</th><th>no</th><th>zh</th></tr></thead><tbody><tr><td rowspan="5">Forward l.</td><td><b>BASE</b><sub>syn</sub></td><td>67.88</td><td>48.52</td><td>74.55</td><td>63.57</td><td>71.81</td><td>60.62</td><td>87.02</td><td>90.92</td></tr><tr><td><b>SUBS</b><sub>syn</sub></td><td>54.0</td><td>42.0</td><td>55.6</td><td>33.0</td><td>52.1</td><td>50.76</td><td>51.7</td><td>43.33</td></tr><tr><td><b>SPCONV</b><sub>syn</sub></td><td>18.5</td><td>36.5</td><td>44.6</td><td>48.3</td><td>36.2</td><td>53.11</td><td>53.0</td><td>46.14</td></tr><tr><td><b>SUBS</b></td><td>59.11</td><td>63.06</td><td>61.87</td><td>58.81</td><td>59.94</td><td>51.15</td><td>64.83</td><td>62.50</td></tr><tr><td><b>SPCONV</b></td><td>16.60</td><td>41.71</td><td>43.18</td><td>58.37</td><td>33.55</td><td>42.34</td><td>52.26</td><td>53.95</td></tr><tr><td rowspan="5">Other</td><td><b>BASE</b><sub>syn</sub></td><td>23.84</td><td>40.96</td><td>16.96</td><td>28.16</td><td>22.59</td><td>33.59</td><td>10.48</td><td>7.38</td></tr><tr><td><b>SUBS</b><sub>syn</sub></td><td>36.1</td><td>47.9</td><td>36.2</td><td>52.3</td><td>38.0</td><td>36.09</td><td>40.6</td><td>44.29</td></tr><tr><td><b>SPCONV</b><sub>syn</sub></td><td>51.2</td><td>39.7</td><td>36.1</td><td>31.4</td><td>34.0</td><td>14.27</td><td>20.9</td><td>32.62</td></tr><tr><td><b>SUBS</b></td><td>26.35</td><td>24.78</td><td>25.74</td><td>26.95</td><td>26.89</td><td>37.49</td><td>22.85</td><td>28.74</td></tr><tr><td><b>SPCONV</b></td><td>53.12</td><td>37.24</td><td>39.01</td><td>17.86</td><td>41.52</td><td>20.11</td><td>22.93</td><td>32.44</td></tr><tr><td rowspan="5">Assessment</td><td><b>BASE</b><sub>syn</sub></td><td>5.06</td><td>5.55</td><td>5.36</td><td>5.31</td><td>2.47</td><td>3.39</td><td>1.86</td><td>1.08</td></tr><tr><td><b>SUBS</b><sub>syn</sub></td><td>5.5</td><td>4.4</td><td>3.9</td><td>9.3</td><td>3.6</td><td>7.52</td><td>3.3</td><td>7.42</td></tr><tr><td><b>SPCONV</b><sub>syn</sub></td><td>16.8</td><td>7.5</td><td>4.3</td><td>13.8</td><td>15.5</td><td>19.78</td><td>18.7</td><td>9.62</td></tr><tr><td><b>SUBS</b></td><td>9.08</td><td>7.07</td><td>7.72</td><td>9.29</td><td>8.34</td><td>6.48</td><td>6.53</td><td>5.00</td></tr><tr><td><b>SPCONV</b></td><td>16.50</td><td>9.11</td><td>4.62</td><td>15.49</td><td>12.64</td><td>15.74</td><td>17.05</td><td>6.96</td></tr><tr><td rowspan="5">Backchannel</td><td><b>BASE</b><sub>syn</sub></td><td>2.05</td><td>3.05</td><td>1.69</td><td>2.17</td><td>1.11</td><td>2.04</td><td>0.52</td><td>0.44</td></tr><tr><td><b>SUBS</b><sub>syn</sub></td><td>2.5</td><td>2.8</td><td>1.9</td><td>2.8</td><td>2.8</td><td>4.18</td><td>1.6</td><td>3.80</td></tr><tr><td><b>SPCONV</b><sub>syn</sub></td><td>16.8</td><td>13.0</td><td>13.5</td><td>2.4</td><td>6.8</td><td>9.89</td><td>2.2</td><td>9.69</td></tr><tr><td><b>SUBS</b></td><td>3.49</td><td>3.72</td><td>3.44</td><td>3.48</td><td>3.45</td><td>3.74</td><td>3.47</td><td>3.00</td></tr><tr><td><b>SPCONV</b></td><td>11.57</td><td>10.79</td><td>11.96</td><td>4.28</td><td>5.73</td><td>18.96</td><td>2.67</td><td>5.65</td></tr><tr><td rowspan="5">Yes/no ans.</td><td><b>BASE</b><sub>syn</sub></td><td>1.16</td><td>1.93</td><td>1.44</td><td>0.80</td><td>2.01</td><td>0.36</td><td>0.11</td><td>0.18</td></tr><tr><td><b>SUBS</b><sub>syn</sub></td><td>2.0</td><td>2.9</td><td>2.4</td><td>2.6</td><td>3.5</td><td>1.45</td><td>2.9</td><td>1.16</td></tr><tr><td><b>SPCONV</b><sub>syn</sub></td><td>1.4</td><td>3.2</td><td>1.5</td><td>4.1</td><td>7.5</td><td>2.94</td><td>5.2</td><td>1.92</td></tr><tr><td><b>SUBS</b></td><td>1.97</td><td>1.37</td><td>1.68</td><td>1.47</td><td>1.38</td><td>1.15</td><td>2.32</td><td>0.76</td></tr><tr><td><b>SPCONV</b></td><td>2.22</td><td>1.15</td><td>1.24</td><td>4.00</td><td>6.55</td><td>2.84</td><td>5.09</td><td>1.00</td></tr></tbody></table>

dialogues in which the first turns included fewer feedback phenomena tended to reproduce this pattern in the subsequent turns.

## 7. Multimodal Feedback Cues in Movies

Conversational feedback in human face-to-face interaction makes extensive use of nonverbal multi-modal behaviors such as head gestures (nods, tilts, shakes; e.g., Allwood et al. 2007; Malisz et al. 2016), facial expressions (smiles, frowns; e.g., Bavelas and Chovil 2018), and gaze (Bavelas et al., 2002). Because film is an inherently visual medium, it could be argued that conversational feedback
<table><caption>Table 8: Amount and type of multimodal feedback in manually annotated video content.</caption><thead><tr><th rowspan="2">Title</th><th rowspan="2">Length (min)</th><th colspan="3">Nonverbal</th><th rowspan="2">Vocal</th><th rowspan="2">Subtitles</th></tr><tr><th>Smile</th><th>Nod</th><th>Shake</th></tr></thead><tbody><tr><td>Enola Holmes</td><td>122</td><td>20</td><td>8</td><td>3</td><td>8</td><td>8</td></tr><tr><td>White Collar</td><td>55</td><td>23</td><td>37</td><td>15</td><td>8</td><td>7</td></tr></tbody></table>

in film dialogues takes place in the visual realm as well and therefore may not be fully captured in movie subtitles.

To investigate whether this is the case, we analyze the subtitles of a randomly selected movie (Enola Holmes, 122 min) and one episode of a series (White Collar, S1E1, 55 min). We annotate the conversational nonverbal feedback behavior of the characters, transcribed their verbal/vocal feedback (i.e., backchannels and the like) and compare these to the subtitles from the MovieSubtitles corpus.

As Table 8 shows, both movies differ with respect to the frequency of use of nonverbal feedback behaviors. While we find 20 instances of smile, 8 instances of head nod and 3 shakes in Enola Holmes, the White Collar episode we analyze (which is less than half in length) contains 23 instances of smiles, 37 instances of head nods and 15 shakes. Only few instances (8) of verbal/vocal feedback (mhm, hm, mm, huh, etc.) are present in each movie and all but one of them are part of the subtitles. As also these numbers indicate, only a small fraction of all feedback phenomena are verbalized and mentioned in subtitles. Therefore, for a more complete modeling of conversational feedback, incorporating information from multimodal sources is essential.

# 8. Looking Ahead: The Latest Chatbots and Conversational Feedback

There has been a recent surge in the availability of large language models fine-tuned to interact with human users through conversation. Details about the training data behind many of these chat models are often not (or only partly) disclosed, but they most likely include some dialogue-related sources such as subtitles, discussion forums and possibly also transcriptions of spontaneous speech, although in much smaller quantities given their limited availability. Despite the emphasis on the conversational aspect, it is unclear to date how well these systems can actually handle communicative feedback signals.

## 8.1 Understanding Feedback Signals

To get a first impression of how well latest models understand feedback cues, we test the reaction of CHATGPT (OpenAI, 2023) to some short feedback cue words within two short conversations, one focusing on positive, the other on negative feedback. Example (3) shows part of the interaction with positive feedback signals.<fnref n="12" />

(3) **Prompt:** Uhm

**Response:** It seems like you might have something on your mind. If you have a question or if there's a topic you'd like to discuss, please feel free to share, and I'll be happy to assist you!

**Prompt:** ok

<footnote n="12">12. For all examples involving CHATGPT, we used the CHATGPT August 3, 2023 Version (relying on GPT-3.5).</footnote>
<table><caption>Table 8: Amount and type of multimodal feedback in manually annotated video content.</caption><thead><tr><th rowspan="2">Title</th><th rowspan="2">Length (min)</th><th colspan="3">Nonverbal</th><th rowspan="2">Vocal</th><th rowspan="2">Subtitles</th></tr><tr><th>Smile</th><th>Nod</th><th>Shake</th></tr></thead><tbody><tr><td>Enola Holmes</td><td>122</td><td>20</td><td>8</td><td>3</td><td>8</td><td>8</td></tr><tr><td>White Collar</td><td>55</td><td>23</td><td>37</td><td>15</td><td>8</td><td>7</td></tr></tbody></table>

in film dialogues takes place in the visual realm as well and therefore may not be fully captured in movie subtitles.

To investigate whether this is the case, we analyze the subtitles of a randomly selected movie (Enola Holmes, 122 min) and one episode of a series (White Collar, S1E1, 55 min). We annotate the conversational nonverbal feedback behavior of the characters, transcribed their verbal/vocal feedback (i.e., backchannels and the like) and compare these to the subtitles from the MovieSubtitles corpus.

As Table 8 shows, both movies differ with respect to the frequency of use of nonverbal feedback behaviors. While we find 20 instances of smile, 8 instances of head nod and 3 shakes in Enola Holmes, the White Collar episode we analyze (which is less than half in length) contains 23 instances of smiles, 37 instances of head nods and 15 shakes. Only few instances (8) of verbal/vocal feedback (*mhm*, *hm*, *mm*, *huh*, etc.) are present in each movie and all but one of them are part of the subtitles. As also these numbers indicate, only a small fraction of all feedback phenomena are verbalized and mentioned in subtitles. Therefore, for a more complete modeling of conversational feedback, incorporating information from multimodal sources is essential.

# 8. Looking Ahead: The Latest Chatbots and Conversational Feedback

There has been a recent surge in the availability of large language models fine-tuned to interact with human users through conversation. Details about the training data behind many of these chat models are often not (or only partly) disclosed, but they most likely include some dialogue-related sources such as subtitles, discussion forums and possibly also transcriptions of spontaneous speech, although in much smaller quantities given their limited availability. Despite the emphasis on the conversational aspect, it is unclear to date how well these systems can actually handle communicative feedback signals.

## 8.1 Understanding Feedback Signals

To get a first impression of how well latest models understand feedback cues, we test the reaction of CHATGPT (OpenAI, 2023) to some short feedback cue words within two short conversations, one focusing on positive, the other on negative feedback. Example (3) shows part of the interaction with positive feedback signals.<fnref n="12" />

(3) **Prompt:** Uhm

**Response:** It seems like you might have something on your mind. If you have a question or if there's a topic you'd like to discuss, please feel free to share, and I'll be happy to assist you!

**Prompt:** ok

<footnote n="12">12. For all examples involving CHATGPT, we used the CHATGPT August 3, 2023 Version (relying on GPT-3.5).</footnote>
**Response:** If you're ready to continue the conversation or if you have a specific question or topic in mind, just let me know. I'm here to help with any information or guidance you need.

**Prompt:** well

**Response:** If you're feeling uncertain or if there's something you'd like to discuss, don't hesitate to let me know. Whether you have a question, need advice, or just want to chat, I'm here to assist you. Just provide me with some context or a topic you're interested in, and we can continue the conversation from there.

As we can see in Example (3), the system responses to the cue words often contain a spelled-out interpretation of what it supposes its interlocutor's communicative intention is. It understands most of them correctly, however, in a few cases, the evaluative sentence or an appropriate reaction is missing and the system resorts to repeating whether it could help answering any questions as illustrated in Example (4).

(4) **Prompt:** oh my

**Response:** Is there anything specific you'd like to discuss or ask about? I'm here to engage in conversation and help with any questions or topics you have in mind. Whether it's something light-hearted, informative, or just a friendly chat, feel free to let me know!<fnref n="13" />

Inspired by the regular occurrence of a communicative intent “summary” in the system outputs, we conduct a small qualitative evaluation in a zero-shot setting, where we prompt CHATGPT with the following instruction: *In the conversation below, can you explain in square brackets after each line what the speaker is trying to communicate?* As input text, we provide a conversation from the Switchboard corpus (file sw_0001_4325), as one utterance per line, delimited by double quotation marks and separated with a new line from the instruction. Punctuation, transcription symbols (e.g., indicating pause or simultaneity in speech) and speaker information are not included in the utterances. For the utterances annotated as feedback, the system quite consistently adds plausible explanations such as *Acknowledging, Agreeing, Affirming*. Interestingly, however, for ca. 14% (21 out of 152) of utterances, CHATGPT does not provide a tag even after a follow-up prompt asking it to add one for the lines that it has skipped at the first attempt (with *some lines are missing an explanation in square brackets, could you add those?*). Half of these “ignored” utterances were annotated in Switchboard as abandoned turns (e.g., *So, but we don't, Um*), where the utterance type could not be clearly determined by annotators, and about a third of them were labeled as *Backchannel*. (One utterance of this type is even removed from the conversation by CHATGPT, possibly already during the preprocessing stage of the system.)

For comparison, we also test the same task with another language model, namely the smaller, but open source Llama 2-CHAT 7B (Touvron et al., 2023). Since Llama 2-CHAT outputs are rather verbose, we use an additional sentence (*Use maximum 3 words as explanation.* ) in the prompt to obtain explanations that would be easier to analyze and more comparable to the CHATGPT output. We observe a similar trend, namely that a number of utterances (13%) are disregarded (completely removed, not only not tagged) in the output.<fnref n="14" /> About half of skipped utterances (6 out of 13) are annotated in Switchboard either as *Acknowledgment* or as *Agree/accept*. Moreover, multi-line utterance

<footnote n="13">13. For additional context, see the complete dialogue in Example (6) in the Appendix C.</footnote>

<footnote n="14">14. The system only tagged 101 lines from the original dialogue before a time out error, possible connected to its token limit.</footnote>
**Response:** If you're ready to continue the conversation or if you have a specific question or topic in mind, just let me know. I'm here to help with any information or guidance you need.

**Prompt:** well

**Response:** If you're feeling uncertain or if there's something you'd like to discuss, don't hesitate to let me know. Whether you have a question, need advice, or just want to chat, I'm here to assist you. Just provide me with some context or a topic you're interested in, and we can continue the conversation from there.

As we can see in Example (3), the system responses to the cue words often contain a spelled-out interpretation of what it supposes its interlocutor's communicative intention is. It understands most of them correctly, however, in a few cases, the evaluative sentence or an appropriate reaction is missing and the system resorts to repeating whether it could help answering any questions as illustrated in Example (4).

(4) **Prompt:** oh my

**Response:** Is there anything specific you'd like to discuss or ask about? I'm here to engage in conversation and help with any questions or topics you have in mind. Whether it's something light-hearted, informative, or just a friendly chat, feel free to let me know!<fnref n="13" />

Inspired by the regular occurrence of a communicative intent “summary” in the system outputs, we conduct a small qualitative evaluation in a zero-shot setting, where we prompt CHATGPT with the following instruction: *In the conversation below, can you explain in square brackets after each line what the speaker is trying to communicate?* As input text, we provide a conversation from the Switchboard corpus (file sw_0001_4325), as one utterance per line, delimited by double quotation marks and separated with a new line from the instruction. Punctuation, transcription symbols (e.g., indicating pause or simultaneity in speech) and speaker information are not included in the utterances. For the utterances annotated as feedback, the system quite consistently adds plausible explanations such as *Acknowledging, Agreeing, Affirming*. Interestingly, however, for ca. 14% (21 out of 152) of utterances, CHATGPT does not provide a tag even after a follow-up prompt asking it to add one for the lines that it has skipped at the first attempt (with *some lines are missing an explanation in square brackets, could you add those?*). Half of these “ignored” utterances were annotated in Switchboard as abandoned turns (e.g., *So, but we don't, Um*), where the utterance type could not be clearly determined by annotators, and about a third of them were labeled as *Backchannel*. (One utterance of this type is even removed from the conversation by CHATGPT, possibly already during the preprocessing stage of the system.)

For comparison, we also test the same task with another language model, namely the smaller, but open source Llama 2-CHAT 7B (Touvron et al., 2023). Since Llama 2-CHAT outputs are rather verbose, we use an additional sentence (*Use maximum 3 words as explanation.* ) in the prompt to obtain explanations that would be easier to analyze and more comparable to the CHATGPT output. We observe a similar trend, namely that a number of utterances (13%) are disregarded (completely removed, not only not tagged) in the output.<fnref n="14" /> About half of skipped utterances (6 out of 13) are annotated in Switchboard either as *Acknowledgment* or as *Agree/accept*. Moreover, multi-line utterance

<footnote n="13">13. For additional context, see the complete dialogue in Example (6) in the Appendix C.</footnote>

<footnote n="14">14. The system only tagged 101 lines from the original dialogue before a time out error, possible connected to its token limit.</footnote>
removals occur both times after backchannels. For example explanations provided by the two tested
systems see Table 19 in Appendix C.

When repeating the same prompt, or a slightly modified version of it, we find that the output
of both systems varies to some degree both in terms of the output format (e.g., more verbose tags,
guessing also the speaker for each utterance, line numbering) and in terms of the type of skipped
utterances, if any. Typically, prompting for shorter labels seems to reduce the utterance-skipping
behavior.

## 8.2 Production of Feedback Cues

We also briefly test CHATGPT's ability to generate short feedback cues and find that despite follow-up
prompts specifically requesting basic conversational feedback, the system seems unable to produce
these, regardless of the use of context conversations or example cues provided (e.g., *mh* vs. *yeah*).
Example prompts exhibiting this behavior are shown in Example (5). Short feedback cues are not
only not produced, but they are even discarded as the last *mh* in the example (and the system response
following it) demonstrate.

(5) **Prompt:** Hi, I would like you to help me with simple dialogues in English since I am not a native speaker. Can you propose me follow-up utterances to the short conversations I give you below?

A: It was when I was in college

B: mh

A: We travelled to UK with friends..

B: mh

A: during summer holidays

**Response:** B: Oh, that sounds like a fun trip! What did you do there?

**Prompt:** Thanks, can you now suggest follow-up that are minimal and simply display that you are following the conversation?

A: I remember one time we were invited for dinner

B: mh

A: but not at our friends place

B: mh

A: at friends of our parents

B: mh

A: and the food was horrible

B: mh

**Response:** B: Oh no, that doesn't sound pleasant. How was the dinner experience at your parents' friends' place?
removals occur both times after backchannels. For example explanations provided by the two tested
systems see Table 19 in Appendix C.

When repeating the same prompt, or a slightly modified version of it, we find that the output
of both systems varies to some degree both in terms of the output format (e.g., more verbose tags,
guessing also the speaker for each utterance, line numbering) and in terms of the type of skipped
utterances, if any. Typically, prompting for shorter labels seems to reduce the utterance-skipping
behavior.

## 8.2 Production of Feedback Cues

We also briefly test CHATGPT's ability to generate short feedback cues and find that despite follow-up
prompts specifically requesting basic conversational feedback, the system seems unable to produce
these, regardless of the use of context conversations or example cues provided (e.g., *mh* vs. *yeah*).
Example prompts exhibiting this behavior are shown in Example (5). Short feedback cues are not
only not produced, but they are even discarded as the last *mh* in the example (and the system response
following it) demonstrate.

(5) **Prompt:** Hi, I would like you to help me with simple dialogues in English since I am not a native speaker. Can you propose me follow-up utterances to the short conversations I give you below?

A: It was when I was in college

B: mh

A: We travelled to UK with friends..

B: mh

A: during summer holidays

**Response:** B: Oh, that sounds like a fun trip! What did you do there?

**Prompt:** Thanks, can you now suggest follow-up that are minimal and simply display that you are following the conversation?

A: I remember one time we were invited for dinner

B: mh

A: but not at our friends place

B: mh

A: at friends of our parents

B: mh

A: and the food was horrible

B: mh

**Response:** B: Oh no, that doesn't sound pleasant. How was the dinner experience at your parents' friends' place?
The limited evidence summarized here thus indicates that recent conversational models might have
some difficulties producing and handling certain short communicative feedback signs, but the reason
behind this and its systematicity remains unclear. Nevertheless, recent studies (Zhan et al., 2023;
Labruna et al., 2023) have already started exploring these language models also for generating datasets
of synthetic dialogues to alleviate the cost and time required for collecting data from human-human
interactions. A more in-depth evaluation of the conversational ability of these models would therefore
be timely.

9. Conclusion and Future Work

As described in this paper, movie and TV subtitles exhibit notable linguistic differences to actual spontaneous dialogues in the amount and type of conversational feedback they include. Based on a collection of corpora of both spontaneous dialogues and subtitles across eight languages, we provide both lexical statistics and dialogue act estimates derived with a fine-tuned neural language model. The results of those quantitative analyses show that the proportion of conversational feedback is considerably lower in subtitles than in spontaneous dialogues across the corpora and languages included. Furthermore, the type of conversational feedback also differs, as negative feedback is proportionally more frequent in subtitles. The difference in the proportion of feedback phenomena is also evidently propagated to synthetic dialogues generated by models trained on subtitles and unscripted dialogues.

Beyond their linguistic interest, these results can also provide useful insights into the development of data-driven neural conversational models, as those are often trained on scripted dialogues such as movie scripts and subtitles, and might therefore struggle to (1) understand communicative feedback from the user and (2) produce such feedback themselves. Our small-scale evaluation of conversational language models revealed, in fact, that while the latest systems have a rather good understanding of the communicative intent behind feedback cues, there are indications that they do not interpret and build on some of these while progressing within the dialogue. Integrating a larger amount of spontaneous dialogues in the training process when available could, therefore, prove valuable to mitigate these potential limitations. Currently there is a substantial lack of transparency about the training data used for fine-tuning most chat models. Consequently, investigating their ability to handle and reproduce phenomena typical of spontaneous dialogues more in depth constitutes a promising direction for further research. This could help prevent the propagation of any potential incorrect use or under-representation of conversational cues when fine-tuning language models with quasi-spontaneous dialogues such as subtitles or synthetic dialogues.

Acknowledgments

This work was carried out within the *GraphDial* project (https://graphdial.nr.no/), supported
by the Research Council of Norway. The work was initiated during the bilateral PHC-Aurora project
*French Norwegian Research Effort on Applied Dialogue Modelling*. Laurent Prévot would also like to
acknowledge continuous support from the Institute for Language Communication and the Brain (ILCB
ANR-16-CONV0002) and the Excellence Initiative of Aix-Marseille University (A*MIDEX AAP-ID-
17-46-170301-11.1). Hendrik Buschmeier was supported by the German Research Foundation (DFG)
in the Collaborative Research Center TRR 318/1 2021 ‘Constructing Explainability’ (438445824).
We would like to thank Hiro Yamazaki for helping with the Japanese lexical items.
The limited evidence summarized here thus indicates that recent conversational models might have
some difficulties producing and handling certain short communicative feedback signs, but the reason
behind this and its systematicity remains unclear. Nevertheless, recent studies (Zhan et al., 2023;
Labruna et al., 2023) have already started exploring these language models also for generating datasets
of synthetic dialogues to alleviate the cost and time required for collecting data from human-human
interactions. A more in-depth evaluation of the conversational ability of these models would therefore
be timely.

9. Conclusion and Future Work

As described in this paper, movie and TV subtitles exhibit notable linguistic differences to actual spontaneous dialogues in the amount and type of conversational feedback they include. Based on a collection of corpora of both spontaneous dialogues and subtitles across eight languages, we provide both lexical statistics and dialogue act estimates derived with a fine-tuned neural language model. The results of those quantitative analyses show that the proportion of conversational feedback is considerably lower in subtitles than in spontaneous dialogues across the corpora and languages included. Furthermore, the type of conversational feedback also differs, as negative feedback is proportionally more frequent in subtitles. The difference in the proportion of feedback phenomena is also evidently propagated to synthetic dialogues generated by models trained on subtitles and unscripted dialogues.

Beyond their linguistic interest, these results can also provide useful insights into the development of data-driven neural conversational models, as those are often trained on scripted dialogues such as movie scripts and subtitles, and might therefore struggle to (1) understand communicative feedback from the user and (2) produce such feedback themselves. Our small-scale evaluation of conversational language models revealed, in fact, that while the latest systems have a rather good understanding of the communicative intent behind feedback cues, there are indications that they do not interpret and build on some of these while progressing within the dialogue. Integrating a larger amount of spontaneous dialogues in the training process when available could, therefore, prove valuable to mitigate these potential limitations. Currently there is a substantial lack of transparency about the training data used for fine-tuning most chat models. Consequently, investigating their ability to handle and reproduce phenomena typical of spontaneous dialogues more in depth constitutes a promising direction for further research. This could help prevent the propagation of any potential incorrect use or under-representation of conversational cues when fine-tuning language models with quasi-spontaneous dialogues such as subtitles or synthetic dialogues.

Acknowledgments

This work was carried out within the *GraphDial* project (https://graphdial.nr.no/), supported
by the Research Council of Norway. The work was initiated during the bilateral PHC-Aurora project
*French Norwegian Research Effort on Applied Dialogue Modelling*. Laurent Prévot would also like to
acknowledge continuous support from the Institute for Language Communication and the Brain (ILCB
ANR-16-CONV0002) and the Excellence Initiative of Aix-Marseille University (A*MIDEX AAP-ID-
17-46-170301-11.1). Hendrik Buschmeier was supported by the German Research Foundation (DFG)
in the Collaborative Research Center TRR 318/1 2021 ‘Constructing Explainability’ (438445824).
We would like to thank Hiro Yamazaki for helping with the Japanese lexical items.
References

Jens Allwood. Om det svenska systemet för språklig återkoppling. In Per Linell, Viveka Adelswärd, Torbjörn Nilsson, and Per A. Pettersson, editors, *Svenskans Beskrivning 16*, volume 1, pages 89–106. Linköping University, Tema Kommunikation, Linköping, Sweden, 1988.

Jens Allwood, Joakim Nivre, and Elisabeth Ahlsén. On the semantics and pragmatics of linguistic feedback. *Journal of Semantics*, 9:1–26, 1992. doi:10.1093/jos/9.1.1.

Jens Allwood, Stefan Kopp, Karl Grammer, Elisabeh Ahlsén, Elisabeth Oberzaucher, and Markus Koppensteiner. The analysis of embodied communicative feedback in multimodal corpora: A prerequisite for behaviour simulation. *Language Resources and Evaluation*, 41:255–272, 2007. doi:10.1007/s10579-007-9056-2.

Anne H. Anderson, Miles Bader, et al. The HCRC map task corpus. *Language and Speech*, 34: 351–366, 1991.

Nicholas Asher, Julie Hunter, Mathieu Morey, Farah Benamara, and Stergos Afantenos. Discourse structure and dialogue acts in multiparty dialogue: The STAC corpus. In *Proceedings of the 10th International Conference on Language Resources and Evaluation*, pages 2721–2727, 2016. URL https://aclanthology.org/L16-1432.

Agnes Axelsson, Hendrik Buschmeier, and Gabriel Skantze. Modelling feedback in interaction with conversational agents – a review. *Frontiers in Computer Science*, 4:744574, 2022. doi:10.3389/fcomp.2022.744574.

Janet Bavelas and Nicole Chovil. Some pragmatic functions of conversational facial gestures. * Gesture*, 17:98–127, 2018. doi:10.1075/gest.00012.bav.

Janet B. Bavelas, Linda Coates, and Trudy Johnson. Listener responses as a collaborative process: The role of gaze. *Journal of Communication*, 52:566–580, 2002. doi:10.1111/j.1460-2466.2002.tb02562.x.

Todd Berliner. Hollywood movie dialogue and the "real realism" of john cassavetes. *Film Quaterly*, 52(3):2–16, 1999.

Philippe Blache, Roxane Bertrand, Gaëlle Ferré, Berthille Pallaud, Laurent Prévot, and Stéphane Rauzy. The corpus of interactional data: A large multimodal annotated resource. *Handbook of Linguistic Annotation*, pages 1323–1356, 2017.

Harry Bunt. Context and dialogue control. *Think Quarterly*, 3(1):19–31, 1994.

Harry Bunt. Multifunctionality in dialogue. *Computer Speech and Language*, 25:222–245, 2011. doi:10.1016/j.csl.2010.04.006.

Hendrik Buschmeier and Stefan Kopp. Communicative listener feedback in human-agent interaction: Artificial speakers need to be attentive and adaptive. In *Proceedings of the 17th International Conference on Autonomous Agents and Multiagent Systems*, pages 1213–1221, Stockholm, Sweden, 2018. doi:10.5555/3237383.3237880.
## References

Jens Allwood. Om det svenska systemet för språklig återkoppling. In Per Linell, Viveka Adelswärd, Torbjörn Nilsson, and Per A. Pettersson, editors, *Svenskans Beskrivning 16*, volume 1, pages 89–106. Linköping University, Tema Kommunikation, Linköping, Sweden, 1988.

Jens Allwood, Joakim Nivre, and Elisabeth Ahlsén. On the semantics and pragmatics of linguistic feedback. *Journal of Semantics*, 9:1–26, 1992. doi:10.1093/jos/9.1.1.

Jens Allwood, Stefan Kopp, Karl Grammer, Elisabeh Ahlsén, Elisabeth Oberzaucher, and Markus Koppensteiner. The analysis of embodied communicative feedback in multimodal corpora: A prerequisite for behaviour simulation. *Language Resources and Evaluation*, 41:255–272, 2007. doi:10.1007/s10579-007-9056-2.

Anne H. Anderson, Miles Bader, et al. The HCRC map task corpus. *Language and Speech*, 34: 351–366, 1991.

Nicholas Asher, Julie Hunter, Mathieu Morey, Farah Benamara, and Stergos Afantenos. Discourse structure and dialogue acts in multiparty dialogue: The STAC corpus. In *Proceedings of the 10th International Conference on Language Resources and Evaluation*, pages 2721–2727, 2016. URL https://aclanthology.org/L16-1432.

Agnes Axelsson, Hendrik Buschmeier, and Gabriel Skantze. Modelling feedback in interaction with conversational agents – a review. *Frontiers in Computer Science*, 4:744574, 2022. doi:10.3389/fcomp.2022.744574.

Janet Bavelas and Nicole Chovil. Some pragmatic functions of conversational facial gestures. * Gesture*, 17:98–127, 2018. doi:10.1075/gest.00012.bav.

Janet B. Bavelas, Linda Coates, and Trudy Johnson. Listener responses as a collaborative process: The role of gaze. *Journal of Communication*, 52:566–580, 2002. doi:10.1111/j.1460-2466.2002.tb02562.x.

Todd Berliner. Hollywood movie dialogue and the "real realism" of john cassavetes. *Film Quaterly*, 52(3):2–16, 1999.

Philippe Blache, Roxane Bertrand, Gaëlle Ferré, Berthille Pallaud, Laurent Prévot, and Stéphane Rauzy. The corpus of interactional data: A large multimodal annotated resource. *Handbook of Linguistic Annotation*, pages 1323–1356, 2017.

Harry Bunt. Context and dialogue control. *Think Quarterly*, 3(1):19–31, 1994.

Harry Bunt. Multifunctionality in dialogue. *Computer Speech and Language*, 25:222–245, 2011. doi:10.1016/j.csl.2010.04.006.

Hendrik Buschmeier and Stefan Kopp. Communicative listener feedback in human-agent interaction: Artificial speakers need to be attentive and adaptive. In *Proceedings of the 17th International Conference on Autonomous Agents and Multiagent Systems*, pages 1213–1221, Stockholm, Sweden, 2018. doi:10.5555/3237383.3237880.
Jean Carletta. Unleashing the killer corpus: experiences in creating the multi-everything AMI Meeting Corpus. *Language Resources and Evaluation*, 41:181–190, 2007.

Neda Chepinchikj and Celia Thompson. Analysing cinematic discourse using conversation analysis. *Discourse, Context & Media*, 14:40–53, 2016. doi:10.1016/j.dcm.2016.09.001.

Christopher Cieri, David Miller, and Kevin Walker. The Fisher corpus: A resource for the next generations of speech-to-text. In *LREC*, volume 4, pages 69–71, 2004.

Herbert H. Clark. *Using Language*. Cambridge University Press, Cambridge, UK, 1996. doi:10.1017/CBO9780511620539.

Herbert H. Clark and Edward F. Schaefer. Contributing to discourse. *Cognitive Science*, 13:259–294, 1989. doi:10.1207/s15516709cog1302_7.

Shauna Concannon and Marcus Tomalin. Measuring perceived empathy in dialogue systems. *AI & SOCIETY*, pages 1–15, 2023.

Mark G. Core and James F. Allen. Coding dialogs with the damsl annotation scheme. In *Proceedings of the AAAI Fall Symposium on Communicative Action in Humans and Machines*, Cambridge, MA, USA, 1997.

Cristian Danescu-Niculescu-Mizil and Lillian Lee. Chameleons in imagined conversations: A new approach to understanding coordination of linguistic style in dialogs. In *Proceedings of the 2nd Workshop on Cognitive Modeling and Computational Linguistics*, pages 76–87, Portland, OR, USA, 2011.

Mark Davies. The TV and movies corpora: Design, construction, and use. *International Journal of Corpus Linguistics*, 26:10–37, 2021.

Yasuharu Den and John Fry. CallHome Japanese corpus (in Japanese). *Journal of the Phonetic Society of Japan*, 4(2):24–30, 2000.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)*, pages 4171–4186, Minneapolis, MN, USA, 2019. doi:10.18653/v1/N19-1423.

Christina Dideriksen, Morten H Christiansen, Kristian Tylén, Mark Dingemanse, and Riccardo Fusaroli. Quantifying the interplay of conversational devices in building mutual understanding. *Journal of Experimental Psychology: General*, 2022.

Mark Dingemanse and Andreas Liesenfeld. From text to talk: Harnessing conversational corpora for humane and diversity-aware language technology. In *Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 5614–5633, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi:10.18653/v1/2022.acl-long.385. URL https://aclanthology.org/2022.acl-long.385.
Jean Carletta. Unleashing the killer corpus: experiences in creating the multi-everything AMI Meeting Corpus. *Language Resources and Evaluation*, 41:181–190, 2007.

Neda Chepinchikj and Celia Thompson. Analysing cinematic discourse using conversation analysis. *Discourse, Context & Media*, 14:40–53, 2016. doi:10.1016/j.dcm.2016.09.001.

Christopher Cieri, David Miller, and Kevin Walker. The Fisher corpus: A resource for the next generations of speech-to-text. In *LREC*, volume 4, pages 69–71, 2004.

Herbert H. Clark. *Using Language*. Cambridge University Press, Cambridge, UK, 1996. doi:10.1017/CBO9780511620539.

Herbert H. Clark and Edward F. Schaefer. Contributing to discourse. *Cognitive Science*, 13:259–294, 1989. doi:10.1207/s15516709cog1302_7.

Shauna Concannon and Marcus Tomalin. Measuring perceived empathy in dialogue systems. *AI & SOCIETY*, pages 1–15, 2023.

Mark G. Core and James F. Allen. Coding dialogs with the damsl annotation scheme. In *Proceedings of the AAAI Fall Symposium on Communicative Action in Humans and Machines*, Cambridge, MA, USA, 1997.

Cristian Danescu-Niculescu-Mizil and Lillian Lee. Chameleons in imagined conversations: A new approach to understanding coordination of linguistic style in dialogs. In *Proceedings of the 2nd Workshop on Cognitive Modeling and Computational Linguistics*, pages 76–87, Portland, OR, USA, 2011.

Mark Davies. The TV and movies corpora: Design, construction, and use. *International Journal of Corpus Linguistics*, 26:10–37, 2021.

Yasuharu Den and John Fry. CallHome Japanese corpus (in Japanese). *Journal of the Phonetic Society of Japan*, 4(2):24–30, 2000.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)*, pages 4171–4186, Minneapolis, MN, USA, 2019. doi:10.18653/v1/N19-1423.

Christina Dideriksen, Morten H Christiansen, Kristian Tylén, Mark Dingemanse, and Riccardo Fusaroli. Quantifying the interplay of conversational devices in building mutual understanding. *Journal of Experimental Psychology: General*, 2022.

Mark Dingemanse and Andreas Liesenfeld. From text to talk: Harnessing conversational corpora for humane and diversity-aware language technology. In *Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 5614–5633, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi:10.18653/v1/2022.acl-long.385. URL https://aclanthology.org/2022.acl-long.385.
Mark Dingemanse, Seán G. Roberts, Julija Baranova, Joe Blythe, Paul Drew, Simeon Floyd, Rosa S. Gisladottir, Kobin H. Kendrick, Stephen C. Levinson, Elizabeth Manrique, Giovanni Rossi, and Nick J. Enfield. Universal principles in the repair of communication problems. *PLoS ONE*, 10: e0136100, 2015. doi:10.1371/journal.pone.0136100.

Paolo Dragone and Pierre Lison. Classification and resolution of non-sentential utterances in dialogue. *IJCoL. Italian Journal of Computational Linguistics*, 2(2-1):45-61, 2016. doi:https://doi.org/10.4000/ijcol.353.

Jens Edlund, Mattias Heldner, and Antoine Pelcé. Prosodic features of very short utterances in dialogue. In Vainio Martti, Aulanko Reijo, and Olli Aaltonen, editors, *Nordic Prosody. Proceedings of the Xth Conference, Helsinki 2008*, pages 57-68. Peter Lang, Frankfurt am Main, Germany, 2009.

Raquel Fernández, Jonathan Ginzburg, and Shalom Lappin. Classifying Non-Sentential Utterances in Dialogue: A Machine Learning Approach. *Computational Linguistics*, 33(3):397-427, 09 2007. ISSN 0891-2017. doi:10.1162/coli.2007.33.3.397. URL https://doi.org/10.1162/coli.2007.33.3.397.

Riccardo Fusaroli, Kristian Tylén, Katrine Garly, Jakob Steensig, Morten H Christiansen, and Mark Dingemanse. Measures and mechanisms of common ground: Backchannels, conversational repair, and interactive alignment in free and task-oriented social interactions. In *the 39th Annual Conference of the Cognitive Science Society (CogSci 2017)*, pages 2055-2060. Cognitive Science Society, 2017.

Jonathan Ginzburg. *The Interactive Stance: Meaning for Conversation*. Oxford University Press UK, 2012.

John J Godfrey, Edward C Holliman, and Jane McDaniel. Switchboard: Telephone speech corpus for research and development. In *Acoustics, Speech, and Signal Processing, IEEE International Conference on*, volume 1, pages 517-520. IEEE Computer Society, 1992.

Charles Goodwin. Between and within: Alternative sequential treatments of continuers and assessments. *Human studies*, 9(2-3):205-217, 1986.

Jan Gorisch, Corine Astésano, Ellen Gurman Bard, Brigitte Bigi, and Laurent Prévot. Aix map task corpus: The French multimodal corpus of task-oriented dialogue. In *Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14)*, pages 2648-2652, Reykjavik, Iceland, May 2014. European Language Resources Association (ELRA). URL http://www.lrec-conf.org/proceedings/lrec2014/pdf/719_Paper.pdf.

Henrik Gottlieb. Subtitles: readable dialogue? *Eye tracking in audiovisual translation*, pages 37-82, 2012.

Agustín Gravano, Julia Hirschberg, and Štefan Beňuš. Affirmative cue words in task-oriented dialogue. *Computational Linguistics*, 38(1):1-39, 2012. doi:10.1162/COLI_a_00083. URL https://www.aclweb.org/anthology/J12-1001.
Mark Dingemanse, Seán G. Roberts, Julija Baranova, Joe Blythe, Paul Drew, Simeon Floyd, Rosa S. Gisladottir, Kobin H. Kendrick, Stephen C. Levinson, Elizabeth Manrique, Giovanni Rossi, and Nick J. Enfield. Universal principles in the repair of communication problems. *PLoS ONE*, 10: e0136100, 2015. doi:10.1371/journal.pone.0136100.

Paolo Dragone and Pierre Lison. Classification and resolution of non-sentential utterances in dialogue. *IJCoL. Italian Journal of Computational Linguistics*, 2(2-1):45-61, 2016. doi:https://doi.org/10.4000/ijcol.353.

Jens Edlund, Mattias Heldner, and Antoine Pelcé. Prosodic features of very short utterances in dialogue. In Vainio Martti, Aulanko Reijo, and Olli Aaltonen, editors, *Nordic Prosody. Proceedings of the Xth Conference, Helsinki 2008*, pages 57-68. Peter Lang, Frankfurt am Main, Germany, 2009.

Raquel Fernández, Jonathan Ginzburg, and Shalom Lappin. Classifying Non-Sentential Utterances in Dialogue: A Machine Learning Approach. *Computational Linguistics*, 33(3):397-427, 09 2007. ISSN 0891-2017. doi:10.1162/coli.2007.33.3.397. URL https://doi.org/10.1162/coli.2007.33.3.397.

Riccardo Fusaroli, Kristian Tylén, Katrine Garly, Jakob Steensig, Morten H Christiansen, and Mark Dingemanse. Measures and mechanisms of common ground: Backchannels, conversational repair, and interactive alignment in free and task-oriented social interactions. In *the 39th Annual Conference of the Cognitive Science Society (CogSci 2017)*, pages 2055-2060. Cognitive Science Society, 2017.

Jonathan Ginzburg. *The Interactive Stance: Meaning for Conversation*. Oxford University Press UK, 2012.

John J Godfrey, Edward C Holliman, and Jane McDaniel. Switchboard: Telephone speech corpus for research and development. In *Acoustics, Speech, and Signal Processing, IEEE International Conference on*, volume 1, pages 517-520. IEEE Computer Society, 1992.

Charles Goodwin. Between and within: Alternative sequential treatments of continuers and assessments. *Human studies*, 9(2-3):205-217, 1986.

Jan Gorisch, Corine Astésano, Ellen Gurman Bard, Brigitte Bigi, and Laurent Prévot. Aix map task corpus: The French multimodal corpus of task-oriented dialogue. In *Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14)*, pages 2648-2652, Reykjavik, Iceland, May 2014. European Language Resources Association (ELRA). URL http://www.lrec-conf.org/proceedings/lrec2014/pdf/719_Paper.pdf.

Henrik Gottlieb. Subtitles: readable dialogue? *Eye tracking in audiovisual translation*, pages 37-82, 2012.

Agustín Gravano, Julia Hirschberg, and Štefan Beňuš. Affirmative cue words in task-oriented dialogue. *Computational Linguistics*, 38(1):1-39, 2012. doi:10.1162/COLI_a_00083. URL https://www.aclweb.org/anthology/J12-1001.
Mattias Heldner, Jens Edlund, and Julia Hirschberg. Pitch similarity in the vicinity of backchannelers. In *Proceedings of INTERSPEECH 2010*, pages 3054–3057, Makuhari, Japan, 2010. doi:10.21437/Interspeech.2010-58.

Mattias Heldner, Anna Hjalmarsson, and Jens Edlund. Backchannel relevance spaces. In *Proceedings of Nordic Prosody XI*, pages 137–146, Tartu, Estonia, 2013. URL http://www.diva-portal.org/smash/get/diva2:628698/FULLTEXT02.pdf.

Julian Hough and David Schlangen. Investigating fluidity for human-robot interaction with real-time, real-world grounding strategies. In *Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue*, pages 288–298, Los Angeles, CA, USA, 2016.

Christine Howes and Arash Eshghi. Feedback relevance spaces: Interactional constraints on processing contexts in Dynamic Syntax. *Journal of Logic, Language and Information*, 30(2):331–362, 2021. URL https://link.springer.com/article/10.1007/s10849-020-09328-1.

HZSK. HAMATAC. The Hamburg MapTask Corpus, 2010. URL http://hdl.handle.net/11022/0000-0000-6330-A.

Tim Isbister, Fredrik Carlsson, and Magnus Sahlgren. Should we stop training more monolingual models, and simply use machine translation instead? In *Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa)*, pages 385–390, Reykjavik, Iceland (Online), May 31–2 June 2021. Linköping University Electronic Press, Sweden. URL https://aclanthology.org/2021.nodalida-main.42.

Gail Jefferson. Side Sequences. *Studies in social interaction*, 1972.

Gail Jefferson. Notes on a systematic deployment of the acknowledgement tokens “Yeah”; and “Mm Hm”; *Paper in Linguistics*, 17(2):197–216, 1984. ISSN 0031-1251. doi:10.1080/08351818409389201.

Janne Bondi Johannessen, Kristin Hagen, Joel James Priestley, and Lars Nygaard. An advanced speech corpus for Norwegian. In *Proceedings of the 16th Nordic Conference of Computational Linguistics (NODALIDA 2007)*, pages 29–36, Tartu, Estonia, May 2007. University of Tartu, Estonia. URL https://aclanthology.org/W07-2406.

Dan Jurafsky, Elizabeth Shriberg, Barbara Fox, and Traci Curl. Lexical, prosodic, and syntactic cues for dialog acts. In *Discourse Relations and Discourse Markers*, 1998.

Jason Kessler. Scattertext: a browser-based tool for visualizing how corpora differ. In *Proceedings of ACL 2017, System Demonstrations*, pages 85–90, Vancouver, Canada, July 2017. Association for Computational Linguistics. URL https://aclanthology.org/P17-4015.

Göran Kjellmer. Where do we backchannel?: On the use of mm, mhm, uh huh and such like. *International Journal of Corpus Linguistics*, 14(1):81–112, 2009. doi:https://doi.org/10.1075/ijcl.14.1.05kje.

Tiziano Labruna, Sofia Brenna, Andrea Zaninello, and Bernardo Magnini. Unraveling ChatGPT: A critical analysis of AI-generated goal-oriented dialogues and annotations. *arXiv preprint*, 2023. URL https://arxiv.org/abs/2305.14556.
Mattias Heldner, Jens Edlund, and Julia Hirschberg. Pitch similarity in the vicinity of backchannels. In *Proceedings of INTERSPEECH 2010*, pages 3054–3057, Makuhari, Japan, 2010. doi:10.21437/Interspeech.2010-58.

Mattias Heldner, Anna Hjalmarsson, and Jens Edlund. Backchannel relevance spaces. In *Proceedings of Nordic Prosody XI*, pages 137–146, Tartu, Estonia, 2013. URL http://www.diva-portal.org/smash/get/diva2:628698/FULLTEXT02.pdf.

Julian Hough and David Schlangen. Investigating fluidity for human-robot interaction with real-time, real-world grounding strategies. In *Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue*, pages 288–298, Los Angeles, CA, USA, 2016.

Christine Howes and Arash Eshghi. Feedback relevance spaces: Interactional constraints on processing contexts in Dynamic Syntax. *Journal of Logic, Language and Information*, 30(2):331–362, 2021. URL https://link.springer.com/article/10.1007/s10849-020-09328-1.

HZSK. HAMATAC. The Hamburg MapTask Corpus, 2010. URL http://hdl.handle.net/11022/0000-0000-6330-A.

Tim Isbister, Fredrik Carlsson, and Magnus Sahlgren. Should we stop training more monolingual models, and simply use machine translation instead? In *Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa)*, pages 385–390, Reykjavik, Iceland (Online), May 31–2 June 2021. Linköping University Electronic Press, Sweden. URL https://aclanthology.org/2021.nodalida-main.42.

Gail Jefferson. Side Sequences. *Studies in social interaction*, 1972.

Gail Jefferson. Notes on a systematic deployment of the acknowledgement tokens “Yeah”; and “Mm Hm”; *Paper in Linguistics*, 17(2):197–216, 1984. ISSN 0031-1251. doi:10.1080/08351818409389201.

Janne Bondi Johannessen, Kristin Hagen, Joel James Priestley, and Lars Nygaard. An advanced speech corpus for Norwegian. In *Proceedings of the 16th Nordic Conference of Computational Linguistics (NODALIDA 2007)*, pages 29–36, Tartu, Estonia, May 2007. University of Tartu, Estonia. URL https://aclanthology.org/W07-2406.

Dan Jurafsky, Elizabeth Shriberg, Barbara Fox, and Traci Curl. Lexical, prosodic, and syntactic cues for dialog acts. In *Discourse Relations and Discourse Markers*, 1998.

Jason Kessler. Scattertext: a browser-based tool for visualizing how corpora differ. In *Proceedings of ACL 2017, System Demonstrations*, pages 85–90, Vancouver, Canada, July 2017. Association for Computational Linguistics. URL https://aclanthology.org/P17-4015.

Göran Kjellmer. Where do we backchannel?: On the use of mm, mhm, uh huh and such like. *International Journal of Corpus Linguistics*, 14(1):81–112, 2009. doi:https://doi.org/10.1075/ijcl.14.1.05kje.

Tiziano Labruna, Sofia Brenna, Andrea Zaninello, and Bernardo Magnini. Unraveling ChatGPT: A critical analysis of AI-generated goal-oriented dialogues and annotations. *arXiv preprint*, 2023. URL https://arxiv.org/abs/2305.14556.
Andreas Liesenfeld and Mark Dingemanse. Bottom-up discovery of structure and variation in response tokens ('backchannels') across diverse languages. In *Proceedings of Interspeech 2022*, pages 1126–1130, Incheon, Korea, 2022. doi:10.21437/Interspeech.2022-11288.

Pierre Lison and Raveesh Meena. Automatic turn segmentation of movie & tv subtitles. In *Proceedings of the 2016 Spoken Language Technology Workshop*, pages 245–252, San Diego, CA, USA, 2016. IEEE.

Pierre Lison and Jörg Tiedemann. OpenSubtitles2016: Extracting large parallel corpora from movie and TV subtitles. In *Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)*, pages 923–929, Portorož, Slovenia, May 2016. European Language Resources Association (ELRA). URL https://aclanthology.org/L16-1147.

Pierre Lison, Jörg Tiedemann, and Milen Kouylekov. OpenSubtitles2018: Statistical Rescoring of Sentence Alignments in Large, Noisy Parallel Corpora. In Nicoletta Calzolari (Conference chair), Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Koiti Hasida, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, Stelios Piperidis, and Takenobu Tokunaga, editors, *Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)*, Miyazaki, Japan, May 7-12, 2018 2018. European Language Resources Association (ELRA). ISBN 979-10-95546-00-9. URL https://aclanthology.org/L18-1275.pdf.

Yang Liu, Kun Han, Zhao Tan, and Yun Lei. Using context information for dialog act classification in DNN framework. In *Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing*, pages 2170–2178, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi:10.18653/v1/D17-1231. URL https://aclanthology.org/D17-1231.

Zofia Malisz, Marcin Wlodarczak, Hendrik Buschmeier, Joanna Skubisz, Stefan Kopp, and Petra Wagner. The alico corpus: Analysing the active listener. *Language Resources and Evaluation*, 50: 411–442, 2016. doi:10.1007/s10579-016-9355-6.

Vladislav Maraev, Bill Noble, Chiara Mazzocconi, and Christine Howes. Dialogue act classification is a laughing matter. In *Proceedings of the 25th Workshop on the Semantics and Pragmatics of Dialogue*, Potsdam, Germany, 2021. URL http://semdial.org/anthology/Z21-Maraev_semdial_0015.pdf.

Stefano Mezza, Alessandra Cervone, Evgeny Stepanov, Giuliano Tortoreto, and Giuseppe Riccardi. ISO-standard domain-independent dialogue act tagging for conversational agents. In *Proceedings of the 27th International Conference on Computational Linguistics*, pages 3539–3551, Santa Fe, New Mexico, USA, August 2018. Association for Computational Linguistics. URL https://aclanthology.org/C18-1300.

Philippe Muller and Laurent Prévot. An empirical study of acknowledgment structures. In *7th workshop on the semantics and pragmatics of dialogue*, 2003.

Bill Noble and Vladislav Maraev. Large-scale text pre-training helps with dialogue act recognition, but not without fine-tuning. In *Proceedings of the 14th International Conference on Computational Linguistics*.
Andreas Liesenfeld and Mark Dingemanse. Bottom-up discovery of structure and variation in response tokens ('backchannels') across diverse languages. In *Proceedings of Interspeech 2022*, pages 1126-1130, Incheon, Korea, 2022. doi:10.21437/Interspeech.2022-11288.

Pierre Lison and Raveesh Meena. Automatic turn segmentation of movie & tv subtitles. In *Proceedings of the 2016 Spoken Language Technology Workshop*, pages 245-252, San Diego, CA, USA, 2016. IEEE.

Pierre Lison and Jörg Tiedemann. OpenSubtitles2016: Extracting large parallel corpora from movie and TV subtitles. In *Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)*, pages 923-929, Portorož, Slovenia, May 2016. European Language Resources Association (ELRA). URL https://aclanthology.org/L16-1147.

Pierre Lison, Jörg Tiedemann, and Milen Kouylekov. OpenSubtitles2018: Statistical Rescoring of Sentence Alignments in Large, Noisy Parallel Corpora. In Nicoletta Calzolari (Conference chair), Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Koiti Hasida, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, Stelios Piperidis, and Takenobu Tokunaga, editors, *Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)*, Miyazaki, Japan, May 7-12, 2018 2018. European Language Resources Association (ELRA). ISBN 979-10-95546-00-9. URL https://aclanthology.org/L18-1275.pdf.

Yang Liu, Kun Han, Zhao Tan, and Yun Lei. Using context information for dialog act classification in DNN framework. In *Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing*, pages 2170-2178, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi:10.18653/v1/D17-1231. URL https://aclanthology.org/D17-1231.

Zofia Malisz, Marcin Wlodarczak, Hendrik Buschmeier, Joanna Skubisz, Stefan Kopp, and Petra Wagner. The alico corpus: Analysing the active listener. *Language Resources and Evaluation*, 50: 411-442, 2016. doi:10.1007/s10579-016-9355-6.

Vladislav Maraev, Bill Noble, Chiara Mazzocconi, and Christine Howes. Dialogue act classification is a laughing matter. In *Proceedings of the 25th Workshop on the Semantics and Pragmatics of Dialogue*, Potsdam, Germany, 2021. URL http://semdial.org/anthology/Z21-Maraev_semdial_0015.pdf.

Stefano Mezza, Alessandra Cervone, Evgeny Stepanov, Giuliano Tortoreto, and Giuseppe Riccardi. ISO-standard domain-independent dialogue act tagging for conversational agents. In *Proceedings of the 27th International Conference on Computational Linguistics*, pages 3539-3551, Santa Fe, New Mexico, USA, August 2018. Association for Computational Linguistics. URL https://aclanthology.org/C18-1300.

Philippe Muller and Laurent Prévot. An empirical study of acknowledgment structures. In *7th workshop on the semantics and pragmatics of dialogue*, 2003.

Bill Noble and Vladislav Maraev. Large-scale text pre-training helps with dialogue act recognition, but not without fine-tuning. In *Proceedings of the 14th International Conference on Computational Linguistics*.
*Semantics (IWCS)*, pages 166–172, Groningen, The Netherlands (online), June 2021. Association for Computational Linguistics. URL https://aclanthology.org/2021.iwcs-1.16.

OpenAI. ChatGPT (August 3 Version), 2023. https://chat.openai.com/chat.

Tim Paek and Eric Horvitz. Grounding criterion: Toward a formal theory of grounding. Technical Report MSR-TR-2000-40, Microsoft Research, Redmond, WA, USA, 2000.

Maria Pavesi. I shouldn't have let this happen': Demonstratives in film dialogue and film representation. *Telecinematic Stylistics*, pages 19–38, 2020. doi:10.5040/9781350042889.0007.

Laurent Prévot, Jan Gorisch, and Sankar Mukherjee. Annotation and classification of French feedback communicative functions. In *Proceedings of the 29th Pacific Asia Conference on Language, Information and Computation*, pages 298–306, Shanghai, China, October 2015. URL https://aclanthology.org/Y15-1034.

Laurent Prévot, Jan Gorisch, and Roxane Bertrand. A CUP of CoFee: A large collection of feedback utterances provided with communicative function annotations. In *Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)*, pages 3180–3185, Portorož, Slovenia, May 2016. European Language Resources Association (ELRA). URL https://aclanthology.org/L16-1507.

Laurent Prevot, Pierre Magistry, and Pierre Lison. Should we use movie subtitles to study linguistic patterns of conversational speech? a study based on French, English and Taiwan Mandarin. In *Third International Symposium on Linguistic Patters of Spontaneous Speech*, 2019. URL https://hal.science/hal-02385689/document.

Matthew Purver. *The Theory and Use of Clarification Requests in Dialogue*. PhD thesis, King's College, University of London, London, UK, 2004.

Paulo Quaglio. Television dialogue and natural conversation: Linguistic similarities and functional differences. *Corpora and discourse: The challenges of different settings*, pages 189–210, 2008.

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. Technical report, OpenAI, 2019. URL https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf.

Christoph Rühlemann. What dialog is absent from constructed dialog? *English Text Construction*, 13:132–157, 2020. doi:10.1075/etc.00038.ruh.

Renata Savy and Francesco Cutugno. CLIPS diatopic, diamesic and diaphasic variations in spoken Italian. In *Proceedings of the 5th Corpus Linguistics Conference: CL2009*, 2009.

Emanuel A Schegloff. Discourse as an interactional achievement: Some uses of ‘uh huh’ and other things that come between sentences. *Analyzing discourse: Text and talk*, 71:71–93, 1982.

Emanuel A Schegloff. Overlapping talk and the organization of turn-taking for conversation. *Language in society*, 29(1):1–63, 2000.
*Semantics (IWCS)*, pages 166–172, Groningen, The Netherlands (online), June 2021. Association for Computational Linguistics. URL https://aclanthology.org/2021.iwcs-1.16.

OpenAI. ChatGPT (August 3 Version), 2023. https://chat.openai.com/chat.

Tim Paek and Eric Horvitz. Grounding criterion: Toward a formal theory of grounding. Technical Report MSR-TR-2000-40, Microsoft Research, Redmond, WA, USA, 2000.

Maria Pavesi. I shouldn't have let this happen': Demonstratives in film dialogue and film representation. *Telecinematic Stylistics*, pages 19–38, 2020. doi:10.5040/9781350042889.0007.

Laurent Prévot, Jan Gorisch, and Sankar Mukherjee. Annotation and classification of French feedback communicative functions. In *Proceedings of the 29th Pacific Asia Conference on Language, Information and Computation*, pages 298–306, Shanghai, China, October 2015. URL https://aclanthology.org/Y15-1034.

Laurent Prévot, Jan Gorisch, and Roxane Bertrand. A CUP of CoFee: A large collection of feedback utterances provided with communicative function annotations. In *Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)*, pages 3180–3185, Portorož, Slovenia, May 2016. European Language Resources Association (ELRA). URL https://aclanthology.org/L16-1507.

Laurent Prevot, Pierre Magistry, and Pierre Lison. Should we use movie subtitles to study linguistic patterns of conversational speech? a study based on French, English and Taiwan Mandarin. In *Third International Symposium on Linguistic Patters of Spontaneous Speech*, 2019. URL https://hal.science/hal-02385689/document.

Matthew Purver. *The Theory and Use of Clarification Requests in Dialogue*. PhD thesis, King's College, University of London, London, UK, 2004.

Paulo Quaglio. Television dialogue and natural conversation: Linguistic similarities and functional differences. *Corpora and discourse: The challenges of different settings*, pages 189–210, 2008.

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. Technical report, OpenAI, 2019. URL https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf.

Christoph Rühlemann. What dialog is absent from constructed dialog? *English Text Construction*, 13:132–157, 2020. doi:10.1075/etc.00038.ruh.

Renata Savy and Francesco Cutugno. CLIPS diatopic, diamesic and diaphasic variations in spoken Italian. In *Proceedings of the 5th Corpus Linguistics Conference: CL2009*, 2009.

Emanuel A Schegloff. Discourse as an interactional achievement: Some uses of ‘uh huh’ and other things that come between sentences. *Analyzing discourse: Text and talk*, 71:71–93, 1982.

Emanuel A Schegloff. Overlapping talk and the organization of turn-taking for conversation. *Language in society*, 29(1):1–63, 2000.
Elizabeth Shriberg. Disfluencies in switchboard. In *Proceedings of the International Conference on Spoken Language Processing*, volume 96, pages 11–14, 1996.

Andreas Stolcke, Elizabeth Shriberg, Rebecca Bates, Noah Coccaro, Daniel Jurafsky, Rachel Martin, Marie Meteer, Klaus Ries, Paul Taylor, and Carol Van Ess-Dykema. Dialog act modeling for conversational speech. In *Proceedings of the 1998 AAAI Spring Symposium on Applying Machine Learning to Discourse Processing*, pages 98–105, 1998.

Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. Dialogue act modeling for automatic tagging and recognition of conversational speech. *Computational Linguistics*, 26: 339–373, 2000a. doi:10.1162/089120100561737.

Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. Dialogue Act Modeling for Automatic Tagging and Recognition of Conversational Speech. *Computational Linguistics*, 26(3):339–373, 09 2000b. ISSN 0891-2017. doi:10.1162/089120100561737. URL https://doi.org/10.1162/089120100561737.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. *arXiv preprint arXiv:2307.09288*, 2023. URL https://arxiv.org/abs/2307.09288.

David R. Traum. *A Computational Theory of Grounding in Natural Language Conversation*. PhD thesis, University of Rochester, Rochester, NY, USA, 1994.

Khiet P. Truong, Ronald Poppe, Iwan de Kok, and Dirk Heylen. A multimodal analysis of vocal and visual backchannels in spontaneous dialogs. In *Proceedings of INTERSPEECH 2011*, pages 2973–2976, Florence, Italy, 2011. doi:10.21437/Interspeech.2011-744.

Thomas Visser, David R. Traum, David DeVault, and Rieks op den Akker. A model for incremental grounding in spoken dialogue systems. *Journal on Multimodal User Interfaces*, 8:61–73, 2014. doi:10.1007/s12193-013-0147-7.

Tamás Váradi. A Budapesti Szociolingvisztikai Interjú. In Ferenc Kiefer, editor, *A magyar nyelv kézikönyve*, pages 339–360. Akadémiai Kiadó, Budapest, 2003.

Nigel Ward. Non-lexical conversational sounds in American English. *Pragmatics & Cognition*, 14: 129–182, 2006. doi:10.1075/pc.14.1.08war.

Valentin Werner. TV discourse, grammaticality, and language awareness. *The Electronic Journal for English as a Second Language*, 24(3):1–23, 2020.

Barbara Wheatley. CALLHOME Mandarin Chinese Transcripts. FTP FILE. Philadelphia, USA: *Linguistic Data Consortium*, 1996. URL https://catalog.ldc.upenn.edu/LDC96T16.

Ramin Yaghoubzadeh, Karola Pitsch, and Stefan Kopp. Adaptive grounding and dialogue management for autonomous conversational assistants for elderly users. In *Proceedings of the 15th
Elizabeth Shriberg. Disfluencies in switchboard. In *Proceedings of the International Conference on Spoken Language Processing*, volume 96, pages 11–14, 1996.

Andreas Stolcke, Elizabeth Shriberg, Rebecca Bates, Noah Coccaro, Daniel Jurafsky, Rachel Martin, Marie Meteer, Klaus Ries, Paul Taylor, and Carol Van Ess-Dykema. Dialog act modeling for conversational speech. In *Proceedings of the 1998 AAAI Spring Symposium on Applying Machine Learning to Discourse Processing*, pages 98–105, 1998.

Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. Dialogue act modeling for automatic tagging and recognition of conversational speech. *Computational Linguistics*, 26: 339–373, 2000a. doi:10.1162/089120100561737.

Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. Dialogue Act Modeling for Automatic Tagging and Recognition of Conversational Speech. *Computational Linguistics*, 26(3):339–373, 09 2000b. ISSN 0891-2017. doi:10.1162/089120100561737. URL https://doi.org/10.1162/089120100561737.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. *arXiv preprint arXiv:2307.09288*, 2023. URL https://arxiv.org/abs/2307.09288.

David R. Traum. *A Computational Theory of Grounding in Natural Language Conversation*. PhD thesis, University of Rochester, Rochester, NY, USA, 1994.

Khiet P. Truong, Ronald Poppe, Iwan de Kok, and Dirk Heylen. A multimodal analysis of vocal and visual backchannels in spontaneous dialogs. In *Proceedings of INTERSPEECH 2011*, pages 2973–2976, Florence, Italy, 2011. doi:10.21437/Interspeech.2011-744.

Thomas Visser, David R. Traum, David DeVault, and Rieks op den Akker. A model for incremental grounding in spoken dialogue systems. *Journal on Multimodal User Interfaces*, 8:61–73, 2014. doi:10.1007/s12193-013-0147-7.

Tamás Váradi. A Budapesti Szociolingvisztikai Interjú. In Ferenc Kiefer, editor, *A magyar nyelv kézikönyve*, pages 339–360. Akadémiai Kiadó, Budapest, 2003.

Nigel Ward. Non-lexical conversational sounds in American English. *Pragmatics & Cognition*, 14: 129–182, 2006. doi:10.1075/pc.14.1.08war.

Valentin Werner. TV discourse, grammaticality, and language awareness. *The Electronic Journal for English as a Second Language*, 24(3):1–23, 2020.

Barbara Wheatley. CALLHOME Mandarin Chinese Transcripts. FTP FILE. Philadelphia, USA: *Linguistic Data Consortium*, 1996. URL https://catalog.ldc.upenn.edu/LDC96T16.

Ramin Yaghoubzadeh, Karola Pitsch, and Stefan Kopp. Adaptive grounding and dialogue management for autonomous conversational assistants for elderly users. In *Proceedings of the 15th
*International Conference on Intelligent Virtual Agents*, pages 28–38, Delft, The Netherlands, 2015.
doi:10.1007/978-3-319-21996-7_3.

Victor H. Yngve. On getting a word in edgewise. In Mary Ann Campbell et al., editors, *Papers from the Sixth Regional Meeting of the Chicago Linguistic Society*, pages 567–577. Chicago Linguistic Society, Chicago, IL, USA, 1970.

Haolan Zhan, Zhuang Li, Yufei Wang, Linhao Luo, Tao Feng, Xiaoxi Kang, Yuncheng Hua, Lizhen Qu, Lay-Ki Soon, Suraj Sharma, Ingrid Zukerman, Zhaleh Semnani-Azad, and Gholamreza Haffari. Socialdial: A benchmark for socially-aware dialogue systems. In *Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval*, SIGIR '23, page 2712–2722, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9781450394086. doi:10.1145/3539618.3591877. URL https://doi.org/10.1145/3539618.3591877.
*International Conference on Intelligent Virtual Agents*, pages 28–38, Delft, The Netherlands, 2015.
doi:10.1007/978-3-319-21996-7_3.

Victor H. Yngve. On getting a word in edgewise. In Mary Ann Campbell et al., editors, *Papers from the Sixth Regional Meeting of the Chicago Linguistic Society*, pages 567–577. Chicago Linguistic Society, Chicago, IL, USA, 1970.

Haolan Zhan, Zhuang Li, Yufei Wang, Linhao Luo, Tao Feng, Xiaoxi Kang, Yuncheng Hua, Lizhen Qu, Lay-Ki Soon, Suraj Sharma, Ingrid Zukerman, Zhaleh Semnani-Azad, and Gholamreza Haffari. Socialdial: A benchmark for socially-aware dialogue systems. In *Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval*, SIGIR '23, page 2712–2722, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9781450394086. doi:10.1145/3539618.3591877. URL https://doi.org/10.1145/3539618.3591877.
## Appendix A. Conversational Feedback Lexical Statistics

In Table 9, we present the list of cue words used for computing the lexical statistics in Section 4. **Content warning:** the lists contain potentially offensive language.

<table><thead><tr><th>Language</th><th>Feedback</th><th>Lexical cues</th></tr></thead><tbody><tr><td rowspan="4">de</td><td>+</td><td>ja, jaa, jaha, jap, jep, jo, joa, aha, hey, ach, achso, okay, ok, richtig, sicher, verstehe, cool, wow, klar, gut, definitiv, absolut, genau, natürlich, ja ja, jaja, ja okay, okay ja, ja genau, ja klar, ja gut, gut okay, ah ja, ja richtig, aber sicher, aber klar, na klar, ich weiß, weiß ich, das stimmt, du hast recht, sie haben recht, ja genau richtig, vermutlich, ja vermutlich, aber wirklich</td></tr><tr><td>-</td><td>nein, nee, nö, niemals, stimmt nicht, das glaube ich nicht, glaube nicht, das glaub ich nicht, glaub nicht, vermutlich nicht</td></tr><tr><td>?</td><td>wirklich, bitte, entschuldige, häh, was, wo, warum welchen, welcher, welche, welches, echt, bist du sicher, sind sie sicher</td></tr><tr><td>=</td><td>mhm, m, mm, hm, ähm, mh, oh, äh</td></tr><tr><td rowspan="4">en</td><td>+</td><td>yes, yeah, yep, okay, oh, right, alright, good, ok, sure, ah, nice, cool, exactly, absolutely, true, great, oh wow, right right, oh okay, oh yeah, yeah right, um-hum yeah, that's great, yes yes, yeah yeah, uh-huh yeah, that's right, right yeah, oh yes, i see, i know, that right, that's true, that's good, all right, of course, got it, is he, oh that's nice, oh that's good, well that's nice, oh i see, oh that's great, yeah that's true, well that's good, well that's great, right that's right, oh yeah yeah, that sounds good, yeah that's right, yeah yeah yeah, yeah oh yeah, oh yeah oh, well that's true, i guess so, yeah i agree, yeah it is, i think so, oh i know, yeah i know, it really is, it is, i agree, definitely, i do too, you bet, you're right, it does, i think so too, that's it, i think you're right, i know it, i agree with you, it was, i agree with that, they are, deal, indeed, obviously, clearly, precisely, certainly, no doubt, so do I, I guess so, they really are, it did, they were, they did, me too, to me too, for me too</td></tr><tr><td>-</td><td>no, wait, gosh, nope, my goodness, oh no, but um, but uh, stop it, oh my goodness, oh my gosh, wait a minute, oh my god, not really, not much, no way, shit, fuck, oh no</td></tr><tr><td>?</td><td>what, really, oh really, why not, you sure, is that right</td></tr><tr><td>=</td><td>um-hum, uh-huh, huh-uh, uh, hum, hm, hey, well, wow, um, huh, mh, mm, mm, mm, mm, mm, mm, mm</td></tr><tr><td>fr</td><td>+</td><td>oui, ouais, ok, ah, voilà, bien, daccord, super, parfait, exactement, ah ouais, ouais ouais, et ouais, d'accord, ah oui, oui oui, c'est ça, eh ouais, ah ouais, je sais, très bien, je comprends, bien sûr, ouais ouais ouais, ah ouais ouais, c'est vrai, ah ouais d'accord, ah d'accord, ah ouais OK, ah ouais ok, ah oui oui, ah ben oui, tu m'étonnes, c'est bien, sans doute, tout à fait, absolument, vachement, je suis d'accord, moi aussi, c'est vrai, c'est juste, c'est exactement ça</td></tr></tbody></table>
## Appendix A. Conversational Feedback Lexical Statistics

In Table 9, we present the list of cue words used for computing the lexical statistics in Section 4. **Content warning:** the lists contain potentially offensive language.

<table><thead><tr><th>Language</th><th>Feedback</th><th>Lexical cues</th></tr></thead><tbody><tr><td rowspan="4">de</td><td>+</td><td>ja, jaa, jaha, jap, jep, jo, joa, aha, hey, ach, achso, okay, ok, richtig, sicher, verstehe, cool, wow, klar, gut, definitiv, absolut, genau, natürlich, ja ja, jaja, ja okay, okay ja, ja genau, ja klar, ja gut, gut okay, ah ja, ja richtig, aber sicher, aber klar, na klar, ich weiß, weiß ich, das stimmt, du hast recht, sie haben recht, ja genau richtig, vermutlich, ja vermutlich, aber wirklich</td></tr><tr><td>-</td><td>nein, nee, nö, niemals, stimmt nicht, das glaube ich nicht, glaube nicht, das glaub ich nicht, glaub nicht, vermutlich nicht</td></tr><tr><td>?</td><td>wirklich, bitte, entschuldige, häh, was, wo, warum welchen, welcher, welche, welches, echt, bist du sicher, sind sie sicher</td></tr><tr><td>=</td><td>mhm, m, mm, hm, ähm, mh, oh, äh</td></tr><tr><td rowspan="4">en</td><td>+</td><td>yes, yeah, yep, okay, oh, right, alright, good, ok, sure, ah, nice, cool, exactly, absolutely, true, great, oh wow, right right, oh okay, oh yeah, yeah right, um-hum yeah, that's great, yes yes, yeah yeah, uh-huh yeah, that's right, right yeah, oh yes, i see, i know, that right, that's true, that's good, all right, of course, got it, is he, oh that's nice, oh that's good, well that's nice, oh i see, oh that's great, yeah that's true, well that's good, well that's great, right that's right, oh yeah yeah, that sounds good, yeah that's right, yeah yeah yeah, yeah oh yeah, oh yeah oh, well that's true, i guess so, yeah i agree, yeah it is, i think so, oh i know, yeah i know, it really is, it is, i agree, definitely, i do too, you bet, you're right, it does, i think so too, that's it, i think you're right, i know it, i agree with you, it was, i agree with that, they are, deal, indeed, obviously, clearly, precisely, certainly, no doubt, so do I, I guess so, they really are, it did, they were, they did, me too, to me too, for me too</td></tr><tr><td>-</td><td>no, wait, gosh, nope, my goodness, oh no, but um, but uh, stop it, oh my goodness, oh my gosh, wait a minute, oh my god, not really, not much, no way, shit, fuck, oh no</td></tr><tr><td>?</td><td>what, really, oh really, why not, you sure, is that right</td></tr><tr><td>=</td><td>um-hum, uh-huh, huh-uh, uh, hum, hm, hey, well, wow, um, huh, mh, mm, mm, mm, mm, mm, mm</td></tr><tr><td>fr</td><td>+</td><td>oui, ouais, ok, ah, voilà, bien, daccord, super, parfait, exactement, ah ouais, ouais ouais, et ouais, d'accord, ah oui, oui oui, c'est ça, eh ouais, ah ouais, je sais, très bien, je comprends, bien sûr, ouais ouais ouais, ah ouais ouais, c'est vrai, ah ouais d'accord, ah d'accord, ah ouais OK, ah ouais ok, ah oui oui, ah ben oui, tu m'étonnes, c'est bien, sans doute, tout à fait, absolument, vachement, je suis d'accord, moi aussi, c'est vrai, c'est juste, c'est exactement ça</td></tr></tbody></table>
- non, putain, pff, si, merde, oh putain, non non, mon dieu, oh mon dieu, je sais pas, non non non, pas trop, pas vraiment, pas possible

? hein, quoi, vraiment, comment ça

= ah, mh, euh, oh, han, ben, bon, hm, hum, peut-être, m, mh mh, mh ouais, ah bon, mh mh mh, eh, hé, hey

no + ja, jo, å ja, ok, oi, greit, presis, wow, riktig, sant, nettopp, absolutt, jepp, definitivt, åpenbart, deal, selvfølgelig, sikkert, akkurat, god, bra, helt sikkert, jeg vet, jeg skjønner, helt riktig, det stemmer, klart det, uten tvil, det er riktig, det er greit, det er sant, det er det, jeg er enig, du har rett, det gjør det, jeg tror det, jeg vet det, det var det, det gjør jeg, jeg antar det, det gjorde det, det gjør jeg også, det tror jeg også, jeg tror du har rett, jeg er enig med deg, jeg er enig i det, de er det, de var, det gjorde de, meg også, til meg også, for meg også

- nei, faen, javel, herregud, ikke helt, ikke mulig, ikke i det hele tatt

? virkelig, hva, hæ

= m, mhm, mh, hmm, mm, mmm, mmhmm, hm, uh-huh, ikke sant

hu + igen, tényleg, úgy van, helyes, jogos, igaz, valóban, pontosan, tudom, rendben, ok, oké, oksi, okés, okszi, igen az, de az, bizony, természetesen, határozottan, feltétlenül, mindenképp, egyetértek, szerintem is, ó igen, hogyne, tényleg az, én is, nekem is, engem is, tőlem is, bennem is, igazad van, naná, mi az hogy, meghiszem azt, biztosra veheted, biztos lehetsz benne, jó, ja, szerintem igen, szerintem is, én is így gondolom, én is úgy gondolom, ennyi, ez az, így van, úgy van, szerintem igazad van, szerintem igazatok van, tudom, jól tudom, egyetértek, az volt, ez volt, de, azok, igen, azok, megegyeztünk, egyértelműen, azt hiszem, kétségtelenül, biztosan, persze, értem, tudod, stimmel, valóban, hát igen, hát dehogynem

- nem, nem igazán, nem létezik, a francba, a fenét, ne, a csodát, hogy a csodába, hát nem

? ó tényleg, micsoda, tényleg, miért ne, biztos

= aha, hú, ú, ó, óh, hű, ja, mhm, mm, mmm, hmm, hmmm, wow, azta, ejha, nahát, ühüm

it + ehi, okay, okay, ok, sì, si, vabbè, ecco, perfetto, wow, esatto, certamente, esattamente, assolutamente, sicuramente, decisamente, ovviamente, precisamente, di sicuro, sono d'accordo, concordo, eccellente, grandioso, ottimo, certo, infatti, fantastico, magnifico, naturalmente, giusto, bene, già, lo ben so, ah ah, ah ha, vero, é vero, lo so, lo è, davvero, vero, oh sì, lo è veramente, anch'io, anche io, hai ragione, d'accordo, va bene, benissimo, bello, buono, penso di sì, credo di sì, mi sa di sì, mi pare di sì, anche secondo me, lo penso anch'io, è così, penso che tu abbia ragione, penso tu abbia ragione, credo che tu abbia ragione, credo tu abbia ragione, mi sa che hai ragione, sono d'accordo con te, sono d'accordo con voi, lo era, lo è stato, lo è stata, sono d'accordo con ciò, lo sono, senza dubbio, a posto, ci sto, lo sono stati, lo erano, anche a me

- oddio, merda, no, non proprio, non molto, non è possibile, cazzo, oh no, macché
? come, davvero, cosa
<table><tr><td>-</td><td>non, putain, pff, si, merde, oh putain, non non, mon dieu, oh mon dieu, je sais pas, non non non, pas trop, pas vraiment, pas possible</td></tr><tr><td>?</td><td>hein, quoi, vraiment, comment ça</td></tr><tr><td>=</td><td>ah, mh, euh, oh, han, ben, bon, hm, hum, peut-être, m, mh mh, mh ouais, ah bon, mh mh mh, eh, hé, hey</td></tr></table>

<table><tr><td>no</td><td>ja, jo, å ja, ok, oi, greit, presis, wow, riktig, sant, nettopp, absolutt, jepp, definitivt, åpenbart, deal, selvfølgelig, sikkert, akkurat, god, bra, helt sikkert, jeg vet, jeg skjønner, helt riktig, det stemmer, klart det, uten tvil, det er riktig, det er greit, det er sant, det er det, jeg er enig, du har rett, det gjør det, jeg tror det, jeg vet det, det var det, det gjør jeg, jeg antar det, det gjorde det, det gjør jeg også, det tror jeg også, jeg tror du har rett, jeg er enig med deg, jeg er enig i det, de er det, de var, det gjorde de, meg også, til meg også, for meg også</td></tr><tr><td>-</td><td>nei, faen, javel, herregud, ikke helt, ikke mulig, ikke i det hele tatt</td></tr><tr><td>?</td><td>virkelig, hva, hæ</td></tr><tr><td>=</td><td>m, mhm, mh, hmm, mm, mmm, mmhmm, hm, uh-huh, ikke sant</td></tr></table>

<table><tr><td>hu</td><td>igen, tényleg, úgy van, helyes, jogos, igaz, valóban, pontosan, tudom, rendben, ok, oké, oksi, okés, okszi, igen az, de az, bizony, természetesen, határozottan, feltétlenül, mindenképp, egyetértek, szerintem is, ó igen, hogyne, tényleg az, én is, nekem is, engem is, tőlem is, bennem is, igazad van, naná, mi az hogy, meghiszem azt, biztosra veheted, biztos lehetsz benne, jó, ja, szerintem igen, szerintem is, én is így gondolom, én is úgy gondolom, ennyi, ez az, így van, úgy van, szerintem igazad van, szerintem igazatok van, tudom, jól tudom, egyetértek, az volt, ez volt, de, azok, igen, azok, megegyeztünk, egyértelműen, azt hiszem, kétségtelenül, biztosan, persze, értem, tudod, stimmel, valóban, hát igen, hát dehogynem</td></tr><tr><td>-</td><td>nem, nem igazán, nem létezik, a francba, a fenét, ne, a csodát, hogy a csodába, hát nem</td></tr><tr><td>?</td><td>ó tényleg, micsoda, tényleg, miért ne, biztos</td></tr><tr><td>=</td><td>aha, hú, ú, ó, óh, hű, ja, mhm, mm, mmm, hmm, hmmm, wow, azta, ejha, nahát, ühüm</td></tr></table>

<table><tr><td>it</td><td>ehi, okay, okay, ok, sì, si, vabbè, ecco, perfetto, wow, esatto, certamente, esattamente, assolutamente, sicuramente, decisamente, ovviamente, precisamente, di sicuro, sono d'accordo, concordo, eccellente, grandioso, ottimo, certo, infatti, fantastico, magnifico, naturalmente, giusto, bene, già, lo ben so, ah ah, ah ha, vero, é vero, lo so, lo è, davvero, vero, oh sì, lo è veramente, anch'io, anche io, hai ragione, d'accordo, va bene, benissimo, bello, buono, penso di sì, credo di sì, mi sa di sì, mi pare di sì, anche secondo me, lo penso anch'io, è così, penso che tu abbia ragione, penso tu abbia ragione, credo che tu abbia ragione, credo tu abbia ragione, mi sa che hai ragione, sono d'accordo con te, sono d'accordo con voi, lo era, lo è stato, lo è stata, sono d'accordo con ciò, lo sono, senza dubbio, a posto, ci sto, lo sono stati, lo erano, anche a me</td></tr><tr><td>-</td><td>oddio, merda, no, non proprio, non molto, non è possibile, cazzo, oh no, macché</td></tr><tr><td>?</td><td>come, davvero, cosa</td></tr></table>
<table><caption>Table 9: Lists of cue phrases employed in the lexical overview of Section 4.1. We distinguish between four core categories of feedback, namely positive feedback/acknowledgment (+), neutral/continuer (=), negative (-), and clarification request (?).</caption><tr><td></td><td>=</td><td>eh, Mm-hmm, hmm, mmm, mh, eh, mhmh, eh, m, hm, ah, oh, beh, uh-huh, mmh, eeh</td></tr><tr><td>ja</td><td>+</td><td>そう、はい、之え、そうか、はあ、どうぞ、本当、は、あっ、ああ、あ、ね</td></tr><tr><td></td><td>=</td><td>うん、ふーん、之っ、へえ、うーん、ふん、之、う</td></tr><tr><td></td><td>-</td><td>ううん、いいえ、いや、いえ、ない、全くない、ちょっと</td></tr><tr><td></td><td>?</td><td>何</td></tr><tr><td>zh</td><td>+</td><td>okay, yeah, yes, ok, 对, 哦, 好, 是, 有, 真的, 还行, 然, , 太好了, 耶, 行, 一定, 没错, 那好, 对了, 真好, 好啊, 好吧, 可以, 太棒了, 太棒了, 好极了, 说得对, 没问题, 我同意, 懂了, 一樣, 我也是, 不错, 是啊, 就是这样, 当然可以</td></tr><tr><td></td><td>-</td><td>不, 没有, 不起, 不是</td></tr><tr><td></td><td>?</td><td>啊, 是吗, 什麼, 什么, 为什么</td></tr><tr><td></td><td>=</td><td>hey, oh, 嘿, 嗯, 呃, 哼, 哈, 嘘, 喔, 呵, 呵, 噢, 哇, 哦, 哟, 噢</td></tr></table>
<table><caption>Table 9: Lists of cue phrases employed in the lexical overview of Section 4.1. We distinguish between four core categories of feedback, namely positive feedback/acknowledgment (+), neutral/continuer (=), negative (-), and clarification request (?).</caption><tr><td></td><td>=</td><td>eh, Mm-hmm, hmm, mmm, mh, eh, mhmh, eh, m, hm, ah, oh, beh, uh-huh, mmh, eeh</td></tr><tr><td>ja</td><td>+</td><td>そう、はい、之え、そうか、はあ、どうぞ、本当、は、あっ、ああ、あ、ね</td></tr><tr><td></td><td>=</td><td>うん、ふーん、之っ、へえ、うーん、ふん、之、う</td></tr><tr><td></td><td>-</td><td>ううん、いいえ、いや、いえ、ない、全くない、ちょっと</td></tr><tr><td></td><td>?</td><td>何</td></tr><tr><td>zh</td><td>+</td><td>okay, yeah, yes, ok, 对, 哦, 好, 是, 有, 真的, 还行, 然, , 太好了, 耶, 行, 一定, 没错, 那好, 对了, 真好, 好啊, 好吧, 可以, 太棒了, 太棒了, 好极了, 说得对, 没问题, 我同意, 懂了, 一樣, 我也是, 不错, 是啊, 就是这样, 当然可以</td></tr><tr><td></td><td>-</td><td>不, 没有, 不起, 不是</td></tr><tr><td></td><td>?</td><td>啊, 是吗, 什麼, 什么, 为什么</td></tr><tr><td></td><td>=</td><td>hey, oh, 嘿, 嗯, 呃, 哼, 哈, 嘘, 喔, 呵, 呵, 噢, 哇, 哦, 哟, 噢</td></tr></table>
<figure><img src="image_9.png" /><figcaption>Figure 7: Norwegian across genres (rule-based, based on cue word lists).</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>Figure 8: Italian across genres (rule-based, based on cue word lists).</figcaption></figure>

Figures 7–12 present statistics for utterance and feedback type as well as common feedback-related lexical items for different languages.

Figure 13 shows politeness keywords and emojis in our English and French corpora.
<figure><img src="image_8.png" /><figcaption>Figure 7: Norwegian across genres (rule-based, based on cue word lists).</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Figure 8: Italian across genres (rule-based, based on cue word lists).</figcaption></figure>

Figures 7–12 present statistics for utterance and feedback type as well as common feedback-related lexical items for different languages.

Figure 13 shows politeness keywords and emojis in our English and French corpora.
<figure><img src="image_11.png" /><figcaption>Figure 9: German across genres (rule-based, based on cue word lists).</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>Figure 10: Hungarian across genres (rule-based, based on cue word lists).</figcaption></figure>
<figure><img src="image_10.png" /><figcaption>Figure 9: German across genres (rule-based, based on cue word lists).</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>Figure 10: Hungarian across genres (rule-based, based on cue word lists).</figcaption></figure>
<figure><img src="image_13.png" /><figcaption>Figure 11: Mandarin Chinese across genres (rule-based, based on cue word lists).</figcaption></figure>

<figure><img src="image_14.png" /><figcaption>Figure 12: Japanese across genres (rule-based, based on cue word lists).</figcaption></figure>
<figure><img src="image_12.png" /><figcaption>Figure 11: Mandarin Chinese across genres (rule-based, based on cue word lists).</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>Figure 12: Japanese across genres (rule-based, based on cue word lists).</figcaption></figure>
<figure><img src="image_15.png" /><figcaption>Figure 13: Short utterance distribution including politeness and emojis.</figcaption></figure>

<table><caption>Table 10: Amount of feedback phenomena using the neural tagger DA-tagger on spontaneous dialogue corpora. Non-English datasets were automatically translated into English with Google Translate API before inference.</caption><thead><tr><th>Lang</th><th>Corpus</th><th># utt</th><th># feedback</th><th>%</th></tr></thead><tbody><tr><td>de</td><td>hamatac</td><td>4,012</td><td>1,126</td><td>28.07</td></tr><tr><td rowspan="4">en</td><td>ami</td><td>83,085</td><td>20,044</td><td>24.12</td></tr><tr><td>fisher</td><td>2,117,748</td><td>421,069</td><td>19.88</td></tr><tr><td>maptask</td><td>26,949</td><td>8,366</td><td>31.04</td></tr><tr><td>stac</td><td>5,841</td><td>514</td><td>8.8</td></tr><tr><td rowspan="3">fr</td><td>cid</td><td>12,326</td><td>1,754</td><td>14.23</td></tr><tr><td>dvd</td><td>7,578</td><td>1,323</td><td>17.46</td></tr><tr><td>maptask</td><td>6,046</td><td>1,226</td><td>20.28</td></tr><tr><td>hu</td><td>buszi</td><td>30,979</td><td>6,125</td><td>19.77</td></tr><tr><td>it</td><td>clips</td><td>24,289</td><td>4,461</td><td>18.37</td></tr><tr><td>ja</td><td>chome</td><td>38,701</td><td>13,432</td><td>34.71</td></tr><tr><td>no</td><td>nota</td><td>85,506</td><td>16,861</td><td>19.72</td></tr><tr><td>zh</td><td>cn-chome</td><td>17,853</td><td>2,251</td><td>12.61</td></tr></tbody></table>

# Appendix B. Detailed Dialogue Act Tagging Results

## B.1 Results per Corpus

Tables 10 and 11 present the results of our dialogue act tagger per (sub)corpus used. Here, we only make a binary distinction by grouping the feedback-relevant classes *Backchannel* and *Assessment* into a single *Feedback* category. The number of utterances refers to the final version of the data after pre-processing with meta-linguistic information removed.
<figure><img src="image_14.png" /><figcaption>Figure 13: Short utterance distribution including politeness and emojis.</figcaption></figure>

<table><caption>Table 10: Amount of feedback phenomena using the neural tagger DA-tagger on spontaneous dialogue corpora. Non-English datasets were automatically translated into English with Google Translate API before inference.</caption><thead><tr><th>Lang</th><th>Corpus</th><th># utt</th><th># feedback</th><th>%</th></tr></thead><tbody><tr><td>de</td><td>hamatac</td><td>4,012</td><td>1,126</td><td>28.07</td></tr><tr><td rowspan="4">en</td><td>ami</td><td>83,085</td><td>20,044</td><td>24.12</td></tr><tr><td>fisher</td><td>2,117,748</td><td>421,069</td><td>19.88</td></tr><tr><td>maptask</td><td>26,949</td><td>8,366</td><td>31.04</td></tr><tr><td>stac</td><td>5,841</td><td>514</td><td>8.8</td></tr><tr><td rowspan="3">fr</td><td>cid</td><td>12,326</td><td>1,754</td><td>14.23</td></tr><tr><td>dvd</td><td>7,578</td><td>1,323</td><td>17.46</td></tr><tr><td>maptask</td><td>6,046</td><td>1,226</td><td>20.28</td></tr><tr><td>hu</td><td>buszi</td><td>30,979</td><td>6,125</td><td>19.77</td></tr><tr><td>it</td><td>clips</td><td>24,289</td><td>4,461</td><td>18.37</td></tr><tr><td>ja</td><td>chome</td><td>38,701</td><td>13,432</td><td>34.71</td></tr><tr><td>no</td><td>nota</td><td>85,506</td><td>16,861</td><td>19.72</td></tr><tr><td>zh</td><td>cn-chome</td><td>17,853</td><td>2,251</td><td>12.61</td></tr></tbody></table>

# Appendix B. Detailed Dialogue Act Tagging Results

## B.1 Results per Corpus

Tables 10 and 11 present the results of our dialogue act tagger per (sub)corpus used. Here, we only make a binary distinction by grouping the feedback-relevant classes *Backchannel* and *Assessment* into a single *Feedback* category. The number of utterances refers to the final version of the data after pre-processing with meta-linguistic information removed.