How Much Does Nonverbal Communication Conform to Entropy Rate Constancy?: A Case Study on Listener Gaze in Interaction (2024)

Abstract: According to the Entropy Rate Constancy (ERC) principle, the information density of a text is approximately constant over its length. Whether this principle also applies to nonverbal communication signals is still under investigation. We perform empirical analyses of video-recorded dialogue data and investigate whether listener gaze, as an important nonverbal communication signal, adheres to the ERC principle. Results show (1) that the ERC principle holds for listener gaze; and (2) that the two linguistic factors syntactic complexity and turn transition potential are weakly correlated with local entropy of listener gaze.

heading: 1 Introduction
text: In human communication, information is conveyed via various channels. In addition to conveying meaning through linguistic units, interlocutors may express themselves through multimodal signals such as hand gestures, head movements, or prosody.
In face-to-face dialogue, interlocutors monitor signals from different modalities to decide when to take the turn, give feedback, or interject a brief response token (backchannel). Humans may also change their dialogue strategies based on their listeners’ reactions, for example, repeating or explaining what has been said (Clark, 1996, p. 39, 378). By investigating such signals, we can gain insights about (1) how engaged dialogue participants are with the content of a conversation, or (2) how common ground (Clark, 1996) and mutual understanding evolve over the course of an interaction.
One important concept for measuring the growth of common ground is alignment of linguistic information (Pickering and Garrod, 2004). Paralinguistic information such as prosody, or nonverbal information such as body posture or gaze direction, showing
∗Corresponding author: y.wang@uni-bielefeld.de
different levels of convergence during interaction, can serve as evidence of linguistic alignment (Pickering and Garrod, 2004). Recent work introduces methods for investigating alignment of linguistic information through information theoretic measures such as entropy (Shannon, 1948). It is widely argued that linguistic information is uniformly distributed in language use, e.g., in the Uniform Information Density hypothesis (UID; Jaeger, 2010), or the Entropy Rate Constancy principle (ERC; Genzel and Charniak, 2002, 2003). Xu and Reitter (2017)’s analysis of spoken dialogue data shows that entropy of dialogue partners’ speech, interactively evolves and converges as the discourse develops. This phenomenon indicates that mutual understanding has been consistently reached over the shifting of topics. Maës et al. (2022) shows that, while global entropy of dialogue remains constant, topic shift is a linguistic factor which leads to peaks in local entropy. Inspired by Xu et al. (2022) and Maës et al. (2022) this paper focuses on listener gaze during interaction.
In this paper, we quantify the information conveyed through gaze. Based upon and extending our previous work (Wang and Buschmeier, 2023), we first investigate gaze local entropy to add further evidence that non verbal communication conforms to the entropy rate constancy principle, complementing the finding of Xu et al. (2022). Our motivation for this is based on four points: (1) During interaction, interlocutors spend most of their time looking at each other (Rogers et al., 2018); (2) New gaze behaviour will emerge while the conversation develops; (3) Interlocutors are less likely to shift gaze from one corner of their visual field to the other when listening, as it can be perceived as being distracted (but see Goodwin 1985, p. 231); (4) If listener gaze targets the speaker, a gaze shift is less likely to happen, unless there is a cognitive processing-based need for gaze aversion (Argyle and Cook, 1976).
3533 Findings of the Association for Computational Linguistics ACL 2024, pages 3533–3545
August 11–16, 2024 ©2024 Association for Computational Linguistics
Based on these four points, if we consider gaze as meaningful communicative units (similar to lexical units), listeners’ gaze behavior should be predictable to some extent. We also take a closer look at two high level linguistic factors: syntactic complexity (Xu and Reitter, 2016), and transition relevance places (potential turn-taking cues) (Sacks et al., 1974) in order to investigate if there is a correlation between gaze local entropy and these two linguistic factors. We do this as an initial attempt to explain why nonverbal communication (specifically listener gaze) may conform to the ERC principle. In this paper we specifically investigate the following two research questions:
• Does listener gaze, as a nonverbal signal, adhere to the Entropy Rate Constancy (ERC) principle?
• If so, why does listener gaze conform to ERC principle. Do syntactic complexity and transition relevance places correlate with this phenomenon?
The two main contributions of our work, compared to previous studies, are as follow:
• We add to the existing evidence that the ERC principle also holds for nonverbal signals (specifically listener gaze). We do this here for dyadic conversations in a controlled linguistic context, namely task oriented dialogue. This finding is based upon and extends our previous work (Wang and Buschmeier, 2023), improving its experiment setting.
• We investigate the effects of the two linguistic factors syntactic complexity and turn transition potential, which is based on the prediction of transition relevant places, on listener gaze entropy in conversational interaction.

heading: 2 Related Work

heading: 2.1 Studies on Gaze in Multimodal Communication
text: Gaze is an important social cue in human interaction that is used for indicating attention, eliciting feedback, or taking the turn (Kendon, 1967; Duncan, 1975; Goodwin and Goodwin, 1986). Brône et al. (2017) summarize these communicative functions of gaze as following two roles: a participation role, where interlocutors show attentiveness during interaction, and a regulation role, which coordinates the dynamics of speaker-listener role shifting, in
addition to other cues, e.g., prosodic cues such as intonation, duration, loudness, and voice quality (Ward, 2019). A recent lab-based study of human interaction (Kendrick et al., 2023), in which gaze was recorded with professional eye trackers, found gaze direction not to be a significant predictor for turn shifts, while gaze aversion was a strong cue for turn holding (which is different from previous finding Nakano et al. 2003).
From the computational side, gaze has been thoroughly investigated for its communicative function in dialogue modeling tasks. For spoken dialogues systems, as well as for the design of human robot interaction, gaze modeling has been used to facilitate the grounding process (Nakano et al., 2003; Skantze et al., 2014). As a multimodal signal in interaction, gaze can also serve as an important feature for turn management (Jokinen et al. 2013; but see Kendrick et al. 2023, above). Onishi et al. (2023) combine gaze direction with other multimodal features (e.g., head rotation) to the baseline model of a self-supervised voice activity projection model (Ekstedt and Skantze, 2022b), and show that they further improve the accuracy of the model’s performance on next speaker prediction. From a more theoretical perspective, Eberle et al. (2022) investigated transformer’s self-attention mechanism and reveals that it has similar predictive power as task specific human gaze fixation pattern.

heading: 2.2 Measuring Information Content in Communication
text: In both written text and spoken dialog, information theory has been used extensively as a theory for uncovering properties of language. Let 𝑋𝑖 denote a linguistic unit (e.g., a single lexical unit or a long utterance). As the relevant context grows, the predictability of the next linguistic unit will be higher, given that enough contextual cues are available as a prior, which increases the reliability of the posterior estimation. The information density of the random variable 𝑋𝑖 is estimated as the entropy 𝐻 (𝑋) (Shannon, 1948).
Genzel and Charniak (2002, 2003) initiated the idea of measuring the information density through an 𝑛-gram model on written text. In this study, we follow Xu and Reitter (2018) and Giulianelli et al. (2021) in calculating information content: For a linguistic unit, e.g., a sentence 𝑋 which comprises of a sequence of smaller units: ⟨𝑤1, . . . , 𝑤𝑖⟩, where 𝑤𝑖 ∈ 𝜗, with 𝜗 being the set of vocabulary. The information content of the sentence is the aver-
age of the sum of negative logarithm conditional probability defined as:
𝐻 (𝑋) = −1 𝑛 ∑︁ 𝑤𝑖∈𝜗 log 𝑃(𝑤𝑖 | 𝑤1, . . . , 𝑤𝑖−1)
Genzel and Charniak (2002, 2003) further proposed the so called Entropy Rate Constancy (ERC) principle, which stipulates that the rate of transmitted information of a given linguistic unit, from a global perspective of a written text, is roughly constant.
Here, we follow Genzel and Charniak (2002) to explain the ERC principle in details: Let 𝐻 (𝑋𝑖 | 𝐶𝑖 , 𝐿𝑖) denote the conditional entropy of the word 𝑋𝑖, where 𝐿𝑖 = 𝑋𝑖−𝑛+1, . . . , 𝑋𝑖−1 is the local 𝑛- gram context, and 𝐶𝑖 = 𝑋0, 𝑋1, . . . , 𝑋𝑖−1 the context which contains all of the words preceding the word 𝑋𝑖 . The conditional entropy of the word 𝑋𝑖 is then decomposed as:
𝐻 (𝑋𝑖 | 𝐶𝑖 , 𝐿𝑖)︸ ︷︷ ︸ global entropy = 𝐻 (𝑋𝑖 | 𝐿𝑖)︸ ︷︷ ︸ local entropy − 𝐼 (𝑋𝑖;𝐶𝑖 , 𝐿𝑖)︸ ︷︷ ︸ mutual information
𝐻 (𝑋𝑖 | 𝐶𝑖 , 𝐿𝑖) is considered roughly constant based on following reasoning1: Given that 𝐼 (𝑋𝑖;𝐶𝑖 , 𝐿𝑖) – as the mutual information between 𝑋𝑖 under its local context 𝐿𝑖 and its global context𝐶𝑖 – increases because the global context increases, the local entropy 𝐻 (𝑋𝑖 | 𝐿𝑖) would have to increase in order for 𝐻 (𝑋𝑖 | 𝐶𝑖 , 𝐿𝑖) to remain roughly constant, though a variation of the value is possible.
A theory similar to the ERC principle, the Uniform Information Density (UID) Hypothesis (Jaeger, 2010), states that speakers tend to distribute information uniformly throughout an utterance so that less processing load is given to the listeners. Formally, the information expressed by a linguistic signal 𝑦 (e.g., an utterance or, in this study, a sequence of gaze labels) as the so-called surprisal, is 𝑦’s negative log probability: 𝑠(𝑦) = − log𝑝ℓ (𝑦), which can be further factorized. Recently, surprisal is commonly approximated using language models with learned parameters (Smith and Levy, 2013; Goodkind and Bicknell, 2018; Wilcox et al., 2020). Besides constraining speaker choice of words, it has also been discussed how UID influences reading time (Meister et al., 2021), or speech duration (Pimentel et al., 2021). Moreover, recent studies investigated whether nonverbal signals adhere to the ERC principle as well. Xu et al. (2022), for example,
1Note that Verma et al. (2023) re-evaluated the ERC principle in text with a neural sequential model and failed to find decisive evidence for it (with inconclusive results though).
encode co-speech gestures (in monological videos) into discrete labels. We used a similar approach in our previous work (Wang and Buschmeier, 2023) for listeners’ gaze in interaction. These two studies show that gesture and gaze as nonverbal signal, conform to ERC principle as well.
In this study, we use surprisal (namely local entropy) to further explain how we transfer the ERC principle, which was originally developed for texts, to gaze. Here is an intuitive example: let us assume the surprisal of the word sequence ⟨a, good, day⟩ is computed as −(log 𝑃(a) + log 𝑃(good | a) + log 𝑃(day | a, good))/3 (no 𝑛-gram model assumed here), the words are the representation of meaning. Analogously, gaze labels are representations of gaze (see Section 5 for details), which semantically represents the distance (distance features) of the listener’s gaze toward speaker, e.g., close, far, left, right, attention, aversion, etc. Given an example gaze label sequence, e.g., ⟨41, 10, 12⟩, the surprisal is can thus be computed as −(log 𝑃(41) + log 𝑃(10 | 41) + log 𝑃(12 | 10, 41))/3. The local context of gaze labels, e.g., ⟨41, 41, 41, 10, 10, . . . ⟩, is based on the input size, the global context of gaze labels, is a similar sequence, e.g., ⟨41, 41, 41, 10, 10, 41, 41, 41, 10, 10, . . .⟩, ahead of the current local context. Since mutual information is increasing with expansion of previous context (for both word context and gaze context), gaze can be treated analogously to words in research on written text. In this study, we thus investigate whether gaze local entropy increases to confirm our hypothesis.

heading: 3 Data Collection
text: Nonverbal communication takes place in parallel to verbal communication (Stivers and Sidnell, 2005). In order to control the variables in our study, we (roughly) control and constraint verbal information so that we can investigate the variation of nonverbal communication. The interactions we analyze are explanations of a board game (Deep Sea Adventure; Sasaki and Sasaki 2014) and were collected in an interaction study that resulted in the MUNDEX corpus (Türk et al., 2023). In each interaction an experienced explainer explains the games to an explainee who does not know the it yet (see Figure 1).
The interactions are organized into three phases: In the first phase, the game is explained without the game material being present. In a second phase, the game is put on the table and the explanation continues. In the third phase the two participants
start playing the game. The interactions can therefore be considered task-oriented dialogues. In this study we include the videos of 58 interactions from the corpus and extract the first phase (explanation without game). These explanations vary in length from 2:12 min to 17:36 min (mean length 7:04 min, standard deviation 3:15 min). All participants speak German.

heading: 4 Hypothesis
text: Our primary hypothesis is as follows: Similar to verbal information, which is considered to conform to the entropy rate constancy (ERC) principle, nonverbal communication signals such as the listener’s gaze – which, similar to gestures, is usually coordinated with speech during interaction – also conform to it. The original hypothesis was posed in our previous work (Wang and Buschmeier, 2023), but the finding was not conclusive. In this study, we revisit the original hypothesis by refining the experimental setting, e.g., by detecting and removing local outliers and partitioning the video recordings into several groups in a more reasonable way.
We further test the ERC principle in nonverbal communication by extending the primary hypothesis as follows: the ERC principle in nonverbal communication in dialogue is based on the combination of multiple linguistic factors, of which we investigate two: syntactic complexity and turn transition potential of lexical items.

heading: 5 Processing Listener Gaze and Dialogue in Interaction
text: For each recorded video from the corpus, we extract the explainee’s gaze information using the Openface framework (Baltrusaitis et al., 2018). Openface generates two types of gaze features (i) gaze direction values in radians averaged from both eyes (two
dimensions) and (ii) gaze vectors in world coordinates (three dimensions for the left and right eye each). We integrate both representations so that the final gaze label does not only reflect gaze dynamics in three dimensions but also incorporates the effects from listener’s head orientation during interaction. The computation of a gaze label follows these steps:
1. For the gaze direction values, we use the DBSCAN clustering algorithm (Ester et al., 1996) to find the spatial distribution of gaze and identify its “dense region” (Tran et al., 2020), both horizontally and vertically. The motivation of finding the “dense region” is based on the intuition that interlocutors spend a large proportion of time during the conversation looking at each other. Based on the minimal and maximal gaze direction values, a 3 × 3 grid is set up. The gaze direction values that are in this dense region are given the label ‘5’, where the minimum and maximum gaze direction values among all gaze direction values in the dense region are used to decide on its boundary. The gaze direction values that are not in the dense region are assigned eight other number-based labels. The overall process follows the ICE algorithm (Tran et al., 2020) while the idea of using a 3 × 3 grid is based on Xu et al. (2022) and Xu and Cheng (2023).
2. For the gaze vectors, we only take the depth dimension into account and process it similarly to the previous step. DBSCAN-clustering is used to find the dense region where the gaze of the left and right eye are located in the depth dimension. The eye gaze vector inside this dense region is again given the label ‘5’. Based on how close the left eye vector or right eye vector are to the dense region, eight different labels are used. This process can sometimes fail, so that only one cluster is generated. We cope with these exceptional cases, by normalizing the depth value to a range [0, 1] and consider values in the range [0.25, 0.75] to be inside the ‘dense region’.
3. Given the label 𝑥𝑑 ∈ [1, 9] for the gaze direction value and the label 𝑥𝑣 ∈ [1, 9] for the gaze vector, a combined label 𝑦, that represents the gaze information, is generated as y = (𝑥𝑑 − 1) · 9 + 𝑥𝑣 (with 𝑦 ∈ [1, 81] representing the set of possible eye gaze labels). See Appendix A for further details on encoding eye gaze.
Figure 2 shows the distribution of the 15 most frequent gaze label in our data. The six most frequent
labels (41, 37, 38 ,40, 42, 45) represent the explainees looking in the direction of explainers with various distance (closer or more distant). The less frequent labels (68, 50, . . . ) instead start to show explainees averting their gaze from explainers.
Dialogues were automatically transcribed using ‘Whisper’ (Radford et al., 2022), which generates speech segments with a start and an end time. In order to calculate word timings, we approximated word onset times by calculating the duration of each speech segment, dividing it by its length in words, and approximating word duration (assuming, for this study, that words have uniform length). Eye gaze labels are then aligned with words based on the video’s frame rate (50 fps).
We consider eye movements as sequences of gaze labels (Figure 9 shows an example of speech gaze alignment). To be able to compute the entropy for gaze, we train a transformer model (Vaswani et al., 2017) to estimate the underlying probability distribution of the gaze sequences.We have chosen a transformer model since it has a stronger psychometric predictive power compared to LSTM-RNNs models (Wilcox et al., 2020). To compute the local entropy of a gaze sequence, we first calculate its negative log probability:
NLL(𝑒1, 𝑒2, . . . , 𝑒𝑇 )
= − ( log 𝑃(𝑒1) + 𝑇∑︁ 𝑖=2 log 𝑃(𝑒𝑖 | 𝑒1, . . . , 𝑒𝑖−1) )
where 𝑇 is the maximum index of a given eye movement sequence and 𝑒𝑖 ∈ [1, 81] are the gaze labels. The local entropy 𝐻 (𝑒1, 𝑒2, . . . 𝑒𝑖) of the gaze sequence is then the exponential of NLL (i.e., perplexity). The learning task is thus to predict the next gaze label 𝑒𝑖 based on the preceding sequence ⟨𝑒1, . . . , 𝑒𝑖−1⟩ and minimize its negative
log probability NLL.

heading: 6 Preliminary Result
text: After computing the local entropy, we first calculate statistical significance, removing outliers. We use a simple local outlier detection method by calculating the standard score (Z-score) of each data point. If the standard score is greater than a threshold (for the gaze local entropy, we define it as 4 so that only very extreme values are considered outliers), it is considered an outlier and replaced by the median of the local entropy.
Figure 3 shows the change of local entropy of the combined eye gaze sequences from all 58 interaction videos after removing outliers. The 𝑥-axis represents the dialogue position of the speech segments (with each dialogue position corresponding to about 7 seconds of speech) and the y-axis represents the local entropy of the gaze sequences. Although there seems to be a general increasing trend of the local entropy, we also have to consider the different length of the videos, which may bias the result. Therefore, we decided to divide the videos into three groups based on length to mitigate the length bias and replot the local entropy (see Figure 4).
We then use a linear model and conduct statistical significance tests between dialogue position and gaze local entropy for each of the three sub groups. The results show us that, coefficient values in all three sub groups are positive, with 𝑝-values smaller than 0.05. The increase of local entropy of gaze
(shown in Figure 4) is thus statistically significant, that is, the preliminary result supports our basic hypothesis. The general increasing trend of local entropy indicates that more new gaze labels appear over the course of the explanations.

heading: 7 Analysis of Linguistic Factors
text: The goal of this section is to answer the question why listener gaze, as a nonverbal signal in interaction, conforms to the ERC principle (see Section 6). Specifically, we focus on exploring linguistic factors which may influence the information amount conveyed by listeners’ gaze during interaction. We select two metrics – syntactic complexity and turn transition potential – as the linguistic factors to investigate and perform statistical analyses to check for correlations between gaze local entropy and these two factors.

heading: 7.1 Syntactic Complexity and Gaze Local Entropy
text: Studies of human sentence processing show that when humans are exposed to written language that is structurally complex, processing difficulties accumulate, and this is reflected in the collected eye movement data, e.g., increasing eye fixation can be found in the syntactically complex area among all of the sentences given (cf. Clifton and Staub, 2011). That is, gaze behavior correlates with cognitive load, which can be increased by syntactic complexity. For example, it has been shown that cognitive
load can cause gaze aversion during interaction (Argyle and Cook, 1976). This correlation between syntactic complexity and eye movement found in the literature leads us to the following conjecture: Is syntactic complexity also relevant to the increase of local entropy of listeners’ gaze in face-to-face interaction?

heading: 7.1.1 Calculating Syntactic Complexity
text: Xu and Reitter (2016) show two metrics for calculating syntactic complexity: (1) tree depth of a syntactic tree, and (2) branching factor, i.e., the average number of children in non-leaf nodes when parsing a syntactic tree. We use dependency parsing (specifically ‘Stanza’; Qi et al. 2018) to parse the automatically recognized speech and calculate its syntactic complexity. The advantage of using dependency parsing is that the input does not have to be a complete sentence to be parsed. We calculate syntactic complexity similarly to Xu and Reitter (2016). Given a speaker’s utterance, we compute its length 𝐿 and use dependency parsing to get the number of heads 𝛼 as well as the maximum tree depth 𝛽. The syntactic complexity 𝑆𝐶 of the utterance is then computed as
𝑆𝐶 =
{ 𝜆 · 𝐿
𝛼 + (1 − 𝜆) · 𝛽 if 𝛼 > 0,
(1 − 𝜆) · 𝛽 otherwise.
𝜆 is a tuning factor set to 0.5 by default. First, an utterance which is lengthy is considered more complex, since it has a bigger tree depth value. Second, the more complex phrases an utterance contains, the more complex the utterance is. We take utterances from explainers as input to the linear model. Figure 5 shows a plot of the development of syntactic complexity of utterances over the course of the interactions. Within the defined topic we analyze (explanations of the board game), the syntactic complexity of utterances increases, as verified by a linear model which weakly correlates utterance position to syntactic complexity (𝛽 = 0.0009, 𝑝 < 0.001). Although the beta coefficient is quite low, we consider that the effect size is still valid given that the value range of syntactic complexity is quite small (from 2.8 to 3.05 in Figure 5 2 and 1.5 to 5.9 in the whole data).

heading: 7.1.2 Correlation with Gaze Local Entropy
text: Based on these results (Figure 5), we further investigate whether there is a correlation between the
2While the syntactic complexity is decreasing at the early stage, it is increasing after dialogue position 25.
local entropy of listeners’ gaze and the syntactic complexity of speech 3. A linear model shows a statistically significant relation between the two variables (𝛽 = 0.02, 𝑝 < 0.05). A weak positive correlation between syntactic complexity and local entropy of gaze indicates that, as syntactic complexity is in general increasing, gaze local entropy increases accordingly. A possible explanation for the correlation between syntactic complexity and local entropy of gaze could be that, with the development of the board game explanation, the speech from explainers becomes more syntactically complex, the explainees instead, take more effort to process the information which may lead to, e.g., longer periods of gaze shift, and thus more variation in gaze labels.
Two illustrative examples from our data can be seen in Figure 6. In the chosen dialogue segments (from the two videos), an increase of local entropy is in general correlated with an increase of syntactic complexity (An exception is A of Example 2 in Figure 6b). Moreover, the chosen examples suggest that a higher local entropy value is generally caused by shifts between frequent gaze labels to less frequent ones and vice versa (e.g., shift from label 41 to 71 in A of example 1 (Figure 6a), shift from label 41 to 62 in C of example 2 (Figure 6b). This is in line with the UID theory: the appearance of less frequent units leads to a higher surprisal value
3Here we use all of the utterances instead of only explainers’ utterance since gaze labels are aligned with all of the utterances. Despite some noise, we still think it is the most optimal way based on our observation that explainee utterances are very limitted in our explanation data.
(Jaeger, 2010).

heading: 7.2 Turn Transition Potential and Gaze Local Entropy
text: The interaction of gaze behavior and turn-taking have constantly attracted research attention (e.g., Skantze et al. 2014; Kendrick et al. 2023). For this study, we assume that, turn-taking signals are a linguistic factor which can influence listeners’ gaze behavior and thus gaze local entropy. Our assumption is based on the intuition that listeners will usually wait until speakers finish speaking and then start to talk. And when listeners want to take a turn or give the turn back to previous speakers in interaction, they usually gaze towards the speakers. According to Sacks et al. (1974), in dyadic conversations, some lexical units, which are defined as transition relevant places (TRP), have higher probability that a turn shift can happen. These usually occur at syntactic or pragmatic completeness.
For our study, we did not use the real turn-taking occurrences from our data to test our hypothesis (because explainees are less likely to take the turn during the initial game explanation phase; Fisher et al. 2022), but instead analyze the transition rel-
evance places as turn-taking potential in our data. The experimental setting is as follows: We train a neural sequential model (TurnGPT; Ekstedt and Skantze 2020) based on two available German dialogue data sets (VM2 and DialogueSUM, see below) to approximate human turn-taking behaviour. We aim at calculating the turn-taking potential of each lexical unit in dialogue, represented as a probability 4. After getting the turn-taking potential for all of the lexical units, we analyze the interaction of turn-taking potential, dialogue position, and gaze local entropy.

heading: 7.2.1 Estimating Turn Transition Potential using TurnGPT
text: We use TurnGPT (Ekstedt and Skantze, 2020) to calculate the turn transition potential for our speech data. TurnGPT extends the GPT-2 architecture (Radford et al., 2019) with additional speaker embeddings. The turn-taking ground truth is encoded by adding a token (<ts>) after lexical items where turn-taking occurred. To process German data, we replace the original GPT-2 model with GermanGPT2 (Schweter, 2020) and fine-tune it on two datasets:
• VERBMOBIL (VM2; Kay, 1992): The dataset is based on recordings of various appointment scheduling scenario and consists of 30 800 utterances collected in face-to-face interactions. Utterances are annotated with dialogue acts and include a corresponding speaker ID.
• DialogueSum (Chen et al., 2021): DialogSum is a large-scale dialogue summarization dataset, consisting of 13 460 dialogues. Each utterance in the dataset is aligned with a speaker label. The original data is in English; we used a German translation of the data (Dialogsum-German) available on HuggingFace.
Figure 7 shows two examples of model predictions (with English translations). Lexical items are aligned with the turn transition probabilities predicted by our German TurnGPT model. Lexical items with higher probabilities indicate syntactic and pragmatic completeness, and are considered reasonable places for turns in real conversation. Additional examples can be found in the Appendix A.5.
4It is worth noting that a lexical item classified as a place with a high turn-taking potential probability, does not necessarily coincide with turn-taking in the real dialogue. We interpret the high probability as a reasonable point for taking the turn in a real conversation.
(a) Example 1: Sobald wir uns einen Schatz schnappen dann wird es kritisch dann müssen wir Geld dafür bezahlen (‘As soon as we grab a treasure, things become critical and we have to pay money for it’). (b) Example 2: wir sind also eigentlich Gegner wir spielen gegeneinander aber wir nutzen den gleichen Sauerstoff (‘So we are actually opponents, we play [against each other], but we use the same oxygen’).
(a) Turn transition potential probability in three bins (from left to right: A, B, C) and the relevant gaze local entropy. The red dot inside the boxes are the means of gaze local entropy.
(b) Regression model of turn transition potential probability and gaze local entropy. The red line shows the general trend of gaze local entropy with the increase of turn transition potential probability.
Figure 8: Statistics of turn transition potentials and gaze local entropy.

heading: 7.2.2 Correlation with Gaze Local Entropy
text: We first use a linear model to investigate whether turn transition probabilities (obtained from the process shown in Section 7.2.1) correlate with dialogue position. No statistically significant correlation is found between the two variables (𝛽 = −3.28, 𝑝 = 0.17, 𝛼 = 0.05). However, the negative coefficient value indicates that, in general, higher turn transition probabilities are distributed more at the beginning of dialogues.
After feeding each utterance into the TurnGPT model to obtain the corresponding turn transition potential probability, we divide the data in to three groups by turn transition potential probability 𝑝𝑇 . A/comparatively low: 𝑝𝑇 < 0.5; B/comparatively high: 𝑝𝑇 ∈ [0.5, 0.75]; C/high: 𝑝𝑇 > 0.75. We then aggregate and analyze gaze local entropy by group (Figure 8a). In group B and C, where higher
turn transition probability is aggregated, the corresponding gaze local entropy generally has a slightly lower mean and median value compared to group A. A possible explanation for this result could be that listeners tend to look at the speaker’s face when taking turns or returning the turn to the speaker, resulting in more uniform and predictable gaze labels defined in our study and thus a comparatively low gaze local entropy value.
Using a linear model, we find a weak, but statistically insignificant correlation (𝛽 = −0.0064, 𝑝 = 0.143, 𝛼 = 0.05) between turn transition potential probability and gaze local entropy. Figure 8b shows this general decreasing linear trend (but data points are distributed quite sparsely).

heading: 8 Conclusion
text: In this study, we extracted listeners’ gaze from video-recorded human interaction dialogues (specifically from explainees in explanatory dialogues) and found evidence that listener gaze, as an important nonverbal signal, adheres to the Entropy Constancy Rate principle – a property of human language use (Genzel and Charniak, 2002). This finding is supported by an increasing trend of the local entropy of listeners’ gaze over the course of the dialogues we analyzed. We believe that this finding can provide future insight into the evaluation of co-speech gaze generation and interpretation in the field of human-agent interaction, e.g., in explanatory settings.
We further investigated two linguistic factors - syntactic complexity and turn-taking potential - that may contribute to this increase in gaze entropy, as they have been considered to interact with gaze behavior in previous studies. Our statistical analysis provides initial evidence that these two factors are correlated (weakly) with the local entropy value of gaze, which could potentially provide an explanation for why listeners’ gaze may adhere to the ERC principle.

heading: 9 Limitations
text: At the current stage, our study has a number of limitations that need to be taken into account when interpreting its results.
A first limitation is that the calculation of gaze local entropy based on a model that takes only gaze labels as input is not optimal, since listener gaze is affected by other modalities, speech, speaker’s prosody, etc. in the interaction. The neural sequen-
tial model (transformer) is autoregressive in that it predicts the current gaze label based only on previous gaze labels. This approach should be better than random guessing, but ignores the effects of the other modalities. This should be addressed in future work, e.g. by building a model that takes different modalities as input and calculates gaze local entropy based on them.
A second limitation concerns the the two weak correlations found in our study. They might be due to other linguistic factors that may influence the consolidation of the ERC principle for listener gaze (or nonverbal communication in general). For example, Giulianelli et al. (2022) found that the repetition of constructions (e.g., ‘yes yeah I’, ‘uhm I think’) in dialogue reduces its informational content. Similarly, Bowers et al. (2010) show that listeners pay more attention to fluent than to disfluent speech, suggesting that disfluencies influence listeners’ gaze behavior. Possible question to investigate, then, might be whether repetitions of constructions or disfluencies influence the information content conveyed by listener gaze in dialogue.
A third limitation concerns the computation of the turn transition probabilities, which is based only on textual representations of the interactions. In practice, prosody is also an important factor that affects turn-taking behavior (Ekstedt and Skantze, 2022a) and such features should be taken into account as well.
A fourth limitation concerns the processing of the video data. The extraction of eye movements using OpenFace is suboptimal. For more reliable gaze estimation, dedicated eye-tracking hardware should be used instead – although this may affect the naturalness of the interactions.

heading: 10 Ethics Statement
text: The used data in the study is for research purposes only and commercial use is not allowed. The participants (i.e., dialogue participants) were mainly university students. They gave informed consent to participate in the study and to have their data used, and were paid 10 euros per hour. The study was approved by the university’s internal ethics and data protection review boards.
Supplementary materials Code and data of analyses is available in the following data publication: https://doi.org/10.17605/OSF.IO/A4RHV
Acknowledgments This research was supported by the German Research Foundation (DFG) in the Collaborative Research Center TRR 318/1 2021 ‘Constructing Explainability’ (438445824). Yang Xu is supported by National Science Foundation of the United States (CRII-HCC: 2105192).

heading: A Appendix
text: A.1 Model Parameter Settings The following parameters were set for training the transformer model for gaze local entropy: batch size 35; input size 25; initial learning rate 0.05. The data was divided into 80% for training and 20% for testing. Both the training and testing data sets are used to compute the local entropy.
The following parameters were set for fine-tuning the German TurnGPT model: batch size 3; weight decay 0.01; dropout rate 0.1; learning rate 0.0001. A
total of 15 epochs were used to finish the fine-tuning tasks. After each epoch, one checkpoint (model) was generated, saving the weight parameters gained during training. The model with minimal loss value (1.0083) was chosen as the final model for estimating the turn transition potential probability.
A.2 DBSCAN Algorithm Parameters We set up a criterion to ensure that a point can only be considered a core point for a cluster if it is surrounded by at least ten samples.
A for-loop was used to find an optimal 𝜖 value (required by DBSCAN) in the range [0.01, 0.1] such that the ratio between the two most frequent cluster labels is just below 15%. In this way, the cluster with the most frequent label is considered to be the region where an explainee’s gaze is mostly located.
A.3 Example of Speech-Gaze Alignment Figure 9 shows an example of gaze and speech alignment.
A.4 Examples ASR-Results Figure 10 shows an example of a diarized speech transcript created automatically with the automatic speech recognition Whisper. We selected the model large-v3. Whisper’s word error rate (WER) for German is given as 5.7% the in Common Voice 15 dataset and as 4.9% in the Fleurs datasets (see https://github.com/openai/whisper/ blob/main/README.md). The transcripts were not corrected.
A.5 Additional Examples of Turn Transition Potential Estimation
Four additional examples of model predictions for turn transition potential probability are shown in Figure 11.
(a) Das Problem ist wir haben nur ein billiges Uboot (‘the problem is, we only have a cheap submarine’). (b) und wir haben dann eine Karte das ist ein Uboot und da ist eine Sauerstoffanzeige drin mit so einem kleinen Stein (‘and then we have a card it’s a submarine and there’s an oxygen gauge with a little stone in it).
(c) Wie bringt man die zum Uboot zurück (‘How do you get them back to the submarine’). (d) und sobald man dann so einen Schatz genommen hat dann muss man aufpassen (‘and once you have taken such a treasure, you have to be careful’).
Figure 11: Four additional examples of model predictions for turn transition potential probability, with translations in parentheses.

heading: Future Work
text: This should be addressed in future work, e. g. by building a model that takes different modalities as input and calculates gaze local entropy based on them. A second limitation concerns the the two weak correlations found in our study. They might be due to other linguistic factors that may influence the consolidation of the ERC principle for listener gaze (or nonverbal communication in general). For example, Giulianelli et al. (2022) found that the repetition of constructions (e. g. , ‘yes yeah I’, ‘uhm I think’) in dialogue reduces its informational content. Similarly, Bowers et al. (2010) show that listeners pay more attention to fluent than to disfluent speech, suggesting that disfluencies influence listeners’ gaze behavior. Possible question to investigate, then, might be whether repetitions of constructions or disfluencies influence the information content conveyed by listener gaze in dialogue. A third limitation concerns the computation of the turn transition probabilities, which is based only on textual representations of the interactions. In practice, prosody is also an important factor that affects turn-taking behavior (Ekstedt and Skantze, 2022a) and such features should be taken into account as well. A fourth limitation concerns the processing of the video data. The extraction of eye movements using OpenFace is suboptimal. For more reliable gaze estimation, dedicated eye-tracking hardware should be used instead – although this may affect the naturalness of the interactions