Does Listener Gaze in Face-to-Face Interaction Follow the Entropy Rate Constancy Principle: An Empirical Study
Authors: Yu Wang
Hendrik Buschmeier
Venue: EMNLP 2023 Findings (2023)
Keywords: entropy rate constancy principle
information density
dialogue
nonverbal behaviour
gaze
TL;DR: We investigate whether the entropy rate constancy principle also holds for listener's nonverbal behaviour (specifically gaze) in dialogue.

Abstract: It is generally assumed that language (written and spoken) follows the entropy rate constancy (ERC) principle, which states that the information density of a text is constant over time. Recently, this has also been found for nonverbal gestures used in monologue, but it is still unclear whether the ERC principle also applies to listeners' nonverbal signals. We focus on listeners' gaze behaviour extracted from video-recorded conversations and trained a transformer-based neural sequence model to process the gaze data of the dialogues and compute its information density. We also compute the information density of the corresponding speech using a pre-trained language model. Our results show (1) that listeners' gaze behaviour in dialogues roughly follows the ERC principle, as well as (2) a congruence between information density of speech and listeners' gaze behaviour.

Does Listener Gaze in Face-to-Face Interaction Follow the Entropy Rate Constancy Principle: An Empirical Study
Yu Wang and Hendrik Buschmeier
Digital Linguistics Lab, Department Linguistics Faculty of Linguistics and Literary Studies Bielefeld University, Bielefeld, Germany {y wang, heuschhee -ieeii--bellell.ddee
Abstract
2002)? (ii) there correlation between verbal
information and listener gaze, in terms of local entropy, dialogue (similar to recent findings on manual gesture in monologue; Xu et al., 2022)?
is generally assumed that language (written and spoken) follows the entropy rate constaney (ERC) principle, which states that the infor- mation density of text is constant over time. Recently, this has also been found for nonverbal gestures used monologue, but it still unclear whether the ERC principle also applies lis- teners' nonverbal signals. We focus on listeners' gaze behaviour extracted from video-recorded conversations and trained a transformer-base neural sequence model to process the gaze data of the dialogues and compute its informa- tion density. We also compute the information density the corresponding speech using pre- pre-trained language model. Our results show (1) that listeners' gaze behaviour in dialogues roughly follows the ERC principle, well as (2) congruence between information density of speech and listeners' gaze behaviour.
Related work
2.1 Communicative functions of gaze
In human interaction, gaze is an important and mul- tifunctional nonverbal signal with functions such as indicating attention, allocating space, and eliciting and monitoring feedback (Kendon, 1967; Duncan, 1975; Harness Goodwin and Goodwin, 1986). Ac- cording to Brône et al. (2017), these functions follow two important roles of gaze: participation role and regulation role. They also emphasise the importance of gaze for turn management. During dialogue interaction, listeners' continuous gaze to- wards the speaker signals attention and engagement (Kendon, 1967; Rossano, 20.
1 Introduction
From the technical side, the importance of gaze has been studied and integrated in recovering mean- ing in interactional dialogues (Alaçam al., 2021), facilitation grounding in interaction with embodied agents (Nakano et al., 2003) or in human-robot interaction (Skantze et al., 2014).
Human social interaction is intrinsically multimodal (Stivers and Sidnell, 2005). Face-to-face commu- nication as multimodal process includes verbal information as well as non-verbal cues, such as gaze, head movements, and speech-accompanying man- ual gestures from both interlocutors. Previous stud- ies have demonstrated that non-verbal behaviours is rich in communicative functions (Wagner et al., 2014; Holler and Levinson, 2019).
2.2 Entropy-rate constancy
Information theory (Shannon, 1948) is mathemat- ical framework that has proved useful for linguistic analysis it can explain aspects of language use. Genzel and Charniak (2002, 2003), for example, propose the 'Entropy Rate Constancy' (ERC) prin- ciple, which states that entropy is constant over the length of a written text or other use of language.
In this paper, we look at gaze behaviour of lis- teners video-recorded specifically, explanation dialogues in which a board game is explained by one interlocutor (the 'ex- plainer') to another (the 'explainee'; Türk et al., 2023). We apply information theoretical measures to the gaze behaviour of the explainees as well the corresponding utterances of both interlocutors and aim at answering the following questions: (i) How informative is listener gaze from an interactional perspective and does it follow the 'entropy rate constancy' principle (ERC; Genzel and Charniak,
The claim by Genzel and Charniak (2002) is follows: Let H(X;CC;, Li) denote the con- ditional entropy of the word X,, where L, X, Xi-ntl...., X is the local n-gram context, and C, = Xo, XI...., X;-1 the context which contains all of the words preceding the word X, The condi-

---

tional entropy of the word X; can then be decom- posed as:
4
Methods
4.1 Data collection
The dialogue interactions on which this study is based are explanations of board games (Türk al., 2023). The explainer, who is familiar with the game, explains it to the explainee, who is unfamiliar with the game (see Figure 1). There are three reasons why we focused on task-oriented dialogue, namely the game explanation scenario, in our study: (i) Based on the theory of topic shifts (Ng and Bradac, 1993; Linell, 1998), we know that some interlocutors play more active role, controlling the dialogue and introducing new topics, while the others have more passive role and follow these topic shifts. Therefore, we identify explanations as common and representative of daily dialogue (there is always a more dominant speaker the explainer - and a ative log probability s(y) = - -log,, (y), where pr is the underlying probability distribution of y. Since the true probability distribution of y is not avail- able, a language model with learned parameters is usually used to approximate the surprisal value of the corresponding linguistic unit (Smith and Levy, 2013; Goodkind and Bicknell, 2018; Wilcox et al., 2020). The linguistic unity y as large sequence (c.g.. an utterance or text), can be further divided into a sequence of smaller units: (yI...... yn), where Yn E 2 and is the set of vocabulary. The surprisal of the current linguistic unit yn is then expressed as the conditional negative log probability given its previous context: s(yx)= = log (ynly<e)). Ac- cording to the UID hypothesis, drastic variations in the per-unit information density of an utterance can place a heavy processing burden on the listener, and thus make communication more difficult. The evenly distributed information density of speech on the other hand, promotes 'rational' (Xu and Reitter, 2018) communication.
H(XIC;; L.) H(XIL)) I(X;; C,, Li)
global entropy
local cntropy mutual information
The assumption of ERC is that H(X;/C; L,) is con- stant. Given the fact that [(;;; C, L)) the mu- tual information between X; under its local context Li and its global context G - increases because the global context increases, the local entropy L4) would have to increase in order for (X Li) to remain constant.
A similar theory, the 'Uniform Information Den- sity' (UID) hypothesis, states that speakers tend distribute information uniformly throughout an utterance (Jaeger, 2010). In UID, the information of a linguistic unit y, defined by its surprisal, its neg-
more passive listener). Furthermore, we assume that the passive listeners instead use more non-verbal signals (e.g. gaze, facial expression) than verbal signals to give feedback to the dominant speakers (Buschmeier and Kopp, 2018), which potentially provides different gaze data for this study. (ii) By focusing on one type of dialogue (explanations) and additionally one topic (a specific game), we assume that the dialogue contents are quite static at the lexical and semantic level and can be considered as invariant in our experiment (and effects on the entropy rate of gaze are not by the domain). (iii) Based on this, can, in principle, even look at the behaviour of individual speakers and whether their speech behaviour follows the ERC principle.
The interactions are divided into two phases: In the first phase, the game is explained without the game material being present. In a second phase, the game is put on the table and the two participants start playing (but usually the explanation continues at least while the game is being set up). All participants speak German. The interaction can therefore be considered task-oriented dialogue. For this study we use the videos of 58 interactions from the corpus and extract the explanation part (first phase). The explanations vary in length from 2:12 to 17:36 min (mean length 7:04 min, standard deviation 3:15 min).
More recent studies have applied the ERC princi- ple and/or UID hypothesis to spoken dialogue (Xu and Reitter, 2017, 2018), task oriented dialogue (Giulianelli et al., 2021), as well as non-verbal sig- nals (specifically manual gesture) in monological speech (Xu et al., 2022).
3 Hy hypttsssss
4.2 Preparing gaze sequences
In this paper, we investigate the hypothesis that, dur- ing face-to-face interaction, listener gaze, being an important verballbal communication mechanism, also conforms to the ERC principle.
For each dialogue, we extract the explainee's gaze information using the 'Openface' framework (Bal- trusaitis et al., 2018) and create a 'gaze sequence" that is used in the neural sequential model (see

---

left of
DR
right DR
Above DR
7
8
9
DR
4
5
6
Below DR
1
2
3
Table Label decisions criterion for direction value representation of eye gaze (DR: dense region).
RE forward
DR
RE backward
LE forward
9
8
7
DR LE backward
6
5
4
3
2
Figure Scene from the data collections. The participant left (the explainer) explains the board game to the participant on the right (the explainee). The explainee's eye gaze is captured with a camera behind the explainer.
Table 2: Label decisions criterion for vector representa- tion of eye gaze (RE: right eye; LE: left eye).
tion, is generated ys ()--1)--+t)) 9 (y E [1,81] representing the set of possible eye gaze labels).
Section 4.4). Openface generates two types of gaze features (i) gaze vectors in world coordinates (three dimensions for the left and right eye each), and (ii) gaze direction values in radians averaged from both eyes (two dimensions). We integrate both represen- tations because gaze direction is easier to use, but does not contain depth information, which is po- tentially relevant as listeners regularly change their posture and head orientation during the interaction.
4.3 Automatic speech recognition for the interaction
Transcriptions of the dialogues were created auto- matically using 'Whisper' (Radford al., 2022),
which creates speech segments with a start and an end time. In order calculate word timings, we approximated word onsets by calculating the duration of each speech segment, dividing it by its length in words, and approximating word duration (assuming, for this study, that words have uniform length). Eye gaze labels are then aligned with words based video frame rate (50 fps). Figure 2 shows an example of this alignment. After pre-processing. we concatenate all of the utterance-alige aligned gaze sequences and use them training and test data for the neural sequential model.
We cluster the two gaze features using the DB- SCAN algorithm (Ester et al., 1996) to find the spatial distribution gaze and identify its 'dense region' (Tran et al., 2020), both horizontally and vertically. This dense region typically represents the target which explainees gaze at most of the time during interaction. After the dense region has been detected, scheme (inspired by Xu al., 2022) and label the gaze points inside the dense region with '5'. Depend- ing on whether the gaze direction horizontally or vertically away from the dense region, eight other number-based labels are used for the gaze points outside the dense region (see Table I). A similar approach is used for the depth-component selected from the eye gaze vectors. DBSCANN-custering is used to find the dense region where the gaze of the left eye and right eye are located in the depth dimension. The eye gaze vector inside the dense region is again given the label '5'. Based on how close the left eye vector or right eye vector is to the dense region, eight different labels are used (see Table 2).
3 3 grid-based labelling
4.4 Processing gaze sequences
Analogous to the processing the information density of linguistic units according in the UID hypothesis (see section 2.2), we consider eye movements
Das Problem ist wir haben nur ein billiges U-Boot wo
42
42
42 42 42
42
42
42
42
42
wenig Sauerstoff drinist und wir müssen uns den tellen
42
42
42 42 42 41
45
42
45
45
Wir sind also eigentlich Gegner wir spieten gegeneinander
45
44
aber wir nutzen den gleichen Sauerstoff
41
41
41
41
41
41
Given the label X E [1, 9] for the gaze direction value and the label x, E [1,9] for the gaze vector, a combined label y, that represents the gaze informa-
Figure 2: Example of the alignment of speech segments (grey) and gaze label sequences (white).

---

sequences of gaze labels and need estimate its underlying probability distribution. We approach this by training an autoregressive model, more specifically a Transformer model (Vaswani et al., 2017), which we have chosen since it has strong psychometric predictive power compared to LSTM- RNNs models(Wilcox et al., 2020). To compute the local entropy of gaze sequence, we first calculate its negative log probability:
A
10.0
7.3
Perplexty
5.0-
2.5
0.0
150
Dia Dialogue position
B
NLL(e1, e2. ,2....,rrr)
Perplextay
2.50
=-(log P(ei)
Σ log P(eilel............
2.25
where the maximum index of given eye move- ment sequence and e E [1, [1]. The local entropy F2,1, e2. e;) of the gaze sequence is then the exponential of NLL (perplcxity). The learning task is thus to predict the next gaze label e; based on the preceding sequence (ei.....e, e.11)) and minimise its negative log probability NLL.
150
Dialogue position
Figure 3: Change of the perplexity (local entropy) of eye gaze during the dialogues. shows the general trend, B shows the normalised trend without the effect from the entropy spikes.
4.5 Processing dialogue data
values), and yields globally rising trend, but also decreasing trend for the first 75 dialogue positions. The 58 explanations vary in length (see Sec- tion 4.1). so plotting them together introduces some bias into the visualisation. We have therefore di- vided the dataset into four groups (based on the total length of the explanations) and plotted the local entropy of the gaze sequences separately in Figure Besides the common rising trend in all of four sub-plots Figure 4A-D, they also share a decreasing tendency at the beginning, resulting (roughly) convex shapes. One explanation for this phenomenon may be that, at the beginning the interaction, explainees focus on the explanation of the game and direct their gaze (and attention)
To compare the local entropy of the gaze se- quences with the local entropy of the corre- sponding speech segments, we compute the lat- ter using pre- mre-trained language model (specifi- cally fomdz/german 9pt2; Schweter, 2020). For the computation we use whole dialogues, i.c.. both explainer and explainee utterances, as both con- tribute to the dialogue and explainee utterances are conditioned on what the explainer has said before and vice versa. Some of the utterances contain only backchannels (such as "yes", "uh-huh", or "okay"; Yngve, 1970), which play important role in the dialogue linguistic 'feedback' to previous ut- terances (Allwood al., 1992; Clark, 1996). In our data, backchannels are usually produced by ex- plainees and we do not exclude them as semantically irrelevant words.
mainly to the explainer, thus signalling their partic- ipation role (Brône et al., 2017). As result, there is little variation of gaze labels. However, the explanation progresses, the explainees' cognitive load may increase, known cause gaze aversive behaviour (Morency et al., 2006; Doherty-Snedder and Phelps, 2005). Another explanation for the in- creasing trend could be that explainers are likely to provide non-verbal feedback to explainers (e.g. head nodding other head movements; Heylen, 2006), signalling understanding etc., which could also lead to changes in gaze behaviour. In any case, the result a higher diversity of gaze labels and thus an increase in local entropy.
5 Results and discussion
Figure shows the combined eye gaze sequences from all 58 videos. The x-axes represent the dia- logue position of the speech segments (with each dialogue position corresponding about 7x of speech) and the y-axes represent the local entropy of the gaze sequences. Figure 3A, shows generally rising trend of the local entropy, Figure shows a replot of the data without the effect from entropy spikes shown in 3A (which we consider as outlier
We chose board game explanation so that the

---

A
B
2.5
Perplexky llocal entropy)
Perloxdy
Perplesty
2.5-
40-
3-
20-
15
30 Dialogue pasition
10
20
Dialogue position
C
D 3.00
50
100
150
Dialogue posttion
2.3-
2.75-
Perplextt
Perplextay
Figure 5: Change of perplexity(local entropy) of the speech segments during the dialogue. The shaded area is the bootstrapped 95% confidence intervals
2.00
1.75
0
20
40 60
50
15c
150
follows the ERC principle (at least to some extent; Xu et al., 2022) although the result is inconclusive in that the information density of listeners' gaze fluctuates along the interaction. A congruence be- tween dialogues content and listeners' gaze can be roughly confirmed, as the local entropy of listeners' gaze and the local entropy of speech both show an increasing tendency. The fluctuation of the entropy rate value (Figure 3A as well as Figure 4) indicates that the property of non-verbal communication can- not simply be explained by ERC principles.
Dialogue position
Dialogue posibion
Figure 4: General trends in perplexity (local entropy) explainees' eye gaze during explanations, grouped by length of dialogues (in dialogue positions): A: 4 dialogues shorter than 25; B: 29 dialogues between 25 to 49; C: dialogues between 50 74; D: 15 dialogues longer than 74.
linguistic content of the dialogue is limited. This gives us good opportunity to look at whether different individuals organise their speech in a 'ra- tional' way (following the ERC principle; Xu and Reitter, 2018). According to Figure 5, it turns out that, given the same topic (the explanation of the board game), different explainers organise the ex- planation in rational way, evidenced by the increasing trend of the local entropy. Moreover, this increasing trend of local entropy for both the speech segments across dialogues and their corresponding gaze sequences indicates a potential congruence between the information density of speech and the gaze behaviour of explainees/tisteers.
As a next step, we plan to look more closely at listeners' eye gaze and further analyse whether sudden changes in the local entropy of gaze be- haviour can be aligned with changes in the local entropy of speech, and if so, under what kinds of context. We also plan to perform further linguistic analysis on the entropy spikes, looking at dialogue acts and calculating the syntactic complexity of the corresponding dialogue positions. Possible future research includes simplifying or compressing the dialogue context based on changes in the entropy values of the listener's gaze.
6 Conclusions
Supplementary materials Code and data of anal- yses is available in the following data publication: https://do..orr/100
This study attempts find out whether (i) gaze as an important non-verbal communication signal follows the entropy rate constancy principle, and whether (ii) the information density of listeners' gaze behaviour correlates with that of the dialogue content. We recorded interaction videos, trained a transformer model, and used pre-trained language model to approximate the information density of listeners' gaze and dialogue content.
htt.:/i...rrr1..0000/////// www.60044//mmmm hig.0000/.....saaree22244222 24400077.24408073
Acknowledgements This work was supported by the German Research Foundation (DFG) in the Collaborative Research Center TRR 318/1 2021 'Constructing Explainability' (438445824).
We would also like acknowledge Olcay Türk, Stefan Lazarov, and our student assistants for the data collection, Özge Alaçam for feedback on the manuscript, and the anonymous EMNLP reviewers for their valuable comments.
We find that listeners' gaze roughly follows the ERC principle, which can be taken as further ev- idence that verbal communication generally

---

Limitations
in multimodal interaction management. Cognitive Linguistics, 28:449-45 483.
One limitation of the study is that the dialogues are not very diverse in content and activity. They are all explanations of a specific board game.
Hendrik Buschmeier and Stefan Kopp. 2018. Commu- nicative listener feedback in human-agen interaction: Artificial speakers need be attentive and adaptive. In Proceedings of the 17th International Conference on A Autonomous Agents and Multiagent Systems, pages 1213-1221, Stockholm, Sweden.
A second limitation that because we rely on video-based gaze information collection rather than dedicated eye-tracking hardware, the gaze data de- rived from Openface is sometimes sub-optimal the camera setup was not optimised for gaze and face tracking, as the camera shots were over the shoulder and thus slightly elevated, rather than at eye level- so the data may contain noise. This com- promise was necessary to meet multiple analysis objectives and also to avoid disrupting the natural- ness of the conversational setting (c.g., by using wearable eye-trackers).
Herbert H. Clark. 1996. Using Language. Cambridge University Press, Cambridge, UK.
G. Doherty-Sneddon and F. G. Phelps. 2005. Gaze aversion: response cogmitive or social difficulty? Memory & Cognition, 33:727-733.
Starkey Duncan. 1975. Interaction units during speaking turns in dyadic, face-to-face conversations. In Adam Kendon, Richard M. Harris, and Mary R. Key, editors, Organization of Behavior in Face-to-Face Interac- rion, pages 199-214. De Gruyter Mouton, Berlin, Germany.
A third limitation concerns the alignment of utterances with gaze labels. Since German words can be very long (e.g. "Tiefseeabenteuerr, the name of the board game), the simplified assumption that all words have the same duration and align with the same number of gaze labels will cause some noise.
Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xi- aowei Xu. 1996. A density-based algorithm for discov- ering clusters in large spatial databases with noise. In Proceedings of the Second International Conference Knowledge Discovery and Data Mining, pages 226-231, Portland, Oregon.
Ethics statement
Dmitriy Genzel and Eugene Charniak. 2002. Entropy rate constancy in text. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 199-206, Philadelphia, PA, USA.
The data collected for this study is for research purposes only and commercial use is allowed. The participants recruited are mainly university students. Participants gave informed consent for their participation in the study and the use of their data, and were paid 10 euros per hour. The study was approved by the university's 's internal ethics and data protection review boards.
Dmitriy Genzel and Eugene Charniak. 2003. Variation of entropy and parse trees sentences as a func- tion of the sentence number. In Proceedings of the 2003 Conference Empirical Methods in Natural Language Processing, pages 65-72.
Mario Giulianelli, Arabella Sinclair, and Raquel Fer- nández. 2021. Is information density uniform in task-oriented dialogues? In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 8271-8283, Online and Punta Cana, Dominican Republic.
References
Özge Alaçam, Ganeshan Malhotra, Eugen Ruppert, and Chris Biemann. 2021. Gaze-baser multimodal mean- ing recovery for noisy/complex environments. In Proceedings of the 2021 International Conference on Multimodal Interaction, pages 673-681, Montréal, Canada.
Adam Goodkind and Klinton Bicknell. 2018. Predictive power of word surprisal for reading times is linear function of language model quality. In Proceedings of the 8th Workshop on Cognitive Modeling and Computational Linguistics (CMCL 2018), pages 10- 18, Salt Lake City, UT, USA.
Jens Allwood, Joakim Nivre, and Elisabeth Ahlsén 1992. On the semanties and pragmatics of linguistic feedback. Journal of Semantics, 9:1-26.
Marjorie Harness Goodwin and Charles Goodwin. 1986. Gesture and coparticipation in the activity of search- ing for a word. Semiotica, 62:51-76.
Tadas Baltrusaitis, Amir Zadeh, Yao Chong Lim, and Louis-Philippe Morency, 2018. Openface 2.0: Facial behavior analysis toolkit. In Proceedings of the 13th IEEF International Conference on Automatic Face Gesture Recognition, pages 59 66, Xi' an. China.
Dirk Heylen. 2006. Head gestures, gaze and the principle of conversational structure. International Journal of Humanoid Robotics, 3:241-267.
Judith Holler and Stephen C. Levinson. 2019. Multi- modal language processing human communication. Trends in Cognitive Sciences, 23:639-652.
Geert Brône, Bert Oben, Annelies Jehoul, Jelena Vranjes, and Kurt Feyaerts. 2017. Eye gaze and viewpoint

---

T. Florian Jaeger. 2010. Redundancy and reduction: Speakers manage syntactic information density. Cog- nitive Psychology, 61 61:23-62.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Sys- tems, page 6000-6010, Long Beach, CA, USA.
Adam Kendon. 1967. Some functions of gaze-direction in social interaction. Acta Psychologica, 26:22-63.
Per Linell. 1998. Approaching Dialogue: Talk, Interac- sion and Contexts in Dialogicul Perspectives. John Benjamins, Amsterdam, The Netherlands.
Petra Wagner, Zofia Malisz, and Stefan Kopp. 2014. Ges- ture and speech interaction: An overview. Speech Communication, 57:209-232.
Louis-Philippe Morency, C. Mario Christoudias, and Trevor Darrell. 2006. Recognizing gaze aversion gestures in embodied conversational discourse. In Proceedings of the 8th International Conference on Multimodal Interfaces, page 187-294, Banff, Canada.
Ethan G. Wilcox, Jon Gauthier, Jennifer Hu, Peng Qian, and Roger P Levy. 2020. On the predictive power of neural language models for human real-time com- prehension behavior. Proceedings of the 42nd Annual Meering of the Cognitive Science Society, pages 1707-1713, Virtual.
Yukiko I. Nakano, Gabe Reinstein, Tom Stocky, and Justine Cassell. 2003, Towards model of face-to-face grounding. In Proceedings of the st Annual Meeting of the Association for Computational Linguistics, pages 553-561, Sapporo, Japan.
Yang Xu, Yang Cheng, and Riya Bhatia. 2022. Gestures are used rationally: Information theoretic evidence from neural sequential models. In Proceedings of the 29th International Conference Computational Linguistics, pages 134-140, Gyeongju, Republic of Korea.
Sik Hung Ng and James J. Bradac. 1993. Power in Lan- guage: Verbal Communication and Social Influence. Sage, Los Angeles, CA, USA.
Yang Xu and David Reitter. 2017. Spectral analysis of information density in dialogue predicts collabora- tive task performance. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 623-633 Vancouver, Canada.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brock- man, Christine Mcl.eavey, and lya Sutskever. 2022. Robust speech recognition via large-scale weak su- pervision. Audio and Speech Processing (eess.AS), arXiv:2212.04356
Yang Xu and David Reitter. 2018. Information den- sity converges in dialogue: Towards information- theoretic model. Cognition, 170: 147-163.
Federico Rossano. 2012. Gaze Behavior in Face-to- Face Interaction. Ph.D. thesis, Radboud University, Nijmegen, The Netherlands.
Victor Yngve. 1970. On getting a word edgewise. In Papers from the Sixth Regional Meeting of the Chicago Linguistic Society, pages 567-577 Chicago, IL. USA. Chicago Linguistic Society.
Stefan Schweter. 2020. (dbmdz/german-gpp2)
German GPT-2 model
Claude E. Shannon. 1948. A mathematical theory of communication. The Bell Systems Technical Journal, 42:379-423.
A Fragment of a game explanation
Gabriel Skantze, Anna Hjalmarsson, and Catharine Oer- tel. 2014. flunnttaking, feedback and joint attention in situated human-robol interaction. Speech Commu- nication, 65:50-66.
Figure shows an example fragment of a game explanation. The timestamped speech seg- ments were automatically recognised from the audio of the interaction videos using 'Whis- per' (Radford et al., 2022). Whisper's word er- ror rate (WER) for German is given as 4.5% (see https: thub. homm//penai/whisppe blob/main/README.co md). The transcripts were not corrected.
Nathaniel J. Smith and Roger Levy. 2013. The effect of word predictability reading time is logarithmic. Cognition, 128:302-319 -199.
Tanya Stivers and Jack Sidnell. 2005. Introduction: Multimodal interaction. Semiotica, 2005(156):1
Minh Tran, Taylan Sen, Kurtis Haut, Mohammad Rafayet Ali, and Ehsan Hoque. 2020. Are you really looking at me? a feature-extraction framework for estimating interpersonal eye gaze from conventional video. IEEE Transactions on Affective Computing, 13:912-925.
B Hyperparameters selection for DBSCAN clustering
For the DBSCAN algorithm, set up a criterion that only if point is surrounded by at least ten samples. can be considered core point for a cluster. for-loop is used find an optimal epsilon value required by the DBSCAN algorithm. The epsilon value is in the range [0.01, 0.1]. We wanted
Olcay Türk, Petra Wagner, Hendrik Buschmeier, Angela Grimminger. Yu Wang, and Stefan Lazarov. 2023, Mundex: A multimodal corpus for the study of the understanding of explanations. In Proceedings of the Ist International Multimodal Communication Symposium, pages 63-64, Barcelona, Spain.

---

00:00.000 00:04.820
Also grundsätzlich das Spiel heißt Tiefsoeabenteuer geht quasi darum, du bist ein Taucher So basically the game is called Deep Sea Adventure and it's basically about you being diver in einem U-Boot
00:04.820 00:05.820
submarine.
00:05.820 00:09.280
musst mógllchht lange raustauchen, um die Schätze anzusammeln.
You have to dive out as long as possible to collect the treasures.
00:09.280 00:12.800
Wer Ende mehr Schátze hat, gewonnen, das ist der Grundaulbau. Whoever has more treasures at the end wins, that's basic structure.
00:12.800 00:18.400
Du musst dir vorstellon, du hast hier das U-Bool, da starten wir auch drin und da hast You have imagine that you have the submarine here, start in there and there you have
00:18.400 00:25.480
denn quasi 25 Felder, bei bis 0, das ist im Ende unsere Sauerstoffanzeig und die you then basically 25 fields, at 25 to 0, that's ultimately our oxygen display and these nicht jeder einzeln, wir tellen
00:25.480 00:28.240
feilen wir das wird gleich noch wichtig, we'll share, that'll be important in moment, not everyone has them individually. we'll share
00:28.240 + 00:31.720 zusammen einen Sauerstofftank und dann haben wir gleich ganz viele kleine Plättchen, da together Oxygen tank and then have lot of small plates right there
00:31.720 00:37.680
sind Punkte drauf von bis 4, die geringsten die höchsten werden dann eben von are points on it from to 4, the lowest 4, the highest are then just from to
00:37.680 00:43.240
einem U-Boof entlang ausgelegt, dass quasi ein Weg entsteht und dann haben wir out along submarine so that path created then we have it
00:43.240
Figure 6: Example ASR transcript of fragment of an explanation dialogue (with English translations).
find epsilon value that ensures that the ratio between the second most frequent cluster label and the most frequent cluster label is just below 15%, so that the cluster with the most frequent label considered to be the region where an explainec's gaze is mostly located.
C Hyperparameter selection for the training procedure
For the neural sequence model (Transformer) the batch size is 35, the input size set to 25, the hidden layer size is Data is divided into 80% for training and 20% for testing. The initial learning rate is 0.05. Both training and testing data sets are used to compute the local entropy.
To compute the local entropy for the speech segments, we use the pre-trained language model peritaiggemannngee
(Schweter, 2020), the input size is set to 25, which is consistent with the gaze sequence input size for the neural sequence model.


Reviews:
[{"review_id": "oT7LidUspE", "reviewer": "Reviewer_H3tb", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. In future work, we indeed plan to use more in-depth linguistic analyses in order to investigate why there are sudden increase/decrease of perplexity (see our conclusion). We plan to analyse it from two aspects: syntactic complexity and dialogue acts.\n\nConcerning the \"reproducibility issue\": We will release the code of our analyses alongside the paper, enabling reproduction on other data. We cannot release the multimodal data due to privacy reasons (GDPR) and limitations in the consent of the participants of our study.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Alignment study between non-verbal (eye gaze) and verbal (speech)", "reasons_to_reject": "Dialogue position and gaze sequence alignment require more in-depth analysis\n\nSpecifically, Figure 4 demonstrates dialogue_positions and the change of perplexity of listeners' eye gaze. \nA better analysis would be the linguistic features of dialogues and the change of perplexity of listeners' eye gaze.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper studies whether the adherence of eye-gaze features to the ‘entropy rate constancy’ principle\n(ERC). Toward that end, the authors conducted their study in the context of a board game explanation. The listeners are unfamiliar with the game and learn the game rules as an expert describes the game.\nThe empirical results demonstrate that listeners' eye gaze features adhere to the ERC principle of the experts' dialogue features.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZEcTjgusDR", "reviewer": "Reviewer_Ck9q", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review of our paper. We are still cautious about our current results and will continue our ongoing work to reinforce our hypothesis on ERC in multimodal interactive communication (i.e., face-to-face dialogue) more generally.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-written and present a potentially interesting novel pipeline to link eye-gaze data during natural conversation with occurring speech.\n\nResults are somewhat inconclusive, but the methods are themselves compelling and I think this is a good fit for a short paper of this type.", "reasons_to_reject": "No reasons to reject.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper builds a computational pipeline to analyze the entropy of eye-gaze patterns during natural interlocutor dialogue and correlate this measure with entropy estimated from the time-aligned speech. This is a novel methods and while the results' consistency (or potential inconsistency) with entropy rate constancy is slightly murky, I think the methodological and paradigm contributes make this an interesting short paper for EMNLP", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pLDZAXUVpP", "reviewer": "Reviewer_V3pQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for reading our paper and giving us your valuable feedback. We consider your three concerns as follows:\n\n1. (limited generalisability of the experiment results cause by one type of dialogue): There are three reasons, why we focussed only on task-oriented dialogue in this study. (i) Based on the theory of topic shifts (e.g., Ng and Bradac, 1993; Linell, 1998) we know that some interlocutors play a more active role controlling the dialogue and introducing new topics, while the others has a more passive role and follows those topic shifts. Based on this we identified explanation as common and representative of daily dialogue (there is always a more dominant speaker – the explainer – and a more passive listener). We thus think that our chosen domain is quite representative for dialogues in general. (ii) Focussing on one type of dialogue (explanations) and additionally one topic (a specific game) we assume that the content is quite static in the dataset (from the lexical and semantic levels), and can be considered invariant in our experiment. With the dialogue contents as the invariant, the results on entropy rate of the gaze will be more convincing. (iii) Based on this, we can even look at the behaviour of individual speakers and whether their speech behaviour follows the ERC theory (see figure 5). More generally we also see that it would be interesting to look at a more diverse set of dialogues (e.g., more casual conversations, different board games, …) and would like to see our result being replicated with other data.\n\n2. (outlier exclusion): We would like to clarify that our criteria for excluding outliers are not arbitrary but based on empirical observation. Based on figure 3A, we found that in some limited dialogue positions, the perplexity jumps over 3.75 and falls back, which means that perplexity is usually below 3.75. Based on this point, we set up a threshold in our visualisation to exclude the effect of these spikes, which helps to generate the figure 3B where an increasing trend is more obvious. Thanks for pointing this out, we would explain it in more detail in a revised version.\nThe perplexity-spike is a question we would like to answer, with more in-depth linguistic analysis,  in future work.\n\n3. (introduction fails to adequately explain their practical significance): Our contribution is a theoretical one answering the two questions posed in the introduction (is there a relation between verbal information and listeners’ gaze; how informative is listener gaze). Our experiment shows a congruence between explanation (verbal information) and listener gaze, and that gaze behaviour roughly follows the ERC principle. On a more practical note, we think that these theoretical insights could serve as a basis for NLP applications incorporating gaze behaviour (mentioned briefly in the conclusion). We will make this more explicit.\n\n## References\n\n* Linell (1998). Approaching dialogue: Talk, interaction and contexts in dialogical perspectives. John Benjamins.\n* Ng and Bradac (1993). Power in language: Verbal communication and social influence. Sage.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- While linguistic features on ERC have been studied before, this research is the first attempt to investigate gaze behavior.\n- The use of OpenFace for gaze information extraction may not be perfect, but the methodology is carefully explained, ensuring reproducibility.", "reasons_to_reject": "- The current study only uses one type of dialogue, limiting the generalizability of the experimental results.\n- The criteria for excluding outliers in the analysis appear arbitrary.\n- Although the experimental results are intriguing, the Introduction fails to adequately explain their practical significance.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This short paper investigates whether the entropy of the listener's gaze in face-to-face conversations follows the law of entropy rate constancy (ERC).\nThe current study focuses on scenes where participants describe board games without using game equipment.\nGaze information is extracted using OpenFace, and then the analysis confirms that Perplexity increases with the length of the conversation.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper explores whether the listener’s non-verbal signal – mainly gaze, follows entropy rate constancy in dyadic conversation. Moreover, the paper also investigates if there is any alignment between the information density of speech content and the listener’s gaze behavior.
The study is inspired by Xu et al.'s 2022 study, which found that ERC applies to non-verbal gestures in monologues, this paper extends to test the principle to the listener's perspective in conversations.

While the reviews are mixed, most reviewers concur that the results presented are inconclusive. The paper could benefit from more comprehensive explanations and deeper analysis, especially concerning dialogue positions and gaze sequences. Despite being a short paper, the paper should aim for self-contained explanations, detailing the models used rather than solely referencing other works. Additionally, the paper lacks adequate justification for certain design and experimental choices, such as the removal of outliers, and should clarify the reliability and generalizability of the results (e.g., in Fig 4). The authors have acknowledged some of these issues and committed to addressing them in future iterations of the paper.

Given that the paper's research findings and methodologies hold some interest to the community, we recommend that the authors refine their work for the next version, taking the reviewers' feedback into account.
Decision: Accept-Findings