<figure><img src="image_1.png" /></figure>

HAL
open science

# Data-Driven Control of Eye and Head movements for Triadic Human-Robot Interactions

Léa Haefflinger, Frédéric Elisei, Gérard Bailly

► To cite this version:

Léa Haefflinger, Frédéric Elisei, Gérard Bailly. Data-Driven Control of Eye and Head movements for Triadic Human-Robot Interactions. International Journal of Social Robotics, 2025, 10.1007/s12369-025-01245-2. hal-05046833

HAL Id: hal-05046833

https://hal.science/hal-05046833v1

Submitted on 25 Apr 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Data-Driven Control of Eye and Head movements for Triadic Human-Robot Interactions

Léa Haefflinger¹,²*, Frédéric Elisei¹† and Gérard Bailly¹†

¹*GIPSA-lab, Univ.Grenoble Alpes, CNRS, Grenoble INP, 11, rue des Mathématiques, St Martin d'Hères, 38400, France.

²Innolab, Atos, 3 Rue de Provence, Échirolles, 38130, France.

*Corresponding author(s). E-mail(s): lea.haefflinger@gipsa-lab.fr;

Contributing authors: frederic.elisei@gipsa-lab.fr; gerard.bailly@gipsa-lab.fr;

†These authors contributed equally to this work.

## Abstract

The importance of gaze in human-robot interaction (HRI) is well documented, particularly for its contribution in regulating turn-taking and managing roles in conversations. However, few gaze models have yet been proposed for multi-party interactions, i.e. several people facing the robot. In this paper, we propose to build and evaluate a gaze control for a robot animating a collaborative game engaging two humans. This model has the particularity of predicting for the robot, both gaze targets and head movements using machine learning algorithms. In this way, we hope to achieve subtle, natural control of eyes-head coordination, which is usually neglected. This model was trained with an original corpus recorded using immersive teleoperation of a robot, providing precise data of a human-like behavior in HRI. Once trained, this model was subjectively evaluated with two online third-person evaluations. The first evaluation shows that the head movements predicted by our model are preferred to those of a heuristic model based purely on the distance between two fixations, and therefore without knowledge of the interaction context. The second evidences that, by biasing the model inputs, the predictions enabled our robot to communicate to whom it speaks (one or both players) simply by controlling its gaze. These results are promising and suggest that this type of model is entirely viable for the generation of gaze behavior in interaction, and that it offers efficient levers for its control.

**Keywords:** Human-robot interaction, Gaze, Head movement, Multi-party

## 1 Introduction

Human communication is not just verbal: a significant amount of information is conveyed through non-verbal cues, such as gaze, head, facial movements and gestures (Argyle, 1972; Birdwhistell, 1955). All these cues have a wide range of functions, including expressing emotions and providing feedback, as well as managing speaking turns and

facilitating understanding of the intent (Kendon, 1967; Krauss et al, 1996).

To integrate social robots into our daily lives and enable an effective and natural interaction with humans, robots need both abilities: to decode and interpret as well as to plan and generate these non-verbal cues. Several studies have already shown the positive impact of endowing a robot with such behaviors (Admoni et al, 2016; Mutlu
et al, 2006; Boucher et al, 2012). Based on this,
many control models for these signals have already
been proposed, in particular for gaze, which is one
of the most important cues (Admoni and Scas-
sellati, 2017; Tashiro et al, 2023). But, most of
the proposed control models are limited to dyadic
face-to-face interactions (i.e one person facing the
robot), where only two roles are possible: speaker
vs. addressee. In multi-party interactions (i.e sev-
eral people facing the robot), multiple addresses
and/or bystanders are likely. In this case, the
use of non-verbal cues such as gaze is even more
important for managing the conversational regime
and roles in the conversation, and are partly
monitored by gaze (Goffman, 1979; Mutlu et al,
2009).

Since few models are designed for multi-party
interactions, this paper proposes a control for
managing the gaze of a robot in three-party inter-
actions. The proposed control is a data-driven
end-to-end model that incrementally – i.e. causal
model with no information on the future – maps
robot’s intentions and partners’ observed behav-
iors to the robot’s eye targets and head movements
via machine learning. We believe that this type of
model could identify and capture more complex
patterns in the data – and reproduce them with
more subtle behavior – than rule-based models.
To train our model, we use original mocap data
from a collaborative game between two human
players and a robot that is immersively teleop-
erated by a human pilot. This setup provides
precise data about the human-like behavior of
the pilot, and takes into account the differences
that inherently exist between human-human vs.
human-robot interactions (Johansson et al, 2013;
Shriberg et al, 2012; Yu et al, 2012), in particular
because of users’ expectations and asymmetrical
communication skills. The model aims to gen-
erate the gaze behavior of the robot animator.
It should allow the robot to animate the game
autonomously in the future. To do this, the control
model takes as input both endogenous (is it speak-
ing?, about what?, to whom?) and exogenous
(which human partner is speaking?, to whom? and
where are they looking at?) multimodal features.

In addition, the proposed control model gen-
erates both a gaze target and the head trajecto-
ries to be performed by the robot. Indeed gaze

is a combination of chest, head and eye move-
ments (Land, 2004), and each of these three
segments of attention conveys its own communica-
tive information. In particular, head can direct
attention or give feedback on its own (Otsuka and
Tsumori, 2020; Hietanen, 1999; Liu et al, 2012). In
a previous study, we highlighted the difference in
a speaker's head pose while speaking to either two
people or one of them (Haefflinger et al, 2023b).
Consequently, we believe that the robot's head
movements should be generated from contextual
elements of the interaction (who is speaking, to
whom, about what, etc.) and not be purely depen-
dent on the gaze target to be reached. However,
current control models seem to pay insufficient
attention to eyes-head coordination when manag-
ing gaze.

Therefore, this paper pursue two objectives:
(a) testing the viability of this data-driven control
model, which is currently not widely used for the
generation of gaze in multi-party interaction; (b)
validating that the generation of head movements
that takes into account the interaction context is
better perceived than that of a control policy using
no contextual information.

To question these two hypothesis, this paper
presents the design of the proposed end2end con-
trol model and its subjective evaluation through
two online experiments. Both were conducted with
a (virtual) Furhat (Al Moubayed et al, 2012)
robot. The first evaluation (see Section 5) is a com-
parison of the head movements generated by our
model with those generated by a heuristic model
that ignores the interaction context. The second
evaluation (see Section 6) tests the ability of our
gaze control to communicate to whom the robot
is addressing, i.e. one or both players.

2 Related work

2.1 Gaze in multi-party interaction

In Human-Human Interactions (HHI), gaze is a
key signal, in particular for its impact on turn-
taking and role management in multi-party inter-
actions (Duncan, 1972; Goodwin, 1981). The gaze
can be used to indicate to whom you are address-
ing when you speak, and whether you want to hold
or yield the floor (Goodwin, 1981; Jokinen et al,
2010). Among the gaze patterns already described,
a speaker will mostly look at the person he/she
is addressing (Vertegaal et al, 2001), shift his/her gaze away at the start of a turn to announce taking the floor, and look at the addressee when yielding the floor (Kendon, 1967). Gaze is therefore an essential parameter in the models used to detect who is speaking to whom, or whether the speaker is going to hold or release the floor (Vertegaal et al, 2001; Jokinen et al, 2013) and, more generally, to detect the different *conversational regimes* during interaction (Otsuka et al, 2005, 2007). It is also widely used to detect who will be the next speaker (Jokinen et al, 2010; Ishii et al, 2016).

The influence of the gaze on turn-taking and participation regulation in the conversation have also been highlighted in Human-Robot Interaction (HRI) (Admoni and Scassellati, 2017). Mutlu et al (Mutlu et al, 2009) successfully used the gaze of their robot to communicate to people their role in a multi-party conversation (i.e. addressee vs. side-participant) and to indicate when the robot releases the floor. Similarly, some studies have shown that, by using relevant gaze cues, the robot can regulate speaking turns, influencing whether the speaker keeps or yields the floor, and having an impact on who will be the next speaker (Bohus and Horvitz, 2010; Skantze et al, 2015).

Beyond this ability to shape multi-party interactions, appropriate gaze behavior can also affect people's perception of the robot. Using a responsive gaze control model in a teamwork task, Correia et al (Correia et al, 2023) notably showed that the robot was perceived as a better teammate. The robot can also be perceived as more or less polite (Moujahid et al, 2022), thoughtful (Andrist et al, 2014), depending on its gaze. Two models designed with data from two humans with different personalities result in a distinct personality for the robot (Shintani et al, 2022). Overall, appropriate control of the robot's gaze results in more engaged partners in the interaction (Sidner et al, 2004; Kuno et al, 2007).

## 2.2 Existing gaze controls

Given the importance of endowing social robots with appropriate gaze behavior in HRI, a number of controls have already been developed and evaluated. While most of the proposed models are designed for dyadic interactions, this Section focuses on those for multi-party interactions as the one presented in this paper.

### 2.2.1 Approaches

Admoni & Scassellati (Admoni and Scassellati, 2017) identify three common approaches for modelling social gaze behavior:

*Biologically inspired models* try to reproduce the human neurobiological and cognitive mechanisms, like the saliency models proposed by Itti et al (Itti and Koch, 2001; Itti et al, 2006) for scene analysis. Several proposals (Sun and Fisher, 2003; Picot et al, 2007) have enriched this model with features such as hierarchical selectivity of attention shifts. *Heuristic or rule-based models* are based on a set of rules extracted from the analysis of human behavior. For multi-party conversations, Zaraki et al (Zaraki et al, 2014) proposed a gaze control for their robot that derives an attention score for each interacting person based on their activity (gaze, gestures, proximity), defining the person with the highest score as the gaze target. In a three-party game, Mishra et al (Mishra and Skantze, 2022) also proposed a model computing an attention score for both players, but tried to predict this score over a time window of 2seconds to adapt their robot's gaze, and in particular eyes-head coordination. Other researchers have proposed to compare simpler models in a teamwork situation, where the robot looks at its partner if that partner looks at it (mutual gaze) and looks at the person its partner is looking at (joint attention) (Correia et al, 2023). However, this approach can only cover a few of the rules regulating human interaction, making it limited and difficult to adapt when going beyond a defined context.

*Data-driven control models* attempt to reproduce human behavior, observed in a well-defined scenario. Several researchers have proposed such control models: Shintani et al (Shintani et al, 2021) mimics human gaze aversion and distribution according to their role in the interaction (speaker, addressee, sub-listener) while Nakano et al (Nakano et al, 2015) exploits information about which participant is more or less dominant in the current interaction. Some work already proposed the use of machine learning tools to model gaze, such as Stefanov et al (Stefanov et al, 2019), who use Long Short Term Memory (LSTM) models, taking into account the robot's speaker or listener state, but perform a purely objective evaluation. Support Vector Machines (SVMs) were also tested for the prediction of a robot's gaze in three-party
interaction, which was perceived to be as natural as when the robot reproduced the labelled gaze from the training data (Huang et al, 2020). Except these few studies, the use of machine learning tools for the generation of a robot's gaze in multi-party interaction remains scarce, or limited to scene observation models where the robot is not directly involved (Domingo et al, 2022). In previous work, we already proposed a gaze control using LSTMs and showed that the gaze of a robot managed by our control was perceived as relevant as a control reproducing human behavior (Haefflinger et al, 2023a).

### 2.2.2 Eyes-Head coordination

Coordination between eyes and head has been extensively studied in humans. As mentioned above, gaze is the result of these components, and each has its own physical characteristics: the movement of the eyes is faster than that of the head, and eyes generally precede head movement (Freedman and Sparks, 2000), while the head only moves if the distance to be performed exceeds a certain threshold (Fuller, 1992a). This coordination can differ between two people, with variable amplitude of the head movements (Sidenmark and Gellersen, 2020). These findings led to several models for scene observation tasks or gaze shift (Itti and Koch, 2001; Itti et al, 2006).

For many gaze models in HRI, eyes-head coordination is either not taken into account, as the robot cannot move its eyes independently of its head (Huang et al, 2020; Sidner et al, 2004), or does not use the context of interaction, and head and eye angles are only derived from the trajectory between two fixations (Zaraki et al, 2014; Shin-tani et al, 2021). However, both eyes and head have their own communicative functions (Otsuka and Tsumori, 2020). In particular, the head is a strong vector for signaling the focus of our attention (Hietanen, 1999; Stiefelhagen and Zhu, 2002). Recent studies using this coordination for communicative purposes have shown convincing results: Gillet et al (Gillet et al, 2021) showed that by adjusting the position of their robot's head, it was able to balance the speaking times between a pair of players; other researchers have implemented and positively evaluated a gaze control that takes eyes-head coordination into account by

increasing the angle achieved by the head in correlation with the estimation of the duration of gaze fixation (Mishra and Skantze, 2022).

In a previous study, Haefflinger et al (Haefflinger et al, 2023b) found that a robot using both head and eye movements in multi-party interaction was perceived as more natural than one using only one of these two components, and that the preference for more or less head or eye movements was dependent on the context of the interaction. These results are the initial motivation of training and testing a data-driven model generating both eyes and head movements.

## 3 Data collection

For the construction of our data-driven model, we used the annotated, open-access RoboTrioV2 dataset (Elisei et al, 2023). This corpus is a collection of triadic interactions between a robot and 2 humans conversing in French.

### 3.1 Corpus description

The originality of this corpus consists in the use of immersive teleoperation to control the robot and endow it with smooth, human-like behavior. The robot used in this corpus is the iCub robot, which has been modified to improve its communicative functions by adding an articulated jaw (Parmiggiani et al, 2014) and redesigning its eyes (Foerster et al, 2015). Thanks to the degrees of freedom of its head (3) and eyes (3), this robot is able to replicate the movements of a human pilot wearing a virtual reality headset equipped with a binocular eyetracker. Moreover it can transmit pilot's voice through the speaker in its mouth that is articulated through motion capture. In addition, thanks to the 2 cameras positioned in its eyes, and the two microphones in its ears, it enables the pilot to take possession of its body and interact through it, while preserving the feedback of face-to-face interaction. The immersive teleoperation platform used for this corpus is described in (Cambuzat et al, 2018). Our immersive platform captures precise data of head and eye movements of the human pilot, with an acquisition frequency of 60 Hz, delivering valuable data for training a gaze management model.

In RoboTrioV2, the male pilot – and thus the teleoperated robot – is involved in a collaborative
game with several human dyads, where he acts as
the animator. The aim of the game for both play-
ers is to find out which words are the most quoted
from a pool of people for a given theme. As an
example, for the seed theme “plant” associated
answers are: flower, green, pot, tree, garden, leaf,
etc. As facilitator, the robot monitors the game
flow by giving the theme and scores, and incit-
ing players to collaborate and propose answers.
The robot’s pilot has access to a tablet (located
in front of the robot) to consult all these informa-
tion. Figure 1 presents how the robot and players
are seated while interacting.

<figure><img src="image_3.png" /><figcaption>**Fig. 1** Scheme illustrating the corpus recording setup (Haeflinger et al, 2023b)</figcaption></figure>

We used 11 interaction sequences from this
corpus, each divided into 9 themes requiring 5
responses, for a total of 4 hours of recorded inter-
actions. Of these 11 sequences, 6 correspond to
pairs of male players, and 5 to pairs of female
players, with each person participating only once.
The robot pilot is common to all interactions, to
ensure consistency in the behavior to be modeled.
In addition to capturing the pilot's behavior and
egocentric audiovisual perception (binaural audio
and binocular video), the audio of the interaction
was recorded in stereo, and the players' faces were
filmed using two fixed HD cameras facing each of
them.

## 3.2 Annotations of multimodal behavior

In addition to recording head and eye trajecto-
ries, these multimodal data have been annotated
to analyze pilot and player behavior, and to train
the model proposed in this work. Annotations are
split into 3 categories: verbal, non-verbal and a
combination of the two.

**Verbal annotations** have been manually car-
ried out using Elan software (Brugman et al,
2004), which facilitates the combination of multi-
ple audio and video channels, and the structuring
of speech annotations:

*   **Pilot's verbal activity:** annotation of all utterances, including the start and end of each utterance with its verbal content. Each utterance is further classified into 23 speech intentions, set to best describe the interactions in this corpus. These include: giving the theme, the score, an explanation, a feedback, requesting a proposal, asking for validating a word, etc.

*   **Players’ verbal activity:** start and end of each utterance with its verbal content for each of the 2 players, respectively called LeftUser for the leftmost player, and RightUser for the rightmost player from the robot’s perspective. No intentions are associated for now.

**Non-Verbal annotations** have been com-
puted automatically using the pilot's eye and head
data and by tracking the players' videos from fixed
HD cameras.

*   **Pilot's gaze:** In order to train our gaze management model, the pilot's gaze was classified into 5 categories for each frame (60 Hz): "Saccade" (part of an ocular saccade), "LeftUser", "RightUser", "Tablet" (for current game information), "Elsewhere" (if not matching targets). To classify gaze, the pilot's focal point was projected onto a plane 1.2 m away (distance between the robot and players during interaction) at each instant by using the recorded head and eyes angles. For each interaction sequence, all focal points were then classified using Gaussian Models Mixture (GMM) with 4 clusters. Occular saccades were triggered when the focal point distance between two frames was higher than 40 mm.
*   **Players' gaze:** To classify the players' gaze, the OpenFace toolkit (Amos et al, 2016) was used to extract the relevant facial landmarks (head and eye positions) from the fixed HD cameras. The eye and head orientations were then used to calculate the gaze focal points. As with the pilot's gaze, after the detection of ocular saccades ("Saccade" class), the set of focal points for each sequence was classified using GMM, with 3 clusters: "Robot", "OtherUser" (left or right depending on the player analyzed), and "Elsewhere".

**Multi-modal annotations** combining information from verbal and non-verbal annotations are also added.

*   **Pilot's addressee:** For each pilot utterance, we annotate who the pilot was addressing with 4 possible classes: "Both" when addressing both players, "LeftUser" (resp. "RightUser") when addressing the player on the left (resp. right), and "Unknown" when it was not possible to determine the addressee. For this classification, both the verbal pronouns or first names used by the pilot and the target of his gaze were used. In French, the pronoun "Tu" (you) is used to address a single person, while the pronoun "Vous" (you) is used to address several people. If a sentence contains neither of these pronouns, it is automatically classified as "Unknown". Conversely, if it contains the pronoun "Vous", the sentence is classified as "Both". If the sentence contains a "Tu", we compare the percentages of the pilot's gaze to the two players to determine the addressee as the most-gazed-at partner. The same is done if a player's first name is mentioned, as he/she may be addressed or just referred to.

## 3.3 Data analysis

This dataset is used to train the proposed gaze model. Preliminary analysis is necessary to understand the data contents, especially the distribution of the pilot's gaze, which the model aims to replicate.

Figure 2, represents the distribution of the pilot and players gaze across the 11 annotated sequences. We can observe that the pilot looks at both players equally and mostly. However, the

"Elsewhere" category is poorly represented, presumably due to the presence of the tablet, which allows him to disengage from the conversation whenever he wants. Since the training data is unbalanced, this must be taken into account when evaluating the model.

Regarding the distributions of the two players' gazes, we can see that the robot is frequently looked at, as well as their playing partner. This suggests a real collaboration between the two players, and that the robot has not been excluded from the interaction. It's not surprising to see that the "Elsewhere" class is highly represented for both players, as the task involves a lot of individual thinking. Finally, even if we can observe some differences in the distribution of their gaze, the sets of left players vs. right players seem to behave quite similarly.

<figure><img src="image_4.png" /><figcaption>**Fig. 2** Gaze distributions of the pilot and players for the 11 annotated sequences.</figcaption></figure>

<caption>_figures/distribution_gaze-eps-converted-to.pdf</caption>

Table 1 presents the distribution of the pilot's gaze based on his activity during the interaction. When speaking, the pilot looks at the tablet most of the time, probably because of the need to know the information of the game in progress. When one of the players is speaking, he or she is mostly looked at, about 51% of the time, although the other player is not ignored at all, about 30% of the time. This is definitely due to the pilot's role as facilitator, who must ensure the collaboration and participation of both players. We hope that the proposed model will be able to understand
and reproduce these gaze distributions, generat-
ing more complex behavior than just looking at
the speaker.

<table><caption>**Table 1** Distribution of the pilot's gaze depending on his activity as speaker or as listener.</caption>
  <thead>
    <tr>
      <th rowspan="2">Pilot's target</th>
      <th rowspan="2">Self</th>
      <th colspan="2">Who speaks?</th>
    </tr>
    <tr>
      <th>LeftUser</th>
      <th>RightUser</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>LeftUser</td>
      <td>23.6%</td>
      <td>50.8%</td>
      <td>31.6%</td>
    </tr>
    <tr>
      <td>RightUser</td>
      <td>24.0%</td>
      <td>31.4%</td>
      <td>51.6%</td>
    </tr>
    <tr>
      <td>Tablet</td>
      <td>39.0%</td>
      <td>8.6%</td>
      <td>8.8%</td>
    </tr>
    <tr>
      <td>Elsewhere</td>
      <td>6.3%</td>
      <td>4.0%</td>
      <td>3.9%</td>
    </tr>
    <tr>
      <td>Saccade</td>
      <td>7.2%</td>
      <td>5.0%</td>
      <td>5.1%</td>
    </tr>
  </tbody>
</table>

In terms of verbal annotations, the pilot speaks approximately twice as much as the players (86 minutes compared to about 40 minutes for each left and right player, over the 4 hours of interactions). Although the number of utterances is similar between the pilot and the players, the pilot's average utterance duration is much longer due to game regulation, while the players mainly make short utterances with lots of feedback. Among the more than 2.800 pilot utterances, the pronouns "Vous" and "Tu" were respectively used in 632 and 172 of them, and player's first name in 139 (note that same utterance can contain both the pronoun "Tu" and a player's first name), allowing the detection of his addressee(s) for these utterances.

We showed in (Haefflinger et al, 2023b) the
difference in the distribution of the pilot's yaw
angle when using the different pronouns "Tu" and
"Vous" on 5 annotated sequences. These prelimi-
nary results highlighted the relevance of the head
in signalling attention, and the need to generate its
movements taking into account the context of the
interaction. Figure 3 presents for the 11 sequences,
the distribution of the pilot's median head yaw
angle (left/right) and gaze angle (sum of head
yaw and eye version) for utterances where the
addressee is known. The utterances were filtered
based on the addressee and the pilot's head direc-
tion during the utterance. Utterances where the
pilot's head was predominantly directed towards
the tablet (filtered based on the up/down pitch
angle) were removed from the analysis. After
calculating the median angle for each selected
utterance, a Gaussian distribution was estimated

for each addressee. Figure 3 confirms that the
pilot's head is more centered when addressing
both players than when addressing only one, while
the median gaze is not and alternates between the
players.

All these analyses have been carried out on all 11 sequences. Obviously, the behaviors of both the pilot and the players differ from one sequence to the next. The players’ personalities, for example, can play an important role: our analysis of players’ embeddings (Haefflinger et al, 2024) of the same model as described in Section 4 shows that inter-player dominance significantly biases this distribution.

3.4 RoboTrio replay on Furhat

The multimodal signals gathered via the teleoper-
ated iCub robot can be integrally replayed on any
robot with corresponding degrees-of-freedom. By
synchronizing the audio with recorded gaze and
head movements, the robot can reproduce them
simultaneously as the interaction audio is played.
We have also developed a tool for replaying inter-
actions on the Furhat robot. To do this, the audio
of the sequence is replayed, and the pilot's speech
is synthesized by Furhat based on the verbal anno-
tations. Furhat's gaze is controlled by giving the
pilot's annotated gaze target at each instant. Gaze
targets are limited to "LeftUser", "RightUser"
and "Tablet" to facilitate replay. For each target,
a position in (x,y,z) has been assigned in order
to indicate where Furhat should look. These posi-
tions correspond to the same player and tablet
positions set in RoboTrio. To reach the target,
Furhat's head movements are specified as the head
angles performed by the pilot, and eye movements
are controlled by the model already embedded in
Furhat.

4 End2end gaze model

4.1 Model description

The proposed model is a cascaded model where
the first output is a gaze target which is fed
as input to the second output generating an
appropriate head trajectory. The model predicts
the robot's gaze according to the 5 classes from
the corpus annotations: "LeftUser", "RightUser",
"Tablet", "Elsewhere" and "Saccade". The predic-
tion of head movements is done on the 3 human
head angles: Pitch (Up/Down), Roll (tilt), Yaw
(Left/Right), also available in the corpus.

The model structure is given in Figure 4. For each output, the model consists of one causal Convolutional Neural Network (CNN) layer, followed by one unidirectional (left-right) Long-Short Term Memory (LSTM) layer and one linear layer. The output is finally passed through a softmax() function for gaze prediction. To best predict the robot's gaze, it's essential to consider temporal events, which motivated the proposed combination of CNN and LSTM layers. The characteristics of the CNN layers are as follows: 20 filters, a dilation coefficient of 2 and a kernel size of 7 for the prediction of gaze targets, and 9 for that of the prediction of head angles, finally the padding is fixed to the size of the kernel divided by two and multiplied by the dilation coefficient. The LSTM layers are composed of 50 units.

The proposed model has a sequence-to-sequence architecture (seq2seq) enabling continuous predictions, with temporal sequences corresponding to a game theme (around 180 seconds per batch, 10,800 frames at 60Hz). In order to train the model, it is necessary to fix the size of the inputs. This is achieved by using the padding method in conjunction with the use of output masks to ignore these values. All input features are encoded in embeddings, allowing the model to detect common characteristics between the inputs. The input features are multimodal signals that have already been annotated in the RoboTrio corpus. These signals can be easily perceived by the robot, facilitating the real-time implementation of the model on a robot in the future. These are separated into two categories; those relating to robot activity (endogenous) and those relating to users activity (exogenous):

*   **Pilot activity: Speech** (speaking or not embeddings of dimension 2), *Speech Intent* (the 23 annotated speech intents, embeddings of dimension 3), *Addressee* (1 input vector for each player to cover the 0, 1 or 2 addressees cases, embeddings of dimension 2).

*   **Users activity: SpeechL/SpeechR** (whether left (resp. right) user is speaking or not, embeddings of dimension 2), *GazeL/GazeR* (user's gaze target, embeddings dimension: 2).

In addition to these multimodal signals, we decided to bias the model with the dyad number.

For each batch given as input to the model, we indicate which sequence this one originates from. This allows the model to take account of differences in social behavior between each pair, which may have influenced the pilot's behavior.

*   **Dyad#:** number of the 11 players pairs facing the robot, embedding of dimension 3.

Finally, during the training phase of the model, the head sub-model is fed with both predicted and real gaze targets from the dataset. This process ensures that the head sub-model actually exploits contextual information on its own.

## 4.2 Model training

For model training, the global loss is the sum of three contributions: on the gaze prediction output, one loss is calculated with the categorical cross-entropy function, on the head angle prediction output, two losses are calculated with the Mean Squared Error (MSE) function, one when the input gaze data are those predicted, and one when they are ground truth data. The training phase is performed on 100 epochs, the first 50 with a learning rate set to 0.005 and the next 50 with a learning rate of 0.001. The batch size is set to 8 and the chosen optimizer is Adam (Kingma and Ba, 2014). In order to best evaluate the performances of the proposed model, we applied the 9-fold cross-validation method. As a result, 9 models were trained and evaluated on different training and test datasets. The test datasets were formed by taking the Nth game theme played from each sequence (total of 11 game themes), leaving the training datasets with the remaining 8 themes from each sequence (total of 88 game themes). The results are presented in the following subsection.

## 4.3 Objective evaluation

Gaze target predictions are evaluated with the F1_score metric, and head trajectory prediction with the Root Mean Squared Error (RMSE). The overall results are: the mean F1-score for gaze classification is 0.50±0.01 and the mean RMSE for head generation is 3.0°±0.1 with predicted gaze, and 2.1°±0.1 with ground truth gaze.

To provide a better analysis of the proposed model's prediction, Table 2 shows the F1_score obtained on the final predictions (combination of
predictions obtained from each test dataset) and
those for each class and according to the robot's
activity. First of all, the "Elsewhere" and "Sac-
cade" classes are extremely poorly predicted, and
act as garbage collectors. Indeed, these classes are
the least represented in the input data, don't sup-
port specific targets, and can occur for a variety
of factors, which explains why the model seems
to focus on the other 3 classes. Moreover, we can
observe that predictions are better when the robot
is speaking, certainly due to the knowledge of the
robot's speech intention. On the contrary, predic-
tions are worse when no one is speaking, because
of the lack of contextual information.

Table 3 provides more details on head movement prediction performance. It can be confirmed that head movements are logically better predicted when the input gaze targets are those of the ground truth, but there are also differences between the 3 angles. The roll angle seems to be the best predicted, involving the least variation. On the other hand, the pitch angle is the hardest to predict, especially when the input gaze targets are the predicted ones, this angle being strongly impacted by a misclassification between players and tablet classes.

4.3.1 Ablation study

To understand how each input feature influences
the model's prediction, we have conducted an
ablation study. The model has been retrained,
with the removal of one or more input features at
each run. We applied the 9-fold cross-validation
method to each ablation study, in order to best
evaluate the effects on performance. Figure 5
shows the global performances obtained for each
ablation study. The gray barplots correspond to
the performance of the model without ablation,
the red ones correspond to ablations performed
on robot-endogenous features, the blue ones are
robot-exogenous, and finally the yellow barplot
refers to the ablation of the players pair bias. To
test if there were significant differences between
the performance distributions of the model with-
out ablation and with ablation, Wilcoxon tests
were conducted between them for each ablation
study.

First of all, we can see that head prediction
performance varies little when the model is fed

with ground truth gaze targets. The gaze informa-
tion largely compensate for the exclusion of other
features. However, removing the dyad bias sig-
nificantly impacts head prediction performance,
regardless of the gaze input used. The addition of
this bias appears to be beneficial to the model's
predictions, mainly concerning head angles.

The other two predictions are far more influ-
enced. Removing the feature related to the pilot's
speech intention seems to considerably reduce
F1_score and degrade head prediction. This infor-
mation in fact distinguishes between times when
the robot needs to look at the tablet (to give a
theme, a score, etc.) and times when it needs to
look at the players (to ask for a proposal, give feed-
back, etc.), thus influencing the gaze target, and
consequently the pitch angle of the head.

In addition, head and gaze predictions are
also impacted when the robot addressee input
is removed. This is consistent with the dataset
analysis presented in Section 3.3, validating the
importance of this feature even for a low number
of occurrences with a non-"Unknown" value.

The removal of other features doesn't seem
to have any impact on the results, but when all
endogenous or exogenous features are removed,
performances are significantly affected, suggest-
ing interleaved influences. We thus complemented
this objective analysis with two subjective evalu-
ations that will focus on two critical questions :
(1) does the gaze direction solely determine head
movements or should the predictor be aware of
communicative context? (2) do third parties actu-
ally decode addressees through gaze and head
control performed by our data-driven model?

4.3.2 Discussion

The objective evaluation of the proposed model
is relatively informative. Mainly, concerning gaze,
where the “Elsewhere” class is poorly considered
by the model. However, gaze in the direction of
the “Tablet” seems to be well predicted, thanks to
strong indicators such as the robot’s speech inten-
tion, which informs the model whether the robot
needs to consult the game’s information or not.
Nevertheless, these results are insufficient to con-
clude on the relevance of the predictions made by
our model. Indeed, failure to reproduce exactly the
behavior of the human pilot during the game does
<table><caption>**Table 2** F1-score of the predicted gaze targets according to the pilot activity.</caption><thead><tr><th>Gaze Target</th><th>Pilot is Speaking</th><th>Pilot is Listening</th><th>No One is Speaking</th><th>ALL</th></tr></thead><tbody><tr><td>UserL</td><td>0.49</td><td>0.53</td><td>0.51</td><td>0.51</td></tr><tr><td>UserR</td><td>0.52</td><td>0.60</td><td>0.52</td><td>0.55</td></tr><tr><td>Tablet</td><td>0.74</td><td>0.29</td><td>0.53</td><td>0.66</td></tr><tr><td>Elsewhere</td><td>0.05</td><td>0.00</td><td>0.01</td><td>0.03</td></tr><tr><td>Saccade</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td></tr><tr><td>Weighted F1-score</td><td>0.53</td><td>0.50</td><td>0.46</td><td>0.50</td></tr></tbody></table>

<table><caption>**Table 3** RMSE of the predicted head movements for each head angle in degrees.</caption><thead><tr><th>Gaze Input</th><th>Pitch</th><th>Roll</th><th>Yaw</th></tr></thead><tbody><tr><td>Predicted</td><td>4.2 ± 0.2</td><td>2.1 ± 0.1</td><td>3.7 ± 0.2</td></tr><tr><td>Ground Truth</td><td>2.7 ± 0.1</td><td>1.7 ± 0.1</td><td>2.7 ± 0.1</td></tr></tbody></table>

not mean that the proposed solution is less natural, or irrelevant to the interaction context. In a previous study with a similar data-driven control and similar objective performances, we were able to show that the predicted gaze was judged as relevant as the one performed by the pilot (Haefflinger et al, 2023a). The following sections will therefore focus on the subjective evaluation of the essential properties of the proposed model.

# 5 Study 1: Evaluation of head movements

## 5.1 Goal

We here compare the head movements generated by our model, which takes into account the context of the interaction (who is talking, to whom, who is looking where, etc), with movements generated by a simpler control, unaware of the context, where the head is a passive carrier of the eyes. The subjective evaluation consists in comparing a robot whose head movements have been predicted by these two different methods. The final purpose is to validate the contribution of context to the active generation of robot's eyes-head coordination.

In order to focus on head evaluation, the evaluated movements are those of RoboTrio interactions predicted by our model with ground truth gaze targets. Each sequence is thus fully predicted, using for each of the Nth game theme the Nth

model that has not been trained on it. Finally, the predictions of the 3 head angles are smoothed with a Blackman filter of window size 30.

## 5.2 Method

### 5.2.1 Baseline heuristic model

The heuristic model used for comparison with the model proposed in this paper is similar to that used in (Haefflinger et al, 2023a). This model is based exclusively on the distance to be covered between two fixations. The head angles to be performed are calculated as percentages of this distance, these percentages have been set to reproduce the variance of head movements made by the pilot during RoboTrio. As a result, the calculated pitch angle corresponds to 45% of the up/down angle to be performed, and the yaw angle to 30% of the left/right angle. The roll angle is set as the average angle achieved by the pilot over all sequences, i.e. 3,6°.

In addition, to make the predicted movements of this model more realistic, we implemented additional characteristics of human head movements. The equations for eyes-head coordination proposed by Itti et al (Itti et al, 2006) were used to calculate the angle threshold at which the head begins to move and to generate the trajectories between two fixations, once the final angle had been calculated. Finally, the midline effect (Fuller, 1992b) is also implemented, allowing the head to return to the midline (0°) when the next fixation and the current head position are on both sides of the midline. The final head movements to be compared with those of the proposed model were then calculated on all the 11 sequences, detecting changes in fixations between the three targets of the gaze annotations: “LeftUser”, “RightUser” and “Tablet”. Finally, the resulting movements
are filtered with a Blackman filter of window size 30.

### 5.2.2 Samples selection

To obtain samples for comparison in an interaction context, RoboTrio sequences are replayed by the virtual Furhat (see Section 3.4). For this study, the head movements given as input are no more those performed by the pilot but those predicted by the proposed model vs. the heuristic model. The gaze targets are not changed from those annotated, for both head movement conditions. By capturing video extracts using this replay method, only the robot is visible on the video, and only the audio provides information about the players, thanks to stereo diffusion (the left player is heard on the left channel, and conversely for the right player). To add more context, we decided to add human silhouettes to the left and right of the video, simulating the gaze of the two players, by turning either towards the other silhouette, or towards the robot depending on the annotation of their gaze (see Figure 6).

One extract per RoboTrio sequence was selected, resulting in 22 video clips (11 extracts multiplied by the two head movement conditions). The duration of these videos varies from 20s (demonstration extract) to 165s, with an average duration of 90s.

### 5.2.3 Online pairwise comparison test

To compare these two models, we decided to conduct an online pairwise preference test. To this end, we've designed a website where each page includes a comparison of two videos showing one robot each, one at the top and one at the bottom. Robots are engaged in the same interaction extract (same verbal, and same silhouette behaviors) but with head movements generated by our model for one, and those generated by the heuristic model for the other. The participants are then required to view the two videos in their entirety, which are played simultaneously on top of each other in order to facilitate a comparative analysis of the robots. Following this, the evaluators must respond to the three questions below, with the only possible responses being "top robot" or "bottom robot."

* Q1: Which of these 2 robots would you prefer to interact with?

* Q2: Which of these 2 robots is the most attentive to its interlocutors?

* Q3: Which of these 2 robots shows the most engagement with its interlocutors?

Evaluators are informed that only head movements vary, and are asked to focus on these.

To avoid bias, each interaction extract is evaluated twice, so that each condition is once on top, once on bottom. All subjects will see all extracts, but in random order. Figure 6 shows an example of a web page included in the evaluation.

In order to check that the evaluation was completed in the proper conditions, a material verification page and two attention checks pages were included to the evaluation. The first page of the website ensures that the user is receiving stereo sound from their computer and that the left channel is on their left side, and the right channel on their right side. If not, the user is invited to repeat the test by changing the orientation of their headphones, and if they fail again, they are not allowed to perform the rest of the experiment.

As a result, the final evaluation comprises 25 pages, 1 for the stereo test, 1 for training, 20 pages of preference testing (10 extracts multiplied by the two conditions top/bottom, bottom/top), 2 for attention checks and finally 1 page of feedback questionnaire.

## 5.3 Participants

We recruited 54 subjects through the crowdsourcing platform Prolific<fnref n="1" />, with a gender balance, all native French speakers from France, Belgium or Switzerland only. Among the 54 subjects who completed the experiment, 4 were removed from the final results after failing the attention checks, resulting in a total of 50 subjects with an average age of 34±10years. The median time taken to complete the experiment was 47 minutes, and all subjects received a reward of £8.

## 5.4 Results

Figure 7 presents the results obtained for each of the 3 questions, showing the percentage of times each model was chosen. To determine if there is

<footnote n="1">¹https://www.prolific.com/</footnote>
a significant difference in preference between the proposed model and the heuristic model, we performed a logistic regression for each question using the *glmer* function in the *lme4* package (Bates et al, 2015) of R software (R Core Team, 2022). For each regression, there is no fixed effect, and the random effect is the *Subject_id*. A comparison test was then carried out using the *glht* function in the multcomp package (Hothorn et al, 2008) of R software to test the null hypothesis *P(Y = "ProposedModel")* = 1/2. We found significant results for each of the three questions q1 (z= 2.68, pvalue=0.003), q2 (z=1.78, pvalue=0.037), q3 (z=3.13, pvalue=0.0008). The null hypothesis is therefore rejected, indicating that for the 3 questions, the *probability P(Y = "ProposedModel")* is significantly higher than 1/2.

The robot with head movements generated by our model is preferred to the one with head movements generated by the heuristic model, and it is perceived as more engaged and attentive. While these differences may vary between extracts, in general it seems more effective to use a model that takes into account the context of the interaction, rather than a context-unaware model.

# 6 Study 2: Evaluation of multi-party handling

## 6.1 Goal

The proposed model aims to provide automated gaze management for a robot that acts as an ani- mator in a collaborative game. In this role, the robot is expected to either address both players or a single one, to encourage collaboration and promote balanced participation. In such cases, it's essential that the robot can easily designate to whom it is addressing when speaking. With this in mind, we've added this information as an input embedding to the model. This subjective evalua- tion aims to validate that controlling this single input is effectively perceived as a faithful indica- tion of the robot addressee(s) through its gaze and head orientation.

## 6.2 Method

### 6.2.1 Analysis of biased predictions

Thanks to the ablation study carried out on our model in 4.3.1, we can assume that addressee

information is relevant to our model. However, before conducting the subjective evaluation, we checked how the control of this feature would affect the predictions of our model. To do this, we used a model trained on the entire corpus, to predict gaze and head movements in sequences where the addressee value is modified during a short clip. The addressee value is only updated during each pilot utterance, setting the addressee as “Left-User”, “RightUser” or “Both”. Figure 8 presents the predictions obtained for a 10s time window for each addressee value specified.

This shows that head movements (upper sub-figure) and gaze target predictions (lower sub-figure) behave differently depending on the chosen *Addressee* bias. We can see that the head is more centered when the bias is set to “Both”, i.e. when the robot is instructed to address both players. The model faithfully reproduces the pilot behaviors analyzed in Section 3.3. Moreover, when the addressee is a single player, gaze prediction focuses exclusively on that player. In contrast, when the addressee bias is “Both”, prediction varies (i.e. alternating between LeftUser/RightUser) within and between speech utterances. We hypothesize that these observed differences can be effectively perceived by third parties, and influence their perception of to whom the robot is addressing. For the selection of samples to be evaluated, it is essential to present an extract including several utterances by the robot, as gaze prediction in the “Both” condition may not vary during a single utterance.

### 6.2.2 Samples selection

To capture our samples, we again used the replay method on the virtual Furhat. This time, we used the head movements and gaze targets predicted by the proposed gaze control model. To limit the verbal content bias in the evaluation of our extract, the speech annotations resynthesized by Furhat were modified to replace all markers of the French pronoun “Tu” (one addressee) with “Vous” markers (which can refer to one or more addressees, in the case of the politeness “Vous”). In addition, the sound of the interaction is played in mono, removing the possibility of guessing which player is speaking. A total of 18 (+1 for demonstration) interaction clips were selected (at least 1 per RoboTrio sequence), with durations varying from 13s to 40s and an average duration of 25s.
These extracts were selected to ensure that when
the robot speaks, the context does not favor any
addressee scenario (both players or one of them).
Each extract resulted in 3 video clips, 1 for each
addressee bias, and only the robot is displayed (see
Figure 9).

6.2.3 Online evaluation website

To evaluate the captured video clips, we built a
website, where evaluators have, on each page, to
watch one clip in its entirety before responding to
the following two questions:

* Over the whole clip, who do you think the robot is addressing when it speaks?

* Indicate your level of confidence in this answer.

For the first question, three answers are pos-
sible: the player on the A side, the player on the
B side, or both players. We have chosen to use
letters and not left/right, to avoid laterality con-
fusion. Figure 9 shows a page from the website
used, where the position of these letters on either
side of the video can be seen.

For the second question, 4 answers are pos-
sible: very confident, fairly confident, not very
confident, extremely non-confident.

Before the evaluation begins, evaluators are briefed on the game context of the interaction, and are explicitly asked to focus on the robot's gaze and head movements to answer the asked questions.

To prevent biased responses, the same inter-
action extract is presented only once and for a
single condition to the same evaluator. However,
each evaluator will see each extract at least once,
and each condition the same number of times
in total. To comply with these constraints, the
evaluation was conducted over 3 blocks, thus con-
taining 18 extract/bias condition combinations.
All evaluators in the same block saw the same
clips (extract/bias condition combinations), but in
random order.

In the end, an evaluation consists in 21 pages:
1 of context description, 1 of demonstration, 18 of
evaluation, and the final feedback form.

6.3 Participants

We recruited 90 subjects on the crowdsourcing platform Prolific<fnref n="2" />, i.e 30 subjects for each evaluation block, with a gender balance. All subjects are French native speakers, and live in France, Belgium or Switzerland only. Among the 90 subjects who completed the experiment, 3 were removed from the final results for not following the instructions, 1 subject in the evaluation block 2, and 2 subjects in block 3. As a result, each extract/bias condition combination was evaluated a minimum of 28 times and a maximum of 30 times. The remaining 87 subjects have an average age of 35±11 years. The median time taken to complete the experiment was 13 minutes, and all subjects received a reward of £2.5.

6.4 Results

First, concerning the question of to whom the
robot is addressing, a confusion matrix of the
answers given by evaluators as a function of
addressee bias is shown Figure 10. It can be
observed that, in the majority of cases, evaluators
identification of the robot’s addressee matches the
one specified as input to the model. The overall
F1_score calculated from these results is 0.78 with
an accuracy of 78%. The F1_score is slightly lower
for the “Both” bias than for “Left” or “Right”,
with a value of 0.70 versus 0.82 for the other two.
This result is not surprising: it’s understandable
that it’s harder to identify whether the robot is
addressing both players or just one, rather than
confusing left player with right player. In fact,
even if the robot has looked at both players, if one
of them is looked at slightly more than the other,
the evaluators may interpret this as the robot
addressing that player more than the other. In
contrast, the game remains collaborative, suggest-
ing sometimes that the robot is addressing both
players and not just one. In summary, 78% of the
time, the evaluators’ answers match the addressee
bias given as input, the chance level being 33%. It
can be concluded that this bias has a real impact
on the prediction of the robot’s behavior, and that
this impact is perceived and enables the robot to
convey its communicative intentions: to whom it
wants to address, simply with its gaze.

<footnote n="2">²https://www.prolific.com/</footnote>
Additionally, concerning the question on the
confidence level of the evaluator in his answer, we
wanted to check if this level depended on whether
or not the evaluator's answer was in agreement
with the bias. To answer this question, we have
created a binary variable *Agreement* which takes
the value 1 if the subject's response corresponds to
the addressee bias given, and 0 otherwise. Then,
we carried out an ordinal regression with random
effect using the clmm function of the ordinal pack-
age (Christensen, 2023) of the R software, with
the chosen random effects: *Subject_id* and *Clip_id*.
Finally, we applied a likelihood ratio test with the
Anova function of the R software which allowed
us to find a significant impact of the *Agreement*
variable on the level of confidence (chisq(1)=32.7,
p-value<0.0001). Figure 11 shows the distribu-
tion of the level of confidence depending of this
*Agreement* variable. It can be seen that although
subjects are quite confident in their answers, pro-
portions of high and very high confidence levels
decrease when answers are not aligned with the
addressee bias. This supports our previous conclu-
sion, showing that even when the robot failed to
communicate to whom it wanted to address, it still
influenced the evaluators' perception, represented
by their doubt in the chosen response.

# 7 Discussion

## 7.1 Implications

The general idea of this paper is to propose a
suitable approach for generating the gaze and
the eyes-head coordination of a robot in multi-
party interaction, and to demonstrate its benefits.
We propose a data-driven end-to-end model, with
the assumption that this allows to capture and
reproduce complex patterns of social behavior. A
rule-based model can also produce complex behav-
iors, but each rule must be considered and planned
in advance, and the design process can become
overcomplex by adding a wide variety of rules. The
proposed approach provides a model whose policy
is based on the parameters given as input and the
dataset used for training. One of its advantages is
that it offers both a model for generating gaze and
a tool for understanding its behavior: we can try
to determine what it learns and what information
it uses, which can then be used as levers to refine
its control. Although this second option has been

little exploited in the current work, it remains
important to mention when comparing it with
other approaches. For example, the utilization of
embedding layers to modulate model predictions
has already demonstrated its efficiency in other
fields such as image, text or speech generation (Ho
et al, 2022; Wang et al, 2018).

Unfortunately, it is difficult to compare the
performance of this model with other existing
studies, notably because the context of interac-
tion strongly affects social behavior. However, we
can summarize the main results presented and
their implications for HRI. Objective evaluations
carried out on our model show fairly moder-
ate objective results, with performances such as
the F1_score of the gaze at around 0.50. Please
note however that, depending on whether the
robot is in listening or speaking state, perfor-
mance differs. We then found that our model is
better at predicting gaze during speaking states,
which contrasts with previous studies that pro-
posed machine learning models performing better
in listening state (Huang et al, 2020; Stefanov
et al, 2019). There are 2 hypotheses to explain
this: firstly, the context of interaction is different
(in our case, the robot is acting as a facilita-
tor, rather than engaging in a casual discussion);
secondly, our model has as input not only the
information about when the robot is speaking,
but also its intention and its addressee(s), which
is not the case when it is listening to the play-
ers. The ablation study of input features (see
Section 4.3.1) reinforces the claim that taking into
account the robot's communicative intentions has
a real benefit for gaze generation. In addition,
this study reveals that head-movement predictions
are impacted by the player-pair bias, even when
predicted from ground-truth gaze. This result sug-
gests that it is useful to consider the differences in
social behavior that may exist among humans.

However, as discussed in Section 4.3.2, these
results are not sufficient to conclude on the rel-
evance of our model's predictions, and require
subjective evaluations. We therefore proposed two
subjective evaluations, to assess two different
aspects of our model: the proper communicative
functions of head movements and whether the
predictions allow the robot to communicate with
whom it wants to address. Firstly, these evalua-
tions were carried out with third-person viewers,
instead of persons interacting with the robot as
proposed by Mishra et al (Mishra and Skantze, 2022) for the evaluation of their gaze control. To do this, we used an original method involving a replay of the interaction by the virtual robot Furhat. This method is similar to that proposed by Shintani et al (Shintani et al, 2021), where the robot replaces a participant in the scene by broadcasting her voice, not by resynthesizing it as in our case. Thanks to this process, the evaluation can be carried out considering the context of interaction and not, for example, evaluating eye-head coordination exclusively on the physical aspect, as previous studies have done (Peters and Qureshi, 2010; Pejsa et al, 2015), and neglecting the communicative and social functions of head movement.

As a result, we showed in our first evaluation that a robot with head movements predicted by our model is significantly preferred to those predicted by a heuristic model where head is slaved by gaze. Even if this preference is slight, taking into account contextual information to generate eyes-head coordination, such as who is speaking to whom, is beneficial for the perception of the robot. We hope that future gaze models will take those aspects into account, especially for multi-party interactions.

The second evaluation showed that by controlling the *Addressee* embeddings of our model, we could impact its predictions depending on whether the robot addresses one or both players. These intents are clearly perceived by humans, allowing them to detect in 78% of cases who the robot aims to address when it is speaking. Endowing a robot with the proposed model enables it to communicate to whom it wants to address via its gaze only. This is particularly important in many practical applications, not only our case where the robot is the animator of a collaborative game and needs to balance speaking time.

The results of these two studies prove that the use of machine learning models to control a robot's gaze is fully relevant. It generates natural, smooth behaviors, and more importantly, enables the robot to convey its communicative intentions. Machine learning models are already widely used to generate non-verbal behavior for virtual agents in discourse situations (Fares et al, 2023; Yoon et al, 2020), but are poorly represented for interactions where the behavior of interlocutors is decisive. This paper aims to promote the use of

this approach in HRI. As these initial results are promising, we hope that the use of these models will be extended in the future.

## 7.2 Limitations

At the core of our model's training, data can be a potential limitation to its performance and use. A primary limitation is the two-person collaborative game context. While the game task has been designed to be transferable to more general contexts involving form filling, the interaction is limited to two people facing the robot, so the model can't handle more for now. Regarding the adaptability of this model in new scenarios, we have proposed a first work suggesting a possible approach through the use of embeddings (Haeflinger et al, 2024).

Moreover, the recording of this corpus is based on the immersive teleoperation of a robot, to capture human-like movements in HRI. While this method allows us to obtain precise data, and probably captures the HRI/HHI differences, it raises the question of how far the pilot's behavior has been impacted by wearing a virtual reality headset, limiting his perception and action capabilities and resulting in less natural behavior. The results of recent studies comparing head movements with and without virtual reality diverge, finding differences (Pfeil et al, 2018) or not (Sidenmark and Gellersen, 2020). The fact that our experimental setup involved real camera streams with mixed reality but no 3D/virtual reconstruction at all makes it all even more difficult to compare with the literature and conclude on this potential impact.

Finally, data annotation can also be a limitation to the model's gaze prediction. In particular, the gaze annotations of the players and the pilot were performed automatically and are perfectible. When compared with a few manually annotated extracts (560 frames), the Kappa score of agreement between these two annotation methods was 0.70 for the players' gaze and 0.66 for the pilot's. These results can be interpreted as a good agreement according to the Landis & Koch scale (Landis and Koch, 1977). In addition, a qualitative assessment of the pilot's gaze annotation was carried out by replaying the exact gaze data recorded on the iCub robot, and the annotated gaze on the
Furhat robot, showing a high degree of similar-
ity in behavior. These results suggest that, even
if imperfect, the proposed annotations are highly
relevant for the model training.

Concerning the two subjective evaluations of
the proposed model, none of them assess the
naturalness or the relevance of the attention tar-
get's predictions. In a previous work, an already
successful evaluation had been carried out on
a similar model (Haefflinger et al, 2023a). We
therefore preferred to focus on other aspects in
this paper and show the other benefits of such
a model. Moreover, both evaluations were con-
ducted online, using a "third-person" method, i.e.
people were not involved in any interaction with
the robot. Unfortunately, this method does not
allow us to measure the impact of our robot on
people's behavior, but only a subjective feedback.
Pereira et al (Pereira et al, 2020) argue that using
a third-person method achieves similar results to a
first-person evaluation in HRI. Finally, to simplify
the process, the replay and capture of video clips
was carried out on the virtual Furhat, and not on
the physical robot. The Mona-Lisa effect (Todor-
ović, 2006), known to affect the perception of an
agent's gaze when displayed in 2D, can be pointed
out. However, the evaluators were not asked to
judge the robot's gaze relative to them, but rela-
tive to its behavior in the conversation, where the
targets of attention are very distinct and fixed.

Despite these limitations, the methods and
results proposed in this work are promising and
deserve further analysis, especially evaluation with
persons interacting in presence with the robot.

8 Conclusions and perspectives

In this paper, we propose a gaze control model
for a robot in three-party interaction. This model,
composed of CNN and LSTM layers and trained
with a dataset recorded through immersive teleop-
eration of a robot, predicts gaze and corresponding
head movements. This model receives input fea-
tures relating to the interaction: is the robot
speaking, about what to whom, and are its inter-
locutors speaking, and where are they looking?

A first evaluation conducted online showed
that head movements predicted by our model were
preferred to those predicted by a heuristic model

based purely on the distance between two fixa-
tions. This result shows that a model using contex-
tual information in addition to the gaze target to
reach is better perceived than a context-unaware
model. Interaction context should therefore not be
neglected when generating eyes-head coordination
for a robot.

A second evaluation showed that the proposed
control allows our robot to convey through its gaze
who it intends to address when it speaks (one or
two people). By modifying the value of the input
feature related to who the robot is addressing (the
person on the right/left or both), the model has
been shown to adapt the robot's gaze patterns.
This skill is especially important as, in this work,
the robot's role is game facilitator, requiring it to
balance and motivate speech between the players.

In conclusion, we have highlighted the via-
bility of this model, and its benefits compared
with a model lacking knowledge of the interaction
context. Future work will focus on implement-
ing this model on the physical Furhat robot, to
validate its use in real-time, and to obtain an
autonomous robot facilitator. This experimental
setup will enable us to evaluate the impact of the
proposed gaze control on the behavior of people
interacting with our robot. It would then be inter-
esting to investigate whether, by controlling the
addressee bias, the robot can successfully balance
turns and promote collaboration between players.

**Supplementary information.** A supplementary video showing examples of the subjective evaluations conducted is available at this link: https://ssw2023.org/GIPSA_expe/Videos/supplementaryVideo_ijsr_v2.mp4.

**Acknowledgements.** This work is supported
by the ANR 19-P3IA-0003 MIAI. The first author
is financed by a CIFRE PhD granted by ANRT
(2021/0836). The authors thank Silvain Gerber for
his help with the statistical analysis of the results.

Declarations

* Funding: This work is supported by the ANR 19-P3IA-0003 MIAI. The first author is financed by a CIFRE PhD granted by ANRT (2021/0836)

* Conflict of interest/Competing interests: The authors declare that they have no conflict of interest
* Ethics approval and consent to participate: Not applicable

* Consent for publication: Not applicable

* Data availability: The authors declare that the dataset used to train the model is freely available online at this link: https://hdl.handle.net/11403/robotrio/v2.

* Materials availability: Not applicable

* Code availability: Not available yet. A Github will be released upon acceptance

## References

Admoni H, Scassellati B (2017) Social eye gaze in human-robot interaction: a review. Journal of Human-Robot Interaction Steering Committee 6(1):25–63. https://doi.org/10.5898/JHRI.6.1.Admoni

Admoni H, Weng T, Hayes B, et al (2016) Robot nonverbal behavior improves task performance in difficult collaborations. In: International Conference on Human-Robot Interaction (HRI). IEEE, International Conference on Human-Robot Interaction (HRI), pp 51–58, https://doi.org/10.1109/HRI.2016.7451733

Al Moubayed S, Beskow J, Skantze G, et al (2012) Furhat: A Back-Projected Human-Like Robot Head for Multiparty Human-Machine Interaction. In: Cognitive Behavioural Systems: COST 2102 International Training School. Springer, Cognitive Behavioural Systems: COST 2102 International Training School, pp 114–130, https://doi.org/10.1007/978-3-642-34584-5_9

Amos B, Ludwiczuk B, Satyanarayanan M (2016) Openface: A general-purpose face recognition library with mobile applications. Tech. rep., CMU-CS-16-118, CMU School of Computer Science

Andrist S, Tan XZ, Gleicher M, et al (2014) Conversational gaze aversion for humanlike robots. In: International Conference on Human-Robot Interaction (HRI). ACM, New York, NY, USA, International Conference on Human-Robot Interaction (HRI), pp 25–32, https://doi.org/10.1145/2559636.2559666

Argyle M (1972) Non-verbal communication in human social interaction. In: Hinde RA (ed) Non-verbal communication. Cambridge U. Press, Oxford, England, p 243–249

Bates D, Mächler M, Bolker B, et al (2015) Fitting linear mixed-effects models using lme4. Journal of Statistical Software 67(1):1–48

Birdwhistell RL (1955) Background to Kinesics. ETC: A Review of General Semantics 13(1):10–18. URL https://www.jstor.org/stable/42581570, publisher: Institute of General Semantics

Bohus D, Horvitz E (2010) Facilitating multi-party dialog with gaze, gesture, and speech. In: International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction. ACM, New York, NY, USA, International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction, pp 1–8, https://doi.org/10.1145/1891903.1891910

Boucher JD, Pattacini U, Lelong A, et al (2012) I reach faster when I see you look: gaze effects in human-human and human-robot face-to-face cooperation. Frontiers in neurorobotics 6:3

Brugman H, Russel A, Nijmegen X (2004) Annotating multi-media/multi-modal resources with elan. In: International Conference on Languages Resources and Evaluation (LREC). European Language Resources Association (ELRA), International Conference on Languages Resources and Evaluation (LREC), pp 2065–2068

Cambuzat R, Elisei F, Bailly G, et al (2018) Immersive teleoperation of the eye gaze of social robots assessing gaze-contingent control of vergence, yaw and pitch of robotic eyes. In: International Symposium on Robotics (ISR). IEEE, Munich, Germany, International Symposium on Robotics (ISR), pp 232–239

Christensen RHB (2023) ordinal—Regression Models for Ordinal Data. URL https://CRAN.R-project.org/package=ordinal, r package version 2023.12-4
Correia F, Campos J, Melo FS, et al (2023)
Robotic Gaze Responsiveness in Multiparty
Teamwork. International Journal of Social
Robotics 15(1):27–36. https://doi.org/10.1007/
s12369-022-00955-1

Domingo JD, Gómez-García-Bermejo J, Zalama
E (2022) Optimization and improvement
of a robotics gaze control system using
LSTM networks. Multimedia Tools and Appli-
cations 81(3):3351–3368. https://doi.org/10.
1007/s11042-021-11112-7

Duncan S (1972) Some signals and rules for tak-
ing speaking turns in conversations. Journal of
personality and social psychology 23(2):283

Elisei F, Haefflinger L, Prévot L, et al (2023)
The robotrio corpus. URL https://hdl.handle.
net/11403/robotrio/v2, ORTOLANG (Open
Resources and TOols for LANGuage) –
www.ortolang.fr

Fares M, Pelachaud C, Obin N (2023) Zero-shot
style transfer for gesture animation driven by
text and speech using adversarial disentangle-
ment of multimodal style encoding. Frontiers in
Artificial Intelligence 6:1142997

Foerster F, Bailly G, Elisei F (2015) Impact of iris
size and eyelids coupling on the estimation of
the gaze direction of a robotic talking head by
human viewers. In: International Conference on
Humanoid Robots (Humanoids), IEEE. IEEE,
International Conference on Humanoid Robots
(Humanoids), pp 148–153

Freedman EG, Sparks DL (2000) Coordination
of the eyes and head: movement kinemat-
ics. Experimental Brain Research 131(1):22–32.
https://doi.org/10.1007/s002219900296

Fuller JH (1992a) Comparison of Head Move-
ment Strategies among Mammals. In: Berthoz
A, Graf W, Vidal PP (eds) The Head-
Neck Sensory Motor System. Oxford University
Press, p 0, https://doi.org/10.1093/acprof:oso/
9780195068207.003.0013

Fuller JH (1992b) Head movement propensity.
Experimental Brain Research 92:152–164

Gillet S, Cumbal R, Pereira A, et al (2021) Robot
gaze can mediate participation imbalance in
groups with different skill levels. In: Interna-
tional Conference on Human-Robot Interac-
tion (HRI). ACM, Boulder CO USA, Interna-
tional Conference on Human-Robot Interaction
(HRI), pp 303–311, https://doi.org/10.1145/
3434073.3444670

Goffman E (1979) Footing. Semiotica 25(1-2):1-30

Goodwin C (1981) Conversational organization,
Interaction between speakers and hearers. Aca-
demic press

Haefflinger L, Elisei F, Bouchot B, et al (2023a) Data-driven generation of eyes and head movements of a social robot in multiparty conversation. In: International Conference on Social Robotics (ICSR), Springer, International Conference on Social Robotics (ICSR), pp 191–203, https://doi.org/10.1007/978-981-99-8715-3{–}17

Haefflinger L, Elisei F, Gerber S, et al (2023b) On the Benefit of Independent Control of Head and Eye Movements of a Social Robot for Multiparty Human-Robot Interaction. In: International Conference on Human-Computer Interaction (HCI). Springer, International Conference on Human-Computer Interaction (HCI), pp 450–466, https://doi.org/10.1007/978-3-031-35596-7_29

Haefflinger L, Elisei F, Varini B, et al (2024)
Probing the inductive biases of a gaze model
for multi-party interaction. In: Companion of
the 2024 ACM/IEEE International Conference
on Human-Robot Interaction (HRI). ACM,
Companion of the 2024 ACM/IEEE Interna-
tional Conference on Human-Robot Interac-
tion (HRI), pp 507–511, https://doi.org/https:
//doi.org/10.1145/3610978.3640702

Hietanen JK (1999) Does your gaze direction
and head orientation shift my visual atten-
tion? NeuroReport: For Rapid Communication
of Neuroscience Research 10:3443–3447. https:
//doi.org/10.1097/00001756-199911080-00033
Ho J, Saharia C, Chan W, et al (2022) Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research 23(47):1–33

Hothorn T, Bretz F, Westfall P (2008) Simultaneous inference in general parametric models. Biometrical Journal 50(3):346–363

Huang HH, Kimura S, Kuwabara K, et al (2020) Generation of Head Movements of a Robot Using Multimodal Features of Peer Participants in Group Discussion Conversation. Multimodal Technologies and Interaction 4(2):15. https://doi.org/10.3390/mti4020015

Ishii R, Otsuka K, Kumano S, et al (2016) Prediction of who will be the next speaker and when using gaze behavior in multiparty meetings. ACM Transactions on Interactive Intelligent Systems (TIIS) 6(1):1–31

Itti L, Koch C (2001) Computational modelling of visual attention. Nature reviews neuroscience 2(3):194–203

Itti L, Dhavale N, Pighin F (2006) Photorealistic Attention-Based Gaze Animation. In: International Conference on Multimedia and Expo. IEEE, International Conference on Multimedia and Expo, pp 521–524, https://doi.org/10.1109/ICME.2006.262440

Johansson M, Skantze G, Gustafson J (2013) Head pose patterns in multiparty human-robot team-building interactions. In: International Conference on Social Robotics (ICSR). Springer, International Conference on Social Robotics (ICSR), pp 351–360

Jokinen K, Nishida M, Yamamoto S (2010) On eye-gaze and turn-taking. In: Workshop on Eye gaze in intelligent human machine interaction. ACM, Hong Kong China, Workshop on Eye gaze in intelligent human machine interaction, pp 118–123, https://doi.org/10.1145/2002333.2002352, URL https://dl.acm.org/doi/10.1145/2002333.2002352

Jokinen K, Furukawa H, Nishida M, et al (2013) Gaze and turn-taking behavior in casual conversational interactions. ACM Transactions

on Interactive Intelligent Systems 3(2):1–30. https://doi.org/10.1145/2499474.2499481, URL https://dl.acm.org/doi/10.1145/2499474.2499481

Kendon A (1967) Some functions of gaze-direction in social interaction. Acta Psychologica 26(1):22–63. https://doi.org/10.1016/0001-6918(67)90005-4

Kingma DP, Ba J (2014) Adam: A method for stochastic optimization. arXiv preprint arXiv:14126980

Krauss RM, Chen Y, Chawla P (1996) Non-verbal behavior and nonverbal communication: What do conversational hand gestures tell us? Advances in Experimental Social Psychology 28:389–450

Kuno Y, Sadazuka K, Kawashima M, et al (2007) Museum guide robot based on sociological interaction analysis. In: International Conference on Computer-Human Interaction (CHI). ACM, New York, NY, USA, International Conference on Computer-Human Interaction (CHI), p 1191–1194

Land MF (2004) The coordination of rotations of the eyes, head and trunk in saccadic turns produced in natural situations. Experimental Brain Research 159(2):151–160. https://doi.org/10.1007/s00221-004-1951-9

Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. biometrics pp 159–174

Liu C, Ishi CT, Ishiguro H, et al (2012) Generation of nodding, head tilting and eye gazing for human-robot dialogue interaction. In: International Conference on Human-Robot Interaction (HRI). ACM, New York, NY, USA, International Conference on Human-Robot Interaction (HRI), p 285–292, https://doi.org/10.1145/2157689.2157797

Mishra C, Skantze G (2022) Knowing Where to Look: A Planning-based Architecture to Automate the Gaze Behavior of Social Robots. In: International Conference on Robot and
Human Interactive Communication (RO-MAN). IEEE, International Conference on Robot and Human Interactive Communication (RO-MAN), pp 1201–1208, https://doi.org/10.1109/RO-MAN53752.2022.9900740

Moujahid M, Hastie H, Lemon O (2022) Multi-party Interaction with a Robot Receptionist. In: International Conference on Human-Robot Interaction (HRI). IEEE Press, Sapporo, Hokkaido, Japan, International Conference on Human-Robot Interaction (HRI), pp 927–931

Mutlu B, Forlizzi J, Hodgins J (2006) A Storytelling Robot: Modeling and Evaluation of Human-like Gaze Behavior. In: International Conference on Humanoid Robots (Humanoids). IEEE, International Conference on Humanoid Robots (Humanoids), pp 518–523, https://doi.org/10.1109/ICHR.2006.321322

Mutlu B, Shiwa T, Kanda T, et al (2009) Footing in human-robot conversations: how robots might shape participant roles using gaze cues. In: International Conference on Human Robot Interaction (HRI). ACM, New York, NY, USA, International Conference on Human Robot Interaction (HRI), p 61–68, https://doi.org/10.1145/1514095.1514109

Nakano YI, Yoshino T, Yatsushiro M, et al (2015) Generating Robot Gaze on the Basis of Participation Roles and Dominance Estimation in Multiparty Interaction. ACM Transactions on Interactive Intelligent Systems 5(4):22:1–22:23. https://doi.org/10.1145/2743028

Otsuka K, Tsumori M (2020) Analyzing Multifunctionality of Head Movements in Face-to-Face Conversations Using Deep Convolutional Neural Networks. IEEE Access 8:217169–217195. https://doi.org/10.1109/ACCESS.2020.3041672

Otsuka K, Takemae Y, Yamato J (2005) A probabilistic inference of multiparty-conversation structure based on Markov-switching models of gaze patterns, head directions, and utterances. In: International Conference on Multimodal Interfaces. ACM, New York, NY, USA, International Conference on Multimodal Interfaces, pp 191–198, https://doi.org/10.1145/1088463.

1088497

Otsuka K, Sawada H, Yamato J (2007) Automatic inference of cross-modal nonverbal interactions in multiparty conversations: "who responds to whom, when, and how?" from gaze, head gestures, and utterances. In: International Conference on Multimodal Interfaces. ACM, Nagoya Aichi Japan, International Conference on Multimodal Interfaces, pp 255–262, https://doi.org/10.1145/1322192.1322237, URL https://dl.acm.org/doi/10.1145/1322192.1322237

Parmiggiani A, Randazzo M, Maggiali M, et al (2014) An articulated talking face for the icub. In: International Conference on Humanoid Robots (Humanoids). IEEE, International Conference on Humanoid Robots (Humanoids), pp 1–6, https://doi.org/10.1109/HUMANOIDS.2014.7041309

Pejsa T, Andrist S, Gleicher M, et al (2015) Gaze and attention management for embodied conversational agents. ACM Transactions on Interactive Intelligent Systems 5:1–34

Pereira A, Oertel C, Fermoselle L, et al (2020) Effects of different interaction contexts when evaluating gaze models in hri. In: International Conference on Human-Robot Interaction (HRI). ACM, New York, NY, USA, International Conference on Human-Robot Interaction (HRI), p 131–139

Peters C, Qureshi A (2010) Graphics for serious games: A head movement propensity model for animating gaze shifts and blinks of virtual characters. Computers & Graphics 34:677–687

Pfeil K, Taranta EM, Kulshreshth A, et al (2018) A comparison of eye-head coordination between virtual and physical realities. In: ACM Symposium on Applied Perception. ACM, ACM Symposium on Applied Perception, pp 1–7

Picot A, Bailly G, Elisei F, et al (2007) Scruzinizing natural scenes: controlling the gaze of an embodied conversational agent. In: Intelligent Virtual Agents (IVA). Springer, Intelligent Virtual Agents (IVA), pp 272–282
R Core Team (2022) R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria, URL https://www.R-project.org/

Shintani T, Ishi CT, Ishiguro H (2021) Analysis of Role-Based Gaze Behaviors and Gaze Aversions, and Implementation of Robot's Gaze Control for Multi-party Dialogue. In: International Conference on Human-Agent Interaction (HAI). ACM, Virtual Event Japan, International Conference on Human-Agent Interaction (HAI), pp 332-336, https://doi.org/10.1145/3472307.3484653

Shintani T, Ishi CT, Ishiguro H (2022) Expression of Personality by Gaze Movements of an Android Robot in Multi-Party Dialogues. In: International Conference on Robot and Human Interactive Communication (RO-MAN). IEEE, Napoli, Italy, International Conference on Robot and Human Interactive Communication (RO-MAN), pp 1534-1541, https://doi.org/10.1109/RO-MAN.53752.2022.9900812

Shriberg E, Stolcke A, Hakkani-Tür D, et al (2012) Learning when to listen: Detecting system-addressed speech in human-human-computer dialog. In: Interspeech. ISCA, Interspeech, pp 334-337

Sidenmark L, Gellersen H (2020) Eye, Head and Torso Coordination During Gaze Shifts in Virtual Reality. ACM Transactions on Computer-Human Interaction 27(1):1-40. https://doi.org/10.1145/3361218

Sidner CL, Kidd CD, Lee C, et al (2004) Where to look: A study of human-robot engagement. In: International Conference on Intelligent User Interfaces. ACM, New York, USA, International Conference on Intelligent User Interfaces, p 78-84

Skantze G, Johansson M, Beskow J (2015) Exploring Turn-taking Cues in Multi-party Human-robot Discussions about Objects. In: International Conference on Multimodal Interaction (ICMI). ACM, New York, NY, USA, International Conference on Multimodal Interaction (ICMI), pp 67-74, https://doi.org/10.1145/2818346.2820749

Stefanov K, Salvi G, Kontogiorgos D, et al (2019) Modeling of Human Visual Attention in Multiparty Open-World Dialogues. ACM Transactions on Human-Robot Interaction 8(2):1-21. https://doi.org/10.1145/3323231

Stiefelhagen R, Zhu J (2002) Head orientation and gaze direction in meetings. In: Extended Abstracts on Human Factors in Computing Systems (CHI). ACM, Extended Abstracts on Human Factors in Computing Systems (CHI), pp 858-859

Sun Y, Fisher R (2003) Object-based visual attention for computer vision. Artificial Intelligence 146(1):77-123. https://doi.org/10.1016/S0004-3702(02)00399-5

Tashiro A, Imamura M, Kumano S, et al (2023) Analyzing and recognizing interlocutors' gaze functions from multimodal nonverbal cues. In: International Conference on Multimodal Interaction (ICMI). ACM, International Conference on Multimodal Interaction (ICMI), pp 33-41

Todorović D (2006) Geometrical basis of perception of gaze direction. Vision research 46(21):3549-3562

Vertegaal R, Slagter R, van der Veer G, et al (2001) Eye gaze patterns in conversations: there is more to conversational agents than meets the eyes. In: SIGCHI Conference on Human Factors in Computing Systems (CHI). ACM Press, Seattle, Washington, United States, SIGCHI Conference on Human Factors in Computing Systems (CHI), pp 301-308, https://doi.org/10.1145/365024.365119

Wang Y, Stanton D, Zhang Y, et al (2018) Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis. In: International Conference on Machine Learning (ICML). PMLR, International Conference on Machine Learning (ICML), pp 5180-5189

Yoon Y, Cha B, Lee JH, et al (2020) Speech gesture generation from the trimodal context of text, audio, and speaker identity. ACM Trans Graph 39(6). https://doi.org/10.1145/3414685.3417838
Yu C, Schermerhorn P, Scheutz M (2012) Adaptive eye gaze patterns in interactions with human and artificial agents. ACM Transactions on Interactive Intelligent Systems (TiiS) 1(2):1–25

Zaraki A, Mazzei D, Giuliani M, et al (2014) Designing and Evaluating a Social Gaze-Control System for a Humanoid Robot. IEEE Transactions on Human-Machine Systems 44(2):157–168. https://doi.org/10.1109/THMS.2014.2303083

<figure><img src="image_5.png" /><figcaption>**Fig. 3** Histogram distribution of the pilot's median head yaw angle (top figure) and median gaze angle (bottom figure) depending on his addressee(s). The left axis reports the number of occurrences and the right axis the probability distribution of the Gaussian approximation.</figcaption></figure>
<figure><img src="image_6.png" /><figcaption>**Fig. 4** Diagram representing the architecture of the proposed model (Haefflinger et al, 2024)</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>**Fig. 5** Results of the ablation study performed on the model's input features. Each barplot represents the average value of the 9-fold cross-validation results for one ablation study, with the error defined as standard deviation. Significant differences in performance with the model without ablation are shown in this way: \* pvalue < 0.05, \*\* pvalue < 0.01.</figcaption></figure>
_figures/eval_study_2.png

**Fig. 6** Example of a page from the French evaluation site. The first paragraph reminds participants to focus on head movements. By clicking on the yellow button, the two videos on the left are launched simultaneously. After a few seconds, the answer buttons under the questions on the right unlock. At bottom right is the button for sending answers and continuing to the next page. To unlock it, the videos must be completed and all three questions answered.
<figure><img src="image_8.png" /><figcaption>**Fig. 7** Distribution of preference answers for each question asked during the subjective evaluation. Significant p-values are indicated as follows: \* *p* < 0.05, \*\* *p* < 0.01, \*\*\* *p* < 0.001</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>**Fig. 8** Head and gaze prediction by the proposed model and for different addressee biases imposed during a 10-second extract of a RoboTrio sequence. Biases are only specified during the pilot's utterances, represented on this graph by the gray areas.</figcaption></figure>
<figure><img src="image_10.png" /><figcaption>**Fig. 9** Example of an evaluation page from the website. The first box reminds the evaluator to focus on the robot's gaze and head movements. On the left is the video to evaluate, which is launched by clicking on it (instruction given before the list of questions). Markers A and B can be seen on either side of the robot, used to answer the two questions on the right of the screen. At the bottom right is the button for sending answers and going on to the next page, activated only after watching the video and answering the questions.</figcaption></figure>
<figure><img src="image_11.png" /><figcaption>**Fig. 10** Confusion matrix of responses concerning robot addressee(s) in relation to the video's addressee bias. The value in each cell represents the number of occurrences for each case. The diagonal reflects the agreement between the evaluator's response and the given bias of the evaluated video.</figcaption></figure>