[image] [image] Ressource protégée - Inria - Institut national de recherche en sciences et technologies du numérique Accéder directement au contenu

[image] [image]

* Documentation

* FR
  Français (FR)

* Anglais (EN)
  Se connecter

* Portail HAL Inria

* Recherche
  Loading...

## Recherche avancée

Information de documents

Titres

○ Titres

○ Sous-titre

○ Titre de l'ouvrage

○ Titre du volume (Série)

○ Champ de recherche par défaut (multicritères) + texte intégral des PDF

○ Résumé

○ Texte intégral indexé des documents PDF

○ Mots-clés

○ Type de document

○ Sous-type de document

○ Tous les identifiants du document

○ Identifiant HAL du dépôt

○ Langue du document (texte)

○ Pays (Texte)

○ Ville

○ À paraître (true ou false)

Ajouter
<figure><img src="image_1.png" /></figure>

HAL
open science

# Attention Sharing Handling Through Projection Capability Within Human-Robot Collaboration

Benjamin Camblor, David Daney, Lucas Joseph, Jean-Marc Salotti

<figure><img src="image_2.png" /></figure>

## To cite this version:

Benjamin Camblor, David Daney, Lucas Joseph, Jean-Marc Salotti. Attention Sharing Handling Through Projection Capability Within Human-Robot Collaboration. International Journal of Social Robotics, 2024, 10.1007/s12369-024-01101-9. hal-04399124

HAL Id: hal-04399124

https://inria.hal.science/hal-04399124v1

Submitted on 17 Jan 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
## Auteur

### Auteur (multicritères)

○ Auteur (multicritères)

○ Auteur : Nom complet

○ Auteur : Nom de famille

○ Auteur : Prénom

○ Auteur : Complément de nom, deuxième prénom

○ Auteur : Organisme payeur

○ Auteur : IdHal (chaîne de caractères)

○ Auteur : Fonction

○ Auteur : personID (entier)

○ Auteur : Identifiant de l'organisme payeur

○ Auteur : Identifiant de la structure

○ Directeur de thèse

○ Éditeur

○ Éditeur scientifique

○ Éditeur de la série

## Ajouter

### Structure

### Structure (multicritères)

○ Structure (multicritères)

○ Structure : Acronyme

○ Structure : Nom

○ Structure : Code

○ Structure : Pays

○ Structure : Type

○ Structure : État dans le référentiel

○ Structure : Identifiant HAL de la structure

○ Structure/équipe de recherche : Pays

○ Structure/regroupement d'équipes : Pays

○ Structure/laboratoire : Pays

○ Structure/regroupement de laboratoires : Pays

○ Structure/institution : Pays

○ Structure/regroupement d'institutions : Pays

## Ajouter
# Attention Sharing Handling Through Projection Capability Within Human-Robot Collaboration

Benjamin Camblor¹*, David Daney¹, Lucas Joseph¹, Jean-Marc Salotti¹,²

¹*Auctus, Inria, 200 Avenue de la Vieille Tour, Talence, 33405, France.

²Bordeaux INP, Avenue des Facultes, Talence, 33400, France.

*Corresponding author(s). E-mail(s): benjamin.camblor@inria.fr;

Contributing authors: david.daney@inria.fr; lucas.joseph@inria.fr;

jean-marc.salotti@ensc.fr;

## Abstract

The link between situation awareness (SA) and the distribution of human attention, has been explored within a human robot collaboration framework. According to Endsley [19], SA is divided into three levels: perception, comprehension and projection. It is involved in the process of making decisions and carrying out actions in a dynamic environment. This work investigates three hypotheses. First, that the ability to project a robot's future actions improves performance in a collaborative task. Second, that the more participants are involved in tasks in a collaborative environment, the better their SA will be. Finally, that the use of a robot's non-verbal communication motions attracts a participant's attention more promptly than if the robot remains motionless. A within-participants study has been designed to investigate our three hypotheses. Participants were asked to perform a collaborative task with a robot. It required them to assist the robot at different moments while they were engaged in a distracting task that was catching their attention (tower of Hanoi puzzle). These moments could either be anticipated and taken into account in the human decision-making and action loop or not. Lastly, the robot could either use non-verbal communication gestures to draw human attention or not. The results have demonstrated the significance of considering the human capability to project a robot next actions in their own personal attention management. Moreover, the subjective measures showed no difference in the assessment of SA, in contrast to the objective measures, which are in line with our second hypothesis. Finally, it seems that standing stationary can be considered a gesture of non-verbal communication. In the present work, robot waiting was more salient in capturing human attention when the robot remained motionless rather than making a signaling motion.

**Keywords:** Human Robot Interaction; Situation Awareness; Projection; Attention; Non-Verbal Communication; Collaborative Robot

## 1 Introduction

Collaborative robots, or cobots offer a compelling solution for executing tasks that, when performed by humans alone, may encounter issues related to ergonomics, accessibility, effort, precision or

potential hazards, and which cannot be easily automated [13]. The introduction of collaborative robots means that industrial robots are moving out of their cages allowing direct interaction with human operators [58]. Historically, robotics research focused on replacing human labor with
Autres

Champ de recherche par défaut (multicritères)

○ <u>Champ de recherche par défaut (multicritères)</u>

○

○ <u>Revues (multicritères)</u>

○ <u>Revue : Éditeur</u>

○ <u>Revue : Titre abrégé</u>

○ <u>Revue : Titre</u>

○ <u>Revue : Identifiant interne</u>

○ <u>Revue : Date de début de publication</u>

○ <u>Revue : ISSN électronique</u>

○ <u>Revue : ISSN</u>

○ <u>Revue : Éditeur</u>

○ <u>Revue : Couleur dans SHERPA/RoMEO</u>

○ <u>Revue : État dans le référentiel</u>

○

○ <u>Colloque (multicritères)</u>

○ <u>Colloque : Titre</u>

○ <u>Colloque : Organisateur</u>

○ <u>Colloque : date de début (Année)</u>

○ <u>Colloque : date de fin (Année)</u>

○ <u>Conférence invitée (oui/non)</u>

○

○ <u>Projets ANR (multicritères)</u>

○ <u>Projet ANR : Acronyme</u>

○ <u>Projet ANR : Acronyme du programme</u>

○ <u>Projet ANR : Code décision (référence)</u>

○ <u>Projet ANR : Nom</u>

○ <u>Projet ANR : Identifiant interne</u>

○ <u>Projet ANR : État dans le référentiel</u>

○

○ <u>Projets européens (multicritères)</u>

○ <u>Projet européen : Acronyme</u>

○ <u>Projet européen : Identifiant de l'appel à projet</u>

○ <u>Projet européen : Référence</u>

○ <u>Projet européen : Nom</u>

○ <u>Projet européen : Date de fin</u>

○ <u>Projet européen : Financement</u>

○ <u>Projet européen : Date de début</u>
robotic work. However, the emergence of col-
laborative robotics requires a focus on robots
capability to collaborate with humans from both
cognitive [59] and physical [28] standpoints.

Fluidity and efficiency of Human-Robot Col-
laboration (HRC) must therefore be designed to
be analogous to collaboration between humans
[31]. Drawing inspiration from the concept of joint
action between two humans [12], robots should
be endowed with capabilities that enable them to
dynamically coordinate and adapt their behavior
in real time [32]. One of the benefits of this would
be to ensure the safety of interacting humans. Tak-
ing into account the Situation Awareness (SA) of
the human partner is one of the ways of guar-
anteeing safe interaction. SA has been defined as
"the perception of the elements in the environment
within a volume of time and space, the comprehen-
sion of their meaning and the projection of their
status in the near future" [19]. According to End-
sley [20], it can be divided in three consecutive
stages: perception, comprehension and projection.
Regarding the cognitive aspects of the human that
a robot can support, Endsley [19] highlights that
attention is a major limitation to having good
SA in a dynamic system, indicating the impor-
tance of being permanently attentive, particularly
in complex decision-making.

<figure><img src="image_4.png" /><figcaption>**Fig. 1** Endsley three levels of situation awareness model [19].</figcaption></figure>

Based on Endsley [20], the first step in build-
ing a good SA is perception (SA Level 1) of the
status, attributes and dynamics of relevant ele-
ments in the environment. Perception may involve
sensory modalities. Verbal and non-verbal com-
munication is an additional source of information
that contributes to level 1 of SA. The second stage
corresponds to the comprehension (SA Level 2)
of the data and the meaning of the relevant cues
perceived in relation to the predefined goals and
objectives. It is based on a synthesis of the disor-
dered elements of Level 1 and a comparison of this

information with the individual's objectives. The
final stage is projection (SA Level 3). The individ-
ual must be able to predict, at least in the short
term, the evolution of elements after knowledge
and determination of their meaning in relation
to the current objective. To reach SA level 3, an
individual needs to have a good understanding
of the situation and the dynamics of the sys-
tem with which they interact. This level enables
proactive decision-making. By constantly project-
ing into the future, an expert is able to develop a
set of strategies and responses to events [20].

Prior research has extensively examined vari-
ous aspects of SA within robotics [9, 36], includ-
ing focus on collaborative robotics [6, 27, 34,
48]. Effective coordination among team members
engaged in a joint activity requires setting expec-
tations of future actions [37]. However, to the best
of our knowledge, there is no previous work that
takes into account the projection capabilities, i.e.
level 3 of SA, of the human on the task and on the
state of the environment in the interaction loop.
Furthermore, only a few studies have looked at
how a robotic system [3, 14, 24, 54] or a collab-
orative robot alone [6] can communicate with a
human partner to enhance their SA. In particular,
this previous work demonstrated that task-related
non-verbal communication motions could improve
the short-term projection capability of people
interacting with a cobot.

In this work, the link between the third level of
SA, i.e. projection, and the distribution of human
attention within a HRC framework is investi-
gated. Non-verbal cobot communication gestures
[6] in signaling the expectation of human assis-
tance during collaborative tasks is also studied.
Three hypotheses are formulated on the basis of
these two research questions and the literature:

* **H1:** *The capability to project a robot subsequent action enhances performance in collaborative tasks.*

This hypothesis is based on our previous work in which performance on the cooperation task was better when the robot supported the participant ability to project [6].

This hypothesis is based on our previous work in
which performance on the cooperation task was
better when the robot supported the participant
ability to project [6].

* **H2:** *Individuals have a better SA when they are involved in a greater number of tasks.*

This hypothesis is based on the work of Hopko et al. [34], who showed that increased robotic assistance reduced a person SA. In the present

This hypothesis is based on the work of Hopko
et al. [34], who showed that increased robotic
assistance reduced a person SA. In the present
○ Projet européen : État dans le référentiel

○ Projet européen : Identifiant interne

○

○ Date de production : année

○ Date de mise en ligne : année

○ Date de publication : année

○ Date d'écriture : année

○ Date de modification du dépôt : année

○ Date de dépôt : année

○ Date de publication électronique : année

○

○ Collection HAL (multicritères)

○ Collection HAL : catégorie

○ Collection HAL : Code

○ Collection HAL : Nom

○ Collection HAL : Identifiant interne

○

○ Identifiant interne du contributeur/déposant

○ Nom complet du contributeur/déposant

○

○ Domaines

○ Domaine primaire

○ Domaine racine

○ Sous-domaine niveau 1

○ Sous-domaine niveau 2

○ Sous-domaine niveau 3

○

○ Statut du document

○ Version du document

○ Type de dépôt

○ Type de document

○ ISBN

○ Numéro - référence

○ Identifiant DOI

○

○ Classification

○ Audience

○ Vulgarisation

○ Comité de lecture - texte (oui ou non)

○ Actes de colloque
work, it was supposed that involving the human
in more tasks would reintroduce them into the
loop, and therefore their SA would be better.

* H3: *The attention of a human is more promptly captured by a robot when it is in motion than when it is inactive.*

This hypothesis is also linked to our previous
work, in which task-related non-verbal commu-
nication motions improved operators SA.

2 Related Works

**2.1 Human-Robot Collaboration**

Human-robot collaboration is a vast field that cov-
ers a wide range of research topics. Numerous
works focus on endowing robots projection capa-
bilities, grounded in either their own trajectory [1,
40] or predicated on the behavior of their human
partners [40, 60] or their physical characteristics
[46, 52]. Other studies have focused on address-
ing human factors such as stress [44], anxiety [39],
cognitive workload [25, 53], risk perception [38] or
trust [10, 53].

Concurrently, other studies focus on more the-
oretical constructs such as joint action [11, 37]
and theory of mind [16]. Joint action is said to
be successful when the agents, whether a team of
humans or a human-robot team, coordinate their
intentions and actions [11]. For this purpose, both
verbal and non-verbal communication between the
different agents are required to coordinate team-
work and thus accomplish a shared goal [37].
It has been argued that non-verbal communica-
tion renders the robot more understandable and
transparent to people in interaction with it[5].

Within the scope of human-robot interaction,
transparency is defined as the observability and
predictability of system behavior: the understand-
ing of what the system is doing, why and what it
will do next [21]. It has been suggested that trans-
parency is one mechanism to facilitate effective
interactions between humans and their robotic
teammates [41]. It was further concluded that
automation transparency has a positive effect on
operator SA and performance [57]. According to
Alonso and De La Puente [2], transparency can be
considered as a means of achieving shared SA and
mutual intentions between human beings and the
system.

To this end, Chen et al. [8, 9] presented
the SA-based agent transparency model, designed
to assist operators in human-agent teams. This
model comprises three levels helping an operator
to understand the reasoning process underlying
the action and assisting them in their decision-
making.

## 2.2 Situation Awareness and Human-Robot Collaboration

SA is an internal cognitive process involved in
decision-making and prior to the execution of an
action in a dynamic environment [20]. Inherently,
SA mainly depends on mental representations of
the situation [19]. Making a decision and per-
forming an action under degraded SA often has
negative outcomes in terms of safety and perfor-
mance. In most cases, this is due to a perception
issue and more specifically, to a poor allocation
of attention during the task [7]. It has been
shown that taking into account an operator cogni-
tive availability is beneficial for Search & Rescue
scenarios [26].

SA has not been widely studied within the
robotics literature. Regarding automation effects,
studies have indicated that intermediate automa-
tion facilitate higher SA without necessarily
improving performance or diminishing workload
[36]. Gombolay et al. [27] highlighted that human
participants’ awareness of their team’s actions
decreased as the degree of robot autonomy
increased. In terms of physiological metrics, it was
observed through heart rate variability measures
that increased robot assistance reduced the atten-
tional supply and task engagement of participants
[34]. Finally, in order to maintain human SA, it
has also been suggested to endow robot with a
lower level of autonomy than they are actually
capable of [48]. However, one of the major chal-
lenge remains in the difficulty of assessing SA in
real time [35].

**2.3 Situation Awareness Assessment**

According to the literature, two methods are com-
monly employed to assess SA [23]. The first is
the Situation Awareness Global Assessment Tech-
nique (SAGAT), an objective method involving a
pre-prepared questionnaire relevant to the task at
hand [18]. The second is the Situation Awareness
○ Référence interne

○ Financement

○ Collaborations

Ajouter

Lancer la recherche

Recherche experte (SolR)

## Recherche experte (SolR)

Lancer la recherche
Vers la recherche avancée

• Déposer

Portail INRIA.HAL.SCIENCE
Les publications, logiciels... des scientifiques Inria

• Accueil

• Publications Inria

○ Consulter les derniers dépôts

○ Par équipe de recherche

○ Par auteur

○ Rapports de recherche et techniques

○ Livres blancs

○ Rapports d'activité des équipes

○ Statistiques

• Boite à Outils

○ X2Hal : importer par lot

○ Logiciels : créer le Codemeta.json

○ Créer mon IDHal

○ Créer mon CVHal

○ Haltools : créer ma page de publications

○ Haltools : extraire des données (réservé Inria)

○ Les bonnes pratiques

■ Comment bien s'affilier ?
Rating Technique (SART), a subjective evaluation method also reliant on a questionnaire [55]. However, these methods introduce an ecological discrepancy by necessitating task interruption for assessment [22, 49].

Another option currently being explored in the literature is to use behavioral [42] and physiological [4] measurements including eye-tracking, cardiovascular activity (HR and HRV), electroencephalography (EEG) and electrodermal activity (EDA) [61]. In their systematic review about physiological measurements of SA, Zhang et al. [61] have identified 16 relevant studies incorporating eye-tracking, 6 associated SA with indirect cardiovascular activity, 3 used measurements related to EEG and one using EDA. All the studies included in the review also performed SA direct assessments. They accounted the use of SAGAT in 14 studies and SART in 10. A positive correlation between SA direct assessments and SA assessed through eye-tracking measures has been demonstrated with SART [30, 45] and SAGAT [15, 43, 45].

Finally, eye-tracking has been emphasised to assess real-time SA using predictive algorithms involving Areas Of Interest (AOI), fixations time on task-relevant objects and other features in human-robot interaction [17]. The use of attention-related metrics therefore appears to be a promising way of managing and supporting human SA within HRC scenarios. Eye-tracking could be a valuable solution to detect whether an individual attention fixation on a specific task element might lead them to neglect essential information, potentially yielding an incomplete mental model of the main task [26].

## 2.4 Non-Verbal Communication and Situation Awareness

Effective joint action requires both verbal and non-verbal communication between the different agents to coordinate teamwork and thus accomplish shared goal [37]. Breazeal et al. [5] explored the impact of non-verbal cues and behavior on task performance by a human-robot team. It was argued that non-verbal communication makes the robot more understandable and transparent for the people interacting with it. Additionally, robot motions have been envisioned as a mean to

communicate robot states and actions [33] without overloading other communication channels. This avoids the need to add interfaces within the workspace or additional alarms that could leave the operator confused about their meaning [20].

In a previous work, it has been shown that the use of non-verbal communication motions had a positive impact on a person SA [6]. Two types of communication gestures were implemented using the robot's redundant degrees of freedom. One of the benefits of this method is that it enables the robot to perform tasks whilst communicating about its states at the same time. The cobot is no longer just a worker or a social being, but a combination of both. Within the industrial context, a robot's priority is to perform tasks. However, it never needs to use all its degrees of freedom to act. Unused cobot degrees of freedom can therefore be used to superimpose communication gestures onto the robot's task.

# 3 Methods

A within-participants study has been designed to investigate both of our research questions and the affiliated hypotheses. Participants were asked to perform a collaborative task with a cobot. Two orders of experimental runs were set up to avoid any potential bias and participants were assigned a random number, each associated with one of two orders.

## 3.1 Participants

Twenty-eight volunteers (10 females and 18 males) were enrolled for the experiment on a university campus using email lists and networking. They were aged between 18 and 38 ($M = 22.77, SD = 3.89$) and were not familiar with the task. Two participants were unable to complete the experiment and were not included in the analysis of results due to technical issues. The experiment was carried out with the agreement of an ethics committee. All participants completed a consent form to take part in the experiment and to use their data for the analysis of this work.

## 3.2 The Tasks

Two main tasks have to be completed by the human-robot team during the experiment. The
■ Quelle version puis-je déposer ?

■ Choisir le bon « type de document »

■ Mettre en valeur ses publications

○ Les tutos IES Inria

■ Lier les notices dans HAL

■ Umbrhal Inria pour l'accessibilité

■ Personnaliser son CVHAL

■ Créer et Personnaliser sa collection

• Documentation

○ Aide en ligne HAL

○ Aide en ligne Haltools

○ Aide en ligne X2hal

○ Tutoriels video HAL

○ Documentation API HAL

○ Ajouter des vignettes

○ Identifiant auteur IdHAL et CV

○ Compléter son dossier ORCID avec ses dépôts HAL

○ Inria et la Science Ouverte

X

X

X

Loading...

X

Le document que vous demandez est en access restreint
jusqu'au 2025-02-13 !

Revenir a la notice

Retourner à l'accueil

Contact

•

Ressources
aim of our experiment was to get the human and
the robot to collaborate on a task at times that
could or could not be anticipated, while the human
performed a cognitively-intensive task. Our objec-
tive was to design gamified tasks that were faithful
to those encountered within industrial environ-
ments.

3.2.1 Tower of Hanoi puzzle

The first task consists of the completion of mul-
tiple rounds of the Tower of Hanoi puzzle. The
Towers of Hanoi puzzle is widely recognized as a
test of executive functions, particularly problem-
solving ability [51]. By its very nature, this task
requires a high level of concentration and atten-
tional focus in order to be completed. It consists
of moving disks of different sizes from an initial
peg to a final peg, respecting two rules: (i) only
one disk can be moved at a time and, (ii) a larger
disk cannot be stacked on a smaller one.

This task is exclusively carried out by the par-
ticipant and is presented to them as the main
task. However, it is meant to draw the participant
attention away from the environment. To this end,
participants are instructed to engage in the Towers
of Hanoi puzzle repeatedly, ensuring a continuous
engagement throughout each experimental phase.
This approach is adopted to prevent a scenario
wherein participants swiftly acquire the solution
algorithm or become familiar with the task, leav-
ing them with idle time once the task is completed.
Thus, it is ensured that most of the participant
attention is focused on the Towers of Hanoi puzzle.

3.2.2 Collaborative building task

The second task consists to build three stacks of
four bricks each. This task is carried out jointly
by the robot and the participant. The role of the
robot is to build the stacks and the participant
has to assist or to monitor the robot. Assisting the
robot can be done in two ways:

*   **Hand-over** - If there are no more brick of the desired color in the robot stock, the participant must hand it one - from a storage box - by placing it in the square of the sharing zone (see Figure 3). The handing-over action cannot be anticipated, as the person does not have access to the model of the stacks the robot has to make. As a result, the human is taken out of the loop

of the robot task and its ability to project (SA
Level 3) is hampered. Handovers are random,
controlled and independent of the phase.

*   Removal - Once the robot has finished build-
    ing the three stacks of bricks, a screen mes-
    sage informing the participant is displayed (see
    Figure 4). Then, the participant has to remove
    the stacks from the tray and put the bricks
    away in the robot storage using the schema dis-
    played on the screen. Bricks not stored in the
    robot stockpile have to be stored in the box.
    The removal action can be anticipated because
    the number and size of the piles is always the
    same throughout the phases and is known by
    the participant. In this case, the SA loop is not
    hampered.

The robot adopts a waiting posture above the
brick tray while it waits for the human to inter-
vene. When the robot is waiting for the human to
intervene, it stands on top of the board with its
gripper turned slightly towards the human. When
it reaches its standby position, the robot begins to
communicate non-verbally about its waiting state.
It can either remain motionless (default behav-
ior) or perform a signaling motion as proposed by
Camblor et al. [6]. The motion symbolize breath-
ing and had been tested to express to participants
that the robot was active and could resume its
activity at any time.

The general given instructions is to complete
as many Tower of Hanoi puzzle as possible, but to
keep the robot waiting as short as possible. This
is presented as a performance constraint on suc-
cessful completion of the task as might be found
in an ecological environment.

Monitoring consists of checking that the robot
catches a brick of the color shown on the screen.
To ensure that she/he is supervising the robot
correctly, the participant is asked to verbally com-
municate the color displayed on the screen and the
color caught by the robot (example: green - green).

## 3.3 Materials and Experimental
set-up

We used a Pupil Core <fnref n="1" /> to collect eye-tracking
data and provide objective data relating to partic-
ipants attention. The two tasks were carried out

<footnote n="1">¹https://pupil-labs.com/products/core/</footnote>
* Documentation

* FAQ

* API

* OAI-PMH

* AuréHAL

## Informations

* À propos

* Données personnelles

* Mentions légales

* Accessibilité

* Conformité RGAA

## Questions juridiques

* Je publie, quels sont mes droits ?

* Loi pour une République numérique

* Stratégie de non-cession des droits

* SHERPA/RoMEO

## Portails

* Portails institutionnels

* HAL

* HAL SHS

* HAL Thèses

* MédiHAL

## CCSD

* CCSD

* Episciences

* Sciencesconf
<figure><img src="image_5.png" /><figcaption>**Fig. 2** Diagram of human and robot tasks during each phase. All arrows coming out of rectangular shapes propagate when the action is performed. Arrows coming out of diamond shapes only propagate if the condition is met. The process start at "Complete Hanoi" box for the human and "Stack bricks" for the robot.</figcaption></figure>

Human must complete the Tower of Hanoi puzzles unless the robot needs assistance. If this is the case, the human must either hand over a piece or remove the buildings, depending on whether or not the robot has completed its task. The robot has to stack bricks continuously until it needs human assistance. The robot remains on standby until the human has provided assistance. The robot can then resume its task.

<figure><img src="image_6.png" /><figcaption>**Fig. 3** Example of one sequence of hand-over. (1) Robot is waiting for assistance while Human completes Hanoi puzzle; (2) Human grabs a brick of the desired color (green) in the storage box; (3) Human places the brick in the sharing square; (4) Robot grabs the brick; (5) Robot stacks the brick on the current build; (6) Robot no longer needs assistance.</figcaption></figure>

in shared space. A Tower of Hanoi puzzle with three pegs and five disks is positioned on the left-hand side of the table. A screen is used to display information about the experiment and calibrate the eye-tracker. Throughout the experiment, the

<figure><img src="image_7.png" /><figcaption>**Fig. 4** Example of one sequence of removal. (1) Robot is waiting for assistance while Human completes Hanoi puzzle; (2) Human removes buildings from the trail; (3) Human stores the bricks in stocks; (4) Human stores the remaining bricks in the storage box.</figcaption></figure>

screen displays the brick the robot wishes to stack. The screen was positioned so that the operator had to look up to see it. In addition, the brightness and image display were kept to a minimum, to avoid participants being flashed during their task. The display screen changed each time the robot placed a brick, announcing the next desired brick.

A Panda cobot (Franka Emika) and the tray on which it built the towers of bricks is located on the right-hand side of the table. The bricks that the robot manipulated is placed in the center of the table, grouped by color. A squared-zone for
hand-over is located in the center of the different colored bricks. A storage box is also positioned in the center of the table to stock the additional bricks after a removal (see Fig 5).

<figure><img src="image_8.png" /><figcaption>**Fig. 5** Experimental set-up. (a) Participant and Hanoi area in red zone; (b) screen in cyan zone; (c) cobot in white zone; (d) trail in blue zone; (e) stockpile and square for sharing in grey zone; (f) storage box in green.</figcaption></figure>

A noise-cancelling headphone is used to ensure that participants cannot hear and use the noise of the robot motors as a source of information. In particular, this enables us to ensure that it is indeed the robot motions that catch participants attention and not the noise, or lack of noise, of the robot. Participants wore the noise-canceling headphone throughout the phases, removing it only after the second removal of each phase (i.e. the end of the phase). This allowed the experimenter to present the instructions for the next phase and ask the participant to complete the questionnaires.

Two questionnaires have to be filled by the participants at the end of each phase of the experiment:

* The Situation Awareness Rating Technique (SART) [55] that assesses SA subjectively and is completed after the task. It consists of 10 questions asking participants to assess three dimensions of the cognitive process involved in the task: attentional demand, attentional reserve and comprehension. Each question is evaluated using a Likert scale with seven levels of precision, and an overall SA score is then calculated.

* The RAW-TLX [29] to measure participants workload. It consists of six questions in which participants rate their perceived workload using a Likert scale with 21 levels of precision. The

first three questions assess task constraints: mental demand, physical demand and time pressure. The last three focus on the user capabilities: performance, effort and frustration. The use of RAW-TLX is another source of help to trick the participant about the true nature of the experiment.

Finally, two computers are used to set up the experiment. The first is used to launch the experiment, administer the questionnaires and record the eye-tracking data and the robot recordings. The second computer is linked to the first and is used to control the robot. These computers are placed behind the subject on the experimenter desk. For technical reasons and to carry out the experiment, the experimenter have to be present in the room. Finally, for safety reasons, an emergency stop button for the robot is placed on the experimenter desk.

## 3.4 Procedure

The experimenter welcomes the participant and explains what she/he would have to do during the experimental session. The phases are presented in such a way that the participants believes that the experimenter is going to assess their ability to complete the Towers of Hanoi puzzle under different workload conditions. The participant then fills in a consent form. Next, the experimenter fits the eye-tracker to the participant face and adjusts the pupil cameras. Participants do not know what the eye-tracker is measures truly. The experimenter only tells them that it allows to check that they are correctly completing Tower of Hanoi puzzle.

The experiment is divided into three distinct phases and a training phase. Each phase begins with a re-explanation of the instructions to ensure that the participant has correctly understood the tasks to be carried out. The experimenter calibrates the eye-tracker after the explanations and the evaluation then begins.

During the training phase, participants have to complete as many 5-disks towers of Hanoi as possible within 7 minutes. This duration was determined during a pre-test of the experiment and is consistent with the duration of the other phases. This phase is used for training and to check the quality of eye-tracker measurements. The robot remains inactive throughout this phase. During the subsequent phases, participants are
still required to complete as many towers of Hanoi
as possible.

From the first phase, the robot carries out
the building task. During this phase, participants
have to both supervise and assist the robot whilst
playing Tower of Hanoi puzzle. In total for this
phase and the following ones, the robot builds
two times three towers of four bricks each. At the
end of this and subsequent phases, participants
are asked to complete both SART and RAW-TLX
questionnaires.

During the second and third phases, the partic-
ipants only have to assist the robot. The difference
between these two phases lies in the way the
robot waits for the human to intervene. The cobot
can either wait for the human to intervene by
remaining motionless or by performing a breath-
ing motion. The order of these two phases depends
on the order initially assigned to the participant:
if she/he belongs to order 1, the robot performs
the breathing motion in phase 2 and remains
motionless in phase 3, and vice versa for order 2.

There are two handovers and two removals per
phase. The measurements are interrupted between
the first removal and the resumption of the robot
once the participants have correctly replaced all
the bricks.

**3.5 Independent Variables**

Three independent variables are manipulated dur-
ing the experiment. Order, which is an between-
participants variable, and Modality and Action,
which are within-participants variables.

**3.5.1 The Order**

*The Order* {*Order 1*, *Order 2*}. This variable is
used to attenuate the order effects. The counter-
balancing was carried out before the experimental
runs by associating random numbers with one of
the two orders. A random number is then drawn
by the participant at the start of their passage.

**3.5.2 The Modality**

*The Modality* {*Monitoring, Breathing, No Breathing*}. This variable corresponds to the name of the phases 1, 2 and 3, according to the participants order presented in the Procedure section.

The Monitoring modality allows us to evaluate
the participants attention in a situation where the

instruction was to pay attention to more things in
the environment. This allowed us to have a refer-
ence level for each participant in the event that
they were efficient and attentive in the following
phases.

The Breathing modality enables us to check
whether the use of a non-verbal communication
motion has a positive influence on participants SA
and attention. The No Breathing modality is used
as a reference for the Breathing modality to assess
the effect of the motion.

**3.5.3 The Action**

*The Action* {*Hand-over, Removal*}. This variable
corresponds to the action the participant has to
perform when the robot is waiting for an inter-
vention. For each phase, there are two hand-overs
and two removals.

The Hand-over action allows us to evaluate the
case where the participant cannot project the next
action of the Robot. The removal action is used
as a reference to assess the effect of projection
because it can be predicted by the participant.

**3.6 Measures**

Objective and subjective data are recorded dur-
ing the experiment from the Monitoring phase
onwards. Objective data is obtained using logs
retrieved by the robot during each phase and using
the eye-tracking device. Subjective data are col-
lected using questionnaires completed after each
phase.

**Robot waiting time** - This measure can be expressed in two ways, either as the waiting time for a handing or the waiting time for a removal. It is calculated by taking the difference between the time at which the robot goes into waiting mode and the time at which the human intervenes. Since there are two hand-overs and two removals per phase, the waiting times for both have been summed up independently.

**The gaze rate in each surface** - The workspace was separated into two zones of interest. The first comprised the entire left side of the set-up (a and b) in figure 3. The second included the entire right-hand side of the set-up (c to f). The gaze rate is obtained by calculating the frequency of observation of each of the two surfaces. The two values are complementary and their sums are equal to 1.
**SART score** - It is calculated from the participants answers using the method proposed by Taylor [55].

## 3.7 Hypothesis

In line with the assumptions defined above, we define four operational hypotheses. Hypothesis H2 is divided into two sub-hypotheses (H2.1 and H2.2). The first is tested using the gaze rate measure and the second using SART scores. In this way, hypothesis H2 is tested from both an objective and subjective point of view.

* **H1:** The robot waiting time will be shorter when the participant has to make a removal than when she/he has to make a hand-over, whatever the phase.

* **H2.1:** Participants will have a higher gaze rate in the robot area during the Monitoring phase compared to the No Breathing and Breathing phases.

* **H2.2:** Participants will score higher on the SART questionnaire after the Monitoring phase compared to the Breathing and No Breathing phases. In line with H2.1, participants are expected to score higher on the SART in the Breathing phase than in the No Breathing phase.

* **H3:** The robot waiting time will be shorter if it waits while breathing rather than while standing motionless. More specifically, lower times are expected in the Breathing phase compared with the No Breathing phase.

# 4 Results

All results were considered significant at the 95% confidence level (i.e. a p value < .05), and Rstudio software (version R 4.3.1) was used to carry out the statistical analysis. For each statistical test, we first verified the normality condition using a Shapiro-Wilk test. The distributions followed a normal distribution for the score on the SART questionnaire ($p = .535$), but not for the robot waiting times ($p < .001$) or for the gaze rate ($p < .001$). The homogeneity of the variances on the SART scores was also verified using a Levene test ($p = .988$). For robot waiting times and gaze rates, Friedman tests according to Action or Modality were carried out to analyze the results. As there is no non-parametric alternative to ANOVA with

at least two factors, we performed our calculations using the parametric version of ANOVA to analyze waiting times by Modality and Action. This enabled us to study whether there was an interaction effect between Modality and action.

## 4.1 Order effect

The running Order effect was examined despite counterbalancing. To do this, 2 x 3 mixed-design ANOVAs were run with the Order (between-subjects factor) and Modality (within-subjects factor). Sphericity was tested for each of our dependent variables. All sphericity tests were significant ($p < .05$) meaning that the condition was not met. A Greenhouse-Geisser correction was applied for each ANOVA. No Order effect was found for the robot waiting time ($F(1.17, 27.97) = 0.155, p = .735$), the gaze rate for human surface ($F(1.2, 28.77) = 0.310, p = .623$), the gaze rate for robot surface ($F(1.2, 28.77) = 0.310, p = .623$), and the SART score ($F(1.56, 37.54) = 1.955, p = .164$). The hypothesis of an effect of Order on our different measures is therefore not supported. In the rest of the paper, the data will be compared without taking the Order variable into account.

## 4.2 Robot waiting time

To test hypotheses H1 and H3, we performed a 3 x 2 repeated measures ANOVA to study the effect of Action (H1) and Modality (H3) on robot waiting time. The ANOVA did not show any interaction between the Modality and the Action on the time ($F(2, 50) = 2.426, p = .0987$). As there was no interaction effect between Modality and Action, the results were further analyzed separately using non-parametric tests.

A Wilcoxon signed-rank test revealed that the waiting times for a hand-over were longer (Mdn = 16.91) than for a removal (Mdn = 4.29), $Z = 14.2, p < .001$. The results support hypothesis H1. Regardless of the experimental Modality, the participants generally made the robot wait less when it came to removing the bricks (see fig 6).

A Friedman test has been calculated to evaluate the effect of Modality on the robot waiting time for a hand-over and revealead significant differences ($\chi^2(2) = 10.2, p < .01$). Post hoc multiple comparisons using Bonferroni correction further showed that robot waiting times were shorter for
<figure><img src="image_9.png" /><figcaption>**Fig. 6** Boxplot of the measured waiting times for each Action. Hand-over on the left in grey and Removal on the right in orange. The average waiting time for hand-overs was 21.84s (SD = 15.48) and 6.68s (SD = 10.58) for removals. The median of hand-overs was 16.91s and 4.29s for removals. Waiting times for hand-overs ranged from 7.45s to 98.78s and ranged from 2.54s to 92.37s for removals.</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>**Fig. 7** Boxplot of the measured waiting times for hand-over for each Modality. Monitoring on the left in orange, No Breathing in the centre in blue and Breathing on the right in grey.</figcaption></figure>

hand-over during the No Breathing phase (Mdn = 13.9) than during the Monitoring phase (Mdn = 19.0), Z = 2.95, *p* < .01 and the Breathing phase (Mdn = 18.6), Z = 4.51, *p* < .01. On the other hand, the Monitoring and Breathing modalities did not differ, Z = 1.02, *p* = 1.00 (see fig 7).

<figure><img src="image_11.png" /><figcaption>**Fig. 8** Boxplot of the measured waiting times for removal for each Modality. Monitoring on the left in orange, No Breathing in the centre in blue and Breathing on the right in grey.</figcaption></figure>

A Friedman test to assess the effect of Modality on the robot waiting time for a removal was also calculated. The results of the tests did not show any significant differences between the modalities for the time spent waiting for a removal ($\chi^2(2) = 1.46, p = .482$) (see fig 8).

These two results regarding the effect of Modality on robot waiting time, whether the action is a hand-over or a removal, contradict our hypothesis H3.

## 4.3 Gaze rate

To test hypothesis H2.1, we performed a Wilcoxon signed-rank test to evaluate differences between gaze rates on human and robot surface. The test revealed significant differences ($Z = 3081, p < .001$) between the gaze on human surface (Mdn = 0.939) and robot surface (Mdn = 0.061).

Friedman tests to evaluate the effect of the Modality on the gaze rates on both human and robot surface have been calculated. The test revealed a significant effect for the human ($\chi^2(2) = 21, p < .001$) and the robot ($\chi^2(2) = 21, p < .001$) surface. Post hoc multiple comparisons using Bonferroni correction showed that the participants gaze rates were higher on their surface in the Breathing (Mdn = 0.953) and No Breathing (Mdn = 0.958) phases than in the Monitoring phase (Mdn = 0.898), respectively $Z = 0.048, p < .001$ and $Z = -0.053, p < .001$. Alternatively,
<figure><img src="image_12.png" /><figcaption>**Fig. 9** Boxplot of the measured gaze rates within Human surface for each Modality. Monitoring on the left in orange, No Breathing in the centre in blue and Breathing on the right in grey.</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>**Fig. 10** Boxplot of the measured gaze rates within Robot surface for each Modality. Monitoring on the left in orange, No Breathing in the centre in blue and Breathing on the right in grey.</figcaption></figure>

gaze rates were higher on the robot surface in the Monitoring phase (Mdn = 0.102) compared to the Breathing (Mdn = 0.047) and No Breathing (Mdn = 0.042) phases, respectively $Z = -0.048, p < .001$ and $Z = 0.053, p < .001$. Finally, there was no significant interaction effect on the gaze rate between the Breathing and No Breathing modalities for the human surface, $Z = -0.004, p = 1.00$, and the robot surface, $Z = 0.004, p = 1.00$.

Taken together, these results show that hypothesis H2.1 is supported.

<figure><img src="image_14.png" /><figcaption>**Fig. 11** Boxplot of the calculated SART scores for each Modality. Monitoring on the left in orange, No Breathing in the centre in blue and Breathing on the right in grey.</figcaption></figure>

## 4.4 SART questionnaire

To test hypothesis H2.2, we ran a repeated measures ANOVA and found no statistically significant differences of the modalities on SART Score ($F(2, 50) = 1.825, p = 0.172$). These results do not support the hypothesis of better SA in the Monitoring (M = 20.9, SD = 6.43) phase rather than in the No Breathing (M = 23.3, SD = 6.55) or Breathing (M = 22.6, SD = 6.48) phase and even between the latter two (see fig 11).

The correlation between the SART and our other dependent variables was also measured. We found no significant correlation between the SART score and hand-over actions ($p > 0.05$), removal actions ($p > 0.05$) and gaze rates ($p > 0.05$) (see tab 1).

These results do not support the hypothesis H2.2 which assumed that participants would assign themselves a better SA when they were more involved in the task.

<table><caption>**Table 1** Correlation between the SART and the other objective measures.</caption><thead><tr><th></th><th>Handover</th><th>Removal</th><th>Rate</th><th>SART</th></tr></thead><tbody><tr><th>Handover</th><td>1.00</td><td></td><td></td><td></td></tr><tr><th>Removal</th><td>0.40 ***</td><td>1.00</td><td></td><td></td></tr><tr><th>Rate</th><td>0.03</td><td>0.07</td><td>1.00</td><td></td></tr><tr><th>SART</th><td><strong>-0.14</strong></td><td><strong>-0.07</strong></td><td><strong>-0.04</strong></td><td><strong>1.00</strong></td></tr></tbody></table>
# 5 Discussion

The aim of this study was to investigate the link between the third level of SA, i.e. projection, and the distribution of human attention within a HRC framework. In addition, it was proposed to re-investigate a non-verbal communication gesture symbolizing breathing successfully tested in Camblor et al. [6] within direct HRC. Based on these two scientific objectives, three hypotheses have been proposed:

* **H1:** The capability to project a robot subsequent action enhances performance in collaborative tasks.

* **H2:** Individuals have a better SA when they are involved in a greater number of tasks.

* **H3:** The attention of a human is more promptly captured by a robot when it is in motion than when it is inactive.

Once operationalized, hypothesis H2 has been divided into two sub-hypotheses (H2.1 and H2.2). The first was tested using objective measures, and the second using subjective measures.

The analysis of the results supports H1 and H2.1. However, H2.2 and H3 are not supported by our results. More importantly, our results supported the opposite of H3: the robot would capture participants attention more quickly by standing stationary rather than moving when it was waiting for assistance. In this section, our results are discussed under each of these hypotheses.

## 5.1 Situation Awareness projection and performance

In line with H1, the results showed an effect of the type of action on the robot waiting time. In general, the participants made the robot wait longer when it had to do a hand-over rather than a removal. We believe the reason lies in the capability to anticipate or not the two actions. Generally speaking, participants made the robot wait almost three times as long when they had to perform a hand-over (3.34s for the removal and 10.92s for the hand-over) regardless of the modality.

It can be explained by the fact that the hand-over could only be partially anticipated. The participant could expect to have to give a brick to the robot (i.e. one of the stocks of a colour is empty) but this was not guaranteed. Indeed, there

was no way of knowing whether the first missing color will be the one requested by the robot, or when the robot would need it. For these reasons, it was ensured that the participants projection would be hampered.

Conversely, the removal action could be anticipated at any time during the phase. Whatever happened, this action was only requested once the robot had completed its task. The determinism of this action meant that the participant only needed a glance to check the progress of the robot task.

The fact that there were only differences in reaction times between modalities for the hand-over action have reinforced the importance of participants ability to project. Regardless of the communication modality and the amount of attention paid to the robot (Monitoring phase), the participants always took very little time to remove the robot constructions. Conversely, for the hand-over, the communication method does not seem to have influenced the waiting time and therefore the lack of projection. This underlines the importance of supporting the complete SA process of people in collaboration with a robot.

## 5.2 Situation Awareness evaluation

Regarding the data on gaze rates, these were consistent with the fact that participants complied with the instructions. Our results have shown a validation of H2.1 and independently of the robot waiting time, an appropriate general behavior of the participants. An interesting indicator for the evaluation of SA would have been to find any correlation between the distribution of participants gaze and the robot waiting time. It could be expected that a participant who often pays attention to the robot would have shorter reaction times and would therefore make the robot wait less. However, the results presented in table 1 seem to suggest that there is no correlation between gaze rates and waiting times.

Contrary to hypothesis H2.2, participants did not assess they had better SA in the first phase of the experiment. This result may seem surprising according to the literature [27, 34, 36, 48], given that the additional instruction in the monitoring modality was to watch the robot and pay as much attention to it as possible. This required them to share their attention significantly. By doing so, they should had greater SA, as the perception
stage of the SA loop was improved. This result
was also supported by the fact that no significant
correlation has been found between the SART and
our objective measures.

This result might then be linked to an incorrect
assessment of the SA. The SART is an often-
criticized questionnaire for assessing SA. SAGAT
is more often recommended as a measure of SA
[23]. Its nature makes it an objective rather than
a subjective measure. On the other hand, one of
the negative points of this questionnaire is that
it requires to freeze the task. This was at odds
with the aim of grabbing participants attention.
Indeed, even if the Tower of Hanoi puzzle was
a distracting task, it was desired to absorb the
participants attention as much as possible on the
Tower of Hanoi task. Interrupting the task four
times (two handovers and two removals) per phase
could already be a bias in participants concentra-
tion on the tower of Hanoi. Adding the SAGAT
evaluation also seemed to compromise the quality
of the experience.

5.3 Effect of the breathing motion
on attention

Whilst in the previous work [6], the breathing
motion increased people vigilance and thus their
safety, this was not the case from the performance
perspective in this work. In the present work,
hypothesis H3 has been rejected. This result can
be explained in several ways. Firstly, the partici-
pants had not been informed about the meaning
of the motion or even its occurrence. However,
our previous work had shown a positive effect
of another signaling motion in terms of perfor-
mance in a context of HRC, both with and without
information about the motion.

In the present work, the context of the task
and the use of the motion were different. Previ-
ously the gesture had been used to indicate a state
of passivity and temporary inactivity. This time,
the notion of waiting for a human to intervene
while performing a continuous task was added.
The difference lies in the activity of the robot
that precedes this state of waiting. In the present
experiment, the robot was in constant motion
near the human operator. Therefore, having the
robot breathe to show that it was waiting did not
have the expected salient effect. During this phase,
the participants saw the robot in their peripheral

vision, active and moving all the time. Conversely
to central vision, the ability to discriminate fea-
tures drops drastically in peripheral vision [47].
Following the results, it seems that the partic-
ipants were unable to differentiate between the
robot motions to perform its task and its breathing
motion. However, when the robot remained
motionless, the participants seem to have been
distracted by the sudden change of state of the
robot. The robot stop would therefore be more
easily noticed. We believe that this problem occurs
at SA level 1, i.e. at the perception level. Indeed,
researchers have pointed to our ability to interpret
social signals in a multitude of forms, particu-
larly abstract ones [33]. This explanation is all the
more plausible given that the robot was sometimes
delayed during the experiments for technical rea-
sons. Although the experimenters were not able to
provide concrete evidence of this from the data,
they reported that the participants were very
often alerted by these sudden stops of the robot
and turned their gaze towards it.

This highlights the importance of taking into
account the context in which the robot motion
is used to communicate non-verbally while per-
forming a task. Additionally, the results suggest
that standing stationary can also be considered
as a gesture of non-verbal communication. In
the present collaborative context, the robot was
performing a continuous task. Using motion to
communicate made no difference. On the other
hand, stopping the robot's motion and keeping
it motionless caused a breakdown in the activity
of the robot and the workspace, and therefore in
the perception of the human. In Camblor et al.
[6], the task of the robot was to remain inactive.
Thus, for the operator, the fact that it was mov-
ing alerted them directly, as moving also broke the
context. All this underlines the importance that
a robot's non-verbal communication through ges-
tures, or lack of them, must take into account the
actual context of the robot's task. If the robot is
to attract the attention of a human partner, non-
verbal communication should breaks the global
context of the task.

Another interesting aspect of this type of
movement is the benefits it provides in terms of
the social perception of humans over robots. The
use of zoomorphic gestures of collaborative robots
has been proposed as a solution to enhance better
social interaction between people and robots [50]. The breathing motion as proposed can be likened to a zoomorphic gesture, as suggested by the authors in their work. More specifically, another work investigated the effect of using animation principles and studied the breathing motion as a social cues [56]. The authors have also highlighted the social benefits of interaction when a collaborative robot generates breathing movements.

## 5.4 Limits

There was a potential bias in relation to the completion of the towers of Hanoi in our experiment. Completing the puzzle can be divided into several sub-problems, corresponding to intermediate stages to be reached before completing the game. Attention resources were mobilized in accordance with these intermediate stages. For example, one of the intermediate stages of completion corresponds to place the largest disk on the final peg, i.e. halfway through the resolution. Thus, when a participant was close to one of these easily identifiable stages, their cognitive resources and attention were less solicited, providing the opportunity to focus attention on the robot for a few moments. If this stage end was temporarily correlated with the robot waiting moment, the probability of human intervention would be higher. Experimenters observed that participants tended to glance at the robot each time they completed a puzzle. This was especially true for those who managed to complete the puzzle several times. Some participants only completed the puzzle once per phase, so this limit did not apply in this case. For those who could not solve the problem, the frustration of not succeeding could be a reason to look at the robot more often.

Concerning SA evaluation, the SAGAT has been unused, as it implies interrupting the task to be administered. The advantage of the SART is it allows post-task evaluation. To keep participants involved in the game of Hanoi towers, the task was not interrupted too often. This ensured the majority of their attention was focused on the game. Interrupting the task to administer the SAGAT would have provided off-task opportunities to potentially schedule a hand-over in the time that followed. Finally, the participants concentration would have been degraded. Furthermore, adding an evaluation of the situation could have

been detrimental to our desire to hide our true objectives to the participants.

## 6 Conclusion

Two questions were addressed in this study. First, the link between the human projection capability and the distribution of human attention within a HRC framework has been studied. The results indicated that the ability to project the robot next actions and its task was beneficial for managing attentional sharing. Notably, participants showed increased attentional engagement in tasks where they were unable to project when the robot would need their help. This highlights the need to take into account human attentional capacities in joint action scenarios involving a robotic partner.

Secondly, the study re-examined the effect of non-verbal robot communication in conveying task information. The study revealed that robot stops were more salient in attracting the attention of a human partner within a collaborative task context. This study underlines the contextual dependence of non-verbal communication, stating that the same gesture cannot be used universally. While the breathing motion effectively signaled potential danger within a shared space lacking direct interaction for task execution, it did not serve to communicate the anticipation of assistance intervention in this instance. It seems that standing stationary can be considered as a gesture of non-verbal communication. In the present work, robot waiting was more salient in capturing human attention when the robot remained motionless rather than making a signaling motion.

One of the perspectives for future work is to be able to estimate SA in real time within a collaborative context using eye-tracker measurements. The robot could adapt its behaviour and communicate to its human partner an imminent need for intervention. The assumption is that a person attentional capacity reflects their SA and the associated task-related mental models. In the present work, the action of the robot were carried out by default and were not conditioned by an assumption of the human SA.
**Supplementary information**

**Acknowledgments.** The authors would like to thank Axel Plantey-Veux for his help in carrying out the experimental runs.

**Declarations**

**Fundings.** The authors acknowledge the support of the French Agence Nationale de la Recherche (ANR) under reference ANR-20-CE10-0005 (PACBOT Project).

**Conflict of interest.** The authors declare that they have no conflict of interest.

**Ethical Approval.** This study has been approved by the Comité Opérationnel d'Évaluation des Risques Légaux et Éthiques (COERLE) Ethics Committee (Application ID: n°2023-05).

**Data Availability.** The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.

**References**

[1] Nicolas Torres Alberto, Antun Skuric, Lucas Joseph, Vincent Padois, and David Daney. Model predictive control for robots adapting their task space motion online. 2023.

[2] Victoria Alonso and Paloma De La Puente. System transparency in shared autonomy: A mini review. *Frontiers in neurorobotics*, 12: 83, 2018.

[3] Cade E Bartlett and Nancy J Cooke. Human-robot teaming in urban search and rescue. In *Proceedings of the Human Factors and Ergonomics Society Annual Meeting*, volume 59-1, pages 250–254. SAGE Publications Sage CA: Los Angeles, CA, 2015.

[4] Bethany Bracken, Sean Tobyne, Aaron Winder, Nina Shamsi, and Mica R Endsley. Can situation awareness be measured physiologically? In *Advances in Neuroergonomics and Cognitive Engineering: Proceedings of the*

AHFE 2021 Virtual Conferences on Neuroergonomics and Cognitive Engineering, Industrial Cognitive Ergonomics and Engineering Psychology, and Cognitive Computing and Internet of Things, July 25-29, 2021, USA, pages 31–38. Springer, 2021.

[5] Cynthia Breazeal, Cory D Kidd, Andrea Lockerd Thomaz, Guy Hoffman, and Matt Berlin. Effects of nonverbal communication on efficiency and robustness in human-robot teamwork. In *2005 IEEE/RSJ international conference on intelligent robots and systems*, pages 708–713. IEEE, 2005.

[6] Benjamin Camblor, Nassim Benhabib, David Daney, Vincent Padois, and Jean-Marc Salotti. Task-consistent signaling motions for improved understanding in human-robot interaction and workspace sharing. In *2022 17th ACM/IEEE International Conference on Human-Robot Interaction (HRI)*, pages 275–283. IEEE, 2022.

[7] Benjamin Camblor, Jean-Marc Salotti, Charles Fage, and David Daney. Degraded situation awareness in a robotic workspace: accident report analysis. *Theoretical Issues in Ergonomics Science*, 23(1):60–79, 2022.

[8] Jessie Y Chen, Katelyn Procci, Michael Boyce, Julia Wright, Andre Garcia, and Michael Barnes. Situation awareness-based agent transparency. *US Army Research Laboratory*, pages 1–29, 2014.

[9] Jessie YC Chen, Shan G Lakhmani, Kimberly Stowers, Anthony R Selkowitz, Julia L Wright, and Michael Barnes. Situation awareness-based agent transparency and human-autonomy teaming effectiveness. *Theoretical issues in ergonomics science*, 19(3): 259–282, 2018.

[10] Min Chen, Stefanos Nikolaidis, Harold Soh, David Hsu, and Siddhartha Srinivasa. Planning with trust for human-robot collaboration. In *Proceedings of the 2018 ACM/IEEE international conference on human-robot interaction*, pages 307–315, 2018.
[11] Aurélie Clodic, Elisabeth Pacherie, Rachid Alami, and Raja Chatila. Key elements for human-robot joint action. *Sociality and normativity for robots: philosophical inquiries into human-robot interactions*, pages 159–177, 2017.

[12] Philip R Cohen and Hector J Levesque. *Teamwork. Nous*, 25(4):487–512, 1991.

[13] J Edward Colgate, Witaya Wannasuphoprasit, and Michael A Peshkin. Cobots: Robots for collaboration with human operators. In *Proceedings of the 1996 ASME international mechanical engineering congress and exposition*, 1996.

[14] Jacob Crossman, Robert Marinier, and Edwin B Olson. A hands-off, multi-robot display for communicating situation awareness to operators. In *2012 International Conference on Collaboration Technologies and Systems (CTS)*, pages 109–116. IEEE, 2012.

[15] Joost CF de Winter, Yke Bauke Eisma, CDD Cabrall, Peter A Hancock, and Neville A Stanton. Situation awareness based on eye movements in relation to the task environment. *Cognition, Technology & Work*, 21(1): 99–111, 2019.

[16] Sandra Devin and Rachid Alami. An implemented theory of mind to improve human-robot shared plans execution. In *2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI)*, pages 319–326. IEEE, 2016.

[17] Amir Dini, Cornelia Murko, Saeed Yahyanejad, Ursula Augsdörfer, Michael Hofbaur, and Lucas Paletta. Measurement and prediction of situation awareness in human-robot interaction based on a framework of probabilistic attention. In *2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)*, pages 4354–4361. IEEE, 2017.

[18] Mica R Endsley. Situation awareness global assessment technique (sagat). In *Proceedings of the IEEE 1988 national aerospace and electronics conference*, pages 789–795. IEEE, 1988.

[19] Mica R Endsley. Toward a theory of situation awareness in dynamic systems. *Human factors*, 37(1):32–64, 1995.

[20] Mica R Endsley. *Designing for situation awareness: An approach to user-centered design*. CRC press, 2016.

[21] Mica R Endsley. From here to autonomy: lessons learned from human-automation research. *Human factors*, 59(1):5–27, 2017.

[22] Mica R Endsley. The divergence of objective and subjective situation awareness: A meta-analysis. *Journal of cognitive engineering and decision making*, 14(1):34–53, 2020.

[23] Mica R Endsley, Stephen J Selcon, Thomas D Hardiman, and Darryl G Croft. A comparative analysis of sagat and sart for evaluations of situation awareness. In *Proceedings of the human factors and ergonomics society annual meeting*, volume 42-1, pages 82–86. SAGE Publications Sage CA: Los Angeles, CA, 1998.

[24] Akuadasuo Ezenyilimba, Margaret Wong, Alexander Hehr, Mustafa Demir, Alexandra Wolff, Erin Chiou, and Nancy Cooke. Impact of transparency and explanations on trust and situation awareness in human-robot teams. *Journal of cognitive engineering and decision making*, 17(1):75–93, 2023.

[25] Étienne Fournier, Dorilys Kilgus, Aurélie Landry, Belal Hmedan, Damien Pellier, Humbert Fiorino, and Christine Jeoffrion. The impacts of human-cobot collaboration on perceived cognitive load and usability during an industrial task: an exploratory experiment. *IISE Transactions on Occupational Ergonomics and Human Factors*, 10(2):83–90, 2022.

[26] Thibault Gateau, Caroline P Carvalho Chanel, Mai-Huy Le, and Frédéric Dehais. Considering human's non-deterministic behavior and his availability state when designing a collaborative human-robots system. In *2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)*, pages 4391–4397. IEEE, 2016.
[27] Matthew Gombolay, Anna Bair, Cindy Huang, and Julie Shah. Computational design of mixed-initiative human-robot teaming that considers human factors: situational awareness, workload, and workflow preferences. *The International journal of robotics research*, 36(5-7):597-617, 2017.

[28] Sami Haddadin and Elizabeth Croft. Physical human-robot interaction. *Springer handbook of robotics*, pages 1835-1874, 2016.

[29] Sandra G Hart. Nasa-task load index (nasa-tlx); 20 years later. In *Proceedings of the human factors and ergonomics society annual meeting*, volume 50-9, pages 904-908. Sage publications Sage CA: Los Angeles, CA, 2006.

[30] Sogand Hasanzadeh, Behzad Esmaeili, and Michael D Dodd. Examining the relationship between construction workers' visual attention and situation awareness under fall and tripping hazard conditions: Using mobile eye tracking. *Journal of construction engineering and management*, 144(7):04018060, 2018.

[31] Guy Hoffman. Evaluating fluency in human-robot collaboration. *IEEE Transactions on Human-Machine Systems*, 49(3):209-218, 2019.

[32] Guy Hoffman and Cynthia Breazeal. Effects of anticipatory action on human-robot teamwork efficiency, fluency, and perception of team. In *Proceedings of the ACM/IEEE international conference on Human-robot interaction*, pages 1-8, 2007.

[33] Guy Hoffman and Wendy Ju. Designing robots with movement in mind. *Journal of Human-Robot Interaction*, 3(1):91-122, 2014.

[34] Sarah K Hopko, Riya Khurana, Ranjana K Mehta, and Prabhakar R Pagilla. Effect of cognitive fatigue, operator sex, and robot assistance on task performance metrics, workload, and situation awareness in human-robot collaboration. *IEEE Robotics and Automation Letters*, 6(2):3049-3056, 2021.

[35] Debra G Jones and Mica R Endsley. Use of real-time probes for measuring situation awareness. *The international journal of aviation psychology*, 14(4):343-367, 2004.

[36] David B Kaber and Mica R Endsley. The effects of level of automation and adaptive automation on human performance, situation awareness and workload in a dynamic control task. *Theoretical issues in ergonomics science*, 5(2):113-153, 2004.

[37] Ross A Knepper, Christoforos I Mavrogianis, Julia Proft, and Claire Liang. Implicit communication in a joint action. In *Proceedings of the 2017 acm/ieee international conference on human-robot interaction*, pages 283-292, 2017.

[38] Markus Koppenborg, Peter Nickel, Birgit Naber, Andy Lungfiel, and Michael Huelke. Effects of movement speed and predictability in human-robot collaboration. *Human Factors and Ergonomics in Manufacturing & Service Industries*, 27(4):197-209, 2017.

[39] Dana Kulic and Elizabeth Croft. Anxiety detection during human-robot interaction. In *2005 IEEE/RSJ International Conference on Intelligent Robots and Systems*, pages 616-621. IEEE, 2005.

[40] Przemyslaw A Lasota, Terrence Fong, Julie A Shah, et al. A survey of methods for safe human-robot interaction. *Foundations and Trends in Robotics*, 5(4):261-349, 2017.

[41] Joseph B Lyons. Being transparent about transparency: A model for human-robot interaction. In *2013 AAAI Spring Symposium Series*, 2013.

[42] Angus McKerral and Kristen Pammer. Identifying objective behavioural measures of expert driver situation awareness. *Accident Analysis & Prevention*, 163:106465, 2021.

[43] S Merchant, Yongjin Kwon, T Schnell, T Etherington, and T Vogl. Evaluation of synthetic vision information system (svis) displays based on pilot performance. In *20th DASC. 20th Digital Avionics Systems Conference (Cat. No. 01CH37219)*, volume 1, pages 2C1-1. IEEE, 2001.
[44] Costanza Messeri, Gabriele Masotti, Andrea Maria Zanchettin, and Paolo Rocco. Human-robot collaboration: Optimizing stress and productivity based on game theory. *IEEE Robotics and Automation Letters*, 6(4):8061–8068, 2021.

[45] Lucas Paletta, Amir Dini, Cornelia Murko, Saeed Yahyanejad, Michael Schwarz, Gerald Lodron, Stefan Ladstätter, Gerhard Paar, and Rosemarie Velik. Towards real-time probabilistic evaluation of situation awareness from human gaze in human-robot interaction. In *Proceedings of the Companion of the 2017 ACM/IEEE International Conference on Human-Robot Interaction*, pages 247–248, 2017.

[46] Luka Peternel, Nikos Tsagarakis, Darwin Caldwell, and Arash Ajoudani. Adaptation of robot physical behaviour to human fatigue in human-robot co-manipulation. In *2016 IEEE-RAS 16th International Conference on Humanoid Robots (Humanoids)*, pages 489–494. IEEE, 2016.

[47] Ruth Rosenholtz. Capabilities and limitations of peripheral vision. *Annual review of vision science*, 2:437–457, 2016.

[48] Sayanti Roy, Trey Smith, Brian Coltin, and Tom Williams. I need your help... or do i? maintaining situation awareness through per- formative autonomy. In *Proceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction*, pages 122–131, 2023.

[49] Paul M Salmon, Neville A Stanton, Guy H Walker, Daniel Jenkins, Darshna Ladva, Laura Rafferty, and Mark Young. Measur- ing situation awareness in complex systems: Comparison of measures study. *International Journal of Industrial Ergonomics*, 39 (3):490–500, 2009.

[50] Vanessa Sauer, Axel Sauer, and Alexander Mertens. Zoomorphic gestures for commu- nicating cobot states. *IEEE Robotics and Automation Letters*, 6(2):2179–2185, 2021.

[51] Herbert A Simon. The functional equiva- lence of problem solving skills. *Cognitive psychology*, 7(2):268–288, 1975.

[52] Antun Skuric, Vincent Padois, and David Daney. On-line force capability evaluation based on efficient polytope vertex search. In *2021 IEEE International Conference on Robotics and Automation (ICRA)*, pages 1700–1706. IEEE, 2021.

[53] Matthew Story, Phil Webb, Sarah R Fletcher, Gilbert Tang, Cyril Jaksic, and Jon Carberry. Do speed and proximity affect human-robot collaboration with an industrial robot arm? *International Journal of Social Robotics*, 14 (4):1087–1102, 2022.

[54] Aaquib Tabrez, Matthew B Luebbers, and Bradley Hayes. Descriptive and prescriptive visual guidance to improve shared situational awareness in human-robot teaming. In *Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems*, pages 1256–1264, 2022.

[55] Richard M Taylor. Situational awareness rat- ing technique (sart): The development of a tool for aircrew systems design. In *Situational awareness*, pages 111–128. Routledge, 2017.

[56] Yunus Terzioğlu, Bilge Mutlu, and Erol Şahin. Designing social cues for collabora- tive robots: the role of gaze and breathing in human-robot collaboration. In *Proceed- ings of the 2020 ACM/IEEE international conference on human-robot interaction*, pages 343–357, 2020.

[57] Koen van de Merwe, Steven Mallam, and Salman Nazir. Agent transparency, situation awareness, mental workload, and operator performance: A systematic literature review. *Human Factors*, page 00187208221077804, 2022.

[58] Milos Vasic and Aude Billard. Safety issues in human-robot interactions. In *2013 IEEE inter- national conference on robotics and automa- tion*, pages 197–204. IEEE, 2013.
[59] Astrid Weiss, Ann-Kathrin Wortmeier, and Bettina Kubicek. Cobots in industry 4.0: A roadmap for future practice studies on human-robot collaboration. *IEEE Transactions on Human-Machine Systems*, 51(4): 335-345, 2021.

[60] Andrea Maria Zanchettin, Andrea Casalino, Luigi Piroddi, and Paolo Rocco. Prediction of human activity patterns for human-robot collaborative assembly tasks. *IEEE Transactions on Industrial Informatics*, 15 (7):3934-3942, 2018.

[61] Ting Zhang, Jing Yang, Nade Liang, Brandon J Pitts, Kwaku O Prakah-Asante, Reates Curry, Bradley S Duerstock, Juan P Wachs, and Denny Yu. Physiological measurements of situation awareness: a systematic review. *Human factors*, page 0018720820969071, 2020.