AR Presentation of Team Members' Performance and Inner Status to Their Leader: A Comparative Study

Thomas Rinnert, James Walsh, Cédric Fleury, Gilles Coppin, Thierry Duval,

Bruce Thomas

► To cite this version:

Thomas Rinnert, James Walsh, Cédric Fleury, Gilles Coppin, Thierry Duval, et al. AR Presentation of Team Members' Performance and Inner Status to Their Leader: A Comparative Study. Applied Sciences, 2023, 14 (1), pp.123. 10.3390/app14010123. hal-04417267

HAL Id: hal-04417267

https://hal.science/hal-04417267v1

Submitted on 25 Jan 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Article

# AR Presentation of Team Members' Performance and Inner Status to Their Leader: A Comparative Study

Thomas Rinnert <fnref n="1" />,<fnref n="2" />,<fnref n="3" />,<fnref n="4" />* , James Walsh <fnref n="1" />,<fnref n="4" /> , Cédric Fleury <fnref n="2" />,<fnref n="3" /> , Gilles Coppin <fnref n="2" />,<fnref n="3" />,<fnref n="4" /> , Thierry Duval <fnref n="2" />,<fnref n="3" />
and Bruce H. Thomas <fnref n="1" />,<fnref n="4" />

**Featured Application:** Authors are encouraged to provide a concise description of the specific application or a potential application of the work. This section is not mandatory.

**Abstract:** Real-time and high-intensity teamwork management is complex, as team leaders must ensure good results while also considering the well-being of team members. Given that stress and other factors directly impact team members' output volume and error rate, these team leaders must be aware of and manage team stress levels in combination with allocating new work. This paper examines methods for visualizing each team member's status in mixed reality, which, combined with a simulated stress model for virtual team members, allows the team leader to take into account team members' individual statuses when choosing who to allocate work. Using simulated Augmented Reality in Virtual Reality, a user study was conducted where participants acted as team leaders, putting simulated team members under stress by allocating a number of required work tasks whilst also being able to review the stress and status of each team member. The results showed that providing Augmented Reality feedback on team members' internal status increases the team's overall performance, as team leaders can better allocate new work to reduce team members' stress-related errors whilst maximizing output. Participants preferred having a graph representation for stress levels despite performing better with a text representation.

**Keywords:** Augmented Reality, Virtual Reality, Empathic Computing, User Status Awareness.

## 1. Introduction

In order for team leaders to maximize the effectiveness and efficiency of their team as individuals and as a whole, they need to understand the current status of their team members, as described in the empathic computing research field [1]. For these leaders, workspace awareness defines the knowledge required for workers to interact with a system, utilizing the up-to-the-moment understanding of other people's actions [2]. However, actively communicating this information can be complex, given the team constitutes many individuals. Additionally, each team member has to manage their own tasks, workload, and stress levels, which team leaders must be aware of and actively monitor.

We envision that a team leader could view the other team members with Augmented Reality (AR) workspace awareness information. Given one of the roles of a team leader is to accomplish work through other people [3], we aim to minimise the time it takes for a leader to make decisions that optimise the whole team's performance and the well-being of the team members. While team leaders can obtain feedback on their team performance and status through 2D interfaces, AR presents in-situ information related to team members in

**Citation:** Rinnert, T.; Walsh, J.; Fleury, C.; Coppin, G.; Duval, T.; Thomas B. AR presentation of team members' performance and inner status to their leader: a comparative study. *Appl. Sci.* **2023**, *1*, 0. https://doi.org/

Received:

Revised:

Accepted:

Published:

**Copyright:** © 2024 by the authors. Submitted to *Appl. Sci.* for possible open access publication under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

<footnote n="1">¹ Crossing IRL 2010, French National Center for Scientific Research (CNRS), Adelaide 5005, Australia;</footnote>

<footnote n="2">² Lab-STICC UMR CNRS 6285, 29238 Brest, France;</footnote>

<footnote n="3">³ Department of Computer Science, IMT Atlantique, 29238 Brest, France; cedric.fleury@imt-atlantique.fr (C.F.); gilles.coppin@imt-atlantique.fr (G.C.); thierry.duval@imt-atlantique.fr (T.D.)</footnote>

<footnote n="4">⁴ Australian Research Centre for Interactive and Virtual Environments, University of South Australia, Adelaide 5095, Australia; james.walsh@unisa.edu.au (J.W.); bruce.thomas@unisa.edu.au (B.T.)</footnote>

<footnote>* Correspondence: thomas.rinnert@cnrs.fr</footnote>
the physical space [4]. This information could be represented in various ways, e.g. textual
versus graphical [2,5], and could even include state information, for example, whether the
team member is browsing information or performing focused work on it [6].

Previous work has examined the effects of sharing physiological states in collaborative Virtual Reality (VR) (e.g. heart rate [7,8]), as well as the development of collaborative awareness by using realistic avatars [9,10], sharing gaze [11? ,12], and even emotions [13]. Understanding a team member's direct and passive actions and behaviours provides team leaders with a much broader range of information when making decisions that impact a team member [14]. This work explores the development of AR visual cues to help team leaders understand the state of their working team members when allocating new tasks to them, as explored by Lee et al. where a local worker shares visual cues with a remote helper [15]. This allows the team leader to individually manage the workload and well-being of individual members while maximizing task outputs for the team overall.

We developed different AR presentation methods of status information to a team leader, communicating individual team members' status. The goal is not to investigate methods of physiological sensing or human tracking technologies but to determine the effectiveness of different visual attributes of AR presentations. This is why simulated team members can be employed in our experiments, as we focus on the effect of sharing the team members' status and not on how we gather it. The AR information is simulated in a VR environment, and the simulated team members are placed in that VR setting. The simulated team members' well-being is presented to the team leader labelled as stress<fnref n="1" />, following Lazarus and Folkman's definition as "a particular relationship between the person and the environment that is appraised by the person as taxing or exceeding his or her resources and endangering his or her well-being" [16]. This investigation focuses on improving information presented to the team leaders who are required to know who is over or under-loaded and how they are coping with that work. The presentation of this information aims to allow the team leader to balance between having the work done and maintaining a healthy and efficient working environment. Future research will investigate AR-presented information in a physical setting with human team members and physiological sensing.

In this paper, two research questions are addressed:

* RQ1 - Does providing information about a team member's stress status and current workload help team leaders manage a team?

* RQ2 - Which AR visualizations are most effective in helping team leaders understand the status of their team members?

After reviewing related work, this paper presents the developed visualization tech-
niques and experimental platform, then the study design and results, and ends with a
discussion and final thoughts.

## 2. Related work

Based on previous work, we detail which information is relevant to support awareness of collaboration, how physiological data can further improve awareness, and how all this information can be visualized.

### 2.1. Awareness in collaborative XR

When working together in a shared space, the perceptive elements of the environment enable people to be aware of the collaborative situation. This awareness of collaboration is defined by Gutwin and Greenberg [5] as the set of knowledge and the environmental understanding for answering the questions Who?, What?, How?, Where?, and When? about the collaborative tasks. The notion of awareness has been widely studied and characterized,

<footnote n="1">¹ Stress is an extremely complex concept and topic. Our use of this term is meant to provide a high-level term covering a person who is experiencing such emotions as being overworked, unable to finish tasks on time, pressured, and falling behind on their work.</footnote>
whether by focusing on how people interact with each other and with the system [17] or by
focusing on the proximity between the perceptive elements and their observer [18].

AR can be used in many situations to improve collaborative work by providing additional awareness cues. For example, capturing a team member's active and passive actions and behaviours would provide other team members and team leaders with a much broader range of information when confronted with decision-making [14]. In order to improve the awareness of collaboration, several techniques have been explored: representing hands, arms and full body gestures [19-22]; sharing the view frustum as a visible cone or pyramid, head or eye gazes as seeable rays [1,6,11,12,21,23? -25]; giving people a pointer, with a shared visual representation, to clarify what people want to designate [20,23,26,27]; or providing the ability to sketch on surfaces or mid-air [20,27].

## 2.2. Improving awareness with physiological data

Improving awareness goes beyond just understanding people's actions, but requires more information about how they feel, why they acted, how they did what they did, and other elements that may influence collaboration. To help people to get awareness from their perceptions, previous work proposed to provide verbal communication, to embody people with realistic avatars [9,10,25], to share emotions [1,13,25,28,29], and physiological data [6-8,25]. For example, Dey et al. shared the heart rate between two participants with a heart icon in their field of view [7] and later shared the heart rate between two participants through VR controllers' vibration [8]. Luong et al. gathered mental workload by integrating sensors in a head-mounted display, and ocular activity was the best indirect indicator of mental workload [30]. They then showed that designing complex training scenarios with multiple parallel tasks while modulating the user's mental workload over time (MATB-II) was possible [31].

## 2.3. Visualizing information

Information can be displayed either with a textual presentation (raw data) or a graphical one (with abstraction) [2,5]. In terms of placement, information could be displayed at one or more fixed locations for the user, such as signs and panels; as a virtual movable tablet that could be resized, enabled, and disposed of on-demand (e.g., Tablet Menus [32]); or attached to the users' body (e.g., on their wrist, fingers, legs, torso, etc... [32,33]). The gaming industry already uses this variety of placement options (such as Lone Echo [34] or Zenith [35]) as well as academic research [2,6,36,37]. In addition, Yang et al. [4] explored collaborative sensemaking tasks for groups of users immersed in VR. The outcomes of their study demonstrate the potential benefits of spatially distributing information in VR compared to aggregating them in a traditional desktop environment.

## 2.4. Synthesis of our literature review

According to these related works, effective collaboration hinges on individuals with a good awareness of collaboration: a profound understanding of their collaborators and actions. Our analysis underscores the necessity for shared environments to provide perceptive elements, utilizing both natural means like visible bodies, gestures, and facial expressions, and XR to present internal information such as feelings, physiological data, and well-being. This synthesis of findings emphasizes the essential role of awareness of collaboration in optimizing shared environments for enhanced collaborative experiences, laying the foundation for further exploration in this study. As Gutwin and Greenberg wrote, "The input and output devices used in groupware systems and a user's interaction with a computational workspace generate only a fraction of the perceptual information that is available in a face-to-face workspace. Groupware systems often do not present even the limited awareness information that is available to the system." [5]. Through this assertion, they identify two research directions: generating more perceptual information and presenting already available data, with existing literature suggesting methods for both.
This study focuses on the latter, investigating how to effectively provide team members' performance and physiological data feedback to their leader using AR visualisations.

## 3. Visualization techniques

<figure><img src="image_2.png" /><figcaption>**Figure 1.** Awareness components with a textual presentation.</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>**Figure 2.** Awareness components with a graphical presentation.</figcaption></figure>

Our review of the literature showed the potential benefits of presenting information with XR, with this work we aim to apply this observation to collaborative teamwork with

130

131

132

133

134
AR visual cues. This is why we developed AR visualization techniques to support the team leader in assessing the current workflow situation and team members' well-being by providing feedback on the team members' status. We define the *team member's status* as the sum of the team member's task status and inner status. *Task status* refers to all information related to the user's current, previous, and future tasks. *Inner status* encapsulates one's well-being-related information, e.g., physiological levels (such as heart rate, body temperature, skin conductance), emotions, and fatigue. For the purposes of this work, we do not examine a complete set of *inner status* attributes. We instead measure how the team leader's actions put team members under *stress*. In our experiment, we do not directly measure stress, rather stress is calculated from known parameters such as the amount of work and rest time the team leader provides and will be addressed as *simulated stress*. The formal definition of how we determine *simulated stress* is presented in Section 4.2.

A previous study [38] showed that providing *task status* to team members helped them to know how well others were performing. We can thus infer that *task status* can also help the team leader determine which team member is best suited to be assigned a task. Our study shares similarities with prior work where a local worker shares visual cues (gaze, facial expressions, physiological signals, and environmental views) with a remote helper who provides guidance [15], but our research diverges in its collaborative context. Indeed, we specifically focus on team monitoring and task allocation, while concurrently sharing a worker's status to understand their actions and well-being. To present the task status and the inner status with AR, the graphical assets must be easy to understand, and avoid occlusion [39]. Both text and visual elements can be used, text must be simple and written on a billboard to contrast with the background [40,41], and icons, symbols and simple 2D graphic elements can support or replace text in AR-presented information [42]. Thus, to answer RQ1 and RQ2, we developed a set of AR visualizations of *team member's status* as a set of awareness components with two different presentation styles: *textual* (Fig. 1) and *graphical* (Fig. 2). The textual style employs numbers and text to present the different attributes of user status information. All numbers and text were displayed in white with a black background, where the graphical presentation employs graphs and simple 3D models. All components are displayed on 2D panels floating around workers to avoid occlusion. A description of each component is as follows:

*   **Task Queue** displays the number and the type of tasks a team member has yet to complete. The textual style had four numbers, one for each task type. The graphical presentation had a virtual shelf that held stacks of each of the different tasks represented in the cube as on the team leader's table (see Fig. 3).

*   **Current Task Completion** shows which task a team member is currently performing and its progress. The textual style presents this as a numeric percentage, and the graphical presentation employs a progress bar.

*   **All Task Completion** shows the progress of a team member on a task batch they have received. Both styles employ the **Current Task Completion** visualizations.

*   **Task History** shows the number and the type of tasks completed from the start of the work session. Both styles employ scaled-down versions of the **Task Queue** component.

*   **Portion of Time Working** represents the proportion of time a team member spent working since the start of the scenario, and the complement to 100% represents the time spent resting. Both styles employ the **Current Task Completion** visualizations.

*   **Success** shows if a team member succeeded or failed their assigned tasks. The textual style presents success and failure as integer values and the ratio of success/failure as a percentage. Instead of using numbers, the graphical style presents a disc that fills radially with the success percentage.

*   **Score** shows how well a team member completed their tasks. Each task is rated on a scale from 0 to 1 inclusive (see Section 4.2 for more details). The component displays a cumulative score since the beginning of the scenario (the sum of all the individual task scores) and the average score per task. The textual style presents the current score as a
real number and the score per task as a numeric percentage. The graphical style only
presents the score per task percentage as a disc that fills radially.

*   **Stress** shares a team member's *inner status* summarized under the *simulated stress* term. The textual style presents a history of which stress levels an operator has been going through, and the current level and value of their *simulated stress*. The written value of the current *simulated stress* is colored in accordance with the stress level (Low - Green; Medium - Yellow; High - Red, see Section 4.2 for more details). The graphical style presents a 2D graph with time as the x-axis and *simulated stress* as the y-axis. The line drawn on this graph takes the colour of the stress level recorded at that time.

The organization of components within panels was refined through pilot experiments, where valuable feedback was obtained. Three main categories emerged based on this feedback: the present and future workload of a team member (comprising Task Queue, Current Task Completion, and All Tasks Completion components), their past performance (including Task History, Portion Of Time Working, Success, and Score components), and their past and current *inner status* (represented by the Stress component). To effectively organize and present this information, we opted for a structured approach. Instead of having everything plotted in one unique panel, we created three dedicated panels, each tailored to one of these categories. The final placement and design of these components aimed to achieve two key objectives. Firstly, it sought to minimize overlap between panels of different team members. Secondly, the design was crafted with a focus on improving the overall readability of all components.

4. Simulated AR in VR application: MATB-II-Collab

To evaluate the eight awareness components’ of the *textual* and *graphical* AR presenta-
tions, we developed a Unity experimental platform where people were immersed with an
HTC Vive Cosmos headset. The experimental platform enables a participant to function as
the team leader, assigning tasks to virtual team members, the *operators*, and being presented
with AR visualizations concerning those operators. *Operators* are part of the experimental
platform and were fully simulated to keep the system invariant from one participant to
another. They are virtual team members who have their behaviour influenced across
iterations of the simulation by the tasks assigned, the platform then gathers data from a
team leader who assigns tasks to this **simulated team**. The experimental platform is in VR,
and the AR is simulated in the platform. The remainder of this section presents both the
experimental platform and the operator model.

<figure><img src="image_4.png" /><figcaption>**Figure 3.** The virtual environment used in the experiment was populated by six virtual team members. The team leader’s table is in the centre of the room, and the six black buckets are where tasks are allocated. The tasks are represented as cubes on the table.</figcaption></figure>
## 4.1. *Experimental platform*

We developed an experimental platform where a participant performs the role of a team leader facing virtual team members labelled *operators*. The number and placement of operators in the virtual environment can be set or modified depending on the experiment conducted. The system automatically creates tasks that the team leader has to assign to operators. Cubes are used to represent the tasks the team leader has to allocate to operators by throwing the task cube in the corresponding basket of each operator (see Fig. 3). The system drops a set of tasks onto the table in front of the team leader. The team leader picks up the tasks from the table with either hand and assigns them to the different operators. The sets of tasks are in blocks of 5 to 20 cubes every 30 seconds.

We modelled the operators to perform tasks at different degrees of success, which is represented by the score, a decimal value between 0 and 1 inclusive given to an operator after performing a task, depending on how stressed they are; Section 4.2 fully describes this model. The operators have humanoid avatars to tackle the same occlusion and placement issues we would have in an AR workspace. The experimental platform integrates the visualization techniques presented in Section 3, providing AR-displayed information around each operator, as feedback on the operator's status. This AR display provides data to inform the team leader's choice of an operator to assign a task to, thus optimizing the team's overall success. The experiment was designed for the participants to receive tasks over time to assign to the operators to achieve the greatest team performance. The team performance was calculated by how well the operators performed the given tasks. The team score is calculated as the sum of the score of all the tasks performed by the operators. The experimental platform enabled two factors to be manipulated: 1) the presentation of indicators (textual and graphical) and 2) if the *simulated stress* feedback is shown or not shown.

## 4.2. *Operator model*

Operators are automated virtual team members controlled by the experimental platform. Their avatar is either controlled by inverse kinematics animation for the four tasks or by skeleton animation when the avatar is idle. The operators virtually perform the four tasks of the NASA MATB-II [43] on a virtual screen interface in front of them, see Fig. 3 The avatars of the operators replay previously recorded movements with inverse kinematics (using FinalIK). The movement data set comprises more than ten different recordings for each task recorded from a person performing the four MATB-II tasks in a VR setting with the experimental platform. The playback speed for each animation was adjusted to last for 10s. If not replaying a recorded animation, the avatar of the operator plays one randomly chosen idle animation among the six skeleton animations from the Mixamo library. Inverse kinematics interpolates the positions of the hands and head to ensure consistently smooth movement when the avatar transitions between recorded movements and other animations [44].

The team leader puts tasks in the operators' task queues. Operators will perform their tasks one after another without resting between them until the queue is empty, following a FIFO (First In, First Out) pattern. When an operator's task queue is empty, they are on a rest break until new tasks are placed in their queue.

Each operator has a speed multiplier; with a speed of 1.0, an operator will execute a task in 10s (the normalized time for all tasks); with a speed of 1.5, they will execute the task in 6.7s and, with a speed of 0.75, they will execute the tasks in 13.3s. Two operators have a speed multiplier of 0.75, two operators have a speed multiplier of 1.0, and two have a speed multiplier of 1.5.

Operators have a *simulated stress* attribute to refer to their internal status (normalized to a 0.0-1.0 range). The team leader's action of adding a task to an operator's task queue progressively increases the *simulated stress* experienced by that operator, which we call *stress charge*. The total amount of *simulated stress* per task experienced by an operator is not impacted by the speed at which the operator completes the task; thus, the *stress charge* is
obtained by multiplying the speed of the operator by the *stress charge* parameter. Operators' *simulated stress* decreases with time, which we call *stress cooldown*, all operators have the same rate of *stress cooldown*, another parameter of the experimental platform. The *stress charge* parameter has to be stronger than the *stress cooldown* parameter to increase operators' *simulated stress* when they work, as their *simulated stress* evolves by the following Algorithm: *As long as the simulation is running, the operator's simulated stress decreases by the cooldown rate, then it increases by the charge rate if the operator has a task to perform.*

The experimental platform makes a probabilistic determination of whether an operator is successful for each task. The more stressed an operator is, the more likely they will not succeed with the task. We distinguish three different levels of *simulated stress*: LOW, MEDIUM, and HIGH, with two parametrable thresholds to separate them. With a LOW level of *simulated stress*, there is a low chance of failure by an operator, and at a HIGH level of *simulated stress*, the operator is almost certain to fail at the task. A parametrable function defines a task's failure probability according to the *simulated stress* level. As a performance measure, a score of 1 minus the failure probability is given to an operator after performing a task. Operators have no preference or skill differences among tasks.

Initial conditions of the virtual environments and operators' attributes will be described in Section 5.1.

# 5. Experimental design - Material and Methods

We performed a 2x2 within study to evaluate if the presence of stress feedback (*Stress* and *NoStress*) and which presentation of data (*textual* or *graphical*) best helped the team leader. The study thus compared four conditions: *Textual~NoStress*, *Textual~Stress*, *Graphical~NoStress*, and *Graphical~Stress*. The *Textual~NoStress* – *Textual~Stress* conditions and *Graphical~NoStress* – *Graphical~Stress* conditions were designed to evaluate **RQ1** on the effect of providing stress feedback to team leaders to improve team performance. The *Textual~Stress* and *Graphical~Stress* conditions enable the evaluation of the impact of the representation of that stress on user performance to assess **RQ2**. With this study, we investigate the effect and relevance of providing team members' status to their leaders in a work where team members must perform their tasks reliably with little to no mistakes.

## 5.1. Parameters and initial conditions

Pilot experiments determined the study's parameters and initial conditions. The *stress charge* parameter was set to 2% per second, and the *stress cooldown* parameter was set to -0.1% per second. The threshold between LOW and MEDIUM levels of *simulated stress* was set to 0.4, and the threshold between MEDIUM and HIGH levels of *simulated stress* has been set to 0.7. For the failure probability function, we used a simple sigmoid function to be easy for the participants to understand. The virtual scene has six operators (see Section 4.2) arranged in an arc, centred on the participant, to add a constraint to the experiment. This way, the participant won't be able to see all operators at once and will have to turn their head along the experiment while not putting excessive effort as there could be with a full circle arrangement or having to navigate in a complex workspace. Each operator is assigned a speed factor, a *simulated stress* offset, and some initial tasks. At the start of each trial, two operators are randomly chosen to become *slow operators* with a 0.75-speed multiplier, two others are randomly chosen to become *fast operators* with a 1.5-speed multiplier, and the last two operators will stay *normal speed operators* with a speed multiplier of 1.0. At the same time, two operators are randomly chosen to begin with *LOW simulated stress* (0.3), two others are randomly chosen to begin with *MEDIUM simulated stress* (0.5), and the two last ones will begin with *no simulated stress* at all (0.0). The trial then starts by assigning 12 tasks among operators: two operators are randomly chosen to receive one task, two other operators receive two tasks each, and the last two operators are given three tasks each. The random assignment of initial conditions for the operators removes any learning effects between trials on each of the operator's abilities. In the worst case, there will only
be one failure with the initial conditions, which will have minimal impact on overall team performance.

325

326

## 5.2. Hypotheses

327

To answer RQ1 and RQ2, we made the following hypotheses:

328

**H1** - *Providing operators' stress feedback to team leaders improves team performance.* During the training, participants were informed that the stress directly impacted team members' performance, and they were informed on how their actions increased and decreased team members' stress level. Providing participants with information on team members' stress levels will help them to make better decisions, and improve the overall team performance.

329

330

331

332

333

**H2** - *Team leaders will prefer the presentation of operators' stress in graphical form rather than textual.* A graph helps to read multidimensional data, and the participants will prefer to read a graph rather than multiple lines of text to have information on the team members' stress over time.

334

335

336

337

## 5.3. Data collection

338

Experiment independent variables are the presence of simulated stress feedback (*Stress* and *NoStress*) and presentation (*Textual* or *Graphical*).

339

340

341

Experiment-dependent variables are as follows:

* The team score (real between 0.0 and 172.0): sum of the score of all the tasks performed by the operators.

342

343

* The trial duration (real from 455s to 1052s): time the team takes to complete all tasks.

344

* The answers to the NASA-TLX (Task Load indexX [45]), a survey on six different dimensions: the mental demand, physical demand, temporal demand, the performance, effort, and frustration felt by the subject during the task. Each dimension has been assessed on a five-level Likert scale.

345

346

347

348

* The preferred condition for each component (what helped them the most).

349

* Which components the participant prefers in the textual presentation.

350

* Which components the participant prefers in the graphical presentation.

351

Additional data was collected during the experiment:

352

* The adopted strategy of assigning tasks after every trial.

353

* A further explanation of the evolution of the participant's strategy at the end of the experiment.

354

355

* Participants' opinion on the importance of stress feedback.

356

## 5.4. Apparatus

357

The study was conducted in a quiet room exclusively dedicated to the study for its duration. Protocols to fight against COVID-19 have been deployed (disinfection of equipment, wearing a mask, etc.). The experiment was run on a VR-ready computer (DELL laptop, i7-10850H, 32GB RAM, NVIDIA Quadro RTX 3000), using an immersive Head Mounted Display (HTC Vive Cosmos) with controllers. The experimental platform was the MATB-II-CVE application, a custom-developed software using Unity 2018.4.30f1, C#, and the SteamVR framework. The Unity project of the experimental platform application is open-access: https://github.com/ThomasRinnert/MATB-II_CollabVR.

358

359

360

361

362

363

364

365

## 5.5. Participants

366

We recruited 24 volunteers (6 females, 18 males) aged from 18 to 54 years old (Mean: 29.3, SD: 8.4), who were rewarded with gift cards after the experiment. The participant self-reported their handedness (1 left-handed, 23 right-handed), and they rated on a scale from 0 ("Not at all") to 4 ("A lot") their familiarity with and use of video games (Mean: 3.6, SD: 1.2), 3D environments (Mean: 3.5, SD: 1.5) and immersive technologies (Mean: 3.5, SD: 1.4).

367

368

369

370

371

372
## 5.6. Protocol

The participants were first welcomed, asked to read an information sheet, and if they agreed to participate, they signed a consent form. They were then informed of the objectives of this experiment: to help team leaders manage their team by being better informed concerning the team members' status feedback. The participants were then informed how operators work (see Section 4.2), particularly how the task score is affected by the operator's stress level. The textual and graphical presentations of the proposed visualization techniques were then presented to the participants (see Section 3), asking them to achieve the best team performance while minimising operators' stress. Participants were aware that time was not a constraint. The university's ethics committee approved this experimental protocol.

### 5.6.1. Training

After the introduction, the participants were equipped with the VR headset and its controllers, and the training phase then started. There were five phases of training: the initial training and a training phase before each trial. During the initial training, participants could experience how to assign tasks to operators. They could ask to retry as many times as they wanted until they were ready to begin a trial. Before each condition, the awareness components were shown to participants, and they could assign some tasks, like during the initial training phase to understand how the awareness components evolved.

### 5.6.2. Task Assignment Trial

During the experiment, the participant had to complete four trials in the virtual environment of assigning tasks to the six operators. A trial had no fixed duration: 160 task artifacts were given to the participants over approximately 8 minutes (5 to 20 every 30 seconds) for them to assign the tasks to operators. The trials end when all tasks are assigned and completed. This number of tasks had been determined during preliminary pilot experiments, where we investigated if there were enough tasks to place the virtual operators in a high-stress state. The experimental parameters produced tasks that even when assigned with care, operators can reach a point where they are fully stressed. Even though the number of tasks is constant (160), the cubes depicting the tasks are generated to be one of the four MATB-II tasks randomly. The only difference in tasks is the name. The tasks all last the same time range for task completion, and operators have no preferences or skill differences among the four types of tasks. Participants were asked to answer questionnaires between each trial and at the end of the experiment. The experiment lasted for approximately one hour and a half for each participant.

## 6. Results

Following the methods of the reference books [46,47], we analysed all quantitative data following the same process, starting with a Shapiro-Wilk test to check for normality. For normally distributed data, 2-ways ANOVAs were used (after checking that the sphericity assumption was met with a Maulchy test) with Tukey HSD tests ran as posthoc analysis, and for data not following a normal distribution, we ran two different analyses:

1. Both conditions with a textual presentation were aggregated and compared with a Wilcoxon test to the two other conditions with a graphical presentation, and the same was done for conditions with and without *simulated stress* feedback;

2. Friedman $\chi^2$ tests were used to check for significant differences between conditions, then pairwise Wilcoxon tests were used as a posthoc analysis with an alpha correction to 0.01 to compare the conditions.

### 6.1. *Objective measures*

#### 6.1.1. Team score

A significant effect of the stress feedback on *Team Score* was found, participants achieved better performance with the *Stress* conditions than with the *NoStress* conditions
<figure><img src="image_5.png" /><figcaption>**Figure 4.** Sum of all individual scores obtained by operators. Boxplots show the quartiles with median values in green, while blue lines show the mean values with 95% confidence intervals.</figcaption></figure>

(Stress: median = 135, SD = 45; NoStress: median = 94, SD = 45; W = 266, p < 0.001, r = -0.489, see Fig. 4). This supports **H1**: providing **stress feedback improves performance**. The presentation alone had no significant effect on *Team Score*. Further analysis showed that the *Textual~Stress* condition led to a significantly higher *Team Score* than the *Textual~No Stress* and *Graphical~No Stress* conditions (*Textual~Stress: median = 141, SD = 47; Textual~No Stress: median = 107, SD = 46, W = 59, p = 0.008, r = -0.542; Graphical~No Stress: median = 83, SD = 45, W = 50, p = 0.003, r = -0.603).

### 6.1.2. Trial duration

There was no explicit request to complete the experiment as fast as possible, however, in reviewing the overall *Trial duration*, we could observe a significant effect from both factors and an interaction effect (stress feedback: $F(1,92) = 5.5, p = 0.021, \omega^2 = 0.039$; presentation: $F(1,92) = 10.2, p = 0.002, \omega^2 = 0.080$; interaction: $F(1,92) = 5.5, p = 0.021, \omega^2 = 0.039$, see Fig. 5). The posthoc analysis showed that the *Graphical~Stress* condition led to a longer *Trial Duration* compared to all other conditions (*Graphical~Stress: mean = 766 sec, SD = 71 sec; Textual~NoStress: mean = 648 sec, SD = 121 sec, p < 0.001; Textual~Stress: mean = 648 sec, SD = 91 sec, p < 0.001; Graphical~NoStress: mean = 666 sec, SD = 126 sec, p = 0.007*).

## 6.2. Questionnaires

### 6.2.1. NASA Task Load Index

In reviewing the overall NASA-TLX score, no significant difference was found. However, a detailed assessment of the questionnaire revealed participants felt significantly less mental demand, effort and frustration when provided with stress feedback (mental demand - *Stress: median* = 3, *SD* = 0.92; *NoStress: median* = 4, *SD* = 1.04, *W* = 88, *p* = 0.037, *r* = -0.301; effort - *Stress: median* = 3, *SD* = 0.94; *NoStress: median* = 4, *SD* = 0.98, *W* = 132, *p* = 0.009, *r* = -0.376; frustration - *Stress: median* = 2, *SD* = 0.96; *NoStress: median* = 2, *SD* = 1.29, *W* = 65, *p* = 0.040, *r* = -0.297).

### 6.2.2. Strategy

After each trial, participants were asked to explain their task assignment strategy with open feedback. It revealed that when stress feedback was not provided, the main focus was on the Success component when participants needed to choose who to assign a task to, as an indirect way to assess operators' stress and chances to succeed in tasks. Eighteen participants reported doing so after the *Textual NoStress* condition and 17 after the
<figure><img src="image_6.png" /><figcaption>**Figure 5.** Time (in seconds) trials lasted.</figcaption></figure>

Graphical NoStress condition. Otherwise, the Stress component was the main focus when available. Ten participants stated that their strategy was based on the Stress Component after the *Textual Stress* condition and 16 reported the same after the *Graphical Stress* condition. Having operators' stress feedback, participants aimed at assigning tasks to operators with a LOW or MEDIUM *simulated stress* level. Sixteen participants reported doing so after the *Textual Stress* condition and 12 after the *Graphical Stress* condition. The presentation seems to have little effect on the task assignment strategy except for the Stress component: more participants reported focusing on the stress indicator when it had a graphical presentation than a textual one, which supports **H2**.

### 6.2.3. Component feedback

In the final questionnaire, participants had to choose which components they found most helpful for each condition. The only significant difference is for the Stress component: participants preferred having it displayed with a graphical presentation, which also supports **H2** ($\chi^2(1) = 6.0, p = 0.014$).

An open-ended question asked participants to share their points of view on the textual and graphical presentations of the awareness components (listed in Section 3). Regarding the graphical presentation, 15 participants reported that they liked the Stress component, and 5 said that this presentation eased their understanding of the data. For the textual presentation, 5 participants said they enjoyed the Success component, 6 the Score component, 5 stated that the written numbers were a perk of the textual presentation, and 6 reported that it was easier to understand the data with this presentation.

Asked if they felt that having stress feedback is essential, all participants answered “yes”. They were then asked to explain their answer, and what 11 participants reported was that having stress feedback helps to get a better score. One participant explicitly mentioned that having stress feedback was better than determining the stress level from other components.

In evaluating the possibility of hybrid components (as a mix of textual and graphical presentations), participants were asked which hybrid components they may find helpful. Some participants suggested not doing hybrid presentations of some components and keeping a textual presentation. Conversely, some participants suggested keeping some components with a graphical presentation. No strong tendencies could be identified except for the Task Queue and the Stress components. Among participants having an opinion on the Task Queue component, 6 of them recommended keeping it with a textual presentation, 2 recommended keeping a graphical presentation, and only 1 suggested changing it to a hybrid component. Opinions concerning the Stress component confirmed the preference
for the graphical presentation by 11 participants mentioning it; however, 7 participants still suggested turning it to a Hybrid component, with only 1 suggesting keeping it with a textual presentation.

# 7. Discussion

In answering **RQ1** and **RQ2**, the paper presents three contributions: a set of visualization techniques for team leaders to present members' stress and workload to help manage allocating tasks; a study evaluating the effectiveness of the proposed visualization techniques for team leaders; and an open-access Unity collaborative VR experimental platform we developed: MATB-II-Collab.

> The usage of virtual workers in our experiments enables the investigation of the impact of providing feedback on their workload, individual performance, and inner status on overall team performance. However, it's essential to note that before deploying such visualizations in a team of real people, obtaining explicit consent from team members for sharing their personal data would be mandatory. This process would not be imposed on them but conducted with their informed consent. Importantly, any use of such data should be solely dedicated to improving their well-being at work and must not, under any circumstances, be a tool to exploit people for their ultimate resources.

## 7.1. *Study outcomes*

Results showed that stress feedback significantly improves user performance, supporting **H1**. Stress feedback also resulted in changes to the participants' task assignment strategy. Participants mainly focused on the stress level of operators as it was their primary way to ensure better performance. When the participants did not have stress feedback, they had to conjecture the stress level of operators, having their judgment based on other indicators. With stress feedback, the participants found the process easier to manage the stress level of operators. Overall, most participants could monitor their operators' well-being, which led to better team performance in our system. Providing operators' status feedback to team leaders is more direct and efficient than having them assess how well their team is via performance measures. For future AR information displays to team leaders, the AR information should contain the stress level information of the team members who could potentially receive workload.

Given that a stress indicator improves team performance, the question is which stress presentation provides the most improvement. The results showed that the textual presentation of stress gave a better score, but the participants preferred the graphical presentation, supporting **H2**. Based on user feedback, some of the AR information (the Success and Score components) presented to the team leader is more understandable with a textual presentation due to the presence of numbers and the accuracy they bring. Other AR information, such as stress, was easier to understand with a graphical presentation because it gives a better understanding of the evolution of the value and a global idea of the value. The *Textual Stress* condition was the best performance without the downside of being longer than the *Graphical Stress* condition. Indeed, the analysis of the participants' strategy explanations revealed that whilst participants did not focus on time, the graphical presentation of the stress component was easier and quicker for them to comprehend and complete their tasks, thus, before assigning tasks, participants waited for the stress level of operators to get lower than with the textual presentation. Another factor is that the graph takes more time to read than the *Textual Stress* panel and the other textual indicators. Our experiment did not put time pressure on participants to allow inexperienced people to take part in the study with only basic training for the task. To further study the effect of the presentation style and the presence of stress feedback on the trial duration, a future study would propose a similar task to people with a managing background.

We recommend that collaborative setting designers provide both textual and graphical presentations of information to team leaders. Indeed, the textual information leads to better performance being more precise and processed faster, and the graphical presentation
information is preferred and is enough to get a rough idea of the information at a glance. Pursuing research on sharing people's status should investigate hybrid presentations, to take advantage of both textual precision and graphical abstraction.

## 7.2. Limitations

There are some limitations to this study. We compared only two levels of abstraction of the information; we used a team of virtual operators; we considered that the team leader could see the team members or at least an avatar of theirs; and we ran one-hour-only experiments.

While our study focused on comparing two extremes in terms of abstraction levels in visualizations, it is important to acknowledge the limitation of this approach. The continuum of abstraction offers numerous possibilities, but our goal was to highlight the advantages of each extreme to later recommend a suitable approach. Both graphical and textual information, as evidenced by quantitative results and participant feedback, demonstrated their respective benefits. Consequently, we recommended a balance between presenting raw information textually and offering abstracted information through symbols, a balance that has to be adapted to specific use cases.

We studied the impact of different feedback provided to a team leader with a virtual team where team members were simulated in our simple Operator Model, see Section 4.2. While our experimental setup allowed us to gain valuable insights, it is essential to consider the potential limitations in generalizing our findings to real-world scenarios with human teams and authentic stress measures. Would the same effects of the indicators' presentation and stress feedback be observed in a real team of humans with real stress measures, would the team leaders adopt the same strategy, and would they feel the same empathy towards the team members? Our experimental design intentionally simplified the way stress affected the operators for ease of understanding among participants. However, this simplicity may not fully capture the intricacies of how real stress influences human performance and team dynamics. In our experimental context, the lower limit of the failure probability occurred when stress levels were at the lowest. This differs with theories such as the 'Flow Theory' [48] and 'Yerkes-Dodson's law' [49], suggesting that human performance tends to peak under moderate stress levels rather than low stress. To address the limitations of our current study, a crucial next step would involve conducting experiments with human operators and utilizing physiological sensors to accurately measure stress levels. This approach would offer a more realistic understanding of how stress influences human performance in team settings. Importantly, readers should interpret our findings with the awareness that the simulated stress in our study might not fully capture the nuances of real-world stress, and the extrapolation of results to human teams requires cautious consideration of these experimental constraints.

An additional limitation of our study was that team leaders had visibility of operators; however, the collaborative presence was not systematically measured. The ability of team leaders to visually perceive operators through avatars during the study may have influenced the dynamics of the interaction, creating empathy toward the operators. This empathetic connection might have been further accentuated in the presence of stress feedback, especially when presented graphically. During the study, a subset of participants explicitly reported heightened assertiveness tied to their empathetic response, particularly in situations where stress feedback was incorporated, and even more so with graphical representation. This limitation is crucial to acknowledge as it introduces a variable that may have influenced the team leader's behaviour beyond the specific factors we were measuring. The absence of a systematic measurement of collaborative presence makes it challenging to isolate and quantify the impact of this variable. Consequently, readers need to consider this potential influence when interpreting and generalizing the results of our study. Future research could explore this aspect more explicitly, incorporating measurements of collaborative presence to further understand its role in the dynamics of team interactions within the context of AR-supported environments.
Another limitation of our study pertains to the constrained time allocated to each participant during the experiments. While one hour may seem relatively brief for an investigation into workload balancing and stress management, longer experiments that delve deeper into these aspects could offer valuable insights. However, extending the duration of experiments poses challenges. In fields such as AR VR, one hour is a standard duration, and surpassing this timeframe can be impractical for various reasons, including participant availability and associated costs. Moreover, spending more than an hour in VR raises ethical concerns due to potential mental and physical stress. Despite these limitations, we remain confident that our findings hold relevance for longer team management scenarios. Incorporating workers' inner status feedback is expected to consistently benefit team leaders in making nuanced decisions when assigning workload to a team. While the graphical representation of workers' stress may prove advantageous in longer cases, as suggested in the Study Outcomes section, our recommendation is to maintain a balanced approach by combining both textual and graphical information for a more comprehensive design choice. For future research, it would be valuable to explore longer experiments conducted in a physical setup with human teams to validate and extend our assumptions.

## 8. Conclusion

This study shows us that different presentations of status information and how team members cope affect team performance with a simulated team. Investigating the effect of providing feedback on the users' inner status, we found that it enhances team performance and potentially improves how team leaders consider team members' status. This shows that we might have to change how we manage teams to achieve better team performance and members' well-being. Indeed, providing users' status to team leaders should be highly considered in works where team members must perform their tasks reliably with little to no mistakes. The comparison we conducted between graphical and textual presentations of visual cues suggests that the textual presentation of inner status will achieve better performance, even if people prefer the graphical presentation. The raw performance information presented to a team leader is more understandable with a textual presentation due to the presence of numbers and the accuracy they bring. On the other hand, more complex information, such as inner status feedback, seems more straightforward to understand with a graphical presentation due to the abstraction this presentation provides.

The implications of this study extend beyond the simulated team environment, prompting considerations for implementation in various industries. The application of different presentations of status information and strategies for team coping, as identified in this research, has the potential to reshape team management practices. In real work settings, implementing feedback on users' inner status for team leaders could enhance team performance and contribute to the overall well-being of team members. This shift towards a more empathic teamwork approach may prove valuable in industries where reliable task performance with minimal errors is crucial. While our study focused on colocated teams, the insights gained are transferable to diverse teamwork scenarios, including remote collaborations facilitated through shared workspaces such as Collaborative Virtual Environments.

However, we recognize that the practical implementation of these techniques comes with its challenges. Of course, as soon as real people are implied in such settings, ethical problems will have to be addressed: obviously, sharing user status must follow privacy rules and ethics and must require the consent of monitored subjects. Moreover, factors such as adapting the feedback mechanisms to specific industry needs, addressing potential resistance to change, and integrating these techniques into existing workflows should be considered. Further research and collaboration with industry professionals can provide valuable insights into overcoming these challenges and optimizing the application of our findings in real-world work settings.

Future work will investigate interactions between the team leader and the indicators (resize/dispose of indicators, alerts...). This includes the effect of providing an estimation
of the evolution of the stress level of team members to team leaders. A future direction is to investigate more complex and realistic virtual operators in the experiments.

648

649

**Author Contributions:** Conceptualization, T.R., J.W., C.F., G.C., T.D., and B.T.; data curation, T.R.; formal analysis, T.R., C.F., G.C., and B.T.; funding acquisition, T.D.; investigation, T.R.; methodology, T.R., J.W., C.F., T.D., and B.T.; project administration, T.D. and B.T.; resources, J.W., and B.T.; software, T.R.; supervision, J.W., C.F., G.C., T.D., and B.T.; validation, J.W., C.F., T.D., and B.T.; visualization, T.R.; writing—original draft, T.R.; writing—review and editing, T.R., J.W., C.F., T.D., and B.T. All authors have read and agreed to the published version of the manuscript.

650

651

652

653

654

655

**Funding:** This research was supported by the Centre National de la Recherche Scientifique (CNRS) with international PhD funding, and by the University of South Australia (UniSA) through a PhD scholarship. This work was also supported by French government funding managed by the National Research Agency under the Investments for the Future program (PIA) grant ANR-21-ESRE-0030 (CONTINUUM).

656

657

658

659

660

**Institutional Review Board Statement:** The study was conducted in accordance with the Declaration of Helsinki and approved by the Ethics Committee of the University of South Australia (approved on 20 September 2022, application ID: 204935).

661

662

663

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

664

665

**Data Availability Statement:** The data presented in this study are openly available on GitHub at https://github.com/ThomasRinnert/MATB-II_CollabVR, accessed on 16 November 2023. The MATB-II-Collab application can be found as a Unity 2018 project in this repository. The measures acquired during the user study can be found in the same repository as "XP2_Results.csv".

666

667

668

669

**Acknowledgments:** We would like to thank Nicolas Barbotin, Pierre Bégout, Nicolas Delcombel, and Anthony David for their help with Unity development; Crossing lab members for helping to sort out some ideas - Jean-Philippe Diguet, Cédric Buche, Maëlic Nau, Thomas Ung; IVE PhD students for pilot testing my study (Spencer O’Keeffe, Radhika Jain, Kieran May, Chenkai Zhang); Thomas Clarke, Allison Jing, Adam Drogemuller, and Brandon Matthews for helping with the statistical analysis; and all participants, especially those who told their friends and family about the study.

670

671

672

673

674

675

**Conflicts of Interest:** The authors declare no conflict of interest.

676

## Abbreviations

677

The following abbreviations are used in this manuscript:

678

679

AR Augmented Reality

VR Virtual Reality

MATB-II Multi Attribute Task Battery II

680

## References

681

1. Piumsomboon, T.; Lee, Y.; Lee, G.A.; Dey, A.; Billinghurst, M. Empathic Mixed Reality: Sharing What You Feel and Interacting with What You See. *Proceedings - 2017 International Symposium on Ubiquitous Virtual Reality, ISUVR 2017*, pp. 38–41. https://doi.org/10.1109/ISUVR.2017.20.

682

683

684

2. Sereno, M.; Wang, X.; Besancon, L.; Mcguffin, M.J.; Isenberg, T. Collaborative Work in Augmented Reality: A Survey. *IEEE Transactions on Visualization and Computer Graphics* **2020**, pp. 1–1. https://doi.org/10.1109/tvcg.2020.3032761.

685

686

3. Hotek, D.R. Skills for the 21st Century Supervisor: What Factory Personnel Think. *Performance Improvement Quarterly* **2002**, *15*, 61–83, [https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1937-8327.2002.tb00250.x]. https://doi.org/10.1111/j.1937-8327.2002.tb00250.x.

687

688

689

4. Yang, Y.; Dwyer, T.; Wybrow, M.; Lee, B.; Cordeil, M.; Billinghurst, M.; Thomas, B.H. Towards immersive collaborative sensemaking. *Proceedings of the ACM on Human-Computer Interaction* **2022**, *6*, 722–746.

690

691

5. Gutwin, C.; Greenberg, S. A descriptive framework of workspace awareness for real-time groupware. In *Proceedings of the Computer Supported Cooperative Work. Kluwer Academic Publishers*, 2002, Vol. 11, pp. 411–446. https://doi.org/10.1023/A:1021271517844.

692

693

694

6. Jing, A.; Gupta, K.; Mcdade, J.; Lee, G.A.; Billinghurst, M. Comparing Gaze-Supported Modalities with Empathic Mixed Reality Interfaces in Remote Collaboration. *ISMAR conference track* **2022**, pp. 837–846. https://doi.org/10.1109/ISMAR55827.2022.00102.

695
7. Dey, A.; Piumsomboon, T.; Lee, Y.; Billinghurst, M. Effects of sharing physiological states of players in collaborative virtual reality gameplay. In Proceedings of the Conference on Human Factors in Computing Systems - Proceedings. Association for Computing Machinery, may 2017, Vol. 2017-May, pp. 4045-4056. https://doi.org/10.1145/3025453.3026028.

697

698

699

8. Dey, A.; Chen, H.; Hayati, A.; Billinghurst, M.; Lindeman, R.W. Sharing manipulated heart rate feedback in collaborative virtual environments. *Proceedings - 2019 IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2019*, pp. 248-257. https://doi.org/10.1109/ISMAR.2019.00022.

700

701

702

9. Beck, S.; Kunert, A.; Kulik, A.; Froehlich, B. Immersive Group-to-Group Telepresence. *IEEE Transactions on Visualization and Computer Graphics* **2013**, *19*, 616-625. https://doi.org/10.1109/TVCG.2013.33.

703

704

10. Yoon, B.; Kim, H.I.; Lee, G.A.; Billinghurst, M.; Woo, W. The effect of avatar appearance on social presence in an augmented reality remote collaboration. *26th IEEE Conference on Virtual Reality and 3D User Interfaces, VR 2019 - Proceedings 2019*, pp. 547-556. https://doi.org/10.1109/VR.2019.8797719.

705

706

707

11. Fraser, M.; Benford, S.; Hindmarsh, J.; Heath, C. Supporting awareness and interaction through collaborative virtual interfaces. In Proceedings of the UIST (User Interface Software and Technology): Proceedings of the ACM Symposium, 1999, Vol. 1, pp. 27-36. https://doi.org/10.1145/320719.322580.

708

709

710

12. Bai, H.; Sasikumar, P.; Yang, J.; Billinghurst, M. A User Study on Mixed Reality Remote Collaboration with Eye Gaze and Hand Gesture Sharing. *Conference on Human Factors in Computing Systems - Proceedings 2020*, pp. 1-13. https://doi.org/10.1145/3313831.3376550.

711

712

713

13. Niewiadomski, R.; Hyniewska, S.J.; Pelachaud, C. Constraint-based model for synthesis of multimodal sequential expressions of emotions. *IEEE Transactions on Affective Computing* **2011**, *2*, 134-146. https://doi.org/10.1109/T-AFFC.2011.5.

714

715

14. Irlitti, A.; Smith, R.T.; Itzstein, S.V.; Billinghurst, M.; Thomas, B.H. Challenges for Asynchronous Collaboration in Augmented Reality. *Adjunct Proceedings of the 2016 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2016* **2017**, pp. 31-35. https://doi.org/10.1109/ISMAR-Adjunct.2016.0032.

716

717

718

15. Lee, Y.; Masai, K.; Kunze, K.; Sugimoto, M.; Billinghurst, M. A Remote Collaboration System with Empathy Glasses. In Proceedings of the 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct), 2016, pp. 342-343. https://doi.org/10.1109/ISMAR-Adjunct.2016.0112.

719

720

721

16. Lazarus, R.S.; Folkman, S. *Stress, appraisal, and coping*; Springer publishing company, 1984.

722

723

17. Benford, S.; Fahlén, L. A Spatial Model of Interaction in Large Virtual Environments. In Proceedings of the Proceedings of the Third European Conference on Computer-Supported Cooperative Work 13-17 September 1993, Milan, Italy ECSCW '93, 1993, pp. 109-124. https://doi.org/10.1007/978-94-011-2094-4_8.

724

725

18. LeChénéchal, M. Awareness Model for Asymmetric Remote Collaboration in Mixed Reality. PhD thesis, INSA de Rennes, 2016.

726

727

19. Becquet, V.; Letondal, C.; Vinot, J.L.; Pauchet, S. How do Gestures Matter for Mutual Awareness in Cockpits? *DIS '19* **2019**, pp. 593-605. https://doi.org/10.1145/3322276.3322306.

728

729

20. Huang, W.; Kim, S.; Billinghurst, M.; Alem, L. Sharing hand gesture and sketch cues in remote collaboration. *Journal of Visual Communication and Image Representation* **2019**, *58*, 428-438. https://doi.org/10.1016/j.jvcir.2018.12.010.

730

731

21. Piumsomboon, T.; Day, A.; Ens, B.; Lee, Y.; Lee, G.; Billinghurst, M. Exploring enhancements for remote mixed reality collaboration. *SIGGRAPH Asia 2017 Mobile Graphics and Interactive Applications, SA 2017* **2017**, pp. 1-5. https://doi.org/10.1145/3132787.3139200.

732

733

22. LeChénéchal, M.; Duval, T.; Gouranton, V.; Royan, J.; Arnaldi, B. Help! I Need a Remote Guide in My Mixed Reality Collaborative Environment. *Frontiers in Robotics and AI* **2019**, *6*, https://doi.org/10.3389/frobt.2019.00106.

734

735

23. Gupta, K.; Lee, G.A.; Billinghurst, M. Do you see what I see? the effect of gaze tracking on task space remote collaboration. *IEEE Transactions on Visualization and Computer Graphics* **2016**, *22*, 2413-2422. https://doi.org/10.1109/TVCG.2016.2593778.

736

737

24. Piumsomboon, T.; Dey, A.; Ens, B.; Lee, G.; Billinghurst, M. The effects of sharing awareness cues in collaborative mixed reality. *Frontiers Robotics AI* **2019**, *6*, 1-18. https://doi.org/10.3389/frobt.2019.00005.

738

739

25. Hart, J.D.; Piumsomboon, T.; Lee, G.; Billinghurst, M. Sharing and Augmenting Emotion in Collaborative Mixed Reality. *Adjunct Proceedings - 2018 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2018* **2018**, pp. 212-213. https://doi.org/10.1109/ISMAR-Adjunct.2018.00069.

740

741

26. Duval, T.; Fleury, C. An asymmetric 2D Pointer / 3D Ray for 3D Interaction within Collaborative Virtual Environments. *Proceedings of Web3D 2009: The 14th International Conference on Web3D Technology* **2009**, *1*, 33-41. https://doi.org/10.1145/1559764.1559769.

742

743

27. Teo, T.; Lee, G.A.; Billinghurst, M.; Adcock, M. Investigating the use of different visual cues to improve social presence within a 360 mixed reality remote collaboration. *Proceedings - VRCAI 2019: 17th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and its Applications in Industry* **2019**, https://doi.org/10.1145/3359997.3365687.

744

745

746

28. Ajili, I.; Mallem, M.; Didier, J.Y. Human motions and emotions recognition inspired by LMA qualities. *Visual Computer* **2019**, *35*, 1411-1426. https://doi.org/10.1007/s00371-018-01619-w.

747

748

29. Ajili, I.; Ramezanpanah, Z.; Mallem, M.; Didier, J.Y. Expressive motions recognition and analysis with learning and statistical methods. *Multimedia Tools and Applications* **2019**, *78*, 16575-16600. https://doi.org/10.1007/s11042-018-6893-5.

749

750

30. Luong, T.; Martin, N.; Raison, A.; Argelaguet, F.; Diverrez, J.M.; Lecuyer, A. Towards Real-Time Recognition of Users Mental Workload Using Integrated Physiological Sensors into a VR HMD. *Proceedings - 2020 IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2020* **2020**, pp. 425-437. https://doi.org/10.1109/ISMAR50242.2020.00068.

751

752

753

754

755
31. Luong, T.; Argelaguet, F.; Martin, N.; Lecuyer, A. Introducing Mental Workload Assessment for the Design of Virtual Reality Training Scenarios. In Proceedings of the Proceedings - 2020 IEEE Conference on Virtual Reality and 3D User Interfaces, VR 2020. IEEE, 2020, pp. 662-671. https://doi.org/10.1109/VR46266.2020.1581086856229.

754

755

756

32. Bowman, D.A.; Wingrave, C.A. Design and evaluation of menu systems for immersive virtual environments. *Proceedings - Virtual Reality Annual International Symposium 2001*, pp. 149-156. https://doi.org/10.1109/vr.2001.913781.

757

758

759

33. Wagner, J.; Nancel, M.; Gustafson, S.; Wendy, E.; Wagner, J.; Nancel, M.; Gustafson, S.; Body, W.E.M.A.; Wagner, J.; Nancel, M.; et al. A Body-centric Design Space for Multi-surface Interaction. *CHI '13: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems 2013*.

760

761

762

34. Ready at Dawn. *Lone Echo*, 2017.

763

764

35. Ramen VR. *Zenith: The Last City*, 2022.

765

766

36. Bégout, P.; Duval, T.; Kubicki, S.; Charbonnier, B.; Bricard, E. WAAT: A Workstation AR Authoring Tool for Industry 4.0. *Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 2020*, 12243 LNCS, 304-320.

767

768

769

37. Sasikumar, P.; Pai, Y.S.; Bai, H.; Billinghurst, M. PSCVR: Physiological Sensing in Collaborative Virtual Reality. *IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 2022*. https://doi.org/10.1109/ISMAR-Adjunct57072.2022.00137.

770

771

772

773

38. Rinnert, T.; Walsh, J.; Fleury, C.; Coppin, G.; Duval, T.; Thomas, B.H. How Can One Share a User's Activity during VR Synchronous Augmentative Cooperation? *Multimodal Technologies and Interaction 2023*, 7. https://doi.org/10.3390/mti7020020.

774

775

776

39. Gattullo, M.; Scurati, G.W.; Evangelista, A.; Ferrise, F.; Fiorentino, M.; Uva, A.E. Informing the Use of Visual Assets in Industrial Augmented Reality. In Proceedings of the Design Tools and Methods in Industrial Engineering; Rizzi, C.; Andrisano, A.O.; Leali, F.; Gherardini, F.; Pini, F.; Vergnano, A., Eds., Cham, 2020, pp. 106-117.

777

778

779

40. Gattullo, M.; Uva, A.E.; Fiorentino, M.; Monno, G. Effect of Text Outline and Contrast Polarity on AR Text Readability in Industrial Lighting. *IEEE Transactions on Visualization and Computer Graphics* **2015**, *21*, 638-651. https://doi.org/10.1109/TVCG.2014.2385056.

780

781

41. Gattullo, M.; Uva, A.E.; Fiorentino, M.; Gabbard, J.L. Legibility in Industrial AR: Text Style, Color Coding, and Illuminance. *IEEE Computer Graphics and Applications* **2015**, *35*, 52-61. https://doi.org/10.1109/MCG.2015.36.

782

783

784

42. Gattullo, M.; Scurati, G.W.; Fiorentino, M.; Uva, A.E.; Ferrise, F.; Bordegoni, M. Towards augmented reality manuals for industry 4.0: A methodology. *Robotics and Computer-Integrated Manufacturing* **2019**, *56*, 276-286. https://doi.org/https://doi.org/10.1016/j.rcim.2018.10.001.

785

786

787

788

43. Santiago-Espada, Y.; Myer, R.R.; Latorella, K.A.; Comstock, J.R. The Multi-Attribute Task Battery II (MATBII): Software for human performance and workload research: A user's guide. *NASA Tech Memorandum 217164 2001*.

789

790

44. Aristidou, A.; Lasenby, J.; Chrysanthou, Y.; Shamir, A. Inverse kinematics techniques in computer graphics: A survey. In Proceedings of the Computer graphics forum. Wiley Online Library, 2018, Vol. 37, pp. 35-58.

791

792

45. Hart, S.G.; Staveland, L.E. Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research. In *Human Mental Workload; Hancock, P.A.; Meshkati, N., Eds.; North-Holland, 1988; Vol. 52, Advances in Psychology*, pp. 139-183. https://doi.org/https://doi.org/10.1016/S0166-4115(08)62386-9.

793

794

46. Field, A.; Miles, J.; Field, Z. *Discovering statistics using R;* Sage publications, 2012.

795

796

47. Robertson, J.; Kaptein, M. *An introduction to modern statistical methods in HCI;* Springer, 2016.

797

798

48. Csikszentmihalyi, M. The flow experience and its significance for human psychology. *Optimal experience: Psychological studies of flow in consciousness* **1988**, *2*, 15-35.

799

800

49. Yerkes, R.M.; Dodson, J.D.; et al. The relation of strength of stimulus to rapidity of habit-formation. *Journal of Comparative Neurology & Psychology* **1908**.

801

802

803

804

**Disclaimer/Publisher's Note:** The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

805

806

807

808