# Pilot study on virtual assistant design for new generation aircraft cockpit

Marlène Gilles, Cédric Bach, Mathieu Quentel, Nico Pallamin, Guillaume

Jégou, Ronan Querrec, Elisabetta Bevacqua

► To cite this version:

Marlène Gilles, Cédric Bach, Mathieu Quentel, Nico Pallamin, Guillaume Jégou, et al. Pilot study on virtual assistant design for new generation aircraft cockpit. Behaviour and Information Technology, 2025, pp.1-12. 10.1080/0144929X.2025.2507691. hal-05322062

HAL Id: hal-05322062

https://hal.science/hal-05322062v1

Submitted on 20 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Pilot study on virtual assistant design for new generation aircraft cockpit

Marlène Gilles¹, Cédric Bach², Mathieu Quentel³, Nico Pallamin³, Guillaume Jégou³, Ronan Querrec¹, and Elisabetta Bevacqua¹

¹ ENIB, Lab-STICC UMR 6285 CNRS, Brest, France

² HDG - Human Design Group, Versailles, France

³ IRT b<>com, Brest, France

May 13, 2025

**Abstract**

In fields like military aviation, virtual assistance is crucial to manage the overwhelming amount of information pilots deal with, reducing their workload. This study seeks to determine the key factors for effective collaboration between pilots and assistants, focusing on embodiment and communication style, which may vary based on the situation criticality. A pilot evaluation show that both embodiment and communication style should be adapted to operational context, especially critical ones (i.e. presence of VA after a loss of consciousness and task oriented communication style during combat phases).

***Keywords***— Virtual assistant; adaptive behavior; embodiment; communication

1 Introduction

Advancements in artificial intelligence (AI), natural language processing, and embodied conversational agents (ECA) have paved the way for virtual assistants (VA) with human-like cognitive and expressive abilities. These VAs can engage users in intuitive and natural interactions, offering more efficient assistance [Breazeal et al., 2005, Kontogiorgos et al., 2019, Kim et al., 2018].

This pilot study focuses on VAs designed specifically for military aircraft crews. In military aviation, assistance is becoming increasingly crucial due to the substantial information burden and intense mental pressure, especially in single-pilot operations where the presence of a second pilot for support is absent [Vu et al., 2018]. We argue that a VA could be beneficial in such scenarios and aim to investigate some characteristics that a VA should possess for effective VA design and their optimal utilization. We focus on two characteristics: the VA's representation, which encompasses their humanoid embodiment with non-verbal capabilities, and their communication style, which pertains to how assistants interact verbally with the user.

We hypothesize that these two characteristics should be adaptive, tailored to meet the specific needs of pilots. While the concept of user-adaptive VAs is not new, integrating both cognitive and social dimensions into their interaction, especially in terms of dialogue and representation, remains unexplored.

2 Related work

2.1 Virtual assistants

Gilles and Bevacqua [2022] conducted an extensive
review to pinpoint essential human characteristics and skills that VAs should be endowed with to improve communication and collaboration between humans and machines, especially in transportation.

Primarily, cognitive abilities enabling the VA to accurately comprehend user input are crucial (Bernard, 2017). Additionally, effective communication skills are important [Bernard, 2017]. The VA should introduce itself, self-reference, and use pronouns to facilitate exchanges and minimize misunderstandings [Large et al., 2019]. It must also consider context, anticipate or identify misunderstandings, and rectify them through rephrasing or direct questions [Skantze, 2005]. Communication style significantly influences users' perception of the virtual assistant [Chattaraman et al., 2019]. A socially-oriented approach, characterized by verbose and friendly communication, enhances interactivity and trust [Rheu et al., 2021]. Conversely, a concise and task-oriented style reduces information overload and enhances perceived efficiency [Perez Garcia and Saffon Lopez, 2019].

Embodiment is another characteristic that enhances interactions. Beyond helping users perceive the assistant's boundaries and increasing its influence [Phillips et al., 2017], embodiment enables non-verbal behaviors that convey substantial information (e.g., turn-taking management [Ehret et al., 2023], agreement/disagreement [Heylen et al., 2007], personality Gebhard et al. [2014], persuasiveness and politeness [Zojaji et al., 2023], making interactions more natural and fluid [Holroyd et al., 2011]. Being embodied with social behavior heightens the perception of social presence [Kontogiorgos et al., 2019] and subsequently boosts trust in the assistant Phillips et al. [2017]. However, embodiment may be inappropriate or counterproductive when users need to stay focused on a task [Kontogiorgos et al., 2019].

Virtual assistants must also possess a natural voice, with expressiveness and speed adapted to the context, and consistent with the VA's physical representation, if any [Martin et al., 2005, Aylett et al., 2013]. Personality traits and emotional state are also crucial, aligning with the user's traits for better efficiency and to avoid unnecessary mental workload [Braun et al., 2019, Nass et al., 2005].

The subsequent discussion in this article will exclusively focus on embodiment and communication style, as research by Gilles and Bevacqua [2022] highlights these as central to optimal VA characteristics.

## 2.2 Assistants in military domain

In recent decades, there has been a growing interest in integrating cognitive assistants into military aviation to support pilots during flight tasks. There is particular emphasis on aiding pilots in single-pilot operations, which is considered the future of aviation, driven by advancements in computer systems, artificial intelligence (AI), and human-machine interfaces (HMI) [Vu et al., 2018, Lim et al., 2018]. These assistants play a crucial role because pilots alone cannot manage the overwhelming volume of information and options. The system should intelligently respond to user commands, provide necessary information, assess situations, offer execution assistance if requested, engage in dialogue implicitly or explicitly, and maintain a user-friendly, non-intrusive interface [Eggleston, 1997]. While the system design must be flexible, it should also be simple and consistent to prevent cognitive overload [Taylor et al., 2001].

In single-pilot operations, there is a need for functionalities to evaluate the pilot's condition and adapt the system accordingly [Lim et al., 2018]. Various adaptive system concepts have emerged over the years, from Super Cockpit in the 1980s [Furness, 1986] to CAMMI [Piuzzi et al., 2014], all aiming to provide human-like assistance. These concepts envision real-time adaptation of interfaces to minimize the pilot's cognitive load while maximizing situational awareness, although some pilots prefer configurable options [Banks and Lizza, 1991]. Additionally, the CAMA system integrates a dialogue management module to enhance situational awareness across all flight conditions [Onken and Walsdorf, 2001]. Tested in simulators and actual flights, CAMA proved highly beneficial, improving situational awareness and gaining acceptance from pilots.

More recently, Estes et al. [2016] developed the Digital Copilot. This system incorporates built-in on-demand assistance, utilizing speech recognition for queries and visual/auditory interfaces for context-
tual notifications. Like CAMA, the Digital Copilot aims to reduce pilot workload and enhance situational awareness. Thoroughly tested, the Digital Copilot seems to yield promising results.

While the existing prototypes mentioned above advocate for adaptive systems, none have explored the assistant's embodiment or communication style.

# 3 Aims and objectives

This study aims to determine the most effective type of virtual assistant for specific operational scenarios and cognitive states. It seeks to pinpoint the optimal level of embodiment and communication style for seamless interaction with aircraft crews. While past research underscores the crucial impact of these dimensions [Gilles and Bevacqua, 2022], consensus on their ideal implementation remains elusive. Both communication styles and embodiments can vary based on the situation, environment, and interaction goals. The study outlines two primary objectives.

The main objective concerns the embodiment and tries to answer the following questions:

- Does the humanoid embodiment of the VA has an impact on the pilot's performance?

- Does the transition from embodiment to non-embodiment impact the pilot's performance?

- Should the level of embodiment be adaptive?

The secondary objective focuses on the communication style and raises the following research questions:

- Does communication style (task-oriented/social-oriented) impact the pilot's performance?

- Should the communication style be adaptive?

- If so, should the communication style be adaptive in the same way as the embodiment level? Is there a correlation between the level of embodiment and the communication style?

Therefore, we focused on these two aspects to design the study, but other interrelated points, which are strategic issues, could be further discussed during debriefing sessions.

## 3.1 Selection of parameters and experimental variables

Based on the research questions and objectives outlined earlier, we identified the dependent, independent, and controlled variables. To reduce development complexity, we adopted the Wizard of Oz (WOz) technique. In this approach, an operator simulates the VA's intelligence, adapting to the operational context and the pilot's requests to mimic how a VA would function in an actual aircraft. The same operator was used throughout the experiment to minimize variability in responses to similar requests across different pilots.

Secondary dimensions crucial for primary evaluation, such as physical support, emotion, and personality, are predefined based on current knowledge. However, variables like the VA's gender or familiarity, which are more challenging to conceptualize, are left to participants' discretion. The study's variables focus on embodiment levels and communication style. For embodiment, we test the following variables:

- VA vocal materialization,

- physical embodiment (that is the assistant human-like representation),

- adaptive embodiment fluctuates between vocal-only interaction and physical materialization, depending on the operational context. For example, the vocal-only mode is used during combat or high-workload phases, while physical materialization occurs during low-workload periods or following a loss of consciousness.

Those corresponding to the style of communication are:

- social-oriented (social, prolix) communication,

- task-oriented (directive, brief) communication,

- adaptive communication, which varies between social and task-oriented approaches depending on the situation and workload. While the latter is considered an “ideal” communication style, it remains fixed by expertise.
# 4 Pilot study protocol

This section presents all aspects of the study design, from the scenario and the experimental organization, the profile of the participants, the devices used, their communication and the assistant's animation, to the data collected.

## 4.1 Scenario

Approximately ten iterations with domain experts were required to develop an operational scenario that aligned with the research objectives and ensured the credibility of the VA-related results concerning the embodiment levels and communication styles under study. An intermediary, such as an operational engineer in military operations or a military test pilot, facilitated communication between the project team and the experts. This intermediary provided general examples of cockpit interactions without disclosing classified information, guiding iterative improvements to the scenario until it was fully scripted.

The chosen scenario, designed to balance complexity and operational realism, consists of four main phases: (1) initial cockpit preparation for a mission, (2) mission execution, (3) mid-flight mission re-tasking and cockpit preparation, and (4) hostile attack resulting in simulated loss of consciousness. These phases were designed to simulate realistic military scenarios and progressively increase participant workload, with the final phase presenting the highest demand.

Although the scenario's structure remains consistent across participants, operational variables (such as mission location, available aircraft resources, and the type of hostile aircraft) are varied between sessions to effectively assess situational awareness.

The scenario duration ranges from 7 to 12 minutes, depending on pilot-VA interactions. Pilots interact with the assistant exclusively through voice, without control over the virtual environment or aircraft operations.

## 4.2 Plan of the pilot study

Since the pilot study involved active pilots, who were instructed not to fly within 24 hours of using a VR headset, each participant was allocated a full day for the experiment. The study was divided into two phases to allow time for debriefing and to manage the risk of cybersickness. In the morning, the embodiment dimension was explored using adaptive communication. In the afternoon, pilots selected two embodiment levels for the communication style tests.

This design encompassed 7 experimental conditions (vocal only/adaptive communication, always embodied/adaptive communication, adaptive embodiment/adaptive communication, 2 embodiment choices/social-oriented communication, and 2 embodiment choices/task-oriented communication). Conditions were not counterbalanced to ensure all pilots experienced conditions relevant to the main objectives, considering the potential for cybersickness that could interrupt the experiment too early. We hypothesize that an adaptive embodiment will be largely favored, as well as an adaptive communication style.

A typical day began with briefing and demographic collection, followed by VR helmet adjustment and VA selection. Pilots had the option to choose one of the four available assistants and personalize the dialogue (e.g., pilot's name, level of formality). A scenario without the VA served as the baseline, followed by a debriefing. Three experimental sessions (10 minutes each) were then conducted to test different levels of embodiment, with immediate debriefings after each. Post-experiment interviews were used to evaluate embodiment characteristics and to select preferred levels for the communication tests.

The afternoon consisted of four sessions, each followed by a debriefing. Pilots were then allowed to complete one final session using the combination of parameters of their choice to conclude the day. A final debriefing was conducted to evaluate the overall experiment, the assistant's characteristics, and the pilots' opinions on this type of assistance.

Experimental choices were based on a plan allowing for testing 7 conditions efficiently, considering logistical and safety concerns. Having each participant test
all conditions would have prolonged the duration of
headset use, which is known to increase the risk of cy-
bersickness, as outlined in previous studies [Stanney
et al., 2020, Khenak et al., 2023].

4.3 Participants

The participants were active military pilots selected
for their operational experience, providing valuable
insights into the potential applications of the VA in
current and future missions. These five experienced
French male pilots, approximately 30 years old, had
minimal prior exposure to virtual reality, primarily
limited to simulator training.

Each participant signed an informed consent form
detailing the main aspects of the experiment. The
consent form adhered to standard protocols and in-
cluded a generic General Data Protection Regula-
tion (GDPR) agreement covering the recording of
physiological and computer data.

4.4 Setup

Due to the unavailability of a flight simulator, the
experiment takes place in virtual reality. The ex-
perimental setup comprises three main components:
the virtual environment where participants are im-
mersed, the Wizard of Oz (WOz) interface used by
the experimenter to control this environment and the
VA which can engages in dialogue with the pilot. The
virtual environment and the VA are developed using
Unity3D and communicate with the WOz interface
via a WebSocket server.

Participants engage with the virtual environment
using HTC Vive Pro Eye headsets, which include eye
tracking sensor. While these sensors are not directly
integrated into scenario development, they are used
to simulate the capability to detect loss of conscious-
ness.

Initially, participants find themselves in the cock-
pit of a fighter aircraft stationed at a generic military
airbase<fnref n="1" />, with a night atmosphere chosen to limit vis-
ibility and maintain simplicity.

Throughout the experiment, participants receive
information solely through the VA and cockpit
screens displaying static web pages, with content

updated by the WOz operator based on predefined
parameters (such as the aircraft destination, its re-
sources, targets, etc.). The WOz operator, situated
in a separate room, controls the scenario through a
dedicated interface—a static web page querying the
same APIs as participants' interfaces to load environ-
ment states and trigger events.

Regarding audio, the WOz operator accesses dia-
logue between the VA and participants via the open-
source software Mumble<fnref n="2" />. Participants, who were in-
formed that they were taking part in a WoZ-type
methodology, could only hear the VA and, in the case
of technical issues, an observer present in the room.
This observer, unseen by the participant in the vir-
tual environment, takes notes on the experiment's
progress and any comments.

The WOz operator oversees the experiment's progress using an interface that guides them through scenario transitions after each state, ensuring prompt and accurate responses. To facilitate the responsiveness of the operator, scripting of the mission was implemented using state machines in Mascaret language [Querrec et al., 2013]. The scenario's macroscopic steps were structured using various state machines, each consisting of individual states that the WOz operator could trigger at will through the interface. Each state corresponds to specific reactions from the VA, as well as changes to be automatically applied to the attributes of the virtual environment's entities. These attributes include mission details, no-strike lists, targets, rules of engagement, aircraft status, screen parameters, and mission presets. In the event of an unexpected occurrence, such as an unplanned question from the pilot, the WOz operator can assume control and manipulate in real-time the value of the entities' attributes, as well as the VA behavior by inputting sentences or selecting non-verbal behaviors, such as backchannel signals, for the agent to carry out. The interface was rigorously tested for consistency and usability in multiple pre-test series by the designer, who also served as the sole WOz

<footnote n="1">1. Scenery from https://assetstore.unity.com/packages/3d/environments/roadways/military-airport-scene-162161</footnote>

<footnote n="2">2. Mumble software: https://www.mumble.info/</footnote>
<figure><img src="image_2.png" /><figcaption>Figure 1: (a) Deictic gesture: Self-pointing; (b) emblem gesture “wave”, used for greeting; (c-d) beat gestures: hand down and hand up; (e) emblem gesture “wipe”. From [Kipp, 2005].</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>Figure 2: The VA humanoid representation in the virtual environment</figcaption></figure>

operator during official tests. This testing process allowed for control optimization and the development of reflexes needed for smooth real-time manipulation during experiments.

When the VA possesses a humanoid representation, it appears on the left of the user and overlaid onto the cockpit (as shown in Figure 2). It is embodied through an ECA architecture based on the standard SAIBA framework [Vilhjálmsson et al., 2007] for generating virtual agent behaviors. Our in-house conversational agent utilizes Behavior Markup Language (BML) messages to specify verbal and non-verbal behaviors, which are then transformed into animations. Any agent's verbal response causes the

invocation of the voice synthesizer CereVoice³ [Aylett and Pidcock, 2007] which transforms the text into a synthetic voice (male or female) and provides temporal information to synchronize speech with gestures or other non-verbal behaviors.

In this study, the assistant exhibits neutral, positive (smile), and negative (frown) expressions, as well as directing its gaze and nodding. Within the cockpit, the assistant can point to interfaces, itself, or the pi-lot, and perform conversational gestures sourced from existing literature [Kipp, 2005] (see Figure 1).

Four agent renderings, varying in gender and eth-
nicity, were provided for pilot selection, generated using the Autodesk Character Generator web application⁴. Non-verbal signals, described at an abstract level, remain consistent across agents but adapt to individual morphology.

Regarding dialogue, the agent initiates by introducing itself and inquiring about the pilot's preferences for naming and familiarity. The scenario is meticulously crafted to identify various communicative intentions for the agent during interactions with the pilot. Multiple sentences are composed for each intention, in both social and task-oriented communication styles. Whenever the VA intends to communicate, a sentence is randomly chosen based on the current communication style. Facial expressions and gestures are then added to emphasize keywords, improving communication. These non-verbal behaviors occur randomly but not systematically, introducing variability to the agent's responses.

## 4.5 Data collection

The data for this user test is varied and collected in different forms. Standard questionnaires were used for measures of workload, cybersickness, sense of presence and acceptance of new technologies. These questionnaires were complemented by physiological measurements and debriefing sessions.

The ISA scale [Jordan, 1992] allowed the participants to estimate by expertise their level of workload for each sequence of the scenario.

The feeling of presence is an important aspect in the field of Virtual Reality. It is characterized by the feeling of being in the virtual environment and not disconnected from it. We used the Witmer's scale [Witmer et al., 2005] to measure the presence.

The last scale measure the acceptability of new technologies developed from the TAM model Alharbi and Drew [2014] (see the filled in questionnaire in Figure 3).

During the experiment, participants were outfitted with various physiological sensors. Cardiac and electrodermal activities were measured using a BITalino sensor setup, and eye tracking was captured with an integrated sensor from the HTC Vive Pro Eye. The presence of these sensors is coherent with the scenario

in which the VA can detect the mental workload and the loss of consciousness based on the monitoring of physiological data. [Luong et al., 2020]

After each condition, at the end of the morning and at the end of the day, a debriefing interview was conducted using a semi-structured interview grid. It addressed the general feeling about the tested condition, some details about the awareness situation, the feeling of having identified and integrated the essential information, the ability to give details about the hostile aircrafts and about the non-strike list of the missions, the help or hindrance that the VA constituted for the awareness situation, as well as the positives and the negatives of the experimental condition being evaluated. The end-of-morning interview was based on the following questions:

- We tested 3 experimental conditions in addition to the baseline situation, which one did you prefer?

- We will test your two preferred embodiments, but with other modes of communication from the agent. Which embodiment would you like to test with these other modalities?

- What did you think of the way the VA spoke to you?

- Can you imagine other ways to present the VA?

- How do you imagine cooperation with this VA in the future, in a real cockpit?

- Are the tasks delegated to the VA appropriate? Can you imagine other tasks?

- Can we imagine other avatars in the cockpit, such as avatars representing drones or ground personnel?

- In the end, is the VA useful?

Finally, during the end-of-day interview, we asked the participant for their impressions of the day and which combination they preferred among all those tested.

4. Autodesk Character Generator web application: https://charactergenerator.autodesk.com/
# 5 Results of the pilot study and discussion

This section presents the most significant results, par-
ticipant feedback, and perspectives from the study.
The findings are primarily qualitative, based on a the-
matic analysis of debriefing interviews conducted us-
ing the MaxQDA qualitative data analysis software⁵,
as well as data from various evaluation scales. These
results are detailed in the following sections.

## 5.1 Embodiment

The embodiment dimension refers to the different
forms of representation of the VA, tested under three
conditions: no embodiment (entirely vocal interac-
tion), a permanent humanoid embodiment, and an
adaptive embodiment that varies according to the
scenario and workload.

The embodiment selected by pilots at the end of
the morning session, to continue with the communi-
cation style tests, is particularly noteworthy. Con-
trary to our expectations, all pilots selected the ver-
sion without any embodiment. Two also chose the
fully embodied version, while the remaining pilots
opted for the adaptive embodiment.

Although their selections might suggest otherwise,
the analysis of post-session verbatim reveals a unan-
imous rejection of the permanent humanoid embod-
iment. The reasons for this rejection are mainly as
follows (with the number of pilots mentioning each
theme in parentheses):

- permanent embodiment attracts too much attention and/or generates distraction (5);

- permanent embodiment is not useful (5);

- the positioning of the VA is not appropriate (3);

- permanent embodiment is too intrusive (3);

- permanent embodiment generates too many parasitic gestures for the VA (2).

The study found that three pilots preferred an adap-
tive humanoid form of the VA, provided its appear-
ance was limited to specific conditions: at the pilot's

request, in response to predefined events, or during
situations directly impacting the aircraft's or pilot's
survivability. These pilots felt that the humanoid em-
bodiment conveyed a sense of team membership, in
contrast to the impersonal nature of a voice-only in-
terface. Moreover, its presence was described as reas-
suring following episodes of consciousness loss, posi-
tively influencing perceived performance.

## 5.2 Communication

The communication dimension refers to the style of
communication that can be generated. As a re-
minder, the latter is either directive (also called task-
oriented communication) or more prolix (also called
social-oriented communication). Participants com-
mented extensively on the communication style of
the VA, which was found to be critical to the per-
formance of the cooperation between the pilot and
the VA, especially when the voice intonation conveys
the criticality and seriousness of the situation.

All the pilots agreed that, as it stands, even
the task-oriented communication was not sufficiently
concise for combat times. The pilots' opinions im-
prove over the course of the day, as participants
learned to control the interactions and requests with
the VA that was piloted by the WOz operator. But
in any case, the most critical point for the pilots was
the fact that the VA cannot be interrupted on de-
mand or asked to be quiet, a consequence of technical
test limitations.

Table 1 shows the preferred conditions at the end
of the day. It can be seen that there is no conver-
gence that emerges from the experimental combina-
tions. Nevertheless, all the pilots preferred the last
condition they tested during the day. It is explained
by the fact that the pilots modulated their coopera-
tion with the VA thanks to the flexibility of the WOz
methodology. In other words, the pilots transformed
the experimental session into a real fine-tuning ses-
sion of their cooperation with the VA. In this way,
over the course of the day, they designed a more ef-
fective cooperation over the duration of the scenario.
In all cases, a concise and controlled communication

5. MaxQDA software: https://www.maxqda.com/
with intonations adapted to the criticality of the con-
text was the consensus.

Debriefing interviews from a study on pilot-Virtual
Assistant (VA) communication revealed key factors
for enhancing interaction. Pilots underscored the
importance of adapting VA intonation to the spe-
cific context, urgency, criticality, and risk levels,
advising for communication that is factual, con-
cise, and task-focused. This approach, supported by
[Yaghoubzadeh et al., 2015], aims to minimize misun-
derstandings and discourages unnecessary politeness
or feedback, calling for consistent language use. The
establishment of distinct communication modes for
normal and combat scenarios was proposed to im-
prove pilot-VA cooperation. Additionally, the priori-
tization of communication tasks and alignment with
shared responsibilities were emphasized, with a no-
table focus on allowing pilots to interrupt the VA
under high workload conditions, reflecting concerns
identified by Budiu and Laubheimer [2018] about the
usability of commercial personal assistants.

Evaluation sessions demonstrated that at least
three pilots were able to effectively initiate and man-
age dialogue with the VA, suggesting that a well-
structured voice modality can enhance situational
awareness and workload management, complement-
ing the aircraft's existing alert systems. The lack of
requests for additional testing runs on different em-
bodiments or communication strategies suggests pi-
lots believed the experiments thoroughly addressed
their experience and perception, affirming the exper-
imental protocol's approach to not require each pilot
to test every combination but ensure comprehensive
coverage.

5.3 Sense of presence

A total of thirty-five presence scales were adminis-
tered to participants to gauge their sense of pres-
ence during interactions with the Virtual Assistant
(VA) across various embodiments and communica-
tion styles, with the results detailed in Table 2. The
scale's maximum score is 154, with a median of 76.
Utilizing validation data from the University of Que-
bec's Sense of Presence Questionnaire, a score below
66 is indicative of significant non-presence, around

104 signifies average presence, and scores above 104 denote good presence, differentiated in the table by color coding (green for scores above 104 and red for below 66). The bold score denotes the final session for the pilots, notably with Pilot 4 opting to forgo the embodied/social condition in favor of revisiting the vocal/directive condition, hence the duplicate score for this condition for him.

The presence scores revealed that Pilot 1 experi-
enced a marked lack of VA presence, attributed to
a limited perception of action possibilities within the
virtual environment, aligning with the inability to in-
teract with the cockpit. Conversely, the other pilots
generally reported a positive perception of presence,
with the last session of the day — invariably their
preferred condition — receiving the highest ratings,
corroborating the qualitative findings.

5.4 Acceptance

Again, thirty-five TAM scales were completed by the
participants. A score above 60 constitutes a rather
good acceptability, while a score below this threshold
is rather negative. These two trends are represented
in red and green in Table 3, and the bold score reflects
the last passage. The maximum score that can be
obtained is 120. The trends in the responses converge
towards an increase in VA acceptance over the course
of the day. The participants' answers to the last run
of the day is indicated on Figure 3. It is noted that
Pilot 4 rated the acceptability of the VA at the same
level during his second run with the voice/directive
condition.

5.5 Physiological measures

It was said to the participants that they could have
their physiological state monitored using physiolog-
ical sensors present in the virtual reality headset to
infer their mental load and adapt VA characteristics
in adaptive sessions. Although the data treatment
chain was already developed (based on [Luong et al.,
2020]), it only served as a follow-up display to the
WOz operator who gave arbitrary values to the pilot
on his request. However, the presence of physiolog-
ical sensors was consistent with the scenario, which
<table><caption>Table 1: Preferred combination at the end of the day</caption><thead><tr><th></th><th>No VA</th><th>Vocal only</th><th>Full embodiment</th><th>Adaptive embodiment</th></tr></thead><tbody><tr><th>No VA</th><td>Baseline</td><td>n/a</td><td>n/a</td><td>n/a</td></tr><tr><th>Task-oriented</th><td>n/a</td><td>1</td><td></td><td></td></tr><tr><th>Social-oriented</th><td>n/a</td><td></td><td>1</td><td>3</td></tr><tr><th>Adaptive communication</th><td>n/a</td><td></td><td></td><td></td></tr></tbody></table>

<table><caption>Table 2: Sense of presence score. One pilot expressed the desire to test the same scenario a second time instead of a new one, hence the two values appearing in the same cell. Vocal (V), Full Embodiment (FE), Adaptive Embodiment (AE); Task-oriented Communication (TC), Social-oriented Communication (SC), Adapted Communication (AC)</caption><thead><tr><th></th><th>V/TC</th><th>V/SC</th><th>V/AC</th><th>FE/TC</th><th>FE/SC</th><th>FE/AC</th><th>AE/TC</th><th>AE/SC</th><th>AE/AC</th></tr></thead><tbody><tr><th>Pilot 1</th><td>65</td><td>63</td><td>74</td><td></td><td></td><td>63</td><td>60</td><td>68</td><td>74</td></tr><tr><th>Pilot 2</th><td>77</td><td>81</td><td>80</td><td></td><td></td><td>79</td><td></td><td>95</td><td>90</td></tr><tr><th>Pilot 3</th><td>97</td><td>90</td><td>68</td><td>84</td><td>108</td><td>99</td><td>80</td><td></td><td>109</td></tr><tr><th>Pilot 4</th><td>123;131</td><td>99</td><td>113</td><td>107</td><td></td><td>102</td><td></td><td></td><td>112</td></tr><tr><th>Pilot 5</th><td>121</td><td>122</td><td>108</td><td></td><td></td><td>100</td><td>126</td><td>148</td><td>142</td></tr></tbody></table>

provides a report of the pilot's physiological state by the VA following a loss of consciousness (simulated by the operator). It increased the credibility of the assistant's answers and therefore the pilot's trust.

Concerning the acceptability aspect of the physiological capture in operation, the feedback from the participants is quite positive. They did not express any concern or apprehension related to the collection and analysis of their physiological data. In contrast, some of the participants considered that an analysis of mental load can be a real asset, especially to allow the VA to adapt to the mental load of the pilot.

The scenario was relatively basic, and its complexity was not sufficient to significantly increase the workload of experimented fighter pilots. Therefore, the usefulness of displaying the workload estimate on the WOz interface could not be evaluated.

Finally, from an operational point of view, it can also be noted that no participant complained about a possible discomfort induced by the presence of the sensors, suggesting that their integration in the helmet is not particularly intrusive.

## 5.6 Other results from debriefing

The debriefing interviews underscored the critical factors for successful cooperation with the Virtual Assistant (VA), including the impact of the VA's speech tempo on pilots' workload and the necessity for its communication style to align with pilots' mental states, as noted by Nass et al. [2005]. The pilots highlighted the potential of the VA to manage workload effectively, with the aim of maintaining it around 3/5 on the ISA scale, and the importance of improving situational awareness through timely information provision, consistent with the findings of Tran et al. [Tran et al., 2018] regarding VA assistance in cockpit emergency management. The proactive behavior of the VA in preparing for upcoming tasks was also appreciated.

The study highlighted mutual learning between pilots and the Virtual Assistant (VA) through Wizard of Oz (WOz) simulations, showing a preference for customized communication matching risk levels and task urgency to enhance cooperation. Trust in the VA, crucial for its acceptability, was seen to depend on personalization and the VA's reliability, as sup-
<table><caption>Table 3: Acceptance score. One pilot expressed the desire to test the same scenario a second time instead of a new one, hence the two values appearing in the same cell. Vocal (V), Full Embodiment (FE), Adaptive Embodiment (AE); Task-oriented Communication (TC), Social-oriented Communication (SC), Adapted Communication (AC)</caption><thead><tr><th></th><th>V/TC</th><th>V/SC</th><th>V/AC</th><th>FE/TC</th><th>FE/SC</th><th>FE/AC</th><th>AE/TC</th><th>AE/SC</th><th>AE/AC</th></tr></thead><tbody><tr><td>Pilot 1</td><td>68</td><td>75</td><td>80</td><td></td><td></td><td>50</td><td>67</td><td>79</td><td>70</td></tr><tr><td>Pilot 2</td><td>77</td><td>52</td><td>67</td><td></td><td></td><td>47</td><td>77</td><td>77</td><td>53</td></tr><tr><td>Pilot 3</td><td>80</td><td>69</td><td>52</td><td>52</td><td>92</td><td>89</td><td></td><td></td><td>42</td></tr><tr><td>Pilot 4</td><td>95;95</td><td>26</td><td>74</td><td>74</td><td></td><td>67</td><td></td><td></td><td>42</td></tr><tr><td>Pilot 5</td><td>100</td><td>100</td><td>99</td><td></td><td></td><td>61</td><td>101</td><td>108</td><td>99</td></tr></tbody></table>

ported by Braun et al. [2019]. Debriefing sessions identified key areas for the VA's contribution, such as efficient communication in combat and roles in interface management and decision support. The potential for VA integration in future combat operations and administrative tasks was also noted, emphasizing the need for a clear definition of the VA's cockpit role.

The efficacy of the WOz method in developing VA-pilot cooperation was affirmed, suggesting its use in future combat aircraft design, despite the study's small scale and lack of complex scenarios. The findings offer qualitative insights into improving VA usability and pilot interaction, pointing toward greater VA adaptability and interaction realism.

## 5.7 Limitations

Although the number of participants was limited, preventing statistically significant results, qualitative insights were in line with Nielsen and Landauer [1993] explaining the cost / benefit ratio that five users are enough to identify 80% of usability problems. This was supported by pilots' feedback aligning with literature on immediate speech cut-off [Budiu and Laubheimer, 2018], concise communication [Yaghoubzadeh et al., 2015], and the potential downsides of agent embodiment [Kontogiorgos et al., 2019, Gilles and Bevacqua, 2022], confirming the sample's representativeness for broader VA user research.

The study faced limitations due to simplified scenarios, the absence of a counterbalanced protocol, and a short virtual exposure designed to limit the risk of cybersickness. These limitations are further compounded by the lack of haptic feedback and realistic cockpit interaction in the virtual environment.

Additionally, the VA's adaptability was restricted, with only minor adjustments to voice and no capability for speech interruption or advanced gestures, highlighting areas for future research.

# 6 Conclusion and perspectives

This pilot study, based on the WOz technique, allowed us to address the test objectives related to estimating the level of embodiment and the communication style required for effective pilot-VA cooperation, adapted to the operational context. While the technique itself was unanimously accepted, the test also provided insights into future cockpit designs involving virtual assistants, particularly in terms of embodiment and communication style. The debriefings with the pilots showed that the VA has its place in future combat aircraft and could have been decisive in many past cases. It will then be necessary to determine the role of the VA within an operational setting that more accurately reflects the complexity of future missions.

Regarding the detection of mental workload through the analysis of physiological responses, participants did not express any discomfort about having their physiological state monitored. On the contrary, some even viewed mental workload analysis as a valuable
<table><caption>Figure 3: Cumulative TAM answers of participants' responses to the last case of the test day</caption><thead><tr><th rowspan="2">TAM questionnaire<br>Response items use a 7 point likely likely - unlikely scale from extremely likely to extremely unlikely</th><th colspan="4">Likely</th><th colspan="4">Unlikely</th></tr><tr><th>Extremely</th><th>Quite</th><th>Slightly</th><th>Neither</th><th>Slightly</th><th>Quite</th><th>Extremely</th></tr></thead><tbody><tr><td>I feel that using Virtual Assistant like this one would be easy for me</td><td>X</td><td>XX</td><td>XX</td><td></td><td></td><td></td><td></td></tr><tr><td>I feel that my interaction with Virtual Assistant like this one would be clear and understandable</td><td>X</td><td>XX</td><td>XX</td><td></td><td></td><td></td><td></td></tr><tr><td>I feel that it would be easy to become skillful at using Virtual Assistant like this one</td><td>X</td><td>XX</td><td>XX</td><td></td><td></td><td></td><td></td></tr><tr><td>I would find Virtual Assistant like this one to be flexible to interact with</td><td></td><td>XX</td><td>XXX</td><td></td><td></td><td></td><td></td></tr><tr><td>Learning to operate Virtual Assistant like this one would be easy for me</td><td></td><td>XXX</td><td>XX</td><td></td><td></td><td></td><td></td></tr><tr><td>It would be easy for me to get Virtual Assistant like this one to do what I want to do</td><td>X</td><td>XX</td><td>XX</td><td></td><td></td><td></td><td></td></tr><tr><td>I feel that my ability to determine Virtual Assistant like this one ease of use is limited by my lack of experience</td><td></td><td></td><td></td><td>XX</td><td>X</td><td></td><td>X</td></tr><tr><td>Using Virtual Assistant like this one in my job would enable me to accomplish tasks more quickly</td><td></td><td>XX</td><td>XXX</td><td></td><td></td><td></td><td></td></tr><tr><td>Using Virtual Assistant like this one would improve my job performance</td><td>X</td><td>XXX</td><td>X</td><td></td><td></td><td></td><td></td></tr><tr><td>Using Virtual Assistant like this one in my job would increase my productivity</td><td>X</td><td>XXX</td><td>X</td><td></td><td></td><td></td><td></td></tr><tr><td>Using Virtual Assistant like this one would enhance my effectiveness on the job.</td><td>X</td><td>XX</td><td>XX</td><td></td><td></td><td></td><td></td></tr><tr><td>Using Virtual Assistant like this one would make it easier to do my job.</td><td></td><td>XXXX</td><td>X</td><td></td><td></td><td></td><td></td></tr><tr><td>I would find Virtual Assistant like this one useful in my job</td><td>X</td><td>XXX</td><td>X</td><td></td><td></td><td></td><td></td></tr><tr><td>I believe it is a good idea to use Virtual Assistant like this one</td><td></td><td>XXX</td><td>XX</td><td></td><td></td><td></td><td></td></tr><tr><td>I like the idea of using Virtual Assistant like this one</td><td>X</td><td>X</td><td>XX</td><td>X</td><td></td><td></td><td></td></tr><tr><td>Using Virtual Assistant like this one is a positive idea</td><td>X</td><td>X</td><td>XXX</td><td></td><td></td><td></td><td></td></tr><tr><td>I plan to use Virtual Assistant like this one in the future</td><td></td><td>XX</td><td>X</td><td>XX</td><td></td><td></td><td></td></tr><tr><td>Assuming that I have access to Virtual Assistant like this one, I intend to use it</td><td></td><td>XXX</td><td>XX</td><td></td><td></td><td></td><td></td></tr><tr><td>In my job, the usage of Virtual Assistant like this one is important</td><td></td><td>XX</td><td>XX</td><td>X</td><td></td><td></td><td></td></tr><tr><td>In my job, the usage of Virtual Assistant like this one is relevant</td><td></td><td>XXX</td><td></td><td>XX</td><td></td><td></td><td></td></tr></tbody></table>

able asset—particularly for enabling the VA to adapt to the pilot's workload. This capability could be integrated into future tests using the algorithm proposed by Luong et al. [2020].

Several questions remain to be studied. Although the pilots did not comment on the quality of the scene or the appearance of the agent, it would be interesting to study whether the quality of the rendering had an impact on their perception of the assistant. In this work, we did not focus on this aspect as well, and the agent and its non-verbal behavior remain relatively straightforward to avoid falling into the Uncanny valley [Perez Garcia and Saffon Lopez, 2019].

Another fascinating subject to study could be a symbolic representation of the assistant, its appearance and disappearance at the pilot's request or according to events that could endanger the pilot or the aircraft, or even the position of the assistant in the cockpit of the future. In fact, during the evaluation, the pilots pointed out that the representation of

the VA and its position should be flexible and better defined according to a future cockpit.

But other dimensions have to be integrated in future work in order to evolve the overall maturity of the VA concept. In particular, the complexity and credibility of the operational scenario must evolve to get closer to future missions. A first approach could be to integrate other actors such as the command center, agents on the ground, teammates evolving in coordinated patrols, drones, etc. The tasks to be performed should also increase in complexity and variability. Thus, the role of the VA in the management of system failures or in decision-making at critical moments are situations that should be explored.

## Acknowledgements

This work has been conducted within the project AVIATOR. This project is supported by the Man-
Machine Teaming (MMT) initiative, funded by the
Directorate General of Armaments (DGA) and run
by Dassault Aviation and Thales.

References

C. Breazeal, C. D. Kidd, A. L. Thomaz, G. Hoffman, and M. Berlin. Effects of nonverbal communication on efficiency and robustness in human-robot teamwork. In *IEEE/RSJ International Conference on Intelligent Robots and Systems*, pages 708–713, 2005. doi: 10.1109/IROS.2005.1545011.

Dimosthenis Kontogiorgos, Andre Pereira, Olle Andersson, Marco Koivisto, Elena Gonzalez Rabal, Ville Vartiainen, and Joakim Gustafson. The Effects of Anthropomorphism and Non-verbal Social Behaviour in Virtual Assistants. In *19th ACM International Conference on Intelligent Virtual Agents*, pages 133–140, Paris, France, 2019. ACM Press. ISBN 978-1-4503-6672-4. doi: 10.1145/3308532.3329466.

Kangsoo Kim, Luke Boelling, Steffen Haesler, Jeremy Bailenson, Gerd Bruder, and Greg F Welch. Does a digital assistant need a body? the influence of visual embodiment and social behavior on the perception of intelligent virtual agents in ar. In *2018 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)*, pages 105–114. IEEE, 2018. doi: 10.1109/ISMAR.2018.00039.

K.-P. L. Vu, J. Lachter, V. Battiste, and T. Z. Strybel. Single pilot operations in domestic commercial aviation. *The Journal of the Human Factors and Ergonomics Society*, 2018.

Marlène Gilles and Elisabetta Bevacqua. A Review of Virtual Assistants’ Characteristics: Recommendations for Designing an Optimal Human–Machine Cooperation. *Journal of Computing and Information Science in Engineering*, 22(5), 03 2022. ISSN 1530-9827. doi: 10.1115/1.4053369.

Denys Bernard. Cognitive interaction: Towards” cog-
nitivity” requirements for the design of virtual as-
sistants. In *2017 IEEE International Conference*

*on Systems, Man, and Cybernetics (SMC)*, pages 210–215. IEEE, 2017. doi: 10.1109/SMC.2017.8122604.

David R. Large, Gary Burnett, and Leigh Clark. Lessons from Oz: design guidelines for automotive conversational user interfaces. In *Proceedings of the 11th International Conference on Automotive User Interfaces and Interactive Vehicular Applications Adjunct Proceedings - AutomotiveUI '19*, pages 335–340, Utrecht, Netherlands, 2019. ACM Press. ISBN 978-1-4503-6920-6. doi: 10.1145/3349263.3351314.

Gabriel Skantze. Exploring human error recovery strategies: Implications for spoken dialogue systems. *Speech Communication*, 45(3):325-341, March 2005. ISSN 01676393. doi: 10.1016/j.specom.2004.11.005.

Veena Chattaraman, Wi-Suk Kwon, Juan E Gilbert, and Kassandra Ross. Should ai-based, conversational digital assistants employ social-or task-oriented interaction style? a task-competency and reciprocity perspective for older adults. *Computers in Human Behavior*, 90:315-330, 2019. doi: 10.1016/j.chb.2018.08.048.

Minjin Rheu, Ji Youn Shin, Wei Peng, and Jina Huh-Yoo. Systematic review: Trust-building factors and implications for conversational agent design. *Int. J. Hum. Comput. Interact.*, 37(1):81–96, 2021. doi: 10.1080/10447318.2020.1807710.

Marta Perez Garcia and Sarita Saffon Lopez. Exploring the Uncanny Valley Theory in the Constructs of a Virtual Assistant Personality. In Yaxin Bi, Rahul Bhatia, and Supriya Kapoor, editors, *Intelligent Systems and Applications*, volume 1037, pages 1017–1033. Springer International Publishing, Cham, 2019. ISBN 978-3-030-29515-8 978-3-030-29516-5. doi: 10.1007/978-3-030-29516-5_76.

Elizabeth Phillips, Daniel Ullman, Maartje M. A. de Graaf, and Bertram F. Malle. What Does A Robot Look Like?: A Multi-Site Examination of User Expectations About Robot Appearance. *Proceedings of the Human Factors and
*Ergonomics Society Annual Meeting*, 61(1):1215-1219, September 2017. ISSN 1541-9312. doi: 10.1177/1541931213601786.

Jonathan Ehret, Andrea Bönsch, Patrick Nossol, Cosima A. Ermert, Chinthusa Mohanathasan, Sabine J. Schlittmeier, Janina Fels, and Torsten W. Kuhlen. Who's next? integrating non-verbal turning cues for embodied conversational agents. New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9781450399944. doi: 10.1145/3570945.3607312.

Dirk Heylen, Elisabetta Bevacqua, Marion Tellier, and Catherine Pelachaud. Searching for prototypical facial feedback signals. pages 147-153, 09 2007. ISBN 978-3-540-74996-7. doi: 10.1007/978-3-540-74997-4_14.

Patrick Gebhard, Tobias Baur, and Ionut Damian. Exploring Interaction Strategies for Virtual Characters to Induce Stress in Simulated Job Interviews. In *13th International Conference on Autonomous Agents and Multiagent Systems AAMAS'14*, pages 661-668, 2014.

Sahba Zojaji, Adam Červeň, and Christopher Peters. Impact of multimodal communication on persuasiveness and perceived politeness of virtual agents in small groups. In *Proceedings of the 23rd ACM International Conference on Intelligent Virtual Agents*, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9781450399944. doi: 10.1145/3570945.3607356.

Aaron Holroyd, Charles Rich, Candace L. Sidner, and Brett Ponsler. Generating connection events for human-robot collaboration. In *IEEE International Workshop on Robot and Human Communication RO-MAN*, pages 241-246, Atlanta, GA, USA, July 2011. IEEE. ISBN 978-1-4577-1571-6. doi: 10.1109/ROMAN.2011.6005245.

Alan Martin, Gregory M. P. O'Hare, Brian R. Duffy, Bianca Schön, and John F. Bradley. Maintaining the Identity of Dynamically Embodied Agents. In *5th International Conference on Intelligent Virtual Agents*, volume 3661 of LNAI, pages 454-465, Kos, Greece, 2005. Springer Berlin, Heidelberg. ISBN 978-3-540-28739-1. doi: 10.1007/11550617_38.

R. Aylett, M. Kriegel, I. Wallace, E. Márquez Segura, J. Mecurio, S. Nylander, and P. Vargas. Do I remember you? Memory and identity in multiple embodiments. In *2013 IEEE RO-MAN*, pages 143-148, 2013. doi: 10.1109/ROMAN.2013.6628435.

Michael Braun, Anja Mainz, Ronee Chadowitz, Bayesian Pfleging, and Florian Alt. At Your Service: Designing Voice Assistant Personalities to Improve Automotive User Interfaces. In *Conference on Human Factors in Computing Systems - CHI'19*, pages 1-11, Glasgow, Scotland Uk, 2019. ACM Press. ISBN 978-1-4503-5970-2. doi: 10.1145/3290605.3300270.

Clifford Nass, Ing-Marie Jonsson, Helen Harris, Ben Reaves, Jack Endo, Scott Brave, and Leila Takayama. Improving automotive safety by pairing driver emotion and car voice emotion. In *Extended Abstracts on Human factors in computing systems - CHI'05*, page 1973, Portland, OR, USA, 2005. ACM Press. ISBN 978-1-59593-002-6. doi: 10.1145/1056808.1057070.

Yixiang Lim, Alessandro Gardi, Roberto Sabatini, Subramanian Ramasamy, Trevor Kistan, Neta Ezer, Julian Vince, and Robert Bolia. Avionics Human-Machine Interfaces and Interactions for Manned and Unmanned Aircraft. *Progress in Aerospace Sciences*, 102:1-46, October 2018. ISSN 03760421. doi: 10.1016/j.paerosci.2018.05.002.

R.G. Eggleston. *Design of computing systems: cognitive considerations: proceedings of the Seventh International Conference on Human-Computer Interaction, (HCI International '97), San Francisco, California, USA, August 24 - 29, 1997.* Advances in human factors/ergonomics. Elsevier, Amsterdam, 1997. ISBN 978-0-444-82183-6.

Robert M. Taylor, Michael C. Bonner, Blair Dickson, Howard Howells, Christopher A. Miller, Nocholas Milton, Kit Pleydell-pearce, Nigel Shadbolt, Jeni Tennison, and Sharon Whitecross. Cognitive Cockpit Engineering: Coupling Functional State Assess-
ment, Task Knowledge Management, and Decision-Support for Context-Sensitive Aiding. *Human Systems IAC Gateway*, 12:254–314, 2001.

Thomas A. Furness. The Super Cockpit and its Human Factors Challenges. *Proceedings of the Human Factors Society Annual Meeting*, 30(1): 48–52, September 1986. ISSN 0163-5182. doi: 10.1177/154193128603000112.

Barbara Piuzzi, Alessandro Cont, and Magda Balerna. The workload sensing for the human machine interface of Unmanned Air Systems. In *2014 IEEE Metrology for Aerospace (MetroAeroSpace)*, pages 50–55, Benevento, Italy, May 2014. IEEE. ISBN 978-1-4799-2069-3. doi: 10.1109/MetroAeroSpace.2014.6865893.

S.B. Banks and C.S. Lizza. Pilot's Associate: a cooperative, knowledge-based system application. *IEEE Expert*, 6(3):18–29, June 1991. ISSN 0885-9000. doi: 10.1109/64.87681.

Reiner Onken and Anton Walsdorf. Assistant systems for aircraft guidance: cognitive man-machine cooperation. *Aerospace Science and Technology*, 5 (8):511–520, December 2001. ISSN 12709638. doi: 10.1016/S1270-9638(01)01137-3.

S. Estes, K. Burns, J. Helleberg, K. Long, J. Stein, and M. Pollack. Digital copilot: Cognitive assistance for pilots. In *The 2016 AAAI Fall Symposium Series: Cognitive Assistance in Government and Public Sector Applications*, pages 141–144, Palo Alto, CA, USA, 2016. AAAI Press.

Kay Stanney, Ben D Lawson, Bas Rokers, Mark Dennison, Cali Fidopiastis, Thomas Stoffregen, Séamas Weech, and Jacqueline M Fulvio. Identifying causes of and solutions for cybersickness in immersive technology: reformulation of a research and development agenda. *International Journal of Human-Computer Interaction*, 36(19):1783–1803, 2020.

Nawel Khenak, Cédric Bach, Stéphane Drouot, and Florence Buratto. Evaluation of virtual reality training: effectiveness on pilots' learning.

In *IHM '23-34e Conférence Internationale Francophone sur l'Interaction Humain-Machine*, 2023.

M. Kipp. *Gesture Generation by Imitation: From Human Behavior to Computer Character Animation*. Universal Publishers, 2005. ISBN 9781581122558. URL https://books.google.fr/books?id=0IyArJx7mEQC.

R. Querrec, P. Vallejo, and C. Buche. Mascaret: create virtual learning environments from system modelling. In *Engineering Reality of Virtual Reality (SPIE Electronic Imaging Conference)*, San Francisco, USA, 2013.

Hannes Vilhjálmsson, Nathan Cantelmo, Justine Cassell, Nicolas E. Chafai, Michael Kipp, Stefan Kopp, Maurizio Mancini, Stacy Marsella, Andrew N. Marshall, Catherine Pelachaud, Zsofi Ruttkay, Kristinn R. Thórisson, Herwin van Welbergen, and Rick J. van der Werf. The behavior markup language: Recent developments and challenges. In Catherine Pelachaud, Jean-Claude Martin, Elisabeth André, Gérard Chollet, Kostas Karpouzis, and Danielle Pelé, editors, *Intelligent Virtual Agents*, pages 99–111, Berlin, Heidelberg, 2007. Springer Berlin Heidelberg. ISBN 978-3-540-74997-4.

Matthew Aylett and Christopher Pidcock. The cerevoice characterful speech synthesiseratif. In *7th International Conference on Intelligent Virtual Agents*, pages 413–414, 09 2007. ISBN 978-3-540-74996-7.

C. Jordan. Experimental study of the effect of an instantaneous self-assessment workload recorder. Technical Report, Portsmouth: DRA Maritime Command and Control Division, 1992.

B.J. Witmer, C.J. Jerome, and M.J. Singer. The factor structure of the presence questionnaire. *Presence*, 14(3):298–312, 2005.

Saleh Alharbi and Steve Drew. Using the technology acceptance model in understanding academics' behavioural intention to use learning management
systems. *International Journal of Advanced Computer Science and Applications*, 5(1), 2014. doi: 10.14569/IJACSA.2014.050120.

T. Luong, N. Martin, A. Raison, F. Argelaguet, J. M. Diverrez, and A. Lécuyer. Towards real-time recognition of users mental workload using integrated physiological sensors into a vr hmd. In *2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)*, pages 425–437, 2020. doi: 10.1109/ISMAR50242.2020.00068.

Ramin Yaghoubzadeh, Karola Pitsch, and Stefan Kopp. Adaptive Grounding and Dialogue Management for Autonomous Conversational Assistants for Elderly Users. In Willem-Paul Brinkman, Joost Broekens, and Dirk Heylen, editors, *15th International Conference on Intelligent Virtual Agents*, volume 9238 of LNAI, pages 28–38, Delft, The Netherlands, 2015. Springer International Publishing. ISBN 978-3-319-21995-0 978-3-319-21996-7. doi: 10.1007/978-3-319-21996-7_3.

R. Budiu and P. Laubheimer. Intelligent Assistants Have Poor Usability: A User Study of Alexa, Google Assistant, and Siri. Nielsen Norman Group, 2018. URL https://www.nngroup.com/articles/intelligent-assistant-usability/. Last accessed 21 March 2024.

The Huy Tran, Ferdinand Behrend, Niels Funning, and Andres Arango. Single Pilot Operations with AR-Glasses using Microsoft HoloLens. In *IEEE/AIAA 37th Digital Avionics Systems Conference (DASC)*, pages 1–7, London, September 2018. ISBN 978-1-5386-4112-5. doi: 10.1109/DASC.2018.8569261.

Jakob Nielsen and Thomas K. Landauer. A mathematical model of the finding of usability problems. In *Proceedings of the INTERACT '93 and CHI '93 Conference on Human Factors in Computing Systems*, CHI '93, pages 206–213, New York, NY, USA, 1993. Association for Computing Machinery. ISBN 0897915755. doi: 10.1145/169059.169166.