<figure><img src="image_1.png" /></figure>

HAL
open science

# A Survey on Affective and Cognitive VR

Tiffany Luong, Anatole Lécuyer, Nicolas Martin, Ferran Argelaguet Sanz

► To cite this version:

Tiffany Luong, Anatole Lécuyer, Nicolas Martin, Ferran Argelaguet Sanz. A Survey on Affective and Cognitive VR. IEEE Transactions on Visualization and Computer Graphics, 2022, 28 (12), pp.5154-5171. 10.1109/TVCG.2021.3110459. hal-03469058

HAL Id: hal-03469058

https://inria.hal.science/hal-03469058

Submitted on 7 Dec 2021

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
# A Survey on Affective and Cognitive VR

Tiffany Luong, Anatole Lecuyer, Nicolas Martin, and Ferran Argelaguet

**Abstract**—In Virtual Reality (VR), users can be immersed in emotionally intense and cognitively engaging experiences. Yet, despite strong interest from scholars and a large amount of work associating VR and Affective and Cognitive States (ACS), there is a clear lack of structured and systematic form in which this research can be classified. We define “Affective and Cognitive VR” to relate to works which (1) induce ACS, (2) recognize ACS, or (3) exploit ACS by adapting virtual environments based on ACS measures. This survey clarifies the different models of ACS, presents the methods for measuring them with their respective advantages and drawbacks in VR, and showcases Affective and Cognitive VR studies done in an immersive virtual environment (IVE) in a non-clinical context. Our article covers the main research lines in Affective and Cognitive VR. We provide a comprehensive list of references with the analysis of 63 research articles and summarize future directions.

**Index Terms**—Artificial, augmented, and virtual realities – Affective Computing - Physiological Measures – Social and Behavioral Sciences

<figure><img src="image_2.png" /></figure>

## 1 INTRODUCTION

VIRTUAL REALITY (VR) technologies enable users to feel “present” in a synthetic 3D scene simulated artificially [138]. This gives VR the potential to engage users in a virtual world and to be a strong emotional driver. At the same time, VR allows to simulate complex, real, and unrealistic situations while immersing users in a highly controlled and safe virtual world. This makes it an ideal media to induce, study and understand Affective and Cognitive States (ACS). Affective States (AS) refer to states such as emotions, mood, and feelings [132], and Cognitive States (CS) refer to states like cognitive load or mental workload, which influence how information is processed (e.g., reasoning, deliberation, planning) [22]. While there are some distinctions in their definitions and models, AS and CS are interwoven [78]. They have various similarities as multicomponent constructs [110, 132] and in the methods to measure them [103].

The interest in associating ACS and computers was raised early. In 1995, Picard et al. defined “Affective Computing” as a type of computing which “relates to, arises from, or influences emotions” [118]. She highlighted the fact that emotions can act as powerful motivators, influence perception, cognition, coping, and have an important role in creativity [118]. In that sense, giving computers the ability to recognize, express and have emotions could contribute to a richer quality of interactions, which is essential in VR. Similarly, taking into account users’ cognitive abilities and limited cognitive resources [153] could make VR experiences more fitted to users. For these reasons, many researchers have shown their interest in associating VR and ACS by inducing, recognizing, and exploiting ACS in VR.

The adaptation of Immersive Virtual Environments (IVEs) by taking into account users’ ACS can be done following these steps (see Fig. 1): VR stimuli are presented to the user, which triggers AS and CS responses that can be assessed using different metrics (e.g., self-reports, physiological measures). Then, objective

metrics can be used to recognize the user’s perceived AS or CS in real-time using Machine Learning (ML) algorithms or rule-based model. Finally, this recognized or measured AS or CS can be exploited to modulate the IVE content. Such a process is not new as similar frameworks applied to emotions and games specifically [23, 82], outside VR [26, 71, 118], already exist. However, few studies went through the whole adaptation course in IVEs. Furthermore, no review currently summarizes the findings considering the different steps of the adaptation loop. Works that study ACS in VR can usually be grouped in at least one of these categories:

* The **induction of ACS** corresponds to the study, design, or development of content, parameters, or methods which can influence or induce changes in the users’ ACS.

* The **recognition of ACS** corresponds to the study, design, or development of recognition models or methods to identify users’ ACS.

* The **exploitation of ACS**, corresponds to the study, design, or development of feedback or adaptation methods, logic, and parameters based on users’ ACS measures.

Based on this categorization, we propose to define “**Affective and Cognitive VR**” as “*the study of virtual reality systems and applications that induce, recognize, or exploit affective and cognitive states*”. In this definition, AS and CS are grouped because of the similarities they share in the way they are processed in VR studies. However, this can be discussed as they can be differentiated in their concepts. In that case, works can speak of “*Affective VR*” when they relate to the study of AS, and “*Cognitive VR*” when they relate to the study of CS.

In this paper, we provide an overview of Affective and Cognitive VR studies performed in IVEs in a non-clinical context. The objective is to present the current findings in each induction, recognition, and exploitation of ACS in IVEs category to go towards adaptive VR applications that consider the user’s ACS. For reviews, SWOT and meta-analysis in VR in the clinical context (e.g., therapy, pathological phobia), interested readers can refer to [27, 116, 120, 123, 125, 127]. Specific aspects such as interaction design for positive usages or the training field can be respectively be found in [70] and [156].

* T. Luong, A. Lecuyer, and F. Argelaguet work with Univ. Rennes, Inria, IRISA, CNRS, Rennes, France.

* T. Luong and N. Martin work with IRT b<>com, Cesson-Sévigné, France.

* Corresponding E-mails: {tiffany.luong, ferran.argelaguet, anatole.lecuyer}@inria.fr
<figure><img src="image_3.png" /><figcaption>FIG. 1: Loop depicting the adaptation of VR based on users' ACS, as well as the categorization of Affective and Cognitive VR studies in the survey. Stimuli in VR first *induce* AS or CS responses to the users. Then, these responses are measured from the users, and used to *recognize* the user's AS or CS. Finally, the recognized AS or CS is *exploited* to adjust the VR stimuli and parameters in real-time.</figcaption></figure>

## 2 METHODOLOGY

We performed the following queries on Google Scholar and IEEE Xplore websites to collect the papers for this survey. Search criteria included all publications from 1999 to March 2020. The following keywords were used: ("virtual reality" OR "immersive virtual environments") AND ("emotions" OR "affective state" OR "personality trait" OR "mood" OR "anxiety" OR "cognitive state" OR "mental workload" OR "cognitive load" OR "cognitive performance" OR "psychological state" OR "physiological signals"). Systematic reviews, meta-analysis, frameworks, and methodological approaches were retrieved and classified into the proposed induction, recognition, and exploitation categories. However, these works were excluded from the principal analysis. Inclusion criteria were: "written in English", "mentioning the use a VR Head-Mounted Displays (HMD) or a CAVE-like system.", respectively considered as fully-immersive and semi-immersive systems [65], "measuring an AS or a CS or aiming to induce, influence, recognize, or exploit an AS or a CS.". The exclusion criterion was "targeting clinical patients or clinical use cases.". Upon completion of the search, titles and abstracts of the identified articles were assessed for suitability for the review. Then, the full texts of the suitable articles were retrieved for further examination of their contents. If they met the inclusion criteria, they were independently reviewed, screened by the researchers, and classified into one or multiple of the categories "induction, recognition, and exploitation of ACS". The reference lists of the selected articles, as well as previous systematic reviews and meta-analyses, were also examined for additional publications that might have been overlooked in the search. A resulting panel of 63 studies was retained in the scope of this survey, among which 43 studies deal with AS, 14 with CS, and 6 with both ACS.

## 3 AFFECTIVE AND COGNITIVE STATES: DEFINITIONS AND MODELS

In this survey, the focus is set on emotions, anxiety, and cognitive load/mental workload as they are the main ACS studied in the

VR community. There is no real consensus over the definitions of psychological states, and AS and CS can overlap in their definitions and in the methods to measure them [103]. Still, a way to differentiate AS and CS can be proposed based on the ways they are elicited and in the processes and responses they are going to provoke. On the one hand, AS mostly relate to primal instincts and derive from personality traits and personal experiences [17]. They can be manipulated, for instance, by stories, narrative content, and preferences, and they influence decision making. On the other hand, CS derive from cognitive resources, cognitive skills, and influence how information is processed [22]. They can be manipulated, for instance, by the intrinsic difficulty of a task, the number of distractors, and the instructions presentation format. These will have consequences on the errors one will make and metrics such as the reaction time. The definition of an AS or a CS is mainly chosen depending on the field, the context, and the aim of the study [72]. While AS and CS are treated in similar ways in IVEs (see Fig. 1), the selection of an appropriate psychological model is crucial for the choice of the measurement method, and all the other steps in a research dealing with ACS [103].

In this section, definitions and models of emotions and anxiety will first be addressed, followed by those of cognitive load and mental workload. The occurrence of the ACS measured in the 63 studies retained in this review are depicted in Fig. 4. Finally, findings on the link between ACS and presence will be presented.

### 3.1 Affective States

Affective states can be distinguished in terms of time (duration), intention (event focus), cause, and impacts on the behaviour and physiological responses among other criteria [18, 45, 132]. In particular, a distinction can be made between states and traits. States can be defined as a concurrent experience or transient mood while traits are more permanent. As such, eliciting a state too often can result in a trait change [141]. In IVEs, the most studied affects are emotions and anxiety.

A well-accepted definition of **emotion** is given by Scherer [132]: "*an episode of interrelated, synchronized changes in the states of all or most of the five organismic subsystems in response to the evaluation of an external or internal stimulus event as relevant to major concerns of the organism*" [131]. In his definition, the five organismic subsystems correspond to a categorization of the physiological responses. As for the emotional models, Kołakowska et al. [73] proposed to classify them into 3 categories following different perspectives:

* Emotions, from a **dimensional perspective**, are considered as continuous phenomena that can be represented in N-dimensional space, usually within 2 or 3 fundamental dimensions. Among the multiple models, Mehrabian and Russell [99] proposed a 3D environmental emotional scale of "*nearly independent bipolar dimensions*", abbreviated as PAD for Pleasure, also referred to as valence (from negative to positive), Arousal, which corresponds to the excitation, mental activation or level of alertness, and Dominance, which is the degree of feeling of having control over a specific situation. Another popular model is Russell's circomplex model of affect [128], which defines emotions based on 2 dimensions: the emotional Valence on the x-axis and the emotional Arousal, on the y-axis.

* From a **discrete perspective**, each emotion is studied independently, as if all emotions were distinct. The most popu-
lar discrete category of emotions is the Facial Action Coding System (FACS [51]) one for expressions recognition: the Big Six (Happiness, Sadness, Fear, Anger, Disgust and Surprise). These fundamental emotions are considered innate, basic, and determined in both psychological and biological sense from this perspective. To this basic set, some added the Neutral expression [63] or extended it with other emotions such as Guilt, Pride and Shame, which have a more complex behavioural manifestation. Other sets of basic emotions are proposed and reviewed in [68, 107].

* Emotions can also be defined from a **hybrid perspective**, combining discrete and dimensional viewpoints. A good example is Plutchik hierarchical model [119] where complex emotions are combinations of pairs of more basic emotions.

Studying emotions from a discrete perspective is at first very simple and intuitive. However, it describes emotions without any inter-relationship, which makes the recognition of non-basic affective states more difficult. On the other hand, the dimensional perspective offers the possibility to study emotions as a function of continuous and nearly independent variables. However, this perspective tends to not be as expressive as the discrete perspective. The hybrid perspective was not used in VR studies to our knowledge yet, hypothetically because of its large panel of emotions.

As for other AS, **anxiety** has been of particular interest in the VR community, even more in VR exposure therapy. Spielberger defined anxiety as “*a complex emotional reaction or state that varies in intensity and fluctuates over time as a function of the intrapsychic or situational stresses that impinge upon an individual*” [141]. He made a distinction between State Anxiety and Trait Anxiety [141], and also between State Anger and Trait Anger [142]. Upon his model, anxiety and anger can manifest themselves as personality traits, but also as emotions for shorter times. As such, State-Anxiety corresponds to an emotional state that “*consists of feelings of tension and apprehension and heightened autonomic nervous system activity*”, and Trait-Anxiety corresponds to a personality trait that refers to “*individual differences in anxiety-proneness*” [141].

## 3.2 Cognitive States

The cognitive states which were the most studied in IVEs are cognitive load and mental workload. Those often designate the same concept in VR studies [121, 150], while not necessarily referring to the same one in the psychological field [57]. They can sometimes be considered as even or distinct, and can especially be confounded with their components or other concepts such as mental load, mental effort, performance, or cognitive workload [110].

Among popular definitions, Paas et al. [110] defined **cognitive load** as “*a multidimensional construct representing the load that performing a particular task imposes on the learner's cognitive system*” [111]. It has multiple aspects such as mental load, mental effort, and performance, and can be manipulated through the task and instructions presentation (see the Cognitive Load Theory [110]). On the other hand, **mental workload** is a subcomponent of workload, which can be defined as the sum of the demands a task imposes on an individual. Among the numerous proposed definitions, papers usually agree to determine mental workload as a multidimensional construct determined by characteristics of the task(s) (e.g., demands) and the operator (e.g.,

skills) [42, 61, 122]. For example, Wickens described MW as “*the relation between the (quantitative) demand for resources imposed by a task and the ability to supply those resources by the operator*” [153]. One theory is often associated with mental workload: Wickens [153]'s **multiple resources theory**. This theory proposes that human resources do not have only one information processing, but multiple resources which can be exploited simultaneously or sequentially depending on their type. It allows system designers to predict when certain tasks can be performed concurrently or will interfere with each other.

Cognitive load and mental workload are very close concepts that are tackled through different perspectives. Cognitive load particularly focuses on the interaction between the instructional design and the user's cognitive process architecture, and mental workload focuses more on the interaction between the nature of the task and the user's elicited pool of cognitive resources [153]. They are used in different contexts (i.e., respectively, mostly in the educational field and in ergonomics) and mainly relate to the user's ability to process task and information.

## 3.3 Presence and Affective and Cognitive States

VR technologies and IVEs were found to be efficient to induce and assess ACS, notably because they engage participants [124]. Users in VR are not just passive observers watching images on a screen, but active participants in a VE. From that aspect arises the sense of presence, an essential component of VR defined by Slater et al. as the sense of “*being there*”, inside a virtual world [138]. It has also two other aspects: “*the extent to which the VE becomes the dominant one*”, and “*the extent to which participants, after the VE experience, remember it as having visited a 'place' rather than just having seen images generated by a computer*” [137]. Indeed, presence in VR is important to evaluate the quality of immersion experienced by users, and it can be treated as a neuropsychological phenomenon [123]. It should, therefore, be taken into account as it can interfere and play a great role when studying other ACS.

The link between presence and ACS has been widely studied in previous research. Riva et al. found a circular interaction between presence and emotions [124]. On one side, the feeling of presence is greater in “emotional” environments, and on the other side, the emotional state is influenced by the level of presence.

Indeed, most studies agree to say presence is influenced by the intensity of emotions [5, 6, 14, 43, 60, 90, 124, 146] and cognitive load [83]. Emotional content [14, 15] and stories which are emotionally powerful and richly narrated [146] could contribute more to presence than technological factors (i.e., the degree of immersion [15, 146] and stereoscopy [14]). Furthermore, the nature of emotions could have an effect on presence [11, 105], even if some results do not support this statement [56]. Valence and arousal were found to have a positive relationship with presence [11, 105], and relaxation, a negative relationship with presence [105]. Some studies also found that presence is influenced by cognitive abilities [5, 43] and personality traits such as trait anxiety [5].

On the other hand, fewer results found an influence of presence on emotions. Studies usually agree to say this link exists but is more complex [14, 15, 55, 56, 60, 124]. For example, Gromer et al. found that presence due to the quality of the VE did not influence fear, but that presence due to individual variabilities predicted later fear responses [60]. As such, Felnhofer et al. suggested that presence was a precondition for emotions to be felt, but that it did not influence the intensity of emotions [55].
In summary, presence, which is an important factor widely studied in IVEs, was found to have a circular relationship with ACS. On the one hand, the intensity of emotions and cognitive load influences presence. On the other hand, the effect of presence on ACS is more complex. As such, it was suggested presence was a precondition for emotions to be felt in IVEs.

# 4 MEASURING AFFECTIVE AND COGNITIVE STATES IN VIRTUAL REALITY

Various methods for measuring ACS have been proposed in the last decades. Kaplan et al. exposed 3 groups of strategies to measure emotions [66]: self-reports, observational methods, and psychophysiological measurements. Similarly, O'Donnell classified the methods to measure mental workload in 3 groups [104]: subjective (or self-reports), physiological, and task performance measures. We propose to merge their propositions as they have many common aspects by dividing the methods to measure ACS into 4 categories: self-reports, observational methods, task performances, and psychophysiological measures. Self-report methods are subjective as they collect the users' perceived and communicated experiences. On the other hand, observational, performance, and psychophysiological methods are objective as they can collect data at a high frequency without any intervention from the users. We will expose each 4 measurement categories and discuss their advantages and drawbacks in an IVE context.

## 4.1 Self-report Measures

Self-reports measures correspond to the methods in which individuals are asked to describe their ACS during a passed time period or at a given time. It usually involves the use of surveys or questionnaires, which can be paper-and-pencil, online or oral. It can also take other forms (i.e., interviews, self-confrontation).

### 4.1.1 Affective States

For the measure of **emotions from a dimensional perspective**, Bradley and Lang [31] developed the Self-Assessment Manikin (SAM), which depicts the three PAD dimensions of emotions by representing each of them by five graphic characters along a nine-point scale. This method was successfully used for the assessment of the user's emotion, especially the valence and the arousal dimensions. For the emotional valence dimension, the Positive and Negative Affect Schedule (PANAS) is a very popular method proposed by Watson et al. [152]. This method consists of 2 scales to measure positive affect and negative affect using ten psychometric items each.

Self-reports is also often used to measure **AS from a discrete perspective** by using Likert-Scales (LS), Visual Analogue Scales (VAS), and other custom questionnaires. Among standardized questionnaires, the State-Trait Anxiety Inventory (STAI) is a popular form often used to measure anxiety [142]. This test is composed of two scales of 20 propositions each: the STAI-S (state), also referred to as STAI form Y-A or Y1, and the STAI-T (trait), also referred to as STAI form Y-B or Y2. The STAI-T is usually used at the beginning of a protocol to recognize generally anxious people (who have higher scores). On the other side, the STAI-S score is a good indicator of short-term anxiety, and it can be used after each condition of a protocol. Spielberger et al. [142] also created a similar multi-scales for anger: the State-Trait Anger Expression Inventory (STAXI), which evaluates trait and state angers.

<figure><img src="image_4.png" /><figcaption>FIG. 2: Example of self-reporting method in an IVE. Instanteous Self-Assessment (ISA, measure of the user's mental workload using 5 different ratings [148]) in a VR Flight Simulator [87].</figcaption></figure>

### 4.1.2 Mental Workload

To measure **Mental Workload**, Cain [33] divided the self-report methods into 2 categories: multidimensional and unidimensional scales. The most popular scales are **multidimensional** ones: the NASA-TLX [61], the Subjective Workload Assessment Technique (SWAT) [122], and the Workload Profile (WP) [151]. The NASA-TLX consists of 6 scales regarding workload: the mental demand, the physical demand, the temporal demand, the performance demand, the effort, and the frustration. Each of them is rated individually on a 0-100 scale, weighted, and aggregated for a global Workload Score. The SWAT considers the mental workload as composed of three components: the time load, the mental effort and the psychological stress load. It uses a method based on interval properties. Finally, the WP is based on the assumption that mental workload can be defined by the dimensions outlined in Wickens's multiple resources theory [153]. Among **unidimensional** scales, the Rating Scale of Mental Effort (RSME) [158] evaluates the mental effort on a continuous vertical axis from 0 to 150, and the Instantaneous Self Assessment (ISA) evaluates mental workload using 5 different ratings [148] (see Fig. 2). The latter has been especially used during flight training [148]. Unidimensional scales are quicker to respond but they can have a smaller diagnosticity as they consider only one dimension.

Self-report methods are considered as cheap and convenient. It is a well-accepted measure of ACS thanks to their self-referential nature and high face validity [66]. One of the main issues concerns the ability of individuals to reveal their ACS and their interpretation of the latter. Results are restricted by human language and can be biased based on what individuals believed they felt, or on which ACS they wanted to report to be more socially desirable [84, 135]. Moreover, since real-time is an important aspect of VR, it is difficult to get a high-frequency measure of the ACS using self-reports. Depending on the way they are done, self-report measures can also break immersion in a Virtual Environment (VE) [139] and have an impact on the user's subjective impression.

## 4.2 Observational Measures

Unlike self-reports, observational methods are based on an exterior point of view and evaluation of body or facial behaviors. Those behaviors are usually associated with body language and include nonverbal indicators such as facial expressions, body postures (e.g., position of hands, style of walking), gestures, touch and the
use of personal space [129]. One can identify patterns of behavior, as well as their frequency of appearance and correlates them with ACS [102].

For the recognition of emotions, **facial expressions recognition** is from far the most popular observational method outside VR. Ekman et al. developed recognition methods for facial expressions of the Big Six emotions [51]: the FACS. It has proven to be a good standard for categorization and measurement of emotional expression but has some limitations in recognizing some expressions across different cultures due to facial deformations and skin colour [17, 64]. Moreover, it can be more challenging to do facial recognition in VR as parts of the face are most of the time occluded by the HMD. Multiple research paths were explored to try answering this problem using Electromyography (EMG) [21, 95], photoreflective sensors [67, 147], proximity sensors [81], strain gauges to measure foam deformation on the HMD [80], and RGB-D camera on the mouth region [80, 106].

On the other hand, VR often implies body engagement, which makes **body gesture** recognition particularly interesting to study ACS. Noroozi et al. provided a survey on the emotional body gesture recognition, as well as a table with general movement protocols associated with the Big Six [102]. Other methods such as **voice treatment** have been less explored in VR but could be interesting given the growing interest in multi-user VR applications and their implication on oral interactions.

Interaction is one of the main components of VR [144], which makes users' behaviors particularly interesting to study in this context. In that sense, observational measures provide rich data and can be useful for examining ACS manifestation in real-time [66]. However, there are some limitations. This method can hardly be universal. ACS such as emotions are built over a lifetime, and their manifestation can vary across different individuals depending on factors such as their cultures, personalities, and genders [17, 94]. Also, people can intentionally express, suppress or hide their psychological states during an observation. Their behaviors can be modulated by the environment (e.g., real or virtual) and the situation (e.g., experimental or entertainment). Moreover, VR equipment can be cumbersome and sometimes make the behavior in IVEs less natural. It is also worthwhile noting that most observational measures greatly depend on the context and interactions to perform in the application.

## 4.3 Tasks Performance Measures

Task performance measures have mainly been explored in works studying CS. O'Donnell distinguished 2 major types of performance-based measures [104]: primary task measures and secondary task measures. Primary task measures attempt to directly assess the user's performance on the task of interest. Typical measures of task performances include accuracy, reaction or response times, and error rates. They can be insensitive to the user's CS if the variability in task demands are insufficient to result in observable changes as they do not give information on the user's remaining cognitive capacities. On the contrary, secondary task measures provide an index of the remaining operator capacity while performing the primary tasks. There are 2 methodologies for secondary tasks: auxiliary task and loading task [33]. In auxiliary task methods, users are instructed to maintain consistent performances on the primary task regardless of the difficulty of the overall task. Therefore, the secondary task performances are an indirect indicator of the user's reserve capacity. In the loading

task approach, the secondary task deliberately causes degradation of the primary task, which require consistent performances on the secondary task. The primary task performance measures are, in that case, more sensitive to the users' mental workload variations.

Task performance measures have great advantages as they can be assessed easily and continuously. However, similarly to observational measures, they are context-dependent. They were shown to have great correlations with CS but can be modulated by users' engagements which can make them inaccurate for the measure of ACS [48, 104, 151].

## 4.4 Psychophysiological Measures

The term psychophysiology refers to the physiological responding to psychological phenomenons. In physiological computing [53], physiological activities and changes are direct reflections of processes in the Autonomic Nervous System (ANS) and in the Central Nervous System (CNS). Since physiological responses are issued from psychological processes, it was hypothesized that it is possible to translate them to psychological states via the extraction of some specific features [118].

### 4.4.1 Measure of the ANS

The most frequently used physiological measures of ACS are indicators of the ANS. Those are uncontrolled consequences of the CNS activity on physical processes. Kreibig provided a review of the ANS activity, detailing the physiological signal feature reliability and behaviour depending on the different elicited emotions [76]. Although there are still debates about the relation between ANS measures and psychological activities [94], the use of these measures to assess negative arousal is uncontested [66]. The most popular measures of the ANS activity include cardiovascular, electrodermal, and pupillometry indicators.

**Cardiovascular activity** can be measured by means of electrocardiography (ECG), and photoplethysmography (PPG). On the one hand, **ECG** detects electrical changes that originate from the ventricles contracting and expelling blood. On the other hand, **PPG** uses a small optical sensor in conjunction with a light source to measure changes in the skin light absorption as blood perfuses through the skin after each heartbeat. PPG sensors usually are less obtrusive, but also less accurate than ECG methods. Among the popular features, both sensors can be used to extract the Heart Rate (HR) and the Heart Rate Variability (HRV), which corresponds to the adaptation changes in the time intervals between two consecutive heartbeats in response to an environmental and/or an internal stimuli [97]. For instance, anxiety is usually associated with an increase in HR and a decrease in HRV [76]. On the other hand, **electrodermal Activity (EDA)**, often referred to as Galvanic Skin Response (GSR), is the term used to define autonomic changes in the electrical properties of the skin [136]. The activity of the sympathetic nervous system is directly linked to the activity of the sweat glands, which is in turn related to the activity of the skin epidermis [28]. The EDA can be represented by its 2 main components: the tonic skin conductance and the phasic skin conductance. The tonic skin conductance corresponds to the normal conductance of an individual in the absence of any stimulus of discrete changes in the experimental environment and is related to the Skin Conductance Level (SCL). The phasic skin conductance happens in accordance to an affective event [85] and is related to the Skin Conductance Responses (SCRs). The EDA has proven strong content validity and is widely used to measure
a user's affective and cognitive changes [53]. For instance, fear was reliably associated with an increase in SCR and SCL responses [76]. As for **pupillometry measures**, they can be assessed using eye-tracking devices. Changes in pupil size are caused by 2 antagonistic muscles: the dilator pupillae, which dilates the pupil, and the sphincter pupillae, which constrict it [50]. The pupil dilation is provoked by activity in the sympathetic pathway of the ANS (fight-or-flight), and the constriction by activity in the parasympathetic division of the ANS (rest-and-digest) [50]. Pupil dilation data was especially shown to be influenced by cognitive efforts [40, 50]. Derived behavioural measures such as gaze direction, blinks, fixation, and eyes saccades were also largely exploited in the analysis of the human's ACS in the literature [34, 37, 40]. Aside from these measures, **respiration (Resp)**, **skin temperature (Temp)**, **ElectroOculoGraphy (EOG)** and EMG measures can also be affected by the ANS processes linked to a change in the user's ACS [85].

There are good practices that suggest placing ANS sensors at specific places [157]. In VR studies, ECG electrodes usually are placed on the user's torso (e.g., via a belt), PPG sensors on the earlobes or the wrist, and EDA sensors, on the fingers, wrist, or directly on the face via the headset. Eye-trackers are also more and more directly built in VR HMDs. To take into account within and between-individual variability, most studies use a baseline to observe changes or to normalize the obtained signals.

### 4.4.2 Measure of the CNS

Electroencephalography (EEG) and other neuroimaging methods measure directly the activity caused by the functioning of the CNS. The frontal lobe of the cortex, which is the largest part of the human brain, is notably responsible for conscious thought, which makes it the most impacted by the ACS [3].

EEG measures rhythmic macroscopic neural oscillations produced by the synchronized activity of brain neurons by placing electrodes on the scalp and amplifying the signals. These oscillations can notably be observed in specific frequency ranges named as $\alpha$ (8-13 Hz), $\delta$ (1-4 Hz), $\theta$ (4-8Hz), $\beta$ (13-30 Hz) and $\gamma$ (30-70 Hz) frequency bands. Specifically, $\theta$, $\alpha$, and $\beta$ waves have been found to be impacted by cognitive performances and the mental effort [54]. Another popular and affordable neuroimaging methods is the functional Near-Infrared Spectroscopy (fNIRS), a multi-wavelength optical spectroscopy technique that measures hemodynamic brain responses resulting from intentional sensory, motor, or cognitive activities [9]. Similarly to PPG sensors, fNIRS systems use skin tissue's particular absorption properties in the near-infrared range to measure and localize changes in oxygenated (i.e., HbO2) and deoxygenated (i.e., HbR) hemoglobin concentration following neural activation. Overall, increasing task difficulty was found to increase HbO2 and decrease HbR [134]. Studies using both systems corroborate the fact that positive and negative affects are respectively associated with the frontal left and right hemisphere of the brain [3]. EEG is known to have a high temporal resolution but a low spatial resolution on the brain [41]. Compared to EEG, fNIRS offers a less accurate temporal assessment of the brain activity but a better spatial resolution and is less sensitive to movements. Other neuroimaging methods such as magnetoencephalography and functional Magnetic Resonance Imaging offer a more precise assessment of brain activity location and a good temporal resolution. They have been used with VR devices [35] but can hardly be used outside a clinical context as they are not mobile.

<figure><img src="image_5.png" /><figcaption>FIG. 3: PhysioHMD [21]. Example of module which integrates physiological sensors directly on the headset.</figcaption></figure>

Measures of the CNS can provide reliable estimations of the emotions [3, 133], but also of the cognitive load [7] or mental workload [10]. However, they can be cumbersome, sensitive to motions and thus, complex to use with a VR headset.

Overall, psychophysiological measures have been widely studied in and outside VR. The measures of the ANS and CNS are both interesting since they provide high frequency and objective measures of the physiological activity influenced by the user's psychological state. It is one of the most considered solutions to measure ACS from an objective perspective in VR (see Tab. 1). One downside is that the sensors can be cumbersome, thus reduce immersion in VEs. For that reason, many laboratories and companies have focused on integrating physiological sensors directly on VR headsets in recent years. For example, PhysioHMD introduces EEG, EMG, EOG, and EDA directly on the headset (see Fig. 3) [21], EmotionalBeasts study introduced EDA and PPG on the HMD [20], and LooxidVR<fnref n="1" /> and Neurable<fnref n="2" />, EEG on the HMDs. Still, one matter of concerns is that physiological sensors are often sensitive to movements, which makes interactions and locomotion in VR sources of artefacts in the signals. Additionally, they can be greatly influenced by within and between-individual variabilities [149] and by cybersickness [44, 69].

## 5 AFFECTIVE AND COGNITIVE VIRTUAL REALITY

This section provides an overview of research in Affective and Cognitive VR in IVEs in a non-clinical context. Combining ACS and VR has advantages in several application fields. Among the 63 studies retained in this survey, most focused on **global VR** applications to understand how users felt in a VR context and how to enhance VR application using the user's mental state (i.e., 37 studies) [5, 11, 13, 14, 15, 16, 19, 20, 29, 32, 34, 36, 37, 46, 47, 48, 49, 55, 56, 58, 60, 74, 79, 83, 86, 88, 89, 90, 91, 98, 105, 108, 113, 117, 124, 140, 143]. 10 focused on **artificial intelligence (AI)** applications by training ACS recognition models in VR [2, 21, 24, 92, 93, 96, 100, 121, 126, 150]. 5 were done in a **gaming** context [1, 62, 82, 109, 112], 3 in a **military** context [114, 115, 154], 2 in a **sport** context [8, 145], 2 were applied in an **educational** context [39, 43], 2 were done to improve immersive **training** [38, 87], 1 targeted an **aerospace** application [6], and 1 focused on the benefits of immersion for **journalism** [146]. The measured ACS in the 63 studies are summarized in Fig. 4.

We decided to treat the 63 studies following the 3 categories: induction, recognition, and exploitation of ACS (Fig. 1). The first group of studies explored methods to **induce ACS**, as well as the

<footnote n="1">1. Looxidvr. https://looxidlabs.com/looxidvr/. Accessed: 2020-03-02.</footnote>

<footnote n="2">2. Neurable. https://neurable.com/. Accessed: 2021-03-17.</footnote>
<figure><img src="image_6.png" /><figcaption>FIG. 4: Illustration of the relative occurrence of the different measured ACS in the 63 Affective and Cognitive VR studies retained in this survey. Light blue corresponds to emotions from a dimensional perspectives. Dark blue corresponds to discrete AS. Green corresponds to measures linked to CS. The most measured ACS were Arousal and Valence with 16 occurrences each.</figcaption></figure>

influence of specific IVEs parameters on users' ACS (see Tab. 1). The second group deals with studies which developed models to **recognize ACS** using physiological signals and ML algorithms in IVEs (see Tab. 3). Finally, the last identified group targets **the exploitation of ACS** to modulate IVEs content and parameters based on ACS measures (see Tab. 2). Some studies were therefore classified in several of these categories.

## 5.1 Induction of Affective and Cognitive States in VR

This section aims to depict the results of studies that found an influence of IVEs variables on the user's AS or CS. Therefore, stimuli used to elicit ACS or which have shown to influence ACS are detailed in Table. 1, as well as the methods used to measure ACS. The studies can be divided into 2 sections depending on their purposes in regards to the induction of AC or CS: studies that aim to induce a variety of different ACS or different levels of ACS, and studies whose objectives are to study the influence of specific parameters on an AS or a CS.

### 5.1.1 Induction of Several Affective and Cognitive States

As such, studies that intended to induce different ACS or different levels of ACS have mainly used emotional videos, emotional VEs, emotional scenarios, or cognitive tasks in IVEs.

**Emotional videos** designate panoramic 360°videos displayed in a VR device. There is no 3D object nor interaction and users can only navigate in a limited space and/or move their head. As such, Li et al. developed a library of 73 immersive VR clips which were labelled using arousal and valence responses [79]. This was done similarly to previous works which developed databases of affective stimuli based on pictures (e.g., the International Affective Picture System (IAPS) [77]) or audio sources (e.g., the International Affective Digital Sounds (IADS) [30]) in VR. In another study, Mavridou et al. presented 20 non-panoramic emotional videos from an affective film library [130] along with 20 neutral videos in a VR cinema environment. The 20 emotional videos managed to induce balanced classes of emotions (i.e., 2 levels of arousal × 2 levels of valence, 5 videos per classes). In the same line, Marín-Morales et al. presented 4 architectural 360° panorama environments which succeeded to elicit 2 levels of arousal × 2 levels of valence by varying colours and geometries [92] (see

Fig. 5). On the other hand, Macedonio et al. presented 10 anger-provoking scenes via panoramic videos to users [88]. Those managed to induce anger responses to users (see Table. 1). In the journalism field, Sundar et al. presented 2 stories in a VR HMD: a sad one about refugees, and a more positive one about dolphins [146]. Those induced the expected valence. It is also the case for Anderson et al.'s study, which presented a neutral 360° classroom class scene, and 2 natural panoramic views [6].

As for **emotional VEs**, those imply 3D objects but no other interactions than navigating in the VE and moving the head. The Empathic Computing Lab developed 5 VEs based on a jungle safari to induce happiness, anxiety, fear, disgust, and sadness [36, 37, 47]. Users are on a car that moves through the safari, and they are exposed to audio-visual stimuli. A pilot study was conducted to make sure the environments triggered the appropriate emotions [36, 47]. In the EMMA Project [4], a procedure was developed to induce different moods/emotions using a virtual park scene. The user explores a park, in which light and other audio-visual stimuli change depending on the targeted mood. The latter was shown to induce sadness, joy, anxiety, relaxation, and neutral states [14, 15, 16, 124]. In the same line, Felnhofer et al. developed 5 virtual parks which induced joy, sadness (no effect), boredom, anger, and anxiety by varying weather, sounds, light, and animated contents [56]. Robitaille and McGuffin also developed a virtual forest that could change from a calm to a stressful condition by varying similar parameters [126]. Other studies made use of distinct VEs to elicit different emotions [34, 82, 105]. On the stressful and anxiety side, a famous study in VR consists of the virtual pit, which exposes a user to a fear of height situation [98]. This was found successful to induce presence and changes in psychophysiological responses on users [60, 90, 98].

Furthermore, **emotional scenarios** are emotional VEs in which users can interact with virtual entities. Those can be horror games [46, 62, 112], games which are supposed to elicit sadness [109], joy [24, 46], neutral games [11], or serious games [38, 43, 87]. VR games and scenarios are usually very engaging and were shown to influence emotions [38, 43, 46, 62, 109, 112], CS [38, 43, 87], and flow [24]. Moreover, one large field of interest in VR concerns virtual public speaking situations [55, 74, 83, 117, 140]. Those were found to greatly influence anxiety, which can also be modulated by the virtual audience type [117, 140] and its reaction [74]. More specifically, some studies focused on the effect of threatening [114, 154] or emergency situations [2] on physiological signals and tasks performance. Others used everyday anxiety-provoking situations (e.g., a test day [5]), and VR sport to validate anxiety stimuli such as the crowd, and the presence and reaction of characters in a competitive situation [8, 145].

On the other hand, works studying CS are using most of the time **cognitive tasks**. As such, the n-back task is a standardized cognitive task targeting working memory which can be declined in different difficulty levels and has been used extensively in VR to induce different levels of mental workload. It can be auditory [86] or visual, and the stimuli can take different forms such as color balls [121, 150], letters (for verbal task) or oriented symbols (for spatial task) [32]. Steed et al. also used a letter recall task, which targets users' working memory, to test the influence of some independent variables on cognitive task performance [113, 143]. On the other hand, Bergström et al. used a counting backward task [19], which targets executive functioning. Similarly, Banakou et al. used a post-experiment Tower Of London task [13], which
<figure><img src="image_7.png" /><figcaption>FIG. 5: Induction of 2 levels of arousal and 2 levels of valence using 4 architectural 360° panorama environments in a VR HMD [92]. Left: negative valence; right: positive valence; bottom: low arousal; top: high arousal.</figcaption></figure>

especially target planning and problem-solving skills. On the
mental rotation side, Collins et al. [39] used a spatial rotation
task to induce different levels of cognitive load. Spatial rotation
tasks were also used as distractors in [113, 143]. Dey et al. [48]
and Gerry et al. [58] made use of a visual searching task, where
users had to find a target shape among several colored visual
distractors, to generate different task load levels. Recall of objects
questionnaires are also often used to assess spatial awareness in
VEs [49, 89, 91]. Cognitive tasks can also be used to induce
arousal, such as arithmetic tasks (e.g., [6]), or the Virtual Reality
Stroop Task used in [114, 154], adapted from the paper and pencil
Stroop Test. These tasks were used as single standardized tasks to
induce different levels of CS using different difficulty levels, but
some studies also explored multitasking in IVEs to modulate the
user's mental workload or arousal level [87, 114].

5.1.2 Other Factors Influencing Affective and Cognitive States

Factors such as display types, immersion levels, avatars, and user
profiles were found to impact users' ACS.

The relation between the **display types, immersion levels** and users' ACS can be complex as some studies found an effect of IVEs on ACS [89, 93, 112], and others did not [74, 86]. For example, Pallavicini et al. found no effect of VR vs non-immersive console on anxiety. However, they found that VR provoked more happiness than non-IVEs [112]. These results go in the same line as other studies' findings [89, 93]. A supposition is that the novelty of VR could elicit this difference in emotional responses [93, 112]. Moreover, immersive display can induce more intense AS [38, 146], improve spatial awareness [89], and learning retention [38] compared to non-immersive methods and display. Furthermore, simulation fidelity was found to increase sport anxiety [145] and memory of objects [91]. On the other hand, stereoscopy did not show an influence on emotions [14, 83].

Avatars were also found to influence ACS. For example, Osimo et al. proposed a self-therapy where the user is describing a problem and answering to themselves by embodying first the patient, and then the counsellor body [108]. The experiment, and the body ownership illusion were found to improve mood and emotions, especially when the user was embodied in Freud's body as the counsellor. Furthermore, embodying a virtual body associated with high cognitive abilities such as Einstein was found to result in better cognitive performance and a decrease of age-based discrimination in [13]. Self- avatar with hand gestures allowance [143], as well as avatar types (full-body, real-body,

hands-only, no body) [113] were also found to have significant
effects on cognitive task performances. These results suggested
that a higher sense of embodiment was associated with greater
cognitive task performances. In the same line, being embodied in
an uncomfortable posture resulted in changes in the physiological
signals and more mistakes in a cognitive task [19]. Bourdin et al.
also discovered that a virtual out of body experience resulted in a
lower fear of death [29].

Aside from avatars, **personality traits** such as trait anxiety can influence emotions [5, 55, 117]. Users who are originally anxious will experience greater anxiety and discomfort in anxiety-triggering IVEs [5, 55]. **Other paradigms** like redirected walking were also found to increase cognitive task performances and impact walking behaviour [32]. On the interactions side, body participation and voice control improved emotions and enjoyment compared to more classical interactions [11, 109]. In another study, Dinh et al. found that multi-sensory feedback such as tactile, olfactory and auditory ones resulted in a greater memory for objects in the VEs [49]. In the same line, Chen et al. discovered HR feedback in emotional VEs to be more enjoyable than no HR feedback, especially audio-haptic HR feedback [36]. They also found that altering the HR feedback could change participants' emotions but not their physiological signals themselves [47]. However, there was no effect of sharing HR to another player in collaborative games on the observer's emotion and empathy [46]. In a horror game, Houzangbe et al. found that users who tried to influence their HR (biofeedback to control fear stimuli and stressors) experienced more fear than those who did not [62].

In conclusion, studies that intended to induce different AS or different levels of AS mainly used emotional videos, VEs, and scenarios. As for the induction of CS, studies mainly adapted standardized cognitive tasks in VR. Cognitive tasks were also found to increase the user's arousal level. Furthermore, immersion, high-fidelity displays, and multi-sensory feedbacks were found to increase AS intensity, improve AS and cognitive performances. Altering the avatar's control and appearance, and redirected walking was also found to increase cognitive demands. Also, embodiment in VR can enhance the user's cognitive experience and performance. Finally, personality traits and physiological feedback can also influence the user's ACS.

5.2 Recognition of Affective and Cognitive States in VR

Recent studies tried to passively measure users' ACS in IVEs by developing recognition models. This enables to assess users' psychological states without having to disrupt their immersion in the VE. However, recognition models typically require the use of objective measures to be continuous. In IVEs, studies mainly relied on physiological signals (see Tab. 3). Due to the complexity of physiological data, Machine Learning (ML) methods are usually required in order to extract the users' ACS. These algorithms aim to infer the function between the input (e.g., physiological data) and the output data (e.g., the subjective AS or CS measure) using supervised techniques [75]. A challenge remains the training of ML models as they require the gathering of labeled data and specific induction protocols to ensure that different classes of ACS are generated. A summary of the different works which developed an ACS recognition model in an IVE is given in Table 3.
<table><caption>TABLE 1: Summary of works that induced ACS in IVEs. Significant changes are indicated in bold in "Objective Measures" and "Subjective Measures" columns. Significant effects are indicated in bold in the "Inducer" column. In the "Inducer" column, the emotional VEs, videos, and games are not detailed when several of them are used in the study.</caption><thead><tr><th>Ref.</th><th>ACS</th><th>Measured ACS</th><th>Subjects Field</th><th>Inducer</th><th>Objective Measures</th><th>Subjective Measures</th></tr></thead><tbody><tr><td>[105]</td><td>AS</td><td>Arousal, Valence, Dominance</td><td>146 VR</td><td>Emotional VEs, presence</td><td>-</td><td>SAM</td></tr><tr><td>[37]</td><td>AS</td><td>Arousal, Valence, Dominance</td><td>20 VR</td><td>Emotional VEs, HR sensory feedback</td><td>Behaviour, Pupil Dilatation</td><td>PANAS, SAM</td></tr><tr><td>[47]</td><td>AS</td><td>Arousal, Valence, Dominance</td><td>19 VR</td><td>Emotional VEs, HR feedback frequency</td><td>HR (ECG), EDA</td><td>PANAS, SAM</td></tr><tr><td>[79]</td><td>AS</td><td>Arousal, Valence</td><td>95 VR</td><td>Emotional videos</td><td>Behaviour</td><td>SAM (2D)</td></tr><tr><td>[96]</td><td>AS</td><td>Arousal, Valence</td><td>11 AI</td><td>Emotional videos</td><td>HR (PPG and ECG)</td><td>Continuous Affect Self-Rating</td></tr><tr><td>[92]</td><td>AS</td><td>Arousal, Valence</td><td>38 AI</td><td>Emotional VEs</td><td>HR (ECG), EEG</td><td>SAM (2D)</td></tr><tr><td>[93]</td><td>AS</td><td>Arousal, Valence</td><td>60 AI</td><td>Emotional VEs, baseline with IAPS and emotional VEs</td><td>HR (ECG), EEG</td><td>SAM (2D)</td></tr><tr><td>[11]</td><td>AS</td><td>Arousal, Valence</td><td>56 VR</td><td>Presence, body participation in a pick and place game</td><td>-</td><td>SAM (2D)</td></tr><tr><td>[146]</td><td>AS</td><td>Arousal, Valence, enjoyment</td><td>129 Journalism</td><td>Emotional videos, Emotional VEs</td><td>-</td><td>Valence and emotional intensity scales, Empathy adjec- tive scale</td></tr><tr><td>[108]</td><td>AS</td><td>Arousal, Valence, "global Mood"</td><td>22 VR</td><td>VR self-therapy, Avatar type (Freud vs look-alike), Avatar control (synchrone, asynchrone)</td><td>-</td><td>Profile Of Mood States, SAM (2D), SUDS</td></tr><tr><td>[126]</td><td>AS</td><td>Arousal</td><td>12 AI</td><td>Stressful situation</td><td>Behaviour, HR (ECG), EDA,</td><td>SAM</td></tr><tr><td>[88]</td><td>AS</td><td>Arousal, Anger</td><td>41 VR</td><td>Emotional videos</td><td>HR (ECG and PPG), EDA, Resp, Temp</td><td>STAXI-2, PANAS, VAS (arousal), LS (10 points, Anger)</td></tr><tr><td>[46]</td><td>AS</td><td>Valence</td><td>26 VR</td><td>Emotional games, HR of a collaborator feedback</td><td>Eye Behaviour</td><td>PANAS, Inclusion of Other in Self scale</td></tr><tr><td>[16]</td><td>AS</td><td>Valence, Anxiety, Joy, Sadness</td><td>110 VR</td><td>Emotional VEs</td><td>-</td><td>LS (7 points - Sadness, joy, anxiety, relaxation), PANAS</td></tr><tr><td>[124]</td><td>AS</td><td>Valence, Anxiety, Anger, Dis- gust, Happiness, Sadness, Sur- prise</td><td>61 VR</td><td>Emotional VEs, presence</td><td>-</td><td>LS (10 points - Sad, Happy, Anxious, Relaxed), VAS (Happiness, Sadness, Anger, Surprise, Disgust, Anxiety, and Quietness), PANAS, STAI-S</td></tr><tr><td>[145]</td><td>AS</td><td>Anxiety</td><td>25 Sport</td><td>Anxiety triggers, field of regard, simulation fidelity, trait- anxiety, sport experience</td><td>Task perf., Behaviour, HR (ECG), EDA</td><td>State-Trait anxiety Inventory of Cognitive and Somatic Anxiety (state), Competitive state anxiety inventory-2</td></tr><tr><td>[5]</td><td>AS</td><td>Anxiety</td><td>210 VR</td><td>Anxiety triggering situation, trait anxiety, personality, individual spatial abilities, computer experience</td><td>-</td><td>STAI-S, SUDS</td></tr><tr><td>[8]</td><td>AS</td><td>Anxiety (Social Anxiety)</td><td>18 Sport</td><td>Sport competition scenario</td><td>Task perf., Behaviour, HR (ECG), EDA</td><td>LS (7 points - on mental state, motivation, perception of the VE during the competition)</td></tr><tr><td>[56]</td><td>AS</td><td>Anxiety, Boredom, Joy, Sadness</td><td>120 VR</td><td>Emotional VEs, presence</td><td>EDA</td><td>modified DES (Differential Emotions Scales)</td></tr><tr><td>[60]</td><td>AS</td><td>Anxiety, Discomfort</td><td>49 VR</td><td>Realism, presence, fear of height situation</td><td>HR (ECG), EDA</td><td>Acrophobia Questionnaire, STAI-S, SUDS</td></tr><tr><td>[83]</td><td>AS</td><td>Public Speaking Anxiety</td><td>86 VR</td><td>Stereoscopy, presence, virtual public speaking situation</td><td>HR (PPG)</td><td>MPRCS</td></tr><tr><td>[74]</td><td>AS</td><td>Public Speaking Anxiety/Fear (stress physiological response)</td><td>66 Therapy</td><td>Virtual/real, public speaking situation</td><td>HR (ECG), Salivary Cortisol</td><td>PRCS (Personal Report of Confidence as a Speaker), STAI-S</td></tr><tr><td>[117]</td><td>AS</td><td>Public Speaking Anxiety/Fear</td><td>40 VR</td><td>Virtual public speaking audience types, prior public- speaking anxiety, immersion</td><td>-</td><td>MPRCS (Modified Personal Report of Confidence as a Speaker), somatic response, self-rating of talk perf.</td></tr></tbody></table>

continued on next page
continued from previous page

<table><thead><tr><th>Ref.</th><th>ACS</th><th>Measured ACS</th><th>Subjects Field</th><th>Inducer</th><th>Objective Measures</th><th>Subjective Measures</th></tr></thead><tbody><tr><td>[140]</td><td>AS</td><td>Public Speaking Anxiety/Fear</td><td>41 VR/Therapy</td><td>Public speaking phobic participants, virtual public speaking situation</td><td>HR (ECG)</td><td>MPRCS, APQ</td></tr><tr><td>[90]</td><td>AS</td><td>Anxiety, Fear of Height (via HR and Discomfort)</td><td>41 VR</td><td>Threatening situations (predicted or not)</td><td>HR (ECG)</td><td>SUD (Subjective Units of Discomfort)</td></tr><tr><td>[34]</td><td>AS</td><td>Anxiety, Happiness, Sadness, (Physiological signals)</td><td>20 VR</td><td>Emotional VEs, cybersickness inducer</td><td>HR (PPG), Eye-tracking data</td><td>LS (7 points - happy, sad, comfortable, anxious)</td></tr><tr><td>[14]</td><td>AS</td><td>Anxiety, Joy, Sadness, Valence</td><td>40 VR</td><td>Emotional VEs, stereoscopy</td><td>-</td><td>LS (10 points - Sadness, joy, anxiety, relaxation), PANAS</td></tr><tr><td>[112]</td><td>AS</td><td>Anxiety, Happiness, Surprise</td><td>26 Gaming</td><td>Horror game, immersion, trait anxiety</td><td>HR (ECG), EDA</td><td>VAS (Anxiety, Happiness, Surprise)</td></tr><tr><td>[55]</td><td>AS</td><td>Anxiety (social)</td><td>65 VR</td><td>Presence, social anxiety, virtual public speaking situation</td><td>HR (ECG)</td><td>STAI-S</td></tr><tr><td>[109]</td><td>AS</td><td>Enjoyment, Sadness</td><td>22 Gaming</td><td>Sad game, voice control interaction</td><td>-</td><td>LS (7 points - enjoyment, emotionally affected by the game)</td></tr><tr><td>[98]</td><td>AS</td><td>Fear of Height (via physiological signals)</td><td>259 VR</td><td>Fear of height situation, multiple exposures, passive haptics, frame rate, latency</td><td>HR (PPG and ECG), EDA, Temp</td><td>-</td></tr><tr><td>[29]</td><td>AS</td><td>Fear of Death</td><td>32 VR</td><td>Out of body experience types, visuotactile and visuomotor synchrony</td><td>-</td><td>Collett-Lester Fear of Death scale</td></tr><tr><td>[62]</td><td>AS</td><td>Fear</td><td>32 Gaming</td><td>Horror game, HR biofeedback</td><td>Task perf., HR</td><td>LS (5 points - fear (in engagement items))</td></tr><tr><td>[2]</td><td>AS</td><td>Excitement trend</td><td>20 AI</td><td>Emergency situation</td><td>EEG, Eye Behaviour</td><td>-</td></tr><tr><td>[24]</td><td>AS</td><td>Flow</td><td>36 AI</td><td>Shooting game</td><td>HR (ECG), Resp, EMG</td><td>Flow Short Scale</td></tr><tr><td>[82]</td><td>AS</td><td>(Physiological signals)</td><td>26 VR</td><td>Emotional VEs</td><td>EDA, HR (ECG)</td><td>-</td></tr><tr><td>[36]</td><td>AS</td><td>(Physiological signals)</td><td>7 VR</td><td>Emotional VEs, multi-sensory HR feedback</td><td>Eye-tracker data</td><td>-</td></tr><tr><td>[15]</td><td>AS</td><td>Emotional Engagement</td><td>60 VR</td><td>Emotional VEs, immersion</td><td>-</td><td>LS (10 points - emotional engagement and indifference)</td></tr><tr><td>[6]</td><td>ACS</td><td>Arousal (physiological), Valence</td><td>18 Aerospace, medicine</td><td>Emotional VEs, VE preference, cognitive task</td><td>HR (ECG), EDA</td><td>PANAS</td></tr><tr><td>[114]</td><td>ACS</td><td>Arousal (physiological)</td><td>50 Military</td><td>Threat situation levels, cognitive task presence</td><td>HR (ECG), EDA, Resp</td><td>(Cognitive workload levels = cognitive task levels)</td></tr><tr><td>[154]</td><td>ACS</td><td>Arousal (physiological), Cognitive perf.</td><td>18 Military and AI</td><td>Threat situation levels, cognitive task levels</td><td>Task perf., Behaviour, HR (ECG), EDA, EEG, Resp, EOG</td><td>-</td></tr><tr><td>[38]</td><td>ACS</td><td>Arousal (physiological), Fear</td><td>48 Transport</td><td>Aviation safety education method (card vs VR), aviation emergency scenario</td><td>Task perf. (knowledge retention), EDA, HR (PPG)</td><td>LS (7 points, 6 fear-related adjectives, 6 engagement-related statements)</td></tr><tr><td>[43]</td><td>ACS</td><td>Positive and Negative emotions</td><td>23 Learning</td><td>(Positive emotions, cognitive abilities, immersions) on presence, educational VEs</td><td>-</td><td>LS (6 points, positive emotions: enjoyment, hope, pride and positive deactivating emotion: relief)</td></tr><tr><td>[89]</td><td>ACS</td><td>Enjoyment, Fear, Cognitive Perf.</td><td>63 VR cinema</td><td>Emotional videos, display types</td><td>Tasks perf. (among which, spatial awareness)</td><td>LS (5 points - enjoyment; fear, nervousness), Narrative Engagement Questionnaire</td></tr><tr><td>[87]</td><td>CS</td><td>Mental Workload</td><td>53 Training</td><td>Piloting, communication, resources management tasks</td><td>Tasks perf.</td><td>ISA</td></tr><tr><td>[39]</td><td>CS</td><td>Cognitive Load</td><td>24 Learning</td><td>Spatial rotation task (3 levels) for classification</td><td>Task perf., HR (PPG), EDA, Temp, and Accelerometer (filter)</td><td>Self-reported moments of insight, (Cognitive workload levels = cognitive task levels)</td></tr><tr><td>[48]</td><td>CS</td><td>Cognitive Load</td><td>14 VR</td><td>Visual search task (20 levels)</td><td>Task perf., EEG</td><td>-</td></tr><tr><td>[86]</td><td>CS</td><td>Mental Effort</td><td>27 VR</td><td>N-back task (1, 2, 3), walking</td><td>Task perf., Behaviour, HR (PPG), EDA</td><td>RSME</td></tr></tbody></table>

continued on next page
<table><caption>TABLE 2: Summary of works that exploited ACS in IVEs. **Explicit** refers to an adaptation of the IVE knowingly to users, and **Implicit** refers to an adaptation of the IVE unknowingly to users. "Explicit exploitation" has 2 subcategories: **BF**: Biofeedback, which refers to studies that incite users to control their physiological state through feedback in the IVE; and **PF**: Physiological Feedback, which refers to studies that directly display the user's physiological signals features in the virtual environment or to a third party interface.</caption><caption>continued from previous page</caption><thead><tr><th>Ref.</th><th>ACS</th><th>Measured ACS</th><th>Subjects Field</th><th>Inducer</th><th>Objective Measures</th><th>Subjective Measures</th></tr></thead><tbody><tr><td>[121]</td><td>CS</td><td>Mental Workload</td><td>10 AI</td><td>N-back task (1, 2, 3)</td><td>fNIRS</td><td>(Mental workload levels = cognitive task levels)</td></tr><tr><td>[150]</td><td>CS</td><td>Cognitive Workload</td><td>15 AI</td><td>N-back task (0, 1, 2)</td><td>EEG</td><td>(Cognitive workload levels = cognitive task levels)</td></tr><tr><td>[13]</td><td>CS</td><td>Cognitive Perf.</td><td>30 VR</td><td>Virtual body (Einstein vs young male)</td><td>Task perf.</td><td>Implicit Association Test</td></tr><tr><td>[19]</td><td>CS</td><td>Cognitive Perf.</td><td>31 VR</td><td>Comfortable or not virtual body posture, body ownership illusion, cognitive task</td><td>Task perf., HR (ECG)</td><td>Autonomic Perception Questionnaire</td></tr><tr><td>[32]</td><td>CS</td><td>Cognitive Perf.</td><td>16 VR</td><td>Cognitive task (verbal vs spatial vs none), redirected walking curvature gains</td><td>Tasks perf., Behaviour</td><td>-</td></tr><tr><td>[113]</td><td>CS</td><td>Letter Recall Task perf.</td><td>32 VR</td><td>Avatar types (no body, hands-only representation, articulated body and real)</td><td>Task perf., Behaviour</td><td>-</td></tr><tr><td>[143]</td><td>CS</td><td>Letter Recall Task perf.</td><td>40 VR</td><td>Self avatar presence, hands gestures allowed or not</td><td>Tasks perf., Behaviour</td><td>-</td></tr><tr><td>[91]</td><td>CS</td><td>Object Location Memory</td><td>30 VR</td><td>Rendering quality</td><td>Task perf.</td><td>-</td></tr><tr><td>[49]</td><td>CS</td><td>Object Location Memory</td><td>322 VR</td><td>Multi-sensory modalities (tactile, olfactory, auditory), visual details</td><td>Task perf.</td><td>-</td></tr></tbody></table>

<table><thead><tr><th>Ref.</th><th>ACS</th><th>Exploitation</th><th>Field</th><th>Targeted ACS</th><th>Feedback</th><th>Feedback Ref.</th><th>Feedback Triggering Event</th><th>Exp. validation</th></tr></thead><tbody><tr><td>[48] [58]</td><td>CS</td><td>Implicit</td><td>VR</td><td>Cognitive Load</td><td>Visual search task difficulty (10 levels)</td><td>EEG</td><td>Threshold (based on EEG during baseline)</td><td>yes</td></tr><tr><td>[115]</td><td>ACS</td><td>Implicit</td><td>Military</td><td>Arousal (physiological), Cognitive Workload</td><td>Following a vehicle while driving in threatening situations: Audio-visual-olfative stressors for arousal, task difficulty for cognitive workload</td><td>ECG, EDA & EOG</td><td>Thresholds (based on a normative database)</td><td>no</td></tr><tr><td>[1]</td><td>AS</td><td>Implicit</td><td>Gaming</td><td>Excitement, Frustration</td><td>Obstacle game difficulty (2 levels)</td><td>EEG</td><td>EEG excitement and frustration indexes</td><td>yes</td></tr><tr><td>[20] [21]</td><td>AS</td><td>Implicit</td><td>AI</td><td>Anger, Stress, Neutral, Happiness (2 valence x 2 arousal)</td><td>Avatar appearance: Fur (2 happiness levels), particles brightness (2 stress levels) and colour (2 anger levels)</td><td>EDA, HR (PPG)</td><td>Emotion recognition</td><td>no</td></tr><tr><td>[21]</td><td>AS</td><td>Implicit</td><td>AI</td><td>Fear</td><td>360° horror video: Scene occlusion following user's gaze direction</td><td>EDA</td><td>Real-time</td><td>no</td></tr><tr><td>[21]</td><td>AS</td><td>Explicit</td><td>Therapy</td><td>Fear</td><td>Insects spawn rate, speed, behaviour and size (3 arousal levels)</td><td>EDA</td><td>Thresholds (manually set for each participant)</td><td>no</td></tr><tr><td>[62]</td><td>AS</td><td>Explicit - BF</td><td>Gaming</td><td>Fear</td><td>Horror game difficulty (5 levels) via audio-visual fearful stimuli & field of view reduction</td><td>HR (PPG)</td><td>Thresholds (compared to HR during baseline)</td><td>yes</td></tr><tr><td>[100]</td><td>AS</td><td>Explicit - BF</td><td>Gaming</td><td>(Physiological Regulation)</td><td>Rescuing a cat game: Auditory indicators of cat's location</td><td>HR (PPG)</td><td>Threshold (at 95% of the mean HR during baseline)</td><td>no</td></tr><tr><td>[47]</td><td>AS</td><td>Explicit - PF</td><td>VR</td><td>Valence, Arousal, Dominance</td><td>Altered audio-haptic HR feedback</td><td>HR (PPG)</td><td>Real-time</td><td>yes</td></tr><tr><td>[36]</td><td>AS</td><td>Explicit - PF</td><td>VR</td><td>Valence, Arousal, Dominance</td><td>Multi-sensory HR feedback modalities</td><td>HR (PPG)</td><td>Real-time</td><td>yes</td></tr><tr><td>[46]</td><td>AS</td><td>Explicit - PF</td><td>VR</td><td>Valence</td><td>Visual feedback of another user's HR in a collaborative game</td><td>HR (PPG)</td><td>Real-time</td><td>yes</td></tr></tbody></table>
<table><caption>TABLE 3: Summary of works that recognized ACS in IVEs.</caption><thead><tr><th>Ref.</th><th>ACS</th><th>Classification of:</th><th>Input</th><th>Algorithm</th><th>Subjects</th><th>Labeled Data</th><th>Model Evaluation</th><th>Accuracy</th></tr></thead><tbody><tr><td>[96]</td><td>AS</td><td>2 arousal levels</td><td>PPG; ECG</td><td>SVM</td><td>11</td><td>10560</td><td>Bradley's test</td><td>-</td></tr><tr><td rowspan="2">[154]</td><td rowspan="2">AS</td><td rowspan="2">3 arousal levels</td><td>EDA, RESP, EEG, ECG</td><td>SVM</td><td>18</td><td>2700</td><td>5-fold cross-validation<br>leave-1 out participant</td><td>96.90%<br>36.90%</td></tr><tr><td></td><td></td><td></td><td></td><td></td></tr><tr><td rowspan="2">[92]</td><td rowspan="2">AS</td><td>2 arousal levels</td><td>EEG, ECG</td><td>SVM</td><td>38</td><td>152</td><td>leave-5 out participants</td><td>75%</td></tr><tr><td>2 valence levels</td><td>EEG, ECG</td><td>SVM</td><td>38</td><td>152</td><td>leave-5 out participants</td><td>71.21%</td></tr><tr><td rowspan="2">[93]</td><td rowspan="2">AS</td><td>2 arousal levels</td><td>EEG, ECG</td><td>SVM</td><td>&lt; 30</td><td>-</td><td>leave-1 out participant</td><td>75%</td></tr><tr><td>2 valence levels</td><td>EEG, ECG</td><td>SVM</td><td>&lt; 30</td><td>-</td><td>leave-1 out participant</td><td>71.08%</td></tr><tr><td>[2]</td><td>AS</td><td>2 excitement trends</td><td>Eye movement</td><td>Deep neural network (best)</td><td>20</td><td>220</td><td>Train(70%) / Test(30%) Split</td><td>92%</td></tr><tr><td>[150]</td><td>CS</td><td>3 cognitive load levels (task levels)</td><td>EEG</td><td>Regularized LDA</td><td>15</td><td>3420</td><td>4-fold cross-validation</td><td>63.90%</td></tr><tr><td>[121]</td><td>CS</td><td>3 cognitive load levels (task levels)</td><td>fNIRS</td><td>Shrinkage LDA</td><td>10</td><td>220</td><td>10-fold cross-validation</td><td>42%</td></tr><tr><td rowspan="3">[39]</td><td rowspan="3">CS</td><td>3 cognitive load levels (task levels)</td><td>EDA</td><td>Random Forest (best one)</td><td>24</td><td>345</td><td>10-fold cross-validation</td><td>50.83%</td></tr><tr><td>3 cognitive load levels</td><td>HR</td><td>Random Forest</td><td>24</td><td>345</td><td>10-fold cross-validation</td><td>91.71%</td></tr><tr><td>2 moments of insight</td><td>EDA</td><td>Random Forest</td><td>13</td><td>43</td><td>10-fold cross-validation</td><td>83.65%</td></tr></tbody></table>

### 5.2.1 Classifiers

There is no consensus over which **ML algorithm** to use depending on the targeted AS or CS and the used sensors. Support Vector Machines (SVM) and Linear Regression Analysis (LDA) are popular methods that have been used for years in the ML community. However, the random forest algorithm also has shown great results in recent years [2, 39, 52]. For example, Collins et al. used Principal Component Analysis (PCA) and SVM with linear basis function, MultiLayer Perception, K-nearest neighbours (k=1), J48 Decision Tree, and Random Forest (100 trees) to classify 3 cognitive load levels and 2 moments of insight [39]. They found random forest to perform the best among the different classifiers. As for the models evaluation, it should be noted that most studies used a user-dependent approach (by using k-fold cross-validation, or leave-n out participant after having pooled all participants data together), meaning that these models require to be trained using the user's data to reach similar classification performances.

ML models training require to **label** the dataset for the classification of the user's AS or CS. As such, Mavridou et al. [95] and Marín-Morales et al. [92, 93] used self-report measures to label their physiological dataset. However, the user's psychological state was sometimes inferred in other studies. For example, Abdessalem et al. extracted the excitement trend (increase or decrease) based on the excitement index given by the Emotiv EEG device, which algorithm is currently not public [2]. In CS studies, task difficulty levels were often used to label the physiological dataset to recognize cognitive load and mental workload. As such, Tremmel et al. and Putze et al. both used the difficulty levels of the n-back task to label their dataset [121, 150]. Similarly, Collins et al. extracted 3 difficulty levels of a spatial rotation task based on the users' overall performances. Using another approach, the users' arousal level was identified based on the user's performance (response time) and the Yerkes-Dodson Law [155], an empirical law between cognitive performances and arousal in [154].

### 5.2.2 Physiological input

Concerning the sensors used in the classification of AS, cardiovascular sensors [92, 93, 95, 154], EDA sensors [154], RESP

sensors [154] and EEG [92, 93, 154] have proved to contribute to the classification of arousal and valence. Wu et al. classified 3 arousal levels using EDA, RESP, EEG, ECG measures and SVM [154]. They ordered the physiological signals considering their importance in the classification performance. EDA and RESP features were found to be the most important, EEG features were moderately important (especially θ waves), followed by ECG features (interbeat intervals features, then HR). Using EEG and ECG features, Marín-Morales et al. found that EEG features were more important than ECG features for both the classification of arousal and valence [92]. Their results are in agreement with Wu et al. [154]'s ones as they found that EEG θ waves played essentially a great role in the classification of emotions [92]. In another study, Marín-Morales et al. also compared the classification of the 2 emotional dimensions in a virtual vs. in a physical museum environment and found out the classifier for the virtual environment needed fewer features than the classifier for the physical environment [93]. They also tested the classification of emotional valence and arousal using a "baseline" which induced a range of the emotions they targeted, using the IAPS, and the 4 emotional VEs used in the previous study [92] (see Fig. 5). By concatenating these baseline features to the tested condition features (i.e., features map), they improved the classification in both the physical (using the IAPS features) and virtual (using the 4 emotional VEs features) setups. Eye movements were also used to recognize the excitement trend (increase or decrease) in VR [2].

For CS, EEG [150], fNIRS [121], HR [39] and EDA [39] have been explored to classify different cognitive load levels in IVEs. In contrast to the recognition of emotions for which EEG θ waves were especially important [92, 154], Tremmel et al. found that the most consistent EEG features signals across participants to discriminate the cognitive load levels were the frontal β and γ signals [150]. Using the sensors separately, Collins et al. results showed greater cognitive load classification performances using HR than EDA features [39] (i.e., 91.71% vs. 50.83% in the best classification setup). Their classification of 2 moments of insight however reached a high accuracy using EDA features (i.e., 83.65% in the best setup).
To conclude, previous results in these studies are encouraging for the use of physiological computing in VR. However, most of them were presented using a user-dependent approach, which depicts a true challenge for the generalization of the classification of ACS using physiological signals between users. While interesting, findings for the ACS recognition in VR have to be considered carefully as there is still currently little studies on this topic. No consensus exists regarding which sensors to use for the classification of ACS, and on which classifier to use, in particular, in a VR setting where users are prone to motions. The studies also used a variety of methods to label their dataset, from the users' ACS subjective measures to the task difficulty levels for CS. Also, few studies exploited their algorithms in real-time to adapt IVEs.

## 5.3 Exploitation of Affective and Cognitive States in VR

The exploitation of ACS refers to the modulation of IVEs content based on the user's ACS measure or recognition (see Fig. 1). Table 2 details studies making an exploitation of ACS in real-time. We retained 13 studies that dealt with the exploitation of ACS in IVEs. However, 2 studies presented a framework of IVEs adaptation based on users' ACS and did not propose an example of possible adaptation [39, 82], which is why they are not presented in Table 2. The methods to exploit ACS measures in real-time can be divided into 2 categories:

*   **Explicit exploitation:** the user's ACS measures are used to adjust the IVE content and parameters, to raise awareness by the users about their own ACS.

*   **Implicit exploitation:** the user's ACS is measured and used independently from the user's knowledge to adjust the IVE content and parameters.

Explicit and implicit exploitations have several corresponding denominations in the literature. For example, Nacke et al. respectively use the terms “direct” and “indirect physiological control” in the game field [101], and Gilleade et al. use the terms “straight-forward biofeedback” and “affective gaming” [59] for AS specifically.

Among explicit exploitations, a famous category of exploitation is biofeedback. In experiments relying on this type of feedback, users are given indications of their physiological state in the VE or via multi-sensory stimuli and they are encouraged to control their physiological state through these feedbacks. This is a type of exploitation which was particularly explored in VR therapy (e.g., [25]) and in 2 studies retained in this survey (i.e., [62, 100]). Another subcategory of explicit exploitations is physiological feedback. It corresponds to the direct display of users' physiological states to themselves in the IVE interface or to a third party. In this category, users are not particularly encouraged to control their physiological state and the feedback is mostly informative [36, 46, 47].

The target of the adaptation (i.e., the adjusted IVE parameter or content) can take various forms, depending on the targeted ACS and the study objective. As such, task difficulty was often adapted to control cognitive load [48, 115], improve game experience [1], or invite users to control their physiological signals in a horror game (i.e., a biofeedback case) [62]. Another classical category of feedback consists in audio-visual stimuli such as stressors (e.g., explosive device blasts [115], insects spawning [21], music box activation frequency [62], scene occlusion [21, 62]) to control

user's arousal level and fear level. In Muñoz et al.'s study, they used auditory feedback to indicate the cat's location in a rescuing cat game based on the user's physiological signals to encourage users to control their physiological state [100]. The change of avatars appearance [20] and the display of the physiological signals in the user interface [36, 46, 47] was also explored to communicate the users' states to themselves or to a third party. Other types of feedback such as olfactory ones or haptic ones were also used to enhance arousal stimuli [115] or as informative indications of the user's physiological state [36, 47].

As for the events used to trigger changes in the IVE content and parameters, ACS recognition was often set as a target method in previous frameworks [23, 26, 39, 82]. However, as seen in Table 2, only two protocols [1, 20] were proposed to adapt VR parameters based on the recognition of ACS using physiological signals and ML methods. Nonetheless, it was not validated through an experimental protocol in the case of Bernal and Maes [20]'s study, and the classification of emotions was not presented in the paper. Abdessalem et al. [1]'s used the excitement and frustration indexes from the Emotiv EEG device as a reference to trigger changes in the game difficulty, which information about the algorithm is still not yet published. This lack of studies doing an adaptation based on ACS recognition can be explained by the difficulty to recognize users' ACS based on physiological signals, as underlined in Section 5.2. To palliate this issue, other studies focused on determining thresholds based on physiological signals measured during the baseline [48, 62, 100], compared to a normative database [115], or in a custom way per participant [21] to trigger changes in the IVEs. On the other hand, some studies directly projected the users' physiological signals to them [36, 47] or to a collaborator [46] in an explicit way. Bernal et al. also directly used the user's EDA responses in real-time to continuously control the scene occlusion in a horror video [21].

To summarize, some studies adapted IVEs content based on ACS measures knowingly (i.e., explicit exploitation) or unknowingly to users (i.e., implicit exploitation). These adaptations take various forms depending on the targeted changes in users' ACS, from tasks difficulty to haptic and olfactory stimuli. However, few of these studies used ACS recognition as a reference to perform the adaptation, while proposed in many frameworks [23, 26, 39, 82]. Most triggered changes in the IVEs based on physiological signals using a predefined threshold or directly in real-time.

# 6 DISCUSSION

In this survey, the most common definitions, models, and standard methods to measure ACS in VR were first presented. Then, previous Affective and Cognitive VR studies in IVEs in a non-clinical context were developed following (1) the induction of ACS, (2) the recognition of ACS, and (3) the exploitation of ACS.

AS and CS definitions and models are still the center of debates, which makes the centralization of data and results difficult. This can be explained by the fact that models are mainly theoretical, and hard to prove with the current tools. Furthermore, the definitions and models chosen in IVEs studies are rarely depicted, which can lead to confusions. For example, the term “cognitive load” refers most of the time to the concept of mental workload in IVEs and designates another cognitive phenomenon in the psychological field. Thus, we encourage VR studies to clarify the definitions of the targeted AS or CS early in their paper. Emotions (from a dimensional and a discrete perspectives) and
anxiety are the AS that were the most studied in VR. Fewer studies dealt with purely cognitive tasks and CS in an IVE, but recent studies show a growing interest in CS such as mental workload.

While there are some distinctions in their definitions and models, AS and CS are closely linked in the methods to measure them [12, 103, 154]. Based on previous categorizations [66, 104], we proposed to divide the methods to measure ACS into 4 groups: self-report, observational, performance and psychophysiological methods. These methods are not exclusive to IVEs, but the VR context involves some adaptations and care compared to usual methods because of the cumbersomeness of VR equipment, the number of motions, and cybersickness (see Section 4). In this survey, most Affective VR studies relied on self-report and psychophysiological measures, and most Cognitive VR studies relied on task performance and observational measures (see Table 1). There is however a trend in recent years as more Cognitive VR studies tend to use psychophysiological signals [39, 121, 150]. Overall, self-report methods have the advantage to show high-face validity but can disrupt the immersion; and observational, performance and psychophysiological methods have the advantage to be continuous, but can be context-dependent and complex to analyze.

In this survey, we retained 63 studies that induced, recognized, and/or exploited ACS in IVEs. Starting with the methods of **induction of ACS**, various IVEs variables were found to influence healthy users' ACS, despite individuals' differences (see Table 1). We gathered the stimuli used in past studies in 2 categories: (a) studies that aimed to induce different ACS or different levels of ACS, and (b) studies that aimed to study the effect of a parameter on an AS or a CS. As such, in the first category, works mainly used audio-visual stimuli through emotional videos, emotional VEs, emotional scenarios, and cognitive tasks. Among methods that can be considered as standardized, Li et al. developed a database of 73 public immersive 360 panoramic videos to induce different emotions along the valence and arousal dimensions [79]. In another study, Mavridou et al. used 20 videos from an affective film library in an immersive virtual cinema to induce valence (2 levels) and arousal (2 levels) responses [96]. The virtual pit experience and virtual public speaking situations are also classical experiments that were found to induce increases in state-anxiety and physiological responses [98, 140]. Otherwise, other studies changed a variety of audio-visual contents in IVEs applications to induce different AS. For the induction of different levels of CS, studies mainly relied on standardized single task difficulty levels variation such as the n-back task [86, 121, 150], spatial rotation task [39], visual search task [48, 58] difficulties, or on multitasking [87, 114]. Cognitive tasks also were found to increase the user's arousal level [6, 114, 154]. These studies demonstrated the fact VR can be a strong ACS inducer. In the second category, other factors such as the type of display, the level of immersion, avatars, personality traits, redirected walking, and multi-sensory feedback was found to influence ACS.

As for the **recognition of ACS**, IVEs studies mainly relied on physiological signals and ML algorithms (see Table 3). The passive recognition of ACS is promising given the growing interest and recent performances of ML algorithms but can be tricky because of individual differences, the fact responses can be context-dependent, and because of the necessity of having large labelled datasets. For now, models of recognition of ACS

in IVEs have shown great performances (see Table 3). However, they were mostly evaluated following user-dependent approaches. The recognition performance of user-independent models (i.e., the generalization of recognition on unseen participants) was rarely explored. The field could benefit from large datasets (e.g., ImageNet for object recognition using deep learning), which can be difficult to acquire given the anonymity issues, the setup time and expenses required by VR and physiological equipment. In conclusion, previous results in the recognition models of ACS in IVEs are hardly generic but pave the way for future works. The recognition of users' ACS could allow the automatic adaptation of IVEs based on the user's ACS without the need to disrupt the immersion by asking users to self-report their mental state.

Two types of **exploitation of ACS** were identified from existing affective and cognitive VR studies in the scope of this survey: (1) explicit exploitation and (3) implicit exploitation. Among the few studies which presented an adaptation of IVEs parameters based on users' ACS, half validated their methodology with an experimental protocol (see Table 2). Most of these studies have set physiological thresholds as reference events to trigger changes in IVEs content based on the user physiological signals. Despite the presentation of frameworks that integrate the recognition of ACS to trigger changes in VR content [39, 82], only two proposed an adaptation based on ACS recognition: one without experimental validation [20], and one, based on Emotiv excitement index [1]. This can be explained by the difficulty to recognize ACS and interpret ACS responses.

# 7 RESEARCH PERSPECTIVES

Those studies demonstrate the will to build more user-centered IVE applications by taking into account the user's ACS. Most works focused on the induction of ACS, followed by a few works in the exploitation of ACS, and finally the recognition of ACS. This can be explained by the fact ACS was mostly used as an offline metrics as a way to evaluate the user experience. The recognition of ACS can be complex to achieve as studies rely on Supervised ML, which usually require a high quantity of data and complex signal processing for physiological data. As for the exploitation of ACS, there are a few more studies than in the recognition of ACS as the feedback can be based on the features of the physiological signals directly.

Most studies used a combination of many factors such as light, colours, characters appearance and behaviour, and music to induce multiple emotions or emotions variations. The effect of each of these factors individually is not clearly understood, which could be interesting to explore. Moreover, only emotional panoramic videos were used to develop standardized stimuli databases [79, 96] for emotional valence and arousal. Interactivity is an important component of VR [144], and certain interactions were already found to impact the user's ACS [11, 109]. Thus, developing standardized methods which introduce interactions with 3D entities to induce different ACS could be interesting to fully exploit VR potential and engagement power. Also, some factors linked to users' profiles were found to have a significant effect on the stimuli influence [5, 55, 117]. More works studying the effect of personality traits and cognitive abilities on the impact of ACS stimuli could help understanding how to make VR application more adapted to users.

Indeed, individual differences are one of the main issues in the recognition of ACS. Studying ways to normalize physiological
data and subjective responses could help make the recognition models more generic. There is also no work yet targeting the genericity challenge, demonstrating the performance of a recognition model on unseen participants in a different context than in the one the model was trained. Another problem is that studies recognizing CS mainly used indirect methods to label their datasets, such as tasks difficulty levels and tasks performances data. A task can induce different mental workload levels to the users. In the same way, different users showing similar task performances might experience different mental workload levels as seen in [48]. Overall, we would suggest the use of self-report methods for the labelling of datasets as they show a higher face validity than other metrics (see Section 4).

As for the exploitation of ACS, there is currently a lack of experimental validations of the proposed exploitation. There is also no proposition of adaptation models based on ACS in IVEs. Future works could focus on presenting new adaptive models, to determine what type of content should be adapted based on the nature of the detected ACS, on demonstrating them, and on studying their impact on users in IVEs. Furthermore, the implicit exploitation of ACS based on the recognition of ACS was very little explored as the recognition of ACS tend to be complex. This is an important path for future work to build more user-centered applications as taking into account the user AS and CS is going to provide more robust and adapted immersive experiences to the users' individual differences.

## 8 CONCLUSION

In this paper, we defined Affective and Cognitive VR to relate to works that (1) induce affective and cognitive states (ACS), (2) recognize ACS, or (3) exploit ACS in VR. First, the different definitions, models, and standard methods to measure ACS in VR were introduced and discussed. Then, we presented a survey of previous Affective and Cognitive VR works in Immersive Virtual Environments (IVEs) in a non-clinical context. Methods that induced ACS were largely explored in the literature. Those were categorized into several groups, mainly studies that aimed to induce different ACS or several ACS levels, and studies that focused on the influence of a parameter on an AS or a CS. As for the recognition of ACS, methods mainly used supervised machine learning and physiological signals. Previous work results are promising but are hardly generic. Finally, the exploitation of ACS can be divided into 2 groups: the modulation of IVEs knowingly to users (i.e., explicit exploitation) or unknowingly to users (i.e., implicit exploitation). Most works used physiological signals features directly or through thresholds to trigger changes in the IVEs and only half validated their adaptation proposition with an experimental protocol. These studies pave the way for future Affective and Cognitive VR work, for which we provide different research perspectives.

## REFERENCES

[1] Hamdi Ben Abdessalem, Marwa Boukadida, and Claude Frasson. Virtual reality game adaptation using neurofeedback. In *The 31st international flairs conference*, 2018.

[2] Hamdi Ben Abdessalem, Maher Chaouachi, Marwa Boukadida, and Claude Frasson. Toward Real-Time System Adaptation Using Excitement Detection from Eye Tracking. In *International Conference on Intelligent Tutoring Systems*, pages 214–223. Springer, 2019.

[3] Soraia M Alarcao and Manuel J Fonseca. Emotions recognition using eeg signals: A survey. *IEEE Transactions on Affective Computing*, 10(3):374–393, 2017.

[4] Mariano Alcañiz, Rosa Baños, Cristina Botella, and Beatriz Rey. The EMMA project: emotions as a determinant of presence. *PsychNology Journal*, 1(2):141–150, 2003.

[5] Ivan Alsina-Jurnet and José Gutiérrez-Maldonado. Influence of personality and individual abilities on the sense of presence experienced in anxiety triggering virtual environments. *International journal of human-computer studies*, 68(10):788–801, 2010.

[6] Allison Anderson, Michael Mayer, Abigail Fellows, Devin Cowan, Mark Hegel, and Jay Buckey. Relaxation with Immersive Natural Scenes Presented Using Virtual Reality. *Aerospace Medicine and Human Performance*, 88:520–526, 2017. doi: 10.3357/AMHP.4747.2017.

[7] Pavlo Antonenko, Fred Paas, Roland Grabner, and Tamara Van Gog. Using electroencephalography to measure cognitive load. *Educ. Psych. Rev.*, 22(4):425–438, 2010.

[8] Ferran Argelaguet Sanz, Franck Multon, and Anatole Lécuyer. A methodology for introducing competitive anxiety and pressure in VR sports training. *Frontiers in Robotics and AI*, 2:10, 2015.

[9] Hasan Ayaz, Patricia A Shewokis, Scott Bunce, Maria Schultheis, and Banu Onaral. Assessment of cognitive neural correlates for a functional near infrared-based brain computer interface system. In *Int. Conf. on Foundations of Augmented Cognition*, pages 699–708. Springer, 2009.

[10] Hasan Ayaz, Patricia A Shewokis, Scott Bunce, Kurtulus Izzetoglu, Ben Willems, and Banu Onaral. Optical brain monitoring for operator training and mental workload assessment. *Neuroimage*, 59(1):36–47, 2012.

[11] Laura Aymerich-Franch. Presence and emotions in playing a group game in a virtual environment: the influence of body participation. *Cyberpsychology, Behavior, and Social Networking*, 13(6):649–654, 2010.

[12] Ryan SJd Baker, Sidney K D'Mello, Ma Mercedes T Rodrigo, and Arthur C Graesser. Better to be frustrated than bored: The incidence, persistence, and impact of learners' cognitive-affective states during interactions with three different computer-based learning environments. *International Journal of Human-Computer Studies*, 68(4):223–241, 2010.

[13] Domna Banakou, Sameer Kishore, and Mel Slater. Virtually being einstein results in an improvement in cognitive task performance and a decrease in age bias. *Frontiers in psychology*, 9:917, 2018.

[14] Rosa Baños, Cristina Botella, Isabel Rubió, Soledad Quero, Azucena García-Palacios, and Mariano Alcañiz. Presence and emotions in virtual environments: The influence of stereoscopy. *CyberPsychology & Behavior*, 11(1):1–8, 2008.

[15] Rosa María Baños, Cristina Botella, Mariano Alcañiz, Víctor Liaño, Belén Guerrero, and Beatriz Rey. Immersion and emotion: their impact on the sense of presence. *CyberPsychology & Behavior*, 7(6):734–741, 2004.

[16] Rosa María Baños, Víctor Liaño, Cristina Botella, Mariano Alcañiz, Belén Guerrero, and Beatriz Rey. Changing induced moods via virtual reality. In *International Conference on Persuasive Technology*, pages 7–15. Springer, 2006.

[17] Lisa Feldman Barrett. *How emotions are made: The secret life of the brain*. Houghton Mifflin Harcourt, 2017.
[18] Christopher Beedie, Peter Terry, and Andrew Lane. Distinctions between emotion and mood. *Cognition & Emotion*, 19 (6):847–878, 2005.

[19] Ilias Bergström, Konstantina Kilteni, and Mel Slater. First-person perspective virtual body posture influences stress: a virtual reality body ownership study. *PloS one*, 11(2), 2016.

[20] Guillermo Bernal and Pattie Maes. Emotional Beasts: Visually Expressing Emotions through Avatars in VR. In *ACM SIGCHI Conference on Human Factors in Computing Systems, Extended Abstracts*, pages 2395–2402, 2017.

[21] Guillermo Bernal, Tao Yang, Abhinandan Jain, and Pattie Maes. Physiohmd: a conformable, modular toolkit for collecting physiological data from head-mounted displays. In *Proceedings of the 2018 ACM International Symposium on Wearable Computers*, pages 160–167, 2018.

[22] Kyle A Bernhardt, Dmitri Poltavski, Thomas Petros, F Richard Ferraro, Terra Jorgenson, Craig Carlson, Paul Drechsel, and Colt Iseminger. The effects of dynamic workload and experience on commercially available eeg cognitive state metrics in a high-fidelity air traffic control environment. *Applied ergonomics*, 77:83–91, 2019.

[23] Daniel Bersak, Gary McDarby, Ned Augenblick, Phil McDarby, Daragh McDonnell, Brian McDonald, and Rahul Karkun. Intelligent biofeedback using an immersive competitive environment. Paper at the Designing Ubiquitous Computing Games Workshop at UbiComp, 2001.

[24] Yulong Bian, Chenglei Yang, Fengqiang Gao, Huiyu Li, Shisheng Zhou, Hanchao Li, Xiaowen Sun, and Xiangxu Meng. A framework for physiological indicators of flow in VR games: construction and preliminary evaluation. *Personal and Ubiquitous Computing*, 20(5):821–832, 2016.

[25] E Bisson, Bruce Contant, Heidi Sveistrup, and Yves Lajoie. Functional balance and dual-task reaction times in older adults are improved by virtual reality and biofeedback training. *Cyberpsychology & behavior*, 10(1):16–23, 2007.

[26] Boyan Bontchev. Adaptation in affective video games: A literature review. *Cybernetics and Information Technologies*, 16(3):3–34, 2016.

[27] Cristina Botella, Javier Fernández-Álvarez, Verónica Guillén, Azucena García-Palacios, and Rosa Baños. Recent progress in virtual reality exposure therapy for phobias: a systematic review. *Current psychiatry reports*, 19(7):42, 2017.

[28] Wolfram Boucsein. *Electrodermal activity*. Springer Science & Business Media, 2012.

[29] Pierre Bourdin, Itxaso Barberia, Ramon Oliva, and Mel Slater. A virtual out-of-body experience reduces fear of death. *PloS one*, 12(1), 2017.

[30] Margaret Bradley and Peter J Lang. *The International affective digitized sounds (IADS)!: stimuli, instruction manual and affective ratings*. NIMH Center for the Study of Emotion and Attention, 1999.

[31] Margaret M Bradley and Peter J Lang. Measuring emotion: the self-assessment manikin and the semantic differential. *Journal of behavior therapy and experimental psychiatry*, 25(1):49–59, 1994.

[32] Gerd Bruder, Paul Lubas, and Frank Steinicke. Cognitive resource demands of redirected walking. *IEEE Transactions on Visualization and Computer Graphics*, 21(4):539–544, 2015.

[33] Brad Cain. A review of the mental workload literature.

Technical report, Defence Research And Development Toronto (Canada), 2007.

[34] Berk Cebeci, Ufuk Celikcan, and Tolga K Capin. A comprehensive study of the affective and physiological responses induced by dynamic virtual reality environments. *Computer Animation and Virtual Worlds*, 30(3-4):e1893, 2019.

[35] Marcus Cheetham, Andreas Pedroni, Angus Antley, Mel Slater, and Lutz Jäncke. Virtual milgram: empathic concern or personal distress? evidence from functional mri and dispositional measures. *Frontiers in human neuroscience*, 3:29, 2009.

[36] Hao Chen, Arindam Dey, Mark Billinghurst, and Robert W Lindeman. Exploring the design space for multi-sensory heart rate feedback in immersive virtual reality. In *Proceedings of the 29th Australian Conference on Computer-Human Interaction*, pages 108–116. ACM, 2017.

[37] Hao Chen, Arindam Dey, Mark Billinghurst, and Robert W Lindeman. Exploring pupil dilation in emotional virtual reality environments. In *Proceedings of the 27th International Conference on Artificial Reality and Telexistence and 22nd Eurographics Symposium on Virtual Environments*, pages 169–176, 2017.

[38] Luca Chittaro and Fabio Buttussi. Assessing knowledge retention of an immersive serious game vs. a traditional education method in aviation safety. *IEEE Transactions on Visualization and Computer Graphics*, 21(4):529–538, 2015.

[39] Jonny Collins, Holger Regenbrecht, Tobias Langlotz, Yekta Said Can, Cem Ersoy, and Russell Butson. Measuring Cognitive Load and Insight: A Methodology Exemplified in a Virtual Reality Learning Context. In *IEEE International Symposium on Mixed and Augmented Reality*, pages 351–362, 2019.

[40] Melissa Patricia Coral. Analyzing cognitive workload through eye-related measurements: A meta-analysis. 2016.

[41] Anders M Dale and Martin I Sereno. Improved localizadon of cortical activity by combining EEG and MEG with MRI cortical surface reconstruction: a linear approach. *Journal of cognitive neuroscience*, 5(2):162–176, 1993.

[42] Dick De Waard. *The measurement of drivers' mental workload*. Groningen University, Traffic Research Center Netherlands, 1996.

[43] Andreas Dengel and Jutta Mägdefrau. Presence Is the Key to Understanding Immersive Learning. In *Int. Conf. on Immersive Learning*, pages 185–198. Springer, 2019.

[44] Mark S Dennison, A Zachary Wisti, and Michael D'Zmura. Use of physiological signals to predict cybersickness. *Displays*, 44:42–52, 2016.

[45] Pieter Desmet. *Designing emotions*. Delft University of Technology, Department of Industrial Design, 2002.

[46] Arindam Dey, Thammathip Piumsomboon, Youngho Lee, and Mark Billinghurst. Effects of Sharing Physiological States of Players in a Collaborative Virtual Reality Game-play. In *ACM SIGCHI Conference on Human Factors in Computing Systems*, pages 4045–4056, 2017.

[47] Arindam Dey, Hao Chen, Mark Billinghurst, and Robert W Lindeman. Effects of manipulating physiological feedback in immersive virtual environments. In *Proceedings of the 2018 Annual Symposium on Computer-Human Interaction in Play*, pages 101–111, 2018.
[48] Arindam Dey, Alex Chatourn, and Mark Billinghurst. Exploration of an eeg-based cognitively adaptive training system in virtual reality. In *IEEE Conference on Virtual Reality and 3D User Interfaces*, pages 220–226, 2019.

[49] Huong Q Dinh, Neff Walker, Larry F Hodges, Chang Song, and Akira Kobayashi. Evaluating the importance of multisensory input on memory and the sense of presence in virtual environments. In *IEEE Virtual Reality*, pages 222–228, 1999.

[50] Maria K Eckstein, Belén Guerra-Carrillo, Alison T Miller Singley, and Silvia A Bunge. Beyond eye gaze: What else can eyetracking reveal about cognition and cognitive development? *Developmental cognitive neuroscience*, 25: 69–91, 2017.

[51] Paul Ekman, E Richard Sorenson, and Wallace V Friesen. Pan-cultural elements in facial displays of emotion. *Science*, 164(3875):86–88, 1969.

[52] Colin Elkin and Vijay Devabhaktuni. Comparative Analysis of Machine Learning Techniques in Assessing Cognitive Workload. In Hasan Ayaz, editor, *Advances in Neuroergonomics and Cognitive Engineering*, pages 185–195, Cham, 2020. Springer. ISBN 978-3-030-20473-0.

[53] Stephen H Fairclough. Fundamentals of physiological computing. *Interacting with computers*, 21(1-2):133–145, 2008.

[54] Stephen H Fairclough, Louise Venables, and Andrew Tattersall. The influence of task demand and learning on the psychophysiological response. *International Journal of Psychophysiology*, 56(2):171–184, 2005.

[55] Anna Felnhofer, Oswald D Kothgassner, Thomas Hetterle, Leon Beutl, Helmut Hlavacs, and Ilse Kryspin-Exner. Afraid to be there? Evaluating the relation between presence, self-reported anxiety, and heart rate in a virtual public speaking task. *Cyberpsychology, Behavior, and Social Networking*, 17(5):310–316, 2014.

[56] Anna Felnhofer, Oswald D Kothgassner, Mareike Schmidt, Anna-Katharina Heinzle, Leon Beutl, Helmut Hlavacs, and Ilse Kryspin-Exner. Is virtual reality emotionally arousing? Investigating five emotion inducing virtual park scenarios. *International journal of human-computer studies*, 82:48–56, 2015.

[57] Edith Galy, Magali Cariou, and Claudine Mélan. What is the relationship between mental workload factors and cognitive load types? *International Journal of Psychophysiology*, 83(3):269–275, 2012.

[58] Lynda Gerry, Barrett Ens, Adam Drogemuller, Bruce Thomas, and Mark Billinghurst. Levity: A virtual reality system that responds to cognitive load. In *ACM SIGCHI Conference on Human Factors in Computing Systems, Extended Abstracts*, pages 1–6, 2018.

[59] Kiel Gilleade, Alan Dix, and Jen Allanson. Affective videogames and modes of affective gaming: assist me, challenge me, emote me. *DiGRA 2005: Changing Views–Worlds in Play*, 2005.

[60] Daniel Gromer, Max Reinke, Isabel Christner, and Paul Pauli. Causal interactive links between presence and fear in virtual reality height exposure. *Frontiers in psychology*, 10: 141, 2019.

[61] Sandra G Hart and Lowell E Staveland. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In *Advances in psychology*, volume 52,

pages 139–183. Elsevier, 1988.

[62] Samory Houzangbe, Olivier Christmann, Geoffrey Gorisse, and Simon Richir. Fear as a biofeedback game mechanic in virtual reality: Effects on engagement and perceived usability. In *Proceedings of the 13th International Conference on the Foundations of Digital Games*, pages 1–6, 2018.

[63] Isabelle Hupont, Sandra Baldassarri, and Eva Cerezo. Facial emotional classification: from a discrete perspective to a continuous emotional space. *Pattern Analysis and Applications*, 16(1):41–54, 2013.

[64] Rachael E Jack. Culture and facial expressions of emotion. *Visual Cognition*, 21(9-10):1248–1286, 2013.

[65] Roy S Kalawsky. Exploiting virtual reality techniques in education and training: Technological issues. http://www.man.ac.uk/MVC/SIMA/vrtech/title.html, 1996.

[66] Seth Kaplan, Reeshad Dalal, and Joseph Luchman. Measurement of emotions. pages 61–75, 2013.

[67] Keisuke Kawahara, Mose Sakashita, Amy Koike, Ippei Suzuki, Kenta Suzuki, and Yoichi Ochiai. Transformed human presence for puppetry. In *Proceedings of the 13th International Conference on Advances in Computer Entertainment Technology*, pages 1–6, 2016.

[68] Theodore D Kemper. How many emotions are there? Wedding the social and the autonomic components. *American journal of Sociology*, 93(2):263–289, 1987.

[69] Young Youn Kim, Hyun Ju Kim, Eun Nam Kim, Hee Dong Ko, and Hyun Taek Kim. Characteristic changes in the physiological components of cybersickness. *Psychophysiology*, 42(5):616–625, 2005.

[70] Alexandra Kitson, Mirjana Prpa, and Bernhard E Riecke. Immersive interactive technologies for positive change: a scoping review and design considerations. *Frontiers in psychology*, 9:1354, 2018.

[71] J Matias Kivikangas, Guillaume Chanel, Ben Cowley, Inger Ekman, Mikko Salminen, Simo Järvelä, and Niklas Ravaja. A review of the use of psychophysiological methods in game research. *Journal of Gaming & Virtual Worlds*, 3 (3):181–199, 2011.

[72] Paul R Kleinginna and Anne M Kleinginna. A categorized list of emotion definitions, with suggestions for a consensual definition. *Motivation and emotion*, 5(4):345–379, 1981.

[73] Agata Kołakowska, Agnieszka Landowska, Mariusz Szwoch, Wioleta Szwoch, and Michał R Wróbel. Modeling emotions for affect-aware applications. *Information Systems Development and Applications*, pages 55–69, 2015.

[74] Oswald D Kothgassner, Anna Felnhofer, Helmut Hlavacs, Leon Beutl, Rupert Palme, Ilse Kryspin-Exner, and Lisa M Glenk. Salivary cortisol and cardiovascular reactivity to a public speaking task in a virtual and real-life environment. *Computers in human behavior*, 62:124–135, 2016.

[75] Sotiris B Kotsiantis, I Zaharakis, and P Pintelas. Supervised machine learning: A review of classification techniques. *Emerging artificial intelligence applications in computer engineering*, 160(1):3–24, 2007.

[76] Sylvia D. Kreibig. Autonomic nervous system activity in emotion: A review. *Biological Psychology*, 84(3):394–421, 2010.

[77] Peter J Lang, Margaret M Bradley, and Bruce N Cuthbert. International affective picture system (IAPS): Technical manual and affective ratings. *NIMH Center for the Study of
*Emotion and Attention*, pages 39–58, 1997.

[78] Howard Leventhal and Klaus Scherer. The relationship of emotion to cognition: A functional approach to a semantic controversy. *Cognition and emotion*, 1(1):3–28, 1987.

[79] Benjamin J Li, Jeremy N Bailenson, Adam Pines, Walter J Greenleaf, and Leanne M Williams. A public database of immersive vr videos with corresponding ratings of arousal, valence, and correlations between head movements and self report measures. *Frontiers in psychology*, 8:2116, 2017.

[80] Hao Li, Laura Trutoiu, Kyle Olszewski, Lingyu Wei, Tristan Trutna, Pei-Lun Hsieh, Aaron Nicholls, and Chongyang Ma. Facial performance sensing head-mounted display. *ACM Transactions on Graphics*, 34(4):1–9, 2015.

[81] Richard Li and Gabriel Reyes. Buccal: low-cost cheek sensing for inferring continuous jaw motion in mobile virtual reality. In *Proceedings of the 2018 ACM International Symposium on Wearable Computers*, pages 180–183, 2018.

[82] Yi Li, Adel S Elmaghraby, Ayman El-Baz, and Estate M Sokhadze. Using physiological signal analysis to design affective VR games. In *IEEE Int. Symp. on Signal Processing and Information Technology*, pages 57–62, 2015.

[83] Yun Ling, Willem-Paul Brinkman, Harold T Nefs, Chao Qu, and Ingrid Heynderickx. Effects of stereoscopic viewing on presence, anxiety, and cybersickness in a virtual reality environment for public speaking. *Presence: Teleoperators and Virtual Environments*, 21(3):254–267, 2012.

[84] Changchun Liu, Karla Conn, Nilanjan Sarkar, and Wendy Stone. Online affect detection and robot behavior adaptation for intervention of children with autism. *IEEE Transactions on Robotics*, 24(4):883–896, 2008.

[85] Changchun Liu, Pramila Agrawal, Nilanjan Sarkar, and Shuo Chen. Dynamic difficulty adjustment in computer games through real-time anxiety-based affective feedback. *International Journal of Human-Computer Interaction*, 25 (6):506–529, 2009.

[86] Tiffany Luong, Nicolas Martin, Ferran Argelaguet, and Anatole Lécuyer. Studying the Mental Effort in Virtual Versus Real Environments. In *IEEE Conference on Virtual Reality and 3D User Interfaces*, pages 809–816, 2019.

[87] Tiffany Luong, Ferran Argelaguet, Nicolas Martin, and Anatole Lecuyer. Introducing mental workload assessment for the design of virtual reality training scenarios. In *IEEE Conference on Virtual Reality and 3D User Interfaces*, pages 662–671, 2020.

[88] Mary F Macedonio, Thomas D Parsons, Raymond A Diguseppe, Brenda A Weiderhold, and Albert A Rizzo. Immersiveness and physiological arousal within panoramic video-based virtual reality. *Cyberpsychology & Behavior*, 10(4):508–515, 2007.

[89] Andrew MacQuarrie and Anthony Steed. Cinematic virtual reality: Evaluating the effect of display type on the viewing experience for panoramic video. In *IEEE Virtual Reality*, pages 45–54, 2017.

[90] Eric Malbos, Ronald M Rapee, and Manolya Kavakli. Behavioral presence test in threatening virtual environments. *Presence: Teleoperators and Virtual Environments*, 21(3): 268–280, 2012.

[91] Katerina Mania, Dave Wooldridge, Matthew Coxon, and Andrew Robinson. The effect of visual and interaction fidelity on spatial cognition in immersive virtual environments. *IEEE Transactions on Visualization and Computer Graphics*, 12(3):396–404, 2006.

[92] Javier Marín-Morales, Juan Luis Higuera-Trujillo, Alberto Greco, Jaime Guixeres, Carmen Llinares, Enzo Pasquale Scilingo, Mariano Alcañiz, and Gaetano Valenza. Affective computing in virtual reality: emotion recognition from brain and heartbeat dynamics using wearable sensors. *Scientific reports*, 8(1):1–15, 2018.

[93] Javier Marín-Morales, Juan Luis Higuera-Trujillo, Alberto Greco, Jaime Guixeres, Carmen Llinares, Claudio Gentili, Enzo Pasquale Scilingo, Mariano Alcañiz, and Gaetano Valenza. Real vs. immersive-virtual emotional experience: Analysis of psycho-physiological patterns in a free exploration of an art museum. *PloS one*, 14(10), 2019.

[94] Iris B Mauss and Michael D Robinson. Measures of emotion: A review. *Cognition and emotion*, 23(2):209–237, 2009.

[95] Ifigeneia Mavridou, James T McGhee, Mahyar Hamedi, Mohsen Fatoorechi, Andrew Cleal, Emili Balaguer-Balester, Ellen Seiss, Graeme Cox, and Charles Nduka. FACETEQ interface demo for emotion expression in VR. In *IEEE Virtual Reality*, pages 441–442, 2017.

[96] Ifigeneia Mavridou, Ellen Seiss, Theodoros Kostoulas, Charles Nduka, and Emili Balaguer-Ballester. Towards an effective arousal detection system for virtual reality. In *Proceedings of the Workshop on Human-Habitat for Health (H3)*, pages 1–6, 2018.

[97] Rollin McCraty and Fred Shaffer. Heart rate variability: new perspectives on physiological mechanisms, assessment of self-regulatory capacity, and health risk. *Global Advances in Health and Medicine*, 4(1):46–61, 2015.

[98] Michael Meehan, Sharif Razzaque, Brent Insko, Mary Whitton, and Frederick P Brooks. Review of four studies on the use of physiological reaction as a measure of presence in stressfulvirtual environments. *Applied psychophysiology and biofeedback*, 30(3):239–258, 2005.

[99] Albert Mehrabian and James A Russell. *An approach to environmental psychology*. the MIT Press, 1974.

[100] John Edison Muñoz, Teresa Paulino, Harry Vasanth, and Karolina Baras. PhysioVR: A novel mobile virtual reality framework for physiological computing. In *IEEE 18th International Conference on e-Health Networking, Applications and Services*, pages 1–6, 2016.

[101] Lennart Erik Nacke, Michael Kalyn, Calvin Lough, and Regan Lee Mandryk. Biofeedback game design: using direct and indirect physiological control to enhance game interaction. In *ACM SIGCHI Conference on Human Factors in Computing Systems*, pages 103–112, 2011.

[102] Fatemeh Noroozi, Ciprian Adrian Corneanu, Dorota Kamińska, Tomasz Sapiński, Sergio Escalera, and Gholamreza Anbarjafari. Survey on emotional body gesture recognition. *arXiv preprint arXiv:1801.07481*, 2018.

[103] Domen Novak, Matjaž Mihelj, and Marko Munih. A survey of methods for data fusion and system adaptation using autonomic nervous system responses in physiological computing. *Interacting with computers*, 24(3):154–172, 2012.

[104] Robert D O'Donnell. Workload assessment methodology. *Cognitive processes and performance*, 1986.

[105] Tiago Oliveira, Paulo Noriega, Francisco Rebelo, and Regina Heidrich. Evaluation of the Relationship Between Virtual Environments and Emotions. In *International Con-
*ference on Applied Human Factors and Ergonomics*, pages 71–82. Springer, 2017.

[106] Kyle Olszewski, Joseph J Lim, Shunsuke Saito, and Hao Li. High-fidelity facial and speech animation for vr hmds. *ACM Transactions on Graphics*, 35(6):1–14, 2016.

[107] Andrew Ortony and Terence J Turner. What's basic about basic emotions? *Psychological review*, 97(3):315, 1990.

[108] Sofia Adelaide Osimo, Rodrigo Pizarro, Bernhard Spanlang, and Mel Slater. Conversations between self and self as Sigmund Freud—A virtual body ownership paradigm for self counselling. *Scientific reports*, 5:13899, 2015.

[109] Hunter Osking and John A Doucette. Enhancing Emotional Effectiveness of Virtual-Reality Experiences with Voice Control Interfaces. In *International Conference on Immersive Learning*, pages 199–209. Springer, 2019.

[110] Fred Paas, Juhani E Tuovinen, Huib Tabbers, and Pascal WM Van Gerven. Cognitive load measurement as a means to advance cognitive load theory. *Educational psychologist*, 38(1):63–71, 2003.

[111] Fred GWC Paas and Jeroen JG Van Merriënboer. Instructional control of cognitive load in the training of complex cognitive tasks. *Educational psychology review*, 6(4):351–371, 1994.

[112] Federica Pallavicini, Ambra Ferrari, Alessandro Pepe, Giacomo Garcea, Andrea Zanacchi, and Fabrizia Mantovani. Effectiveness of virtual reality survival horror games for the emotional elicitation: Preliminary insights using Resident Evil 7: Biohazard. In *Int. Conf. on Universal Access in Human-Computer Interaction*, pages 87–101. Springer, 2018.

[113] Ye Pan and Anthony Steed. Avatar Type Affects Performance of Cognitive Tasks in Virtual Reality. In *25th ACM Symposium on Virtual Reality Software and Technology*, pages 1–4, 2019.

[114] Thomas D Parsons and Christopher G Courtney. Interactions between threat and executive control in a virtual reality stroop task. *IEEE Transactions on Affective Computing*, 9(1):66–75, 2016.

[115] Thomas D Parsons and James L Reinebold. Adaptive virtual environments for neuropsychological assessment in serious games. *IEEE Trans. on Consumer Electronics*, 58(2), 2012.

[116] Thomas D Parsons and Albert A Rizzo. Affective outcomes of virtual reality exposure therapy for anxiety and specific phobias: A meta-analysis. *Journal of behavior therapy and experimental psychiatry*, 39(3):250–261, 2008.

[117] David-Paul Pertaub, Mel Slater, and Chris Barker. An experiment on public speaking anxiety in response to three different types of virtual audience. *Presence: Teleoperators & Virtual Environments*, 11(1):68–78, 2002.

[118] Rosalind Wright Picard et al. Affective computing. 1995.

[119] Robert Plutchik. *The emotions*. University Press of America, 1991.

[120] Mark B Powers and Paul MG Emmelkamp. Virtual reality exposure therapy for anxiety disorders: A meta-analysis. *Journal of anxiety disorders*, 22(3):561–569, 2008.

[121] Felix Putze, Christian Herff, Christoph Tremmel, Tanja Schultz, and Dean J Krusienski. Decoding mental workload in virtual environments: a fNIRS study using an immersive n-back task. In *41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society*, pages 3103–3106, 2019.

[122] Gary B Reid and Thomas E Nygren. The subjective workload assessment technique: A scaling procedure for measuring mental workload. In *Advances in psychology*, volume 52, pages 185–218. Elsevier, 1988.

[123] Giuseppe Riva. Virtual reality in psychotherapy. *Cyberpsychology & behavior*, 8(3):220–230, 2005.

[124] Giuseppe Riva, Fabrizia Mantovani, Claret Samantha Capdeville, Alessandra Preziosa, Francesca Morganti, Daniela Villani, Andrea Gaggioli, Cristina Botella, and Mariano Alcañiz. Affective interactions using virtual reality: the link between presence and emotions. *CyberPsychology & Behavior*, 10(1):45–56, 2007.

[125] Albert “Skip” Rizzo and Gerard Jounghyun Kim. A SWOT analysis of the field of virtual reality rehabilitation and therapy. *Presence: Teleoperators & Virtual Environments*, 14(2):119–146, 2005.

[126] Patrice Robitaille and Michael J McGuffin. Increased affect-arousal in VR can be detected from faster body motion with increased heart rate. In *Proceedings of the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games*, pages 1–6, 2019.

[127] F David Rose, Barbara M Brooks, and Albert A Rizzo. Virtual reality in brain damage rehabilitation. *Cyberpsychology & behavior*, 8(3):241–262, 2005.

[128] James A Russell. A circumplex model of affect. *Journal of personality and social psychology*, 39(6):1161, 1980.

[129] Horst Ruthrof. *The body in language*. Bloomsbury Publishing, 2015.

[130] Andrea Samson, Sylvia Kreibig, Blake Soderstrom, Ayanna Wade, and James Gross. Eliciting positive, negative and mixed emotional states: A film library for affective scientists. *Cognition and Emotion*, 30(5):827–856, 2016.

[131] Klaus R Scherer. On the nature and function of emotion: A component process approach. *Approaches to emotion*, 2293:317, 1984.

[132] Klaus R Scherer. What are emotions? And how can they be measured? *Social science information*, 44(4):695–729, 2005.

[133] Louis A Schmidt and Laurel J Trainor. Frontal brain electrical activity (EEG) distinguishes valence and intensity of musical emotions. *Cognition & Emotion*, 15(4):487–500, 2001.

[134] Felix Scholkmann, Stefan Kleiser, Andreas Jaakko Metz, Raphael Zimmermann, Juan Mata Pavia, Ursula Wolf, and Martin Wolf. A review on continuous wave functional near-infrared spectroscopy and imaging instrumentation and methodology. *Neuroimage*, 85:6–27, 2014.

[135] Andreas Schwerdtfeger. Predicting autonomic reactivity to public speaking: don't get fixed on self-report data! *Int. Journal of Psychophysiology*, 52(3):217–224, 2004.

[136] Henrique Sequeira, Pascal Hot, Laetitia Silvert, and Sylvain Delplanque. Electrical autonomic correlates of emotion. *Int. journal of psychophysiology*, 71(1):50–56, 2009.

[137] Mel Slater. Measuring presence: A response to the Witmer and Singer presence questionnaire. *Presence*, 8(5):560–565, 1999.

[138] Mel Slater, Martin Usoh, and Anthony Steed. Depth of presence in virtual environments. *Presence: Teleoperators & Virtual Environments*, 3(2):130–144, 1994.

[139] Mel Slater, Andrea Brogni, and Anthony Steed. Physiological responses to breaks in presence: A pilot study. In
*Presence 2003: The 6th annual international workshop on presence*, volume 157, 2003.

[140] Mel Slater, David-Paul Pertaub, Chris Barker, and David M Clark. An experimental study on fear of public speaking using a virtual environment. *CyberPsychology & Behavior*, 9(5):627–633, 2006.

[141] Charles D Spielberger. State-trait anxiety inventory. *The Corsini encyclopedia of psychology*, pages 1–1, 2010.

[142] Charles D Spielberger, Sumner J Sydeman, Ashley E Owen, and Brian J Marsh. *Measuring anxiety and anger with the State-Trait Anxiety Inventory (STAI) and the State-Trait Anger Expression Inventory (STAXI)*. Lawrence Erlbaum Associates Publishers, 1999.

[143] Anthony Steed, Ye Pan, Fiona Zisch, and William Steptoe. The impact of a self- avatar on cognitive load in immersive virtual reality. In *IEEE Virtual Reality*, pages 67–76, 2016.

[144] Jonathan Steuer. Defining virtual reality: Dimensions determining telepresence. *Journal of communication*, 42(4): 73–93, 1992.

[145] Cheryl Stinson and Doug A Bowman. Feasibility of training athletes for high-pressure situations using virtual reality. *IEEE Transactions on Visualization and Computer Graphics*, 20(4):606–615, 2014.

[146] S Shyam Sundar, Jin Kang, and Danielle Oprean. Being There in the Midst of the Story: How Immersive Journalism Affects Our Perceptions and Cognitions. *Cyberpsychology, Behavior, and Social Networking*, 20(11):672–682, 2017.

[147] Katsuhiro Suzuki, Fumihiko Nakamura, Jiu Otsuka, Katsu-toshi Masai, Yuta Itoh, Yuta Sugiura, and Maki Sugimoto. Recognition and mapping of facial expressions to avatar by embedded photo reflective sensors in head mounted display. In *IEEE Virtual Reality*, pages 177–185, 2017.

[148] Andrew J Tattersall and Penelope S Foord. An experimental evaluation of instantaneous self-assessment as a measure of workload. *Ergonomics*, 39(5):740–748, 1996.

[149] Grace Teo, Gerald Matthews, Lauren Reinerman-Jones, and Daniel Barber. Adaptive aiding with an individualized workload model based on psychophysiological measures. *Human-Intelligent Systems Integration*, pages 1–15, 2019.

[150] Christoph Tremmel, Christian Herff, Tetsuya Sato, Krzysztof Rechowicz, Yusuke Yamani, and Dean J Krusien-ski. Estimating Cognitive Workload in an Interactive Virtual Reality Environment Using EEG. *Frontiers in Human Neuroscience*, 13, 2019.

[151] Pamela S Tsang and Velma L Velazquez. Diagnosticity and multidimensional subjective workload ratings. *Ergonomics*, 39(3):358–381, 1996.

[152] David Watson, Lee Anna Clark, and Auke Tellegen. Development and validation of brief measures of positive and negative affect: the panas scales. *Journal of personality and social psychology*, 54(6):1063, 1988.

[153] Christopher D Wickens. Multiple resources and mental workload. *Human factors*, 50(3):449–455, 2008.

[154] Dongrui Wu, Christopher G Courtney, Brent J Lance, Shrikanth S Narayanan, Michael E Dawson, Kelvin S Oie, and Thomas D Parsons. Optimal arousal identification and classification for affective computing using physiological signals: Virtual reality stroop task. *IEEE Transactions on Affective Computing*, 1(2):109–118, 2010.

[155] Robert M Yerkes and John D Dodson. The relation of strength of stimulus to rapidity of habit-formation. *Journal of comparative neurology and psychology*, 18(5):459–482, 1908.

[156] Maryam Zahabi and Ashiq Mohammed Abdul Razak. Adaptive virtual reality-based training: a systematic literature review and framework. *Virtual Reality*, pages 1–28, 2020.

[157] Clint Zeagler. Where to wear it: functional, technical, and social considerations in on-body location for wearable technology 20 years of designing for wearability. In *Proceedings of the 2017 ACM International Symposium on Wearable Computers*, pages 150–157, 2017.

[158] Ferdinand Rudolf Hendrikus Zijlstra. *Efficiency in work behaviour: A design approach for modern tools*. PhD thesis, 1993.

<figure><img src="image_8.png" /></figure>

**Tiffany Luong** received a MSc degree in Computer Science and an Engineering degree from the french national engineering school Arts et Métiers in 2018. She is now a PhD student in the Hybrid team at Inria (Rennes, France) and in the Human Factor Technologies team at IRT b<>com (Cesson-Sévigné, France). Her research interests include virtual reality and physiological computing.

<figure><img src="image_9.png" /></figure>

**Anatole Lécuyer** is director of research and head of Hybrid team at Inria, Renne, France. He is currently Associate Editor of IEEE Transactions on Visualization and Computer Graphics, Frontiers in Virtual Reality and Presence. He was Program Chair of IEEE VR 2015-2016 and General Chair of IEEE ISMAR 2017. Anatole Lécuyer obtained the IEEE VGTC Technical Achievement Award in Virtual/Augmented Reality in 2019.

<figure><img src="image_10.png" /></figure>

**Nicolas Martin** is a research scientist at the IRT b<>com (Cesson-Sévigné, France) since 2017. He received his PhD degree from the Université Rennes 2 in 2017. His main research interests include physiological computing, machine learning and human computer interaction.

<figure><img src="image_11.png" /></figure>

**Ferran Argelaguet** is an Inria research scientist at the Hybrid team (Rennes, France) since 2016. He received his PhD degree from the Universitat Politècnica de Catalunya (UPC), in Barcelona, Spain in 2011. His main research interests include 3D user interfaces, virtual reality and human-computer interaction. He was program co-chair of the IEEE Virtual Reality and 3D User Interfaces conference track in 2019 and 2020.