# Technological Competence Is a Pre-condition for Effective Implementation of Virtual Reality Head Mounted Displays in Human Neuroscience: A Technological Review and Meta-Analysis

Panagiotis Kourtesis, Simona Collina, Leonidas a A Doumas, Sarah E Macpherson

► To cite this version:

Panagiotis Kourtesis, Simona Collina, Leonidas a A Doumas, Sarah E Macpherson. Technological Competence Is a Pre-condition for Effective Implementation of Virtual Reality Head Mounted Dis- plays in Human Neuroscience: A Technological Review and Meta-Analysis. Frontiers in Human Neuroscience, 2019, 13 - Article 342, 10.3389/fnhum.2019.00342. hal-03472807

HAL Id: hal-03472807

https://inria.hal.science/hal-03472807

Submitted on 9 Dec 2021

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

Check for
updates

# Technological Competence Is a Pre-condition for Effective Implementation of Virtual Reality Head Mounted Displays in Human Neuroscience: A Technological Review and Meta-Analysis

Panagiotis Kourtesis<fnref n="1" />,<fnref n="2" />,³,⁴*, Simona Collina³,⁴, Leonidas A. A. Doumas<fnref n="2" /> and Sarah E. MacPherson<fnref n="1" />,<fnref n="2" />

## OPEN ACCESS

**Edited by:**
Valerio Rizzo,
University of Palermo, Italy

**Reviewed by:**
Camila Rosa De Oliveira,
Faculdade Meridional (IMED), Brazil
URI Maoz,
Chapman University, United States

*Correspondence:
Panagiotis Kourtesis
pkourtes@exseed.ed.ac.uk

**Specialty section:**
This article was submitted to
Cognitive Neuroscience,
a section of the journal
Frontiers in Human Neuroscience

**Received:** 27 June 2019
**Accepted:** 18 September 2019
**Published:** 02 October 2019

**Citation:**
Kourtesis P, Collina S, Doumas LAA and MacPherson SE (2019)
Technological Competence Is a Pre-condition for Effective Implementation of Virtual Reality Head Mounted Displays in Human Neuroscience: A Technological Review and Meta-Analysis.
Front. Hum. Neurosci. 13:342.
doi: 10.3389/fnhum.2019.00342

Immersive virtual reality (VR) emerges as a promising research and clinical tool. However, several studies suggest that VR induced adverse symptoms and effects (VRISE) may undermine the health and safety standards, and the reliability of the scientific results. In the current literature review, the technical reasons for the adverse symptomatology are investigated to provide suggestions and technological knowledge for the implementation of VR head-mounted display (HMD) systems in cognitive neuroscience. The technological systematic literature indicated features pertinent to display, sound, motion tracking, navigation, ergonomic interactions, user experience, and computer hardware that should be considered by the researchers. Subsequently, a meta-analysis of 44 neuroscientific or neuropsychological studies involving VR HMD systems was performed. The meta-analysis of the VR studies demonstrated that new generation HMDs induced significantly less VRISE and marginally fewer dropouts. Importantly, the commercial versions of the new generation HMDs with ergonomic interactions had zero incidents of adverse symptomatology and dropouts. HMDs equivalent to or greater than the commercial versions of contemporary HMDs accompanied with ergonomic interactions are suitable for implementation in cognitive neuroscience. In conclusion, researchers' technological competency, along with meticulous methods and reports pertinent to software, hardware, and VRISE, are paramount to ensure the health and safety standards and the reliability of neuroscientific results.

**Keywords:** virtual reality, VRISE, HMD, cybersickness, neuroscience, neuropsychology, psychology, VR

<footnote n="1">¹Human Cognitive Neuroscience, Department of Psychology, University of Edinburgh, Edinburgh, United Kingdom,</footnote>

<footnote n="2">²Department of Psychology, University of Edinburgh, Edinburgh, United Kingdom, ³Lab of Experimental Psychology, Suor Orsola Benincasa University of Naples, Naples, Italy, ⁴Interdepartmental Centre for Planning and Research “Scienza Nuova”, Suor Orsola Benincasa University of Naples, Naples, Italy</footnote>
# INTRODUCTION

In recent years, virtual reality (VR) technology has attracted attention, demonstrating its utility and potency in the field of neuroscience and neuropsychology (Rizzo et al., 2004; Bohil et al., 2011; Parsons, 2015). Traditional approaches in human neuroscience involve the utilization of static and simple stimuli which arguably lack ecological validity (Parsons, 2015). VR offers the usage of dynamic stimuli and interactions with a high degree of control within an ecologically valid environment which enables the collection of advanced cognitive and behavioral data (Rizzo et al., 2004; Bohil et al., 2011; Parsons, 2015). VR can be combined with non-invasive imaging techniques (Bohil et al., 2011; Parsons, 2015) and has been effective in the assessment of cognitive and affective functions and clinical conditions (e.g., social stress disorders) which require ecological validity (Rizzo et al., 2004; Parsons, 2015) for their assessment, rehabilitation and treatment (e.g., post-traumatic stress disorder) (Rizzo et al., 2004; Bohil et al., 2011).

However, researchers and clinicians have reported caveats with the implementation of immersive VR interventions and assessments, particularly when head mounted display (HMD) systems are utilized (Sharples et al., 2008; Davis et al., 2015; de França and Soares, 2017; Palmisano et al., 2017). A predominant concern is the presence of adverse physiological symptoms (i.e., cyber/simulation-sickness which includes nausea, disorientation, instability, dizziness, and fatigue). These undesirable effects are categorized as VR Induced Symptoms and Effects (VRISE) (Sharples et al., 2008; Davis et al., 2015; de França and Soares, 2017; Palmisano et al., 2017), and are evaluated by using questionnaires such as the Simulator Sickness Questionnaire (Kennedy et al., 1993) and the Virtual Reality Sickness Questionnaire (Kim et al., 2018).

VRISE may risk the health and safety of participants or patients (Kane and Parsons, 2017; Parsons et al., 2018), which raises ethical considerations for the adoption of VR HMDs as research and clinical tools. Additionally, the presence of VRISE has modulated substantial decline in reaction times and overall cognitive performance (Plant and Turner, 2009; Nalivaiko et al., 2015; Plant, 2016; Nesbitt et al., 2017; Mittelstaedt et al., 2018), as well as increasing body temperature and heart rates (Nalivaiko et al., 2015). Also, the presence of VRISE robustly increases cerebral blood flow and oxyhemoglobin concentration (Gavgani et al., 2018), the power of brain signals (Arafat et al., 2018), and the connectivity between stimulus response brain regions and nausea-processing brain regions (Toschi et al., 2017). Thus, VRISE could be considered confounding variables, which significantly undermine the reliability of neuropsychological, physiological, and neuroimaging data.

VRISE are predominantly mediated by an oculomotor discrepancy between what is being perceived through the oculomotor (optic nerve) sensor and what is being sensed via the rest of the afferent nerves in the human body (Sharples et al., 2008; Davis et al., 2015; de França and Soares, 2017; Palmisano et al., 2017). Nevertheless, technologically speaking, VRISE are derivatives of hardware and software inadequacies, i.e., the type of display screen, resolution, and refresh rate of the image,

the size of the field of view (FOV) as well as non-ergonomic movements within an interaction in the virtual environment (VE; de França and Soares, 2017; Palmisano et al., 2017). Notably, VR HMDs have substantially evolved during the last two decades. Important differences may be seen between the HMDs released before 2013 (old generation) and those released from 2013 onwards (new generation). While the last old generation HMD was released in 2001 (i.e., nVisor SX111), the year 2013 is used to distinguish between old and new generation HMDs, since it is the year that the first new generation HMD prototype (i.e., Oculus Development Kit 1) was released. This systematic review attempts to clarify the technological etiologies of VRISE and provide pertinent suggestions for the implementation of VR HMDs in cognitive neuroscience and neuropsychology. In addition, a meta-analysis of the neuroscience studies that have implemented VR HMDs will be conducted to elucidate the frequency of VRISE and dropout rates as per the VR HMD generation.

# TECHNOLOGICAL SYSTEMATIC REVIEW

In Table 1, a glossary of the key terms and concepts is provided to assist with comprehension of the utilized terminology. We followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines using a decremental stepwise method to perform the literature review (see Figure 1). The selected papers and book chapters included an explicit explanation and discussion of VRISE and users’ experiences pertinent with the specified technological features of the VR hardware and software. Digital databases specialized in technologies were used: (1) IEEE Xplore Digital Library; (2) ACM Digital Library; (3) ScienceDirect; (4) MIT CogNet; and (5) Scopus. Two categories of keywords were used, where each category had three or more keywords and each paper had to include at least one keyword from each category in the main body of the text. The categories were: (1) “virtual reality” OR “immersive virtual reality” OR “head-mounted display;” AND (2) “VRISE” OR “motion sickness” OR “cyber sickness” OR “simulation sickness.” Finally, the extracted information from the identified papers was clustered together under common features (i.e., display, sound, motion tracking, navigation, ergonomic interactions, user experience, and computer hardware).

## Technological Etiologies of VRISE
### Display

VR HMDs use the following three types of screens: Cathode Ray Tubes (CRT); Liquid Crystal Display (LCD); and organic light emitting diode (OLED). LCD screens replaced CRT ones due to VRISE (Costello, 1997). LCD, in comparison to CRT, alleviated the probability of visual complications and physical burdens (e.g., fatigue) (Costello, 1997). However, the suitability of LCD was challenged by the emergence of OLED screens. While old generation VR HMDs mainly utilize LCD screens (Costello, 1997), the commercial versions of new generation VR HMDs predominantly use OLED screens (Kim J. W. et al., 2017). The OLED screens have been found to be
<table><caption>**TABLE 1 | Glossary of key terms and concepts.**</caption><thead><tr><th></th><th>Terms and concepts</th><th>Explanation/definition</th></tr></thead><tbody><tr><td rowspan="3">Headsets</td><td>Head mounted display (HMD)</td><td>A display device which is worn on the head and provides an immersive virtual reality for the wearer</td></tr><tr><td>Development kit (DK) HMD</td><td>A prototype device, which is utilized by the VR Software developers to develop VR software before the commercial version of an HMD. The DKs are not provided for general use</td></tr><tr><td>Commercial version (CV) HMD</td><td>The final version of an HMD, which is dispersed to the market for general use</td></tr><tr><td rowspan="5">Display</td><td>Liquid crystal display (LCD)</td><td>A type of display/screen that uses the light-modulating properties of liquid crystals. Liquid crystals emit light indirectly, instead of using a reflector to produce images</td></tr><tr><td>Organic light emitting diode (OLED)</td><td>A type of display/screen that uses an organic compound film that emits light in response to an electric current. OLEDs are used as displays in devices such as television screens, computer monitors, and smartphones</td></tr><tr><td>Field of view (FOV)</td><td>The area captured by the display device. The size of the FOV and the size of the display device directly affect the quality of the image</td></tr><tr><td>Refresh rate and frame rate</td><td>The refresh rate is the number of times that the hardware updates its display per second. It involves the repeated display of identical frames. The frame rate indicates the frequency that software can add new data to a display</td></tr><tr><td>Resolution</td><td>The number of distinct pixels in each dimension displayed in a frame</td></tr><tr><td rowspan="5">Interactions</td><td>Motion tracking</td><td>The process of tracking the movement of objects or people. It is facilitated by motion sensors which detect the position of motion trackers embedded in devices (e.g., HMDs and 6DoF controllers)</td></tr><tr><td>Controllers/Wands with 6 degrees of freedom (DoF)</td><td>Controllers which have 6DoF of movement in 3-D space on three directional axes (i.e., Forward-Back, Left-Right, Up-Down) and three rotational axes (i.e., Roll, Pitch, Yaw)</td></tr><tr><td>Direct hand interaction</td><td>A motion tracking device (i.e., a motion sensor) which directly tracks hand movements</td></tr><tr><td>Teleportation</td><td>A navigation system, which allows the user to be transferred to a new location in the virtual environment without physically moving in the real environment</td></tr><tr><td>Ergonomic interactions</td><td>These resemble real-life interactions, which optimize user experience and overall VR system performance (see also Definition of Ergonomic Interactions). Ergonomic interactions are facilitated by and restricted to the capabilities of the VR hardware and software</td></tr><tr><td>Virtual environment (VE)</td><td>A three-dimensional artificial environment which is displayed on a display device and allows the users to interact with it</td></tr></tbody></table>

better than LCD screens for general implementation in VR, because of their faster response times, lighter weight, and better color quality (Kim J. W. et al., 2017). OLED screens decrease the likelihood of VRISE and offer an improved VR display (Kim J. W. et al., 2017).

Three more factors related to display type are crucial for the avoidance of VRISE: the width of the FOV (Rakkolainen et al., 2016; Kim J. W. et al., 2017); the resolution of the image per eye (Hecht, 2016; Rakkolainen et al., 2016; Kim J. W. et al., 2017; Brennesholtz, 2018); and the latency of the images (frames per second) (Hecht, 2016; Rakkolainen et al., 2016; Kim J. W. et al., 2017; Brennesholtz, 2018). A wider FOV significantly decreases the chance of VRISE and increases the level of immersion (Rakkolainen et al., 2016; Kim J. W. et al., 2017). The canonical guidelines suggest a lowest threshold of 110° FOV (diagonal) (Hecht, 2016; Rakkolainen et al., 2016; Kim J. W. et al., 2017; Brennesholtz, 2018). In addition, an increased refresh rate and resolution alleviates the danger of discomfort or VRISE (Hecht, 2016; Rakkolainen et al., 2016; Kim J. W. et al., 2017; Brennesholtz, 2018). The refresh rate should be ≥75 Hz (i.e., ≥75 frames per s) (Goradia et al., 2014; Hecht, 2016; Brennesholtz, 2018), while the resolution is required to be higher than 960 × 1,080 sub-pixels per eye (Goradia et al., 2014).

## Sound

A second important consideration for a user's experience in VR is the sound quality. The integration of spatialized sounds (e.g., ambient and feedback sounds) in the VE may increase the level of immersion, pleasantness of the experience, and successful navigation (Vorländer and Shinn-Cunningham, 2014), while they significantly decrease the likelihood of VRISE (Viirre et al., 2014). However, the volume and localization of sounds need to be optimized in terms of audio spatialization to ensure a user's experience is pleasant without adverse VRISE (Viirre et al., 2014; Vorländer and Shinn-Cunningham, 2014).

## Motion Tracking

Motion tracking in VR is a pre-condition for naturalistic movement within an immersive VE (Slater and Wilbur, 1997; Stanney and Hale, 2014). Motion tracking allows the precise tracking of the user's physical body within the VE (i.e., it allows the computer to provide accurate environmental feedback, which modulates and consolidates the awareness of the position and movement of the user's body). This phenomenon is called proprioception or kinesthesia (Slater and Wilbur, 1997) and is linked with vestibular and oculomotor mediated VRISE (Slater and Wilbur, 1997; Plouzeau et al., 2015; Caputo et al., 2017). Hence, motion tracking should be adequately
<figure><img src="image_3.png" /><figcaption>FIGURE 1 | Decremental stepwise method for the technological literature review.</figcaption></figure>

rapid and accurate to facilitate ergonomic interactions in
the VE (Caputo et al., 2017).

Navigation

A highly important factor in the quality of VR software
and to avoid VRISE is the movement of the user in the
VE (Porcino et al., 2017). New generation HMDs deliver an
adequate play area for interactions to facilitate ecologically
valid scenarios (Porcino et al., 2017; Borrego et al., 2018).
However, there are restrictions in the size of the play area,
which does not permit navigation solely by physical walking

(Porcino et al., 2017; Borrego et al., 2018). Teleportation
allows movement beyond the play area size and elicits a
high-level of immersion and pleasant user experience, whilst
alleviating VRISE (Bozgeyikli et al., 2016; Frommel et al., 2017;
Porcino et al., 2017). In contrast, movement dependent on
a touchpad, keyboard, or joystick results to high occurrences
of VRISE (Bozgeyikli et al., 2016; Frommel et al., 2017;
Porcino et al., 2017). Therefore, teleportation in conjunction
with physical movement (i.e., free movement of the upper
limbs and walking in a small-restricted area) is the most
suitable method for movement in VR (Bozgeyikli et al., 2016;
<table><caption>**TABLE 2** | Minimum hardware criteria: old and new generation VR HMDs.</caption><thead><tr><th>Product</th><th>Generation</th><th>Resolution (per eye)</th><th>Display screen</th><th>Refresh rate</th><th>FOV (Diagonal)</th><th>Motion trackers and sensors (Type and quantity)</th></tr></thead><tbody><tr><td>VFX 3D</td><td>Old</td><td>480 × 240</td><td>LCD</td><td>45 Hz</td><td>45°</td><td>–</td></tr><tr><td>VUZIX Wrap 1200</td><td>Old</td><td>852 × 480</td><td>LCD</td><td>60 Hz</td><td>35°</td><td>Unknown type (1), 3 magnetometers, 3 accelerometers, and 3 gyroscopes</td></tr><tr><td>eMagin Z800 3DVisor</td><td>Old</td><td>800 × 600</td><td>OLED</td><td>60 Hz</td><td>40°</td><td>–</td></tr><tr><td>nVisor SX111</td><td>Old</td><td>1,280 × 1,024</td><td>LCD</td><td>60 Hz</td><td>110°</td><td>–</td></tr><tr><td>Oculus rift development kit 1</td><td>New</td><td>640 × 800</td><td>LCD</td><td>60 Hz</td><td>110°</td><td>–</td></tr><tr><td>Oculus rift development kit 2</td><td>New</td><td>960 × 1,080</td><td>OLED</td><td>75 Hz</td><td>110°</td><td>–</td></tr><tr><td>Minimum hardware criteria for the avoidance of VRISE</td><td>NA</td><td>>960 × 1,080</td><td>OLED or LCD</td><td>≥75Hz</td><td>≥110°</td><td>Tracking should be adequately rapid and accurate to facilitate ergonomic interactions</td></tr><tr><td>Oculus rift commercial version</td><td>New</td><td>1,080 × 1,200</td><td>OLED</td><td>90 Hz</td><td>110°</td><td>Accelerometer, gyroscope, magnetometer, 360° constellation tracking camera</td></tr><tr><td>HTC VIVE commercial version</td><td>New</td><td>1,080 × 1,200</td><td>OLED</td><td>90 Hz</td><td>110°</td><td>Sensors (>70) including MEMS, magnetometer, gyroscope, accelerometer, and laser position sensors, lighthouse laser tracking system (2 base stations emitting pulsed InfraRed lasers), front-facing camera</td></tr></tbody></table>

MEMS, Microelectromechanical systems.

<table><caption>**TABLE 3** | Criteria for suitable VR software in cognitive neuroscience and neuropsychology.</caption><thead><tr><th>Domains</th><th>User experience</th><th>Game mechanics</th><th>In-game assistance</th><th>VRISE</th></tr></thead><tbody><tr><td rowspan="5">Criteria</td><td>An adequate level of immersion</td><td>A suitable navigation system (e.g., Teleportation)</td><td>Digestible tutorials</td><td>Absence or insignificant presence of nausea</td></tr><tr><td>Pleasant VR experience</td><td>Availability of physical movement</td><td>Helpful tutorials</td><td>Absence or insignificant presence of disorientation</td></tr><tr><td>High quality graphics</td><td>Naturalistic picking/placing of items</td><td>Adequate duration of tutorials</td><td>Absence or insignificant presence of dizziness</td></tr><tr><td>High quality sounds</td><td>Naturalistic use of items</td><td>Helpful in-game instructions</td><td>Absence or insignificant presence of fatigue</td></tr><tr><td>Suitable hardware (HMD and computer)</td><td>Naturalistic 2-handed interaction</td><td>Helpful in-game prompts</td><td>Absence or insignificant presence of instability</td></tr></tbody></table>

Frommel et al., 2017; Porcino et al., 2017). Yet, there are additional factors such as external hardware (i.e., controllers and wands), which are needed to facilitate optimal ergonomic interactions in VR.

## Ergonomic Interactions

Ergonomic and naturalistic interactions are essential to minimize the risk of VRISE, while non-ergonomic and non-naturalistic interactions increase the occurrence of them (Slater and Wilbur, 1997; Stanney and Hale, 2014; Plouzeau et al., 2015; Caputo et al., 2017; Porcino et al., 2017). Importantly, controllers, joysticks, and keyboards do not support ergonomic and naturalistic interactions in VR (Plouzeau et al., 2015; Bozgeyikli et al., 2016; Caputo et al., 2017; Frommel et al., 2017; Porcino et al., 2017; Sportillo et al., 2017; Figueiredo et al., 2018). Instead, wands with 6 degrees of freedom (DoF) of movement (e.g., Oculus Rift and HTC Vive wands), and realistic interfaces with direct hand interactions (e.g., Microsoft's Kinect) facilitate

naturalistic and ergonomic interactions (Sportillo et al., 2017; Figueiredo et al., 2018). Both hardware systems facilitate easy familiarization with their controls and their utilization (Sportillo et al., 2017; Figueiredo et al., 2018). However, direct hand interactions are easier than 6DoF controllers-wands in terms of familiarization with their controls and efficiency (Sportillo et al., 2017; Figueiredo et al., 2018). Direct hand interactions were also found to offer more pleasant user experiences (Sportillo et al., 2017; Figueiredo et al., 2018), although, they are substantially less accurate than 6DoF controller-wands (Sportillo et al., 2017; Figueiredo et al., 2018).

## User Experience

Notably, ergonomic interactions might be available to the user; however, the user is required to learn the necessary interactions and how the VE functions to facilitate a pleasant user experience (Gromala et al., 2016; Jerald et al., 2017; Brade et al., 2018). The inclusion of comprehensible tutorials where the user may
spend an adequate amount of time acquiring the necessary skills (i.e., navigation, use and grab of items, two-handed interactions) and knowledge of the VE (i.e., how it reacts to your controls) is crucial (Gromala et al., 2016; Jerald et al., 2017; Brade et al., 2018). Additionally, in-game instructions and prompts should be offered to the user through interactions in the VE (e.g., directional arrows, non-player characters, signs, labels, ambient sounds, audio, and videos) (Gromala et al., 2016; Jerald et al., 2017; Brade et al., 2018).

## Computer Hardware

The computer hardware (i.e., the processor, graphics card, sound card) should at least meet the minimum requirements of the VR software and HMD (Anthes et al., 2016). The performance of VR HMDs is analogous to the computing power and the quality of the hardware (Stanney and Hale, 2014; Anthes et al., 2016; Borrego et al., 2018). The processor, graphics card, sound card, and operating system (e.g., Windows) need to be considered and reported because they modulate the performance of the software (Plant and Turner, 2009; Plant, 2016; Kane and Parsons, 2017; Parsons et al., 2018). Research software developers and researchers are required to be technologically competent in order to opt for the appropriate hardware and software to achieve their research and/or clinical aims (Plant, 2016; Kane and Parsons, 2017; Parsons et al., 2018).

## Conclusions

Based on the outcomes of the above technological review, VR HMDs should have a good quality display-screen (i.e., OLED or upgraded LCD), an adequate FOV (i.e., diagonal FOV ≥ 110°), adequate resolution per eye (i.e., resolution > 960 × 1,080 sub-pixels per eye), and an adequate image refresh rate (i.e., refresh rate ≥ 75 Hz) to safeguard the health and safety of the participants and the reliability of the neuroscientific results (see **Table 2**). Also, the VR HMD should have external hardware which offers an adequate VR area, fast and accurate motion tracking, spatialized audio, and ergonomic interactions. The computer's processor, graphics card, and sound card should meet the minimum requirements of the VR software and HMD too. New generation VR HMDs appear to have all the necessary hardware characteristics (i.e., graphics, level of immersion, and sound) to be used in ecological valid research and clinical paradigms (Borrego et al., 2018; see **Table 2** for a comparison between old and new generation HMDs). New generation VR HMDs have the required hardware to support and produce high-quality spatialized sounds in VEs (Borrego et al., 2018). Additionally, new generation VR HMDs have integrated rapid and precise motion tracking which facilitates naturalistic and ergonomic interactions within the VE (Borrego et al., 2018).

Both the Oculus development kit (DK) 1 and DK2 do not meet the minimum hardware features highlighted by the technological review, despite being new generation VR HMDs (see **Table 2**). The DK1 has substantially lower resolution per eye and image refresh rates, while the DK2 has marginally acceptable refresh rates, yet a slightly lower resolution per eye. These DKs are not available for general use but are used by professional

developers to produce beta (early) versions of their games or apps (Goradia et al., 2014; Suznjevic et al., 2017). Moreover, they were removed from the market after the release of the Oculus Rift CV. VR HMDs should have hardware characteristics equal to or better than the commercial versions (CV) of the Oculus Rift and HTC Vive in order to ensure the health and safety of the participants, as well as the reliability of the neuroscientific results (i.e., physiological, neuropsychological, and neuroimaging data). The researchers and clinicians should have the technological competence to choose an HMD which is equal to or greater than the CVs of the Oculus Rift and HTC Vive (e.g., Valve Index, HTC Vive Pro, Oculus Quest, Pimax VR, and StarVR).

However, the VR software's features are equally important. The VR software should include an ergonomic interaction and navigation system, as well as tutorials, in-game instructions, and prompts. A suitable navigation system should combine teleportation and physical movement, while ergonomic interactions should include those that simulate real-life interactions by using a direct hands system or 6DoF controllers. Also, the tutorials, in-game instructions, and prompts should be informative and easy to follow, especially for experimental or clinical purposes where users should be equally able to interact with the VE (Plant and Turner, 2009; Plant, 2016; Kane and Parsons, 2017; Parsons et al., 2018), and/or jeopardize the health and safety of their participants/patients (Kane and Parsons, 2017; Parsons et al., 2018). The criteria for effective VR software are displayed in **Table 3**. These criteria should be met before implementing VR software for research and/or clinical purposes. Otherwise, researchers or clinicians may compromise the reliability of their study's results (Plant and Turner, 2009; Plant, 2016; Kane and Parsons, 2017; Parsons et al., 2018), and/or jeopardize the health and safety of their participants/patients (Kane and Parsons, 2017; Parsons et al., 2018).

The above features enable researchers or clinicians to administer a sophisticated and pleasant VR experience, which substantially alleviates or eradicates adverse VRISE. Therefore, the technological competency of neuroscientists and neuropsychologists is a pre-condition for the efficient adoption and implementation of innovative technologies like VR HMDs in cognitive neuroscience or neuropsychology.

# META-ANALYSIS OF VR STUDIES IN COGNITIVE NEUROSCIENCE

## Literature Research and Inclusion Criteria

We followed the PRISMA guidelines to conduct the literature research using a decremental approach, where the selection commenced with a relatively vast accumulation of abstracts and concluded with a diminished list of full papers that comprise standardized and detailed VR research paradigms. The procedure is described in **Figure 2**. The following databases were used for the literature research: (1) PsycInfo; (2) PsycArticles; (3) PubMed; and (4) Medline. Two categories of keywords were used, with three keywords in each category. The minimum threshold for each study was the inclusion of at least one keyword from each category in the main body of text. The keywords
<figure><img src="image_4.png" /></figure>

for each category were: (1) “virtual reality;” OR “Immersive;”
OR “Head Mounted Display;” AND (2) “Psychology;” OR
“Neuropsychology” OR “Neuroscience.” Additional filters and
criteria were: (1) chronological specification (2004 and later); and
(2) a comprehensive description of the VR research methods
in conjunction with the research aims and results. Finally, the
selected studies were allocated into two groups according to the
generation of the implemented VR HMD. Two tables display the
studies that utilize old generation (Table 4) and new generation
(Table 5) HMDs.

Data Collection and Coding

Target Variables

The principal aim of the meta-analysis was to measure the
frequency of VRISE in neuroscience or psychology studies
using a VR HMD. However, only six studies reported VRISE
quantitatively (i.e., using a questionnaire). For this reason,
we considered only the presence or absence of VRISE. The
dichotomous VRISE variable (i.e., presence or absence of VRISE)
was quantified (i.e., absent VRISE = 0; present VRISE = 1) to
facilitate a comparison (i.e., Bayesian t-tests) between the studies
<table><caption>TABLE 4 | Neuroscience studies employing old generation VR HMDs.</caption><thead><tr><th rowspan="2">References</th><th rowspan="2">Topic</th><th rowspan="2">HMD</th><th>Ergonomic interactions</th><th rowspan="2">Clinical condition</th><th rowspan="2">Age group</th><th rowspan="2">N</th><th colspan="2">Dropouts</th></tr><tr><th>YES = 12<br>NO = 10</th><th>YES = 14<br>NO = 8</th><th>YES = 9<br>NO = 13</th></tr></thead><tbody><tr><td>Kim et al. (2004)</td><td>Visuospatial<br>functions</td><td>Eye-trek<br>FMD-250W</td><td>YES</td><td>Brain injury</td><td>MA</td><td>52</td><td>YES</td><td>NO - 0</td></tr><tr><td>Moreau et al. (2006)</td><td>Executive<br>functions</td><td>Eye-trek<br>FMD-250W</td><td>YES</td><td>ADHD and<br>autism</td><td>YA</td><td>22</td><td>YES</td><td>YES - 1</td></tr><tr><td>Botella et al. (2007)</td><td>Therapy<br>(VRET)</td><td>V6 VR</td><td>NO</td><td>Panic<br>disorder</td><td>YA and MA</td><td>46</td><td>YES</td><td>YES - 9</td></tr><tr><td>Matheis et al. (2007)</td><td>Memory</td><td>eMagin z800</td><td>YES</td><td>Brain injury</td><td>MA</td><td>40</td><td>NO</td><td>NO - 0</td></tr><tr><td>Parsons et al. (2007)</td><td>Executive<br>functions</td><td>eMagin z800</td><td>NO</td><td>ADHD</td><td>C</td><td>20</td><td>YES</td><td>YES - 1</td></tr><tr><td>Banville et al. (2010)</td><td>Memory</td><td>eMagin z800</td><td>NO</td><td>Brain injury</td><td>YA</td><td>62</td><td>NO</td><td>NO - 0</td></tr><tr><td>Rizzo et al. (2010)</td><td>Therapy<br>(VRET)</td><td>eMagin z800</td><td>NO</td><td>PTSD</td><td>YA</td><td>20</td><td>YES</td><td>YES - 5</td></tr><tr><td>Reger et al. (2011)</td><td>Therapy<br>(VRET)</td><td>eMagin z800</td><td>NO</td><td>PTSD</td><td>YA</td><td>24</td><td>YES</td><td>YES - 6</td></tr><tr><td>Bioulac et al. (2012)</td><td>Executive<br>functions</td><td>eMagin z800</td><td>YES</td><td>ADHD</td><td>YA</td><td>36</td><td>NO</td><td>NO - 0</td></tr><tr><td>Carlozzi et al. (2013)</td><td>Rehabilitation</td><td>eMagin z800</td><td>YES</td><td>Spinal cord<br>injury</td><td>MA</td><td>54</td><td>YES</td><td>YES - 10</td></tr><tr><td>Meyerbroeker et al. (2013)</td><td>Therapy<br>(VRET)</td><td>nVISOR<br>SX111</td><td>YES</td><td>Agoraphobia</td><td>MA</td><td>55</td><td>YES</td><td>YES - 17</td></tr><tr><td>Parsons et al. (2013)</td><td>Attention<br>assessment</td><td>eMagin Z800</td><td>YES</td><td>Healthy</td><td>YA</td><td>50</td><td>YES</td><td>NO - 0</td></tr><tr><td>Peck et al. (2013)</td><td>Racial<br>biases</td><td>nVISOR<br>SX111</td><td>YES</td><td>Healthy</td><td>YA</td><td>60</td><td>NO</td><td>NO - 0</td></tr><tr><td>Freeman et al. (2014)</td><td>Social<br>cognition</td><td>nVISOR<br>SX111</td><td>YES</td><td>Paranoia</td><td>YA</td><td>60</td><td>YES</td><td>NO - 0</td></tr><tr><td>Rothbaum et al. (2014)</td><td>Therapy<br>(VRET)</td><td>eMagin Z800</td><td>NO</td><td>PTSD</td><td>YA and MA</td><td>156</td><td>NO</td><td>NO - 0</td></tr><tr><td>Veling et al. (2014)</td><td>Paranoid<br>thoughts</td><td>eMagin Z800</td><td>NO</td><td>Psychosis</td><td>YA</td><td>41</td><td>YES</td><td>NO - 0</td></tr><tr><td>Hartanto et al. (2014)</td><td>Social<br>stress</td><td>eMagin Z800</td><td>NO</td><td>Healthy</td><td>YA</td><td>54</td><td>NO</td><td>NO - 0</td></tr><tr><td>Gaggioli et al. (2014)</td><td>Stress<br>levels</td><td>Vuzix Wrap<br>1200VR</td><td>NO</td><td>Healthy</td><td>MA</td><td>121</td><td>YES</td><td>NO - 0</td></tr><tr><td>Shiban et al. (2015)</td><td>Therapy<br>(VRET)</td><td>eMagin Z800</td><td>NO</td><td>Arachnophobia</td><td>YA</td><td>58</td><td>YES</td><td>YES - 8</td></tr><tr><td>Freeman et al. (2016)</td><td>Therapy<br>(VRET)</td><td>nVISOR<br>SX111</td><td>YES</td><td>Persecutory<br>delusions</td><td>MA</td><td>30</td><td>YES</td><td>YES - 1</td></tr><tr><td>Parsons and Carlew<br>(2016)</td><td>Attention<br>assessment</td><td>eMagin Z800</td><td>YES</td><td>Healthy</td><td>YA</td><td>50</td><td>NO</td><td>NO - 0</td></tr><tr><td>Parsons and Barnett<br>(2017)</td><td>Attention<br>assessment</td><td>eMagin Z800</td><td>YES</td><td>Healthy</td><td>YA and OA</td><td>89</td><td>NO</td><td>NO - 0</td></tr></tbody></table>

HMD, Head-Mounted Display; VRISE, VR induced adverse symptoms and effects; YA, Young Adults; MA, Middle-Aged Adults; OA, Older Adults; C, Children; VRET, VR Exposure Therapy; PTSD, Post-Traumatic Stress Disorder; ADHD, Attention Deficit Hyperactivity Disorder.

that used old generation HMDs, new generation DK HMDs, and new generation CV HMDs, as well as the examination of potential correlations with other variables (i.e., Bayesian Pearson's correlation analysis).

secondary aim of the meta-analysis was to inspect the dropout rates in neuroscience or psychology studies that used VR HMDs. However, as the vast majority of studies had no dropouts, studies with some dropouts (e.g., 3, 5,

6) were statistically considered as outliers. For this reason, we considered the existence of dropouts in each study. The dropout variable was dichotomized as presence = 1 and absence = 0. This dichotomous dropout variable was used to investigate whether using a certain generation HMD (i.e., old generation HMDs, new generation DK HMDs, or new generation CV HMDs) could increase/decrease the dropout size. We compared (i.e., Bayesian t-tests)
<table><caption>TABLE 5 | Neuroscience studies employing new generation VR HMDs.</caption><thead><tr><th rowspan="2">References</th><th rowspan="2">Topic</th><th rowspan="2">HMD</th><th>Ergonomic interactions</th><th rowspan="2">Clinical condition</th><th rowspan="2">Age group</th><th rowspan="2">N</th><th>VRISE</th><th>Dropouts</th></tr><tr><th>YES = 18<br>NO = 4</th><th>YES = 4<br>NO = 18</th><th>YES = 4<br>NO = 18</th></tr></thead><tbody><tr><td>Foerster et al. (2016)</td><td>Attention assessment</td><td>Oculus DK2</td><td>NO</td><td>Healthy</td><td>YA</td><td>44</td><td>NO</td><td>YES - 2</td></tr><tr><td>Quinlivan et al. (2016)</td><td>Attention assessment</td><td>Oculus DK2</td><td>YES</td><td>Healthy</td><td>YA</td><td>40</td><td>NO</td><td>NO - 0</td></tr><tr><td>Kim A. et al. (2017)</td><td>VR presence</td><td>Oculus DK2</td><td>YES</td><td>PD</td><td>OA</td><td>33</td><td>NO</td><td>NO - 0</td></tr><tr><td>Montenegro and Argyriou (2017)</td><td>Memory, attention, executive functions</td><td>Oculus DK2</td><td>YES</td><td>AD (early stages)</td><td>OA</td><td>20</td><td>NO</td><td>NO - 0</td></tr><tr><td>Parsons and McMahan (2017)</td><td>Memory assessment</td><td>HTC vive</td><td>YES</td><td>Healthy</td><td>YA</td><td>103</td><td>NO</td><td>NO - 0</td></tr><tr><td>Kelly et al. (2017)</td><td>Spatial perception</td><td>HTC vive</td><td>YES</td><td>Healthy</td><td>YA</td><td>76</td><td>NO</td><td>NO - 0</td></tr><tr><td>Bourdin et al. (2017)</td><td>Fear of death</td><td>Oculus DK2</td><td>YES</td><td>Healthy</td><td>YA</td><td>36</td><td>NO</td><td>NO - 0</td></tr><tr><td>Hasler et al. (2017)</td><td>Racial bias</td><td>Oculus DK2</td><td>YES</td><td>Healthy</td><td>YA</td><td>36</td><td>NO</td><td>NO - 0</td></tr><tr><td>Mottelson and Hornbæk (2017)</td><td>Navigation, attention, B-P</td><td>HTC vive</td><td>YES</td><td>Healthy</td><td>YA and MA</td><td>31</td><td>NO</td><td>NO - 0</td></tr><tr><td>Rooney et al. (2017)</td><td>Social cognition</td><td>Oculus rift CV</td><td>YES</td><td>Healthy</td><td>YA and MA</td><td>103</td><td>NO</td><td>NO - 0</td></tr><tr><td>Zimmer et al. (2018)</td><td>Social stress</td><td>Oculus DK2</td><td>NO</td><td>Healthy</td><td>YA and MA</td><td>93</td><td>YES</td><td>YES - 5</td></tr><tr><td>Hsieh et al. (2018)</td><td>Spatial perception and navigation</td><td>HTC vive</td><td>YES</td><td>Healthy</td><td>YA</td><td>70</td><td>NO</td><td>NO - 0</td></tr><tr><td>Yeh et al. (2018)</td><td>Anxiety</td><td>HTC vive</td><td>YES</td><td>Healthy</td><td>YA</td><td>34</td><td>NO</td><td>NO - 0</td></tr><tr><td>Collins et al. (2018)</td><td>Psychoeducation on DBS</td><td>Oculus rift CV</td><td>YES</td><td>Movement disorder</td><td>OA</td><td>30</td><td>NO</td><td>NO - 0</td></tr><tr><td>Barberia et al. (2018)</td><td>Fear of death</td><td>Oculus DK2</td><td>YES</td><td>Healthy</td><td>YA</td><td>31</td><td>YES</td><td>YES - 1</td></tr><tr><td>Banakou et al. (2018)</td><td>Embodiment, cognition-IQ</td><td>HTC vive</td><td>YES</td><td>Healthy</td><td>YA</td><td>30</td><td>NO</td><td>NO - 0</td></tr><tr><td>Christou et al. (2018)</td><td>Motor-rehabilitation</td><td>HTC vive</td><td>YES</td><td>Stroke patients</td><td>YA and MA</td><td>29</td><td>NO</td><td>NO - 0</td></tr><tr><td>Gómez-Jordana et al. (2018)</td><td>Balance and walking rehabilitation</td><td>Oculus DK2</td><td>YES</td><td>PD</td><td>OA</td><td>22</td><td>NO</td><td>NO - 0</td></tr><tr><td>Lubetzky et al. (2018)</td><td>Sensory integration and balance</td><td>Oculus DK2</td><td>NO</td><td>Healthy</td><td>YA and MA</td><td>21</td><td>YES</td><td>NO - 0</td></tr><tr><td>Oagaz et al. (2018)</td><td>Memory assessment</td><td>HTC vive</td><td>YES</td><td>Healthy</td><td>YA</td><td>20</td><td>NO</td><td>NO - 0</td></tr><tr><td>George et al. (2018)</td><td>Working memory and attention assessment</td><td>HTC vive</td><td>YES</td><td>Healthy</td><td>YA</td><td>20</td><td>NO</td><td>NO - 0</td></tr><tr><td>Detez et al. (2019)</td><td>Gambling</td><td>HTC vive</td><td>NO</td><td>Healthy</td><td>YA and MA</td><td>60</td><td>YES</td><td>YES - 3</td></tr></tbody></table>

HMD, Head-Mounted Display; VRISE, VR induced adverse symptoms and effects; YA, Young Adults; MA, Middle-Aged Adults; OA, Older Adults; PD, Parkinson's disease; AD, Alzheimer's disease; DK, Development Kit; CV, Commercial Version; B-P, Body Perception; DBS, Deep Brain Stimulation.

the dropout rate across studies that used old generation HMDs, new generation DK HMDs, and new generation CV HMDs. We also inspected whether the dropout rates correlated with other variables by using Bayesian Pearson's correlation analysis.

## Grouping Variables

We subdivided studies into groups based on the HMD generation they used. Hence, two groups of studies were created and compared by using Bayesian *t*-tests; the first group included studies that utilized old generation HMDs, while the second
group included studies which utilized new generation HMDs (i.e., both DKs and CVs).

The new generation studies were further distinguished and compared by using Bayesian *t*-tests based on the type of new generation HMDs adopted (i.e., DK or CV). Two sub-groups were formed; the first group included studies that utilized DK HMDs, and the second group included studies that utilized CV HMDs.

Furthermore, the recency of the HMD technology was compared by using an ordinal variable where 1 indicated old generation HMDs, 2 indicated new generation DKs, and 3 indicated new generation CVs. This ordinal variable allowed us to inspect whether the HMD generation correlated with other variables by using Bayesian Pearson's correlation analysis.

Lastly, we considered the type of interactions, where the type of interactions were expressed in a binary form (i.e., non-ergonomic interactions = 0 and ergonomic interactions = 1). This allowed a comparison between the VR studies which had ergonomic interactions and the VR studies which had non-ergonomic interactions by using a Bayesian *t*-test. It also allowed us to inspect whether the interaction type correlated with other variables (i.e., Bayesian Pearson's correlation analysis).

## Definition of Ergonomic Interactions

In line with the definition of ergonomic interactions in our technological review, we considered interactions to be ergonomic or non-ergonomic based on their proximity to real-life interactions. We provide some examples below to clarify our criteria:

Example 1—Ergonomic Interaction: if the VR software required the participant to look around moving his or her head.

Example 2—Non-Ergonomic Interaction: if the VR software required the participant to look around by using a joystick or mouse.

Example 3—Ergonomic Interaction: if the VR software required the user to interact with objects (e.g., pushing a button, holding an item) in the VE or to navigate within the VE by using either 6DoF controllers or direct-hand interactions.

Example 4—Non-Ergonomic Interaction: if the VR software required the user to interact with objects (e.g., pushing a button, holding an item) in the VE or to navigate within the VE by using a keyboard or joystick (e.g., Xbox controller).

## Statistical Analyses

Bayesian statistics were preferred over null hypothesis significance testing (NHST). The Bayesian factor ($BF_{10}$) was therefore used instead of *p*-values for statistical inference, although we do report both $BF_{10}$ and *p*-values. *p*-values measure the difference between the data and the null hypothesis ($H_0$) (e.g., the assumption of no difference or no effect), while the $BF_{10}$ calibrates *p*-values by converting them into evidence in favor of the alternative hypothesis ($H_1$) over the $H_0$ (Cox

and Donnelly, 2011; Bland, 2015; Held and Ott, 2018). $BF_{10}$ is considered substantially more parsimonious than the *p*-value in evaluating the evidence against the $H_0$ (Cox and Donnelly, 2011; Bland, 2015; Held and Ott, 2018). Also, the difference between $BF_{10}$ and the *p*-value in evaluating the evidence against $H_0$ is even greater in small sample sizes (Held and Ott, 2018). Bayesian Factor ($BF_{10}$) threshold ≥ 10 was set for statistical inference in all analyses, which indicates strong evidence in favor of the $H_1$ (Rouder and Morey, 2012; Wetzels and Wagenmakers, 2012; Marsman and Wagenmakers, 2017), and corresponds to a *p*-value < 0.01 (e.g., $BF_{10} = 10$) (Cox and Donnelly, 2011; Bland, 2015; Held and Ott, 2018). JASP software was used to perform the statistical analyses (JASP Team, 2018). Bayesian independent samples *t*-tests were conducted to investigate the difference in VRISE frequency and dropout occurrence between old and new generation HMDs, as well as between new generation DKs and CVs. A Bayesian Pearson's correlations analysis examined the possible statistical relationships amongst the HMD generations, VRISE presence, the type of interactions, and dropout occurrences.

# Results

## The Implementation of Old and New Generation HMDs in Cognitive Neuroscience

The studies that utilized old generation HMDs are displayed in **Table 4** and recruited 1,200 participants in total. Nine out of 22 studies examined stress disorders, 7 of these were VR exposure therapy (VRET) studies either for phobias or post-traumatic stress disorder (PTSD), while 2 studies attempted to assess stress levels in context (e.g., assessment of social stress during a job interview). In 9 studies, there were VR assessments of cognitive functions, 2 studies assessed memory, 3 studied attention, 3 examined executive functions, and 1 examined visuospatial ability. Two of the studies involved social cognition while only one involved paranoid thinking. Lastly, only one study provided rehabilitation sessions in VR for patients with spinal injuries. The targeted age groups were young adults in 18 studies, middle-aged adults in 8 studies, older adults in one study, and children in one study.

The studies that utilized new generation HMDs are displayed in **Table 5** and recruited 982 individuals in total. Specifically, 376 individuals were recruited in 10 studies where new generation DKs were used, while 606 individuals were recruited in 12 studies where new generation CVs were used. Nine out of the 22 studies attempted to assess cognitive functions (i.e., memory, attention, visuospatial ability, executive functions), 4 investigated anxiety disorders (i.e., fear of death, social stress, general anxiety disorder), 3 provided sensorimotor rehabilitation interventions, 3 studies examined the effects of presence in specific VEs, 2 assessed social cognition and 1 study offered a psychoeducational session to patients with motor-related disorders. Lastly, the targeted age groups were young adults in 18 studies, middle-aged adults in 6 studies, and older adults in 4 studies.
<figure><img src="image_5.png" /><figcaption>**FIGURE 3** | VRISE per HMD generation and ergonomic interactions. ABSENT, Absence of VRISE; PRESENT, Presence of VRISE; OLD, Old Generation HMD; DK, New Generation HMD—Development Kit; CV, New Generation HMD—Commercial Version; Ergonomic, Ergonomic Interactions; Non-Ergonomic, Non-Ergonomic Interactions.</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>**FIGURE 4** | Dropouts and sample size per HMD, VRISE presence, and ergonomic interactions. ABSENT, Absence of VRISE; PRESENT, Presence of VRISE; OLD, Old Generation HMD; DK, New Generation HMD—Development Kit; CV, New Generation HMD—Commercial Version; Ergonomic, Ergonomic Interactions; Non-Ergonomic, Non-Ergonomic Interactions.</figcaption></figure>

## Meta-Analysis

The descriptive statistics are presented in **Figures 3, 4**. In **Figure 3**, the number of studies with VRISE are displayed according to their HMD generation and interaction type. In **Figure 4**, the dropouts and sample sizes are presented according to their HMD generation, VRISE presence, and interaction type. The presence of VRISE substantially becomes less frequent when new generation HMDs are implemented (**Figure 3**).

In new generation HMDs, VRISE are present in only 4 out of 22 studies, while across 982 participants, there are only 11 dropouts. In contrast, in old generation HMDs, VRISE are present in 14 out of 22 studies, while in a total sample size of 1,200 participants, there are 58 dropouts.

In the 14 old generation HMDs studies where VRISE are present, half of them involved ergonomic interactions and the other half involved non-ergonomic interactions. Similarly, there
is an equal distribution of dropouts (29 in each) between the old generation HMDs studies that had ergonomic and non-ergonomic interactions. When only old HMDs with ergonomic interactions are considered, VRISE are present in 7 out of 12 studies, while in an entire sample size of 598, the dropouts are 29. In the studies with new generation DKs, non-ergonomic interactions had an increased presence of VRISE than the ones with ergonomic interactions. Also, in the studies which used DKs with non-ergonomic interactions, 7 participants out of 158 dropped out, while in studies with ergonomic interactions, only one participant out of 218 dropped out. Importantly, when new generation CVs with ergonomic interactions are exclusively considered, there are no VRISE or dropouts in any of the 11 studies with 546 participants. Finally, VRISE were only present in one study using a new generation CV HMD, where 3 participants dropped out. This study was the only one with a new generation CV HMD that did not involve ergonomic interactions.

The Bayesian independent samples t-test highlighted that studies involving new generation VR HMDs have significantly less frequent VRISE (BF₁₀ = 144.68; *p* < 0.001). The difference in the existence of dropouts was not substantial, yet the studies with new generation HMDs have less frequent dropouts (BF₁₀ = 4.69; *p* < 0.05) than studies with old HMDs. Notably, the studies which used a new generation CV HMD have significantly less frequent VRISE (BF₁₀ = 46.39; *p* < 0.001) but not less frequent dropouts (BF₁₀ = 1.66; *p* = 0.16) than the studies which used a new generation DK HMD. Finally, the studies which implemented VR software with ergonomic interactions had substantially less frequent VRISE (BF₁₀ = 19.54; *p* < 0.001) and dropouts (BF₁₀ = 16.01; *p* < 0.001) than studies which used VR software with non-ergonomic interactions.

The Bayesian Pearson's correlations demonstrated a substantial negative correlation between the presence of VRISE and the HMD generation [BF₁₀ = 328.03; *r*_(44) = -0.56, *p* < 0.001], while the presence of VRISE robustly demonstrated a positive correlation with the existence of dropouts [BF₁₀ = 83510.53; *r*_(44) = 0.68, *p* < 0.001]. Also, the utilization of ergonomic interactions was significantly negatively correlated with VRISE [BF₁₀ = 20.11; *r*_(44) = -0.42, *p* < 0.001] and the existence of dropouts [BF₁₀ = 16.11; *r*_(44) = -0.41, *p* < 0.001].

## Discussion

The results of the meta-analysis indicated that VR HMDs have been implemented in diverse clinical conditions and age groups, as well as the unquestionable difference between old generation and new generation HMDs. There were significantly more frequent VRISE in the studies involving old generation VR HMDs compared to studies with new generation HMDs. Additionally, the frequency of VRISE correlated negatively with the HMD generation. Hence, the older the utilized HMD, the higher the VRISE frequency. Moreover, the existence of dropouts significantly and positively correlated with the presence of VRISE.

Nevertheless, one potential reason for the higher dropouts in old generation studies is that several studies included follow-up sessions (e.g., VRET) and participants may have opted not to return to complete the remaining sessions for reasons other

than the presence of VRISE. However, in the old generation studies, the dropout rates were low in relation to the size of the population, albeit there were VRISE present. The low dropout rates in the old generation HMD studies may be due to the fixed intervals between the VR sessions where the participants were able to rest and obtain relief from any adverse effects they were experiencing.

Furthermore, the incidence of VRISE in old generation HMD studies may be due to anxiety levels (Bouchard et al., 2011) or be self-induced (Almeida et al., 2017) as several of the studies had either stress-related aims or included participants with stress disorders. However, several of the new generation HMD studies also had comparable aims and/or populations and included patients with clinical conditions (e.g., Alzheimer's disease, Parkinson's disease, stroke, and movement disorders) which have high comorbidity with stress and anxiety (Factor et al., 1995; Smith et al., 2000; Jenner, 2003; Allen and Bayraktutan, 2009). Also, the rates of self-induced VRISE are expected to be equal in both new and old generation HMD studies. In addition, the reporting of VRISE may be for reasons unrelated to the quality of the hardware or software (e.g., subjectivity in the reporting of VRISE, individual differences in the experience of VRISE) (Kortum and Peres, 2014; Almeida et al., 2017). However, this modulation is again expected to have affected both new and old generation HMD studies in a similar way.

Beyond the difference between old and new generations HMDs, a substantial difference is observed between DK and CV new generation HMDs. Significantly fewer VRISE were present in the studies that used a CV, indicating the superiority of new generation CV HMDs compared to new generation DK HMDs. Furthermore, the studies (i.e., both old and new generation studies) which utilized VR software with ergonomic interactions had robustly less frequent VRISE and dropouts than the studies which implemented VR software with non-ergonomic interactions. However, the ergonomic interactions do not appear to mitigate the dropout frequency and the incidence of VRISE in old generation HMDs. In contrast, VRISE were present in more DK studies with non-ergonomic interactions compared to DK studies with ergonomic interactions. Similarly, more participants dropped out from DK studies with non-ergonomic interactions. Notably, there were no VRISE or dropouts in CV studies with ergonomic interactions. Therefore, the contribution of ergonomic interactions in the reduction of VRISE increases when newer and better HMDs are utilized. To conclude, the findings of the meta-analysis are aligned with the outcomes of the technological review.

## GENERAL DISCUSSION

### Technological Competence in VR Neuroscience and Neuropsychology

The findings of our technological literature review suggest that the hardware features of old generation HMDs and new generation DKs do not meet the minimum hardware features that alleviate or eradicate VRISE. Instead, the technological
literature review postulates the suitability of new generation CVs which have specific hardware capabilities to alleviate VRISE. However, VR software attributes (e.g., ergonomic interactions) are equally vital.

Secondly, the findings of our meta-analysis of 44 neuroscientific or neuropsychological studies using VR are aligned with the outcomes of our technological review, where VRISE were substantially less frequent in studies which utilized new generation VR HMDs. In particular, the studies which used new generation CVs accompanied by ergonomic interactions did not have any VRISE or dropouts. Therefore, the combined outcomes of the technological review and the meta-analysis indicate that the appropriate VR HMDs are those with hardware characteristics equal to or greater than the HTC Vive and Oculus Rift, though the VR HMD should be implemented in conjunction with VR software which offers ergonomic interactions.

However, researchers may have to opt for an HMD based on their available budget. For example, the Oculus Rift costs around $400, while the HTC Vive costs around $500. Moreover, the majority of HMDs also require a VR-ready desktop PC or a laptop to be operated, so a researcher needs to additionally spend around $500–$1,500 for a desktop computer or laptop to utilize these HMDs. Hence, the combined cost would be between $800 and $1,900. The cost of VR equipment (e.g., both HMD and computer) may lead researchers to use HMDs that are cheaper, albeit that they are more likely to result in VRISE. However, in the market, there are plenty of cost-effective alternatives that meet the minimum hardware criteria. For example, the Oculus Quest is a standalone HMD (i.e., it does not require a PC, a laptop, or a smartphone to be operated) and it costs approximately $400. Hence, a researcher can spend the equivalent of the price of a neuropsychological test or a smartphone to acquire and use an HMD that meets the minimum hardware criteria to lower the presence of VRISE.

Nonetheless, the selection of an appropriate VR HMD and software requires technological competency from the researchers, clinicians, and/or research software developers. Unfortunately, the meta-analysis results do not indicate that technological knowledge of VR has been well-established in neuroscience. Of course, the utilization of old generation HMDs and new generation DKs pre-2016 is justified as the new generation CVs were not available. However, in our meta-analysis, 25 studies were conducted between 2016 and 2019, where half of these studies (13/25) implemented an inappropriate HMD (i.e., old generation HMD or new generation DK). However, 10 studies used a DK2 which has a marginally lower resolution than the minimum hardware criteria, while our meta-analysis results indicated that its utilization in conjunction with ergonomic interactions appears to alleviate the frequency of VRISE, but not as effectively as the CV HMDs. Furthermore, one fifth of the studies did use a new generation HMD, but they did not have ergonomic interactions in their VR software. Therefore, at this time, VR technological competence does not seem to have been well-established in neuroscience. As a result, in the studies since 2016, the health and safety of the participants may not substantially guaranteed, and the reliability of the results may be

questionable, as VRISE substantially decreases reaction times and overall cognitive performance (Plant and Turner, 2009; Nalivaiko et al., 2015; Plant, 2016; Nesbitt et al., 2017; Mittelstaedt et al., 2018), as well as confounding neuroimaging and physiological data (Toschi et al., 2017; Arafat et al., 2018; Gavgani et al., 2018). The selection of an appropriate HMD is paramount for successfully implementing VR HMDs in cognitive neuroscience and neuropsychology.

However, the implementation of the currently available and appropriate HMDs in neuroscience and neuropsychology should be compatible with the research aims. For example, in research designs where the user should be active (i.e., navigating, walking, and interacting within the VE) instead of being idle, or in a standing or a seated position, the researcher should opt for the best HMD that permits intense body movement and activity. In this setting, the Oculus Rift was found to be inferior to the HTC Vive on pick-and-place (i.e., relocating objects) tasks, whilst the HTC Vive also provided a substantially superior VR experience for users compared to the Oculus Rift (Suznjevic et al., 2017). Moreover, the HTC Vive provides an interactive area that is twice the size (25 m²) of the Oculus Rift, albeit that both are very accurate in tracking (Borrego et al., 2018). Nevertheless, the HTC Vive was found to lose motion-tracking and the ground level becomes slanted when the user goes out of bounds (Niehorster et al., 2017). This shortcoming solely affects studies where the participant needs to go out of the tracking area. In most neuroscientific designs, the recommended maximum play area by HTC (6.25 m²) or by Borrego et al. (2018) (25 m²) are both substantially adequate for conducting ecological valid experiments (Borges et al., 2018; Borrego et al., 2018). Nonetheless, the slanted floor or lost tracking is not a hardware problem but a software one (Borges et al., 2018). In cases where the participant is required to go out of the tracking area, the tracking problem or the slanted floor may be easily corrected by adding 3 additional trackers (Peer et al., 2018), using software with an improved algorithm (freely distributed by NASA Ames Research Center) (Borges et al., 2018), or by simply updating the firmware of the lighthouse base stations. In summary, the researchers should be technologically competent to not only identify and implement a safe HMD and software, but an HMD and software that facilitate the optimal research methods pertinent to their research needs and aims.

As discussed in our technological review, the quality of the implemented VR software is equally important to avoid VRISE. Our meta-analysis of VR studies indicated that the utilization of ergonomic interactions is crucial albeit with the utilization of an appropriate HMD. For example, Detez et al. (2019) used the HTC Vive to investigate physiological arousal and behavior during gambling. However, the interactions and navigation within the VE were facilitated by using a typical controller (Detez et al., 2019). Hence, their VR software did not support the utilization of the ergonomic 6DoF controllers (both hands) of the HTC Vive, which facilitate naturalistic navigation (e.g., teleportation) and interaction within the VE. Consequently, Detez et al.'s (2019) participants experienced VRISE and 3 of their participants discontinued their sessions and so their data were discarded (Detez et al., 2019). Importantly, Detez et al. (2019) only reported
the presence of VRISE and dropout size. They did not provide any quantitative data on the intensity of VRISE, or the quality of their software attributes (e.g., graphics, sound, tutorials, in-game instructions and prompts) (Detez et al., 2019). Indeed, only six of the studies in the meta-analysis provided adequately explicit reports on VRISE and VR software. Since Detez et al. (2019) assessed reaction times and heart rates, these data are likely to be affected by VRISE, despite the study having a rigorous experimental design and using the HTC Vive. Therefore, it is important to use appropriate VR software and external hardware to prevent risks to the health and safety of the participants as well as the reliability of the results.

## Limitations and Future Studies

The above technological review and meta-analysis of VR studies evidenced the importance of technological and methodological features in VR research and clinical designs. However, our meta-analysis of VR studies has some limitations. The meta-analysis considered VR studies with diverse populations and designs, which may have affected the frequency of VRISE and the existence of dropouts. Uniformity across studies (e.g., considering only VRET, assessments or a specific clinical population) was not possible due to the scarcity of neuroscience studies involving VR, especially using new generation HMDs. Moreover, the review did not consider any software details due to the scarcity of such descriptions in published studies. Future VR studies should report software and hardware features to allow an in-depth meta-analysis. Equally, only six studies provided quantitative reports of VRISE intensity, consequently, only the presence or absence of VRISE was considered. The dichotomous consideration of VRISE is susceptible to reports based on subjective criteria and individual differences, but this is likely to have affected the VRISE rates in both old and new generation studies. Future studies should aim to appraise the quality of the software and intensity of VRISE (e.g., using questionnaires). Studies should also attempt to clarify the acceptable duration of immersive VR sessions, which will aid researchers in designing their studies appropriately. Importantly, the cost of the VR software development should also be considered. Finally, studies should attempt to provide software development guidelines that enable researchers and/or research software developers

to develop VR research software without depending on third parties (e.g., freelance developers or software development companies) and these guidelines should embed suggestions and instructions for VR software development, which meet the criteria discussed above.

## Conclusion

The use of VR HMDs is becoming more popular in neuroscience either for clinical or research purposes and VR technology and methods have been well-accepted by diverse populations in terms of age groups and clinical conditions. A more pleasant VR experience and a reduction in VRISE symptomatology has been found using new generation CV HMDs, which deliver an adequately high display resolution, rapid image refresh rate, ergonomic design and has controllers which allow naturalistic navigation and movement within the VE environment, especially when there is restricted teleportation. The outcomes of the current technological review and meta-analysis support the feasibility of new generation VR CV HMDs to be implemented in cognitive neuroscience and neuropsychology. The findings of the technological review suggest methods that should be considered in the development or selection of VR research software, as well as hardware and software features that should be included in the research protocol. The selected VR HMD and the VR research software should enable suitable ergonomic interactions, locomotion techniques (e.g., teleportation), and kinetic mechanics which ensure VRISE are reduced or completely avoided. A meticulous approach and technological competence are compulsory to consolidate the viability of VR research and clinical designs in cognitive neuroscience and neuropsychology.

## DATA AVAILABILITY STATEMENT

All datasets generated for this study are included in the manuscript/supplementary files.

## AUTHOR CONTRIBUTIONS

PK had the initial idea and contributed to every aspect of this study. SC, LD, and SM contributed to the methodological aspects and the discussion of the results.

## REFERENCES

* Allen, C. L., and Bayraktutan, U. (2009). Oxidative stress and its role in the pathogenesis of ischaemic stroke. *Int. J. Stroke* 4, 461–470. doi: 10.1111/j.1747-4949.2009.00387.x

* Almeida, A., Rebelo, F., Noriega, P., and Vilar, E. (2017). “Virtual reality self induced cybersickness: an exploratory study,” in *International Conference on Applied Human Factors and Ergonomics* (Cham: Springer), 26–33. doi: 10.1007/978-3-319-60582-1_3

* Anthes, C., García-Hernández, R. J., Wiedemann, M., and Kranzlmüller, D. (2016). “State of the art of virtual reality technology,” in *2016 IEEE Aerospace Conference (Big Sky, MT)*, 1–19. doi: 10.1109/AERO.2016.7500674

* Arafat, I. M., Ferdous, S. M. S., and Quarles, J. (2018). “Cybersickness-provoking virtual reality alters brain signals of persons with multiple sclerosis.” in *2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)* (Reutlingen: IEEE), 1–120. doi: 10.1109/VR.2018.8446194

* Banakou, D., Kishore, S., and Slater, M. (2018). Virtually being Einstein results in an improvement in cognitive Task performance and a decrease in age bias. *Front. Psychol.* 9:917. doi: 10.3389/fpsyg.2018.00917

* Banville, F., Nolin, P., Lalonde, S., Henry, M., Dery, M. P., and Villemure, R. (2010). Multitasking and prospective memory: can virtual reality be useful for diagnosis? *Behav. Neurol.* 23, 209–211. doi: 10.1155/2010/320707

* Barberia, I., Oliva, R., Bourdin, P., and Slater, M. (2018). Virtual mortality and near-death experience after a prolonged exposure in a shared virtual reality may lead to positive life-attitude changes. *PLoS ONE* 13:e0203358. doi: 10.1371/journal.pone.0203358

* Bioulac, S., Lallemand, S., Rizzo, A., Philip, P., Fabrigoule, C., and Bouvard, M. P. (2012). Impact of time on task on ADHD patient’s performances in a virtual classroom. *Eur. J. Paediatr. Neurol.* 16, 514–521. doi: 10.1016/j.ejpn.2012.01.006

* Bland, M. (2015). *An Introduction to Medical Statistics, 4th Edn.* Oxford: Oxford University Press.
Bohil, C. J., Alicea, B., and Biocca, F. A. (2011). Virtual reality in neuroscience research and therapy. *Nat. Rev. Neurosci.* 12:752. doi: 10.1038/nnr3122

Borges, M., Symington, A., Coltin, B., Smith, T., and Ventura, R. (2018). "HTC Vive: analysis and accuracy improvement," in *2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)* (Madrid: IEEE), 2610-2615. doi: 10.1109/IROS.2018.8593707

Borrego, A., Latorre, J., Alcaniz, M., and Llorens, R. (2018). Comparison of oculus rift and HTC vive: feasibility for virtual reality-based exploration, navigation, exergaming, and rehabilitation. *Games Health J.* 7, 151–156. doi: 10.1089/g4h.2017.0114

Botella, C., García-Palacios, A., Villa, H., Baños, R. M., Quero, S., Alcañiz, M., et al. (2007). Virtual reality exposure in the treatment of panic disorder and agoraphobia: a controlled study. *Clin. Psychol. Psychother.* 14, 164–175. doi: 10.1002/cpp.524

Bouchard, S., Robillard, G., Renaud, P., and Bernier, F. (2011). Exploring new dimensions in the assessment of virtual reality induced side effects. *J. Comput. Inform. Technol.* 1, 20–32. Available online at: http://pubs.drdc-rddc.gc.ca/BASIS/pcandid/www/engpub/DDW?W%3DSYSNUM=536530&tr=0

Bourdin, P., Barberia, I., Oliva, R., and Slater, M. (2017). A virtual out-of-body experience reduces fear of death. *PLoS ONE* 12:e0169343. doi: 10.1371/journal.pone.0169343

Bozgeyikli, E., Raij, A., Katkoori, S., and Dubey, R. (2016). "Point and teleport locomotion technique for virtual reality," in *Proceedings of the 2016 Annual Symposium on Computer-Human Interaction in Play* (Austin, TX: ACM), 205–216. doi: 10.1145/2967934.2968105

Brade, J., Dudczig, M., and Klimant, P. (2018). "Using virtual prototyping technologies to evaluate human-machine-interaction concepts," in *Aw&I Conference* (Chemnitz), 3.

Brennesholtz, M. S. (2018). "3-1: Invited Paper: VR standards and guidelines," in *SID Symposium Digest of Technical Papers*, 49 (Pleasantville, NY), 1–4. doi: 10.1002/sdtp.12476

Caputo, F., Greco, A., Egidio, D. A., Notaro, I., and Spada, S. (2017). "A preventive ergonomic approach based on virtual and immersive reality," in *International Conference on Applied Human Factors and Ergonomics* (Cham: Springer), 3–15. doi: 10.1007/978-3-319-60582-1_1

Carlozzi, N. E., Gade, V., Rizzo, A. S., and Tulsky, D. S. (2013). Using virtual reality driving simulators in persons with spinal cord injury: three screen display versus head mounted display. *Disabil. Rehabil.* 8, 176–180. doi: 10.3109/17483107.2012.699990

Christou, C. G., Michael-Grigoriou, D., Sokratous, D., and Tsiakoulia, M. (2018). "A virtual reality loop and wire game for stroke rehabilitation," in *28th International Conference on Artificial Life and 23rd Telexistence and Eurographics Symposium on Virtual Environments* (Limassol).

Collins, M. K., Ding, V. Y., Ball, R. L., Dolce, D. L., Henderson, J. M., and Halpern, C. H. (2018). Novel application of virtual reality in patient engagement for deep brain stimulation: a pilot study. *Brain Stimul.* 11, 935–937. doi: 10.1016/j.brs.2018.03.012

Costello, P. J. (1997). *Health and Safety Issues Associated With Virtual Reality: A Review of Current Literature*. Loughborough: Advisory Group on Computer Graphics.

Cox, D. R., and Donnelly, C. A. (2011). *Principles of Applied Statistics*. Cambridge: Cambridge University Press

Davis, S., Nesbitt, K., and Nalivaiko, E. (2015). "Comparing the onset of cybersickness using the Oculus Rift and two virtual roller coasters," in *Proceedings of the 11th Australasian Conference on Interactive Entertainment (IE 2015)*, 27 (Sydney), 30.

de França, A. C. P., and Soares, M. M. (2017). "Review of virtual reality technology: an ergonomic approach and current challenges," in *International Conference on Applied Human Factors and Ergonomics* (Cham: Springer), 52–61. doi: 10.1007/978-3-319-60582-1_6

Detez, L., Greenwood, L. M., Segrave, R., Wilson, E., Chandler, T., Ries, T., et al. (2019). A psychophysiological and behavioural study of slot machine near-misses using immersive virtual reality. *J. Gambl. Stud.* 35, 929–944. doi: 10.1007/s10899-018-09822-z

Factor, S. A., Podskalny, G. D., and Molho, E. S. (1995). Psychogenic movement disorders: frequency, clinical profile, and characteristics. *J. Neurol. Neurosurg. Psychiatry* 59, 406–412. doi: 10.1136/jnnp.59.4.406

Figueiredo, L., Rodrigues, E., Teixeira, J., and Techrieb, V. (2018). A comparative evaluation of direct hand and wand interactions on consumer devices. *Comput. Graph.* 77, 108–121. doi: 10.1016/j.cag.2018.10.006

Foerster, R. M., Poth, C. H., Behler, C., Botsch, M., and Schneider, W. X. (2016). Using the virtual reality device Oculus Rift for neuropsychological assessment of visual processing capabilities. *Sci. Rep.* 6:e37016. doi: 10.1038/srep37016

Freeman, D., Bradley, J., Antley, A., Bourke, E., DeWeever, N., Evans, N., et al. (2016). Virtual reality in the treatment of persecutory delusions: randomised controlled experimental study testing how to reduce delusional conviction. *Br. J. Psychiatry* 209, 62–67. doi: 10.1192/bjp.bp.115.176438

Freeman, D., Evans, N., Lister, R., Antley, A., Dunn, G., and Slater, M. (2014). Height, social comparison, and paranoia: an immersive virtual reality experimental study. *Psychiatry Res.* 218, 348–352. doi: 10.1016/j.psychres.2013.12.014

Frommel, J., Sonntag, S., and Weber, M. (2017). "Effects of controller-based locomotion on player experience in a virtual reality exploration game," in *Proceedings of the 12th International Conference on the Foundations of Digital Games* (Hyannis, MA: ACM), 30. doi: 10.1145/3102071.3102082

Gaggioli, A., Pallavicini, F., Morganti, L., Serino, S., Scaratti, C., Briguglio, M., et al. (2014). Experiential virtual scenarios with real-time monitoring (interreality) for the management of psychological stress: a block randomized controlled trial. *J. Med. Internet Res.* 16:e167. doi: 10.2196/jmir.3235

Gavgani, A. M., Wong, R. H., Howe, P. R., Hodgson, D. M., Walker, F. R., and Nalivaiko, E. (2018). Cybersickness-related changes in brain hemodynamics: a pilot study comparing transcranial Doppler and near-infrared spectroscopy assessments during a virtual ride on a roller coaster. *Physiol. Behav.* 191, 56–64. doi: 10.1016/j.physbeh.2018.04.007

George, C., Demmler, M., and Hussmann, H. (2018). "Intelligent interruptions for IVR: investigating the interplay between presence, workload and attention," in *Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems* (Montreal, QC: ACM), LBW511. doi: 10.1145/3170427.3188686

Gómez-Jordana, L. I., Stafford, J., Peper, C. L. E., and Craig, C. M. (2018). Virtual footprints can improve walking performance in people with Parkinson's disease. *Front. Neurol.* 9:681. doi: 10.3389/fneur.2018.00681

Goradia, I., Doshi, J., and Kurup, L. (2014). A review paper on oculus rift and project morpheus. *Int. J. Curr. Eng. Technol.* 4, 3196–3200. Available online at: http://inpressco.com/wp-content/uploads/2014/09/Paper183196-3200.pdf

Gromala, D., Tong, X., Shaw, C., Amin, A., Ulas, S., and Ramsay, G. (2016). Mobius floe: an immersive virtual reality game for pain distraction. *Electron. Imaging* 2016, 1–5. doi: 10.2352/ISSN.2470-1173.2016.4.ERVR-413

Hartanto, D., Kampmann, I. L., Morina, N., Emmelkamp, P. G., Neerincx, M. A., and Brinkman, W. P. (2014). Controlling social stress in virtual reality environments. *PLoS ONE* 9:e92804. doi: 10.1371/journal.pone.0092804

Hasler, B. S., Spanlang, B., and Slater, M. (2017). Virtual race transformation reverses racial in-group bias. *PLoS ONE* 12:e0174965. doi: 10.1371/journal.pone.0174965

Hecht, J. (1997). Optical dreams, virtual reality. *Optics Photon. News* 27, 24–31. doi: 10.1364/OPN.27.6.000024

Held, L., and Ott, M. (2018). On p-values and Bayes factors. *Annu. Rev. Stat. Appl.* 5, 393–419. doi: 10.1146/annurev-statistics-031017-100307

Hsieh, T. J. T., Kuo, Y. H., and Niu, C. K. (2018). "Utilizing HMD VR to improve the spatial learning and wayfinding effects in the virtual maze," in *International Conference on Human-Computer Interaction* (Cham: Springer), 38–42. doi: 10.1007/978-3-319-92285-0_6

JASP Team (2018). JASP (Version 0.9) [*Computer Software*].

Jenner, P. (2003). Oxidative stress in Parkinson's disease. *Ann. Neurol.* 53, S26–S38. doi: 10.1002/ana.10483

Jerald, J., LaViola, J. J. Jr., and Marks, R. (2017). "VR interactions," in *ACM SIGGRAPH 2017 Courses* (Los Angeles, CA: ACM), 19. doi: 10.1145/3084873.3084900

Kane, R. L., and Parsons, T. D. (eds.). (2017). *The Role of Technology in Clinical Neuropsychology*. New York, NY: Oxford University Press.

Kelly, J. W., Cherep, L. A., and Siegel, Z. D. (2017). Perceived space in the HTC vive. *ACM Trans. Appl. Percept.* 15:2. doi: 10.1145/3106155

Kennedy, R. S., Lane, N. E., Berbaum, K. S., and Lilienthal, M. G. (1993). Simulator sickness questionnaire: an enhanced method for quantifying
simulator sickness. *Int. J. Aviation Psychol.* 3, 203–220. doi: 10.1207/s15327108ijap0303_3

Kim, A., Darakjian, N., and Finley, J. M. (2017). Walking in fully immersive virtual environments: an evaluation of potential adverse effects in older adults and individuals with Parkinson's disease. *J. Neuroeng. Rehabil.* 14:16. doi: 10.1186/s12984-017-0225-2

Kim, H. K., Park, J., Choi, Y., and Choe, M. (2018). Virtual reality sickness questionnaire (VRSQ): motion sickness measurement index in a virtual reality environment. *Appl. Ergon.* 69, 66–73. doi: 10.1016/j.apergo.2017.12.016

Kim, J. W., Choe, W. J., Hwang, K. H., and Kwag, J. O. (2017). “78-2: the optimum display for virtual reality,” in *SID Symposium Digest of Technical Papers*, 48, 1146–1149. doi: 10.1002/sdtp.11845

Kim, K., Kim, J., Ku, J., Kim, D. Y., Chang, W. H., Shin, D. I., et al. (2004). A virtual reality assessment and training system for unilateral neglect. *Cyberpsychol. Behav.* 7, 742–749. doi: 10.1089/cpb.2004.7.742

Kortum, P., and Peres, S. C. (2014). The relationship between system effectiveness and subjective usability scores using the System Usability Scale. *Int. J. Human Comput. Int.* 30, 575–584. doi: 10.1080/10447318.2014.904177

Lubetzky, A. V., Kary, E. E., Harel, D., Hujsak, B., and Perlin, K. (2018). Feasibility and reliability of a virtual reality oculus platform to measure sensory integration for postural control in young adults. *Physiother. Theory Pract.* 34, 935–950. doi: 10.1080/09593985.2018.1431344

Marsman, M., and Wagenmakers, E. J. (2017). Bayesian benefits with JASP. *Eur. J. Dev. Psychol.* 14, 545–555. doi: 10.1080/17405629.2016.1259614

Matheis, R. J., Schultheis, M. T., Tiersky, L. A., DeLuca, J., Millis, S. R., and Rizzo, A. (2007). Is learning and memory different in a virtual environment? *Clin. Neuropsychol.* 21, 146–161. doi: 10.1080/13854040601100668

Meyerbroeker, K., Morina, N., Kerkhof, G. A., and Emmelkamp, P. M. G. (2013). Virtual reality exposure therapy does not provide any additional value in agoraphobic patients: a randomized controlled trial. *Psychother. Psychosom.* 82, 170–176. doi: 10.1159/000342715

Mittelstaedt, J. M., Wacker, J., and Stelling, D. (2018). VR aftereffect and the relation of cybersickness and cognitive performance. *Virtual Real.* 23, 143–154. doi: 10.1007/s10055-018-0370-3

Montenegro, J. M. F., and Argyriou, V. (2017). Cognitive evaluation for the diagnosis of Alzheimer's disease based on turing test and virtual environments. *Physiol. Behav.* 173, 42–51. doi: 10.1016/j.physbeh.2017.01.034

Moreau, G., Guay, M. C., Achim, A., Rizzo, A., and Lageix, P. (2006). The virtual classroom: an ecological version of the continuous performance test–A pilot study. *Annu. Rev. CyberTherapy Telemed.* 4, 59–66. doi: 10.1037/e695432011-008

Mottelson, A., and Hornbæk, K. (2017). “Virtual reality studies outside the laboratory,” in *Proceedings of the 23rd ACM Symposium on Virtual Reality Software and Technology* (Gothenburg: ACM), 9. doi: 10.1145/3139131.3139141

Nalivaiko, E., Davis, S. L., Blackmore, K. L., Vakulin, A., and Nesbitt, K. V. (2015). Cybersickness provoked by head-mounted display affects cutaneous vascular tone, heart rate and reaction time. *Physiol. Behav.* 151, 583–590. doi: 10.1016/j.physbeh.2015.08.043

Nesbitt, K., Davis, S., Blackmore, K., and Nalivaiko, E. (2017). Correlating reaction time and nausea measures with traditional measures of cybersickness. *Displays* 48, 1–8. doi: 10.1016/j.displa.2017.01.002

Niehorster, D. C., Li, L., and Lappe, M. (2017). The accuracy and precision of position and orientation tracking in the HTC vive virtual reality system for scientific research. *i-Perception* 8:2041669517708205. doi: 10.1177/2041669517708205

Oagaz, H., Schoun, B., Pooji, M., and Choi, M. H. (2018). Neurocognitive assessment in virtual reality through behavioral response analysis. *IEEE J. Biomed. Health Inform.* 23, 1899–1910. doi: 10.1109/JBHI.2018.2881455

Palmisano, S., Mursic, R., and Kim, J. (2017). Vection and cybersickness generated by head-and-display motion in the Oculus Rift. *Displays* 46, 1–8. doi: 10.1016/j.displa.2016.11.001

Parsons, T. D. (2015). Virtual reality for enhanced ecological validity and experimental control in the clinical, affective and social neurosciences. *Front. Human Neurosci.* 9:660. doi: 10.3389/fnhum.2015.00660

Parsons, T. D., and Barnett, M. (2017). Virtual Apartment-Based Stroop for assessing distractor inhibition in healthy aging. *Appl. Neuropsychol. Adult* 26, 144–154. doi: 10.1080/23279095.2017.1373281

Parsons, T. D., and Carlew, A. R. (2016). Bimodal virtual reality stroop for assessing distractor inhibition in autism spectrum disorders. *J. Autism Dev. Disord.* 46, 1255–1267. doi: 10.1007/s10803-015-2663-7

Parsons, T. D., Courtney, C. G., and Dawson, M. E. (2013). Virtual reality Stroop task for assessment of supervisory attentional processing. *J. Clin. Exp. Neuropsychol.* 35, 812–826. doi: 10.1080/13803395.2013.824556

Parsons, T. D., and McMahan, T. (2017). An initial validation of the virtual of the virtual environment grocery store. *J. Neurosci. Methods* 291, 13–19. doi: 10.1016/j.jneumeth.2017.07.027

Parsons, T. D., McMahan, T., and Kane, R. (2018). Practice parameters facilitating adoption of advanced technologies for enhancing neuropsychological assessment paradigms. *Clin. Neuropsychol.* 32, 16–41. doi: 10.1080/13854046.2017.1337932

Parsons, T. D., Rizzo, A. A., Bamattre, J., and Brennan, J. (2007). Virtual reality cognitive performance assessment test. *Annu. Rev. CyberTherapy Telemed.* 5, 163–171. doi: 10.1037/e688982007-001

Peck, T. C., Seinfeld, S., Aglioti, S. M., and Slater, M. (2013). Putting yourself in the skin of a black avatar reduces implicit racial bias. *Conscious. Cogn.* 22, 779–787. doi: 10.1016/j.concog.2013.04.016

Peer, A., Ullich, P., and Ponto, K. (2018). “Vive tracking alignment and correction made easy,” in *2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)* (Reutlingen: IEEE), 653–654. doi: 10.1109/VR.2018.8446435

Plant, R. R. (2016). A reminder on millisecond timing accuracy and potential replication failure in computer-based psychology experiments: an open letter. *Behav. Res. Methods* 48, 408–411. doi: 10.3758/s13428-015-0577-0

Plant, R. R., and Turner, G. (2009). Millisecond precision psychological research in a world of commodity computers: new hardware, new problems? *Behav. Res. Methods* 41, 598–614. doi: 10.3758/BRM.41.3.598

Plouzeau, J., Paillot, D., Chardonnet, J. R., and Merienne, F. (2015). “Effect of proprioceptive vibrations on simulator sickness during navigation task in virtual environment,” in *International Conference on Artificial Reality and Teleexistence* (Kyoto).

Porcino, T. M., Clua, E., Trevisan, D., Vasconcelos, C. N., and Valente, L. (2017). “Minimizing cyber sickness in head mounted display systems: design guidelines and applications,” in *2017 IEEE 5th International Conference on Serious Games and Applications for Health (SeGAH)* (Perth, WA: IEEE), 1–6. doi: 10.1109/SeGAH.2017.7939283

Quinlivan, B., Butler, J. S., Beiser, I., Williams, L., McGovern, E., O'Riordan, S., et al. (2016). Application of virtual reality head mounted display for investigation of movement: a novel effect of orientation of attention. *J. Neural Eng.* 13:056006. doi: 10.1088/1741-2560/13/5/056006

Rakkolainen, I., Turk, M., and Höllerer, T. (2016). “A superwide-FOV optical design for head-mounted displays,” in *Proceedings of the Joint 26th International Conference on Artificial Reality and Teleexistence and the 21st Eurographics Symp. on Virtual Environments (ICAT-EGVE 2016)* (Little Rock, AR), 45–48.

Reger, G. M., Holloway, K. M., Candy, C., Rothbaum, B. O., Dife de, J., Rizzo, A. A., et al. (2011). Effectiveness of virtual reality exposure therapy for active duty soldiers in a military mental health clinic. *J. Traumat. Stress* 24, 93–96. doi: 10.1002/jts.20574

Rizzo, A., Dife de, J., Rothbaum, B. O., Reger, G., Spitalnick, J., Cukor, J., et al. (2010). Development and early evaluation of the Virtual Iraq/Afghanistan exposure therapy system for combat-related PTSD. *Ann. N. Y. Acad. Sci.* 1208, 114–125. doi: 10.1111/j.1749-6632.2010.05755.x

Rizzo, A. A., Schultheis, M., Kerns, K. A., and Mateer, C. (2004). Analysis of assets for virtual reality applications in neuropsychology. *Neuropsychol. Rehabil.* 14, 207–239. doi: 10.1080/09602010343000183

Rooney, B., Burke, C., Bálint, K., O'Leary, T., Parsons, T., Lee, C. T., et al. (2017). “Virtual reality, presence and social cognition: the effect of eyegaze and narrativity on character engagement,” in *2017 23rd International Conference on Virtual System and Multimedia (VSMM)* (Dublin: IEEE), 1–6. doi: 10.1109/VSMM.2017.8346272

Rothbaum, B. O., Price, M., Jovanovic, T., Norrholm, S. D., Gerardi, M., Dunlop, B., et al. (2014). A randomized, double-blind evaluation of D-cycloserine or alprazolam combined with virtual reality exposure therapy for posttraumatic