Title: Adventures in mirror world: Uncovering the cognitive and sensory basis for natural behaviour in virtual reality
Funder: Economic and Social Research Council

Abstract: Our proposal has four main objectives: 1) Advancing the theoretical knowledge about the interrelationship of cognitive, perceptual, and visuo-motor processes in sensorimotor decision-making 2) Evaluation of the (external) validity of virtual reality (VR) environments for training and investigating skilled human actions 3) Exploration of strategies to compensate for the limitations of VR environments 4) Demonstrating the potential of our methodological approach that simulates fundamental constraints of VR environments independent of changes in perceptual complexity by using mirrors (1) The features of virtual environments whose effects on behaviour we propose to investigate (i.e., reduction of visual complexity, virtuality of objects and virtuality of movements) directly influence factors that have been identified as important determinants of goal-directed motor behaviour in the fields of sensorimotor decision-making (i.e., visual uncertainty, consequences of actions and biomechanical effort). Therefore, each of our experiments investigating features of VR simultaneously addresses research questions that are highly relevant to the sensorimotor decision-making and motor control literature and will make important contributions to the knowledge in these fields. (2) While isolated perceptual aspects of VR devices have been studied extensively (e.g., the role of cues to depth), we propose an innovative research programme that comprehensively evaluates perceptual and cognitive aspects of virtual environments. Specifically, we will investigate how the key features inherent to virtual environments (i.e., reduction of visual complexity, virtuality of objects and virtuality of movements) jointly influence actions using highly ecologically valid experimental tasks (i.e., obstacle avoidance and grasping). Crucially, this also includes evaluating the hypothesis that the knowledge alone that a given environment is virtual, rather than real, is sufficient to alter motor behaviour. The overall goal is to uncover the sensory and cognitive basis for natural behaviour in VR and to provide insights into the nature and magnitude of the differences between actions performed in natural and (perceptually matched) virtual settings. (3) Some of the problems of current VR devices, such as missing or ambiguous implementations of cues to depth, might be resolved by technological advances; other aspects, such as the virtuality of objects and movements, are fundamental and unavoidable consequences of an environment being "virtual". We aim to identify and test the efficacy of tools and methods that can compensate for such inherent differences between virtual and natural environments to enable natural behaviour in VR. Implementing compensatory techniques will also help us to understand and define the upper limit on how much correspondence can be achieved between the two settings. (4) The method we employ in this proposal simulates VR environments using mirrors. This allows us to directly compare behaviour in natural and virtual conditions independent of the limitations of currently available VR devices. Specifically, using mirrors means that we can simulate the key features of VR environments that affect behaviour (i.e., virtuality of objects and movements) independent of the confounding effects of reduced perceptual complexity - which is impossible with currently available VR devices. We believe this method can serve as a general benchmark for the degree of correspondence that can theoretically be achieved between behaviour in natural and virtual environments. Our final objective is therefore to demonstrate the utility of mirrors as a standard to which VR devices can be compared, and for the investigation of motor behaviour in general.

Lay abstract: Besides their obvious value for entertainment, VR devices are becoming increasingly attractive for the systematic experimental investigation of human perception and behaviour because they promise the possibility of experimental control over environments that approach the complexity of natural settings. If we investigate behaviour or train certain real-world tasks (e.g., in sports or medical training) with the help of VR devices, we expect that the insights we gain or the skills we acquire transfer to real-world settings. Yet, it is currently still unclear if and to what degree behaviour in virtual and natural environments is comparable. One reason for differences between behaviour in natural and virtual environments is that currently available VR devices simulate environments that are perceptually less complex than their natural counterparts and lack many sources of information that we use to plan and execute our actions in the real world (e.g., missing or inappropriately implemented cues to depth). However, perceptual limitations of currently available VR devices may not be the only source of differences in behaviour. What has so far largely been neglected is how cognitive factors shape behaviour in VR. Particularly, the awareness of interacting with virtual objects can change expectations about the consequence of our actions. For example, while we have clear expectations regarding the immediate consequences of collisions with real objects (e.g., pain or injury and/or damage to the object), collisions with virtual objects usually do not have such direct consequences. Thus, even if VR was perceptually perfect, as long as people are cognisant of its virtual nature, the cognitive effects are likely to alter the way we act in VR (e.g., how cautiously we move). It is therefore essential to understand the exact nature of these cognitive effects. Another feature of many VR applications that might affect our actions is that users often do not directly interact with real or virtual objects but instead use some device (e.g., a joystick) to manipulate them. In that case, the relation between biomechanical effort and resulting movement differs from their natural relation. This is crucial as we know that natural movements are shaped by biomechanical demands and the biomechanical constraints of our body. What makes the systematic investigation of the influence of these factors difficult is that due to the limitations of currently available VR devices, it is impossible to disentangle their effects. For example, the effects of the reduced perceptual complexity on movements cannot be easily separated from those resulting from the knowledge of interacting with virtual objects. Here, we propose a method that allows us to directly compare behaviour in natural and virtual environments independent of the limitations of currently available VR devices. We will simulate a virtual environment by using a powerful but in this kind of research underutilised tool: mirrors. Mirrors are simple and easy to use, yet they are visually the most complete and convincing virtual reality device available. Using mirrors to simulate VR has the advantage that the mirrored environment provides the same visual information as the corresponding natural environment. Therefore, we can investigate behaviour in virtual environments while being able to control the confounding effects of reduced perceptual complexity which is impossible with currently available VR devices. With this simulated virtual reality, we will systematically investigate the three features of VR influencing behaviour (i.e., reduction of perceptual complexity, virtuality of objects and virtuality of movements) using ecologically valid obstacle avoidance and grasping tasks. Our goal is to provide the scientific basis for implementations of VR that are useful for the investigation of human actions and advance knowledge on the interrelationship of perceptual, cognitive and visuomotor processes.

Research activity codes: 1.2 PSYCHOLOGICAL AND SOCIOECONOMIC PROCESSES
Health categories: MENTAL HEALTH