# Exploring Human Perception Using Virtual Reality

Katja Zibrek

► To cite this version:

Katja Zibrek. Exploring Human Perception Using Virtual Reality. Cognitive Science 2024, Oct 2024, Ljubljana, Slovenia. pp.1-4, 10.70314/is.2024.cog.12. hal-04871624

HAL Id: hal-04871624

https://inria.hal.science/hal-04871624v1

Submitted on 7 Jan 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Exploring Human Perception Using Virtual Reality

Katja Zibrek

katja.zibrek@inria.fr

Inria centre at Rennes University

France

## Abstract

Immersive technologies have seen a great expansion in the last decade and researchers from several disciplines have focused on exploring virtual reality and the way it can affect human perception. Virtual reality is a unique medium which has the ability to transfer the user from the physical environment to a digitally created illusion of space, events and interactions which mimic real life. In this paper, some basic concepts of perception in virtual reality are introduced, followed by the summary of our research which primarily focused on the perception of virtual agents. Our method is based on the concept of interpersonal distance when people meet in social settings and where the distances they keep between each other signal the nature of their relationship. We studied these distances to evaluate realism, attractiveness and even personality traits of virtual agents in virtual reality. We discuss how our results can give valuable insights into the human mind and how we can use this knowledge for training and rehabilitation applications in virtual reality.

## Keywords

virtual reality, perception, virtual agents, proximity

## 1 Introduction

Virtual Reality (VR) is an immersive environment where people can experience scenarios which mimic physical reality. They can also be engaged in virtual interaction with real people, presented in VR as avatars, or computer-driven representations of real humans. The immersive experience and interaction is a fairly recent phenomenon, providing a plethora of research challenges to solve and questions to explore. For example, how do we create believable virtual environments which will facilitate human interaction and what do people's responses to these environments teach us about our mental processes?

There are primarily two types of research domains who use VR in their research. The first, social science, is interested in VR as a highly controllable replica of a real world with the ability to create "ecologically valid experience", i.e. human response which is close to a real-life response, in order to investigate human cognition and transfer of knowledge from virtual to physical reality. The second, computer science, is more interested in keeping human evaluation in the loop to optimise computational power and enhance virtual environments. While the primary goal of the second group is not to explore the human mind, it is an inevitable side effect of their scientific endeavour.

There is, however, a third group of researchers. This group presents a bridge between the social and computer science by

exploring the potential of virtual immersive technology to enhance human abilities. Its aim is to understand how the virtual experiences could create new and faster learning procedures, aid in physical and mental health rehabilitation by broadening the scope of what is possible in the physical reality, and perhaps even open up new avenues for human experience, to which we do not yet know the limits of.

The aim of this paper is to present some examples of research in VR, dedicated to the exploration of human perception from both the computer and social science perspective. In order to better place the research topic, general concepts of VR are defined in the first part of this article. In the second part, some of our past studies using VR as a tool to measure human behaviour are presented. Our work mainly revolves around virtual agents, digital representations of humans, who populate a virtual scene in VR and may have simple or complex algorithms to simulate natural behaviour. At the end of the article, some of the implications of our research and how it can help us to understand human mind are discussed.

## 2 Virtual Reality

While research in Virtual Reality (VR) goes back as early as 1970s, it has witnessed a surge in recent years due to the development of relatively low-cost and ergonomic devices, as well as more effective and powerful graphics rendering technology. Entertainment industry began to launch VR-specific games (e.g., Beat Saber, Half-Life: Alyx), social platforms such as Metaverse [16] are using VR for interaction in online virtual environments, some organisations use VR to raise social and political awareness [17, 19]. Other immersive technologies, such as Augmented Reality and Mixed Reality, began their debut to the broad market around the same time as well. Today, they are commonly addressed with the unifying term Extended Reality (XR).

VR, however, is unlike any other immersive system. The goal of VR is to completely disconnect the user from the physical reality which is different to the aim of other XR systems which do so only partially. Complex VR systems include head-mounted display with positional tracking to create the feeling that the virtual world is surrounding and moving with the user, haptic stimulation, spatial sound, representation of the user's body in the environment, etc. This complexity of the system increases system immersion [27, 20]. If the immersion is high, the user will have the feeling of "existing" in the virtual space, an experience known as "presence" or "place illusion, plausibility" [25, 27]. Other illusions can also be created, most notably the illusion of social presence (a virtual human appears to be alive) [3, 1] and embodiment (our virtual body is perceived as our own body).

### 2.1 Presence

Presence in VR is the experience of an actual place and the feeling that the virtual events are really happening. Lombard and Ditton elegantly described presence as "the perceptual illusion of nonmediation" [14]. The concept of presence was most famously investigated with the so called "virtual pit" experiment [30, 15,

<footnote>Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s).
*Information Society* 2024, 7–11 October 2024, Ljubljana, Slovenia
© 2024 Copyright held by the owner/author(s).
https://doi.org/10.70314/is.2024.cog.12</footnote>
6], where the participants were standing on the top of a narrow
ledge in a virtual room, looking down to another room through a
wide gap in the floor. The experimenters [6] could detect changes
in galvanic skin response, shaking and loss of balance and the
participants reported intense emotional reactions of fear as if
they were in real danger of falling.

Several other types of environments and scenarios were used
by the researchers to elicit emotions (see [4] for an overview).
Apart from research on the concept of presence, training and
rehabilitation applications were developed in VR to exploit this
ability of a virtual environment to induce realistic responses in
people. Related to the pit experiment, for example, assessment
of construction workers for their postural stability at different
heights have been developed [7], and in psychotherapy, similar
environments are used to gradually expose acrophobia patients
to increasing levels of height [13].

## 2.2 Social Presence

Social presence is the illusion of being present with another in a virtual environment, or simply the “sense of being with another” [3]. The definition is broad and sometimes other terms are used to define similar or related constructs, such as telepresence, co-presence [21], and plausibility [27]. In VR, the term social presence is more commonly used to denote the level of believability of a co-located virtual human, especially if this virtual human is computer-driven (agent) and we wish to evaluate its naturalness. Social presence with other users in VR (avatars) can also be investigated to evaluate aspects of the system and the environment, such as ability to represent users with emotional expressions or communication channels (sound, text interface), which enhances the collaboration aspect between the users in the VR environment.

Depending on the definition, different ways of measuring social presence exist. Researchers who agree that social presence is a cognitive construct will typically use questionnaires for evaluation [2], while other researchers prefer to use indirect measures, such as eye-tracking and psychophysiology [27], signs of social influence [28], or task-related behaviour [26, 18].

## 2.3 Embodiment

Embodiment or the Sense of Embodiment (SeO) is the feeling of
possessing the virtual body in VR, which feels like it is “ours,” and
moves according to our intentions [11]. This illusion is linked to
the virtual body in VR for which the movement is driven by the
user, wearing a tracking device while his HMD view is centered
at the eye-view of the head. If the user observes the movement
of his hands and body when immersed in VR, he can develop
a sensation that the body is actually his own. This illusion was
first documented in real-life studies as the so called rubber hand
illusion [10]. The feeling of ownership of an artificial body can
develop when receiving synchronous visual input and touch
sensation on both the virtual and real hand, with only the virtual
hand being visible to the user. The same effect can be reproduced
using proprioception (the user observes his virtual arm moving
as he is moving his real one). Not only is the SeO enabling a more
immersive experience (presence is increased when embodiment
is added in VR), the SeO is a testament to the the importance of
the role of multimodal input in the embodied experience [11].

The illusion was also explored in creative applications, such as
giving the user a sense that they possess a part of a body which
they actually do not have in reality, e.g., having a sixth finger [9].

Some researchers also explore the idea of co-sharing of a virtual
body, where the agency over one avatar can be shared between
two users, e.g. one user possesses the left arm and the other the
right, as well as different percentages of possession of the full
avatar body [5].

## 3 Using VR to study the perception of virtual agents

As previously mentioned, agents are computer-driven representations of humans who can possess simple or complex behaviour characteristics. Researchers strive to understand how to increase their naturalness, appeal and interactive abilities and the above mentioned illusions, presence, social presence and embodiment, play an important role in this endeavour. Presence increases the believability of the scenario with the agent, social presence influences the user to exhibit social behaviour, and embodiment gives us the opportunity to measure user's body position and movement in relation to the agent in the virtual space.

### 3.1 Proximity

Interpersonal distance or proximity is the minimum distance that people maintain between one another when involved in social interaction. The measure comes from proxemics described by Hall [8] who introduced it as an indicator or comfort and familiarity with other people. Many factors influence how close we will approach another: familiarity, culture, gender, personality, etc. Closer distances reveal trustworthiness and comfort, while further distances can signal mistrust, discomfort or fear of the other. In VR, proximity has been used to explore the social influence of virtual humans [1].

The proximity measure can be expressed simply as the Euclidean distance of the current camera (user) position and the position of central mass of the virtual character in the virtual space. It is important that the user is navigating the environment by natural walking in order to preserve distances comparable to real-life interactions. The proximity tasks can vary. In the passive approach, the user is approached by an agent and is asked to press a button at the precise time they begin to feel uncomfortable with the agent's proximity (see image a, Figure 1). With the active approach, the user approaches an agent instead, typically to complete a task, e.g., read the name tag on the agent's chest. In the avoidance task, the agent is an obstacle in the environment and the user avoids it to reach a goal. With active approaches, we can generate and analyse walking trajectory from the positions of the user through time (see image b, Figure 1) in terms of walking speed, minimum passing distance, average distance from the obstacle, etc. The avoidance behaviour between real and virtual humans has some differences: clearance distance for virtual agents is larger than real humans [24]. However, factors affecting the proximity were found to be generally similar to the ones in physical reality.

### 3.2 Previous Research

Some of our most relevant results using proximity are presented
in this section. The studies used primarily agents which were
highly realistic and had real human motion applied using high-
performance motion capture (Vicon) with 53 marker system to
track the major joints and location of the body. The VR environ-
ments were created with Unreal Engine 4 or 5, and we primarily
used HTC Vive with natural locomotion (participants could tra-
verse the environment by walking) to immerse our participants
<figure><img src="image_2.png" /><figcaption>Figure 1: Examples of our experimental stimuli and measures of proximity: a) passive approach, where the virtual agent approaches the user and signals by pressing a button at the precise moment they feel uncomfortable with the agent. The Euclidean distance between the central mass of the agent and user's head-mounted display is recorded as the value of proximity; b) active approach, where the user (dark-grey character on the right of the image) circumvents the agent while walking through the virtual environment. Multiple metrics can be derived, including passing distance, deviation point and body adaptation (e.g., shoulder rotation).</figcaption></figure>

in the virtual scenario. In all our studies, the participants also possessed a virtual body.

### 3.2.1 User agency.
In Zibrek et al. [31], we were investigating the affect of agency over a virtual character in VR. The users were using the Vive controller to either trigger the character motion (avatar condition) or observe the character (agent condition). Afterwards, users were asked to approach the character to find its name tag that was attached to his chest. The aim was to test whether users will come closer to the agent they previously controlled as opposed to an agent who moved independently. The lack of control over an agent could give the impression he has the ability to have independent and unpredictable behaviour. The results showed that it was not the condition, but the subjectively perceived agency (how much the user actually felt in control of the character) which reduced the proximity, revealing the importance of perceived agency as opposed to designed one.

### 3.2.2 Gender and attractiveness.
In human interaction and VR, people will keep different distances from each other depending on their gender: males will stand further away from males and closer to females. Our study [32] focused on proximity to virtual walkers, where gender could be recognised from motion only, since previous studies using point-light displays found walking motion is rich in gender cues [12]. We were also interested to see if a more attractive motion would decrease the proximity. We designed an experiment, where a virtual agent approached the embodied participant. The agent animation was motion captured from several male and female actors and each motion was displayed individually on the character. Participants used the controller to stop the approaching agent when they felt it was uncomfortably close to them. Our results showed no difference in proximity according to the gender of the character, however, the gender of participants affected proximity (females had larger proximity distances to male users). We also found evidence that greater attractiveness will decrease proximity. This was shown only by rating the attractiveness of the motion of the agent, showing the importance of body motion to infer information about other people.

### 3.2.3 Agent animation.
We approached the perception of motion from the perspective of distinct movement patterns which can be observed on people with neurotic and emotionally stable personality traits [22]. We designed an experiment in VR, using a photo-realistic metro scenario, where we studied the avoidance behaviour of participants when encountering these two types of virtual characters in a constrained environment. Our results

indicate that neurotic motion increases the proximity even in tight spaces and also affects the choice of metro exit where they would be less likely to exit from a door which is obstructed by a neurotic agent.

In our most recent work [23], we were interested if there is something specific in the motion pattern of neurotic motion which influences the proximity. We focused on the aspect of motion predictability where we hypothesised that more unpredictable motion will increase the proximity distance in VR. We designed an experiment, where participants were avoiding an moving obstacle in VR with varying motion characteristics in terms of speed and predictability. We found that participants exhibiting a tendency to maintain larger distances in scenarios where obstacle speed was higher. Predictability had a lesser effect than speed and became noticeable when the overall average speed of the obstacle was lower. Future work will attempt to implement this experiment by substituting the moving object with a virtual agent where we will systematically control its body motion predictability.

## 4 Discussion

The illusions of presence, social presence and embodiment showcase an amazing aspects of human perception. Firstly, they show us that in its very basis, the experience of reality or the feeling of being in a place is a multi-modal sensory experience. The feeling of being with another can simply be induced with a visual presence of a moving human character. Embodiment can be achieved with synchronising haptic/proprioceptive and visual signals.

Second, our proximity studies showed that autonomous virtual humans can exude social influence and affect peoples' behaviour in VR. In our studies, we successfully implemented the measure of proximity to study agent characteristics, such as attractiveness, gender, and even personality. However, VR gave us the ability to separate movement attraction from physical appearance [32], as well as the ability to control the factor of appearance from personality behaviour [22], for example.

Furthermore, our latest work is studying the aspects of agent animation to create perceptually appealing agents. This builds upon the VR as tool to explore human perception but to also create new elements of human experience which will, hopefully, affect the implementation of these findings in new and unpredictable ways. By understanding and controlling aspects of agent motion and behaviour, we could anticipate the creation of 'appealing agents', who would be likable and comforting to the VR users and have the ability to improve the outcome of training
and rehabilitation applications. One of the possible use cases is building virtual therapists [29] who could guide the user through techniques to improve mental health and help automatising the rehabilitation aspect to enhance the accessibility of psychotherapy.

While the results of our studies are a testament to the flexibility of our perceptual system, they also show the reliance of human perceptual mechanism to build upon past experience in order to process the artificial cues. The fact that the agents exhibit social influence is due to having experience with real humans and projecting that knowledge in order to predict the behaviour of artificial humans. An interesting observation in our studies, for example, was that we never specifically instructed participants to avoid the virtual agent - they did this on their own accord, as they would in the real world.

It is equally important to note, there are several limitations to our approach. Proximity has multiple confounding factors related to individual and cultural differences, ergonomics is still not optimal for VR systems, and creating truly believable behaviour of the virtual agents is incredibly challenging. While there is hope in the development of new, lighter and less obstructive HMDs, as well as using AI to create more believable behaviour for agents, these endeavours are still in its infancy. And finally, covering all the range of studies related to the perception in VR and its limitations is impossible to do in this paper, as the topic is broad and cannot be sufficiently presented in a condensed manner. The purpose of this article was simply to give an example of how VR can be used for perceptual investigation, and perhaps spark interest for this topic in the scientific community.

## References

[1] Jeremy N Bailenson, Jim Blascovich, Andrew C Beall, and Jack M Loomis. 2001. Equilibrium theory revisited: mutual gaze and personal space in virtual environments. *Presence: Teleoperators & Virtual Environments*, 10, 6, 583–598.

[2] Frank Biocca and Chad Harms. 2004. Internal consistency and reliability of the networked minds social presence measure. In *Seventh Annual International Workshop on Presence*.

[3] Frank Biocca, Chad Harms, and Jenn Gregg. 2001. The networked minds measure of social presence: pilot test of the factor structure and concurrent validity. In *4th annual international workshop on presence, Philadelphia, PA*, 1–9.

[4] Julia Diemer, Georg W Alpers, Henrik M Peperkorn, Youssef Shiban, and Andreas Mühlberger. 2015. The impact of perception and presence on emotional reactions: a review of research in virtual reality. *Frontiers in psychology*, 6, 26.

[5] Rebecca Fribourg, Nami Ogawa, Ludovic Hoyet, Ferran Argelaguet, Takuji Narumi, Michitaka Hirose, and Anatole Lécuyer. 2020. Virtual co-embodiment: evaluation of the sense of agency while sharing the control of a virtual body among two individuals. *IEEE Transactions on Visualization and Computer Graphics*, 27, 10, 4023–4038. DOI: 10.1109/TVCG.2020.2999197.

[6] Henrique Galvan Debarba, Sidney Bovet, Roy Salomon, Olaf Blanke, Bruno Herbelin, and Ronan Boulic. 2017. Characterizing first and third person viewpoints and their alternation for embodied interaction in virtual reality. *PloS one*, 12, 12, e0190109.

[7] Mahmoud Habibnezhad, Jay Puckett, Houtan Jebelli, Ali Karji, Mohammad Sadra Fardhosseini, and Somayeh Asadi. 2020. Neurophysiological testing for assessing construction workers’ task performance at virtual height. *Automation in Construction*, 113, 103143.

[8] Edward T Hall et al. 1968. Proxemics [and comments and replies]. *Current anthropology*, 9, 2/3, 83–108.

[9] Ludovic Hoyet, Ferran Argelaguet, Corentin Nicole, and Anatole Lécuyer. 2016. “wow! i have six fingers!”: would you accept structural changes of your hand in vr? *Frontiers in Robotics and AI*, 3, 27. DOI: 10.3389/frobt.2016.00027.

[10] Marjolein PM Kammers, Frederique de Vignemont, Lennart Verhagen, and H Chris Dijkerman. 2009. The rubber hand illusion in action. *Neuropsychologia*, 47, 1, 204–211.

[11] Konstantina Kilteni, Raphaela Groten, and Mel Slater. 2012. The sense of embodiment in virtual reality. *Presence: Teleoperators and Virtual Environments*, 21, 4, 373–387. DOI: 10.1162/PRES_a_00124.

[12] Lynn T Kozlowski and James E Cutting. 1977. Recognizing the sex of a walker from a dynamic point-light display. *Perception & psychophysics*, 21, 575–580.

[13] Merel Krijn, Paul MG Emmelkamp, Roeline Biemond, Claudius de Wilde de Ligny, Martijn J Schuemie, and Charles APG van der Mast. 2004. Treatment of acrophobia in virtual reality: the role of immersion and presence. *Behaviour research and therapy*, 42, 2, 229–239.

[14] Matthew Lombard and Theresa Ditton. 1997. At the heart of it all: the concept of presence. *Journal of computer-mediated communication*, 3, 2, JCMC321. DOI: 10.1111/j.1083-6101.1997.tb00072.x.

[15] Michael Meehan, Brent Insko, Mary Whitton, and Frederick P Brooks Jr. 2002. Physiological measures of presence in stressful virtual environments. *Acm transactions on graphics (tog)*, 21, 3, 645–652.

[16] Meta. 2024. Metaverse. Retrieved August 23, 2024 from https://about.meta.com/metaverse.

[17] Tanvi Misra. 2016. What it feels like to be forced from home. Retrieved August 23, 2024 from https://www.bloomberg.com/news/articles/2016-10-14/how-virtual-reality-can-build-empathy-towards-refugees.

[18] Maria Murcia-Lopez, Tara Collingwoode-Williams, William Steptoe, Raz Schwartz, Timothy J Loving, and Mel Slater. 2020. Evaluating virtual reality experiences through participant choices. In *2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)*. IEEE, 747–755. DOI: 10.1109/VR46266.2020.000098.

[19] Unated Nations. 2016. Virtual reality: creating humanitarian empathy. Retrieved August 23, 2024 from https://www.youtube.com/watch?v=vAEjX9S80zk.

[20] Niels Chr Nilsson, Rolf Nordahl, and Stefania Serafin. 2016. Immersion revisited: a review of existing definitions of immersion and their relation to different theories of presence. *Human technology*, 12, 2, 108–134. DOI: 0.17011/ht/urn:hu:201611174652.

[21] Kristine L Nowak and Frank Biocca. 2003. The effect of the agency and anthropomorphism on users' sense of telepresence, copresence, and social presence in virtual environments. *Presence: Teleoperators & Virtual Environments*, 12, 5, 481–494.

[22] Yuliya Patotskaya, Ludovic Hoyet, Anne-Hélène Olivier, Julien Pettré, and Katja Zibrek. 2023. Avoiding virtual humans in a constrained environment: exploration of novel behavioural measures. *Computers & Graphics*, 110, 162–172. DOI: 10.1016/j.cag.2023.01.001.

[23] Yuliya Patotskaya, Ludovic Hoyet, Katja Zibrek, and Julian Pettre. 2024. Entropy and speed: effects of obstacle motion properties on avoidance behavior in virtual environment. In *SAP'24: ACM Symposium on Applied Perception 2024*, 1–13. DOI: 10.1145/3675231.3675236ff.

[24] Ferran Argelaguet Sanz, Anne-Hélène Olivier, Gerd Bruder, Julien Pettré, and Anatole Lécuyer. 2015. Virtual proxemics: locomotion in the presence of obstacles in large immersive projection environments. In *2015 IEEE virtual reality (vr)*. IEEE, 75–80. DOI: 10.1109/VR.2015.7223327.

[25] Thomas B Sheridan. 1996. Further musings on the psychophysics of presence. *Presence: Teleoperators & Virtual Environments*, 5, 2, 241–246.

[26] Richard Skarbez, Solene Neyret, Frederick P Brooks, Mel Slater, and Mary C Whitton. 2017. A psychophysical experiment regarding components of the plausibility illusion. *IEEE transactions on visualization and computer graphics*, 23, 4, 1369–1378. DOI: 10.1109/TVCG.2017.2657158.

[27] Mel Slater. 2009. Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments. *Philosophical Transactions of the Royal Society B: Biological Sciences*, 364, 1535, 3549–3557. DOI: 10.1098/rstb.2009.0138.

[28] Mel Slater, Angus Antley, Adam Davison, David Swapp, Christoph Guger, Chris Barker, Nancy Pistrang, and Maria V Sanchez-Vives. 2006. A virtual reprise of the stanley milgram obedience experiments. *PloS one*, 1, 1, e39. DOI: 10.1371/journal.pone.0000039.

[29] Mel Slater, Solène Neyret, Tania Johnston, Guillermo Iruretagoyena, Mercè Álvarez de la Campa Crespo, Miquel Alabèria-Segura, Bernhard Spanlang, and Guillem Feixas. 2019. An experimental study of a virtual reality counselling paradigm using embodied self-dialogue. *Scientific reports*, 9, 1, 10903. DOI: doi.org/10.1038/s41598-019-46877-3.

[30] Martin Usoh, Kevin Arthur, Mary C Whitton, Rui Bastos, Anthony Steed, Mel Slater, and Frederick P Brooks Jr. 1999. Walking> walking-in-place> flying, in virtual environments. In *Proceedings of the 26th annual conference on Computer graphics and interactive techniques*, 359–364.

[31] Katja Zibrek, Elena Kokkinara, and Rachel McDonnell. 2017. Don't stand so close to me: investigating the effect of control on the appeal of virtual humans using immersion and a proximity-based behavioral task. In *Proceedings of the ACM symposium on applied perception*, 1–11. DOI: 10.1145/319881.3119987.

[32] Katja Zibrek, Benjamin Niay, Anne-Hélène Olivier, Ludovic Hoyet, Julien Pettré, and Rachel McDonnell. 2020. The effect of gender and attractiveness of motion on proximity in virtual reality. *ACM Transactions on Applied Perception (TAP)*, 17, 4, 1–15. DOI: 10.1145/3419985.