# Performative Virtual Scenes: A Dynamic VR Environment Design Approach

Nuno N. Correia, Stephan Jürgens, Raul Masu, Jochen Feitsch, Ivana Druzetic

► To cite this version:

Nuno N. Correia, Stephan Jürgens, Raul Masu, Jochen Feitsch, Ivana Druzetic. Performative Virtual Scenes: A Dynamic VR Environment Design Approach. 20th International Conference on Entertainment Computing (ICEC), Nov 2021, Coimbra, Portugal. pp.100-114, 10.1007/978-3-030-89394-1_8 . hal-04144358

HAL Id: hal-04144358

https://inria.hal.science/hal-04144358

Submitted on 28 Jun 2023

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Performative Virtual Scenes: A Dynamic VR Environment Design Approach

Nuno N. Correia¹,²[0000-0002-9999-9162], Stephan Jürgens²[0000-0003-4739-904X], Raul Masu²,³[0000-0002-9725-6584], Jochen Feitsch⁴, and Ivana Druzetic⁴[0000-0001-7697-335X]

¹ Tallinn University, Tallinn, Estonia nuno.correia@tlu.ee

² ITI/LARSyS, Funchal, Portugal {stephan.jurgens,raul.masu}@iti.larsys.pt

³ DI, FCT, Universidade Nova de Lisboa, Caparica, Portugal

⁴ University of Applied Sciences Düsseldorf, Düsseldorf, Germany {jochen.feitsch,ivana.druzetic}@hs-duesseldorf.de

**Abstract.** This paper proposes a dynamic VR environment design approach to enhance the way a user experiences a computer generated character's movement. We transferred concepts and practices from performance studies to create a dynamic VR environment, in which design elements are interconnected and changing with the character's movement. Our focus was on space and light as two essential elements of both dance performances and virtual environments. We employed and tested our dynamic VR environment design approach by means of two studies (on space and on light) with 24 users in total. The results of these studies are discussed with regard to the enhancement of, and benefits for the user experience, followed by suggestions for the applicability of our approach to other fields in VR design.

**Keywords:** VR · light · space · user experience · performing arts · contemporary dance · VR environment · CG character · movement.

## 1 Introduction

The perception of computer generated characters in virtual environments is a research topic that has recently received attention, aiming at designing for a more lively and embodied user experience (e.g., [27], [28]). We support that there is potential in applying performance studies concepts to Computer Generated (CG) characters, movement and their environment in VR, in order to improve user experience. The expertise of performing arts directors and practitioners has been identified as very relevant for VR design. Smith has investigated the translation of live performance into VR, and refers to Google Experience Engineer Elly Nattinger, who states: "the expertise of performing arts directors in controlling the audience's focus provides them with the necessary skills for developing work in VR and suggests that these techniques should be studied and used by all disciplines working in VR" [26]. However, some issues that are considered key in performance studies have not gained the same degree of attention in VR design.
Among the topics that may support VR development and user experience design, we identify research potential in investigating the *connections between character movement and the virtual environment*; in particular the connection between movement and changes in space and light. Space and light are two central elements in dance performance, particularly including digital technology. For instance, Correia et al. [6] included set (equivalent to space in our terminology), light, sound and costume design in their Performance Network diagram. Blom and Beckhaus have mapped the design space of dynamic interactive virtual environments. They propose that “what makes environments interesting and engaging is having worlds that are both active and reactive” [3]. Under their dynamic components taxonomy, our intended elements, space and light, are situated under ‘scene attributes’. Although their research does not relate directly to dance, nor to CG characters, their findings identify the relevance of dynamic elements in VR environments.

Based on this identified potential, we propose a *dynamic VR environment* design approach: an environment that is interconnected and changing with the character’s movement. This leads to our research question: *How does a dynamic VR environment change the way a user experiences a CG character’s movement?*. We focus on space and light as dynamic elements of the VR environment.

Based on that dynamic VR environment, we designed two studies, using motion capture data to generate CG character movement. We use the term ‘CG character’ in the same way as Vincs et al.: “combining motion capture data with 3D character animation” [28]. Following the approach of transferring performing arts expertise to VR design, we used the artistic concepts and choreographic strategies from an interactive dance piece conceived by Outi Valanto, who worked with digital spaces and light-sensitive movement explorations. We propose that this approach can also be useful in VR design outside of performing arts.

# 2 Literature Review

## 2.1 VR, Dance and Movement

The topic of dance in Virtual Reality has emerged in recent Human-Computer Interaction (HCI) literature in different contexts. Smith [26] analysed the current practice in dance performance and virtual reality. The author classified dance performances according to the content and type of virtual environment, from a realistic representation of the environment to computer generated imagery. Thomas et al. [27] identified a gap between how bodies are seen and how bodies are felt in virtual environments. To address this, they explored a combination of somatic dance practices and VR technology. This is also an important topic for dance education, which Raheb et al. [21] discussed in their survey on Dance Interactive Learning Systems, including examples of dance in VR.

Choreographic processes have also entered the design of VR environments. For instance, Fdili Alaoui et al. [7] presented a VR version of the choreographic performance *Radical Choreographic Object*. Altizer Jr. et al. [1] and Jürgens et al.
[14] explored the use of VR and choreographic thinking, visualizing the creative process. Movement is fundamental in dance, but also in other VR fields studied in HCI, such as exergames. Wolf et al. [31] presented a jump-based locomotion augmentation technique, which “combines realistic physical movement with hyper-realistic virtual outcome”, leading to an exaggeration of physical actions.

## 2.2 User Experience in VR: Presence and Immersion

Mestre et al. discussed the term ‘presence’ referring to the initial concept of ‘spatial presence,’ understood as “self-orientation and self-location with respect to a media environment, not the real environment” [18]. In a recent study on experiencing stories in a virtual environment, Shin [25] concluded that the VR experience can be divided in two phases. The author describes the first phase as a ‘two-tiered process of immersion,’ which comprises presence and flow. In the next phase, empathy and embodiment “are selectively experienced by users.” In Shin’s approach, the user has a fundamental role in “processing immersion, forming presence, and influencing user engagement” [25]. Consequently, the author suggests to harmonize the technological features and respective cognitive processes of different user groups. This can be done according to the designer’s intended level of presence and flow to impact on desired empathy and embodiment. We see specific research opportunities arising from Shin’s approach, to take the cognitive processes of the user into account.

Space is an important component to take into account regarding presence. Wirth et al. [29] proposed a model of spatial presence consisting of a two-step formative process. In the first step, the user verifies that the virtual environment is a space through the construction of a “spatialized mental model of the mediated environment”. The second step consists of accepting the mediated environment as the primary frame of self-reference and self-location. We will combine the frameworks of Wirth et al. [29] and Shin [25] in our design approach.

## 2.3 Space and Realism in VR

Space is also relevant in VR in terms of level of realism. For instance, in their project *Reality Check*, Hartmann et al. [8] focused on the situated physical reality as a starting point to investigate real-time 3D reconstruction, aiming to merge reality with a VR environment, comparing different degrees of blending between both. Kitson et al. [15] proposed that ‘lucid dreaming’, a phenomenon where dreamers are conscious of their dreaming, and know that no event or action have real-life consequences, can be relevant for VR design. These authors demonstrate that there are effective alternatives to realism in the design of VR environments.

Brown [5] described the potential of the virtual stage (beyond traditional scenographic conventions) as an evolving ‘relational space’: “In learning to dance with data, spaces unfold, [...] the stage metamorphoses from a physical location - grounded, fixed, actual - to a relational space - incorporating the ungrounded, the fluid and the virtual.” Brown’s ‘relational space’ concept informed the interrelated logic of our dynamic approach.
## 2.4 Light, User Experience and VR

In HCI literature, light has been identified as an important element for user experience design. To describe the concept of user experience, Hassenzahl began by analysing the example of the Wake-Up Light, a crossing between an alarm clock and a bedside lamp: “half an hour before the set alarm, the lamp starts to brighten gradually, simulating sunrise” [9]. Höök et al. [11] designed *Breathing Light*, which measured the movements of the user’s chest to control the dimming of the ambient light, providing ‘subtle guidance’ for the user to reach precise bodily introspection. In the installation *Dichroic Wade* [23], light was conceived “as a material and concrete element” for artistic creation. These examples illustrate the importance of light as an element that can help “transcending the material”, for designing “experiences created and shaped through technology (aka User Experience)” [9].

Despite the identified potential of light for user experience, there is scarce related literature applying it to VR. We present some recent examples pointing to this area. In the *Aura Garden* VR environment, light painting photography is explored: users can create their own light sculptures through a physical wand controller [24]. He et al. [10] studied audience experiences of live theatre performances in VR. The authors compared conventional and panoramic video to find out which media format provoked a stronger desire in the viewer to watch the live performance of the piece in an actual theatre venue. In the results, the authors mentioned two examples of participants who connected light changes and lighting effects to their vivid memory of a specific scene. This particularly relates to our light study in VR.

# 3 Study Design

## 3.1 Design of the VR Environments

We designed two studies, each focusing on dynamic space and dynamic light. We based the dynamic changes of space and light on the movement of a CG character – our proposed *dynamic VR environment* design approach. To assess our approach, we tested the dynamic condition of each element against a neutral (static) condition. We collaborated with choreographer Outi Valanto, following the recommendation from Jürgens et al. [13] to foster “integration of contemporary dance practitioners as researchers in interdisciplinary projects”. Her dance piece *E-motional Landscapes*, developed in a previous stage of the project *Moving Digits* (https://movingdigits.eu), has a strong connection to space and light.

Regarding **space**, the choreographer stated “*the moving body is the center of engagement and experience*” and navigates “*in between the physical and imagi-native spaces*” (from the *E-motional Landscapes* project description). Regarding **light**, the choreographer wanted to explore “*how we direct ourselves to the light and how we direct ourselves away from the light, and how this affects our body and our movement*”, for example “*trying to search the shadows and avoiding the*
*light*" (excerpt of interview with choreographer). To obtain data for the movement of the character, we conducted a motion capture session with a professional dancer, using the OptiTrack system (with 24 cameras covering a 5 x 5 meters tracking area). In this session, the dancer conducted improvisations informed by the above concepts around space and light.

In the design of the **dynamic space** condition, we mapped the walls and the ceiling to the head position of the dancer, in a non-linear way (the x-, y- and z-axis of the head were mapped to different space elements). Regarding **dynamic light**, we created six trigger areas for light within the virtual space, reacting to the dancer's horizontal head position. Additionally, the light intensity could be controlled via the vertical head position. We aimed to avoid one-to-one mappings, and planned to implement more complex and less obvious ones, following recent recommendations from literature on dance and technology [17]. In both cases, the setting for the virtual environment was a room with a 'plain' appearance, similar to a dance studio (figure 1). The rationale for this 'plain' appearance was to emphasize the CG character's movement. Some elements, such as bricks and columns, were designed with the intention of providing reference points for the viewer in terms of scale and location. Both studies and both conditions were designed for being experienced by a sitting viewer, the same setting as s/he would experience in a theatrical context (i.e., the viewer could look around but not move around). There was no sound in both conditions.

<figure><img src="image_3.png" /><figcaption>Fig. 1: Top: Study 1 - Space. Bottom: Study 2 - Light. Neutral conditions on the left, dynamic conditions on the right.</figcaption></figure>

## 3.2 Methods

Study 1 and 2 focused on the impact of dynamic space and light, respectively, on user experience. In each study, we compared two conditions: the dynamic condition, where one element (either space or light) is mapped to the CG character's movement, against a neutral condition, where space and light are static.
Each study used a specific segment of the choreography, which related to space or light. The two studies followed the same procedure. We adopted a repeated measures, within-subjects approach as done in other VR studies, such as [2] and [16], therefore each participant experienced both the dynamic and neutral conditions. The duration of the segments were 2 minutes (Study 1) and 2 minutes and 42 seconds (Study 2). The studies combined questionnaires and interviews.

**Questionnaires** To investigate user experience in VR, we decided to assess presence (the sense of “being there in the environment” [30]) and engagement (“a quality of user experience that is characterized by the depth of an actor’s investment in the interaction [with a digital system]” [19]). Presence data was collected using the Igroup Presence Questionnaire (IPQ), a validated questionnaire with a general presence factor, PRES, and three subscales: SP, Spatial Presence, the sense of being physically present in the virtual environment; INV, Involvement, focus of attention between real and virtual environments; REAL, Experienced Realism, judgment of the reality of the virtual environment [12]. It has 14 items, using 7-point scales.

To assess user engagement, we adopted the short form of the User Engagement Scale (UES), that is, UES-SF [20]. UES-SF consists of 12 items instead of UES's 31, therefore “ideal in within-subject studies where participants are completing multiple tasks or trials” [20] and fitting for our studies. UES-SF is a validated questionnaire with four subscales: FA, Focused Attention, “feeling absorbed in the interaction and losing track of time”; PU: Perceived Usability, “affect experienced as a result of the interaction and the degree of control and effort expended”; AE: Aesthetic Appeal, “attractiveness and visual appeal of the interface”; and RW: Rewarding, how much the experience was rewarding and worthwhile. It relies on 5-point scales. The order of the questions within IPQ and UES-SF was randomised. Combining IPQ with UES-SF resulted in 26 items for each condition.

We implemented the **questionnaires inside the VR environment**, following the approach recommended by Schwind et al. [22]. This way, participants do not need to remove their Head-Mounted Display (HMD) and put it on again to experience the next condition. According to [22], adopting presence questionnaires in VR can reduce “distracting or biasing cues from the real-world, such as the experimenter or experimental setup”. Consequently, it potentially requires a smaller sample than real-world questionnaires to reveal statistically significant results between two or more conditions [22].

To **analyse the resulting quantitative data** we used Python (v. 3.6.9, libraries: statsmodels, statistics, scipy). To assess parametric assumptions of our datasets, we tested the distribution using a normality test (Shapiro test, given the smaller sample size). Only PRES in experiment 1 passed the normality test. The data that met parametric assumptions was analysed using t-test to compare conditions; while the remaining datasets were analysed using Wilcoxon signed-rank test (also known as Wilcoxon t-test) to compare conditions. Only significant comparisons are reported in detail.
**Interviews** We also conducted semi-structured interviews with participants, to gather further insights. The questions focused on: a general impressions of both conditions; impressions on the CG character's movement; perceived connection (if any) between character's movement and space or light; and any other thoughts provoked by the experience. Interviews lasted, on average, 10 minutes. The transcribed interviews were later coded and analysed using thematic analysis [4] by two members of our team, and cross-checked by a third member.

**Participants** For each of the 2 studies, there were 12 participants (24 in total), testing both the dynamic and the neutral conditions. Participants were recruited from the local academic and artistic communities. As incentive, they received merchandising from the research project and a voucher for a dance performance. Before the studies, we conducted an online demographics questionnaire with participants, including questions about experience with VR and dance.

Participants had ages ranging from 24 to 58 years. 10 identified as male, 13 as female, and 1 preferred not to say. In terms of VR experience, 8 used VR once a month or more; 10 between once a month and once a year; 2 once a year or less; and 4 had never used it. Regarding dance experience, 2 attended dance performances once a month or more; 9 between once a month and once a year; 9 once a year or less; and 4 had never attended. Participants were distributed between the studies in order to achieve diversity of age, gender, experience of VR and dance in each study. For anonymity, participants were numbered first (P1, P2 etc.) and distributed later between studies, as part of this process.

<figure><img src="image_4.png" /><figcaption>Fig. 2: Study procedure: Participant with HMD (left) and an image from the in-VR questionnaire (right)</figcaption></figure>

**Procedure** Participants undertook the following procedure: 1) welcome, briefing about the study, consent form signing, and introduction to the VR set-up; 2) VR study, consisting of: first condition and questionnaires (IPQ and UES-SF), followed by second condition and questionnaires, all in VR; 3) interview. To counterbalance the learning effect, the order of the conditions (neutral and dynamic) was randomised. Participants were sitting during the experiment (figure 2), in order to resemble the audience experience of a traditional dance performance.
The average durations of the in-VR experiences were 14 minutes and 11 seconds for Study 1 (space), and 15 minutes and 12 seconds for Study 2 (light). These were presented using an HTC Vive Pro. To guarantee fluid VR experiences and a constant refresh rate (90 FPS) for the HMD, we used state-of-the-art gaming hardware: an Intel i7-8086K CPU (4 GHz) with 32GB of memory and an NVIDIA GeForce GTX 1080 (8 GB memory) video card.

# 4 Results and Analysis

## 4.1 Results from Questionnaires

The descriptive statistics (mean and standard deviation) for each of the subscales of IPQ and UES-SF of both studies can also be seen in table 1. The results of the Wilcoxon signed-rank test analysis show a statistically significant difference in some of the subscales of both IPQ and UES-SF questionnaires between our two conditions.

<table><caption>Table 1: Mean scores and (±) standard deviation for IPQ and UES-SF subscales across conditions for Study 1 – Space and Study 2 – Light. The * indicates a statistically significant difference between conditions, with higher values in bold.</caption><thead><tr><th>Study</th><th>Questionnaire</th><th>Subscale</th><th>Neutral Condition</th><th>Dynamic Condition</th></tr></thead><tbody><tr><td rowspan="10">Study 1</td><td rowspan="4">IPQ</td><td>PRES</td><td>4.333 ± 1.312</td><td>4.250 ± 1.639</td></tr><tr><td>SP</td><td>4.200 ± 1.610</td><td>4.150 ± 1.768</td></tr><tr><td>INV*</td><td>*3.521 ± 1.633</td><td>*4.333 ± 1.736</td></tr><tr><td>REAL*</td><td>*2.750 ± 1.689</td><td>*2.125 ± 1.333</td></tr><tr><td rowspan="4">UES-SF</td><td>FA*</td><td>*2.500 ± 1.093</td><td>*3.222 ± 0.885</td></tr><tr><td>PU</td><td>4.250 ± 1.010</td><td>4.083 ± 1.064</td></tr><tr><td>AE*</td><td>*2.806 ± 0.937</td><td>*3.389 ± 0.826</td></tr><tr><td>RW*</td><td>*3.083 ± 1.115</td><td>*3.806 ± 1.023</td></tr><tr><td rowspan="10">Study 2</td><td rowspan="4">IPQ</td><td>PRES*</td><td>*3.917 ± 1.320</td><td>*4.667 ± 1.312</td></tr><tr><td>SP*</td><td>*3.900 ± 1.620</td><td>*4.333 ± 1.650</td></tr><tr><td>INV</td><td>3.729 ± 1.604</td><td>3.938 ± 1.819</td></tr><tr><td>REAL</td><td>2.646 ± 1.315</td><td>2.875 ± 1.615</td></tr><tr><td rowspan="4">UES-SF</td><td>FA</td><td>2.833 ± 1.280</td><td>3.194 ± 1.126</td></tr><tr><td>PU</td><td>4.417 ± 0.862</td><td>3.944 ± 1.246</td></tr><tr><td>AE*</td><td>*2.750 ± 1.037</td><td>*3.333 ± 0.972</td></tr><tr><td>RW</td><td>3.500 ± 1.093</td><td>3.667 ± 1.054</td></tr></tbody></table>

In **Study 1 – Space**, for IPQ, the following statistically significant differences emerged in the analysis: the INV subscale (Involvement, focus of attention between real and virtual environments) was ranked higher in the dynamic condition than in the neutral one (Z=76.500, p=0.001); in addition the REAL subscale (Experienced Realism) was ranked lower in the dynamic condition than in the neutral one (Z=174.500, p=0.05). In UES-SF, we found the following statistically significant difference as follows: FA subscale (Focused Attention, “feeling
absorbed in the interaction and losing track of time” [20]) was ranked higher
in the dynamic condition (Z=78.000, p=0.002); additionally the AE subscale
(Aesthetic Appeal, “the attractiveness and visual appeal of the interface” [20])
was higher in the dynamic condition (Z=30.000, p=0.001); and to conclude the
RW subscale (Rewarding, how much the experience was worthwhile) was also
higher in the dynamic condition (Z=28.000, p=0.001).

In **Study 2 – Light**, for IPQ, the following statistically significant differences emerged: the PRES factor (General Presence) was ranked higher in the dynamic condition (Z=2.500, p=0.047); and the SP subscale (Spatial Presence, the sense of being physically present in the Virtual Environment) was also ranked higher in the dynamic condition (Z=287.500, p=0.037). In the UES-SF test the AE subscale (Aesthetic Appeal, “the attractiveness and visual appeal of the interface” [20]) was ranked higher in the dynamic condition (Z=52.500, p=0.002).

**4.2 Analysis and Discussion of Interviews**

In this section, we present the results of our thematic analysis, and discuss these
in light of the questionnaire results and the literature review. Six main themes
emerged from our analysis, which became the headings to the sections below.
The first three themes relate to Study 1; the fourth and fifth themes corresponds
to Study 2; and the last theme was identified in both studies.

**Higher Involvement in a Less Realistic VR Environment** The results of **Study 1**, on space, show that our participants felt more involved in the dynamic condition. This is supported by the higher scores in the dynamic condition in the ‘involvement’ subscale of the IPQ questionnaire and by the interviews. Moreover, there is a connection between higher involvement and the less realism of the dynamic environment. Although participants experience a higher realism in the neutral condition (from IPQ, ‘experienced realism’ subscale), they report preference for the dynamic environment, as it is less realistic and hence provides more surprise (P5), unpredictability (P5), novelty (P13) and challenge (P2).

There are straightforward parallels to Wolf et al. [31], who successfully intro-
duced a less realistic virtual outcome as a reward in another exergame system.
Our findings also integrate well with Wirth et al.’s framework of a two-step
formative process of presence [29]. Once participants understood the less realis-
tic elements in the dynamic VR environment (corresponding to step one in the
model by Wirth et al.) they could embrace this mediated environment as their
primary reference of self-reference and self-location (step two). Our interview
results demonstrate that the participants reported a benefit of the less realis-
tic dynamic space condition, different from a ‘plain’ real-world experience. This
benefit was absent in the neutral, more realistic condition. This is an important
observation: realism does not necessarily lead to involvement, and in fact the
opposite can occur.

**Facilitation of Embodiment and Empathy** The dynamic space condition of **Study 1** provoked different feelings and emotions: “made me emotional” (P6),
"you can develop feelings" (P13), "it moved me" (P4), "I have felt more" (P6), it was "more moody" (P10). This led to a heightened perception of embodiment: "the walls coming nearer, I definitely felt more my body" (P4). Some participants reported a change in the way they felt their own body during the experience of the dynamic space condition, and a stronger feeling of action in the VR environment. These outcomes are mirrored in the results of the UES-SF questionnaire 'rewarding' subscale, with higher scores in the dynamic space condition.

These findings correlate to the framework presented by Shin [25]. For this author, embodiment and empathy are experienced in a second phase of the VR experience (following the initial phase of immersing in the VR environment). Therefore, by enabling embodiment and empathy, the dynamic space condition facilitates this second phase.

**Understanding and Empowerment of CG Character Movement** In **Study 1**, the participants' comments to the dynamic space condition eluci- date how the CG character's movement exists inseparable from its context. As a representative example, P3 experienced the neutral space condition as *"static, lifeless, empty"*, while the dynamic space condition was described as *"alive like it was linked to the person"*. Participants further experienced the CG character movement as *"easier to understand"* (P5) and following a *"storyline"* (P8). Ad- ditionally, two participants experienced the space as empowering the movement of the character (P1, P3). These observations are reflected in the results of the UES-SF questionnaire in the 'focused attention' subscale, with higher scores in the dynamic space condition.

This is a fine example for the design of a virtual stage, where a grounded and fixed physical location has metamorphosed into an evolving "relational space" [5]. This relational space (the dynamic space condition) led to an experience of empowered character movement, while also making it more understandable.

**Directing Attention to Body and Movement** A notable aspect of the VR experience in **Study 2** was the participants' perception of the relationship between movement and light. Some participants detected a link between the dancer and the direction (P12), timing (P17) and intensity (P14) of the light. The perception by these participants of various subtle mapping strategies (as outlined in the design of the VR environment section) indicates an awareness of the performance ecology in the VR environment. Other participants stated that the neutral light condition did not direct the attention to anything specific (P18, P20, P21), whereas the dynamic condition focused attention on movement elements or body parts through the correlation of changes in movement and light.

Despite some similarity (of the dynamic condition) with a physical theatre setting, participants did not experience the light element as a conventional "lighting cue" for the performer(s) on stage (as for example in [10]). Instead, they perceived the causal connections between movement and light, which is more similar to the work of Seevinck [23] and Seo et al. [24], who use light as material, as an interconnected element in interactive environments. The mappings
contributed to the success of the dynamic condition in directing attention to the CG character's body and movement.

**Enabling Presence and Self-Orientation in a Mediated Environment**

The results of **Study 2** show that our participants did not feel as present in the neutral light condition of the VR environment. A representative statement: *"it was more like watching the scene and not being immersed"* (P22). Conversely, most participants reported strong feelings of expectation (P14, P18, P20, P24), surprise (P18), intrigue (P20) and curiosity (P17) related with the dynamic light condition. This is supported by the IPQ questionnaire results in the 'general presence' subscale. Additionally, the results in the 'spatial presence' subscale of IPQ also show higher scores in the dynamic light condition.

Mestre et al. [18] link presence to self-orientation and self-location “with respect to a media environment, not the real environment.” We suggest combining this idea with Wirth et al.’s framework of a two-step formative process of presence [29]. Wirth et al. call the first step the construction of a “spatialized mental model of the mediated environment”. Once our participants formed a clear idea about the media environment as a first step (e.g., P17 interpreted the neutral condition as a rehearsal space and the dynamic condition as a performance venue), they could embrace the media environment as a primary reference for self-orientation and self-location (second step). We suggest that certain mental models (such as being placed in a dark stage environment to watch a performance) imply the participant’s acceptance of challenges in self-orientation and self-location in anticipation of a gratifying VR experience. In fact, our results showed that most participants reported strong feelings of expectation, surprise, intrigue and curiosity with the dynamic light condition. By fostering a mental model of a mediated environment, this condition facilitated self-orientation and self-location, where associated challenges were accepted and welcomed.

**Interconnected Elements Contributing to Aesthetic Experience**

Overall, the dynamic space condition of **Study 1** was appealing to the participants as a *"whole"* (P6, P8, P9, P15, P16) experience, where a feeling of harmony and complementarity between movement of CG character and of space existed. For example, the interconnectedness between the elements in the VR environment led to the experience of *"emotions that got transferred"* from the character in the dynamic space condition to the participant (P5). Again, this matches results from the UES-SF questionnaire, showing higher scores in the dynamic condition for the 'aesthetic appeal' subscale.

The aforementioned process of embodiment and empathy [25] in this case occurred through the experience of an interconnected VR environment, in which the design elements mutually reinforce each other. Thus, the dynamic space condition, through the interconnectedness of its elements, led to a more aesthetically appealing user experience.

Similarly, in **Study 2** some participants also reported a feeling of all-encompassing experience, of being more in contact with the *"whole"* (P9, P15) in the dynamic
condition. Participants experienced the dynamic light condition as appealing (P12, P17, P21, P22), impressive (P17), visually pleasing (P23), mysterious (P14, P18, P20, P24), dark (P22), and even scary (P17, P18, P22, P24), in direct positive correlation with self-reported higher mental engagement. The higher scores in the UES-SF questionnaire for ‘aesthetic appeal’ of the dynamic light condition support the interpretation that the aesthetic experience was particularly important for engagement.

Their experience also confirms the immersive process described by Shin [25] who observed that embodiment and empathy follow successful presence in the VR environment. Therefore, as in the dynamic space, the interconnectedness of elements in the dynamic light condition led to a more holistic and more aesthetically appealing user experience.

# 5 Conclusion

To answer our research question we conducted two studies, each focusing on a different VR environment element: space and light. We compared a dynamic approach, interconnecting environment elements with CG character movement, against a neutral condition. In both studies, the dynamic approach led to higher presence and engagement values in questionnaires than the neutral condition, which was further supported and clarified by the interviews.

We discussed how this dynamic approach can lead to several user experience benefits with the design of CG characters’ movement in VR: higher involvement in a less realistic VR environment; facilitation of embodiment and empathy; understanding and empowerment of CG character movement; directing attention to body and movement; enabling presence and self-orientation in a mediated environment; and interconnected elements contributing to aesthetic experience. Wirth et al.’s framework of a two-step formative process of presence [29], in combination with Shin’s model of immersion [25], have been very useful in discussing the participants’ experiences in both studies.

We propose that the dynamic VR environment design approach can be useful for designers – toward improving the user experience of VR environments, where CG character movement plays a key role. Our studies suggest that our approach can successfully improve user experience in VR, compared to a neutral approach, for two VR environment elements: space and light. However, our investigation on this novel topic remains exploratory, is based on two studies only, and further research is needed.

We suggest that there is potential in further studying space (e.g., positive and negative space) and light (e.g., hue and duration), while applying our dynamic approach to additional environment elements (e.g., sound and costume design, as identified in [6]), pointing to further research pathways. Taking into account the fields identified in our literature review, we propose that our dynamic VR environment design approach could be applied to other fields in VR design, for instance gaming (e.g., exergaming), VR storytelling, and cultural heritage.
**Acknowledgements** This research is co-funded by the Creative Europe programme of the EU (597398-CREA-1-2018-1-PT-CULT-COOP1, Moving Digits project). We acknowledge the support of LARSyS to this research (Projeto - UIDB/50009/2020). The third author acknowledges ARDITI -Agencia Regional para o Desenvolvimento e Tecnologia under the scope of the Project M1420-09-5369-FSE-000002 - PhD Studentship.

## References

1. Altizer Jr., R., Handman, E., Bayles, G., Jackman, J., Cheng, K., Ritchie, S., Newell, T., Wright, C.: Choreografish: Co-designing a Choreography-based Therapeutic Virtual Reality System with Youth Who Have Autism Spectrum Advantages. In: Proceedings of the 2018 Annual Symposium on Computer-Human Interaction in Play Companion Extended Abstracts - CHI PLAY '18 Extended Abstracts. pp. 381–389. ACM Press, Melbourne, VIC, Australia (2018), http://dl.acm.org/citation.cfm?doid=3270316.3271541

2. Bala, P., Masu, R., Nisi, V., Nunes, N.: "When the Elephant Trumps": A Comparative Study on Spatial Audio for Orientation in 360° Videos. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. pp. 1–13. CHI '19, Association for Computing Machinery, New York, NY, USA (May 2019), https://doi.org/10.1145/3290605.3300925

3. Blom, K.J., Beckhaus, S.: The design space of dynamic interactive virtual environments. Virtual Reality **18**(2), 101–116 (Jun 2014), https://doi.org/10.1007/s10055-013-0232-y

4. Braun, V., Clarke, V.: Using thematic analysis in psychology. Qualitative Research in Psychology **3**(2), 77–101 (Jan 2006), https://www.tandfonline.com/doi/abs/10.1191/1478088706qp063oa, publisher: Routledge

5. Brown, C.: Learning to Dance with Angelfish: Choreographic Encounters Between Virtuality and Reality. In: Broadhurst, S., Machon, J. (eds.) Performance and Technology: Practices of Virtual Embodiment and Interactivity, pp. 85–99. Palgrave Macmillan UK, London (2006), https://doi.org/10.1057/9780230288157

6. Correia, N.N., Masu, R., Pham, A.H.D., Feitsch, J.: Connected Layers: Evaluating Visualizations of Embodiment in Contemporary Dance Performances. In: Fifteenth International Conference on Tangible, Embedded, and Embodied Interaction. pp. 1–12. TEI '21, Association for Computing Machinery, New York, NY, USA (Feb 2021), https://doi.org/10.1145/3430524.3440621

7. Fdili Alaoui, S., Bevilacqua, F., Jacquemin, C.: Interactive Visuals As Metaphors for Dance Movement Qualities. ACM Trans. Interact. Intell. Syst. **5**(3), 13:1–13:24 (Sep 2015), http://doi.acm.org/10.1145/2738219

8. Hartmann, J., Holz, C., Ofek, E., Wilson, A.D.: RealityCheck: Blending Virtual Environments with Situated Physical Reality. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI '19. pp. 1–12. ACM Press, Glasgow, Scotland Uk (2019), http://dl.acm.org/citation.cfm?doid=3290605.3300577

9. Hassenzahl, M.: User Experience and Experience Design. In: The Encyclopedia of Human-Computer Interaction. Interaction Design Foundation, 2nd edition edn. (2013), https://www.interaction-design.org/literature/book/the-encyclopedia-of-human-computer-interaction-2nd-ed/user-experience-and-experience-design
10. He, L., Li, H., Xue, T., Sun, D., Zhu, S., Ding, G.: Am I in the theater?: usability study of live performance based virtual reality. In: Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology. pp. 1–11. ACM, Tokyo Japan (Nov 2018), https://dl.acm.org/doi/10.1145/3281505.3281508

11. Höök, K., Ståhl, A., Jonsson, M., Mercurio, J., Karls- son, A., Johnson, E.C.B.: Somaesthetic design. interactions 22(4), 26–33 (Jun 2015). https://doi.org/10.1145/2770888, http://dl.acm.org/citation.cfm?doid=2797212.2770888

12. Igroup.org – project consortium: Igroup presence questionnaire (IPQ), http://www.igroup.org/pq/ipq/index.php

13. Jürgens, S., Correia, N.N., Masu, R.: The Body Beyond Movement: (Missed) Op- portunities to Engage with Contemporary Dance in HCI. In: Fifteenth Interna- tional Conference on Tangible, Embedded, and Embodied Interaction. pp. 1–9. TEI ’21, Association for Computing Machinery, New York, NY, USA (Feb 2021), https://doi.org/10.1145/3430524.3440624

14. Jürgens, S., Fernandes, C., Kuffner, Rafael: Choreographed 4D visuals: experienc- ing Sylvia Rijmer’s Body Logic Method in a VR Dance installation. Performance Research 25(6), 94–97 (2020), publisher: Routledge

15. Kitson, A., Schiphorst, T., Riecke, B.E.: Are You Dreaming?: A Phenomenologi- cal Study on Understanding Lucid Dreams as a Tool for Introspection in Virtual Reality. In: Proceedings of the 2018 CHI Conference on Human Factors in Com- puting Systems - CHI ’18. pp. 1–12. ACM Press, Montreal QC, Canada (2018), http://dl.acm.org/citation.cfm?doid=3173574.3173917

16. Masu, R., Bala, P., Ahmad, M., Correia, N.N., Nisi, V., Nunes, N., Romão, T.: VR Open Scores: Scores as Inspiration for VR Scenarios. In: Michon, R., Schroeder, F. (eds.) Proceedings of the International Conference on New Interfaces for Musical Expression. pp. 109–114. Birmingham City University, Birmingham, UK (Jul 2020), https://www.nime.org/proceedings/2020/nime2020_paper21.pdf, iSSN: 2220-4806

17. Masu, R., Correia, N.N., Jürgens, S., Druzetic, I., Primett, W.: How do Dancers Want to Use Interactive Technology?: Appropriation and Layers of Meaning Be- yond Traditional Movement Mapping. In: Proceedings of the 9th International Conference on Digital and Interactive Arts. pp. 1–9. ACM, Braga Portugal (Oct 2019), https://dl.acm.org/doi/10.1145/3359852.3359869

18. Mestre, D., Fuchs, P., Berthoz, A., Vercher, J.L.: Immersion et présence. In: Fuchs, P., Moreau, G., Berthoz, A., Vercher, J.L. (eds.) Le traité de la réalité virtuelle, pp. 309–38. Ecole des Mines de Paris, Paris (2006)

19. O’Brien, H.: Theoretical Perspectives on User Engagement. In: O’Brien, H., Cairns, P. (eds.) Why Engagement Matters: Cross-Disciplinary Perspectives of User En- gagement in Digital Media, pp. 1–26. Springer International Publishing, Cham (2016), https://doi.org/10.1007/978-3-319-27446-1

20. O’Brien, H.L., Cairns, P., Hall, M.: A practical approach to measuring user en- gagement with the refined user engagement scale (UES) and new UES short form. International Journal of Human-Computer Studies 112, 28–39 (Apr 2018), http://www.sciencedirect.com/science/article/pii/S1071581918300041

21. Raheb, K.E., Tsampounaris, G., Katifori, A., Ioannidis, Y.: Choreomorphy: a whole-body interaction experience for dance improvisation and visual experimen- tation. In: Proceedings of the 2018 International Conference on Advanced Visual Interfaces. pp. 1–9. ACM, Castiglione della Pescaia Grosseto Italy (May 2018), https://dl.acm.org/doi/10.1145/3206505.3206507
22. Schwind, V., Knierim, P., Haas, N., Henze, N.: Using Presence Questionnaires in Virtual Reality. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI '19. pp. 1–12. ACM Press, Glasgow, Scotland Uk (2019), http://dl.acm.org/citation.cfm?doid=3290605.3300590

23. Seevinck, J.: Dichroic Wade. In: Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems - CHI EA '16. pp. 3889–3892. ACM Press, San Jose, California, USA (2016), http://dl.acm.org/citation.cfm?doid=2851581.2891094

24. Seo, J.H., Bruner, M., Ayres, N.: Aura Garden: Collective and Collaborative Aesthetics of Light Sculpting in Virtual Reality. In: Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems. pp. 1–6. ACM, Montreal QC Canada (Apr 2018), https://dl.acm.org/doi/10.1145/3170427.3177761

25. Shin, D.: Empathy and embodied experience in virtual environment: To what extent can virtual reality stimulate empathy and embodied experience? Computers in Human Behavior **78**, 64–73 (Jan 2018), http://www.sciencedirect.com/science/article/pii/S0747563217305381

26. Smith, S.: Dance performance and virtual reality: an investigation of current practice and a suggested tool for analysis. International Journal of Performance Arts and Digital Media **14**(2), 199–214 (Jul 2018), https://doi.org/10.1080/14794713.2018.1509256, publisher: Routledge

27. Thomas, L.M., Glowacki, D.R.: Seeing and feeling in VR: bodily perception in the gaps between layered realities. International Journal of Performance Arts and Digital Media **14**(2), 145–168 (Jul 2018), https://doi.org/10.1080/14794713.2018.1499387, publisher: Routledge

28. Vincs, K.: Virtualizing Dance. In: The Oxford Handbook of Screendance Studies. Oxford University Press, New York (2016)

29. Wirth, W., Hartmann, T., Böcking, S., Vorderer, P., Klimmt, C., Schramm, H., Saari, T., Laarni, J., Ravaja, N., Gouveia, F.R., Biocca, F., Sacau, A., Jäncke, L., Baumgartner, T., Jäncke, P.: A Process Model of the Formation of Spatial Presence Experiences. Media Psychology **9**(3), 493–525 (May 2007), https://doi.org/10.1080/15213260701283079, publisher: Routledge

30. Witmer, B.G., Singer, M.J.: Measuring Presence in Virtual Environments: A Presence Questionnaire. Presence **7**(3), 225–240 (Jun 1998). https://doi.org/10.1162/105474698565686

31. Wolf, D., Rogers, K., Kunder, C., Rukzio, E.: JumpVR: Jump-Based Locomotion Augmentation for Virtual Reality. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. pp. 1–12. CHI '20, Association for Computing Machinery, New York, NY, USA (Apr 2020), https://doi.org/10.1145/3313831.3376243