# Experimenting the interaction effect between a physical and a human-controlled virtual actor in theatrical performances

Anastasiia Ternova, Geoffrey Gorisse

► To cite this version:

Anastasiia Ternova, Geoffrey Gorisse. Experimenting the interaction effect between a physical and a human-controlled virtual actor in theatrical performances. JIT 2022 - Journées d'Informatique Théâtrale 2022, Inria Grenoble Rhône Alpes; ENSATT Lyon; Laboratoire Passages XX-XXI de l'Université Lyon 2, Oct 2022, Lyon, France. hal-04154469

HAL Id: hal-04154469

https://inria.hal.science/hal-04154469

Submitted on 6 Jul 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# EXPERIMENTING THE INTERACTION EFFECT BETWEEN A PHYSICAL AND A HUMAN-CONTROLLED VIRTUAL ACTOR IN THEATRICAL PERFORMANCES

Anastasiia Ternova

Université Paris 8

anastasiia.ternova02@univ-paris8.fr

Geoffrey Gorisse

Arts et Métiers Institute of Technology

geoffrey.gorisse@ensam.eu

## ABSTRACT

With this paper, we aim at introducing an empirical experiment to explore the impact of different interactions between a physical actor and a virtual avatar (digital character controlled by a second actor equipped with a motion-capture suit) on spectators' sense of social presence in the frame of theatrical performances. This investigation relies on previous creation-researches that permitted us to put forward a hypothesis that different types of interactions could impact spectators' attitude, making them drive their attention towards the physical and/or the virtual environment during the performance. With the proposed experiment we want to apply an empirical scientific approach to our artistic practice in order to share guidelines, techniques and exercises with theater makers confronting physical actors and digital characters in their practice.

## 1. INTRODUCTION

With this paper, we aim at introducing an empirical experiment to explore the impact of different interactions between a physical actor and a virtual avatar (digital character controlled by a second actor equipped with a motion-capture suit) on spectators' sense of social presence [1, 6, 10] in the frame of theatrical performances. Social presence can be defined as the sense of being there together [10]. Numerous researches in the field of human-computer interaction focused on factors impacting the sense of social presence in both multi-user and virtual character interaction contexts [7]. In this research, we are interested in investigating the way spectators can experience social presence with a physical and a virtual actor. We will focus on three subdimensions of the sense of social presence as introduced by Biocca and colleagues [1] : co-presence, attention allocation and perceived affective understanding. In the frame of this experiment, "virtual actor" refers to a "virtual marionette" as introduced by Cédric Plessiet [8], that is to say a rigged digital puppet that is fully controlled using a motion capture suit.

To our knowledge, previous research putting together theatrical practice and scientific investigations mainly focuses theater makers : rehearsing in virtual environments [11], using 3D models to learn texts [9], motion capture and 3D environment visualization while performing [5].

However, in the proposed investigation we will focus on spectators' perception of the performance.

Following a creation-research process [12], Anastasiia Ternova introduced four types of interactions between physical actors and digital avatars affecting the spectators' perception of the scene by :

1. drawing their attention more towards the physical world or towards the virtual world by emphasizing one over the other;

2. letting them choose the object of their attention by presenting unrelated and parallel actions in both universes;

3. creating a harmonious attention flow by connecting the two worlds;

4. disturbing their attention by superimposing the two worlds.

These interactions were introduced during the staging of a creation-research *Patrick and Venus* (Figure 1a), a performance with avatars animated in real time by actors equipped with motion capture suits. They were partly tested afterwards in two subsequent creation-research projects entitled *The Wizard without Shadow* [14], [15], [3] (a performance with pre-recorded animations) (Figure 1b) and *Salomé* [13] (real time motion capture) (Figure 1c). However, the relevance of such interactions in drawing spectators' attention in the expected way cannot be stated as the aforementioned performances involved many uncontrolled variables such as attractive and distracting digital environments surrounding the avatars or unbalanced scenography with one world being more prominent than the other, inconsistent virtual camera movements, dynamic lighting in both the physical and the virtual environments, varying avatar appearances, etc. In addition, spectators were not formally invited to share their perception of the scenes and to discuss the way they directed their attention toward the proposed content. Thus, the goal of this experiment is to create an ecologically valid environment to investigate the impact of each proposed interaction type for both the actors and the spectators. In other words, we are interested in applying an empirical scientific approach to our artistic practice in order to share guidelines, techniques and exercises with theater artists working with human- avatar interactions.
<figure><img src="image_2.png" /><figcaption>**Figure 1** – Creation-research projects where actors interact with virtual avatars.</figcaption></figure>

# 2. MATERIALS AND METHODS

## 2.1. Tasks and Conditions

The proposed experiment will consist in inviting the participants (spectators) to watch a sequence of minima-list scenes presenting an ecologically valid experimental context. Such an experimental design will ensure a viable statistical analysis of spectators' attention. As a single ex-periment will not make it possible to explore the impact of the four aforementioned interaction types, this first study will focus on the effect of actors' activity with two consi-dered levels : active and passive. The goal of the expe-riment will be to investigate on which actor (physical or virtual) the spectators draw their attention depending on the acting intensity. In both cases, the two worlds will remain connected and the actors will maintain a contin-nuous interaction. Participants will be exposed to three short scenes :

- **Scene 1, Active Physical World (A-PW)** : Active Physical Actor (A-PhA) and Passive Avatar (P-Av).

- **Scene 2, Active Virtual World (A-VW)** : Passive Physical Actor (P-PhA) and Active Avatar (A-Av).

- **Scene 3, Active Worlds (A-Ws)** : Active Physical Actor (A-PhA) and Active Avatar (A-Av).

In each scene the active performer will reproduce simi-lar movements. Actors will be physically separated at the beginning of the scene. As they act, the distance will be progressively reduced until they touch each other.

In the first scene (A-PW), the physical actor approaches the avatar. The latter observes the physical actor's behav-vior while remaining passive. In the second scene (A-VW), the situation is reversed. The physical actor observes the avatar performing the same action. Finally in the third scene (A-Ws), the two actors move towards each other.

After each scene, the participants will complete a ques-tionnaire to assess their level of attention and of social presence with the actors. As in classical theater, the spec-tators will be seated in front of the stage. The stage will consist of an empty performing space with a white screen at the back of the stage to project the virtual space.

## 2.2. Virtual Character and Digital Environment

In line with previous creation-research projects (Figure 1), a non-organic white avatar scaled to match the size of the physical actor will be used as virtual actor. The same

avatar will be used in the three scenes. The virtual content of the experience will be developed using Unreal Engine 4. We intend to produce a dark and minimalist environ-ment, as we want the spectators to focus on the avatar and not be distracted by its surroundings.

## 2.3. Actors and Physical Environment

Two actors will participate in this experiment. The ac-tor chosen to control the virtual avatar will have previous experience in motion capture and real-time avatar anima-tion in order to be conscious of the two circle of presence introduced by Gagneré [2] : proprioceptive consciousness of the virtual body and awareness of the virtual space of acting. On the other hand, the physical actor will have a good understanding of the specificities and constraints in-duced by motion capture. The both actors have to share a common circle of attention, a definition derived from Sta-nislavski's system [16] that means a vivid consciousness of the presence of the second actor. To balance the real and virtual actors, the physical actor will be dressed in white. The physical environment will be split in three spaces (Fi-gure 2) :

1. Performing space of the physical actor : empty stage with simple lighting ensuring that participants will focus on the gestures and actions of the actor. The screen on which the avatar will be projected will be positioned at the back of the stage behind the physical actor's performing space. The scene will be filmed with a camera capturing the actor and the projection screen. The content will be displayed in the second space.

2. Performing space of the mocaptor<fnref n="1" /> : hidden space in the backstage allowing the actor to animate the avatar and to perform the interactions remotely. This space ensures that the spectators do not focus on the mocaptor. A Perception Neuron 3 motion cap-ture suit will be used to control the avatar. A video projector will display the virtual environment be-hind the physical actor on a white screen. A camera fixed on a tripod will film the stage and display the recorded content on a screen in the backstage to al-low the mocaptor to interact with the physical actor.

3. Spectator seats in front of the stage.

<footnote n="1">1. Actor controlling an avatar using a motion capture suit.</footnote>
<figure><img src="image_3.png" /><figcaption>Figure 2 – Experiment setup.</figcaption></figure>

## 2.4. Participants - Spectators

48 participants (6 groups of 8 spectators) aged from 18 to 60 years with prior theater experience but who do not attend very often (not more often than once or twice a month, not less than once every six months) will be recruited for this experiment. They will be invited to watch the six scenes following a within subjects design. It is best to recruit participants whose professional field is not related to theater and 3D art.

## 2.5. Procedure

### 2.5.1. Phase 1 : Instructions and Pre-experiment Questionnaire

Participants will sign a consent form and answer a pre-experiment questionnaire to collect anonymous demographic data : gender, age, professional field, previous experience with theater and new technologies expertise. After filling up the first questionnaire, the required instructions will be given to the participants. The experimenter will explain to the participants that they are about to see a staging with new technologies where a physical actor will be confronted with a digital avatar animated in real time using a motion capture suit. Next, participants will be informed that they will have to fill out a questionnaire between each of the three scenes to evaluate their perception and feelings regarding the actors' performance.

### 2.5.2. Phase 2 : Experiment

The actors will perform the three scenes sequentially. Each scene will last approximately 3 minutes. A 7 minutes break between the scenes will allow the participants to fill out the questionnaire. Overall, the experiment should last between 30 and 45 minutes. The sequence of the scenes will be counterbalanced. Each group of participants will see them in a different order.

- Group 1 : 1, 2, 3

- Group 2 : 1, 3, 2

- Group 3 : 2, 1, 3

- Group 4 : 2, 3, 1

- Group 5 : 3, 1, 2

- Group 6 : 3, 2, 1

### 2.5.3. Phase 3 : Post-experiment Interviews

After completing the last questionnaire, the participants and the actors, along with the experimenter, will engage in a discussion to gather subjective qualitative feedback. This discussion will be recorded and further analyzed afterwards.

## 2.6. Measures

After each scene, participants will fill out a post-experiment questionnaire adapted from the Networked Minds Social Presence Questionnaire [1, 4] to assess their level of co-presence, their attention allocation and their perceived affective understanding with each actor (Table 1). Three out of six dimensions of the original questionnaire will be considered in this study as the others require either a verbal interaction (perceived message understanding) or a mutual interaction between the interactants (perceived emotional interdependence, perceived behavioral interdependence). In addition, half of the items per dimension will be discarded due to the lack of reciprocity in the spectator-actor interaction. The final post-experimental questionnaire is composed of 9 items based on seven-point semantic differential scales to be completed twice per scene (one for each actor).

<table><caption>Table 1 – Post Experiment Questionnaire</caption><tr><td><strong>Co-presence</strong></td></tr><tr><td>I noticed (the physical actor/the avatar).<br>(The physical actor/The avatar) presence was obvious to me.<br>(The physical actor/The avatar) caught my attention.</td></tr><tr><td><strong>Attention Allocation</strong></td></tr><tr><td>I was easily distracted from (the physical actor/the avatar).<br>I remained focused on (the physical actor/the avatar) throughout the action.<br>(The physical actor/The avatar) did not receive my full attention.</td></tr><tr><td><strong>Perceived Affective Understanding</strong></td></tr><tr><td>I could tell how (the physical actor/the avatar) felt.<br>(The physical actor's/The avatar's) emotions were not clear to me.<br>I could describe (the physical actor's/the avatar's) feelings accurately.</td></tr></table>
### 3. HYPOTHESES

- **H1**: Spectators feel a higher sense of co-presence (H1.1), a higher attention allocation towards (H1.2) and a higher perceived affective understanding with (H1.3) the active performer for each scene regardless of his/her nature (physical or virtual).

- **H2**: When both actors are active, the unusual aspect of the virtual actor induces a higher sense of co-presence (H2.1) and a higher attention allocation (H2.2).

- **H3**: When both actors are active, the expressiveness of the physical actor induces a higher perceived affective understanding.

### 4. REFERENCES

[1] Biocca, F., Harms, C., & Burgoon, J. K. (2003). Toward a more robust theory and measure of social presence: Review and suggested criteria. *Presence: Teleoperators & virtual environments*, 12(5), 456-480.

[2] Gagneré, G., Plessiet, C. (2019). Espace virtuel interconnecté et Théâtre (2). Influences sur le jeu scénique. Revue : Internet des objets, (1).

[3] Gagneré G., Ternova A. (2021) Keeping the living bond between actors and remote audiences in distributed virtual theater. In Proceedings of 10th International Conference on Digital and Interactive Arts (AR-TECH'21), Aveiro, Portugal, ACM, New York, NY, USA, 12 pages.

[4] Harms, C., & Biocca, F. (2004). Internal consistency and reliability of the networked minds measure of social presence. In Seventh annual international workshop : Presence (Vol. 2004). Universidad Politecnica de Valencia Valencia, Spain.

[5] Kade, D., Lindell, R., Ürey, H., & Özcan, O. (2016). Supporting acting performances through mixed reality and virtual environments. In Fifth International Conference on Software and Emerging Technologies for Education, Culture, Entertainment, and Commerce (SETECEC 2016) : New Directions in Multimedia Mobile Computing, Social Networks, Human-Computer Interaction and Communicability : : Venice, Italy : : March 29–31.

[6] Lee, K. M. (2004). Presence, explicated. Communication theory, 14(1), 27-50.

[7] Oh, C. S., Bailenson, J. N., & Welch, G. F. (2018). A systematic review of social presence : Definition, antecedents, and implications. Frontiers in Robotics and AI, 114.

[8] Plessiet, C., Gagneré, G., & Sohier, R. (2019). A Proposal for the Classification of Virtual Character. In VISIGRAPP (2 : HUCAPP) (pp. 168-174).

[9] Roberts-Smith, J., DeSouza-Coelho, S., Dobson, T. M., Gabriele, S., Rodriguez-Arenas, O., Ruecker, S., Sinclair, S., Akong, A., Bouchard, M., Hong, M., Jakkacki, D., Lam, D., Kovacs, A., Northam, L., & So, D.

(2013). Visualizing Theatrical Text : From Watching the Script to the Simulated Environment for Theatre (SET). DHQ : Digital Humanities Quarterly, 7(3).

[10] Schroeder, R. (2006). Being there together and the future of connected presence, *Presence : Teleoperators & Virtual Environments*, 15(4), 438–454.

[11] Slater, M., Howell, J., Steed, A., Pertaub, D. P., & Garau, M. (2000, September). Acting in virtual reality. In Proceedings of the third international conference on Collaborative virtual environments (pp. 103-110).

[12] Ternova A., (2019). Patrick and Venus. Director, playwright, 3D artist. Featuring Remy Gorski, Alissa Menshykova (mocaptors), Cécile Roque-Alsina, Louis Hemet (actors), Naël Bernard, Justine Waller (actors, programmers, 3D artists), Georges Gagneré (programmer).

[13] Ternova A., (2022). Salomé. Director, programmer, 3D artist. Bsed on the homonimous play of Oscar Wilde. Featuring Mathieu Capella (mocaptor), Remy Gorski, Louis Hemet, Charles Halphen, William Bourgine, Kilian Cailliez, Samir Chiguer (actors), Gaëtan Henry (actor, scenographer, 3D artist), Julien Lomet (3D artist), Sophia Kourkoulakou (ava-tar designer), Kacper Gosrki (digital operator), Julia Grosjean (costume designer), Georges Gagneré (AVA-tarStaging concept). With support of Patrick Grosjean and Etat Libre d'Orange.

[14] Ternova A., (2021). The Wizard without Shadow. Director, playwright, avatar designer and programmer. Featuring Naël Bernard (actor), Georges Gagneré (programmer), Remy Gorski (mocaptor), Justine Waller (set designer, programmer). Produced by Didascalie.net.

[15] Ternova A., Gagneré G., (2021) « The Wizard without Shadow », in Proceedings of 10th International Conference on Digital and Interactive Arts (AR-TECH'21), Aveiro, Portugal, ACM, New York, NY, USA, 4 pages.

[16] Wegner, W. H. (1976). The creative circle : Stanislavski and yoga. Educational theatre journal, 28(1), 85-89.