# Interactive Storytelling in a Mixed Reality Environment:
The Effects of Interactivity on User Experiences

Marija Nakevska, Anika van Der Sanden, Mathias Funk, Jun Hu, Matthias Rauterberg

► To cite this version:

Marija Nakevska, Anika van Der Sanden, Mathias Funk, Jun Hu, Matthias Rauterberg. Interactive Storytelling in a Mixed Reality Environment: The Effects of Interactivity on User Experiences. 13th International Conference Entertainment Computing (ICEC), Oct 2014, Sydney, Australia. pp.52-59, 10.1007/978-3-662-45212-7_7. hal-01408504

HAL Id: hal-01408504

https://inria.hal.science/hal-01408504

Submitted on 5 Dec 2016

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Interactive Storytelling in a Mixed Reality Environment:
The Effects of interactivity on User Experiences

Marija Nakevska, Anika van der Sanden, Mathias Funk, Jun Hu, and Matthias Rauterberg

Department of Industrial Design Eindhoven University of Technology, P.O. Box 513, 5600 MB Eindhoven, The Netherlands

m.nakevska@tue.nl, m.j.v.d.sanden@student.tue.nl, m.funk@tue.nl, j.hu@tue.nl, g.w.m.rauterberg@tue.nl

**Abstract.** Interactive storytelling in a mixed reality environment merges digital and physical information and features. It usually uses an augmentation of the real-world and physically-based interaction to create an immersive experience that corresponds to the dramatic storyline of the interactive narrative influenced by the actions of the user. Immersiveness is a crucial aspect of such an installation, and can be influenced by multiple factors such as video, sounds, interaction and, finally, the density of all combined stimuli. We used one of the stages from our interactive ALICE installation to investigate immersiveness and its contributing factors in a between-group design with a special focus on the effects of interactivity, and the feedback and feedforward stimuli of the environment on the users' experiences. The study was carried out with 41 participants and the results showed that immersiveness not necessarily depends on the modality of stimuli, but instead on their time-density.

**Keywords:** interactive storytelling, mixed reality, immersiveness

## 1 Introduction

Interactive storytelling in a mixed reality environment merges digital and physical information and features. It usually uses an augmentation of the real-world and physically-based interaction. The dramatic storyline of the interactive narrative is influenced by the actions of the user. The participants are engaged in an interaction taking place in a real physical environment that does not involve direct use of a computer and interaction devices.

Dow [6] addresses three experiential pleasures of immersive and interactive stories: presence, agency and dramatic involvement. The features of the medium that can be manipulated by the design are: perceptually immersive interfaces, interactivity, and narrative structures. The terms immersion and presence often are used together, immersion describes a set of physical properties of the media technology that may give rise to presence [9]. Presence in a storytelling environment is defined as the feeling of being in a story world, while dramatic involvement is the feeling of being caught up in the plot and with characters of a story [6].
Interactivity refers to the degree to which users of a medium can influence the form or content of the mediated environment. While, agency refers to the empowerment of the user to take meaningful actions in the world which effect relate to her intention [10]. The motivation for a user to act in an interactive narrative may be very different from common interaction with a product or in gaming. In interactive storytelling, source for agency may be the ability to navigate and to influence the environment, to interact with characters or to have an effect on the course of events and outcome of the narrative. Feedback and feedforward are one of the most common used design principles in interaction design. Through feedback the user receives information about the effectiveness of her action, whereas feedforward communicates what kind of action is possible and how it can be carried out.

We use the third stage “Eat me, drink me” from the ALICE project [2], to explore the challenges in designing an interactive narrative. The ALICE installation consists of six consecutive stages, creating an experience based on selected parts from the novel “Alice’s Adventures in Wonderland” by L. Carroll [3]. The user takes the role of the character Alice and experiences the sequence of emotional and behavioral states as Alice did in the narrative. In this paper we present the technical and storytelling mechanisms and we study the effects of interactivity on the user experience. A between subjects experiment was conducted to explore potential differences in sense of presence, agency and satisfaction with different levels of interactivity. The independent variable was the interaction environment. We hypothesize that an interactive setting should lead to higher levels of presence, agency and satisfaction.

After reviewing relevant research, we present the experimental setup and results regarding the relation between these variables of interactivity, presence and agency, resulting in several conclusions in the last paragraph.

## 2 Related work

Existing physically interactive spaces usually implement full body interaction, augmentation of a physical space or manipulation of real, physical objects. The MIT Media Lab created several physically interactive story environments, that used a “less-choice, more responsiveness” approach to engage physically the users as characters in a story; concluding that “compelling interactive narrative story systems can be perceived as highly responsive, engaging and interactive” [8]. Dow et. al., present results of a qualitative, empirical study by using augmented reality (AR) interactive drama AR Facade [5], which showed that “immersive AR can create an increased sense of presence”, and “increased presence does not necessarily lead to more engagement” [4]. The multimodal mixed reality installation, Synthetic Oracle, is used for an empirical research that indicates the importance of the choice of interaction mode and shows that “the activity level and behavior of users modulates their experience, and that in turn, the interaction mode modulates their behavior” [1]. This empirical research suggests that the interactivity and interaction type can have an impact on the behavior and the personal experience of the user. It is important to further evaluate and quantify the experiences from empirical perspective. In the next section we de-
scribe the setup of the “Eat me, drink me” stage and the results from the experiment that involved 41 participants.

# 3 Experiment

## 3.1 Eat me, drink me

The “Eat me, drink me” stage is designed to induce similar experience to the one described in the original narrative. Following is the summary from the narrative [3]:
> "Alice enters a room with doors all around that differ in size. She finds a key that unlocks one small door, but she is too big to fit through it. After she drinks and eats, she undergoes several changes, she grows and shrinks. Eventually she has the right size and the key from the small door."

The participant finds herself trapped in a cube room, to continue further out of this room, the user needs to find the right relation between her size and the room and to have the key to “open” the door. Since the ALICE installation has six stages in total, the participant has to go through each stage in a limited amount of time. We aimed the interaction design to support each participant to move on to next stage in three to five minutes.

**Spatial design.** The 5-sided Cave Automatic Virtual Environment (CAVE) is 3x3x3 meter cube made of white semi-translucent canvas, see Figure 1. The floor has pressure sensors to measure the position of the participant in the room. On each wall of the room, a virtual (VR) door (Fig. 2b) is projected, on the sliding side is projected a white VR door smaller than the others and features a doorknob (Fig. 2c) as a character from the story.

**Physical props.** On one side of the room on a table the participant finds a cookie box labeled “Eat Me” and in the other corner is a bottle labeled “Drink Me”(Fig. 2a). These objects contain sensors to register interaction accordingly. The box is equipped with an IR sensor that detects movement when the participant takes a cookie. The bottle contains a wireless connected tilt sensor which detects if the participant is drinking from the bottle. Behind one of the tables, a physical key with a label “Take me” is hidden.

**Interaction design.** When the participant performs an action, takes a cookie (eats) or drinks, the projected room becomes smaller; and on the second eat/drink action the room becomes bigger. Both actions feature an appropriate sound, which emphasizes the impression that the participant is getting smaller or bigger. During the experiment we observe if the participant takes the physical key, and with Wizard of Oz method we indicate in the system, which is coupled with a virtual key that appears at the doorknob featuring VR sparkles and a piano “fantasy” sound. Each step on a pressure sensor results in a cracking sound played on loudspeakers. The cracking sounds are different depending on the previously taken actions. If the participant is “big”, the cracking sound of the floor is heavier, and vise versa, the cracking sounds are shorter and lighter. An ambient sound is played in the background that consists of fantasy music and dropping water. The “water drop” sound features a different echo depending of the relative size of the VR room.
<figure><img src="image_2.png" /><figcaption>**Fig. 1.** Overview of “Eat me, drink me” stage. (a) left, the schematic overview of the physical setting (b) right, the scale model of the CAVE, from top to bottom: (1) the entrance is opened, (2) the entrance and exit are closed, (3) the exit is opened</figcaption></figure>

The interactive doorknob gives hints for participant's actions. When the participant comes closer to the VR door (Fig. 2c) and is too big to fit through, the doorknob says: “*Sorry! You're way too big.*” When the participant approaches the door and has no key with her, the doorknob says: “*No use. Haha! I forgot to tell you. I'm locked.*” If the participant approaches the door and has the key in her hand, the virtual door will open. If the VR door opens the participant sees the White Rabbit in a beautiful garden waving and saying: “*Oh, dear! Oh, dear! You will be late*” (Fig. 2d).

A narrator voice gives guidance based on the participant's behavior. As the participant moves around in the environment, the number of triggered pressure sensors is counted. When the number of steps passes eight the narrator voice says: “*Are you just moving around in here, will you ever find the way out?*” If standing on the same position is detected the narrator voice says; “*Oh dear! You are just standing here!*” To facilitate the progress through the story, we introduced explicit feed-forward hints from the interactive doorknob, like “*Alright, try the bottle*”, “*Now try the box on the table*”. After three minutes, the doorknob gives the appropriate guidance, depending on the last taken action from the participant and waits for the participant to finish it.
<figure><img src="image_3.png" /><figcaption>**Fig. 2.** The virtual room and characters. (a)The physical objects (b) User inside the CAVE (c) VR door with interactive doorknob (d) White Rabbit in the Garden</figcaption></figure>

**3.2 Procedure and participants**

The participants were invited to take part of the “Alice’s Adventures in Wonderland”
and they were led into the room with the instruction to “have fun”. They experienced
one of the following interaction modes:

- Interactive environment (IE): The environment used all the available interaction features described in the interaction design section, based on the behavior of the user.

- Non-interactive environment (NIE): A pre-programmed scenario (that uses 10 from the described features) of the narrative is played without taking in consideration the behavior of the user.

- Non-interactive with minimum stimuli (NIMS): A pre-programmed scenario (that uses 4 features) of the narrative is played without taking in consideration the behavior of the user.

Forty-one participants joined the study, all university students from 18 to 33 years old (13 female, 28 male, mean age 23 with a standard deviation of 3). Twelve participants joined the IE setting; sixteen joined the NIE and thirteen the NIMS setting. All experimental sessions took less than 20 minutes including the experience about 4 minutes and a survey about 15 minutes.

**3.3 Measurements**

**Presence.** Participants were administered with the ITC-Sense of Presence Inventory (ITC-SOPI) to evaluate their levels of physical presence [7]. The ITCSOPI is a validated 44-item self-report questionnaire that was used in this study to measure how physically present and involved the users feel in the storytelling environment through four factors: spatial presence, engagement, ecological validity and negative effects.

**Agency.** We measured the subjectively perceived agency based on the perceived proficiency, the perceived responsiveness and technical aspects of the environment and how much the participants are aware of their influence on the events in the environment. The following items were created: "I felt proficient in my actions with the environment during the experience", "I was aware of my influence on control mechanisms in the environment", "I felt that the environment was responsive to my behaviors". Since, agency is achieved when the actions of the user are causing the intended
effect on the mediated world; we added items to check if the user's intention and the
hints from the environment match. "I knew what actions I should take to do to go
out", "I knew what I should do because the environment gave me a hint", and "The
physical objects were obvious hint for interaction".

**Satisfaction.** We also measured how much the participants appreciated the experience. They were asked to rate the experience on several scales: "The experience was: terrible, okay, good, great, best thing of entertainment experiences, best thing in my life", "I have really enjoyed myself during this experience". And to choose on a 5-points Likert scale between: "Very dissatisfied" - "Very satisfied" and "Terrible" - "Delighted".

We also observed the users' behavior via video records from the surveillance sys-
tem. The actions of the users recognized by the sensing mechanisms (pressure sen-
sors, IR and tilt sensors) were recorded in a text file.

4 Results

Figure 3 illustrates the means of the factors generated by the ITC-SOPI questionnaire, the agency and satisfaction questionnaires. Differences between the means for the three conditions for presence, engagement, naturalness, negative effect and satisfaction were examined for significance using a one-way ANOVA for independent groups design. The results showed no significant differences between the three conditions for presence, engagement, naturalness, negative effect and satisfaction.

A one-way between subjects ANOVA was conducted to compare the effect on agency for IE, NIE and NIMS conditions. There was a significant effect on agency for the three conditions [F(2, 38) = 8.209, p = 0.001]. Post hoc comparisons using the Tukey HSD test indicated that the mean score for the IE condition (M = 3.56, SD = 0.54) was significantly different than the NIE condition (M = 3.02, SD = 0.54) and the NIMS condition (M = 2.71, SD = 0.49). The NIE condition did not significantly differ from NIMS condition.

Taking the data from the sensing mechanisms, we counted the number of actions (eat, drink, trigger feedback) that were triggered by the users. We compared the number of actions by the participants in each setting for the IE, NIE and NIMS conditions with one-way ANOVA. There was a significant difference for the three conditions [F(2, 25) = 6.237, p = 0.006]. Post hoc comparisons using the Tukey HSD test showed that the mean score for the IE condition (M = 5.08, SD = 2.46) was significantly different than the NIE condition (M = 9.06, SD = 4.14). The NIE condition is significantly different from the NIMS condition (M = 4.66, SD = 3.44). However, the number of actions from the participant in the IE condition does not differ significantly from the NIMS condition.
<figure><img src="image_4.png" /><figcaption>Fig. 3. Dimensions of the presence (spatial presence, engagement, naturalness, negative effects), agency and satisfaction by treatment conditions: IE, NIE and NIMS</figcaption></figure>

# 5 Discussion

The results showed that the interaction types did not influence the feelings of presence and the satisfaction from the experience. We originally expected that the presence factors of the interactive environment will be significantly higher than that of NIE and NIMS environment. We assume that the CAVE as a strongly immersive environment, contributes for high feelings of presence even when the environment is not responsive to the actions of the user.

Through observation of the actions of the users and by quantifying the number of actions we noticed differences in the users' behavior. The participants that were immersed in a not responsive environment (NIE) were more active and tried out more interaction possibilities (touch, walk, look around). The participants who experienced minimum stimuli in a non-interactive environment (NIMS) did not perform as many actions; instead they would rather stand and look around. The participants in the non-interactive environment (NIE) more often showed confusion and frustration, while the participants in the interactive environment (IE) seemed satisfied every time they discovered an interaction asset. The stimuli provided from the environment evoke different behavior and with that also a different personal user experiences.

In the interactive setting (IE) everyone had slightly different experience depending on the triggered stimuli and the actual context. Not everyone would reveal all of the events from the narrative, e.g. the virtual garden was visible only if the participant approached the small VR door and had the key. The order in which they would discover the events or the pace in which the story would be played different for different participants. The events from the story were context related and they would trigger only if the person was at the right place on a right time.
# 6 Concluding Remarks

This paper has described the fully realized interactive story “Eat me, drink me” in-
spired by one of the chapters from the narrative “Alice’s adventures in Wonderland”.
We present the interactive story, the technology and design decisions that went into
building the system. Furthermore, we investigate the user interaction and the overall
experience. The study contributes to our knowledge about the design of interactive
and mixed reality spaces, and how the responsiveness and the amount of stimuli in-
duce or bias behavior and experiences. We have to point out that results obtained with
the different settings over short durations of time have to be taken with precaution
since its effects may vary over longer time periods. One limitation of this study is the
usage of subjective post hoc measures of experience such as ITC-SOPI, where pres-
ence and engagement are measured based on the overall perception of the immersive
environment. Further studies would explore the user experience in an enriched inter-
active setting that implements more challenging scenario of the interactive narrative.

References

1. Badia, S.B.i., Valjamae, A., Manzi, F., Bernardet, U., Mura, A., Manzolli, J., Verschure, P.F.J.: The effects of explicit and implicit interaction on user experiences in a mixed reality installation: The synthetic oracle. Presence: Teleoperators and Virtual Environments 18(4), 277–285 (2009)

2. Bartneck, C., Hu, J., Salem, B., Cristescu, R., Rauterberg, M.: Applying virtual and augmented reality in cultural computing. IJVR 7(2), 11–18 (2008)

3. Carroll, L.: Alice's adventures in wonderland. Broadview Press (2011)

4. Dow, S., Mehta, M., Harmon, E., MacIntyre, B., Mateas, M.: Presence and engagement in an interactive drama. In: Proceedings of the SIGCHI conference on Human factors in computing systems. pp. 1475–1484. ACM (2007)

5. Dow, S., Mehta, M., Lausier, A., MacIntyre, B., Mateas, M.: Initial lessons from an ar facade, an interactive augmented reality drama. In: Proceedings of the 2006 ACM SIGCHI international conference on Advances in computer entertainment technology. p. 28. ACM (2006)

6. Dow, S.P.: Understanding user engagement in immersive and interactive stories. ProQuest (2008)

7. Lessiter, J., Freeman, J., Keogh, E., Davidoff, J.: A cross-media presence questionnaire: The itc-sense of presence inventory. Presence: Teleoperators and virtual environments 10(3), 282–297 (2001)

8. Pinhanez, C.S., Davis, J.W., Intille, S., Johnson, M.P., Wilson, A.D., Bobick, A.F., Blumberg, B.: Physically interactive story environments. IBM Systems Journal 39(3.4), 438–455 (2000)

9. Riva, G., Davide, F., IJsselsteijn, W.: I being there: The experience of presence in mediated environments. Being there: Concepts, effects and measurements of user presence in synthetic environments pp. 3–16 (2003)

10. Wardrip-Fruin, N., Mateas, M., Dow, S., Sali, S.: Agency reconsidered. Breaking New Ground: Innovation in Games, Play, Practice and Theory. Proceedings of DiGRA 2009 (2009)