# Creating an Immersive Mixed Reality Theater: Reconciling Artistic Intent and Technical Constraints

Pierrick Uro, Florent Berthaut, Thomas Pietrzak, Laurent Grisoni, Marcelo M Wanderley, Chris Salter

► To cite this version:

Pierrick Uro, Florent Berthaut, Thomas Pietrzak, Laurent Grisoni, Marcelo M Wanderley, et al. Creating an Immersive Mixed Reality Theater: Reconciling Artistic Intent and Technical Constraints. Université de Lille. 2023. hal-04999956v2

HAL Id: hal-04999956

https://hal.science/hal-04999956v2

Submitted on 26 Mar 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Creating an Immersive Mixed Reality Theater: Reconciling Artistic Intent and Technical Constraints

Pierrick Uro

IDMIL, CIRMMT

McGill University

Université de Lille

pierrick.uro@univ-lille.fr

Laurent Grisoni

Université de Lille

laurent.grisoni@univ-lille.fr

Florent Berthaut

Université de Lille

florent.berthaut@univ-lille.fr

Marcelo M. Wanderley

IDMIL, CIRMMT

McGill University

marcelo.wanderley@mcgill.ca

Thomas Pietrzak

Université de Lille

thomas.pietrzak@univ-lille.fr

Chris Salter

Immersive Arts Space

Zurich University of the Arts

christopher.salter@concordia.ca

## ABSTRACT

Artistic performances and installations making use of Mixed Reality (MR) that can be experienced by large groups of people are becoming ever more present in the entertainment landscape, in particular theater. Such audiovisual productions often run into similar technical challenges, namely occlusion, spatial calibration, shared experience of MR, and interaction from outside the virtual world. These challenges need to be addressed according to the underlying artistic vision and specific practical constraints of the project. We describe each of these persistent challenges as well as some of the solutions existing in the literature. Concrete approaches to solve these issues are discussed in the framework of the immersive MR theater production *Animate*, notably for the behavior of an interactive audio-visual virtual environment. These descriptions are then compared to those in previous works in order to identify three categories of solutions that can then be employed as design strategies in future productions: Limit interaction or environment complexity, exploit non-noticeability of imperfections, and integrate imperfections within the story. We also argue that these strategies, which attempt to meet a goal with what the available technology can afford, line up with the concept of "Affect over Effects", and as such are anchored within a frugal approach to technology.

## Author Keywords

Mixed Reality, Theater, Interactive Performance, Multi-user, Augmented Reality

## CCS Concepts

* Human-centered computing → Mixed / augmented reality;

* Applied computing → Performing arts;

## 1. INTRODUCTION

As an ever-increasing number of artistic productions employ immersive displays in Mixed or Virtual Reality, they often run into

the issue of compromising between what the technology enables and the artist's ambition. In particular, when an installation involves shared immersion, i.e. with multiple spectators and actors, specific challenges come into play, such as occlusion, spatial calibration, and shared experiences. Although the commonly occurring challenges can be identified, no unique solution exists for every context, since the intent and constraints are unique to each production and sometimes each performance. Instead, this problem demands strategies that can be followed, adapted, and serve as guidelines for future designers of productions that use the same type of technology.

This paper discusses critical notions such as Mixed Reality (MR) [19], immersion [24, 15], Immersive Mixed Reality Theater (IMRT) [6], and existing approaches to accord technology and performance. We then list challenges that are susceptible to arise in the technical production of interactive artistic productions that use these concepts and some of their solutions found in the literature.

A case study of Animate, an artistic installation that combines spatialized sounds and virtual objects in an IMRT, is presented, along with the solutions that were brought to solve the dilemmas opposing artistic intentions and practical constraints that its production ran into. Building on this specific example and the literature, we identify categories of solutions, or strategies, for IRMT productions. We also comment on how such strategies and the concept of "Affect over Effects", specifically in the context of technology and performance [9], meet a frugal approach to reconciling art and technology.

## 2. RELATED WORK

### 2.1 Mixed Reality and Immersion

The term Mixed Reality (MR) along with its adjacent concepts Augmented Reality (AR), Virtual Reality (VR) and Extended Reality (XR) have been used, but not always defined, in various contexts and with varying meanings, leading to possible ambiguity, confusion and misconceptions [26, 20].

Specifically for MR, most definitions stem from Milgram and Kishino's taxonomy [19] that considers MR as a virtuality continuum spreading from AR to Augmented Virtuality (AV), described as an entirely virtual environment on which elements from the real world are added. The taxonomy consists of three dimensions: Extent of World Knowledge, Reproduction Fidelity, and Extent of Presence Metaphor. This definition has however evolved with Skarbez et al. [23] who point out that the continuum is discontinuous as perfect VR is unreachable, include standard VR under MR, and emphasize the role of users in the characterization of the experience.

In this paper, the concept of MR follows Rauschnabel et al.'s

<figure><img src="image_2.png" /></figure>

Licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0). Copyright remains with the author(s).
XReality framework [20] which considers it a subset of AR, separate from VR. It constitutes one end of the Assisted Reality-Mixed Reality continuum (cf. 1). The two ends of the continuum are contrasted by several elements, namely in this context the stability, consistency, and dimensionality of the content, interactivity, or shared experiences.

<figure><img src="image_3.png" /><figcaption>**Figure 1:** Rauschnabel et al.'s XReality framework for Augmented and Virtual Reality [20].</figcaption></figure>

The concept of immersion, critical in MR experiences, is defined as the extent to which technology can achieve an illusion of reality [24], and has three dimensions: Physical Presence, Social/Self Presence, and Involvement, which relate respectively to Plausibility, Interactivity, and Interestingness [15]. It is thus a culmination of these six aspects, and all should be taken into account in order to achieve an immersive experience.

## 2.2 Immersive Mixed Reality Theater

The concept of Mixed Reality Performance (MRP) is described by Benford and Giannachi as the staging of theatrical performances in MR environments [1]. This is a followup from many collaborations between Blast Theory and the Mixed Reality Lab, which started with the MRP *Desert Rain* [11]. It featured live performers, and a highly interactive environment shared between six participants, though in separate rooms, and underlined the importance of enabling illusion for the audience.

Immersive Mixed Reality Theater (IMRT) was later coined by Gochfeld *et al.* [6] to describe performances that combine the immersion of VR technology with actual people and physical objects. For them, IMRT brings to previous storytelling experiences like theater or cinema almost limitless options in terms of visual and audio effects. IMRT is illustrated with the piece *Holojam in Wonderland* which connects two actors and an audience of four inside a shared VR environment.

Building up from this work, Herscher *et al.* [8] designed the CAVRN framework and management system for larger-scale experiences. It is able to bring over 30 audience members into a shared co-located VR environment and is applied in *CAVE* [13], a show that only partially falls under the definition of IMRT as no live actors are present and the physical world had little involvement. They conducted surveys and interviews, which cemented the value of co-localized shared experiences, and noted the interest of the audience for interactivity without necessary consequence on the narrative.

A further iteration of a large-scale shared co-localized narrative experience this time in AR instead of VR was *Mary and the monster* [12]. It also challenges the definition of IMRT as there are no live actors and the audience is seated around a table where the scenes happen on a miniature scale, because of the limited field of view afforded by the AR glasses used.

An example of shared co-localized VR for 4 to 8 maximum participants can be found in Dreamscape Immersive's experiences<fnref n="1" />. They use external sensors to track the people as well as the props inside of the space, allowing their audience to interact with physical objects while in VR.

The use of props is a key aspect of *Bury The Wren* [9], a narrative performance which has a single spectator share the virtual space with a performer. Some of the props are shared between the two people and are also modeled and tracked in the virtual world as "Carbon Realities" using an external tracking system. The show also features AR segments, which brings it closer to the definition of IMRT and makes it share most of its challenges, despite its small amount of simultaneous users.

## 2.3 Reconciling technology and performance

Some IMRT creators give valuable comments on how they recorded technology and performance in their work. Benford and Giannachi developed the concept of trajectories, which corresponds to the unique experience each participant goes through [1]. They are not completely controllable, and are manipulated both by dramatical decisions and what the technology affords.

Through the design of *Bury The Wren* [9] a strong emphasis is put on the need for "*Affect over Effects*", or in other words the fact that what ultimately matters is the story and the experience provided to the audience rather than the technology used to achieve it. An illustration of this is the choice to lean into the low quality of their passthrough cameras and adding shaders to amplify its resemblance to thematically relevant daguerreotypes. This technical limitation guided their creative process, but remained a catalyst for the story, rather than making it an excuse for using any specific technology.

# 3. COMMON CHALLENGES IN IMMERSIVE MIXED REALITY THEATER

Commonly reoccurring challenges can be identified in existing IMRT productions. In particular, the four design hypotheses and choices for CAVRN [8] outline them well, they are:

1. Audience should be represented in the experience

2. Local proximity between members should be preserved

3. Audience should remain seated

4. Interaction should not be required to drive the experience

These hypotheses are, however, specific to the context of the production which strays slightly from the definition of IMRT as it is in VR, features no live actors, and has a seated audience.

## 3.1 Occlusion

The first challenge is occlusion, which is a major concern in AR settings, and relates to the first two hypotheses. Within this context, occlusion is defined as the process happening when the virtual and real layers of the image are merged and an element from the background layer is displayed in front of the foreground layer, usually because it is positioned closer [22]. This challenge is even more so important for MR, whose aim is to merge as closely as possible the physical and virtual environments [21], and thus becomes a critical issue in IMRT.

The two main approaches to solving occlusion are model-based and depth-based [22]. The model-based approach is most relevant for known static objects and consists in preemptively building and placing 3D models within the virtual environment, so they can mask other virtual elements [3, 10]. This approach is more

<footnote n="1">¹https://dreamscapeimmersive.com/</footnote>
complex for dynamic objects, though they can be tracked and the
model moved accordingly [5]. The depth-based approach is use-
ful for unknown environments, mainly using depth cameras [2] or
stereovision [5].

## 3.2 Spatial calibration

In the context of IMRT, spatial calibration refers to the alignment
of the virtual world onto the real world, allowing reliable place-
ment of the virtual objects in relation to the physical world, which
is necessary for model-based occlusion and shared experiences.

Systems may rely on different methods related to outside-in or
inside-out tracking to achieve spatial calibration. Outside-in track-
ing is based on cameras placed around the space that track markers
on the users [6, 9], since the position of the cameras and thus their
coordinate system is known, the physical position of each user is
easily accessible as long as they stay within the monitored space.
A device that uses inside-out tracking identifies features in the en-
vironment around itself and tracks their movement to deduce its
own position. Spatial anchors may be placed, which access<fnref n="2" /> or
build [7] the fixed coordinate system associated to the features of
the environment, and can thus be translated to physically located
points. Such spatial anchors may need to be initialized for each
headset, but they can also be shared across multiple devices [25].
Various protocols also exist to manually provide a correspondence
with the physical world to the coordinate system of the headset
[16].

Consistent spatial alignment is critical for hypothesis 2, shared co-located experiences, as everyone should be able to see objects at the same locations. Hypothesis 3, the choice of a seated audience, is also relevant to this challenge, as it was made precisely because of technological limitations preventing acceptable tracking on a large scale so that it ensures that avatars will remain aligned with the physical audience.

## 3.3 Shared experience of Mixed Reality

Sharing the experience of IMRT means in most cases that everyone
should be able to see the same things, at the same positions, at the
same time, which means that the environments are synchronized.
This is essential in order to ensure that the audience can share their
reactions, and that external stimuli remain coherent, such as exter-
nal speakers in VR [4]. Synchronizing all objects only truly be-
comes a challenge when these objects can be interacted with and
their behavior becomes unpredictable.

For non-interactive objects, synchronizing the time and space
between all headsets is sufficient. Space synchronization is tied to
spatial calibration in order to align the position of the virtual envi-
ronment on all headsets. A scripted timeline makes time synchro-
nization trivial and controllable [8], but cues can be implemented
for more flexibility [6].

Synchronizing interactive objects is possible through the net-
work as long as they are in limited number, such as articulation
points on avatars [6, 8], or tracked objects [9], since, as CAVRN's
fourth hypothesis points out [8], the strain put on the network
scales heavily with the number of objects and headsets involved.

How much of the environment is shared also plays a role in the
system's strain, and can be classified into three levels of collabo-
ration, ranging from the perception of each other to exclusive or
shared interaction with virtual objects [18]. A way to smoothly
allow a collective interaction with the position of the same object
is to introduce continuous forces instead of absolute control of the
position [17].

# 4. CASE STUDY: ANIMATE

Studying a concrete example of IMRT, its particular artistic and
practical context, and the choices that went into its design and pro-
duction, allows to learn more on ways to solve common IMRT
challenges.

## 4.1 Description

Animate is an IMRT work at the crossroads of performance, ra-
dio play and installation focused on a near-future Canada radically
transformed by climate change. The show premiered in August
2022 at the Kunstfest in Weimar, Germany and was experienced
by over 400 people in 10 days.

### 4.1.1 Artistic Intent

The plot of the story Animate focuses on two characters, Daniel
and Laurie, who are fleeing a near future of climate disasters. Emo-
tionally distraught from their hidden pasts, the pair set out on a
journey through the wild landscape of Newfoundland, Canada.
While news of global climate catastrophes is broadcast on their
car radio, they drive towards the Tablelands, a real Mars-like land-
scape in the middle of Gros Morne National Park where one can
walk directly on the Earth's mantle. Gradually, the climate-change
transformed landscape exerts a force on them as if it was a con-
scious, living, and breathing entity. It emits strange sounds and
affects Daniel and Laurie's personal relationship. But it is only in
the dramatic conclusion, where the earth comes alive in an apoc-
alyptic scene of rocks rising from the ground and attacking the
characters, that the intricate interconnectedness of humans and a
natural world under the throes of radical environmental change is
symbolically and viscerally manifested.

The basic dramaturgical strategy that evolved in the early phases
of the project involved collaborating with the author Kate Story to
adapt the short story and turn it into four dramatic scenes like a the-
ater text, then record actors speaking the text for a radio play and
finally use the radio play as a complex sonic accompaniment with
additional visual elements in MR for a live theater-like installation
with actors.

### 4.1.2 Technical Constraints

The set for the premiere of the show was the KET-Halle, a mas-
sive decommissioned agriculture machinery factory whose main
area spans over 144 by 120 meters with a ceiling of 10 meters
high at its lowest, though the actual area used both as a stage and
a wandering space for the audience still spanned 16 by 32 me-
ters. Inside this space, the Berlin-based MONOM studio has set
up its 4DSound system: an array of 48 omnidirectional speakers
that allow the physical placement of sounds within a 16 x 16 x 5
meters volume together with sub-woofers placed on the periphery
of the space. The audience wears Meta Quest 2 headsets, chosen
because of their inside-out tracking and accessible pass-through
features, and for their affordable cost allowing easier up-scaling to
12 people.

The two actors pull a rope to guide the audience, who is hold-
ing onto it, and who can only see a completely virtual scene. The
virtual environment then opens up into AR, revealing the physi-
cal stage, the rocks, and the actors, who are not wearing headsets
themselves. As the scene proceeds, the rocks start moving, rush af-
ter the audience, forms choreographies around the room, and gain
in intensity before falling together to the ground.

## 4.2 Addressing the technical challenges

Implementing the show means considering each of the technical
challenges in light of the artistic intentions with the practical con-
straints caused by the very specific context of the show. Not every

<footnote n="2">²https://developer.oculus.com/documentation/unity/unity-spatial-anchors-overview/</footnote>
issue can be completely solved, and some have to be mitigated or avoided with the final objective to limit as much as possible the negative impact it can have on the audience's experience while staying true to the underlying artistic intent. It will often come down to what is deemed acceptable for the experience of the audience and will not threaten their immersion.

### 4.2.1 Occlusion

The issue of occlusion only occurs for the rocks in the last part of the show where they begin floating and moving around since it is only then that the audience leaves the completely virtual environment. In this context, there are two categories: the stage and the setting are static objects, and the audience and actors moving around are dynamic objects. Preserving immersion would require implementing occlusion for both of those categories.

The use of models is possible for any of the static objects in the room and its walls as long as the spatial calibration works correctly, that is, if the headset is accurately aware of where it is in the real space. This has the added benefit of enabling collisions between the rocks and the modeled objects, which serves the intended merging of both environments.

The question of which elements of the room should be modeled and thus occlude the objects can however be raised. In fact, objects can be part of the natural decor, or be completely practical, such as fences or speakers, and should be considered carefully. Each object may thus provide occlusion based on factors like how conveniently they can be modeled and positioned, or how much their eventual occlusion will impact the overall experience. Occlusion can indeed take away from the show, hence it should not be applied blindly to everything. For instance, providing occlusion to fences or speakers would accentuate them by bringing them into the performance, when they should have been ignored.

<figure><img src="image_4.png" /><figcaption>**Figure 2:** View from a participant attempting to protect themselves with their hands, which are displayed in front of the rocks.</figcaption></figure>

In the case of dynamic objects, the hand tracking provided by the headsets is used to dynamically model a person's own hands (cf. Fig. 2). Attempting to model the other dynamic objects, the rest of the audience and the actors, with non-immersion-threatening quality would require at least partial body outside-in tracking, which quickly becomes too complex, both in terms of space and amount of people. Depth estimations would not only risk low accuracy with the higher distances encountered, but also would indiscriminately occlude any detected shape, without consideration for its value for the performance.

### 4.2.2 Spatial calibration

The large scale at which the tracking needs to operate excludes outside-in tracking, leaving its inside-out alternative which is already handled by the Meta Quest 2. However, the specific conditions of the space bring issues of instability to the system, namely

the high ceiling, varying natural light, and the feature-rich but spatially periodic environment, in particular the highly similar pillars that the headsets can mistake for each other. These unfavorable conditions can be alleviated to an extent. For instance, varying light can be tackled with more consistent electric light, or environment-related tracking issues can be helped by removing parts of the set that appear problematic, like mulch that had been spread on the ground. It is however not enough to guarantee perfect tracking at this scale, as tracking was found to be unreliably lost and reset in the middle of a performance, and even walking from one side of the stage to the other could introduce a drift from the initial calibration by over 10 centimeters.

Since the tracking system itself could not be trusted, the solution was to refrain from using highly repeatable spatial calibration protocols that relied on placing multiple points in order to guarantee the most identical alignment for all headsets. Instead, the issues that could happen at any point during the performance were acknowledged, and the focus was put on either shrinking their impact on the overall experience or making them fixable with little to no interruption of the person's experience. The positional offset caused by the drift was not experience-breaking in either scene, so only the radical change in alignment caused by an autonomous reset was a major issue.

The calibration protocol was downgraded to only a single point, the headset itself, to set both orientation and position. This allows technicians to make immediate decisions on the balance between speed and quality of calibration, by moving the person to the optimal spot, simply orienting them in place, or timing the reset to when the spectator looks in the right direction.

The choice between these options is left to the operators on a case-by-case basis, depending on how responsive the person seems to be, and whether it is more important to have the proper position at the cost of moving the person or if the orientation suffices in that part of the scene.

### 4.2.3 Shared experience of Mixed Reality

One of the goals of the last scene is to virtually drown the audience in a massive amount of flying rocks while giving them the option to protect themselves with their hands. Synchronization is an essential aspect of the audience experience here, as the rocks' visual behavior should match what the external sound system plays for all users.

Spatial synchronization can rely on the loosely similarly calibrated headsets. Coordinating the time for all headsets can be easily done by sending them a start message, which triggers a static timeline. When interaction comes into play, however, the state of the objects needs to be synchronized, which becomes tricky with how complex the environment is, as hundreds of independent rocks move, collide, and are interacted with in a massive physical simulation.

Because of limited network resources, the chosen solution was simply not to synchronize the state of the rocks and instead rely on the illusion and audience's assumption that the environment is actually synchronized.

#### Behavior of the Environment.

Not synchronizing interactable objects is in fact possible by designing the behavior of the rocks to be flexible around a fixed timeline. The rocks are pulled towards scripted targets which are the same for everyone thanks to time and space synchronization, yet they can still be pushed off course by the audience.

In order for the rocks to keep an acceptable similarity between headsets, the audience cannot be allowed to unsettle them too much. To achieve this, instead of using realistic simulated collisions on hands, which would allow slapping the rocks away with high bursts of energy, force fields with limited power centered around each hand continuously push away the rocks that are in a range of one
<figure><img src="image_5.png" /><figcaption>**Figure 3:** A participant appearing to be interacting with rocks as seen by another member of the audience.</figcaption></figure>

meter. The energy that can be given to a rock is thus more spread out in time, and less disruptive of its intended movements, which also ties in with the artistic intention of allowing the audience to connect with the rocks without letting them take the objects' agency away. Interacting from a distance rather than in contact also makes it less obvious to other members of the audience that what they each see is slightly different (cf. Fig 3).

Even though every audience member sees a marginally different environment based on how the rocks were pushed for each of them, the behavior of the environment on a macroscopic scale is the same. It would have been possible to also synchronize everyone's hands in order to still trigger the interaction with rocks floating close by in every instance of the environment. However, after weighing the instability of the spatial calibration against the limited impact of that synchronization, this was not implemented, since test audiences appeared easily overwhelmed by the chaotic swarms of rocks and paid small attention to the other people in the room.

### Audiovisual Coherence.

In Animate, rocks are audiovisual objects, as they have audio and visual representations, which should both be taken into account in order to maintain consistency and therefore the perception of immersion. The most important aspect of this coherence is that audio and visuals match both in position and nature. Thanks to the behavior of the rocks being determined on a macroscopic scale, the sounds they make can afford to be entirely scripted, which drastically simplifies the sound design work as it can focus on a single optimal case instead of a large spectrum of potential states.

The early approach of triggering sounds based on collisions of rocks made the difference between the static audio and the dynamic visuals too obvious since a collision could happen in one representation of the rocks and not the other, thus breaking the feeling of immersion. Instead, the nature of the sounds matches their scripted macroscopic behavior, like gathering into a tight cluster, forming a tornado in the center of the stage, or dropping all at once. The position of these sounds can also acceptably match the visual state of the environment by making them occupy a larger area, which makes their exact location harder to identify, and thus their slight incoherence harder to notice.

# 5. DISCUSSION

Design strategies can be deduced from the solutions that have been implemented for Animate and similar productions, and can be useful for the design practices of future IMRT works. They also each follow a frugal approach, whose aim is to meet a goal with technical restrictions.

## 5.1 Strategies for reconciling artistic intent and technological constraints

The strategies described here are all ways to mitigate the impact of technical issues on the desired audience experience. They can be used together in certain contexts, though they all come with varying impacts on the initial artistic vision.

### 5.1.1 Limit interaction or environment complexity

The first strategy consists in limiting the interaction or environment of the audience. The limit can be put on the interaction only, like by keeping the audience seated in order to reduce the size of the tracking area and to avoid line of sight issues for the outside-in tracking system [13, 14]. The limit can also be put on the environment only, for instance by miniaturizing it to the scale of a table and placing it in the middle of the audience, thus sidestepping the issue of occlusion [12].

Mostly, though, interaction and environment are intertwined and altered together. For instance, addressing networking limitations in shared environments can be done by limiting the proportion of the environment that is interactable: only avatars [6, 13], partial avatars [17] or a few narratively relevant objects [9]. This same issue can be addressed with the counterpart method of reducing the number of objects that are being synchronized, while still handling a larger amount of interactable objects, such as in Animate.

### 5.1.2 Exploit non-noticeability of imperfections

Not all imperfections are obvious to a non-vigilant audience, and can thus be preferred without much impact on the artistic goals to investing a large amount of resources and complexity on a more solvable issue.

For instance, in Animate, as long the spatial alignment is not completely different between headsets, the imperfection is hard to notice based solely on the reactions of the rest of the audience or the actors since it is impossible to identify exactly what part of the environment they are reacting to. This allows the designers to cope with the potential failure of headset tracking, which requires re-calibrating the headset. In fact, by drastically simplifying the calibration procedure, which is then faster to run at the expense of repeatability, they can limit the impact it may have on the audience when it has to be repeated during the performance. This does however impact parallel choices in the production, as it forbids occlusion on the audience or other critical objects since the misalignment would quickly become obvious.

Since a lack of occlusion can only be noticed when a real object is behind a virtual one, the staging can be decided accordingly. For example, the virtual rocks in Animate were placed in a specific manner, such that they were never behind the massive sub-woofers on the stage from the point of view of the audience when they discover them, thus preventing immediate visual discrepancies caused by the lack of occlusion.

We also used the limited capability of the audience to notice differences between their environment and the ones displayed for the others, to avoid the issues that come with the synchronization of dynamic objects at a big scale while preserving their feeling of a shared environment.

### 5.1.3 Integrate imperfections within the story

In specific contexts, or if the challenge was identified early in the production process, an issue can become a part of the narrative, and sometimes even strengthen it.

The poor quality of a pass-through camera and its gray-scale image can thus be adjusted with shaders in order to resemble a dawgureotype and thus match the historical context of the story [9]. Complementarily, in Animate, the quality of the virtual environ-
ment is altered in a way that blends the two together, reinforcing
the feeling that the virtual and physical worlds have merged. This
same merging can also be challenged by the absence of occlusion
between the rocks and the humans in the environment, underlining
the inherent frictions between both characters, as well as with their
environment.

## 5.2 A frugal approach to technology

What unites the previously described strategies is the attempt to
meet a goal with what the available technology can afford. They
also reveal that more technology, in greater complexity and variety,
is not always the right call in artistic contexts, and that showing
technological restraint and downscaling is a valid solution.

For instance, choosing less expensive headsets means that more
of them can be bought, although with lower quality. However,
this less than ideal quality can be worked with, like how grayscale
passthrough images can be integrated within the artistic vision in a
constructive way.

Entire technological layers can also be bypassed with the right
conditions and design decisions. For example, completely forgo-
ing synchronization between headsets and instead counting on a
reliable macroscopic behavior sidesteps many issues. One such is-
sue is informing the actors of the current state of the environment
without a headset, which we were able to accomplish by relying
on their sense of timing and occasional signals from technicians.

An overabundance of technology is rarely the right choice and
choosing which technology to use should always come with criti-
cal thought on its role in the dramaturgical vision for the produc-
tion. This principle lines up with the concept of "Affect over Ef-
fects" coined by Garrett [9], which pushes for technologically as-
sisted performances to make their audiences feel something rather
than experience it.

# 6. CONCLUSION

We presented the common technical challenges of Occlusion, Spa-
tial Alignment, and Shared Experience of Mixed Reality in IMRT
productions, and discussed some of their existing solutions. Then,
a case study of Animate provided detailed descriptions of solutions
to these challenges in a concrete context that weaves visual and
audio elements. The solutions were grouped into three strategies
consisting in limiting the complexity of the environment or interac-
tion, exploiting non-noticeability of imperfections, and integrating
the imperfections within the story. They outline the importance of
prioritizing the artistic aim and dramaturgical decisions over the
drive to amass technological solutions to achieve an effect and op-
timize an immersion, which is part of frugal approach to technol-
ogy. These strategies are meant to be used by future designers
of IMRT or similar experiences who wish to answer compromises
between artistic intentions and practical constraints.

# 7. ACKNOWLEDGEMENTS

This work was partially funded by grants from the Fonds de Re-
cherche du Québec (FRQ PRISME-ART) and the Hauts de France
region.

8. REFERENCES

[1] S. Benford and G. Giannachi. *Performing Mixed Reality*. The MIT Press, 2011.

[2] Y. Berdnikov and D. Vatolin. Real-Time Depth Map Occlusion Filling and Scene Background Restoration for Projected-Pattern-Based Depth Cameras. In *Graphic Conf. IETP*, 2011.

[3] D. E. Breen, R. T. Whitaker, E. Rose, and M. Tuceryan. Interactive Occlusion and Automatic Object Placement for

Augmented Reality. *Computer Graphics Forum*,
15(3):11–22, 1996.

[4] C. DeFanti. *Co-Located Augmented and Virtual Reality Systems*. Ph.D., Ann Arbor, United States, 2019. ISBN: 9781085678018.

[5] P. Fortin and P. Hebert. Handling Occlusions in Real-time Augmented Reality : Dealing with Movable Real and Virtual Objects. In *The 3rd Canadian Conference on Computer and Robot Vision (CRV'06)*, pages 54–54, June 2006.

[6] D. Gochfeld, C. Brenner, K. Layng, S. Herscher, C. DeFanti, M. Olko, D. Shinn, S. Riggs, C. Fernández-Vara, and K. Perlin. *Holojam in Wonderland* : Immersive Mixed Reality Theater. *Leonardo*, 51(4):362–367, Aug. 2018.

[7] W. He, M. Xi, H. Gardner, B. Swift, and M. Adcock. Spatial Anchor Based Indoor Asset Tracking. In *2021 IEEE Virtual Reality and 3D User Interfaces (VR)*, pages 255–259, Mar. 2021. ISSN: 2642-5254.

[8] S. Herscher, C. DeFanti, N. G. Vitovitch, C. Brenner, H. Xia, K. Layng, and K. Perlin. CAVRN: An Exploration and Evaluation of a Collective Audience Virtual Reality Nexus Experience. In *Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology*, pages 1137–1150, New Orleans LA USA, Oct. 2019. ACM.

[9] B. Kates. Incandescent Edges of the Future: Performance Creation with Virtual, Augmented and Carbon Realities. Master's thesis, Arts, Sept. 2020.

[10] N. M. Khan, X. Nan, N. Dong, Y. He, M. Kyan, J. James, L. Guan, and C. Davis. Towards a shared large-area mixed reality system. In *2016 IEEE International Conference on Multimedia & Expo Workshops (ICMEW)*, pages 1–6, July 2016.

[11] B. Koleva, I. Taylor, S. Benford, M. Fraser, C. Greenhalgh, H. Schnädelbach, D. vom Lehn, C. Heath, J. Row-Farr, and M. Adams. Orchestrating a mixed reality performance. In *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems*, pages 38–45, Seattle Washington USA, Mar. 2001. ACM.

[12] K. Layng, S. Herscher, and K. Perlin. XR for Everyone, Everywhere. In *ACM SIGGRAPH 2020 Educator's Forum, SIGGRAPH '20*, page 1, New York, NY, USA, 2020. Association for Computing Machinery.

[13] K. Layng, K. Perlin, S. Herscher, C. Brenner, and T. Meduri. CAVE : Making Collective Virtual Narrative: *Best Paper Award. Leonardo*, 52(4):349–356, Aug. 2019.

[14] K. Layng, K. Perlin, S. Herscher, and G. Zetter. Mary and the monster: chapter one. In *ACM SIGGRAPH 2019 Virtual, Augmented, and Mixed Reality, SIGGRAPH '19*, page 1, New York, NY, USA, 2019. Association for Computing Machinery.

[15] H. Lee. A Conceptual Model of Immersive Experience in Extended Reality. preprint, PsyArXiv, Sept. 2020.

[16] C. E. Leisi and O. Sahli. Co-Experiencing Virtual Spaces: A Standalone Multiuser Virtual Reality Framework. Dec. 2020.

[17] D. Lobser, K. Perlin, L. Fang, and C. Romero. FLOCK: a location-based, multi-user VR experience. In *ACM SIGGRAPH 2017 VR Village*, pages 1–2, Los Angeles California, July 2017. ACM.

[18] D. Margery, B. Arnaldi, and N. Plouzeau. A General Framework for Cooperative Manipulation in Virtual Environments. In M. Gervautz, D. Schmalstieg, and A. Hildebrand, editors, *Virtual Environments 99*, Eurographics, pages 169–178, Vienna, 1999. Springer.

[19] P. Milgram and F. Kishino. A taxonomy of mixed reality visual displays. *IEICE TRANSACTIONS on Information and Systems*, 77(12):1321–1329, 1994.
[20] P. A. Rauschnabel, R. Felix, C. Hinsch, H. Shahab, and F. Alt. What is XR? Towards a Framework for Augmented and Virtual Reality. *Computers in Human Behavior*, 133:107289, Aug. 2022.

[21] S. Rokhsaritalemi, A. Sadeghi-Niaraki, and S.-M. Choi. A Review on Mixed Reality: Current Trends, Challenges and Prospects. *Applied Sciences*, 10(2):636, Jan. 2020.

[22] M. M. Shah, H. Arshad, and R. Sulaiman. Occlusion in augmented reality. In *2012 8th International Conference on Information Science and Digital Content Technology (ICIDT2012)*, volume 2, pages 372–378, June 2012.

[23] R. Skarbez, M. Smith, and M. C. Whitton. Revisiting Milgram and Kishino's Reality-Virtuality Continuum. *Frontiers in Virtual Reality*, 2, 2021.

[24] M. Slater and S. Wilbur. A Framework for Immersive Virtual Environments (FIVE): Speculations on the Role of Presence in Virtual Environments. *Presence: Teleoperators and Virtual Environments*, 6(6):603–616, 12 1997.

[25] I. Sluganovic, M. Serbec, A. Derek, and I. Martinovic. HoloPair: Securing Shared Augmented Reality Using Microsoft HoloLens. In *Proceedings of the 33rd Annual Computer Security Applications Conference*, ACSAC '17, pages 250–261, New York, NY, USA, 2017. Association for Computing Machinery.

[26] M. Speicher, B. D. Hall, and M. Nebeling. What is Mixed Reality? In *Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems*, pages 1–15, Glasgow Scotland Uk, May 2019. ACM.