# Macbeth Metahuman: performing Craig's über-marionette in hyper-reality

Topher Maraffi

► To cite this version:

Topher Maraffi. Macbeth Metahuman: performing Craig's über-marionette in hyper-reality. Journées d'informatique théâtrale, Oct 2024, Avignon, France. hal-05138650

HAL Id: hal-05138650

https://hal.science/hal-05138650v1

Submitted on 1 Jul 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Left and Middle: Edward Gordon Craig stage designs for *Hamlet* (1910) and *Macbeth* (1909); Right: *Macbeth Metahuman* scene test in the NCSU Virtual Production Lab (2024)

<figure><img src="image_2.png" /></figure>

# MACBETH METAHUMAN: PERFORMING CRAIG'S ÜBER-MARIONETTE IN HYPER REALITY

Topher Maraffi

North Carolina State University

cmaraff@ncsu.edu

## ABSTRACT

An exciting application for virtual production is where actors appear to perform with life-sized digital doubles in a large LED wall for a location-based experience like a live show. The *Macbeth Metahuman* project combines Edward Gordon Craig's über-marionette or super-puppet concept with Curtis Hickman's hyper-reality design framework to create the illusion of intelligent actor-Metahuman interaction in a virtual production volume. Applying Craig's critique of theatre acting and descriptions of an über-marionette to digital doubles, I develop practical criteria for super-puppets in immersive experiences. Then I discuss the initial results of adapting the opening scene of Shakespeare's *Macbeth* for a hyper-reality treatment in a virtual production volume. Stage magic principles are used to hide the control mechanism for changing states in the Unreal Engine scene, so that a live actor wearing an inertial motion capture suit under their costume can puppeteer a hidden MetaHuman double that hits invisible trigger boxes in the volume, driving the dramatic spectacle through special effects and animated MetaHuman witches. I argue that this project fulfills the criteria for Craig's concept of a super-puppet in his Theatre of the Future, and that this expanded virtual production pipeline offers live actors a new venue of creative expression as performance capture artists.

## 1. INTRODUCTION

Recent Screen Actors Guild (SAG) strikes over fears of movie and game studios replacing actors with AI-generated digital doubles, along with the American Federation of Television and Radio Artists (AFTRA) strike by writers worried about studios using generative-AI like ChatGPT to write scripts, brought Hollywood production nearly to a standstill in 2023 [19, 22]. These strikes echo a similar outcry from actors more than a century earlier when influential theatre director Edward Gordon Craig called for all stage actors to be replaced with super-puppets. In his 1908 self-published essay *The Actor and the Über-marionette* [4], Craig argued that “acting is not an art” because it is not iteratively designed like painting or music and suggested that life-sized super-puppets could solve the problem. However, his description of an über-marionette was beyond the capabilities of any puppet that existed in Craig’s lifetime, bordering on an impossible or supernatural performing object that has only become technologically feasible with digital doubles in a location-based hyper-reality experience. “The actor must go, and in his place comes the inanimate figure-the Über-marionette we may call him, until he has won for himself a better name” [4]. Whether you call them digital doppelgangers, avatars, Synthespians, Personas or MetaHumans, we can now use photogrammetry and performance capture to simulate the likeness and gesture style of a physical performer for cinema, games and live experiences. A digital double,
unlike a digital twin that is a real-time simulation of a
physical person, is a performative facade that creates the
illusion of an actor's characterization. The versatility of
digital doubles is one of the reasons industry leaders like
Apple, Meta, Nvidia, Disney and Epic Games are
investing heavily in their development, and why Craig's
1908 argument for using life-sized super-puppets in live
theatre is relevant today.

Craig's vision of extending the art of puppetry into a
new actor-driven virtual performance medium devoid of
the physical actor was ahead of its time and can be a
useful way to frame digital doubles as super-puppets. But
a digital double becomes a super-puppet only when it
fulfills Craig's criteria of being performed live, with
actors or spectators, as an expressive performing object
in a technologically mediated space like a theater stage or
virtual production volume. While there continues to be
ethical issues with the unlicensed use of performer data
for generating digital doubles, the potential for actors to
have a new venue to express characters and stories is
what we are exploring in NCSU College of Design's
Virtual Production Lab.

The *Macbeth* MetaHuman Theatre project combines Craig's futuristic vision of a super-puppet with Curtis Hickman's stage magic concept of hyper-reality [8] to extend our virtual production pipeline into live performance applications that create the illusion of improvisation with MetaHumans in an LED wall. To create a believable stage illusion, it is important for the audience to know that they are being tricked but not directly perceive the mechanism for the illusion. This project takes advantage of the narrative framing of supernatural witches in the opening scene of Shakespeare's *Macbeth* to demonstrate virtual production technology can hide the control mechanism for hyper-reality illusions with MetaHumans in a live show. This is now technologically possible because a wireless Livelink connection allows an actor wearing an inertial motion capture suit under their costume to manipulate a hidden digital double in the virtual production volume with their full-body movements. Once the live actor is spatially represented in the Unreal Engine scene, the entire virtual production volume becomes a super-puppet mechanism through hidden triggers, so that the actor can perform the illusion of intelligent interaction with the virtual figures and world in the LED wall. This project is intended to demonstrate that hyper-reality design of actor-MetaHuman interaction in a virtual production volume fulfills Craig's criteria for an über-marionette, addressing the live acting problems he critiqued, and promises a new hyper-reality acting medium with real actors as the source artist.

## 2. REVISITING CRAIG'S ÜBER-MARIONETTE

Edward Gordon Craig (1872-1966) was a British
theatre actor, director, designer and author who self-
published his own performative periodical out of
Florence aptly titled *The Mask* (1907), where he wrote
under many pseudonyms to espouse his theories about the

future of theatre. As one of the first modern auteur
directors and theatrical designers, Craig took issue with
actors not correctly representing his artistic vision and
accidentally breaking character on stage due to nerves at
being watched, uncontrollable emotions from method
acting, or relying on too much personal charisma or
“personality”. He proposed to “...get the actor out of the
skin of the part altogether” by removing them from
audience view to replace them on stage with an über-
marionette crafted to symbolically represent the actor’s
refined character performance without the “...living
figure in which the weakness and tremors of the flesh
were perceptible” [4]. Craig scholar Eynat-Confino noted
that the Actors’ Union had recently been established in
England when Craig proposed his über-marionette
concept, which became “...a physical threat for the
actors” not intended since “...according to his theory, the
victory of the über-marionette or of any other instrument
for movement could be brought about only through the
actor’s transformation into an artist of the theatre”
through a “pure and perfect simulacrum” that Craig
described as “a make-believe” of the actor controlled by
“unseen forces” [7]. This was not a metaphor for the
perfect actor or mime as some believed after Craig
softened his tone in subsequent publications [10], nor a
Noh or Bunraku performer that impressed Craig and
likely influenced his über-marionette concept [9, 21], but
instead the vision of a virtual figurative instrument that
he did not have the technology to invent yet. He indicated
the virtual nature of his concept in 1912 when he wrote
“And did you think when I wrote five years ago of this
new figure who should stand as the symbol of man... to
see real metal or silken threads? What the wires of the
Über-marionette shall be and what shall guide him, who
can say? I do not believe in the mechanical ... nor in the
material.” [4]. Craig continued to develop his vision for
an über-marionette, along with a similarly inspired
Kinetic Stage of movable screens that his son considered
using remote control for in 1923 [7], but although Craig
indicated some type of remote control may be the
solution, he never saw his super-puppet concept realized
in his lifetime.

Craig scholar and curator of the Craig Collection at the
National Library of France, Patrick Le Bœuf, discounts
the speculation that Craig's über-marionette was just a
metaphor for the ideal performer and offers newly found
evidence that indicates Craig intended a full-body puppet
that covered a performer like armor and a mask [11].
Other recent scholarship from Degli Esposti questions the
contradiction in Craig's descriptions of an über-
marionette as a "mechanistic conception of ideal acting"
[6] with his public appreciation of contemporary actors
like Henry Irving and Eleonora Duse. Degli Esposti
points out that Craig's discourse is ontological and not
practical, with no concrete, detailed definition of the
über-marionette, and was a goal for Western acting
technique to develop "...a working technical solution for
the imperfection of acting" [6]. But Craig's imprecise and
vague statements make sense if he was searching for a
revolutionary new theatre tool that had not been invented
yet, as is his use of negations to “...define his ideal movement by elimination, by instructing ‘what not to do’.” [7]. My position is that we can take Craig seriously, even literally, when he said “If you can find in nature a new material, one which has never yet been used by man to give form to his thoughts, then you can say that you are on the high road towards creating a new art. For you have found that by which you can create it. It then only remains for you to begin. The theatre, as I see it, has yet to find that material.” [4]. The contradictions in Craig’s discourse fade away if what he had in mind was a virtual full-body puppet that was to be remote controlled through an actor’s refined performance – today’s digital double. I’m not saying that Craig knew he was describing a digital double in his lifetime. That kind of virtual production technology wasn’t feasible for live theatre in the twentieth century. I am arguing that the digital double realizes his über-marionette vision and that it has evolved organically as he predicted it would, only taking a century longer to be invented than he anticipated. When Craig described the über-marionette as, “...the actor plus fire, minus egoism; the fire of the gods and demons, without the smoke and steam of mortality.” [6], he makes poetic and practical sense when applied to a digital double as a super-puppet in a game engine, because the presence and liveness of a motion captured actor can be refined by animators to create a perfect live character performance without ego, accidents or physical limitations. This not only fulfills many of Craig’s über-marionette requirements, it is also why studios are adopting digital doubles in every type of production today. As Craig described, the digital double is an artistic representation of the actor, refined according to arts principles by sculptors and animators to display a likeness according to the director’s vision. It is a mediated illusion that captures and interprets the source actor’s physical presence from movement data, what Craig called “fire” and “noble artificiality”, and what Theatre Anthropology theorist Eugenio Barba would term a “dilated body” through “extra-daily” or stylized performance technique [1]. Digital doubles function as full-body puppets as Le Bœuf deduced, but also require the source actor to enhance their physical technique as Degli Esposti proposed, by projecting their characterization through the technology onto a virtual figure that has no inherent life. To facilitate better motion capture data, I teach my students to apply Disney’s Principles of Animation to performance capture, which Disney animators originally developed from studying live performers in the 1930s, and have been industry and critically proven to create the “illusion of life” in animated figures [20].

## 2.1. Criteria for Performing a Super-Puppet

By approaching Craig’s discourse as a guide for developing über-marionette criteria, we can identify practical features for designing a theatrical scene with a digital double that will function as a super puppet.

### 2.1.1. *Life-sized or larger than life figure.*

Craig described the über-marionette as life-sized or larger than life, “The Puppet had once a more generous form than yourselves.” [4], and Degli Esposti noted that his Über-Marionette International Theatre project manuscripts, “...provide specifications about their height, which should be 5 to 5½ feet, except for heroes and noteworthy people, which could be up to 6 feet tall, and gods, which should be 6½ feet tall and over” [6]. I’ve written about life-sized interactive Star Wars characters in The VOID’s *Secrets of the Empire* location-based VR experience, and how they created an aesthetic of being a spect-actor in a live show [17]. But life-sized digital doubles can also be seen in themed experiences like *The Bourne Stuntacular* (2020) and live concerts like *ABBA Voyage* (2022) that feature LED walls.

### 2.1.2. *Hidden mechanism for live character control.*

Craig wanted to hide the physical performer from the spectator’s view to reduce their nerves and ego, so that a super-puppet would function to get the actor, “out of the skin of the part altogether” [4]. He wanted to display an artistic representation of the actor, so that there would no longer be “... a living figure to confuse us into connecting actuality and art; no longer a living figure in which the weakness and tremors of the flesh were perceptible”. Craig repeatedly described the control mechanism for the über-marionette as unknown, hidden and not physical, “...many centuries ago these figures had a rhythmical movement and not a Jerky one; had no need for wires to support them” and “What the wires of the Über-marionette shall be and what shall guide him, who can say? I do not believe in the mechanical ... nor in the material.” [4]. Wireless actor control of a digital double is now possible using motion capture and a Livelink connection to a game engine. Using inertial suits that can be hidden under a costume hides the control mechanism as Craig described and can create a magical aesthetic through hyper-reality illusions.

*Iteratively refined performance based on actor data.*

Craig’s main complaint against live acting was that it was subject to “accidents” due to the performer’s human nature, and that, “Art only arrives by design... Therefore in order to make any work of art it is clear we may only work in those materials with which calculate. Man is not one of these materials.” [4]. He compared acting to the arts of painting and music and considered them superior because the product was iteratively refined until it perfectly captured the artist’s vision, whereas it was impossible to have an actor that always produced a perfect performance. However, performance capture allows actors to improvise and make creative choices with their bodies that interpret the script, and then animators refine the captured data according to arts principles and the director’s vision to perfect the characterization and remove any unwanted accidents.
When the refined performance is used in a real-time game engine, then state machines allow technical artists to trigger and mix the data in a live show to give the appearance of intelligent interaction with live actors.

### 2.1.3. Illusion of liveness and intelligent interaction.

It is clear that Craig expected the über-marionette to have an appeal of presence and liveness coming from the source actor that will be at least as expressive as the source actor and should go beyond through artistic refinement. “The über-marionette will not compete with Life-but will rather go beyond it. Its ideal will not be the flesh and blood but rather the body in Trance- it will aim to clothe itself with a death-like Beauty while exhaling a living spirit” [4]. Performance capture using MetaHuman Animator and motion capture suits can represent an actor’s performance through data, but the technology can capture more when the actor exaggerates their physical technique towards more stylistic gesture, as Craig suggested, “They must create for themselves a new form of acting, consisting for the main part of symbolical gesture. To-day they impersonate and interpret; tomorrow they must represent and interpret” [4]. Then artists must refine the data by setting keyframes based on animation and performing arts principles [20, 1] that push the performance further towards an expressive goal. This has the capacity to take an animated representation past uncanniness to a presence and liveness that makes an emotional impact, suggesting that the so-called uncanny valley effect is just an acting problem. Anyone who has cried at a Disney or Pixar movie can attest to the emotional impact of animated characters. A super-puppet should be capable of performing such an illusion of life effect in a real-time experience.

## 3. DIGITAL DOUBLES IN HYPER REALITY

I began making the argument for using Craig's über-marionette concept as the model for digital doubles in live theatre back in 2010 with my MFA thesis performances *Theatre of the Avatar: The Magic Mirror Game* at the University of California Santa Cruz, and in a 2011 Digital Humanities poster publication *Roots of Performatology: From Über-marionette to Embodied Performative Agent* [15]. My Performatology Master's thesis in computer science at UCSC demonstrated that Barba's extra-daily techniques are quantifiable in motion capture data of skilled performers, and support the premise that motion capture could function as a type of full-body control for a super-puppet [16]. In this initial work, I speculated on the real-time motion capture technology needed to complete the illusion of improvisation on stage with a digital double as Craig intended, but there were still some missing conceptual and technical components needed for the design and control of a super-puppet. The missing link was applying Curtis Hickman's hyper-reality design framework to motion capture control of a digital double in an LED wall so that the entire virtual production volume functions as

a magic theater to hide the control mechanism from the audience. Eynat-Confino pointed out that one reason Craig gave for desiring the über-marionette was “to restore Mystery to the theatre”, and that he associated mystery with “magical procedures of the director” [7]. Craig indicated an appreciation for stage illusions in his short dialogue *On Learning Magic*, “A great poet told us that all Art is a trick; therefore do not despise tricks.” [5]. Hickman, a professional magician turned visual effects artist who co-founded The VOID VR Theaters, used hyper-reality design principles to produce the most advanced location-based VR experiences from 2015-2019, after which all The VOID theaters closed due to the pandemic. In 2023, Hickman authored *Hyper-Reality: The Art of Designing Impossible Experiences* and detailed his fifty-two design “laws” to create the “...practical illusion of an impossible reality so convincing the mind accepts it as reality itself.” [8]. Hickman’s definition is distinguished from Baudrillard’s hyperreality concept [2] by both the hyphen and a practicing magicians focus on simulation as illusionism. However, he has stated that the goal of hyper-reality is experiencing an impossible effect, a simulacrum or copy of something that doesn’t exist, and quoted Baudrillard in his own book “Only what exceeds reality can go beyond the illusion of reality.” [3].

Since Hickman’s definition of hyper-reality is technology agnostic and not confined to VR, we began applying it to augmented reality and virtual production in my NCSU MADTech courses, and Hickman guest lectured to our students over Zoom annually from 2023 to 2025. I had already written about the importance of stage magic in immersive VR experiences [17], and how VOID location-based VR experiences like *Star Wars: Secrets of the Empire* (2017) functioned like a nineteenth century magic theatre. I also connected Hickman’s work to Dan North’s *Performing Illusions* thesis that stage magicians like Georges Méliès were foundational to cinema by developing an “aesthetics of the impossible” through special effects that ultimately led to virtual actors in film [18]. The illusionism North traces through the history of special effects and stop-motion film has accelerated with the convergence of digital postproduction technology and real-time game engines, moving special effects and digital doubles into virtual production sets where the illusions are performed live like in Victorian magic theaters. Digital doubles activate spectators to spot the trick in the spectacle, “The synthespian shows how, in the field of AI, reality itself is under inspection – there are attempts to assess the differences between the real and it’s representation.” [18]. This assessment is evident in YouTube reaction videos for Luke Skywalker’s appearance at the end of the *Mandalorian*, Season 2 (2020), where fans are frozen with eyes wide and shaking as they try to make sense of the impossibility of a seemingly young Mark Hamill, and then burst into emotion based on the success or failure of the digital illusion. These reactions to a digital double generally reflect how audiences respond to magic shows and other live illusions, such as acrobatic puppeteer
Pasha’s mannequin dance illusion on *America’s Got Talent* (2021), and location-based hyper-reality shows like *The Bourne Stuntacular* (2020) themed experience at Universal Studios and *ABBA’s Voyage* (2022) virtual concert in London. I argue that the VOID, Bourne, and ABBA experiences all create live illusions of digital doubles in hyper-reality that to varying degrees fulfill Craig’s criteria for a super-puppet. Non-location-based VR experiences like Agile Lens’ *Christmas Carol VR* (2021) and Ferryman Collective’s *The Severance Theory: Welcome to Respite* (2021) are also performed live through mocap, with life-sized or larger than life characters that to varying degrees fulfill our super-puppet criteria. While these VR theatre experiences create a live aesthetic through improvisation and audience interaction, such as the Dickens digital double breaking the fourth wall to say “Pardon me Topher” when moving into your avatar’s space (audience members are represented by colored lights with screen names), the degree of refined liveness varies. A behind the scenes tour of *Welcome to Respite* revealed many hidden triggers that actors are manipulating live to play real-time animation and effects.

## 4. MACBETH METAHUMAN THEATRE

Because the magic of special effects and hyper-reality are about giving the audience an impossible experience, some dramatic scenes and characters lend themselves to a hyper-reality treatment more than others. North argued that all special effects announce themselves by their performative spectacle and impossible aesthetics, and began his book with a quote from special effects historian Norman Klein’s nod to stage magic, “What good is sleight of hand if you cannot see the hand?” [18] to point out that illusions must be visible to be appreciated. The reason most special effects driven films and hyper-reality experiences are in the fantasy and science fiction genres is because they support illusionism through supernatural or speculative narrative framing to distinguish the aesthetics from everyday reality. As North puts it, “If it became totally ‘real’, it would cease to hold fascination as an illusion of reality.” [18], and Hickman’s goal for hyper-reality was “to take people to alternate realms that are beyond our own; to push them to higher levels of experience.” [8]. Such realms may be populated by digital doubles or fictional characters that have abilities that are beyond real people, such as larger than life superheroes, aliens, monsters or ghosts. Craig often described his über-marionette concept as like a god or ancient stone statues of Death come to life, and Le Bœuf pointed out “One could not stress too much how important, not to say central, ghosts, spirits, and the supernatural in general were in Craig’s theatrical conceptions.” [12]. Shakespeare in particular fascinated Craig, especially “impossible” scenes to stage that featured the supernatural, such as the opening witches scene in *Macbeth*. From 1920-22 he wrote an unfinished essay titled *Nine Ways of Opening Macbeth* [13] that I used as a guide for developing a hyper-reality adaptation with live actors and super-puppet witches in our virtual

production volume. Le Bœuf, who published the unfinished essay from the Craig collection, concluded “...what makes this scene so attractive to Craig is that it is imbued with the supernatural element.” [12]. Craig seems to confirm this reading by stating his appreciation for the mystical in the scene, and the need to take a nonstandard approach that goes beyond the meaning of the text, “...there is no modern playwright who deals so much in the mystical and the beyond words imaginative. His ghosts, his fairies, his magic islands and his magic transformations, are nothing to treat in a scholarly way. When you are forced to show them on a stage, you must take liberties.” [13].

It was also the shortness and dramatic flexibility of the opening scene, with only ten lines of dialogue, that was ideal for an imaginative interpretation that would “...strike the keynote of the character of the whole drama.”, as he quoted from the poet Coleridge [13]. In the unpublished essay, Craig also conjured the ghost of Shakespeare himself to performatively imply that an impossible approach was needed, which only Craig himself could deliver, “Shakespeare...drew in an immense breath (which took away the breath of London for the next three centuries), and slowly, deliberately, delivered himself of his ‘impossible’ plays—’for which an ‘impossible’ theatre may be built—by that fellow Craig, who is coming along,’ said he. ‘But which must be built,’ he added, ‘if ever they are to be performed in my way ... in my ‘impossible’ way.” [13]. Taking these hyper-reality friendly design suggestions as inspiration, the *Macbeth MetaHuman* project is intended to demonstrate that we can now build Craig’s kinetic theatre for performing “impossible” Shakespeare witches with MetaHuman super-puppets in a virtual production volume.

### 4.1. Hyper-Reality Adaptation of *Macbeth* Act 1, Scene 1: Three Witches Speaking

In his unfinished essay, Craig ruled out treating the three witches in *Macbeth’s* opening scene as old Scottish fortune tellers who take advantage of the superstitious, but instead treated them as supernatural immortals who are able to “vanish into the air” and “read the future”, and are better “...related to the old Greek idea of the Three inexorable Fates, than to any English and Scotch notion of hanky-panky.” [13]. This supernatural approach to the witches relates to many of his nine ways of opening *Macbeth*, including 2, 3, 4, 6, 7 and arguably 9. I blend elements of all these ways in my hyper-reality adaptation, except the first way, which Craig discounted as the “scholarly” or “usual way”, which constitute “a far too sane, a far too reasonable frame of mind.” [13].

Craig listed the nine ways in his essay [13]:

1. In the scholarly way, the so-called Shakespearean way, an attempt at reconstruction of what happened in the Globe Theatre in 1610.

2. The witches as the Three Fates.

3. As sorceresses of the Middle Ages.

4. As shades in the air.
5. As machine women, born of industrialism and all its terrors.

6. As demons—things which assume unheard-of shapes, as in the tales of the East.

7. Ditto, as they might be (if Philosophy could find it out) in the West.

8. The serio-comic way—something powerfully terrible, but comic too.

9. The marionette way of dealing with the whole thing.

In the article Craig equated the witches to artists as well as to the Greek Fates, much as he described the über-marionette, “Witches and artists are not unlike; at least they have no doubts; and so, being alike, we both are utterly in sympathy with ordinary hob-goblins, terrific banshees, and the whole line of mystery-workers, up and up, till we arrive at the Three Fates. Here our sympathy is tinged with more than a little respect—with even a little terror.” [13]. While he listed nine ways to open *Macbeth*, later in the article Craig described several other ways that are unnamed, and then more thoroughly proposes the impossible or ideal “Actors Way” that features famous ghosts and demons as spectators, and stars Isadora Duncan and Eleonora Duse who were the inspirations for his über-marionette concept [7], implying they are neither women nor human, “...nearer to the angels than are most humans, and nearer to the fiends than are most angels... You should be women, yet your genius forbids me to interpret that you are so”. [13]. The immortal Fates and three weird sisters are often described with beards and androgynous, so my adaptation can be played by any actor or digital double. Craig also considered the text of the scene to be practically meaningless, except the first and last two lines, but noted how “...extraordinary that without saying anything of value to the play, the mere sound of the words, the mere look of the thing, is so valuable.” [13]. Le Bœuf interprets his interest in the scene as the best way to “...illustrate the difference between drama and theater, between literature and mise-en-scène.” [12], which Craig also discussed in his über-marionette essays in relation to acting and the importance of symbolic representation over a literal interpretation of the text [4].

The ten lines of the scene as listed in Craig's essay are
[13]:

1st witch. When shall we three meet again? 1.

In thunder, lightning, or in rain? 2.

2nd witch. When the hurlyburly's done, 3.

When the battle's lost or won. 4.

3rd witch. That will be ere set of sun. 5.

1st & 2nd witch. Where the place? upon the heath. 6.

3rd witch. There to meet with Macbeth. 7.

All: I come, Graymalkin; Paddock calls:—anon. 8.

All: Fair is foul, and foul is fair 9.

Hover through the fog and filthy air. 10.

In my adaptation, the text is playfully interpreted into six sections that feature hyper-reality effects and magical transformations that are intended to generate a spectacle

of the impossible and blur the audience perception between live actors and MetaHumans in our virtual production volume. For instance, the first two lines that Craig said “...suit none but the terrible figures of high tragedy.” [13], give each witch their personas based on the elements of thunder, lightning, and rain, informing their magical entrances in my initial demo. Hickman categorizes these “overt illusions” as a magical “production (or appearance)” [8], considering it one of the easiest (and weakest) hyper-reality illusions to stage, making it a good choice for launching the scene. The elemental characterization guided my own live performance as the Rain character, and motivated my gesture style, voice choices, and special effects for the Thunder and Lightning MetaHuman animations. As Craig suggested, I take liberties in directing which character speaks the lines to enhance the illusion of interactive dialogue between the live actor and MetaHumans. Here is my script adaptation for the first two lines (the full scene script is available on the project webpage):

*   **Witch 1 enters from stage left carrying a two-faced skull with glowing coals in the open crown.** Witch 1 walks to center stage a couple of feet in front of the LED wall showing the barren and dark Scottish heath scene. Upon setting the skull on center stage close to the audience, explosions occur in the wall scene and physical lights, and witch 1 notices their shadow on central standing stone. After playing with their digital shadow and breaking the 4th wall, Witch 1 does a rolling gesture towards the left standing stone to conjure some fog and says first line "When shall we meet again?" to the empty landscape like someone hidden is there.

*   **Witch 2 MetaHuman steps out of the fog fluidly and says in a rolling deep voice, "In thunder,...".** Witch 1 spins around and does a fast gesture towards the right standing stone to trigger a lightning strike and Witch 3 MetaHuman drops into scene with a flourish and says "lightning,..." in a sharp high voice, making live Witch 1 jump back towards center stage. Then Witch 1 gestures upward to hit third trigger and pulls down a cloud and rain effect, while saying their line "or in rain?".

Craig designed several Shakespeare productions that influenced my stage design, such as his scenic and costume designs for both *Macbeth* (1908) and *Hamlet* (1910/1929 Cranach Press) including the cubist look of his etchings being reflected in my standing stones that also function as projection surfaces for digital shadow play by the live actor, and his Cranach Press bird-themed Hamlet costumes informed the black crow aesthetic of my masked MetaHuman witches.

## 4.2. Actor-MetaHuman Interaction Research in a Virtual Production Volume

This hyper-reality adaptation of the opening scene of *Macbeth* is meant to demonstrate some of the ways that a
theatrical treatment of MetaHuman interaction in a
virtual production volume can create mixed or extended
reality effects (MR/XR) for a live audience that don't
require expensive and distracting headsets, while still
maintaining the aesthetics of a traditional play experience
in a shared theater space. My research question for this
project has been: Can an actor use motion capture to drive
life-sized MetaHuman reactions in a virtual production
volume so that there appears to be expressive,
improvised, and supernatural interplay between the real
and virtual figures in a live performance? To answer this
question, we are treating MetaHumans as performing
objects that should fulfill Craig's criteria for an über-
marionette or super-puppet. I therefore designed a series
of increasingly sophisticated hyper-reality effects in the
prototyping phase that require real-time blending of
refined performance capture data in Unreal Engine 5.x,
and experimentation with new tools like motion matching
for MetaHumans. My design takes advantage of a space
in front of our LED wall where live actors can appear to
affect the virtual world using a hidden MetaHuman
double they control through a Livelink connection to our
Rokoko or Noitom inertial motion capture suit and
gloves. By taking this approach, the entire virtual
production volume, including the Unreal Engine scene
with MetaHumans in the LED wall and the actor wearing
the mocap suit on the physical stage with practical
lighting and effects, functions as a super-puppet control
mechanism being performed by the actor because the
MetaHuman super-puppet cannot be separated from the
digital "strings" that move it.

As my initial demo performance shows on the project
webpage, wearing the Rokoko Smartsuit Pro 2 motion
capture suit and gloves under a basic feathered costume
and crow mask, I proceed to puppeteer a hidden
MetaHuman digital double that only shows up as a digital
shadow on the standing stones in the LED wall. As I
move through a series of dramatic actions in the virtual
production volume, I cause my MetaHuman's right hand
to hit a series of hidden trigger boxes placed strategically
in the Unreal Engine scene and overlapping the real space
in front of the wall. The four trigger boxes are designed
to change the global game engine state so that specific
visual effects, sounds, and animation will play when
triggered, and can only be fired when the prerequisite
state is active to avoid accidental triggering while I'm
moving through the space. The hidden MetaHuman has a
collider attached to a socket on the right hand joint, so
that as a performer I can work out the spatial positioning
of the trigger boxes when rehearsing and then hit my
"marks" according to improvised timing and rhythm (see
Figure 1). This allows for some creative choices in how
and when I deliver my lines and strike the next trigger
box with my right hand. For instance, in the beginning
when I walk in carrying a hollow skull containing
glowing LED "coals", how long I take or where I walk is
up to me, as the scene won't progress to the initial "Fire"
state until I move my right hand down to the ground to
hit the first trigger box. The fact that I am holding a
physical object in my hand is irrelevant to the interaction

dynamics of the scene. When the hand collides,
explosions occur and the flickering campfire light that
casts the digital shadow is turned on, along with a chime
sound that alerts me as a performer that the state has been
successfully entered. The remaining trigger states of
Thunder, Lightning and Rain are hit in the same way to
move the scene forward dramatically and reflect the
elemental characterizations of each witch through sound,
visual effects, and refined animation in the wall and
simultaneous physical effects in the volume.

<figure><img src="image_3.png" /><figcaption>**Figure 1.** Trigger-box testing by the live actor performing their MetaHuman doubles through spatial hyper-reality design in the virtual production volume (NCSU, Fall 2024).</figcaption></figure>

5. DEMONSTRATION & FUTURE WORK

The initial demonstration on the [project website](https://macbeth unreal engine.com/) shows the hyper-reality illusion of the *Macbeth* Unreal Engine scene in the LED wall dynamically reacting to the presence and movement of a live actor in costume, by casting a believable and responsive shadow onto the virtual standing stones in the background, followed by magical productions of life-sized MetaHuman witches that seem to respond to the live actor's dialogue by delivering their own Shakespeare lines. The success of the hyper-reality effects and supernatural aesthetic of the *Macbeth* scene relies on hiding the control mechanism that the actor is using to puppeteer the virtual witches. The hidden "strings" of the über-marionette in this case are the entire virtual production system functioning like a magic theatre to mediate perceptual illusions between the actor and the audience that blur staged realities. The virtual effects are enhanced by physical lighting and smoke that reflect the dynamic Unreal Engine environment in the LED wall. The virtual production controls in both the physical space and virtual world include an inertial motion capture suit worn under a Craig inspired crow costume that gives the actor real-time manipulation of a hidden MetaHuman double, and invisible trigger boxes in the volume that are spatially and temporally designed to drive the Unreal Engine scene forward through global state changes. Rehearsal for the live actor is an intuitive extension of stage blocking and improvisation by physically learning where the trigger boxes are positioned in the volume (aided by an attached sphere to visualize where the hidden MetaHuman's right
hand is located in the virtual space), and then puppeteering the scene through full-body gesture while performing the live witch role (see Figure 1). This design approach improved on earlier tests that had the actor controlling a visible MetaHuman in the scene like the dynamics of a Bunraku puppeteer, also with an inertial motion capture suit and hidden trigger boxes, but did not create the same perceptual illusion between the live actor and virtual scene. Equally important for the actor, the super-puppet control was not intuitive because the performer had to continually watch the LED wall and mentally mirror their full-body gesture to effectively puppeteer the MetaHuman in the virtual space.

In spring semester 2025, I will be working with NCSU MADTech graduate students to further develop my adaptation for the opening of *Macbeth*, with more sophisticated hyper-reality effects that blend a library of performance capture variations for each witch character to increase the illusion of intelligent MetaHuman reactions to the actor's live improvisation on stage. This development will include weighted blending of MetaHuman data with real-time gaze tracking to have the virtual witches appear to look at the live actor or track objects in the actor's hand and then react accordingly. For instance, one of the sections is designed so the actor picks up some LED "coals" from the skull prop, and then threatens to throw them at a MetaHuman witch. Hidden trigger boxes placed high and low will allow the live performer to improvise whether to throw the LED lights towards the MetaHuman's head or feet, triggering different fire visual effects and animation variations making the MetaHuman duck or jump. We also plan to enhance the hyper-reality effects by building up the physical set through collaborations with NCSU College of Design's Virtual Production Lab personnel and University Theater faculty, including a more immersive set, better DMX image-based lighting, and computer-controlled practical effects. Lastly, the goal of this work is to have professional Shakespearean actors perform the witch roles for the live actor and MetaHuman doubles, to show the potential for this technology to revolutionize live drama in the hands of performing artists, as Craig intended for his Theatre of the Future.

## 6. REFERENCES

[1] Barba, E., & Savarese, N. *A Dictionary of Theatre Anthropology: The Secret Art of the Performer* (2nd ed.). Routledge. 2005.

[2] Baudrillard J, Glaser SF. *Simulacra and simulation*. Ann Arbor: University of Michigan press. 1994.

[3] Baudrillard J. *The perfect crime*. Verso. 1996.

[4] Craig, E. G. *The Actor and The Über-Marionette*. The Mask Periodical, Vol 1 No 2, Florence: Self-published. 1908. *Gentlemen, the Marionette!* Vol 5 No 2. 1912.

[5] Craig, E. G. *Craig On Theatre: On Learning Magic*. Pg 98, Internet Archive. 1983.

[6] Degli Esposti, P. "The Fire of Demons and the Steam of Mortality: Edward Gordon Craig and the Ideal Performer". Theatre Survey, 56(1), 4-27. 2015.

[7] Eynat-Confino, I. Beyond the Mask: Gordon Craig, Movement, and the Actor. Southern Illinois University Press. 1987.

[8] Hickman, C. *Hyper-reality: The art of designing impossible experiences*. Independently published on Amazon. 2023.

[9] Innes, C. *Edward Gordon Craig: A Vision of the Theatre*. London: Routledge. 1998.

[10] Leabhart, T., and Leabhart, S. "Edward Gordon Craig's Übermarionette And Étienne Decroux's "Actor Made Of Wood"," Mime Journal: Vol. 26, Article 6. 2017. DOI: 10.5642/mimejournal.20172601.06

[11] Le Boeuf, P. "On the Nature of Edward Gordon Craig's Über-Marionette". New Theatre Quarterly 26(2): 102-114. 2010.

[12] Le Boeuf, P. "Two Unknown Essays By Craig On The Production Of Shakespeare's Plays". Mime Journal, Vol. 26, Article 14. 2017.

[13] Le Boeuf, P. "Nine Ways Of Opening Macbeth," Mime Journal: Vol. 26, Article 15. 2017.

[14] Lyons, C. R. "Gordon Craig's Concept of the Actor." Educational Theatre Journal, vol. 16, no. 3, pp. 258-69. 1964.

[15] Maraffi, C. "Roots of Performatology: From Über-marionette to Embodied Performative Agent". Proceedings of Digital Humanities Conference, Stanford. 2011.

[16] Maraffi, C., S. Ishikawa, and A. Jhala. "Inferring Performer Skill from Aesthetic Quality Features in a Dance Game Gesture Corpus". Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, vol. 9, no. 2, 2021.

[17] Maraffi, C. "Stage Magic as a Performative Design Principle for VR Storytelling". Cinergie-Il Cinema E Le Altre Arti, 10(19), 93-104. 2021.

[18] North, D. *Performing Illusions: Cinema, Special Effects, and the Virtual Actor*. London – New York: Wallflower Press. 2008.

[19] Scherer, M. "The SAG-AFTRA Strike is Over, But the AI Fight in Hollywood is Just Beginning". AI Policy & Governance, Privacy & Data, Jan 24, Center for Democracy and Technology. 2024.

[20] Thomas, F. and Johnston, O. *The Illusion of Life: Disney Animation*. New York, Hyperion, 1995.

[21] Yamaguchi, Y. "Edward Gordon Craig and the International Reception of Bunraku: Transmission of the Puppet Images in Gakuya zue shūi, and the Networks of Modernism between Japan, Europe, and the United States." Forum Modernes Theater, vol. 28 no. 2, 2013.

[22] Yandoli, K., and Richardson, K. ""An Existential Threat": Actors Troubled by SAG's AI Deal With Hollywood". Rolling Stone Magazine, Nov 17th. 2023.