Motion Capture Body Tracking and Functional Safety in Dynamically
Controlled Theatre Automation Systems
Authors: Lisowski Daniel 
Published: July 2023

Live theatrical performance is an ever-evolving art form in which
visionary theater makers are incorporating evolving technologies into
performances to connect and engage modern audiences. Recent developments in
theatrical motion control systems are enabling vibrant and adaptive control
through dynamic automation. Traditional theatrical motion control systems
use a set path motion profile to produce predictable movements of scenery
and people through space and time. New dynamic control systems utilize an
external generated set point parameter to specify the desired motion. This
flexibility enables a DJ to control the movement of flown digital
chandeliers above a dance floor with their regular beat control or for Alice
to control her flown decent as she tumbles down the rabbit hole by changing
the position of her arms. Putting the control of the equipment into the
hands of the performers is a significant leap in evolution stage automation
control.To accomplish the motion tracking of performers on stage, the
research team developed a unique set-up of Microsoft Kinect sensors to
monitor the stage environment. The skeletal models of up to four performers
were captured and transmitted via an ethernet network to the video game
engine. The main benefit of this tracking method is that it allowed for both
traditionally costumed performers and regularly dressed audience members to
be captured by the system. Most motion capture systems in entertainment
require the use of custom body suits which can disenchant and distract
audiences. Our system maintains the traditional relationship between costume
and character while collecting the necessary interactivity data.The natively
3D nature of the video game environment is optimally suited to handle the
interaction between performer and the digital environment. Using the
skeletal tracking models from the sensor system, the video game engine
allowed the performers to dynamically interactive with the stage
environment. They could open doors, swat away birds, and flap their wings to
fly away. The added benefit of the system was that it allowed members of
youth young audiences to be called onto stage to further the narrative with
their actions. The resulting system enables a new performance methodology
with exciting new options for theatrical storytelling, educational training,
and interactive entertainment. This evolution of control adds risks beyond
those present in deterministic systems.Dynamic control of entertainment
automation systems requires strong conformance to functional safety
principles to mitigate the risks to affected personal and environments.
Functional Safety is the mitigation of unacceptable injury risks through the
implementation of one or more automatic protection functions (often called
safety functions). In traditional theatre environments, safety functions
have historically been limited to “Emergency Stop” functions which halt all
motion when an operator presses “the big red button”. As systems become more
complex and utilize dynamic control, theatrical automation control systems
need to utilize a functional safety evaluation process to better protect
persons and the environment for systematic and random failures in the
systems. Industry experts develop best practice procedures to address safety
concerns which are written into documents called standards.The presenter
conducts research at the intersection of entertainment and engineering, has
actively participated in the development of numerous American national
standards with the Technical Standards Program of ESTA (Entertainment
Services and Technology Association), and is a dual certified functional
safety engineer (TÜV Rheinland and Underwriters Laboratories). This
presentation will go through the functional safety considerations required
to implement dynamic control per national and international standards using
examples from realized performance projects led by the presenter.