Boiling Mind: Amplifying the Audience-Performer Connection
through Sonification and Visualization of Heart andElectrodermal Activities
Moe Sugawa
Taichi Furukawa
George Chernyshovmoe.sen.se321@gmail.com
taichi@kmd.keio.ac.jp
chernyshov@kmd.keio.ac.jp
Keio University

Karola Marky

marky@tk.tu-darmstadt.deTechnische Universität Darmstadt

Danny Hynds
Jiawen Han

Marcelo Padovani
Dingding Zheng

dannyhynds@kmd.keio.ac.jp
hanjiawen@kmd.keio.ac.jp
Keio University

mapama@kmd.keio.ac.jp
zheng208@kmd.keio.ac.jp
Keio University

Kai Kunze

Kouta Minamizawa

kai@kmd.keio.ac.jp
Keio University

kouta@kmd.keio.ac.jp
Keio University

Figure 1: "Boiling Mind" dance performance, using audience physiological signals as visual/sound elements on stage.

ABSTRACT
Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than theauthor(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specific permission
and/or a fee. Request permissions from permissions@acm.org.
TEI ’21, February 14–17, 2021, Salzburg, Austria© 2021 Copyright held by the owner/author(s). Publication rights licensed to ACM.
ACM ISBN 978-1-4503-8213-7/21/02. . . $15.00
https://doi.org/10.1145/3430524.3440653

In stage performances, an invisible wall in front of the stage often
weakens the connections between the audience and performers.
To amplify this performative connection, we present the concept"Boiling Mind". Our design concept is based on streaming sensor
data related to heart and electrodermal activities from audience
members and integrating this data into staging elements, such as
visual projections, music, and lighting. Thus, the internal states of
the audience directly influence the staging. Artists can have a more
direct perception of the inner reactions of audience members andcan create physical expressions in response to them. In this paper,

TEI ’21, February 14–17, 2021, Salzburg, Austria

we present the wearable sensing system as well as design considerations of mapping heart and electrodermal activity to changes
in the staging elements. We evaluated our design and setup overthree live performances.

CCS CONCEPTS• Applied computing → Performing arts.

KEYWORDS
Interactive Dance performance, Audience engagement, Physiological signals, Embodiment
ACM Reference Format:
Moe Sugawa, Taichi Furukawa, George Chernyshov, Danny Hynds, Jiawen Han, Marcelo Padovani, Dingding Zheng, Karola Marky, Kai Kunze,
and Kouta Minamizawa . 2021. Boiling Mind: Amplifying the AudiencePerformer Connection through Sonification and Visualization of Heart andElectrodermal Activities. In Fifteenth International Conference on Tangible, Embedded, and Embodied Interaction (TEI ’21), February 14–17, 2021,
Salzburg, Austria. ACM, New York, NY, USA, 10 pages. https://doi.org/10.
1145/3430524.3440653

1

INTRODUCTION

Live performances can be enchanting experiences for both audiences and artists [3]. However, even if artists carefully follow astrict script, each live performance is different since the involvement of the audience changes. Although the audience is involved,
there often is a separation between the audience and the artists onstage. Thus, the connection between the audience and the artists israther low.
In this work, we are exploring means to strengthen this connection. We present a stage production aiming to integrate theaudience into the process of aesthetic creation via physiologicalsensing. We call this approach "Boiling Mind".
To realize "Boiling Mind", we expressed physiological signals
from the audience during the performance via changes in common
staging elements, such as projections, sound, and light elements.
The reactions of the audience are expressed through these staging
elements and become a part of the performance. Additionally, artists
on stage adapt their performance to the audience which createsa performative feedback loop. With the projections covering thewhole stage performers can see the representation of the audience’s
physiological data and adjust their movements to create a calmer
or more intense atmosphere.
The project is based on the artistic practice and embodied philosophy of Artistic Director Moe Sugawa [47]. The team included amulti-disciplinary artist (artistic director who is a dancer/researcher),
one choreographer, six dancers, one composer, one interactive video
developer, one interactive lighting developer, three wearable developers, one lighting engineer, one sound engineer, and one stage
manager. As a team, we produced a 60-minute dance performancein a semi-democratic process.
This paper focuses on the artistic aspects and concept design
rather than data analysis and lab studies. The dataset of the three
performances will be published for the research community. This
work was conducted "in-the-wild", with a contemporary dance ensemble and audience members (not recruited lab participants). The

choreography of the performance and other artistic decisions weremade by the choreographer and the artistic team under feedback
and discussions with the engineering and design members of theteam.
The main contributions of this paper are as follows: (1) We
present the concept of "bringing the audience on stage" by mapping
physiological signals from audience members to stage elements
to amplify the connection between artists and the audience. We
detail information about the design process, focusing on sensing
heart rate and electrodermal activity (EDA), as well as how thesesignals are translated to the onstage elements through sonificationand visualizations. (2) We describe an open integrated wearable
system that consists of a wrist band, back-end processing, and output. Our proposed system enables the capturing and translating ofphysiological signals into stage design elements. (3) We realized
our proposed concept as a proof-of-concept with the performance"Boiling Mind". The performance involved dancers, choreographers,
researchers, visual and sound designers in the process. It was performed three times in front of an audience of 57, 37 and 45 people. In
each performance, around 30 audience members wore the wearable
wrist bands. (4) We conclude with discussing qualitative feedback,interviews and lessons learned from both the audience and artists.

2

BACKGROUND

The fundamentals of our work are based upon findings on physiological signals, audience engagement, interactive performances.

2.1

Psychophysiological Signals

Electrodermal Activity (EDA) is a property of human skin, in particular, changes in the skin conductance level. It can be viewed as
a combination of two components: 1) tonic (slow) and 2) phasic(fast). The phasic component is associated with emotions or stress.
Despite being over a century old [26], it remains one of the mostwidespread tools for the measurement of autonomic nervous system responses in psychology and psychotherapy [37, 45]. EDA is avery sensitive marker and is often used to assess emotional arousal[1, 6, 31, 46].
Heart rate (HR) and its variability (HRV) are considered to result
from the autonomic system (ANS) activities. The neurovisceralintegration model describes HRV as the result of prefrontal cortex
activities that affect modulation of the parasympathetic (PSNS) andsympathetic (SNS) nervous systems balance [44]. In other words, theneural circuitry that affects HRV goes very deep into our brain andreflects higher-level cognitive processes and emotional states. HRV
has been shown to reflect worry and anxiety [9, 25] and emotions[12, 36] among many other characteristics. In this work, we use
the ratio Low Frequency/High Frequency (LF/HF) as an indicatorof the balance between PSNS and SNS activities, as it is one of the
HRV features that is easy to interpret as the excitement indicator
[42].

2.2

Audience Engagement

Audiences see active and connected forms of engagement as indicators of performance quality and that this influences the reattendance of a performance [38]. The engagement of an audience
is difficult to assess since it is an implicit emotion involving both

Boiling Mind: Amplifying the Audience-Performer Connection through Sonification and Visualization of Heart and ElectrodermalTEI
Activities’21, February 14–17, 2021, Salzburg, Austria

valence and arousal[30]. Within this scope, subjective assessments,
such as interview results or questionnaires, are general methods
to understand audience engagement [7, 33]. To gather information
about the engagement of the audience in real-time, the availabilityof physiological measures which can reflect ANS activity have been
explored [2].
Some researchers investigated the possibilities of Brain-Computer
Interfaces (BCI) via electroencephalography (EEG) sensing to reveal audience responses. Ramchurn etal. developed an interactive
brain-controlled film where each scene’s narrative was determinedby one audience member’s EEG attention data [39]. Furthermore,in Hassib etal. ’s research, they extended the use of EEG sensing
beyond personal tracking to the aggregation of multiple users’ engagement information as feedback to presenters[20]. In our study,
we recorded and presented all of the audiences’ physiological data
as performance elements to provide a comprehensive and close
connection between the audience and performance.
Existing literature has also explored the feasibility of using cardiovascular values recorded by an electrocaardiogram (EKG) and
EDA to measure audience involvement and engagement. Latulipe
et al. have conducted a series of studies to explore measurement
for the arousal component of engagement via EDA. Their study
indicated EDA related to the performance as interpreted by arts experts and self-reported engagement by audiences, which supported
EDA’s validity of audience engagement [32]. Lin et al. found that
EDA positively correlated with subjective ratings for presence, emotional arousal and engagement but not for HR. In addition, Perttulaet al. found that HR measurement could be used to increase thepositive event experience when designed properly. However, theyalso stated that their HR measurement belts need to be placed underclothes which seemed to be at risk of violating audience privacy. In
our setup, we offered the audience wristbands to reduce hesitationand insecurity. Instead of using HR directly, we used HRV measurements and EDA together to collect subtle engagement changes in
the audience.

2.3

Interactive Performance

Dancing was originally an art where bodies moved in response tounconscious emotions to interact with others in rituals and festivals
such as African dance and Kecak [17]. As a co-dependent activity,
the connection between dancers and audience has fascinated bothresearchers and practitioners [5, 16, 24, 38]. Staging elements andempathetic choreography are used to elicit an interpretive andaffective response from a passive audience [13].
Throughout the last century, artists like Nam June Paik [18] andJohn Cage [14] have often relied on interactions from the audiencein their art pieces and performances. John Cage considered musicarising through both performers and so-called audiences’ activity as
a valuable element [28]. In Cage’s work, the audience can contributeto the performance process by contributing seemingly non-musicalsounds such as adjusting their seat or coughing [19].
In recent years, readily available technologies, such as wearable bio sensors and digital media, have enabled artists to drive
the performance creation with the support of HCI researchers[4, 7, 13, 41, 43]. Some of them have been exploring the interactions
between performers and stage design[10, 27, 40]. There are also

attempts to integrate the audiences’ senses and feelings into the performance environment, which is similar to the main concept of ourstudy. Klaus Obermaier’s team has implemented an interactive system where audience members can co-create the sound environment
via the Letterbox smartphone application consciously [34]. Audience’s involuntary emotional feelings can also be transformed into
messages [11], emotional indicator icons [22] and virtual avatars[8] during a live performance.
In this work, we aimed to build an audience-performer connection across the invisible barrier of the stage by including theaudience’s emotional feedback as part of the performance. Inspiredby previous researches, we designed Boiling Mind, in which all
the staging elements including visualizations, music, and lightingchanged dynamically based on physiological signals collected by
wearable sensors attached to the audience. The audience can contribute to creating and shaping the performance environment aswell as the dancers’ expressions.

3 CONCEPT DEVELOPMENT
3.1 Concept
The stage design and described setup came out of a collaboration
between the performing artists, designers and researchers in asemi-democratic process.
Both researchers and artists contributed to developing the concept during the iterative process to balance research and artistic
interests. This process was mainly led by artists, especially thechoreographer. Besides the predetermined story and design, thechoreographer also created some improvisational sections where
dancers could interact with stage elements affected by audience
physiological response.
During the performance, the physiological data of the audience
was captured using custom built smart wristbands. The physiological data includes heartbeats, EDA, and the LF/HF ratio related tothe heart rate variability. The staging elements changed based on
the audience’s physiological data.
This design aimed to deepen the connection between the audience and the dancers into a collaborative relationship by building
a collective experience through the use of physiological data andreactive stage design.

3.2

Design Process

We applied a co-design process. In particular, we developed a dance
performance in cooperation with a dance group. The choreographer
and the dance group worked in close relations with visualizationand sound designers, and researchers in wearable computing, neuroscience, and performing arts. Over the course of a year, we held
regular meetings, discussions, tryouts and a test performances,demonstrating each others skills and expertise in a democratic
process.
To test our concept, we conducted a proof-of-concept performance. The initial performances involved eyetracking, heart rate,
and an electrooculography (EOG) sensing system. We worked together with the performers to test visualizations and feedback mechanisms. From the initial tests (seen in Figure 2) we acquired someinsights on how to improve our design:

TEI ’21, February 14–17, 2021, Salzburg, Austria

Figure 2: Pictures from tryouts and an initial trial screening mostly with people working in the performing arts sector andtheater aficionados.
(1) Involve more audience members. Audience members who did
not get to use any sensing devices reported feeling left out andwanting to be part of the physiological recording. (2) Provide a
multi-modal experience. Dancing and all the staging elements need
to be designed together. Visuals that were disjointed from the performance were perceived as distracting. (3) Provide heart related
and electrodermal sensing. EEG and EOG were very noisy in thelive performance. Eyetracking and pupil dilation measurements
seemed interesting, yet took relatively long to setup and calibrate
to be used reliably. Among the test recordings with dancers andaudiences, heart rate and electrodermal activity were the easiest tointerpret as manifestations of the emotional response.
We also conducted several test screenings to evaluate the sensor
data. It was important that the performance could go on even withsensor failures and errors, so we prepared a backup plan using synthetic data which each designed element (lighting, sound, visuals)
would use in case the sensor streams were failing.
These three core insights lead to the performance Boiling Mind
that we describe in next Section. The resulting piece deals withsearching and finding your place in society. The choreography aims
to excite and stress the audience in the beginning half. The second
half is more about reflection and relaxation. Each scene is describedin detail later. The goal of the design is to transform the audience’s
unconscious reactions onto the performance space. The dancers
perceive these changes, especially with light and sound, and canadjust their body movements accordingly. The audience can see thedancers’ response to changes in the staging elements representing
their own unconscious reactions. For example, when the audience’s
physiological data suggests an increasing level of arousal, dancers
can see the reflection of it in the staging elements. In order to takethe excitement even further, the dancers will perform even more
intensely. This leads the audience to believe that the dancers arereacting to the audience’s excitement. Calming and relaxing parts ofthe performance can be performed in a similar fashion. We decided
that the sound and visualization would be the main elements forinteracting with the dancers. In case the live streaming failed, we
had synthetic streams prepared reflecting the expected reactions
from the audience (pre-recorded in test performances).

4

IMPLEMENTATIONIn the following we describe the stage design and implementation.

Figure 3: Data Flow.

Figure 4: Wristband with electrodermal, heart activity, acceleration and gyro sensors.

4.1

Setup and Staging

Structure of the setup and the data flow is shown in the Fig.3. The
collected physiological data of the audience is transmitted to theserver over WiFi. The server re-streams the processed data to othercomputers controlling each staging element.
4.1.1 Sensing Electrodermal and Heart Activity of the Audience. For this project we built wrist-worn devices measuring EDA
from 2 electrodes on the fingers, and the heart rate using an optical
blood volume pulse sensor (See Fig.4). EDA signal was sampled at
4.545 Hz and the heart rate sensor at 50Hz. The server software

Boiling Mind: Amplifying the Audience-Performer Connection through Sonification and Visualization of Heart and ElectrodermalTEI
Activities’21, February 14–17, 2021, Salzburg, Austria

the faster the EDA changed, the more intense the movement of thevisualizations became. As a result, the audience and the dancers may
understand the extent of the audience reaction from the movementof the visualization.
Figure 5: Mapping LF/HF Parameters to Colors.

Figure 6: Grouping behavior of the lights, left to right showsgroup growing and right to left movements
was recording and re-streaming the data to other machines used
for staging controls (See Fig.3). Stress tests showed that our system
could handle up to 60 devices using a Netgear Nighthawk series
WiFi router. Calculation of the HRV parameters such as LF/HF ratio
was done on the server side. The aggregated and processed data
was sent out to other machines in OSC format over UDP protocol
via an Ethernet link.
4.1.2 Visualizations. We projected the visualizations renderedbased on the audience physiological responses. Since the projection
was covering the whole stage, it played a significant role in setting
the mood and atmosphere. In the design process of the visualisations
we had the following considerations and requirements:(1) Visualisations should reflect audiences’ physiological reactions in real-time.
(2) Visualisations have to be easy to understand by both audience and dancers at a glance.
(3) Visualisations have to reflect the atmosphere, tone and storyof each scene.
Visualisations were rendered and projected onto the stage using
TouchDesigner1 software. We used 3 projectors so that the visualizations could cover the whole space. We based the visual design
around an intuitively comprehensible color scheme. A number ofstudies indicate that individuals tend to associate red colors witharousal and excitation whereas blue colors are associated with quietness and relaxation [21, 23, 29, 35, 49]. Therefore, in our design weused warm colors such as red to represent arousal and cold colors
such as blue to represent relaxation. The LF/HF ratio was mapped
to the blue and red extremes of a gradient as shown in Figure 5.
The predominant color of the visualisations changed according toeither the real-time average LF/HF ratio of the whole audience or
one particular spectator’s data.
The EDA signal was mapped to the visuals as well. EDA signal
interpretation concentrates on the rate of change rather than theabsolute values. For each visualization, the rate of change of the
EDA signal affected the visualizations’ movement speed. That is,
1 https://derivative.ca

4.1.3 Audio. The audio setup for this work consisted of 11 speakers: 9 small mono speakers placed on the ceiling, and two large
speakers on the floor at the back of the stage. The small speakers
on the ceiling were used primarily for supplemental sound design.
These sounds were faded in along with excitement levels of theaudience. We decided upon this approach as a way to not only allow
the audience data to guide another aspect of the performance, butalso to give the dancers a sonic cue of how engaged the audience
was during these moments. The upper speakers were also responsible for projecting additional timbral layers in several scenes. Overall,
the disbursement of the cumulative soundscapes was divided up
with roughly three-quarters of the audio layers assigned to the main
speakers and the remaining quarter of the audio layers assigned
to the ceiling speakers. This was decided after conducting several
trials, in which this balance between consistent audio and supplemental/reactive audio was found to be strongest for the production
team and the performers.
4.1.4 Lighting. The interactive lighting system consisted of 6
moving head lights on two axes placed around the stage. There
were 2 light modes: mode 1 was used in several scenes, while mode
2 was used during one of the performance scenes. In the first mode,
6 members from the audience were selected with their HR data
mapped to the intensity of the corresponding light. When morethan one person had similar beats per minute (BPM) rate, theywould be considered as a group.
The lights corresponding to this group would move to pointat each other. When someone’s BPM changed to a different value
from the rest of the group, the corresponding light would moveback to its initial position. The light would then be re-assigned toa different member. All the moving head lights of a given groupwould keep forwarding to each other (See Figure 6 ). On the second
mode, all the lights were linked to one audience member.
Once a mode was switched on, the lighting can function automatically until switched to a different mode. Thus, the lighting
design complements the visualization of the audience as a group oras an individual. In addition, it also affected the dancers by highlighting the inner states of the audience through brightness levels
and lighting movements.

4.2

Performance

In March 2020 we conducted three performances. The performanceswere advertised mentioning that we will be using the physiologicalsignals and a brief introduction of what would be expected to happen during the performance. All the audience members were asked
to read through the consent form on their seats (See Fig.4 Left). By
signing the form, they agreed to participate in this work and tooffer their physiological data. The performance, including the data
streaming and recording, was conducted according to the ethics
rules and regulations of the ethics board of university blinded
for review.

TEI ’21, February 14–17, 2021, Salzburg, Austria

Figure 7: Performance scenes. (a) Introduction. (b) Business
Card scene. (c) Bolero scene. (d) Ghostly Amenities scene. (e)Romeo scene. (f) Chaos scene. (g) Loving Animal Companion scene. (h) Ending
Each live performance lasted for about 1 hour and involved 7
dancers enacting the scenes.
a- Introduction (5 minutes): Before the performance started,
we introduced the design of the stage elements and physiologicalsensing to the audience to explain how the stage elements wouldchange depending on their physiological responses (See in Figure7(a)). We explained how sensors impact the discrete parts of theperformances with raw data and stage element examples. We alsoemphasized the informed consent regarding the sensor data, explaining the kind of data and potential risks with EDA/heartraterecordings. Some audience members could not wear the sensors or
decide not to be recorded. As the audience members were informed
over flyers and brochures beforehand about the recording, only few
opted out completely. The audience members could see raw dataand several visualizations based on an individual volunteer andaggregates.
b- Business Card Scene (3 minutes): The performance starts
from Ravel’s classical piece Boléro. Shortly after the beginning, the7 dancers blend into the audience and talk to audience members
while handing them their own business cards. The physiological
data from the audience members of this scene was plotted individually in the form of a real-time graph, with all the audience’s
graphs displayed side-by-side as shown in Figure 7(b). Each graphcontained the heart rate waveform, BPM, LF/HF ratio value, EDA
value, and ID of the spectator. The audience was able to see visualizations of their own data reflecting on the stage.As a result of
initial tests, many audience members expressed a desire to see how
their biometric information changed while watching the dance performance. The act of watching a dance is a very personal experience
for the audience, and they are desperate to find common groundbetween their own experience and what happens on stage. To assist
the audience in this quest, we designed the piece to visualize eachaudience member’s biometric information as the key to connecting
the audience and the stage. As the emotional response of those who
received business cards from dancers spiked drastically, it was also
clearly visible in the graphs.
c- Boléro (15 minutes): As Boléro continued to build up, thedancers started moving slowly and quietly, wearing suits and highheels. They gradually took them off and exploded in sync withthe rhythm and crescendo of Boléro. As the rhythm of Boléro andthe rhythm of the high heels hitting the floor synchronized, thevisualization changed to a more abstract representation as shownin Figure 7(c). In this scene, each spectator’s data was transformed

into an orb that changed color depending on the respective LF/HF
value. The shape of this orb was indeterminate, and the speed ofits movement depended on the EDA signal changes. Also, Mode1
was used for lighting.
d- Ghostly Amenities (10 minutes): Each performer dancedwith a puppet that represented their alter ego. In the end, thedancers handed the puppet to an audience member. The choreography and music set a mysterious tone in this scene. The visuals
were designed to amplify this tone. We used a fluid simulationand created a smoke-filled space. The amount and frequency of
appearance of the smoke clouds was varied by means of the LF/HF
value. The mean of the EDA values also affected the parametersof the fluid simulation of the smoke. When the EDA data from theaudience suggested sedation, there was less smoke, and vice versa(See Figure 7(d)).
The musical score consisted of four movements which were separated and played back depending on the pacing of the choreography.
The physiological data from the audience was used as a graphic
score guiding the implementation of reverb and a low pass filter(LPF). As the excitement levels increased, the LPF allowed more
sub-frequencies into the soundscape. This design was intended tofurther increase the excitement levels due to the physical nature of
the low frequencies, while the reverb was used as an auditory cue
for the changes occurring.
e- Romeo (15 minutes): One dancer spoke the line, "If I see
your mind, I will dance with it". The phrase and idea came fromthe lighting technician during discussions leading to the artistic
decision to include a more directly linked performance between
dancer and audience member. Next, a different dancer picked an
audience member and invited him to the stage to play the role ofher Romeo. The dancer chose a male spectator from the first row attheir own discretion. The spectator needed to have an active sensor
and consented to the public data use.
In this scene, the visualization showed the LF/HF and EDA plot
similar to Figure 7(a), except the data used was picked up from thechosen Romeo as shown in Figure 7(e). For all three performances,the LF/HF ratio of the Romeo indicated a high level of excitement.
For this scene, we linked all the lights to the Romeo audience
member using Mode2 lighting.
The dancers could perceive the emotional state of the chosen
Romeo based on the color of the visual projection. When it turned
red, the dancers would start to improvise and reach out to Romeo,
as if saying they were happy to see him here. If he smiled, thedancers would dance with him in a playful way.
f- Chaos (5 minutes): This scene described the conflicting andcomplex feelings of instability, confusion, and joy that we all experience as we grow from childhood to adulthood. The visualization
moved in the form of a wave, with the average LF/HF value changing the color of the wave, and the average EDA value changing theheight of the wave as shown as Figure 7(f). Also, Mode1 was used
for lighting.
The score for this movement began with a looping drum beat which
was then manipulated based on the physiological data. The EDA
levels would trigger a stuttering effect which was played backthrough the ceiling speakers. The LF/HF values dictated the pitch
variance in the drum sounds. Lastly, the movements of the solo
performer dictated the duration of the drum solo and the entrance

Boiling Mind: Amplifying the Audience-Performer Connection through Sonification and Visualization of Heart and ElectrodermalTEI
Activities’21, February 14–17, 2021, Salzburg, Austriaof the subsequent synthesizer passages. Solo dancers improvised
to the rhythm of the audience’s interlocking LF/HF ratio. As theaudience’s excitement level increased, the music became disjointed,
forcing the dancers to move with unnatural pauses and a complex
juxtaposition of tension and relaxation.
g- Loving Animal Companion (15 minutes): One dancer followed another dancer like a playful animal companion (e.g. dog)
willingly walking after its owner, clinging to her legs. In this scene,
we used particle colors linked to the LF/HF values and particle density linked to EDA as shown in Figure 7(g). The particle effect was
rendered to illustrate the change from red to blue as the audience
was more sedated, mirroring the slow choreography and aesthetic
tone.
The music for this scene was also quite sedated and relaxed. The
musical director used the incoming data to guide when various
musical layers would enter and exit. In addition to these musicallayers, heavily manipulated sounds of high heels were played backthrough the ceiling speakers as a reference to the earlier parts ofthe performance. These heel sounds were faded in and out basedon the LF/HF and EDA levels.
h- Ending (10 minutes): The music and choreography of the final scene was a reference to the first Boléro scene. The visual design
was also the same as in the business card and Boléro scenes. The
similarity and references between the ending scene and the beginning scene are purposely delivered to the audience so that they cancompare the different states they perceived with the visualization
before and after experiencing the entire performance.
To sum up, Boiling Mind was a one-hour contemporary dance
performance based on "Boléro" composed by Maurice Ravel. The
choreography in Boiling Mind incorporated the mechanism of interpersonal synchrony [15] to create a feedback loop between the audience and the dancers. This was accomplished by sharing rhythms
between interpersonal actors in a biosensor-based reactive performance [48]. 57 people attended the first performance, 37 the second,and 45 the third performance. For the 3 performances, we had physiological data related to the emotional responses of 40, 35 and 40
members of the audience used in each respective performance.

5

FEEDBACK AND DISCUSSION

After each performance, each spectator received an online questionnaire link. We received 35 audience (Female:18, Male: 16, Unknown:1) responses in total. Among 35 audience members, 3 ofthem attend dance performances every week, 8 of them attend
dance performances several times a month, 19 of them attend dance
performances several times a year, and 5 of them mentioned thiswas the first time they attended a dance performance. We also collected informal interviews and feedback from the choreographer,dancers, theater staff and audience members. Additionally, we conducted an online semi-structured interview with five of the seven
dancers. Overall, the feedback was very positive.

5.1

Feedback

The online questionnaire covered the following areas: demographic
and background questions (e.g. how often do you visit theater/
dance performances) and their experience of "Boiling Mind." For
the experience enjoyment levels, we used the 1-9 Likert Scale (1 not

Figure 8: EDA data of one of the participants (performance3, subject 23). EDA is measured by the change of Skin Conductivity (CS). Light blue - raw EDA data from 12-bit ADC(0..4095 range); Blue - low-pass filtered data; Orange markers - recognized peaks in skin conductivity (local maxima);
Purple markers - recognized valleys (local minima). The distribution of phasic peaks suggest that there is a relation between the excitement peaks and the performance scenes.

Figure 9: LF/HF ratio data of one of the participants (performance 1, subject 21). LF/HF ratio is a measurement of heart
rate variability which represents the ratio between SNS andPSNS activity. Blue - LF/HF ratio data; Red - Average LF/HF
ratio of this subject during the whole performance.A high
LF/HF ratio indicates sympathetic dominance.The trend suggests the emotional arousal during the unfolding of the performance.

at all, 9 very much) as well as several open-ended questions about
what they liked best/worst compared to other dance performancesand general feedback. Almost all the audience members enjoyed
the performance and liked the concept of "inviting the audience onstage".
From the questionnaires, 95.6% of the audience members reported they enjoyed the performance and 81.1% reported they felt
this performance was more engaging than other performances theyattended. We found most attendees considered "Boiling Mind" asan engaging and exciting performance because of its interaction
and novelty. 86.5% of them perceived the dance itself as the mostimpacting factor related to their feelings and emotions. In termsof positive reactions to the staging elements, 56.8% of them noted
the audio, 37.8% noted the the visuals, while 21.6% noted the lighting. Overall, most of them (95.6%) specified that they want to seeanother piece based on the same design concept.
From the free form discussions, we got more detailed responses.
An audience member commented that "thinking of combining myheart rate with the stage design made me already excited." 5 audience members mentioned that they enjoyed the Business CardTEI ’21, February 14–17, 2021, Salzburg, Austria

scene, as they saw the changes in posture of some audience members as well as spikes in heart rate and EDA on some graphs. Thisconnection was something they could relate to and identify. However, we also found some of the audience liked the concept, yetoften failed to see a direct effect of their reactions reflected on thestage. 2 members described that they could not really recognize
how their heart rate or EDA influenced the dancers and that a direct
feedback loop was missing.
The interview with the dancers centered around their overall
perceptions of the performances and the audience engagement.
We presented them with the graph regarding audience physiological data when dancers and the choreographer reflected on theperformances during the interview.
The dancers expressed their enthusiasm for the performancepiece and their willingness to create similar performances in the
future. They could recognize the staging elements’ differences foreach performance, especially the audio and color changes in projections and lighting effects. However, they wanted more clarity
on what the changes in staging elements represent and how theyshould react to them.
Moreover, they would like to deliver more physical interactions
to respond to the audience’s physiological data. For example, one
dancer said when she saw several projections of audience states
turning blue, she tried to increase her movement speed and intensity
to engage the audience more. However, it may not be easy fordancers to engage with the staging elements all the time as forsome part they are preoccupied with the story line. Regarding
that, some dancers suggested adding more improvisational scenes
allowing them to perform accordingly to the change of staging
elements to strengthen the communicative flow between dancers
and the audience.

5.2

Lessons Learned"A more direct feel on how I influence the stage" A criticism
from several members of the audience (4) was that they cannot recognize how their own data changes the stage elements. For someit’s quite disjunctive. 2 of them also remarked that they didn’t recognize any difference compared to a standard dance performance.
One remedy could be to play with the sense of agency. Since direct control for large audiences is very difficult, one could devise
interactions that give the audience a feeling of control.
"Don’t dismiss Raw Data as a visual" Related to the first insight, several participants (5) mentioned that they enjoyed theBusiness Card scene, as they saw the posture reactions of someaudience members as well as spikes in heart rate and skin responseon some graphs. They could relate to them. We thought that raw
time series data might not be optimal, yet "Keeping it close to theraw signals" for visualization seems to be one take away. In our scenario with 40 simultaneous streams it is definitely feasible, thoughbeyond that there will be scalability issues.
"Best Practices on How to Dance to Biometric Visualization" Four of the dancers still mentioned that they didn’t know
how to respond to the biometric data from the audience directly.
They recognized changes in their behavior (e.g. dancing more erraticly when the visualization showed that the audience was notengaged in a part of the performance that was supposed to be stressful); yet they mentioned that they need more practice to acquirephysical techniques to respond to the audience’s biometric data. We
need to create better practices and training routines surrounding
this concept.
Post-Performance Insights When the dancers and choreographer reflected on the performances, we also presented them withthe graph regarding to audience bio data. Dancers discussed reasons
why some scenes had the audience more engaged than others suchas Romeo while referring to the summarized audience bio feedback.
They especially liked comparing different performances and could
identify individual dance variations that seemed to have an impact
on the audience.
"Even tighter Co-Design between Artists and TechnicalStaff" Even though the dance, visualizations, sonification, andlighting were co-designed together with dancers, choreographer,
designers and researchers, we need an even tighter loop including
the physiological sensing more in the process. Given the limitations,
we had just a couple of training sessions with 2 or 3 test subjects
posing as audience. For dancers to have a better grasp of what thevisualization changes mean, new forms of co-design (e.g. training
with a small audience) have to be created.

6

CONCLUSIONS AND FUTURE WORKThis paper introduces Boiling Mind, a novel interactive performance
approach that breaks the fourth wall between the audience andperformers by using sonification and visualization of heart beats
and EDA data. We have developed a system to reflect physiological
data collected from a large number of sensors to stage elements
in real time, and have investigated a method to enhance audienceengagement and participation in the performance.
From an initial data inspection, the data seems sound and captured the intended effects from the audience (stress versus relaxation). Yet, this paper focused on the design process and providing
the dataset to the community in general (with video recordings ofthe performance). The analysis of the data is future work and will
give more quantitative insights into how to design these experiences.
Although the reception of the three performances was very positive, the intention of creating a seamless feedback loop between audience and performers is still not fully achieved. For future work, wewill add the dancers’ real-time recording for the whole performance.
This will allow us to audio-visualize how Audience-Performer physiological data interacts with and influences each other. We will also
focus on improving the connection between the data and stage
design elements to let the audience feel more influence over thestage. For example, the audience can play with the stage elements
to increase their sense of agency and control over the staging elements before the performance starts. Since implementing a systemthat gives a sense of control to a large audience is very difficult,one could guide the audience to perform some synchronous actions
that would give the audience a feeling of control over the staging
elements. In addition, we believe this approach can be applied in thefield of remote performances in which the audience is physicallyseparated from the performance space.

Boiling Mind: Amplifying the Audience-Performer Connection through Sonification and Visualization of Heart and ElectrodermalTEI
Activities’21, February 14–17, 2021, Salzburg, Austria

In order to create stronger "continuous feedback loops" between
audience and performers, stage elements and maybe dancers themselves could be physically affected by the audience biometeric data.
For example, as the audience’s heartbeat increases, the floor on thestage starts pounding, or artificial muscles will provide the rhythm
directly to the dancers body.
We hope this research can inspire/help performers and designers
to explore novel methods by developing more interactive works.

ACKNOWLEDGMENTS
We would like to thank the Session House Theater for performance
space. Especially choreographer Naoko Ito, Mademoiselle Cinema
dancers as well as Stage staff for providing us with the necessary
support for this project. We also want to thank Dr. Jamie Ward,
Goldsmiths, London, for his continuous support. This work is also
partly supported by JST CREST Experiential Supplements Grant
No. JPMJCR16E1.

REFERENCES[1] John L Andreassi. 2010. Psychophysiology: Human behavior and physiologicalresponse. Psychology Press.
[2] John L Andreassi. 2013. Psychophysiology: Human behavior & physiologicalresponse. Psychology Press.
[3] Philip Auslander. 2008. Liveness: Performance in a mediatized culture. Routledge.
[4] Steve Benford, Chris Greenhalgh, Andy Crabtree, Martin Flintham, Brendan
Walker, Joe Marshall, Boriana Koleva, Stefan Rennick Egglestone, Gabriella Giannachi, Matt Adams, Nick Tandavanitj, and Ju Row Farr. 2013. Performance-led
research in the wild. ACM Transactions on Computer-Human Interaction (TOCHI)
20, 3 (2013), 1–22. https://doi.org/10.1145/2491500.2491502[5] Miranda Boorsma. 2006. A strategic logic for arts marketing: Integrating customer
value and artistic objectives. International journal of cultural policy 12, 1 (2006),
73–92.
[6] Wolfram Boucsein. 2012. Electrodermal activity: Springer Science & BusinessMedia. Broek, EL vd, Schut, MH, Westerink, JHDM, Herk, J. v., & Tuinenbreijer, K(2012).
[7] Astrid Breel. 2015. Audience agency in participatory performance: a methodology
for examining aesthetic experience. Participations: Journal of Audience & Reception
Studies 12, 1 (2015), 368–387.
[8] Susan Broadhurst. 2002. Blue Bloodshot Flowers: interaction, reaction and performance. Digital Creativity 13, 3 (2002), 157–163. https://doi.org/10.1076/digc.
13.3.157.7340[9] Jos F Brosschot, Eduard Van Dijk, and Julian F Thayer. 2007. Daily worry is
related to low heart rate variability during waking and the subsequent nocturnal
sleep period. International journal of psychophysiology 63, 1 (2007), 39–47.
[10] Courtney Brown. 2019. Machine Tango: An Interactive Tango Dance Performance.
565–569. https://doi.org/10.1145/3294109.3301263[11] Teresa Cerratto-Pargman, Chiara Rossitto, and Louise Barkhuus. 2014. Understanding Audience Participation in an Interactive Theater Performance. In
Proceedings of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast,
Foundational (Helsinki, Finland) (NordiCHI ’14). Association for Computing Machinery, New York, NY, USA, 608–617. https://doi.org/10.1145/2639189.2641213[12] Kwang-Ho Choi, Junbeom Kim, O Sang Kwon, Min Ji Kim, Yeon Hee Ryu, andJi-Eun Park. 2017. Is heart rate variability (HRV) an adequate tool for evaluating
human emotions?–A focus on the use of the International Affective Picture
System (IAPS). Psychiatry Research 251 (2017), 192–196.
[13] Greg Corness, Kristin Carlson, and Thecla Schiphorst. 2011. Audience empathy:
a phenomenological method for mediated performance. In Proceedings of the 8th
ACM conference on Creativity and cognition. 127–136. https://doi.org/10.1145/
2069618.2069641[14] Stephen Davies. 1997. John Cage’s 4’ 33 ": Is it music ? Australasian Journal of
Philosophy 75, 4 (1997), 448–462.
[15] E. Delaherche, M. Chetouani, A. Mahdhaoui, C. Saint-Georges, S. Viaux, and D.
Cohen. 2012. Interpersonal Synchrony: A Survey of Evaluation Methods acrossDisciplines. IEEE Transactions on Affective Computing 3, 3 (2012), 349–365.
[16] Steve Dixon and Barry Smith. 2007. Digital performance: a history of new media
in theater, dance, performance art, and installation. MIT press.
[17] Barbara Ehrenreich. 2007. Dancing in the streets: A history of collective joy.
Macmillan.
[18] Byeongwon Ha. 2015. A Pioneer of Interactive Art: Nam June Paik as MusiqueConcrète Composing Researcher. In ISEA (Vancouver, International Symposium

on Electronic Art, 2015).
[19] Rob Haskins. 2013. John Cage. Reaktion Books.
[20] Mariam Hassib, Stefan Schneegass, Philipp Eiglsperger, Niels Henze, Albrecht
Schmidt, and Florian Alt. 2017. EngageMeter: A system for implicit audienceengagement sensing using electroencephalography. In Proceedings of the 2017
Chi conference on human factors in computing systems. 5114–5119. https://doi.
org/10.1145/3025453.3025669[21] Michael Hemphill. 1996. A note on adults’ color–emotion associations. The
Journal of genetic psychology 157, 3 (1996), 275–280.
[22] Kristina Höök. 2009. Affective loop experiences: designing for interactional
embodiment. Royal Society 364, 1535, Article B (12 December 2009). https://doi.org/10.1098/rstb.2009.0202[23] Keith W Jacobs and Frank E Hustmyer Jr. 1974. Effects of four psychologicalprimary colors on GSR, heart rate and respiration rate. Perceptual and motor
skills 38, 3 (1974), 763–766.
[24] Corinne Jola. 2016. The magic connection: Dancer-audience interaction. Zwischenleiblichkeit und Bewegtes Verstehen–Intercorporeity, movement and tacit knowledge (2016), 269–287.
[25] Peter Jönsson. 2007. Respiratory sinus arrhythmia as a function of state anxiety
in healthy individuals. International journal of psychophysiology 63, 1 (2007),
48–54.
[26] Carl Gustav Jung. 1918. Studies in word-association. Heinemann.
[27] Keina Konno, Richi Owaki, Yoshito Onishi, Akiko Takeshita, Ryo Kanda, Naoko
Shiomi, Kyle McDonald, Satoru Higa, Motoi Shimizu, Yosuke Sakai, Yasuaki
Kakehi, Kazuhiro Jo, Yoko Ando, Kazunao Abe, and Takayuki Ito. 2016. DividualPlays Experimental Lab-An installation derived from Dividual Plays. TEI ’16,
February 14 - 17, 2016, Eindhoven, Netherlands (2016). https://doi.org/10.1145/
2839462.2856346[28] Richard Kostelanetz. 2003. Conversing with cage. Routledge.
[29] Rikard Küller, Byron Mikellides, and Jan Janssens. 2009. Color, arousal, andperformance—A comparison of three experiments. Color Research and Application34, 2 (2009), 141–152. https://doi.org/10.1002/col.20476[30] Peter J Lang. 1995. The emotion probe: studies of motivation and attention.
American psychologist 50, 5 (1995), 372.
[31] Peter J Lang, Mark K Greenwald, Margaret M Bradley, and Alfons O Hamm.
1993. Looking at pictures: Affective, facial, visceral, and behavioral reactions.
Psychophysiology 30, 3 (1993), 261–273.
[32] Celine Latulipe, Erin A. Carroll, and Danielle Lottridge. 2011. Love, Hate, Arousal
and Engagement: Exploring Audience Responses to Performing Arts. Association
for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/1978942.
1979210[33] Jane Lessiter, Jonathan Freeman, Edmund Keogh, and Jules Davidoff. 2001. A
cross-media presence questionnaire: The ITC-Sense of Presence Inventory. Presence: Teleoperators & Virtual Environments 10, 3 (2001), 282–297.
[34] Christopher Lindinger, Martina Mara, Klaus Obermaier, Roland Aigner, Roland
Haring, and Veronika Pauser. 2013. The (St)Age of Participation: audience involvement in interactive performances. Digital Creativity 24, 2 (2013), 119–129.
https://doi.org/10.1080/14626268.2013.808966[35] Banu Manav. 2007. Color-emotion associations and color preferences: A casestudy for residences. Color Research and Application 32, 2 (2007), 144–150. https://doi.org/10.1002/col.20294[36] Gewnhi Park and Julian F Thayer. 2014. From the heart to the mind: cardiac
vagal tone modulates top-down and bottom-up visual perception and attention
to emotional stimuli. Frontiers in psychology 5 (2014), 278.
[37] TW Picton, I Martin, and PH Venables. 1980. Techniques in psychophysiology.
(1980).
[38] Jennifer Radbourne, Katya Johanson, Hilary Glow, and Tabitha White. 2009. The
Audience Experience: Measuring Quality in the Performing Arts. International
Journal of Arts Management 11 (01 2009).
[39] Richard Ramchurn, Sarah Martindale, Max L Wilson, and Steve Benford. 2019.
From Director’s Cut to User’s Cut: to Watch a Brain-Controlled Film is to Edit it.
In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems.
1–14. https://doi.org/10.1145/3290605.3300378[40] Danilo Rodrigues, Emily Grenader, Fernando Nos, Marcel Dall’Agnol, Troels
Hansen, and Nadir Weibel. 2013. MotionDraw: a tool for enhancing art andperformance using kinect. In CHI ’13. https://doi.org/10.1145/2468356.2468570[41] Asreen Rostami, Donald McMillan, Elena Márquez Segura, Chiara Rossito, andLouise Barkhuus. 2017. Bio-sensed and embodied participation in interactive
performance. In Proceedings of the Eleventh International Conference on Tangible,
Embedded, and Embodied Interaction. 197–208.
[42] Fred Shaffer and JP Ginsberg. 2017. An overview of heart rate variability metrics
and norms. Frontiers in public health 5 (2017), 258.
[43] Catherine Stevens, Renee Glass, Emery Schubert, Johnson Chen, and Heather
Winskel. 2007. Methods for measuring audience reactions. In Proceedings of theinaugeral International Conference on Music Communication Science. Citeseer,155.
[44] Julian F Thayer and Esther Sternberg. 2006. Beyond heart rate variability: vagal
regulation of allostatic systems. Annals of the New York Academy of Sciences 1088,

TEI ’21, February 14–17, 2021, Salzburg, Austria

1 (2006), 361–372.
[45] Christian Tronstad, Gaute E Gjein, Sverre Grimnes, Ørjan G Martinsen, AnneLene Krogstad, and Erik Fosse. 2008. Electrical measurement of sweat activity.
Physiological measurement 29, 6 (2008), S407.
[46] Marieke van Dooren, Joris H Janssen, et al. 2012. Emotional sweating acrossthe body: Comparing 16 different skin conductance measurement locations.
Physiology & behavior 106, 2 (2012), 298–304.
[47] Francisco J Varela, Evan Thompson, and Eleanor Rosch. 2016. The embodied
mind: Cognitive science and human experience. MIT press.

[48] Jamie A Ward, Daniel Richardson, Guido Orgs, Kelly Hunter, and Antonia
Hamilton. 2018. Sensing Interpersonal Synchrony between Actors and Autistic Children in Theatre Using Wrist-Worn Accelerometers. Proceedings of the2018 ACM International Symposium on Wearable Computers, 148–155. https://doi.org/10.1145/3267242.3267263[49] Lois B Wexner. 1954. The degree to which colors (hues) are associated withmood-tones. Journal of applied psychology 38, 6 (1954), 432.