Boiling Mind - A Dataset of Physiological Signals during an
Exploratory Dance Performance
Zhuoqi Fu
Jiawen Han
George Chernyshov

fzq1996,hanjiawen,chernyshov@kmd.keio.ac.jp
Keio UniversityJapan

Karola Marky

marky@tk.tu-darmstadt.deTechnische Universität Darmstadt
Germany

Moe Sugawa
Dingding Zheng
Danny Hynds

moe.sen.se321@gmail.com
zheng208,dannyhynds@kmd.keio.ac.jp
Keio UniversityJapan

Jamie A Wardjamie@jamieward.net
Goldsmiths, University of London
United Kingdom

Taichi Furukawa
Marcelo Padovani
Kouta Minamizawa
taichi,mapama,kouta@kmd.keio.ac.jp
Keio UniversityJapan

Kai Kunze

kai@kmd.keio.ac.jp
Keio UniversityJapanABSTRACT
The relationship between audience and performers is crucial towhat makes live events so special. The aim of this work is to develop a new approach amplifying the link between audiences andperformers. Specifically, we explore the use of wearable sensors ingathering real-time audience data to augment the visuals of a live
dance performance. We used the J!NS MEME, smart glasses withintegrated electrodes enabling eye movement analysis (e.g. blinkdetection) and inertial motion sensing of the head (e.g. nodding
recognition). This data is streamed from the audience and visualised live on stage during a performance, alongside we also collected heart rate and eye gaze of selected audience. In this paper we
present the recorded dataset, including accelerometer, electrooculography(EOG), and gyroscope data from 23 audience members.

CCS CONCEPTS• Human-centered computing → Interaction design; Interaction design process and methods.

KEYWORDS
Physiological Signals, Visualization, Audience Engagement
ACM Reference Format:
Zhuoqi Fu, Jiawen Han, George Chernyshov, Moe Sugawa, Dingding Zheng,Danny Hynds, Taichi Furukawa, Marcelo Padovani, Kouta Minamizawa,Karola Marky, Jamie A Ward, and Kai Kunze. 2021. Boiling Mind - A Datasetof Physiological Signals during an Exploratory Dance Performance. In
Augmented Humans International Conference 2021 (AHs ’21), February 22–24, 2021, Rovaniemi, Finland. ACM, New York, NY, USA, 3 pages. https://doi.org/10.1145/3458709.3459006
Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
AHs ’21, February 22–24, 2021, Rovaniemi, Finland© 2021 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-8428-5/21/02.
https://doi.org/10.1145/3458709.3459006

301

Figure 1: Visualizations while performers were dancing

1

INTRODUCTION

Each live performance is unique, even when following the samechoreography, or script. Reactions between audiences and performers can have noticeable effects on the overall experience. This work
attempts to understand and amplify these effects by integrating
real-time audience physical and physiological data directly into theperformance itself.
We present an exploratory performance to integrate various
physical and physiological audience signals, obtained using wearable sensors, into the visual effects of a live dance performance.
Specifically, the work shows: (1) the technical feasibility of streaming multiple sensors using electrooculography (EOG) and head
movement with smart eyewear, Blood Volume Pulse (BVP), and agaze tracker (2) an integrated wearable system that consists of asmart eyewear, back-end processing and output visualization (3) amulti-person, multi-modal audience dataset (available here).
Connecting performers and audience via sensing technology is
not new. Previously, researchers have attempted to develop an interactive system where sound and visual changes were manipulatedby dancers’ movement [10, 12]. However, even though tracking
performers movement can enrich the stage design, that work does
not involve the audience directly. Some works explore audience
participation by using linguistic feedback [3, 6], collective heart
rate [15], electroencephalography (EEG) [8], and galvanic skin response (GSR) [17, 20]. Here we build on these ideas by integrating
EOG and head movement signals obtained using commercial eyewear with heart rate and gaze.

AHs ’21, February 22–24, 2021, Rovaniemi, FinlandFu, et al.

Eyewear-recorded EOG has been used to recognize activity, concentration, and potential social interaction [4, 7, 9, 13, 19]. Heartrate (HR) and heart rate variability (HRV) have also been used to
track affect and mental state [5, 14]. By using multi-modal sensors like these on audience, we hope to pave the way for systems
that can characterise audience mental states like engagement andenjoyment both cheaply and in real-time.

2(sampled with 50Hz). We also include audience HR data from BVP
sensor (20Hz) and eye movement data from the Pupil Core (120 Hz
sampling binocular) as well as dancers’ motion data in our dataset
for more comprehensive information.

PERFORMANCE AND HARDWARE SETUP

Figure 2: Audience Wearing Blood Volume Pulse Sensor (a),Pupil Core (e) and Jins Meme (g), EOG-enabled Smart Eyewear. (c), (f), and (h) Visualization of the physiological datacollected from the audience.
A 20 minute experimental public dance performance titled "HerChair", was performed on November 4th 2019, at the Session House
Theater (Tokyo, Japan), in collaboration with a contemporary dancegroup Mademoiselle Cinema.
The audience were aware that their signals would be visualised
and projected onto a screen as part of the performance, and wehave informed consent from everyone whose data is included in theanonymized dataset. The setup and performance were conductedwith the full approval of the ethics board of our university.
In order to measure the response of the audience to the dancers’
motion we use a combination of audience EOG, eye tracking andBVP. EOG was used to calculate audience blinks [9], and BVP wasused to calculate HR and HRV.
Audience data was visualized in real time using TouchDesigner
software and projected on the stage. When three audience members
blink simultaneously, the visuals were changed from blue to red
as shown in Fig.2 (h). To abstractly express the audience’s interest
to the stage, the eye movement streamed from the Pupil Core wasvisualized as shown in Fig.2 (f). HR data was presented as a planar
image flowing horizontally as shown in Fig.2 (c). The ratio of low
frequency band (LF) and high frequency (HF) band of HRV signal(LF/HF) was used to set the display coloring. As an example of
how this translated into the visuals,an LF/HF ratio of 0.0 to 2.0 wasassumed to be in a relaxed state and was visualised as blue. Conversely, a ratio of 2.0-5.0 and beyond was interpreted as audience
excitement, and was represented by red colors.

3DATASET DESCRIPTION

The dataset mainly consists of MEME recordings including 3-axis
accelerometer data, 2-side EOG data, and three gyroscopes data

302

Figure 3: Example x,y,z accelerometer (above) and EOG (below) data from three audience members during the performance
We present an initial exploration of the 3-axis accelerometer toshow the quality of the data recorded. Fig. 3 (above) depicts accelerometer data from three audience members’ glasses recordings.
The accelerometer data looks usable and some patterns between theaudience members can be seen, e.g. potential head nods and handclap timings. To further quantify the synchrony among physiological signals, we will apply various techniques such as windowed
cross-lagged correlation (WCLC) [2] and dynamic time warping(DTW) [1, 11].
We also investigate the EOG data recorded from MEME. Fig. 3(below) shows an example of EOG taken from three audience members. The data seems usable and not more or less noisy then similar
data we collected in the laboratory before. With the feasibility toreflect people’s cognitive state [4, 18], EOG data could also provide
us with implicit engagement information. From initial observations,
we found that audiences blinked less as the performance went on,
which potentially suggests growing visual attention [16].

4DISCUSSION AND FUTURE WORKS

We present an approach at collecting and streaming multi-person,multi-modal sensor data using commercial eyewear. We give a brief
analysis of of the recorded data and provide a full dataset for download. Future work includes 1) building the feedback loop from theaudience to the dancers, by visualizing physiological data to thestage and enable dancers to adjust their performance accordingBoiling Mind - A Dataset of Physiological Signals during an Exploratory Dance Performanceto the audience status, 2) applying more intuitive methods of human cognition analysis including combining blink frequency, blink
strength, and head gesture to make precise rating on engagement.

REFERENCES[1] Donald J. Berndt and James Clifford. 1994. Using Dynamic Time Warping toFind Patterns in Time Series. In Proceedings of the 3rd International Conference onKnowledge Discovery and Data Mining (Seattle, WA) (AAAIWS’94). AAAI Press,359–370.
[2] Steven Boker, Minquan Xu, Jennifer Rotondo, and Kadijah King. 2002. Windowed
cross-correlation and peak picking for the analysis of variability in the association
between behavioral time series. Psychological methods 7 (10 2002), 338–55. https://doi.org/10.1037/1082-989X.7.3.338[3] Teresa Cerratto-Pargman, Chiara Rossitto, and Louise Barkhuus. 2014. Understanding Audience Participation in an Interactive Theater Performance. In
Proceedings of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast,Foundational (Helsinki, Finland) (NordiCHI ’14). Association for Computing Machinery, New York, NY, USA, 608–617. https://doi.org/10.1145/2639189.2641213[4] George Chernyshov, Kirill Ragozin, Benjamin Tag, and Kai Kunze. 2019. EOG
Glasses: an Eyewear Platform for Cognitive and Social Interaction Assessments inthe Wild. In Proceedings of the 21st International Conference on Human-Computer
Interaction with Mobile Devices and Services. 1–5.
[5] Kwang-Ho Choi, Junbeom Kim, O Sang Kwon, Min Ji Kim, Yeon Hee Ryu, andJi-Eun Park. 2017. Is heart rate variability (HRV) an adequate tool for evaluating
human emotions?–A focus on the use of the International Affective Picture
System (IAPS). Psychiatry Research 251 (2017), 192–196.
[6] Sabrina A. L. Frohn, Jeevan S. Matharu, and Jamie A. Ward. 2020. Towards a Characterisation of Emotional Intent during Scripted Scenes Using In-Ear Movement
Sensors. In Proceedings of the 2020 International Symposium on Wearable Computers (Virtual Event, Mexico) (ISWC ’20). Association for Computing Machinery,
New York, NY, USA, 37–39. https://doi.org/10.1145/3410531.3414292[7] Aman Gupta, Finn L Strivens, Benjamin Tag, Kai Kunze, and Jamie A Ward. 2019.
Blink as You Sync: Uncovering Eye and Nod Synchrony in Conversation Using
Wearable Sensing. In Proceedings of the 23rd International Symposium on Wearable
Computers (London, United Kingdom) (ISWC ’19). Association for Computing
Machinery, New York, NY, USA, 66–71. https://doi.org/10.1145/3341163.3347736[8] Mariam Hassib, Stefan Schneegass, Philipp Eiglsperger, Niels Henze, Albrecht
Schmidt, and Florian Alt. 2017. EngageMeter: A System for Implicit Audience Engagement Sensing Using Electroencephalography. In Proceedings of the 2017 CHIConference on Human Factors in Computing Systems. Association for Computing
Machinery, New York, NY, USA, 5114–5119. https://doi.org/10.1145/3025453.
3025669[9] Shoya Ishimaru, Kai Kunze, Koichi Kise, Jens Weppner, Andreas Dengel, Paul
Lukowicz, and Andreas Bulling. 2014. In the Blink of an Eye: Combining HeadMotion and Eye Blink Frequency for Activity Recognition with Google Glass. In
Proceedings of the 5th Augmented Human International Conference (Kobe, Japan)(AH ’14). Association for Computing Machinery, New York, NY, USA, Article 15,4 pages. https://doi.org/10.1145/2582051.2582066[10] Jodi James, Todd Ingalls, Gang Qian, Loren Olsen, Daniel Whiteley, Siew Wong,and Thanassis Rikakis. 2006. Movement-Based Interactive Dance Performance.
In Proceedings of the 14th ACM International Conference on Multimedia (SantaBarbara, CA, USA) (MM ’06). Association for Computing Machinery, New York,NY, USA, 470–480. https://doi.org/10.1145/1180639.1180733[11] Eamonn Keogh. 2002. Exact Indexing of Dynamic Time Warping. In Proceedingsof the 28th International Conference on Very Large Data Bases (Hong Kong, China)(VLDB ’02). VLDB Endowment, 406–417.
[12] Celine Latulipe, Erin A. Carroll, and Danielle Lottridge. 2011. Evaluating Longitudinal Projects Combining Technology with Temporal Arts. In Proceedings ofthe SIGCHI Conference on Human Factors in Computing Systems (Vancouver, BC,Canada) (CHI ’11). Association for Computing Machinery, New York, NY, USA,1835–1844. https://doi.org/10.1145/1978942.1979209[13] Hang Li, Julien Epps, and Siyuan Chen. 2020. Think before you speak: An investigation of eye activity patterns during conversations using eyewear. International Journal of Human-Computer Studies 143 (2020), 102468. https://doi.org/10.1016/j.ijhcs.2020.102468[14] Gewnhi Park and Julian F Thayer. 2014. From the heart to the mind: cardiac
vagal tone modulates top-down and bottom-up visual perception and attention
to emotional stimuli. Frontiers in psychology 5 (2014), 278.
[15] Arttu Perttula, Pauliina Tuomi, Marko Suominen, Antti Koivisto, and Jari Multisilta. 2010. Users as sensors: creating shared experiences in co-creational spaces
by collective heart rate. In Proceedings of the 14th International Academic MindTrek
Conference: Envisioning Future Media Environments. 41–48.
[16] Tsugunosuke Sakai, Haruya Tamaki, Yosuke Ota, Ryohei Egusa, Shigenori Inagaki, Fusako Kusunoki, Masanori Sugimoto, and H. Mizoguch. 2017. Eda-based
estimation of visual attention by observation of eye blink frequency. International Journal on Smart Sensing and Intelligent Systems 10 (01 2017), 296–307.

303

AHs ’21, February 22–24, 2021, Rovaniemi, Finland

https://doi.org/10.21307/ijssis-2017-212[17] Moe Sugawa, Taichi Furukawa, George Chernyshov, Danny Hynds, Jiawen Han,
Marcelo Padovani, Dingding Zheng, Karola Marky, Kai Kunze, and Kouta Minamizawa. 2021. Boiling Mind: Amplifying the Audience-Performer Connection
through Sonification and Visualization of Heart and Electrodermal Activities. InFifteenth International Conference on Tangible, Embedded, and Embodied Interaction (Salzburg, Austria) (TEI ’21). Association for Computing Machinery, New
York, NY, USA, Article 34, 10 pages. https://doi.org/10.1145/3430524.3440653[18] Benjamin Tag, Andrew W. Vargo, Aman Gupta, George Chernyshov, Kai Kunze,and Tilman Dingler. 2019. Continuous Alertness Assessments: Using EOG Glasses
to Unobtrusively Monitor Fatigue Levels In-The-Wild. In Proceedings of the 2019
CHI Conference on Human Factors in Computing Systems (Glasgow, ScotlandUk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–12.
https://doi.org/10.1145/3290605.3300694[19] Yuji Uema and Kazutaka Inoue. 2017. JINS MEME Algorithm for Estimation andTracking of Concentration of Users. In Proceedings of the 2017 ACM International
Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the2017 ACM International Symposium on Wearable Computers (Maui, Hawaii) (UbiComp ’17). Association for Computing Machinery, New York, NY, USA, 297–300.
https://doi.org/10.1145/3123024.3123189[20] Chen Wang, Erik N. Geelhoed, Phil P. Stenton, and Pablo Cesar. 2014. Sensing
a Live Audience. In Proceedings of the SIGCHI Conference on Human Factors
in Computing Systems (Toronto, Ontario, Canada) (CHI ’14). Association for
Computing Machinery, New York, NY, USA, 1909–1912. https://doi.org/10.1145/2556288.2557154