Affective Umbrella – Towards a Novel Sensor Integrated
Multimedia Platform Using Electrodermal and Heart Activity inan Umbrella Handle
Kanyu Chen

cady.cky@kmd.keio.ac.jp
Keio University
Japan

Christopher Kim

chris.kim@kmd.keio.ac.jp
Keio University
Japan

Jiawen Han

hanjiawen@kmd.keio.ac.jp
Keio University
Japan

George Chernyshov

chernyshov@kmd.keio.ac.jp
Keio University
Japan

Ismae Rasa

ismaelrasa@kmd.keio.ac.jp
Keio University
Japan

Kai Kunze

kai@kmd.keio.ac.jp
Keio University
Japan

Figure 1: Affective Umbrella to accompany and understand us more deeply. (a) Overview of the experimental environment anddevices used (b) the dominant hand holding the umbrella during the experiment (c) Wearing a wristband with the other handfor a baseline recording (d) Comparison of the sensor((blood volume pulse) BVP and (electrodermal activity) EDA sensors)placement design of the umbrella handle with the fingers.

ABSTRACTCCS CONCEPTS

We present our first steps towards an umbrella-based novel multimedia platform using physiological data as an integrated feedback
loop. In this paper, we demonstrate the viability of using an umbrella
handle as a form factor to measure electrodermal activity(EDA) andheart rate(HR) in real-time. We compared the performance of thedevice with that of a more conventional finger sensor placement.
Although the finger sensor placement is more widespread and considered to be more reliable, yet we are able to derive meaningful
data from the umbrella handle in both stationary and dynamic
contexts in the presented feasibility study.

• Computer systems organization → Sensors and actuators.

Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
MUM 2021, December 5–8, 2021, Leuven, Belgium© 2021 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-8643-2/21/05.
https://doi.org/10.1145/3490632.3497835

208

KEYWORDS
umbrella handle, electrodermal activity, heart rate
ACM Reference Format:
Kanyu Chen, Jiawen Han, George Chernyshov, Christopher Kim, Ismae
Rasa, and Kai Kunze. 2021. Affective Umbrella – Towards a Novel Sensor
Integrated Multimedia Platform Using Electrodermal and Heart Activity
in an Umbrella Handle. In 20th International Conference on Mobile andUbiquitous Multimedia (MUM 2021), December 5–8, 2021, Leuven, Belgium.
ACM, New York, NY, USA, 3 pages. https://doi.org/10.1145/3490632.3497835

1INTRODUCTION

Umbrellas are one of the most ubiquitous accessories available tous. The motivation of Affective Umbrella is to explore ways ofour emotional connection to nature. Affective Umbrella presents
opportunities for a novel and unobtrusive means to gather physiological data. Umbrella redesign related research focuses on energy

MUM 2021, December 5–8, 2021, Leuven, Belgium

Kanyu Chen, Jiawen Han, George Chernyshov, Christopher Kim, Ismae Rasa, and Kai Kunze

conversion[7, 9, 13], sound space[7, 8], umbrella surface as an Internet interactive screen[3, 11, 12] , etc. We are unaware of any
umbrella redesign embedded with physiological sensors [4, 15, 17].
The main contributions of this paper are: (1) We present anapproach to measure EDA and heart rate data from an umbrella
handle. We describe an initial design as well a layout of the sensors
together with a functional prototype. (2) We present a dataset of 8participants, which is available under the anonymized google drive
link1 , comparing the measured heart rate and EDA to data fromthe participant’s fingers (of the other hand). (3) We show initialinsights that the umbrella handle is feasible to collect consistent,
accurate physiological signal according to the comparison between
the baseline device.

2Manikin) questionnaire, finally received an 8-minutes interview.
During the experiment, participants were required to hold the affective umbrella with their dominant hand and wear the wristband
on the other hand (Figure1), being exposed to audio stimuli listed
to over wireless blue-tooth headphones. Both the umbrella andwristband device used the same analog front-end PCB circuitry,with a wheatstone bridge connected to a differential ADC that communicates to an ESP32 chip over I2C. The resulting data is streamedwirelessly to a nearby computer running in-house software as adata receptacle.

3INITIAL DATA ANALYSIS, DISCUSSION
AND FUTURE WORKPROTOTYPE IMPLEMENTATION AND
INITIAL EVALUATIONFigure 2: Affective umbrella setup: (a) Original sketch (b)Model and render effect (c) Fabrication in a 3D printer (d,e)
Circuit board soldering (f) Using hot melt glue to assemble
the umbrella handle
Taking into account the product design that conforms to usage habits, it is also conducive to improving the stability of datacollection. We consider of the handle material, weight, and appearance. The prototyping of Affective Umbrella: (a) Sensor placement
design -(b) Modeling-(c) Printing-(d,e) Soldering circuit board-(f)Assembling the umbrella ( See as figure?? ). The material is madeof Polylactic Acid(PLA), a light, hard, and environmentally friendly
polymer, which will not bring more burden to users.The handle uses
the same (blood volume pulse) BVP and (electrodermal activity)
EDA sensors as the wristband used in similar research [2, 6, 10, 16].
Placement of the EDA electrodes was chosen to be in line with ourhabits of holding umbrella, EDA sensors are placed diagonally on
the umbrella handle, matching the position of the palm: under thethumb, and under the pad of the index finger; the BVP plethysmograph is located under the middle finger. To check the rationality ofsensor placement and the viability of data collection, we conducted
the following feasibility test.
The main purpose of the experiment is to test the effectivenessof biometric sensors placement and feasibility of the data acquisition, and to further explore the response of physiological data
to different audio effects (rain sound and jump-scare effects) anddifferent states (imitating the actual scenarios of an umbrella: stationary or walking around), so as to understand the potential valueof Affective Umbrella as the ubiquitous multimedia platform.
There were 8 participants in total, four male and four female,ranging from 23 to 32 years old. The experiment included two 7minute sessions, before and after each resting and experimental
period, participants were asked to fill out the SAM (Self Assessment
1 https://drive.google.com/file/d/1CfrYq4YySX6ctSjNVFvawRrUWN0PRkN_/view

209

Figure 3: Graph(A) EDA changes with EDA extrema
counts,EDA extrema peaks (highlighted in orange), trends
in subject07 and subject08 from wristband(Left Y scale) andhandle(Right Y scale), corresponding with audio effects;
Graph(B) Trends in all subjects of HRV PNN50 (Left Y scale)and value from Self Assessment Manikin(SAM) (Right Yscale).
During the study we recorded EDA (4.54 Hz) and BVP (50 Hz)
data alongside the SAM questionnaire. For EDA data analysis, wemainly focus on the fast changes in EDA response (see Figure ??).
The different patterns of EDA across different sessions of experiment are expected. And we picked (subject07) sub07 and (subject08)
sub08’s EDA as an example to show the difference( See Figure 3(Graph A)). Graph(A) shows the extreme values of EDA locatedunder the sound effect stimulus in session 1. Through the audio
decibel peak corresponding to the extreme EDA peak, we found thatthe overall accuracy of the data collected from the umbrella
handle is higher than data from the fingers, (sum amounts ofpeaks with jump scared effects from sub07 and sub08 ) 25 > 6 (peaksfrom sub07 + sub08 with jump scared effects).This can potentiallybe explained by stronger gripping force participants use tohold the umbrella.
HRV features were extracted from BVP data. For simplicity(SeeGraph(B) from Figure??), the changes in PNN50 (strongly relatedto PSNS, less affected by SNS (related to excitement)[1, 5, 14],) andAffective Umbrella – Towards a Novel Sensor Integrated
Multimedia Platform Using Electrodermal and Heart Activity in an Umbrella HandleSAM arousal were used to analyze the average changes of all subjects. It is not difficult to find that the two verify each other. Therefore, the sensor placement and the physiological sensing ofAffective Umbrella are both reasonable and effective, that is,
the umbrella handle can be used as an functional prototype
measuring EDA and HR data.
We propose a method to measure physiological data (EDA, HRdata) using Affective Umbrella, briefly describe and analyze therecorded data, and provide a complete dataset for download. Inaddition, we also found that the space created by the rain andthe umbrella can affect emotions to a certain extent. Future work
includes: (1) Real-time detection of the effects of umbrella spaceand weather (like rain) on our physiology (2) Exploring the reasons
behind different mapping relationships, such as the loudness andfrequency of rain; the intensity of ultraviolet radiation and thephysiological response relation.

ACKNOWLEDGMENTS
This work was supported by JST, PRESTO Grant Number JPMJPR2132,
Japan.

REFERENCES[1] Solange Akselrod, David Gordon, Jeffrey B Madwed, Nancy C Snidman, Danied C
Shannon, and Richard J Cohen. 1985. Hemodynamic regulation: investigation byspectral analysis. American Journal of Physiology-Heart and Circulatory Physiology 249, 4 (1985), H867–H875.
[2] Adrian Borrego, Jorge Latorre, Mariano Alcañiz, and Roberto Llorens. 2019. Reliability of the Empatica E4 wristband to measure electrodermal activity to emotional
stimuli. In 2019 International Conference on Virtual Rehabilitation (ICVR). IEEE,
1–2.
[3] Tianqin Cai, Liming Liu, and Jiahui Tang. 2018. E-umbrella: A Multifunction
Emotion Improvement Umbrella. (2018).
[4] Yannick Daviaux, Emilien Bonhomme, Hans Ivers, Étienne de Sevin, Jean-Arthur
Micoulaud-Franchi, Stéphanie Bioulac, Charles M Morin, Pierre Philip, and Ellemarije Altena. 2020. Event-Related Electrodermal Response to Stress: Results
From a Realistic Driving Simulator Scenario. Human factors 62, 1 (2020), 138–151.

210

MUM 2021, December 5–8, 2021, Leuven, Belgium[5] JPA Delaney and DA Brodie. 2000. Effects of short-term psychological stress on
the time and frequency domains of heart-rate variability. Perceptual and motorskills 91, 2 (2000), 515–524.
[6] Thomas Fritz, Andrew Begel, Sebastian C Müller, Serap Yigit-Elliott, and ManuelaZüger. 2014. Using psycho-physiological measures to assess task difficulty insoftware development. In Proceedings of the 36th international conference onsoftware engineering. 402–413.
[7] Kazuyuki Fujita, Yuichi Itoh, Ai Yoshida, Maya Ozaki, Tetsuya Kikukawa, RyoFukazawa, Kazuki Takashima, Yoshifumi Kitamura, and Fumio Kishino. 2009.
Funbrella: recording and replaying vibrations through an umbrella axis. In Proceedings of the International Conference on Advances in Computer Enterntainment
Technology. 66–71.
[8] Yusuke Kamiyama, Mai Tanaka, and Hiroya Tanaka. 2008. Oto-Shigure: An
Umbrella-Shaped Sound Generator for Musical Expression.. In NIME. 352–353.
[9] Yuichiro Katsumoto and Masa Inakage. 2007. Amagatana. In Proceedings of the15th ACM international conference on Multimedia. 361–362.
[10] Basel Kikhia, Thanos G Stavropoulos, Stelios Andreadis, Niklas Karvonen, Ioannis
Kompatsiaris, Stefan Sävenstedt, Marten Pijl, and Catharina Melander. 2016.
Utilizing a wristband sensor to measure the stress level for people with dementia.
Sensors 16, 12 (2016), 1989.
[11] Takashi Matsumoto, Sho Hashimoto, and Naohito Okude. 2008. The embodied
Web: embodied Web-services interaction with an umbrella for augmented cityexperiences. Computer Animation and Virtual Worlds 19, 1 (2008), 49–66.
[12] Sohichiro Mori and Makoto Oka. 2017. Proposal of Interaction Used Umbrella for
Smartphone. In Human Interface and the Management of Information: Information,
Knowledge and Interaction Design, Sakae Yamamoto (Ed.). Springer International
Publishing, Cham, 579–588.
[13] Xi Li Ni, Li Gao, Xi Xi Zhang, Yan Feng, and Yu Jing Jia. 2012. New Materials
Application in Multifunctional Environmental-Protection Umbrella. In Advanced
Materials Research, Vol. 600. Trans Tech Publ, 242–245.
[14] David Nunan, Gavin RH Sandercock, and David A Brodie. 2010. A quantitative
systematic review of normal values for short-term heart rate variability in healthy
adults. Pacing and clinical electrophysiology 33, 11 (2010), 1407–1417.
[15] Hugo F Posada-Quintero and Ki H Chon. 2020. Innovations in electrodermal
activity data collection and signal processing: A systematic review. Sensors 20, 2(2020), 479.
[16] Moe Sugawa, Taichi Furukawa, George Chernyshov, Danny Hynds, Jiawen Han,Marcelo Padovani, Dingding Zheng, Karola Marky, Kai Kunze, and Kouta Minamizawa. 2021. Boiling Mind: Amplifying the Audience-Performer Connection
through Sonification and Visualization of Heart and Electrodermal Activities. In
Proceedings of the Fifteenth International Conference on Tangible, Embedded, andEmbodied Interaction. 1–10.
[17] Min Cheol Whang, Joa Sang Lim, and Wolfram Boucsein. 2003. Preparing computers for affective communication: A psychophysiological concept and preliminaryresults. Human Factors 45, 4 (2003), 623–634.