Smartphone Drum: Gesture-based Digital Musical Instruments
Application for Deaf and Hard of Hearing People
Ryo Iijima

ryoiijima@digitalnature.slis.tsukuba.ac.jp
University of Tsukuba
Tsukuba, Japan

Sayan Sarcar

mailtosayan@gmail.com
University of Tsukuba, Japan
Birmingham City University, UK

Akihisa Shitara

theta-akihisa@digitalnature.slis.tsukuba.ac.jp
University of Tsukuba
Tsukuba, Japan

Yoichi Ochiai

wizard@digitalnature.slis.tsukuba.ac.jp
University of Tsukuba
Tsukuba, Japan

ABSTRACT
Smartphone applications that allow users to enjoy playing musicalinstruments have emerged, opening up numerous related opportunities. However, it is difcult for deaf and hard of hearing (DHH)
people to use these apps because of limited access to auditory information. When using real instruments, DHH people typically feelthe music from the vibrations transmitted by the instruments or
the movements of the body, which is not possible when playing
with these apps. We introduce “smartphone drum,” a smartphone
application that presents a drum-like vibrotactile sensation when
the user makes a drumming motion in the air with their smartphone
like a drumstick. We implemented an early prototype and received
feedback from six DHH participants. We discuss the technical implementation and the future of new instruments of vibration.

KEYWORDSMusic, Game, User study, Deaf, Hard of hearing
ACM Reference Format:
Ryo Iijima, Akihisa Shitara, Sayan Sarcar, and Yoichi Ochiai. 2021. Smartphone Drum: Gesture-based Digital Musical Instruments Application for
Deaf and Hard of Hearing People. In Symposium on Spatial User Interaction(SUI ’21), November 9–10, 2021, Virtual Event, USA. ACM, New York, NY,USA, 2 pages. https://doi.org/10.1145/3485279.3488285

1INTRODUCTION AND RELATED WORK

Playing music can strengthen our creativity [1]. In an efort to makethese benefts available to more diverse groups of people, accessible
digital musical instruments (ADMI) are gaining interest in the feldof computer music [3]. Prior research has aided DHH people inplaying musical instruments through sensory substitution. However, it has been suggested that cost is an obstacle to both creation
and utilization of accessibility technologies because products andservices that require specialized devices can be difcult to afordby those with limited income.
Considering this, we aimed to create a musical instrument that
does not rely on sound, using widespread smartphones to realize
SUI ’21, November 9–10, 2021, Virtual Event, USA© 2021 Copyright held by the owner/author(s).
This is the author’s version of the work. It is posted here for your personal use. Notfor redistribution. The defnitive Version of Record was published in Symposiumon Spatial User Interaction (SUI ’21), November 9–10, 2021, Virtual Event, USA, https://doi.org/10.1145/3485279.3488285.

Figure 1: A User playing the smartphone drum. It detects
the drumming motion and presents a drum-like vibrotactile
sensation.
accessible instruments. To achieve this, we introduce the “smartphone drum,” a smartphone application that presents a drum-likevibrotactile sensation when a user makes a drumming motion in
the air using their smartphone as a drumstick.

2 INITIAL PROTOTYPE
2.1 Instrument Design
We frst discussed with our co-author, Shitara, who has profoundhearing loss, regarding the considerations to be made for DHH
people to enjoy musical instruments on their smartphones. We
identifed the following requirements:• Requires less musical knowledge. Some DHH people
may have little or no musical experience. Therefore, thedesign should be enjoyable for both novices and experts.
• Model a musical instrument whose vibrations are easily perceived. Among real instruments, percussion instruments have more recognizable vibrations than pianos orguitars. Such instruments might be better accepted by DHH
people [2].
• Present rich feedback. The sound of acoustic instruments
difers depending on subtle diferences in the performer’smovements. Rich feedback should be presented through vibrations to make the system work as a musical instrument
and to convey the nuances.
With the above requirements in mind, we decided to design an
instrument modeled after a drum. This is because drums can be

SUI ’21, November 9–10, 2021, Virtual Event, USA

enjoyed even without any prior experience or knowledge, and thevibrations transmitted from the drumsticks to the fngertips canbe easily perceived. Furthermore, the beating motion is simple andeasy to recognize, making it easy to follow the rhythm with thewhole body.
We created a prototype application that uses the smartphone
as a drumstick, and when the user makes a drumming motion in
the air, the smartphone produces a vibration as if the user has hit
an actural drum. The envelope of the vibration changes dependingon the strength of the beating motion. We prepared three types ofdrums: hi-hat, snare, and bass; therefore, the characteristics of thevibration would vary depending on the type of drum.

2.2Implementation

The prototype was developed as an iOS application. Core Haptics,
a framework for providing customized haptic feedback, is used topresent a vibrotactile sensation.
Detection of Drumming Motion. The drumming motion is detected using the values obtained from the built-in accelerometerand gyroscope. When the user presses and holds a fnger somewhere on the screen, the system starts capturing the values of themotion sensors at 100Hz until the user takes the fnger of thescreen. When all of the following conditions were met, a drumming
motion was considered to have been performed.
• (current z-axis acceleration) < 1• (previous x-axis gyro) < 0• (current x-axis gyro) < (previous x-axis gyro)
Generating Haptic Pattern. When a drumming motion is detected,
the attitude of the device at that time is obtained, and the pitch
value is used to determine the drum (among hi-hat, snare, andbass) that was hit. A vibration pattern is then generated using thedrum-type and acceleration values as inputs. Core haptics provides
two types of haptic patterns: transient (short and fxed duration)and continuous (customizable duration). In addition, intensity andsharpness (i.e., frequency) can be fnely controlled. This gives us
fne control over the intensity and frequency of drum vibrations,as well as their time variations.

3 USER FEEDBACK
3.1 ExperimentParticipants. We recruited six DHH participants (three femalesand three males) through email, social media, and snowball sampling . Participants were 24 years old on average (SD=4.86, range=1933). Participants were paid ¥430 for the 30-minute study period.
Procedure. We shared our iOS app with the participants to complete the entire study remotely. In the frst half of the study, westarted with a background questionnaire, explained how to use theapplication, and then let the participants play freely for a minimumof 90 seconds. In the second half, semi-structured interviews wereconducted to ask them about their experiences using the app. The
purpose of the semi-structured interview was to receive feedback
on a possible direction for design and functionality.

Iijima, et al.

3.2

Result

The results of the semi-structured interviews were open-coded andorganized into the following three main themes:
Past musical experience. All participants had played the drums
before, but it was only P3 who had heard the sound. P3 was interested in knowing how close the feedback was to the real feelingof the sound, rather than the vibration of the drum . Meanwhile,
P6 said “the vibration from the app is more amusing than that fromactual drums but is lesser.” Users who have heard the actual soundseem to prefer feedback that is closer to the sound of actual drums,
whereas users who have never heard the sound seem to prefer
feedback similar to the vibrations transmitted by actual drums.
Variation of Musical Instruments. After experiencing our app,four participants said they would like to play other instrumentswith vibrations. Suggestions include other types of drums, such as
Japanese drums (P1), shakers like maracas (P2), string instruments
like viola, castanets, and tambourines (P3), and guitar (P4). There
were also several suggestions for performing gestures. For example,
P6 said it would be fun to have more variations of the drumming
motions, such as swinging down with a wrist twisting motion.
Cooperative Play. P1 and P2 said that they wanted to play anensemble with several people, with one person in charge of one
scale. P2 refected on his experiences of playing handbell ensembles
at school, and said that he wanted to play with his friends.

4

CONCLUSION AND FUTURE WORKWe designed the vibration patterns empirically, which may havecaused discrepancies between a smartphone drum and acoustic
drums, specifcally for those who have heard the sound of acoustic
drums. Exploring the design space of vibration patterns that are
comfortable for users will be the subject of a future study.
The primary objective behind this research was to assess theacceptance of the concept of a vibration-based musical instrument
smartphone-app by DHH users and to understand their needs andcapabilities. Therefore, it remains as a future task to investigate theefects of the nature and delay of vibrations on user experience.
Based on feedback from users, we would like to consider creating
instruments other than drums, as well as creating a system for
multiple users to play together.

ACKNOWLEDGMENTS
This work was supported by JST CREST Grant Number JPMJCR19F2,Japan.

REFERENCES[1] Birgitta Cappelen and Anders-Petter Andersson. 2012. Musicking Tangibles for Empowerment. In Computers Helping People with Special Needs, Klaus Miesenberger,
Arthur Karshmer, Petr Penaz, and Wolfgang Zagler (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 254–261. https://doi.org/10.1007/978-3-642-31522-0_38[2] Rumi Hiraga and Nobuko Kato. 2006. Understanding Emotion through Multimedia:
Comparison between Hearing-Impaired People and People with Hearing Abilities.
In Proceedings of the 8th International ACM SIGACCESS Conference on Computersand Accessibility (Portland, Oregon, USA) (Assets ’06). Association for Computing
Machinery, New York, NY, USA, 141–148. https://doi.org/10.1145/1168987.1169012[3] Alon Ilsar and Gail Kenning. 2020. Inclusive Improvisation through Sound andMovement Mapping: From DMI to ADMI (ASSETS ’20). Association for Computing
Machinery, New York, NY, USA, Article 49, 8 pages. https://doi.org/10.1145/3373625.3416988