STAMPER: Human-machine Integrated Drumming
Daisuke Uriu

The University of Tokyo
Tokyo, Japanuriu@star.rcast.u-tokyo.ac.jp

Takumi Handa

Waseda University
Tokyo, Japan
taku.handa@gmail.com

Shuta Iiyama

Taketsugu Okada

The University of Tokyo
Tokyo, Japansyuta.iiyama1905@gmail.com

Azumi Maekawa

The University of Tokyo
Tokyo, Japanazumi@star.rcast.u-tokyo.ac.jp

The University of Tokyo
Tokyo, Japan
take2ogu@gmail.com

Masahiko Inami

The University of Tokyo
Tokyo, Japaninami@star.rcast.u-tokyo.ac.jp

Figure 1: Left, Center: a STAMPER installation; Right: Detecting the drummer’s motions on OpenPose.

ABSTRACTSTAMPER enables drummers to generate innovative performancesby controlling multiple bass drums. STAMPER consists of several“Actuated Pedals (APs),” bass drum pedals embedded with an EC
motor, paired with bass drums, and a machine vision system. The
APs are used both as input and output. One AP in input mode senses
the position of the pedal being used by the drummer. Other APs in
actuated mode respond to the input mode AP’s motion. Machine
vision is used to monitor the drummer’s movements to control the
behavior of the actuated APs. In this paper, we describe the current
prototype of STAMPER and what kind of drum performances canbe achieved with the system by reporting a study conducted withan experienced drummer.

CCS CONCEPTS• Human-centered computing → Human computer interaction (HCI).

KEYWORDS
Drums, Human Augmentation, Human-machine Integration, NewInterfaces for Musical Expression, Digital Musical Instruments
Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor proft or commercial advantage and that copies bear this notice and the full citationon the frst page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
CHI EA ’23, April 23–28, 2023, Hamburg, Germany© 2023 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-9422-2/23/04.
https://doi.org/10.1145/3544549.3585619

ACM Reference Format:
Daisuke Uriu, Shuta Iiyama, Taketsugu Okada, Takumi Handa, Azumi
Maekawa, and Masahiko Inami. 2023. STAMPER: Human-machine Integrated Drumming. In Extended Abstracts of the 2023 CHI Conference on
Human Factors in Computing Systems (CHI EA ’23), April 23–28, 2023, Hamburg, Germany. ACM, New York, NY, USA, 5 pages. https://doi.org/10.1145/
3544549.3585619

1

INTRODUCTION

Drumming is a musical performance using all four human limbs.
The drum set is a manual machine designed for the structure of
the human body (four limbs), with which the drummer createsvarious sound expressions. The drummer beats the drums withboth arms, while the right leg pedals beat the bass drum and the leftfoot operates the high-hat. This can be interpreted as an example
of a performance achieved through human-machine integration.
There are currently several emerging research domains studying integrated and cooperative movement between humans andmachines in the HCI community. These are human augmentation(e.g., [7, 14, 15]), human-computer integration [3, 13], and humanmachine mutual actuation [10, 11]. A common feature of these
domains is that they focus on designing and implementing systems
for sensing and assisting human movement through computer operated machinery. Previous works have attempted to develop new
sports [7, 8], support physical motion [5, 10, 11], generate supernumerary limbs [14, 15, 17], and expand musical expression [2, 6]. We
believe that applying these technologies to musical instruments canenable musicians to create new and attractive musical expressions
that were previously impossible.

CHI EA ’23, April 23–28, 2023, Hamburg, Germany

With this background in mind, we invented a human-machine
integrated drumming system called “STAMPER” which enables
drummers to explore and generate novel drum expressions whichcannot be achieved with traditional drum sets.1 STAMPER consistsof a generic drum set with a set of modifed bass drum pedals called“Actuated Pedals” (APs: Figure 2), additional bass drums, and a machine vision system that monitors the drummer’s motions (Figure
1 Right) and refects them on controlling APs for the additionalbass drums. The APs work together to allow the drummer to play
multiple bass drums at the same time. One AP senses the drummer’s kicking of the bass drum while the others actively kick theadditional drums. The machine vision system allows the drummer
to freely control how the bass drums play by opening and closing
the knees to control parameters such as 1) counts: how many times
the additional bass drums beat and 2) delay time: how long therepeated beats are delayed. Diferent from playing with the beat ofa computer-generated rhythm, a drummer playing STAMPER is incontrol of the computer-mediated machine and is free to investigate and invent new rhythms and unique drum expressions while
keeping a sense of agency.
Artistic expressions generated by drummers using STAMPER canbe defned as computer/machine mediated music performances [12,16] as well as human-machine cooperative music performances [1,9]. In the professional music scene, it is known that Rick Allen (hardrock band Def Leppard’s drummer) who lost one of his arms in an
accident plays a specially made drum set integrating analog and electric drums. In a similar vein, there is research on robotic drumming
using a machine-made arm [6]. The STAMPER project has been conducted to design human-machine integrated musical performances,synthesizing the multiple research areas of technology-enabledmusical performances and the human augmentations.

Uriu, et al.

2 THE STAMPER SYSTEM
2.1 Design Rationale
As mentioned, STAMPER, as a human-machine integrated drumming system, consists of an integrated interactive system and adrummer using unique ways to control this new musical instrument. While the hardware part (the APs) was logically designed andimplemented (described below), we iteratively conducted trials anderrors to investigate how a drummer could control multiple APs.
Before deciding on the way of opening and closing the knees, there
were other ideas, such as preparing special drumsticks attachedwith a button to control the APs, sensing the tilt of the upper bodyduring drumming, controlling the APs by raising and lowering
the elbows, etc. We prepared rapid prototypes for these ideas andtested them. Finally, we decided to focus on the knee movementsthat do not interfere with the drumming. After preparing the initial prototype, we invited a drummer and collaboratively updated
the system such as the accuracy of sensing the drummer’s movements, and the usability as a musical instrument. For example, we
found out the swinging of the drummer’s upper body center during
drumming and this hindered sensing the opening and closing of
the knees. To address this, we attached an AR marker (Figure. 6C) to the drum chair to detect its rotation and refected the tilt of
the drummer’s upper body in detecting the knee movements. Our
work in progress describes the current version of the STAMPER
system and presents the possibility of the construction of future
human-machine integrated drum expressions.

2.2

Actuated Pedal (AP)The AP (Figure 2) is a pedal device with an embedded EC motor.
It can be used as both an input interface to sense the drummer’s
pedaling and a computer controlled actuator. When in input mode,the drummer can use the AP in the same way they might use anormal drum pedal, while it senses the drummer’s pedaling via
the EC motor’s encoder. When in actuated mode, the AP can be
actuated through the EC motor. As such, the AP can be controlled
both manually and via a computer. The STAMPER system requires
one input mode AP and one or more actuated mode APs.

16”
Clash
CymbalCamera for
OpenPose

14”
Tom

UI
Monitor

14”
Hi-hat14”
Snare

Camera detecting
AR marker

6”
Electric
CymbalMIDI Signal

6”Electric
Cymbal22” Bass

22” Bass

22” Bass

AP 1

AP 2

AP 3
Actuation Signal
Received Signal (Pedal Angle)

PC

Figure 2: Components of Actuated Pedal
Figure 3: Overview of STAMPER: Three APs are placed oneach bass drum. A camera set overhead observes the drummer’s movements on OpenPose. Another camera detects an
AR marker put on the swivel drum stool to detect its rotation.
1 Please watch the introductory video for the STAMPER system at https://www.youtube.

com/watch?v=4y3OBM9zNIQ.

STAMPER: Human-machine Integrated Drumming

CHI EA ’23, April 23–28, 2023, Hamburg, Germany

Figure 4: UI in Metronome Mode

Figure 5: UI in Delay Mode

2.3

Modes and Functions

2.3.1 Play Modes. STAMPER allows the drummer to choose twomodes to manipulate AP2 and 3. The two modes are called Metronome
and Delay. Each mode has parameters that can be adjusted by the
drummer, and the drummer can manipulate the parameters by moving his knees during the performance. Each mode is described indetail below.
1) Metronome Mode: AP2 and AP3 alternately hit the bass drums
with a constant rhythm. In this mode, the drummer canadjust the bpm at which AP2 and AP3 are hit. One of the rolesof the drummer in a band is to provide the band memberswith a constant tempo. We created this mode in the hopethat, by having a machine replace this role, the drummer
will not have to keep the tempo and will be able to play withmore variety.
2) Delay Mode: AP2 and AP3 hit the bass drums at a slight delay
after the drummer hits the bass drum. In this mode, thedelay time of AP2 and AP3 can be adjusted by opening andclosing the knees. By reproducing the delay efect, which is
widely used with guitars and bass guitars, with the APs, weaimed to allow the drummer to express a single sound more
profoundly and create a rhythmic efect using overlapping
sounds.

2.4

Playing STAMPER

When using the STAMPER system (Figure 3), in addition to performing as they might usually, the drummer controls the behaviorof multiple APs by moving their knees. A web camera set in front of
the drummer monitors their body movement, and their knee positions are used to determine the behavior of the APs. In Metronome
Mode, the right knee is assigned to BPM and the left knee is assigned to the volume, as shown in Figure 4. In Delay Mode, theright knee is assigned to delay time and the left knee is assigned
to the number of repetitions (delay count), as shown in Figure 5.
While drumming, the drummer can see the status of each parameteron a monitor. By striking the electric cymbals, they can lock thevalue of each parameter and may lock and unlock the parameters if
needed. One may note, however, that drummers may achieve moregroovy drumming using STAMPER by learning how to move theknees precisely and independently.

Figure 6: A screen captured image taken from the videos: The
Drummer Plays STAMPER (screen capture of the attachedvideos). A&B: Overview of the system. C: An AR Marker detecting the rotation of the drum chair. D: AP1 in input mode.
E: AP2&3: in actuated mode. F: Detecting the drummer’s motions on OpenPose. G: UI the drummer is seeing.

3

DRUM EXPRESSIONS WITH STAMPER

We invited an experienced drummer to play the STAMPER systemand observed what kind of drumming expressions he created. In
our design process, we sometimes invited him to test and play with
our earlier prototypes. He had already adopted STAMPER wellthroughout the previous practices over fve hours, so this section
reports a set of drum expressions with STAMPER by a profcientmusician instead of a beginner. This submission has videos [V1-4]
attached as supplemental materials (Figure. 6).2 Please refer to each
video clip when reading the descriptions below.

3.1

Playing in Metronome ModeHe frst selected Metronome Mode and played 8 beats to see how
the system worked. During this time, he did not move his left knee
but only slightly moved his right knee to control the BPM, trying
to match his rhythm of playing. Occasionally, he stopped beating
the hi-hat and snare and played only the bass drum to check the
speed of AP2’s and AP3’s beats.
In Metronome Mode, an interesting performance was observedin which he beat the bass drum with a subtle shift from the timing of AP2 and AP3 [V1]. During this performance, he locked theparameters to precisely understand a unique rhythm happening
since the blurring of AP2’s and AP3’s tapping timing was reduced.
It seemed that he was trying to create a groovy rhythm with slightshifts between human-made beats and machine-mediated ones.
It happened that the rhythm was shufed [V2]. This was unintentionally caused by a phenomenon that AP2 and AP3 were
not evenly spaced, and only AP3’s beats were delayed due to aslight diference in hardware maintenance. Interestingly, he not
only accidentally encountered the shufed rhythms but also sometimes intentionally changed the rhythm by subtly delaying his own
drumming to delay from the APs.
2 The supplementary videos can be found on this YouTube playlist: https://youtube.

com/playlist?list=PLUe7uE8VX9nmoGNP96AROYmi19iglTil_.

CHI EA ’23, April 23–28, 2023, Hamburg, Germany

3.2

Playing in Delay Mode

Next, he selected Delay Mode and checked the system’s response.
First, he beat the bass drum only. Once he recognized a consistent
response, he added the beats of the hi-hat and snare. He was carefully testing the system’s operation by subtly adjusting the openingof his knees. Occasionally, he locked the parameter and repeatedthe same pattern of beats.
In Delay Mode, a scene was observed in which he intentionallygenerated a 3-beat with the system [V3]. Although there is a slightdelay between AP2 and AP3, the system’s response was consistentenough and he was able to play a 3-beat on a variety of parameters.
In an interesting performance, he noticed that the AP does notdetect weak pedaling, so intentionally combined weak and strong
pedaling to make a new sound [V4]. In Delay Mode, AP1 in the
input mode senses the rotation angle of the embedded EC motor(Figure 2) when the drummer is pedaling the bass drum. He was able
to mix weak pedaling not to beat AP2 and AP3 with strong pedaling
to beat them, creating a more complex rhythmic performance.

3.3

Feedback from the Drummer

After he had played STAMPER for one hour, we interviewed him
for his impressions. The interview was semi-structured, and three
general questions were prepared in advance concerning; 1) overall
impression and evaluation, 2) control of knees opening and closing,and 3) the range of the parameter settings.
3.3.1 Overall Impression and Evaluation. “I enjoyed the system as
a new instrument. I really liked the gap between the strength andrhythm of AP2 and AP3. Some people might not want this gap, as aless maintained system. But I thought the movements were musical.
I felt like they knew the rhythm and groove despite the machines! It
was fun to be able to beat in 3 beats. It was very good, very stable. I
could see the potential for an interesting level.”
We were a little bit worried about the completeness of the current
system. In fact, as he said, there was a subtle diference in thebeater volume between AP2 and AP3. This was probably caused bydiferences in the adjustment of the beater angle, beater length, andspring strength between them. Since AP2 and AP3 were originallyintended to have the same volume settings, this phenomenon canbe viewed as a malfunction. But surprisingly, he interpreted theseissues as if humans play the instrument and make natural rhythmic
grooves, and totally enjoyed the machine-mediated drumming.
3.3.2 Control of Knees Opening and Closing. “I felt like I could do
it because the system response was stable (better than before). I wasable to understand how much I needed to open the knee to get thedesired reaction. However, I felt that the left knee was more unstable
than the right knee. I need more practice.”
Operating the parameters by the knees is one of the most uniquefeatures of the STAMPER system. Though we do not describe thedetailed design process in this paper, we iteratively tested diferent patterns of control methods. Drummers have to use all limbs
when drumming. Thus, we needed to develop a way of controllingparameters without using the limbs. Though we did not conduct formal usability testing with many drummers, he showed an amazing
performance with highly unique drumming.

Uriu, et al.

3.3.3 The Range of the Parameter Setings. “The left knee tends toshake easily, and I thought that a smaller range might be acceptable.
The right knee is OK in the current setting, but I would like to try anarrower setting. Depending on the value of the parameters, I do notthink it is necessary to make it wider. However, I think it is better tokeep the function of being able to make it wider. It might be easier tohandle a subtle change in groove rather than a big change in tempo.
It is actually difcult for drummers to control the knees accurately and there is a limitation to moving the knees during drumming. As such, throughout the collaborative process with drummers,
we were iteratively testing the range of parameters and trying tofgure out the best settings. We found issues and refected them inthe subsequent updates.
3.3.4 Others. “Though AP3 seemed to be a little faster than AP2, therhythmic blurring was regular and consistent. I personally enjoyed
the blurring and gaps in the system but felt that it is not suitable fortwo-bass pedal performances such as in rock and metal music. These
need to be in perfect rhythm. The timing of AP2 and AP3 is slightlyof, so it is not possible to hit accurately in rapid succession. It would
be nice if the drummer could adjust this discrepancy.”
While we encountered how he generated new rhythms in thisstudy, we also noticed the limitations of our current STAMPER
system as of now. To revise the system to support wider genres ofmusic, we need to refne the system’s accuracy and add customizable functions to ft the diferent requirements of drummers.

4

CONCLUSION

This paper introduced STAMPER, a new instrument to explore
human-machine integrated drumming performances. Surprisingly,
the drummer we invited adapted to playing the new instrument
quickly, passionately sought a new technique for drumming, andsometimes produced highly attractive sounds no one has ever heard
before. We believe our fndings will contribute to designing humanmachine integrated performances, not only for music but for otherartistic performances, such as performing arts, and sports, and othereveryday activities.

ACKNOWLEDGMENTS
We would like to express our gratitude to Hidetaka Kikuchi, thedrummer who made a signifcant contribution to this research,
particularly as described in section 3. This research was supported
by JST ERATO Grant Number JPM-JER1701, Japan (INAMI JIZAIBODY PROJECT [4]).

REFERENCES[1] Scott Barton, Ethan Prihar, and Paulo Carvalho. 2017. Cyther: a human-playable,
self-tuning robotic zither.. In NIME. 319–324.
[2] Mason Bretan, Deepak Gopinath, Philip Mullins, and Gil Weinberg. 2016. A
robotic prosthesis for an amputee drummer. arXiv preprint arXiv:1612.04391(2016).
[3] Valdemar Danry, Pat Pataranutaporn, Adam Haar Horowitz, Paul Strohmeier,
Josh Andres, Rakesh Patibanda, Zhuying Li, Takuto Nakamura, Jun Nishida, Pedro
Lopes, Felipe León, Andrea Stevenson Won, Dag Svanæs, Florian Floyd Mueller,
Pattie Maes, Sang-won Leigh, and Nathan Semertzidis. 2021. Do Cyborgs Dreamof Electric Limbs? Experiential Factors in Human-Computer Integration Design
and Evaluation. In Extended Abstracts of the 2021 CHI Conference on Human
Factors in Computing Systems. Association for Computing Machinery, New York,NY, USA, Article 123, 6 pages. https://doi.org/10.1145/3411763.3441355[4] Masahiko Inami, Daisuke Uriu, Zendai Kashino, Shigeo Yoshida, Hiroto Saito,
Azumi Maekawa, and Michiteru Kitazaki. 2022. Cyborgs, Human Augmentation,

STAMPER: Human-machine Integrated Drumming

Cybernetics, and JIZAI Body. In Augmented Humans 2022 (Kashiwa, Chiba, Japan)(AHs 2022). Association for Computing Machinery, New York, NY, USA, 230–242.
https://doi.org/10.1145/3519391.3519401[5] Shunichi Kasahara, Jun Nishida, and Pedro Lopes. 2019. Preemptive Action:
Accelerating Human Reaction Using Electrical Muscle Stimulation Without Compromising Agency. In Proceedings of the 2019 CHI Conference on Human Factors in
Computing Systems. Association for Computing Machinery, New York, NY, USA,
1–15. https://doi.org/10.1145/3290605.3300873[6] Roozbeh Khodambashi, Gil Weinberg, William Singhose, Shima Rishmawi, Varun
Murali, and Euisun Kim. 2016. User oriented assessment of vibration suppression
by command shaping in a supernumerary wearable robotic arm. In 2016 IEEE-RAS
16th International Conference on Humanoid Robots (Humanoids). IEEE, 1067–1072.
[7] Hideki Koike, Jun Rekimoto, Junichi Ushiba, Shinichi Furuya, and Asa Ito. 2021.
Human Augmentation for Skill Acquisition and Skill Transfer. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, Article 93, 3 pages.
https://doi.org/10.1145/3411763.3441354[8] Kai Kunze, Kouta Minamizawa, Stephan Lukosch, Masahiko Inami, and Jun
Rekimoto. 2017. Superhuman Sports: Applying Human Augmentation to PhysicalExercise. IEEE Pervasive Computing 16, 2 (March 2017), 14–17. https://doi.org/
10.1109/MPRV.2017.35[9] Sang-won Leigh and Jeonghyun (Jonna) Lee. 2021. A Study on Learning Advanced
Skills on Co-Playable Robotic Instruments. NIME 2021. https://doi.org/10.21428/
92fbeb44.002be215 https://nime.pubpub.org/pub/h5dqsvpm.
[10] Azumi Maekawa, Seito Matsubara, Sohei Wakisaka, Daisuke Uriu, Atsushi
Hiyama, and Masahiko Inami. 2020. Dynamic Motor Skill Synthesis with HumanMachine Mutual Actuation. In Proceedings of the 2020 CHI Conference on Human
Factors in Computing Systems. Association for Computing Machinery, New York,NY, USA, 1–12. https://doi.org/10.1145/3313831.3376705[11] Azumi Maekawa, Hiroto Saito, Daisuke Uriu, Shunichi Kasahara, and Masahiko
Inami. 2022. Machine-Mediated Teaming: Mixture of Human and Machine in
Physical Gaming Experience. In Proceedings of the 2022 CHI Conference on Human

CHI EA ’23, April 23–28, 2023, Hamburg, Germany

Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). Association
for Computing Machinery, New York, NY, USA, Article 618, 11 pages. https://doi.org/10.1145/3491102.3517555[12] Andrew McPherson. 2010. The magnetic resonator piano: Electronic augmentation of an acoustic grand piano. Journal of New Music Research 39, 3 (2010),
189–202.
[13] Florian Floyd Mueller, Pedro Lopes, Paul Strohmeier, Wendy Ju, Caitlyn Seim,
Martin Weigel, Suranga Nanayakkara, Marianna Obrist, Zhuying Li, Joseph Delfa,
Jun Nishida, Elizabeth M. Gerber, Dag Svanaes, Jonathan Grudin, Stefan Greuter,
Kai Kunze, Thomas Erickson, Steven Greenspan, Masahiko Inami, Joe Marshall,Harald Reiterer, Katrin Wolf, Jochen Meyer, Thecla Schiphorst, Dakuo Wang, andPattie Maes. 2020. Next Steps for Human-Computer Integration. In Proceedingsof the 2020 CHI Conference on Human Factors in Computing Systems. Association
for Computing Machinery, New York, NY, USA, 1–15. https://doi.org/10.1145/
3313831.3376242[14] Federico Parietti, Kameron Chan, and H. Harry Asada. 2014. Bracing the humanbody with supernumerary Robotic Limbs for physical assistance and load reduction. In 2014 IEEE International Conference on Robotics and Automation (ICRA).
141–148. https://doi.org/10.1109/ICRA.2014.6906601[15] MHD Yamen Saraiji, Tomoya Sasaki, Kai Kunze, Kouta Minamizawa, andMasahiko Inami. 2018. MetaArms: Body Remapping Using Feet-Controlled
Artifcial Arms (UIST ’18). Association for Computing Machinery, New York, NY,
USA, 65–74. https://doi.org/10.1145/3242587.3242665[16] Gil Weinberg and Scott Driscoll. 2006. Toward Robotic Musicianship. Comput.
Music J. 30, 4 (dec 2006), 28–45. https://doi.org/10.1162/comj.2006.30.4.28[17] Nahoko Yamamura, Daisuke Uriu, Mitsuru Muramatsu, Yusuke Kamiyama,
Zendai Kashino, Shin Sakamoto, Naoki Tanaka, Toma Tanigawa, Akiyoshi Onishi,
Shigeo Yoshida, Shunji Yamanaka, and Masahiko Inami. 2023. Digital Cyborgs:
The Collaborative Design Process of JIZAI ARMS. In Proceedings of the 2023
CHI Conference on Human Factors in Computing Systems (Hamburg, Germany)(CHI ’23). Association for Computing Machinery, New York, NY, USA, 19 pages.
https://doi.org/10.1145/3544548.3581169