Conun-drum: A drumming bot
Aaron Garza
Ayano Hiranaka
Abhishek Raghunathan
Paula Stocco
Spring 2022

1

Abstract

This projects consists of Toro, the robot, playing the drums in a virtual simulation environment.
The environment consists of Toro seated at the center, 5 different drum set instruments, and a base.
The user specifies a music score and the tempo as beats per minute using a custom designed GUI.
This information is transferred to the controller using text files, which subsequently commands
the bot to reach the target points at each time step. Each leg and arm of Toro are controlled
independently, allowing the simulation to play up to 4 instruments at any given time. The controller
is also linked to a python module that simultaneously plays the sounds of the instruments, adding
authenticity in the form of sound to the simulation.

2

Introduction

There have been several attempts at programming robots to play musical instruments. Our
project was inspired by Shimon, a robot that plays the Marimba (figure 1). The robot mimics user
input from a keyboard and plays out the same notes on the Marimba.

Figure 1: Shimon playing the Marimba

1

It should be noted that all the target points for Shimon are present in the horizontal plane. We
wanted to take this one step further by controlling a bimanual robot such as Toro, to play a complete
drum set, require 3D spacial manipulation and control.
The challenge for us was to be able to take user input in an artistic manner(using a GUI), efficiently
transfer this information to the controller, and manipulate the 4 limbs of Toro to reach all the desired
target points at the right times, while also ensuring that the sound module was synchronized andplaying the right sounds at each time step. Other aspects of the project included importing a drum
set and integrating it into the simulation environment, seating Toro, replacing his end effectors withdrumsticks, and determining appropriate arm and leg positions, choosing target points on each drumfrom the CAD information,

3

Final Implementation

3.1

Environmental Modeling and System Overview

To mimic a human drummer, we select Toro, a bimanual humanoid for our simulation. Toro isequipped with a custom drumstick end effector, and is seated at a drumset placed onto a stage.
In terms of code structure, the project consists of two main modules that communicate throughredis: the Sai2.0 simulation and the external Python modules enhancing the simulation. First,
the user inputs a sequence of drum beats and a tempo through a GUI. The controller then receives
necessasry information from the GUI to simulate Toro playing the beats inputted by the user. When
Toro’s drum sticks arrive at an instrument, the controller signals the audio player module to play
the corresponding sound. This workflow as well as the simulation environment are illustrated infigure 2.

Figure 2: Simulation environment (left) and system architecture (right)

3.2

State Machines

Figure 3 below shows the high level state machine for the left and right arms of Toro. The
state machine initializes into a HOME state when the user starts the GUI. In this state, the robot’s
joints and end effector positions are fixed in a starting configuration. Once the user has input abeat and pressed the play button in the GUI, the controller will indicate that the robot has started
playing and transition to the FIRST MOVING state. During this transition, the data from the2

GUI is copied into a text file with time data corresponding to the times at which Toro will play the
specified drum, and position data corresponding to the desired positions Toro’s end effector mustreach to play each drum. The controller then reads the data from the text files and places them into
vectors that the state machine can index into. In the FIRST MOVING state, both arms move to astarting position above the first drum they must play. For example, if the left arm’s first drum toplay is the snare, then the end effector will move to (xsnare , ysnare , zsnare +dz) where dz corresponds
to a fixed distance above the drum. Once both arms are in position, the arms will wait to start
playing until a unified start time has been reached. In this case, we have specified that time to be 5
seconds after the play button has been pressed. This is the time at which Toro plays the first beat.
In order for the drum stick to hit the drum on time, Toro’s arm must start moving ahead of time.
To account for the time it takes Toro’s arms to reach each drum from the ready position above thedrum, the state machine factors in the velocity of the end effector when hitting, and the amount oftime it might take the arm to accelerate to this velocity. The max velocity of the end effector is set
to vhit = 0.5 m/s when hitting so the time to hit variable is calculated as dz/vhit . The time theend effector takes to accelerate to this velocity, tbuf f er , was estimated empirically. As an example,
if Toro has to play a drum at 15 seconds, then Toro will start moving at 15 − time to hit − tbuf f er
seconds. Once this time has been reached, the state transitions to the HITTING state. Once thevector norm of the current position of the end effector is within 0.0001 meters to the vector normof the desired drum position (i.e. the drum has been hit), the state machine transitions to the
LIFTING STICK state. In this state, each arm moves back up to the z + dz position. When the
drum stick has been lifted, the state machine will move on to the next drum that has to be hit andmove to the MOVING STATE. At this point, the state machine will move from MOVING STATE
to HITTING DRUM to LIFTING DRUMSTICK in a loop as Toro loops through the beat. If thestop button is pressed at any point in time and at any state in the machine, the state will transition
back to HOME and Toro will move to the initial position and joint configuration. The user can
input a new beat or simply press play again to let the state machine run its course.

Figure 3: State machine for left and right arms of Toro

3

The state machine for Toro’s legs shown in Figure 4 functions similarly to the state machine forthe arms. The difference here is that only the joint position of the ankle is controlled since playing
the bass drum and hi hat involves stomping the foot. Again, starting the GUI initializes the statemachine in the HOME state and pressing play will cause a transition into the MOVING STATE
where the ankle joint moves to an initial joint position to get ready for the first stomp. Like the arms,
the feet start moving ahead of time so that they finish the stomp at the right time. This calculationaccounts for the angular velocity of the joints and a buffer time to account for the acceleration of
the joint to the desired angular velocity. Once the time to initiate the stomp is reached, the statemachine moves to the STOMP state. In this state, the machine checks if the joint angle is close to
the desired angle for the stomp. After this position is reached, the state machine transitions to the
WAIT state where the feet move back to their ready positions to wait for the time to initiate thenext stomp. The state machine then transitions between STOMP and WAIT for the duration of
Toro’s playing. When the user presses the stop button, the state moves to HOME and waits until
the user presses the play button again.

Figure 4: State machine for left and right legs of Toro

3.3

Controller

A hierarchical controller with successive Nullspaces and constrained Jacobians is used throughout
the state machine. All operational space control laws are proportional derivative controllers withdynamic decoupling and trajectory tracking. Each subsequent operational space task command
torques are projected onto the combined Nullspace of the previous tasks. The lowest priority law
is a PD joint task. In order of highest to lowest priorities, the control laws are: hip-base position,right end effector position and orientation, left end effector position and orientation, and finally
joint position. The desired joint position for left foot and right foot were dependent on the statemachine, while the remaining joints were commanded to maintain their original position, preventing
for example Toro’s arms from rotating about his shoulders once his drum stick reached its desired
position. Collisions were avoided through the state machine driven positions and trajectory planning,rather than additional repulsion fields between limbs and instruments.

4

T
T
T
Ft|rh|lh + Jt|rh|lh|j
Ft|rh|lh|jΓ = JtT Ft + Jt|rh
Ft|rh + Jt|rh|lh

whereΓ is the total commanded torques
J is a constraint-consistent and task-consistent posture Jacobian
F is the calculated forces including dynamic decoupling in the operational or joint spaces
This hierarchical controller with dynamic decoupling was chosen to isolate the limb tasks andreduce motion coupling. Velocity saturation was initially used, under the hypothesis that achieving amaximum velocity as quickly as possible would produce the most repeatable downward motion from
lifted drumstick to hit position. However, this method stops applying motor torques once velocity
is reached and thus performed less reliably than trajectory tracking which computes intermediate
desired positions using our specified maximum velocity, acceleration, and jerk, and will actively
attempt to correct for position error during travel.

3.4

User Interface

A graphical user interface (GUI), shown in figure 5, is developed using PyQt5 library. The gridof toggle buttons located in the center of the window represents a measure of a drum score, and eachcolumn represents an eighth note. The time signature is common type. The user can click buttons
in the grid to select which instrument Toro will play at each beat, and set a tempo using the tempo
slider. Clicking the CLEAR button clears the user’s beat selection. When the PLAY button isclicked, the GUI first confirms that the user input is valid (i.e. the input beat does not require morethan two arms), and displays an error message if the input is impossible. If the input is valid, each
instrument is assigned to each of Toro’s four limbs in a way that prevent Toro’s arms from crossing
each other. Then, a text file containing desired end effector position and time pairs for a measureare exported for each limb. This file is read by the controller. Finally, a redis flag is set signaling thecontroller that the data preparation step is complete. At any point while the simulation is running,
the user can click the STOP button, which sets a redis flag signaling the controller to reset, putting
the simulation in a state allowing new user input.

Figure 5: Graphical user interface accepting drum score and tempo from users

5

3.5

Sound Generation

An important aspect of this project was to include drum sounds to accompany Toro’s playing.
A separate Python module using the Pygame library’s audio capabilities was used for playback of
downloaded drum samples. Since Toro can play multiple drums at once, Pygame’s mixer module
was perfect for our application since it allows for multiple sounds to be played without incorporating
blocking code or obstructing sounds that have already started playing. The Redis server communicates between the controller module and sound module. Separate keys for each drum were made to
indicate whenever a drum’s corresponding sound should play. In the controller, whenever Toro has
played a drum, the appropriate drum Redis key is set to 1. The Python sound module constantly
checks the values of these keys, plays the appropriate sound whenever a key is read as 1, and sets
the value of the key back to 0.

3.6

Results

We found a 0.02 second difference between instrument sounds is audibly detectable. The final
controller is able to play most configurations with error between any of the four limbs consistently
below 0.02 seconds. The following analysis is an illustration of the team’s challenge in perfecting
consistent timing, and is not intended as a deep investigation into the cause, which is contained in
the challenges section. The musical measure shown in Table 1 was set through the GUI and thedata visualized as an example of the entire systems behavior:
Instrument
Tom2
Tom1
Snare
Bass
Hi-hatBeat2, 4, 6, 7, 8
3, 5, 6, 8
1, 5, 73, 4, 5, 6, 7, 8
1, 2, 5, 6, 7, 8

Table 1: Instruments playing each of eight beats, repeated after each measure
Figure 6 shows the largest difference between all the limbs for a repeated eight beat measure.
For this note combination, every fifth beat when both arms and legs are playing is more than the
audible delay between notes.
If the hit times for the instruments playing during the note of interest, beat 5, are reviewed using
Figure 7, it can be seen that it is not a general delay but rather an early foot hits causing the large
difference between instrument hit times.
Additionally, during other musical combinations not pictured, one limb may be off audibly but
not consistently. There are multiple factors affecting note playing, which are discussed further inthe challenges section along with suggested improvements.

3.7

Discussion

One of the team’s goals was to have Toro play at 15 BPM, and not only appear visually on time
but have a sound component that truly reflected the controller capabilities. As described in the
Final Implementation section, one of the challenges was tuning the controller such that all limbs
always played within the non-distinguishable 0.02s window from one another. This task was madeharder at higher speeds.

6Figure 6: Largest difference between any two instruments at every beatDuring tuning, we found that print statements and redis key reads in occasionally accessed portions of the code could affect processing time and thus controller performance. While we attempted
to optimize the code, this effect may still account for some notes being off beat.
The current controller design could contribute to the challenge of tuning Toro, because the natureof the hierarchical control prevents lower priority tasks from affecting higher priority ones but notvice versa.
To illustrate,the closed loop system equations to tune the PD parameters of the right hand areshown below. Gravity, centrifugal, and Coriolis affects are left out of the following analysis.
The dynamics are written using the operational space inertia matrix and operation space coordinates for the right hand end effector, and the dynamically consistent Jacobian inverse used to map
the control torques into the appropriate space.
Λ̂rh ẍrh = J¯t|rh Γ
The left hand end effector and joint tasks are only in the Nullspace of the right hand end effector
task therefore dropping away, and being in its own space the right hand end effector control force isleft.
Λ̂rh ẍrh = J¯t|rh JtT Ft + Ft|rh
The PD constants to be tuned are within the right hand force equation. However, regardless ofdynamic decoupling and any chosen kv and kp values, the affect of the hip control task J¯t|rh (JtT Ft )will cause an unaccounted for disturbance. If the hip-base position task happens to fall within theright arm task space, it can cause an error in position.
Methods we would consider if stating anew for addressing this are included in the challenges
section.
7

Figure 7: Largest difference at every beat for all instruments

4

Challenges

The Conon-drum project gave us the opportunity to explore multi-limb control with the Tororobot. Working with Toro posed many challenges which we reflected on to consider how similar
problems might best be addressed in the future.

4.1

Faster BPM and Consistent Timing

We found that differences in controller and simulation loop rate made a large difference; teammembers who could not run at 1 kHz often encountered repeated drum hits or large delays in hits
that were never seen at 1 kHz run time. Similarly, we found that print statement and Redis key
reads would affect the processing time and therefore the hit time. In addition to optimizing run time,several improvements could be made to the controller to address current weaknesses that posed achallenge.
Trajectory tracking was implemented for both arm tasks, and found to be fairly consistent.
However, an error in the start position, which was set as a given vertical displacement from thedrum hit location, would generate different trajectories and therefore different hit times. To remove
steady state error from the initial position and integral term could be added to the current PD
controller.
A difficulty in producing consistent timing at faster tempos in particular was the non-instantreaction of the hand position reaching its desired pose. Although the threshold distance for reachingdesired position was reached and registered, Toro was unable to stop or reverse direction quickly
enough with an enforced acceleration and jerk. One way to address this could be to implementa collision between the end effector and the drum. The hit would force Toro to stop and return
immediately; inconsistent trajectories upwards caused by impact would be acceptable because time8

to reach goal position for states other than hitting drum does not affect instrument hit time as longas the wait position is reached in time. See 8 in the Appendix for a screen shot showing a preliminary
implementation of this idea on the Conun-drum bot.
The challenge of higher priority tasks affecting lower ones could be addressed with a feed forward
term added to all non-first priority tasks.
Another method to address this effect would be to combine all tasks into a single Jacobian,which could make the error between them smaller as one task would not be enforced above another.
Compared to the equations shared in the controller section, the PD constant equations would be
solved simultaneous to achieve a critically damped system.
Additionally, centrifugal and Coriolis forces were not compensated for in our controller, anddoing so could also improve its accuracy.

4.2

Head Nod

Early in the project, nodding Toro’s head to the beat was incorporated into the joint task to help
visualize the count in time. However, when fine tuning the hit times we discovered this task affected
the final controller accuracy and it was removed. This could be remediated using a hierarchical joint
controller, see 9 in the Appendix for preliminary code implementation. Alternatively, the aforementioned (subsection 4.1) combined task controller could be used with the head task subsequently
projected into the null space.

5

Conclusion

Playing the drums like a human is certainly an unusual task for a robot. While we often viewrobots as accurate, tireless, and lifeless machines, our experience working with ”Toro the ConunDrummer” contradicted with this norm. Customizing Toro’s drum sets, implementing, improving,
and tuning our controller, and watching Toro get better at drumming was a heartwarming experience- as if we were watching a child learning something new. As robots take on more and more roles insociety, it is not only important for them to be safe and reliable, but also likeable to humans. We
believe robots like ”Toro the Conun-Drummer” challenging itself with atypical, human-like tasks
can help change the impression of robots as a human’s ”tool” to a ”skilled collaborator.” In our
attempt to mimic a human drummer, we were also greatly impressed by the human drummer’s
skills, consistently hitting the instruments at millisecond-scale accuracy. Although our ”ConunDrummer” is not skilled enough to jam with human musicians yet, we hope to see robots joining
humans in creative tasks in the near future.
Visit https://www.youtube.com/watch?v=IfrgMBxHBiM to see Toro in action.

96

References
Free CAD files used for drum set:https://grabcad.com/library/drum-set-1/details?folderi d = 1505224
Free Bass Drum sound effects:
https://www.fesliyanstudios.com/royalty-free-sound-effects-download/bass-drum-274

10

7

Appendix

Figure 8: Collision objects on the drum and Toro’s hand were added and collision checking incorporated.

11

Figure 9: Code to create a hierarchical joint controller keeping Toro’s head nod in the Nullspace

12