RoboPianist: Dexterous Piano Playing with Deep Reinforcement Learning
Authors: Kevin Zakka
Philipp Wu
Laura Smith
Nimrod Gileadi
Taylor Howell
Xue Bin Peng
Sumeet Singh
Yuval Tassa
Pete Florence
Andy Zeng
Pieter Abbeel
Venue: CoRL 2023 Poster (2023)
Keywords: high-dimensional control
bi-manual dexterity

Abstract: Replicating human-like dexterity in robot hands represents one of the largest open problems in robotics. Reinforcement learning is a promising approach that has achieved impressive progress in the last few years; however, the class of problems it has typically addressed corresponds to a rather narrow definition of dexterity as compared to human capabilities. To address this gap, we investigate piano-playing, a skill that challenges even the human limits of dexterity, as a means to test high-dimensional control, and which requires high spatial and temporal precision, and complex finger coordination and planning. We introduce RoboPianist, a system that enables simulated anthropomorphic hands to learn an extensive repertoire of 150 piano pieces where traditional model-based optimization struggles. We additionally introduce an open-sourced environment, benchmark of tasks, interpretable evaluation metrics, and open challenges for future study. Our website featuring videos, code, and datasets is available at https://kzakka.com/robopianist/

ROBOPIANIST: Dexterous Piano Playing with Deep Reinforcement Learning
Kevin Zakka 9,, Philipp Wu Laura Smith
Nimrod Gileadi
Taylor Howell", Xue Bin Peng", Sumeet Singh', Yuval Tassa' Pete Florence', Andy Zeng", Pieter Abbeel #UC Berkeley, Gooole DeepMind, "Stanford University, Simmn Fraser University Correspondence to: zakka@berkeley. edu
Replicating human-like dexterity in robot hands represents one of the largest open problems in robotics. Reinforcement learning promising ap- proach that has achieved impressive progress in the last few years; however, the class of problems it has typically addressed corresponds to rather narrow def- inition of dexterity as compared to human capabilities. To address this gap, we investigate piano-playing, a skill that challenges even the human limits of dexter- ity, as means to test high-dimensional control, and which requires high spatial and temporal precision, and complex finger coordination and planning. We intro- duce ROBOPIANIST, a system that enables simulated anthropomorphic hands learn an extensive repertoire of 150 piano pieces where traditional model-based optimization struggles. We additionally introduce an open-sourced environment, benchmark of tasks, interpretable evaluation metrics, and open challenges for future study. Our website featuring videos, code, and datasets is available at Kttps://kzakka.coo http://aakkaacco/oooopol/// anist/.
Abstract:
Keywords: high-dimensional control, bi- bi-manual dexterity
1
Introduction
Despite decades-long research into replicating the dexterity of the human hand. high- sigh-dimensional control remains grand challenge in robotics. This topic has inspired considerable research from both mechanical design [1, 2, 3] and control theoretic points view [4, 5, 6, 7, 8]. Learning- bbased approaches have dominated the recent literature, demonstrating proficiency with in-hand cube orien- tation and manipulation [9, 10, 11] and have scaled a wide variety of geometries [12, 13, 14, 15]. These tasks, however, correspond human capabilities. In particular, most tasks are well-specified using single goal state or termina- tion condition, limiting the complexity of the solution space and often yielding unnatural-looking behaviors so long as they satisfy the goal state. How can we bestow robots with artificial embodied intelligence that exhibits the same precision and agility as the human motor control system?
narrow set of dexterous behaviors relative to the breadth of
In this work, we seek challenge our methods with tasks commensurate with this complexity and with the goal of emergent human-like dexterous capabilities. To this end, introduce family of tasks where success exemplifies many of the properties that we seek in high-dimensional control policies. Our unique desiderata are (i) spatial and temporal precision, (ii) coordination. and (iii) planning. We thus built an anthropomorphic simulated robot system, consisting two robot hands situated at a piano, whose goal is to play a variety of piano pieces, i.e., correctly pressing sequences of keys on keyboard, conditioned on sheet music, the form of a Musical Instrument Digital Interface (MIDI) transcription (see Figure 1). The robot hands exhibit high degrees of freedom (22 actuators per hand, for total of 44), and partially underactuated, akin to human hands. Control- ling this system entails sequencing actions that the hands are able to hit exactly the right notes at exactly the right times; simultaneously achieving multiple different goals, in this case, fingers on
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

B
Embodiment detrespomershee hande 44 DOn poseiton control
at Piano Key with
generates sound when depresand
Fingering
Piano
key with comesponninn (samle color flnger
"pectal"
Figure 1: ROBOPIANIST simulation featuring full-size digital keyboard (A) with piano keys modeled as linear springs (B). In the piano playing task, two (left and right) anthropomorphic Shadow hands (C) are tasked with playing musical piece encoded as trajectory of key presses (D).
each hand hitting different notes without colliding; planning how to press keys in anticipation of how this would enable the hands to reach later notes under space and time constraints.
We propose ROBOPIANIST, an end-to-end system that leverages deep reinforcement learning (RL) synthesize policies capable of playing diverse repertoire of musical pieces on the piano. We show that combination of careful system design and human priors (in the form of fingering anno- tations) is crucial its performance. Furthermore, we introduce ROBOPIANISTE REPERTOIRE a benchmark of 150 songs, which allows us comprehensively evaluate our proposed system and show that it surpasses a strong model-based based approach by over 83%. Finally, we demonstrate the effectiveness of multi-task imitation learning in training a single policy capable of playing multiple songs. To facilitate further research and provide a challenging benchmark for high-dimensional con- trol, we open source the piano-playing environment along with https://kzakke, /wtak/zaaaaaaaaaaa com hwtpp/anooppi........o//
Related Work
We address related work within two primary areas: dexterous nigh-dimensional control, and robotic pianists. For more comprehensive related work, please see Appendix A.
Dexterous Manipulation as a High--iimesiooal Control Problem. nipulation literature uses fower-dimensional systems (i.e., single-arm, simple end-effectors), which circumvents challenges that arise in more complex systems. Specifically, only a handful of general- purpose policy optimization methods have been shown work on high-dimensional hands, even for a single hand [10, 9, 12, 11, 16, 14, 17, 7J, and of these, only subset has demonstrated results in the real world [10, 9, 12, 11, 16). Results with bi-manual hands are rarer [15, 18]. In addi- tion. the class of problems generally tackled in these settings corresponds to definition of dexterity pertaining to traditional manipulation skills [19], such re-orientation, relocation, manipulating simply-articulated objects (e.g., door-opening, ball throwing and catching), and using simple tools (e.g.. hammer) [20, 21, 15, 11, 22, 12]. This effectively reduces the search space for controls to pre- dominantly single
The vast majority of the ma-
behavior space per task. In contrast, our piano-playing task encompasses more complex notion of a goal, extendable arbitrary difficulty by only varying the musical score.
Robotic Piano Playing. Robotic pianists have rich history within the literature, with several works dedicated to the design of specialized hardware [23, 24, 25, 26, 27, 28), and/or customized controllers for playing back a song using pre- programmed commands [29, 30]. The works of Scholz [31], Yeon [32] use a dexterous hand to play the piano by leveraging a combination of inverse kine- matics and offline trajectory planning. In Xu et al. [33], the authors formulate piano playing as an RL problem for single Allegro hand miniature piano and leverage tactile sensor feedback. The piano playing tasks considered in these prior works are relatively simple (c.g., play up six succes-

---

sive notes, or three successive chords with only two simultaneous keys pressed for each chord). On the other hand, ROROPIANIST allows a general bi-manual controllable agent to emulate a pianist's growing proficiency by providing a large range of musical pieces with graded difficulties.
Experimental Setup
In this section, we introduce the simulated piano-playing environment as well as the musical suite used train and evaluate our agent.
Simulation details. We build our simulated piano-playing environment (depicted in Figure us- ing the open-source MuJoCo [34, 35] physics engine. The piano model is full--sie digital keyboard with 52 white keys and 36 black keys. We use Kawai manual [36] reference for the keys' po- sitioning and dimensions. Each key is modeled as joint with a linear spring and is considered "active" when its joint position is within 0.5% of its maximum range, at which point a synthesizer used to generate a musical note. We also implement a mechanism to sustain the sound of any currently active note to mimic the mechanical effect of sustain pedal on real piano. The left and right hands are Shadow Dexterous Hand [37] models from MuJoCo Menagerie [38], which have been designed to closely reproduce the kinematics of the human hand.
Musical representation. We use the Musical Instrument Digital Interface (MIDI) standard to represent a musical piece as sequence of time-stamped messages corresponding to note- or note-off events. A message carries additional pieces of information such as the pitch of a note and its velocity. We convert the MIDI file into a time-indexed note trajectory (a.k.a, piano roll), where each note represented representation for the agent, informing it which keys must be pressed each time step.
one-hot vector of length 88. This trajectory used as the goal
Musical evaluation. We precision, recall, and Fl scores evaluate the proficiency of our agent. These metrics are computed by comparing the state of the piano keys at every time step with the corresponding ground-truth state, averaged across all time steps. If at any given time there are keys that should be "on" " and keys that should be "off", precision measures how good the agent is at not hitting any of the "off" keys, while recall measures how good the agent is hitting the "on" keys. The Fl score combines precision and recall into one metric, and ranges from (if either precision or recall is 0) to I (perfect precision and recall). We primarily use the F1 score for our evaluations as it is common heuristic accuracy score in the audio information retrieval literature [39]. and found empirically that it correlates with qualitative performance on our tasks.
MDP formulation. We model piano-playing as a finite-horizon Markov Decision Process (MDP) defined by a tuple (S,A, P.P, r.7. H) where S C IR" is the state space, A C R" is the action space, p(-) is the initial state distribution, p( p(1|8, a) governs the dynamics, r A + R defines the rewards, 7 E [0, 1) the discount factor, and H is the horizon. The goal of an agent maximize its total expected discounted reward over the horizon: E
ΣH Ei ("r(,,
The agent's observations consist of proprioceptive and goal state information. The proprioceptive state contains hand and keyboard joint positions. The goal state infor- mation contains vector of key goal states obtained by indexing the piano roll at the current time step, as well as a discrete vector indicating which fingers of the hands should be used at that timestep. To successfully play the piano, the agent must be aware of at least few seconds' worth of its next goals in order to be able to plan appro- priately. Thus the goal state is stacked for some lookahead horizon L. A detailed description of the observation space is given in Table 1. The agent's 's action is 45 dimensional and consists of target joint angles for the hand with an additional scalar value for the sustain pedal. The agent predicts
Observations
Unit
Size
Hand and forearm joints Foreurm Cartesian position
Piano key joints
rad
88
Active fingers
discrete
L. 10
Piano key goal state
discrete
L- 88
The agent's observation space. corresponds to the lookahead horizon.
Table 1:
3

---

Twinkle Twinkle
Clair de Lune
Nocturne
1.00
1.00
1.00
0.75
0.75
0.75
0.50
0.50
0.50
0.25
0.25
0.25
0.00
0.00
0.00
0
2.5M
5M
0
2.5M
5M
2.5M
5M
Samples
Samples
Samples
Sparse Rew.
Fingering Min. Energy
+ Lookahead
Reduced Action
Figure 2: ROBOPIANIST Design Considerations. The Fl performance for 3 songs of increasing difficulty.
From left to right as specified the legend, each curve inherits all MDP attributes of the curve before it but makes one additional modification as deseribed by its label.
target joint angles at 20Hz and the targets are converted to torques using PD controllers running at 500Hz. Since the reward function is crucial for learning performance, we discuss its design in Sec- tion
Fingering labels and dataset. Piano fingering refers the assignment of fingers to notes, e.g., "C4 played by the index finger of the right hand". Sheet music will typically provide sparse fingering labels for the tricky sections of piece help guide pianists, and pianists will often develop their own fingering preferences for given piece. Since fingering labels aren't 't available in MIDI files by default, we used annotations from the PIG dataset [40] to create corpus of 150 annotated MIDI files for use in the simulated environment. Overall this dataset we call REPERTOIRE-156 contains piano pieces from 24 Western composers spanning baroque, classical and romantic periods. The pieces vary in difficulty, ranging from relatively easy (e.g., Mozart's Piano Sonata 545 in C major) to significantly harder (e.g., Scriabin's 's Piano Sonata No. 5) and range in length from tens of seconds over 3 minutes.
4 ROBOPIANIST System Design
Our aim is to enable robots to exhibit sophisticated, nigh-dimensional control necessary for success- fully performing challenging musical pieces. Mastery of the piano requires (i) spatial and temporal precision (hitting the right notes, at the right time), (ii) coordination (simultancously achieving mul- tiple different goals, in this case, fingers on each hand hitting different notes, without colliding), and (iii) planning (how a key is pressed should be conditioned on the expectation how would enable policy to reach future notes). These behaviors not emerge if we solely optimize with a sparse reward for pressing the right keys at the right times. The main challenge exploration, which is further exacerbated by the high-dimensional nature of the control space.
We overcome this challenge with careful system design and human priors, which we detail in this section. The main results are illustrated in Figure 2. We pick 3 songs in increasing difficulty from ROBOP/ANISTREPPRRTRRE 150. We note that "Twinkle Twinkle" the easiest while "Nocturne" the hardest. We train for 5M samples, 3 seeds. We evaluate the Fl every 10K training steps for episode (no stochasticity in the environment).
4.1 Human priors
We found that the agent struggled to play the piano with sparse reward signal due to the explo- ration challenge associated with the high-dimensional action space. To overcome this issue, we incorporated the fingering labels within the reward formulation (Table 2). When we remove this prior and only reward the agent for the key press, the agent's Fl stays at zero and no substamtial learning progress is made. We suspect that the benefit of fingering comes not only from helping the agent achieve the current goal, but facilitating key presses subsequent timesteps. Having the policy discover its own preferred fingering, like experienced pianist, is exciting direction for future research.

---

4.2 Reward design
We first include reward proportional to how depressed the keys that should be active are. We then add constant penalty if any inactive keys are pressed hard enough to produce sound. This gives the agent some leeway to rest its fingers on inactive keys so long as they don't produce sound. We found that giving a constant penalty regardless of the number of false positives was crucial for learning; otherwise, the agent would become too conservative and hover above the keyboard without pressing any keys. In contrast, the smooth reward for pressing active keys plays an important role in exploration by providing dense learning signal. We introduce two additional shaping terms: (i encourage the fingers to be spatially close to the keys they need to press (as prescribed by the fingering labels) to help exploration, and (ii) we minimize energy expenditure, which reduces variance (across seeds) and erratic behavior control policies trained with RL are prone to generate. The total reward at a given time step is a weighted sum over the aforementioned components. A detailed description of the reward function can be found in Appendix
4.3 Peeking into the future
We observe additional improvements in performance and variance from including future goal states in the observation, i.c., increasing the lookahead horizon L. Intuitively, this allows the policy to better plan for future notes for example by placing the non-finger joints (e.g., the wrist) in manner that allows more timely reaching of notes at the next timestep.
4.4 Constraining the action space
To alleviate exploration even further, we explore disabling degrees freedom [41] in the Shadow Hand that either do not exist in the human hand (e.g., the little finger being opposable) or are not strictly necessary for most songs. We additionally reduce the joint range of the thumb. While this speeds up learning considerably, we observe that with additional training time, the full action space eventually achieves similar Fl performance.
5 Results
In this section, present experimental findings on ROBOPIANIST-ETUDE-L set of ROBOPIANIS-REPERTTRR-- ROBOPIANIST-REPERROIII 150 can be found in Appendix C. We design our experiments to an- swer the following questions:
sub- consisting of 12 songs. The results on the full
How does our method compare to a strong baseline in being able to play individual pieces? How can enable single policy to learn play more than one song?
What effects do our design decisions have on the feasibility of acquiring highly complex, dexterous control policies?
5.1 Specialist Policy Learning
For our policy optimizer, we use a state-of-the-art model-free RL algorithm DroQ [42], one of sev- eral regularized variants of the widely-used Soft-Actor-Crinn [43] algorithm. We evaluate online predictive control (MPC) as a meaningful point of comparison. Specifically, we use the implemen- tation from Howell et al. [44] that leverages the physics engine dynamics model, and which was shown to solve wrwriiosll--oossdeeed--hheee simulation. Amongst various planner options in [44], we found the most success with Predictive Sampling, derivative-free sampling-basse method. A detailed discussion of this choice of base- line and its implementation can be found in Appendix F. Our method uses million samples to train for each song using the same set of hyperparameters, and the MPC baseline is run one-tenth of real-time speed to give the planner adequate search time.
dexterous manipulation tasks [9, 13, 11] in
5

---

Figure 3: Policies in the ROBOPIANIST environment displaying skilled piano behaviors such (left) simul- taneously controlling both hands to reach for notes on opposite ends the keyboard, (middle) playing chord with the left hand by precisely and simultaneously hitting a note triplet, and (right) playing a trill with the right hand, which involves rapidly alternating between two adjacent notes.
1.0
The quantitative results are shown in Figure 4. We ob- serve that the ROBOPIANIST agent significantly outper- forms the MPC baseline, achieving an average F1 score of 0.79 compared to 0.43 for MPC. We hypothesize that the main bottleneck for MPC is compute: the planner strug- gles with the large search space which means the qual- ity of the solutions that can be found in the limited time budget is poor. Qualitatively, our learned agent displays remarkably skilled piano behaviors such as (1) simultane- ously controlling both hands to reach for notes on oppo- site ends of the keyboard, (2) playing chords by precisely and simultaneously hitting note triplets, and (3) play- ing trills by rapidly alternating between adjacent notes (see Figure 3). We encourage the reader to listen to these policies on the supplementary !
RoboPianis: VPC
0.8
0.6
0.4
0.2
0.0
Faom Smota tass
the ary
Corld Creen Pee coas
Task
Figure 4: The Fl scores achieved hy ROBOPIANIST (blue) and MPC (green) for cach of the ROROPIANIST-ETUDD- 12 tasks.
5.2 Multi-Song Policy Learning
Multitask RL Ideally, a single agent should be able learn how to play all songs. Secondly, given enough songs to practice on, we would like this agent to zero-shot generalize ones. To investigate whether this is possible, we create multi-task environment where the num- ber of tasks in the environment corresponds to the number of songs available in the training set. Note that in this multi-song setting, envi- ronments of increasing size are additive (i.c., 2-song environment contains the same song as the I-song environment plus additional one). We use Für Elise as the base song and report the performance of single agent trained on increasing amounts of training songs in Fig- ure 5. We observe that training on an increas- ing amount of songs is significantly harder than training specialist policies on individual songs. Indeed, the F1 score on Für Elise continues drop as the number of training songs increases, from roughly 0.7 Fl for I song (i.c., the specialist) to almost 0 F1 for 16 songs. We also evaluate the agent's s zero-shot performance (i.e., no fine-tuning) on ROBOPIANIST-ETUDE
Multitask RL Training
(Train) Für Elise F1
(Test) Etude-12
0.060
0.6
0.056
0.4
0.052
0.048
0.0
0.044
3e5
6e5
9c5
3e5
6e5
9e5
Samples
Samples
Number of Training Songs
2
8
16
Figure 5: Multi-song training with RI. We test the ability to learn multiple songs simultancously. The training performance on the shared song. Für Elise, is shown on the left. We see adding more songs degrades performance on individual songs. On the right, eval- uate on the held out ROBOPIANIST-ETUDA 12 songs, suggesting the difficulty of generalization with multi- task RL training.
(which does not have overlap with the training songs) to test our RL agent's ' ability to generalize. We see, perhaps surprisingly, that multitask RL. training fails to posi- tively transfer on the test set regardless of the size of the pre-training tasks.
hattp:://keakka.ccm/roooooo

---

Multitask BC- Scaling Songs
Multita Songs Scallng Model
(Train) For Elise
(est Etude: 12
(Train) Für Else F2
(Test) Etude 12 F1
0.8
0.150
0.6
0.125
0.4
0.135
0.100
0.2
0.2
0.075
2e5
Samples
Samples
Samples
Samples
Train Songs
Hidden Dimens lon
64
128
512
1024
(a) Scaling songs for model with hidden dimension of 1024.
(b) Scaling the model size trained on 64 songs.
Figure 6: Here show the learning curves of BC, showing the interplay between the attempted number of training songs and model size. The song Für Elise, the score across Für Elise and ROROPIANIST-FTUDE 12. see that for small number of songs, the model quickly overfits, suggesting that for the given model size the amount of data and diversity is lacking. We see that adding more songs results in higher test F1, suggesting that the model is beginning to generalize. However, zero-shot test performance is far from that of the expert.
the training dataset for each experiment.
show
As we increase the model size, the Fl score on the training song improves. suggesting that larger models can better capture the complexity of the piano playing task across multiple songs.
Multitask Behavioral Cloning Since multitask training with RL is challenging, we instead distill the specialist RL policies trained in Subsection 5.1 into a single multitask policy with Behavioral Cloning (BC) [45]. To do so, we collect 100 trajectories for each song the RODOPINIIS-REPERRIIRR- 1 150, hold out 2 for validation, and use the ROBOPIANIST-ETUDE- trajectories for testing. We then train feed-forward neural network to predict the expert actions conditioned on the state and goal using mean squared error loss. For more direct comparison with multi-task RL, the dataset subsets use trajectories from the same songs used in the equivalently sized multi-task RL experiments. We observe in Figure 6a that as we increase the number of songs in the training dataset, the model's s ability to generalize improves, resulting in higher test Fl scores. We note that for large model (hidden size of 1024), training on too few songs results in overfitting because there isn't 't enough data. Using a smaller model alleviates this issue, as shown in the more detailed multitask BC results found in Appendix E, but smaller models are unable to perform well on multiple songs. Despite having better generalization performance than RL, zero-shot performance on ROBOPIANIST-ETUDE 12 falls far below the performance achieved by the specialist policy. Ad- ditionally, we investigate the effect of model size on the multitask BC performance. We train models with different hidden dimensions (fixing the number of hidden layers) on a dataset of 64 songs. As shown in Figure 6b, a smaller hidden dimension results in lower performance which most likely indicates underfitting.
5.3 Further analysis
In this section, we discuss the effect of certain hyperparameters on the performance of the RL agent.
Control frequency and lookahead horizon: Figure 7 illustrates the interplay between the control frequency (defined as the reciprocal of control timestep) and the lookahead horizon L, and their effect on the Fl score. Too large of control timestep can make it impossible to play faster songs, while a smaller control timestep increases the effective task horizon, thereby increasing the compu- tational complexity of the task. Lookahead on the other hand controls how far into the future the agent can see goal states. We observe that as the control timestep decreases, the lookahead must be increased to maintain the agent ability to see and reason about future goal states. A control frequency of 20Hz (0.05 seconds) a sweet spot, with notably 100Hz (0.01 seconds) drastically reducing the final score. At 100Hz, the MDP becomes too long-horizon, which complicates exploration, and at 10Hz, the discretization of the MIDI file becomes too coarse, which negatively impacts the timing of the notes.
7

---

Twinkle Twinkle
Elise
1.0
1.0
0.8
0.8
0.6
0.6
0.2
0.0
0.0
2.5M
5M
0
2.5M
5M
Samples
Samples
0.84
0.88
0.92
0.96
0.99
645 2218
Figure 7: Lookahead & Control Timestep. These parameters jointly modulate the fore- sight and control granularity of the agent. We find control timestep of 0.05 offers optimal control granularity, and performance generally improves with the lookahead up to a point.
Figure 8: Discount factor. find the discount factor to have a high impact on F1, especially as the task complexity increases. While simpler songs (Twinkle Twinkle) the dis- count factor does not have any visible effect, it critical for faster pucerppaced songs (Für Elise) The curves show that any dis- count in the range of 0.84-0.92 can be usd with similar effect.
Discount factor: As shown Figure 8, the discount factor has a significant effect on the F1 score. We sweep over range of discount factors on two songs of varying difficulty. Notably, find that discounts in the range 0.84 to 0.92 produce policies with roughly the same performance and high discount factors (e.g., 0.99, 0.96) result in lower F1 performance. Qualitatively, on Für Elise, we noticed that agents trained with higher discounts were often conservative, opting to skip entire note sub-segments. However, agents trained with lower discount factors were willing to risk making mistakes in the early stages of training and thus quickly learned how to correctly strike the notes and attain higher F1 scores.
Discussion
Limitations While ROBOPIANIST produces agents that push the boundaries bi-manual dexter- ous control, it does so simplified simulation of the real world. For example, the velocity of note, which modulates the strength of the key press, is ignored in the current reward formulation. Thus, the dynamic markings of the composition are ignored. Furthermore, our RL training approach can be considered wasteful, in that we learn by attempting to play the entire piece at the start of every episode, rather than focusing on the parts of the song that need more practicing. Finally, our results highlight the challenges of multitask learning especially in the RL setting.
Conclusion In this paper, we introduced ROBOPIANIST, which provides simulation framework and suite tasks the form of a corpus of songs, together with a high-quality baseline and vari- ous axes evaluation, for studying the challenging high- high-dimensional control problem mastering piano-playing with two hands. Our results demonstrate the effectiveness of our approach in learning broad repertoire of musical pieces, and highlight the importance of various design choices re- quired for achieving this performance. There is array of exciting future directions explore with ROBOPIANIST including: leveraging human priors to accelerate learning (e.g., motion priors from YouTube), studying zero-shot generalization to new songs, incorporating multimodal data such as sound and touch. We believe that ROBOPIANIST serves as valuable platform for the research com- munity, enabling further advancements in high high-dimensional control and dexterous manipulation.
Acknowledgments
We would like to thank members of the Robot Learning Lab and anonymous reviewers for their feedback and helpful comments. This project supported in part by ONR #N00014-22-1-2122 under the Science of Autonomy program.

---

References
[1] S. Yuan, L. Shao, C. L. Yako, A. M. Gruebele, and J. K. Salisbury. Design and control of roller grasper v2 for in-hand manipulation. 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 9151-9158, 2020.
[2] Z. Xu and E. Todorov. Design of a highly biomimetic anthropomorphic robotic hand towards artificial limb regeneration. In 2016 IEEE International Conference Robotics and Automa- tion (ICRA), pages 3485-3492, 2016.
[3] C. McCann, V. Patel, and A. Dollar. The stewart hand: A highly dexterous, six-degrees-of- freedom manipulator based on the stewart-gough platform. icra, 28(2):23-36, 2021. doi: 10.11099MRA22221:30447773
[4] R. Fearing. Implementing force strategy for object re-orientation. In Proceedings. 1986 IEEE International Conference on Robotics and Automation, volume 3, pages 96-102, 1986. 10. 10:110111000 111119000TT.1990 198.199966..8775.
[5] D. Rus. In-hand dexterous manipulation of piecewise-smooth 3-d objects. The International Journal of Robotics Reseurch, 18(4):355-38, 1999.
[6] A. Okamura, N. Smaby, and M. Cutkosky. An overview of dexterous manipulation. In Pro- ceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No.00CH37065), volume 1, pages 255-262 vol.1, 2000. doi: 1e.1100.1100 100.100000BT..0000000000 844007.
[7] T. Pang, J. T. Suh, L. Yang, and R. Tedrake. Global planning for contact-rich manipulation via local smoothing of quasi-dynamic contact models. ArXiv, abw./200..1787, 2022.
[8] R. R. Ma and A. M. Dollar. On dexterity and dexterous manipulation. In 2011 15th Interna- tional Conference on Advanced Robotics (ICAR), pages 1-7, 2011. doi: 100.111/////./0011111 6088576.
[9] M. Andrychowicz, B. Baker, M. Chociej, Józefowicz. B. McGrew, J. W. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, J. Schneider, Sidor, J. Tobin, Welinder, I.. Weng, and W. Zaremba. Learning dexterous in-hand manipulation. The International Jour- nal of Robotics Research, 39:20 - 2018
[10] OpenAl, L. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, Powell, R. Ribas, J. Schneider, N. A. Tezak, Tworek, P. Welinder, L. Weng, Q. Yuan, W. Zaremba, and L. M. Zhang. Solving rubik's cube with robot hand.
ArXiv, abs/1910.07113,, 2019.
[11] A. Handa, A. Allshire, V. Makoviychuk, A. Petrenko, R. Singh, J. Liu, D. Makoviichuk, K. Van Wyk, A. Zhurkevich, B. Sundaralingam, et al. Dextreme: Transfer of agile in-hand manipulation from simulation to reality. arXiv preprint 2rXiv:2210.13700 2022.
[12] T. Chen, J. Xu, and P. Agrawal. A system for general in-hand object re-orientation. In Confer- ence on Robot Learning, pages 297-307, 2022.
[13] A. Nagabandi, K. Konolige, S. Levine, and V. Kumar. Deep dynamics models for learning dexterous manipulation. Conference on Robot Learning (CoRL), abs/1909. 2019.
[14] Т. Chen, M. Tippur, S. Wu, V. Kumar, E. H. Adelson, and P. Agrawal. Visual dexterity: In-hand dexterous manipulation from depth. ArXiv, 1. 2022.
[15] Y. Chen, Y. Yang. Wu, S. Wang, X. Feng. Jiang, S. M. McAleer, H. Dong, Z. Lu, and S.-
C. Zhu. Towards human-level bimanual dexterous manipulation with reinforcement learning. arXiv preprint arXiv: 08686668666......8888 2022.

---

[16] H. Qi, A. Kumar, R. Calandra, Y. Ma, and J. Malik. In-Hand Object Rotation via Rapid Motor Adaptation. In Conference on Robot Learning (CoRL), 2022.
[17] Mordatch, E. Todorov, and Z. Popović. Discovery of complex behaviors through contact- invariant optimization. ACM Trans. Graph., 31(4), jul 2012. ISSN 0730-0301. doi:10.1145/ 2185520.2185539, URL stttps://doi. httg//...145/2185000 2185539.
[18] A. M. Castro, F. N. Permenter, and X. Han. An unconstrained convex formulation of compliant contact. IEEE Transactions on Robotics, 2022.
[19] R. R. Ma and A. M. Dollar. On dexterity and dexterous manipulation. 2011 15th Interna- tional Conference Advanced Robotics (ICAR), pages 1-7. IEHE, 2011.
[20] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4RL: Datasets for deep data-driven reinforcement learning, arXiv preprint arXiv:2004.072111 2020.
[21] C. Smith, Y. Karayiannidis, Nalpantidis, X. Gratal, P. Qi, D. V. Dimarogonas, and Kragic. Dual arm manipulation-a survey. Robotics and Autonomous systems, 2012.
[22] H. J. Charlesworth and G. Montana. Solving challenging dexterous manipulation tasks with trajectory optimisation and reinforcement learning. In International Conference on Machine Learning, pages 1496-1506. PMLR, 2021.
[23] I. Kato, S. Ohteru, K. Shirai, T. Matsushima, S. Narita, S. Sugano, T. Kobayashi, and E. Fuji- sawa. The robot musician 'wabot-2' (wwasddwww robot-2). ())))),. Robotics, 3(2):143-155, 1987.
[24] J.C. Lin, H.-H. Huang, Y.-F. Li, J.C.. Tai, and I.-W. Liu. Electronic piano playing robot. In 2010 International Symposium on Computer, Communication, Control and Automation (3CA), volume 2, pages 353-356. IEEE, 2010.
[25] T. Maloney. Piano-playing robotic arm. Technical report, 2019.
[26] J. Hughes and P. Maiolino. An anthropomorphic soft skeleton hand exploiting conditional models for piano playing. Science Robotics, 3(25), 2018.
[27] R. Castro Ornelas. Robotic Finger Hardware and Controls Design for Dynamic Piano Playing. PhD thesis, Massachusetts Institute of Technology, 2022.
[28] D. Zhang, J. Lei, B. Li, D. Lau, and C. Cameron. Design and analysis of a piano playing robot. In 2009 International Conference on Information and Automation, pages 757-761, 2009, doi: 10. N0.1199CINFFA2200952200000
[29] Y.-F. Li and L.-L. Chuang. Controller design for music playing appo-eppllied to the anthro- pomorphic piano robot. In 2013 IEEE 10th International Conference on Power Electronics and Drive Systems (PEDS), pages 968-973. IEEE, 2013.
[30] A. Zhang, M. Malhotra, and Y. Matsuoka. Musical piano performance by the act hand. In 2011 IEEE international conference on robotics and automation, pages 3536-3541. IEEE, 2011.
[31] B. Scholz. Playing Piano with a Shadow Dexterous Hand. PhD thesis, Universität Hamburg. 2019.
[32] S. Yeon. Playing piano with a robotic hand, 2022. URL https://github.com com/ seongho-yeon paaoon/plaing..... waayying-pinn---e aamoo-itth---rrbbtee hannd..
[33] H. Xu. Y. Luo, S. Wang, T. Darrell, and Calandra. Towards learning play piano with dexterous hands and touch. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 10410-10416. IEEE, 2022.
10

---

[34] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RS International Conference on Intelligent Robots and Systems, pages 5026- 5033. IEEE, 2012. doi:10.1 (O9/IROS.2002.633880
[35] S. Tunyasuvunakool, A. Muldal, Y. Doron, S. Liu, S. Bohez, Merel, Erez, T. Lil- licrap. N. Heess, and Y. Tassa. dm_contro) Software and tasks for continuous con- trol. Soffware Impacts, 6:100022, 2020. ISSN 2665-9638. dorhhtpss//bbi.rrr//0000016
simpa.2020.100002. URL $2665963820300099.
[36] K. M. Instruments, Kawai Vertical Piano Regulation Manual, 2019. URL https://kavaiuus
http:p/ccooaaee///lpaaaa///////a harpaa/sssaaatao/o///aaaaaaaaeee Piant--eeano--e-itiiion Manuall.
pdf.
[37] Shadow
Dexterous
Hand, 2005.
URL https://www.hhaaorrbbbtttt
wwkterous-hand-seriie
[38] M. M. Contributors, MuJoCo Menagerie: A collection of high-quality simulation models for MuloCo, 2022. URL http: http::///gttbbb cww ceem//deemiddmmuu httpa//dmmjjjoaoaaaaaaaaae
[39] C. Raffel, B. McFee, E. Humphrey, Salamon, O. Nieto, D. Liang, and D. W. Ellis. _vale A transparent implementation of common mir metries. In International Society for Music Information Retrieval Conference, 2014.
[40] E. Nakamura, Y. Saito, and K. Yoshii. Statistical learning and estimation of piano fingering. ArXiv, abs/ 2019.
[41] L. Smith, I. Kostrikov, and Levine. walk in the park: Learning to walk in 20 minutes with model-free reinforcement learning. ArXiv, abs/2208 07860, 2022.
[42] T. Hiraoka, T. Imagawa, T. Hashimoto, T. Onishi, and Y. Tsuruoka. Dropout q-functions for doubly efficient reinforcement learning. International Conference Learning Representa- tions (ICLR), 2022.
[43] T. Haarmoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In ICML, 2018
[44] A. Howell, N. Gileadi, Tunyasuvunakool, K. Zakka, T. Erez, and Y. Tassa. Predictive sampling: Real-time behaviour synthesis with mujoco. ArXiv, abs/2212.00541, 2022.
[45] D. A. Pomerleau. Alvinn: An autonomous land vehicle in neural network. In NIPS, 1988.
[46] A. M. Okamura, N. Smaby, and M. R. Cutkosky. An overview of dexterous manipulation. In Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No. 00CH37065), volume 1, pages 255-262. IEFE, 2000.
[47] A. Rajeswaran, V. Kumar, A. Gupta, J. Schulman, E. Todorov, and S. Levine. Learning com- plex dexterous manipulation with deep reinforcement learning and demonstrations, ArXiv, abs/1709. 10087, 2017.
[48] B. Sundaralingam and T. Hermans. Relaxed-rigidity constraints: kinematic trajectory opti- mization and collision avoidance for in-grasp manipulation. Autonomous Robots, 43:469-483, 2019.
[49] I. Radosavovic, X. Wang, L. Pinto, and J. Malik. State-only imitation learning for dexterous manipulation. In 2021 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 7865-7871. 1. IEEE, 2021.

---

[50] N. Heess, D. TB, S. Sriram, J. Lemmon, J. Merel, G. Wayne, Y. Tassa, T. Erez, Z. Wang, S. Eslami, et al. Emergence of locomotion behaviours in rich environments. arXiv preprint arXiv: 722802286, 2017.
[51] A. Sharma, S. Gu, S. Levine, V. Kumar, and K. Hausman. Dynamics-aware unsupervised discovery of skills. In International Conference on Learning Representations, 2020.
[52] S. Tunyasuvunakool, A. Muldal, Doron, S. Liu, S. Bohez, J. Merel, T. Erez, T. Lillicrap, N. Heess, and Y. Tassa. dm_control: Software and tasks for continuous control. Software Impacts, 6: 6:100022, 2020.
[53] X. B. Peng, Z. Ma, P. Abbeel, S. Levine, and A. Kanazawa. AMP: Adversarial motion priors for stylized physics-based character control. ACM Transactions on Graphics (TOG), 40(4): 1-20, 2021.
[54] J. Merel, L.. Hasenclever, A. Galashov, A. Abuja, V. Pham, G. Wayne, Y. W. Teh, and N. Heess. Neural probabilistic motor primitives for humanoid control. In International Conference on Learning Representations, 2019.
[55] J. Merel, S. Tunyasuvunakool, A. Ahuja, Y. Tassa, L. Hasenclever, V. Pham, T. Erez, G. Wayne, and N. Heess. Catch & carry: reusable neural controllers for vision-guided whole-body tasks. ACM Transactions on Graphics (TOG), 39(4)33--1 2020.
[56] A. Escontrela, X. B. Peng, W. Yu, T. Zhang, A. Iscen, K. Goldberg, and P. Abbeel. Adver- sarial motion priors make good substitutes for complex reward functions. In 2022 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 25-32. IEEE, 2022.
[57] J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, S. Vanderman-Milne and Q. Zhang, JAX: composable transfor- mations of Python+NumPy programs, 2018. URL battp://g/ rituub...... com/ conn/ooglee/axx
[58] H. van Hasselt, Guez, and D. Silver. Deep reinforcement learning with double g-larnning. arXiv e-prints, 2015.
[59] S. Fujimoto, H. van Hoof, and D. Meger. Addressing function approximation error in actor- critic methods. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmassan, Stockholm, Sweden, July 10-15, 2018, 2018.
[60] N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res., 15:1 15:1929-1958, 2014.
[61] J. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. ArXiv, abs/1600.064400 2016.
[62] X. Glorot and Bengio. Understanding the difficulty of training deep feedforward neural networks. In International Conference on Artificial Intelligence and Statistics, 2010.
[63] D. Kingma and J. Ba. Adam: method for stochastic optimization. CoRR, abs/1412.6990, 2014.
[64] Y. Tassa, T. Erez, and E. Todorov. Synthesis and stabilization of complex behaviors through online trajectory optimization. 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 4906 2012.
12

---

A Extended Related Work
We address related work within two primary areas: dexterous high-dimensional control, and robotic pianists.
Dexterous Manipulation and High-Dimensional Control The vast majority of the control lit- erature uses much lower-dimensional systems (i.e., single-arm, simple end-effectors) than high- dimensional dexterous hands. Specifically, only a handful of general- methods have been shown to work on high-dimensional hands, even for single hand [10, 9, 12, 11, 16, 14, 17, 7]. and of these, only a subset has demonstrated results in the real world [10, 9, 12, 11, 16]. Results with bi-manual hands are even rarer, even in simulation only [15, 18].
policy optimization
As a benchmark, perhaps the most distinguishing aspect of ROBOPIANIST is in the definition of "task success". As an example, general manipulation tasks are commonly framed as the continual application of force/torque on an object for the purpose of a desired change in state (e.g., SE(3) pose and velocity). Gradations of dexterity are predominantly centered around the kinematic redundancy of the arm the complexity of the end-effector, ranging from parallel jaw-grippers to anthropomor- phic hands [46, 15]. A gamut of methods have been developed accomplish such tasks, ranging from various combinations of model-based and model-free RL, imitation learning, hierarchical con- trol, etc. [47, 10, 13. 12, 48, 49]. However, the class of problems generally tackled corresponds to a definition of dexterity pertaining to traditional manipulation skills [19], such as re-orientation, relocation, manipulating simply-articulated objects (e.g., door opening, ball throwing and catching), and using simple tools (e.g., hammer) [20, 21, 15, 11, 22]. The only other task suite that we know of that presents bi-manual tasks, the recent Bi-Dex [15] suite, presents broad collection of tasks that fall under this category.
While these works represent an important class of problems, we explore an alternative notion of dexterity and success. particular, for most all the aforementioned suite of manipulation tasks, the "goal" state is some explicit, specific geometric function of the final states; for instance, an open/closed door, object re-oriented, nail hammered, etc. This effectively reduces the search space for controls predominantly single bhasin-of-attractionn in behavior space per task. In contrast, the ROBOPIANIST suite of tasks encompasses more complex notion of a goal, which is encoded through a musical performance. In effect, this becomes highly combinatorially variable sequence of goal states, extendable to arbitrary difficulty by only varying the musical score. "Success" is graded on accuracy over an entire episode; concretely, via a time-varying non-analytic output of the environment, i.e., the music. Thus, it is not matter of the "final-state" that needs to satisfy certain termination/goal conditions, criterion which is generally permissive of less robust execution through the rest of the episode, but rather the behavior of the policy throughout the episode needs to be precise and musical.
Similarly, the literature on humanoid locomotion and more broadly, "character control", another im- portant area of high-dimensional control, primarily features tasks involving the discovery of stable walking/running gaits [50, 51, 52], or the distillation of a finite set of whole-body movement pri- ors [53, 54, 55], to use downstream for training a task-leve) policy. Task success is typically encoded via rewards for motion progress and/or reaching terminal goal condition. is well-doce that the endless pursuit of optimizing for these rewards can yield unrealistic yet "high-reward" behaviors. While works such as [53, 56] attempt to capture stylistic objectives via leveraging demonstration data, these reward functions are simply appended to the primary task objective. This scalarization of multiple objectives yields an arbitrarily subjective Pareto curve of optimal policies. In contrast, performing piece of music entails both objectively measurable precision with regards to melodic and rhythmic accuracy, as well as subjective measure of musicality. Mathematically, this translates as stylistic constraint satisfaction, paving the way for innovative algorithmic advances.
Robotic Piano Playing Robotic pianists have rich history within the literature, with several works dedicated to the design of specialized hardware [23, 24, 25, 26, 27, 28], and/or customized con- trollers for playing back song using pre- pro-programmed commands (open-loop) [29, 30]. The work
13

---

in [31] leverages combination of inverse kinematics and trajectory stitching to play single keys and playback simple patterns and a song with a Shadow hand [37]. More recently, [32], the author simulated robotic piano playing using offline motion planning with inverse kinematics for a 7-DoF robotic arm, along with an Iterative Closest Point-based heuristic for selecting fingering for four-fingered Allegro hand. Each hand simulated separately, and the audio results are com- bined post-hoc. Finally, in [33], the authors formulate piano playing as RL problem for single Allegro hand (four fingers) on a miniature piano, and additionally leverage tactile sensor feedback. However, the tasks considered are rather simplistic (e.g., play up to six successive notes, three successive chords with only two simultaneous keys pressed for each chord). The ROBOPIANIST benchmark suite is designed to allow a general bi-manual controllable agent to emulate a pianist's growing proficiency on the instrument by providing curriculum of musical pieces, graded in diffi- culty. Leveraging two underactuated anthropomorphic hands actuators provides level of realism and exposes the challenge of mastering this suite of high-dimensional control problems.
14

---

B Detailed Reward Function
Reward
Formula
Weight
Explanation
Key Press
0.5 a( kyll2) 0.5 (1 (raise posiiive)
Press right keys and only right keys Minimize energy expendiuure
Energy Penalty
Moints [Miinns
-e-3
Finger Close to Key
sl Pella)
Shaped reward to bring fingers to key
Table The reward function used to train ROBOPIANIST agents. represents the joint torque, velocity, py and ps represent the position the finger and key in the world frame respectively, k, and kg represent the current and the goal states of the key respectively, and g is a function that transforms the distances
joint
rewards in the [0, 1] range.
15

---

C
Full Repertoire Results
RoboPlanist
MPC
Task
0.00
0.25
0.50
0.75
1.00
F1
Figure 9: Results on the full repertoire of 150 songs.
16

---

D ROBOPIANIST Training details
Computing infrastructure and experiment running time
Our model-free RL codebase is implemented in JAX [57]. Experiments were performed Google Cloud m1 -1-highmem--i 64 machine with an Intel Xeon E5-2696V3 Processor hardware with 32 cores (2.3 GHz base clock), 416 GB RAM and 4 Tesla K80 GPUs. Each "run", iee, the training and evaluation of a policy on one task with one seed, took an average of hrs wall clock time. These run times are recorded while performing up to runs in parallel.
Network architecture
We use a regularized variant of clipped double Q-learning [58, 59], specifically DroQ [42]. for the critic. Each Q-function is parameterized by a 3-layer multi-layer perceptron (MLP) with ReLU activations. Each linear layer followed by dropout [60] with a rate 0.01 and layer normaliza- tion [61]. The actor is implemented as a tanh laaeidiiiignguasssse 3-layer MLP that outputs mean and covariance. Both actor and critic MLPs have hidden layers with 256 neurons and their weights are initialized with Xavier initialization [62], while their biases are initialized to zero.
and is also parameterized by
Training and evaluation
We first collect 5000 seed observations with uniform random policy, after which we sample actions using the RL policy. We then perform gradient update every time receive new environment observation. We use the Adam [63] optimizer for neural network optimization. Evaluation happens in parallel in background thread every 10000 steps. The latest policy checkpoint is rolled out by taking the mean of the output (i.e., no sampling). Since our environment is "fixed", we perform only one rollout per evaluation.
Reward formulation
The reward function for training the RL agent consists of three terms: 1) key press term Pkey 2) move finger to key term Flinger. and 3) an energy penalty term Tenergy.
Tkey encourages the policy to press the keys that need be pressed and discourages it from pressing keys that shouldn't 't be pressed. It is implemented as:
Tkey 0.5
K Σ a(1 111z) -11la) + 0.5 (1 1 (false positive)) )
where is the number of keys that need to be pressed the current timestep, k, is the normal- ized joint position of the key between 0 and 1, and (false positive)
indicator function that is 1 if any key that should not be pressed creates a sound. g is the tolerance function from the dm_ control [52] library: it takes the L2 distance of k, and 1 and converts it into a bounded positive number between and 1. We use the parameters bounds=0. 05 and margin=0.. 5.
Fiinger encourages the fingers that are active at the current timestep to move as close as possible to the keys they need to press. It is implemented as:
Ffinger Σ s(lpp) 1k1))).
where py the Cartesian position of the finger and p; is the Cartesian position of point centered at the surface of the key. for this reward is parameterized by bounds=0. 01 and margin=O. 1. Finally, T Fenergy penalizes high energy expenditure and is implemented as:
T Tenergy Tjoints
where Tjoints is vector of joint torques and Vjoints is vector of joint velocities.
The final reward function sums up the aforementioned terms follows:
Trotal Tkey + F'inger 0.005 Fenergy
17

---

Other hyperparameters
For a comprehensive list of hyperparameters used for training the model-free RL policy, see Table 3.
Hyperparameter
Value
Total train steps
5M
Optimizer
Type
ADAM 3 x 10 4
Learning rate
B1
0.9
32
0.999
Critic
Hidden units
256
Hidden layers
3 ReLU 0.01
Non-linearity
Dropout rate
Actor
Hidden units
256
Hidden layers
3 ReLU
Non-linearity
Misc.
Discount factor
0.99
Minibatch size
256
Replay period every
step
Eval period every
10000 step
Number of eval episodes
1
Replay buffer capacity
1M
Seed steps
5000
Critic target update frequency
1
Actor update frequency
1
Critic target. EMA momentum (TQ)
0.005
Actor log std dev. bounds
[-20, 2]
Entropy temperature
1.0
Learnable temperature
True
Table 3: Hyperparameters for all model-free RL experiments.
E
Multitask BC Results
18

---

Train)
(Test) Etude-12 F1
Validation
Loss
0.8
1.2
0.12
0.45
0.20
0.6
0.30
0.4
0.10
0.2
0.04
0.15
0.3
0.0
0.00
0.00
0.00
0.0
2e5
4e5
2e5
4e5
2e5
4e5
0
2e5
4e5
2e5
4e5
0.16
0.60
0.25
0.8
1.2
0.12
0.45
0.20
0.6
.28
0.08
0.30
0.4
0.10
0.2
0.04
0.15
0.05
0.3
0.0
0.00
0.00
0.00
0.0
2e5
4e5
2e5
4e5
2e5
4e5
0
2e5
4e5
2e5
4e5
0.16
0.60
0.25
0.8
1.2
0.12
0.45
0.20
0.6
0.15
55
0.4
0.08
0.30
0.10
0.6
0.2
0.15
0.3
0.0
0.00
0.00
0.00
0.0
2e5
4e5
2e5
4e5
2c5
4e5
2e5
4e5
2e5
4e5
0.16
0.60
0.25
0.12
0.45
0.20
0.6
0.9
0.15
0.4
0.08
0.30
0.10
0.6
0.2
0.3
0.0
0.00
0.00
0.00
0.0
2e5
4e5
2e5
4e5
2e5
4e5
2e5
4e5
2e5
4e5
0.16
0.60
0.25
1.2
0.45
0.6
0.9
1024
0.15
0.4
0.08
0.30
0.10
0.6
0.2
mmm
0.15
0.05
0.3
0.0
0.00
0.00
0.00
0.0
2e5
4e5
2e5
4e5
2e5
4e5
2e5
4e5
2e5
4e5
Samples
Samples
Samples Sings
Samples
Samples
16
32
64
Figure Caption the figure.
F
Baselines
Computing infrastructure and experiment running time
Our MPC codebase implemented in C++ with MJPC [44]. Experiments were performed on 2021 MI Max Macbook Pro with 64 GB of RAM.
Algorithm
We use MPC with Predictive Sampling (PS) as the planner. PS a derivative-free sampling-bart based algorithm that iteratively improves nominal sequence of actions using random search. Concretely, N candidates are created every iteration by sampling from a Gaussian with the nominal as the mean and fixed standard deviation a. The returns from the candidates are evaluated, after which the highest scoring candidate is set the nominal. The action sequences represented with cubic splines to reduce the search space and smooth the trajectory. In our experiments, used
19

---

N = 10, o 0.05, and spline dimension of 2. We plan over horizon of 0.2 seconds, use planning time step of 0.01 seconds and a physics time step of 0.005 seconds.
Cost formulation
The cost function for the MPC baseline consists of terms: 1) key press term Ckey, 2) and move finger to key term Cringer
The costs are implemented similarly to model-free baseline, but don't make use of the g function, ie., they solely consist in unbounded 12 distances.
The total cost is thus:
Ciotal
Ckey + Cringer
Note that we experimented with a control cost and an energy cost but they decreased performance so we disabled them.
Alternative baselines
We also tried the optimized derivative- -bsed implementation of iLQG [64] also provided by [44], but this was not able to make substantial progress even at significantly slower than real-time speeds. iLQG is difficult to make real time because the action dimension is large and the algorithm theoreti- cal complexity is O[|A|S H). The piano task presents additional challenges due to the large number of contacts that are generated at every time step. This make computing derivatives for iLQG very expensive (particularly for implementation which used fnite-differencing compute them). A possible solution would be to analytical derivatives and differentiable collision detection.
Besides online MPC, we could have used offline trajectory optimization to compute short reference trajectories for each finger press offline and then track these references online (in real time) using LQR. We note, however, that the (i) high dimensionality, (ii) complex sequence of goals adding many constraints, and (iii) overall temporal length (tens of seconds) of the trajectories pose chal- lenges for this sort of approach.
20


Reviews:
[{"review_id": "sSI8P5Q86L", "reviewer": "Reviewer_Dnam", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors present an RL based agent that is trained to play piano using bi-manual anthropomorphic hands. The approach uses DroQ as backbone model. The authors contribute by: (1) developing a novel environment; (2) analyzing several choices for observation and reward shaping; (3) proposing a multi-task imitation learning setting for training a single agent able to play multiple songs. This paper may represent a starting point for future research in several domains, such as robotics, machine learning and arts. ", "summary_of_recommendation": "The paper is clear and the approach is sound. The authors address several problems that arise in the development of the agent and propose effective solutions. The contributions are clear and set the basis for interesting future work.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "The paper is interesting and well presented. However, the relevance of this work may be increased with some further experiments/corrections:\n- as most of the contributions rely on the choices of observations and rewards, the authors should present an ablation study regarding the role of such choices;\n- as the MPC results are consistently lower than the model, the authors should also test whether using another backbone model could play an essential role in the agent performance: other exploration and representation techniques may positively impact the performance\n- in the abstract the authors state that the produced tracks are \"acoustically pleasing\", but they do not provide metrics showing the similarity between the track and the correct execution, or any user-based study that demonstrates such property", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is clear and the approach is sound. The authors address several problems that arise in the development of the agent and propose effective solutions. The contributions are clear and set the basis for interesting future work.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "lkQB51oaRR", "reviewer": "Reviewer_eFF7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "Existing simulated tasks for studying dexterous manipulation do not require diverse bimanual behaviors. The authors propose a benchmark consisting of playing 150 piano songs with two Shadow hands. They evaluate this RoboPianist trained with DroQ and find that it outperforms an MPC baseline. Extensive ablations show the impact of various design choices in simulation, reward, and optimization hyperparameters. The 150 songs are shown to have diverse difficulties and are distinct enough that generalizing to a new song or learning multiple songs from scratch is difficult.", "summary_of_recommendation": "RoboPianist makes a novel high-dimensional control task accessible to practitioners and thoroughly evaluates a strong baseline for performance. Clarifying the motivation behind atypical algorithm selection and adding an expanded discussion of sim-to-real feasibility will resolve some outstanding questions.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- tl:dr: If I were trying to either (i) train a multitask policy from scratch or (ii) train a sim-to-real policy with domain randomization, I'd want to use PPO with a huge batch size. What (if anything) prevents this in RoboPianist?\n\n  - All of [9, 10, 11, 12, 14, 15, 16] use PPO as the policy optimizer. Why choose DroQ? I haven't seen a comparison of DroQ vs PPO in dexterous manipulation works or the original DroQ paper. DroQ seems to be focused on sample efficiency and in the original paper, it's evaluated in domains with very few samples (~1e5). If it's more suitable than PPO for dexterous manipulation tasks, that would be novel and useful information for practitioners. \n  - By my math, 5M samples, with a control frequency of 20Hz, is equivalent to 70 hours of data, and this is collected in 5 hours, yielding a 14x realtime factor. This is 1-2 orders of magnitude less than most recent works in dexterous manipulation, mostly implemented in IsaacGym. Is this because of (a) MuJoCo being a bit slower? (b) the task is more complex involving two higher-dof hands and an articulated piano? (c) or maybe the bottleneck is actually DroQ taking a long time to compute the policy update? \n\n- The benchmark lacks environmental aspects that approximate sim-to-real transfer. Specifically, domain randomization, lag modeling, sensor noise, and the infeasibility of directly observing the key angles. These are worth mentioning in the limitations section when discussing how RoboPianist might transfer to a real robot.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n- This paper is dense and its contributions to the environment design for a novel task are significant. Designing the environment and task specification from scratch for such a complex task is a large effort. Its release will allow the community to study bimanual control with a dramatically lowered barrier to entry.\n- The evaluation and ablations of the policy optimization and baseline are very thorough. They evaluate the impact of many interesting hyperparameters and design choices in good detail. \n- The question of generalization across piano songs is interesting to explore. I agree with the assertion in Section 2 that traditional manipulation skills have a dominant \"basin of attraction\" while piano playing requires a more open-ended and diverse set of motor skills. And because the tasks are shown to be very diverse in difficulty and character, any method that improves generalization in this space must be more powerful. That seems like a quality of a good benchmark.\n\nWeaknesses:\n- The policy optimization approach departs from all prior references on learning dexterous manipulation by using DroQ instead of PPO. The motivation behind this design choice should be elaborated in the text.\n- There is no sim-to-real evaluation. Also, no dynamics randomization, sensor noise, or lag modeling is mentioned. Also, the observation includes the piano key joints, which would be hard to sense in the real world. The benchmark could be strengthened by introducing these aspects. Otherwise, I am not sure if advancing progress on this benchmark brings us closer to a real robot that can play the piano.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "MSB7b5NuCG", "reviewer": "Reviewer_F4CG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "- This paper aims to challenge learning-based dexterity with the task of piano playing with bi-manual anthropomorphic hands in simulation.\n- A new simulation environment is created, powered by the MuJoCo engine with two Shadow Hands and a simulated piano.\n- A state space and reward function for RL is designed and justified.\n- Evaluation is based on the alignment between key presses and the \"piano rolls\" extracted from the corresponding MIDI files.\n- A dataset of 150 music pieces is constructed for training and evaluation. \n- A standard RL algorithm (DroQ) is adopted for policies training.\n- Different training paradigms are explored, including training one policy per piece, training one policy for multiple pieces, and training per piece experts followed by distillation to multi-piece policies using behavior cloning (BC).\n- Multitask RL polices struggles with training, while per task policies and multitask BC can excel to a large extent.\n- Zero-shot generalization remains challenging.", "summary_of_recommendation": "The presentation and experiments look very well executed. However, I struggle see a strong merit otherwise. The task itself, without human evaluation, is questionable. Methodology-wise the contribution is also limited and specific to the piano playing problem.\n\n### Post Rebuttal\nThe rebuttal addresses my concern on the over-claimed contributions. While falling short on some of the interesting aspects (e.g. \"aligning with human values of pleasantness\" and \"generalizing to new songs\") at its current form, I agree with the authors that the publication of this work can lead to future development in these areas. I also like the \"unique dimension to policy evaluation\" (\"does my policy sound good\") of the piano task. Therefore I'm willing to raise my recommendation from weak reject to weak accept.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Address the weaknesses listed above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Strengths\n- The paper is looking at a potentially profoundly interesting problem (i.e., robot pianist), which can go beyond just the topic of dexterous control but more deeply into the space of AI creativity and music performing arts. For example, can a robot play classical pieces in idiosyncratic but acoustically pleasant manner? Progress on this front could even make impact in the music domain ultimately.\n- Experiments are thorough and well executed. Technical and experimental details are well presented.\n- The paper is quite open and clear about the limitations of it's approach and the limitations of the results.\n- Overall, the paper is well written and a pleasant read. Figures and illustrations are beautifully made.\n\n### Weaknesses\n- Certain contributions are over-claimed and not well-supported with proper empirical evidence. The abstract claims that the proposed method can \"produce visually and acoustically pleasing performances\" (L11). The same claim is also made in the intro section (L42: \"... producing visually and acoustically pleasing performances,\"). But there is no evaluation nor any reported metrics on visual or acoustical \"pleasantness\". It is also rather counter-intuitive that DRL can learn \"pleasing\" performance without training with any human feedback.\n- Following the last point, one expected metric for \"naturalness\" or \"pleasantness\" is through human evaluation. However, no human evaluation has been provided. Also, from my little personal biased view, I find the provided examples, though impressive in some way, still not quite \"acoustically pleasant\" yet. It still makes some entry-level mistakes that are common among piano beginners.\n- Without evaluating at the level of \"artisticness\" or \"pleasantness\", the task is only aimed to be solved at a \"control\" level, i.e. aligning key presses with the piano roll, which really downplays the artistic side of the task, and rather unuseful. One can build mechanical systems that can \"play piano\" with precise key strokes---why bother with human-like hands?\n- Speaking of merely \"pushing robot dexterity\", the result is rather unexciting. Standard RL can nail the task to a large extent but that can also somewhat be foreseen. Generalization to unseen pieces would be a significant deal, but this paper is not there yet still.\n- Also on the note of \"pushing dexterity\", the techniques presented in this paper seems very specific to the task of piano playing (e.g., state space and reward defined on key presses, fingering, etc.). It is unclear how easy to adapt when one wants to solve a different dexterous tasks.\n- One little thought on the training of multitask RL: rather than \"learning multiple songs simultaneously\" (Fig. 5), why not \"progressively\". I.e., start from one song; once converged, add a new song and continue training on both songs, etc. I thought this was what the authors intended to do but it looks like Fig. 5 is comparing training in different $N$-song envs \"from scratch\".\n\n### Question and Suggestions\n- [L90] \"We use the Musical Instrument Digital Interface (MIDI) standard to represent musical pieces and synthesize sounds.\" -> Sentence duplicated with L84-85.\n- [L112] \"We define actions to be target joint angles for the 22 joints in each hand ... convert them to torques at the 52 hand joints.\" -> $52 - 22 \\times 2 = 8$. What do these 8 extra joints account for? Movement of the forearms' base?\n- [Fig. 2] The term \"Action Reward\" is a bit confusing. Why not just use \"Energy Penalization\"?\n- [L150] \"... sheet music will typically provide sparse fingering information, ...\" -> Is this what is actually used for \"human-prior\" fingering information? This can be made more explicit in the paragraph.\n- [L266] \"can result make it impossible\" -> \"can make it impossible\"\n- [Fig. 7] What is the unit of y-axis (Lookahead)? Timesteps (i.e., 20 means 1 second for 20 Hz control timestep, which seems a bit short for planning)? Or seconds (whereas 20 seconds seems a bit long)?", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "1cZBy3xXHId", "reviewer": "Reviewer_28kq", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper introduces a simulation setup for bi-manual anthropomorphic hands for the task of playing a piano. The task is intricate in itself, requiring coordination among the fingers, planning for efficiently performing the next set of actions, as well as spatial and temporal precision for hitting the right set of notes with appropriate intensity, simultaneously and at multiple stages. The key challenge as highlighted in the paper is the high-dimensional controls for dexterous manipulation which involve continuous on-and-off contact points between the piano keys and the robot hands. The paper proposed a deep reinforcement learning framework which can learn a collection of 150 piano pieces, called the ROBOPIANIST-REPERTOIRE-150 with both, single as well as multi-task policies which can learn to play more than 1 song in a single policy.", "summary_of_recommendation": "The paper poses an interesting problem setup, despite less novelty. It would be good to address the comments posted above. I recommend accepting the paper, but am keen to see some results with the real robot hands and piano setup.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "All the questions/comments that need to be addressed by the author are highlighted in the \"Weaknesses\" section. \n\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Strengths: \n\n- The paper is well-written and the introduction section illustrates clarity about why this problem is important and challenging. Moreover, the problem setting is interesting and different, compared to a majority of tasks usually tackled under the umbrella of dexterous manipulation.\nThe simulation is built and released with an open-source physics simulator (MuJoCo), that makes it easier to reproduce and build on the proposed approach.\n- The Robopianist-repertoire-150 is open-sourced, which helps in evaluating the proposed results in detail and also provides a good benchmark for future research in this direction. \n- The paper discusses the generalization capabilities of the proposed framework in detail. While they show that multi-task RL does not show any benefit for generalization, multi-task behavioral cloning policy performs well and does show generalization capabilities with more data and larger models.\n- The example result provided is an impressive qualitative depiction of the proposed methodology. \n\n#### Weaknesses:\n\n- The results on the website, although impressive, only show very few examples in a single video. Moreover,  the link provided does not provide code or dataset, unliked claimed in the abstract.\n- The demo video does not mention the playback speed, which makes it harder to infer the actual efficacy and latency of the system. Also, no failure results are discussed on published on the website. It would be good if the author can provide a more detailed analysis of these results, commenting on when and why the system fails to better understand the current shortcomings.\n- The current system assumes the availability of the fingering pattern for finger-to-key assignment, similar to (speed) typing. As pointed out by the author, without this information it is hard to optimize for emergent behaviors with sparse rewards. Although this can result in limited behavior emergence through deep RL and is left for future work to explore, it would be good to see some baseline results as an ablation study without these assumptions. This can provide a better picture for qualitative and quantitative comparisons, and also justify the proposed design choices more profoundly.\n- The MPC baseline in Figure 3 is hypothesized to be much worse due to the computational bottleneck of the planner. Can the author explain and show if MPC works comparably or not with the RoboPianist for easier song pieces with fewer notes? \n- No discussion is available for transfer to real world and the challenges that might accompany.\n\n##### Minor errors:\n- Figure 10 in the supplementary is missing a caption. \n- L266 contains an error/type in the sentence - “A control time step that is too large can result in making it impossible…”\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Scores:  Dnam: Weak Accept, eFF7: Weak Accept, 28kq: Weak Accept, 28kq: Weak Accept

Quality:  The paper presents an RL based agent that is trained to play piano in simulation using bi-manual anthropomorphic hands. 
It introduces a multi-task imitation learning setting for training a single agent which is able to play multiple songs.
All reviewers have acknowledged that the study is challenging and very well executed. 

Clarity: The paper is clear and well-organized 

Originality: Good

Significance: The paper presents a novel high-dimensional control task and evaluates a strong baseline for performance, which can serve as a good starting point for future studies. 

Cons:

- lack of a user-based study that demonstrates that the agent plays "acoustically pleasing"

Pros:

- novel high-dimensional dexterous control task highly relevant to robotics 
- clear contributions with a good baseline for future work
- thorough evaluation of the model, extensive ablation study and good discussion of generalization capabilities
Decision: Accept (Poster)