RP1M: A Large-Scale Motion Dataset for Piano Playing with Bi-Manual Dexterous Robot Hands
Authors: Yi Zhao
Le Chen
Jan Schneider
Quankai Gao
Juho Kannala
Bernhard Schölkopf
Joni Pajarinen
Dieter Büchler
Venue: CoRL 2024 (2024)
Keywords: Bi-manual dexterous robot hands
dataset for robot piano playing
imitation learning
robot learning at scale

Abstract: Endowing robot hands with human-level dexterity is a long-lasting research objective. Bi-manual robot piano playing constitutes a task that combines challenges from dynamic tasks, such as generating fast while precise motions, with slower but contact-rich manipulation problems. Although reinforcement learning based approaches have shown promising results in single-task performance, these methods struggle in a multi-song setting. Our work aims to close this gap and, thereby, enable imitation learning approaches for robot piano playing at scale. To this end, we introduce the Robot Piano 1 Million (RP1M) dataset, containing bi-manual robot piano playing motion data of more than one million trajectories. We formulate finger placements as an optimal transport problem, thus, enabling automatic annotation of vast amounts of unlabeled songs. Benchmarking existing imitation learning approaches shows that such approaches reach state-of-the-art robot piano playing performance by leveraging RP1M.

RPIM: A Large-Scale Motion Dataset for Piano Playing with Bimanual Dexterous Robot Hands
Yi Zhao +11 Le Chen x2 Jan Schneider 2 Quankai Gao
Juho Kannala 1.4 Bernhard Schölkopf Joni Pajarinen Dieter Büchler Aalto University, Finland Max Planck Institute for Intelligent Systems, Germany University of Southern California, USA Unniversity of Oulu, Finland
Abstract: It has been long-standing research goal to endow robot hands with human-level dexterity. Bimanual robot piano playing constitutes task that com- bines challenges from dynamic tasks, such generating fast while precise motions, with slower but contact-rich manipulation problems. Although reinforcement learning-bai approaches have shown promising results single-task perfor- mance, these methods struggle in a multi-song setting. Our work aims to close this gap and, thereby, enable imitation learning approaches for robot piano playing scale. To this end, we introduce the Robol Piano / Million (RPIM) dataset, containing bimanual robot piano playing motion data of more than one million trajectories. We formulate finger placements an optimal transport problem, thus, enabling automatic annotation of vast amounts of unlabeled songs. Benchmarking existing imitation learning approaches shows that such approaches reach promising robot piano playing performance by leveraging RPIM
Keywords: Bimanual dexterous robot hands, dataset for robot piano playing, imitation learning, robot learning at scale
Introduction
Empowering robots with human-level dexterity notoriously challenging. Current robotics research on hand and arm motions focuses on manipulation and dynamic athletic tasks. Manipulation, such grasping reorienting [1], requires continuous application of acceptable forces at moderate speeds to various objects with distinct shapes and weight distributions. Environmental changes, like humidity or temperature, alter the already complex contact dynamics, which adds to the complexity of manipulation tasks. Dynamic tasks, like juggling [2] and table tennis [3] involve making and breaking contact, demanding high precision and tolerating less inaccuracy due to rarer contacts. High speeds in these tasks necessitate greater accelerations and introduce a precision-sper speed tradeoff.
Robot piano playing combines various aspects of dynamic and manipulation tasks: the agent is required to coordinate multiple fingers to precisely press keys for arbitrary songs, which is high- dimensional and rich control task. At the same time, the finger motions have to be highly dynamic, especially for songs with fast rhythms. Well-practiced pianists can play arbitrary songs, but this level of generalization extremely challenging for robots. this work, we build the foundation to develop methods capable of achieving human-level bi-manual dexterity at the intersection of manipulation and dynamic tasks, while reaching such generalization capabilities in multi-task environments.
While reinforcement learning (RL) is a promising direction, traditional RL approaches often struggle to achieve excellent performance multi-task settings [4]. The advent of scalable imitation learning techniques [5] enables representing complex and multi-modal distributions. Such large models
Equal contribution. Correspondence yi. zhao@aalto..... fi, wwen@tuebingen..ngg... Work done during an internship at Max Planck Institute for Intelligent Systems.
Project website: https:://ppt github.iii..
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Hands
Piano
MIDI
Fingering
Collect
O Agent
Cost Matrix
KM ILWLWW
TH
FF
LF
Action
Key
00
06
Key
RP1M
Key
Figure 1: Overview of RPIM. (Left) RPIM is a large-scale motion dataset for piano playing with bi-manual dexterous robot hands. The dataset includes ~IM expert trajectories collected by ~2k RL specialist agents. (Right) To collect a diverse motion dataset of playing sheet music available on the Internet, we lift the requirement of human-annotated fingering by formulating the finger placement as an optimal transport problem such that the robot hands play piano in an energy-efficient way.
most effective when trained on massive datasets that combine the state evolution with the corresponding action trajectories. So far, creating large datasets for robot piano play is problematic due to the time-consuming fingering annotations. Fingering annotations map which finger is supposed to press a particular piano key at each time step. With fingering information, the reward is less sparse, making the training significantly more effective. These labels usually require expert human annotators [6]. preventing the agent from leveraging the large amounts of unlabeled music pieces on the internet [7]. Besides, human-labeles fingering may be infeasible for robots with morphologies different from human hands, such as different numbers of fingers distinct hand dimensions.
In this paper, we propose the Robot Piano I Million datases (RP1M). This dataset comprises the motion data of high-quality bi-manual robot piano play. In particular, we train RL agents for each of the 2k songs and roll out each policy 500 times with different random seeds. To enable the generation of RPIM, we introduce method to learn the fingering automatically by formulating finger placement optimal transport (OT) problem [8, 9]. Intuitively, the fingers are placed in way such that the correct keys are pressed while the overall moving distance of the fingers is minimized. Agents trained using our automatic fingering match the performance of agents trained with human-amnotated fingering labels. Besides, our method is easy to implement with almost no extra training time. The automatic fingering also allows learning piano playing with different embodiments, such as robots with four fingers only. With RPIM, scale. We benchmark various behavior cloning approaches and find that using RPIM, existing methods perform better in terms of generalization capabilities in multi-song piano play. This work contributes in various ways:
now possible to train and test imitation learning approaches
To facilitate the research dexterous robot hands, release RPIM, dataset of piano playing motions that includes more than 2k music pieces with expert trajectories generated by agents.
We formulate fingering an optimal transport problem, enabling the generation of vast amounts of robot piano data with RL, as well as allowing variations in the embodiment.
Using RPIM, we benchmark various approaches to robot piano playing, wherehy existing imitation learning approaches reach promising results in motion synthesis on novel music pieces due to scaling with RPIM.
2
Related Work
Piano Playing with Robots Piano playing with robot hands has been investigated for decades. It a challenging task since bimanual robot hands should precisely press the right keys at the right time, especially considering its high-dimensiona action space. Previous methods require specific robot designs [10, 11, 12, 13, 14, 15] or trajectory proprogramming [16, 17]. Recent methods enable piano playing with dexterous hands through planning [18] or RL [19] but are limited simple music pieces. RoboPianist [4] introduces benchmark for robot piano playing and demonstrates strong RL performance, but requires human fingering labels and performs worse in multi-task learning.

---

Table I: Existing datasets on dexterous or bimanual robotic manipulation.
Dexterous hands
Dynamic tasks
Demonstra-
Dataset
Task
Bimanual
tions
DexGraspNet [28]
grasping
1.3M
RealDex [31]
grasping
/
2.6K
UniDexGrasp [30]
grasping
/
1.IM
ALOHA [33]
manipulation
825
Bi-DexHands [34]
manipulation
V
partially
20K
D4RL [32] (Adroit)
manipulation
V
30K
RPIM (ours)
piano
IM
Yang et al. [20] improves the policy training performance by considering the bionic constraints of the anthropomorphic robot hands.
Human fingering informs the agent of the correspondence between fingers and pressed keys each time step. These labels require expert annotators and are, therefore, expensive acquire in practice. Several approaches learn fingering from human-annotated data with different machine learning methods [6. 21, 22]. Moryossef et al. [23] extract fingering from videos to acquire fingering labels cheaply. Ramoneda et al. [24] proposes to treat piano fingering as sequential decision-making problem and use RL to calculate fingering but without considering the model of robot hands. Shi al. [25] automatically acquires fingering via dynamic programming, but the solution is limited simple tasks. Concurrent work [26] obtains fingering labels from YouTube videos and trains diffusion policy play hundreds of songs. In paper, do not introduce separate fingering model, instead, similar human pianists, fingering is discovered automatically while playing the piano, hereby largely expanding the pool of usable data to train generalist piano-playing agent.
Datasets for Dexterous Robot Hands Most large-scale datasets dexterous robot hands focus on grasping various objects. To get suitable grasp positions, some methods utilize planners [27, 28, 29]. while others use learned grasping policies [30], or track grasping motions of humans and imitate these motions robot hand [31]. Compared to the abundance of datasets for grasping, there exist relatively few datasets for object manipulation with dexterous robot hands. The D4R1. benchmark [32] provides small sets of expert trajectories for four such tasks, consisting of human demonstrations and rollouts of trained policies. Zhao et al. [33] provides a small object manipulation dataset that utilizes a low-cost bimanual platform with simple parallel grippers. Chen et al. [34] collects offline datasets for two simulated bimanual manipulation tasks with dexterous hands. Furthermore, Fan et al. [35] proposes large-scale dataset for bimanual hand-object manipulation with human hands rather than robot hands. Table 1 summarizes the characteristics of these existing datasets. To the best of our knowledge, our RP M dataset is the first large-scale dataset of dynamic, bimanual piano playing with dexterous robot hands.
We further discuss related work on dexterous robot hands and generalist agents in Appendix A.
3 Background
Task Setup The simulated piano-playine environment built upon RoboPianist [4]. includes a robot piano-playing setup, an RL-based agent for playing piano with simulated robot hands, and multi-task learner. To avoid confusion, refer to these components RoboPianist, RoboPianist-RR RL, and RoboPianist-MY, respectively. The piano playing environment features full-size keyboard with 88 keys driven by linear springs, two Shadow robot hands [36], and a pseudo sustain pedal.
Sheet music is represented by Musical Instrument Digital Interface (MIDI) transcription. Fach time step in the MIDI file specifies which piano keys press (active keys). The goal of piano-playing agent is press active keys and avoid inactive keys under space and time constraints. This requires the agent coordinate its fingers and place them properly in highly dynamic scenario such that
3

---

Piano sonala 2022332
French suite no.5 sarabande
Intermezze miner op. 117 no.2
Fleurs de varsevie op.57
1.00
1.00
0.8
0.75
0.75
0.75
3T (Duns)
Sooee
0.50
Robs
0.50
0.50
0.4
No Fingering
0.25
0.00
0.00
0.00
2.5
Enviromment Stepp (1.6))
Envtroement Stepp (1:4g)
Ennvfooment Steps (1:46)
Steps (1.w6)
Figure 2: Comparison of the RI. performance with our OT fingering. human-annotated fingering, and no fingering. Our method matches the performance of RoboPianist-RL which is trained with human fingering. We also outperforms the baseline without any fingering information by a large margin. The plots show the mean over 3 random seeds and the shaded areas represent the 95% confidence interval.
target keys, at not only the current time step but also the future time steps, can be pressed accurately and timely. The original RoboPianist uses MIDI files from the PIG dataset [6] which includes human fingering information annotated by experts. However, as mentioned earlier, this limits the agent to only play human-labeled music pieces, and the human annotation may not be suitable for robots due the different morphologies.
The observation includes the state of the two robot hands, fingertip positions, piano sustain state, piano key states, and a goal vector, resulting in an 144--immnsioonal observation space. The goal includes 10-step active keys and 10-step target sustain states obtained from the MIDI file, represented by a binary vector. RoboPianst further includes 10-step human-labeled fingering the observation space but we remove this observation in our method since we do not need human-labeled fingering. For the action space, remove the DoFs that do not exist in the human hand or are used in most songs, resulting in a 39-dimensional action space, consisting of the joint positions of the robot hands, the positions of forearms, and a sustain pedal. We evaluate the performance of the trained preeision-rocall
agent with an average Fl score calculated by Fi 2 precision+ For piano playing, recall and precision measure the agent's performance on pressing the active keys and avoiding inactive keys respectively [4].
Playing Piano with RL We use RL to train specialist agents per song control the bimanual dexterous robot hands to play the piano. We frame the piano playing task as finite Markov Decision Process (MDP). At time step t, the agent To (a parameterized by 8, receives state 8, and takes action 0e to interact with the environment and receives new state 8++1 and reward rt- The state and action spaces are described above and the reward r. gives an immediate evaluation of the agent's behavior. We will introduce reward terms used for training in Section 4.1. The agent's goal maximize the expected cumulative rewards over an episode of length H, defined as J En 7 (8)) where is discount factor ranging from O to 1.
Large-Scale Motion Dataset Collection
In this section, we describe our RPIM dataset in detail. We first introduce how to train a specialist piano-playing agent without human fingering labels. Removing the requirement of human fingering labels allows the agent to play any sheet music available on the Internet (under copyright license). We then analyze the performance of our specialist RL agent well the learned fingering. Lastly, introduce our collected large-scale motion dataset, RPIM, which includes ~ IM expert trajectories for robot piano playing, covering ~22 pieces of music.
4.1 Piano Playing without Human Fingering Labels
To mitigate the hard exploration problem posed by the sparse rewards, RoboPianist-PP RL adds dense reward signals by using human fingering labels. Fingering informs the agent of the "ground-truthh fingertip positions, and the agent minimizes the Euclidean distance between the current fingertip posi- tions and the "ground-truth" positions. We now discuss our OT based method to lift the requirement of human fingering.

---

Although fingering is highly personalized,, generally speaking, it helps pianists to press keys timely and efficiently. Motivated by this, apart from maximizing the key pressing rewards, we also aim to minimize the moving distances of fingers. Specifically, at time step t, for the i-th key ke to press, we use the j-th finger I to press this key such that the overall moving cost is minimized. We define the minimized cumulative moving distance between fingers and target keys as de Rt given by
pT min We
Σ
we(k', Ck, f9), S.L., 1)
Σ we (
for E K₁,
(
JEF
(1)
i1) Σ (k),
11 for jE m) wn 8)) E {0,1}, for (ix))
KEK.
K, represents the set of keys to press at time step and F represents the fingers of the robot hands. Ce(k', f)) represents the cost of moving finger f to piano key k" time step t calculated by their Euclidean distance. (k'k,, fz) is boolean weight. In case, it enforces that each key in K2 will be pressed by only one finger in F, and each finger presses al mosi one key. The constrained optimization problem in Eq. (1) is an optimal transport problem. Intuitively, it tries to find the best "transport" strategy such that the overall cost of moving (a subset of) fingers F to keys K2 is minimized. We solve this optimization problem with a modified Jonker- Volgenant algorithm [37] from SciPy [38] and use the optimal combinations (i" as the fingering for the agent. The fingering is calculated on the fly based on the hands' state, so during the RL training, the fingering adjusts according the robot hands' state.
We define a reward rt to encourage the agent to move the fingers close to the keys K, which is defined as:
Sexple- (d.T - 0.01)²)
d > 0.01, if or < 0.01.
7 = [1.0 1.0
(2)
is constant scale value as used in Tassa et al. [39] and d. is the distance between fingers and target keys obtained by solving Eq. (1). OT increases exponentially as de decreases and is set as I once ooT is smaller than a pre-defined threshold (0.01). The overall reward function is defined as:
Tt Ft.
Press + r2
+ Sustain
+ ai rr. + 022
Collision
Energy
y Press and SSustain represent the reward for correctly pressing the target keys and the sustain pedal. Collision encourages the agent avoid collision between forearms and rt. Energy prefers energy-saving
?
behaviors. a1 and 02 are coefficient terms, and a₁ = 0.5 and a2 = 5- 5-10-3 are adopted. Our method is compatible with any RL methods, and we use DroQ [40] in our paper.
4.2 Analysis of Specialist RL Agents
The performance of the specialist RL agents decides the quality of our dataset. In this section, investigate the performance of our specialist RL agents. We are interested in i) how the proposed OT- based finger placement helps learning, ii) how the fingering discovered by the agent itself compares to human fingering labels, and iii) how our method transfers to other embodiments.
Results In Fig. 2. we compare our method with RoboPianist-RU both with and without human fingering. We use the same DroQ algorithm with the same hyperparameters for all experiments. RoboPianist-RL RL includes human fingering in its inputs, and the fingering information is also used in the reward function to force the agent to follow this fingering. Our method, marked as OT, removes the fingering from the observation space and uses OT-based finger placement to guide the agent to discover its own fingering. We also include a baseline, called No Fingering, that removes the fingering entirely. The first two columns of Fig. 2 show that our method without human-annotuan annotated fingering matches RoboPianst-Rl. performance on two different songs. Our method outperforms the baseline without human fingering by a large margin, showing that the proposed OT-based finger placement boosts the agent learning. The proposed method works well even on challenging songs. We test our method on Flight of the Bumblebee and achieve 0.79 Fl score after 3M training steps. To
5

---

Five-finger Hands
Humon
Four-finger Honds
OT
Humon
Thumb
First Finger
Middle Finger
Ring Finger
Little Finger
Figure 3: Comparison of fingering discovered by the agent itself and human annotations.
the best of our knowledge, we the first to play the challenging song Flight of the Bumblebee with general-purpose bimanual dexterous robot hands.
Analysis of the Learned Fingering We now compare the fingering discovered by the agent itself and the human annotations. Fig. 3, we visualize the sample trajectory of playing French Suite No.5 Sarabande and the corresponding fingering. We found that although the agent achieves strong performance for this song (the second plot Fig. 2), our agent discovers different fingering compared to humans. For example, for the right hand, humans mainly use the middle and ring fingers, while our agent uses the thumb and first finger. Furthermore, in some cases, human annotations are not suitable for the robot hand due to different morphologies. For example, in the second time step of Fig. 3, the human uses the first finger and ring finger. However, due to the mechanical limitation of the robot hand, it can not press keys that far apart with these two fingers, thus mimicking human fingering will miss one key. Instead, our agent discovered use the thumb and little finger, which satisfies the hardware limitation and accurately presses the target keys.
Cross Emboidments Labs usually have different robot platforms, thus having method that works for different embodiments is highly desirable. We test our method on a different embodiment. To simplify the experiment, we disable the little finger of the Shadow robot hand and obtain a four-finger robot hand, which has a similar morphology to Allegro [41] and LEAP Hand [42]. We evaluate the modified robot hand on the song French Suite No.5 Sarabande (first 550 time steps). where our method achieves 0.95 F1 score, similar to the 0.96 achieved with the original robot hands. In the bottom row of Fig. 3, we visualize the learned fingering with four-finger hands. The agent discovers different fingering compared to humans and the original hands but still accurately presses active keys, meaning our method is compatible with different embodiments.
4.3 RPIM Dataset
To facilitate the research on dexterous robot hands, we collect and release a large-scale motion dataset for piano playing. Our dataset includes 1M expert trajectories covering ~2k musical pieces. For each musical piece, we train an individual DroQ agent with the method introduced in Section 4.1 for 8 million environment steps and collect 500 expert trajectories with the trained agent. We chunk each sheet music every 550 time steps, corresponding to 27.5 seconds, that each run has the same episode length. The sheet music used for training is from the PIG dataset [6] and a subset (1788 pieces) of the GiantMIDI-Pian dataset [7].
In Fig. 4, we show the statisties of our collected motion dataset. The top plot shows the histogram of the pressed keys. We found that keys close to the center are more frequently pressed than keys the corner. Also, white keys, taking 65.7%. are more likely to be pressed than black keys. In the bottom left plot, we show the distribution of the number of active keys over all time steps. It roughly follows

---

Histogram Pressed Keys
75000
Key Color
white
50000
black
25000
Key
Num of Musical Pieces
Num of Active Keys over All Time Steps
Num of Musical Pieses
Histogram of F1 Scores
1.00
60
75
0.75
40
60
0.50
25
0.25
1000
2000
3000
4000
5000
6200
7000
1.0 1.00
Keys
Figure 4: Statistics of our RPIM dataset. (Top) Histogram of pressed keys in our RPIM dataset.
(Bottom Left) Distribution of the number of active keys over all time steps. (Bottom Right) Distribution of Fl scores in our dataset.
Gaussian distribution, and 90.70% musical pieces in our dataset include 1000-4000 active keys. We also include the distribution of Fl scores of trained agents used for collecting data. We found most agents (79.00%) achieve F1 scores larger than 0.75, and 99.89% of the agents' F1 scores are larger than 0.5. The distribution of F1 scores reflects the quality of the collected dataset. We empirically found agents with FI score 0.75 are capable of playing sheet music reasonably well with only minor errors. Agents with < 0.5 Fl scores usually have notable errors due to the difficulty of songs or the mechanical limitations of the Shadow robot hand. We also include the Fl scores for each piece in our dataset so users can filter the dataset according to their needs.
5 Benchmarking Results
The analysis in the previous section highlighted the diversity of highly dynamic piano-playing motions in the RPIM dataset. In this section, we assess the multi-task imitation learning performance of several widely used methods on our benchmark. To be specific, the objective to train a single multi-task policy capable of playing various music pieces on the piano. We train the policy on portion of the RPIM dataset and evaluate in-distribution performance (F1 scores on songs included the training data) and its generalization ability (FI scores on songs not present the training data).
Baselines We evaluated Behavior Cloning (BC) [43], Behavior Transformer (Be'T) [44], Diffusion Policy [5] with U-Net (DP-U) [45] and with Transformer (DP-T) [46]. BC directly learns a policy by using supervised learning on observation-action pairs from expert demonstrations. BeT clusters con- tinuous actions into discrete bins using k-means, allowing it to model high-dimensional, continuous, multimodal action distributions as categorical distributions [44]. Diffusion Policy learns to model the action distribution by inverting process that gradually adds noise to sampled action sequence. We evaluated both the CNN-based (U-Net) Diffusion Policy (DP-U) and the Transformer- based Diffusion Policy (DP-T) with DDPM [47]. We use the same code and hyperparameters as Chi et al. [5]. Detailed descriptions of the baselines as well as hyperparameters are given in Appendix C.
Experiment Setup We train the policies on subsets of the RPIM dataset with different sizes: 12, 25, 50, 100, 150. We then evaluate the trained policies on both i) 12 in-distribution songs: music pieces that overlap with the training sets, and ii) 20 out-of-distribution (OOD) songs: music pieces that do not overlap with the training songs. The selected songs are very challenging and contain diverse motions and long horizons. In the experiment, we report zero-shot evaluation results without fine-tuning. We report the average Fl scores of each group of music pieces for policies trained with each baseline method. We list the selected songs for evaluation in Appendix C.2.
Discussion We present the benchmarking performance of multi-task agents in Table 2. For the in in-distribution evaluation, compared to Fl scores obtained with our RL specialist agents in Fig, 4, we notice performance gap across all baselines. This gap widens as the data size increases. When
7

---

Table 2: Comparison results of multi-task imitation learning.
In-disribution
Out-of-distrihution
# music
12
25
50
100
150
12
25
50
100
150
BC-MLP
0.529
0.315
0.319
0.193
0.250
0.079
0.119
0.100
0.108
0.187
BeT
0.062
0.080
0.065
0.078
0.088
0.094
0.110
0.111
0.120
0.125
DP-U
0.539
0.541
0.546
0.505
0.454
0.181
0.189
0.198
0.215
0.256
DP-T
0.357
0.304
0.297
0.301
0.318
0.186
0.210
0.230
0.291
0.316
trained on smaller dataset with 12 training songs, DP-U performs comparably to BC-MLP and slightly outperforms DP-T, while BeT experiences significant performance drop. This decline may be attributed to hyperparameter choices, such as the number of action bins. Although we used the same number of action bins as the official implementation, the complexity of our tasks suggests that this configuration may be inadequate, and increasing the number of bins could improve performance.
As the dataset size increases, we observe that Diffusion Policy outperforms the other baselines. DP-U and DP-T show performance drops of 15. 77% and 10.92%, respectively, while BC-MLP suffers more significant decline of 52.74%. Similar performance gaps have been noted in previous work [4] and concurrent research [26] suggests a hierarchical policy structure, although it still lags behind RL specialists. This highlights the need for future research to address the performance gap between RL specialists and multi-task agents.
In the zero-shot out-of-distien distriitittitiiinnn evaluation, we find that performance improves for all evaluated baselines as the training data size increases. Specifically, the FI scores for DP-U and DP-T rise from 0.181 to 0.256 and from 0.186 to 0.316, respectively, when the number of training songs is increased from 12 to 150. This suggests that larger datasets enhance the generalization capabilities of multi-task agents. We hope that releasing our large-scale RPIM dataset will contribute to the development of robust generalist piano-playing agents within the research community.
Limitations & Conclusion
Limitations Our paper has limitations in several aspects. Firstly, although our method lifts the requirement of numan-annotade fingering, enabling RL training on diverse songs, our method still fails to achieve strong performance challenging songs due to fast rhythms and mechanical limitations of the robot hands. Improving the RL method and hardware design could help address this. Secondly, the evaluation metric, Fl score, may not adequately capture musical performance and the position-based controller missing the target velocity would hinder the performance. Thirdly, our dataset includes only proprioceptive observations, whereas humans play piano using moulil inputs like vision, touch, and hearing; incorporating these could enhance the agent's capabilities. Furthermore, there are several challenges deploying the learned agent on real-world robot. This includes the challenges of obtaining the state of the piano and the hands (e.g., tracking the precise fingertip positions), optimizing precise position controller at high speed well as the sim-to-real gap for the highly dynamic piano-playing task, etc. Lastly, although we demonstrate better zero-shot generalization performance than RoboPianist-MT [4]. there is still a gap between our best multi-task agent and RL specialists, which requires future investigation.
Conclusion In this paper, we propose large-scale motion dataset named RPIM for piano playing with bimanual dexterous robot hands. RP1M includes 1 million expert trajectories for playing 2k musical pieces. To collect such diverse dataset for piano playing, we lift the need for human- annotated fingering in the previous method by introducing novel automatic fingering annotation approach based on optimal transport. On single songs, our method matches the baselines with Human-annotater fingering and can be adopted across different embodiments. Furthermore, we benchmark various imitation learning approaches for multi-song playing. We report promising results in motion synthesis for novel music pieces when increasing the data size and identify the gap to achieving human-level paano-- ability. We believe the RPIM dataset, with its scale and quality, forms solid step towards empowering robots with human-level dexterity.
8

---

Acknowledgments
We thank the support of the Max Planck Institute for Intelligent Systems, Tübingen (Germany). We acknowledge CSC IT' Center for Science, Finland, for awarding this project access to the LUMI supercomputer, owned by the EuroHPC Joint Undertaking, hosted by CSC (Finland) and the LUMI consortium through CSC. Yi Zhao, Juho Kannala, and Joni Pajarinen acknowledge funding by the Research Council of Finland (345521 353138, 327911). We thank Yuxin Hou and Wenyan Yang for the insightful discussion, We thank all reviewers for their detailed and constructive comments.
References
[1] OpenAI, M. Andrychowicz, B. Baker, M. Chociej. R. Józefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, J. Schneider, S. Sidor, J. Tobin, P. Welinder, L. Weng. and W. Zaremba. Learning Dexterous In-Hand Manipulation. The International Journal of Robotics Research (IJRR), 2020.
[2] K. Ploeger, M. Lutter, and Peters. High acceleration reinforcement learning for real-world juggling with binary rewards. Conference on Robot Learinng pages 642-653. PMLR, 2021.
[3] D. Büicheer, S. Guist, R. Calandra, V. Berenz, B. Schölkopf, and J. Peters. Learning to play table tennis from scratch using muscular robots. IEEE Transactions Robotics, 3806)3380038 2022.
[4] K. Zakka, P. Wu, L. Smith, N. Gileadi, T. Howell, X. B. Peng. S. Singh, Y. Tassa, P. Florence, A. Zeng, et al. RoboPianist: Dexterous Piano Playing with Deep Reinforcement Rearning. In Conference on Robot Learning (CoRL), 2023.
[5] C. Chi, S. Feng, Du, Z. Xu, E. Cousineau, B. Burchfiel, and Song. Diffusion policy: Visuomotor policy learning via action diffusion. Roboties: Science and Systems (RSS), 2023.
[6] E. Nakamura, Y. Saito, and K. Yoshii. Statistical Learning and Estimation of Piano Fingering. Information Sciences, 2020.
[7] Q Kong, B. J. Chen, and Y. Wang. GiamMMDDI-Pano: A large-scale midi dataset for classical piano music. Transactions of the International Society for Music Information Retrieval, 2022.
[8] C. Villani et al. Optimal transport: Old and new, volume 338. Springer, 2009.
[9] G. Peyré, M. Cuturi, al. Computational optimal transport: With applications data science. Foundations and Trends@ in Machine Learning, 11(5-6):355-600 2011, 2019.
[10] I. Kato, S. Ohteru, K. Shirai, T. Matsushima, S. Narita, S. Sugano, T. Kobayashi, and E. Fujisawa. The Robot Musician 'wabot-2' waabot-2"wwaseda robot-2). Robotics, 1987.
[11] J.-C. Lin, H.-H. Huang, Y.-F. Li, Tai, and L.-W. Liu. Electronic Piano Playing Robot. In International Symposium on Computer, Communication, Control and Automation (3CA), 2010.
[12] A. Topper, T. Maloney, S. Barton, and X. Kong. Piano-Playing Robotic Arm. Worcester MA, 2019.
[13] J. Hughes, P. Maiolino, and F. lida. An Anthropomorphic Soft Skeleton Hand Exploiting Conditional Models for Piano Playing. Science Robotics, 2018.
[14] R. Castro Ornelas. Robotic Finger Hardware and Controls Design for Dynamic Piano Playing, 2022.
[15] D. Zhang, J. Lei, B. D. Lau, and C. Cameron. Design and Analysis of Piano Playing Robot. In International Conference Information and Automation, 2009.

---

[16] Y.-F. Li and L.-L. Chuang. Controller Design for Music Playing Robot Robot-Applie to the Anthropomorphic Piano Robot. In International Conference on Power Electronics and Drive Systems (PEDS), 2013.
[17] A. Zhang, M. Malhotra, and Y. Matsuoka. Musical Piano Performance by the ACT Hand. In International Conference Robotics and Automation (ICRA), 2011.
[18] B. Scholz. Playing Piano with a Shadow Dexterous Hand, 2019.
[19] H. Xu, Y. Luo, S. Wang, T. Darrell, and Calandra. Towards Learning Play Piano with Dexterous Hands and Touch. In International Conference Intelligent Robots and Systems (IROS), 2022.
[20] Y. Yang, Z. Wang. D. Xing, and P. Wang. Learning playing piano with bionic-constrained diffusion policy for anthropomorphic hand. Cyborg and Bionic Systems, 5:0104, 2024.
[21] P. Ramoneda, D. Jeong, E. Nakamura, X. Serra, and M. Miron. Automatic Piano Fingering from Partially Annotated Scores using Autoregressive Neural Networks. In International Conference on Multimedia, 2022.
[22] D. A. Randolph, B. Di Fugenio, and Badgerow. Modeling Piano Fingering Decisions with Conditional Random Fields. 2023.
[23] A. Moryossef, Y. Elazar, and Y. Goldberg. At Your Fingertips: Extracting Piano Fingering Instructions from Videos. arXiv preprint arXiv:2303. 03745, 2023.
[24] Ramoneda, M. Miron, and Serra. Piano Fingering with Reinforcement Learning. arXiv preprint arXiv:2 2021.
[25] W. Shi, Y. Li, Y. Guan, X. Chen, S. Yang, and S. Mo. Optimized Fingering Planning for Automatic Piano Playing Using Dual-Arm Robot System. In International Conference on Robotics and Biomimetics (ROBIO), 2022.
[26] C. Qian, J. Urain, K. Zakka, and J. Peters. Pianomime: Learning generalist, dexterous piano player from internet demonstrations. arXiv preprint arXiv:2407.18/77, 2024.
[27] M. Liu, Z. Pan, K. Xu, K. Ganguly, and D. Manocha. Deep differentiable grasp planner for high-dof grippers. Robotics: Science and Systems, 2020.
[28] R. Wang. J. Zhang, J. Chen, Y. Xu, P. Li, T. Liu, and H. Wang. DexGraspNet: A large- scale robotic dexterous grasp dataset for general objects based on simulation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 1 11359-11366. IEEE, 2023.
[29] L. C. Murrilo, N. Khargonkar, B. Prabhakaran, and Y. Xiang. MultiGripperGrasp: A dataset for robotic grasping from parallel jaw grippers dexterous hands. arXiv preprint arXiv:2403.09841 2024.
[30] Y. Xu, W. Wan, J. Zhang, H. Liu, Z. Shan, H. Shen, R. Wang, H. Geng, Y. Weng, J. Chen, al. Dnasex Universal Robotic Dexterous Grasping via Learning Diverse Proposal Generation and Coal-Conditioned Policy. In Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[31] Y. Liu, Y. Yang, Y. Wang. X. Wu, J. Wang, Y. Yao, S. Schwertieger, S. Yang, W. Wang, J. Yu, et al. RealDex: Towards human-like grasping for robotic dexterous hand. arXiv preprint arXiv:2402. 13853, 2024.
[32] J. Fu. A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4RL: Datasets for deep data-driven reinforcement learning, arXiv preprint 2xXv:2004.007299 2020.
10

---

[33] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn. Learning fine- fine-grained bimanual manipulation with low-cost hardware. Robotics: Science and Systems, 2023.
[34] Y. Chen, Geng, F. Zhong. Ji, Jiang, Z. Lu, H. Dong, and Y. Yang. Bi-DexHands: Towards human-level bimanual dexterous manipulation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[35] Z. Fan, O. Taheri, D. Tzionas, M. Kocabas, M. Kaufmann, M. J. Black, and O. Hilliges. Aretie: dataset for dexterous bimanual hand-ohject manipulation. In Proceedings of the JEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 12943-12954, 2023.
[36] ShadowRobot.
ShadowRobot Dexterous Hand.
httts::/www.ghaddorobot.t com/
productss/dexterros--taaa 2005.
[37] D. F. Crouse, On implementing 2d rectangular assignment algorithms. JEEE Transactions on Aerospace and Electronic Systems, 52(4):1679-1699 2016.
[38] Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski,
P. Peterson, W. Weckesser, J. Bright, et al. Scipy 1.0: fundamental algorithms for scientific computing in python. Nature methods, 17(3):611-27, 2020.
[39] Y. Tassa, Y. Doron, A. Muldal, T. Erez, Li, D. d. L. Casas, D. Budden, A. Abdolmaleki, Merel, A. Lefrancq, et al. Deepmind control suite. arXiv preprint 1rXiv:1801. 18000,0 2018.
[40] T. Hiraoka, T. Imagawa, T. Hashimoto, T. Onishi, and Y. Tsuruoka. Dropout Q-functions for doubly efficient reinforcement learning. International Conference on Learning Representations, 2022.
[41]
hand.
[42] K. Shaw, A. Agarwal, and D. Pathak. Leap hand: Low-cost, efficient, and anthropomorphic hand for robot learning. Robotics: Science and Systems, 2023.
[43] D. A. Pomerleau. Alvinn: An autonomous land vehicle in a neural network. Advances in neural information processing systems, 1, 1988.
[44] N. M. Shafiullah, Cui, A. A. Altanzaya, and L. Pinto. Behavior transformers: Cloning modes with stone. Advances in neural information processing systems, 35:22955-22968, 2022.
[45] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted assisted intervention -iceettiio-MCCAALL 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part 18, pages 234 241. Springer, 2015.
[46] A. Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
[47] Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. International Conference on Learning Representations, 2021.
[48] D. Rus. In-Hand Dexterous Manipulation of Piecewise-Smoolle 3-D Objects. The International Journal of Robotics Research (IJRR), 1999.
[49] A. Bicchi and R. Sorrentino. Dexterous Manipulation through Rolling. International Conference on Robotics and Automation (ICRA), 1995.
[50] L. Han and J. C. Trinkle. Dextrous Manipulation by Rolling and Finger Gaiting. In nternational Conference Robotics and Automation (ICRA), 1998.

---

[51] Y. Bai and C. K. Liu. Dexterous Manipulation using both Palm and Fingers, In International Conference on Robotics and Automation (ICRA), 2014.
[52] I. Mordatch, Z. Popović, and E. Todorov. Contact-Invariant Optimization for Hand Manipulation. In Proceedings of the ACM SIGGRAPt//Eumgraphiss Symposium on Computer Animation, 2012.
[53] N. C. Dafle, A. Rodriguez, R. Paolini, B. Tang, S. S. Srinivasa, M. Erdmann, M. T. Mason, Lundberg, Staab, and T. Fuhlbrigge. Extrinsic Dexterity: In-Hand Manipulation with External Forces. In International Conference on Robotics and Automation (ICRA), 2014.
[54] N. Chavan-Dafle and A. Rodriguez. Sampling-Be Seapling-BBsed Planning of In-Hand Manipulation with External Pushes. In Robotics Research: The 18th International Symposium ISRR, 2020.
[55] V. Kumar, Y. Tassa, T. Erez, and E. Todorov. Real-Time Behaviour Synthesis for Dynamic Hand-Manipulation. In International Conference on Robolics and Automation (ICRA), 2014.
[56] M. Li, Y. Bekiroglu, D. Kragic, and A. Billard. Learning of Grasp Adaptation through Expe- rience and Tactile Sensing. In International Conference on Intelligent Robots and Systems, 2014.
[57] J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning Quadrupedal Locomo- tion over Challenging Terrain. Science robotics, 2020.
[58] I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, et al. Solving Rubik's Cube with Robot Hand. arXiv preprint arXiv:1910.07113. 2019.
[59] X. B. Peng, P. Abbeel, S. Levine, and M. Van de Panne. DeepMimic: Example-Guie Guided Deep Reinforcement Learning of Physics- Based Character Skills. ACM Transactions On Graphics (TOG), 2018.
[60] V. Kumar, A. Gupta, E. Todorov, and S. Levine. Learning Dexterous Manipulation Policies from Experience and Imitation. arXiv preprint arXiv: wrXiv:6611.05095 2016.
[61] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine. Learn- ing Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations. Robotics: Science and Systems, 2018
[62] I. Radosavovic, X. Wang, L. Pinto, and Malik. State-Only Imitation Learning for Dexterous Manipulation. In International Conference on Intelligett Robots and Systems (IROS), 2021.
[63] R. Jeong, T. Springenberg. J. Kay, Zheng, Y. Zhou, Galashov, N. Heess, and F. Nori. Learning Dexterous Manipulation from Suboptimal Experts, arXiv preprint arXiv22010.08587, 2020.
[64] Lin, Y. Zhang, Li, two multifingered hands. arXiv preprint arXiv:2404.1682,, 2024.
Li, H. Qi, Yi. S. Levine, and Malik. Learning visuotactile skills with
[65] T. Chen, J. Xu, and P. Agrawal. A System for General In-hand Object Re-Orientation. In Conference Robot Learning (CoRL), 2022.
[66] Z. Yang, K. Yin, and L. Liu. Learning Use Chopsticks in Diverse Gripping Styles. 2022.
[67] Y. Chen, T. Wu, S. Wang, X. Feng, Jiang, Z. Lu, S. McAleer, H. Dong. S.-C. Zhu, and Y. Yang. Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learning. Ad- vances in Neural Information Processing Systems (NeurIPS), 2022.
12

---

[68] A. Allshire, M. Mittal, V. Lodaya, V. Makoviychuk, D. Makoviichuk, F. Widmaier, M. Wüthrich, S. Bauer, A. Handa, and A. Garg. Transferring Dexterous Manipulation from GPU Simulation to a Remote Real- World Trifinger. In International Conference on Intelligent Robots and Systems (IROS), 2022.
[69] Y. Qin, B. Huang, Z.-H. Yin, H. Su, and X. Wang. DexPoint: Generalizable Point Cloud Reinforcement Learning for Sim-to-Real Dexterous Manipulation. In Conference on Robot Learning (CoRI.), 2023.
[70] T. Lin, Z.-H. Yin, H. Qi, P. Abbeel, and J. Malik. Twisting lids off with two hands. arXiv preprint arXiv:2403.02333, 2024.
[71] S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y. Sulsky, J. Kay, J. T. Springenberg, et al. generalist agent. Transactions on Machine Learning Research (TMLR), 2022.
[72] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, et al. RT-1: Robotics Transformer for Real-World Control at Scale. Robotics: Science and Systems (RSS). 2023.
[73] H. Bharadhwaj. J. Vakil, M. Sharma, A. Gupta, S. Tulsiani, and V. Kumar. Roboagent: Generalization and efficiency in robot manipulation via semantic augmentations and action chunking. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 4788-4795. IEEE, 2024.
[74] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, et al. RT-2: Vision-aanguage-Aciin Models Transfer Web Knowledge to Robotic Control. Conference on Robot Learning (CoRI.), 2023.
[75] H. Ha, P. Florence, and S. Song. Scaling and distilling down: Language-guided robot skill acquisition. In Conference on Robot Learning (CoRL), pages 3766-3777. PMLR, 2023.
[76] Y. Du, S. Yang, B. Dai, H. Dai, O. Nachum, J. Tenenbaum, D. Schuurmans, and P. Abbeel. Learning universal policies via text-guided video generation. Advances in Neural Information Processing Systems (NeurIPS), 36, 2024.
[77] R. Bonatti, S. Vemprala, S. Ma, F. Frujeri. S. Chen, and A. Kapoor. Pact: Perception-action causal transformer for autoregressive robotics pre-training. in 2023 ieee. In RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3621-3627, 2022.
[78] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Conference on Computer Vision and Patlern Recognition (CVPR), 2022.
[79] J. Ho, T. Salimans, A. Gritsenko, W. Chan, M. Norouzi, and J. Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633-8666 8646, 2022.
[80] J. Ho, W. Chan, C. Saharia, J. Whang, R. Gao, A. Gritsenko, D. P. Kingma, B. Poole, M. Norouzi, D. J. Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint 1rXiv:2210.02303, 2022.
[81] B. Poole, A. Jain, J. T. Barron, and B. Mildenhall. DreamFusion: Text-to-3d using 2d diffusion. International Conference on Learning Representations, 2023.
[82] Z. Liu, Y. Feng, M. J. Black, D. Nowrouzezahrai, L. Paull, and W. Liu. MeshDiffusion: Score- based generative 3d mesh modeling. International Conference on Learning Representations, 2023.
[83] M. Reuss, M. Li. X. Jia, and R. Lioutikov. Goal-conditioned imitation learning using score-based diffusion policies. Robotics: Science and Systems (RSS), 2023,
13

---

[84] O. M. Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, T. Kreiman, C. Xu, et al. Octo: An open-source generalist robot policy. Robotics: Science and Systems (RSS), 2024.
[85] X. Huang, Y. Chi, R. Wang. Z. Li, X. B. Peng. S. Shao, B. Nikolic, and K. Sreenath. Diffuseloco: Real-time legged locomotion control with diffusion from offline datasets. Conference on Robot Learning (CoRL), 2024.
14

---

Appendix
A More Related Work
Dexterous Robot Hands The research of dexterous robot hands aims replicate the dexterity of human hands with robots. Many previous works [48, 49, 50, 51, 52, 53, 54, 55] use planning to compute trajectory followed by controller, thus require accurate model of the robot hand. Closed-loop approaches have been developed by incorporating sensor feedback [56]. These methods also require an accurate model the robot hand, which can be difficult to obtain in practice, especially considering the large number of active contacts between the hand and objects.
Due to the difficulty of actually modeling the dynamies of the dexterous robot hand, recent meth- ods resort to learning-basse
approaches, especially RL, which has achieved huge success in both robotics [57, 58, 1] and computer graphics [59]. To ease the training of dexterous robot hands with large number of degrees of freedom (DoFs), demonstrations are commonly used [60, 61, 62, 63, 64]. Due to the advance of both RL algorithms and simulation, recent work shows impressive results on dexterous hand manipulation tasks without human demonstrations. Furthermore, the policy trained in the simulator can further be deployed on real dexterous robot hands via sim-to-real transfer [65, 66, 67, 30, 68, 69, 70].
Generalist Agents RL methods usually perform well on single tasks, however, as human beings, we can perform multiple tasks. Generalist agents are proposed to master a diverse set of tasks with a single agent (71, 57, 72, 73J. These methods typically resort to scalable models and large datasets [72, 74, 75, 76, 77). Recently, diffusion models have achieved many state-of-the-are results across image, video, and 3D content generation [78, 79, 80, 81, 82] In the context of roboties, diffusion models have been used policy networks for imitation learning in both manipulation [5, 75, 83, 84] and locomotion tasks [85]. The same technique has also been investigated in multi-task learning [75, 84). We investigate the application of diffusion policy in high-dimensional control tasks, that is, playing piano with bimanual dexterous robot hands.
B RPIM Dataset Collection Details
B.1 Reward Formulation
In Eq. (3), we give the overall reward function used in our paper. We now give details of each term. Press indicates whether the active keys correctly pressed and inactive keys are not pressed. We rf
use the same implementation as [4], given as: Press 0.5 (k ΣK o(Ibk 1[]2)) + 0.5 K is the number of active keys, kj is the normalized key states with range [O, 1]. where 0 means the i-th key is not pressed and means the key is pressed. g is tolerance from Tassa et al. [39], which is similar to the one used in Equation (2). 1p indicates whether the inactive keys are pressed, which encourages the agent to avoid pressing keys that should not be pressed. r² Sustaan encourages the agent to press the pseudo sustain pedal at the right time, given as r Sustain = g( .urget). 8 and target are the state of current and target sustain pedal respectively. r Collision penalizes the agent from collision, defined as rt ,,Collision- = 1 collision where Lcollision is 1 if collision happens and 0 otherwise. Fnergy Energy prioritizes energy-saving behavior. It is defined as T. |ijints [T |jjintss// Tjoints and Vjoints are joint torques and joint velocities respectively.
1p))
B.2 Training Details
Observation Space Our 1444-dimessional observation space includes the proprioceptive state of dexterous robot hands and the piano as well as L-step goal states obtained from the MIDI file. In our case, we include the current goal and 10-step future goals in the observation space (I==11). At each time step, an 89-dimensional binary vector used to represent the goal, where 88 dimensions are for key states and the last dimension is for the sustain pedal. The dimension of each component in the observation space is given in Table 3.
15

---

Table 3: Observation space.
Observations
Dim
Piano goal state
L. 88
Sustain goal state
L
Piano key joints
88
Piano sustain state
Fingertip position
3 10
Hand state
Training Algorithm & Hyperparameters Although our proposed method is compatible with any reinforcement learning method, we choose the DroQ [40] as Zakka et al. [4] for fair comparison. DroQ is a model-free RL method, which uses Dropout and Layer normalization in the Q function to improve sample efficiency. We list the main hyperparameters used in our RL training in Table 4.
Table 4: Hyperparameters used in our RL agent.
Hyperparameter
Value
Training steps
8M
Episode length
550
Action repeat
Warm-up steps
5k
Buffer size
1M
Batch size
256
Update interval
2
Piano environment
Lookahead steps
10
Gravity compensation
True
Control timestep
0.05
Stretch factor
1.25
Trim slience
True
Agent
MLPs
[256, 256, 256]
Num. Q
2
Activation
GeLU
Dropout Rate
0.01
EMA momentum
0.05
Discount factor
0.88
Learnable temperature
True
Optimization
Optimizer
Adam
Learning rate
3e-4
81
0.9
82
0.999
eps
1e-8
16

---

B.3
Computational Resources
We train our RL agents on the LUMI cluster equipped with AMD MI250X GPÜs, 64 cores AMD EPYC "Trento" CPUs, and 64 GBs DDR4 memory. Each agent takes 21 hours to train. The overall data collection cost is roughly 21 hours 2089 agents 43,869 GPU hours.
B.4 MuJoCo XLA Implementation
To speed up training, we re-implement the RoboPianist environment with MuJoCo XLA (MJX), which supports simulation in parallel with GPUs. MJX has a slow performance with complex scenes with many contacts. To improve the simulation performance, we made the following modifications:
We disable most of the contacts but only keep the contacts between fingers and piano keys as well as the contact between forearms.
Primitive contact types are used whenever possible.
The dimensionality of the contact space is 3.
The maximal contact points are set to 20.
We use Newton solver with iterations=2 and Is iterations=6.
After the above modifications, with 1024 parallel environments, the total steps per second is 159,376.
We use PPO implementation implemented with Jax to fully utilize the paralleled simulation. The PPO with MJX implementation is much faster than the DroQ implementation, which only takes hours and 7 minutes for 40M environment steps on the Twinkle Twinkle Little Star song while as a comparison, DroQ needs roughly 21 hours for 8M environment steps. However, the PPO implementation fails to achieve a comparable Fl score as the DroQ implementation as shown in Fig. Therefore, we use the DroQ implement with the CPU version of the RoboPianist environment.
Twinkle Twinkle Little Star
1.00
0.75
F1 Score
0.50
0.25
DroQ
PPO+MJX
0.00
20
40
Environment Steps (1e6)
Figure 5: Comparison of the RL performance between DroQ and PPO with the MJX implementation of the RoboPianist environment. PPO+MJX is faster to run but has worse performance than DroQ. We use DroQ with the CPU-version RoboPianist environment when training our RL agents.
C Multitask Benchmarking Details
single multi-task policy capable of playing various songs is highly desirable. However, playing different music pieces on the piano results in diverse behaviors, creating a complex action distribution, particularly for dexterous robot hands with large number of degrees of freedom (DoFs). This section introduces the baseline methods we have compared and the hyperparameters we have used. We also talk about the details of our multitask training and evaluation.
17

---

C.1
Baselines and Hyperparameters
C.1.1 Behavior Cloning
Behavior Cloning (BC) [43] directly learns a policy using supervised learning on observation-action pairs from expert demonstrations, one of the simplest methods to acquire robotic skills. Due to its straightforward approach and proven efficacy, BC is popular across multiple fields. The method employs MultiiLayer Perceptron (MLP) as the policy network. Given expert trajectories, the policy network learns to replicate expert behavior by minimizing the Mean Squared Error (MSE) between predicted and actual expert actions. Despite its advantages, BC tends to perform poorly in generalizing to unseen states from the expert demonstrations. The MLP we used features three hidden layers, each with 512 units, followed by Layer Normalization and an Exponential Linear Unit (ELU) activation function to stabilize training and introduce non-linearity.
Table 5: Hypermeters used in BC
Hyperparameter
Value
Batch Size
1024
Optimizer
Adam
Learning Rate
le-4
Observation Horizon
Prediction Horizon
Action Horizon
C.1.2 BeT
Behavior Transformers (BeT) [44] uses a transformer-decoder based backbone with discrete action mode predictor coupled with continuous action offset corrector model continuous actions sequences. It clusters continuous actions into discrete bins using k-means to model high-dimensional, continuous multi-modal action distributions as categorical distributions without learning complicated generative models. We adopted the implementation and hyperparameters from the Diffusion Policy codebase [5].
Table 6: Hyperparamerters used in BeT
Hyperparameter
Value
Batch Size
512
Optimizer
AdamW W
Learning Rate
le-4
Num of bins
64
MinGPT n_layer
MinGPT h_ae
8
MinGPT embd
120
Observation Horizon
Prediction Horizon
Action Horizon
C.1.3 Diffusion Policy
Diffusion models have achieved many state-of-the-art results across image, video, and 3D content generation [78, 79, 80, 81, 82]. In the context of robotics, diffusion models have been used as policy networks for imitation learning in both manipulation [5, 75, 83, 84] and locomotion tasks [85],
showing remarkable performance across various robotic tasks. Diffusion Policy [5] proposed to learn
18

---

an imitation learning policy with conditional diffusion model. It models the action distribution by inverting process that gradually adds noise to a sampled action sequence, conditioning on a state and a sampled noise vector. We evaluated both the U-Net-base Diffusion Policy (DP-U) and the transformer-based Diffusion Policy (DP-T). We build our diffusion policy training pipeline based on the original Diffusion Policy [5] codebase, which provides high-quality implementations.
Table Hyperparameters used in DP-U
Hyperparameter
Value
Batch Size
1024
Optimizer
Adam W
Learning Rate
le-4
Weight Decay
1e-6
Diffusion Method
DDPM
Number of Diffusion Iterations
100
EMA Power
0.75
U-Net Hidden Layer Sizes
[256, 512, 1024]
Diffusion Step Embedding Dim.
256
Observation Horizon
1
Prediction Horizon
4
Action Horizon
Table 8: Hyperparameters used in DP-T
Hyperparameter
Value
Batch Size
1024
Optimizer
AdamW
Learning Rate
1e-3
Weight Decay
le-4
Diffusion Method
DDPM
EMA Power
0.75
layer
n head
n_emb p_drop_emi
156
0.0
p.drop attn
0.3
Observation Horizon
Prediction Horizon
Action Horizon
C.2 Training and Evaluation
We train the policies with 5 different sizes of expert data: 12, 50, 100, and 150 songs, respectively. Subsequently, we assess the trained policies using two distinct categories of musical pieces. The first category, in-distribution songs, includes pieces that are part of the training datasets. Evaluating with mn-distribttion songs tests the multitasking abilities of the policies and checks if a policy can accurately recall the songs on which it was trained. The second group of songs for evaluation are out-of-distribution songs: those music pieces do overlap with the training songs. The selected songs contain diverse motions and long horizons, making them challenging to play. This out-of- distribution evaluation measures the zero-shot generalization capabilities of the policies. Analogous
19

---

to an experienced human pianist who can play new pieces at first sight, we aim to determine if is feasible to develop a generalist agent capable of playing the piano under various conditions.
Additionally, our framework is designed with flexibility mind, allowing users to select songs not included in our dataset for either training data collection evaluation. Furthermore, users have the option to assess their policies specific segments of song rather than the entire piece.
Table 9: In-distribution songs
RobePPanissccuueee---PPaiiaalal
Table 10: Out-of-distributtionnnn songs
GianhilDDKKeeerrraaPPtttoo-000
20


Reviews:
[{"review_id": "datBmaoimU", "reviewer": "Reviewer_iGED", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors propose a method for automatically generating fingering charts for piano music, by posing the problem of mapping fingers to piano keys as an \"optimal transport\" problem that can be solved with an off-the-shelf solver. They show that giving these fingering charts to RL agents and adding an auxiliary reward for following the fingering sequence greatly improves the agents' ability to learn to play a song. The authors present a new dataset, RP1M, which is the result of training a specialist agent for each of 2k songs, and rolling out 500 trajectories from each agent. The authors demonstrate that this approach works for 4-finger hands, and that the data can be used to train piano-playing agents using various imitation-learning methods. The environment and evaluation metric are based on the benchmark presented by (Zakka et. al., 2023).", "summary_of_recommendation": "Because of the limitations of the benchmark itself, I don't think this work will have impact on real robot piano playing. That said, it is a big improvement on the existing benchmark, and we should accept it on the merits of its robotics work.", "technical_quality": 3, "clarity_of_presentation": 5, "potential_impact": 2, "robotics_focus": 2, "quality_of_limitations": 2, "questions_for_rebuttal": "- Address the downsides of the benchmark in the limitations section.\n- Compare the proposed fingering generation method to others in the literature.\n- Check whether the generated fingering is followed by the RL specialists.\n- Please address the minor comments listed above.\n\nRequest to area chair: Maybe reach out to reviewers outside of robotics, who wrote about piano fingering generation.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5HWsNbPAQb", "reviewer": "Reviewer_cZsS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents a new dataset for piano playing. The dataset consists of 2k songs. The paper makes data collection a bit cheaper by using Optimal Transport to assign finger to keys such as the overall finger movement is minimized.", "summary_of_recommendation": "I am recommending to accept the paper mainly because I like the dataset and it is an unexplored problem.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "Please confirm if the dataset would be released in public.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xQN0NlzpjD", "reviewer": "Reviewer_qnEm", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work introduces the RP1M dataset, which provides extensive bi-manual piano playing data with over one million trajectories. By matching finger positions, the dataset enables the automatic annotation of songs, significantly reducing the need for human-labeled data and making it feasible to scale imitation learning.", "summary_of_recommendation": "Overall, this work is a good contribution to the community. However, due to the missing baseline and the potential challenge of the position-based controller, I will start with a weak reject and will adjust my rating based on the authors' response.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "Same as the prior question section", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BbDWOnhiLQ", "reviewer": "Reviewer_3sJW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work proposes a bi-manual, dexterous, piano-playing dataset with 1M demonstrations. The dataset is diverse by forgoing human annotations and utilizing an optimal transport reward for training the RL agent. The work is competitive with human-annotated methods for individual songs and provides preliminary results for a multi-task piano-playing policy.", "summary_of_recommendation": "Overall, this dataset will be helpful to the community and the paper is both clear and has sufficient experimental support. However, the paper would benefit greatly from additional ablations and deployment on real world hands as that would clearly show the usefulness of this simulation dataset.", "technical_quality": 4, "clarity_of_presentation": 5, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 2, "questions_for_rebuttal": "-Why can [3] not be considered the “first large-scale dataset of dynamic, bi-manual manipulation with dexterous robot hands”? It seems like this is a strong claim and I would welcome more elaboration for this as it is a strong statement.\n\n-In the related works, it would be helpful to further distinguish from [1] as they also train an RL policy for each song but then distill the policies to create a generalist policy rather than creating a dataset to train a single policy.\n\n-What additional metrics can be included to characterize the dataset?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p14JYRzL1R", "reviewer": "Reviewer_EzFs", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper introduces the Robot Piano 1 Million (RP1M) dataset, which contains bi-manual robot piano-playing motion data for 2k sounds. Considering that the human annotation required in the RoboPianist is hard to collect, it proposes an Optimal-transport-based reward when training the RL specialist without human annotated fingering. To collect the motion dataset, it first trains RL agents for each song and then uses the trained RL agents to collect datasets. It further benchmarks Imitation Learning methods based on the collected dataset.", "summary_of_recommendation": "The paper focuses on a specific task domain and generates a valuable dataset for that domain.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "- How to measure the generated data quality and what can be a good data filtering mechanism?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2tmJRiYkDY", "reviewer": "Reviewer_Sw5X", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper uses dexterous robot hands and introduces a new dataset, called the Robot Piano 1 Million (RP1M) dataset, containing over a million bi-manual piano playing trajectories. This dataset aims to improve performance of robots in the task of piano playing using imitation learning, facilitated by framing finger placements as an optimal transport problem.  The main contributions of the paper include the introduction of a novel dataset for an interesting, complex and previously under-researched task of piano playing, formulating fingering as an optimal transport problem that enables the authors to collect diverse large scale data using RL.", "summary_of_recommendation": "I feel the paper is a step in the right direction towards improving the robotics manipulation with dexterous arms and provides an important benchmark for the robotics community. The lack of exhaustive testing of prior approaches and real-world evaluation could be circumvented by the authors and could improve the overall quality of the paper.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "I would want the authors to consider adding more exhaustive baselines to the evaluation for the reader to get an idea about how the diverse set of approaches available work on the proposed benchmark. It would also be useful to see some real-world transfer experiments on this challenging task of robot piano playing.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5VE8X2ajdJ", "reviewer": "Reviewer_Kzd5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper addresses a big limitation of the recently introduced RoboPianist benchmark: the need for fingering labels to train a piano playing policy with RL. They solve for fingering labels by formulating it as an optimal transport problem. Using this technique, they label 2000 MIDI files and demonstrate that they can effectively train specialist policies per MIDI file to generate a dataset of 1M piano playing motions. Finally, they distill this data into a multitask policy using diffusion policy and compare with other BC baselines.", "summary_of_recommendation": "Overall a good paper that is well written and provides an elegant solution to the piano fingering problem. As it stands, I recommend accepting this paper, but I have some concerns regarding the BC results / dataset. I look forward to the authors clarifying my concerns.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "- It would be good to get some more analysis about the distribution of failures exhibited by the 2k specialist policies. Some questions that popped to mind:\n  - Are there any instances where the actuator gains or energy penalty were limiting the policy from playing fast-paced songs?\n  - Relatedly, how did you decide what DoFs to remove from the wrist? And if anything, why not add more DoFs like roll/pitch/yaw to the forearm?\n  - You mention failures from mechanical limitations of the shadow hand. Why not filter out those MIDI files from the dataset to disentangle hardware from policy learning?\n- I noticed you used a stretch factor 1.25 (mentioned in the appendix). Did you slow down the songs for a specific reason, maybe because it improved overall performance? If so, is this a fair comparison with the RoboPianist curves?\n- What software suite did you use for the BC experiments? Were all the baselines implemented using robomimic/diffusion policy or just diffusion policy used Chi et al's codebase?\n- It would be nice to get a bit more details regarding how you formulated the optimal transport problem. Was any tweaking done to get it to work? I can imagine one could speedup the solver by ignoring keys on the far right/far left of the piano, or basically only considering keys that are within a horizon of time into the future? Also did you consider another fingering algorithm baseline, like the ones mentioned in the related works section?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: # strengths 
- The strength of the paper has been well recognized by all reviewers. 
- Generalization ability of learned agents to beyond individual pieces has also been well recognized.
- The novelty of finger placements as an optimal transport problem has been highlighted
- Evaluations on cross-embodiment has been welcomed by multiple reviewers

# weakness 
- There are concerns raised on the evaluation metrics and their relevance to realistic piano playing. 
- Details missing on baselines and ablation. They additionally are in need of further clarifications 

# recommendation 
Reviewers have provided an overwhelming diversity of suggestions for improvements. Authors are encouraged to improve the submission based on the provided suggestions.

# rebuttal summary 
The authors provided comprehensive responses to the reviewer's feedback. The authors also proposed to add clarifications, improve on baselines, and make the datasets public. Post these clarifications, reviews adjusted their scores. All reviewers are aligned on their recommendation for the paper. Based on their suggestions, AC recommends to accept the paper.
Decision: Accept