RP1M: A Large-Scale Motion Dataset for Piano Playing with Bi-Manual Dexterous Robot Hands

Authors: yi  zhao
  le chen
jan  schneider
quankai  gao
juho  kannala
bernhard  schölkopf
joni  pajarinen
dieter b üchler
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 13:55:48

Abstract: Endowing robot hands with human-level dexterity is a long-lasting research objective. Bi-manual robot piano playing constitutes a task that combines challenges from dynamic tasks, such as generating fast while precise motions, with slower but contact-rich manipulation problems. Although reinforcement learning based approaches have shown promising results in single-task performance, these methods struggle in a multi-song setting. Our work aims to close this gap and, thereby, enable imitation learning approaches for robot piano playing at scale. To this end, we introduce the Robot Piano 1 Million (RP1M) dataset, containing bi-manual robot piano playing motion data of more than one million trajectories. We formulate finger placements as an optimal transport problem, thus, enabling automatic annotation of vast amounts of unlabeled songs. Benchmarking existing imitation learning approaches shows that such approaches reach state-of-the-art robot piano playing performance by leveraging RP1M.

Main paper: 
 introduction: 
 Empowering robots with human-level dexterity is notoriously challenging. Current robotics research on hand and arm motions focuses on manipulation and dynamic athletic tasks. Manipulation, such as grasping or reorienting [1] , requires continuous application of acceptable forces at moderate speeds to various objects with distinct shapes and weight distributions. Environmental changes, like humidity or temperature, alter the already complex contact dynamics, which adds to the complexity of manipulation tasks. Dynamic tasks, like juggling [2] and table tennis [3] involve making and breaking contact, demanding high precision and tolerating less inaccuracy due to rarer contacts. High speeds in these tasks necessitate greater accelerations and introduce a precision-speed tradeoff.Robot piano playing combines various aspects of dynamic and manipulation tasks: the agent is required to coordinate multiple fingers to precisely press keys for arbitrary songs, which is a highdimensional and rich control task. At the same time, the finger motions have to be highly dynamic, especially for songs with fast rhythms. Well-practiced pianists can play arbitrary songs, but this level of generalization is extremely challenging for robots. In this work, we build the foundation to develop methods capable of achieving human-level bi-manual dexterity at the intersection of manipulation and dynamic tasks, while reaching such generalization capabilities in multi-task environments.While reinforcement learning (RL) is a promising direction, traditional RL approaches often struggle to achieve excellent performance in multi-task settings [4] . The advent of scalable imitation learning techniques [5] enables representing complex and multi-modal distributions. Such large models Figure 1 : Overview of RP1M. (Left) RP1M is a large-scale motion dataset for piano playing with bi-manual dexterous robot hands. The dataset includes ∼1M expert trajectories collected by ∼2k RL specialist agents. (Right) To collect a diverse motion dataset of playing sheet music available on the Internet, we lift the requirement of human-annotated fingering by formulating the finger placement as an optimal transport problem such that the robot hands play piano in an energy-efficient way.are most effective when trained on massive datasets that combine the state evolution with the corresponding action trajectories. So far, creating large datasets for robot piano play is problematic due to the time-consuming fingering annotations. Fingering annotations map which finger is supposed to press a particular piano key at each time step. With fingering information, the reward is less sparse, making the training significantly more effective. These labels usually require expert human annotators [6] , preventing the agent from leveraging the large amounts of unlabeled music pieces on the internet [7] . Besides, human-labeled fingering may be infeasible for robots with morphologies different from human hands, such as different numbers of fingers or distinct hand dimensions.In this paper, we propose the Robot Piano 1 Million dataset (RP1M). This dataset comprises the motion data of high-quality bi-manual robot piano play. In particular, we train RL agents for each of the 2k songs and roll out each policy 500 times with different random seeds. To enable the generation of RP1M, we introduce a method to learn the fingering automatically by formulating finger placement as an optimal transport (OT) problem [8, 9] . Intuitively, the fingers are placed in a way such that the correct keys are pressed while the overall moving distance of the fingers is minimized. Agents trained using our automatic fingering match the performance of agents trained with human-annotated fingering labels. Besides, our method is easy to implement with almost no extra training time. The automatic fingering also allows learning piano playing with different embodiments, such as robots with four fingers only. With RP1M, it is now possible to train and test imitation learning approaches at scale. We benchmark various behavior cloning approaches and find that using RP1M, existing methods perform better in terms of generalization capabilities in multi-song piano play. This work contributes in various ways:• To facilitate the research on dexterous robot hands, we release RP1M, a dataset of piano playing motions that includes more than 2k music pieces with expert trajectories generated by our agents. • We formulate fingering as an optimal transport problem, enabling the generation of vast amounts of robot piano data with RL, as well as allowing variations in the embodiment. • Using RP1M, we benchmark various approaches to robot piano playing, whereby existing imitation learning approaches reach promising results in motion synthesis on novel music pieces due to scaling with RP1M. 
related work: 
 Piano Playing with Robots Piano playing with robot hands has been investigated for decades. It is a challenging task since bimanual robot hands should precisely press the right keys at the right time, especially considering its high-dimensional action space. Previous methods require specific robot designs [10, 11, 12, 13, 14, 15] or trajectory pre-programming [16, 17] . Recent methods enable piano playing with dexterous hands through planning [18] or RL [19] but are limited to simple music pieces. RoboPianist [4] introduces a benchmark for robot piano playing and demonstrates strong RL performance, but requires human fingering labels and performs worse in multi-task learning. Yang et al. [20] improves the policy training performance by considering the bionic constraints of the anthropomorphic robot hands.Human fingering informs the agent of the correspondence between fingers and pressed keys at each time step. These labels require expert annotators and are, therefore, expensive to acquire in practice.Several approaches learn fingering from human-annotated data with different machine learning methods [6, 21, 22] . Moryossef et al. [23] extract fingering from videos to acquire fingering labels cheaply. Ramoneda et al. [24] proposes to treat piano fingering as a sequential decision-making problem and use RL to calculate fingering but without considering the model of robot hands. Shi et al. [25] automatically acquires fingering via dynamic programming, but the solution is limited to simple tasks. Concurrent work [26] obtains fingering labels from YouTube videos and trains a diffusion policy to play hundreds of songs. In our paper, we do not introduce a separate fingering model, instead, similar to human pianists, fingering is discovered automatically while playing the piano, hereby largely expanding the pool of usable data to train a generalist piano-playing agent.Datasets for Dexterous Robot Hands Most large-scale datasets of dexterous robot hands focus on grasping various objects. To get suitable grasp positions, some methods utilize planners [27, 28, 29] , while others use learned grasping policies [30] , or track grasping motions of humans and imitate these motions on a robot hand [31] . Compared to the abundance of datasets for grasping, there exist relatively few datasets for object manipulation with dexterous robot hands. The D4RL benchmark [32] provides small sets of expert trajectories for four such tasks, consisting of human demonstrations and rollouts of trained policies. Zhao et al. [33] provides a small object manipulation dataset that utilizes a low-cost bimanual platform with simple parallel grippers. Chen et al. [34] collects offline datasets for two simulated bimanual manipulation tasks with dexterous hands. Furthermore, Fan et al. [35] proposes a large-scale dataset for bimanual hand-object manipulation with human hands rather than robot hands. Table 1 summarizes the characteristics of these existing datasets. To the best of our knowledge, our RP1M dataset is the first large-scale dataset of dynamic, bimanual piano playing with dexterous robot hands.We further discuss related work on dexterous robot hands and generalist agents in Appendix A. 
background: 
 Task Setup The simulated piano-playing environment is built upon RoboPianist [4] . It includes a robot piano-playing setup, an RL-based agent for playing piano with simulated robot hands, and a multi-task learner. To avoid confusion, we refer to these components as RoboPianist, RoboPianist-RL, and RoboPianist-MT, respectively. The piano playing environment features a full-size keyboard with 88 keys driven by linear springs, two Shadow robot hands [36] , and a pseudo sustain pedal.Sheet music is represented by Musical Instrument Digital Interface (MIDI) transcription. Each time step in the MIDI file specifies which piano keys to press (active keys). The goal of a piano-playing agent is to press active keys and avoid inactive keys under space and time constraints. This requires the agent to coordinate its fingers and place them properly in a highly dynamic scenario such that Comparison of the RL performance with our OT fingering, human-annotated fingering, and no fingering. Our method matches the performance of RoboPianist-RL, which is trained with human fingering. We also outperforms the baseline without any fingering information by a large margin. The plots show the mean over 3 random seeds and the shaded areas represent the 95% confidence interval.target keys, at not only the current time step but also the future time steps, can be pressed accurately and timely. The original RoboPianist uses MIDI files from the PIG dataset [6] which includes human fingering information annotated by experts. However, as mentioned earlier, this limits the agent to only play human-labeled music pieces, and the human annotation may not be suitable for robots due to the different morphologies.The observation includes the state of the two robot hands, fingertip positions, piano sustain state, piano key states, and a goal vector, resulting in an 1144-dimensional observation space. The goal includes 10-step active keys and 10-step target sustain states obtained from the MIDI file, represented by a binary vector. RoboPianst further includes 10-step human-labeled fingering in the observation space but we remove this observation in our method since we do not need human-labeled fingering. For the action space, we remove the DoFs that do not exist in the human hand or are used in most songs, resulting in a 39-dimensional action space, consisting of the joint positions of the robot hands, the positions of forearms, and a sustain pedal. We evaluate the performance of the trained agent with an average F1 score calculated by F 1 = 2 • precision•recall precision+recall . For piano playing, recall and precision measure the agent's performance on pressing the active keys and avoiding inactive keys respectively [4] .Playing Piano with RL We use RL to train specialist agents per song to control the bimanual dexterous robot hands to play the piano. We frame the piano playing task as a finite Markov Decision Process (MDP). At time step t, the agent π θ (a t |s t ), parameterized by θ, receives state s t and takes action a t to interact with the environment and receives new state s t+1 and reward r t . The state and action spaces are described above and the reward r t gives an immediate evaluation of the agent's behavior. We will introduce reward terms used for training in Section 4.1. The agent's goal is to maximize the expected cumulative rewards over an episode of length H, defined asJ = E π θ H t=0 γ t r t (s t , a t ), where γ is a discount factor ranging from 0 to 1. 
large-scale motion dataset collection: 
 In this section, we describe our RP1M dataset in detail. We first introduce how to train a specialist piano-playing agent without human fingering labels. Removing the requirement of human fingering labels allows the agent to play any sheet music available on the Internet (under copyright license). We then analyze the performance of our specialist RL agent as well as the learned fingering. Lastly, we introduce our collected large-scale motion dataset, RP1M, which includes ∼1M expert trajectories for robot piano playing, covering ∼2k pieces of music.To mitigate the hard exploration problem posed by the sparse rewards, RoboPianist-RL adds dense reward signals by using human fingering labels. Fingering informs the agent of the "ground-truth" fingertip positions, and the agent minimizes the Euclidean distance between the current fingertip positions and the "ground-truth" positions. We now discuss our OT-based method to lift the requirement of human fingering.Although fingering is highly personalized, generally speaking, it helps pianists to press keys timely and efficiently. Motivated by this, apart from maximizing the key pressing rewards, we also aim to minimize the moving distances of fingers. Specifically, at time step t, for the i-th key k i to press, we use the j-th finger f j to press this key such that the overall moving cost is minimized. We define the minimized cumulative moving distance between fingers and target keys as d OT t ∈ R + , given byEQUATIONK t represents the set of keys to press at time step t and F represents the fingers of the robot hands. c t (k i , f j ) represents the cost of moving finger f j to piano key k i at time step t calculated by their Euclidean distance. w t (k i , f j ) is a boolean weight. In our case, it enforces that each key in K t will be pressed by only one finger in F , and each finger presses at most one key. The constrained optimization problem in Eq. ( 1) is an optimal transport problem. Intuitively, it tries to find the best "transport" strategy such that the overall cost of moving (a subset of) fingers F to keys K t is minimized. We solve this optimization problem with a modified Jonker-Volgenant algorithm [37] from SciPy [38] and use the optimal combinations (i * , j * ) as the fingering for the agent. The fingering is calculated on the fly based on the hands' state, so during the RL training, the fingering adjusts according to the robot hands' state.We define a reward r OT t to encourage the agent to move the fingers close to the keys K t . which is defined as:EQUATIONc is a constant scale value as used in Tassa et al. [39] and d OT t is the distance between fingers and target keys obtained by solving Eq. (1). r OT t increases exponentially as d OT t decreases and is set as 1 once d OT t is smaller than a pre-defined threshold (0.01). The overall reward function is defined as:r t = r OT t + r Press t + r Sustain t + α 1 • r Collision t + α 2 • r Energy t (3)r Press and r Sustain t represent the reward for correctly pressing the target keys and the sustain pedal. r Collision t encourages the agent to avoid collision between forearms and r Energy t prefers energy-saving behaviors. α 1 and α 2 are coefficient terms, and α 1 = 0.5 and α 2 = 5 • 10 -3 are adopted. Our method is compatible with any RL methods, and we use DroQ [40] in our paper.The performance of the specialist RL agents decides the quality of our dataset. In this section, we investigate the performance of our specialist RL agents. We are interested in i) how the proposed OTbased finger placement helps learning, ii) how the fingering discovered by the agent itself compares to human fingering labels, and iii) how our method transfers to other embodiments.In Fig. 2 , we compare our method with RoboPianist-RL both with and without human fingering. We use the same DroQ algorithm with the same hyperparameters for all experiments. RoboPianist-RL includes human fingering in its inputs, and the fingering information is also used in the reward function to force the agent to follow this fingering. Our method, marked as OT, removes the fingering from the observation space and uses OT-based finger placement to guide the agent to discover its own fingering. We also include a baseline, called No Fingering, that removes the fingering entirely. The first two columns of Fig. 2 show that our method without human-annotated fingering matches RoboPianst-RL's performance on two different songs. Our method outperforms the baseline without human fingering by a large margin, showing that the proposed OT-based finger placement boosts the agent learning. The proposed method works well even on challenging songs. We test our method on Flight of the Bumblebee and achieve 0.79 F1 score after 3M training steps. To the best of our knowledge, we are the first to play the challenging song Flight of the Bumblebee with general-purpose bimanual dexterous robot hands.We now compare the fingering discovered by the agent itself and the human annotations. In Fig. 3 , we visualize the sample trajectory of playing French Suite No.5 Sarabande and the corresponding fingering. We found that although the agent achieves strong performance for this song (the second plot in Fig. 2 ), our agent discovers different fingering compared to humans. For example, for the right hand, humans mainly use the middle and ring fingers, while our agent uses the thumb and first finger. Furthermore, in some cases, human annotations are not suitable for the robot hand due to different morphologies. For example, in the second time step of Fig. 3 , the human uses the first finger and ring finger. However, due to the mechanical limitation of the robot hand, it can not press keys that far apart with these two fingers, thus mimicking human fingering will miss one key. Instead, our agent discovered to use the thumb and little finger, which satisfies the hardware limitation and accurately presses the target keys.Cross Emboidments Labs usually have different robot platforms, thus having a method that works for different embodiments is highly desirable. We test our method on a different embodiment. To simplify the experiment, we disable the little finger of the Shadow robot hand and obtain a four-finger robot hand, which has a similar morphology to Allegro [41] and LEAP Hand [42] . We evaluate the modified robot hand on the song French Suite No.5 Sarabande (first 550 time steps), where our method achieves a 0.95 F1 score, similar to the 0.96 achieved with the original robot hands. In the bottom row of Fig. 3 , we visualize the learned fingering with four-finger hands. The agent discovers different fingering compared to humans and the original hands but still accurately presses active keys, meaning our method is compatible with different embodiments.To facilitate the research on dexterous robot hands, we collect and release a large-scale motion dataset for piano playing. Our dataset includes ∼1M expert trajectories covering ∼2k musical pieces. For each musical piece, we train an individual DroQ agent with the method introduced in Section 4.1 for 8 million environment steps and collect 500 expert trajectories with the trained agent. We chunk each sheet music every 550 time steps, corresponding to 27.5 seconds, so that each run has the same episode length. The sheet music used for training is from the PIG dataset [6] and a subset (1788 pieces) of the GiantMIDI-Piano dataset [7] .In Fig. 4 , we show the statistics of our collected motion dataset. The top plot shows the histogram of the pressed keys. We found that keys close to the center are more frequently pressed than keys at the corner. Also, white keys, taking 65.7%, are more likely to be pressed than black keys. In the bottom left plot, we show the distribution of the number of active keys over all time steps. It roughly follows a Gaussian distribution, and 90.70% musical pieces in our dataset include 1000-4000 active keys. We also include the distribution of F1 scores of trained agents used for collecting data. We found most agents (79.00%) achieve F1 scores larger than 0.75, and 99.89% of the agents' F1 scores are larger than 0.5. The distribution of F1 scores reflects the quality of the collected dataset. We empirically found agents with F1 score ≥ 0.75 are capable of playing sheet music reasonably well with only minor errors. Agents with ≤ 0.5 F1 scores usually have notable errors due to the difficulty of songs or the mechanical limitations of the Shadow robot hand. We also include the F1 scores for each piece in our dataset so users can filter the dataset according to their needs.A 0 A # 0 B 0 C 1 C # 1 D 1 D # 1 E 1 F 1 F # 1 G 1 G # 1 A 1 A # 1 B 1 C 2 C # 2 D 2 D # 2 E 2 F 2 F # 2 G 2 G # 2 A 2 A # 2 B 2 C 3 C # 3 D 3 D # 3 E 3 F 3 F # 3 G 3 G # 3 A 3 A # 3 B 3 C 4 C # 4 D 4 D # 4 E 4 F 4 F # 4 G 4 G # 4 A 
benchmarking results: 
 The analysis in the previous section highlighted the diversity of highly dynamic piano-playing motions in the RP1M dataset. In this section, we assess the multi-task imitation learning performance of several widely used methods on our benchmark. To be specific, the objective is to train a single multi-task policy capable of playing various music pieces on the piano. We train the policy on a portion of the RP1M dataset and evaluate its in-distribution performance (F1 scores on songs included in the training data) and its generalization ability (F1 scores on songs not present in the training data).Baselines We evaluated Behavior Cloning (BC) [43] , Behavior Transformer (BeT) [44] , Diffusion Policy [5] with U-Net (DP-U) [45] and with Transformer (DP-T) [46] . BC directly learns a policy by using supervised learning on observation-action pairs from expert demonstrations. BeT clusters continuous actions into discrete bins using k-means, allowing it to model high-dimensional, continuous, multimodal action distributions as categorical distributions [44] . Diffusion Policy learns to model the action distribution by inverting a process that gradually adds noise to a sampled action sequence. We evaluated both the CNN-based (U-Net) Diffusion Policy (DP-U) and the Transformer-based Diffusion Policy (DP-T) with DDPM [47] . We use the same code and hyperparameters as Chi et al. [5] . Detailed descriptions of the baselines as well as hyperparameters are given in Appendix C.1.We train the policies on subsets of the RP1M dataset with different sizes: 12, 25, 50, 100, 150. We then evaluate the trained policies on both i) 12 in-distribution songs: music pieces that overlap with the training sets, and ii) 20 out-of-distribution (OOD) songs: music pieces that do not overlap with the training songs. The selected songs are very challenging and contain diverse motions and long horizons. In the experiment, we report zero-shot evaluation results without fine-tuning. We report the average F1 scores of each group of music pieces for policies trained with each baseline method. We list the selected songs for evaluation in Appendix C.2.We present the benchmarking performance of multi-task agents in Table 2 . For the in-distribution evaluation, compared to F1 scores obtained with our RL specialist agents in Fig. 4 , we notice a performance gap across all baselines. This gap widens as the data size increases. When trained on a smaller dataset with 12 training songs, DP-U performs comparably to BC-MLP and slightly outperforms DP-T, while BeT experiences a significant performance drop. This decline may be attributed to hyperparameter choices, such as the number of action bins. Although we used the same number of action bins as the official implementation, the complexity of our tasks suggests that this configuration may be inadequate, and increasing the number of bins could improve performance.As the dataset size increases, we observe that Diffusion Policy outperforms the other baselines. DP-U and DP-T show performance drops of 15.77% and 10.92%, respectively, while BC-MLP suffers a more significant decline of 52.74%. Similar performance gaps have been noted in previous work [4] and concurrent research [26] suggests a hierarchical policy structure, although it still lags behind RL specialists. This highlights the need for future research to address the performance gap between RL specialists and multi-task agents.In the zero-shot out-of-distribution evaluation, we find that performance improves for all evaluated baselines as the training data size increases. Specifically, the F1 scores for DP-U and DP-T rise from 0.181 to 0.256 and from 0.186 to 0.316, respectively, when the number of training songs is increased from 12 to 150. This suggests that larger datasets enhance the generalization capabilities of multi-task agents. We hope that releasing our large-scale RP1M dataset will contribute to the development of robust generalist piano-playing agents within the research community. 
limitations & conclusion: 
 Limitations Our paper has limitations in several aspects. Firstly, although our method lifts the requirement of human-annotated fingering, enabling RL training on diverse songs, our method still fails to achieve strong performance on challenging songs due to fast rhythms and mechanical limitations of the robot hands. Improving the RL method and hardware design could help address this. Secondly, the evaluation metric, F1 score, may not adequately capture musical performance and the position-based controller missing the target velocity would hinder the performance. Thirdly, our dataset includes only proprioceptive observations, whereas humans play piano using multi-modal inputs like vision, touch, and hearing; incorporating these could enhance the agent's capabilities. Furthermore, there are several challenges to deploying the learned agent on a real-world robot. This includes the challenges of obtaining the state of the piano and the hands (e.g., tracking the precise fingertip positions), optimizing a precise position controller at high speed as well as the sim-to-real gap for the highly dynamic piano-playing task, etc. Lastly, although we demonstrate better zero-shot generalization performance than RoboPianist-MT [4] , there is still a gap between our best multi-task agent and RL specialists, which requires future investigation. 
 
 Appendix: In this paper, we propose a large-scale motion dataset named RP1M for piano playing with bimanual dexterous robot hands. RP1M includes 1 million expert trajectories for playing 2k musical pieces. To collect such a diverse dataset for piano playing, we lift the need for humanannotated fingering in the previous method by introducing a novel automatic fingering annotation approach based on optimal transport. On single songs, our method matches the baselines with human-annotated fingering and can be adopted across different embodiments. Furthermore, we benchmark various imitation learning approaches for multi-song playing. We report promising results in motion synthesis for novel music pieces when increasing the data size and identify the gap to achieving human-level piano-playing ability. We believe the RP1M dataset, with its scale and quality, forms a solid step towards empowering robots with human-level dexterity.Dexterous Robot Hands The research of dexterous robot hands aims to replicate the dexterity of human hands with robots. Many previous works [48, 49, 50, 51, 52, 53, 54, 55] use planning to compute a trajectory followed by a controller, thus require an accurate model of the robot hand. Closed-loop approaches have been developed by incorporating sensor feedback [56] . These methods also require an accurate model of the robot hand, which can be difficult to obtain in practice, especially considering the large number of active contacts between the hand and objects.Due to the difficulty of actually modeling the dynamics of the dexterous robot hand, recent methods resort to learning-based approaches, especially RL, which has achieved huge success in both robotics [57, 58, 1] and computer graphics [59] . To ease the training of dexterous robot hands with a large number of degrees of freedom (DoFs), demonstrations are commonly used [60, 61, 62, 63, 64] . Due to the advance of both RL algorithms and simulation, recent work shows impressive results on dexterous hand manipulation tasks without human demonstrations. Furthermore, the policy trained in the simulator can further be deployed on real dexterous robot hands via sim-to-real transfer [65, 66, 67, 30, 68, 69, 70] .Generalist Agents RL methods usually perform well on single tasks, however, as human beings, we can perform multiple tasks. Generalist agents are proposed to master a diverse set of tasks with a single agent [71, 57, 72, 73] . These methods typically resort to scalable models and large datasets [72, 74, 75, 76, 77] . Recently, diffusion models have achieved many state-of-the-art results across image, video, and 3D content generation [78, 79, 80, 81, 82] In the context of robotics, diffusion models have been used as policy networks for imitation learning in both manipulation [5, 75, 83, 84] and locomotion tasks [85] . The same technique has also been investigated in multi-task learning [75, 84] . We investigate the application of diffusion policy in high-dimensional control tasks, that is, playing piano with bimanual dexterous robot hands.In Eq. ( 3) , we give the overall reward function used in our paper. We now give details of each term. r Press t indicates whether the active keys are correctly pressed and inactive keys are not pressed. We use the same implementation as [4] , given as:r Press t = 0.5 • ( 1 K K t g(||k i s -1|| 2 )) + 0.5 • (1 -1 fp ).K is the number of active keys, k i t is the normalized key states with range [0, 1], where 0 means the i-th key is not pressed and 1 means the key is pressed. g is tolerance from Tassa et al. [39] , which is similar to the one used in Equation ( 2). 1 fp indicates whether the inactive keys are pressed, which encourages the agent to avoid pressing keys that should not be pressed. r Sustain Observation Space Our 1144-dimensional observation space includes the proprioceptive state of dexterous robot hands and the piano as well as L-step goal states obtained from the MIDI file. In our case, we include the current goal and 10-step future goals in the observation space (L=11). At each time step, an 89-dimensional binary vector is used to represent the goal, where 88 dimensions are for key states and the last dimension is for the sustain pedal. The dimension of each component in the observation space is given in Table 3 . To speed up training, we re-implement the RoboPianist environment with MuJoCo XLA (MJX), which supports simulation in parallel with GPUs. MJX has a slow performance with complex scenes with many contacts. To improve the simulation performance, we made the following modifications:• We disable most of the contacts but only keep the contacts between fingers and piano keys as well as the contact between forearms.• Primitive contact types are used whenever possible.• The dimensionality of the contact space is set to 3.• The maximal contact points are set to 20.• We use Newton solver with iterations=2 and ls iterations=6.After the above modifications, with 1024 parallel environments, the total steps per second is 159,376.We use PPO implementation implemented with Jax to fully utilize the paralleled simulation. The PPO with MJX implementation is much faster than the DroQ implementation, which only takes 2 hours and 7 minutes for 40M environment steps on the Twinkle Twinkle Little Star song while as a comparison, DroQ needs roughly 21 hours for 8M environment steps. However, the PPO implementation fails to achieve a comparable F1 score as the DroQ implementation as shown in Fig. 5 . Therefore, we use the DroQ implement with the CPU version of the RoboPianist environment. A single multi-task policy capable of playing various songs is highly desirable. However, playing different music pieces on the piano results in diverse behaviors, creating a complex action distribution, particularly for dexterous robot hands with a large number of degrees of freedom (DoFs). This section introduces the baseline methods we have compared and the hyperparameters we have used. We also talk about the details of our multitask training and evaluation.to an experienced human pianist who can play new pieces at first sight, we aim to determine if it is feasible to develop a generalist agent capable of playing the piano under various conditions. Additionally, our framework is designed with flexibility in mind, allowing users to select songs not included in our dataset for either training data collection or evaluation. Furthermore, users have the option to assess their policies on specific segments of a song rather than the entire piece.  


Decision: # strengths 
- The strength of the paper has been well recognized by all reviewers. 
- Generalization ability of learned agents to beyond individual pieces has also been well recognized.
- The novelty of finger placements as an optimal transport problem has been highlighted
- Evaluations on cross-embodiment has been welcomed by multiple reviewers

# weakness 
- There are concerns raised on the evaluation metrics and their relevance to realistic piano playing. 
- Details missing on baselines and ablation. They additionally are in need of further clarifications 

# recommendation 
Reviewers have provided an overwhelming diversity of suggestions for improvements. Authors are encouraged to improve the submission based on the provided suggestions.

# rebuttal summary 
The authors provided comprehensive responses to the reviewer's feedback. The authors also proposed to add clarifications, improve on baselines, and make the datasets public. Post these clarifications, reviews adjusted their scores. All reviewers are aligned on their recommendation for the paper. Based on their suggestions, AC recommends to accept the paper.

Reviews: review: title: title: Good job on building and expanding upon an existing benchmark, but the benchmark itself is problematic.
paper_summary: paper_summary: The authors propose a method for automatically generating fingering charts for piano music, by posing the problem of mapping fingers to piano keys as an "optimal transport" problem that can be solved with an off-the-shelf solver. They show that giving these fingering charts to RL agents and adding an auxiliary reward for following the fingering sequence greatly improves the agents' ability to learn to play a song. The authors present a new dataset, RP1M, which is the result of training a specialist agent for each of 2k songs, and rolling out 500 trajectories from each agent. The authors demonstrate that this approach works for 4-finger hands, and that the data can be used to train piano-playing agents using various imitation-learning methods. The environment and evaluation metric are based on the benchmark presented by (Zakka et. al., 2023).
main_review: main_review: ## Post-rebuttal update

In the conversation below, the authors convinced me that this paper has enough contributions to warrant publication, despite shortcomings in the benchmark itself. I think I've made my point about applicability of this benchmark to real piano playing, but don't want to block the author's contribution to robotics and open source code.

I have updated my recommendation to weak accept.

## Original review

Clarity:

The paper is written clearly and reads smoothly. It was easy to understand what the authors did.

Originality:

Within the field of robotics, this paper seems unusual and original. Given the goal of optimizing the "F1 score" (from (Zakka et. al., 2023)), reformulating fingering as an optimal transport is clever, and the fact that RL is still used allows agents to use the fingering as a hint, so imperfect or implausible finger annotations are not a complete blocker for performance.

This work contributes to a body of work around automatic piano fingering generation, which I'm not qualified to comment on. I think the originality of the proposed approach should be evaluated compared to that body of work.

Significance:

Again, the work can be evaluated as a robotics paper, or in its merics for automatic piano fingering generation.
In terms of robotics, the method is quite specific to piano playing, and to optimizing the particular metric introduced in Zakka's paper. It will be of interest to others in the field who want to try their hand at the benchmark, but perhaps not to the wider community.
For automatic piano fingering generation, it's possible that with small modifications this work could be a competitive approach for generating piano fingerings: use a realistic human hand model and RL training to improve upon the imperfect fingering generated by the optimal transport heuristic.

Strengths:

- The work greatly expands on the existing piano playing benchmark and presents an approach that is applicable to any piece of piano music.
- They demonstrate an ability of learned agents to generalise beyond individual pieces that they were trained on.

Weaknesses:

- The work inherits the main weakness of the original benchmark, which is nothing to do with robotics, and all to do with music. The F1 metric is not grounded in musical insight, and a near-perfect F1 score does not correspond to a good musical performance. Specifically, the perceived quality of a piano performance depends on the precise timing of the start of each note, and on the velocity with which they are played, neither of which are taken into account by the metric. Additionally, controlling the robot and evaluating the rewards with a 50ms timestep, regardless of the underlying music's rhythm, guarantees that notes will be missed. For example, in Flight of the Bumblebee, each note's duration is about 80ms, so an error of +-25ms represents a significantly incorrect timing. In general, listeners will notice a latency of 15ms. As such, I don't have confidence that the recorded trajectories in this dataset correspond to useful musical performances. This can be heard in the demonstrated performances in the supplementary video. This is not the authors' fault, but limits the usefulness of the data for anything but this particular benchmark.

- When proposing a new way of generating piano fingerings, it seems important to compare it to other automatic fingering generation methods, such as those cited in the introduction. The paper currently only compares to having no fingering charts at all.

- It is not clear how closely the expert agents actually follow the generated fingering. Is its importance in guiding the agents towards a good initial guess, or is it really good enough for performing the pieces? I would like to see some measure of the difference between suggested fingerings and performed fingerings.

Minor comments:

- I think that "bimanual" is more common than "bi-manual". Similarly for "multimodal" vs "multi-modal".
- On line 70, you argue that RL is used because of the difficulty of modelling the dynamics of a hand accurately. This might be a reason for model-free RL in some domains, but in this benchmark there is no modelling error: one could plan and execute in the simulated environment with perfect fidelity. The reason to use RL for this work is because the reward landscape is very challenging for gradient-based trajectory optimization methods, with the large number of possible self-collisions that a bimanual setup has.
- In the final slide of the supplementary video, the audio and video are not well synchronized.
- In figure 4, "Num. of Active Keys Per Musical Piece" is a strange metric to have. Is this the number of notes played in the piece?
questions: questions: - Address the downsides of the benchmark in the limitations section.
- Compare the proposed fingering generation method to others in the literature.
- Check whether the generated fingering is followed by the RL specialists.
- Please address the minor comments listed above.

Request to area chair: Maybe reach out to reviewers outside of robotics, who wrote about piano fingering generation.
limitations: limitations: 2
review_summary: review_summary: Because of the limitations of the benchmark itself, I don't think this work will have impact on real robot piano playing. That said, it is a big improvement on the existing benchmark, and we should accept it on the merits of its robotics work.
title: title: Good dataset on a fairly unexplored topic
paper_summary: paper_summary: The paper presents a new dataset for piano playing. The dataset consists of 2k songs. The paper makes data collection a bit cheaper by using Optimal Transport to assign finger to keys such as the overall finger movement is minimized.
main_review: main_review: The paper presents a new dataset for piano playing, the dataset consists of 2k songs. The paper evaluates several imitation learning baselines and shows that having finger assignments is more beneficial than using ground truth annotations. Further, it shows that diffusion policy is superior to all other baselines but not very sample efficient.  

Strengths

- Largest dataset available for this task with 2k songs.
- Evaluations on cross embodiment, reducing training data, benchmarking multiple imitation learning methods
- Made data collection more efficient by removing the need for finger information and replacing the module with optimal transport where we cast the assignment problem of assigning finger to each stroke


Weaknesses

- Evaluations can be improved -- for instance the number of methods on imitation learning 
- Model improvements were not explored well. The numbers in table 2, show that the diffusion policy's performance decreases as we add more weights. The paper could have also benefitted from model-based methods
- The numbers produced in table 2 are a bit problematic, mainly because the diffusion model performance reduces with increase in data
questions: questions: Please confirm if the dataset would be released in public.
limitations: limitations: 3
review_summary: review_summary: I am recommending to accept the paper mainly because I like the dataset and it is an unexplored problem.
title: title: Review
paper_summary: paper_summary: This work introduces the RP1M dataset, which provides extensive bi-manual piano playing data with over one million trajectories. By matching finger positions, the dataset enables the automatic annotation of songs, significantly reducing the need for human-labeled data and making it feasible to scale imitation learning.
main_review: main_review: Strengths:
1. This is the first large-scale dexterous robot piano playing dataset, generated with RL and does not require human annotations. It is a good resource for this line of research.
2. It is refreshing to see the proposed method is compatible with different embodiments with either four or five fingers.
3. The proposed method is able to handle highly dynamic pieces like "Flight of the Bumblebee," which is truly amazing.
4. The benchmark of multi-task imitation learning based on RP1M highlights the strength of diffusion policy in such a multi-modal case, which is a good finding for multi-task dexterous manipulation.


Questions:
1. My main concern lies in the simplicity of the position-based controller. For piano playing, it seems critical to also align the velocity of the finger movements, not just the positional mapping (Line 150), since it will affect the strength of the sound and should be an important metric of music playing. This becomes more important if we plan to use this dataset on a real-world robot with PD controllers.
2. For multi-task imitation learning, MT-ACT from RoboAgent[1] is an important missing citation and baseline method in the benchmark. It would be very helpful to show MT-ACT’s performance in the dexterous manipulation domain.
3. As a dataset paper, I would recommend the authors use a website for browsing the quality of the data. The current video submission only contains a few pieces, making it hard to estimate the scale and quality of the dataset.
4. Since RP1M contains so many pieces of piano playing, is it possible to incorporate generative models for music composition and synthesizing? If not, what might be the remaining challenges? More data or better model architecture?
5. It would also be beneficial to explain how to use such a dataset on a real-world robot system. For example, how to track the precise fingertip positions? Is there a sim-to-real gap during highly dynamic pieces? Is positional control enough for piano playing in the real world? I’m not sure whether this year's CoRL has the rule of real-world robot testing results. But I do agree that reproducing the system in the real world requires too much engineering effort.

[1]. RoboAgent: Towards Sample Efficient Robot Manipulation with Semantic Augmentations and Action Chunking
questions: questions: Same as the prior question section
limitations: limitations: 3
review_summary: review_summary: Overall, this work is a good contribution to the community. However, due to the missing baseline and the potential challenge of the position-based controller, I will start with a weak reject and will adjust my rating based on the authors' response.
title: title: Clear and well supported work which would benefit from ablations
paper_summary: paper_summary: This work proposes a bi-manual, dexterous, piano-playing dataset with 1M demonstrations. The dataset is diverse by forgoing human annotations and utilizing an optimal transport reward for training the RL agent. The work is competitive with human-annotated methods for individual songs and provides preliminary results for a multi-task piano-playing policy.
main_review: main_review: This work presents a well-curated dataset for training dextrous, bi-manual piano-playing policies. The overall paper is clear and the simulation experiments are complete but could be expanded with more ablations. There is no demonstration of the method on hardware which would greatly improve the paper, but I do not think is required.

Strengths:

-Overall, this paper is very clear and clearly articulates the method and intuition with a few exceptions

-The method does not rely on human annotations and is hand morphology agnostic allowing this method to scale better than alternative methods

-The analysis of the learned fingering due to embodiment constraints is very interesting and could benefit from further detail

Weaknesses:

-It would be helpful to further elaborate on in section 4.1 the design of reward equation 2

-The work currently uses DroQ but could benefit from an ablation study with different RL algorithms 

-The work should also include ablations for hyperparameters of DroQ to ensure baselines have the best possible performance they can attain

-For Section 5, it would be helpful to further discuss why diffusion policies don’t perform as well as the generalist agent instead of keeping that for future work as it is very interesting to see 

-I would recommend expanding on Figure 4 by adding more statistics about the dataset and removing Figure 3 if necessary

-The limitations are good but I would add some details about expected problems when deploying on real hands

Consider to cite the following additional work:
[1] also trains a general piano-playing agent from demonstrations, but they curate their dataset from YouTube videos and have a framework to extract features from the videos, train expert policies for each song, and distill the policies into one agent. Since this work can be used with different hand embodiments, [2] is relevant as it trains a diffusion policy with biological constraints from an anthropomorphic hand and is competitive with SOTA RL results. The authors claim that this dataset is the “first large-scale dataset of dynamic, bi-manual manipulation with dexterous robot hands”, however, [3] also creates “a dataset of two hands that dexterously manipulate objects” albeit from video frames. This claim should be reviewed and ensured it is accurate. 

[1] https://arxiv.org/abs/2407.18178 

[2] https://spj.science.org/doi/full/10.34133/cbsystems.0104

[3] https://arxiv.org/abs/2204.13662
questions: questions: -Why can [3] not be considered the “first large-scale dataset of dynamic, bi-manual manipulation with dexterous robot hands”? It seems like this is a strong claim and I would welcome more elaboration for this as it is a strong statement.

-In the related works, it would be helpful to further distinguish from [1] as they also train an RL policy for each song but then distill the policies to create a generalist policy rather than creating a dataset to train a single policy.

-What additional metrics can be included to characterize the dataset?
limitations: limitations: 2
review_summary: review_summary: Overall, this dataset will be helpful to the community and the paper is both clear and has sufficient experimental support. However, the paper would benefit greatly from additional ablations and deployment on real world hands as that would clearly show the usefulness of this simulation dataset.
title: title: Review
paper_summary: paper_summary: The paper introduces the Robot Piano 1 Million (RP1M) dataset, which contains bi-manual robot piano-playing motion data for 2k sounds. Considering that the human annotation required in the RoboPianist is hard to collect, it proposes an Optimal-transport-based reward when training the RL specialist without human annotated fingering. To collect the motion dataset, it first trains RL agents for each song and then uses the trained RL agents to collect datasets. It further benchmarks Imitation Learning methods based on the collected dataset.
main_review: main_review: Strength:
 - The paper addresses an interesting problem: enabling robots to play piano songs. This task demands precise hand-finger movements and involves significant dynamism. 
 - The use of optimal transport-based rewards for robotic piano playing is, to the reviewer’s knowledge, a novel approach.

Weakness:
 - The main contribution of the paper is the introduction of a domain-specific dataset for robotic piano playing. While the task itself is interesting, the domain remains quite specialized, making it challenging to assess how the dataset could impact other robotics tasks/domains.
 - The quality of the RL-generated dataset can be highly variable, and a more comprehensive evaluation mechanism is needed. Currently, the dataset is collected by running an agent after training for a large number of steps. It would be beneficial to implement a data filtering mechanism to ensure the quality of the collected dataset or to only collect data after the RL agent reaches a certain accumulated reward threshold. For instance, section 4 mentions that “agents with an F1 score ≥ 0.75 are capable of playing sheet music reasonably well with only minor errors.” In this case, filtering out agents or retraining those with an F1 score less than 0.75 is important to maintain dataset quality.
questions: questions: - How to measure the generated data quality and what can be a good data filtering mechanism?
limitations: limitations: 3
review_summary: review_summary: The paper focuses on a specific task domain and generates a valuable dataset for that domain.
title: title: Review for "RP1M: A Large-Scale Motion Dataset for Piano Playing with Bi-Manual Dexterous Robot Hands"
paper_summary: paper_summary: The paper uses dexterous robot hands and introduces a new dataset, called the Robot Piano 1 Million (RP1M) dataset, containing over a million bi-manual piano playing trajectories. This dataset aims to improve performance of robots in the task of piano playing using imitation learning, facilitated by framing finger placements as an optimal transport problem.  The main contributions of the paper include the introduction of a novel dataset for an interesting, complex and previously under-researched task of piano playing, formulating fingering as an optimal transport problem that enables the authors to collect diverse large scale data using RL.
main_review: main_review: The paper focuses on a dynamic task with dexterous bimanual robot hands that requires precise contact-rich control. This task comes across as an important step in developing complex manipulation capabilities in robotics and is a step in the right direction. The background and related works section in the paper seems to be exhaustive and well written. 
For benchmarking of the results on the provided dataset, the paper uses a number of baselines that include variants of behavior cloning and diffusion policy. 

As shown in the results, diffusion policy beats the performance of other baselines by a significant margin. Although the experiments have been done with a variety of settings, the discussion sections lack the explanation behind the obtained results. The experiments section could also improve from the inclusion of more exhaustive set of baselines that could include baselines that are based on transformers (for e.g., behavior transformer [1], PACT [2]), LfD (Robomimic [3]), IL+RL (Relay Policy Learning [4]). This would provide the readers about how these baselines perform on the proposed benchmark.

Also, it would be useful for the reader if the authors could compare with methods that utilize a small number of human demonstrations. Even using videos available on the internet of people playing a piano could be used as a learning signal for the model.
The analysis of the dataset statistics along with the quantitative results showing the performance improvement over method without fingering is thorough and convincing. 

While this work is an important step in the direction of human-level dexterity, it would be more convincing if the authors could justify that by displaying some results on a real-world robot too. The transfer of the robot policy learnt in simulation to a real-world scenario would justify the utility of the proposed dataset to enable dexterity in robots. 
Finally, the dataset proposed, which is the a large scale dataset for dynamic bimanual manipulation with dexterous hands could serve the robotics community in a positive way, but the paper could certainly improve with more elaborate evaluation.

[1] Behavior Transformers: Cloning k modes with one stone. Nur Muhammad Mahi Shafiullah, Zichen Jeff Cui, Ariuntuya Altanzaya, Lerrel Pinto

[2] PACT: Perception-Action Causal Transformer for Autoregressive Robotics Pre-Training. Rogerio Bonatti, Sai Vemprala, Shuang Ma, Felipe Frujeri, Shuhang Chen, Ashish Kapoor

[3] What Matters in Learning from Offline Human Demonstrations for Robot Manipulation. Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, Roberto Martín-Martín

[4] Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning. Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, Karol Hausman
questions: questions: I would want the authors to consider adding more exhaustive baselines to the evaluation for the reader to get an idea about how the diverse set of approaches available work on the proposed benchmark. It would also be useful to see some real-world transfer experiments on this challenging task of robot piano playing.
limitations: limitations: 3
review_summary: review_summary: I feel the paper is a step in the right direction towards improving the robotics manipulation with dexterous arms and provides an important benchmark for the robotics community. The lack of exhaustive testing of prior approaches and real-world evaluation could be circumvented by the authors and could improve the overall quality of the paper.
title: title: Review of paper RP1M
paper_summary: paper_summary: This paper addresses a big limitation of the recently introduced RoboPianist benchmark: the need for fingering labels to train a piano playing policy with RL. They solve for fingering labels by formulating it as an optimal transport problem. Using this technique, they label 2000 MIDI files and demonstrate that they can effectively train specialist policies per MIDI file to generate a dataset of 1M piano playing motions. Finally, they distill this data into a multitask policy using diffusion policy and compare with other BC baselines.
main_review: main_review: ## Strengths

- Clarity of presentation
- Great related works section
- Pretty visualizations
- Open-source release of a massive dataset by robotics standards

## Weaknesses

- While I appreciate the simplicity and effectiveness of using optimal transport to solve for fingering information, and furthermore I understand how time-intensive the experiments can be, I think the experiments section can be improved overall. Specifically, I might even argue that you can remove 1 or 2 BC baselines (e.g. BC-RNN and IBC provide marginal value to the experiments) and focus more on improving the performance of Diffusion Policy and analyzing the failure modes. You have the gift of a huge dataset (1M snippets!) so if anything you should be really asking yourselves why performance is so paltry when the size of the dataset increases. There is either a data issue, or an optimization issue. 
- The test set in the BC results seems pretty small? 12 for in-distribution songs, 5 for easy out-of-distribution, and 5 for hard out-of-distribution. I think you can increase it to get more statistical significance.
- Speaking of statistical significance, why are there no standard deviations in Table 2? Diffusion policy is non-deterministic right?
- Performance of BC methods decreases for H=4096 as the number of songs increases seems fishy. As I said above, this is fishy and leads me to believe there is some issue with the optimization or the data. Can you comment on this?
- Speaking of dataset, what if song snippets repeat? E.g. you chunked up songs into 20 second snippets. But chunks in the same song could have a lot of overlap (especially true in music where things are periodic). Did you account for that during the filtering process?

## Minor

* I appreciated the implementation in MJX and the explanation for falling back to CPU MuJoCo in the appendix.
* There a few grammar and spelling mistakes in the manuscript, please fix them.
* Figures and tables should be self-contained, please add a caption to each one of them, not just a title.
questions: questions: - It would be good to get some more analysis about the distribution of failures exhibited by the 2k specialist policies. Some questions that popped to mind:
  - Are there any instances where the actuator gains or energy penalty were limiting the policy from playing fast-paced songs?
  - Relatedly, how did you decide what DoFs to remove from the wrist? And if anything, why not add more DoFs like roll/pitch/yaw to the forearm?
  - You mention failures from mechanical limitations of the shadow hand. Why not filter out those MIDI files from the dataset to disentangle hardware from policy learning?
- I noticed you used a stretch factor 1.25 (mentioned in the appendix). Did you slow down the songs for a specific reason, maybe because it improved overall performance? If so, is this a fair comparison with the RoboPianist curves?
- What software suite did you use for the BC experiments? Were all the baselines implemented using robomimic/diffusion policy or just diffusion policy used Chi et al's codebase?
- It would be nice to get a bit more details regarding how you formulated the optimal transport problem. Was any tweaking done to get it to work? I can imagine one could speedup the solver by ignoring keys on the far right/far left of the piano, or basically only considering keys that are within a horizon of time into the future? Also did you consider another fingering algorithm baseline, like the ones mentioned in the related works section?
limitations: limitations: 3
review_summary: review_summary: Overall a good paper that is well written and provides an elegant solution to the piano fingering problem. As it stands, I recommend accepting this paper, but I have some concerns regarding the BC results / dataset. I look forward to the authors clarifying my concerns.