PianoMime: Learning a Generalist, Dexterous Piano Player from Internet Demonstrations

Authors: cheng  qian
t u munich
julen  urain
t u darmstadt
kevin  zakka
jan  peters
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 00:03:58

Abstract: In this work, we introduce PianoMime, a framework for training a piano-playing agent using internet demonstrations.
The internet is a promising source of large-scale demonstrations for training our robot agents. 
In particular, for the case of piano-playing, Youtube is full of videos of professional pianists playing a wide myriad of songs.
In our work, we leverage these demonstrations to learn a generalist piano-playing agent capable of playing any arbitrary song.
Our framework is divided into three parts: a data preparation phase to extract the informative features from the Youtube videos,
a policy learning phase to train song-specific expert policies from the demonstrations and a policy distillation phase to distil the policies into a single generalist agent.
We explore different policy designs to represent the agent and evaluate the influence of the amount of training data on the generalization capability of the agent to novel songs not available in the dataset.
We show that we are able to learn a policy with up to 57% F1 score on unseen songs.

Main paper: 
 introduction: 
 The Internet is a promising source of large-scale data for training generalist robot agents. If properly exploited, it is full of demonstrations (video, text, audio) of humans solving an infinite number of tasks [1, 2, 3] that could inform our robot agents on how to behave. However, learning from these databases is challenging for several reasons. First, unlike teleoperation demonstrations, video data does not specify the actions that the robot is performing, which typically requires the use of reinforcement learning to induce the robot's actions [4, 2, 5] . Second, videos typically show a human performing the task while the learned policy is applied to a robot. This often requires retargeting the human motion to the robot body [5, 6, 7] . Finally, as pointed out in [2] , if we want to learn a generalist agent, we need to choose a task for which large databases are available and which allows for an unlimited variety of open-ended goals.From opening doors [6] to manipulating ropes [8] or pick and place tasks [9, 10] , previous work has successfully taught robot manipulation skills through observations. However, these approaches have been limited to robots with low dexterity or to a small variety of goals.In this work, we focus on the task of learning a generalist piano player from Internet demonstrations. Piano-playing is a highly dexterous open-ended task [11] . Given two multi-fingered robot hands and a desired song, the goal of a piano-playing agent is to press the right keys, and only the right keys, at the right time. In addition, the task can be conditioned on arbitrary songs, allowing for large and high-dimensional goal conditioning. In addition, the Internet is full of videos of professional piano players performing a wide variety of songs. Interestingly, these pianists often record themselves from above, making it easy to observe their performances. In addition, they usually share the MIDI files of the song they are playing, making it easier to extract relevant information.Figure 1 : The goal of this work is to train a generalist piano-playing agent (PianoMime) from Youtube videos. We collect a set of videos and accompanying MIDI files and train a single agent to play any song, combining reinforcement learning and behavioral cloning.To learn a generalist piano-playing agent from Internet data, we introduce PianoMime, a framework for training a single policy capable of playing any song (see Figure 1 ). In essence, the PianoMime agent is a goal-conditioned policy that generates actions in the configuration space, given the desired song to be played. At each time step, the agent receives a trajectory of keys to press as goal input. The policy then generates a trajectory of actions and executes them in chunks.To train the agent, we combine both reinforcement learning and imitation learning. We train individual song-specific expert policies using reinforcement learning in conjunction with YouTube demonstrations, and we distill all the expert policies into a single generalist behavior cloning policy.To represent the agent, we perform ablations of different architectural design strategies to model the behavior cloning policy. We investigate the benefit of incorporating representation learning to enhance the geometric information of the goal input. In addition, we explore the effectiveness of a hierarchical policy that combines a high-level policy generating fingertip trajectories with a learned inverse model generating joint space actions (see Figure 2 ). We show that the learned agent is able to play arbitrary songs not included in the training dataset with about 56% F1 score.In summary, the main contribution of this work is a framework for training a generalist piano-playing agent using Internet demonstration data. To achieve this goal, we • Introduce a method for learning policies from Internet demonstrations by decoupling the human movement information from the task-related information.• Present a reinforcement learning approach that combines residual policy learning strategies [12, 13] with style reward-based strategies [5] .• Explore different policy architecture designs, introduce novel strategies to learn geometrically consistent latent features, and perform ablations on different architecture designs.Finally, we release the dataset and trained models as a benchmark for testing Internet-data-driven dexterous manipulation. 
related work: 
 Robotic Piano Playing. Several studies have investigated the development of robots capable of playing the piano. In [14] , multi-target Inverse Kinematics (IK) and offline trajectory planning are used to position the fingers over the intended keys. In [15] , a Reinforcement Learning (RL) agent is trained to control a single Allegro hand to play the piano using tactile sensor feedback. However, the piano pieces used in these studies are relatively simple. Subsequently, in [11] , an RL agent is trained to control two Shadow hands to play complex piano pieces by designing a reward function that includes a fingering reward, a task reward, and an energy reward. In contrast to previous approaches, our approach exploits YouTube piano-playing videos, allowing for faster training and more accurate robot behavior.Motion Retargeting and Reinforcement Learning. Our work has similarities with motion retargeting [16] , especially those works that combine motion retargeting with RL to learn control policies [17, 18, 5, 19, 6] . Given a mocap demonstration, it is common to use the demonstration either as a reward function [5, 19] or as a nominal behavior for residual policy learning [18, 6] . In our work, we extract not only the mocap information, but also task-related information (piano states), which allows the agent to balance between mimicking the demonstrations and solving the task. 
method: 
 The PianoMime framework consists of three phases: data preparation, policy learning, and policy distillation.In the data preparation phase, given the raw video demonstration, we extract the informative signals needed to train the policies. Specifically, we extract the fingertip trajectories and a MIDI file that informs us of the state of the piano at each instant.In the policy learning phase, we train song-specific policies via RL. This step is essential to generate the robot actions that are missing in the demonstrations. The policy is trained with two reward functions: a style reward and a task reward. The style reward aims to match the robot's finger movements with those of the human in the demonstrations to preserve the human style, while the task reward encourages the robot to press the right keys at the right time.In the policy distillation phase, we train a single behavioral cloning policy to mimic all the songspecific policies. The goal of this phase is to train a single generalist policy that can play any song. We explore different policy designs and goal representation learning to improve the generalizability of the policy.We generate the training dataset by web scraping. We download YouTube videos of professional piano artists playing different songs. In particular, we select YouTube channels that also upload MIDI files of the songs played. The MIDI files represent the trajectories of the piano's state (keys pressed/unpressed) throughout the song. We use the video to extract the movement of human pianists and the MIDI file to inform about the target state of the piano during the execution of the song. We choose the fingertip position as the key signal for the robot hand to mimic. While some dexterous tasks may require the use of the palm (e.g., grasping a bottle), we believe that mimicking the fingertip motion is sufficient for the piano-playing task. This also reduces the constraints on the robot, allowing it to adapt its embodiment more freely.To extract the fingertip motion from the videos, we use MediaPipe [20] , an open-source framework for perception. Given a frame from the demonstration videos, MediaPipe outputs the skeleton of the hand. We find that the classic top-view recording in YouTube videos of piano playing is highly beneficial for obtaining an accurate estimate of fingertip positions. Note that since the videos are RGB, we lack the depth signal. Therefore, we predict the 3D fingertip positions based on the piano state. The detailed procedure is explained in Appendix A.In the data preparation phase, we extract two trajectories: a human fingertip trajectory τ x and a piano state trajectory τ ˇ" ( . The human fingertip trajectory τ x : px 1 , . . . , x T q is a T -step trajectory of the 3D fingertip positions of two hands x P R 3ˆ10 (10 fingers). The piano state trajectory τ ˇ" ( : p ˇ" ( 1 , . . . , ˇ" ( T q is a T -step trajectory of piano states ˇ" ( P B 88 , represented by an 88-dimensional binary variable representing which keys should be pressed.Given the ROBOPIANIST [11] environment, our goal is to learn a goal-conditioned policy π θ that plays the song defined by τ ˇ" ( while matching the fingertip movement given by τ x . Note that satis- fying both objectives jointly may be impossible. Perfectly tracking the fingertip trajectory τ x might not lead to playing the song correctly. Although both trajectories are collected from the same source, errors in hand tracking and embodiment mismatches might lead to deviations, resulting in poor song performance. Therefore, we suggest using τ x as a style guide behavior. ˇ" ( : p ˇ" ( t:t`L q at time t, a latent representation τ t z is computed given a pre-trained observation encoder. Then, the policy is decoupled between a high-level fingertip policy that generates a trajectory of fingertip positions τ tx and a low-level inverse model that generates a trajectory of target joint position τ t q . Similar to [11] , we formulate the piano playing as an Markov Decision Process (MDP) with the horizon of the episode H, which is the duration of the song to be played. The state observation is defined by the robot's proprioception s and the goal state g t . The goal state g t at time t informs the desired piano key configurations ˇ" ( in the future g t " p ˇ" ( t`1 , . . . , ˇ" ( t`L q, where L is the lookahead horizon. As claimed in [11] , to successfully learn how to play, the agent must be aware of several steps into the future to plan its actions. The action a is defined as the desired configuration for both hands q P R 23ˆ2`1 , each with 23 joint angles and one dimension for the sustain pedal.We propose to solve the reinforcement learning problem by combining residual policy learning [12, 13, 6] and style mimicking rewards [5, 19] . Residual Policy Architecture. Given the fingertip trajectory τ x , we solve an IK [21] problem to obtain a trajectory of desired joint angles τ ik q : pq ik 0 , . . . , q ik T q for the robot hands. Then we represent the policy π θ pa|s, g t q " π r θ pa|s, g t q`q ik t`1 as a combination of a nominal behavior (given by the IK solution) and a residual policy π r θ . Given the target state at time t, the nominal behavior is defined as the next desired joint angle q ik t`1 . We then learn only the residual term around the nominal behavior. In practice, we initialize the robot at q ik 0 and roll both the goal state and the nominal behavior with a sliding window along τ ˇ" ( and τ ik q respectively. Style Mimicking Reward. We also include a style-mimicking reward to preserve the human style in the trained robot actions. The reward function r " r ˇ" ( `rx consists of a task reward r ˇ" ( and a style-mimicking reward r x . While the task reward r ˇ" ( encourages the agent to press the correct keys, the style reward r x encourages the agent to move his fingertips similar to the demonstration τ x . We provide further details in Appendix D.In the policy learning phase, we train song-specific expert policies from which we roll out state and action trajectories τ s : ps 0 , . . . , s T q and τ q : pq 0 , . . . , q T q. Then we generate a datasetD : pτ i s , τ i q , τ i x , τ i ˇ" ( q N i"1where N is the number of songs learned. Given the dataset D, we apply Behavioral Cloning (BC) to learn a single generalist piano-playing agent π θ pq t:t`L , x t:t`L |s t , ˇ" ( t:t`L q, which outputs configuration space actions q t:t`L and fingertip movements x t:t`L conditioned on the current state s t and the future desired piano states ˇ" ( t:t`L .We explore different strategies to represent and learn the behavioral cloning policy and improve its generalization capabilities. In particular, we explore (1) representation learning approaches to induce spatially informative features, (2) a hierarchical policy structure for sample-efficient training, and (3) expressive generative models [22, 23, 24] to capture the multimodality of the data. Also, inspired by current behavioral cloning approaches [22, 25] , we train policies that output sequences of actions rather than single-step actions and execute them in chunks. Representation Learning. We pre-train an observation encoder over the piano state ˇ" ( to learn spatially consistent latent features. We hypothesize that two piano states that are spatially close should lead to latent features that are close. Using these latent features as a target should lead to better generalization. To obtain the observation encoder, we train an autoencoder with a reconstruction loss over a Signed Distance Field (SDF) defined on the piano state. Specifically, the encoder compresses the binary vector of the goal into a latent space, while the decoder predicts the SDF function value of a randomly sampled query point (the distance between the query point and the closest "on" piano key). For the BC policy, we concatenate L-timestep desired piano states and pass through the pre-trained observation encoder to obtain the latent goal representation. We provide more details in Appendix G. Hierarchical Policy. We represent the piano-playing agent with a hierarchical policy. The highlevel fingertip policy takes a sequence of desired future piano states ˇ" ( and outputs a trajectory of human fingertip positions x. Then, a low-level inverse model takes the fingertip and piano state trajectories as input and outputs a trajectory of desired joint angles q. On the one hand, while fingertip trajectory data is readily available from the Internet, obtaining low-level joint trajectories requires solving a computationally expensive RL problem. On the other hand, while the high-level mapping ( ˇ" ( Þ Ñ x) is complex and involves fingerings, the low-level mapping (x Þ Ñ q) is relatively simple, involving a task space to configuration space mapping. This decoupling allows us to train the more complex high-level mapping on large, cheap datasets, and the simpler low-level mapping on smaller, expensive datasets. We visualize the policy in Figure 2 . Expressive Generative Models. Considering that the human demonstration data of piano playing is highly multimodal, we explore the use of expressive generative models to better represent this multimodality. We compare the performance of different deep generative models based policies, such as Diffusion Policies [22] and Behavioral Transformer [23] , as well as a deterministic policy. 
experimental results: 
 We divide the experimental evaluation into three parts. In the first part, we investigate the performance of our proposed framework in learning song-specific policies via RL. In the second part, we perform ablation studies on policy designs for learning a generalist piano-playing agent by distilling the previously learned policies via BC. Finally, in the third part, we study the influence of the amount of training data on the generalization capabilities. Dataset and Evaluation Metrics All experiments are performed on our collected dataset, which contains the notes and corresponding demonstration videos and fingertip trajectories of 60 piano songs from a Youtube channel, PianoX1 . To standardize the length of each task, each song is divided into several clips, each 30 seconds long (the dataset contains a total of 431 clips, 258K state-action pairs). In addition, we select 12 unseen clips to investigate the generalization ability of the generalist policy. These clips consist of completely new songs that do not appear in the training dataset. We use the same evaluation metrics from RoboPianist [11] , i.e., precision, recall, and F1 score (see Appendix B). We run each policy for the whole song and evaluate its performance. Simulation Environment Our experimental setup uses the ROBOPIANIST simulation environment [11] , implemented in the Mujoco [26] . The agent predicts target joint angles at 20Hz, and the targets are converted to torques using PD controllers running at 500Hz. The notes of the songs are also discretized at a frequency of 20Hz. We use the same setup as [11] with two modifications: 1) The z-axis sliding joints attached to both forearms are enabled to allow more versatile hand movements.2) We increase the proportional gain of the PD controller for the x-axis sliding joints to allow faster horizontal movement, which we feel is essential for some fast-paced piano songs.In this section, we evaluate the song-specific policy learning and aim to answer the following questions: (1) Does the integration of human demonstrations with RL help to achieve better performance? (2) Which elements of the learning algorithm are most important for good performance?We use Proximal Policy Optimization (PPO) [27] because we found that it performs best compared to other RL algorithms. We compare our model to two baselines: RoboPianist [11] We use the RL method introduced in [11] . We keep the same reward functions as in the original work and manually label the fingering from the demonstration videos to provide the fingering reward.Inverse Kinematics (IK) [21] Given a fingertip trajectory demonstration τ x , a Quadratic Programming-based IK solver [21] is used to compute a target, joint position trajectory and execute it open-loop. We select 10 clips from the collected dataset with different levels of difficulty. We individually train specialized policies for each of the 10 clips using both the baselines and our method. We then evaluate and compare their performance based on the obtained F1 score.Performance. As shown in Figure 3 , our method consistently outperforms the RoboPianist baseline for all 10 clips, achieving an average F1 score of 0.94 compared to the baseline's 0.74. We attribute this improvement to the incorporation of human priors, which narrows the RL search space to a favorable subspace, thereby encouraging the algorithm to converge on more optimal policies. In addition, the IK method achieves an average F1 score of 0.70, only slightly lower than the baseline. This demonstrates the effectiveness of incorporating human priors, which provides a strong starting point for RL. We also observe that our method trains faster than the baseline. On an RTX 4090, the baseline took an average of 4 hours to train, while our method took an average of 2.5 hours. Impact of Elements. Our RL method has two main elements: a style-mimicking reward and a residual learning. We exclude each element individually to study their respective influences on policy performance (see Figure 3 ). We clearly observe the critical role of residual learning, which implies the benefit of using human demonstrations as nominal behavior. We observe a marginal performance increase of 0.03 when excluding the style-mimicking reward, but this also results in a larger discrepancy between the robot and human fingertip trajectories. Thus, the weight of the style-mimicking reward can be considered as a parameter that controls the human similarity of the learned robot actions. The ablation study for this weight is discussed in Appendix F. Hand Pose Visualization. 2 We provide Figure 3 and Figure 4 as an example of hand poses in different settings and provide attached videos on the website with further examples. In Figure 4 , we exemplify that our policy places the hands in similar poses to the YouTube videos. We measure the distance between fingertips in YouTube videos and the robot in Appendix F. We observe that the IK nominal behavior leads the robot to place the fingers in positions similar to those in YouTube videos.The RL policy then slightly adapts the fingertip positions to press the keys correctly. Besides, we observe that the RoboPianist baseline sometimes presents visually unhuman-like motions. For ex- ample, in Figure 3 Left, the middle finger and the ring finger of the left robot hand are at relatively unhuman-like positions.This section focuses on the evaluation of the policy distillation for playing different songs. We evaluate the influence of different policy design strategies on the agent's performance. We aim to assess (1) the impact of integrating a pre-trained observation encoder to induce spatially consistent features, (2) the impact of a hierarchical design of the policy, and (3) the performance of different generative models on piano-playing data.Proposed Models. We propose two base policies, Two-stage Diff and Two-stage Diff-res policy. Both use hierarchical policies and goal representation learning, as described in Section 3.3. The only difference between them is that the low-level policy of Two-stage Diff predicts the target joints directly, while Two-stage Diff-res predicts the residual term of an IK solver. A detailed description of the policies can be found in Appendix I.Baselines. We consider as baselines a Multi-task RL policy and a BC policy with MSE Loss from [11] . Additionally, we implement an Adversarial Inverse Reinforcement Learning (AIRL) baseline [28] . We provide further details of the models in Appendix I.Ablation Models. To analyze the impact of our policy design choices, we design three variants of our proposed model, i.e., w/o SDF: We train a policy that directly receives the 88-dimensional binary representation of the goal, without using the SDF observation encoder, to evaluate the impact of the goal's representation learning, One-stage: We train an end-to-end diffusion policy to evaluate the impact of the hierarchical architecture, BeT: We train a two-stage Behavior-Transformer [23] to evaluate the impact of using diffusion models.Results. As shown in Table 1 , our methods (Two-stage Diff and Two-stage Diff-res) outperform the others on both training and test datasets. Multi-task RL and AIRL have higher precision on the test dataset, but this is because they barely press any keys. We observe a large improvement when using both diffusion policies instead of BeT, a hierarchical policy instead of an end-to-end policy and a slight improvement when using a pre-trained observation encoder, especially on the test dataset. We also observe a slight performance improvement when the model predicts the residual term of IK (Two-stage Diff-res).In this section, we investigate the impact of scaling the training data on the generalization capabilities of the agent. We divide the experiments into two parts:(1) We evaluate the impact of scaling the training data on the performance of three policy designs (One-stage Diff, Two-stage Diff, and Two-stage Diff-res) evaluated on the test dataset by training them with different proportion of the dataset (see Figure 5 Top).(2) We evaluate the influence of a good high-level policy of Two-stage Diff by training different high-level policies on different proportions of the dataset (see Figure 5 Bottom). We provide additional results in Appendix M. Impact of high-level policy quality. We further employ different combinations of the high-level and low-level policies of Two-stage Diff trained with different proportions of the dataset and assess their performance. In addition, we introduce a high-level oracle policy that outputs the ground-truth fingertip positions from the human demonstration videos.The results (see Figure 5 Bottom) demonstrate that the overall performance of the policy is significantly influenced by the quality of the high-level policy. Low-level policies paired with Oracle high-level policies consistently outperform the ones paired with other high-level policies. Besides, we observe early performance convergence with increasing training data when paired with a low-quality high-level policy.Inference Speed One of the limitations is the inference speed. The models operate with an inference frequency of approximately 15Hz on an RTX 4090 machine, which is lower than the standard real-time demand on hardware. Future works can employ faster diffusion models, e.g., DDIM [29] , to speed up the inference.Out-of-distribution Data Most of the songs in our collected dataset are of modern style. When evaluating the model on the dataset from [11] , which mainly contains classical songs, the performance degrades. It implies the model's limited generalization across songs of different styles. Future work can collect more diverse training data to improve this aspect. Acoustic Experience Although the policy achieves up to 57% F1-score on unseen songs, we found that higher accuracy is still necessary to make the song acoustically appealing and recognizable. Future work should focus on improving this accuracy to enhance the overall acoustic experience. 
conclusion: 
 In this work, we present PianoMime, a framework for training a generalist robotic pianist using Internet video sources. The proposed framework is composed of three distinct phases: first, extract task-related and human motion-related trajectories from videos, second, train song-specific policies with reinforcement learning and finally, distill all the song-specific policies in a single generalist policy. We found that the resulting policy demonstrates an impressive generalization capability, achieving an average F1-score of 57% on unseen songs. We believe that the findings for learning fine motor skills in piano playing can be applied to other tasks that require high dexterity and precision, and scenarios where robot data collection through teleoperation is challenging. A Retargeting: From human hand to robot hand 
 
 Appendix: Youtube MujocoTo retarget from the human hand to the robot hand, we follow a structured process.Step 1: Homography Matrix Computation Given a top-view piano demonstration video, we firstly choose n different feature points on the piano. These points could be center points of specific keys, edges, or other identifiable parts of the keys that are easily recognizable (see Figure 6 ). Due to the uniform design of the pianos, these points represent the same physical positions in both the video and Mujoco. Given the chosen points, we follow the Eight-point Algorithm to compute the Homography Matrix H that transforms the pixel coordinate in videos to the x-y coordinate in Mujoco (the z-axis is the vertical axis).Step 2: Transformation of Fingertip Trajectory We then obtain the human fingertip trajectory with MediaPipe [20] . We collect the fingertips positions every 0.05 seconds. Then we transform the human fingertip trajectory within pixel coordinate into the Mujoco x-y 2D coordinate using the computed homography matrix H.Step 3: Heuristic Adjustment for Physical Alignment We found that the transformed fingertip trajectory might not physically align with the notes, which means there might be no detected fingertip that physically locates at the keys to be pressed or the detected fingertip might locate at the border of the key (normally a human presses the middle point on the horizontal axis of the key). This misalignment could be due to the inaccuracy of the hand-tracking algorithm and the homography matrix. Therefore, we perform a simple heuristic adjustment on the trajectory to improve the physical alignment. Specifically, at each timestep of the video, we check whether there is any fingertip that physically locates at the key to be pressed. If there is, we adjust its y-axis value to the middle point of the corresponding key. Otherwise, we search within a small range, specifically the neighboring two keys, to find the nearest fingertip. If no fingertip is found in the range or the fingertip has been assigned to another key to be pressed, we then leave it. Otherwise, we adjust its y-axis value to the center of the corresponding key to ensure proper physical alignment.Step 4: Z-axis Value Assignment Lastly, we assign the z-axis value for the fingertips. For the fingertips that press keys, we set their z-axis values to 0. For other fingertips, we set their z-axis value to 2 ¨hkey , where h key is the height of the keys in Mujoco.We use the same metrics from RoboPianist [11] , i.e., Precision, Recall, and F1 score. Here we provide a detailed definitions of them:• True Positive (TP): Keys that should be pressed are pressed.• False Positive (FP): Keys that should not be pressed are pressed.• False Negative (FN): Keys that should be pressed are not pressed.Precision " T P T P `F PRecall " T P T P `F NF1 " 2 ¨Precision ¨Recall Precision `RecallGiven the ground truth and the executed piano state trajectory, we calculate Precision, Recall, and F1 score for each timestep. We then get the overall Precision, Recall, and F1 score by averaging them over timesteps. In this way, precision evaluates the robot's capability of avoiding pressing the wrong keys, while recall evaluates the robot's capability of pressing the correct keys. F1 score combines both of them.The implementation of the IK solver is based on the approach of [21] . The solver addresses multiple tasks simultaneously by formulating an optimization problem and finding the optimal joint velocities that minimize the objective function. The optimization problem is given by:EQUATIONwhere w i is the weight of each task, K i is the proportional gain and v i is the velocity residual. We define a set of 10 tasks, each specifying the desired position of one of the robot's fingertips. We do not specify the desired quaternions. All the weights w i are set to be equal. We use quadprog 3to solve the optimization problem with quadratic programming. The other parameters are listed in Table 2 . We present a detailed representation of the reward functions applied in our method in Table 3 .We use PPO [27] (implemented by StableBaseline 3 [30] ) to train the song-specific policy with residual RL(See Algorithm 1). All of the experiments are conducted using the same network architecture and tested using 3 different seeds. Both actor and critic networks are of the same architecture, containing 2 MLP hidden layers with 1024 and 256 nodes, respectively, and GELU [31] as activation functions. The detailed hyperparameters of the networks are listed in Table 6 .Table 3 : The detailed reward function to train the song-specific policy. The Key Press reward is the same as in [11] , where k s and k g represent the current and the goal states of the key respectively, and g is a function that transforms the distances to rewards in the [0, 1] range. p df and p rf represent the fingertip positions of human demonstrator and robot respectively.Reward Formula Weight Explanation Key Press 0.5 ¨gp}k s ´kg } 2 q `0.5 ¨p1 ´1false positive q 2/3 Press the right keys and only the right keys Mimic gp}p df ´prf } 2 q 1/3 Mimic the demonstrator's fingertip trajectory To numerically evaluate the human-likeness of the robot motion, we include an additional metric, ∆f t, which computes the average Euclidean distance between the robot fingertip positions and the human demonstrators for each timestep. We further make an ablation study to explore the impact of the weight of style-mimicking reward on ∆f t and F1 score, respectively (See Figure 7 ). The result indicates that when the weight of the mimic reward is zero, the F1 score is the highest, but the relative distance between the human fingertip positions and the robot's fingertip positions is also the greatest. As we increase the influence of the mimic reward, the performance decreases, while the relative distance to the human fingertip positions also diminishes. This allows us to balance between improving performance and achieving a behavior more similar to the videos by adjusting the mimic reward. The discrepancy is unavoidable since the robot's embodiment differs from that of a human, and accurately playing the piano song might necessitate some deviation from human behavior.We train an autoencoder to learn a geometrically continuous representation of the goal (See Figure 8 and Algorithm 2). During the training phase, the encoder E, encodes the original 88-dimensional binary representation of a goal piano state ˇ" ( t into a 16-dimensional latent code z. The positional encoding of a randomly sampled 3D query coordinate x is then concatenated with the latent code z and passed through the decoder D. We use positional encoding here to represent the query coordinate more expressively. The decoder is trained to predict the SDF f px, ˇ" ( t q. We define the SDF value of x with respect to ˇ" ( t as the Euclidean distance between the x and the nearest key that is supposed to be pressed in ˇ" ( t , mathematically expressed as: SDFpx, ˇ" ( t q " min pPtpi| ˇ" ( t,i"1u }x ´p},where p i represents the position of the i-th key on the piano. The encoder and decoder are jointly optimized to minimize the reconstruction loss:Lpx, , ˇ" ( t q " pSDFpx, ˇ" ( t q ´DpEpv, xqqq Compute the prior control signal that tracks x t with the IK controller ũt " ikpx t , o t q 8:Run policy to get the residual term r t " π θ po t q 9:Compute the adapted control signal u t " ũt `rt for n " 1 : N do 14:Sample a batch of transitions tps j , u j , r j , s j`1 qu from the collected trajectories 15:Update the actor and critic network with PPO We pre-train the autoencoder using the GiantMIDI dataset4 , which contains 10K piano MIDI files of 2,786 composers. The pre-trained encoder maps the ˇ" ( t into the 16-dimensional latent code, which serves as the latent goal for behavioral cloning. The encoder network is composed of four 1D-convolutional layers, followed by a linear layer. Each successive 1D-convolutional layer has an increasing number of filters, specifically 2, 4, 8, and 16 filters, respectively. All convolutional layers utilize a kernel size of 3. The linear layer transforms the flattened output from the convolutional layers into a 16-dimensional latent code. The decoder network is an MLP with 2 hidden layers, each with 16 neurons. We train the autoencoder for 100 epochs with a learning rate of 1e ´3. for j " 1 : N batch do 5: Two-stage Diff. The model consists of a hierarchical policy with a pre-trained goal observation encoder, as described in Section 3.3. Note that the entire dataset is used for training the high-level policy, while only around 40 % of the collected clips (110K state-action pairs) are trained with RL and further used for training the low-level policy. The detailed network implementation is described in Appendix H. Two-stage Diff w/o SDF. We directly use the binary representation of the goal instead of the SDF embedding representation to condition the high-level and low-level policies.The model is close to Two-stage Diff, with slight changes. We employ an IK solver to compute the target joints given the fingertip positions predicted by the high-level policy. The low-level policy predicts a residual term around the IK solution instead of the robot's actions.Two-stage BeT. We train both high-level and low-level policies with Behavior Transformer [23] instead of DDPM. The hyperparameter of Bet are listed in Table 8 .One-stage Diff. We train a single diffusion model to predict the robot actions given the SDF embedding representation of goals and the proprioception state.Multi-task RL. We create a multi-task environment where for each episode a random song is sampled from the dataset. The observation and action space, as well as the reward function of the environment, follow the same settings as described in [11] . Consequently, we use Soft-Actor-Critic (SAC) [34] to train a single agent within the environment. Both the actor and critic networks are MLPs, each with 3 hidden layers, and each hidden layer contains 256 neurons.We train a feedforward network to predict the robot action of the next timestep conditioned on the binary representation of goal and proprioception state with MSE loss. The feedforward network is an MLP with 3 hidden layers, each with 1024 neurons. AIRL [28] . We use the same multi-task environment as Multi-task RL. We use an opensource implementation of AIRL based on PPO 5 , where the actor and critic networks in PPO are MLPs, each consisting of three hidden layers with 256 neurons per layer. The reward and shaping term of the discriminator also use the same MLP architecture, with three hidden layers and 256 neurons in each layer. We collect expert state-action pairs by rolling out the song-specific policies, where the state consists of the song's notes and proprioceptive state, and the action is the joint-space robot action. The collected data is then fed into the discriminator. In Table 9 and Table 10 , we show the Precision, Recall, and F1 score of each song in our collected test dataset and the Etude-12 dataset from [11] , achieved by Two-stage Diff and Two-stage Diff-res, respectively. We observe an obvious performance degradation when testing on Etude-12 dataset. We suspect that the reason is due to out-of-distribution data, as the songs in the Etude-12 dataset are all classical, whereas our training and test dataset primarily consists of modern songs.8th Conference on Robot Learning (CoRL 2024), Munich, Germany.https://www.youtube.com/channel/UCsR6ZEA0AbBhrF-NCeET6vQThe key colors in Figureand Figuremean the following: green indicates a correctly pressed key, yellow indicates a key that should be pressed but is not, and red indicates a key that should not be pressed but is.https://github.com/quadprog/quadproghttps://github.com/bytedance/GiantMIDI-Pianohttps://github.com/toshikwa/gail-airl-ppo.pytorch 


Decision: **Paper summary**

This paper presents a method for learning piano-playing policies from internet videos as demonstrations. The results indicate that the proposed method is generalizable to new songs and performs well in comparison to baseline RL methods.

**Review summary**

Summary of strengths:
+ The paper addresses an important challenge: leveraging the large amounts of data available through internet videos for robot policy learning.
+ The evaluation demonstrates that the proposed method is effective in modeling human-likeness, performance, and generalization from the training data.
+ The evaluation is thorough and is performed over multiple ablations of the system.

Summary of weaknesses:
- The reviewers (CxL8 and gJnu) note several key clarifications that are necessary for fully understanding the paper. The evaluation analysis needs more explanation and structure (Reviewer Hkur). **[This has been addressed by the rebuttal and revised paper.]**
- The results may not be accurate due to an error in computing F1 scores, as noted in the appendix. The method for computing these metrics is also missing. **[This has been addressed by the rebuttal and revised paper.]**
- The evaluation would be improved with a real-world evaluation and an additional baseline (inverse RL from raw data). **[The authors have added AIRL as a baseline.]**


**Response to rebuttal**

Two of the reviewers have increased their recommendations from weak reject to weak accept, stating that their concerns were addressed by the author rebuttal. The third reviewer maintains their recommendation of weak accept. 

**Recommendations for improvement**

* The paper would be further strengthened by the inclusion of DAgger as a baseline. 
* The human-subjects study should not be included in this paper. The authors are encouraged to revisit this in future work with IRB approval and a more rigorous survey scale and analysis.

Reviews: review: title: title: This paper presents a novel framework for teaching agents to play the piano using YouTube video demonstrations. It shows significant improvements in performance and human-likeness through innovative methods like residual policy architecture and style-mimic rewards. While the paper is well-structured, the analysis is complex and somewhat difficult to follow. Overall, the work makes a substantial and original contribution to robot learning.
paper_summary: paper_summary: The paper utilized online piano playing demonstrations to extract relevant data, developed song-specific policies through reinforcement learning (RL), and subsequently employed behavior cloning techniques to enhance generalization across different pieces (including unseen songs).
main_review: main_review: Quality: The paper is well-structured, presenting a comprehensive framework for learning piano playing through YouTube video demonstrations. The methodology is robust, and the results are thoroughly analyzed, despite the complexity.

Clarity: The paper is generally clear and well-organized. However, the analysis section is somewhat overwhelming due to the multitude of factors considered, making it difficult to trace the results easily.

Originality: The work is highly original, leveraging publicly available online demonstrations to teach agents complex tasks like playing the piano. The use of hierarchical policies and goal representation learning to improve generalization is particularly innovative.

Significance: This work is significant as it advances the field of robot learning by demonstrating a practical application of reinforcement learning and online demonstrations. It has the potential to impact future research and practical applications in robotics and artificial intelligence.

Strengths:
Innovative Approach: Utilizing YouTube videos for training agents is a novel and creative idea.
Improved Performance: The residual policy architecture and style-mimic reward clearly enhance performance and human-likeness.
Generalization: The method effectively generalizes policies using hierarchical policies and goal representation learning.
Clear Structure: The paper is well-organized, with a consistent narrative and clear presentation of the methodology and results.

Weaknesses:
Complex Analysis: The analysis of the results is dense and somewhat difficult to follow due to the numerous factors involved.
Overwhelming Data: The extensive data and variables considered in the analysis can be overwhelming for the reader, potentially obscuring key findings.
Reasoning: Why/how better agents learning how to play piano could be beneficial to other robot learining areas could be discussed more.

Overall, this paper presents a significant and original contribution to the field of robot learning, with clear improvements in performance and generalization. However, simplifying the analysis or providing clearer explanations could enhance the clarity and accessibility of the results.
questions: questions: The performance of the model during reinforcement learning (RL) shows that the version without the mimic component achieves a better F1 score. Later, the paper discusses the style-mimicking reward as a parameter influencing the human-like quality of the robot's actions. To strengthen the argument, the paper should include a metric or evaluation demonstrating the significance of the style-mimicking reward and its optimal weight. Specifically, it would be beneficial to quantify how much the mimic score impacts the overall performance and determine the appropriate weight for this reward.

Additionally, the paper would benefit from incorporating evaluations involving real people to assess the performance of the piano-playing agent. This would provide a practical metric for the effectiveness of the learned robot actions and validate the human-likeness of the performance in a real-world context.
limitations: limitations: 3
review_summary: review_summary: Based on the evaluation, I recommend accepting the paper with minor revisions. The paper presents an innovative framework for teaching agents piano playing using YouTube videos, with notable improvements in performance and human-likeness through advanced techniques like residual policy architecture and style-mimic rewards. However, the analysis is complex and challenging to follow, and the paper lacks a user study involving real humans. Additionally, it does not clearly demonstrate how learning to play the piano can benefit other areas of robot learning. Addressing these issues will enhance the paper's clarity and broader relevance.
title: title: This work has novel learning system design for teaching dexterous robotic hands to play piano.
paper_summary: paper_summary: This work uses human demonstrations from internet to teach two dexterous robotic hands to play piano in simulation. The authors introduced a RL-based song-specific policy learning, and further presented a generalizable policy learning with behavior cloning. The proposed learning framework has been tested on unseen songs and can achieve 57%  F1 score.
main_review: main_review: This work provides a novel framework for piano playing with the combination of song-specific RL policies and generalized BC policies. Learning from human demonstrations on YouTube with top-down views seems natural and efficient to leverage human priors. I believe that the hierarchical system structure can be potentially extended to other dexterous manipulation learning. 

The experiments are thorough and the authors properly evaluated all modules in the system. However, in the Appendix, the authors said there were errors in F1 scores for all methods in the paper. I am concerned about whether the statements from the results will still hold with updated scores. 

The paper clarification needs a lot of improvement. I would suggest the authors to proofread the draft. Some issues are listed here: 
- Please be consistent about terminologies in the paper such as "Internet" vs "internet".
- It seems “Inverse Kinematics”, “Reinforcement Learning”, “Markov Decision Process”, “Behavioral Cloning” are highlighted and linked to empty links.
- What does “cross domain inverse dynamics model” mean? Is it "inverse kinematics model"? I don’t think there is anything about dynamics here since only kinematics are used to map ee poses to the joint poses.
- What is “FK” in Figure 2? Is it “Forward Kinematics”? 
- Broken sentences: 
“At each timestep, the agent receives as goal input a trajectory of the keys to be pressed.”
“we combine both reinforcement learning with imitation learning ”
“Given a mocap demonstration, it has been common to exploit the demonstration rather as a reward function [5, 19] or as a nominal behavior for residual policy learning [18, 6]. ”
“We select the fingertip position as the essential signal to mimic with the robot hand ”
questions: questions: - What are the major failure modes for the policy distillation? I understand that the test data (unseen clips) might be out-of-distribution, but how different are they from the training data?
- What do the colors, i.e. green, yellow, and red represent in the visualization? 
- Please also report the training time and potentially compare it with baseline methods to support the statement: "In contrast with previous approaches, our approach exploits YouTube piano-playing videos, enabling faster training and more accurate and human-like robot behavior.".  
- Please correct the F1 scores and double-check the analysis of the experiments.
limitations: limitations: 2
review_summary: review_summary: I recommend to accept the paper for the novel system design and thorough experiments. The authors addressed my questions and concerns during the rebuttal.
title: title: Review
paper_summary: paper_summary: This paper proposes a framework, PianoMime, that can learn from demonstration and reinforcement learning, to play on piano a variety of songs in a simulated environment. The policy is hierarchical in that it takes as input a sequence of notes to play (MIDI File) and outputs fingertip positions for the robot; next, the fingertip positions are passed to a network that outputs the joint commands, q. The paper compares to baselines in a standard environment and shows strongly positive results against baselines.
main_review: main_review: Strengths:

-The results are generally positive, and ablations are included, such as removing distillation, stages, and SDF. However, it is unclear whether there is still an encoder.

-The paper is easy to read, though, there are missing details noted below.

-The design of the algorithmic system is intuitive and simple.

Weaknesses:

-Please see the questions below, as the paper is omitting important details to help review the paper.

-The goal representation learning is primarily a neural network encoder using a Signed Distance Field (SDF). Removing the SDF makes the approach a little bit worse, but it is unclear what "w/o SDF" means precisely. 

-Overall, the paper appears to be a combination of known, existing techniques and does not have an innovation at the algorithmic level but rather at a systems level. Ablation studies support the design of the system, so there is an incremental, positive improvement.

-It would have been helpful to evaluate the approach in the real-world.

-The approach is missing a baseline for inverse reinforcement learning from the raw data; a baseline for DAgger during policy distillation would also help as would using RL fine-tuning after distillation.

-The qualitative results (Figure 3) are not insightful other than to show that "ours" is better than a baseline. More insights into failure cases would be useful.

-The metrics used in this paper appear to be one-step prediction metrics for precision, recall, and F1-score. It is unclear precisely how these metrics are computed. An issue here, though, might be that the metrics assume "teacher forcing" in that the robot may be put into the correct state at time t, then the model asked to predict and implement the next sound (unclear what exactly is being measured), and then that is it. It would be much more informative to actually have the robot play an entire MIDI file open-loop as if were a real performance and then measure the distance between the ground-truth MIDI file and then the MIDI file that the robot produced. I imagine the performance degradation would be significant and would give a much better assessment of which approach is actually best and playing a song.

-The paper does not appear to measure how well the robot captures the finger movement / style of the human player and instead seems to focus on notes; however, this is a bit unclear. For example, a human-expert eval to compare the style of the robot's play for anthropomorphic properties would be helpful.

Writing:

-Some formatting issues exist, such as no spacing between paragraphs in Section 3.1.

================ Post-Rebuttal =============================

Based upon the rebuttal, I have decided to move to a weak accept.
questions: questions: -The specifics of the music trajectory are unclear. There are orders of magnitude difference in the wall-clock time between a whole note tied to another whole note in a next bar vs. a 64th note. The MIDI file should account for things like staccato and slur, but the discretization of time here is key. It would be helpful if more details are provided.

-Why was imitation learning (DAgger) not used instead of BC?

-The performance improvement by doing distillation is quite modest and not uniformly better. Why is that, and why then include it as a contribution?

-What would happen if you further fine-tuned the distilled policy using RL?

-It would have been helpful to report precisely how precision, recall, and F1-score are being reported. Also, are these averaged over all classes? In the case of class imbalance, this may skew the results and make it harder to understand when/where the model performs well or not.

-In Figure 2, is the observation encoder only receiving one discrete note unit at time, t, or is it receiving a sequence (i.e., the goal)?
limitations: limitations: 2
review_summary: review_summary: The paper is lacking numerous details; the work is positive but more of a collection of existing techniques; and it may not have employed sufficiently robust evaluation metrics.