Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The authors present an RL based agent that is trained to play piano using bi-manual anthropomorphic hands. The approach uses DroQ as backbone model. The authors contribute by: (1) developing a novel environment; (2) analyzing several choices for observation and reward shaping; (3) proposing a multi-task imitation learning setting for training a single agent able to play multiple songs. This paper may represent a starting point for future research in several domains, such as robotics, machine learning and arts. 
review: The paper is clear and the approach is sound. The authors address several problems that arise in the development of the agent and propose effective solutions. The contributions are clear and set the basis for interesting future work.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper is clear and the approach is sound. The authors address several problems that arise in the development of the agent and propose effective solutions. The contributions are clear and set the basis for interesting future work.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: Existing simulated tasks for studying dexterous manipulation do not require diverse bimanual behaviors. The authors propose a benchmark consisting of playing 150 piano songs with two Shadow hands. They evaluate this RoboPianist trained with DroQ and find that it outperforms an MPC baseline. Extensive ablations show the impact of various design choices in simulation, reward, and optimization hyperparameters. The 150 songs are shown to have diverse difficulties and are distinct enough that generalizing to a new song or learning multiple songs from scratch is difficult.
review: Strengths:
- This paper is dense and its contributions to the environment design for a novel task are significant. Designing the environment and task specification from scratch for such a complex task is a large effort. Its release will allow the community to study bimanual control with a dramatically lowered barrier to entry.
- The evaluation and ablations of the policy optimization and baseline are very thorough. They evaluate the impact of many interesting hyperparameters and design choices in good detail. 
- The question of generalization across piano songs is interesting to explore. I agree with the assertion in Section 2 that traditional manipulation skills have a dominant "basin of attraction" while piano playing requires a more open-ended and diverse set of motor skills. And because the tasks are shown to be very diverse in difficulty and character, any method that improves generalization in this space must be more powerful. That seems like a quality of a good benchmark.

Weaknesses:
- The policy optimization approach departs from all prior references on learning dexterous manipulation by using DroQ instead of PPO. The motivation behind this design choice should be elaborated in the text.
- There is no sim-to-real evaluation. Also, no dynamics randomization, sensor noise, or lag modeling is mentioned. Also, the observation includes the piano key joints, which would be hard to sense in the real world. The benchmark could be strengthened by introducing these aspects. Otherwise, I am not sure if advancing progress on this benchmark brings us closer to a real robot that can play the piano.
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: RoboPianist makes a novel high-dimensional control task accessible to practitioners and thoroughly evaluates a strong baseline for performance. Clarifying the motivation behind atypical algorithm selection and adding an expanded discussion of sim-to-real feasibility will resolve some outstanding questions.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper introduces a simulation setup for bi-manual anthropomorphic hands for the task of playing a piano. The task is intricate in itself, requiring coordination among the fingers, planning for efficiently performing the next set of actions, as well as spatial and temporal precision for hitting the right set of notes with appropriate intensity, simultaneously and at multiple stages. The key challenge as highlighted in the paper is the high-dimensional controls for dexterous manipulation which involve continuous on-and-off contact points between the piano keys and the robot hands. The paper proposed a deep reinforcement learning framework which can learn a collection of 150 piano pieces, called the ROBOPIANIST-REPERTOIRE-150 with both, single as well as multi-task policies which can learn to play more than 1 song in a single policy.
review: #### Strengths: 

- The paper is well-written and the introduction section illustrates clarity about why this problem is important and challenging. Moreover, the problem setting is interesting and different, compared to a majority of tasks usually tackled under the umbrella of dexterous manipulation.
The simulation is built and released with an open-source physics simulator (MuJoCo), that makes it easier to reproduce and build on the proposed approach.
- The Robopianist-repertoire-150 is open-sourced, which helps in evaluating the proposed results in detail and also provides a good benchmark for future research in this direction. 
- The paper discusses the generalization capabilities of the proposed framework in detail. While they show that multi-task RL does not show any benefit for generalization, multi-task behavioral cloning policy performs well and does show generalization capabilities with more data and larger models.
- The example result provided is an impressive qualitative depiction of the proposed methodology. 

#### Weaknesses:

- The results on the website, although impressive, only show very few examples in a single video. Moreover,  the link provided does not provide code or dataset, unliked claimed in the abstract.
- The demo video does not mention the playback speed, which makes it harder to infer the actual efficacy and latency of the system. Also, no failure results are discussed on published on the website. It would be good if the author can provide a more detailed analysis of these results, commenting on when and why the system fails to better understand the current shortcomings.
- The current system assumes the availability of the fingering pattern for finger-to-key assignment, similar to (speed) typing. As pointed out by the author, without this information it is hard to optimize for emergent behaviors with sparse rewards. Although this can result in limited behavior emergence through deep RL and is left for future work to explore, it would be good to see some baseline results as an ablation study without these assumptions. This can provide a better picture for qualitative and quantitative comparisons, and also justify the proposed design choices more profoundly.
- The MPC baseline in Figure 3 is hypothesized to be much worse due to the computational bottleneck of the planner. Can the author explain and show if MPC works comparably or not with the RoboPianist for easier song pieces with fewer notes? 
- No discussion is available for transfer to real world and the challenges that might accompany.

##### Minor errors:
- Figure 10 in the supplementary is missing a caption. 
- L266 contains an error/type in the sentence - “A control time step that is too large can result in making it impossible…”

robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper poses an interesting problem setup, despite less novelty. It would be good to address the comments posted above. I recommend accepting the paper, but am keen to see some results with the real robot hands and piano setup.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: - This paper aims to challenge learning-based dexterity with the task of piano playing with bi-manual anthropomorphic hands in simulation.
- A new simulation environment is created, powered by the MuJoCo engine with two Shadow Hands and a simulated piano.
- A state space and reward function for RL is designed and justified.
- Evaluation is based on the alignment between key presses and the "piano rolls" extracted from the corresponding MIDI files.
- A dataset of 150 music pieces is constructed for training and evaluation. 
- A standard RL algorithm (DroQ) is adopted for policies training.
- Different training paradigms are explored, including training one policy per piece, training one policy for multiple pieces, and training per piece experts followed by distillation to multi-piece policies using behavior cloning (BC).
- Multitask RL polices struggles with training, while per task policies and multitask BC can excel to a large extent.
- Zero-shot generalization remains challenging.
review: ### Strengths
- The paper is looking at a potentially profoundly interesting problem (i.e., robot pianist), which can go beyond just the topic of dexterous control but more deeply into the space of AI creativity and music performing arts. For example, can a robot play classical pieces in idiosyncratic but acoustically pleasant manner? Progress on this front could even make impact in the music domain ultimately.
- Experiments are thorough and well executed. Technical and experimental details are well presented.
- The paper is quite open and clear about the limitations of it's approach and the limitations of the results.
- Overall, the paper is well written and a pleasant read. Figures and illustrations are beautifully made.

### Weaknesses
- Certain contributions are over-claimed and not well-supported with proper empirical evidence. The abstract claims that the proposed method can "produce visually and acoustically pleasing performances" (L11). The same claim is also made in the intro section (L42: "... producing visually and acoustically pleasing performances,"). But there is no evaluation nor any reported metrics on visual or acoustical "pleasantness". It is also rather counter-intuitive that DRL can learn "pleasing" performance without training with any human feedback.
- Following the last point, one expected metric for "naturalness" or "pleasantness" is through human evaluation. However, no human evaluation has been provided. Also, from my little personal biased view, I find the provided examples, though impressive in some way, still not quite "acoustically pleasant" yet. It still makes some entry-level mistakes that are common among piano beginners.
- Without evaluating at the level of "artisticness" or "pleasantness", the task is only aimed to be solved at a "control" level, i.e. aligning key presses with the piano roll, which really downplays the artistic side of the task, and rather unuseful. One can build mechanical systems that can "play piano" with precise key strokes---why bother with human-like hands?
- Speaking of merely "pushing robot dexterity", the result is rather unexciting. Standard RL can nail the task to a large extent but that can also somewhat be foreseen. Generalization to unseen pieces would be a significant deal, but this paper is not there yet still.
- Also on the note of "pushing dexterity", the techniques presented in this paper seems very specific to the task of piano playing (e.g., state space and reward defined on key presses, fingering, etc.). It is unclear how easy to adapt when one wants to solve a different dexterous tasks.
- One little thought on the training of multitask RL: rather than "learning multiple songs simultaneously" (Fig. 5), why not "progressively". I.e., start from one song; once converged, add a new song and continue training on both songs, etc. I thought this was what the authors intended to do but it looks like Fig. 5 is comparing training in different $N$-song envs "from scratch".

### Question and Suggestions
- [L90] "We use the Musical Instrument Digital Interface (MIDI) standard to represent musical pieces and synthesize sounds." -> Sentence duplicated with L84-85.
- [L112] "We define actions to be target joint angles for the 22 joints in each hand ... convert them to torques at the 52 hand joints." -> $52 - 22 \times 2 = 8$. What do these 8 extra joints account for? Movement of the forearms' base?
- [Fig. 2] The term "Action Reward" is a bit confusing. Why not just use "Energy Penalization"?
- [L150] "... sheet music will typically provide sparse fingering information, ..." -> Is this what is actually used for "human-prior" fingering information? This can be made more explicit in the paragraph.
- [L266] "can result make it impossible" -> "can make it impossible"
- [Fig. 7] What is the unit of y-axis (Lookahead)? Timesteps (i.e., 20 means 1 second for 20 Hz control timestep, which seems a bit short for planning)? Or seconds (whereas 20 seconds seems a bit long)?
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The presentation and experiments look very well executed. However, I struggle see a strong merit otherwise. The task itself, without human evaluation, is questionable. Methodology-wise the contribution is also limited and specific to the piano playing problem.

### Post Rebuttal
The rebuttal addresses my concern on the over-claimed contributions. While falling short on some of the interesting aspects (e.g. "aligning with human values of pleasantness" and "generalizing to new songs") at its current form, I agree with the authors that the publication of this work can lead to future development in these areas. I also like the "unique dimension to policy evaluation" ("does my policy sound good") of the piano task. Therefore I'm willing to raise my recommendation from weak reject to weak accept.
review_title: null

Meta-review: Scores:  Dnam: Weak Accept, eFF7: Weak Accept, 28kq: Weak Accept, 28kq: Weak Accept

Quality:  The paper presents an RL based agent that is trained to play piano in simulation using bi-manual anthropomorphic hands. 
It introduces a multi-task imitation learning setting for training a single agent which is able to play multiple songs.
All reviewers have acknowledged that the study is challenging and very well executed. 

Clarity: The paper is clear and well-organized 

Originality: Good

Significance: The paper presents a novel high-dimensional control task and evaluates a strong baseline for performance, which can serve as a good starting point for future studies. 

Cons:

- lack of a user-based study that demonstrates that the agent plays "acoustically pleasing"

Pros:

- novel high-dimensional dexterous control task highly relevant to robotics 
- clear contributions with a good baseline for future work
- thorough evaluation of the model, extensive ablation study and good discussion of generalization capabilities
Decision: Accept (Poster)