PianoMotion10M: Dataset and Benchmark for Hand Motion Generation in Piano Performance
Authors: Qijun Gan
Song Wang
Shengtao Wu
Jianke Zhu
Venue: ICLR 2025 Spotlight (2025)
Keywords: Hand pose estimation
piano music
motion generation
Primary area: datasets and benchmarks
TL;DR: We construct the first large-scale piano-motion dataset, PianoMotion10M for hand motion generation.

Abstract: Recently, artificial intelligence techniques for education have been received increasing attentions, while it still remains an open problem to design the effective music instrument instructing systems. Although key presses can be directly derived from sheet music, the transitional movements among key presses require more extensive guidance in piano performance. In this work, we construct a piano-hand motion generation benchmark to guide hand movements and fingerings for piano playing. To this end, we collect an annotated dataset, PianoMotion10M, consisting of 116 hours of piano playing videos from a bird's-eye view with 10 million annotated hand poses. We also introduce a powerful baseline model that generates hand motions from piano audios through a position predictor and a position-guided gesture generator. Furthermore, a series of evaluation metrics are designed to assess the performance of the baseline model, including motion similarity, smoothness, positional accuracy of left and right hands, and overall fidelity of movement distribution. Despite that piano key presses with respect to music scores or audios are already accessible, PianoMotion10M aims to provide guidance on piano fingering for instruction purposes. The source code and dataset can be accessed at https://github.com/agnJason/PianoMotion10M.

Published as a conference paper at ICLR 2025
PIANOMOTION OM: DATASET AND BENCHMARK FOR HAND MOTION GENERATION IN PIANO PERFOR- MANCE
Qijun Gan & Song Wang & Shengtao Wu & Jianke Zhu"
Zhejiang University
(ganqi jun, songw, shengtaowu, jkzhu) juju. edu. on
ABSTRACT
Recently, artificial intelligence techniques for education have been received in- creasing attentions, while it still remains an open problem to design the effective music instrument instructing systems. Although key presses can be directly de- rived from sheet music, the transitional movements among key presses require extensive guidance in piano performance. In this work, construct piano- hand motion generation benchmark to guide hand movements and fingerings for piano playing. To this end, collect annotated dataset, PianoMotion 10M, consisting 116 hours of piano playing videos from a bird' bird's-eye view with 10 million annotated hand poses. We also introduce powerful baseline model that generates hand motions from piano audios through a position predictor and a position-guide gesture generator. Furthermore, series of evaluation metrics are designed to assess the performance of the baseline model, including motion simi- larity, smoothness, positional accuracy of left and right hands, and overall fidelity of movement distribution. Despite that piano key presses with respect to music scores or audios are already accessible, PianaMotion 10M 10M aims to provide guid- ance piano fingering for instruction purposes. The source code and dataset can be accessed at https:://// /tttps:/ggitubb. com/ wgwp/aannaasnniiiaa www.o//ssnnniannottiiio aon.anooooooonnnnn0
INTRODUCTION
The process of learning has been significantly improved with artificial intelligence techniques, which enable individuals enhance their skills under the guidance of an AI coach (Wang et al.] 2019 Zakka et al.) 2023 Wang et al., 2024). This can be extended into learning to play the musical instruments. Particularly, piano performance requires a profound understanding of the underlying relationship between musical compositions and their corresponding physical motions. As the rigor- ous practice and training program are necessary for athletes to master a diverse range of expressive human poses, it entails extensive practices to achieve proficiency in piano fingering and hand move- ment. To facilitate access to playing guidance, the development of Al piano coach has been spurred.
Large-scale piano-motion datasets the foundation of nuanced approach for motion generation, which offer valuable guidance for physical performance and musical expression. The computational challenge of motion generation lies in capturing the nonlinear relationship between musical pieces and the intricate hand motions required for piano playing. The hand poses vary for the same note across different melodies. Moreover, the dynamic nature of musical expression demands level of continuous motion, which is challenging to be learned from small datasets. These limitations underscore the urgent need for large-scale piano-motion dataset.
To address the deficiency of the dataset for guiding hand movements and fingerings in playing piano, we introduce a large-scale 3D piano-motion dataset named PianoMotion1OM. As shown in Fig. I PianoMotion 10M contains piano audio tracks, Musical Instrument Digital Interface (MIDI) files, and annotated hand motions with their corresponding videos, meticulously collected from the Internet. It comprises I 1,966 pairs of video and music, with a total duration of 116 hours and
Corresponding author.

---

Published as a conference paper at ICLR 2025
Piano Videos from Internet
Piano Audio
Hand Motion Generation Model
Motion
Audio
MIDI
PianoMotion Dataset
Hand Motions
Figure 1: Overview of our framework. We collect videos of professional piano performances from the Internet and process them to construct a large-scale dataset, PianoMotion 10M, which comprises piano music, MIDI files and hand motions. Building upon this dataset, establish benchmark for generating hand motions from piano music.
10 million annotated frames. The parametric MANO hand model (Romero et al. 2017) is employed to represent hand gestures. Our constructed dataset offers a diverse range of music styles and piano techniques, which addresses the demands of various preferences and skill levels.
Traditional applications like PianoPlayer (PianoPlayer 2018) are adept at generating static hand gestures and positions for piano scores, which typically make use of classifiers predict the proper fingering combinations. However, they often fail to capture the diversity and continuity inherent of the piano performance in PianoMation 10M. Both rule-based methods (Balliauw et [017 Lin & Liu 2006) and HMM-based approaches Yonebaayshhi et al. [007] Nakamura & Sagayama (015) can estimate fingerings while they primarily focus on the local fingering constraints of continuous notes, Consequently, they often overlook crucial information like long-range fingering relationships, while our task aims estimate the motions of long clips.
To address these limitations, novel baseline model is presented to show the effectiveness of Pi- anoMotion 10M, which is able generate realistic hand motions from piano melodies. Given piece of piano music, our model can locate the positions of both hands and generate long sequence of hand gestures for the performance. It effectively learns the music-position correlation through an efficient position predictor and produces continuous gestures with position-guided gesture gen- crator based on a diffusion probabilistic model. To assess our baseline model, we propose several evaluation metrics, including Fréchet Gesture Distance and Wasserstein Gesture Distance to mea- sure the fidelity of each hand motions, Fréchet Inception Distance with a pre-trained auto-encoder to investigate the motion quality of double hands, and Position Distance assess the accuracy of hand positioning, and Smoothness of the generated motions.
In summary, main contributions are: A large-scale piano-motion dataset PianoMotion comprises 16 hours of music and 10 million annotated frames with hand poses. To the best of our knowledge, it stands the first dataset integrating piano music with its corresponding hand motions, which facilitates the tasks of 3D hand motion generation from piano audio tracks and piano music generation conditioned hand motions. 2) Based PianoMotion 10M, we propose a benchmark with a series of evaluation metrics to investigate the effectiveness on hand motion generation, including the accuracy positions and fidelity of gestures. 3) A novel baseline model bridges piano music with hand motions, which estimates hand location with position predictor and generates hand gestures sequences through a position-guides gesture generator.
RELATED WORK
Motion-musie Datasets, While multi-modal datasets (Schuhmann et al. 2021 Srinivasan et al.] 2021 Miech al. 2019 Lee et al. 2021) become the key of various learning tasks, there remains
2

---

Published as a conference paper at ICLR 2025
significant gap in the availability of repository specially designed for music- motion gen-
eration. Although the existing hand gesture datasets (Moon et al. 2020b a Kwon et al. 2021 Gan
et al.] 2024|] Wu et al. 2023) contain large number of image-hand gesture pairs, they do not have audio or other related information. This limitation hinders them from the generative tasks, since they
mainly focus hand reconstruction and pose estimation. Recently, datasets like AIST++ (Li al.
2021] and TikTok (Zhu et al., (022) are tailored for music-dance learning, which provide limited music segments, typically less than 5 hours in duration. Moryossef et al. (Moryossef et al., 2023) manage to automatically detect which fingers pressed the key of the piano and provide a dataset of piano-fingering. However, it does not provide continuous hand gesture movements. Therefore, there is crucial need to build piano-motion dataset specially tailored for motion generation tasks. To this end, we introduce the PianoMotion 10M dataset, which comprises extensive piano music and their corresponding hand motion annotations.
Piano Hand Datasets. Yamamoto et al. (Yamamoto et al. 2010) employed a key-frame technique to generate natural hand motions for piano playing from inputted music scores. Liang et al. (Liang) ct al. 2016) used an RGB-D camera to collect 7,200 depth images and designed models to esti- mate fingertip movements and predict finger tappings. The CMU Panoptic Studio dataset (Joo al.) 2013 2017] Simon et al. 2017) provided a multi-view dataset of human-obij -bjject interactions, in- cluding annotated hand and face postures. Simon el al. (Simon al. 2017) introduced a real-time hand keypoint detector and a markerless 3D hand motion capture system capable of reconstructing challenging hand-object interactions and musical performances. TwohandsMusic (Seo et al. 2019) presented two shallow networks to estimate 3D hand poses and tap gestures, along with dataset of 85,000 images of hand movements while playing the piano. To help beginners improve their playing techniques, Johnson er al. (Johnson et al. 2020) discussed automatic assessment of pianists' hand postures, which categorizes postures as correct posture, low wrists, or flat hands. Kun et al. (Su al, 2020) collected piano performance videos to generate the music for videos directly. Moryossef et al. (Moryossef et al., (223)) managed to automatically detect which fingers pressed the key of the piano and provided dataset of piano- fingering. Wu et al. (Wu et al. (023) proposed marker- removal approach for collecting bare-hand data, which included a precise ground truth alongside
large-scale pianist 31) hand dataset, PianoHand2.5MM. This dataset provided advanced annotated
images for estimating hand postures in marked- hhane images.
Our dataset and baseline primarily focus researching the relationships between piano music and hand postures. More comparisons with other existing hand datasets are shown in the attached pdf document. Since most of existing piano-hand datasets are not publicly available, our open-source release of PianoMotion 10M will be of substantial importance to advancing research in this field.
3D Human Motion Synthesis. The problem of generating realistic and controllable 3D human motion sequences has been a subject of long-standing study. By taking advantage of 2D keypoint detection (Cao et al.) 2017), the synthesis of 2D skeletons has been extensively explored (Ren al. 2020 Shiratori et al. 2006) |erreira et al. 2021). Considerable research efforts have been devoted
to 2D speech-drivee driven head generation for facial mouth and lip motion generation (Zakharov et al.,
2019 Chen et al. 2019: Guo et al., 1021 Ji et al. 2021 Hong et al. 2022), which usually employ
either image-driven or voice-driven methods to produce realistic videos of speaking individuals. However, the expressive capabilities of 2D pose skeletons are limited and they are not applicable to
3D character models. Recent methods for full-body 3D dance generation have utilized LSTM (Tang
ct al. 2018 Yalta et al. 2019: Zhuang et al., 2022). GANs (Sun et al.] 2020; Ginosar et al. 2019) or transformer encoders (Li et al. 2021: Huang et al. 2020 Siyao et al. 2002). To generate vivid talking head videos, extensive research has been conducted in the field of speech-driven 3D facial animation (Cao et al. 2005; Cudeiro et al. 2019 Tian et al. 2019 Fan et al. [022] Zhang et al., 2021 Emo'Talk (Peng al. 2023) animates emotional 3D faces from speech input by generating controllable personal and emotional styles. Tian et al. (Tian et al. (2024) introduce speed controller and face region controller to enhance stability during the head generation process.
In the domain of hand motion generation, it has been primarily categorized into rule-based methods (Cassell et al. 2001 Huang & Mutlu 2012; Starke al. 2022) and data-driven ap- proaches (Kopp et al. 20066 Chiu et al. [015]] Liu et 2022 Yoon et al. [020] Bhattacharya eet al., (222)) For instance, Speech2Gesture (Giossar et al. (219)) utilizes conditional generative
adversarial networks to generate personalized 2D keypoints from audio. Ahuja el al. (Ahuja et al.) 2020) propose a method for personalized motion transfer. Ao el al. (Van Den Oord et al.) (017)
leam the mapping between the speech and gestures from data using combined network structure
3

---

Published as a conference paper at ICLR 2025
of the vector quantized variational auto-encoder (VQ-VAE). Beyond 2D keypoints generation, Tri- Modal (Yoon et al., 2020) extracts different upper body movements from TED talks and designs a LSTM-based neural network conditioned audio, text, and identity to generate co-speech gestures.
Recently, diffusson models have achieved promising results in generating human motions. Previous works such as MDM (Tevet et al. 2022) and MotionDiffiuse (Zhang al. (024)) have produced realistic motion inspired by denoising diffusion probabilistic models (DDPM) (Ho et al., 2020). PhysDiff (Yuan et al. 2023) extends MDM by imposing physical constraints. MLD (Chen al, 2023) utilizes latent carrier DDPM for forward denoising and reverse diffusion in motion latent space. MAA (Azadi et al., 2023) enhances the performance of -intriittb data by pre-training diffusion models. Zhang et al. (Zhang et al., (203)) introduce improves the eex-tto-motion functionality distribution.
DDPM, which
PIANOMOTION OM DATASET
a formidably challenging task to map piano music to hand motions due to the significant influ- ence of note sequences and positions hand movements and fingering. Lacking diverse data may lead to the inferior performance on estimating various hand poses in piano playing. To capture the variability, we present the first large-scale piano-motion dataset, PianoMotion 10M, which com- prises 1,966 piano performance videos along with 10 million hand poses and their corresponding MIDI files, resulting in overall duration of 1 116 hours. Fig. presents
example of our dataset. These videos are segmented into 16,739 individual clips with length of 30 seconds. To ensure com- prehensive evaluation, extract 7,519 clips for training, 821 for validation and 8,399 for testing. The dataset is divided based on the original videos to avoid the overlap in piano performances. The detailed comparisons with the existing datasets on hands and 3D human motions are summarized into Tab. 1
Table 1: Comparison between different hand and motion datasets. The proposed PianoMo- 10M dataset consists of piano music with corresponding hand poses for hand motion generation. For reference, the table is organized as follows: the first five rows list existing hand-image datasets, the subsequent four rows present music-motion datasets, and the following four rows display various
piano-hand pose datasets. The final row features our dataset.
Dataset
Year
Pose
Size
Subject
Music
MIDI
Duration(h)
Available
CMU Panoptic
2017
31K
X
X
FreiHAND
2019
134K
32
X
X
InterHand2.6M
2020b
2.6M
27
X
X
RGB2Hands
2020
1K
2
X
X
Re:InterHand
2023
1.5M
10
X
X
X
GrooveNet
2017
0.38
DanceNet
2022
0.96
EA-MUD
2020
0.35
AIST++
2021
10.1M
30
5.19
Barehanded Music
2016
7.2K
X
Music
2019
85K
X
Piano-fingering
2023
155K
X
PianoHand2.5M
2023
2.5M
21
4.3
X
PianoMotion 0M
2024
10.5M
14
116
3.1 DATA COLLECTION
There is wealth of videos and live streams dedicated to musical instrument performances and tutorials from the Internet. Note that each individual has a unique playing style, we firstly select 14 piano experts from Bilibil and collect a total of 3,647 candidate videos. To ensure consistency and
https:// www.billbbl com of the most popular video-sharing platforms in China.
4

---

Published as a conference paper at ICLR 2025
Video
Hand Pose
MIDI
Audio
Figure 2:
Illustration of sample from PianoMotion 10M. Each sample in our dataset includes audio, hand pose annotations, and MIDI file along with the corresponding Bilibili video ID.
enhance the quality of our dataset, we conduct pre-processing with five pivotal factors, including resolution, audio quality, camera perspective, presence of multiple individuals, and visibility of hands. Accurate, noise-free audio is essential to our dataset. We manually select pure piano music to ensure the audio contains human vocals sounds from other instruments, rather than employing on automated tools Kong et al. (2020). Moreover, videos with a resolution lower than 1080 x 1920 discarded to ensure the quality of our dataset. To enhance the observation of hand movements and gestures, we choose videos with a bird's-eye view to minimize hand occlusion. Furthermore, remove those videos where hand regions are frequently (more than 20% frames) obstructed or invisible during performance. Additionally, we do not take consideration of videos with multiple piano players, which is beyond the scope of this dataset. By addressing these factors, the overall quality and coherence of the dataset have been substantially enhanced. Following this preprocessing stage, the result collection of 1, 966 high-quality raw videos. Each one showcases the piano playing by individual, which is captured from bird's bird's-eye view along with pure piano audio.
3.2 DATA ANNOTATIONS
MIDI is universal digital protocol store musical data for various musical devices, which is served as digital representation of musical notes, volume, tempo, and other performance parameters. Some creators provide ground-trutt MIDI files for their performances. For the remaining piano performances, automatic piano transcription methods (Kong al. 2021 Maman & Bermano, 2022 Gardner et al. 2021; Toyama et al. 2023 Cheuk et al.) 2023) are employed to convert them into stable and robust performance in wild data. To ensure the accuracy of the conversion results, we replay MIDI files and compare them with the original music tracks. Files are adjusted if they exceed thresholds such timing differences over 30 ms, dynamic variations beyond 10%, or noticeable pitch mismatches that alter melody or harmony.
MIDI files, while (Kong et al. 2022 demonstrates
Hand Pose is captured via the parametric hand model MANO (Romero al. 2017), which serves as our hand prior model. It effectively maps the pose parameter E - with J per-bone parts and the shape parameter p E 12 onto a template mesh M with vertices V. The MANO model enables the simulation of various hand gestures and movements during piano performances, which provides an effective way to study the relationship between hand gestures in playing piano. Due the non- uniformity of hand sizes and positions in the collected videos, we first employ the MediaPipe hand detection framework (Lugaresi et al. 2019) to obtain bounding box of the hand region. Video frames are cropped according to the detected hand bounding boxes to enhance the robustness of the results. To this end, hand poses collected videos are annotated using HaMeR PPavlakos et al. 2023). HaMeR follows fully transformer-basee architecture and reconstructs hand models with increased accuracy and robustness. All images having hands detected by MediaPipe are annotated with hand poses by HaMeR, which result in dataset of 10 million image-pose pairs.
5

---

Published as a conference paper at ICLR 2025
Table 2: Statistics on the distribution of subjects the PianoMation 10M dataset, where subject
names denote the identity ID of experts.
Subject Name
Videos
Clips
Time(sec)
Frames
Subject Name
Videos
Clips
Time(sec)
Frames
1467634
337
4,359
103,293
2,237,181
66683747
535
5,136
130,352
3,520,370
37367458
114
859
20,802
571,525
676539782
19
359
11,314
187,917
442401135
275
1,788
52,030
1,209,905
688183660
264
872
19,074
564,699
478315001
285
2,674
62,615
1,792,707
Others
137
692
18,692
442,863
Total Videos: 1,966, Clips: 16,739, Total Duration(hour): 116.16, Annotated Frames: 10,527,167
To enhance the smoothness and continuity of our dataset, the generated hand poses require to be cleaned and refindd. The results of MediaPipe and HaMeR are usually accurate in most cases, while some inferior results may occur due to rapid motion and image blurring. The Hampel filter (Pearson et al. 2016) with a window size of 20 is utilized to identify these outliers. The outliers and the timestamps where hands are undetected are firstly labeled as missing values. Within a small period, hand movements can be considered can be interpolated bilaterally. To address these missing data in the time series, missing segments with the frame length 8 less than 30 frames are filled by linear interpolation with respect to their surrounding values. The others are considered as periods when the hand is invisible. To ensure the reliability of detected hands, similar strategy is employed label observations with excessively short duration (8 < 15) as invisible. Finally, to ensure the smoothness of hand motions, we make use of a Savitzky-Golay filter (Schafer (20T)) with an order of and a window size of 11 for data smoothing, which delivers better performance in terms of noise separation, artifacts and baseline drifts. The annotated hand poses are manually checked to ensure their quality.
the motion with constant speed. Therefore, the small gap
3.3 DATA STATISTICS
The dataset is made of contributions from several subjects with different experts, and each provides varying amounts of data in terms of videos, clips, duration, and annotated frames. Tab. 2 presents the detailed statistics on the distribution of subjects within our PianoMotion10M dataset.
The whole piano-motion dataset, PianoMotion1OM, consists of 1,966 videos with approximately 116 total hours of footage. Each piece of music is segmented into 30-second clips at 24-second intervals, resulting in total of 16,739 clips. Note that we discard those clips where hand visibility is below 80%. Our dataset has 14 subjects with different playing styles. This variability ensures a rich diversity of playing techniques and music styles, which is essential for training robust models to predict piano hand gestures from musical pieces. All videos our dataset are publicly accessible with provided video IDs on the Bilibili website.
BASELINES
To tackle the challenging task of generating hand motions synchronized with piano audio tracks, introduce a novel motion generation framework upon the PianoMotion dataset. As illustrated in Fig. our framework consists of a position predictor and a gesture generator. The position predictor extracts the hand positions from piano music and integrates them as contextual input for the gesture generator. By leveraging DDPM model (Ho et al. 2020), the gesture generator estimates hand pose sequence based on the piano audio and the predicted positions. Further details on each component are elaborated in the following subsections.
Problem Formulation. Given a piano audio piece A with N frames, our objective is to obtain the hand position sequence P E N x3 and hand gestures E Nx33 for both left and right hands.. The hand position sequence P encompasses the 3D coordinates of the left and right hands. The hand gestures & consist of Euler angle at each joint of both hands.
Due to the highly nonlinear relationship between acoustic signals and hand gestures, it poses significant challenge to estimate motions through discriminative models (Peng et al. 2023). which usually leads to an average pose, as demonstrated in Section 5.2 To address this issue, hand position predictor is introduced to estimate the continuous changes hand positions. Moreover, a generative model is utilized to reconstruct hand gestures from piano music piece. Hereby, the task

---

Published as a conference paper at ICLR 2025
Position Decoder
Loss
Linear
L +
L
R:
Position Predictor
Position
Audio Encoder
Time Embedding
Timestep
Gesture Decoder
MW
θ
Noise
Denoise
Gesture
Gesture Generator Position-guided Diffusion
Piano Audio
Hand Motion
Illustration of our baseline model. Given piece of piano music, our baseline model estimates the hand motions by predicting hand positions and generating hand gestures.
Figure 3:
of hand motion generation becomes more concise and comprehensive by disentangling into hand position estimation and gesture generation. To extract the audio features fa from the audio A, make use of pre-trained audio feature extractor &a (Baevski et al. 2020; Hsu et al. (021). which formulated ʃa (A), Sa NCC, C the feature dimension of Qa.
4.1 POSITION PREDICTOR
The position predictor is employed predict the 3D positions of the left and right hands, respec- tively. Since the audio features fa cannot directly be mapped to the positions, a feature embedding module is treated our position decoder & extract the latent features. It projects the sequential audio features fa onto the latent features fo with more comprehensive temporal information, which is formulated fp = (p(f)), fp E IR N x512 Subsequently, linear mapping is employed to project the feature fp onto the output positions P.
In our experiments, we can make use of either Transformer (Vaswani et al., 2017 or State-Space Model (SSM) (Gu & Dao 2023) as our feature embedding module. Transformer leverages self- attention mechanisms to effectively capture long-range dependencies and contextual information order to learn temporal relationships. Recently, different representation inspired by classical SSM (Kalman] 1960) is proposed to replace the attention mechanism, which is built upon a more contemporary Selective Structured State Space Model (S6) (Gu Dao (023) suitable for deep learning. By sharing similar architecture to the classical RNN, it can efficiently capture information from previous inputs.
4.2 POSITION-GUIDED GESTURE GENERATOR
Leveraging recent achievements Zhi et al. 2023 Azadi et al.] 2023) motion generation, our approach incorporates diffusion probabilistic model (oo et al. 2020). By mastering denoising, this model effectively captures the complex distribution of hand motions observed in large-scale piano- motion datasets, which has the capability generate motions with different conditions. Starting with a clean sample On of motion sequence, the forward diffusion process establishes Markov chain that gradually adds noise O, T steps, which generates series of noisy samples O,,..., O7
((O,)zo) = ( / Va a.)
(1)
where N' denotes the Gaussian distribution. a; -I. (1- 3.) and 3 represent the variance sched- uler for the added noise. Therefore, a model parameterized by a deep neural network G trained to master the inverse process within another Markov chain, which learns the mapping p(0,-1(0) to sequentially denoise samples over steps. Specifically, denoising model G consists of a 4-layer U-Net (Ronneberger et al. 2015) with 256, 512, 1024, 2048 dimensions for each layer.

---

Published as a conference paper at ICLR 2025
To reduce the noise in piano audio, the audio features fa are simultaneously fed into the denoising neural network G as conditions. Similar to the position decoder Dp. a gesture decoder Pg is utilized to extract gesture features fg from the audio features fa fg 4 (f(fa) Considering that the finger movements at different positions for the same pitch are distinct, G requires the guidance of hand positions P from the position predictor. This will enhance the fidelity of the generation process. As for the additional conditions, the time step embeddings of time are concatenated in denoising process. The denoising process can be formulated as follows
O,- G(O,, t; fa P),
where Ou E Nx333 denotes the result of hand motions within N frames.
4.3 IMPLEMENTATION DETAILS
We employ a two-stage scheme to train our proposed network. At the first stage, the position pre- dictor is trained using position error Lp and velocity loss L... The position error L. computes the L1 loss between the predicted position P and the ground truth P. expressed as Lp = || - Pll1- Inspired by (Peng et al., 2023). velocity loss is adopted to induce temporal stability for generating smoothing movements, which formulated C. [|(P, -a- n-th frame in a motion sequence. Specifically, our model is trained by subject 14676363 634 and subject 66685747, which have the similar piano keyboard layout. At the second stage, the parameters of position predictor are frozen, and the estimated positions are employed as a guidance for the gesture generator. As in (Salimans & Ho 2022), the denoising process is modified from noise prediction to velocity prediction. During the whole training process, the parameters of audio feature extractor 4. are frozen.
n denotes the
The baseline model is implemented with PyTorch. We normalize the inputs into 30 FPS through interpolation, where each piece of music lasts seconds. Both stages involve training for 100, 000 iterations with the learning rates of 2 x 10-5 and 5 10-5, respectively. We conducted all the experiments on a PC with single NVIDIA RTX 3090Ti GPU, which has 24GB of GPU RAM.
BENCHMARK
5.1 EVALUATION METRICS
To assess the performance of our proposed baseline, we employ several evaluation metrics to exam- ine the effectiveness of hand poses generated by the input piano music, which are crucial in under- standing the relationship between piano melody and its corresponding playing motions. Specifically, assess the overall motion distribution similarity of the hand, we employ the Fréchet Inception Dis- tance (FID). In piano performance, the left and right hands execute distinct actions and movements. For detailed evaluation of the disparity of single-hand movements, we utilize the Fréchet Gesture Distance (FGD) and Wasserstein Gesture Distance (WGD). Evaluating positional accuracy is also crucial, for which we introduce the Positional Distance (PD) as a metric. Smoothness is utilized to determine the fluidity of the generated motions.
Fréchet Inception Distance (FID). FID is introduced to measure the overall motion similarity by calculating the Fréchet distance between the feature vectors of prediction and ground truth. We pre- train an auto-encoder (Yoon et al. 2020) to project motion sequence onto a latent space for double hands. The FID is adopted assess the fidelity of the overall motions generated by our baseline model. The formulation of FID is derived as follows
FID I|efred 1 Tr(C, (rpee + CG - -
"peee" C..)),
where f represents the latent features obtained by the autoencoder. j and denote the mean vector and covariance matrix, respectively. Tr() is the trace operation of matrix.
Fréchet Gesture Distance (FGD). On the other hand, FGD is utilized to compute the long long-term disparity between predicted gestures and ground truth of one hand. It calculates the Fréchet distance over complete sequences of hand motions O, offering metric for the global alignment between generated and real data. The formula is listed as follows
FGD = I|p
g1l + Tr(C pred + gt 2 *
V * Cpred C.
(4)
8

---

Published as a conference paper at ICLR 2025
Table 3: Quantitative evaluation of our proposed hand motion generation baseline and existing models on the validation set. We present a comparative analysis of various network architectures,
highlighting the performance and efficiency of our baselines in generating hand motion.
Left Hand
Right Hand
Params
Method
Decoder
Step
FID,
FGD1
WGD1
PDL
Smooth
FGD.
WGD1
PDJ
Smooth.
(M)
Emo
2023
0.232
0.353
0.360
0.259
SivelySpeaker
2023
TF
1000
0.538
0.220
0.038
0.406
0.535
0.249
0.030
0.334
4.157
321
Wav2vee
SSM
1000
0.125
0.223
0.042
0.112
0.416
0.216
0.034
0.335
3.587
320
Our- Base
0.219
0.040
0.424
0.246
3.608
SSM
1000
0.432
0.218
0.041
0.407
0.402
0.247
0.033
0.336
3.412
320
TF
1000
0.432
0.219
0.011
0.112
0.418
0.247
0.034
0.338
3.529
323
Wav2vee
SSM
1000
0.430
0.219
0.038
0.253
0.421
0.244
0.031
0.208
3.453
539
Our-LLare
0.372
0.038
0.244
3.376
SSM
0,406
0.217
0.037
0.237
0.403
0.244
0.030
0.214
3.395
HuBERT
TF
1000
0.372
0.217
0.837
0.348
0.351
0.244
0.030
0.205
3.281
557
Wasserstein Gesture Distance (WGD). WGD (Rubner et al. tributions, each of which is represented as Gaussian Mixture Model (GMM) (Kolouri et al. 2018) as below
2000) is computed between two dis-
W (Iz. I,))
inf E(x.)) T~II(I
I| yll].
(5)
where II( Iy) indicates the joint distributions that combine the parametric GMM distributions In and Zy. Distributions In and I, are fitted by predicted gestures and ground truth of single hand. E calculates the expectation of the sample pairs (z, y) with multiple Gaussians, which focuses on short-term similarity by analyzing mixture distributions, allowing it to capture finer, localized motion patterns within the sequence. The WGD metric offers robust measure of dissimilarity between the generated motion and ground truth.
Position Distance (PD). The PD calculates the L2 distance between estimated positions and the ground truth, which is formulated as
PD = || pred 1
(6)
where P is the hand root position. It is essential to evaluate the precision of predicted hand positions to ensure the accuracy required for piano fingerings.
Smoothness. Smoothness measured by computing the acceleration of each joint. However, hands in static state exhibit maximum smoothness, which contrary to the desired outcome. Conse- quently, we consider the acceleration of ground truth as a reference and utilize relative acceleration the evaluation metric for smoothness, which is formulated Smooth = >x f T t; denotes the acceleration of the i-th joint. It reflects the continuance and coherence of the estimated gestures.
5.2 EXPERIMENTS
Experimental Setup. Within the network, we compare two transformer-based audio feature extrac- tors, Wav2Vec2.0 (Baevski et al. 2020) and HuBERT (Hsu et al. 202I), which are popular in the field of self superviedd speech recognition. Regarding the feature embedding module (Decoder), we evaluate the performance of SSM-based model (Gu & Dao 2023) in contrast to the transformer (TF) approach (Vaswani et al. 2017). We conduct experiments using two different model configurations. Both of them employ the same diffusion architecture while differing in the feature extractor setup. 2ww.eee..--bbe/HuBBBBe
The base model incorporates
as the audio feature extractors, fea- turing model dimension of 768 and containing 8-layer TF/SSM decoder. The large model utilizes Wav2Vec2 2ee.Neee..-lareeHuBBBBBBB O-large/HuBBRT--arre
with a model dimension of 1,024 and 16-layer TF/SSM decoder.
Quantitative Results. Tab. 3 provides the results of existing methods and our baseline under various experimental settings on PianoMotion 10M dataset. Due to the absence of prior work on gener- ating gestures for piano music, we refer to the network structures of EmoTaak (Peng et al. 2023) and LivelySpeaker ZZi et al., (203)) and re-implement them to account for our task. Emo di- rectly generates poses from audio, while LivelySpeaker utilizes an MLP-based diffusion backbone for gesture generation. Our baseline models achieve better fidelity on hand motions by estimat- ing positions and gestures, separately. In our baseline models, Ta-based models outperform the 3SM-based models in processing our time-series information, especially in our large model. For the audio feature extractor, the performance of the HuBERT model slightly outperforms the Wav2vec2.

---

Published as a conference paper at ICLR 2025
Music
Talkalk [44]
LiveSpeaker [76]
Ours
GT
Figure 4: Illustration the qualitative results. We display the generated gestures across frames
using different methods. Our method stands out due to its greater fidelity, shown in the examples.
Additionally, we conduct ablation experiments on different denoising steps and our model also achieves competitive results with fewer steps, as shown in Table
Step
FID4
FGDJ
WGD.
Smooth.
3.540
0.361
0.237
0.354
3.682
0,363
0.236
0.310
3.438
0.366
0.232
0.254
Qualitative Results. Fig. demonstrates the visual results of our baseline and the existing models. The output of Emo'Talk method ex- hibits a static average gesture. While MLPs af- ford LivelySpeaker rapid inference speed, they compromise the fidelity of generated motions. Conversely, our model demonstrates notably superior performance compared to previous methods by taking advantage of a two-stage approach, as illustrated Fig. 4 To attain accurate positional information, utilize an end-to-end position predictor rather than making of uncontrollable diffusion model for position generation. Furthermore, we employ a position-guided approach with diffusion-base basod gesture generator for hand motion estimation.
3.360
0.348
0.233
0.240
1000
3.281
0.362
0.231
0.227
Table 4: Ablation study on denoising steps.
CONCLUSION
In this work, we present PianoMotion 10M, a new large-scale piano-motion dataset for hands, which has 116 hours of piano music and 10 million frames annotated with hand poses. We address the critical issue that the current datasets are insufficient for human-piano interaction. Based on PianoMotion 10M.MMionOM,, we develop benchmark model that maps the piano music pieces to hand mo- tions. To simplify the learning target, we divide the motion into position and gesture by introducing position predictor along with gesture generator guided by the estimated positions. We suggest the evaluation metrics to measure the fidelity and smoothness of the hand motions compared to the ground truth. Our dataset and benchmark will further advance the automation of piano performance simulation and assist in learning piano playing.
ACKNOWLEDGMENTS
This work was supported in part by the National Key Research and Development Program of China under Grant (2023YFF0905 104) and in part by the National Natural Science Foundation China under Grants (62376244). It also supported by the Information Technology Center and State Key Lab of CAD&CG, Zhejiang University. We are grateful to all Bilibili creators who contributed to the dataset construction, especially CIPMusic, for their support.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Chaitanya Ahuja, Dong Won Lee, Yukiko I Nakano, and Louis-Phillipp Morency. Style transfer for co-speech gesture animation: A multi-speaker aandtiionllmiixue approach. In ECCV, pp. 248 265, 2020.
Omid Alemi, Jules Françoise, and Philippe Pasquier. Groovenet: Real-time music-driven dance movement generation using artificial neural networks. Networks, 8(17):26, 2017.
Samanch Azadi, Akbar Shah, Thomas Hayes, Devi Parikh, and Sonal Gupta. Make-an-animation animation:
Large-scale text-conditional 3d human motion generation. In ICCV, pp. 15039-15048, 2023.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli, wav2vec 2.0: A frame- work for self-supervised learning of speech representations. NeurIPS, 114112444-246000 2020.
Matteo Balliauw, Dorien Herremans, Daniel Palhazi Cuervo, and Kenneth Sörensen. A variable neighborhood search algorithm to generate piano fingerings for polyphonic sheet music. ITOR, 24(3)500-533, 2017.
Uttaran Bhattacharya, Nicholas Rewkowski, Abhishek Banerjee, Pooja Guhan, Aniket Bera, and Dinesh Manocha. Text2gestures: A transformer-based network for generating emotive body ges- tures for virtual agents. In VR, pp. 1-10, 2021.
Yong Cao, Wen C Tien, Petros Faloutsos, and Frédéric Pighin. Expressive speech-driven facial animation. TOG, 1283-13322 2005.
Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2d pose estimation using part affinity fields. CVPR, pp. 7291-7299, 2017.
Justine Cassell, Hannes Högni Vilhjálmsson, and Timothy Bickmore. Beat: the behavior expression animation toolkit. In SIGGRAPH, pp. 477-486, 2001.
Lele Chen, Ross K Maddox, Zhiyao Duan, and Chenliang Xu. Hierarchical cross-modal talking face generation with dynamic pixel-wise loss. In CVPR, pp. 7832-7841, 2019.
Xin Chen, Biao Jiang, Wen Liu, Zilong Huang, Bin Fu, Tao Chen, and Gang Yu. Executing your commands via motion diffusion in latent space. In CVPR, pp. 18000-18010, 2023.
Kin Wai Cheuk, Ryosuke Sawata, Toshimitsu Uesaka, Naoki Murata, Naoya Takahashi, Shusuke Takahashi, Dorien Herremans, and Yuki Mitsufuji. Diffroll: Diffusion-based generative music transcription with unsupervised pretraining capability. In ICASSP, pp. 1-5. IEEE, 2023.
Chung-Cheng Chiu, Louis-Philippe Morency, and Stacy Marsella. Predicting co-verbal gestures: A deep and temporal modeling approach. In Intelligent Virtual Agents: 15th International Confer- ence, pp. 152-166, 2015.
Daniel Cudeiro, Timo Bolkart, Cassidy Laidlaw, Anurag Ranjan, and Michael Black. Capture, learning, and synthesis of 3d speaking styles. In CVPR, pp. 10101-1011) 2019.
Yingruo Fan, Zhaojiang Lin, Jun Saito, Wenping Wang, and Taku Komura. Faceformer: Speech- driven 3d facial animation with transformers. In CVPR, pp. 18770-187777 2022.
Joao P Ferreira, Thiago M Coutinho, Thiago L Gomes, José F Neto, Rafael Azevedo, Renato Mar- tins, and Erickson R Nascimento. Learning to dance: graph convolutional adversarial network to generate realistic dance motions from audio. Computers & Graphics, 94:11-21 2021.
Qijun Gan, Wentong Li, Jinwei Ren, and Jianke Zhu. Fine-grained multi-view hand reconstruction using inverse rendering. In AAAI, pp. 1779-1787, 2024.
Josh Gardner, Ian Simon, Ethan Manilow, Curtis Hawthorne, and Jesse Engel. Mt3: Multi-task multitrack music transcription. arXiv preprint arXiv:21 2021.
Shiry Ginosar, Amir Bar, Gefen Kohavi, Caroline Chan, Andrew Owens, and Jitendra Malik. Learn- ing individual styles of conversational gesture. In CVPR, pp. 3497-3506, 2019.
11

---

Published as a conference paper at ICLR 2025
Albert Gu and Tri Dao. Mamba: Ainear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023.
Yudong Guo, Keyu Chen, Sen Liang, Yong-Jin Liu, Hujun Bao, and Juyong Zhang. Ad-nerf: Audio 2021.
driven neural radiance fields for talking head synthesis. In ICCV, pp.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurlPS, 33: 6840-6851, 2020.
Fa-Ting Hong, Longhao Zhang, Li Shen, and Dan Xu. Depth-aware generative adversarial network for talking head video generation. In CVPR, pp. 3397-3406, 2022.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. TASLP, 29:3451-3460, 2021.
Chien-Ming Huang and Bilge Mutu. Robot behavior toolkit: generating effecttve social behaviors for robots. In HRI, pp. 25-32 32, 2012.
Ruozi Huang, Huang Hu, Wei Wu, Kei Sawada, Mi Zhang, and Daxin Jiang. Dance rev- olution: Long-term dance generation with music via curriculum learning. arXiv preprint arXiv:2006.066 06/ 19, 2020.
Xinya Ji, Hang Zhou, Kaisiyuan Wang, Wayne Wu, Chen Change Loy, Xun Cao, and Feng Xu. Audio-driven emotional video portraits. In CVPR, pp. 14080-14089, 2021.
David Johnson, Daniela Damian, and George Tzanetakis. Detecting hand posture piano playing using depth data. Computer Music Journal, 43(1):59-78, 2020.
Hanbyul Joo, Hao Liu, Lei Tan, Lin Gui, Bart Nabbe, lain Matthews, Takeo Kanade, Shohei Nobuhara, and Yaser Sheikh. Panoptic studio: massively multiview system for social motion capture. In ICCV, pp. 3334 3342, 2015.
Hanbyul Joo, Tomas Simon, Xulong Li, Hao Liu, Lei Tan, Lin Gui, et al. Panoptic studio: A massively multiview system for social interaction. TPAMI, 2017.
Rudolph Emil Kalman. A new approach to linear filtering and prediction problems. 1960.
Soheil Kolouri, Gustavo K Rohde, and Heiko Hoffmann. Sliced wasserstein distance for learning gaussian mixture models. In CVPR, pp. 3427-3436, 2018
Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, and Mark D Plumbley. Panns: Large-scale pretrained audio neural networks for audio pattern recognition. IEEE/ACM Transac- tions on Audio, Speech, and Language Processing, 28:2880-2894, 2020.
Qiuqiang Kong, Bochen Li, Xuchen Song, Yuan Wan, and Yuxuan Wang. High-resolution piano transcription with pedals by regressing onset and offset times. TASLP, 29:3707-3717. 2021.
Stefan Kopp, Brigitte Krenn, Stacy Marsella, Andrew N Marshall, Catherine Pelachaud, Hannes Pirker, Kristinn R Thórisson, and Hannes Vilhjälmsson. Towards a common framework for multi- modal generation: The behavior markup language. In Intelligent Virtual Agents: 6th International Conference, pp. 205-217, 2006.
Taein Kwon, Bugra Tekin, Jan Stühmer, Federica Bogo, and Mare Pollefeys. H2o: Two hands manipulating objects for first person interaction recognition. In ICCV, pp. 10138-10148, 2021.
Sangho Lee, Jiwan Chung. Youngjae Yu, Gunhee Kim, Thomas Breuel, Gal Chechik, and Yale Song. Acav100m: Automatic curation of large-scale datasets for audio-visual video representa- tion learning. In ICCV, pp. 102844 2021.
Ruilong Li. Shan Yang, David A Ross, and Angjoo Kanazawa. Ai choreographer: Music condi- tioned 3d dance generation with aist++. In ICCV, pp. 13401-13412, 2021.
Hui Liang, Jin Wang, Qian Sun, Yong-Jin Liu, Junsong Yuan, Jun Luo, and Ying He. Barchanded music: real-time hand interaction for virtual piano. In I3D, pp. 87-94, 2016.
12

---

Published as a conference paper at ICLR 2025
Chih-Chun Lin and Damon Shing-Mon Min Liu. An intelligent virtual piano tutor. In VRCIA, pp. 353 356, 2006.
Xian Liu, Qianyi Wu, Hang Zhou, Yinghao Xu, Rui Qian, Xinyi Lin, Xiaowei Zhou, Wayne Wu, Bo Dai, and Bolei Zhou. Learning hierarchical cross-modal association for co-speech gesture generation. In CVPR, pp. 10462-10472, 2022.
Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris McClanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo-Ling Chang, Ming Guang Yong, Juhyun Lee, al. Mediapipe: A framework for building perception pipelines, arXiv preprint urXiv:1966.081 08177, 2019.
Ben Maman and Amit H Bermano, Unaligned supervision for automatic music transcription in the wild. In ICML, pp. 14918-14934, PMLR, 2022.
Antoine Miech, Dimitri Zhukov, Jean-Bay Saan-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howtol00m: Learning a text-video embedding by watching hundred million narrated video clips. In ICCV, pp. 2630-2640, 2019.
Gyeongsik Moon, Takaaki Shiratori, and Kyoung Mu Lee. Deephandmesh: A weakly- suakly-supervieed deep encoder-decoder framework for high-fidelity hand mesh modeling. In ECCV, pp. 440-455, 2020a.
Gyeongsik Moon, Shoou-I Yu, He Wen, Takaaki Shiratori, and Kyoung Mu Lee. Interhand2.66 A dataset and baseline for 3d interacting hand pose estimation from single rgb image. In ECCV, pp. 548-564, 2020b.
Gycongsik Moon, Shunsuke Saito, Weipeng Xu, Rohan Joshi, Julia Buffalini, Harley Bellan, Nicholas Rosen, Jesse Richardson, Mallorie Mize, Philippe De Bree, al. A dataset of relighted 3d interacting hands. NewrIPS, 36, 2023.
Amit Moryossef, Yanai Flazar, and Yoav Goldberg. At your fingertips: Extracting piano fingering instructions from videos. arXiv preprint 1rxXhv:333303344 2023.
Eita Nakamura and Shigeki Sagayama. Automatic piano reduction from ensemble scores based merged-output hidden markov model. In ICMC, 2015.
Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Angioo Kanazawa, David Fouhey, and Jitendra Malik. Reconstructing hands in 3d with transformers. arXiv preprint arXiv:2312.05251, 2023.
Ronald K Pearson, Yrjö Neuvo, Jaakko Astola, and Moncef Gabbouj. Generalized hampel filters. EURASIP, 2016:1-18, 2016.
Ziqiao Peng, Haoyu Wu, Zhenbo Song, Hao Xu, Xiangyu Zhu, Jun He, Hongyan Liu, and Zhaoxin Fan. Emotalk: Speech- emotional disentanglement for 3d face animation. In ICCV, pp. 20687-20697, 2023.
PianoPlayer, 2018.
Xuanchi Ren, Haoran Li, Zijian Huang, and Qifeng Chen. Self-supervised dance video synthesis conditioned on music. ACM MM, pp. 46-54, 2020.
Javier Romero, Dimitrios Tzionas, and Michael Black, Embodied hands: modeling and capturing hands and bodies together. ACM TOG, pp. 245:1-245:17, 2017.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In LNCS, pp. 234-241, 2015.
Yossi Rubner, Carlo Tomasi, and Leonidas Guibas. The earth mover's distance as metric for image retrieval. I/CV, 121, 2000.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.005/2. 2022.
Ronald W Schafer. What is a savitzky-gokka filter?[lecture notes]. IEEE Signal processing magazine, 28(4): 11--17,, 2.
13

---

Published as a conference paper at ICLR 2025
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXtv:2111.02114 2021.
Kyeongeun Seo, Hyeonjoong Cho, Daewoong Choi, Sangyub Lee, Jackyu Lee, and Jaejin Ko. Twohandsmusic: multitask learning-based egocentric piano-playing gesture recognition system for two hands. ICIP, pp. 4614-4618, 2019.
Takaaki Shiratori, Atsushi Nakazawa, and Katsushi Ikeuchi. Dancing-to toahingg-t-uusic character animation. In CGF, pp. 449-44444 2006.
Tomas Simon, Hanbyul Joo, lain Matthews, and Yaser Sheikh. Hand keypoint detection in single images using multiview bootstrapping. In CVPR, pp. 1145-1153, 2017.
Li Siyao, Weijiang Yu, Tianpei Gu, Chunze Lin, Quan Wang, Chen Qian, Chen Change Loy, and Ziwei Liu. Bailando: 3d dance generation by actor-critic gpt with choreographic memory. In CVPR. pp. 11050-11059, 2022.
Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Mare Najork. Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning. In SIGIR, pp. 2443-2449, 2021.
Sebastian Starke, Ian Mason, and Taku Komura. Deepphase: Periodic autoencoders for learning motion phase manifolds. TOG, 4 1133 2022.
Kun Su, Xiulong Liu, and Eli Shlizerman. Audeo: Audio generation for a silent performance video. NewrlPS, 33:3325-3333 -337, 2020.
Guofei Sun, Yongkang Wong, Zhiyong Cheng, Mohan S Kankanhalli, Weidong Geng, and Xiang- dong Li. Deepdance: music-to-dance motion choreography with adversarial learning. TOMM, 23:497-509, 2020.
Taoran Tang, Jia Jia, and Hanyang Mao. Dance with melody: An lstm-autoencoder approach to music-oriented dance synthesis. In ACM MM, pp. 1598-1606, 2018.
Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H Bermano. Human motion diffusion model. arXiv preprint arXiv: 14916, 2022.
Guanzhong Tian, Yi Yuan, and Yong Liu. Audio2face: Generating speech/face animation from single audio with attention-based bidirectional Istm networks. In JCMEW, pp. 366-371, IEFE, 2019.
Linrui Tian, Qi Wang, Bang Zhang, and Liefeng Bo. Emo: Emote portrait alive- geree--eneratingg ex- pressive portrait videos with audio2video diffusion model under weak conditions. arXiv preprint arXiv:2402.1 17485, 2024.
Keisuke Toyama, Taketo Akama, Yukara Ikemiya, Yuhta Takida, Wei-Hsiang Liao, and Yuki Mitsu- fuji. Automatic piano transcription with hierarchical frequency-time transformer. arXiv preprint arXiv:2307.04305. 2023.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. NeurIPS, 30, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, łukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurIPS, 30, 2017.
Jianbo Wang, Kai Qiu, Houwen Peng, Jianlong Fu, and Jianke Zhu. Ai coach: Deep human pose estimation and analysis for personalized athletic training assistance. In ACM MM, pp. 374-382, 2019.
Jiayi Wang, Franziska Mueller, Florian Bernard, Suzanne Sorli, Oleksandr Sotnychenko, Neng Qian, Miguel A Otaduy, Dan Casas, and Christian Theobalt. Rgb2hands: real-time tracking of 3d hand interactions from monocular rgb video. ACM TOG, 39 2020.
14

---

Published as a conference paper at ICLR 2025
Zhe Wang, Petar Veličković, Daniel Hennes, Nenad Tomašev, Laurel Prince, Michael Kaisers, Yoram Bachrach, Romuald Elie, Li Kevin Wenliang, Federico Piccinini, et al. Tacticai: an ai assistant for football tactics. Nature communications, 15(1):1-13, 2024.
Erwin Wu, Hayato Nishioka, Shinichi Furuya, and Hideki Koike. Marker-removal networks to collect precise 3d hand data for rgb-based estimation and its application in piano. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 2977-2986, 2023.
Nelson Yalta, Shinji Watanabe, Kazuhiro Nakadai, and Tetsuya Ogata. Weakly deep recurrent neural networks for basic dance step generation. In IJCNN, pp. 1-8, 2019.
Kazuki Yamamoto, Etsuko Ueda, Tsuyoshi Suenaga, Kentaro Takemura, Jun Takamatsu, and Tsukasa Ogasawara. Generating natural hand motion in playing a piano. In IROS, pp. 3513- 3518, 2010.
Yuichiro Yonebayashi, Hirokazu Kameoka, and Shigeki Sagayama. Automatic decision of piano fingering based on a hidden markov models. In IJCAI, volume 7, pp. 2915-2921, 2007.
Youngwoo Yoon, Bok Cha, Joo-Haeng Lee, Minsu Jang, Jaeyeon Lee, Jachong Kim, and Geehyuk Lee. Speech gesture generation from the trimodal context of text, audio, and speaker identity. TOG, 39(6):1-16, 2020.
Ye Yuan, Jiaming Song. Umar lqbal, Arash Vahdat, and Jan Kautz. Physdiff: Physics-guider human motion diffusion model. In ICCV, pp. 16010-16021, 2023.
Egor Zakharov, Aliaksandra Shysheya, Egor Burkov, and Victor Lempitsky. Few-shot adversarial learning of realistic neural talking head models. In ICCV, pp. 9459 9468,9948, 2019.
Kevin Zakka, Philipp Wu, Laura Smith, Nimrod Gileadi, Taylor Howell, Xue Bin Peng, Sumeet Singh, Yuval Tassa, Pete Florence, Andy Zeng, et al. Robopianist: Dexterous piano playing with deep reinforcement learning. In CoRL, 2023.
Chenxu Zhang, Yifan Zhao, Yifei Huang, Ming Zeng, Saifeng Ni, Madhukar Budagavi, and Xiaohu Guo. Facial: Synthesizing dynamic talking face with implicit attribute learning. In ICCV, pp- 3867-3876, 2021.
Mingyuan Zhang, Xinying Guo, Liang Pan, Zhongang Cai, Fangzhou Hong, Huirong Li, Lei Yang, and Ziwei Liu. Remodiffuse: Retrieval- augmenttd motion diffusion model. ICCV, pp. 364- 373, 2023.
Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondiffuse: Text-driven human motion generation with diffusion model. TPAMI, 2024.
Yihao Zhi, Xiaodong Cun, Xuelin Chen, Xi Shen, Wen Guo, Shaoli Huang, and Shenghua Gao. Livelyspeaker: Towards semantic-aware co-speech gesture generation. In ICCV, pp. 20807- 20817, 2023.
Ye Zhu, Kyle Olszewski, Yu Wu, Panos Achlioptas, Menglei Chai, Yan Yan, and Sergey Tulyakov. Quantized gan for complex music generation from dance videos. ECCV, pp. 182-199, 2022.
Wenlin Zhuang, Congyi Wang, Jinxiang Chai, Yangang Wang, Ming Shao, and Siyu Xia. Mu- sic2dance: Dancenet for music-driven dance generation. TOMM, 18(2):1-21, 2022.
Christian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan Russell, Max Argus, and Thomas Brox. Freihand: A dataset for markerless capture of hand pose and shape from single rgb images. In ICCV, pp. 813-822, 2019.
15

---

Published as a conference paper at ICLR 2025
APPENDIX
In this part, we further provide more details and discussions on our proposed dataset and benchmark:
More statistics of PianoMotion 10M:
B
More visual samples PianoMotion 10M dataset;
More details of our baseline;
Limitation and future work;
Ethical considerations;
Author statement;
A
MORE STATISTICS OF PIANOMOTION10M
Counts (Million)
5
2
13
19
25
31
37
43
49
55
61
67
73
79
85
91
97
103
109
115
121
127
Note
14
Counts (Million)
10
12
10 16
22
25
28
31
34
37
40
43
46
49
52
55
58
61
64
67
70
73
76
79
85
Volume
Figure Al: Distribution of Note Clicks and Volume Levels in the PianoMotion 10M Dataset. The top figure depicts note click frequency, and the bottom one shows the volume distribution.
Fig. Al presents detailed statistical analysis of piano fingerings, which focuses on the frequency of note clicks and the distribution of volume levels.
Note Click Counts. The top figure in Fig. A displays the frequency of each note played, measured in millions. This distribution spans 128 keys of the piano, which indicates frequent usage of those keys in performances. shows higher counts in specific note ranges.
Notes around the middle of the keyboard, particularly from C4 to C6, exhibit significantly higher click counts, aligning with their common use in melodies and harmonic accompaniments. Con- versely, notes in the low (AO to B1) and high (C7 to C8) octaves have markedly fewer clicks, as these ranges are less frequently used and typically reserved for specific musical effects or embel- lishments. It is worth noting that, certain notes, particularly those fundamental to common chords and scales (e.g.. A, C, E, and G). exhibit higher frequencies, reflecting their frequent use in various musical pieces.
Volume Distribution. In addition to note frequency, the figure below presents a comprehensive distribution of volume levels, spanning various ranges to highlight the dynamics of piano playing. Volume counts, measured millions, provide insights into the intensity and expression captured in constructed dataset.
There is a higher count of notes played at moderate volume levels. This reflects the natural dynamics of piano playing, where most notes neither extremely soft loud.
B MORE VISUAL SAMPLES OF PIANOMOTION 1OM DATASET
We provide visual samples of PianaMotion 10M dataset, illustrated in Fig. A2
16

---

Published as a conference paper at ICLR 2025
Hand Pose
Audio
MIDI
Sample I. BVIze4 T7q6
Hand Pose
Audio
MIDI
Sample 2. BVI 1879
Hand Pose
Audio
MIDI
Sample 3. BV1EDAy1D75
Hand Pose
Audio
MIDI
Sample 4. BVISu411v7d
Hand Pose
Audio
MIDI
Sample 5. BVIzo4ylm7qP
Figure A2: More samples from PianoMotion10M dataset. BV*** denote the corresponding video iDs.
17

---

Published as a conference paper at ICLR 2025
C MORE DETAILS OF OUR BASELINE
The audio feature extractor + maps the audio A to the feature vector fa- We use pre-trained Wav2Vec2.0 (Baevski et al. 2020) and HuBERT (Hsu al. 202I) models developed by Facebook AI for a.. Both models leverage extensive unlabeled data for unsupervised pretraining to learn high- dimensional speech representations. HuBERT extends its sumi-supee learning approach with pseudo-labels to self supervised learning. In our experiments, we use wav2vec2- and hubert-base 1s960 for the base model, and www.vec2-large- -aree2--arre--h-----0------ -ewee-990h--vv--eee--- and hubert-la aarelarre--- a960- ft for the large model as the audio feature extractor 4a.
D LIMITATION AND FUTURE WORK
Figure A3: Samples of failure in the generated hand motions when dealing with excessively fast music.
Our dataset is derived from 14 subjects and faces an issue of imbalance, primarily due to the varying levels of performance provided by authorized creators. In extreme performance scenarios, such as rapid, multi-hand, or non-human piano audio, our baseline method may yield incorrect hand transitions due to -iitt---iitiitttionn challenges, as illustrated Fig. A3 Additionally, not all videos meet the required data quality standards. In the future, we plan to engage creators on other platforms, such as YouTube and TikTok, to further enhance the diversity of our dataset.
Our dataset closely associates piano music with hand movements. Due to data diversity, we have not yet aligned piano positions in all videos. We plan to engage extra experts to annotate piano key positions for more precise spatial alignment in the next version. Additionally, the variance in piano tones across recordings may also affect the bascline model's performance.
PianoMotion 10M provides piano music, corresponding MIDI files, and hand poses, offering searchers a valuable resource for studying numan-piano interaction. This dataset enables the analy- sis of piano music through hand gestures and the generation of hand poses from audio tracks. With PianoMotion 10M, we hope to benefit and facilitate further research in relevant fields.
E ETHICAL CONSIDERATIONS
Piano motion datasets may pose significant privacy challenges, particularly concerning the pianist's s identifiable aspect, mainly their hands, during piano performance. Our dataset comprises videos uploaded by users on Bilibili, which are publicly accessible. During the dataset collection process, carefully selected the high-quality piano performance videos with bird's s-eye view Bilibili and obtained permissions from the respective creators to use their videos, and accessed the original content. Those videos from creators who did not grant permission did not respond have been included in the dataset. We informed the creators that we would extract hand poses and audios from the videos to create a publicly available, commmmmmrcaall dataset. We clarified that their personal likenesses would not be used. The dataset also included metadata, such the video URL, to trace the original video. In the camera-ready version, we will acknowledge all the creators' contribution in building our dataset.
F AUTHOR STATEMENT
The authors bear all responsibility in case of violation of rights. We confirm that the PianoMo- t0o dataset is open-sourced under the CC BY-NC 4.0 International license and the released code is publicly available under the Apache-2.0 license, for academic and research purposes.
license, ensuring open access and permissive usage
18


Reviews:
[{"review_id": "VOK8wCFx3a", "reviewer": "Reviewer_RbsM", "summary": "This paper presents PianoMotion10M, a large-scale dataset designed to advance hand motion generation in piano performance, featuring 116 hours of piano videos annotated with 10 million hand poses. To complement this dataset, the authors introduce a baseline model that generates realistic and continuous hand motions from piano audio, using a position predictor and a gesture generator. The dataset and model are aimed at improving AI-driven piano instruction, with metrics to assess motion quality, smoothness, and positional accuracy.", "questions": "Please see Weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Originality and Dataset Contribution: The PianoMotion10M dataset provides a novel resource, combining piano audio and video data with detailed 3D hand pose annotations across 116 hours of performance. This dataset addresses a significant gap by providing high-resolution, continuous hand motion data, which is essential for realistic AI-driven piano instruction and training, especially in guiding hand positioning and transitions.\n\n2. Comprehensive Benchmark and Evaluation Metrics: The paper introduces a set of detailed evaluation metrics, including Frechet Gesture Distance, Wasserstein Gesture Distance, and Position Distance, to assess motion fidelity, positioning accuracy, and smoothness. This comprehensive approach to benchmarking supports the model’s effectiveness and provides a robust foundation for future research in hand motion generation.\n\n3. Well-defined Baseline Model: The baseline model combines a position predictor with a gesture generator to generate realistic hand motions based on audio input, demonstrating the dataset’s utility. The use of a diffusion probabilistic model for gesture generation adds an innovative approach to capturing continuous motion, enhancing the quality of generated hand movements and reinforcing the practical application of PianoMotion10M in piano instruction.", "weaknesses": "1. Supplementary Material and Generation Results: The supplementary material lacks corresponding piano music for the generated hand motions, making it difficult to assess the quality and realism of the generated performance. Including paired audio and visual outputs would provide a more complete evaluation of the model’s effectiveness.\n\n2. Dataset Collection Process:\nThe collection and preprocessing of data, essential to any dataset creation work, lack clarity. Specifically, the paper does not thoroughly explain the data acquisition process in Section 3.1.\nWhile the authors mention in line 213 that \"3,647 candidate videos\" were collected, it remains unclear how these videos were queried or selected.\nIn line 240, the authors state that they \"remove videos where hand regions are frequently obstructed or invisible.\" However, they do not provide a clear definition of what constitutes \"frequently obstructed or invisible,\" making it hard to understand the filtering criteria.\nLine 236 mentions manually selecting pure piano music to ensure no vocals or sounds from other instruments, but it’s unclear why an automated audio event detection approach (such as [1]) wasn’t considered for this task.\n\n3. Annotation Quality and Relevance:\nThe quality of annotations is crucial in generating reliable datasets. In line 251, the authors mention \"we transcribe the piano performances into MIDI files by utilizing a state-of-the-art automatic piano transcription (Kong et al., 2021)\". However, this method, although effective, was developed in 2021. The authors do not address whether they explored more recent transcription methods (such as [2-5]) or provide experiments comparing different methods to validate their choice.\n\n4. Additionally, while MIDI signals are provided, they are not used in the hand motion generation task. It would be beneficial to validate the quality and relevance of MIDI annotations, potentially by incorporating MIDI data into the baseline model to assess its impact on hand motion generation. This could offer valuable insights for future research.\n\n[1] Kong Q, Cao Y, Iqbal T, et al. Panns: Large-scale pretrained audio neural networks for audio pattern recognition[J]. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2020, 28: 2880-2894.\n\n[2] Maman B, Bermano A H. Unaligned supervision for automatic music transcription in the wild[C]//International Conference on Machine Learning. PMLR, 2022: 14918-14934.\n\n[3] Gardner J, Simon I, Manilow E, et al. MT3: Multi-task multitrack music transcription[J]. arXiv preprint arXiv:2111.03017, 2021.\n\n[4] Toyama K, Akama T, Ikemiya Y, et al. Automatic piano transcription with hierarchical frequency-time transformer[J]. arXiv preprint arXiv:2307.04305, 2023.\n\n[5] Cheuk K W, Sawata R, Uesaka T, et al. Diffroll: Diffusion-based generative music transcription with unsupervised pretraining capability[C]//ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023: 1-5.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8mmZAwzgsk", "reviewer": "Reviewer_Jc4e", "summary": "The paper has two main contributions:\n1) A dataset of 100+ hours of piano performance audio and (overhead) video, MIDI, and hand motion sequences, all aligned.\n2) A baseline method for generating hand motion sequences from piano audio.", "questions": "I don't have questions though I'd probably accept the paper as is.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The main strength of the paper is that this is a really cool dataset linking piano performance audio (and MIDI) to hand motion sequences.  Piano playing is a challenging task requiring extreme dexterity, and this dataset seems like an important contribution to the research community.\n\nThe paper is clearly written and provides something new and of value.", "weaknesses": "The biggest thing missing is some sort of verification that a hand motion sequence *actually results in the target audio* when applied to a piano, using a physical model.  I recognize that this may be considered out of scope, but without it it's hard to trust even the ground truth provided in the dataset.  And then all of the metrics for the baseline method end up comparing generated hand motions with the ostensible ground truth hand motions, which has two issues:\n1) The ground truth hand motions may have errors.\n2) Even if the ground truth hand motions are accurate, there may be multiple possible hand motions to generate a given performance audio, none more \"correct\".\nHowever, I think this may be okay if the dataset is positioned as \"how some pianists performed these pieces\" rather than \"how one *should* perform these pieces\".\n\nAn uncited research paper related to the above point is RoboPianist (Zakka et al. 2023): https://kzakka.com/robopianist/\n\nI'm not sure why the baseline method included attempts to infer hand motions directly from audio, rather than going through MIDI.  MIDI (including note velocities) seems like it would contain all of the information needed to reconstruct the hand motions, and is *much* more compact than the raw audio.  If this is incorrect, it would still probably be worth mentioning in the paper.\n\nThe paper doesn't include a discussion of sustain pedal.  It seems as though the piano transcription method used *does* transcribe pedal events, which seem like they would be necessary to include in the baseline method otherwise much piano audio would be impossible to perform hands-only.  However, since the baseline method goes straight from audio to hand motions, the model is forced to implicitly transcribe sustain pedal in order to determine the hand motions.  Is that correct?\n\nThe paper raises a bunch of questions that I am curious about but were not discussed, specifically related to limits of the method and failure modes.  The examples included in the supplementary material are sort of \"easy\".  What happens if you give the baseline method virtuosic piano audio e.g. the MAESTRO dataset?  What happens if you speed up the audio so that it's no longer humanly playable?  What happens if you play intervals that human hands can't accommodate?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DkVUhubepg", "reviewer": "Reviewer_wnCC", "summary": "The paper addresses the challenge of hand motion generation for piano performance, recognizing the gap in current AI systems for musical instruction, particularly for instruments like the piano where hand positioning is critical.  The proposed PianoMotion10M serves as an important dataset aimed at solving this problem.", "questions": "1. Do authors have some human evaluation results for the generated motions? It would be better if these could be incorporated.\n\n2. Ethic concern and copyright issue: Since permissions were obtained from Bilibili users, how does the dataset’s open accessibility align with potential copyright restrictions or privacy concerns, particularly for identifiable hand features? I know that they do have some restrictions on this.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This is a large-scale dataset comprising 116 hours of piano performance from 14 professional players and 10 million annotated frames. The videos are collected under specific constraints (e.g., high resolution, bird’s-eye perspective) to ensure quality and consistency.\n\n2.  The data collection and annotation method is well-structured, using tools like MediaPipe and HaMeR for robust hand pose detection. \n\n3. The dataset is indeed a valuable contribution, likely to support advancements in both piano instruction and broader motion generation research.\n\n4. The two-stage design—featuring a position predictor and a position-guided gesture generator—seems well-suited for this task and shows reasonable quality. The model uses a diffusion-based architecture with a U-Net structure, which allows it to generate natural, fluid gestures that are better suited to piano performance.", "weaknesses": "1. The paper acknowledges the dataset’s limitations, particularly the need for improved diversity in piano positions and video sources, now only include 14 people. Addressing the inconsistency in piano tones and alignment of keyboard positions remains critical for achieving high-fidelity results.\n\n2. A suggestion to include related work: Audeo: Audio Generation for a Silent Performance Video. Kun et,al (NeurIPS 2020).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mxB30nR5y8", "reviewer": "Reviewer_ik5L", "summary": "This paper proposes a large dataset consisted of piano music and aligned MIDI and hand positions containing 116h of recordings and 14 subjects. The goal of the dataset is to train generative processes to renderize 3d human-hand motion videos. This work presents an interesting baseline method to renderize hands.", "questions": "I do have questions regarding the text:\n\nSection 4.3: The learning rates are said to be 2e^{-5} and 5e^{-5}. The math reads: 2 (or five) times exp(-5). The usual reporting method is to use 2 (or 5) times 10^{-5}. Was that a typo, or are you really using the Euler number to define learning rates? If so, is there a reason for such?\n\nSection 3.2: \n- MIDI files with high discrepancies were adjusted: what is the measure for a \"high discrepancy\"? The issue here is that, if this dataset is going to be used as a benchmark, there should be a measure on how much it differs from reality.\n\n- Data uses a Savitzky-Golay filter. This filter is applied to the data in the dataset to increase smoothness. What are the filter parameters (order, frequency response, etc.)? Also, why was this specific type of filter chosen instead of the standard ones (a Butterworth filter seems adequate)? What is the impact of not using the filter?\n\nSection 3.3: All videos are accessible with the IDs in the Bilibili website. Is Bilibili aware and consenting with the idea that there will probably be robots scraping their data without using their website? That is, are there chances that Bilibili will simply include barriers that would harm the dataset? Also, if users take down their own videos, how would that impact the dataset?\n\nSection 4.3: The GPU has 24GB of RAM. How big is the model itself? How long did training take? Also, are you planning on releasing the trained weights?\n\nSection 5.1:\n- Several metrics are proposed, which is a good thing. However, none of the metrics were discussed in terms of their relevance to the real problem. For example: what kind of \"error\" in the rendering could be highlighted by the FID, FGD or WGD? Without this discussion, the metrics simply pose a \"competition\" on who gets better values. A phrase such as \"The WGD metric offers a robust measure of dissimilarity between the generated motion and ground truth.\" does not clarify if this dissimilarity consists of a jittery behavior, an offset, an error in velocity, fingers being rendered too long (or too many!), and so on. Such a discussion, with clear examples, could clarify why these specific metrics were used, and not others.\n- Also, the metrics imply in a comparison with a ground truth. This is not wrong per se, but, when hands are rendered, is there really a \"ground truth\"? That is, if two pianists play the same excerpt, wouldn't their hands be slightly different? So, more than increasing the proximity to a ground truth, the goal of the algorithms should be to obtain results that are within a distribution for human differences, not simply to approach a unique ground truth.\n- In this same line of thought, does it really make a visible difference to have small variations in these metrics? For example, going from FGD=0.360 to FGD=0.351 is a difference of 0.009 - what type of difference does that reflect in the actual performance?\n- How do these measures compare with a Mean Opinion Score (MOS)? It would be great to have some evidence that the proposed measures can substitute MOS, because collecting human opinions is costly. However, we need evidence for such.\n- Minor suggestion: in Table 3, make the Right Hand appear to the right of Left Hand.\n- A not so minor aspect: why are the measurements so different between the hands? Afterall, we have the same amount of data and the same models for them, correct? According to the data, Wav2Vec+TF is better at rendering left hands, while HuBERT+TF is better at right hands (according to WGD), which seems to indicate that their performance is actually the same and the differences are due to noise/chance/variance.\n\nSection 5.2\n- Figure 4 states that \"Ours\" is close to \"GT\" (or: the blue hands are closer to the gray ones). This is true, at least for this example; but in general the green and yellow hands seem to be equally plausible hand positions. In the third line, the yellow hands (LiveSpeaker) seem to be even more emphatic on the fact that the left hand is not playing anything.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "In descending order of importance:\n\n- A large dataset for hand pose generation is proposed. This is original, clear, and significant.\n- A model is proposed. This is technically original, but the significance is not well discussed for the reasons stated in the comments.\n- A set of measures for performance is proposed. They are not original, nor their significance to the problem is discussed.\n\nOverall, I find the article interesting, although I am not sure how many people are specifically trying renderize hand poses in pianos using audio as input. However, this dataset could also be used for other tasks, which broadens its impact.\n\n- The initiative of making a dataset for this task is great, and the ethics statement covers most of the potential problems,\n- The dataset has some potential problems that could be taken care of, especially the long-term availability and the reliance on a specific company and several users to maintain the data,", "weaknesses": "There is a great emphasis in the proposed model whereas the proposed dataset and measures are not well discussed. I would suggest focusing the paper on the dataset and on the discussion on the metrics and skip the model proposal altogether. In special, this is because the model uses more data than any other model, hence it is expected to approach the data distribution better than other models.\n\n- The presented model is interesting, but its performance is close to EmoTalk (more on that in item 4)\n- I miss some discussions on the metrics and their relationship with the problem, and the space taken by describing the baseline method could be used for that. In fact, EmoTalk and LivelySpeaker, according to Table 3, are already interesting for future comparisons and their metrics are very close to the proposed method's.\n\nOne aspect that has not been addressed is a description of styles, genres, levels of expertise, and other biases in the dataset. This leads to thinking that the dataset polarized towards particular styles of playing (for example: ragtime versus classical). This is not a problem per se, as it is unavoidable; however, not *knowing* what these biases are can become a problem.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)