PianoMotion10M: Dataset and Benchmark for Hand Motion Generation in Piano Performance

## Abstract

Abstract Recently, artificial intelligence techniques for education have been received increasing attentions, while it still remains an open problem to design the effective music instrument instructing systems. Although key presses can be directly derived from sheet music, the transitional movements among key presses require more extensive guidance in piano performance. In this work, we construct a piano-hand motion generation benchmark to guide hand movements and fingerings for piano playing. To this end, we collect an annotated dataset, PianoMotion10M, consisting of 116 hours of piano playing videos from a bird’s-eye view with 10 million annotated hand poses. We also introduce a powerful baseline model that generates hand motions from piano audios through a position predictor and a position-guided gesture generator. Furthermore, a series of evaluation metrics are designed to assess the performance of the baseline model, including motion similarity, smoothness, positional accuracy of left and right hands, and overall fidelity of movement distribution. Despite that piano key presses with respect to music scores or audios are already accessible, PianoMotion10M aims to provide guidance on piano fingering for instruction purposes. The dataset and source code can be accessed at https://agnjason.github.io/PianoMotion-page .

## 1 Introduction

The process of learning has been significantly improved with artificial intelligence techniques, which enable individuals to enhance their skills under the guidance of an AI coach [, ].
This can be extended into learning to play the musical instruments. Particularly, piano performance requires a profound understanding of the underlying relationship between musical compositions and their corresponding physical motions.
As the rigorous practice and training program are necessary for athletes to master a diverse range of expressive human poses, it entails extensive practices to achieve proficiency in piano fingering and hand movement. To facilitate access to playing guidance, the development of AI piano coach has been spurred.

Large-scale piano-motion datasets are the foundation of a nuanced approach for motion generation, which offer valuable guidance for physical performance and musical expression.
The computational challenge of motion generation lies in capturing the nonlinear relationship between musical pieces and the intricate hand motions required for piano playing.
The hand poses vary for the same note across different melodies. Moreover, the dynamic nature of musical expression demands a level of continuous motion, which is challenging to be learnt from small datasets.
These limitations underscore the urgent need for a large-scale piano-motion dataset.

To address the deficiency of the dataset for guiding hand movements and fingerings in playing piano, we introduce a large-scale 3D piano-motion dataset named PianoMotion10M.
As shown in Fig. 1, PianoMotion10M contains piano audio tracks, Musical Instrument Digital Interface (MIDI) files, and annotated hand motions with their corresponding videos, meticulously collected from the Internet.
It comprises 1,966 pairs of video and music, with a total duration of 116 hours and 10 million annotated frames.
The parametric MANO hand model [] is employed to represent hand gestures.
Our constructed dataset offers a diverse range of music styles and piano techniques, which addresses the demands of various preferences and skill levels.

Figure: Figure 1: Overview of our framework. We collect videos of professional piano performances from the Internet and process them to construct a large-scale dataset, PianoMotion10M, which comprises piano music, MIDI files and hand motions. Building upon this dataset, we establish a benchmark for generating hand motions from piano music.
Refer to caption: x1.png

Traditional applications like PianoPlayer [] are adept at generating static hand gestures and positions for piano scores, which typically make use of classifiers to predict the proper fingering combinations. However, they often fail to capture the diversity and continuity inherent of the piano performance in PianoMotion10M.
Both rule-based methods [, ] and HMM-based approaches [, ] can estimate fingerings while they primarily focus on the local fingering constraints of continuous notes.
Consequently, they often overlook crucial information like long-range fingering relationships, while our task aims to estimate the motions of long clips.

To address these limitations, a novel baseline model is presented to show the effectiveness of PianoMotion10M, which is able to generate realistic hand motions from piano melodies.
Given a piece of piano music, our model can locate the positions of both hands and generate a long sequence of hand gestures for the performance.
It effectively learns the music-position correlation through an efficient position predictor and produces continuous gestures with a position-guided gesture generator based on a diffusion probabilistic model.
To assess our baseline model, we propose several evaluation metrics, including Frechet Gesture Distance and Wasserstein Gesture Distance to measure the fidelity of each hand motions, Frechet Inception Distance with a pre-trained auto-encoder to investigate the motion quality of double hands, and Position Distance to assess the accuracy of hand positioning, and Smoothness of the generated motions.

In summary, our main contributions are: 1) A large-scale piano-motion dataset PianoMotion10M comprises 116 hours of music and 10 million annotated frames with hand poses. To the best of our knowledge, it stands as the first dataset integrating piano music with its corresponding hand motions, which facilitates the tasks of 3D hand motion generation from piano audio tracks and piano music generation conditioned on hand motions.
2) Based on PianoMotion10M, we propose a benchmark with a series of evaluation metrics to investigate the effectiveness on hand motion generation, including the accuracy of positions and fidelity of gestures.
3) A novel baseline model bridges piano music with hand motions, which estimates hand location with a position predictor and generates hand gestures sequences through a position-guided gesture generator.

## 2 Related Work

Motion-music Datasets. While multi-modal datasets [, , , ] become the key of various learning tasks, there remains a significant gap in the availability of repository specially designed for music-conditional motion generation. Although the existing hand gesture datasets [, , , , ] contain a large number of image-hand gesture pairs, they do not have audio or other related information. This limitation hinders them from the generative tasks, since they mainly focus on hand reconstruction and pose estimation.
Recently, datasets like AIST++ [] and TikTok [] are tailored for music-dance learning, which provide limited music segments, typically less than 5 hours in duration.
Moryossef et al. [] manage to automatically detect which fingers pressed the key of the piano and provide a dataset of piano-fingering. However, it does not provide continuous hand gesture movements.
Therefore, there is a crucial need to build a piano-motion dataset specially tailored for motion generation tasks. To this end, we introduce the PianoMotion10M dataset, which comprises extensive piano music and their corresponding hand motion annotations.

3D Human Motion Synthesis. The problem of generating realistic and controllable 3D human motion sequences has been a subject of long-standing study.
By taking advantage of 2D keypoint detection [], the synthesis of 2D skeletons has been extensively explored [, , ].
Considerable research efforts have been devoted to 2D speech-driven head generation for facial mouth and lip motion generation [, , , , ], which usually employ either image-driven or voice-driven methods to produce realistic videos of speaking individuals.
However, the expressive capabilities of 2D pose skeletons are limited and they are not applicable to 3D character models.
Recent methods for full-body 3D dance generation have utilized LSTM [, , ], GANs [, ] or transformer encoders [, , ].
To generate vivid talking head videos, extensive research has been conducted in the field of speech-driven 3D facial animation [, , , , ].
EmoTalk [] animates emotional 3D faces from speech input by generating controllable personal and emotional styles.
Tian et al. [] introduce a speed controller and a face region controller to enhance stability during the head generation process.

In the domain of hand motion generation, it has been primarily categorized into rule-based methods [, , ] and data-driven approaches [, , , , ].
For instance, Speech2Gesture [] utilizes conditional generative adversarial networks to generate personalized 2D keypoints from audio.
Ahuja et al. [] propose a method for personalized motion transfer.
Ao et al. [] learn the mapping between the speech and gestures from data using a combined network structure of the vector quantized variational auto-encoder (VQ-VAE). Beyond 2D keypoints generation, TriModal [] extracts different upper body movements from TED talks and designs a LSTM-based neural network conditioned on audio, text, and identity to generate co-speech gestures.

Recently, diffusion models have achieved promising results in generating human motions. Previous works such as MDM [] and MotionDiffuse [] have produced realistic motion inspired by denoising diffusion probabilistic models (DDPM) []. PhysDiff [] extends MDM by imposing physical constraints. MLD [] utilizes latent carrier DDPM for forward denoising and reverse diffusion in motion latent space. MAA [] enhances the performance of non-distributed data by pre-training diffusion models. Zhang et al. [] introduce retrieval-enhanced DDPM, which improves the text-to-motion functionality in distribution.

## 3 PianoMotion10M Dataset

It is a formidably challenging task to map piano music to hand motions due to the significant influence of note sequences and positions on hand movements and fingering.
Lacking diverse data may lead to the inferior performance on estimating various hand poses in piano playing.
To capture the variability, we present the first large-scale piano-motion dataset, PianoMotion10M, which comprises 1,966 piano performance videos along with 10 million hand poses and their corresponding MIDI files, resulting in an overall duration of 116 hours. Fig. 2 presents an example of our dataset.
These videos are segmented into 16,739 individual clips with a length of 30 seconds. To ensure comprehensive evaluation, we extract 7,519 clips for training, 821 for validation and 8,399 for testing.
The detailed comparisons with the existing datasets on hands and 3D human motions are summarized into Tab. 1.

**Table 1: Comparison between different hand and motion datasets. The proposed PianoMotion10M dataset consists of piano music with corresponding hand poses for hand motion generation. Existing hand-image datasets are listed in the first four rows, and music-motion datasets are presented in the subsequent four rows for reference.**
| Dataset | Pose | Size | Subject | Music | MIDI | Duration(hour) |
| --- | --- | --- | --- | --- | --- | --- |
| FreiHAND [] | ✓ | 134K | 32 | ✗ | ✗ | - |
| InterHand2.6M [] | ✓ | 2.6M | 27 | ✗ | ✗ | - |
| RGB2Hands [] | ✓ | 1K | 2 | ✗ | ✗ | - |
| Re:InterHand [] | ✓ | 1.5M | 10 | ✗ | ✗ | - |
| GrooveNet [] | ✓ | - | 1 | ✓ | ✗ | 0.38 |
| DanceNet [] | ✓ | - | 2 | ✓ | ✗ | 0.96 |
| EA-MUD [] | ✓ | - | - | ✓ | ✗ | 0.35 |
| AIST++ [] | ✓ | 10.1M | 30 | ✓ | ✗ | 5.19 |
| PianoMotion10M | ✓ | 10.5M | 14 | ✓ | ✓ | 116.16 |
|  |  |  |  |  |  |  |

Figure: Figure 2: Illustration of sample from PianoMotion10M. Each sample in our dataset includes audio, hand pose annotations, and a MIDI file along with the corresponding Bilibili video ID.
Refer to caption: x2.png

### 3.1 Data Collection

There is a wealth of videos and live streams dedicated to musical instrument performances and tutorials from the Internet. Note that each individual has a unique playing style, we firstly select $14$ piano experts from Bilibili(^1^11[https://www.bilibili.com](https://www.bilibili.com), one of the most popular video-sharing platforms in China.) and collect a total of 3,647 candidate videos. To ensure consistency and enhance the quality of our dataset, we conduct pre-processing with five pivotal factors, including resolution, audio quality, camera perspective, presence of multiple individuals, and visibility of hands. We manually select pure piano music to ensure that the audio contained no human vocals or sounds from other instruments. Moreover, videos with a resolution lower than $1080\times 1920$ are discarded to ensure the quality of our dataset. To enhance the observation of hand movements and gestures, we choose videos with a bird’s-eye view to minimize hand occlusion. Furthermore, we remove those videos where hand regions are frequently obstructed or invisible during performance. Additionally, we do not take consideration of videos with multiple piano players, which is beyond the scope of this dataset. By addressing these factors, the overall quality and coherence of the dataset have been substantially enhanced. Following this preprocessing stage, the result is a collection of $1,966$ high-quality raw videos. Each one showcases the piano playing by an individual, which is captured from a bird’s-eye view along with pure piano audio.

### 3.2 Data Annotations

MIDI is a universal digital protocol to store musical data for various musical devices, which is served as a digital representation of musical notes, volume, tempo, and other performance parameters. Once video and pure piano audio pairs are collected, we transcribe the piano performances into MIDI files by utilizing a state-of-the-art automatic piano transcription []. To ensure the accuracy of the conversion results, we replay MIDI files and compare them with the original music tracks. Those files with high discrepancies are adjusted.

Hand Pose is captured via the parametric hand model MANO [], which serves as our hand prior model. It effectively maps the pose parameter $\theta\in\mathbb{R}^{J\times 3}$ with $J$ per-bone parts and the shape parameter $\rho\in\mathbb{R}^{10}$ onto a template mesh $\bar{\mathcal{M}}$ with vertices $V$.
The MANO model enables the simulation of various hand gestures and movements during piano performances, which provides an effective way to study the relationship between hand gestures in playing piano. Due to the non-uniformity of hand sizes and positions in the collected videos, we first employ the MediaPipe hand detection framework [] to obtain bounding box of the hand region. Video frames are cropped according to the detected hand bounding boxes to enhance the robustness of the results. To this end, hand poses in collected videos are annotated using HaMeR []. HaMeR follows a fully transformer-based architecture and reconstructs hand models with increased accuracy and robustness. All images having hands detected by MediaPipe are annotated with hand poses by HaMeR, which result in a dataset of 10 million image-pose pairs.

To enhance the smoothness and continuity of our dataset, the generated hand poses require to be cleaned and refined. The results of MediaPipe and HaMeR are usually accurate in most cases, while some inferior results may occur due to rapid motion and image blurring. The Hampel filter [] with a window size of $20$ is utilized to identify these outliers. The outliers and the timestamps where hands are undetected are firstly labeled as missing values. Within a small period, hand movements can be considered as the motion with constant speed. Therefore, the small gap can be interpolated bilaterally.
To address these missing data in the time series, missing segments with the frame length $\delta$ less than $30$ frames are filled by linear interpolation with respect to their surrounding values. The others are considered as periods when the hand is invisible. To ensure the reliability of detected hands, a similar strategy is employed to label observations with excessively short duration ($\delta<15$) as invisible. Finally, to ensure the smoothness of hand motions, we make use of a Savitzky-Golay filter [] for data smoothing. The annotated hand poses are manually checked to ensure their quality.

### 3.3 Data Statistics

The dataset is made of contributions from several subjects with different experts, and each provides varying amounts of data in terms of videos, clips, duration, and annotated frames.
Tab. 2 presents the detailed statistics on the distribution of subjects within our PianoMotion10M dataset.

**Table 2: Statistics on the distribution of subjects in the PianoMotion10M dataset, where subject names denote the identity ID of experts.**
| Subject Name | Videos | Clips | Time(sec) | Frames | Subject Name | Videos | Clips | Time(sec) | Frames |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 1467634 | 337 | 4,359 | 103,293 | 2,237,181 | 470175873 | 14 | 100 | 2,450 | 64,946 |
| 2084102325 | 11 | 103 | 2,766 | 70,243 | 478315001 | 285 | 2,674 | 62,615 | 1,792,707 |
| 36760114 | 22 | 160 | 4,772 | 106,037 | 494725787 | 1 | 12 | 586 | 7,488 |
| 37367458 | 114 | 859 | 20,802 | 571,525 | 66685747 | 535 | 5,136 | 130,352 | 3,520,370 |
| 403444513 | 12 | 171 | 4,245 | 99,973 | 676539782 | 19 | 359 | 11,314 | 187,917 |
| 434762078 | 74 | 128 | 2,948 | 80,607 | 688183660 | 264 | 872 | 19,074 | 564,699 |
| 442401135 | 275 | 1,788 | 52,030 | 1,209,905 | 864712 | 3 | 18 | 923 | 13,569 |
| Total Videos: 1,966, Clips: 16,739, Total Duration(hour): 116.16, Annotated Frames: 10,527,167 |  |  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |  |

The whole piano-motion dataset, PianoMotion10M, consists of 1,966 videos with approximately 116 total hours of footage.
Each piece of music is segmented into 30-second clips at 24-second intervals, resulting in a total of 16,739 clips. Note that we discard those clips where hand visibility is below 80%. Our dataset has 14 subjects with different playing styles.
This variability ensures a rich diversity of playing techniques and music styles, which is essential for training robust models to predict piano hand gestures from musical pieces. All videos in our dataset are publicly accessible with provided video IDs on the Bilibili website.

## 4 Baselines

To tackle the challenging task of generating hand motions synchronized with piano audio tracks, we introduce a novel motion generation framework upon the PianoMotion10M dataset. As illustrated in Fig. 3, our framework consists of a position predictor and a gesture generator. The position predictor extracts the hand positions from piano music and integrates them as contextual input for the gesture generator. By leveraging a DDPM model [], the gesture generator estimates hand pose sequence based on the piano audio and the predicted positions. Further details on each component are elaborated in the following subsections.

Figure: Figure 3: Illustration of our baseline model. Given a piece of piano music, our baseline model estimates the hand motions by predicting hand positions and generating hand gestures.
Refer to caption: x3.png

Problem Formulation.
Given a piano audio piece $A$ with $N$ frames, our objective is to obtain the hand position sequence $P\in\mathbb{R}^{N\times 3\times 2}$ and hand gestures $\Theta\in\mathbb{R}^{N\times J\times 3\times 2}$. The hand position sequence $P$ encompasses the 3D coordinates of the left and right hands. The hand gestures $\Theta$ consist of Euler angle at each joint of both hands.

Due to the highly nonlinear relationship between acoustic signals and hand gestures, it poses a significant challenge to estimate motions through discriminative models [], which usually leads to an average pose, as demonstrated in Section 5.2. To address this issue, a hand position predictor is introduced to estimate the continuous changes in hand positions. Moreover, a generative model is utilized to reconstruct hand gestures from a piano music piece. Hereby, the task of hand motion generation becomes more concise and comprehensive by disentangling it into hand position estimation and gesture generation. To extract the audio features $f_{a}$ from the audio $A$, we make use of a pre-trained audio feature extractor $\Phi_{a}$ [, ], which is formulated as $f_{a}=\Phi_{a}(A),f_{a}\in\mathbb{R}^{N\times C}$. $C$ is the feature dimension of $\Phi_{a}$.

### 4.1 Position Predictor

The position predictor is employed to predict the 6-dimensional 3D positions for both the left and right hands. Since the audio features $f_{a}$ cannot directly be mapped to the positions, a feature embedding module is treated as our position decoder $\Phi_{p}$ to extract the latent features. It projects the sequential audio features $f_{a}$ onto the latent features $f_{p}$ with more comprehensive temporal information, which is formulated as $f_{p}=\Phi_{p}(f_{a}),f_{p}\in\mathbb{R}^{N\times 512}$. Subsequently, a linear mapping is employed to project the feature $f_{p}$ onto the output positions $P$.

In our experiments, we can make use of either Transformer [] or State-Space Model (SSM) [] as our feature embedding module.
Transformer leverages self-attention mechanisms to effectively capture long-range dependencies and contextual information in order to learn temporal relationships.
Recently, a different representation inspired by classical SSM [] is proposed to replace the attention mechanism, which is built upon a more contemporary Selective Structured State Space Model (S6) [] suitable for deep learning. By sharing a similar architecture to the classical RNN, it can efficiently capture information from previous inputs.

### 4.2 Position-guided Gesture Generator

Leveraging recent achievements [, ] in motion generation, our approach incorporates a diffusion probabilistic model [].
By mastering denoising, this model effectively captures the complex distribution of hand motions observed in large-scale piano-motion datasets, which has the capability to generate motions with different conditions.
Starting with a clean sample $\Theta_{0}$ of motion sequence, the forward diffusion process establishes a Markov chain that gradually adds noise to $\Theta_{0}$ in $T$ steps, which generates a series of noisy samples $\Theta_{1},...,\Theta_{T}$ as

$$ $q(\Theta_{t}|x_{0})=\mathcal{N}(\Theta_{t};\sqrt{\bar{\alpha}_{t}}x_{0},(1- \bar{\alpha}_{t})I),$ (1) $$

where $\mathcal{N}$ denotes the Gaussian distribution. $\bar{\alpha}_{t}=\prod_{s=1}^{t}(1-\beta_{s})$ and $\beta$ represent the variance scheduler for the added noise. Therefore, a model parameterized by a deep neural network $G$ is trained to master the inverse process within another Markov chain, which learns the mapping $p(\Theta_{t-1}|\Theta_{t})$ to sequentially denoise samples over $T$ steps. Specifically, denoising model $G$ consists of a 4-layer U-Net [] with 256, 512, 1024, 2048 dimensions for each layer.

To reduce the noise in piano audio, the audio features $f_{a}$ are simultaneously fed into the denoising neural network $G$ as conditions. Similar to the position decoder $\Phi_{p}$, a gesture decoder $\Phi_{g}$ is utilized to extract gesture features $f_{g}$ from the audio features $f_{a}$ as $f_{g}=\Phi_{g}(f_{a})$. Considering that the finger movements at different positions for the same pitch are distinct, $G$ requires the guidance of hand positions $P$ from the position predictor. This will enhance the fidelity of the generation process. As for the additional conditions, the time step embeddings of time $t$ are concatenated in denoising process. The denoising process can be formulated as follows

$$ $\hat{\Theta}_{0}=G(\Theta_{t},t;f_{g},P),$ (2) $$

where $\hat{\Theta}_{0}\in\mathbb{R}^{N\times J\times 3}$ denotes the result of hand motions within $N$ frames.

### 4.3 Implementation Details

We employ a two-stage scheme to train our proposed network. At the first stage, the position predictor is trained using position error $\mathcal{L}_{p}$ and velocity loss $\mathcal{L}_{v}$. The position error $\mathcal{L}_{p}$ computes the $L_{1}$ loss between the predicted position $\hat{P}$ and the ground truth $P$, expressed as $\mathcal{L}_{p}=||\hat{P}-P||_{1}$. Inspired by [], velocity loss is adopted to induce temporal stability for generating smoothing movements, which is formulated as $\mathcal{L}_{v}=||(\hat{P}_{n}-\hat{P}_{n-1})-(P_{n}-P_{n-1})||_{2}$. $n$ denotes the $n$-th frame in a motion sequence. Specifically, our model is trained by subject 1467634 and subject 66685747, which have the similar piano keyboard layout. At the second stage, the parameters of position predictor are frozen, and the estimated positions are employed as a guidance for the gesture generator. As in [], the denoising process is modified from noise prediction to velocity prediction. During the whole training process, the parameters of audio feature extractor $\Phi_{a}$ are frozen.

The baseline model is implemented with PyTorch. We normalize the inputs into 30 FPS through interpolation, where each piece of music lasts 8 seconds. Both stages involve training for $100,000$ iterations at the learning rates of $2e^{-5}$ and $5e^{-5}$, respectively. We conducted all the experiments on a PC with single NVIDIA RTX 3090Ti GPU, which has 24GB of GPU RAM.

## 5 Benchmark

### 5.1 Evaluation Metrics

To assess the performance of our proposed baseline, we employ several evaluation metrics to examine the effectiveness of hand poses generated by the input piano music, which are crucial in understanding the relationship between piano melody and its corresponding playing motions.

Frechet Inception Distance (FID).
Frechet Inception Distance is introduced to measure the Frechet distance between the feature vectors of prediction and ground truth. We pre-train an auto-encoder [] to project motion sequence onto a latent space for double hands. The FID is adopted to assess the fidelity of the overall motions generated by our baseline model.

Frechet Gesture Distance (FGD). Unlike FID, Frechet Gesture Distance is utilized to compute the disparity between predicted gestures and ground truth of one hand. This metric is instrumental in evaluating the similarity of single hand gestures.

Wasserstein Gesture Distance (WGD). Wasserstein Gesture Distance [] is computed between two distributions, each of which is represented as a Gaussian Mixture Model (GMM) [] as below

$$ $W(\mathcal{I}_{x},\mathcal{I}_{y})=\inf_{\gamma\sim\Pi(\mathcal{I}_{x}, \mathcal{I}_{y})}\mathbb{E}_{(x,y)\sim\gamma}\left[\|x-y\|\right],$ (3) $$

where $\Pi(\mathcal{I}_{x},\mathcal{I}_{y})$ indicates the joint distributions that combine the parametric GMM distributions $\mathcal{I}_{x}$ and $\mathcal{I}_{y}$. Distributions $\mathcal{I}_{x}$ and $\mathcal{I}_{y}$ are fitted by predicted gestures and ground truth of single hand. $\mathbb{E}$ calculates the expectation of the sample pairs $(x,y)$. The WGD metric offers a robust measure of dissimilarity between the generated motion and ground truth.

Position Distance (PD). The Position Distance calculates the $L_{2}$ distance between estimated positions and the ground truth. It is essential to evaluate the precision of predicted hand positions to ensure the accuracy required for piano fingerings.

Smoothness. Smoothness is measured by computing the acceleration of each joint. However, hands in a static state exhibit maximum smoothness, which are contrary to the desired outcome.
Consequently, we consider the acceleration of ground truth as a reference and utilize relative acceleration as the evaluation metric for smoothness, which is formulated as $Smooth=\sum_{i}|\hat{\tau_{i}}-\tau_{i}|$. $\tau_{i}$ denotes the acceleration of the $i$-th joint.
It reflects the continuance and coherence of the estimated gestures.

### 5.2 Experiments

**Table 3: Quantitative evaluation of our proposed hand motion generation baseline and existing models on the validation set. We present a comparative analysis of various network architectures, highlighting the performance and efficiency of our baselines in generating hand motion.**
| Method | Decoder | Step | Right Hand | Left Hand | FID$\downarrow$ | Params<br>(M) |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  |  | FGD$\downarrow$ | WGD$\downarrow$ | PD$\downarrow$ | Smooth$\downarrow$ | FGD$\downarrow$ | WGD$\downarrow$ | PD$\downarrow$ | Smooth$\downarrow$ |  |  |  |  |
| EmoTalk [] | TF | - | 0.360 | 0.259 | 0.033 | 0.313 | 0.445 | 0.232 | 0.044 | 0.353 | 4.645 | 308 |  |  |
| LivelySpeaker [] | TF | 1000 | 0.535 | 0.249 | 0.030 | 0.334 | 0.538 | 0.220 | 0.038 | 0.406 | 4.157 | 321 |  |  |
| Our-Base | Wav2vec | SSM | 1000 | 0.416 | 0.246 | 0.034 | 0.335 | 0.425 | 0.223 | 0.042 | 0.412 | 3.587 | 320 |  |
| TF | 1000 | 0.424 | 0.246 | 0.033 | 0.334 | 0.426 | 0.219 | 0.040 | 0.402 | 3.608 | 323 |  |  |  |
| HuBERT | SSM | 1000 | 0.402 | 0.247 | 0.033 | 0.336 | 0.432 | 0.218 | 0.041 | 0.407 | 3.412 | 320 |  |  |
| TF | 1000 | 0.418 | 0.247 | 0.034 | 0.338 | 0.432 | 0.219 | 0.041 | 0.412 | 3.529 | 323 |  |  |  |
| Our-Large | Wav2vec | SSM | 1000 | 0.421 | 0.244 | 0.031 | 0.208 | 0.430 | 0.219 | 0.038 | 0.253 | 3.453 | 539 |  |
| TF | 1000 | 0.354 | 0.244 | 0.030 | 0.209 | 0.372 | 0.214 | 0.038 | 0.251 | 3.376 | 557 |  |  |  |
| HuBERT | SSM | 1000 | 0.403 | 0.244 | 0.030 | 0.214 | 0.406 | 0.217 | 0.037 | 0.237 | 3.395 | 539 |  |  |
| TF | 1000 | 0.351 | 0.244 | 0.030 | 0.205 | 0.372 | 0.217 | 0.037 | 0.248 | 3.281 | 557 |  |  |  |
|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |

Experimental Setup.
Within the network, we compare two transformer-based audio feature extractors, Wav2Vec2.0 [] and HuBERT [], which are popular in the field of self-supervised speech recognition. Regarding the feature embedding module (Decoder), we evaluate the performance of SSM-based model [] in contrast to the transformer (TF) approach []. We conduct experiments using two different model configurations. Both of them employ the same diffusion architecture while differing in the feature extractor setup. The base model incorporates Wav2Vec2.0-base/HuBERT-base as the audio feature extractors, featuring a model dimension of 768 and containing 8-layer TF/SSM decoder. The large model utilizes Wav2Vec2.0-large/HuBERT-large, with a model dimension of 1,024 and 16-layer TF/SSM decoder.

**Table 4: Ablation study on denoising steps.**
| Step | FID$\downarrow$ | FGD$\downarrow$ | WGD$\downarrow$ | Smooth$\downarrow$ |
| --- | --- | --- | --- | --- |
| 5 | 3.540 | 0.361 | 0.237 | 0.354 |
| 10 | 3.682 | 0.363 | 0.236 | 0.310 |
| 100 | 3.438 | 0.366 | 0.232 | 0.254 |
| 300 | 3.360 | 0.348 | 0.233 | 0.240 |
| 1000 | 3.281 | 0.362 | 0.231 | 0.227 |
|  |  |  |  |  |

Quantitative Results.
Tab. 3 provides the results of existing methods and our baseline under various experimental settings on PianoMotion10M dataset. Due to the absence of prior work on generating gestures for piano music, we refer to the network structures of EmoTalk [] and LivelySpeaker [] and re-implement them to account for our task. EmoTalk directly generates poses from audio, while LivelySpeaker utilizes an MLP-based diffusion backbone for gesture generation. Our baseline models achieve better fidelity on hand motions by estimating positions and gestures, separately.
In our baseline models, TF-based models outperform the SSM-based models in processing our time-series information, especially in our large model. For the audio feature extractor, the performance of the HuBERT model slightly outperforms the Wav2vec2.
Additionally, we conduct ablation experiments on different denoising steps and our model also achieves competitive results with fewer steps, as shown in Table 4.

Qualitative Results. Fig. 4 demonstrates the visual results of our baseline and the existing models. The output of EmoTalk method exhibits a static average gesture. While MLPs afford LivelySpeaker rapid inference speed, they compromise the fidelity of generated motions. Conversely, our model demonstrates notably superior performance compared to previous methods by taking advantage of a two-stage approach, as illustrated in Fig. 4. To attain accurate positional information, we utilize an end-to-end position predictor rather than making use of an uncontrollable diffusion model for position generation. Furthermore, we employ a position-guided approach with a diffusion-based gesture generator for hand motion estimation.

Figure: Figure 4: Illustration of the qualitative results. We display the generated gestures across frames using different methods. Our method stands out due to its greater fidelity, as shown in the examples.
Refer to caption: x4.png

## 6 Conclusion

In this work, we present PianoMotion10M, a new large-scale piano-motion dataset for hands, which has 116 hours of piano music and 10 million frames annotated with hand poses.
We address the critical issue that the current datasets are insufficient for human-piano interaction.
Based on PianoMotion10M, we develop a benchmark model that maps the piano music pieces to hand motions.
To simplify the learning target, we divide the motion into position and gesture by introducing a position predictor along with a gesture generator guided by the estimated positions.
We suggest the evaluation metrics to measure the fidelity and smoothness of the hand motions compared to the ground truth.
Our dataset and benchmark will further advance the automation of piano performance simulation and assist in learning piano playing.

## Appendix

In this part, we further provide more details and discussions on our proposed dataset and benchmark:

- •
§A: More statistics of PianoMotion10M;
- •
§B: More visual samples of PianoMotion10M dataset;
- •
§C: More details of our baseline;
- •
§D: Limitation and future work;
- •
§E: Ethical considerations;
- •
§F: Author statement;
- •
§G: License and consent with public resources.

## Appendix A More Statistics of PianoMotion10M

Figure: Figure 5: Distribution of Note Clicks and Volume Levels in the PianoMotion10M Dataset. The top figure depicts note click frequency, and the bottom one shows the volume distribution.
Refer to caption: x5.png

Fig. 5 presents a detailed statistical analysis of piano fingerings, which focuses on the frequency of note clicks and the distribution of volume levels.

Note Click Counts.
The top figure in Fig. 5 displays the frequency of each note played, measured in millions. This distribution spans 128 keys of the piano, which indicates frequent usage of those keys in performances. It shows higher counts in specific note ranges.

Notes around the middle of the keyboard, particularly from C4 to C6, exhibit significantly higher click counts, aligning with their common use in melodies and harmonic accompaniments. Conversely, notes in the low (A0 to B1) and high (C7 to C8) octaves have markedly fewer clicks, as these ranges are less frequently used and typically reserved for specific musical effects or embellishments.
It is worth noting that, certain notes, particularly those fundamental to common chords and scales (e.g., A, C, E, and G), exhibit higher frequencies, reflecting their frequent use in various musical pieces.

Volume Distribution.
In addition to note frequency, the figure below presents a comprehensive distribution of volume levels, spanning various ranges to highlight the dynamics of piano playing. Volume counts, measured in millions, provide insights into the intensity and expression captured in our constructed dataset.

There is a higher count of notes played at moderate volume levels. This reflects the natural dynamics of piano playing, where most notes are neither extremely soft nor loud.

## Appendix B More Visual Samples of PianoMotion10M Dataset

Figure: Figure 6: More samples from PianoMotion10M dataset. BV*** denote the corresponding video iDs.
Refer to caption: x6.png

## Appendix C More Details of Our Baseline

The audio feature extractor $\Phi_{a}$ maps the audio $A$ to the feature vector $f_{a}$. We use pre-trained Wav2Vec2.0 [] and HuBERT [] models developed by Facebook AI for $\Phi_{a}$. Both models leverage extensive unlabeled data for unsupervised pretraining to learn high-dimensional speech representations. HuBERT extends its semi-supervised learning approach with pseudo-labels to self-supervised learning. In our experiments, we use [wav2vec2-base-960h](https://huggingface.co/facebook/wav2vec2-base-960h) and [hubert-base-ls960](https://huggingface.co/facebook/hubert-base-ls960) for the base model, and [wav2vec2-large-960h-lv60-self](https://huggingface.co/facebook/wav2vec2-large-960h-lv60-self) and [hubert-large-ls960-ft](https://huggingface.co/facebook/hubert-large-ls960-ft) for the large model as the audio feature extractor $\Phi_{a}$.

## Appendix D Limitation and Future Work

Our dataset closely associates piano music with hand movements. Due to data diversity, we have not yet aligned piano positions in all videos. We plan to engage extra experts to annotate piano key positions for more precise spatial alignment in the next version. Additionally, the variance in piano tones across recordings may also affect the baseline model’s performance.

PianoMotion10M provides piano music, corresponding MIDI files, and hand poses, offering researchers a valuable resource for studying human-piano interaction. This dataset enables the analysis of piano music through hand gestures and the generation of hand poses from audio tracks. With PianoMotion10M, we hope to benefit and facilitate further research in relevant fields.

## Appendix E Ethical Considerations

Piano motion datasets may pose significant privacy challenges, particularly concerning the pianist’s identifiable aspect, mainly their hands, during piano performance. Our dataset comprises videos uploaded by users on Bilibili, which are publicly accessible.
To address privacy risks, we adopt a strict policy of releasing only video IDs, not the videos or images directly. Furthermore, we employ the MANO hand prior model [] as a form of robust anonymization. This approach ensures the protection of personal information, mitigates the risk of individual identification, and minimizes privacy concerns. With these precautions, our dataset enables valuable research in piano motion analysis and generation.

## Appendix F Author Statement

The authors bear all responsibility in case of violation of rights. We confirm that the PianoMotion10M dataset is open-sourced under the [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/) International license and the released code is publicly available under the [Apache-2.0](http://www.apache.org/licenses/) license, ensuring open access and permissive usage for academic and research purposes.

## Appendix G License and Consent with Public Resources

### G.1 Tools for Annotation

The piano audios were transcribed by piano_transcription_inference [] and the hand poses of the MANO model [] in videos were annotated with MediaPipe [] and HaMeR []:

- •
piano_transcription_inference(^2^22[https://github.com/bytedance/piano_transcription](https://github.com/bytedance/piano_transcription).) Apache License 2.0
- •
MANO(^3^33[https://mano.is.tue.mpg.de/](https://mano.is.tue.mpg.de/).) CC BY-NC
- •
MediaPipe(^4^44[https://ai.google.dev/edge/mediapipe](https://ai.google.dev/edge/mediapipe).) Apache License 2.0
- •
HaMeR(^5^55[https://github.com/geopavlakos/hamer](https://github.com/geopavlakos/hamer).) MIT License

### G.2 Models for Baseline

Pre-trained Wav2Vec2.0 [] and HuBERT [] were utilized as audio feature extractor, while transformer [] approach and SSM-based model [] were employed as our feature decoder:

- •
Wav2Vec2.0(^6^66[https://github.com/facebookresearch/fairseq/tree/main/examples/wav2vec](https://github.com/facebookresearch/fairseq/tree/main/examples/wav2vec).) MIT license
- •
HuBERT(^7^77[https://github.com/pytorch/fairseq/tree/master/examples/hubert](https://github.com/pytorch/fairseq/tree/master/examples/hubert).) MIT license
- •
Transformer(^8^88[https://github.com/huggingface/transformers](https://github.com/huggingface/transformers).) Apache License 2.0
- •
SSM(^9^99[https://github.com/state-spaces/mamba](https://github.com/state-spaces/mamba).) Apache License 2.0

### G.3 Re-evaluated Methods

In the experimental section, we evaluated EmoTalk [] and LivelySpeaker [], and models were reproduced using the official code:

- •
EmoTalk(^10^1010[https://github.com/psyai-net/EmoTalk_release](https://github.com/psyai-net/EmoTalk_release).) CC BY-NC 4.0
- •
LivelySpeaker(^11^1111[https://github.com/zyhbili/LivelySpeaker](https://github.com/zyhbili/LivelySpeaker).) Unknown

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Please see Weakness.
soundness: 3
strengths: 1. Originality and Dataset Contribution: The PianoMotion10M dataset provides a novel resource, combining piano audio and video data with detailed 3D hand pose annotations across 116 hours of performance. This dataset addresses a significant gap by providing high-resolution, continuous hand motion data, which is essential for realistic AI-driven piano instruction and training, especially in guiding hand positioning and transitions.

2. Comprehensive Benchmark and Evaluation Metrics: The paper introduces a set of detailed evaluation metrics, including Frechet Gesture Distance, Wasserstein Gesture Distance, and Position Distance, to assess motion fidelity, positioning accuracy, and smoothness. This comprehensive approach to benchmarking supports the model’s effectiveness and provides a robust foundation for future research in hand motion generation.

3. Well-defined Baseline Model: The baseline model combines a position predictor with a gesture generator to generate realistic hand motions based on audio input, demonstrating the dataset’s utility. The use of a diffusion probabilistic model for gesture generation adds an innovative approach to capturing continuous motion, enhancing the quality of generated hand movements and reinforcing the practical application of PianoMotion10M in piano instruction.
summary: This paper presents PianoMotion10M, a large-scale dataset designed to advance hand motion generation in piano performance, featuring 116 hours of piano videos annotated with 10 million hand poses. To complement this dataset, the authors introduce a baseline model that generates realistic and continuous hand motions from piano audio, using a position predictor and a gesture generator. The dataset and model are aimed at improving AI-driven piano instruction, with metrics to assess motion quality, smoothness, and positional accuracy.
weaknesses: 1. Supplementary Material and Generation Results: The supplementary material lacks corresponding piano music for the generated hand motions, making it difficult to assess the quality and realism of the generated performance. Including paired audio and visual outputs would provide a more complete evaluation of the model’s effectiveness.

2. Dataset Collection Process:
The collection and preprocessing of data, essential to any dataset creation work, lack clarity. Specifically, the paper does not thoroughly explain the data acquisition process in Section 3.1.
While the authors mention in line 213 that "3,647 candidate videos" were collected, it remains unclear how these videos were queried or selected.
In line 240, the authors state that they "remove videos where hand regions are frequently obstructed or invisible." However, they do not provide a clear definition of what constitutes "frequently obstructed or invisible," making it hard to understand the filtering criteria.
Line 236 mentions manually selecting pure piano music to ensure no vocals or sounds from other instruments, but it’s unclear why an automated audio event detection approach (such as [1]) wasn’t considered for this task.

3. Annotation Quality and Relevance:
The quality of annotations is crucial in generating reliable datasets. In line 251, the authors mention "we transcribe the piano performances into MIDI files by utilizing a state-of-the-art automatic piano transcription (Kong et al., 2021)". However, this method, although effective, was developed in 2021. The authors do not address whether they explored more recent transcription methods (such as [2-5]) or provide experiments comparing different methods to validate their choice.

4. Additionally, while MIDI signals are provided, they are not used in the hand motion generation task. It would be beneficial to validate the quality and relevance of MIDI annotations, potentially by incorporating MIDI data into the baseline model to assess its impact on hand motion generation. This could offer valuable insights for future research.

[1] Kong Q, Cao Y, Iqbal T, et al. Panns: Large-scale pretrained audio neural networks for audio pattern recognition[J]. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2020, 28: 2880-2894.

[2] Maman B, Bermano A H. Unaligned supervision for automatic music transcription in the wild[C]//International Conference on Machine Learning. PMLR, 2022: 14918-14934.

[3] Gardner J, Simon I, Manilow E, et al. MT3: Multi-task multitrack music transcription[J]. arXiv preprint arXiv:2111.03017, 2021.

[4] Toyama K, Akama T, Ikemiya Y, et al. Automatic piano transcription with hierarchical frequency-time transformer[J]. arXiv preprint arXiv:2307.04305, 2023.

[5] Cheuk K W, Sawata R, Uesaka T, et al. Diffroll: Diffusion-based generative music transcription with unsupervised pretraining capability[C]//ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023: 1-5.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: I don't have questions though I'd probably accept the paper as is.
soundness: 2
strengths: The main strength of the paper is that this is a really cool dataset linking piano performance audio (and MIDI) to hand motion sequences.  Piano playing is a challenging task requiring extreme dexterity, and this dataset seems like an important contribution to the research community.

The paper is clearly written and provides something new and of value.
summary: The paper has two main contributions:
1) A dataset of 100+ hours of piano performance audio and (overhead) video, MIDI, and hand motion sequences, all aligned.
2) A baseline method for generating hand motion sequences from piano audio.
weaknesses: The biggest thing missing is some sort of verification that a hand motion sequence *actually results in the target audio* when applied to a piano, using a physical model.  I recognize that this may be considered out of scope, but without it it's hard to trust even the ground truth provided in the dataset.  And then all of the metrics for the baseline method end up comparing generated hand motions with the ostensible ground truth hand motions, which has two issues:
1) The ground truth hand motions may have errors.
2) Even if the ground truth hand motions are accurate, there may be multiple possible hand motions to generate a given performance audio, none more "correct".
However, I think this may be okay if the dataset is positioned as "how some pianists performed these pieces" rather than "how one *should* perform these pieces".

An uncited research paper related to the above point is RoboPianist (Zakka et al. 2023): https://kzakka.com/robopianist/

I'm not sure why the baseline method included attempts to infer hand motions directly from audio, rather than going through MIDI.  MIDI (including note velocities) seems like it would contain all of the information needed to reconstruct the hand motions, and is *much* more compact than the raw audio.  If this is incorrect, it would still probably be worth mentioning in the paper.

The paper doesn't include a discussion of sustain pedal.  It seems as though the piano transcription method used *does* transcribe pedal events, which seem like they would be necessary to include in the baseline method otherwise much piano audio would be impossible to perform hands-only.  However, since the baseline method goes straight from audio to hand motions, the model is forced to implicitly transcribe sustain pedal in order to determine the hand motions.  Is that correct?

The paper raises a bunch of questions that I am curious about but were not discussed, specifically related to limits of the method and failure modes.  The examples included in the supplementary material are sort of "easy".  What happens if you give the baseline method virtuosic piano audio e.g. the MAESTRO dataset?  What happens if you speed up the audio so that it's no longer humanly playable?  What happens if you play intervals that human hands can't accommodate?

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Do authors have some human evaluation results for the generated motions? It would be better if these could be incorporated.

2. Ethic concern and copyright issue: Since permissions were obtained from Bilibili users, how does the dataset’s open accessibility align with potential copyright restrictions or privacy concerns, particularly for identifiable hand features? I know that they do have some restrictions on this.
soundness: 3
strengths: 1. This is a large-scale dataset comprising 116 hours of piano performance from 14 professional players and 10 million annotated frames. The videos are collected under specific constraints (e.g., high resolution, bird’s-eye perspective) to ensure quality and consistency.

2.  The data collection and annotation method is well-structured, using tools like MediaPipe and HaMeR for robust hand pose detection. 

3. The dataset is indeed a valuable contribution, likely to support advancements in both piano instruction and broader motion generation research.

4. The two-stage design—featuring a position predictor and a position-guided gesture generator—seems well-suited for this task and shows reasonable quality. The model uses a diffusion-based architecture with a U-Net structure, which allows it to generate natural, fluid gestures that are better suited to piano performance.
summary: The paper addresses the challenge of hand motion generation for piano performance, recognizing the gap in current AI systems for musical instruction, particularly for instruments like the piano where hand positioning is critical.  The proposed PianoMotion10M serves as an important dataset aimed at solving this problem.
weaknesses: 1. The paper acknowledges the dataset’s limitations, particularly the need for improved diversity in piano positions and video sources, now only include 14 people. Addressing the inconsistency in piano tones and alignment of keyboard positions remains critical for achieving high-fidelity results.

2. A suggestion to include related work: Audeo: Audio Generation for a Silent Performance Video. Kun et,al (NeurIPS 2020).

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: I do have questions regarding the text:

Section 4.3: The learning rates are said to be 2e^{-5} and 5e^{-5}. The math reads: 2 (or five) times exp(-5). The usual reporting method is to use 2 (or 5) times 10^{-5}. Was that a typo, or are you really using the Euler number to define learning rates? If so, is there a reason for such?

Section 3.2: 
- MIDI files with high discrepancies were adjusted: what is the measure for a "high discrepancy"? The issue here is that, if this dataset is going to be used as a benchmark, there should be a measure on how much it differs from reality.

- Data uses a Savitzky-Golay filter. This filter is applied to the data in the dataset to increase smoothness. What are the filter parameters (order, frequency response, etc.)? Also, why was this specific type of filter chosen instead of the standard ones (a Butterworth filter seems adequate)? What is the impact of not using the filter?

Section 3.3: All videos are accessible with the IDs in the Bilibili website. Is Bilibili aware and consenting with the idea that there will probably be robots scraping their data without using their website? That is, are there chances that Bilibili will simply include barriers that would harm the dataset? Also, if users take down their own videos, how would that impact the dataset?

Section 4.3: The GPU has 24GB of RAM. How big is the model itself? How long did training take? Also, are you planning on releasing the trained weights?

Section 5.1:
- Several metrics are proposed, which is a good thing. However, none of the metrics were discussed in terms of their relevance to the real problem. For example: what kind of "error" in the rendering could be highlighted by the FID, FGD or WGD? Without this discussion, the metrics simply pose a "competition" on who gets better values. A phrase such as "The WGD metric offers a robust measure of dissimilarity between the generated motion and ground truth." does not clarify if this dissimilarity consists of a jittery behavior, an offset, an error in velocity, fingers being rendered too long (or too many!), and so on. Such a discussion, with clear examples, could clarify why these specific metrics were used, and not others.
- Also, the metrics imply in a comparison with a ground truth. This is not wrong per se, but, when hands are rendered, is there really a "ground truth"? That is, if two pianists play the same excerpt, wouldn't their hands be slightly different? So, more than increasing the proximity to a ground truth, the goal of the algorithms should be to obtain results that are within a distribution for human differences, not simply to approach a unique ground truth.
- In this same line of thought, does it really make a visible difference to have small variations in these metrics? For example, going from FGD=0.360 to FGD=0.351 is a difference of 0.009 - what type of difference does that reflect in the actual performance?
- How do these measures compare with a Mean Opinion Score (MOS)? It would be great to have some evidence that the proposed measures can substitute MOS, because collecting human opinions is costly. However, we need evidence for such.
- Minor suggestion: in Table 3, make the Right Hand appear to the right of Left Hand.
- A not so minor aspect: why are the measurements so different between the hands? Afterall, we have the same amount of data and the same models for them, correct? According to the data, Wav2Vec+TF is better at rendering left hands, while HuBERT+TF is better at right hands (according to WGD), which seems to indicate that their performance is actually the same and the differences are due to noise/chance/variance.

Section 5.2
- Figure 4 states that "Ours" is close to "GT" (or: the blue hands are closer to the gray ones). This is true, at least for this example; but in general the green and yellow hands seem to be equally plausible hand positions. In the third line, the yellow hands (LiveSpeaker) seem to be even more emphatic on the fact that the left hand is not playing anything.
soundness: 3
strengths: In descending order of importance:

- A large dataset for hand pose generation is proposed. This is original, clear, and significant.
- A model is proposed. This is technically original, but the significance is not well discussed for the reasons stated in the comments.
- A set of measures for performance is proposed. They are not original, nor their significance to the problem is discussed.

Overall, I find the article interesting, although I am not sure how many people are specifically trying renderize hand poses in pianos using audio as input. However, this dataset could also be used for other tasks, which broadens its impact.

- The initiative of making a dataset for this task is great, and the ethics statement covers most of the potential problems,
- The dataset has some potential problems that could be taken care of, especially the long-term availability and the reliance on a specific company and several users to maintain the data,
summary: This paper proposes a large dataset consisted of piano music and aligned MIDI and hand positions containing 116h of recordings and 14 subjects. The goal of the dataset is to train generative processes to renderize 3d human-hand motion videos. This work presents an interesting baseline method to renderize hands.
weaknesses: There is a great emphasis in the proposed model whereas the proposed dataset and measures are not well discussed. I would suggest focusing the paper on the dataset and on the discussion on the metrics and skip the model proposal altogether. In special, this is because the model uses more data than any other model, hence it is expected to approach the data distribution better than other models.

- The presented model is interesting, but its performance is close to EmoTalk (more on that in item 4)
- I miss some discussions on the metrics and their relationship with the problem, and the space taken by describing the baseline method could be used for that. In fact, EmoTalk and LivelySpeaker, according to Table 3, are already interesting for future comparisons and their metrics are very close to the proposed method's.

One aspect that has not been addressed is a description of styles, genres, levels of expertise, and other biases in the dataset. This leads to thinking that the dataset polarized towards particular styles of playing (for example: ragtime versus classical). This is not a problem per se, as it is unavoidable; however, not *knowing* what these biases are can become a problem.

Meta-review: This paper introduces PianoMotion10M, a new dataset of piano performances with video, audio, MIDI, and hand motion data.  The dataset contains over 116 hours of piano videos annotated with 10 million hand poses. The authors also introduce a baseline model that generates realistic and continuous hand motions from piano audio, using a position predictor and a gesture generator. The dataset and model are aimed at improving AI-driven piano instruction, with metrics to assess motion quality, smoothness, and positional accuracy.

Strengths:
Provides a large and unique dataset for piano performance research.
Includes accurate 3D hand pose annotations.
Offers a baseline model for hand motion generation (position predictor with a gesture generator, using a U-net and diffusion model).
Defines clear evaluation metrics (including Frechet Gesture Distance, Wasserstein Gesture Distance, and Position Distance).

Weaknesses:
Lacks verification that the hand motions can reproduce the audio (addressed with a user study in the rebuttals).
Baseline model doesn't utilize MIDI information (addressed in rebuttals with new experiments).
Limited diversity in piano positions and playing styles.
Could benefit from more detailed explanations of data collection and annotation choices (addressed in rebuttals).

Decision: Accept (Spotlight)