From Pose to Muscle: Multimodal Learning for Piano Hand Muscle Electromyography
Authors: RUOFAN LIU
YICHEN PENG
Takanori Oku
Chen-Chieh Liao
Erwin Wu
Shinichi Furuya
Hideki Koike
Venue: NeurIPS 2025 poster ()
Keywords: Multimodal Learning
Electromyography Estimation
Muscle Analysis
Primary area: applications
TL;DR: We introduce a multimodal electromyography (EMG) inference framework trained on a collected piano performance dataset, to enable high-fidelity hand EMG estimation and demonstrate generalization and adaptability across unseen users and tasks.

Abstract: Muscle coordination is fundamental when humans interact with the world. Reliable estimation of hand muscle engagement can serve as a source of internal feedback, supporting the development of embodied intelligence and the acquisition of dexterous skills. However, contemporary electromyography (EMG) sensing techniques either require prohibitively expensive devices or are constrained to gross motor movements, which inherently involve large muscles. On the other hand, EMGs exhibit dependency on individual anatomical variability and task-specific contexts, resulting in limited generalization. In this work, we preliminarily investigate the latent pose-EMG correspondence using a general EMG gesture dataset. We further introduce a multimodal dataset, PianoKPM Dataset, and a hand muscle estimation framework, PianoKPM Net, to facilitate high-fidelity EMG inference. Subsequently, our approach is compared against reproducible competitive baselines. The generalization and adaptation across unseen users and tasks are evaluated by quantifying the training set scale and the included data amount.

From Pose to Muscle: Multimodal Learning for Piano Hand Muscle Electromyography
Ruofan Liu 1.2
Yichen Peng' Takanori Oku?
Chen-Chieh Liao
Erwin Wu² Shinichi Furuya" Hideki Koike
(liu.r. fiuurrac. peng. ag,, liao. c. aa, wu.e. mu.e.aa) aa) @m titech www.ttitt.cc....... jc.
--okusshbbaura-it..... ac. jp, shinichi. furuya@sony. com, koikečc. titech. ac jp 'Inttitute of Science Tokyo, "Sony Computer Science Laboratories
Abstract
Muscle coordination is fundamental when humans interact with the world. Re- liable estimation of hand muscle engagement can serve source of internal feedback, supporting the development of embodied intelligence and the acquisition of dexterous skills. However, contemporary clectromyography (EMG) sensing techniques either require prohibitively expensive devices or are constrained to gross motor movements, which inherently involve large muscles. On the other hand, EMGs exhibit dependency on individual anatomical variability and task-specific contexts, resulting in limited generalization. In this work, we preliminarily investi- gate the latent pose-EMG correspondence using a general EMG gesture dataset. We further introduce multimodal dataset, PianoKPM Dataset, and hand muscle estimation framework, PianoKPM Net, to facilitate high-fidelity EMG inference. Subsequently, our approach compared against reproducible competitive baselines. The generalization and adaptation across unseen users and tasks are evaluated by quantifying the training set scale and the included data amount.
Introduction
Human- world interaction is an interdependent loop involving explicit visual, auditory, haptic cues, and implicit physiological signals. Among these, internal information, such as hand muscle elec- tromyography (EMG), is critical for embodied interaction 212, assistive technology 4 and skill acquisition () yet remains difficult to access in real-world ubiquitous scenarios due to the cost, obtrusiveness, and expertise required by conventional sensing systems 77).
On the other hand, the emergence of large-scale datasets and deep learning networks increasingly facilitates cross- moosl synthesis, which enables transformation between different modalities. For instance, text to image [8], audio to motion (9) or text to music [10]. These developments offer new insights into inferring internal physiological signals like EMG from readily accessible modalities. Also, the invisible yet informative muscle activations are highly correlated with body postures, which validated in several recently released large-scale datasets containing EMG and motion data. The public datasets, like EMGBench [11) and emg2pose [1). demonstrate the feasibility of classifying and estimating gestures from EMG. However, the inverse task infer EMG from postures other modalities, remains substantially challenging and underexplored. The difficulty stems from the richer semantic information embedded in EMG compared posture alone, as there no strict one-to-one mapping and the same postures may correspond to different muscle recruitment patterns.
To pioneer this promising yet insufficiently studied field, this work begins by leveraging emg2pose [12, currently the largest and most diverse paired dataset EMG and hand poses. We pretrain foundational model capable lear latent relationships between muscular activations and hand postures, which serves as a preliminary step toward assessing the feasibility of pose-to-EMG inference.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

We further focus on specific domain of piano performance, a representative scenario that involves intricate and dexterous hand motions. Expert-level pianists rely heavily on precise internal muscle synergies for advanced and technical performances. However, prior research highlights that muscle recruitment patterns can vary significantly across different domains [13), which is also confirmed when we evaluate the pretrained model on PianoMotion [14). hand motion dataset specifically built for piano but without muscle dynamics.
To enhance the model's performance and generalization during piano playing, we introduce the Piano Keystroke- Pas---ossllee Dataset (PianoKPM Dataset) that contains synchronized, high-fidelity data of hand muscle EMG sampled at 2000 Hz, keystroke motions at 1000 Hz, audio, and multi-viww RGB videos 60 FPS. The dataset also includes 3D hand motion data estimated using the MANO model [15] and state-of-the-ar (SOTA) method HaMeR [16]. To the best of our knowledge, this is the largest publicly available EMG dataset of professional piano performance, comprising data from 20 expert pianists performing 7 distinct musical tasks, with 12.64 hours of high-quality recordings.
Leveraging this dataset, we propose the Piano Keystroke-Pose- Mose-M Network (PianoKPM Net) to infer high-frequency EMG from pose data, Additionally, we explore the benefits of incorporating other modalities, such key-pressing events, improve model performance, A comparative evaluation of several baselines and our method is provided to demonstrate the applicability. Moreover, various electrode placement [17 18]. subject anatomy [99 20]. and kinematics may introduce substantial discrepancies in EMG data distribution, posing challenges for generalization. We subsequently investigate the model's performance on held-out users and tasks based on the training set scale and quantify the amount of data required from an unseen distribution to recover high accuracy.
In summary, this paper makes three contributions:
A hand musele EMG estimation framework, PianoKPM Net, leverages human-centric hand motion data and tool-centric piano keystroke data.
A multimodal piano performance dataset, PianoKPM Dataset, contains simultaneous hand postures, keystroke motions, audio, and miniature hand muscle EMG.
A comprehensive evaluation compares our EMG inference approach with several baselines in piano playing and analyzes its generalization and adaptation across users and tasks.
Instructions regarding accessing and using our PianoKPM dataset and network are provided at https://github. wwtp:///uoffimi111111///1/1a1110
2 Related Work
2.1
Museular Dynamies Estimation
Muscular dynamics have shown great potential in augmenting various domains like control interfaces
(2 sign recognition [23 motor optimization [6], and recently mixed reality [27
However, conventional sensor- based muscle measurements demand scarce and resource-intensive
instrumentation [29]. Prior research explores computer-assistin algorithms to estimate myoelectrical signals to avoid the high cost, encompassing simulation based and learning- based methods (30).
Musculoskeletal simulations emerge as promising approaches, while most rely on inverse dynamics and hence are constrained to pre-selected distributions and entail substantial computational overhead 38 On the other hand, data-driven models enhance generalization and lower required resource 39 43 albeit at the inevitable cost of accuracy [44.. All the aforementioned principally focus on
the gross motors of large muscles. Several prior studies [45 [45]46] target muscles involved in non- non-posed facial expressions and hand movements, yet these early efforts suffer from limited accuracy small participant validation. Consequently, the intricate and critical small hand muscles in dexterous activities remain largely underexplored (47
2.2 Relevant Datasets
EMG Datasets. Existing hand-EMG datasets can be categorized into those only involving coarse gesture labels for classification [1] [8) and those involving precise hand postural information for regression. Representative datasets of the latter are summarized in Row 1-3 of Table [ Ninapro 51 (4). one of the most established EMG datasets, comprises multiple subsets collected using 10

---

Table 1: Dataset comparisons. The first three rows list publicly available EMG-hand datasets, the following three correspond to piano performance datasets, and ours is presented in the final row.
Dataset
Year
Public
Size
Subject
Pose
Audio
MIDI
EMG
Ninapro
2014
67
emg2pose
2024
/ 80M
193
/
PiMForce
2024
/
83.2M
21
/
PianoHand2.5M
2023
x
2.5M
21
/
/
FürElise
2024
V
2.2M
15
/
/
PianoMotion
2024
V
10M
14
/
d
PianoKPM (Ours)
2025
V
5M
20
/
/
/
Otto Bock electrodes, 12 Delsys Trigno sensors, dual Myo armbands, and 22-sensor dataglove to record hand postures. emg2pose [12] employs 16-channel sEMG-RD wristband and 26-camera motion capture system, while PiMForce (55) utilizes 8-channel sEMG armband with magnetic sensing-based tracking module, to acquire hand kinematics and muscle dynamics. Some additional EMG datasets on specific skills, such alpine skiing (56), capture only large muscle activations.
Piano Datasets. Various piano datasets are curated to support different research objectives, such music generation [57], fingering extraction [58], and gesture recognition [59] Rows 4-6 of disspayy datasets providing more accurate hand movement annotations. PianoHand2.55 2 [62] offers precise ground-truth (GT) labels with large-scale 3D hand pose dataset from professional pianists. FürFlise (63) captures synchronized MIDI key-press data and multi-view videos using a five-camera setup. PianoMotion 10M [14] aggregates in-the-wild piano performance videos from the internet, including reconstructed motion, audio, and MIDI data. However, to date, no piano performance dataset simultancously provides high-quality pose, audio, and MIDI data, along with hand muscle EMG, which is crucial for underlying fine-grained dynamics.
Preliminary Exploration: Contrastive Pretraining
(1) General EMG Detasel Pretrsining
(2) Domain Transfersbilll Evelusting
(3) Piano Specific Modeling
PiamoModion
Plano (Ours)
Hand Poses (r (
MIGGHLLTTET Keystrokes (Kel)*)
Joint Angles (++ and)
Hand Possss (Pe gon))
Poses (o (P)))))))
Fnondw
Pose-to-EMG Cornespondance Prior
Wrist EMG cg)))
Synthotic Wrist EMG an-)
Hand EMG (seag))
Figure Overall research pipeline. We (1) use a public general EMG dataset to pre-learn latent relationships between hand kinematics and EMG, (2) evaluate this prior on a piano motion dataset but the synthetic EMG fails yield satisfactory performance, (3) construct our specialized pose-EMG paired dataset which is subsequently used to estimate the precise FMG.
To implement cross-modal EMG estimation and mitigate the cost of large-scale paired EMG-pose collection, we exploit knowledge from public datasets. This section claborates on the methods we use and questions encounter in the preliminary study, including pretraining dataset, evaluating in a task-specific domain, and articulating the research motivations. As Figure reveals, the whole pipeline from general pretraining to piano-specific modeling is the cornerstone of our EMG inference framework.
general EMG
General EMG Dataset Pretraining. In Figure (1), to preliminarily investigate the feasibility of estimating EMG from other modalities, we employ the public emg2pose dataset [12 to pretrain deep learning framework capable of capturing semantic correlation between hand poses and EMG.

---

The dataset contains 2000 Hz, 16-channel EMG from a bipolar sEMG-RD wrist band [33), and pose annotations from a 26-camera motion capture system. spanning 193 participants, 370 hours, and 29 stages. A competitive EMG-to-pose baseline is also provided. We then modify its architecture by inverting and adapting the input-outpur channels encode inductive biases, which can understand pose-to-EMG: correspondence prior and facilitate downstream adaptation through fine-tuning.
Domain Transferability Evaluating. Given that EMG may exhibit distributional shifts across unseen domains, we focus on advanced piano performance with diverse dexterous finger movements to assess the generalization the pretrained framework. To this end, we utilize the PianoMotion 10M dataset [14). which comprises 16 hours of bird's-eye s-eye view piano performance videos with 10 million annotated hand poses. The pose-to- to-e-to-EMG correspondence prior evaluated on this dataset, yet fails to yield satisfactory results. Although the absence of paired EMG precludes quantitative evaluation, qualitative visualizations of the synthetic wrist EMG in Figure (2) suggest that the predictions are suboptimal.
Research Questions Articulating. The above observations motivate and sharpen the focus of our research. We underscore three reasons for the degraded performance: (1) The piano hand postures differ substantially from generic datasets, undermining the generalization of learned pose-to-EMc correspondence in the target domain. (2) Surface EMG acquired from the wrist aggregates multiple underlying muscle activities, offering limited localization isolate individual hand muscles. (3) Pose representations are inherently sparse, lacking sufficient contextual and dynamic cues essential for reliable EMG inference.
Methodology
To address the three research questions mentioned in Section respectively (1) construct piano-centric dataset encompassing diverse hand postures; (2) employ EMG sensors to individually monitor six major hand muscles; (3) incorporate high-frequency keystrokes as auxiliary modality. Leveraging this multimodal dataset, we further propose a neural network for EMG inference that achieves reliable and accurate results across general benchmarks and our piano-specific dataset.
4.1 PianoKPM Dataset
(a)
Multiview Videos
Hand Poses
Keystroke
Mctions
Figure 2: Dataset composition. (a) The studio, apparatus, and six hand muscles. (b) PianoKPM Dataset consists of 60 FPS videos captured from three different views, hand pose annotations, and 2000 Hz 6-channel EMG, 1000 Hz 88-channel keystrokes, and audio collected by specific equipment.
As shown in Figure (a), all data collected in studio equipped with a grand piano, three RGB cameras, two EMG sensors, a fixed microphone, and a keystroke sensing system [64]. From professional pianist and neuromuscular science researcher's advice, EMG signals recorded from six hand muscles: Abductor Digiti Minimi (ADM), Pollicis Brevis (PB), First to Fourth Dorsal Interosseous (IDI- 4DI). The annotated hand postures are extracted using the SOTA pose estimation method HaMeR [16]. Appendix B provides more dataset details based on standardized datashect 65
4.1.1 Data Preprocessing
Inter-Frame Discrepancy-based Motion Refinement. Vision-based pose reconstruction achieves promising results, while inferior outcomes may still arise due to inevitable occlusion and rapid

---

movement. To clean and refine hand motions, we detect and remove the abnormal frames, then interpolate and smooth the remaining valid data. Given that outlier frames always exhibit variations from their adjacency, we compute the inter-frame differences d; = |X; x;+1ll2 to find the error frames. We then clean the data using time-informed linear interpolation, given the smooth and consistent nature of the hand motion over short durations. For an anomalous frame f. an interpolation coefficient a is computed with its temporally adjacent valid frames, then the current pose values Pnezt- Once all frames contain valid poses, a moving [66]67 67 is applied to enhance temporal smoothness and reject motion artifacts. We further commission professional data annotation service to verify the final dataset quality.
are interpolated via p (1 a)) Ppree
average filter [66
EMG Preprocessing. To ensure quality, computability, and alignment with other modalities, EMG is preprocessed through filtering. normalizing, downsampling. and task-based cropping. To reduce the noise of raw EMG signals sampled at 2000 Hz, we apply low-pass Butterworth filter (the 5th-order, cutoff frequency: 12 Hz) [68]. Considering inter- and inter-subject variability in EMG amplitude, we normalize each muscle's s EMG by its maximum voluntary contraction (MVC) [0]. which is empirically recorded through an MVC test session before data collection trials. This procedure transforms EMG into relative values between - and 1, enabling more informative within- and across-subject comparisons of muscle engagement. Subsequently, the 2000 Hz EMG is downsampled to 1000 Hz to synchronize with other modalities. All data are cropped based on the onset and offset of keystroke events, and only those periods when dedicated muscle activities relate to piano playing are included. See Appendix B.1.4 for more EMG preprocessing details.
4.1.2 Data Analysis
Data Statistics. PianoKPM Dataset consists of 20 highly-skilled pianists, 7 tasks, 7,000 performances,
and 21,000 videos with a total length of approximately 12.64 hours. Figure illustrates example performance that includes synchronized 720p 60 FPS RGB videos from three views, corresponding 3D hand pose annotations, 1000 Hz EMG, 1000 Hz keystroke motions, and audio. After preprocessing. our dataset has valid data for 5, 5,026,566 frames of videos and hand postures, along with 28, 24,706,,877 frames of EMGs and keystrokes. The scale and variability can ensure diversity and consistency across modalities, serving as the basis for downstream multimodal modeling to accurately predict EMG. Appendix B. 1.2 inc2ude includes more details about the dataset composition.
Data Quality. Following prior work (7I.. we adopt precision, recall, and Fl metrics, to evaluate the quality of the reconstructed hand motions. Our evaluation protocol defines TP as a frame where the model's s prediction validly matches the ground truth. FP refers to a frame that is flagged as invalid by the outlier detection algorithm in Section [4..I] while FN denotes a missing frame with the ground truth presence of a hand. Across all reconstructed hand sequences from the three camera views, we obtain 4.83M TPs, 0.14M FPs, and 0.06M FNs. Based on these counts, our model achieves 97.22 precision, 98.74 recall, and 97.97 F1 over the full dataset. See Appendix B.2 for dataset insights (72).
4.2 PianoKPM Net
Feature e )**)))
Hand EMG
Las
GT Hand
Fusion Pose
Encoder Spac
(BKE RD1)
Lou
EMG (SA E
Latant- Prior
Encoder Eato
Synthetic Wrist EMG
(8v e (
Regresieee Decoder
Pose-to-EMG
Hand (PER mumeer
Correspondence Prior
Keystroke Encoder Bay
Law + Loe
Hine-Channel-Wii
Keystroke K
Encoder &ea
Figure 3: Network architecture. multimodal hand muscle EMG estimation network enhances human-centric hand pose data by integrating tool-centric keystroke motions.
The potential of estimating EMG from human-centric videos grounded in the observation that muscle activities are inherently linked to the executed motor actions (3). Different movement
5

---

patterns lead to distinct muscle coordination dynamics, hence producing discriminative EMG signals However, postural data alone cannot fully decode muscle activity complexity, due to its sparsity and the absence of external force information during human-object interactions. For instance, the same poses performed with or without extrinsic resistance may elicit different muscle engagement, which highlights the need to incorporate tool-centric information for context-aware EMG estimation. Consequently. we augment hand movements with precise keystroke motions. As Figure 3 shows, we devise multi-branch encoding architecture to utilize the complementary inputs: (1) Input-specific encoders, Epose and Ekey, to extract shallow features from postures and keystrokes, (2) Latent-prior encoder, Etatent, to integrate prior correlations between pose dynamics and EMG latent space, (3) ceiee-channel-wine
48/74]
48 74..
encoder, Ejea: to disentangle temporal and feature dependencies. An auto-regressive decoder, Dea is subsequently introduced to reconstruct EMG aligned with the ground-truth channels from the encoded feature representations. Appendix C. provides further details on the structure and parameters of each layer and the design of the loss function, and Appendix C.22 describes the implementation, including hyperparameter settings, computational resources, and possible latency.
4.2.1 Multi-Branch Feature Encoder
Input-Specific Encoder. We first extract shallow features independently from the hand and keystroke motions using input-specific encoders. Due to the inaccuracy of 3D pose depth information mentioned in Appendix B.1.4 the pose encoder uses concatenated projected 2D keypoints P E R x x2xx (T: time length, V: views, J: joints) of each frame perform cross-view fusion, which reshapes, aligns and aggregates the input vector to produce 3D postural features through: Fp = (pose(()) E TTxx,, In parallel, the keystroke encoder takes the ID keystroke sequence K E RT (N: keys) as input and outputs keystroke features by: Fş- Ekey (K) E RTTxD.
Latent-Prior Encoder. To exploit pre-trained priors capturing the statistical heuristics between hand motion and EMG, a latent encoder is introduced to align pose representations with the latent prior module's 's inputs, which is useful for injecting inductive bias into the model (75). Formally, the latent encoder maps the input pose features into physiologically meaningful space by: Z, Etatens (F2) E R''xDi, capturing compositional rules of muscular activations in response to hand motion. It is then integrated into the main time-channel encoder, guiding its feature construction with relational biases to facilitate learning about entities, relations, and rules for composing them.
Time-C Channel- Wise Encoder. We revise the TDS block in previous research (76) as the core component of our cime-chnnel-wwiee encoder, which designed to decouple temporal dynamics from channel-wise feature interactions. All intermediate features are first combined by addition along the feature dimension: F E T = [F, + F + Zı), and then mapped to a nigher-dimensional space by: C = channels X feature_ wettthh Subsequently. the reshaped pseudo-image tensor F' E x''xexw allows us to treat the feature space as a 2D spatial domain and align the input shape with the following blocks. This vector is further passed through module consisting of two sequential blocks, each comprising a 2D convolutional layer. The final embedded feature is computed as:
A = Eea (F') E RTxD,, Compared to other convolutional encoders 777 78). this decomposition facilitates temporal-channe separability, allowing the network to learn across time and channels independently.
Dp. Dr. Dr. and D, refer to the output dimensions of the pose, keystroke, latent prior, and fused embedded features after their respective encoders. More details are introduced Appendix C.I
4.2.2 Auto- Regressere Decoder
Given that the target EMG is sequential, our decoder is designed as an auto-regressive structure. At each timestep I, the model predicts the hand muscle EMG én, E RM (M: muscles) based on the encoded features h and the prediction ej, t-1 from the last timestep, which is thereby computed as: E, = Deaa( (f [h']]e, - Teacher forcing is not employed during training and the model relies solely on its previous predictions. A multilayer perceptron (MLP) is opted as the decoder backbone. Compared to long short-term memory (LSTM) architectures chosen in previous work (79), the MLP achieves competitive accuracy while offering faster convergence and lower computational overhead, making it preferable in the training and inference phases.

---

4.2.3 Precision-Structure Hybrid Loss
In addition to the standard Mean Squared Error (MSE) loss, we incorporate Optimal Transport (OT) loss to balance local accuracy with global structure preservation. MSE is widely used due to its effectiveness in penalizing numerical precision. However, training with it alone leads the model to converge to an wer-thoo solution that suppresses peaks, valleys, and muscle activation transitions. On the other hand, previous studies have demonstrated the effectiveness of OT distance in assessing muscle synergy similarity [80]. Therefore. Sinkhorn baased OT loss [81] 82) is specifically tailored for training, which measures the transport required morph the predicted sequence into the ground truth as a distributional matching problem. OT loss can jointly consider amplitude,
temporal progression, and inter-muscle coordination as: Lot W. (En, En). where W. denotes the entropic-regullrizee Wasserstein distance between two point clouds in R over T time steps. Finally, we combine them as: L = Xmee Lmse + Aot Lot, to encourage low-level numerical precision and preserve higher-level physiological structures of EMG sequences.
Experiments
To evaluate the methodology rigorously, we conduct experiments along two axes. (1) Architectural evaluation:
We benchmark method against several baselines the public emg2pose dataset [12] and the proposed PP Dataset using quantitative and qualitative metrics. (2) Held-out evaluation: We investigate the model's generalization to held-out-distributiin scenarios. including exploring the effect of training dataset scale and quantifying the amount of unseen data required to sustain satisfying performance. To ensure fair comparisons and align with our hybrid loss design, we report results using Root Mean Squared Error (RMSE) and Optimal Transport Distance (OTD), which can evaluate the similarity between synergistic muscle activation patterns.
5.1 Architectural Evaluation
On the emg2pose and PianoKPM datasets, in-distribution experiments across all users and tasks are conducted to compare the proposed network architecture against several baselines. To further validate the usability of multiple modalities, we perform ablation studies, isolating the effect of each input on overall estimation accuracy. Given the absence of existing work on estimating small hand muscle EMG from poses in piano performance, EMG-to-pose regression, to support multiple input modalities. Moreover, we modify CodeTalker [84), widely-used cross-modal synthesis model, to better suit our task and serve as reference to our method. See Appendix D.I for baseline implementation details.
revise NeuroPose [83), originally designed for
Table 2: Results for architectural evaluations. We report the mean and standard deviation across performances. Bold indicates the best method and lowest loss with different input settings.
Dataset
Input
Methods
RMSE
OTD
NeuroPose
83
.067 + .032
.039 + .036
emg2pose
12
Pose
CodeTalker
84
.096 + .027
.075 .043
PianoKPM (Ours)
.055 + .034
.030 = .037
NeuroPose
.166 = .066
.070 .081
Pose
CodeTalker
.171 = .063
.064 = .071
PianoKPM (Ours)
.136 + .062
.033 .060
NeuroPose
196 + .096
118 + .122
PianoKPM
Keystroke
CodeTalker
.152 + .075
.055 .092
PianoKPM (Ours)
.191 + .090
.106 = .14
NeuroPose (multi-input)
.155 + .068
.061 = .077
Pose, Keystroke
CodeTalker (cross-attention)
.143 + .071
.040 + .082
PianoKPM (Ours)
.134 - .061
.031 .058
Quantitative Results. The method comparison and modality ablation results are exhibited in Table In general, PianoKPM Net achieves competitive accuracy when using human-centric video input alone (RMSE: 0.136, OTD: 0.033), and improves more when incorporated with tool-centric keystroke input (RMSE: 134, OTD: 0.031). While PianoKPM Net consistently outperforms other baselines

---

with either video alone or the fused two inputs, CodeTalker obtains the best performance with only keystroke imput. This is because the transformer-bant structure and self-attention mechanism capture long-sequence temporal dependencies. Our model integrates spatial postural semantics with temporal keypressing features to improve the accuracy of EMG prediction.
Hand (engaged finger)
GT
Baseline
Prediction (CodeTTalker)
Preciction [Qurs)
Figure 4: Qualitative results. Color gradients ranging from green (low activation) red (high) used to intuitively convey EMG magnitudes. See Figure 2 for the muscle localization diagram.
Qualitative Results. Since the EMG is normalized to small range, the quantitative improvements in RMSE appear marginal. Qualitative visualizations are provided in Figure compare the intuitive results of the best-performing baseline and ours. We choose task involving alternating little, ring, and middle finger movements. The line plots of 1DI-4DI muscle activities are displayed to compare the GT EMG, CodeTalker's s. and PianoKPM Net's predictions. Our method yields amplitude and dynamics more closely aligned with GT EMG, highlighting its superior capability modeling dexterous muscle activation patterns. See Appendix D.3 for more qualitative visualizations.
5.2 Held-Out Evaluation
Table 3: Dataset split and results for held-out cvaluations. We separate two test sets to measure generalization to new users (Cross-Usen) and different performance tasks Cross-Tassk
Train, Val
Test Tasks
RMSE
OTD
Users
Tasks
Hours
Users
Hours
Cross-User Cross-Task
2
6
1.18
.209 E .071
.095 .097
18
9.54
18
1
1.72
.264 +: .994
.152 1188
Configurations and Overall Results. Robust and effective muscle EMG inference requires models that generalize across different individuals and tasks. Inspired by prior work [12), we similarly construct two held-out test sets intended to evaluate across these axes independently. Table reports detailed statistics and results for each held-out setting. Specifically, Cross- User denotes unseen users during training but in- distribbution performing tasks, while Cross-Tast is the vice versa, involving unseen tasks but in-distribution users. More detailed held-out settings can be found in Appendix D.2 In the held-out evaluation, Cross-user (RMSE: 0.209, OTD: 0.095) and Cross- Task (RMSE: 0.264, OTD: 0. 12)) perform worse compared to the architectural evaluation (RMSE: 0. 134, OTD: 0.031). Notably, the model exhibits lower generalization performance on unseen tasks compared to unseen users. It stems from the lower diversity of tasks (6) relative to users (18) in the training set, and the higher semantic complexity in the task space, particularly in structured domains like musical performance. Novel tasks often engage different muscles, activation timings, and fine-grad fine-grained motor control variations that the model has not learned during training. We analyze the specific reasons in Appendix B.2 and visualize some held-out results in Appendix D.3
Generalization Across Dataset Scale. Several experiments are conducted investigate the effect of the training set scale on generalization Figure 5 (a) exhibits that adding more training users or tasks contributes to reduction in out-o--distribution estimation error since the model exposed to broader range of individual anatomics and hand kinematics, mitigating overfitting, But Cross-Task

---

(e) Generalization Acrosa Datasel Scale
(b) Adieptation to Unseen Data
BMG Extmration (rror
1.10
4.000 EIAL
12
58%
13%
23%
33%
proporiin
propo
Figure 5: Huld-out evaluation results. (a) Model generalization ability concerning the number of users or tasks included in training. (b) Model adaptation ability with different amounts of data from new user task. Distributions are across performance segments. The average EMG estimation errors (the sum of MSE and OTD) are indicated for reference, and error bars are standard deviations.
generalization is less pronounced than Cross-Usen perhaps due to the lower feature overlap between the held-out task and in-distribution tasks. We include an additional held-out test in Appendix D.2]
Adaptation to Unseen Data. We further explore whether folding a small amount of unseen data into training can facilitate few-shot learning. Figure [5] (b) proves that including 30% of the new user data can reduce EMG estimation error by 55.6%, while 30% of new task data yields a greater reduction of 63.1% %. Notably, including just 10% data from the unseen task domain already improves accuracy by approximately 50%, underscoring the critical role of kinematic diversity in enhancing EMG prediction generalization.
Limitations and Future Work
The PianoKPM Dataset is the first large-scale, high-fidelity piano EMG dataset consisting of expert-level pianists' hand muscle activities. However, the lack of novices' and intermediates' performances restricts the dataset's generalization to broader population, and synergistic wrist-forearm muscles remain unexplored in hand control. In addition, the current seven tasks are insufficient capture the complexity of bimanual coordination required in advanced repertoires such as sonatas and fugues. Future investigations include more demographic diversity, muscular variations, and musical tasks. See Appendix B.3 for other dataset limitations.
PianoKPM Dataset:
We propose PianoKPM Net, a promising approach for pose-to-EMe EMG inference that outperforms several comparative baselines. However, f that using only keystrokes as input, Code'Talker achieves the best predictions, which may be attributed to the transformer's s effectiveness at encoding long temporal dependencies, while PianoKPM is superior for multimodal learning. Future work may incorporate a transformer-based encoder or other sequence modeling frameworks extract features from temporal keystroke sequences, for example, advanced diffusion- based techniques [85 86). perception models [87 88). [8). and other multimodal networks [89]. On the other hand, our findings also highlight the persistent challenge of generalization across users and tasks.
Pia Net:
Future work could explore few-shot finetuning and transfer learning techniques, such as correlation alignment (CORAL) (90) for unsupervised domain adaptation and invariant risk minimization (IRM) (91] for domain generalization, to facilitate universally robust models capable of handling distributional variation. Other aspects for network limitations are introduced in Appendix C
7 Conclusions
This paper presents PianoKPM Net, framework for inferring dexterous hand muscle EMG during piano performance from accessible modalities. We conduct preliminary studies on general EMG- to-pose dataset. Due to its unsatisfactory transfer results in piano-playing contexts, we introduce PianoKPM Dataset,
scage,scale, high-fidelity, open-source source multimodal EMG dataset, collected from expert pianists. Trained on this dataset, PianoKPM Net achieves promising estimation from pose input alone and further improves with keystroke integration. Comprehensive evaluations assess the model architectures and generalization capabilities. Together, PianoKPM Net and PianoKPM Dataset establish a foundation for low-cost access to internal physiological and myoelectric signals, advancing human augmentation and high nigh-dimensional human- mamann-achine interaction.

---

Acknowledgments
This study is supported by JST CRONOS under Grant No. JPMJCS24N8, and JST ASPIRE under Grant No. JPMJAP2404.
References
[1] Ethan Eddy, Erik Scheme, and Scott Bateman. A framework and call to action for the future development of emg-based input in hei. In Proceedings of the 2023 CHI Conference Human Factors in Computing Systems, CHI '23, pages 1-23, New York, NY, USA, 2023, Association for Computing Machinery.
[2] T. Scott Saponas, Desney S. Tan, Dan Morris, Ravin Balakrishnan, Jim Turner, and James A. Landay. Enabling always-available input with muscle-computer interfaces. In Proceedings of the 22nd Annual ACM Symposium on User Interface Software and Technology. UIST '09, page 167-176, New York, NY. USA, 2009. Association for Computing Machinery.
[3] Momona Yamagami, Alexandra A Portnova-Faanee
Junhan Kong, Jacob O. Wobbrock, and Jennifer Mankoff. How do people with limited movement personalize upper-body gestures? considerations for the design of personalized and accessible gesture interfaces. Proceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility, ASSETS '23, pages 1- 1-15, New York, NY, USA, 2023. Association for Computing Machinery.
[4] Ray Antonius and Hendra Tjahyadi. Electromyography gesture identification using can-ran neural network for controlling quadeopters. Journal of Physics: Conference Series, 1858(1):0 20755, April 2021.
[5] Jakob Karolus, Annika Kilian, Thomas Kosch, Albrecht Schmidt, and Pawel W. Wozniak. Hit the thumb jack! using electromyography to augment the piano keyboard. In Proceedings of the 2020 ACM Designing Interactive Systems Conference, DIS 20, page 429 440, New York, NY, USA, 2020. Association for Computing Machinery.
[6] Ruofan Liu. Yichen Peng. Takanori Oku, Erwin Wu, Shinichi Furuya, and Hideki Koike. Pianokeystroke- emg: Piano hand muscle electromyography estimation from easily accessible piano keystroke. In SIG- GRAPH Asia 2024 Posters, SA '24, pages 1-2, New York, NY, USA, 2024. Association for Computing Machinery.
[7] Andrea Manca, Andrea Cereatti, Lynn Bar-On, Alberto Botter, Ugo Della Croce, Marco Knaflitz, Nicola Maffiuletti, Davide Mazzoli, Andrea Merlo. Silvestro Roatta, Andrea Turolla, and Franca Deriu. A survey the use and barriers of surface electromyography in neurorehabilitation. Frontiers in Neurology. 11:573616, October 2020.
[8] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorcalistic text-to- image diffusion models with deep language understanding, Advances in neural information processing systems, 35:36479 -74944, 2022.
[9] Simon Alexanderson, Rajmund Nagy, Jonas Beskow, and Gustav Eje Henter. Listen, denoise, action! audio-driven motion synthesis with diffiusion models. ACM Transactions on Gruphics (TOG). 42(4))1-20.
July 2023.
[10] Andrea Agostinelli, Timo I Denk, Zalán Borsos, Jesse Engel. Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. Musiclm: Generating music from text. arXiv preprint #rXiv:2301.11355, 2023.
Jngse
[11] Jehan Yang. Maxwell Soh, Vivianna Lieu, Douglas J Weber, and Zackory Erickson. Emgbench: Bench- marking duttof-dittiibution generalization and adaptation for electromyography, Advances in Neural Information Processing Systems, 37:50313-503422 2024.
[12] Sasha Salter, Richard Warren, Collin Schlager, Adrian Spurr, Shangehen Han, Rohin Bhasin, Yujun Cai, Peter Walkington, Anuoluwapo Bolarinwa, Rohert J Wang, al. emg2pose: large and diverse benchmark for surface electromyographic hand pose estimation, Advances in Neural Information Processing Systems, 37:55703-5 557288, 2024.
[13] Patrick Kaifosh and Thomas R Reardon. A generic noniavasive neuromotor interface for human-computer interaction. Nature, pages 1-10, 2025.
[14] Qijun Gan, Song Wang. Shengtao Wu, and Jianke Zhu. Pianomotion 10m: Dataset and henchmark for hand motion generation in piano performance. arXiv preprint arXiv:2406.09322, 2024.
[15] Javier Romero, Dimitrios Tzionas, and Michael J. Black. Embodied hands: modeling and capturing hands and bodies together. ACM Transactions on Graphics, 36(6):1-17, November 2017.
[16] Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Angjoo Kanazawa, David Fouhey, and Jitendra Malik. Reconstructing hands in 3d with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), pages
June 2024.
10

---

[17] Luca Mesin. Roberto Merletti, and Alberto Rainoldi. Surface eng: The issue of electrode location. Journal of eleccrromyogruphy and kinesiology official journal of the International Society of Electraphysiological Kinesiology, 19:719-26, October 2008.
[18] Yu Mike Chi, Tzyy-Ping Jung, and Gert Cauwenberghs. Dry-contact and noncontact biopotential electrodes: Methodological review. IEEE Reviews Biomedical Engineering, 3:106-119, 2010.
[19] Evan Campbell, Angkoon Phinyomark, and Frik Scheme. Current trends and confounding factors in myoelectric control: Limb position and contraction intensity. Sensors, 20(6), 2020.
[20] Reza Nourbakhsh and Carl Kukulka. Relationship between musele length and moment arm on emg activity of human triceps surac muscle. Journal of electromyography and kinesiology: official journal of the International Society of Electrephysiological Kinesiology, 14:263-73, May 2004.
[21] Faizan Haque, Mathieu Nancel, and Daniel Vogel. Myopoint: Pointing and clicking using forcarm mounted clectromyography and inertial motion sensors. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, CHI '15 page 3653-3656, New York, NY, USA, 2015. Association for Computing Machinery.
[22] T Scott Saponas, Desney S. Tan, Dan Morris, and Ravin Balakrishnan. Demonstrating the feasibility of using forcarm clectromyography for musele-computee interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '08, page 515-524, New York, NY, USA, 2008. Association for Computing Machinery.
[23] Celal Savur and Ferat Sahin. American sign language recognition system by using surface emg signal. In 2016 IEEE International Conference on Systems, Man, and Cyhernetics (SMC), pages 002872-002877, 2016.
[24] Qian Zhang, Dong Wang, Run Zhao, and Yinggang Yu. Myosign: enabling end-to-end sign language recognition with wearables. In Proceedings of the 24th International Conference on Intelligent User Interfaces, IUI '19, page 650-660, New York, NY, USA, 2019, Association for Computing Machinery.
[25] Hans Kainz, Willi Koller, Elias Wallnöfer, Till Bader, Gabriel Mindler, and Andreas Kranzl. A framework based on subject-specitfe musculoskcletal models and monte carlo simulations to personalize musele coordination retraining. Scientific Reports, 14(1):3567 February 2024.
[26] Jan Pieter Clarys and Jan Cabri and. Electromyography and the study of sports movements: review. Journal of Sports Sciences, 1 11(5)3379-448, 1993. PMID: 8301704.
[27] Jessica Sehrt. Tim Wißmann, Jan Breitenbach, and Valentin Schwind. The effects of body location and biosignal feedback modality on performance and workload using electromyography in virtual reality. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI '23, pages 1-16, New York, NY, USA, 2023. Association for Computing Machinery.
[28] Lisa Brown Jaloza. Inside facebook reality labs: Wrist-basee interaction for the next computing platform. Tech at Meta, 18, 2021.
[29] Röisín Howard. Richard Conway, and Andrew Harrison. A survey of sensor devices: use in sports biomechanics. Sports Biomechanics, 15:450-46.. May 2016.
[30] Jennifer L. Hicks, Thomas K. Uchida, Ajay Seth, Apoorva Rajagopal, and Scott L. Delp. Is my model good enough? best practices for verification and validation of musculoskeletal models and simulations of movement. Journal of Biomechanical Engineering, 137(():020905, February 2015.
[31] Ahmet Erdemir, Scott McL.ean, Walter Herzog, and Antonie J. van den Bogert. Model-based estimation of muscle forces exerted during movements. Clinical Biomechanics, 12(2):131-154, 2007.
[32] Scott L.. Delp, Frank C. Anderson, Allison S. Arnold, Peter Loan, Ayman Habib, Chand T. John, Eran Guendelman, and Darryl G. Thelen. Opensim: Open-source software to create and analyze dynamic simulations of movement. IEEE Transactions on Biomedical Engineering, 54(11):1940-1950, 2007.
[33] Ajay Seth, Jennifer L Hicks, Thomas K Uchida, Ayman Habib, Christopher L Dembia, James Dunne, Carmichael F Ong, Matthew S DeMers, Apoorva Rajagopal, Matthew Millard, et al. Opensim: Simulating musculoskeletal dynamics and neuromuscular control to study human and animal movement. PLoS Computational Biology, 1006223, July 2018
[34] Thomas Geijtenbeek. Scone: Open source software for predictive simulation of biological motion. Journal of Open Source Software, 4(38):142,, June 2019.
[35] Steve A Maas, Benjamin J Ellis, Gerard A Ateshian, and Jeffrey A Weiss. Febio: Finite elements for biomechanics. Journal of biomechanical engineering, 134:01 1005, February 2012.
[36] Claudio Pizzolato, David G. Lloyd, Massimo Sartori, Elena Ceseracciu, Thor F. Besier, Benjamin J. Fregly, and Monica Reggiani. Ceinms: A toolbox to investigate the influence of different neural control solutions on the prediction of muscle excitation and joint moments during dynamic motor tasks. Journal of Biomechanics, 48(14):3929 3936, 2015.

---

[37] Antoine Falisse, Gil Serrancoli, Christopher L Dembia, Joris Gillis, llse Jonkers, and Friedl De Groote. Rapid predictive simulations with complex musculoskeletal models suggest that diverse healthy and pathological human gails can emerge from similar control strategies. Journal of The Royal Society Interface, 16(157):2019400)) 194000 2019.
[38] Frank Anderson and Marcus Pandy. Dynamic optimization of human walking. J. Biomech. Eng. 123(5):381 -300, 2001.
[39] Scott D. Uhlrich, Antoine Falisse, Lukasz Kidziński, Julie Muccini, Michael Ko, Akshay S. Chaudhari, Jennifer L.. Hicks, and Scott I. Delp. Opencap: 3d human movement dynamics from smartphone videos. PLoS Computational Biology, 19 101 2023.
[40] Mazen Al Borno, Johanna O' O'Day, Vanessa Ibarra, James Dunne, Ajay Seth, Ayman Habib, Carmichael Ong, Jennifer Hicks, Scott Uhlrich, and Scott Delp. Opensense: An open-source toolbex for inertial- measuremenn--uit--assee measurement of lower extremity kinematies over long durations. Journal of neurvengineering and rehubilitation, 19(1):22, 2022.
[41] Timothy Hewett, Gregory Myer, Kevin Ford, Robert Heidt, Angelo Colosimo, Scott Melcan, Antonic van den Bogert, Mark Paterno, and Paul Succop. Biomechanical measures of neuromuscular control and valgus loading of the knee predict anterior cruciate ligament injury risk in female athletes prospective study. The American journal of sports medicine, 33:492-501, May 2005.
[42] Lukasz Kidziński, Bryan Yang, Apoorva Rajagopal, Scott Delp, and Michael Schwartz. Deep neural networks enable quantitative movement analysis using single-camera videos. Nature Communications, 11(1):0054, August 2020.
[43] M.A. Boswell, S.D. Uhirich, F.. Kidziński, K. Thomas, J.A. Kolesar, G.E. Gold, G.S. Beaupre, and S.I. Delp. A neural network predict the knee adduction moment in patients with osteoarthritis using anatomical landmarks obtainable from video analysis. Osteoarthritis and Cartilage. 2963)3446-36, 2021.
[44] Eni Halilaj, Apoorva Rajagopal, Madalina Fiterau, Jennifer L. Delp. Machine learning in human movement biomechanics: Best practices, common pitfalls, and new opportunities. Journal of Biomechunics, 81:1-11, 2018.
Hicks, Trevor J. Hastic, and Scott L.
[45] Ruofan Liu, Yichen Peng, Takanori Oku, Chen-Chieh Liao, Erwin Wu, Shinichi Furuya, and Hideki Koike. Piamuscle: Improving piano skill acquisition by cost-effectively estimating and visualizing activities of miniature hand muscles. In Proceedings of the 2025 CHI Conference on Human Factors Computing Systems, CHI '25, pages 1-16, New York, NY, USA, 2025, Association for Computing Machinery.
[46] Giuseppe Boceignone, Vittorio Cuculo, Giuliano Grossi, Raffaella I. Lanzarotti, and Raffaella Migliaccio. Virtual emg via facial video analysis. In Image Analysis and Processing ICIAP 2017, pages 197-207, Cham, 2017. Springer International Publishing.
[47] Vittorio Caggiano, Huawei Wang, Guillaume Durandau, Massimo Sartori, and Vikash Kumar. Myosuite-a contact-rich simulation suite for musculoskeletal motor control. arXiv preprint arXiv:2205. 13600, 2022.
[48] Yu Du, Wenguang Jin, Wentao Wei, Yu Hu, and Weidong Geng. Surface emg-based based inter-session gesture recognition enhanced by deep domain adaptation. Sensors, 17(3), 2017.
[49] Ulysse Côté-Allard, Cheikh Latyr Fall, Alexandre Drouin, Alexandre Campeau-Lecoute Clément Gosselin, Kyrre Glette, François Laviolette, and Benoit Gosselin. Deep learning for clectromyographic hand gesture signal classification using transfer learning, JEEE Transactions on Neural Systems and Rehabilitation Engineering, (764) 771, 2019.
[50] Xinyu Jiang. Xiangyu Liu, Jiahao Fan, Xinming Ye, Chenyun Dai, Edward A. Clancy, Metin Akay. and Wei Chen. Open access dataset, toolbox and benchmark processing results high-density surface electromyogram recordings. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 29: 1035-10606, 2021.
[51] Manfredo Atzori. Arjan Gijsberts, Claudio Castellini, Barbara Caputo,
Mittaz Hager, Simone Elsig. Giorgio Giatsidis, Franco Bassetto, and Henning Müller. Electromyography data for non-invasive naturally-conttrlled robotic hand prostheses. Scientific Data, 1, 2014.
[52] Manfredo Atzori, Arjan Gijsberts. Simone Heynen. Anne-Gabrielle Mittaz Hager, Olivier Deriaz, Patrick van der Smagt, Claudio Castellini, Barbara Caputo, and Henning Müller. Building the ninapro database: resouree for the biorobotics community. In 2012 4th IEEE RAS EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob), pages 1258 1265, 2012.
[53] Manfredo Atzori, Arjan Gijsberts, IIja Kuzborskij, Simone Elsig, Anne Gabrillleel Mittaz Hager, Olivier Deriaz, Claudio Castellini, Henning Müller, and Barbara Caputo. Characterization of a benchmark database for myoelectric movement classification. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 23(1):73 83, 2015.
12

---

[54] Stefano Pizzolato, Luca Tagliapietra, Matteo Cognolato, Monica Reggiani, Henning Müller, and Manfredo Atzori. Comparison of six electromyography acquisition setups hand movement classification tasks. PLoS one, 12:e0186132, October 2017.
[55] Kyungjin Sco, Junghoon Sco, Hanscok Jeong. Sangpil Kim, and Sang Ho Yoon. Tosture-informed muscular force learning for robust hand pressure estimation. Advances in Neural Information Processing Systems, 37:87831 87873, 2024.
[56] Erwin Wu, Takashi Matsumoto, Chen-Chieh Liao, Ruofan Liu, Hidetaka Katsuyama, Yuki Inaba, Noriko Hakamada, Yusuke Yamamoto, Yusuke Ishige, and Hideki Koike. Skitech: An alpine skiing and snowboard- ing datasct of 3d body pose, sole pressure, and electromyography. In Proceedings of the 6thh International Workshop on Multimedia Content Analysis in Sports, MMSports '23, page 3-8, New York, NY, USA, 2023. Association for Computing Machinery.
[57] Kun Su, Xiulong Liu, and Eli Shlizerman. Audeo: audio generation for silent performance video. Advances in Neural Information Processing Systems, 33:3325-3337, 2020.
[58] Amit Moryossef, Yanai Elazar, and Yoav Goldberg. At your fingertips: Extracting piano fingering instructions from videos. arXiv preprint arXiv:2303.0374 2023.
[59] Hui Liang, Jin Wang, Qian Sun, Yong-Jin Jin Liu, Junsong Yuan, Jun Luo, and Ying He. Barehanded music: real time hand interaction for virtual piano. In Proceedings of the 20th ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games, 13D 16, page 87-94, New York, NY, USA, 2016. Association for Computing Machinery.
[60] Kyeongeun Sco, Hyeonjoong Cho, Dacwoong Choi, Sangyub Lce, Jackyu Lec, and Jacjin Ko. Twohandsmusic: Multitask learning-based egocentric piano-playing gesture recognition system for two hands. In 2019 IEEE International Conference on Image Processing (ICIP), pages 4614-4618, 2019.
[61]
David Johnson, Daniela Damian, and George Tzanetakis. Detecting hand posture piano playing using depth data. Computer Music Journal, 43(1):59,78, 2020.
[62] Erwin Wu, Hayato Nishioka, Shinichi Furuya, and Hideki Koike, Marker-removal networks to collect precise 3d hand data for rgb-based estimation and its application in piano. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 2977-29977 January 2023.
[63] Ruocheng Wang. Pei Xu. Haochen Shi, Elizabeth Schumann, and C. Karen Liu. Fürelise: Capturing and physically synthesizing hand motion of piano performance. In SIGGRAPH Asia 2024 Conference Pupers. SA '24, pages 1-11, New York, NY, USA, 2024. Association for Computing Machinery.
[64] Takanori Oku and Shinichi Furuya. Noncontact and high-precision sensing system for piano keys identified fingerprints of virtuosity. Sensors, 22(13), 2022.
[65] Timnit Gebru, Jamie Morgenstemn, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé lii, and Kate Crawford. Datasheets for datasets. Communications of the ACM, 64(12):86-92, 2021.
[66] S Conforto and T D' Alessio. Optimal rejection of artifacts in the processing of surface emg signals for movement analysis. In Computer Methods in Biomechanics and Biomedical Engineering pages CRC Press, 2020.
[67] S.M. Kay and S.L. Marple. Spectrum analysis-a modern perspective. Proceedings of the IEEE. 69(11):380-1419, 1981.
[68] I.W. Selesnick and C.S. Burrus. Generalized digital butterworth filter design. IEEE Transactions on Signal Processing, 46(6):1688 1694, 1998
[69] Thomas S Buchanan, David G Lloyd, Kurt Manal, and Thor F Besier. Neuromusculoskeeltal modeling: estimation of musele forces and joint moments and movements from measurements of neural command. Journal of applied biomechanics, 20(4):367-395, 2004.
[70] Rubana H. Chowdhury, Mamun B. Reaz, Mohd Alauddin Bin Mohd Ali, Ashrif A. A. Bakar, Kalaivani Chellappan, and Tae G. Chang. Surface electromyography signal processing and classification techniques. Sensors, 13(9): (3(9):124311 12431-12466, 2013.
[71] Kevin Zakka, Philipp Wu. Laura Smith, Nimrod Gileadi, Taylor Howell, Xue Bin Peng, Sumeet Singh, Yuval Tassa, Pete Florence, Andy Zeng, and Pieter Abbeel. Robopianist: Dexterous piano playing with deep reinforcement learning. arXiv preprint MoXiv:2304.04100 2023.
[72] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(86):2579 26055, 2008
[73] Afroza Sultana, Farruk Ahmed, and Mohammad Alam. A systematic review on surface electromyography- based classification system for identifying hand and finger movements. Healthcare Analytics, 3:100126. November 2022.
13

---

[74] Christoph Amma, Thomas Krings, Jonas Büer, and Tanja Schultz. Advancing muscle-computer interfaces with high-density electromyography. In Proceedings of the 33rd Annual ACM Conference Human Factors in Computing Systems, CHi '15, page 929-938, New York, NY, USA, 2015, Association for
Computing Machinery.
[75] Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Goovaler, Vinicius Zamhaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv: S.fi:/8800.0226 2018.
[76] Awni Hannun, Ann Lee, Qiantong Xu, and Ronan Collobert. Sequence-to-sequence speech recognition with time- dine-depth separable convolutions. arXiv preprint 19X4v:1904.02//1 2019.
[77] Ronan Collobert, Christian Puhrsch, and Gabriel Synnaeve. Way2letter: an end-to-end convnet-based speech recognition system. arXiv preprint arXiv: 1w7Xiv:660.031999 2016.
[78] Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. Convolutional sequence to sequence learning. In International conference on machine learning, pages 1243-1252. PMLR, 2017.
[79] Raul C Simpetru, Andreas Arkudas. Dominik I Braun, Marius Osswald, Daniela Souza de Oliveira, Bjoern Eskofier, Thomas M Kinfe, and Alessandro Del Vecchio. Sensing the full dynamics of the human hand with neural interface and deep learning. BioRxiv, pages 6022-07, 2022.
[80] Arinobu Nújjima and Shoichiro Takeda. Improving putting accuracy with electrical muscle stimulation feedhack guided by muscle synergy analysis. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, CHI '25, pages 1-11, New York, NY, USA, 2025. Association for Computing Machinery.
[81] Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transportation distances. Advances in neural information processing systems, 26, 2013.
[82] Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport: With applications to data science. Foundations and Trends in Machine Learning, 1(5-6))355-6077 2019.
[83] Yilin Liu, Shijia Zhang, and Mahanth Gowda. Neuropose: 3d hand pose tracking using emg wearables. In Proceedings of the Web Conference 2021. WWW 22,, page 1471-1443 New York, NY, USA, 2021. Association for Computing Machinery.
[84] Jinbo Xing, Menghan Xia, Yuechen Zhang, Xiaodong Cun, Jue Wang, and Tien-Tsis Wong. Codetalker: Speech-driven 3d facial animation with discrete motion prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 12780-12790, 2023.
[85] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint arXiv:2009.00 09761, 2020.
[86] Kashif Rasul, Calvin Seward, Ingmar Schuster, and Roland Vollgraf. Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting. In International conference on machine learning, pages 8857-8868. 8868. PMLR, 2021.
[87] Andrew Jacgle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In International conference on machine learning, pages 4651- 4664. PMLR, 2021.
[88] Andrew Jaegle, Sebastian Borgeaud, Jean- Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, al. Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint a07. 14795, 2021.
[89] Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. National Science Review, 1 ((12))mwae403, November 2024.
[90] Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In European conference computer vision, pages 443-450. Springer, 2016.
[91] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paze Paz. Invariant risk minimization. arXiv preprint 190.1007.00893 2019.
[92] Yufei Xu, Jing Zhang, Qiming Zhang, and Dacheng Tao. Vitpose: Simple vision transformer basclines for human pose estimation. Advances in neural information processing systems, 35:38571-38584, 2022.
[93] Shinichi Furuya and Sayuri Yokota. Temporal exploration in sequential movements shapes efficient neuromuscular control. Journal of neurophysiology. 120(1):196-25) 22002 April 2018.
[94] Sara A Winges, Shinichi Furuya, Nathanicl Faber, and Martha Flanders. Patterns of musele activity for digital coarticulation. Journal of neurophysiology, 1 110(1):230 242, April 2013.
[95] Leonardo G Cohen and Mark Hallent. Hand cramps: clinical features and electromyographic patterns in focal dystonia. Neurology. 38(7):000--006 1988.
[96] Zengyi Qin, Zhenyu Jiang, Jiansheng Chen, Chunhua Hu, and Yu Ma. semg-based based tremor severity evaluation for parkinson's 's disease using a light-weight enn. IEEE Signal Processing Letters, 26(4):637- 641, 2019.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Section introduces three main claims, which are respectively achieved through (1) the code published in an anonymous GitHub repository htttps://gil thub anoKPMNet. git, allowing others to reproduce PianoKPM Net for EMG inference, (2) the multimodal piano performance dataset PianoKPM Dataset also linked on the same repo, and (3) the comprehensive evaluations of the model architecture and generalization articulated in Section
com/ruofanliu0129/ wtwpai///i.iiaaiiiitt......
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification; Some of the limitations are discussed in Section 6 Respectively, Appendix B.3 and Appendix C.33
C.3 further supplement and analyze the dataset-related and network-related
limitations, along with their underlying causes.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
15

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section 4.2 and Appendix C fully describe the architecture PianoKPM Net. http:://gtttb....mmmmmmoooooooos htwp://////.......a.......eeete. git for more reproduc-
tion details.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility.
16

---

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We provide anonymous Github repo at httpa://gi tttbb.tttuuu... com/ Nhoffan/uu1199Piamooke git, containing available code and example dataset.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
ceiiionooliiiii for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Dataset splits and reasons are specified in Table 3 and Appendix D.2, also included in configuration JSON files on Github repo. Implementation details are quantified in Appendix C22
including hyperparameters and computational resources.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Mean and standard deviation across performance segments reported in Table and Table 3 Error bars regarding the standard deviations are displayed Figure and Figure
Guidelines:
17

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative етror rates).
If error bars are reported in tables or plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Appendix C.2 includes the compute resources information.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We reviewed the NeurIPS Code of Ethics and all data and code were anonymized.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
18

---

Justification: Appendix B.3 and Appendix E discuss potential negative and positive ethical, societal, and privacy impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We expect that the piano EMG inference framework and the corresponding dataset have low risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We check all asset licenses and cite the original papers.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
19

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The new PianoKPM Dataset is collected and used with participants' consent and the code is created by the authors. Details of the data collection process are mentioned in Appendix 1.3
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details the dataset/code/model as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: Appendix B.1.3 Figure 6 and Figure introduce the collection protocol,
compensation, instruction slides, and sheet music given to participants.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [Yes]
20

---

Justification:
As mentioned in Appendix B.1.3 the study protocol is approved by local
IRB.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: This research does not involve LLMs as important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
21

---

A Overall Appendix Structure
This appendix offers extended details support and complement the main manuscript. Section B elaborates on the specifics of the PianoKPM Dataset, including a datasheet, dataset insights, and limitations. Section deescribss the proposed PianoKPM Net, completing the architecture, training and inference details, and limitations. Section Dootliines the experimental implementation for archi- tectural and held-out evaluations and includes additional visualization results that further substantiate the main findings. Finally, we discuss the broader implications in Section B offering perspectives on potential impacts and future applications.
B
PianoKPM Dataset Details
B.1
Datasheet
standardized datasheet is provided following the methodology proposed by Gebru et al. 65
B.1.1
Motivation
PianoKPM Dataset aims construct the first multimodal dataset capturing professional pianists' muscle activities (EMG), hand postures, audio, and keystroke motions during piano performances. Since EMG provide non-invasive access to internal neuromuscular signals, this dataset enables research on the pose-to-EMG correspondence understanding and data-driven modeling of dexterous motor control. Our primary objective is EMG inference, task that offers significant potential to enhance embodied interaction, skill acquisition, healthcare, and rehabilitation.
B.1.2 Composition
The dataset consists of 35,000 PKL files, including 7,000 sample sets, and each contains EMG, keystrokes, and hand pose data captured from three camera views. All data have been temporally synchronized and undergone standard preprocessing procedures, including filtering, normalizing, downsampling, and cleaning. In total. the dataset comprises performance data collected from 20 professional pianists across 7 designed tasks, each repeated 50 times per participant. The release also includes two dataset configuration JSON files, specifying the training, validation, and test splits used in architectural and held-out evaluations in Section [] All data have been fully anonymized to remove personally identifiable information. The dataset can be found in: httpss //github. com/ruof an Lww0199/PianoKBB. PiinooKMMNet. git We plan to release the raw dataset in the future, additionally including 3-view videos (720p 60FPS), raw EMG signals (2000 Hz), audio, and raw keystroke signals (1000 Hz).
B.1.3 Collection Process
Data Collection for Muscle Activity Estimation Using Deep Learning
Alm: Estimate hand muscle activity in piano
performance using deep leamning techniques.
Main Task: Participants will wear sensors on their left hand and repeatedly perform musical piece. The following four types of data will be collected: Collected Data: Electromyography (EMG) data, hand motion videos, keystrake data, performance audio.
ADI
PB
Duration: Approximately 2 hours Compensation: 6,000 JPY
3DI
m After each performance, please relax your hand and keep horizontal, touch keyboard.
making sure
m We will give a signal before each performance, so please begin only
alter the signal given.
m If you would like to take a break, feel free to let us know at any time.
Figure 6: The introduction slide presented to participants before the study. The upper-right corner illustrates the six specified hand muscles and the two types of EMG sensor setups.
22

---

Apparatus Setup. Figure 2 (a) demonstrates that all modality data is collected in standardized piano studio equipped with a Shigeru Kawai SK2L grand piano (L: 180 em, W: 152 cm, H: 102 cm. 324 kg). As Figure Sauure/Juppe upppeer- corner shows, two types of wireless EMG sensors (Delsys, Natick, MT, USA) are employed to record GT EMG at 2000 Hz with wireless transmission latency - 40 ms. A Trigno Quattro sensor (4-channel, 25 g) records signals from Muscle IDI to 4DI on the dorsal side of the hand, while two Trigno Mini sensors (1-channel, 19 g) captured Muscle ADM and PB on the palmar side. All sensor heads (25 12 X 7 mm) are placed on target muscles, with sensor bodies (27 X 46 X 13 mm) affixed to the forearm non-obstructive positions. The skin is prepped with alcohol to reduce the impedance. To minimize interference with performing, multiview motion capture system comprising three synchronized RGB cameras (1280 x 720, 60 FPS) with audio. Cameras are mounted above the keyboard center, far left, and far right around the piano. Keystroke motions are recorded using contactless optical sensor system from prior work (64), which measures the vertical displacement of all 88 keys ms temporal and 0.01 mm spatial resolution.
employ markerless
Participants Recruitment. We recruit twenty highly skilled professional pianists (16 identified as females, 4 as males), aged 20-42 (M: 2,.1, SD: 5.1), with 10-38 years (M: 20.9, SD: 6.0) of formal piano training. All participants have previously received top awards in piano competitions and are actively engaged in piano-related education and research, indicating their professional-level expertise. Twelve participants (60%) have prior experience performing with EMG sensors, suggesting a high degree of familiarity and comfort with the equipment. This indicates that over half of the participants can minimize potential interference from the sensors, supporting the reliability of the recorded EMG data. Ethical review processes are done by local Institutional Review Board (IRB). Before the study, all participants are informed about the research using an introduction slide in Figure and asked to review and sign IRB- IRBereeiewed consent form. They can ask questions and are free to withdraw from the study at any time. A retraction of consent form is also provided in advance allow them revoke consent if desired. All collected data are fully anonymized to remove any personally identifiable information. Each pianist is compensated at a rate of 3,000 JPY per hour (in total 120,000 JPY across all participants).
ID
Name
Sheet Music
TI
Ascending scale
T2
Descending scale
T3
Third Dyad
T4
Arpeggio
TS
Third Dyad
T6
Third Dyad Trill
17
All Finger Trill
Figure 7: Task descriptions. The name, sheet music, and fingering patterns are provided.
Task Design. The performance tasks are designed to capture a wide range of kinematics. In consultation with a professional pianist and neuromuscular researcher, we design seven left-hand tasks of comparable difficulty: two Scales in opposite directions, one Arpeggio, two Third Dyads, one Third Dyad Trill, and one All-finger Trill. All tasks are exclusively restricted to the left hand, as pianists typically exhibit less accurate strength control over the left hand compared to the right. Descriptions of the detailed sheet music and fingering patterns in each task can be found in Figure 7 In Appendix D.2 we follow the recommendation of an expert piano instructor to select the T4, Arpeggio, as the held-out task, which typically involves wider finger spans and faster positional transitions, encompassing distinct and challenging muscle activation patterns compared to other tasks.
23

---

Performance Styles. Each participant repeats each task fifty times, covering several distinct performance styles to induce varied muscle activations. For each task, trials I to 20 involve a progressive increase in intensity and volume, gradually from pianississimo (ppp) to fortississimo (()). Trials 21 to 30 are performed in a Legato style, emphasizing smooth transitions with relaxed arm movement and equal finger descent and ascent. Trials 31 to 40 follow Staccato style, characterized by short, crisp sounds, impulsive breaths, and rapid finger release. Finally, trials 41 to 50 are executed as fast possible to capture high-speed muscular dynamics. These diverse performing styles provide a rich range of muscle recruitment patterns, laying a solid foundation for the collected data diversity and subsequent estimation algorithm's 's robustness.
B.1.4 Preprocesingcdaannng/hbbiiinnggg
EMG and Keystroke Preprocessing. As detailed in Section 4.1.1 EMG signals undergo a series of preprocessing steps, including filtering, normalization, downsampling, and temporal alignment. Keystroke motions are first normalized to the range to based on the individual key's specific minimum and maximum heights recorded before the collection, followed by synchronization with the corresponding EMG and pose data.
Hand Pose Extracting and Preprocessing. To accurately capture the hand postures, we adopt the differentiable parametric MANO model [15] as the basis representation for hand annotation. The 3D joint positions P3D E 2 and mesh vertices Van E 777xx3 are computed with pose 0 and shape 3, through functions V 3n = M (0, (3 and Pan = P Preg [M 3)). To reduce setup complexity and better align with real-world general scenarios, we deliberately avoid multi-camera calibration. Instead, we first apply ViTPose [92] to detect 2D pose keypoints and bounding boxes of the hand region, based on which each frame is cropped to solve inhomogencity across hand spatial locations. Subsequently, HaMeR [16]. a SOTA transformer- based hand pose estimation model, is employed to reconstruct 3D hand postures with improved accuracy and robustness. The PianoKPM Dataset comprises 5.0 million images from paree-viewpoin cameras, and each frame is annotated with 3D hand postures generated by HaMeR. However, in preliminary experiments, we observe the depth (Z-axis) estimation is not sufficiently accurate to meet the requirements for precise pose inference. As such, in the subsequent network training stage, we instead use the projected 2D keypoints from the right-view depth cues from the top-view as Pan E IR V xxxx2 (V: views, J: joints) and customize a fusion pose encoder to extract semantically meaningful 3D pose representations. To address the limitations of single-fram frame posture estimation algorithms based on visual input, we briefly describe the pipeline for refining and pess-processing the extracted hand pose data in Section [4..1] Specifically, abnormal frames containing artifacts can be categorized into missing frames, where ViTPose (92] fails to detect human body key points, and invalid frames, where HaMeR (16) does not capture the hand joint positions. During error detection, we first compute the frame differences, their global mean μr, and standard deviation a based on which a threshold is predefined as TS μ + 2.5 a. During traversal, the current frame and the adjacent valid frame are compared to obtain the corresponding z-score z; = (dvalid 1)/a.. Notably, when abnormal frames occur consecutively, the growing temporal gap A/ from the last valid frame can inflate the z-score. Therefore, we apply dynamic threshold that adapts to the length of this gap by ts, = TS. : (1 + 0.1 and frame is flagged as abnormal when > ts; In addition, we traverse the sequence in forward and backward directions, and only frames identified as abnormal in their strict intersection are considered anomalous and subsequently corrected via interpolation.
replace the
B. Uses
The dataset, associated code, and dataset split configurations are released to advance academic research in EMG-based learning and modeling for purely non-commercial purposes. The provided codebase, implemented on the popular framework PyTorch, modular and adaptable to alternative application scenarios. We welcome and encourage its use as reference for related research on biosignal inference and multimodal learning.
B. Distribution and Maintenance
An example subset of the PianoKPM Dataset and the code for reproducing experimental results are available at ht/htss:: //ggtttttuuut com/ comm/uufanll wwtllllo1199aaaann/ data will be also hosted on a public cloud storage platform in the future. Contributions are welcome
git The full dataset and raw
24

---

from the broader research community, and ongoing maintenance, updates, and issue tracking will be managed and distributed through the GitHub repository.
B.2
Dataset Insights
100
Task
T6
P14
PLE
-50
-100
-100
-50
50
100
Figure 8: Visualization for EMG features after t-SNE dimensionality reduction. Different colors distinguish different tasks, and dashed circles highlight some example users' EMG data.
The collection of the PianoKPM dataset enables the analysis correlations between EMG and other modalities. Figure 8 iilustrates the visual distribution of EMG features across different tasks and users. We align the EMG of each performance into a two-dimensional map using t-SNE (72), and the results reveal clear and distinct clustering by task. For instance, T7 (brown) predominantly located in the lower-left region, while T5 (light green) is in the lower-right, indicating EMGs in different tasks have different characteristics. On the other hand, different users' EMG features of the same task may appear nearby. For example, P9 and P10 performing T5 (light green, lower-right) exhibit similar EMG patterns. This suggests that the model has the potential to learn cross-user invariant features. Such capacity may partly explain observation in Section 5.2 that the model generalizes more effectively across users than across tasks. However, the left region of the plot shows overlap among different tasks, indicating that the EMGs for certain tasks are less distinguishable. To address this issue, we design the PianoKPM Net, with an advanced feature extraction module, novel network architecture, and specialized loss function.
fixed length then reduce the feature dimensionality
B.3 Dataset Limitations
Data Fidelity. While we leverage the SOTA algorithm for 3D hand annotations, the single-frame- based inference still suffers from temporal jitter and error estimations. Since vision-based approaches rely on camera inputs, these limitations are often difficult to overcome due to occlusion and low lighting. Moreover, our system only captures the 3D positions 21 hand joints, without estimating the hand mesh, elbow, shoulder, or other upper-body parts, which are nevertheless closely related muscle activation patterns. On the other hand, during data collection, we occasionally encounter sensor detachment issues caused by hand perspiration during piano performance. Although immediate measures are taken, for example, cooling the hands, re- ce-sanitizing the skin, and repositioning the sensors, minor variations in sensor placement may inevitably occur and affect the fidelity of the recorded EMG. While this variability, rather than being limitation, may contribute positively the model's 's robustness by exposing it to intra-subject domain shifts. In real-world applications, even EMG signals from the same user may vary across sessions due to subtle changes sensor placement or physiological conditions, making this type of noise meaningful aspect of the learning process.
25

---

Ethics and Privacy. The combination of hand muscle EMG, hand motions, and keystrokes in the PianoKPM Dataset can serve as a unique biometric signature of pianist's neuromotor behavior. Such data may potentially be used to identify individuals or infer sensitive information, physical conditions, or even health status. Appropriate anonymization, encrypted storage, and controlled safeguards are essential to protect participant privacy. Moreover, EMG and motion data recorded during the piano performance may unintentionally capture a pianist's style interpretation. Clear consent procedures should be established to ensure that participants understand how their data may be analyzed, used, or shared. Despite these concerns, there are societal benefits from the development of the dexterous skill EMG-motion dataset, supporting internal muscle state feedback, motor supervision. fatigue monitoring, and even embodied interaction applications. See Appendix E for more potential impacts.
C PianoKPM Net Details
C.1 Network Architecture
Multi-Branch Feature Encoder. To extract meaningful representations from input sequences, we utilize a stack of two 1D convolutional blocks. Each block consists of a ID convolution layer followed by a ReLU activation and dropout with a rate of 0.1. Layer normalization is subsequently applied to stabilize training and improve generalization. Specifically, the two blocks expand the input dimension (postures for 84, keystrokes for 88) to 256 channels with kernel sizes of 11 and 5, strides of 1 and 1, and padding of 5 and 2, preserve the input size. For layer normalization, we apply it across the channel dimension after transposing the temporal axis, ensuring consistency with the input-output shape requirements. This design allows the network jointly model global and local temporal patterns hierarchically. Subsequently, all features are combined via element-wise addition and fed into the dependencies in the feature sequences, we adopt a hierarchical time-channel-wii encoder, inspired by a time-depth separable (TDS) structure (76). The encoder consists of sequential modules, each comprising three main components, ID convolution, a stack of TDS blocks, and a linear projection. Conv IBBlocks apply over the time dimension with kernel sizes of 17 and 9, strides of and 1, and padding of 8 and 4. They map the input channels to higher-dimensionas space, which increases representational capacity and aligns the input shape with the expected configuration of the following TDS blocks. TDS blocks include a special module to perform a 2D convolution with fixed kernel size along the time axis and channel-wise depth separation along the feature axis. The reshaped vectors enable grouped convolution over temporal windows. The output is then fused back to the original shape and combined via a residual connection. A layer normalization follows to stabilize training. Specifically, the kernel size of the Cony2DBlocks layers is (1, 9) and (1, 5), with stride of (1, 1). The input and output channels are 256 and 64 respectively.
Encoder in Section 4.2.1| To capture temporal-channel
Auto-Regge Sett-Reegressvve Decoder. In Section 4.2.2, for the backbone of the decoder, we utilize a lightweight fully connected Multi-Layer Perceptron (MLP) module to map the high-dimensional fused feature vector into a low-dimensional target space. The MLP is designed to support optional layer normalization and output scaling enhance training stability and numerical conditioning. The overall architecture of the decoder is straightforward. At each time step, the model concatenates the 64 14-dimensional embedded encoder features of the current frame with the 6-dimensional predicted EMG from the previous frame. This combined vector is fed into two fully connected layers of size 512 for each, followed by LeakyRel.U activation applied. Layer Normalization is applied after hidden layers to mitigate internal covariate shifts and facilitate faster convergence. The decoder finally outputs a 6-dimensional predicted EMG vector. Additionally, a final multiplicative scaling factor of 0.01 is applied to the output to regularize the prediction, which preserves sufficient capacity to capture the nonlinear mapping between high-level features and target signals.
Precision-Structuue Hybrid Loss. In the preliminary exploration of EMG inference networks, we found that using only MSE loss causes the network to output "safc" stable EMG values, likely since the GT EMG may exhibit subtle fluctuations. To encourage the network to better learn the EMG structural variations driven by pose and keystroke inputs, we introduce an additional loss based on optimal transport, OT loss, mentioned in Section 4.2.3 The original optimal transport problem between two discrete probability distributions f = E, a;dz, and v Σ, byóg, is formulated as: Coriginal oe (a, b) min Eis Ye Cig. But it is a linear programming (LP) problem
26

---

without gradient hence not possible for backprop. Thus, referring to prior work [8T], we add entropy regularization (1) to use the Sinkhorn-bew based optimal transport loss from the Python Geomloss library and compute a soft alignment between the predicted and ground-truth distributions as:
oot Wc(a,
min 7Ell(a.b)
EnyCi cH(r)
Here, 1 Pi log Yay is the entropy of the transport plan and e controls the regularization strength. Our subsequent experiments validate that the combination of MSE and OT losses facilitates EMG inference in high local accuracy and faithful global pattern preservation.
C.2 Implementation Details
Hyperparameters. We train the model for 200 epochs to ensure sufficient convergence. The batch size is set to 64 to balance computational efficiency and training stability. EMG and keystroke signals sampled at 1000 Hz use window length of 1024 for both training and inference. During training, sliding window with an overlap of 256 is applied for data augmentation. For the 60 FPS pose data, the window length is 60 with an overlap of 15. The number of workers and threads is specified as 0 to ensure reproducibility. The model employs an AdamW optimizer with a learning rate of 0.0001, and StepLR scheduler with step size of 20 and decay factor (gamma) of 0.5. The loss function is a weighted combination of MSE and OT losses, where the weights Amee and Aor are both set to 1.
Compute Resources. Model training is performed on high-performance computing system with an AMD EPYC 9654 (6--core/92--theaal processor, 768 GiB DDR5-4800 RAM, and NVIDIA H100 SXM5 GPUs, and the entire process takes approximately 14 hours. Notably, neither multi-GPU parallelism nor mixed-precision training is employed. Model inferring is conducted on a more accessible setup with an Intel Core 19- 10900X CPU, 128 GB RAM, and an NVIDIA GeForce RTX 4090 with 24 GB GPU memory. The model contains 3.7 million parameters and achieves batch inference within 170 ms (latency Ri 170ms). Therefore, the inference is lightweight enough to run on desktop-gun -rrade hardware without delay out-of-memory issues, which confirms its suitability for interactive applications requiring timely feedback.
C.3 Network Limitations
Generalization.
Generalization remains challenging across nearly all EMG-based research, and
our work is no exception. Prior studies have primarily approached this issue through two strategies: (I) expanding dataset scale to capture wider range of intra- and inter-subject variability [12] 13).
or (2) adopting domain generalization methods to enable robust cross-domain transfer [II]. Our future work will extend in both directions.
(1) Addressing dataset and network bias: The current dataset may overrepresent certain demographics (e.g., professional pianists), potentially introducing bias and limiting the model's 'ss upplicability to broader populations. Future models should be trained on more demographically and behaviorally diverse datasets and incorporate explicit evaluations of
generalization across ages, expertise levels, and physical conditions. (2) Improving distributional generalization: Our current experiments focus on the relationship between training set coverage and generalization to unseen users or tasks. A promising direction is to employ transfer learning or few-shot learning to better adapt models across different distributions. Alternatively, future work may leverage large-scale multimodal foundation models to encode stronger muscle-pose priors that facilitate generalization in low-data or domain- scenarios.
Recently, multimodal learning frameworks have been a new rising research hotspot, demonstrating the benefits of incorporating diverse inputs into model training [89]. While the present study focuses on hand motions and keystrokes containing direct physiological and kinematic associations with EMG signals, we retain audio, post-execution modality, to enable future investigations into multimodal fusion strategies. This choice lays the groundwork for extensible research on richer sensory integration. Looking forward, we plan to incorporate additional modalities such as audio, touch pressure, and visual sheet music, to improve the accuracy, robustness, and semantic interpretability of EMG inference. These efforts are expected to support the development of more comprehensive models for high-level EMG reasoning and nuanced performance understanding.
Other Modalities.
27

---

D Experiment Details
D.1 Baselines Implementation in Architectural Evaluation
NeuroPose [83] While NeuroPose infers 3D hand poses from EMG, our objective reverses to estimate EMG from motions, optionally enhanced with auxiliary modalities like keystrokes. To this end, we modify the original NeuroPose U-Net architecture to accommodate differences in input structures and prediction goals. Specifically, NeuroPose utilizes a single-modality input, but our framework incorporates multimodal inputs. Inspired by diffuusion-sepp embeddings in DiffWave [85), we encode keystroke information as constraints and add to the input of residual layers. The encoder composed of three sequential Conv-BN-Red ReLU-MMxXPoO layers, progressively downsampling the feature sizes by (4x4), (2x4), and (2x2) over temporal and spatial domains. This followed by five residual blocks with consistent kemel size of 3x2, while between the first and second residual blocks, we inject the encoded keystroke features into the pose representation. As well, the decoder consists of three similar Conv-BN- Men--B--ReLUS Rep-UUUpU modules, upsampling the feature sizes by (4x4), (8x4), and (8x4). final linear projection maps the output to a 6-channel EMG signal, aligning with our target muscles.
CokeTa [84]: While the original CodeTalker targets speech-driven 3D facial animation, we build upon its Transformer-basee architecture and extend it our pose-to-EMG inference task. Concretely, pose data and keystroke sequences are first processed by their modality-speciff encoders, temporally align with distinct frequency. The encoded pose representations are subsequently fused via addition, yielding unified pose embedding. The core component is transformer block that takes the fused pose embedding as the query (Q) and the encoded keystroke embedding as the key (K/V). This block consists of three sub-modules, each containing an L.N-XATTN-ReSNE layer and an L.-MLP--ResNE feedforward layer, which enable the model to inject keystroke-informed dynamics into the pose features, facilitating more physiologically plausible EMG prediction. A final FC layer maps the 512-dimension hidden features to the 6-channel EMG output.
D.2 Held-Out Evaluation
Detailed Configurations. Here, we provide more detailed elaboration of the held-out protocols described in Section 5.2 as well
additional held-out test set. For Cross-User, the held-out users are randomly sampled and for Cross-Task, the held-out task (T4 in Figure [ is chosen by professional piano teacher, which should be visually -istt--distiitution concerning the training stages. In Figure g (a), besides Cross-User and Cross-Task, most challenging but practically significant scenario, Cross-User-Task is conducted to involve both unseen users and tasks. We partition the dataset into training, validation, and test sets with an approximate ratio of 70% : 10% 20%. Validation sets sampled from the same distribution training sets, while each test set corresponds to a specific condition as described above.
(a) Cross-User- Task Configuration
(b) Cross-User- Task Generalization
0.433
EMG Eslimation Error (1)
004
0.452
0.099
0.239
0.378
0.347
Train,
Users
Tasks
Hours
0.3
0.211
0.299
0298
Test
Users
Tasks
Hours
0.20
0.1
0.0
number of training users
number of training tasks
Figure 9: Cross-User-Task configuration and results. (a) The test set is split include both unseen users and task. (b) Model generalization ability across training dataset scale, The bar charts take the same format in Figure
28

---

Cross-User-Taase From Figure (b), we can draw similar conclusions as those discussed in Sec- 5.2 In Cross-User- Dassk, increasing the number of training users and tasks both contribute to improved performance. Notably, enhancing task diversity leads to a more rapid reduction in EMG estimation error, underscoring the critical role of kinematic and postural variability in facilitating generalization. Moreover, compared to Cross- User and Cross-Tast Task, the model exhibits inferior generalization performance under Cross- -ses-Use----e Task. This highlights the greater complexity simulta- neously adapting to both unseen users and novel tasks, suggesting the need for further methodological optimization this scenario.
D.3 More Qualitative Results
Muscle
4DI
T2
GT
PlanoKPM Prediction
T3
T6
GT
PianoKPM Prediction
T4
T7
GT EMG
PianoKPM Prediction
Figure 10: Visualization results for tasks T2 to T7 from several users. Six hand muscle EMGs represented as line plots. For each task, the first row shows the GT EMG as a reference, while the second row presents the predictions given by PianoKPM Net.
Architectural Evaluation. Figure 4 presents comparative results for a representative task (TI). In this section, we further show visualization results for the remaining six tasks (T2-T7) across some users. As shown in Figure To aside from minor fluctuations in some predicted values, PianoKPM Net successfully captures the general trends of muscle activation and the amplitude-Iewe proximity to the GT EMG. This performance can be attributed to the reccisionsstruttun hybrid loss and the designed network architecture of the PianoKPM Net. Consequently, the network consistently delivers accurate and interpretable EMG predictions across all tasks and users.
Held-Out Evaluation. As shown in Figure I1 the predictions in the left half (Cross-Uie Usees- setting) are relatively more accurate. While the predicted EMG signals may not perfectly align with the GT in magnitude, the structures and activation trends of muscles are partially preserved, indicating that the model can capture user-invariant neuromuscular dynamics to some extent. In contrast, the right half (Cross-Tass Task setting) shows more discrepancies. For example, in the top-right example, PianoKPM erroneously outputs additional activation for Muscle ADM (red); similarly, in the bottom-right example, Musele (orange) activation missing. These errors may be due -istt--iistiitttionn tasks exhibiting kinematic patterns not included during training, which leads to incorrect model
29

---

Muscle
ADM
1DI
2D1
SEX
4D1
Cross-User
Cross-Tase
GT EMG
PlanoKPM Prediction
Cross-User
Cross-Task
GT EMG
PianoKPM Prediction
Figure 11:
Visualization results under Cross-User (Left) and Cross-Task (Right) settings.
fitting. On the other hand, variations across users appear to be partially captured or characterized by advanced network architecture.
E Broader Impacts
The PianoKPM framework is proposed for estimating piano hand muscle EMG during dexterous motor tasks by leveraging multiple modalities, such human-centric pose data and tool-centric keystroke data, which have potential applications across various domains. In embodied interaction, this technology enables muscle-aware user interfaces by recognizing intent and effort enhance adaptive feedback. In healthcare and rehabilitation, our low-cost and non-invasive EMG estimation can support remote muscle monitoring during therapy. In digital twins and biomechanical modeling, the predicted EMG can complement kinematics to construct more realistic, individualized, and physiologically grounded digital human models.
To further clarify the broader impact and prospects of our work, here we enumerate additional downstream applications. In the domain of piano performance, prior research reveals the neuro- muscular control and coarticulation muscle activity patterns in professional pianists with the use of EMG 93 94]. Nevertheless, placing numerous sensors on the hand impedes skillful manual movements by expert pianists, which emphasizes the importance of estimating hand muscular activi- ties in non-contact manner. On the other hand, EMG-based visualization systems are proven to support motor learning in piano training, helping users perceive subtle muscle activations (45). EMG feedback can boost users' improvisational ability by enhancing their motor control strategies (5). EMG-based analyses of the finger muscular activities identify aberrant neuromuscular control of patients with neurological disorders such as focal hand dystonia (95). In general domains, EMG signals serve as key input for prosthesis control, with recent works calling for the further integration of EMG-based input into ubiquitous interactions (T). EMG is increasingly used in classifying neuro- muscular disorders and predicting disease severity 196). Prior work leverages EMG for analyzing and enhancing motor tasks, such as gait retraining [25).
Despite its promising utility, the development of EMG estimation models introduces novel ethical and privacy concerns, such the risk of biometric identification, unintended inference of health conditions, and non-consensual bodily monitoring. Consequently, this study prioritizes informed consent, data security, and transparency to ensure responsible use. We make sure that all participants understand the nature of data collection, provide written consent, and retain the right to withdraw at any time. All data are anonymized protect privacy, ensuring the study's compliance with ethical research standards.
30


Reviews:
[{"review_id": "VQ1E9Ytzfw", "reviewer": "Reviewer_gUcJ", "summary": "This paper addresses the challenging task of estimating hand muscle electromyography (EMG) signals from more accessible modalities during piano performance. The authors introduce **PianoKPM-Net**, a multimodal learning framework designed to infer 6-channel hand muscle EMG signals from hand pose and keystroke information. In addition, they present the **PianoKPM Dataset**, a high-quality, synchronized dataset comprising EMG, pose, keystroke, and audio recordings from 20 expert pianists performing 7 distinct musical tasks. The proposed approach achieves promising results in EMG inference and is supported by comprehensive evaluations on both cross-user and cross-task generalization.", "questions": "Please refer to the **Strengths And Weaknesses**.", "limitations": "Please refer to the **Strengths And Weaknesses**.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\n1. **Novel and Well-Motivated Problem**  \n   The paper tackles the practical challenge of high-cost EMG sensing by exploring a pose-to-EMG inference approach. The initial analysis highlighting the poor transferability of general EMG datasets to piano-specific contexts provides a strong motivation for the development of domain-specific solutions.\n\n2. **Comprehensive Dataset Contribution**  \n   The authors introduce the PianoKPM dataset, offering 12.64 hours of high-quality, synchronized multimodal recordings (2000 Hz EMG, 1000 Hz keystrokes, 60 FPS video) from 20 expert pianists. The dataset is collected with proper IRB approval and is accompanied by detailed preprocessing procedures, making it a valuable resource for future research.\n\n3. **Thorough Experimental Evaluation**  \n   The paper presents comprehensive experiments demonstrating that the proposed model achieves promising performance in EMG inference. Evaluations cover both cross-user and cross-task settings, offering insights into the generalization capabilities of the model.\n\n4. **Insightful Ablations and Visualizations**  \n   The authors provide extensive ablation studies examining the effects of model components (e.g., branch configurations, temporal modules) and input modalities. Qualitative visualizations further illustrate the model’s ability to capture meaningful muscle activation dynamics.\n\n---\n\n### Weaknesses\n\n1. **Limited Architectural Novelty**  \n   While the proposed framework is effective, the multi-branch encoder architecture is relatively standard, and the paper does not introduce any fundamentally new architectural designs.\n\n2. **Missing Direct Mapping Baseline**  \n   The paper lacks a direct pose/keystroke-to-EMG mapping baseline, which would help isolate the benefits of the proposed staged design. Additionally, ablation studies exploring different encoder-decoder combinations are missing and would provide a clearer understanding of architectural trade-offs.\n\n3. **Lack of Runtime Analysis**  \n   Although the model consists of multiple components, there is no discussion of computational cost or inference time. This limits the assessment of its applicability in real-time or interactive scenarios.", "was_revised": true, "final_justification": "For my part, I believe the authors have adequately addressed my main concerns. Having reviewed the other reviewers' comments, I am inclined to maintain my recommendation as Accept.", "rebuttal": "Thank you for the encouraging rating and for clearly recognizing several strengths of our work, including the “novel and well-motivated” problem (Strength 1), the “comprehensive” dataset contribution (Strength 2), the “thorough” experimental evaluation (Strength 3), and the “insightful” ablations and visualizations (Strength 4). We are truly encouraged by your positive review. Please find our responses to the three weaknesses below.\n\n> Weakness 1: Limited Architectural Novelty. While the proposed framework is effective, the multi-branch encoder architecture is relatively standard, and the paper does not introduce any fundamentally new architectural designs.\n\n* We admit the reviewer’s comments regarding the architectural novelty of the proposed multi‑modal network. As acknowledged, our current design builds upon established structures. However, we experiment with some promising alternatives, including the UNet backbone from NeuroPose and the Transformer backbone from CodeTalker (Section 5.1, Appendix D.1), and the tuned encoder-decoder structure relatively yields more precise EMG predictions (0.155 vs. 0.143 vs. 0.134 in Table 2). Several possible factors are also provided in the manuscript (Section 5.1).\n* Moreover, perhaps other novel architectures could further advance EMG estimation. As illustrated in the limitations (Section 6), we identify several promising directions, such as diffusion-based techniques, perception models, and other multimodal networks. We also look forward to exploring more emerging designs in future work.\n\n> Weakness 2: Missing Direct Mapping Baseline. The paper lacks a direct pose/keystroke-to-EMG mapping baseline, which would help isolate the benefits of the proposed staged design. Additionally, ablation studies exploring different encoder-decoder combinations are missing and would provide a clearer understanding of architectural trade-offs.\n\n* It is true that a direct pose/keystroke-to-EMG mapping baseline is important for comparisons. But to the best of our knowledge, there is currently a lack of prior work directly estimating EMG from pose or keystroke data, specifically in dexterous skills such as piano performance. Consequently, we select two representative models as the most relevant baselines, which are originally developed for similar between-modality synthesis tasks. From this perspective, the proposed PianoKPM Net may also serve as a baseline for future studies targeting multi-modal EMG estimation.\n* Regarding the ablation studies on different encoder-decoder combinations, we do not include this analysis in the current submission due to constraints in time and space, as explained in the limitations (Section 6). Future work may incorporate a transformer-based encoder and investigate a broader range of novel networks to better understand the architectural trade-offs.\n\n> Weakness 3: Lack of Runtime Analysis. Although the model consists of multiple components, there is no discussion of computational cost or inference time. This limits the assessment of its applicability in real-time or interactive scenarios.\n\n* We appreciate the reviewer’s attention to the applicability of our model in real-time and interactive scenarios. We include the runtime analysis in Appendix C.2 Implementation Details: Compute Resources, but do not present it in the main text. Specifically, on a desktop equipped with an Intel Core i9-10900X CPU, 128 GB RAM, and an NVIDIA GeForce RTX 4090 GPU with 24 GB GPU memory, our model achieves batch inference within 170 ms. Therefore, the inference is lightweight enough to run on desktop-grade hardware without delay or out-of-memory issues, confirming its suitability for interactive applications that require timely feedback."}, {"review_id": "Ql6JyyCuNG", "reviewer": "Reviewer_kTxH", "summary": "This paper addresses the challenge of estimating hand muscle activity (electromyography, EMG) from external, easily accessible modalities, specifically in the context of piano performance. While prior work has primarily focused on inferring motion from EMG signals, this study explores the inverse problem: inferring fine-grained EMG signals from hand postures and keystroke dynamics.\n\nTo this end, the authors first examine the feasibility of pose-to-EMG inference by pretraining on a general-purpose dataset (emg2pose) and demonstrate the limitations of transferring such models to piano-specific tasks. Motivated by this gap, they introduce the PianoKPM Dataset—a large-scale, high-fidelity, multimodal dataset containing synchronized recordings of hand EMG, keystrokes, pose, and video from 20 expert pianists.\n\nBuilding on this dataset, the authors propose PianoKPM Net, a neural network designed to infer EMG signals by fusing hand pose and keystroke data. The network integrates pretrained priors, multi-branch encoders, and a time-channel-wise architecture with a hybrid loss function to capture both precise and structured EMG dynamics.\n\nThrough comparisons with baseline models and ablation studies, the paper claim that PianoKPM Net achieves superior EMG prediction accuracy. The model also shows promising generalization to unseen users and tasks, and can adapt with limited new data. The authors conclude that their framework sets a foundation for cost-effective EMG estimation and has implications for skill analysis, human-computer interaction, and embodied AI.", "questions": "1. Have you explored or considered the generalizability of PianoKPM Net to other fine-motor tasks (e.g., typing, handwriting, surgical suturing)? If possible, provide either empirical results or discussion on how transferable the learned representations are to different but related tasks.\n2. How well does PianoKPM Net perform with less curated or noisier input data—for example, RGB-only 2D keypoints without multi-view fusion or in-the-wild keystroke data with latency?\n3. How critical is the Optimal Transport (OT) component in your hybrid loss for overall performance? Could you include an ablation study comparing models trained with MSE alone versus the hybrid loss?\n4. What are the envisioned downstream applications that would directly benefit from the inferred EMG signals? Please elaborate on how EMG estimation could enable new capabilities or improve existing pipelines.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "# Strengths\n- This work addresses an underexplored problem: estimating fine-grained hand muscle activity from observable modalities in the context of a highly skilled and dexterous task (piano playing).\n- The proposed PianoKPM Net is carefully constructed and includes thoughtful architectural design choices such as a multi-branch encoder, latent prior injection, and a hybrid loss combining MSE and Optimal Transport.\n- The integration of keystroke data with hand pose for EMG estimation represents a new multimodal approach.\n\n# Weaknesses\n- While the paper convincingly highlights the technical challenge of pose-to-EMG inference, it falls short in articulating why this capability is important for downstream tasks. Without a clearer connection to downstream benefits, it is difficult to fully appreciate the practical significance of high-fidelity EMG estimation in the piano domain or beyond.\n- While the paper reports competitive performance, the improvements over baselines in some metrics (e.g., RMSE) appear modest and may benefit from deeper statistical analysis or error case exploration.\n- While the dataset and model are tailored for piano performance, their generalizability to other fine motor tasks (e.g., typing, surgery) is not empirically explored.", "was_revised": true, "final_justification": "The rebuttal addressed my major concerns, so I raise my score to 4.", "rebuttal": "We thank the reviewer for the valuable comments, and we organize your concerns into three categories: insufficient description of downstream applications (Weakness 1, Q4), incomprehensive analysis of model performance (Weakness 2, Q2, Q3), and lack of investigation into the generalizability to other skills (Weakness 3, Q1). We first briefly revisit the core contributions of this research and subsequently address each of these concerns.\n\n**Recap of Our Core Concept:**\n* We want to clarify the overall motivation and methodology of this research. Our study initially investigates pose-to-EMG mapping using a **general-purpose** EMG dataset. However, **limited transferability** to dexterous skills like piano performance is observed, which hence motivates us to collect a **domain-specific** dataset tailored to piano playing. Accordingly, the proposed method is inherently designed for a downstream task, EMG inference during piano performance. Furthermore, as illustrated (Section 1, Section 2.1), the inferred EMG signals offer clear and meaningful benefits across multiple domains. We also outline their potential applications in other areas below.\n\n**1. Insufficient description of downstream applications (Weakness 1, Q4):**\n> Weakness 1, Q4: The paper falls short in articulating why pose-to-EMG inference is important for downstream tasks. What are the envisioned downstream applications that would directly benefit from the inferred EMG signals?\n* As discussed in the manuscript  (Section 1, Section 2.1), EMG is critical for a wide range of applications. While our work focuses primarily on accurately inferring hand muscle EMG from multimodal inputs during performing one of the most dexterous motor tasks, we agree that evaluating the utility of the inferred EMG in downstream tasks is an important direction to highlight the significance of this study for future work. Specifically, potential downstream applications may include:\n  * **In the piano performance domain:**\n    * Performance Analysis: Prior research reveals the neuromuscular control of muscle activity patterns in professional pianists with the use of EMG [1]. Nevertheless, placing numerous sensors on the hand impedes skillful manual movements by expert pianists, which emphasizes the importance of estimating hand muscular activities in a non-contact manner.\n    * Skill Acquisition: EMG-based visualization systems can support motor learning in piano training, helping users perceive subtle muscle activations [2].\n    * Improvisation Support: EMG feedback may boost users’ improvisational ability by enhancing their motor control strategies [3].\n    * Clinical Application: EMG-based analyses of the finger muscular activities identify aberrant neuromuscular control of patients with neurological disorders such as focal hand dystonia [4]. \n  * **In general domains:**\n    * Interaction Control: EMG signals serve as key input for prosthesis control, with recent works calling for the further integration of EMG-based input into ubiquitous interactions [5].\n    * Motor Skill Optimization: Prior work leverages EMG for analyzing and enhancing motor tasks, such as gait retraining [6].\n* Overall, the inferred EMG can be integrated into the mentioned downstream applications, which have broader potential across various domains (Appendix E).\n\n**2. Incomprehensive analysis of model performance (Weakness 2, Q2, Q3):**\n> Weakness 2: The improvements over baselines in some metrics (e.g., RMSE) appear modest and may benefit from deeper statistical analysis or error case exploration.\n* RMSE is selected as an evaluation metric due to its widespread use in signal processing for quantifying reconstruction errors. However, we acknowledge its limitations, such as sensitivity to outliers in error cases. To mitigate these shortcomings, we complement RMSE with qualitative visualizations (Figure 4, Figure 10) and the quantitative Optimal Transport Distance (OTD), which captures the similarity between synergistic muscle activation patterns (Table 2). Also, we would like to emphasize that the lack of established quantitative metrics for evaluating EMG prediction remains an open challenge, which is also recognized by Reviewer k3DA: “The validation of similarity of real EMG data and predicted data remains somewhat qualitative, and the shown examples are convincing [...] The paper discusses, in a nice and satisfying way the difficulties associated with this question”. Consequently, we think this gap is partially addressed via qualitative results and OTD, and we are optimistic that future research will continue to develop more tailored evaluation metrics for EMG estimation.\n* Regarding the suggestion for error case exploration, we agree that such analysis would provide valuable insights. If the manuscript is accepted, we plan to illustrate and analyze more error cases by including additional comparisons against baselines.\n\n> Q2: How well does PianoKPM Net perform with less curated or noisier input data-for example, RGB-only 2D keypoints without multi-view fusion or in-the-wild keystroke data with latency?\n* **RGB-only 2D keypoints:** In the early-stage explorations, we experiment with only 2D hand keypoints from a top-down view as pose input. However, the results are suboptimal, which is due to the inherent limitations of 2D keypoints in capturing the semantic information necessary for reliable EMG prediction. Top-down 2D keypoints lack critical depth information, such as joint height and press strength, which are essential for distinguishing variations in hand kinematics and muscle engagements. Therefore, we introduce more informative multi-view fused 3D pose data for EMG inference. This richer representation improves the network to model fine-grained hand muscle EMG accurately.\n* **In-the-wild keystroke data:** We have not yet conducted evaluations under such conditions. Given the sequential features of keystroke data, we assume that certain algorithmic adjustments, such as temporal synchronization or filtering strategies, may be applied to mitigate the effects of latency to some extent.\n\n> Q3: How critical is the Optimal Transport (OT) component in your hybrid loss for overall performance? Could you include an ablation study comparing models trained with MSE alone versus the hybrid loss?\n* We agree on the suggestion of conducting an ablation study for the hybrid loss design, which is a constructive direction to strengthen our research. This part is not included in the manuscript due to space limitations. Still, during the initial investigation, we use MSE alone as the loss function, which leads the model to converge to overly smooth and conservative outputs, where muscle activation peaks, valleys, and transitions are suppressed. Thus, a hybrid loss function combining MSE with OT is designed to balance local accuracy with global structure preservation, which can hence encourage the model to capture the dynamically structured nature of muscle activations (Section 4.2.3).\n* We plan to perform an ablation study comparing different loss configurations that we are happy to include in the final version of this paper.\n\n**3. Lack of investigation into the generalizability to other skills (Weakness 3, Q1):**\n> Weakness 3, Q1: The dataset and model’s generalizability to other fine motor tasks is not explored (e.g., typing, handwriting, surgical suturing).\n* As noted in the recap of our core concept, we recognize the appeal of a model that can generalize across diverse motor tasks, while a task-specific model remains of substantial practical significance, particularly in real-world contexts. Reviewer gUcJ also acknowledges this point: “The initial analysis highlighting the poor transferability of general EMG datasets to piano-specific contexts provides a strong motivation for the development of domain-specific solutions”. \n* On the other hand, generalization remains challenging across nearly all EMG-based research, and our work is no exception. Our results show reasonable generalization across users, but noticeably weaker performance across tasks (Section 5.2, Section 6, Appendix C.3). These findings somewhat underscore the necessity of fine-tuning on task-specific domains when transferring to a new motor task.\n* We believe that the overall workflow holds potential generalizability to other motor tasks, such as typing, handwriting, and surgical suturing. By leveraging a general dataset to learn priors for pose-to-EMG mapping, followed by task-specific adaptation through new data collection, the proposed network could be fine-tuned for various downstream tasks. While our current work focuses on validating the feasibility of this workflow within the piano performance domain, we are optimistic that the same framework can be extended to estimate EMG in other motor tasks in future work.\n\n**Reference:**\n\n[1] Furuya S, Yokota S. Temporal exploration in sequential movements shapes efficient neuromuscular control. J Neurophysiol. 2018 Jul 1;120(1):196-210.\n\n[2] Liu R et al. PiaMuscle: Improving Piano Skill Acquisition by Cost-effectively Estimating and Visualizing Activities of Miniature Hand Muscles. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI '25).\n\n[3] Karolus J et al. Hit the Thumb Jack! Using Electromyography to Augment the Piano Keyboard. In Proceedings of the 2020 ACM Designing Interactive Systems Conference (DIS '20).\n\n[4] Cohen LG, Hallett M. Hand cramps: clinical features and electromyographic patterns in a focal dystonia. Neurology. 1988 Jul;38(7):1005-12.\n\n[5] Eddy E et al. A Framework and Call to Action for the Future Development of EMG-Based Input in HCI. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI '23).\n\n[6] Kainz H et al. A framework based on subject-specific musculoskeletal models and Monte Carlo simulations to personalize muscle coordination retraining. Scientific Reports. 14."}, {"review_id": "yy4EvsoYXC", "reviewer": "Reviewer_k3DA", "summary": "The paper presents a carefully validated architecture that succeeds to predict EMG signals with reasonable accuracy from multimodal more easily to access data streams for piano playing. These data streams include video data providing 3D poses of the hand movements, and the keystroke sequences derived for the keyboard. In addition, the study presents a multimodal dataset that includes these inputs an combination with normalized EMG signals from 6 hand muscles for seven different types of piano sequences. The neural network architecture for the prediction of EMG signals combines encoder networks for hand pose and keystroke sequences, where the pose information is augemnted by adding prior information of from and additional model that encodes prior information about the mapping between hand pose and EMG sequences.  The two information streams are combined using a channelwise encoder architecture, providing input to an autoregressive network that predicts the EMG sequence. \n\nI is convincingly demonstrated that the architecture, in contrast to simpler benchmark architectures, results in predictions that show substantial similarities with ground truth EMG data. In addition, it is demonstrated that the addition of the keystroke sequences as inputs again improves the accuracy of the predictions. Further validation holding out data from  individual users, or of parts of the piano movement tasks, demonstrates that the architecture is able (to some degree) to generalize to new users and tasks.  Specifically, it is shown that addition of a small amount of user-specific data improves the prediction accuracy for new users massively, which is important for the realization of accurate individually-specific predictions in different applications.", "questions": "What would be main obstacles to generalize the method to more piano tasks, and potentially larger classes of hand movements?\n\nTable 3 and results for held out evaluation: Here  the table certain values (.209 etc) are given, but it remains unclear whether these values are good or bad. One sentence in the interpreting section would help. \n\nThere is some older work trying to predict facial EMG from video seuences (e.g. Boccignone, Giuseppe & Cuculo, Vittorio & Grossi, Giuliano & Lanzarotti, Raffaella & Migliaccio, Raffaella. (2017). Virtual EMG via facial video analysis.  9th International Conference on Image Analysis and Processing). Maybe it would be suitable to quote such related work in the domain of facial movements and facial action units?\n\nL. 219 ff.: The description of the time encode archietcture is somewhat hard to follow, unless one digs it up in the supplement.. Specifically, it seems not to be clear what these variables C, c and w indicate.  Likewise for D_f.\n\nMinor questions: \nAbstract: what is the maining of 'prohibitive devices', and 'gross motors' ?\n\nFig 1 (2): Evaluationg -> Evaluation (?)", "limitations": "Limitations of the approach are thoroughly discussed, also also in the supplement.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "The paper is very well written and methods and evaluation is very solid. The paper presents one of the fist solutions for the problem of accurate prediction of EMG data from multimodal data, which is likely an important field for future research given the difficulties associated with EMG recordings. In addition, the paper presents a nice multimodal data sets including EMG, video data and keystroke data, which is very useful for benchmarking similar architectures. The main points that the paper makes, that EMG signals can be predicted with a decent accuracy for multimodal data is made convincingly. \n\nThe valiation of similarity of real EMG data and predicted data remains somewhat qualitative, and the shown examples are convinving, but raise the question what the point and criteria are, that would define a satisfying prediction of EMG data form multimodal inputs. The paper discusses, in my view in a nice and satisfying way the difficulties associated with this question (interindividual diferences, etc.).\n\nAnother limitation, which also is adequately discussed, is the constrained set of movements (piano, only 7 sequences), where it still seems to require substantial work to extend the approach to less constrained applications. \n\nA minor point is: \nA lot of details of the algorithms are described in the supplement, while there was a relatively detailed description, how tracking errors of the pose data are interpolated. This made it  somewhat difficult to grasp the main components of the core neural network architecture in the main text. Maybe the manuscript would gain if some more critical details of the main architecture would be given in the main text, and the interpolation of the pose data would be shifted in the supplement, which seems more like a technical detail.", "was_revised": true, "final_justification": "The rebuttal has answered all my concerns in a satisfying manner. Likewise, it seems to do this for the other reviewers. For this reason I maintain my score (accept).", "rebuttal": "Thank you for the encouraging rating. We are especially grateful that the research is recognized as the “first” and “important” solution for the accurate prediction of EMG data, the proposed dataset is “nice” and “useful”, the architecture is “carefully validated” and “convincing”, the evaluation is ”very solid” and “satisfying”, and the paper is “very well written”. Please find our responses to your questions below.\n\n> Weakness: A minor point is: A lot of details of the algorithms are described in the supplement, while there was a relatively detailed description, how tracking errors of the pose data are interpolated. This made it somewhat difficult to grasp the main components of the core neural network architecture in the main text. Maybe the manuscript would gain if some more critical details of the main architecture would be given in the main text, and the interpolation of the pose data would be shifted in the supplement, which seems more like a technical detail.\n\n> Q4: L. 219 ff.: The description of the time encode archietcture is somewhat hard to follow, unless one digs it up in the supplement. Specifically, it seems not to be clear what these variables C, c and w indicate. Likewise for D_f.\n\n* We respectfully take the liberty of addressing a minor weakness and Q4 together, since they pertain to the clarity and completeness of the network architecture description. The main text indeed lacks a focused interpretation of the core network design, while including pose preprocessing details that may not be essential (Section 4.1.1). We apologize for any confusion this may cause, particularly regarding variable definitions and architectural components. More detailed information is currently provided in the appendix (Appendix C.1). Also, we clarify some mentioned variables here: \n  * $C = c \\times w$ represents the reshaped feature dimension, where $c$ denotes the number of channels and $w$ is the feature width.\n  * $D_p$, $D_k$, $D_l$, and $D_f$ refer to the output dimensions of the pose, keystroke, latent prior, and fused embedded features after their respective encoders. These notations are introduced to concisely describe feature dimensions across modules.\n* Additionally, in future revisions, we will carefully reorganize the content of the main manuscript (Section 4) and the appendix (Appendix B.1, Appendix C.1). In detail, we plan to move the pose data preprocessing algorithms to the appendix, and correspondingly incorporate critical model details into the main text to improve clarity. We appreciate your advice, which will help us make the paper more accessible and informative to a broader readership.\n\n> Q1: What would be main obstacles to generalize the method to more piano tasks, and potentially larger classes of hand movements?\n\n* As the reviewer rightly points out, the main obstacles to generalizing our method lie in two key areas: inter-task differences and inter-individual differences.\n  * **Inter-task differences:** Task-level variability introduces greater diversity in hand kinematics. The complexity of piano performance, such as rapid chord transitions or irregular fingering strategies, leads to more diverse and less stereotypical movement patterns. Our results confirm that for unseen tasks, more extensive and heterogeneous training data are required to maintain robust generalization performance (Section 5.2).\n  * **Inter-individual differences:** These differences stem from the non-uniformity of motor strategies across pianists. Different pianists may accomplish the same task using distinct muscle recruitment patterns, which poses challenges for cross-user generalization. We also discuss potential solutions, such as few-shot learning and domain adaptation (Sections 5.2, Section 6, Appendix C.3).\n* Despite these challenges, the framework aims to offer a foundation for generalization. The modular encoder-decoder architecture is flexible and supports fine-tuning with limited additional data (Section 5.2, Appendix D.3). We are optimistic about its potential to scale to more complex tasks through further data collection and architectural refinement.\n\n> Q2: Table 3 and results for held out evaluation: Here the table certain values (.209 etc) are given, but it remains unclear whether these values are good or bad. One sentence in the interpreting section would help.\n\n* We apologize for the insufficient explanation in the manuscript for Table 3 and will revise the corresponding section. Specifically, we will add a statement clarifying that in the held-out evaluation, *Cross-User* and *Cross-Task* settings perform worse compared to the architectural evaluation. Notably, the model exhibits poorer generalization on unseen tasks than on unseen users. This clarification might help readers better understand the reported values.\n\n> Q3: There is some older work trying to predict facial EMG from video seuences (e.g. Boccignone, Giuseppe & Cuculo, Vittorio & Grossi, Giuliano & Lanzarotti, Raffaella & Migliaccio, Raffaella. (2017). Virtual EMG via facial video analysis. 9th International Conference on Image Analysis and Processing). Maybe it would be suitable to quote such related work in the domain of facial movements and facial action units?\n\n* We thank the reviewer for bringing this relevant work to our attention. While our research focuses on hand muscle EMG estimation during dexterous piano performance, the mentioned study provides valuable insights into face EMG simulation from visual input. In the modified manuscript, we will conduct a more comprehensive survey of related work and include a citation of the mentioned paper to better contextualize our contributions.\n\n> Q5: Minor questions: Abstract: what is the maining of 'prohibitive devices', and 'gross motors' ?\n\n* Prohibitive devices refer to costly and inaccessible equipment such as Delsys EMG sensors (Appendix B.1.3). Gross motors denote movements primarily involving large muscles, such as jogging, jumping, or squatting (Section 2.1). In contrast, the dexterous motor skills targeted in our work, like piano performance, mainly engage small muscles. These terms will be clarified in the revised manuscript for better readability.\n\n> Q6, Paper Formatting Concerns: Minor proofreading mistakes and incomplete references.\n\n* Many thanks for your careful attention to these details, and we will revise them correspondingly. A thorough proofreading of the manuscript will be conducted, and all references will be fully completed and formatted to comply with the required style guidelines."}, {"review_id": "srJ4SZdmCT", "reviewer": "Reviewer_DhVG", "summary": "This paper addresses the topic of electromyography (EMG) inference from hand poses. The authors introduce a hand muscle estimation framework, PianoKPM Net. The framework pre-learns latent correspondences between hand poses and EMG signals using the general EMG gesture dataset. It then enables high-fidelity EMG inference by leveraging PianoKPM Dataset, a self-collected multimodal dataset specifically designed for piano-playing tasks. A comprehensive evaluation compares the proposed approach with reproducible competitive baselines in the context of piano playing, and examines its generalization and adaptation across different users and tasks.", "questions": "Was the audio data used solely as part of the multimodal dataset for the piano-playing scenario, or was it also incorporated into the proposed methodology? It is not evident whether the audio modality played any role beyond data collection in this study.", "limitations": "Yes, the authors adequately address the limitations associated with both the dataset and the framework in the paper.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The paper is well-written and easy to follow.\n- To address a specific target task, the authors developed a novel dataset (PianoKPM Dataset) by systematically collecting and integrating multiple modalities tailored for piano-playing scenarios. The accompanying framework (PianoKPM Net) is carefully designed to validate both the utility and the methodological soundness of the dataset.\n- The paper provides a discussion on the EMG modality, and the authors take on the challenging task of inferring EMG signals rather than using them as input. This reverse approach is interesting. The paper effectively demonstrates the validity of this approach through reasonable evaluations and intuitive experimental results.\n\nWeaknesses:\n- Based on Figure 7 in Appendix B.1.3 and the description in the main text, it shows that the data was collected using a scenario involving alternating little, ring, and middle finger movements. In this regard, it is suggested that incorporating data and training under more randomized finger movement scenarios would be beneficial. Repetitive and predictable task designs in biosignal experiments can induce participant anticipation or attentional bias, which may in turn introduce unwanted bias into signal interpretation.\n- One aspect that could be further explored is the evaluation of the predicted EMG signals beyond similarity metrics. The work focuses on EMG inference, but broader utilization of the inferred signals was not pursued, which could be a potential direction for future work. It would be valuable to explore how well the inferred EMG can support downstream tasks, such as pose prediction or reconstruction of piano playing. Given that the dataset is specifically tailored for piano-playing scenario, future work could benefit from exploring these applications further.", "was_revised": true, "final_justification": "This paper aims to infer EMG signals from hand poses. To achieve this goal, the authors designed a dedicated experimental paradigm and proposed a tailored framework accordingly.\n\nHowever, there are concerns that repetitive movements during the experimental process may induce participant anticipation or attentional bias. Additionally, although the dataset was collected in a multimodal manner, modalities such as audio were not utilized in the proposed method.\n\nThe scope of the task is highly specific, and while the proposed EMG inference method shows potential for broader applications, these possibilities remain unexplored and are only mentioned as future work. Based on these aspects, I assigned relatively low ratings for 'Significance' and 'Originality'.\n\nThat said, the paper presents thorough analyses with clear arguments and maintains high readability, which enhances understanding. Therefore, I rated 'Quality' and 'Clarity' highly.\n\nTaking all factors into account, my final recommendation is 'Borderline Accept'.", "rebuttal": "Thank you for your constructive feedback and questions. We are heartened by your recognition that our work tackles a “challenging” task and provides an “interesting” approach (Strength 3). You give positive remarks regarding the “novel” PianoKPM Dataset, the “carefully designed” PianoKPM Net, and the “reasonable and intuitive” experimental evaluation (Strength 2, Strength 3). Also, the paper is “well-written”, “easy to follow”, and “effectively” demonstrates the contributions we aim to highlight (Strength 1, Strength 3). Hereafter, we will address the identified weaknesses and questions point-by-point.\n\n> Weakness 1: Based on Figure 7 in Appendix B.1.3 and the description in the main text, it shows that the data was collected using a scenario involving alternating little, ring, and middle finger movements. In this regard, it is suggested that incorporating data and training under more randomized finger movement scenarios would be beneficial. Repetitive and predictable task designs in biosignal experiments can induce participant anticipation or attentional bias, which may in turn introduce unwanted bias into signal interpretation.\n\n* We appreciate the thoughtful observation regarding the potential risk of attentional bias or participant anticipation arising from repetitive and predictable finger movement patterns. This is indeed a valid concern in biosignal experiments. \n\n* Before data collection, all tasks are designed by a professional pianist and neuromuscular researcher (Appendix B.1.3). The use of structured and alternating finger movements is intended to ensure the involvement of all six hand muscles. We would also like to note that the twenty pianists recruited in our study are all highly skilled professionals, each having won top awards in piano competitions and actively engaged in piano-related education and research. Most of them have previously participated in piano data collections and are trained to deliver consistently high-standard performances while minimizing behavioral or attentional bias (Appendix B.1.3). As a result, we do not observe any systematic artifacts or distortions in the current dataset. \n\n* Moreover, we incorporate several distinct performance styles to induce varied muscle activations (Appendix B.1.3). This design enables us to collect interpretable correspondences between pose and EMG data, which are essential for validating the proposed model.\n\n* On the other hand, we fully agree that incorporating more randomized and naturalistic finger movement scenarios will be essential to evaluate the robustness of our approach. In future dataset expansions, we plan to include less predictable motor tasks to mitigate potential bias and better reflect real-world usages.\n\n> Weakness 2: One aspect that could be further explored is the evaluation of the predicted EMG signals beyond similarity metrics. The work focuses on EMG inference, but broader utilization of the inferred signals was not pursued, which could be a potential direction for future work. It would be valuable to explore how well the inferred EMG can support downstream tasks, such as pose prediction or reconstruction of piano playing. Given that the dataset is specifically tailored for piano-playing scenario, future work could benefit from exploring these applications further.\n\n* While our work focuses primarily on accurately inferring hand muscle EMG from multimodal inputs during performing one of the most dexterous motor tasks, we agree that evaluating the utility of the inferred EMG in downstream tasks is an important direction to highlight the significance of this study for future work. We partially discuss the potential utilization of the inferred EMG signals in the manuscript (Section 1, Section 2.1). To further clarify the broader impact and prospects of our work, here we enumerate additional downstream applications, in the domain of piano performance and general domains:\n  * **In the piano performance domain:**\n    * Performance Analysis: Prior research reveals the neuromuscular control and coarticulation of muscle activity patterns in professional pianists with the use of EMG [1, 2]. Nevertheless, placing numerous sensors on the hand impedes skillful manual movements by expert pianists, which emphasizes the importance of estimating hand muscular activities in a non-contact manner.\n    * Skill Acquisition: EMG-based visualization systems are proven to support motor learning in piano training, helping users perceive subtle muscle activations [3].\n    * Improvisation Support: EMG feedback can boost users’ improvisational ability by enhancing their motor control strategies [4].\n    * Clinical Application: EMG-based analyses of the finger muscular activities identify aberrant neuromuscular control of patients with neurological disorders such as focal hand dystonia [5]. \n  * **In general domains:**\n    * Interaction Control: EMG signals serve as key input for prosthesis control, with recent works calling for the further integration of EMG-based input into ubiquitous interactions [6].\n    * Disease Diagnosis: EMG is increasingly used in classifying neuromuscular disorders and predicting disease severity [7].\n    * Motor Skill Optimization: Prior work leverages EMG for analyzing and enhancing motor tasks, such as gait retraining [8].\n* Overall, the inferred EMG can be integrated into the mentioned downstream applications, for example, in piano performance, real-time feedback during training, performing technique assessment, and EMG-driven interaction interfaces, which also have broader potential across various domains (Appendix E).\n\n> Question: Was the audio data used solely as part of the multimodal dataset for the piano-playing scenario, or was it also incorporated into the proposed methodology? It is not evident whether the audio modality played any role beyond data collection in this study.\n\n* In the current study, the audio data is included as part of the multimodal dataset, but is not incorporated into the proposed methodology. Our intent in providing audio data is to support extensible research on richer sensory integration. We hope that the availability of audio, alongside pose, keystroke, EMG, and other modalities, can encourage the development of more comprehensive models for high-level EMG reasoning and nuanced performance understanding, which is also noted in the appendix (Appendix C.3).\n\n**Reference:**\n\n[1] Furuya S, Yokota S. Temporal exploration in sequential movements shapes efficient neuromuscular control. J Neurophysiol. 2018 Jul 1;120(1):196-210.\n\n[2] Winges SA et al. Patterns of muscle activity for digital coarticulation. J Neurophysiol. 2013 Jul;110(1):230-42.\n\n[3] Liu R et al. PiaMuscle: Improving Piano Skill Acquisition by Cost-effectively Estimating and Visualizing Activities of Miniature Hand Muscles. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI '25).\n\n[4] Karolus J et al. Hit the Thumb Jack! Using Electromyography to Augment the Piano Keyboard. In Proceedings of the 2020 ACM Designing Interactive Systems Conference (DIS '20).\n\n[5] Cohen LG, Hallett M. Hand cramps: clinical features and electromyographic patterns in a focal dystonia. Neurology. 1988 Jul;38(7):1005-12.\n\n[6] Eddy E et al. A Framework and Call to Action for the Future Development of EMG-Based Input in HCI. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI '23).\n\n[7] Qin Z et al. sEMG based Tremor Severity Evaluation for Parkinson's Disease using a Light-weight CNN. IEEE Signal Processing Letters. PP. 1-1.\n\n[8] Kainz H et al. A framework based on subject-specific musculoskeletal models and Monte Carlo simulations to personalize muscle coordination retraining. Scientific Reports. 14."}]

Author rebuttal:

Meta-review: This paper studies a very specific task on estimating muscle electromyography (EMG) signals from more accessible modalities in the context of piano playing. In this specific setup, a multimodal dataset is constructed, and a corresponding neural network is proposed to achieve the goal of solving the task. The experiments show that the importance of this specifically curated dataset and the proposed network.
Decision: Accept (poster)