Title: From Pose to Muscle: Multimodal Learning for Piano Hand Muscle Electromyography
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multimodal Learning

Electromyography Estimation

Muscle Analysis
Abstract: Muscle coordination is fundamental when humans interact with the world. Reliable estimation of hand muscle engagement can serve as a source of internal feedback, supporting the development of embodied intelligence and the acquisition of dexterous skills. However, contemporary electromyography (EMG) sensing techniques either require prohibitively expensive devices or are constrained to gross motor movements, which inherently involve large muscles. On the other hand, EMGs exhibit dependency on individual anatomical variability and task-specific contexts, resulting in limited generalization. In this work, we preliminarily investigate the latent pose-EMG correspondence using a general EMG gesture dataset. We further introduce a multimodal dataset, PianoKPM Dataset, and a hand muscle estimation framework, PianoKPM Net, to facilitate high-fidelity EMG inference. Subsequently, our approach is compared against reproducible competitive baselines. The generalization and adaptation across unseen users and tasks are evaluated by quantifying the training set scale and the included data amount.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the topic of electromyography (EMG) inference from hand poses. The authors introduce a hand muscle estimation framework, PianoKPM Net. The framework pre-learns latent correspondences between hand poses and EMG signals using the general EMG gesture dataset. It then enables high-fidelity EMG inference by leveraging PianoKPM Dataset, a self-collected multimodal dataset specifically designed for piano-playing tasks. A comprehensive evaluation compares the proposed approach with reproducible competitive baselines in the context of piano playing, and examines its generalization and adaptation across different users and tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper is well-written and easy to follow.\n- To address a specific target task, the authors developed a novel dataset (PianoKPM Dataset) by systematically collecting and integrating multiple modalities tailored for piano-playing scenarios. The accompanying framework (PianoKPM Net) is carefully designed to validate both the utility and the methodological soundness of the dataset.\n- The paper provides a discussion on the EMG modality, and the authors take on the challenging task of inferring EMG signals rather than using them as input. This reverse approach is interesting. The paper effectively demonstrates the validity of this approach through reasonable evaluations and intuitive experimental results.\n\nWeaknesses:\n- Based on Figure 7 in Appendix B.1.3 and the description in the main text, it shows that the data was collected using a scenario involving alternating little, ring, and middle finger movements. In this regard, it is suggested that incorporating data and training under more randomized finger movement scenarios would be beneficial. Repetitive and predictable task designs in biosignal experiments can induce participant anticipation or attentional bias, which may in turn introduce unwanted bias into signal interpretation.\n- One aspect that could be further explored is the evaluation of the predicted EMG signals beyond similarity metrics. The work focuses on EMG inference, but broader utilization of the inferred signals was not pursued, which could be a potential direction for future work. It would be valuable to explore how well the inferred EMG can support downstream tasks, such as pose prediction or reconstruction of piano playing. Given that the dataset is specifically tailored for piano-playing scenario, future work could benefit from exploring these applications further.\n\nQUESTIONS:\nWas the audio data used solely as part of the multimodal dataset for the piano-playing scenario, or was it also incorporated into the proposed methodology? It is not evident whether the audio modality played any role beyond data collection in this study.\n\nLIMITATIONS:\nYes, the authors adequately address the limitations associated with both the dataset and the framework in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a carefully validated architecture that succeeds to predict EMG signals with reasonable accuracy from multimodal more easily to access data streams for piano playing. These data streams include video data providing 3D poses of the hand movements, and the keystroke sequences derived for the keyboard. In addition, the study presents a multimodal dataset that includes these inputs an combination with normalized EMG signals from 6 hand muscles for seven different types of piano sequences. The neural network architecture for the prediction of EMG signals combines encoder networks for hand pose and keystroke sequences, where the pose information is augemnted by adding prior information of from and additional model that encodes prior information about the mapping between hand pose and EMG sequences.  The two information streams are combined using a channelwise encoder architecture, providing input to an autoregressive network that predicts the EMG sequence. \n\nI is convincingly demonstrated that the architecture, in contrast to simpler benchmark architectures, results in predictions that show substantial similarities with ground truth EMG data. In addition, it is demonstrated that the addition of the keystroke sequences as inputs again improves the accuracy of the predictions. Further validation holding out data from  individual users, or of parts of the piano movement tasks, demonstrates that the architecture is able (to some degree) to generalize to new users and tasks.  Specifically, it is shown that addition of a small amount of user-specific data improves the prediction accuracy for new users massively, which is important for the realization of accurate individually-specific predictions in different applications.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is very well written and methods and evaluation is very solid. The paper presents one of the fist solutions for the problem of accurate prediction of EMG data from multimodal data, which is likely an important field for future research given the difficulties associated with EMG recordings. In addition, the paper presents a nice multimodal data sets including EMG, video data and keystroke data, which is very useful for benchmarking similar architectures. The main points that the paper makes, that EMG signals can be predicted with a decent accuracy for multimodal data is made convincingly. \n\nThe valiation of similarity of real EMG data and predicted data remains somewhat qualitative, and the shown examples are convinving, but raise the question what the point and criteria are, that would define a satisfying prediction of EMG data form multimodal inputs. The paper discusses, in my view in a nice and satisfying way the difficulties associated with this question (interindividual diferences, etc.).\n\nAnother limitation, which also is adequately discussed, is the constrained set of movements (piano, only 7 sequences), where it still seems to require substantial work to extend the approach to less constrained applications. \n\nA minor point is: \nA lot of details of the algorithms are described in the supplement, while there was a relatively detailed description, how tracking errors of the pose data are interpolated. This made it  somewhat difficult to grasp the main components of the core neural network architecture in the main text. Maybe the manuscript would gain if some more critical details of the main architecture would be given in the main text, and the interpolation of the pose data would be shifted in the supplement, which seems more like a technical detail.\n\nQUESTIONS:\nWhat would be main obstacles to generalize the method to more piano tasks, and potentially larger classes of hand movements?\n\nTable 3 and results for held out evaluation: Here  the table certain values (.209 etc) are given, but it remains unclear whether these values are good or bad. One sentence in the interpreting section would help. \n\nThere is some older work trying to predict facial EMG from video seuences (e.g. Boccignone, Giuseppe & Cuculo, Vittorio & Grossi, Giuliano & Lanzarotti, Raffaella & Migliaccio, Raffaella. (2017). Virtual EMG via facial video analysis.  9th International Conference on Image Analysis and Processing). Maybe it would be suitable to quote such related work in the domain of facial movements and facial action units?\n\nL. 219 ff.: The description of the time encode archietcture is somewhat hard to follow, unless one digs it up in the supplement.. Specifically, it seems not to be clear what these variables C, c and w indicate.  Likewise for D_f.\n\nMinor questions: \nAbstract: what is the maining of 'prohibitive devices', and 'gross motors' ?\n\nFig 1 (2): Evaluationg -> Evaluation (?)\n\nLIMITATIONS:\nLimitations of the approach are thoroughly discussed, also also in the supplement.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of estimating hand muscle activity (electromyography, EMG) from external, easily accessible modalities, specifically in the context of piano performance. While prior work has primarily focused on inferring motion from EMG signals, this study explores the inverse problem: inferring fine-grained EMG signals from hand postures and keystroke dynamics.\n\nTo this end, the authors first examine the feasibility of pose-to-EMG inference by pretraining on a general-purpose dataset (emg2pose) and demonstrate the limitations of transferring such models to piano-specific tasks. Motivated by this gap, they introduce the PianoKPM Dataset—a large-scale, high-fidelity, multimodal dataset containing synchronized recordings of hand EMG, keystrokes, pose, and video from 20 expert pianists.\n\nBuilding on this dataset, the authors propose PianoKPM Net, a neural network designed to infer EMG signals by fusing hand pose and keystroke data. The network integrates pretrained priors, multi-branch encoders, and a time-channel-wise architecture with a hybrid loss function to capture both precise and structured EMG dynamics.\n\nThrough comparisons with baseline models and ablation studies, the paper claim that PianoKPM Net achieves superior EMG prediction accuracy. The model also shows promising generalization to unseen users and tasks, and can adapt with limited new data. The authors conclude that their framework sets a foundation for cost-effective EMG estimation and has implications for skill analysis, human-computer interaction, and embodied AI.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n- This work addresses an underexplored problem: estimating fine-grained hand muscle activity from observable modalities in the context of a highly skilled and dexterous task (piano playing).\n- The proposed PianoKPM Net is carefully constructed and includes thoughtful architectural design choices such as a multi-branch encoder, latent prior injection, and a hybrid loss combining MSE and Optimal Transport.\n- The integration of keystroke data with hand pose for EMG estimation represents a new multimodal approach.\n\n# Weaknesses\n- While the paper convincingly highlights the technical challenge of pose-to-EMG inference, it falls short in articulating why this capability is important for downstream tasks. Without a clearer connection to downstream benefits, it is difficult to fully appreciate the practical significance of high-fidelity EMG estimation in the piano domain or beyond.\n- While the paper reports competitive performance, the improvements over baselines in some metrics (e.g., RMSE) appear modest and may benefit from deeper statistical analysis or error case exploration.\n- While the dataset and model are tailored for piano performance, their generalizability to other fine motor tasks (e.g., typing, surgery) is not empirically explored.\n\nQUESTIONS:\n1. Have you explored or considered the generalizability of PianoKPM Net to other fine-motor tasks (e.g., typing, handwriting, surgical suturing)? If possible, provide either empirical results or discussion on how transferable the learned representations are to different but related tasks.\n2. How well does PianoKPM Net perform with less curated or noisier input data—for example, RGB-only 2D keypoints without multi-view fusion or in-the-wild keystroke data with latency?\n3. How critical is the Optimal Transport (OT) component in your hybrid loss for overall performance? Could you include an ablation study comparing models trained with MSE alone versus the hybrid loss?\n4. What are the envisioned downstream applications that would directly benefit from the inferred EMG signals? Please elaborate on how EMG estimation could enable new capabilities or improve existing pipelines.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenging task of estimating hand muscle electromyography (EMG) signals from more accessible modalities during piano performance. The authors introduce **PianoKPM-Net**, a multimodal learning framework designed to infer 6-channel hand muscle EMG signals from hand pose and keystroke information. In addition, they present the **PianoKPM Dataset**, a high-quality, synchronized dataset comprising EMG, pose, keystroke, and audio recordings from 20 expert pianists performing 7 distinct musical tasks. The proposed approach achieves promising results in EMG inference and is supported by comprehensive evaluations on both cross-user and cross-task generalization.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n1. **Novel and Well-Motivated Problem**  \n   The paper tackles the practical challenge of high-cost EMG sensing by exploring a pose-to-EMG inference approach. The initial analysis highlighting the poor transferability of general EMG datasets to piano-specific contexts provides a strong motivation for the development of domain-specific solutions.\n\n2. **Comprehensive Dataset Contribution**  \n   The authors introduce the PianoKPM dataset, offering 12.64 hours of high-quality, synchronized multimodal recordings (2000 Hz EMG, 1000 Hz keystrokes, 60 FPS video) from 20 expert pianists. The dataset is collected with proper IRB approval and is accompanied by detailed preprocessing procedures, making it a valuable resource for future research.\n\n3. **Thorough Experimental Evaluation**  \n   The paper presents comprehensive experiments demonstrating that the proposed model achieves promising performance in EMG inference. Evaluations cover both cross-user and cross-task settings, offering insights into the generalization capabilities of the model.\n\n4. **Insightful Ablations and Visualizations**  \n   The authors provide extensive ablation studies examining the effects of model components (e.g., branch configurations, temporal modules) and input modalities. Qualitative visualizations further illustrate the model’s ability to capture meaningful muscle activation dynamics.\n\n---\n\n### Weaknesses\n\n1. **Limited Architectural Novelty**  \n   While the proposed framework is effective, the multi-branch encoder architecture is relatively standard, and the paper does not introduce any fundamentally new architectural designs.\n\n2. **Missing Direct Mapping Baseline**  \n   The paper lacks a direct pose/keystroke-to-EMG mapping baseline, which would help isolate the benefits of the proposed staged design. Additionally, ablation studies exploring different encoder-decoder combinations are missing and would provide a clearer understanding of architectural trade-offs.\n\n3. **Lack of Runtime Analysis**  \n   Although the model consists of multiple components, there is no discussion of computational cost or inference time. This limits the assessment of its applicability in real-time or interactive scenarios.\n\nQUESTIONS:\nPlease refer to the **Strengths And Weaknesses**.\n\nLIMITATIONS:\nPlease refer to the **Strengths And Weaknesses**.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 2, "DATE": "2025-07-02"}