Posture-Informed Muscular Force Learning for Robust Hand Pressure Estimation

Authors: kyungjin  seo
junghoon  seo
hanseok  jeong
sangpil  kim
sang ho yoon
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 03:17:23

Abstract: We present PiMForce, a novel framework that enhances hand pressure estimation by leveraging 3D hand posture information to augment forearm surface electromyography (sEMG) signals. Our approach utilizes detailed spatial information from 3D hand poses in conjunction with dynamic muscle activity from sEMG to enable accurate and robust whole-hand pressure measurements under diverse hand-object interactions. We also developed a multimodal data collection system that combines a pressure glove, an sEMG armband, and a markerless finger-tracking module. We created a comprehensive dataset from 21 participants, capturing synchronized data of hand posture, sEMG signals, and exerted hand pressure across various hand postures and hand-object interaction scenarios using our collection system. Our framework enables precise hand pressure estimation in complex and natural interaction scenarios. Our approach substantially mitigates the limitations of traditional sEMG-based or vision-based methods by integrating 3D hand posture information with sEMG signals.
Video demos, data, and code are available online.

Main paper: 
 introduction: 
 Hands are a central tool for humans to interact with the surrounding environment. With the advancement in hand tracking technology, hand inputs, including position, orientation, gesture, and motion, are increasingly used as a primary means of control, especially for emerging interfaces (e.g., augmented/virtual reality and wearables). Using hands as the main interaction medium offers a high level of versatility and flexibility to achieve natural and intuitive interactions.Recent studies have started to utilize hand pressure information to support hand-based interactions such as touching [1] , grasping [2, 3] , and pressing [4] . Researchers also utilized hand pressure to provide effective haptic feedback [5] for a more immersive user experience. Furthermore, precise hand pressure measurement becomes essential for real-world applications, including ergonomic evaluation [6] , hand rehabilitation [7] , and prosthetic hand control [8] . To this end, previous works focus on obtaining real-time and accurate hand pressure information with direct measurement approaches utilizing gloves [9] [10] [11] or load cells [12] . However, these approaches require users to be in physical contact by either wearing or holding the device, which hinders natural hand movements or reduces user comfort. Thus, the necessity of direct contact limits the users from performing hand-based interactions in a natural and unrestricted manner.To this end, non-invasive sensing techniques to estimate exerted hand pressure without embedding sensors on the user's hand have been highlighted. These methods include profiling wrist topography with capacitive sensing [13] , multiple pressure sensing from the wrist [14] , and electromyography Figure 1 : Our sensing framework (PiMForce) leverages 3D hand posture information along with sEMG data to enable a whole-hand pressure estimation during various hand-object interactions. We support real-time pressure estimation on the fingertips and palm regions based on RGB image and sEMG inputs. The intensity of each node's color indicates the pressure level. from the forearm [2] . Recent works utilized only a single RGB-D [15] or RGB camera [16, 17] with computer vision techniques to estimate pressure exerted by the hand. However, previous methods had limitations, where they could only estimate hand pressure when interacting with plane surfaces or required a line-of-sight view of the hand. With these limitations, it is hard to support natural interaction contexts like working with diverse hand grasps.In this work, we introduce PiMForce, a novel multimodal sensing framework that enhances real-time hand pressure estimation by leveraging 3D joint information of the hand and forearm sEMG signals, covering the fingertip to the entire palm. As illustrated in Figure 1 , our framework addresses previous challenges by integrating detailed spatial information from 3D hand poses with dynamic muscle activity from sEMG measurements. This multimodal sensing integration allows us to estimate subtle and comprehensive hand pressure even under diverse grasps. To validate the proposed model, we built a multimodal hand data collection system and created a dataset from 21 participants. We believe our dataset is the first of its kind containing multimodal sensor signals during hand interactions under various grasps. We demonstrated our work on an off-the-shelf system with a single camera to confirm the accuracy and feasibility of the proposed framework. Our contributions are listed as follows:• We propose PiMForce, a novel hand pressure estimation framework that enhances sEMG signals by incorporating hand posture information.• We develop a multimodal hand data collection system with a data collection protocol and create a unique dataset containing simultaneous hand pressure, hand posture, and surface electromyography signals.• Evaluation and analysis of experiments demonstrate the improved performance of our approach, showing its consistent superiority over existing sEMG-based and vision-based methods.2 Related WorksQuestion: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: Section 1 includes the main claims.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
vision-based hand pressure estimation: 
 Previous works explored the interaction between the hand and objects by observing the movement and rotation of the object over time to estimate the pressure exerted by hands. By determining the pressure required to produce these observed changes, the model estimated the aggregate pressure applied by the hand [18] [19] [20] . These approaches enabled the pressure estimation to act upon concealed or non-visible areas where direct visibility of the hand in contact with an object is absent. Still, previous approaches had limitations where interacting with immovable objects would not work due to the absence of dynamic interaction indicators.Researchers also looked into different visual indicators like color changes in fingertips, which represent fluctuation of blood circulation within the fingertips [21, 22] or compression of skin tissues [23, 24] . These physiological behaviors served as indicators of the exerted hand pressure. Moreover, examining shadows cast during hand-object interaction provided further insight into the spatial relationship and dynamics of force between them [25] [26] [27] .Recent works further advanced the existing visual indicator approach where they use a hand image captured by a single camera at a distance to estimate the hand pressure [16, 28, 17, 29] . They employed a deep learning model that facilitates the understanding of visual cues to estimate accurate Table 1 : Comparison with the previous datasets for hand contact and pressure estimation. We modified and updated the table from [17] . NA refers to 'not available'. hand pressure in an end-to-end fashion. However, previous works require a high-quality whole-hand image without occlusion since they rely on visual indicators for the estimation. In our work, we utilize multimodal inputs, including vision-driven 3D hand posture information and wearable-based muscle activation signals, to enhance the estimation of robust hand pressure under various hand-object interaction contexts.Researchers have used forearm/wrist surface electromyography (sEMG) sensors to acquire finger muscle activation information. Here, the sensor captured a train of neuron impulses propagated through the arms from the forearm or wrist [30, 31] . Previously, researchers used sEMG sensors to estimate various types of hand-related force/pressure, including force/pressure from gripping [32] [33] [34] [35] [36] and fingertip [4, [37] [38] [39] [40] [41] . Recent works also enabled the estimation of hand pressure along with hand gesture recognition using sEMG signals [8, 42] . However, previous works only dealt with a limited set of discrete hand poses [43] . Moreover, an issue existed with using sEMG signals for complex hand interactions where similar muscle activation signal behaviors were observed across different hand poses. This could easily confuse the model and lead to false behavior. In this work, we train the model with 3D hand posture to encode distinctive hand pose information alongside sEMG signals.This integration forms a robust and accurate hand pressure estimation framework for similar muscle activation behaviors but different hand poses. It is worth emphasizing that this study is the first to incorporate hand posture information for hand pressure estimation using forearm-worn sEMG.Previous studies primarily focused on estimating pressure at the fingertip or on a single gripping force, but our approach expands this to encompass the whole hand.In the computer vision and machine learning community, researchers have formed various types of hand-object interaction datasets for hand pressure estimation. These vision-based datasets collected rich visual and pose information for hand-object interactions, capturing everything from object affordances to whole-body grasps [44, 28, 17, [45] [46] [47] . On the other hand, sEMG-based datasets have also been proposed and used to estimate hand pressure for AR/VR or prosthetic robotic arm control applications [48, 4] . A highly relevant work is the ActionSense dataset [49] , which focuses on capturing multimodal data of human activities in a kitchen environment using wearable sensors. However, while ActionSense provides a valuable resource for understanding general kitchen activities, our work focuses on the utilization of 3D hand posture in muscular force learning for understanding hand pressure estimation. Furthermore, the temporal resolution of EMG data and the spatial resolution of hand pressure in ActionSense are substantially lower compared to ours, making it challenging to utilize rich sensor input and output effectively. Still, the dataset containing both rich visual and physiological information is missing.In this work, we attempt to set up a new multimodal dataset that contains 3D hand pose information, sEMG signals, and ground truth measurement of hand pressure as shown in Table 1 . Our work provides a holistic view of whole-hand dynamics during various hand-object interactions. This integration enables continuous and comprehensive pressure estimation across the whole palm, addressing the limitations of previous datasets that either infer pressure from visual cues or measure it in isolation.Our dataset not only captures the subtle interplay between visual and tactile information, but also increases the potential candidates for input features for accurate and robust hand pressure estimation.3 Building Multimodal Dataset: Posture, Electromyography, and PressureTo capture multimodal data with various hand-object interactions, we integrated and customized existing hardware, including a pressure glove, an armband with 8-channel sEMG sensors, and a markerless finger tracking module. Figure 7 in supplementary material showcases our data collection setup to capture real-time and synchronous multimodal data, including 3D hand posture, sEMG signals, and exerted hand pressure. More detailed information about the hardware, defined hand postures, data collection protocol, and data processing can be found in Section B.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: Section E includes the limitations of the work.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when the image resolution is low, or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
data collection setup: 
 Pressure Glove. To capture pressure exerted from the hand, we developed a customized pressure glove using a single 65-node pressure sensing glove (TactileGlove, Pressure Profile Systems) attached with a pressure sensor (RA18DIY, Marveldex) at each fingertip. We added flexible sensors to the fingertips to address missed readings when the pressures were exerted on the edge of the fingertip.Our pressure glove supports pressure readings up to 55 N/cm 2 with a sampling rate of 40 Hz.Pose. Recent common approaches to obtaining ground truth 3D hand pose [51] involve using multiple RGB cameras to derive 2D hand poses from each camera, followed by triangulation [52, 53] or hand template fitting [44, 45, 54] . However, these methods are infeasible when a pressure data glove is worn, as the glove obscures the hand, hindering accurate hand pose estimation from RGB images. To acquire accurate 3D hand pose information under this constraint, we employed a magnetic sensing-based markerless finger tracking module (Quantum Mocap Metaglove, Manus).The module provides each finger's 3D position and 3-axis joint angles with a sample rate of 120 Hz and less than 5-millisecond latency. We attached the finger-tracking module to the pressure glove to capture exerted hand pressure and 3D hand pose data simultaneously.8-Channel sEMG Armband. We used 8 sEMG sensors (Trigno Avanti, Delsys) and installed sensors into a customized armband made with semi-flexible material (TPU 95A) to ensure electrode contact for various sizes of forearms. Our system captures muscle action potentials with a sampling rate of 2,000 Hz.Multimodal Data Synchronization. Before training the multimodal dataset, we employed a linear interpolation approach to synchronize high frame rate readings (sEMG) with low frame rate data (3D hand pose). We first joined the data from matched time points based on the collection time of the sEMG data and interpolated missing values of the hand pose data linearly. We applied the same approach to the pressure glove, where we synchronized high frame rate sensors (pressure sensors) with the low frame rate sensing glove (TactileGlove). Then, we adopted a nearest-neighbor-based interpolation to synchronize 3D hand pose and sEMG data with hand pressure [4] .With IRB approval, a total of 21 right-handed participants took part in this study. Of these, 17 were male (81%) and 4 were female (19%). The participants' ages ranged from 20 to 32 years, with a mean age of 24.3 years (SD = 3.9). To ensure good quality hand pressure data using our glove, we chose participants with hand sizes greater than 180 mm. Prior to participation, all participants were provided with a detailed information sheet outlining the purpose, procedures, and potential risks of the study. We obtained written informed consent from each participant, ensuring they understood the nature of the data being collected, their right to withdraw at any time, and the measures taken to ensure data privacy. We equipped participants with our multimodal glove and an 8-channel sEMG armband. Following initial calibration to compensate for each user's hand size, participants performed 22 distinctive hand-object interactions for the data collection task (Figure 8 in supplementary material).Our hand-object interactions consist of 7 hand-plane interactions, 5 pinch interactions, and 10 distinctive hand grasps. We included the same hand-plane and pinch interactions from recent hand pressure estimation work [4] while adding palm-pressing motion. In terms of hand grasps selection, Figure 2 : Our multimodal hand pressure estimation architecture enhances sEMG data by embedding 3D hand pose information. We train the model using a classification-regression joint loss to improve hand pressure estimation.we selected 10 grasps from representative 33 grasp types [55] based on the clustering of palm pressure distribution similarity among grasp taxonomies [56] . We collected 1,980 seconds of synchronized multimodal data per participant (30 seconds × 22 hand-object interactions × 3 sessions). Here, we used data from 2 sessions for training while the remaining session was reserved for evaluation. We provided sufficient rest periods between each trial to prevent muscle fatigue, ensuring the quality of collected data. All collected data was anonymized, complying with relevant data privacy regulations.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
overview: 
 The rationale behind using sEMG to estimate the pressure exerted by the hand lies in the direct relationship between muscle electrical activity and the pressure generated during hand interactions [57] [58] [59] . When muscles contract for movement, they generate bioelectric signals that can be captured by EMG sensors [60] . This indicates that sEMG signals have the potential to estimate the pressure exerted by the hand. The ability to decode sEMG signals from forearm muscles generated by fingerlevel movements will set a robust foundation for understanding complex hand interactions. However, the pressure exerted by the hand cannot be solely represented with muscle activation information.The main reason is that the distribution of hand pressure varies according to different hand postures. For example, similar sEMG patterns may be generated by different hand pressures depending on the related hand postures or grasps [43, 61] . This highlights the importance of considering hand posture information along with sEMG signals to estimate the exerted hand pressure precisely. Section C.1 in supplementary material addresses the specific empirical observation for this motivation.To address these issues, we enhance sEMG signals by leveraging 3D hand posture information. By integrating inputs from forearm-worn sEMG sensors with 3D hand pose information derived from an RGB image, we observe improvements in the accuracy of hand pressure estimation. Our multimodal approach (Figure 2 ) leverages the strengths of both hand posture and muscle activations, offering a comprehensive understanding of hand dynamics for whole-hand pressure estimation. Refer to Section C for more detailed information about the model architecture, training, and inference.To verify the validity of our framework, we devise a deep neural network model to effectively utilize the obtained multi-modalities. We represent the model as f , where the 3D hand pose is denoted by H and the sEMG signal by E. Feature Extraction from 3D Hand Pose. The hand model in our study is represented as a kinematic tree with 15 joint angles θ ∈ R 15×3 , similar to the pose parameters in MANO and its variants [62] [63] [64] [65] . To handle this skeleton-based representation simply like PoseConv3D [66] , we adopt a 3D ResNet [67] to process 3D heatmap volumes of hand joints, transforming θ into 21 3D hand joints J ∈ R 21×3 through the hand skeleton model's forward kinematics. These 3D hand joints are then converted into 3D heatmap volumes H ∈ [0, 1] 21×H×W ×D , with H, W , and D all set to 48, for processing by the 3D ResNet. Unlike the sequential representation of 2D heatmaps in PoseConv3D, our approach uses a single timestep 3D joint representation. After processing through the 3D ResNet, the hand pose feature is flattened and transformed into a 512-dimensional vector using an FC layer, with the 3D ResNet34 model being the model of choice for this operation.Feature Fusion and Estimation. The extracted sEMG feature f EMG (E) and hand pose feature f hand (H) are concatenated to form a 1024-dimensional joint feature vector. This vector is then passed through two FC layers, mapping to 256-dimensional features, followed by a 1-D batch normalization and ReLU non-linearity. Finally, the last FC layer with Sigmoid activation function maps this toI-dimensional output, producing Ĉ = f (E, H) = f pred (f EMG (E), f hand (H)) ∈ [0, 1].The predicted pressure P is defined as P = 2P max • ReLU( Ĉ -0.5) ∈ [0, P max ], thus completing the feature fusion and pressure prediction.The objective function comprises two key components: a classification loss L c and a regression loss L r . The classification loss is designed to accurately identify whether any pressure is exerted by a particular region of the hand (i.e., fingertips or palm areas shown in Figure 5 of Section B.1.1), using a cross-entropy loss to distinguish between pressure and no-pressure instances for each hand region i:EQUATIONwhere C i is the ground-truth label for region i, and Ĉi is the predicted probability of pressure application. Here, C i = 1 indicates the presence of pressure in the ith region, while C i = 0 indicates its absence. In contrast, the regression loss, targets the accurate quantification of pressure levels using an L 2 loss to minimize the difference between the predicted and actual pressure values:EQUATIONwhere Pi represents the model's predicted pressure for region i and P i is the corresponding actual pressure. Our dataset contains pressures from 0∼20 N. Therefore, our model predicts pressure values in [0, 20], organized by Pmax , the maximum of the pressure. To integrate these two aspects into a unified training objective, we introduce a balancing hyper-parameter λ, resulting in a combined loss function:L = L c + λ • L r .This composite loss enables our model to not only discern the presence of pressure but also quantify its magnitude accurately.For the training phase, we employed data acquired from our data collection system to ensure the accurate capture of exerted hand pressure and 3D hand pose data. However, during the inference phase, our framework exploits off-the-shelf hand pose detectors [68] [69] [70] , which extract 3D hand pose from RGB or RGB+D inputs. These detectors can been chosen for their high accuracy and robustness in various conditions, ensuring reliable performance during inference. Thus, users can interact with external objects using their bare hands, without the need for additional hand-worn equipment. This approach ensures our model's practical applicability in real-world scenarios, prioritizing user convenience and natural interaction. By leveraging readily available technology, we make it easier for users to adopt our system in everyday applications. Refer to Section B.4 for details on how we canonicalized 3D hand pose information extracted from RGB images for our model input. 
experiments: 
 In Sections 5.2.1 and 5.2.2, where ground truth hand pressure is necessary, data was collected while participants wore the pressure glove, and hand postures were obtained from the data glove. We also conducted qualitative evaluations (Section 5.2.3 and the demo video) without ground truth, where data was collected without any gloves, and hand postures were inferred solely from RGB images using an off-the-shelf hand pose detector. For this purpose, we employed the pre-trained Attention Collaboration-based Regressor [69] , which has demonstrated superior performance with a mean per joint position error (MPJPE) of approximately 8mm for reconstructing hand poses from a single RGB camera. The high accuracy ensures the reliability of our hand posture inferences in qualitative assessments. To assess our model's performance, we utilize three metrics: Coefficient of Determination (R 2 ), Normalized Root Mean Squared Error (NRMSE), and classification accuracy.The exact definitions and explanations of evaluation metrics can be found in Section D.2. Refer to Section D.3 and D.4 for additional quantitative and qualitative restuls, respectively.This study compares the proposed model against several baseline and state-of-the-art methods to validate its effectiveness in whole-hand pressure estimation. To ensure a fair comparison, we selected methods that quantitatively measure the pressure applied by the hand, rather than solely identifying hand contact. Detailed information about the implementation of comparative methods can be found in Section D.1. The methods included in the comparison are:sEMG Only Model [4] . An sEMG-based approach decodes finger-wise forces in real-time, demonstrating the potential of muscle activation patterns in informing hand activities. This method emphasizes using electromyography sensors for understanding complex hand dynamics but does not incorporate hand posture information.3D Hand Posture Only Model. A variation of our proposed framework that solely utilizes 3D hand posture for pressure estimation, omitting the sEMG signal input. This model tests the efficacy of hand posture information in isolation.sEMG + Hand Angles Model. This model represents a variation of our proposed framework, where instead of utilizing the 3D representation H for hand pose, it employs the angular representation θ of hand joints as the input to the hand pose feature extractor f hand . By substituting the 3D hand pose with direct angle measurements of hand joints, this baseline aims to highlight the benefits of using a 3D representation for hand pose in multimodal sensing.PressureVision++ [17] . This vision-based deep learning model estimates hand pressure from a single RGB image by identifying visual cues related to hand pressure application, showcasing the use of visual information for pressure estimation without physical FSR sensors.PiMForce (Ours). The comprehensive model enhances sEMG signals by leveraging 3D hand posture information for continuous and detailed pressure estimation across the whole hand. This approach aims to mitigate the limitations of sEMG-based methods by integrating the strengths of both modalities for enhanced pressure prediction accuracy. We analyze the performance of our proposed framework in comparison to these methodologies, both quantitatively and qualitatively. Additionally, we investigate the capabilities of our model to accurately estimate hand pressure across a variety of hand postures and parts, providing a thorough assessment of its performance. Our framework enhances sEMG signals by leveraging 3D hand posture information for detailed palm pressure data collection, contrasting with PressureVision++, which relies on visual cues for force estimation. This approach is designed to underscore the distinctive benefits of our multimodal sensing framework in capturing a broad range of hand interactions. The integration of 3D hand posture and sEMG information in our framework shows a clear advantage over approaches relying on a single data modality, as expected. The sEMG Only Model and the 3D Hand Posture Only Model show limited pressure estimation performance when compared to our integrated approach. Interestingly, the improvement in performance with the sEMG + Hand Angles model over the sEMG Only Model is marginal (less than 0.5%p improvements in all metrics). This highlights the importance of incorporating a comprehensive 3D hand posture representation. By embedding comprehensive hand posture knowledge to be used with sEMG data, we develop an effective multimodal approach to capture nuanced variations in hand pressure exerted across different hand regions and postures.Cross-user performance assesses how well the model performs on data from individuals not included in the training set, which is crucial for real-world applications. As shown in Table 3 , our proposed method combining sEMG signals with 3D hand posture data significantly outperforms the sEMGonly baseline across all evaluation metrics in cross-user scenarios. This demonstrates the enhanced generalizability and effectiveness of our approach in estimating hand pressure among different users. To further demonstrate the performance of our model over time, we present Figure 23 in supplementary material, which illustrates the temporal evolution of both ground truth and predicted pressure values for all nine hand regions during consecutive TM-Press and Medium Wrap actions.We delve into the performance of our model across various hand regions and posture types, utilizing data represented in both Table 4 and Figure 3 . This analysis highlights the noticeable impact of incorporating 3D hand pose data, particularly noting a greater improvement in hand palm regions (+1.95%p) over fingertips (+1.05%p) compared to the sEMG Only Model. This distinction emphasizes the crucial role of 3D hand pose for accurate pressure estimation in diverse hand postures.Our findings reveal that the model achieves superior pressure estimation in Press and Pinch interactions, with classification accuracies surpassing 90% and NRMSE values maintained below 6%. However, it encounters challenges with specific postures such as Palm-Press, which, despite a lower classification accuracy of 68.42%, still shows a high regression accuracy of 3.12%. When examining Grasp postures, our model shows a slight dip in performance relative to Press and Pinch, with NRMSE values ranging between 5∼8%. This suggests a moderate pressure estimation capability for these more complex interactions, yet the model consistently maintains a high R 2 range of 0.8 to 0.9 across all posture types. This consistent correlation between predicted values and actual pressure measurements highlights the model's ability to maintain high accuracy and reliability across a diverse On the contrary, more complex grasps like Power Sphere, Fixed Hook, and Parallel Extension, showed lower accuracy and higher NRMSE, implying that there remains room for improvement.For practical inference applications without a data glove, solely relying on EMG data and incorporating 3D hand pose information obtained through an off-the-shelf hand pose detector, we demonstrate the adaptability of our model in Figure 4a . This comparison with the vision-based method, Pressure-Vision++, showcases our PiMForce's capability to estimate hand pressures robustly during diverse interactions. During hand-plane interactions, specifically those involving the tip Press type posture, both approaches appear to perform well. However, our analysis reveals vulnerabilities in handling more complex Grasp and Pinch motions when using PressureVision++. Furthermore, PressureVi-sion++ requires complete visibility of all fingers within the camera's view due to the high reliance on visual cues for pressure inference. In contrast, our framework effectively utilizes the estimated hand pose as long as the hand pose information is sufficiently accurate for inference. This capability underscores the practicality of our method, facilitating more natural user interactions with external objects without the constraints of direct visibility or glove use. Figure 4b shows demo video footage illustrating our PiMForce's capability to accurately estimate hand pressure while continuously changing hand posture, pressure levels, and the objects being grasped. This demonstrates the flexibility and reliability of our approach in real-world scenarios.To further substantiate our model's effectiveness using an off-the-shelf hand pose detector, we conducted a quantitative comparison with PressureVision++, as presented in Table 5 . PiMForce demonstrates largely better performance across all fingertips during both plane and pinch interactions, indicating superior performance in estimating hand pressures compared to PressureVision++. This quantitative evaluation confirms that our framework outperforms existing vision-based methods in terms of accuracy and robustness during diverse interactions.  
conclusion: 
 In this paper, we introduce PiMForce, a pioneering framework for hand pressure estimation by integrating 3D hand posture information with muscle activation signals from forearm-worn sEMG.By embedding 3D hand posture information into a deep neural network, we enable the model to process this data alongside sEMG signals, enhancing its capability to learn complex relationships between muscle activations and hand pressure distributions. This novel approach is the first to combine these modalities, providing a comprehensive analysis of hand dynamics across various interactions. We developed a unique multimodal hand data collection system and protocol, capturing a dataset that includes hand pressure, posture, and electromyography signals. Our method notably improves upon previous techniques, enabling accurate whole-hand pressure estimation through detailed hand posture information. Extensive quantitative and qualitative comparisons demonstrated the consistent superiority of our framework over existing sEMG-based and vision-based methods. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: Table 2 , which is the main quantitative result, includes error bars. Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. 
 
 Appendix: This supplementary material provides added details to complement the main manuscript. Section B elaborates on the specifics of the multimodal dataset creation. Section C offers an extended description of the methodological framework employed in our study. Section D describes implementation details of comparative methods and showcases additional results to further support the findings reported in the main text. Lastly, we note limitations and future works in Section E.B Detailed Description for the Multimodal Dataset To effectively capture comprehensive data on hand pressure and posture, we modified a tactile glove equipped with 65 pressure sensors by integrating additional fingertip pressure sensors along with a position tracking module, as shown in Figure 5 . We customized the design of the glove to focus on the hand pressure measurements across 5 fingertips and 4 palm regions consisting of 16 pressure sensor nodes.To determine the pressure in each region, we use the maximum value among the sensors within that region rather than summing or averaging their readings. This approach accounts for variations in hand size, ensuring that the pressure measurement is not artificially lowered due to inactive sensors that may not be engaged by all participants.For fingertip pressure measurements, we utilized a Force-Sensing Resistor (FSR) type pressure sensor (RA18DIY, Marveldex), characterized by a force range of 0 ∼ 40 N/cm 2 and a thin profile of 0.7 mm with an 8 mm diameter sensing area. Figure 6 presents the typical response behavior of the FSR, highlighting the force-resistance relationship. In our work, we derived a precise fitting model for the sensor's output through repetitive calibrations using a push-pull gauge. The calibration process involved conducting 30 load-unload cycles (ranging from 0 ∼ 30 N, incremented by 1 N) and meticulously recording the resistance values to ensure accuracy.For the grasp postures, we carefully selected 10 representative movements from a comprehensive grasp taxonomy presented in recent literature [55] . This taxonomy classifies grasps based on several factors, including opposition type, virtual fingers, grip type, and thumb position, culminating in Using these data collection tools, we gathered data from each participant for the 22 postures listed in Table 6 , repeating each posture a total of three times. Each posture was collected for a duration of 30 seconds, with participants instructed to repeat the given posture 12 to 15 times. For Plane-type postures, participants were instructed to apply and then release force against a flat surface. For Pinch and Grasp postures, they were directed to fix an object with their left hand and then apply and release force with their right hand. To prevent the accumulation of fatigue in the muscles due to EMG, we incorporated a 10-second rest period after each posture, and after completing all 22 postures, participants were given a 10-minute break to ensure adequate rest.Muscle fatigue, characterized by a decline in a muscle's ability to generate force, occurs when a muscle is repeatedly contracted or held in a sustained contraction over an extended period. As muscles fatigue, their electrical activity changes, manifesting as increased signal amplitude (root mean square) and shifted frequency content [71] . These alterations can negatively impact the quality of sEMG data With a frame length of 1,248 samples, corresponding to 0.624 seconds, we utilized a Short Time Fourier Transform (STFT) for sEMG signal. This method, leveraging a window length of 256 and a hop length of 32 samples, employed a Hamming window function to minimize spectral leakage. To preprocess the sEMG signal, we applied the STFT and considered only the signals below 64Hz to eliminate high-frequency noise, optimizing the signal's relevance for muscle activity analysis [4] . For the pressure data, considering the conventional force that a human can comfortably apply with their hand, we propose a maximum inferable force of 20 N, and forces exceeding 20 N were clipped to be treated as 20 N. Additionally, to exclude noise data caused by subtle sensor presses, values below 0.2 N were processed to be treated as 0. For the 3D hand pose data, we use data obtained from the Quantum Mocap Metaglove for the training dataset. The hand skeleton model adopted in this study is as shown in Figure 11 . The joints of the fingers from the middle to the ring finger are omitted because they are defined identically to those of the index finger. The raw data obtained through the data glove represents the hand as 20-dimensional angular data, denoting each finger with one abduction angle and three flexion angles [72] . For the thumb, it includes the CMC (carpometacarpal joint)'s abduction and flexion angles, along with the MCP (metacarpophalangeal joint) and IP (interphalangeal joint) flexion angles. For the other fingers, it encompasses the MCP's abduction and flexion angles, as well as the PIP (proximal interphalangeal joint) and DIP (distal interphalangeal joint) flexion angles. This 20-dimensional angular data is transformed into a three-dimensional angular representation θ for each of the three joints determining the state of each finger, considering the kinematic tree of the hand skeleton in the MANO hand model [73] . Subsequently, through forward kinematics [74] , this is converted into a hand joint representation J that includes the fingertips and the hand root. The relative bone lengths of each hand skeleton used during forward kinematics are shown in Table 7 . When converting hand joint representation J into 3D heatmap volumes H, J is first min-max linearly scaled to take values in the range [12, 36] , considering the maximum and minimum values for each spatial dimension in the training dataset. Thereafter, J is transformed into 21 3D heatmaps H by converting it into a 3D Gaussian heatmap with a standard deviation of σ = 1 for each dimension.When using a 3D hand pose inferred from vision data, it can be transformed into the same representation as the 3D joint representation J. However, considering the global rotation of the wrist in the photo, it is necessary to align the direction of the global rotation of the hand with the kinematic tree of the hand used in training, and the scale of the hand also needs to match that used during training. Accordingly, the 3D hand pose J inferred from vision data was rescaled so that the distance from the root joint to the MCP of the middle finger is identical, rotated so that the plane formed by the three joints (root joint, MCP of the middle finger, MCP of the index finger) is consistent, and translated so that the root joint is in the same position. Afterward, the 3D hand pose inferred from vision data is transformed into 21 3D heatmaps H through the process described above.To empirically motivate our approach, we observed that sEMG signals alone might not be sufficient for fine-grained pressure localization on the hand. We conducted an empirical analysis demonstrating the limitations of using sEMG signals alone for this purpose. Figure 12 presents detailed sEMG signal patterns for two pairs of hand postures: (1) I-Press (index finger press) versus M-Press (middle finger press), and (2) TI-Pinch (thumb-index pinch) versus TM-Pinch (thumb-middle pinch). In each case, participants were instructed to apply pressure using different fingers, targeting specific regions of the hand. The 8-channel sEMG signals were recorded over a 5-second interval, with each channel representing muscle activity from different forearm muscles. However, as shown in Figure 12 , the sEMG patterns are remarkably similar within each pair of actions, despite the differences in the fingers exerting pressure.This similarity arises because the muscle activations required for pressing or pinching with adjacent fingers can produce overlapping sEMG signals due to the anatomical proximity and shared muscle groups involved. Consequently, relying solely on sEMG signals makes it challenging to distinguish between pressures applied by the index or middle finger. While sEMG signals provide valuable information about the force exerted, they lack the spatial specificity needed for precise pressure localization. These findings highlight the necessity of incorporating 3D hand posture information to disambiguate the source of pressure. By fusing sEMG signals with precise hand pose data, our framework can leverage both the force-related information from sEMG and the spatial context provided by hand posture, leading to more accurate and fine-grained pressure estimation.sEMG Feature Extractor f EMG The sEMG feature extractor employs a conventional encoderdecoder architecture [4] , with the encoder composed of three encoder blocks and the decoder comprising three decoder blocks. Each encoder block includes a 2D convolutional layer that filters the input, extracting pertinent features by convolving the input with a kernel of size (3, 3) and preserving the input size with padding set to same. 2D batch normalization and a ReLU activation function follow this. A max-pooling layer then downsamples the output by selecting the maximum value within each pooling window, as defined by the downsampling parameter, effectively diminishing the spatial dimensions. This block structure is iterated with increasing channel sizes and downsample rates. The initial encoder block processes the raw sEMG signals with 32 filters, followed by subsequent blocks that increase the filter count to 128 and 256.In parallel to the encoder, each decoder block is designed to upscale the condensed features to a higher resolution. It utilizes a 2D convolutional layer with an identical (3, 3) kernel size and same padding, followed by batch normalization and ReLU activation, mirroring the encoder block. An upsampling step follows, augmenting the spatial dimensions to correspond to those in the encoder's earlier stages. The decoder restores the feature map from its condensed state, as generated by the encoder, to a higher resolution, employing decoder blocks in reverse sequence. Subsequently, the 2D feature map, having been processed through both the encoder and decoder, is flattened into a 1D feature vector. Finally, the 1D feature vector is projected to a 512-dimensional space through a single fully connected layer.Hand Pose Feature Extractor f hand The Hand Pose Feature Extractor adopts a modified version of the ResNet34 architecture [75] , incorporating 3D convolutions to adeptly handle spatial structure inherent in hand pose data [67, 76] . This adaptation is crucial for capturing the 3D pose of hand postures, offering a richer representation than traditional 2D models. Central to this architecture is the 3D ResNet block, termed the residual block. Each residual block starts with a 3D convolutional layer, employing a kernel size of (3, 3, 3) and stride of 1, maintaining spatial dimensions through padding. This convolution is followed by 3D batch normalization and a ReLU activation, ensuring non-linearity and normalization of activations. A second 3D convolution with the same kernel size and padding repeats this process. The block is designed to accommodate optional downsampled input, aligning input and output dimensions via a downsample path if necessary, employing either average pooling or a (1, 1, 1) convolution for dimension matching, followed by batch normalization.Residual connections in each block allow inputs to bypass layers, helping to prevent the vanishing gradient problem by directly adding inputs to outputs.The architecture initiates with a 3D convolution layer that processes the input using a kernel size of (7, 7, 7), stride (2, 2, 2), and padding to keep spatial dimensions consistent. It continues with four sequential layers comprising multiple residual blocks. These layers progressively increase channel depth from 64 to 512, adhering to the sequence [64, 128, 256, 512] . The layer structure is [3, 4, 6, 3] , indicating the number of residual blocks per layer. Downsampling occurs in the transition between these layers to reduce spatial dimensions and increase the receptive field. After progressing through the convolutional blocks, an adaptive average pooling layer condenses the 3D feature map to a size of 1 × 1 × 1, effectively distilling the spatial-temporal features into a singular, compact feature vector. This vector is then connected to a fully connected layer, which outputs the 512-dimensional final feature representation.Our Full Model f After concatenating the 512-dimensional feature vectors from sEMG and 3D hand pose data, the model processes the resulting 1024-dimensional vector through a linear layer to reduce it to 256 dimensions. This reduction is immediately followed by a batch normalization step and a ReLU activation to standardize and introduce non-linearity into the features, enhancing their representation for the task at hand. This sequence-comprising a 256-dimensional linear layer, batch normalization, and ReLU activation-is executed twice to further refine the features. The refined features then advance through a Sigmoid-activated linear layer, tailored to the specific classification task with dimensions set by the numbers of interest regions (I = 9), finalizing the model's output preparation.For the training of our model, we configured our setup as follows: We trained the model for 50 epochs to ensure adequate training steps. We set the batch size to 64 to balance computational efficiency and training stability. The model was optimized using the Adam optimizer [77] with a learning rate of 0.0001 and (β 1 , β 2 ) of (0.9, 0.999), along with a weight decay of 0.0001. The model training was conducted on a computing environment equipped with an AMD EPYC 7763 64-Core Processor, 1.0TB RAM, and an NVIDIA RTX 6000A 48Gi. Our model's entire training process took approximately 12 hours.In our inference setup, we used a computer equipped with a 12th Generation Intel Core i9-12900K processor, 32GB of RAM, and an NVIDIA GeForce RTX 4080 with 16GB of GPU memory. The total number of parameters in our model amounts to approximately 66.17 M, with the sEMG feature extractor contributing 1.26 M parameters and the hand pose feature extractor contributing 64.11 M parameters. Our analysis on inference time indicated that our model requires only 4 milliseconds to process a single batch, demonstrating its capability to handle 256 streaming inputs in a batch inference setup without any delay or memory overflow issues on our test environment. Thus, we conclude that our model is sufficiently lightweight and efficient for real-time processing, making it practical for applications that require immediate feedback.The model architecture and data processing details of the sEMG Only Model largely align with the official implementation2 disclosed in [4] . One exception to this conformity is our modification of the final linear layer, where we expanded the number of prediction dimensions from 5 to 9. This alteration was made to estimate the force exerted at the tips of each finger, aligning the output dimensions with those of our study. Thus, this implementation serves as a counterpart solely utilizing EMG data, excluding hand pose information. For PressureVision++, we directly utilized the implementation and pre-trained weights officially released by the authors 3 . It is noteworthy that we initially considered using PressureVision, which infers the entire hand's pressure from photos. However, after testing its official implementation 4 , we found it entirely non-functional in our environment due to its lack of generalization ability. Consequently, we adopted PressureVision++, which is limited to estimating pressure at the fingertips but has a much higher ability for generalization.The 3D Hand Posture Only Model corresponds to our full model with the sEMG feature extractor f EMG removed. Consequently, the linear layer that previously mapped a 1024-dimensional concatenated feature to 256 dimensions has been adjusted to map from 512 dimensions to 256 dimensions. This modification reflects the absence of sEMG features, focusing solely on the utilization of 3D hand pose data. The sEMG + Hand Angles Model modifies our full model by replacing f hand with a 3-layer fully connected network, setting all layers to map features to a 256-vector. Each layer incorporates 1D batch normalization and ReLU activation, aiming for a consistent feature representation. Additionally, θ underwent dimension-wise min-max normalization to adjust values to the [0, 1] range, facilitating standardized input for model processing. For aspects not specifically mentioned, such as data processing, model architecture, and the choice of stochastic optimizer, we adhered to the configurations described for our full model. This approach ensures that our comparative analysis remains grounded in a consistent methodological framework, allowing for a fair evaluation of the different models' performance.We employ a set of evaluation metrics that allow for a comprehensive assessment of both classification and regression capabilities. These metrics are designed to quantify the accuracy, precision, and generalizability of our model in predicting pressure exertions across the hand. The metrics include:Coefficient of Determination (R 2 ). The R 2 value measures the proportion of variance in the dependent variable that is predictable from the independent variables. It is defined as:EQUATIONwhere T is the number of time frames, I is the number of hand regions, P i,t is the actual pressure, Pi,t is the predicted pressure for region i at time t, and Pi,t represents a mean pressure for region i. This metric quantifies how well the model's predictions fit the actual pressure data, with values closer to 1 indicating a stronger correlation between predicted and actual pressures.Normalized Root Mean Squared Error (NRMSE). NRMSE provides a normalized measure of the deviation of the predicted pressure values from the actual pressures applied, offering insight into the model's precision in estimating the magnitude of forces. It is computed as follows:EQUATIONwhere P max is the maximum observed pressure in the dataset.Classification Accuracy. This metric assesses the model's capability to accurately classify whether pressure is being exerted by any region of the hand. Specifically, a prediction is considered correct only if the pressure exertion status of all hand regions (fingertips and palm areas) is accurately classified for a given time frame. This metric is crucial for understanding the model's ability to distinguish between active and inactive pressure application scenarios. To assess the generalizability of our hand pressure estimation framework across different individuals, we conducted cross-user evaluations. Additionally, we performed a quantitative comparison with the vision-based method PressureVision++ to benchmark our model's performance against existing state-of-the-art approaches.Cross-User Evaluation Given the known variability of sEMG signals due to anatomical differences and sensor placement among users, it is crucial to evaluate how our model performs when encountering data from participants not seen during training. We partitioned our dataset of 21 participants into a training set and a test set. Specifically, data from 17 participants were used to train the model, while data from the remaining 4 participants were reserved for testing. This ensures that the model is entirely blind to the test participants during both training and evaluation phases.Based on the cross-user evaluation results presented in Table 3 , our PiMForce combining sEMG signals with 3D hand posture data significantly outperforms the sEMG-only baseline across all evaluation metrics. Specifically, our method achieves an R 2 score of 70.06%, compared to 47.90% for the sEMG-only model, indicating a substantial improvement in the proportion of variance explained by the model. The NRMSE is reduced from 14.14% to 10.70%, and the MAE decreases from 12.24% to 8.54%, demonstrating enhanced precision in pressure estimation. Additionally, the classification accuracy increases from 57.40% to 72.01%, reflecting a superior ability to correctly identify the presence or absence of pressure across hand regions. Table 10 shows the detailed performance in terms of hand regions.These improvements highlight the effectiveness of integrating 3D hand posture information with sEMG signals to capture both muscle activation patterns and spatial context, thereby enhancing the model's generalizability to unseen users. The reduced variability in performance metrics, as indicated by the lower standard deviations, also suggests that our method is more consistent across different individuals. Each participant was instructed to repeat each action for 30 seconds, following our data collection protocol. To ensure optimal visibility for PressureVision++, we carefully adjusted the camera angle to capture both the fingers and palm (see Figure 14 ). Note that the Sensel Morph could not measure thumb force during pinch actions, so this data was excluded from the analysis. Figure 13 illustrates ground truth pressure and predicted pressure under the setting of PressureVision++.Table 5 presents the performance comparison between PressureVision++, our proposed method, and the sEMG-only model in the cross-user setting. It is important to note that our evaluation of PressureVision++ is inherently a cross-user performance report, as the model was trained on a separate dataset. As shown in the table, our method significantly outperforms both PressureVision++ and the sEMG-only model in all three metrics. Specifically, PiMForce achieves an accuracy of 82.20%, compared to 66.00% for PressureVision++ and 67.90% for the sEMG-only model. For a detailed breakdown of fingertip performance during plane and pinch interactions, we present the finger-specific performances in Tables 11. These results indicate that our proposed method consistently outperforms PressureVision++ in estimating fingertip pressures across all fingers during both plane and pinch interactions.The cross-user evaluation confirms that our framework is robust and generalizes well to unseen participants. While there is a general decrease in performance when moving from within-user to cross-user evaluation (e.g., classification accuracy decreases from 83.17% to 72.01% for our method), our model still maintains superior performance compared to existing methods. PressureVi-sion++ shows limitations in cross-user scenarios due to its reliance on visual features that can vary widely among individuals, such as skin texture, color, and hand shape. Additionally, it requires full visibility of the hand, limiting its applicability in scenarios where the hand is partially occluded or when interactions involve grasping objects.Our PiMForce's superior performance can be attributed to the complementary nature of sEMG signals and 3D hand pose data. While sEMG provides insights into muscle activation patterns, 3D hand pose offers spatial context that helps disambiguate similar sEMG signals arising from different hand configurations. By integrating these modalities, our framework achieves more accurate and reliable hand pressure estimation across diverse users and interaction types, demonstrating its potential for practical applications in human-computer interaction and virtual reality systems. The video included in the supplementary material is designed to qualitatively demonstrate our model's inference performance in a time-continuous context. 5 The extended results of Section 5. To provide a more detailed analysis of our model's temporal performance, we include additional figures that showcase the ground truth and predicted pressure values over time for various hand actions. Figure 23 displays the pressure trajectories for all nine hand regions during the execution of the TM-Press and Medium Wrap actions by a participant. In this 20-second interval, the participant performs the TM-Press action for the first 10 seconds, followed by the Medium Wrap action for the next 10 seconds.In these plots, each subplot corresponds to a specific hand region, with time on the horizontal axis and pressure magnitude on the vertical axis. The solid lines denote the ground truth pressures measured by the tactile glove, while the dashed lines represent the pressures predicted by our model. The alignment between the ground truth and predicted pressures across different hand regions and actions demonstrates the model's capability to accurately track pressure dynamics over time. Notably, the pressure shifts from regions associated with the TM-Press action to those corresponding to the Medium Wrap action, reflecting the participant's change in hand posture and interaction.Our framework and model, while innovative, is not without its limitations. One of the primary constraints is its reliance on the accuracy of off-the-shelf hand pose detectors during inference. If the hand pose is inaccurately estimated, it can lead to erroneous pressure estimations. Furthermore, our analysis, as evidenced by Figure 6 of the main text, reveals that performance is not uniformly high across all hand postures. This inconsistency indicates areas that could benefit from further refinement. Due to the time constraints of participants, our dataset does not cover the full range of hand postures described in existing literature on hand posture taxonomy [55, 56] . Additionally, the absence of mesh information for both hand and object means we lack precise data on the surface contact locations.Our study also faces limitations due to the demographic homogeneity of the participants. The lack of diverse demographic representation among participants may limit the generalizability of our current model across different populations.Overcoming these limitations opens up several directions for future work. Exploring additional modalities such as depth sensor, constructing an end-to-end model that integrates vision sensors with sEMG signals, adopting high-density electromyography (HD-EMG) equipment [78, 48] to surpass the limitations of standard sEMG devices, and expanding our dataset through more extensive participant sampling and long-term data collection are promising paths forward. Incorporating hand mesh and object mesh estimation [79] [80] [81] with our muscular force learning framework could provide more accurate information regarding contact points and the corresponding pressure on hand mesh. Demographic limitations can be addressed by conducting studies with a larger and more diverse participant pool. Expanding the participant demographics will enhance the generalizability of our results and ensure the framework's applicability across a broader range of users. These steps will help refine our model, making it more robust and widely applicable across different hand interactions.Additionally, our current framework primarily leverages global features from both sEMG signals and 3D hand pose data, without fully exploring alternative multimodal fusion methods. We did not incorporate fine-grained spatial information or prior structural knowledge of hand poses into the model architecture. Future work could investigate different fusion techniques, such as integrating spatial hierarchies or attention mechanisms, to capture the intricate relationships between hand pose and muscle activity. This exploration could further enhance pressure estimation accuracy and robustness.The proposed framework has several promising applications across various fields. In healthcare and rehabilitation, this technology could enhance therapeutic devices by providing detailed feedback on hand movements and pressure, improving the effectiveness of rehabilitation exercises and assistive devices for individuals with motor impairments. In human-computer interaction, integrating hand pressure data could lead to more intuitive and responsive interfaces, particularly in virtual reality (VR) and augmented reality (AR) environments. In occupational safety and ergonomics, our framework could help design ergonomic tools and workspaces, reducing the risk of repetitive strain injuries and enhancing worker productivity and safety.Despite its potential usefulness, the use of forearm-worn sEMG to estimate hand movements, poses, and pressure could introduce biases in the model's performance due to medical and physiological variations. For instance, some individuals have less muscles than others; for example, the palmaris longus muscle, which flexes the wrist joint, is absent in about 15% of the population [82] . To mitigate these risks, it is essential to consider a broader clinical demographic when recruiting participants and designing research for future studies.While our study did not present any direct clinical risks to the participants, we recognize the importance of ethical data handling, particularly regarding sEMG signals which may contain personal information. Prior to participation, all participants were provided with a detailed information sheet outlining the purpose, procedures, and potential risks of the study. We obtained written informed consent from each participant, ensuring they understood the nature of the data being collected and their right to withdraw at any time. All collected data was anonymized, complying with relevant data privacy regulationsNeurIPS Paper Checklisthttps://github.com/NYU-ICL/xr-emg-force-interfacehttps://github.com/pgrady3/pressurevision2https://github.com/facebookresearch/PressureVisionhttps://hci-tech-lab.github.io/PiMForce/Figure 17: Qualitative visual analysis among comparative models.Figure 20: Pressure estimation results for the vision-aided hand in the test set. 


Decision: The reviewers appreciate the collected dataset and the idea of combining 3D hand poses and electromyography signals for hand pressure estimation. The reviewer raised concerns about the lack of comparison with other vision-based baselines, and novelty in the data fusion framework, etc. The author provided detailed and professional rebuttals to address most of the concerns. After reading these comments, I give my suggestion as acceptance of this paper.

Reviews: review: paper_summary: paper_summary: This paper presents a novel framework for estimating hand pressure during various hand-object interactions using multimodal data. The framework integrates sEMG and 3D hand posture information to enhance the accuracy of pressure estimation. They introduce a dataset to validate their approach. The primary contribution of the study is the exploration of combining vision-driven 3D hand posture data with sEMG to improve the robustness of hand pressure estimation.
strength_weakness: strength_weakness: 1. Although integrating sEMG signals with 3D hand posture data is a promising approach, there is insufficient experimental evidence to support the motivation behind this combination. The paper lacks clear proof of the benefits described, beside the sentence.

2. The paper does not adequately compare their approach with other vision-based datasets or techniques that also extract 3D hand posture information.

3. While combining sEMG and vision-based methods theoretically addresses individual robustness issues, the claim that this combination improves overall robustness is not convincingly supported by the data. The dataset should demonstrate the robustness of the proposed method, not just improvements in accuracy.
questions: questions: Please list up and carefully describe any questions and suggestions for the authors. Think of the things where a response from the author can change your opinion, clarify a confusion, or address a limitation. This is important for a productive rebuttal and discussion phase with the authors.
limitations: limitations: see weaknesses
paper_summary: paper_summary: This paper proposes a hand pressure estimation method based on 3D hand poses and t forearm surface electromyography (sEMG) signals. Accordingly, the paper constructs a multimodal dataset containing pressure, 3D hand poses, and sEMG signals. The paper experimentally verifies that combining 3D hand poses and sEMG signals achieves better results compared to using these two modalities independently. Furthermore, compared to vision-based pressure estimation methods, the method achieves better results in complex hand-object interaction scenarios.
strength_weakness: strength_weakness: 1) From the perspective of multimodal fusion methods, the framework proposed in this paper lacks novelty. This framework directly combines the global features of the electromyography modality and the global features of hand poses without fully utilizing the fine-grained spatial information and prior structural information of the hand poses.

2) The qualitative comparison with PressureVision++ is unfair. Due to biases in the training dataset and the design preferences of the method, PressureVision++ is more suitable for pressure estimation in planar contact scenarios. However, this paper does not compare with PressureVision++ in such scenarios.

3) The training and testing setups are not sufficiently reasonable. During training, hand poses captured by a data glove are used as input, while during testing, the estimated results from a vision-based hand pose estimation model are used as input. There is a significant gap in the distribution of hand poses, which may potentially impair the performance of the hand pose branch.
questions: questions: 1. Would it be beneficial for the overall method's performance to use predicted hand poses as input during training?
2. Have you tried other multimodal information fusion methods?
limitations: limitations: Yes
paper_summary: paper_summary: This manuscript introduces a framework for estimating the pressure of 9 points on the hand during 22 distinct hand object interaction types including pinches, grasps and planar interactions. They use an 8 channel sEMG band placed at the forearm as well as hand pose estimated using a glove or monocular video camera. They train models using a dataset from 20 participant that uses pressure sensors in the glove as ground truth. They evaluate models within participants, and compare pressure estimation models that use different combinations of sEMG and hand pose, including different representations of hand pose using joint angles or embedded 3D coordinates. The best performance is achieved by a multimodal network that fuses an encoding of the spectrogram of the sEMG and the 3D hand pose, the later using a 3D convolutional network. They compare performance across fingers and gestures, and give qualitative comparisons to other vision-based approaches.
strength_weakness: strength_weakness: - All evaluation is within-participant. sEMG is known to be quite variable across participants due to variations in anatomy and placement, and the hierarchy of performance across modalities will likely change. 
- The comparisons with PressureVision++ (Figure 4) and other SOTA approaches are a little underwhelming, as they essentially show the framework only functions when all fingertips are tracked. A more informative set of comparisons would show results, with ground truth, for cases when the fingertips are unoccluded. For figure 16, the quantitative comparison betwene the two would be informative.
- The behavioral generalization to new objects and gestures is unclear. Similarly the performance with raw vision data is only qualitative. It is unclear how the system handles domain mismatches between the glove and vision-based hand pose estimation framework. In combination with the lack of cross-participant information the in the wild utility of the approach is unclear.
questions: questions: ** Questions ** 
- what is the cross participant performance? 
- What is the quantitative performance compared to Pressure Vision? What is the correlation between the two techniques if an absolute ground truth (e.g. from sensel) is not possible. 
- Why does utilizing hand angles not improve the model performance? 
- How was 3D hand pose acquired in cases without the glove? I don't see a description of this method. 
- Can you show raw traces of the discretized pressure to give an intuition for what the real pressure data looks like? 

**Minor**

- **Datasets for Hand Pressure Estimation** — do these datasets include pressure, many of these appear to be object interaction papers
- Fig 4: can you clarify if this participant was in the training dataset?
- L302 ‘significant’ - were significance tests run here?
- Table 3 - what are the relevant error bars for this table? 
- nit: L 336 - pionering ?
- Table 2:unclear what errors are over
- Table 3: unclear what value is being plotted
- all exocentric?
- nit: Figure 3 pixelated
- Manuscript overstates at multiple points ‘L316 ‘exceptional’
limitations: limitations: Yes, they review this in section F.
paper_summary: paper_summary: The authors leverage 3D hand pose and sEMG signals as inputs to facilitate hand pressure estimation. With data gloves, a multimodal hand-object interaction collection system is devised. Empirical experiment results show the efficacy of the proposed method. Furthermore, the feasibility and robustness is proven in a single-camera scenario.
strength_weakness: strength_weakness: - A highly related dataset ''ActionSense: A Multimodal Dataset and Recording Framework for Human Activities Using Wearable Sensors in a Kitchen Environment'' in NeurIPS 2022 should be cited and discussed.

- The evaluation is only conducted in a drop-one-session manner. It might be better if a drop-one-subject evaluation could be provided to demonstrate the cross-subject ability for a more general application.

- The results are impressive in their low relative errors. It would help more with the absolute errors reported. 

- The results are reported with little temporal information. A curve showing the predicted force changing with time would be preferable. Also, quantitative metrics on the temporal consistency would be a bonus if possible.

- For glove-less evaluation, only qualitative results are reported. It appears possible to provide a quantitative comparison given the controlled hand postures. For example, attaching sensors to the contact region of the MR-Press makes it possible to provide GT in glove-less scenarios.

- Reporting quantitative prediction errors for different postures is a missing chance to provide more insights.
questions: questions: - Are the readings from 6 sensors belonging to Region 8 simply added up as the pressure of Region 8?

- Is it possible to briefly explain how the quality would be influenced by muscle fatigue as mentioned in L158?

- Will the data glove introduce different interaction patterns from those without data gloves?
limitations: limitations: Please refer to the Weaknesses and Questions part.

Comments: title: Thank you for the detailed response
Please check the author rebuttal
Response to Ethics Review
Response to Ethics Review
Additional comment for minor points
Additional comment for Q1 and Q2
comment: We deeply appreciate your meticulous review of our manuscript and the insightful suggestions provided. Your feedback has been instrumental in guiding our revisions, and we have diligently addressed the remaining points raised in your comments. As the discussion phase concludes, we are confident that the detailed clarifications and newly suggested empirical motiation, as presented in our rebuttal, provide a compelling testament to the effectiveness of our proposed method. We sincerely hope that these revisions have adequately resolved your concerns. We remain available to address any further questions you may have and are grateful for your valuable contribution to improving the quality of our work.
Thank you for your thorough review and valuable contributions to our work. Your insights are greatly appreciated.

Regarding the PressureVision++ quantitative comparisons, we found your comment about the low scores intriguing. To prevent miscalibration, we took steps to ensure that the units (psi, which is the unit outputted by the Sensel Morph pressure sensing array) and the range of values predicted by PressureVision++ were converted into Newton units, as is done in our paper, and calibrated for two different force ranges and units, enabling pressure estimation up to 20 N, as described in "B.4.1 sEMG Signal and Pressure Data" (Line 937 of our manuscript). This process helped mitigate potential miscalibration concerns, allowing for a fair comparison between the two systems. Moreover, we performed time synchronization between the camera and the Sensel Morph pressure sensing array. We verified the proper synchronization through data collection and preprocessing, ensuring no issues in this regard.

However, it seems that PressureVision++ excels at detecting contact or the presence of force but faces challenges in accurately estimating force magnitude. This is consistent with the trends observed in PressureVision++ [R1]. For clarity, Table 2 (Performance compared to a PressureVision baseline) in [R1] primarily focuses on quantitative performance metrics of PressureVision++. Below is an excerpt from Table 2 in [R1]:
| Method               | Contact Acc. | Contact IoU | Volumetric IoU |
|----------------------|--------------|-------------|----------------|
| PressureVision [R2]  | 72.7%        | 15.2%       | 11.3%          |
| PressureVision++ [R1] | **89.3%**  | **41.9%**   | **27.5%**      |

The primary performance metrics defined in [R1] are (1) Contact Accuracy, (2) Contact IoU, and (3) Volumetric IoU. Contact Accuracy and Contact IoU assess the correct identification of contact or the presence of force, corresponding to our "accuracy" metric. Conversely, Volumetric IoU reflects the prediction accuracy of pressure magnitude (section "5. Evaluation" of [R1]), corresponding to our R² or NRMSE, which represent regression task performance. While PressureVision++ demonstrates high performance in Contact Accuracy (measuring whether force presence is correctly identified regardless of location) and Contact IoU (IoU between the ground truth and predicted contact image), achieving 89.3% and 41.9%, respectively, Volumetric IoU only reaches 27.5%. This suggests that PressureVision++ excels in classification tasks, accurately identifying the presence or absence of force, but faces challenges in regression tasks, specifically estimating the precise force magnitude.

This observation aligns with the trend we observed in our Table R2. In Table R2, PressureVision++ shows comparable accuracy (measuring whether force presence is correctly identified for all five fingers) to the Force-aware Interface (67.90% versus 66.00%). However, PressureVision++ exhibits relatively low values in R-square (40.30%) and NRMSE (32.95%), indicating less accurate performance in estimating force regression. Based on these results, PressureVision++ demonstrates strength in detecting contact or the presence of force but struggles to accurately estimate force magnitude. In comparison, the sEMG only model, while exhibiting similar performance in accuracy to PressureVision++, seems to have a comparative advantage in accurately estimating the precise force magnitude.

Furthermore, to ensure transparency and reproducibility of our evaluation of PressureVision++, we will publicly release all data and code used for quantitative assessment and performance calculation of comparative methodologies. This information will be included in a new section, "Cross-User Evaluation," in Section D (Line 1046-1047) of the revised manuscript.
Thank you for the detailed point by point response, which addresses my original comments. I need more time to reconsider my original score but hopefully this will move it in a positive direction. 

One follow up on the Pressure Vision comparisons. These are great. I am wondering why the R2 is so bad. Is it predicting the pressure at the wrong time, is the pressure variable, or is it simply miscalibrated in intensity?
We again express our sincere gratitude to the reviewer for their insightful comments and constructive feedback. We appreciate the time and effort invested in carefully reviewing our work. The reviewer's suggestions have significantly contributed to enhancing the clarity, depth, and overall quality of our manuscript. We have diligently addressed each concern raised, and we believe the revisions made have strengthened our paper considerably. We are particularly grateful for the valuable guidance on presenting temporal information. We believe that the revised manuscript, incorporating the reviewer's feedback, offers a stronger contribution to the field and we are confident that it will be well-received by the NeurIPS community.
We sincerely appreciate you taking the time to thoroughly review our rebuttal and acknowledge the clarification regarding our comparison with PressureVision++. We understand your perspective on the importance of multimodal fusion methods and agree that there is a clear potential for improving our approach in that aspect. We are grateful for your constructive feedback and insightful suggestions, particularly regarding the exploration of fine-grained spatial information and prior structural knowledge of hand poses.

Thank you again for your valuable contribution to our work.
I have carefully read the author's meticulous rebuttal and the reviews of other reviewers. I thank the authors for providing a detailed rebuttal. First, my concerns about comparison with PressureVision++  have been addressed. Although the authors state that their main contribution is not the multimodal fusion method, I still think it is important. After all, it is “foreseeable ” that fusing more modes can improve the accuracy of estimation. Overall, considering the pioneering nature and adequacy of the experiments, I maintain my initial score, which is borderline accept, but there are still some weaknesses in the depth and innovation of the fusion framework.
Thanks for the thorough responses. Most of my concerns are addressed.
Dear Reviewers of submission 7975,

Thank you for your time and effort so far in reviewing this paper. The author kindly provided the rebuttal to your comments, and the deadline for the author-reviewer discussion is tomorrow (13th August). It becomes more critical for this paper since reviews of this paper is mixed. It would be much appreciated if you could participant in the discussion with author to check if your concerns are addressed by the author or not. Thank you!

Best regards,

Your AC of submission 7975.
We appreciate the thorough ethics review and agree with the recommendations. We acknowledge the importance of transparently addressing ethical considerations in the main paper, not just relegating them to the supplementary material. 

We propose the following additions to incorporate ethical considerations into the Section 3.2 Data Collection Procedure of the main manuscript:
- (To Line 147) Prior to participation, all participants were provided with a detailed information sheet outlining the purpose, procedures, and potential risks of the study. We obtained written informed consent from each participant, ensuring they understood the nature of the data being collected, their right to withdraw at any time, and the measures taken to ensure data privacy.
- (To Line 159) All collected data was anonymized, complying with relevant data privacy regulations.

Moreover, we propose adding the following paragraph to Line 1136 of the "Potential Social and Broader Impacts" section (Section F):
- While our study did not present any direct clinical risks to the participants, we recognize the importance of ethical data handling, particularly regarding sEMG signals which may contain personal information. Prior to participation, all participants were provided with a detailed information sheet outlining the purpose, procedures, and potential risks of the study. We obtained written informed consent from each participant, ensuring they understood the nature of the data being collected and their right to withdraw at any time. All collected data was anonymized, complying with relevant data privacy regulations.

By incorporating these additions, we aim to provide a more comprehensive and transparent discussion of ethical considerations related to our research, demonstrating our commitment to conducting responsible researc
We appreciate the reviewer's insightful comments regarding the ethical considerations of our research. We acknowledge the need to clarify how we managed informed consent, data privacy, and handling of personal data, especially given the sensitive nature of sEMG signals.

To address these concerns, we propose adding the following paragraph to Line 1136 of the "Potential Social and Broader Impacts" section (Section F):
- While our study did not present any direct clinical risks to the participants, we recognize the importance of ethical data handling, particularly regarding sEMG signals which may contain personal information.  Prior to participation, all participants were provided with a detailed information sheet outlining the purpose, procedures, and potential risks of the study.  We obtained written informed consent from each participant, ensuring they understood the nature of the data being collected and their right to withdraw at any time.  All collected data was anonymized, complying with relevant data privacy regulations.  

This addition explicitly addresses the reviewers' concerns and highlights our commitment to ethical research practices. We believe this strengthens our paper by providing a transparent account of our data collection and handling procedures.
**(M1) Datasets for Hand Pressure Estimation — do these datasets include pressure, many of these appear to be object interaction papers**

You correctly pointed out that the line between object interaction papers and hand contact/pressure papers can be blurry, and Table 1 and Section 3.1, Vision-based Hand Pressure Estimation, reflect this. While some datasets focus solely on hand contact, others include pressure information, and some delve into object interaction to varying degrees. Rather than attempting a strict categorization, which can be challenging due to the inherent interconnectedness of these areas, we chose to provide a more comprehensive overview in this section. This approach allows for a broader discussion of related work and a better understanding of the research landscape.

**(M2) Fig 4: can you clarify if this participant was in the training dataset?**

Yes. Of course, this testing session on display does not overlap with the training session.

**(M3) L302 ‘significant’ - were significance tests run here?**

No. Thanks to Reviewer YYDP, we understand that the term ‘significant’ was understood differently for readers. We’ll remove that expression to avoid misunderstanding

**(M3) Table 3 - what are the relevant error bars for this table?**

Table R5 presents the same data as Table 3 in the main manuscript, but includes the standard deviation for each reported value. We will update Table 3 in the revised manuscript to include these standard deviations, providing a more complete picture of the variability in our results.

**(M4) Table 2:unclear what errors are over**

As guided in the Experiments section (Line 248), we present the evaluation metrics in Appendix D.2. To assess our model's performance, we utilize three metrics:  Coefficient of Determination (R²) which measures the proportion of variance in the actual pressure explained by our model, Normalized Root Mean Squared Error (NRMSE) which quantifies the precision of our model's pressure predictions, and Classification Accuracy, assessing the model's ability to correctly identify whether pressure is being applied to each region of the hand. We will add this summary to Line 248  of the main text.

**(M5) Table 3: unclear what value is being plotted**

The caption for Table 3 indicates that metric is NRMSE, but if there is still any ambiguity, please let us know again.

**(M6) all exocentric?**

Yes. In this research, we use an exocentric camera for camera modality. However, we believe it can be extended to usage of egocentric cameras if the hand pose estimation model is working well.

**(M7) nit: Figure 3 pixelated**

It’s a mistake. We’ll fix it.

**(M8) Manuscript overstates at multiple points L316 "exceptional"**

We’ll remove that expression to avoid overstatement. “Specific actions such as I-Press, M-Press, and R-Press exhibit high accuracy and low NRMSE,showing the model’s superior performance in simpler press interactions.”
**(Q1) Are the readings from 6 sensors belonging to Region 8 simply added up as the pressure of Region 8?**

Instead of summing the values from the 6 sensors within Region 8, we define the maximum value among those sensors as the pressure for that region. Consequently, individuals with smaller hands might not activate all 6 pressure sensors in Region 8 during certain grasp postures. Therefore, we opted for the maximum value among the 6 sensors rather than the sum. Similarly, we chose the maximum over the average for the same reason. When only a subset of the 6 sensors are activated due to hand size discrepancies, using the average would result in an artificially low pressure reading due to the inactive sensors.

**(Q2) Is it possible to briefly explain how the quality would be influenced by muscle fatigue as mentioned in L158?**

Muscle fatigue is the decline in a muscle's ability to generate force. It happens when a muscle is repeatedly contracted or held in a sustained contraction for an extended period. Imagine holding a heavy book at arm's length – after a while, your arm will start to shake, and it will become increasingly difficult to keep the book up. This is muscle fatigue. 
The connection between muscle fatigue and EMG-based control is that muscle fatigue can negatively impact the quality of EMG data used for hand pressure estimation. As muscles tire, their electrical activity changes. This can manifest as increased signal amplitude (RMS) and shifted frequency content[R4]. These changes make it harder for the model to accurately distinguish between different hand postures and pressure levels. Moreover, fatigue leads to less consistent muscle activation patterns, making it challenging for the model to learn reliable relationships between EMG signals and exerted pressure[R5].
While muscle fatigue is recognized as an important topic in EMG-based control, it falls outside the scope of this study. To minimize its effects on our results, we ensured sufficient rest periods between data collection trials for both training and testing. To convey this consideration more explicitly, we will include a statement regarding this after Line 930 in Section B.3 Data Collection Protocol.