Posture-Informed Muscular Force Learning for Robust Hand Pressure Estimation
Authors: Kyungjin Seo
Junghoon Seo
Hanseok Jeong
Sangpil Kim
Sang Ho Yoon
Venue: NeurIPS 2024 poster (2024)
Keywords: Hand Pressure Estimation
Multi-modality
Hand-Object Interaction
Primary area: machine_learning_for_other_sciences_and_fields
TL;DR: A novel framework is proposed for hand pressure estimation, augmenting sEMG signals with 3D hand posture information.

Abstract: We present PiMForce, a novel framework that enhances hand pressure estimation by leveraging 3D hand posture information to augment forearm surface electromyography (sEMG) signals. Our approach utilizes detailed spatial information from 3D hand poses in conjunction with dynamic muscle activity from sEMG to enable accurate and robust whole-hand pressure measurements under diverse hand-object interactions. We also developed a multimodal data collection system that combines a pressure glove, an sEMG armband, and a markerless finger-tracking module. We created a comprehensive dataset from 21 participants, capturing synchronized data of hand posture, sEMG signals, and exerted hand pressure across various hand postures and hand-object interaction scenarios using our collection system. Our framework enables precise hand pressure estimation in complex and natural interaction scenarios. Our approach substantially mitigates the limitations of traditional sEMG-based or vision-based methods by integrating 3D hand posture information with sEMG signals.
Video demos, data, and code are available online.

Posture-Informed Muscular Force Learning for Robust Hand Pressure Estimation
Kyungjin Seo't, Junghoon Seo' Hanseok Jeong", Sangpil Kim Sang Ho Yoon 1:2
1 Graduate School of Culture Technology, KAIST, South Korea
2 Graduate School Metaverse, KAIST, South Korea Department of Artificial Intelligence, Korea University, South Korea
Abstract
We present PiMForce, novel framework that enhances hand pressure estimation by leveraging 3D hand posture information to augment forearm surface electromyo- graphy (sEMG) signals. Our approach utilizes detailed spatial information from 3D hand poses in conjunction with dynamic muscle activity from sEMG to enable accurate and robust whole-hand pressure measurements under diverse hand-object interactions, We also developed multimodal data collection system that combines a pressure glove, an sEMG armband, and a markerless finger-tracking module, We created comprehensive dataset from 21 participants, capturing synchronized data hand posture, sEMG signals, and exerted hand pressure across various hand postures and hand-object interaction scenarios using our collection system. Our framework enables precise hand pressure estimation in complex and natural inter- action scenarios. Our approach substantially mitigates the limitations of traditional sEMG-based or vision-based methods by integrating 3D hand posture information with sEMG signals. Video demos, data, and code are available online."
Introduction
Hands are a central tool for humans to interact with the surrounding environment. With the ad- vancement in hand tracking technology, hand inputs, including position, orientation, gesture, and motion, are increasingly used as a primary means of control, especially for emerging interfaces (e.g., augmented/virtual reality and wearables). Using hands the main interaction medium offers high level of versatility and flexibility to achieve natural and intuitive interactions.
Recent studies have started to utilize hand pressure information to support hand-based interactions such as touching [1], grasping [2. 3]. and pressing [4]. Researchers also utilized hand pressure to provide effective haptic feedback [5] for a more immersive experience. Furthermore, precise hand pressure measurement becomes essential for real-world applications. including ergonomic evaluation [6]. hand rehabilitation [7], and prosthetic hand control [8]. To this end, previous works focus on obtaining real-time and accurate hand pressure information with direct measurement approaches utilizing gloves [9-11] or load cells [12]. However, these approaches require users be in physical contact by either wearing or holding the device, which hinders natural hand movements reduces user comfort. Thus, the necessity of direct contact limits the users from performing basd-based interactions in natural and unrestricted manner.
To this end, non-invasive sensing techniques estimate exerted hand pressure without embedding sensors on the user's hand have been highlighted. These methods include profiling wrist topography with capacitive sensing [13], multiple pressure sensing from the wrist [14], and electromyography
Both authors contributed equally research.
Project Page: https: /twp//hicicthhhee lab. laab.githe github. io so/PiMoocce/
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Model Training Session (with Data Glove)
Model Testing Session (without Data Glove)
Multimodal Neural
CAMERA
Hand Tracking Meduk
Trained Model
Clove
GEMG EMMo-rmmondd
Praicuure
Input
FENT Grmband
Inputt
Hand Pressure Estimation
Our sensing framework (PiMForce) leverages 3D hand posture information along with sEMG data to enable a whole-hand pressure estimation during various hand-object interactions. We support real-time pressure estimation the fingertips and palm regions based RGB image and sEMG imputs. The intensity of each node's 's color indicates the pressure level.
Figure I:
from the forearm [2]. Recent works utilized only a single RGB-D [15] or RGB camera [16, 17] with computer vision techniques estimate pressure exerted by the hand. However, previous methods had limitations, where they could only estimate hand pressure when interacting with plane surfaces required line-of-sight view of the hand. With these limitations, it is hard support natural interaction contexts like working with diverse hand grasps.
In this work, we introduce PiMForce, novel multimodal sensing framework that enhances real-time hand pressure estimation by leveraging 3D joint information of the hand and forearm sEMG signals, covering the fingertip the entire palm. As illustrated in Figure our framework addresses previous challenges by integrating detailed spatial information from 3D hand poses with dynamic muscle activity from sEMG measurements. This multimodal sensing integration allows us estimate subtle and comprehensive hand pressure even under diverse grasps. To validate the proposed model, we built multimodal hand data collection system and created dataset from 21 participants. We believe our dataset is the first of its kind containing multimodal sensor signals during hand interactions under various grasps. We demonstrated our work on an off-the-shelf system with a single camera confirm the accuracy and feasibility of the proposed framework. Our contributions are listed as follows:
We propose PiMForce, novel hand pressure estimation framework that enhances sEMG signals by incorporating hand posture information.
We develop multimodal hand data collection system with data collection protocol and create a unique dataset containing simultaneous hand pressure, hand posture, and surface electromyography signals.
Evaluation and analysis of experiments demonstrate the improved performance of our approach, showing its consistent superiority over existing sEMG-based and vision-based methods.
2 Related Works
2.1 Vision-based Hand Pressure Estimation
Previous works explored the interaction between the hand and objects by observing the movement and rotation of the object over time to estimate the pressure exerted by hands. By determining the pressure required produce these observed changes, the model estimated the aggregate pressure applied by the hand [18-20]. These approaches enabled the pressure estimation to act upon concealed non-visible areas where direct visibility of the hand in contact with object absent. Still, previous approaches had limitations where interacting with immovable objects would not work due the absence of dynamic interaction indicators.
Researchers also looked into different visual indicators like color changes in fingertips, which represent fluctuation of blood circulation within the fingertips [21, 22] compression of skin tissues [23, 24]. These physiological behaviors served as indicators of the exerted hand pressure. Moreover, examining shadows cast during hand-object interaction provided further insight into the spatial relationship and dynamics of force between them [25-27].
Recent works further advanced the existing visual indicator approach where they use hand image captured by a single camera at a distance to estimate the hand pressure [16, 28, 17, 29]. They employed deep learning model that facilitates the understanding of visual cues to estimate accurate

---

Table 1: Comparison with the previous datasets for hand contact and pressure estimation. We modified and updated the table from [17]. NA refers to not available'.
Datsset
Iapui Modality
Frames
Participatts
Contact/
Pressure
Fose
Whole Hand
Natural Objects
OakInk [47]
RGBD
230k
12
Inferred from pose
DexYCB [44]
RGBD
582k
10
Inferred from pose
d
HO-3D [45]
RGBD
78k
10
Inferred from pose
GRAR [46]
Pose
1.6M
10
Inferrad pose
ContaccPosc [50]
RGBD
3.0M
SU
Thermal imprint
8
Disionne [Xs]
RGB
3.M
36
Pressure Pad
ContactLabelDB [17]
Pad
Force-Aware Interfice [-]
sEMG
17.8
Pressure Pad
x
HD:EMG [1S]
SEMG
67.6M
20
Custom made Device
ActionSence (49)
M4
9.42M
10
Pressure Glove
8
Ours
Posc+sEMG
83.2M
Pressure Glove
8
hand pressure in an end-to-end fashion. However, previous works require a high-quality whole-hand image without occlusion since they rely on visual indicators for the estimation. In our work, utilize multimodal inputs, including vision-driven 3D hand posture information and wearable-bass muscle activation signals, enhance the estimation of robust hand pressure under various hand-object interaction contexts.
2.2 Wearable-based Hand Pressure Estimation
Researchers have used forearm/wrist surface electromyography (sEMG) sensors to acquire finger muscle activation information. Here, the sensor captured train of neuron impulses propagated through the arms from the forearm or wrist [30, 31]. Previously, researchers used sEMG sensors to estimate various types hand-related force/pressure, including force/pressure from gripping [32-36] and fingertip 37-41. Recent works also enabled the estimation of hand pressure along with hand gesture recognition using sEMG signals [8, 42]. However, previous works only dealt with limited issue existed with using sEMG signals for complex hand interactions where similar muscle activation signal behaviors were observed across different hand poses. This could easily confuse the model and lead to false behavior. In this work, we train the model with 3D hand posture to encode distinctive hand pose information alongside sEMG signals. This integration forms robust and accurate hand pressure estimation framework for similar muscle activation behaviors but different hand poses. It is worth emphasizing that this study is the first to incorporate hand posture information for hand pressure estimation using forearm- worn sEMG. Previous studies primarily focused on estimating pressure at the fingertip or on a single gripping force, but approach expands this encompass the whole hand.
set of discrete hand poses [43]. Moreover,
2.3 Datasets for Hand Pressure Estimation
In the computer vision and machine learning community, researchers have formed various types of hand-object interaction datasets for hand pressure estimation. These vision-based datasets collected rich visual and pose information for hand-object interactions, capturing everything from object affordances to whole-body grasps [44, 28, 17, 45-47]. J. On the other hand, sEMG-based datasets have also been proposed and used to estimate hand pressure for AR/VR or prosthetic robotic arm control applications [48, 4]. A highly relevant work is the ActionSense dataset [49], which focuses on capturing multimodal data of human activities a kitchen environment using wearable sensors. However, while ActionSense provides a valuable resource for understanding general kitchen activities, our work focuses on the utilization of 3D hand posture in muscular force learning for understanding hand pressure estimation. Furthermore, the temporal resolution of EMG data and the spatial resolution of hand pressure ActionSense are substantially lower compared to ours, making it challenging to utilize rich sensor input and output effectively. Still, the dataset containing both rich visual and physiological information is missing.
In this work, we attempt to set up new multimodal dataset that contains 3D hand pose information, sEMG signals, and ground truth measurement hand pressure as shown in Table 1. Our work provides a holistic view of whole-hand dynamics during various hand-object interactions. This inte- gration enables continuous and comprehensive pressure estimation across the whole palm, addressing the limitations of previous datasets that either infer pressure from visual cues measure it in isolation.

---

Our dataset not only captures the subtle interplay between visual and tactile information, but also increases the potential candidates for input features for accurate and robust hand pressure estimation.
3 Building Multimodal Dataset: Posture, Electromyography, and Pressure
To capture multimodal data with various hand-object interactions, we integrated and customized existing hardware, including a pressure glove, an armband with 8-channel sEMG sensors, and markerless finger tracking module. Figure 7 in supplementary material showcases our data collection setup capture real-time and synchronous multimodal data, including 3D hand posture, sEMG signals, and exerted hand pressure. More detailed information about the hardware, defined hand postures, data collection protocol, and data processing can be found in Section B.
3.1 Data Collection Setup
To capture pressure exerted from the hand, developed customized pressure glove using single 65-node pressure sensing glove (TactileGlove, Pressure Profile Systems) attached with a pressure sensor RRAIDDIY, Marveldex) at each fingertip. We added flexible sensors the fingertips to address missed readings when the pressures were exerted on the edge of the fingertip. Our pressure glove supports pressure readings up to 55 N/em2 with sampling rate of 40 Hz.
Pressure Glove.
3D Hand Pose. Recent common approaches to obtaining ground truth 3D hand pose [51] involve using multiple RGB cameras derive 2D hand poses from each camera, followed by triangulation [52, 53] or hand template fitting [44, 45, 54]. However, these methods are infeasible when a pressure data glove is worn, the glove obscures the hand, hindering accurate hand pose estimation from RGB images. To acquire accurate 3D hand pose information under this constraint, we employed a magnetic sensing-bane markerless finger tracking module (Quantum Mocap Metaglove, Manus). The module provides each finger's s 3D position and 3-axis joint angles with a sample rate of 120 Hz and less than 5-millisecond latency. We attached the finger-tracking module to the pressure glove to capture exerted hand pressure and 3D hand pose data simultancously.
8- 8-Channel sEMG Armband. We used 8 sEMG sensors (Trigno Avanti, Delsys) and installed sensors into a customized armband made with semi-flcxible material (TPU 95A) to ensure electrode contact for various sizes of forearms. Our system captures muscle action potentials with sampling rate of 2,000 Hz.
Multimodal Data Synchronization. Before training the multimodal dataset, we employed linear interpolation approach to synchronize high frame rate readings (sEMG) with low frame rate data (3D hand pose). We first joined the data from matched time points based on the collection time of the sEMG data and interpolated missing values of the hand pose data linearly. We applied the same approach to the pressure glove, where we synchronized high frame rate sensors (pressure sensors) with the low frame rate sensing glove (TactileGlove). Then, we adopted a nearest-neighbor-baee interpolation to synchronize 3D hand pose and sEMG data with hand pressure [4].
3.2 Data Collection Procedure
With IRB approval, a total of 21 right-handed participants took part in this study. Of these, 17 were male (81%) and 4 were female (19%). The participants' ages ranged from to 32 years, with a mean age of 24.3 years (SD) = 3.9). To ensure good quality hand pressure data using our glove, we chose participants with hand sizes greater than 180 mm. Prior to participation, all participants were provided with detailed information sheet outlining the purpose, procedures, and potential risks of the study, We obtained written informed consent from each participant, ensuring they understood the nature of the data being collected, their right to withdraw at any time, and the measures taken to ensure data privacy. We equipped participants with our multimodal glove and an 8-chanmel sEMG armband. Following initial calibration to compensate for each user's hand size, participants performed 22 distinctive hand-object interactions for the data collection task (Figure 8 in supplementary material).
Our hand-object interactions consist of 7 hand-plane interactions, 5 pinch interactions, and 10 distinctive hand grasps. We included the same hand-plane and pinch interactions from recent hand pressure estimation work [4] while adding pelms-ressing motion. In terms of hand grasps selection,

---

Train Session with
Raw EMG
Spectrograms
Clessification Outputt (*)
Regression Outputt P)
EMO
Encoder
Decodar (CNN)
Transform
Session without
Concor
21
Posa with Postures
Ground Truth Classification
Ground Truth
Camera Input
Trarsfom
RooMet
Regressioni P)
Pose
Dactor
27
Train Session
Test Session
Deactivation
Activation
Figure 2: Our multimodal hand pressure estimation architecture enhances sEMG data by embedding 3D hand pose information. We train the model using a classification-rereesss regression joint loss to improve hand pressure estimation.
we selected 10 grasps from representative 33 grasp types [55] based on the clustering of palm pressure distribution similarity among grasp taxonomies [56]. We collected 1, 1,980 seconds of synchronized multimodal data per participant (30 seconds x 22 hand-object interactions x 3 sessions). Here, we used data from 2 sessions for training while the remaining session was reserved for evaluation. We provided sufficient rest periods between each trial to prevent muscle fatigue, ensuring the quality of collected data. All collected data was anonymized, complying with relevant data privacy regulations.
Method
4.1
Overview
The rationale behind using sEMG to estimate the pressure exerted by the hand lies in the direct relationship between muscle electrical activity and the pressure generated during hand interactions [57- 59]. When muscles contract for movement, they generate bioelectric signals that can be captured by EMG sensors [60]. This indicates that sEMG signals have the potential to estimate the pressure exerted by the hand. The ability to decode sEMG signals from forearm muscles generated by finger- level movements will set robust foundation for understanding complex hand interactions. However, the pressure exerted by the hand cannot be solely represented with muscle activation information. The main reason is that the distribution of hand pressure varies according to different hand postures. For example, similar sEMG patterns may be generated by different hand pressures depending on the related hand postures or grasps [43, 61]. This highlights the importance of considering hand posture information along with sEMG signals to estimate the exerted hand pressure precisely. Section C.1 in supplementary material addresses the specific empirical observation for this motivation.
To address these issues, we enhance sEMG signals by leveraging 3D hand posture information. By integrating inputs from forearm- worrn worn sEMG sensors with 3D hand pose information derived from an RGB image, we observe improvements in the accuracy of hand pressure estimation. Our multimodal approach (Figure 2) leverages the strengths of both hand posture and muscle activations, offering comprehensive understanding of hand dynamics for whole-hand pressure estimation. Refer to Section C for more detailed information about the model architecture, training, and inference.
4.2
3D Hand Pose and sEMG Feature Extractions
To verifyy the validity of our framework, devise a deep neural network model to effectively utilize the obtained multi- muulti-modlittis.. We represent the model as f. where the 3D hand pose is denoted by Il and the sEMG signal by E. The classification and regression targets for pressure are represented as C and P. respecivelly. the model outputs for pressure classification and regression are indicated, they are denoted by Ć and P. The feature extractor for sEMG data is fEMG. and for hand pose, is fhand- The overall model comprises JEMG, fhand, and pressure predictor fpeed that takes the features extracted from hand pose and sEMG to perform pressure classification and regression.
Feature Extraction from sEMG signals. The sEMG data generally encounters measurement noises, including powerline noise and electromagnetic artifacts. To mitigate these issues, we utilize the short- time Fourier transform (STFT) to convert sEMG time-domain signals into spectrograms, represented
5

---

as E E R8x32x64 isolating high-frequency noise and facilitating the application of convolutional neural network (CNN) models for feature extraction. We employed 2D mncoderr model to extract features from the 21) sEMG signals. The encoder-decoder model processes the data and then flattens the output, which subsequently transformed through a fully connected (FC) layer into a 51 512-dimensional feature vector.
Feature Extraction from 3D Hand Pose. The hand model in our study is represented kinematic tree with 15 joint angles 0 E R 15133, similar to the pose parameters in MANO and its variants [62- 65J. To handle this skeleton-based representation simply like PoseConv3D [66], we adopt 3D ResNet [67] to process 3D heatmap volumes of hand joints, transforming 0 into 21 3D hand joints J 222 through the hand skeleton model's forward kinematics. These 3D hand joints are then converted into 3D heatmap volumes H [o, 1]21 x H xD, with H. W. and D all set to 48, for processing by the 3D ResNet. Unlike the sequential representation of 2D heatmaps in PoseConv3D, our approach uses a single timestep 3D joint representation. After processing through the 3D ResNet, the hand pose feature is flattened and transformed into a 51 512-dimensional vector using an FC layer, with the 3D ResNet34 model being the model of choice for this operation.
Feature Fusion and Estimation. The extracted sEMG feature fEMG (E) and hand pose feature faad H) are concatenated to form a 1024-dimensional joint feature vector. This vector is then passed through two FC layers, mapping 256-dimensional features, followed by a 1-D batch normalization and ReLU non-linearity. Finally, the last FC layer with Sigmoid activation function maps this to T-dimensional output, producing C' = f(E, H) = fpeea (EEMGEE)))))) (E), fhana ()) E [0, 1]. The predicted pressure P is defined P = 2P max ReLUU( 0.5) E [0, P P. thus completing the feature fusion and pressure prediction.
4.3 Joint Training of Classification and Regression
The objective function comprises two key components: a classification loss Le and a regression loss Ly. The classification loss is designed to accurately identify whether any pressure is exerted by a particular region of the hand (i.e.. fingertips or palm areas shown in Figure 5 of Section B.1.1), using a cross-entropy loss to distinguish between pressure and no-pressure instances for each hand region i:
L== = Σ a log C. + (1 C4) log(11 - -G).
(1)
where C. is the ground-truth label for region i, and C₁ is the predicted probability of pressure application. Here, C2 = indicates the presence of pressure in the ith region, while C, = indicates its absence. In contrast, the regression loss, targets the accurate quantification of pressure levels using an Ly loss to minimize the difference between the predicted and actual pressure values:
1
L 1
Σ IA P.122
where P. represents the model's 's predicted pressure for region and P. is the corresponding actual pressure. Our dataset contains pressures from 0~20 N. Therefore, our model predicts pressure values in [0. 20), organized by Pmaz. the maximum of the pressure. To integrate these two aspects into a unified training objective, we introduce balancing hyper-parameter X, resulting in combined loss function: L L. + X L,. This composite loss enables our model to not only discern the presence of pressure but also quantify its magnitude accurately.
4.4 Estimation without the Glove
For the training phase, we employed data acquired from our data collection system to ensure the accurate capture of exerted hand pressure and 3D hand pose data. However, during the inference phase, our framework exploits off-the-shelf hand pose detectors [68-70], which extract 3D hand pose from RGB or RGB+D inputs. These detectors can been chosen for their high accuracy and robustness in various conditions, ensuring reliable performance during inference. Thus, users can interact with external objects using their bare hands, without the need for additional hand-worn equipment. This approach ensures our model's 's practical applicability in real-world scenarios, prioritizing user convenience and natural interaction. By leveraging readily available technology, we make it easier

---

Table 2: Performance among comparative models on evaluation metrics.
Method
NRMSE
Accuracy
sEMG Only [4]
83.49 + 16.40%
8.07 + 2.62%
77.83 + 11.56%
3D Hand Posture Only
66.32 37.01%
11.57 +: 3.95
70.08 + 13.09%
sEMG Hand Angles
84,22 + 17.11%
7.89 + 2.61%
78.22 = 10.57%
PiMForce (Ours)
88.86 11.92%
6.65 2.11%
83.17 9.38%
for users to adopt our system in everyday applications. Refer to Section B.4 for details on how we canonicalized 3D hand pose information extracted from RGB images for our model input.
5 Experiments
In Sections 5.2.1 and 5.2.2, where ground truth hand pressure is necessary, data was collected while participants wore the pressure glove, and hand postures were obtained from the data glove. We also conducted qualitative evaluations (Section 5.2.3 and the demo video) without ground truth, where data collected without any gloves, and hand postures were inferred solely from RGB images using an off-the-shelf hand pose detector. For this purpose, we employed the pre-trained Attention Collaboration-banl -based Regressor [69], which has demonstrated superior performance with mean per joint position error (MPJPE) of approximately 8mm for reconstructing hand poses from single RGB camera. The high accuracy ensures the reliability of our hand posture inferences in qualitative assessments. To assess our model's 's performance, we utilize three metrics: Coefficient of Determination (R*), Normalized Root Mean Squared Error (NRMSE), and classification accuracy. The exact definitions and explanations of evaluation metrics can be found in Section D.2. Refer to Section D.3 and D.4 for additional quantitative and qualitative restuls, respectively.
5.1 Comparative Methods
This study compares the proposed model against several baseline and state-of-the-art methods to validate its effectiveness in whole-hand pressure estimation. To ensure a fair comparison, we selected methods that quantitatively measure the pressure applied by the hand, rather than solely identifying hand contact. Detailed information about the implementation of comparative methods can be found in Section 1. The methods included in the comparison are:
sEMG Only Model [4]. An JEMG-based approach decodes finger-wise wise forces in real-time, demon- strating the potential of muscle activation patterns in informing hand activities. This method em- phasizes using electromyography sensors for understanding complex hand dynamics but does not incorporate hand posture information.
3D Hand Posture Only Model. A variation of our proposed framework that solely utilizes 3D hand posture for pressure estimation, omitting the sEMG signal input. This model tests the efficacy of hand posture information in isolation.
SEMG + Hand Angles Model. This model represents a variation of our proposed framework, where instead of utilizing the 3D representation H for hand pose, employs the angular representation 0 of hand joints as the input to the hand pose feature extractor fuand. By substituting the 3D hand pose with direct angle measurements of hand joints, this baseline aims to highlight the benefits of using 3D representation for hand pose in multimodal sensing.
Pressure Vision+ [17]. This iision-based deep learning model estimates hand pressure from single RGB image by identifying visual cues related to hand pressure application, showcasing the use of visual information for pressure estimation without physical FSR sensors.
PiMForce (Ours). The comprehensive model enhances sEMG signals by leveraging 3D hand posture information for continuous and detailed pressure estimation across the whole hand. This approach aims to mitigate the limitations of sEMG-based methods by integrating the strengths of both modalities for enhanced pressure prediction accuracy.
7

---

Table 3: Cross-user performance on evaluation metries under whole interaction and posture.
Method
R²
NRMSE
MAE
Accuracy
sEMG only [4]
90.90-2 + 9.97%
14.14
2.41%
12.24
2.75%
57.40 =: 5.81%
PiMForce (Ours)
70.06 + 4.025:
10.70 1.4%%
8.54 + 1,56%
72.01 + 2.86%
5.2 Results
We analyze the performance of our proposed framework in comparison to these methodologies, both quantitatively and qualitatively. Additionally, we investigate the capabilities of our model to accurately estimate hand pressure across variety of hand postures and parts, providing a thorough assessment of its performance. Our framework enhances sEMG signals by leveraging 3D hand posture information for detailed palm pressure data collection, contrasting with Pressure Vision++, which relies on visual cues for force estimation. This approach is designed underscore the distinctive benefits of our multimodal sensing framework in capturing a broad range of hand interactions.
5.2.1 Do hand pose and sEMG signals together improve pressure estimation?
Table 2 outlines the performance metrics of various comparative models, including the sEMG Only Model, the 3D Hand Posture Only Model, the sEMG + Hand Angles model, and our model. PiMForce remarkably outperforms the comparative methods, achieving an accuracy of 83.17% NRMSE of 6.65%, and an R² value of 88.86%. This demonstrates the comprehensive capability of our model to accurately classify and quantify the pressures exerted by the hand.
The integration of 3D hand posture and sEMG information in our framework shows clear advantage over approaches relying on a single data modality, as expected. The sEMG Only Model and the 3D Hand Posture Only Model show limited pressure estimation performance when compared to our integrated approach. Interestingly. the improvement in performance with the sEMG + Hand Angles model over the sEMG Only Model is marginal (less than 0.5%p improvements in all metrics). This highlights the importance of incorporating comprehensive 3D hand posture representation. By embedding comprehensive hand posture knowledge to be used with sEMG data, we develop an effective multimodal approach to capture nuanced variations in hand pressure exerted across different hand regions and postures.
Cross-user performance assesses how well the model performs on data from individuals not included in the training set, which is crucial for real-world applications. As shown in Table 3, our proposed method combining sEMG signals with 3D hand posture data significantly outperforms the sEMG- only baseline across all evaluation metrics in cross-user scenarios. This demonstrates the enhanced generalizability and effectiveness of our approach in estimating hand pressure among different users. To further demonstrate the performance of our model over time, we present Figure 23 in supplementary material, which illustrates the temporal evolution of both ground truth and predicted pressure values for all nine hand regions during consecutive TM-Press and Medium Wrap actions.
5.2.2 How does accuracy vary by hand region and posture type?
We delve into the performance of our model across various hand regions and posture types, utiliz- ing data represented in both Table 4 and Figure 3. This analysis highlights the noticeable impact of incorporating 3D hand pose data, particularly noting a greater improvement in hand palm re- gions (+1.95%p) over fingertips (+1 100%%)) compared to the sEMG Only Model. This distinction emphasizes the crucial role of 3D hand pose for accurate pressure estimation in diverse hand postures.
Our findings reveal that the model achieves superior pressure estimation in Press and Pinch inter- actions, with classification accuracies surpassing 90% and NRMSE values maintained below 6%. However, it encounters challenges with specific postures such as Palm-Press, which, despite a lower classification accuracy of 68. (8.42% still shows high regression accuracy of 3. 3.%. When examining Grasp postures, our model shows slight dip in performance relative to Press and Pinch, with NRMSE values ranging between 5~8%. This suggests a moderate pressure estimation capability for these more complex interactions, yet the model consistently maintains a high R2 range of 0.8 to 0.9 across all posture types. This consistent correlation between predicted values and actual pressure measurements highlights the model's ability to maintain high accuracy and reliability across a diverse
8

---

Table Performance comparison of hand regions in terms of NRMSE.
Finger Tip
Hand Palm
Method
Thamb
Tadix
Middle
Ring
Pinky
Moan
Upper Right
Upper Left
Lower Right
Lower Left
Mean
Overall
Moan
sEMG Only [+]
9:78
1,48%
9.26
3.51%
8.15 a:
3,17%
6.65 2:
2.51%
4,09 2:
1.32%
7.21 =:
2.79%
8.79 =
3,88%
3,43 +:
3.765
8.06 a:
3.78%
7.79
3.98%
8,42
3.85%
8,07
3.26%
3D Hand Posture Ouly
13.85
5,39%
13.64 1
5,399
12.18 1
5.31%
9.85
4.49%
6.16 1
1.73%
11.13
4.55%
12.26
6.09%
10.41 1
6,30%
11.47
6.0256
9.71
0.30%
10:96 1
6.18%
11:05 1
5.27%
AEMG- Hund Angles
9:54
9.04
3.50%
7:95
3.21%
6.48 +t
2.48%
4.57 +
1,37%
7.52 +
2.82%
8.56 +
3.90%
7:23 +
4,14%
7,67
4.22%
6.67
4.21%
7:53
7:50
3.40%
PiMForce (Ours)
8.04
1,73
6,01
6.66
7.23
3.12%
6.17
3.32%
6.76
Accuracy (%)
NRMSE (%)
100
10
80
60
40
20
HPress
IM-Press
TM-Pinch
Palmar Pinch
Ring
Parallel Extension
M-Press
PRePs
TIM-Pinch
Medium Wrap
Adducted Thumb Quadpad
R Press
Palm Press
TIMR Pinch
Sphere 3 finger
Foxed Hook
P-Press
TI-Pinch
TIMRP-Pinch
Power Sphere
Figure 3: Quantitative evaluation of the user-independent model, showing the posture- wise perfor- mance on the estimation of hand pressure. 'he error bars indicate standard error.
range of hand parts and postures. Specific actions such as I-Press, M-Press, and R-Press exhibit high accuracy and low NRMSE, showing the model's superior performance in simpler press interactions. On the contrary, more complex grasps like Power Sphere, Fixed Hook, and Parallel Extension, showed lower accuracy and higher NRMSE, implying that there remains room for improvement.
5.2.3 Can inference succeed with an off-the-shelf hand pose detector?
For practical inference applications without a data glove, solely relying on EMG data and incorporat- ing 3D hand pose information obtained through an off- sffeff hand pose detector, we demonstrate the adaptability of our model in Figure 4a. This comparison with the vision-based method, Pressure- Vision++, showcases our PiMForce's 's capability estimate hand pressures robustly during diverse interactions. During hand-plane interactions, specifically those involving the tip Press type posture, both approaches appear perform well. However, our analysis reveals vulnerabilities in handling more complex Grasp and Pinch motions when using Pressure Vision++. Furthermore, Pressure Vi- sion++ requires complete visibility of all fingers within the camera's 's view due the high reliance on visual cues for pressure inference. In contrast, our framework effectively utilizes the estimated hand pose as long as the hand pose information is sufficiently accurate for inference. This capability underscores the practicality of our method, facilitating more natural user interactions with external objects without the constraints direct visibility or glove use. Figure 4b shows demo video footage illustrating our PiMForce's 's capability to accurately estimate hand pressure while continuously chang- ing hand posture, pressure levels, and the objects being grasped. This demonstrates the flexibility and reliability of approach in real-world scenarios.
To further substantiate our model's effectiveness using an off-the-shelf hand pose detector, we conducted a quantitative comparison with PressureVisiont+: demonstrates largely better performance across all fingertips during both plane and pinch interactions, indicating superior performance in estimating hand pressures compared to Pressure Vision++. This quantitative evaluation confirms that our framework outperforms existing vision- based methods in terms of accuracy and robustness during diverse interactions.
presented in Table 5. PiMForce

---

Hand Type
Input Image
[Rendered]
TIMR-
Untletedaalle
Medium Wrap
Quadpod
Und
(b)
Figure (a) Qualitative results in the absence pressure glove. The 3D Hand Pose Estimation [69] represents 3D hand posture, including hand occlusion, using the 3D hand detector. The Pressure- Vision++ [28] column shows the pressure estimation of fingertips. The red rectangles indicate the instances of pressure estimation failure due to hand occlusion. The proposed multimodal framework shows robust whole-hand pressure estimation for diverse hand-object interactions. (b) Illustration of the demo video footage showing robust hand pressure estimation with varying hand postures, pressure levels, and interacting objects.
Table 5: Cross-user performance on evaluation metrics under plane interaction and pinch posture.
Method
R²
NRMSE
Accuracy
Pressure Vision++ [17]
40.30 + 5.14%
32.95 + 2.02%
67.90 3.01%
sEMG only [4]
42.13 1 6.88%
12.57 2.09%
5.84%
PiMForce (Ours)
66.71 + 4.68%
9.27 1.40%
82.20 2.42%
Conclusion
In this paper, we introduce PiMForce, pioneering framework for hand pressure estimation by integrating 3D hand posture information with musele activation signals from forearm-wom sEMG. By embedding 3D hand posture information into a deep neural network, we enable the model to process this data alongside sEMG signals, enhancing capability to learn complex relationships between muscle activations and hand pressure distributions. This novel approach is the first to combine these modalities, providing comprehensive analysis of hand dynamics across various interactions. We developed unique multimodal hand data collection system and protocol, capturing a dataset that includes hand pressure, posture, and electromyography signals. Our method notably improves upon previous techniques, enabling accurate whole-hand pressure estimation through detailed hand posture information. Extensive quantitative and qualitative comparisons demonstrated the consistent superiority of our framework over existing JEMG-based and vision-based methods.
10

---

Acknowledgments and Disclosure of Funding
This work supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. 2022R1 A4A5033689, Contribution Rate: 50% and No. 00210001, Contribution Rate: 25%) and Culture, Sports and Tourism R&D Program through the Korea Cre- ative Content Agency grant funded by the Ministry of Culture, Sports and Tourism in 2023 and 2024 (Project Name: Development of Real-time Virtual Convergence-basee
Performing Arts Educa- tion Platform Technology, Project Number: RS-2023-0021900 190000 Contribution Rate: 15%, Project Name: International Collaborative Research and Global Talent Development for the Development of Copyright Management and Protection Technologies for Generative AI, Project Number: RS-2024- 00345025, Contribution Rate: 10%).
Bibliography
[1] Taichi Tsuchida, Kazuyuki Fujita, Kaori Ikematsu, Sayan Sarcar, Kazuki Takashima, and Yoshifumi Kitamura. Tetraforce: magnetic-basse based interface enabling pressure force and shear force input applied to front and back a smartphone. Proceedings of the ACM on Human Human-Compun Computer Interaction, 6(ISS):
185 185-206, 2022.
[2] Julie Duque, D Masset, and Jacques Malchaire. Evaluation of handgrip force from emg measurements. Applied ergonomics, 26(1):61-66, 1995. 1
[3] Xianta Jiang, Lukas-Karim Merhi, and Carlo Menon. Force exertion affects grasp classification using force myography. JEEE Transactions on Human- Machine Systems, 48(2):219-2 226, 2017. 1
[4] Yunxiang Zhang, Benjamin Liang, Boyuan Chen, Paul M Torrens, S Farokh Atashzar, Dahua Lin, and Qi Sun. Force-aware interface via elcctromyography for natural vi/ar interaction. ACM Transactions on Graphics (TOG), (16):118( 2022. 1, 3, 4, 7, 9, 20, 23
[5] Sang Ho Yoon, Siyuan Ma. Woo Suk Lee, Shantanu Thakurdesai, Di Sun, Flávio P Ribeiro, and James D Holbery. Hapsense: A soft haptic i/o device with uninterrupted dual functionalities of foree sensing and vibrotactile actuation. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology, pages 949 10119996,1220 2019.
[6] Stephen Bao and Barbara Silverstein. Estimation of hand force in ergonomic job evaluations. Er- gonomicx, 48(3):288-301, 2005. doi: 10. 0530000144033342003322222 URL https://doi.orgg 1000/00014013042000327777
PMID: 15764327. 1
[7] Nagisa Matsumoto, Koji Fujita, and Yuta Sugiura. Estimation of grip strength using monocular camera for home-based hand rehabilitation. SICE Journal Control, Measurement, and System Integration, 14 (1):1-11, 2021, doi: 10. 00000/8824889.0000 2020. 1863612. 2020. 186361.
URL https: httt:s:://ooooo/////// org/10 10. 1080/ 18824889.
[8] Bin Fang, Chengyin Wang, Fuchun Sun, Ziming Chen, Jianhua Shan, Huaping Liu, Wenlong Ding, and Wenyuan Liang. Simultaneous semg recognition of gestures and force levels for interaction with prosthetic hand. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 30:2426-2222 2436, 2022. 1,3
[9] Josie Hughes, Andrew Spielberg, Mark Chounlakone, Gloria Chang, Wojeiech Matusik, and Daniela Rus. A simple, inexpensive, wearable glove with hybrid resistive-reessure sensors for computational sensing, proprioception, and lask identification. Advanced Intelligent Systems, 2(6):200000,, 2020.
[10] Yiyue Luo, Yunzhu Li, Pratyusha Sharma, Wan Shou, Kui Wu, Michael Foshey, Beichen Li. Tomás Palacios, Antonio Torralba, and Wojeiech Matusik. Learning human-environment interactions using conformal tactile textiles. Nature Electronics, 4(3):193-201, 2021.
[11] Subramanian Sundaram, Petr Kellnhofer, Yunzhu Li, Jun-Ya 7hu, Antonio Torralba, and Wojciech Matusik. Learning the signatures of the human grasp using scalable tactile glove. Nature, 569(7758): 698 708-702,40 2019. 1
[12] Claudia Cutler, Anita L Hamilton, Emma Hough, Cheyenne M Baines, and Ross A Clark. Open- 3d printed sensors for hand strength assessment: Validation of low-cost load cell and fabric sensor-hased systems. Australian occupational therapy journal, 65(5):412-419, 2018.
[13] Julius Cosmo Romeo Rudolph, David Holman, Bruno De Araujo, Ricardo Jota, Daniel Wigdor, and Valkyrie Savage. Sensing hand interactions with everyday objects by profiling wrist topography. In Sixteenth International Conference on Tangible, Embedded, and Embodied Interaction, pages 1 14, 2022.
11

---

[14] Artem Dementyev and Joseph A Paradiso. Wristflex: low-power gesture input with wrist- wort pressure sensors. In Proceedings of the 27th annual ACM symposium User interface software and technology. pages 161-166, 2014.
[15] Tu-Hoa Pham, Nikolaos Kyriazis, Antonis A Argyros, and Abderrahmane Kheddar. Hand-object contact force estimation from markerless visual tracking. IEEE transactions on pattern analysis and machine intelligence, 4002))2888 2896, 2017. 2
[16] Navid Fallahinia and Stephen A Mascaro. Real-time tactile grasp force sensing using fingernail imaging via deep neural networks. IEEE Robotics and Automation Letters, 7(3):6558-655) 2022.
[17] Patrick Grady, Jeremy A Collins, Chengeheng Tang, Christopher D Twigg, Kunal Aneja, James Hays, and Charles C Kemp. Pressurevision+*: Estimating fingertip pressure from diverse rgb images. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 8698-8708, 2024. 3, 7
[18] Kiana Ehsani, Shubham Tulsiani, Saurabh Gupta, Ali Farhadi, and Abhinav Gupta. Use the force, Luke! Learning to predict physical forces by simulating effects. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 224 223, 2020. 2
[19] Zongmian Li, Jirí Sedlár. Justin Carpentier, Ivan Laptev, Nicolas Mansard, and Josef Sivic. Estimating 3d motion and forces of person-object interactions from monocular video. CVPR, pages 8640-8649, 2019.
[20] Tu-Hoa Pham, Abderrahmane Kheddar, Ammar Qammaz, and Antonis A Argyros. Towards force sensing from vision: Observing hand-object interactions to infer manipulation forces. In Proceedings of the IEEE conference computer vision and pattern recognition, pages
2015. 2
[21] Nutan Chen, Göran Westling, Benoni Edin, and Patrick van der Smagt. Estimating fingertip forces, torques, and local curvatures from fingernail images. Rohotica, 38(7):1242-1266 2020.
[22] Stephen A. Mascaro and H. Harry Asada. Mcasurement of finger posture and three-axis fingertip touch force using fingernail sensors. IEEE Transactions on Robotics and Automation, 20(1):26-33) 35, 2004. 2
[23] Elliott Donlon, Siyuan Dong, Melody Liu, Jianhua Li, Edward Adelson, and Alberto Rodriguez. Gelslim: A high-resolution, compact, robust, and calibrated lactile-sensing finger. 2018 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1927-1934. IEEE, 2018. 2
[24] Mike Lambeta, Pe-Wei Chou, Stephen Tian, Brian Yang, Benjamin Maloon, Victoria Rose Most, Dave Stroud, Raymond Santos, Ahmad Byagowi, Gregg Kammerer, et al. Digit: A novel design for low-cost compact high-resolution tactile sensor with application to in-hand manipulation. IEEE Robotics and Automation Letters, 5(3:33838-8855, 2020. 2
[25] Helen II Hu, Amy A Gooch, William B Thompson, Brian E Smits, John J Rieser, and Peter Shirley. Visual eues for imminent object contact in realistic virtual environments. In Proceedings Visualization 2000., pages 179-185. 185. IEEE, 2000, 2
[26] Helen H Hu, Amy A Gooch, Sarah H Creem Beeem, and William B Thompson. Visual cues for perceiving distances from objects to surfaces. Presence: Teleoperators & Virtual Environments, 1 (1())652-664, 2002.
[27] Geoffrey S. Hubona, Philip N. Wheeler, Gregory W. Shirah, and Matthew Brandt. The relative contributions of stereo, lighting, and background scenes in promoting 3d depth visualization. ACM Transactions on Computer Humam Interaction, 6(3):214-24)) 242, 1999. 2
[28] Patrick Grady, Chengeheng Tang, Samarth Brahmbhatt, Christopher D Twigg, Chengde Wan, James Hays, and Charles C Kemp. Pressurevision: estimating hand pressure from a single rgb image. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part VI. pages 328 345. Springer, 2022. 2,
[29] Yiming Zhao, Taein Kwon, Paul Streli, Mare Pollefeys, and Christian Holz. Egopressure: A dataset for hand pressure and pose estimation in egocentric vision. arXiv preprint arXiv:2409, 02224, 2024. 2
[30] Ning Jiang, Deborah Falla, Andrea d' Avella, Berhard Graimann, and Dario Farina. Myoelectric control in neurorchablitation. Critical ReviewyTM Biomedical Engineering, 38(4), 2010. 3
[31] T Scott Saponas, Desney S Tan, Dan Morris, and Ravin Balakrishnan. Demonstrating the feasibility of using forearm electromyography for muscle-computer interfaces. In Proceedings of the SIGCHI Conference Human Factors in Computing Systems, pages 315-524, 2008.
12

---

[32] Betzalel Fialkoff. Harel Hadad, Darío Santos, Franco Simini, and Marcelo David. Hand grip force estimation via emg imaging. Biomedical Signal Processing and Control, 74:103550, 2022. ISSN 1746- 8094. doi: https://do..ogg/0000166/...p
2022. URL atttps:/wwet http:s//nn..eddeeeeerrrttttt com/
sclence/ cckkac///arrllc/////////44440200
3
[33] Itzel Jared Rodríguez Martínez, Andrea Mannini, Francesco Clemente, and Christian Cipriani. Online grasp force estimation from the transient emg. IEEE Transactions on Neural Systems and Rehahilitation Fingineering, 28(1022333-2341, 2020. doi: 10.1 109//TNSSEE.222000225555
[34] F. Mobasser and K. Hashtrudi-Zasse Hand force estimation using electromyographhy signals. Proceedings of the 2005 IEEE International Conference on Roboticx and Automation, pages 2631-2636, 2005. doi: 10.1 100
[35] Changcheng Wu, Hong Zeng. Aiguo Song, and Baoguo Xu. Grip force and 3d push-pull force estimation based on semg and gmn. Frontiers Neuroscience, 11, 2017. ISSN 1662-453X. doi: 10.3389ffnnns.200 2017. 00343. URL https: //wp://wwww.ooo http://w.rroot..oooo wwwootrrrrsstr.ee..... org/ artiieettiieseeeeeeeeeee 10. 3389/f3191nnn
00343.
[36] Changeheng Wu, Qingqing Cao, Fei Fei, Dehua Yang. Baoguo Xu, Guanglie Zhang, Hong Zeng. and Aiguo Song. Optimal strategy of semg feature and measurement position for grasp force estimation. PLOS ONE, 16(3))1-21, 03 2021. doi: 10. wwwouumallpooe.024477777 URL httpp:://doi.rrr 10. 1 journal.pone 0247883. 3
[37] Berj Bardizbanian, Jennifer Keating, Xinming Huang, and Edward A. Clancy. Estimating individual and combined fingertip forces from forearm emg during constant-pose, force-varying tasks. In 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pages 3134 3137, 2020. doi:
3
[38] Pu Liu, Donald R. Brown, Edward A. Clancy, Francois Martel, and Denis Rancourt. Emg-force estimation for multiple fingers. In 2013 IEEF Signal Processing in Medicine and Biology Symposium (SPMB), pages 1-6, 2013. doi: 10. 1...109/SMMBB.CM 201..01136666 3.6766777.
[39] He Mao, Peng Fang, and Guanglin Li. Simultancous estimation of multi-finger forces by surface clec- tromyography and accelerometry signals. Biomedical Signal Processing and Control, 70:103005, 2021. ISSN 1746-8094, doi: htgp:://doi.og/// 1r..ooo1.rrr.100011.....pp. bttp::.0........2.......... 2030000 URL https: //wp:///wwwwww... www.ewaciinceeirrcctt........... com./scieece/articcl htteee/pi///aaeeeeeeeeeee ////.///177440942000000529
[40] Arinobu Nújjima and Yuki Kubo. Assisting with voluntary pinch force control by using electrical muscle stimulation and active bio- acoustic sensing. Adjunct Proceedings of the 34th Annual ACM Symposium on User Interface Software and Technology, pages 11-13, 2021.
[41] Fengze Zhang, Yunxiang Zhang, Xi Peng, Sky Achitoff, Paul M Torrens, and Qi Sun. May the force be with you: Dexterous finger force-aware interface. In 2024 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE, 2024. 3
[42] Francesca Leone, Cosimo Gentile, Anna Lisa Ciancio, Emanuele Gruppioni, Angelo Davalli, Rinaldo Sacchetti, Eugenio Guglielmelli, and Loredana Zollo. Simultancous semg classification of hand/wrist gestures and forces. Frontiers in neurorohotics, 13:42, 2019. 3
[43] Ethan Eddy, Erik Scheme, and Scott Bateman. A framework and call to action for the future development of emg-based input in hci. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1-23, 2023. 3, 5, 19
[44] Yu- Wei Chao, Wei Yang. Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S Narang. Karl Van Wyk, Umar lqbal, Stan Birchfield, et al. Dexycb: A benchmark for capturing hand grasping of objects. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9044-9053, 2021. 3,4 4
[45] Shreyas Hampali, Mahdi Rad, Markus Oherweger, and Vincent Lepetit. HOnnotate: A method for 3D annotation of hand and object poses. In CVPR, pages 3196-3206, 2020. 3,4
[46] Omid Taheri, Nima Ghorbani, Michael J. Black, and Dimitrios Tzionas, GRAB: A dataset of whole-body human grasping objects. ECCV, 2020.
[47] Lixin Yang. Kailin Li, Xinyu Zhan, Fei Wu, Anran Xu, Liu Liu, and Cewu Lu. Oakink: A large-scale knowledge repository for understanding hand-object interaction. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20953 20962,, 2022. 3
13

---

[48] Nebojša Malešević, Alexander Olsson. Paulina Sager, Elin Andersson, Christian Cipriani, Marco Controzzi, Anders Björkman, and Christian Antfolk. A database of high-density surface electromyogram signals comprising 65 isometric hand gestures. Scientific Data, 8(1):63, 2021. 3, 3.37
[49] Joseph DelPreto, Chao Liu, Yiyue Luo, Michael Foshey, Yunzhu Li, Antonio Toralla, Wojeiech Matusik, and Daniela Rus. Actionsense: multimodal dataset and recording framework for human activities using wearable sensors in a kitchen environment. Advances in Neural Information Processing Systems, 35: 13800-
18800-13813 13813, 13, 2022. 3
[50] Samarth Brahmbhatt, Chengeheng Tang. Christopher D. Twigg, Charles C. Kemp, and James Hays. ContactPose: A dataset of grasps with object contact and hand pose. In ECCV, August 2020. 3
[51] Takehiko Ohkawa, Ryosuke Furuta, and Yoichi Sato. Efficient annotation and learning for 3d hand pose estimation: A survey. International Journal of Computer Vision, (1111:11133320000 2023. 4
[52] Tomas Simon, Hanbyul Joo, Iain Matthews, and Yaser Sheikh. Hand keypoint detection in single images using multiview bootstrapping. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. pages 1145 1153, 2017.
[53] Gycongsik Moon, Shoou-I Yu, He Wen, Takaaki Shiratori, and Kyoung Mu Lec. Interhand2. 6m: A dataset and baseline for 3d interacting hand pose estimation from a single rgb image. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XX 16, pages 548-564. Springer, 2020. 4
[54] Christian Zimmermann, Duygu Ceylan, Jimei Yang. Bryan Russell, Max Argus, and Thomas Brox Freihand: A dataset for markerless capture of hand pose and shape from single rgb images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 813-822, 2019. 4
[55] Thomas Feix, Javier Romero, deinz-Bodo Schmiedmayer, Aaron M Dollar, and Danica Kragic. The grasp taxonomy of human grasp types. IEEE Transactions on manaine systems, 46(1):66 77, 2015. 5, 16, 37
[56] Baharch Abbasi, Ehsan Noohi, Sina Parastegari, and Miloš Zefran. Grasp laxonomy based on force distribution. In 2016 25th IEEE International Symposium on Robot and Haman Inteructive Communication (RO-MAN), pages 1198-1103. IEEE, 2016. 5, 17,
[57] Ctrl labs at Reality Labs, David Sussillo, Patrick Kaifosh, and Thomas Reardon. A generic noninvasive neuromotor interface for human-computer interaction. bioRxiv, pages 2024-02, 2024. 5
[58] W Melzer, E Rios, and MF Schneider. Time course of calcium release and removal in skeletal musele fibers. Biophysical journal, 45(3))633--44) 1984.
[59] W Melzer, A Hermann-Frank, and H Ch I. Lüttgau. The role of ca2+ ions in excitation-coomrrctioo coupling of skeletal muscle fibres. Biochimica et Biophysica Acta (BBA)-Reviewx on Biomembranes, 1241(1): 59-116, 1995.
[60] Marco JM Hoozemans and Jaap H Van Dieen. Prediction of handgrip forces using surface emg of forearm muscles. Journal of electromyography and kinexiology, 15(4)3358-366, 2005. 5
[61] JL McNiu-Gray, DME Hester, Mathiyakom, and BA Munkasy. Mechanical demand and multijoint control during landing depend on orientation of the body segments relative to the reaction force. Journal of biomechanics, 34(1 11::::47111111 2001. 5
[62] Javier Romero, Dimitrios Tzionas, and Michael J. Black. Emhodied hands: modeling and capturing hands and bodies together. ACM Trans. Graph. 36(6), nov 2017. ISSN 0730-0301. doi: 10. 01115//3300000 3130883. URL https: /ttttp:://do.coo org/10 10. 1 1141/ /3100003000000 3130883. 6
[63] Adrian Spurr, Umar Iqbal, Pavlo Molchanov, Otmar Hilliges, and Jan Kautz. Weakly supervised 3d hand pose estimation via biomechanical constraints. In European conference computer vision, pages 21 1-228.. Springer, 2020.
[64] Lixin Yang, Xinyu Zhan, Kailin Li, Wenqiang Xu, Jiefeng Li, and Cewu Lu. Cpf: Learning contact potential field to model the hand-object interaction. In Proceedings of the IEFE/CVE International Conference on Computer Vision, pages 1097-11106, 2021.
[65] Lixin Yang. Kailin Li, Xinyu Zhan, Jun Lv, Wenqiang Xu, Jiefeng Li, and Cewu Lu. Artiboost: Boosting articulated 3d hand-object pose estimation via online exploration and synthesis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2750 2760, 2022. 6
14

---

[66] Haodong Duan, Yue Zhao, Kai Chen. Dahua Lin, and Bo Dai. Revisiting skeleton- action recognition. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2969-
2978, 2022.
[67] Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh. Can spatiotemporal 3d cnns retrace the history of 2d enns and imagenet? In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 6546-666 (166-6555222200 2018. 6, 22
[68] Shreyas Hampali, Sayan Deb Sarkar, Mahdi Rad, and Vincent Lepetit. Keypoint transformer: Solving joint identification in challenging hands and object interactions for accurate 3d pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2022. 6
[69] Zhengdi Yu, Shaoli Huang, Chea Fang. Toby P Breckon, and Jue Wang. Acr: Attention collaboration- based regressor for arbitrary two-hand reconstruction. In Proceedings of the /EEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1295--12944, 2023, 7, 10
[70] Fan Zhang, Valentin Bazarevsky, Andrey Vakunov, Andrei Tkachenka, George Sung, Chuo-Ling Chang. and Matthias Grundmann. Mediapipe hands: On-device real-time hand tracking. arXiv preprint arXiv:2006. 102 14, 2020. 6
[71] Thilina Dulantha Lalitharatne, Yoshikai Hayashi, Kenbu Teramoto, and Kazuo Kiguchi. A study effects of muscle fatigue on emg- based control for human upper-lime power-assist. 2012 IEEE 6th International Conference on Information and Automation for Sustainability, pages 124-128. IEEE, 2012. 18
[72] Esteve Peña Pitarch. Virtual human hand: Grasping strategy and simulation. Universitat Politècnica de Catalunya, 2008, 20
[73] Javier Romero, Dimitrios Tzionas, and Michael J. Black. Embodied hands: Modeling and capturing hands and bodies together. ACM Transactions on Graphics, (Proc. SIGGRAPH Asia), November 2017, URI. http:/ / //oii/////./// acm. wwemooog//.... 10. 144//3300000000 3130883. 20
[74] John Lin, Ying Wu, and Thomas $ Huang. Modeling the constraints of human hand motion. In Proceedings workshop on human motion, pages 121-126 IEEE, 2000. 21
[75] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pottern recognition, pages 770-778, 2016.
22
(76] Christoph Feichtenhafer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEFE/CVF international conference on computer vision, pages 6202- 6211, 2019. 22
[77] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference for Learning Representations, 2015. 23
[78] Christoph Amma, Thomas Krings, Jonas Böer, and Tanja Schultz. Advancing muscle-computer interfaces with high-density cloctromyography. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pages 929-938, 2015. 37
[79] Yufei Ye, Abhinav Gupta, and Shubham Tulsiani. What's in your hands? 3d reconstruction of generic objects in hands. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 3895-3905, 2022. 37
[80] Zerui Chen, Shizhe Chen, Cordelia Schmid, and Ivan Laptev. gsdf: Geometry-driven signed distance functions for 3d hand-object reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12890-12900, 2023.
[81] Hao Xu, Haipeng Li, Yinqiao Wang, Shuaicheng Liu, and Chi-Wing Fu. Handbooster: Boosting 3d hand- mesh reconstruction by conditional synthesis and sampling of hand-object interactions. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. 37
[82] Richard Drake, A Wayne Vogl, and Adam WM Mitchell. Gray's 's anatomy for students F-book. Elsevier Health Sciences, 2009. 37
15

---

A Overall Structure of Supplementary Material
This supplementary material provides added details to complement the main manuscript. Section elaborates the specifics of the multimodal dataset creation. Section C offers an extended description of the methodological framework employed in our study. Section D describes implementation details of comparative methods and showcases additional results to further support the findings reported in main text. Lastly, we note limitations and future works Section
B
Detailed Description for the Multimodal Dataset
B.1 Data Collection Hardware
B.1.1 Pressure Sensor Glove Customization
To effectively capture comprehensive data on hand pressure and posture, we modified tactile glove equipped with 65 pressure sensors by integrating additional fingertip pressure sensors along with position tracking module, as shown in Figure 5. We customized the design of the glove to focus on the hand pressure measurements across fingertips and palm regions consisting nodes. To determine the pressure in each region, we use the maximum value among the sensors within that region rather than summing or averaging their readings. This approach accounts for variations in hand size, ensuring that the pressure measurement is not artificially lowered due to inactive sensors that may not be engaged by all participants.
pressure sensor
B.1.2 Pressure Sensor Characteristics
For fingertip pressure measurements, utilized Force-Sensing Resistor (FSR) type pressure sensor (RA 18DIY, Marveldex), characterized by force range of O ~ 40 N/em² and a thin profile of 0.7 mm with 8 mm diameter sensing area. Figure presents the typical response behavior of the FSR, highlighting the force-resistance relationship. In our work, we derived a precise fitting model for the sensor's output through repetitive calibrations using push-pull gauge. The calibration process involved conducting 30 load-unload cycles (ranging from 0 ~ 30 N, incremented by N) and meticulously recording the resistance values to ensure accuracy.
B.2 Defined Hand Postures
For the grasp postures, we carefully selected 10 representative movements from a comprehensive grasp taxonomy presented in recent literature [55]. This taxonomy classifies grasps based on several factors, including opposition type, virtual fingers, grip type, and thumb position, culminating in
Position Tracking Sensor
Finger Pressure Sensor
2
7
9
MCU
Palm Pressure Sensor
Nodes Hand Pressure
Figure 5: We added position tracking sensors the knuckles and 5 fingertip pressure sensors. Our glove records occlusion-fres hand-tracking data with exerted hand pressures at 9 regions (5 fingertips and palm regions).
16

---

Table The detailed data collection protocol for our study, categorizing hand postures into Plane, Pinch, and Grasp postures. It lists each posture's 's name and abbreviation.
Type
Posture Name
Abbreviation
Plane
Index Press
I-Press
Plane
Middle Press
M-Press
Plane
Ring Press
R- Press
Plane
Pinky Press
P-Press
Plane
Thumb & Index Press
TI-Press
Plane
Index & Middle Press
IM-Press
Plane
Palm Press
Palm-Pree
Pinch
Thumb & Index Pinch
TI-Pin TI-Pinch
Pinch
Thumb & Middle Pinch
TM-Pinch
Pinch
Thumb & Index & Middle Pinch
TIM- HIM-iinch
Pinch
Thumb & Index & Middle & Ring Pinch
TIMR-Pina
Pinch
Thumb & Index & Middle & Ring & Pinky Pinch
HIMR--PinC
Grasp
Palmar Pinch
Grasp
Lateral
Grasp
Sphere Finger
Grasp
Power Sphere
Grasp
Ring
Grasp
Medium Wrap
Grasp
Fixed Hook
Grasp
Quadpod
Grasp
Purallel Extension
Grasp
Adducted Thumb
33 distinct grasp types. Given the constraints of wearing EMG armband and the necessity to minimize experimenter fatigue through sufficient rest, was crucial to narrow down the posture set. Following the clustering of grasp types based pressure distribution in [56], opted for ten hand movements that cpitomize the diverse range of grasps, as shown in Figure This selection process ensured manageable yet comprehensive dataset that accurately reflects wide array of hand-object interactions.
B.3 Data Collection Protocol
As illustrated in Figure 10, we developed software tools to facilitate user convenience and high- quality data acquisition. These tools provide posture-specific guide images to make it easier for users to follow along, as well as visual feedback that allows monitoring of EMG, pressure, and hand movements real time. Additionally, they offer information on data collection and rest times for each posture, enabling systematic data gathering with time synchronization.
(a)
(b)
(c)
Finger Pressure Sensor
50
Asproximation
1.0
40
Thtt22
This33
0.8
Resistance
Tris4
Conductance
300
0.6
20
R 47.98 e*(-0.4FF) 1.57
0.4
Aporaximation Tiall
10
0.2
1Haa22
0.0
Tssad
5
100
15 Force(N)
20
25
30
5
10
15
20
25
30
Force(N)
Figure 6: FSR sensor calibration and response curves. (a) We calibrated each FSR pressure sensor with the calibration setup using the precise push-pull gauge. (b) & (c) The FSR's resistance and conductance characteristics function of applied force, respectively, illustrating the sensor's calibration curve derived from multiple trials for data collection.
17

---

(b)
Finger Tracking
Module
Hand Pressure Modes)
3D Hand (20 Joint Angles)
Pressure Glove
SEMG Armband
044
Pressure Sensor
8-Channel
Pose
SEMG Signals (21 Joint Coordinates)
Figure 7: (a) Data collection setup to capture sine-synchronized hand pressure. 3D hand pose, and sEMG data. We asked participants to interact with 22 action sets while wearing customized pressure gloves integrated with finger tracking module and an 8-channel EMG armband. (b) We captured 9-nodes hand pressure values, 8-channel sEMG signals, 20 joint angles, and 21 joint coordinates (computed from joint angles).
Plane
I-Press
M-Press
R-Press
P-Press
IM-Press
MR-Press
Palm-Press
Pinch
TI-Pinch
TM-Pinch
TIM-Pinch
TIMR-Pinch
TIMRP-Pinch
Grasp
Palmar Pinch
Lateral
Sphere 3 Finger
Power Sphere
Ring
Wrap
Hook
Quadpod
Parallel Extension Adducted Thumb
Figure 8: 22 actions executed by the hand while collecting multimodal sensing data. It includes hand-plane interactions, 5 pinch actions, and 10 hand-object interactions selected from hand grasp taxonomy. Capital letters before hyphen, namely M, R, and P. stand for thumb, index finger, middle finger, ring finger, and pinky finger, respectively.
Using these data collection tools, we gathered data from each participant the postures listed in Table 6, repeating each posture total of three times. Each posture was collected for duration of 30 seconds, with participants instructed to repeat the given posture 12 to times. For Plane-type postures, participants were instructed to apply and then release force against flat surface. For Pinch and Grasp postures, they were directed to fix an object with their left hand and then apply and release force with their right hand. To prevent the accumulation of fatigue in the muscles due to EMG, we incorporated 10-second rest period after each posture, and after completing all 22 postures, participants given 10-minute break ensure adequate rest.
Muscle fatigue, characterized by decline muscle's ability to generate force, occurs when muscle is repeatedly contracted or held in sustained contraction over an extended period. As muscles fatigue, their electrical activity changes, manifesting as increased signal amplitude (root mean square) and shifted frequency content [] These alterations negatively impact the quality of sEMG data
18

---

Grasp Actions
Lateral
Sphere Finger
Power Sphere
Ring
Medium Wrap
Foedd Hook
Parallel
Adducted
Quadpod
Power
Intermediate
Precision
Palm
Pad
Side
Pad
Side
3-5
2-4
3-4
2-3
2-5
Large
Power
Sphere
Priematic: Fager
Type
Adouction Grip
Triped
Palmar Prich
Primmatc Figger
Premetic
Priamati F4
Willing Tripod
Thumb Abducted
Diamoter
Sphere
Fieger
Ttp Pinch
Trppor
Sseepod
Dask
Mediiu
Pincer
Sphere
Index Einenelon
Addlcttdd Thumb
Lateral Traed
Parallel
Thumb Adducted
Stict
Fleed
Figure 9: Representative grasp postures selected for data collection. This illustrates the ten grasp pos- tures chosen for the study, reflecting wide range of hand interactions. The postures categorized by opposition type, virtual finger usage, grip type, and thumb position. The coder diagrams above the images indicate the pressure points for each grasp, corresponding the regions of the hand engaged during the posture.
used for hand pressure estimation, making more challenging for the model to accurately distinguish between different hand postures and pressure levels. Fatigue leads to less consistent muscle activation patterns, complicating the model's ability to learn reliable relationships between sEMG signals and exerted pressure [43]. While muscle fatigue is an important consideration in EMG-based control, addressing it falls outside the scope of this study. To minimize its effects on our results, we ensured sufficient rest periods between data collection trials during both training and testing phases.
19

---

Posture
Pressure
AMOCERI Nore
EMG
Target Power Sphere
28.8
Keep
DELL
Collecting time
Pose
Figure 10:
Data collection interface displaying real time feedback for hand posture, pressure, and EMG signals, along with timer for data acquisition sessions.
B.4
Data Processing
B.4.1
sEMG Signal and Pressure Data
With a frame length of 1,248 samples, corresponding to 0.624 seconds, we utilized a Short Time Fourier Transform (STFT) for sEMG signal. This method, leveraging window length of 256 and hop length of 32 samples, employed Hamming window function to minimize spectral leakage. To preprocess the sEMG signal, applied the STFT and considered only the signals below 64Hz to eliminate high-free fighffreuuency noise, optimizing the signal's 's relevance for muscle activity analysis [4]. For the pressure data, considering the conventional force that human comfortably apply with their hand, we propose a maximum inferable force of 20 N, and forces exceeding 20 N were clipped to be treated 20 N. Additionally, exclude noise data caused by subtle sensor presses, values below 0.2 were processed to be treated as
B.4.2 Hand Pose
For the 3D hand pose data, we use data obtained from the Quan- tum Mocap Metaglove for the training dataset. The hand skele- ton model adopted in this study is as shown in Figure 11. The joints of the fingers from the middle the ring finger ted because they are defined identically to those of the index finger. The the hand as 20-dimensional angular data, denoting each finger with one abduction angle and three flexion angles [72]. For the thumb, it includes the CMC (carpometacarpal joint)'s 's abduction and flexion angles, along with the MCP (metacarpophallngelll joint) and IP (interphalangeal joint) flexion angles. For the other fingers, encompasses the MCP's abduction and flexion angles, as well as the PIP (proximal interphalangeal joint) and DIP (dis- tal interphalangeal joint) flexion angles. This 20-dimensional angular data is transformed into three-dimensional angular Figure 11: Hand skeleton model. representation / for each of the three joints determining the state of each finger, considering the kinematic tree of the hand skelcton in the MANO hand model [73]. Subsequently, through forward
omit-
DIP
data obtained through the data glove represents
MCP
MCP
20

---

Table Relative bone length measurements for different finger sections.
Hand Bone
Thumb
Index
Middle
Ring
Pinky
Root Ma-i-to-MPPPaa (Root-to- CMC for thumih)
0.5134
1,0475
1.0000
0.9871
0.9585
MCP-ooPIPP (CMC-1o-MCC for thumb)
0.4225
0.4509
0.5375
0.5055
0.3399
PIP-to-DIP (MCP-to-IP thumb)
0.3772
0.3014
0.3427
0.3039
0.2551
DIP-to-Tip (IP-to-Tip Tip for thumb)
0.3324
0.2765
0.3061
0.2822
0.2540
kinematics [74], this is converted into a hand joint representation. J/ that includes the fingertips and the hand root. The relative bone lengths each hand skeleton used during forward kinematics are shown in Table 7. When converting hand joint representation J into 3D heatmap volumes H, J is first min-max linearly scaled to take values in the range [12, 36], considering the maximum and minimum values for each spatial dimension in the training dataset. Thereafter, J is transformed into 21 3D heatmaps H by converting it into 3D Gaussian heatmap with standard deviation of a = 1 for each dimension.
When using a 3D hand pose inferred from vision data, it can be transformed into the same representa- tion as the 3D joint representation J. However, considering the global rotation of the wrist in the photo, it is necessary to align the direction of the global rotation of the hand with the kinematic tree of the hand used in training, and the scale of the hand also needs to match that used during training. Accordingly, the 3D hand pose J inferred from vision data was rescaled so that the distance from the root joint the MCP of the middle finger is identical, rotated that the plane formed by the three joints (root joint, MCP of the middle finger, MCP of the index finger) is consistent, and translated so that the root joint is in the same position. Afterward, the 3D hand pose inferred from vision data is transformed into 21 3D heatmaps H through the process described above.
C Detailed Description for Method
C.1 Empirical Motivation of Our Framework
To empirically motivate our approach, we observed that sEMG signals alone might not be sufficient for fine-grained pressure localization on the hand. We conducted an empirical analysis demonstrating the limitations of using sEMG signals alone for this purpose. Figure 12 presents detailed sEMG signal patterns for two pairs of hand postures: (1) I-Press (index finger press) versus M-Press (middle finger press), and (2) TI-Pinch (thumb-index pinch) versus TM-Pinch (thumb-middle pinch). each case, participants were instructed to apply pressure using different fingers, targeting specific regions of the hand. The 8-channel sEMG signals were recorded over 5-second interval, with each channel representing muscle activity from different forearm muscles. However, shown in Figure 12, the sEMG patterns are remarkably similar within each pair actions, despite the differences in the fingers exerting pressure.
This similarity arises because the muscle activations required for pressing or pinching with adjacent fingers can produce overlapping sEMG signals due to the anatomical proximity and shared muscle groups involved. Consequently, relying solely on sEMG signals makes challenging to distinguish between pressures applied by the index middle finger. While sEMG signals provide valuable information about the force exerted, they lack the spatial specificity needed for precise pressure localization. These findings highlight the necessity of incorporating 3D hand posture information to disambiguate the source of pressure. By fusing sEMG signals with precise hand pose data, our framework can leverage both the force-related information from sEMG and the spatial context provided by hand posture, leading to more accurate and fine-graud nne-grained pressure estimation.
C.2 Details of Deep Neural Network Architecture
sEMG Feature Extractor fEMG The sEMG feature extractor employs a conventional encoder- decoder architecture [4], with the encoder composed of three encoder blocks and the decoder comprising three decoder blocks. Each encoder block includes 2D convolutional layer that filters the input, extracting pertinent features by convolving the input with kernel of size (3, 3) and preserving the input size with padding set to same. 2D batch normalization and ReLU activation function follow this. A max-poolle max--ooolig layer then downsamples the output by selecting the maximum value within
21

---

I-Press
M-Press
TI-Pinch
TM-Pinch
Ch
Ch
Ch
Ch
Ch
Ch 7
Figure 12: Visualization of patterns with similar EMG footprint on different postures.
each pooling window, defined by the downsampling parameter, effectively diminishing the spatial dimensions. This block structure is iterated with increasing channel sizes and downsample rates. The initial encoder block processes the raw sEMG signals with 32 filters, followed by subsequent blocks that increase the filter count to 128 and 256.
In parallel to the encoder. each decoder block is designed to upscale the condensed features to higher resolution. utilizes 2D convolutional layer with an identical (3, 3) kernel size and same padding, followed by batch normalization and ReLU activation. mirroring the encoder block. An upsampling step follows, augmenting the spatial dimensions to correspond those in the encoder's earlier stages. The decoder restores the feature map from its condensed state, generated by the encoder, higher resolution, employing decoder blocks in reverse sequence. Subsequently, the 2D feature map, having been processed through both the encoder and decoder, is flattened into a ID feature vector. Finally, the ID feature vector is projected to a 512-dimensional space through single fully connected layer.
Hand Pose Feature Extractor fnand The Hand Pose Feature Extractor adopts a modified version of the ResNet34 architecture [75], incorporating 3D convolutions adeptly handle spatial structure inherent in hand pose data [67, 76]. This adaptation is crucial for capturing the 3D pose of hand postures, offering a richer representation than traditional 2D models. Central this architecture is the 3D ResNet block, termed the residual block. Each residual block starts with 3D convolutional layer, employing a kernel size of (3, 3) and stride of 1, maintaining spatial dimensions through padding. This convolution is followed by 3D batch normalization and ReLU activation, ensuring non-linearity and normalization of activations. A second 3D convolution with the same kernel size and padding repeats this process. The block is designed to accommodate optional downsampled
22

---

input, aligning input and output dimensions via a downsample path if necessary, employing either average pooling or a (1, 1, 1) convolution for dimension matching, followed by batch normalization. Residual connections in each block allow inputs to bypass layers, helping to prevent the vanishing gradient problem by directly adding inputs to outputs.
The architecture initiates with 3D convolution layer that processes the input using kernel size of (7, 7, 7). stride (2, 2, 2). and padding keep spatial dimensions consistent. It continues with four sequential layers comprising multiple residual blocks. These layers progressively increase channel depth from 64 to 512, adhering to the sequence [64, 128, 256, 512]. The layer structure is [3, 4, 6, 3], indicating the number of residual blocks per layer. Downsampling occurs in the transition between these layers reduce spatial dimensions and increase the receptive field. After progressing through the convolutional blocks, an adaptive average pooling layer condenses the 3D feature map to size of 1 x 1, effectively distilling the spatial-temporal features into singular, compact feature vector. This vector is then connected to a fully connected layer, which outputs the 512-dimensional final feature representation.
Our Full Model ʃ After concatenating the 512-dimensional feature vectors from sEMG and 3D hand pose data, the model processes the resulting 1024-dimensional vector through a linear layer to reduce it to 256 dimensions. This reduction is immediately followed by a batch normalization step and a ReI.U activation to standardize and introduce non-linearity into the features, enhancing their representation for the task at hand. This sequence- comprising 256-dimensional linear layer, batch normalization, and ReLU activation-is executed twice to further refine the features. The refined features then advance through Sigmoid-activated linear layer, tailored to the specific classification task with dimensions set by the numbers of interest regions (I = 9), finalizing the model's output preparation.
C,3 Model Training and Inference
For the training of our model, we configured our setup as follows: We trained the model for 50 epochs to ensure adequate training steps. We set the batch size to 64 to balance computational efficiency and training stability. The model was optimized using the Adam optimizer [77] with a learning rate of 0.0001 and (81,8)) of (0.9, 0.999), along with weight decay of 0.0001. The model training was conducted on a computing environment equipped with an AMD EPYC 7763 64-Core Processor, 1.0TB RAM, and an NVIDIA RTX 6000A 48Gi. Our model's entire training process took approximately 12 hours.
In our inference setup, we used computer equipped with 12th Generation Intel Core 19-129 processor, 32GB of RAM, and an NVIDIA GeForce RTX 4080 with 16GB of GPU memory. The total number of parameters in our model amounts to approximately 66.17 17 M, with the sEMG feature extractor contributing 1. 1.26 M parameters and the hand pose feature extractor contributing 64. 11 M parameters. Our analysis on inference time indicated that our model requires only 4 milliseconds to process a single batch, demonstrating its capability to handle 256 streaming inputs batch inference setup without any delay memory overflow issues our test environment. Thus, conclude that our model is sufficiently lightweight and efficient for real-time processing, making practical for applications that require immediate feedback.
D Results
D.I
Implementation Details of Comparative Methods
The model architecture and data processing details of the sEMG Only Model largely align with the official implementation disclosed in [4]. One exception to this conformity is our modification of the final linear layer, where expanded the number of prediction dimensions from 5 to 9. This alteration was made to estimate the force exerted at the tips of each finger, aligning the output dimensions with those of our study. Thus, this implementation serves as a counterpart solely utilizing EMG data, excluding hand pose information. For Pressure Vision++, we directly utilized the implementation and pre-traing pre-trained weights officially released by the authors It is noteworthy that we initially considered
wttggghhub..cm//u---e /nttpe://github.co com/NYU- ICL -or- foree-interec-ccceeeeeeee interface
Phttps://gitai www.ppgrady3/pressueeeeeeeeeeeee
23

---

using Pressure Vision, which infers the entire hand's pressure from photos. However, after testing its official implementation', we found it entirely non-functional in our environment due its lack of generalization ability. Consequently, we adopted PressureVVsioon+t Vision+++, which is limited to estimating pressure at the fingertips but has much higher ability for generalization.
The 3D Hand Posture Only Model corresponds to our full model with the sEMG feature extractor fEMG removed. Consequently, the linear layer that previously mapped 1024-dimensional concatenated feature to 256 dimensions has been adjusted to map from 512 dimensions to 256 dimensions. This modification reflects the absence of sEMG features, focusing solely on the utilization of 3D hand pose data. The sEMG + Hand Angles Model modifies our full model by replacing fhand with a 3-layer fully connected network, setting all layers to map features layer incorporates ID batch normalization and ReLU activation, aiming for consistent feature representation. Additionally, underwent dimension- -iise min-max normalization adjust values to the [0, I] range, facilitating standardized input for model processing. For aspects not specifically mentioned, such as data processing, model architecture, and the choice of stochastic optimizer, we adhered to the configurations described for our full model. This approach ensures that our comparative analysis remains grounded in consistent methodological framework, allowing for fair evaluation of the different models' performance.
256-vector, Each
D.2 Evaluation Metrics
We employ a set of evaluation metrics that allow for comprehensive assessment of both classification and regression capabilities. These metrics are designed to quantify the accuracy, precision, and generalizability of our model in predicting pressure exertions across the hand. The metries include:
Coefficient of Determination (R²). The R value measures the proportion of variance in the dependent variable that is predictable from the independent variables. It is defined as:
R R² 11-1- Ei - P(,))
EP P.,)²*
where T is the number of time frames, is the number of hand regions, P., the actual pressure, P... is the predicted pressure for region at time f, and P.. represents mean pressure for region i. This metric quantifies how well the model's predictions fit the actual pressure data, with values closer to indicating stronger correlation between predicted and actual pressures.
Normalized Root Mean Squared Error (NRMSE). NRMSE provides normalized measure of the deviation the predicted pressure values from the actual pressures applied, offering insight into the model's precision in estimating the magnitude of forces. It is computed follows:
I
NRMSE
(P.. P.,)
P Piaxx
T
where Pmax is the maximum observed pressure in the dataset.
Classification Accuracy. This metric assesses the model's capability accurately classify whether pressure is being exerted by any region of the hand. Specifically, a prediction considered correct only if the pressure exertion status of all hand regions (fingertips and palm areas) is accurately classified for a given time frame. This metric crucial for understanding the model's ability to distinguish between active and inactive pressure application scenarios.
24

---

D.3
Additional Quantitative Results
D.3.1
Mean Average Error
Table 8: Performance comparison of models in terms of MAE. Note that this table reports the performance of Table terms of the MAE metric.
Method
MAE
SEMG Only [4]
6.03 + 2.24
3D Hand Posture Only
9.03 + 4.18
sEMG + Hand Angles
5.75 1: 2.37
PiMForee (Ours)
4,99 + 2.02
Table Performance comparison of hand regions in terms of MAE, Note that this table reports the performance of Table 4 in terms of the MAE metric.
Finger Tip
Method
Thumb
Index
Middle
Ring
Finky
Mean
Upper Right
Upper Left
Lower Right
Left
Mean
sRMG [+]
5.10
6,52
8,84
7.40
7.75
4.30
6:73
Only
7.4
7.20
sEMG Hand Angles
866
4.08
8.04
2.35
2.97
1.01
6.24
3.05
7,66
4:33
6,34
4.44
6.77
4.56
575
4.50
6,63
PiMForee (Ours)
7.38
3.43
6,95
3.44
5.91
2.97
4,55
2.36
2.61 =
0.77
5.48 =
2.59
6.56 =
3.64
5.88 +
3.72
6,07
3.59
4.99
3.76
5.77
3.68
5.61
3.08
D.3.2 Cross-User Evaluation and Comparison with PressureViste Vision++
To assess the generalizability of our hand pressure estimation framework across different individuals, we conducted cross-user evaluations. Additionally, we performed quantitative comparison with the vision-based method Pressure Vision+++ to benchmark our model's performance against existing state-of-the tatt-off-the-at approaches.
Cross-Usee Evaluation Given the known variability of sEMG signals due to anatomical differences and sensor placement among users, it is crucial to evaluate how our model performs when encoun- tering data from participants not seen during training. We partitioned our dataset of 21 participants into a training set and a test set. Specifically, data from 17 participants were used to train the model, while data from the remaining 4 participants entirely blind to the test participants during both training and evaluation phases.
reserved for testing. This ensures that the model is
Based on the cross-user evaluation results presented in Table 3, our PiMForce combining sEMG signals with 3D hand posture data significantly outperforms the sEMG-only baseline across all evaluation metrics. Specifically, our method achieves an R score of 70.06%, compared to 47.90% for the sEMG-only model, indicating a substantial improvement in the propertion of variance explained by the model. The NRMSE is reduced from 14.1 14% to 10.70%, and the MAE decreases from 12.24% to 8.54%, demonstrating enhanced precision in pressure estimation. Additionally, the classification accuracy increases from 57.40% to 72.01 %, reflecting superior ability correctly identify the presence or absence of pressure across hand regions. Table 10 shows the detailed performance in terms of hand regions.
These improvements highlight the effectiveness of integrating 3D hand posture information with sEMG signals to capture both muscle activation patterns and spatial context, thereby enhancing the model's generalizability to unseen users. The reduced variability in performance metries, as indicated by the lower standard deviations, also suggests that our method is more consistent across different individuals.
Table 10: Cross Eross-user performance comparison of whole hand regions in terms of NRMSE.
Fingre Tip
Hand Palm
Method
Thumb
Indes
Ring
Pinky
Upper
Upper
Right
sMMG Only
14.26
1395 a
12.22 &:
9:25
6.53 a
11.24
12.75
10.90
12.07
10.20
11.50
11.35
(Ours)
4.24%
6.41%
3,47%
1.72%
3.515
4.4015
25

---

(a)
(b)
(c)
(d)
(e)
Figure 13: Visualization of ground truth pressure and predicted pressure for the same posture using the existing Pressure PressureVision++ Vision++ hand pressure prediction framework. (a) Original image from camera. (b) Input image for the PeesureVissione (d) Original predicted pressure. (e) Overlaied predicted pressure by Pressure Vision++, projected onto Sensel pressure array. (f) Ground truth pressure, projected onto Sensel pressure array.
model. (c) Overlaied predicted pressure by Pressure Vision++.
26

---

Comparison with PressureVision++ To benchmark our PiMForce against existing vision-based pressure estimation methods, we conducted quantitative evaluation of PressureVision-ti Using the same equipment as the original Pressure-iision+ study Logitech Brio 4K webcam and Sensel Morph pressure sensing array - we collected data from participants. As Pressure Vision+ + estimates pressure only on fingertips and requires full visibility of the hand within the camera view, we focused our evaluation on plane and pinch interaction sets, specifically: I-Press, M-Press, R-Press, P-Press, IM-Press, MR-Press, TI-Pinch, TM-Pinch, TIM-Pinch, TIMR-Pinch, and TIMRP-Pine
Figure 14: Collecting fingertip pressure data from Sensel pad and hand data from an RGB camera to quantitatively evaluate Pressure Vision++ and our model.
Each participant was instructed repeat each action for 30 seconds, following our data collection protocol. To ensure optimal visibility for PressureVisionn++ we carefully adjusted the camera angle to capture both the fingers and palm (see Figure 14). Note that the Sensel Morph could not measure thumb force during pinch actions, this data was excluded from the analysis. Figure 13 illustrates ground truth pressure and predicted pressure under the setting of Pressure Vision++.
Table 5 presents the performance comparison between Pressure Vision++, our proposed method, and the sEMG-only model in the cross-user setting. is important to note that our evaluation of Pressure Vision++ is inherently cross-user performance report, as the model was trained on separate dataset. As shown in the table, our method significantly outperforms both V and the sEMG-only model in all three metrics. Specifically, PiMforce achieves accuracy of 82.20% compared 66.00% for Pressure Vision++ VissonreVision+++++ and 67. 90% for the sEMG-only model. For detailed breakdown of fingertip performance during plane and pinch interactions, we present the finger-specific performances in Tables 11. These results indicate that our proposed method consistently outperforms PressureVision++ Vision+ in estimating fingertip pressures across all fingers during both plane and pinch interactions.
Discussion The cross-user evaluation confirms that framework is robust and generalizes well to unseen participants. While there is a general decrease in performance when moving from within-user to cross-user evaluation (e.g.. classification accuracy decreases from 83.17% 72.01 % for method), our model still maintains superior performance compared to existing methods. Pressure Vi- sion++ shows limitations in cross-user scenarios due to its reliance visual features that vary widely among individuals, such as skin texture, color, and hand shape. Additionally, it requires full visibility of the hand, limiting its applicability in scenarios where the hand is partially occluded or when interactions involve grasping objects.
Our PiMForce's superior performance can be attributed to the complementary nature of sEMG signals and 3D hand pose data. While sEMG provides insights into muscle activation patterns, 3D hand pose offers spatial context that helps disambiguate similar sEMG signals arising from different hand configurations. By integrating these modalities, our framework achieves more accurate and reliable hand pressure estimation across diverse users and interaction types. demonstrating its potential for practical applications in human-computer interaction and virtual reality systems.
Table 1I: Cross-user performance comparison of finger tips in terms of NRMSE.
Method
Index
Middle
Ring
Pinky
Meon
Index
Middle
Ring
Pinky
Mean
Overall
Meam
Vrssenee [17]
$3.23
1.55%
20,14
2,08%
27.25 =
1.67%
15.45
2.27%
27.12
1.37%
43.18
1.57 %
42.45
2.55 %
31.72
2,42
27.77 +
2.39
36.93
1.51 %
32.70
1.05%
JEMG unly
13.90
13.34
8.15%
8.72
4,48%
5.64
10.40 ::
13.45
7.15%
11.12
5.53%
3.68
5.15
8.56
5.92%
10.34
4.57%
PiMFForce (Ursi
10.13
8.49
4.54%
5.45
2.42%
8.54
8.48
4.65%
6.08
7.17
8.34
3-46%
27

---

D.4 Additional Qualitative Results
The video included in the supplementary material is designed to qualitatively demonstrate our model's s inference performance in a time-continuou, context." The extended results of Section 5.2.2 and Figure 4a in the main text are displayed. Figures 15-18 serve as extended figures Section 5.2.2 in the main text. Figures 19-22 act as extended figures to Figure 4a in the main text.
Hand-Object Interaction Type
Hand Only
sEMG
sEMG 3D Hand
Ground
Ours (Rendered)
Angles
I-Press
M-Press
R Press
IM- Press
MR-Press
Figure 15: Qualitative visual analysis among comparative models.
'tttps//hhc--tech. github. 1o/PMMForce/
28

---

Hand-Object interaction Type
Only
3D Hand Posture Only
sEMG Hand Angles
sEMG Hand Posture
Ground Truth
Ours (Rendered)
Palm-Press
Ti-Pinch
M--inch
TIMMPinch
TIMRP-Pinch
Figure 16: Qualitative visual analysis among comparative models.
29

---

Hand-Object Interaction Type
SEMO Only
3D Hand Posture Only
SEMG Hand
sEMG 3D Hand Posture
Ground Truth
Ours (Rendered)
Lateral
Sphere Finneer
Power Sphere
Ring
Medium Wrap
Figure 17: Qualitative visual analysis among comparative models.
30

---

Hand-Object Interaction Type
sEMG Only
3D Hand Posture Only
sEMG Hand Angles
sEMG 3D Hand Posture
Ground Truth
Ours (Rendered)
Parallel Extension
Adducted Thumb
Quadpod
Figure 18: Qualitative visual analysis among comparative models.
31

---

Hand-Object Interaction Type
Input Image
3D Hand Pose Estimation
PressureVision+*
Ours (Rendered)
I-Press
M-Press
P-Press
Undetectable
MR-Press
Figure 19: Pressure estimation results for the vision-aidee aided hand in the test set.
32

---

ObjecO Interaction Type
3D Hand Pose Estimation
Ours (Rendered)
Input Image
PressureVision+*
AUNO
Undetectable
Palm-Press
Undetectable
TI-Pinch
Undetectable
TM-Pinch
Undetectable
TIM-Pinch
Undetectable
TIMRP-Pinch
Undetectable
TIMRP-Pinch
Figure Pressure estimation results for the vision-aided hand in the test set.
33

---

Hand-Object Interaction Type
Input Image
3D Hand Pose Estimation
PressureVision+*
Ours (Rendered)
Undetectable
Palmar Pinch
Undetectable
Lateral
Undetectable
Sphere Finger
Undetectable
Power Sphere
Undetectable
Ring
Undetectable
Medium Wrap
Figure 21: Pressure estimation results for the rision-aided hand in the test set.
34

---

Hand-Object Interaction Type
Input Image
3D Hand Pose Estimation
PressureVision+*
Ours (Rendered)
Undetectable
Fixed Hook
Undetectable
Extension
Undetectable
Adducted Thumb
Quadpod
Undetectable
Figure 22: Pressure estimation results for the vision-aidee hand in the test set.
35

---

Ground-Truth Force (N)
Predicted Force (N)
IM-Press
Medium Wrap
Moddlel
Indexiii
Thumbow:
Upper Righees
Uppoer wite
Tower Rightii
Cower LMRIN)
Timels)
Timets
Figure 23: Comparison of fingertip and palm forces predicted by model with ground truth data over time. To evaluate our model, we use the IM-Press posture, which uses only fingertip pressure, and the Medium Wrap posture, which uses whole-hand pressure, during Subject 5's test session.
D.5 Comparison between Predicted Pressures and Ground Truth over Time
To provide detailed analysis of model's temporal performance, include additional figures that showcase the ground truth and predicted pressure values over time for various hand actions. Figure 23 displays the pressure trajectories for all nine hand regions during the execution of the TM-Press and Medium Wrap actions by participant. In this 20-second interval, the participant performs the TM-Press action for the first 10 seconds, followed by the Medium Wrap action for the next 10 seconds.
In these plots, each subplot corresponds specific hand region, with time the horizontal axis and pressure magnitude on the vertical axis. The solid lines denote the ground truth pressures measured by the tactile glove, while the dashed lines represent the pressures predicted by model. The alignment between the ground truth and predicted pressures across different hand regions and actions demonstrates the model's capability accurately track pressure dynamics over time. Notably, the pressure shifts from regions associated with the TM-Press action to those corresponding to the Medium Wrap action, reflecting the participant's change in hand posture and interaction.
36

---

E Limitations and Future Works
Our framework and model, while innovative, is not without its limitations. One of the primary constraints is its reliance on the accuracy of off-the-shelff hand pose detectors during inference. If the hand pose is inaccurately estimated, can lead to erroneous pressure estimations. Furthermore, our analysis, as evidenced by Figure 6 of the main text, reveals that performance is not uniformly high across all hand postures. This inconsistency indicates areas that could benefit from further refinement. Due to the time constraints of participants, our dataset does not cover the full range of hand postures described existing literature on hand posture taxonomy [55, 56]. Additionally, the absence of mesh information for both hand and object means we lack precise data the surface contact locations. Our study also faces limitations due to the demographic homogeneity of the participants. The lack of diverse demographic representation among participants may limit the generalizability of our current model across different populations.
Overcoming these limitations opens up several directions for future work. Exploring additional modalities such as depth sensor, constructing an end-to-end model that integrates vision sensors with sEMG signals, adopting high-density electromyography (HD-EMG) equipment [78, 48] to surpass the limitations of standard sEMG devices, and expanding our dataset through more extensive participant sampling and long-term data collection are promising paths forward. Incorporating hand mesh and object mesh estimation [79-81] with our muscular force learning framework could provide more accurate information regarding contact points and the corresponding pressure hand mesh. Demographic limitations can be addressed by conducting studies with a larger and more diverse participant pool. Expanding the participant demographics will enhance the generalizability of our results and ensure the framework's 's upplicability across a broader range of users. These steps will help refine our model, making it more robust and widely applicable across different hand interactions.
Additionally, our current framework primarily leverages global features from both sEMG signals and 3D hand pose data, without fully exploring alternative multimodal fusion methods. We did not incorporate fine-grained spatial information or prior structural knowledge of hand poses into the model architecture. Future work could investigate different fusion techniques, such as integrating spatial hicrarchies or attention mechanisms, to capture the intricate relationships between hand pose and muscle activity. This exploration could further enhance pressure estimation accuracy and robustness.
F Potential Social and Broader Impacts
The proposed framework has several promising applications across various fields. In healthcare and rehabilitation, this technology could enhance therapeutic devices by providing detailed feedback on hand movements and pressure, improving the effectiveness of rehabilitation exercises and assistive devices for individuals with motor impairments. In human-computer interaction, integrating hand pressure data could lead more intuitive and responsive interfaces, particularly in virtual reality (VR) and augmented reality (AR) environments. In occupational safety and ergonomies, our framework could help design ergonomic tools and workspaces, reducing the risk of repetitive strain injuries and enhancing worker productivity and safety.
Despite its potential usefulness, the use of forearm-wont sEMG to estimate hand movements, poses, and pressure could introduce biases in the model's 's performance due to medical and physiological variations. For instance, some individuals have less muscles than others; for example, the palmaris longus muscle, which flexes the wrist joint, is absent in about 15% of the population [82]. To mitigate these risks, it is essential to consider a broader clinical demographic when recruiting participants and designing research for future studies.
While our study did not present any direct clinical risks to the participants, recognize the importance of ethical data handling, particularly regarding sEMG signals which may contain personal information. Prior to participation, all participants were provided with a detailed information sheet outlining the purpose, procedures, and potential risks of the study. We obtained written informed consent from each participant, ensuring they understood the nature of the data being collected and their right to withdraw at any time. All collected data was anonymized, complying with relevant data privacy regulations
37

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Section includes the main claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section E includes the limitations of the work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when the image resolution is low, or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
38

---

Justification:
This paper does not include theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Section and C.3 include all the information needed to reproduce the main experimental results of the paper.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
39

---

Answer: [Yes]
Justification:
Section includes URL for open access to the data and code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //nips. wttpccccpppccccceeececccccccccct httpa///ggiieeeeeeeuuuuiiiiiii
/CodeSubmie ssionnollll onPolicy))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section 4, 3.2 and C.3 include all the training and test details. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Table 2, which is the main quantitative result, includes error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
40

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Section C.3 includes information the computer resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https / http///eeepp//eeeeee/eee/ee cc
Answer:
[Yes]
Justification:
We reviewed NeurIPS Code Of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Section F includes societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
41

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper does not include data models that have high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We checked the licenses of assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
42

---

Answer: [Yes]
Justification:
The assets include documentation.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: Section 3.2 and B include the full text of instructions given participants and screenshots.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: Section 3.2 includes the text on IRB approvals. Human study in this work does not give rise to clinical risks.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
43


Reviews:
[{"review_id": "PHO3h93UBG", "reviewer": "Reviewer_3PTe", "summary": "This paper presents a novel framework for estimating hand pressure during various hand-object interactions using multimodal data. The framework integrates sEMG and 3D hand posture information to enhance the accuracy of pressure estimation. They introduce a dataset to validate their approach. The primary contribution of the study is the exploration of combining vision-driven 3D hand posture data with sEMG to improve the robustness of hand pressure estimation.", "questions": "Please list up and carefully describe any questions and suggestions for the authors. Think of the things where a response from the author can change your opinion, clarify a confusion, or address a limitation. This is important for a productive rebuttal and discussion phase with the authors.", "limitations": "see weaknesses", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The integration of sEMG signals with 3D hand posture data sound good.\n\n2. The dataset is extensive, including 83.2 million frames from various hand-object interactions.\n\n3. The proposed model shows high accuracy in estimating hand pressure.", "weaknesses": "1. Although integrating sEMG signals with 3D hand posture data is a promising approach, there is insufficient experimental evidence to support the motivation behind this combination. The paper lacks clear proof of the benefits described, beside the sentence.\n\n2. The paper does not adequately compare their approach with other vision-based datasets or techniques that also extract 3D hand posture information.\n\n3. While combining sEMG and vision-based methods theoretically addresses individual robustness issues, the claim that this combination improves overall robustness is not convincingly supported by the data. The dataset should demonstrate the robustness of the proposed method, not just improvements in accuracy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**(W1) Support for advantages of integrating sEMG signals with 3D Hand Posture data**\n   - We would like to clarify that we performed an ablation study (Table 2 and 3) and discussed study results in the corresponding sections. These tables compare the performance of our model using: (1) sEMG data only[R3], (2) sEMG data with hand posture data using angle representation, and (3) sEMG data with 3D hand pose data(Ours). We primarily discussed this analysis in Section 5.2.1 highlighting the benefits of combining sEMG and 3D hand pose information.\nWe acknowledge that our initial explanation of the motivation might not be sufficiently convincing for readers. To provide stronger empirical motivation for our approach, we will add a new figure illustrating cases where similar sEMG patterns are observed despite pressure being applied to different parts of the hand. Figure R4 demonstrates this phenomenon. The figure displays the time-series representation of 8-channel sEMG signals over 5 seconds for two pairs of hand postures: (1) I-Press and M-Press, (2) TI-Pinch and TM-Pinch. Each row represents a different sEMG channel. In both pairs, the pressure is applied differently, targeting either the index fingertip or the middle fingertip. However, we observe very similar sEMG patterns between I-Press and M-Press, and between TI-Pinch and TM-Pinch.\nThis observation demonstrates that while sEMG signals provide valuable information about force, they may not be sufficient for fine-grained pressure localization on their own. This highlights the potential of incorporating hand posture information to guide pressure estimation. We will incorporate this discussion and Figure R4 into the revised manuscript, adding a new second paragraph in Section 4.1 (Llines 172-173) and a new Appendix section titled \"Empirical motivation of our framework\" to further clarify the rationale behind our approach.\n\n**(W2) Comparison with vision-based datasets/techniques extracting 3D hand posture info**\n   - We agree that we did not comprehensively cover various vision-based datasets or methodologies for 3D hand estimation as related works or comparative methodologies. However, please take it into consideration that the ultimate goal of our research is to estimate hand pressure, not to derive 3D hand posture. For this reason, we selected comparative methodologies and datasets aimed at estimating hand contact or hand pressure rather than 3D hand posture estimation. Since the ultimate goal of our research is hand pressure estimation, we believe that comparisons with 3D hand pose estimation methodologies and datasets may not be contextually consistent. However, we also found out that we missed highly related dataset paper such as ActionSense from NeurIPS 2022 which will be added as part of our related works.\n\n**(W3) Clarification on the meaning of the robustness in our paper**\n- In our manuscript, the 'robustness' of the model refers to the model’s capability to ensure high accuracy across various postures over different hand regions. Here, we have detailed the performance across different hand regions (Table 3) for various postures (Figure 3) to show our model’s robustness. Thanks to Reviewer 3PTe, we understand that the term 'robustness' was understood differently by readers, we will revise it to clearly convey our intended meaning. Specifically, we will revise the following lines as follows:\n\n- (Line 313) “This consistent correlation between predicted values and actual pressure measurements highlights the model’s robustness and adaptability in handling a wide range of hand movements.” \n→ “This consistent correlation between predicted values and actual pressure measurements highlights the model’s ability to maintain high accuracy and reliability across a diverse range of hand parts and postures, thereby demonstrating its robustness.”\n- (Line 331) “Figure 4b shows demo video footage illustrating our framework’s robustness in estimating hand pressure while continuously changing hand posture, pressure levels, and the objects being grasped.”\n→ “\"Figure 4b shows demo video footage illustrating our framework’s capability in accurately estimating hand pressure while continuously changing hand posture, pressure levels, and the objects being grasped.\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UfdS4GPapx", "reviewer": "Reviewer_nF88", "summary": "This paper proposes a hand pressure estimation method based on 3D hand poses and t forearm surface electromyography (sEMG) signals. Accordingly, the paper constructs a multimodal dataset containing pressure, 3D hand poses, and sEMG signals. The paper experimentally verifies that combining 3D hand poses and sEMG signals achieves better results compared to using these two modalities independently. Furthermore, compared to vision-based pressure estimation methods, the method achieves better results in complex hand-object interaction scenarios.", "questions": "1. Would it be beneficial for the overall method's performance to use predicted hand poses as input during training?\n2. Have you tried other multimodal information fusion methods?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The paper is well written and easy to follow.\n2) The idea of integrating 3D hand poses and electromyography signals for hand pressure estimation in this paper is intuitive, interesting, and practical. The experiments in this paper further demonstrate that combining hand poses and electromyography signals is beneficial.\n3) The data collection system constructed in this paper is impressive, as collecting high-quality synchronized multimodal hand data is very challenging. This dataset could potentially advance the field of hand pressure perception.", "weaknesses": "1) From the perspective of multimodal fusion methods, the framework proposed in this paper lacks novelty. This framework directly combines the global features of the electromyography modality and the global features of hand poses without fully utilizing the fine-grained spatial information and prior structural information of the hand poses.\n\n2) The qualitative comparison with PressureVision++ is unfair. Due to biases in the training dataset and the design preferences of the method, PressureVision++ is more suitable for pressure estimation in planar contact scenarios. However, this paper does not compare with PressureVision++ in such scenarios.\n\n3) The training and testing setups are not sufficiently reasonable. During training, hand poses captured by a data glove are used as input, while during testing, the estimated results from a vision-based hand pose estimation model are used as input. There is a significant gap in the distribution of hand poses, which may potentially impair the performance of the hand pose branch.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**(W1) Clarifications on the novelty of the proposed framework**\n - We acknowledge that there might be room for improvement in terms of methodological novelty. In this paper, however, our primary objective of this paper was to pioneer the use of electromyography and 3D hand posture data simultaneously and demonstrate its efficacy. To achieve this, we presented a data collection platform and a pipeline for using our model in inference without a data glove. Subsequently, to validate the approach experimentally, we deliberately opted for the simplest form of machine learning modeling approach instead of focusing on introducing a new model architecture by incorporating geometric structures of the model and data, our work lays a strong foundation for future research in this direction. We will add this point in the Limitations and Future Work section (Line 1107) as follows:\n- “While our current framework demonstrates the value of combining sEMG and 3D hand pose data, it primarily focuses on leveraging global features from both modalities. Future work could explore incorporating fine-grained spatial information and prior structural knowledge of hand poses into the model architecture to further enhance pressure estimation accuracy and robustness.”\n\n**(W2) Clarification on the comparison with PressureVision++**\n- We agree that this is not a perfectly fair comparison. Regarding this, we would like to clarify the rationale for our comparison and explain the additional quantitative comparison with PressureVision++ we carried out during our rebuttal.\nUnlike PressureVision, the authors of PressureVision++ specifically highlighted its effectiveness in interacting with diverse surfaces (3rd, 4th, and 6th paragraphs of Introduction in [R1]), even presenting training and data collection methodologies designed for such scenarios. This led us to believe that PressureVision++ would be a suitable candidate for comparison with our vision-based method, which is why the primary reason we chose PressureVision++[R1] over PressureVision[R2]. \nWhen considering vision-based methods for comparison, we found PressureVision and PressureVision++ to be the only viable options available in existing research. While it may not meet your standards for a completely fair comparison, we believe we made the best possible choice, given the available options.\nWe carried out a quantitative comparison with PressureVision++ to support our claim with a fairer condition. \n\n**(W3) Clarification on utilizing vision-based hand pose estimation model for testing setup**\n- As outlined in Section 4.4 (Line 228), we canonicalize 3D hand pose data derived from a vision-based hand pose estimator (More details shown in Section B.4.2, Line 942). During inference, we align the 3D hand pose estimated from visual data with the training data representation. This process involves rescaling the hand pose to match the bone lengths used during training, rotating it to align with the kinematic tree of the MANO hand model, and translating it to ensure consistent root joint positioning. This canonicalization minimizes the discrepancy between hand pose data from the glove and the vision-based estimator, improving the generalizability of our model during inference. Also our framework inherently relies on the accuracy of off-the-shelf hand pose detectors during inference (Line 1097). We acknowledge that this dependency is a limitation of the proposed modality and inference pipeline and discuss this limitation in Section E. Limitations and Future Works (Line 1077).\n\n**(Q1) Benefits of utilizing predicted hand poses as input for training**\n - While using predicted hand poses during training might seem beneficial, our training dataset requires participants to wear a glove to collect the ground truth hand pressure data. We utilize a pressure-sensing glove (TactileGlove, Pressure Profile Systems) for this purpose. Unfortunately, most current hand pose detectors are not trained to work on hands wearing gloves, or their performance significantly degrades in such situations. Therefore, it is hard to incorporate predicted hand poses during the training phase due to the need for more reliable hand pose estimation for gloved hands.\n\n\n**(Q2) Have you tried other multimodal information fusion methods?**\n - We haven't explored alternative multimodal information fusion methods in this work. However, we acknowledge that there are numerous potential improvements to be made in both the model design and implementation.  Investigating different fusion techniques is a promising direction for future work, and we believe it holds the potential to enhance the accuracy and robustness of our hand pressure estimation framework. We will add this point into limitation and future works section as follows:\n“While our current framework demonstrates the value of combining sEMG and 3D hand pose data, it primarily focuses on leveraging global features from both modalities. Future work could explore incorporating fine-grained spatial information and prior structural knowledge of hand poses into the model architecture to further enhance pressure estimation accuracy and robustness.”", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aKkU48Qwku", "reviewer": "Reviewer_YYDP", "summary": "This manuscript introduces a framework for estimating the pressure of 9 points on the hand during 22 distinct hand object interaction types including pinches, grasps and planar interactions. They use an 8 channel sEMG band placed at the forearm as well as hand pose estimated using a glove or monocular video camera. They train models using a dataset from 20 participant that uses pressure sensors in the glove as ground truth. They evaluate models within participants, and compare pressure estimation models that use different combinations of sEMG and hand pose, including different representations of hand pose using joint angles or embedded 3D coordinates. The best performance is achieved by a multimodal network that fuses an encoding of the spectrogram of the sEMG and the 3D hand pose, the later using a 3D convolutional network. They compare performance across fingers and gestures, and give qualitative comparisons to other vision-based approaches.", "questions": "** Questions ** \n- what is the cross participant performance? \n- What is the quantitative performance compared to Pressure Vision? What is the correlation between the two techniques if an absolute ground truth (e.g. from sensel) is not possible. \n- Why does utilizing hand angles not improve the model performance? \n- How was 3D hand pose acquired in cases without the glove? I don't see a description of this method. \n- Can you show raw traces of the discretized pressure to give an intuition for what the real pressure data looks like? \n\n**Minor**\n\n- **Datasets for Hand Pressure Estimation** — do these datasets include pressure, many of these appear to be object interaction papers\n- Fig 4: can you clarify if this participant was in the training dataset?\n- L302 ‘significant’ - were significance tests run here?\n- Table 3 - what are the relevant error bars for this table? \n- nit: L 336 - pionering ?\n- Table 2:unclear what errors are over\n- Table 3: unclear what value is being plotted\n- all exocentric?\n- nit: Figure 3 pixelated\n- Manuscript overstates at multiple points ‘L316 ‘exceptional’", "limitations": "Yes, they review this in section F.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The combination of different modalities integrated in the baseline is I believe new, and the method is evaluated across a sufficient number of participants (20) and gestures. \n- They compare across relevant single-modality baselines, and break out evaluations broken out per-behavior.\n- The framework is clearly defined and contextualized. I think the 3D pose embedding and joint angle comparison is nice.", "weaknesses": "- All evaluation is within-participant. sEMG is known to be quite variable across participants due to variations in anatomy and placement, and the hierarchy of performance across modalities will likely change. \n- The comparisons with PressureVision++ (Figure 4) and other SOTA approaches are a little underwhelming, as they essentially show the framework only functions when all fingertips are tracked. A more informative set of comparisons would show results, with ground truth, for cases when the fingertips are unoccluded. For figure 16, the quantitative comparison betwene the two would be informative.\n- The behavioral generalization to new objects and gestures is unclear. Similarly the performance with raw vision data is only qualitative. It is unclear how the system handles domain mismatches between the glove and vision-based hand pose estimation framework. In combination with the lack of cross-participant information the in the wild utility of the approach is unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**(W1) Additional experiments for cross-user evaluation**\n\nPlease refer (Joint Response 2).\n\n**(W2-1) Clarification on the comparison with PressureVision++**\nYou are correct that the current comparison might not be entirely fair, and we acknowledge that our framework currently performs best when all fingertips are tracked. However, we'd like to clarify our approach in two ways. First, this limitation is inherent to the PressureVision and PressureVision++ methods themselves. As you pointed out, PressureVision++ relies on visual cues like color and shape changes on the hand caused by pressure. Consequently, it can only estimate hand pressure when all relevant visual features are visible in the camera image. In contrast, our approach doesn't require full hand visibility; as long as sufficient hand posture information can be inferred, pressure estimation is possible. Second, while we aim for comprehensive comparisons, PressureVision and PressureVision++ represent the only viable options available in current research for vision-based hand pressure estimation. While not a perfect comparison, we believe it offers valuable insights given the limited alternatives.\n\n**(W2-2) “The quantitative comparison between the two would be informative.”**\n\nPlease refer (Joint Response 1).\n\n**(W3-1) Limitation on behavioral generalization to new objects and gestures**\n\nYou correctly pointed out that our study does not demonstrate the performance of our framework on unseen gestures during the training phase. We acknowledge this limitation. While constructing our dataset, we strived to collect data encompassing a wide range of hand interactions, including plane interactions, pinch interactions, and grasp interactions, to facilitate comparisons with previous works [R1,R3]. However, we also had to consider the limited time and potential fatigue of our participants. Therefore, we made a selection of representative postures, as shown in Figure 8 and Figure 9. Collecting additional data for novel gestures with new participants is infeasible in this rebuttal phase, preventing us from conducting further experiments on generalization. However, as highlighted in Lines 908 and 152, our selection of postures covers a considerable portion of representative movements from the reference grasp taxonomy and those presented in [R3]. This suggests a degree of generalizability, but further research with new gestures and objects is certainly needed to solidify these points. Following this point, we will add this text to the Limitation section (Line 1107): \n- While our dataset covers a range of representative hand postures, it is limited in its exploration of novel gestures and object interactions unseen during training. Further research is necessary to investigate the generalizability of our framework to new objects and gestures. Expanding the dataset with a broader range of hand-object interaction scenarios and evaluating the model's performance on these unseen examples would provide valuable insights into its robustness and adaptability.\n\n**(W3-2) Clarification on matching the glove and vision-based hand pose estimation framework**\n\nYou are correct, and we don't claim this to be a perfectly fair comparison. However, we'd like to provide some context on our rationale in two points. First, the authors of PressureVision++ specifically highlight its effectiveness in interacting with diverse surfaces (3rd, 4th, and 6th paragraphs of Introduction [R1]), even presenting training and data collection methodologies designed for such scenarios. This led us to believe that PressureVision++ would be a suitable candidate for comparison with our vision-based method. Second, when considering vision-based methods for comparison, we found PressureVision and PressureVision++ to be the only viable options available in existing research. While it may not meet your standards for a completely fair comparison, we believe we made the best possible choice given the available options.\n\n**(Q2) Why does utilizing hand angles not improve the model performance?**\nBased on the current results of our ablation study, we believe that hand angles alone lack sufficient inductive bias to effectively capture the spatial information inherent in hand postures. \n\n**(Q3) How was 3D hand pose acquired in cases without the glove?**\nWe specify the utilization of an off-the-shelf hand pose detector in Section 5's introduction and Section 4.4, explicitly stating that we chose to employ ACR (Line 244). Additionally, as mentioned earlier, Section B.4.2, 3D Hand Pose, describes the canonicalization process for transforming the results of vision-based pose estimation into a standardized 3D hand pose representation (Lines 943-975).\n\n**(Q4) Can you show raw traces of the discretized pressure to give an intuition for what the real pressure data looks like?**\n\nTo visually illustrate the performance of our model over time, we will include figures showing the temporal evolution of both ground truth and predicted pressure values. Figure R1 file provides an example of such visualization. The figure presents the GT pressure and the pressure predicted by our model for all 9 hand regions during TM-Press and Medium Wrap actions. The horizontal axis represents time, each row corresponds to a specific hand region, and the vertical axis in each plot shows the pressure values. In this example, the user performs the TM-Press for 10 seconds, followed by the Medium Wrap for another 10 seconds.  We believe that these temporal visualizations will enhance the readers' understanding of our framework by demonstrating how data is collected and predictions are made over time. We will include these figures and the corresponding discussion in a new Section D.4. Moreover, we’ll add one of those figures in the main body of the paper as in (Line 299) of Section 5.2.1.\n\nDue to limitations on the length of our rebuttal, we would like to address the minor points in separated comment.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NR7G2OdYf4", "reviewer": "Reviewer_saS6", "summary": "The authors leverage 3D hand pose and sEMG signals as inputs to facilitate hand pressure estimation. With data gloves, a multimodal hand-object interaction collection system is devised. Empirical experiment results show the efficacy of the proposed method. Furthermore, the feasibility and robustness is proven in a single-camera scenario.", "questions": "- Are the readings from 6 sensors belonging to Region 8 simply added up as the pressure of Region 8?\n\n- Is it possible to briefly explain how the quality would be influenced by muscle fatigue as mentioned in L158?\n\n- Will the data glove introduce different interaction patterns from those without data gloves?", "limitations": "Please refer to the Weaknesses and Questions part.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The collected dataset is superior in scale and provides a holistic collection for various hand-object interactions.\n\n- The ablation studies are helpful in providing useful insights.", "weaknesses": "- A highly related dataset ''ActionSense: A Multimodal Dataset and Recording Framework for Human Activities Using Wearable Sensors in a Kitchen Environment'' in NeurIPS 2022 should be cited and discussed.\n\n- The evaluation is only conducted in a drop-one-session manner. It might be better if a drop-one-subject evaluation could be provided to demonstrate the cross-subject ability for a more general application.\n\n- The results are impressive in their low relative errors. It would help more with the absolute errors reported. \n\n- The results are reported with little temporal information. A curve showing the predicted force changing with time would be preferable. Also, quantitative metrics on the temporal consistency would be a bonus if possible.\n\n- For glove-less evaluation, only qualitative results are reported. It appears possible to provide a quantitative comparison given the controlled hand postures. For example, attaching sensors to the contact region of the MR-Press makes it possible to provide GT in glove-less scenarios.\n\n- Reporting quantitative prediction errors for different postures is a missing chance to provide more insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**(W1) Additional dataset reference**\n\nAs you suggest, we will cite and discuss in related work section as follows:\nA highly relevant work is the ActionSense dataset, which focuses on capturing multimodal data of human activities in a kitchen environment using wearable sensors. Similar to our approach, ActionSense emphasizes the importance of rich, synchronized, multi-modal data streams for understanding human actions and gathers . However, while ActionSense provides a valuable resource for understanding general kitchen activities, our work focuses on utilization of 3D hand posture into muscular force learning for understanding hand pressure estimation.\n\n**(W2) Additional cross-user evaluation**\n\nPlease refer (Joint Response 2).\n\n**(W3) Additional results regarding absolute errors**\n\nWe have included the Mean Absolute Error (MAE) results in the rebuttal PDF file. Table R6 and Table R7 in the rebuttal PDF file correspond to Table 2 and Table 3 in the main manuscript, respectively, and present the MAE metrics for each comparison.\n\n**(W4) Providing temporal information for predicted forces**\n- **“A curve showing the predicted force changing with time would be preferable.”**: We acknowledge the reviewer's insightful suggestion regarding the need for temporal information in our results.  To illustrate the performance of our model over time, we will include figures showing the temporal evolution of both ground truth and predicted pressure values. Figure R1 in the rebuttal PDF file provides an example of such visualization. The figure presents the ground truth pressure and the pressure predicted by our model for all 9 hand regions during TM-Press and Medium Wrap actions. The horizontal axis represents time, each row corresponds to a specific hand region, and the vertical axis in each plot shows the pressure values. In this example, the user performs the TM-Press action for 10 seconds, followed by the Medium Wrap action for another 10 seconds.  As evident from the figure, the location of pressure exertion and prediction shifts according to the action performed. We believe that these temporal visualizations will enhance the readers' understanding of our framework by clearly demonstrating how data is collected and predictions are made over time. We will include these figures and the corresponding discussion in a new Section D.4 titled \"Comparison between Predicted Pressures and Ground Truth along with time\" in the revised manuscript. Moreover, we’ll add one of those figures in the main body of the paper as in (Line 299) of Section 5.2.1.\n- **“Quantitative metrics on temporal consistency would be a bonus if possible.”**: We would appreciate it if the reviewer could provide specific examples or references for \"quantitative metrics on temporal consistency\". This would allow us to better understand their suggestion and provide a relevant analysis.\n\n**(W5) Additional experiments and results for glove-less evaluation**\n\nPlease refer (Joint Response 1).\n\n**(W6) quantitative prediction errors for different postures**\n\nYou correctly pointed out that our study does not demonstrate the performance of our framework on unseen gestures during the training phase. We acknowledge this limitation. While constructing our dataset, we strived to collect data encompassing a wide range of hand interactions, including plane interactions, pinch interactions, and grasp interactions, to facilitate comparisons with previous works [R1,R3]. However, we also had to consider the limited time and potential fatigue of our participants. Therefore, we made a selection of representative postures, as shown in Figure 8 and Figure 9. Collecting additional data for novel gestures with new participants is infeasible in this rebuttal phase, preventing us from conducting further experiments on generalization. However, as highlighted in Lines 908 and 152, our selection of postures covers a considerable portion of representative movements from the reference grasp taxonomy and those presented in [R3]. Further research with new gestures and objects is certainly needed to solidify these points. Following this point, we will add this text into the limitation section:\n- While our dataset covers a range of representative hand postures, it is limited in its exploration of novel gestures and object interactions unseen during training. Further research is necessary to investigate the generalizability of our framework to new objects and gestures. Expanding the dataset with a broader range of hand-object interaction scenarios and evaluating the model's performance on these unseen examples would provide valuable insights into its robustness and adaptability.\n\n**(Q3)Will the data glove introduce different interaction patterns from those without data gloves?**\n\nAs outlined in Section 4.4 of the main text, we describe how we canonicalize 3D hand pose data derived from a vision-based hand pose estimator in Section B.4.2, 3D Hand Pose. During inference, we align the 3D hand pose estimated from visual data with the training data representation. This process involves rescaling the hand pose to match the bone lengths used during training, rotating it to align with the kinematic tree of the MANO hand model, and translating it to ensure consistent root joint positioning. This canonicalization minimizes the discrepancy between hand pose data from the glove and the vision-based estimator, improving the generalizability of our model during inference. Therefore, if the inference of hand pose is good enough, there will be no difference between wearing gloves and not. As noted in the Section E. Limitations and Future Works in the appendix, our framework inherently relies on the accuracy of off-the-shelf hand pose detectors during inference (Line 1097). This dependency is acknowledged as a limitation of the proposed modality and inference pipeline.\n\nDue to limitations on the length of our rebuttal, we would like to address (Q1) and (Q2) in seperated comment.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank reviewers for the constructive feedback. 

We have been putting our best effort to address the weaknesses and questions pointed by reviewers to strengthen the paper. Our rebuttal PDF includes new experimental results, diagrams, and tables to support our work. In our rebuttal comments, tables and figure  in the rebuttal PDF are referenced as Figure Rx & Table Rx while figures and tables in the main paper are denoted as Figure x and Table x. Below, we have listed the references used in our rebuttal.

We summarize the two major changes and updates we have made during the rebuttal period:
### **(Joint Response 1) Quantitative comparison with vision-based method (PressureVision++[R1])**
We appreciate the reviewer's suggestion for a quantitative comparison with vision-based methods. Yes, while PressureVision methods are inherently limited in their application due to the necessity of a glove for whole-hand pressure measurement, a quantitative comparison would provide valuable insights.

To address this, we conducted a quantitative evaluation of PressureVision++ using the same equipment as the original PressureVision++ study: a Logitech Brio 4k webcam and a Sensel Morph pressure sensing array. As PressureVision++ estimates pressure only on fingertips and requires full visibility of the hand within the camera view, we focused our evaluation on plane and pinch interaction sets, specifically: I-Press, M-Press, R-Press, P-Press, IM-Press, MR-Press, TI-Pinch, TM-Pinch, TIM-Pinch, TIMR-Pinch, and TIMRP-Pinch. Following our data collection protocol, each participant was instructed to repeat each action for 30 seconds. To ensure optimal visibility for PressureVision++, we carefully adjusted the camera angle to capture both the fingers and palm. Figure R2 and Figure R3 illustrate the data collection setup for PressureVision++. We collected data from 5 participants using this setup, mirroring the methodology of PressureVision++. Please note that the Sensel Morph could not measure thumb force during pinch actions, so this data was excluded from the analysis. Therefore, Table R2 and Table R4 present the performance comparison between PressureVision++ and our model, focusing on the pressure exerted by the tips of the index, middle, ring, and pinky fingers for the specified action set. It is important to note that our quantitative evaluation of PressureVision++ is inherently a cross-user performance report, as the model was trained on a separate dataset. For fairness and consistency, we report the cross-user performance for "sEMG Only[R3]" and "sEMG + 3D Hand Posture(Ours)" in Table R2 and Table R4 as well. Table R2 provides a comparison across all action sets for the three main metrics, while Table R4 offers a detailed breakdown of fingertip performance for plane interactions and pinch actions. As expected, there is a general decrease in performance when moving from within-user to cross-user evaluation. For instance, in our method, we observe a decrease in R² from 88.86% to 66.71%, in NRMSE from 6.65% to 9.27%, and in accuracy from 83.17% to 82.20%. At the same time, as shown in Tables R2 and R4, our method still significantly outperforms both PressureVision++ (66.60% in accuracy) and the sEMG only model (67.90% in accuracy) in the cross-user evaluation. This indicates the superiority of our method in both within-user and cross-user scenarios.

### **(Joint Response 2) Cross-user Evaluation**
We acknowledge the reviewer's concern regarding the limited within-participant evaluation and the potential variability of sEMG signals across individuals. To address this, we conducted additional experiments in a cross-user setting to measure the generalizability of our model. Specifically, we used all session data acquired from 16 participants as our training set, while data from the remaining 4 participants was allocated to the test set. This ensures that the model is completely blind to the test participants during both training and evaluation. Table R1 and R3 present the results of this cross-user evaluation, comparing our model's performance against the primary baseline method Force-aware interface[R3]. Furthermore, in the rebuttal PDF file, Table R2 and Table R4 provide a breakdown of our performance for index, middle, ring, and pinky finger pressures during plane and pinch interactions, specifically for comparison against PressureVision++. The results from these cross-user evaluations demonstrate that our approach consistently outperforms existing methods[R1,R3] even when encountering unseen participants. For a more detailed analysis of our performance compared to PressureVision++, please refer to our response regarding the quantitative evaluation of PressureVision++ below. We will add new results and discussion as a new subsection titled “Cross-User Evaluation” in Section D (Line 1046-1047) of the revised manuscript.

Please refer to reviewer-specific feedback and a one-page PDF with a summary of added experimental results.

### References

- [R1] Grady, Patrick, et al. "PressureVision++: Estimating Fingertip Pressure from Diverse RGB Images.", CVPR 2024.
- [R2] Grady, Patrick, et al. "PressureVision: estimating hand pressure from a single RGB image." ECCV 2022.
- [R3] Zhang, Yunxiang, et al. "Force-aware interface via electromyography for natural VR/AR interaction." ACM Transactions on Graphics (TOG) 2022.
- [R4] Lalitharatne, Thilina Dulantha, et al. "A study on effects of muscle fatigue on EMG-based control for human upper-limb power-assist." ICIAfS 2012.
- [R5] Eddy, Ethan, Erik J. Scheme, and Scott Bateman. "A framework and call to action for the future development of EMG-based input in HCI." ACM CHI 2023.

Meta-review: The reviewers appreciate the collected dataset and the idea of combining 3D hand poses and electromyography signals for hand pressure estimation. The reviewer raised concerns about the lack of comparison with other vision-based baselines, and novelty in the data fusion framework, etc. The author provided detailed and professional rebuttals to address most of the concerns. After reading these comments, I give my suggestion as acceptance of this paper.
Decision: Accept (poster)