SATPose: Improving Monocular 3D Pose Estimation with Spatial-aware Ground Tactility

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-09 08:17:43

Abstract: Estimating 3D human poses from monocular images is an important research area with many practical applications. However, the depth ambiguity of 2D solutions limits their accuracy in actions where occlusion exits or where slight centroid shifts can result in significant 3D pose variations. In this paper, we introduce a novel multimodal approach to mitigate the depth ambiguity inherent in monocular solutions by integrating spatial-aware pressure information. To achieve this, we first establish a data collection system with a pressure mat and a monocular camera, and construct a large-scale multimodal human activity dataset comprising over 600,000 frames of motion data. Utilizing this dataset, we propose a pressure image reconstruction network to extract pressure priors from monocular images. Subsequently, we introduce a Transformer-based multimodal pose estimation network to combine pressure priors with monocular images, achieving a world mean per joint position error (W-MPJPE) of 51.6mm, outperforming state-of-the-art methods. Extensive experiments demonstrate the effectiveness of our multimodal 3D human pose estimation method across various actions and joints, highlighting the significance of spatial-aware pressure in improving the accuracy of monocular 3D pose estimation methods. Our dataset is available at: https://anonymous.4open.science/r/SATPose-51DD.

Main paper: 
 related work 2.1 vision-based 3d human pose estimation: 
 3D human pose estimation (HPE) is a traditional computer vision task [1, 2, 38, 48, 62] , existing works of 3D HPE can broadly be classified into two mainstreams. The first involves the one-stage estimation [12, 38, 46] , where the 3D poses are directly predicted from the images. The second is the two-stage estimation, where the 2D poses are first extracted from the images, and then a lift from 2D to 3D is performed. As for two-stage estimation, benefiting from the excellent performance of state-of-the-art 2D pose detectors [7, 19, 34] , numerous works [25, 29, 39, 44, 49, 59, 61] engage in improving the performance of 2D-3D pose lifting. Most of them can be divided into TCN (Temporal Convolutional Network)based methods [29, 39] , GCN (Graph Convolutional Network)-based methods [8, 20, [54] [55] [56] 60] , and Transformer-based methods [18, 26-28, 31, 41, 43, 49, 63] . Compared to TCN-and-GCN-based structures, Transformer-based architecture is better suited for modeling long sequnences of 2D poses due to its well-designed attention mechanism. In this paper, we innovatively incorporate a multimodal Transformer-based structure to fuse the 2D poses extracted from monocular images and ground pressure data.Pressure sensors can provide comprehensive biomechanical information. Their integration in various fields such as interactive control [14, 36, 40] , robot touch [24, 57] , gesture recognization [3, 15, 30, 47, 51, 52] , etc., highlights their unique advantages. For example, TouchEditor [58] supports text editing for AR glasses through tactile gestures on a flexible touchpad. SmartSleeve [36] enables real-time sensing of surface and deformation gestures on a textile pressure sensor. In addition to the aforementioned small-area pressure sensors, studies by Clever et al. [9] and Casas et al. [5] utilize 2D pressure data to simulate lying postures, while Luo et al. [32] propose a method for 3D pose estimation using pressure sensors integrated into a mat, showing promising prospects in depth estimation and lower limb pose estimation. However, it is difficult to provide effective support for upper limb posture estimation using a single pressure sensor. Therefore, we integrate visual and tactile information, leveraging the complementary nature of both to enhance the pressure-based pose estimation, while addressing issues such as depth information absence in monocular images and estimation errors from centroid shifts.Recent research [10, 17, 23, 33, 37, 64] indicates that integrating data from various sensors can improve the accuracy of 3D human pose estimation. For instance, Pan et al. [35] merge monocular RGB and sparse IMU data to facilitate robust motion capture even when visual signals are unavailable. [16] proposes an architecture fusing RGB and LiDAR data for precise pedestrian localization and pose prediction. [37]  
pvm dataset: 
 In this section, we detail the experimental setup employed for the collection of the large-scale multimodal human activity dataset, PVM (Pressure, Vision, Mocap) Dataset.Figure 2 : Diagram of the multimodal data collection system, comprising a pressure mat and a monocular camera situated in an optical motion capture space, and all devices centrally controlled by a computer. 3.1.2 Actions. We focus on 3D human pose estimation for actions with some body parts being occluded and actions sensitive to slight shifts in body centroid. It is challenging to address these issues with monocular images due to depth ambiguity, yet they can impair the effectiveness of applications requiring high-precision pose estimation, such as sports training. Thus, by filtering and integrating existing human activity datasets [22, 32] and the functional movement screen (FMS) action set [11] , we designed a set of 16 actions (see Fig. 3 ). Participants performed each action for approximately 90 seconds (1800 frames), with the flexibility to take breaks as needed. Prior to the commencement of the experiment, the researcher conducted a brief demonstration of each action for the participant. Once the experiment began, no additional guidance was provided, allowing participants the freedom to perform actions at any location according to their individual preferences.3.2.1 Pressure Sensing Mat. The ground pressure images were collected using a commercial, large-format, high-resolution pressure 3.2.2 Monocular Camera. We employed the C93 monocular camera manufactured by AONI2 to record videos of human activities (Fig. 2 ). The videos were captured at a resolution of 1280 × 720 with a frame rate of 30Hz. The camera was securely mounted on a tripod, positioned 1.7m directly in front of the pressure sensing mat at a height of 1.2m. This camera setup was designed to ensure the capture of the complete body of all participants at any location on the pressure sensing mat. The camera was connected to a computer through a wired connection, facilitating control over video recording and the transmission and storage of the video stream. Capture System. We utilized a optical motion capture (Mocap) system manufactured by NOKOV 3 , which consists of 12 Mars2H series optical cameras, to record the 3D human pose (Fig. 2 ). This data serves as the ground truth for subsequent model training. The Mocap system determines the 3D positions of the 18 body joints by capturing the reflective markers on human body, ultimately recording the human motion with millimeter-level precision (error range of ±0.15 mm) and a frame rate of 60Hz.The three aforementioned data acquisition devices were connected to the same computer for data collection and storage control, ensuring data synchronization. The computer receives eight 64 × 32 pressure matrices from the pressure mat and ultimately concatenates them into a 256 × 64 pressure matrix. Mocap data is exported in .bvh format and further converted into the 3D world coordinates of various body keypoints, resulting in data of size 18 × 3 for each frame. For monocular videos, we retain the original image frames, and extract 2D skeleton keypoints with the open-source human pose estimation library, OpenPose [4] ; Finally, we obtain 2D skeleton with 18 keypoints corresponding one-to-one with the 3D one.After post-processing, data from all modalities were unified to a frame rate of 20Hz. In the end, we collected over 600,000 frames of multimodal human activity data from 20 participants performing 16 actions. The dataset was divided into a training set and a test set with a ratio of 16:4 participants. 
method: 
 In this section, we introduce our data-driven pressure image reconstruction method along with the multimodal 3D human pose estimation framework. The spatial-aware grounding tactility is crucial for enhancing the 3D perception capability of monocular-visionbased human pose estimation networks. However, the reliance on the pressure mat may limit the application of this system to more scenarios. There is a need for an adaptive method that can utilize pressure information during testing while overcoming the restrictions associated with the pressure mat. A dual-pronged strategy involves extracting pressure priors from 2D poses, and subsequently utilizing both 2D poses and pressure images reconstructed from 3 https://www.nokov.com/ We propose a Transformer-based framework (Fig. 7 ) for estimating 3D pose from sequences of 2D keypoints and pressure images.Embedding. The encoding of 2D poses is executed with a convolutional network. The input consists of multiple frames of 2D poses, denoted as 𝑃𝑜 ∈ R 𝑁 ×𝐽 ×2 , where 𝑁 = 351 represents the number of frames, and 𝐽 = 18 represents the number of body joints. The 𝑁 frames encompass the current frame along with the past 𝑁 -1 2 frames and future 𝑁 -1 2 frames. The pose embedding concatenates the 2D coordinates (𝑥, 𝑦) of the 𝐽 joints for each frame, resulting in 𝑁 tokens 𝑃𝑜 ′ ∈ R 𝑁 × (𝐽 •2) . Subsequently, a 1D convolutional layer embeds each token into a high-dimensional feature 𝑍 𝑝𝑜 ∈ R 𝑁 ×𝑑 𝑚 , (𝑚 = 256).Embedding. The encoding of pressure images is realized through a Vision Transformer (ViT)-based architecture [13] . The input pressure images, denoted as 𝑃𝑟 ∈ R 𝑁 ×𝐻 ×𝑊 , have 𝑁 frames in the sequence, with 𝐻 = 256 representing the height and 𝑊 = 64 representing the width of each pressure image. Prior to the encoding, pressure images are resized from 256 × 64 to 120 × 120, aligning them with the input format of Transformer. Subsequently, each pressure image is divided into a sequence of tokens, which are then fed into Transformer's encoder. ViT encodes this information into a high-dimensional feature representation, resulting in 𝑍 𝑝𝑟 ∈ R 𝑁 ×𝑑 𝑚 . Leveraging the global relationship modeling capability of ViT for sequential data, it effectively captures spatial relationships within pressure images.Transformer. The 2D pose embedding and pressure image embedding are added to obtain a fused feature vector, denoted as 𝑍 0 ∈ R 𝑁 ×𝑑 𝑚 = 𝑍 𝑝𝑜 + 𝑍 𝑝𝑟 . 𝑍 0 is further encoded using Vanilla Transformer [50] to obtain an intermediate feature vector 𝑍 1 ∈ R 𝑁 ×𝑑 𝑚 . Vanilla Transformer operates based on self-attention mechanisms, allowing it to better understand the spatial relationships between different joints and contextual information between different frames. By stacking these layers, the network can effectively encode information about multi-frame human movements, capturing the spatiotemporal correlations between body keypoints.Our ultimate goal is to estimate the 3D human pose for the current frame. Therefore, we use Stride Transformer [26] to compress the frame dimension of the features 𝑍 1 . Unlike Pure Transformer, Stride Transformer replaces the feedforward neural network (FFN) layer with a convolutional feedforward neural network (CFFN) layer. This involves downsampling the input vector through convolutional operations, resulting in the final feature representation 𝑍 2 ∈ R 1×𝑑 𝑚 for the current frame.To better model multiframe pose information, we supervise the model at both the full sequence scale and the single target frame scale. Specifically, regression is performed on the outputs 𝑍 1 and 𝑍 2 of Pure Transformer and Stride Transformer, respectively, to obtain the estimated 3D poses 𝑋 1 ∈ R 𝑁 ×𝐽 ×3 and 𝑋 2 ∈ R 𝐽 ×3 . 𝑋 1 and 𝑋 2 represent the estimation results for the 3D poses across 𝑁 frames and the 3D pose for the current frame, respectively. The regression head consists of a batch normalization layer and a 1D convolutional layer. Given the 3D pose sequence ground truth 𝑌 1 ∈ R 𝑁 ×𝐽 ×3 and the target frame 3D pose ground truth 𝑌 2 ∈ R 𝐽 ×3 , the L2 norm is computed between 𝑋 1 and 𝑌 1 , and between 𝑋 2 and 𝑌 2 , serving as two (𝑌 1 , 𝑋 1 ) = 𝑁 𝑛=1 𝐽 𝑗=1 𝑌 1 𝑛 𝑗 -𝑋 1 𝑛 𝑗 2 , 𝐿 2 (𝑌 2 , 𝑋 2 ) = 𝐽 𝑗=1 𝑌 2 𝑗 -𝑋 2 𝑗 2 .Therefore, the total loss for the 3D pose estimation network is 𝐿 = 𝐿 1 + 𝐿 2 .Training. We train the pose estimation model for 25 epochs with one 3090 GPU, which takes about 20 hours to converge. We employ the Adam optimizer with a peak learning rate of 1e-3 that gradually decreases following a cosine learning rate schedule.MPJPE (mean per joint position error) is a commonly used metric for evaluating the performance of human pose estimation networks. In this paper, the variant of MPJPE is used, i.e., W-MPJPE (protocol #0), measured in millimeters (mm). It indicates the Euclidean distance between the predicted joint positions and the true joint positions in the world coordinate system. 
experiments 5.1 comparison with state-of-the-art results: 
 We compared our multimodal method for 3D human pose estimation with state-of-the-art (SOTA) monocular-vision-based methods. Utilizing the official code of these methods, we performed training and testing on our PVM dataset. Table 1 presents the results of the 16 actions for these SOTA methods and our method on protocol #0. The results of our method encompass estimations based on real pressure and predicted pressure.Our estimations based on real pressure (51.6mm) and predicted pressure (51.8mm) both outperform SOTA methods, achieving optimal results in terms of the mean values and for the majority of actions. Despite the challenges in some complex actions, such as "Rotation Stability", where various methods exhibit poor performance (around 80mm), our approach consistently delivers a relatively favorable result (below 70mm). This underscores the superiority of our multimodal pose estimation method. Additionally, the test results based on predicted pressure demonstrate performance comparable to those based on real pressure. This presents a reliable approach for obtaining prior pressure information during testing, mitigating the constraints of pressure mats and allowing adaptation to a broader range of scenarios.Fig. 8 illustrates qualitative comparison results for four actions. The causes of estimation errors include: i) Information loss due to body parts being obscured (e.g. the knee joints during bending); ii) Depth ambiguity resulting in errors in estimating joint positions in depth (e.g. the feet joints during bending and waiting), while our method has achieved significant improvement in the z-axis direction (depth direction) of 4.7mm compared with the monocular method; iii) Subtle variations in the body centroid that are challenging to capture from 2D vision (e.g. the shoulder-neck joints during bending and the hip joints during waist turning), while our method has achieved significant improvement ranging from 5 to 10mm in these joints compared with the monocular method; iv) Occasional deviations in the horizontal direction due to global displacement, such as the feet joints during waist turning.In summary, the spatial perception characteristics of pressure images offer significant gains in the depth information required for 3D human pose estimation through the following mechanisms: i) Facilitating the localization of various body joints in contact with the ground and providing accurate global displacement information by utilizing the position distribution of pressure; ii) Reflecting the body centroid through the differences in pressure magnitude across these distributions, aiding in inferring the spatial relationships between different body joints.Fig. 9 depicts box plots illustrating estimation errors for various joints across different action distributions, showcasing errors along the x, y, and z axes for each joint. The multimodal human pose estimation network was adjusted by separately removing pressure embedding and 2D pose embedding, and models were trained with only 2D poses and only pressure inputs on the PVM dataset.Methods excel in estimating lower limb joints compared to upper limb joints. This is attributed to the more complex joint structure and increased degrees of freedom in the upper body. Additionally, for upper limb joints, estimation errors gradually increase from the root joint (hip joint) to the middle joints (shoulder, elbow joints), and then to the end joints (wrist joints). Similarly, the end joints exhibit a larger error range, showing significant performance variations under different actions. This complexity stems from the motion transfer through the multi-joint chain, involving rotations, translations, or their combinations, making end joints more flexible in terms of degrees of freedom.Methods based on real pressure (see Fig. 9a ) and predicted pressure (see Fig. 9b ) exhibit comparable performance, both outperforming results of the monocular-vision-based method in Fig. 9c . This is primarily evident in the following aspects: i) Significant reduction [44] , P-STMO [43] , MHFormer [27] , GLA-GCN [56] . Skeletons drawn with black line represent the ground truth poses.in estimation errors across all joints, notably with a 5.9mm decrease in the upper limb joints and a 4.2mm decrease in the lower limb joints; ii) Error reduction is more noticeable in joints closer to the root node, with a 6.0mm decrease in the hip, followed by sequential decreases from the knees to the toes (3.4mm, 3.3mm, and 3.1mm) and from the shoulders to the wrists (6.0mm, 3.4mm, and 2.5mm); iii) The z-axis direction, representing the depth dimension, demonstrates the most substantial error reduction (4.7mm) in predictions compared to the other two axes (1.7mm and 2.3mm).On one hand, ground pressure sensors capture pose-related information by sensing the contact between the human body and the ground. As a result, they complements reliable and stable data for pose estimation of the feet and leg joints. When it comes to upper limb joints, it plays a significant role in estimating the positional relationships between joints, thereby substantially enhancing the accuracy of upper limb joints. On the other hand, introducing spatial-aware pressure knowledge helps mitigate the inherent limitation of depth information loss, resulting in notable reduction in estimation errors along the z-axis. In addition, the purely pressure-based method demonstrates the weakest performance overall (120.9mm) but exhibits relatively superior and stable performance for lower limb joints (83.6mm) compared to upper limb joints (158.3mm) (see Fig. 9d ). This showcases its potential for estimating lower limb joints.In Table 2 , we have summarized the 3D human pose estimation errors for different modal combinations. The method that integrates 2D poses with reconstructed pressure and the one that integrates 2D poses with real pressure achieved comparable results, surpassing the performance of the purely monocular-vision-based approach. This outcome strongly validates the efficacy of introducing spatial-aware pressure information in enhancing the performance of monocular-vision-based 3D human pose estimation methods. 3 indicate that as the resolution of the pressure images increases, there is a general trend of decreasing error in 3D human pose estimation. Higher-resolution pressure images provide more detailed information, enabling the model to capture the nuances of human pose more accurately. However, it is noteworthy that when the image resolution increases from 120 × 120 to 160 × 160, the estimation error does not decrease but rather increases. Beyond a certain resolution level, additional details in the images may not be crucial for the pose estimation task and could instead lead to difficulties in network convergence. Moreover, higher-resolution images may escalate computational costs and model complexity. Therefore, we ultimately adopted the 120 × 120 resolution scheme for optimal performance.  
conclusion: 
 In this paper, we propose a novel 3D human pose estimation method that integrates ground pressure images with monocular images. As a foundation, we construct a large-scale multimodal dataset incorporating pressure images, monocular images, and ground truth 3D poses recorded during 16 distinct actions from 20 participants, totaling over 600,000 frames. Furthermore, to accommodate situations where a pressure mat is unavailable, we propose a pressure image reconstruction network capable of reconstructing ground pressure images from monocular images. Experimental results indicate that both combining monocular vision with real pressure and combining monocular vision with predicted pressure outperformed all monocular-vision-based SOTA methods. This demonstrates the exceptional performance of pressure information in enhancing the accuracy of 3D human pose estimation task, and provides viable new directions for future research in this field.Limitations and Future Work: For pressure embedding, we conduct image-wide feature extraction, which, although effective, entails redundant information. Exploring more compact feature extraction methods may enhance information density, potentially improving estimation accuracy and efficiency for the multimodal pose estimation network. For pressure image reconstruction, there are various methods to introduce pressure priors. In this paper, we employ a data-driven image reconstruction approach, yet future exploration could involve biomechanical analysis to understand force relationships and generate pressure images, offering an intriguing and promising direction for future research.  
: 
 Figure 1 : We constructed a real-time 3D human pose estimation system, consisting of a monocular camera and an optional pressure mat (left). With pressure images, we mitigate the inherent depth ambiguity in monocular vision solutions, enabling accurate prediction of 3D human poses (right). [20, 29, 43, 56] to improve its accuracy. However, the fact that monocular vision cannot reliably gauge depth makes it difficult for these efforts to be fully effective [26, 56] . Specifically, different body movements can be projected into the image space in the same 2D pose. During certain actions like sideways poses, some parts of the human body may be occluded by other body parts; slight shifts in the body centroid that may not be easily discernible from monocular images can lead to different body poses; deficiencies in depth information estimation can also detrimentally impact the effectiveness of global displacement estimation. To alleviate the impact of depth ambiguity, previous research either resorts to the temporal context captured by successive action frames [26, 39] or the spatial context of the input monocular image modeled by Graph Neural Networks (GNNs) [8, 20] . Although effective, these methods rely solely on monocular 2D visual cues and lack true 3D spatial awareness, thus limiting their performance.Parallel to monocular-vision-based methods, grounding tactility has also been used to estimate 3D human poses. The benefit of this approach originates from the large size and flexible nature of pressure sensors, enabling their embedding into everyday exercise equipment such as fitness mats. This expands the applicability of pressure sensors in various domains including sports, fitness and medical rehabilitation, etc. To achieve these, most existing works rely on array-type pressure sensors as tactile sensing devices, which are highly sensitive to external forces and offer precise localization of force areas and perception of force magnitude. For example, some prior works [6, 45] utilized pressure insoles and pressure mats to study the distribution of plantar pressure during human motion; Luo et al. [32] proposed a method utilizing solely pressure mat information to estimate 3D human pose. Despite achieving lower full-body estimation accuracy compared to visual approaches, pressure sensors demonstrated excellent performance in lower limb joint estimation, highlighting their potential to address the depth ambiguity inherent in vision-based solutions. Inspired by this, we propose an integrative approach that ingeniously leverages the sensitivity of pressure sensors to grounding positions to mitigate the depth ambiguity of monocular vision method, thereby improving the accuracy of 3D human pose estimation. However, this is a challenging task as: i) to the best of our knowledge, there are no existing systems that can simultaneously collect both tactile and vision data, while collecting this type of multimodal dataset requires substantial effort. ii) Developing an algorithm to fuse the different modalities introduced by the new system poses another challenge. iii) Introducing new hardware increases the system's complexity.To tackle these challenges, in this paper, we established a human activity collection system incorporating a pressure mat and a monocular camera, and introduced a Transformer-based multimodal framework designed for accurate 3D human pose estimation from both monocular images and pressure images (Fig. 1 ). Compared to monocular methods, our approach enhances the 3D perception of the system by integrating additional spatial knowledge from ground pressure, resulting in more accurate 3D pose estimation. Specifically, utilizing the data collection system we built, we first created a large-scale multimodal human activity dataset, namely the PVM (Pressure, Vision, Mocap) Dataset, which includes pressure images, monocular images and ground truth 3D poses. This dataset covers data from 20 volunteers performing 16 different actions, totaling over 600,000 frames. Particularly, to enhance the generalizability and performance of our method in various scenarios, especially those where a pressure mat is unavailable, we proposed a test-time adaptation strategy that reconstructs pressure images from monocular images and feeds them into the pose estimation network. Leveraging the large-scale PVM Dataset, we can reliably reconstruct ground pressure images from monocular images, and then achieve accurate 3D human pose estimation even if no real pressure information is available during testing. Experimental results show that our multimodal 3D pose estimation network achieved a world mean per joint position error (W-MPJPE) of 51.6mm and 51.8mm with real pressure and predicted pressure, respectively, both surpassing state-of-the-art methods. This demonstrates the effectiveness and flexibility of our approach, allowing for choosing whether to use the pressure mat depending on the requirement for higher accuracy or simpler system complexity.In summary, our contributions can be summarized as follows:• A novel human activity collection system including a pressure mat and a monocular camera. Leveraging this system and optical motion capture cameras, we constructed the PVM Dataset, a large-scale multimodal human activity dataset. • A test-time adaptation strategy that predicts pressure data from monocular images, enabling accurate 3D human pose estimation solely from monocular images during testing. This makes our system applicable to more diverse real-world scenarios, especially those without a pressure mat. • A Transformer-based multimodal fusion framework that incorporates spatial-aware pressure information to mitigate the inherent depth ambiguity of monocular-vision-based 3D pose estimation methods and improve their accuracy. 
 
 Appendix: https://www.moxiantech.com/https://www.aoni.cc/ 


Decision: This paper has a clear technical contribution in the collection of a dataset  integrating spatial-aware pressure information that is used to mitigate the depth ambiguity inherent in monocular 3D pose estimation solutions.The paper is very easy to follow and presents  solid experimental results.
All reviewers agreed on accepting the paper after the rebuttal. Therefore I reccommend acceptance.

Reviews: review: paper_summary: paper_summary: The paper presents a multimodal approach that integrates both vision and pressure data to address depth ambiguity in monocular 3D human pose estimation. To achieve this, the authors developed a data collection system that combines a pressure mat and a monocular camera, paired with marker-based motion capture ground truth, resulting in a comprehensive multimodal human activity dataset. The proposed system highlights the significance of spatial-aware pressure information in enhancing the accuracy of monocular 3D pose estimation methods.
strength_weakness: strength_weakness: 1. Interesting Idea: The integration of pressure information with vision data to jointly alleviate depth ambiguity is innovative and demonstrates significant improvements over existing methods.
2. Beneficial Dataset: The proposed PVM dataset seems highly beneficial to the relevant community, and provides a robust foundation for training and evaluating the proposed models.
3. Non-trivial effort went into conducting extensive experimental validation. 
4. The paper is well-structured and easy to follow.
limitations: limitations: 1. Limited Effectiveness for Upper Body: The proposed paradigm seems to alleviate depth ambiguity primarily for the lower body, with limited usefulness for the upper body.
2. Lack of Evaluation for Test-Time Adaptation: The authors proposed a test-time adaptation strategy that predicts pressure data from monocular images, which is important for adapting to existing monocular settings. However, the manuscript lacks evaluation of this module within existing state-of-the-art methods, which would demonstrate its ability to enhance these methods. Without this evaluation, the novelty of the proposed approach is somewhat diminished.
3. Concerns with Marker-Based System: The capture system utilizes three systems: pressure, vision, and an infrared marker-based system. However, the marker-based approach can interfere with the appearance in the vision data, making it unsuitable for monocular settings. From my perspective, in single-person scenarios without object interaction, there is typically less occlusion. Therefore, a multi-RGB capture setting (marker-less setting) should be sufficient for achieving motion capture quality.
paper_summary: paper_summary: Estimating 3D human poses from monocular images is challenging due to depth ambiguity, especially in cases of occlusion or slight centroid shifts. This paper introduces a multimodal approach that integrates spatial-aware pressure information to improve accuracy. The authors created a large dataset using a pressure mat and a monocular camera, and developed a pressure image reconstruction network to extract pressure priors. A Transformer-based multimodal pose estimation network combines these priors with monocular images, achieving stoa performance. The proposed method demonstrates the importance of spatial-aware pressure in enhancing monocular 3D pose estimation.
strength_weakness: strength_weakness: - The technical contribution of this paper is significant, especially the collection of a new dataset containing the pressure modality.
- The authors also provide a large number of experiments and ablation studies to demonstrate the performance of proposed method.
- The paper writing is clear and easy to follow.
limitations: limitations: - The network framework is a straightforward integration of previous methods (e.g., ViT, stride transformers). This is not a major limitation, as a significant portion of the technical contribution stems from the dataset.

- This paper evaluates only the W-MPJPE metric, without assessing the MPJPE metric (with joint root aligned) or the PA-MPJPE metric (MPJPE with Procrustes analysis).

- The evaluation methodology for the W-MPJPE metric is unclear. Specifically, is the world coordinate the same as the camera coordinate in a monocular motion capture setup? Furthermore, what is the coordinate system when using pressure information and images as simultaneous inputs?

- There is a concern about whether the method relies on the shape of the pressure mattress and if it can generalize well to different shapes of pressure mattresses.

- It is also uncertain if the pressure prediction network performs well with subtle human movements, such as when the feet are slightly above the ground. If it does not, will this negatively impact the final predicted pose when only the RGB image is used as input?
paper_summary: paper_summary: This paper introduces a novel approach that integrates ground pressure images with monocular images to enhance the accuracy of 3D human pose estimation. The authors construct a large-scale multimodal dataset and propose a pressure image reconstruction network for scenarios where a pressure mat is unavailable. The experimental results demonstrate good performance over SOTA approaches.
strength_weakness: strength_weakness: Multimodal Dataset Construction: The creation of a dataset that includes ground pressure images, monocular images, and ground truth 3D poses enhances the generalizability of the model.
Pressure Image Reconstruction Network: This network addresses the limitation of requiring pressure mats, making the approach more flexible and widely applicable.
Performance Metrics: The method has been evaluated against existing SOTA methods, showing good improvements in 3D pose estimation accuracy.
limitations: limitations: -	Duplicated Information in feature Embedding: The image-wide feature extraction method for pressure embedding can lead to redundant information, which may affect efficiency and accuracy.
-	Simpler Pressure Reconstruction: The data-driven approach for pressure image reconstruction lacks biomechanical insights, which could further refine the accuracy by understanding force relationships.
-	Limited Evaluation Metrics: While the paper shows improvements in error metrics, it could benefit from additional evaluation metrics to provide a more comprehensive assessment of the model's performance.
-	Upper Limb Performance: Despite improvements, the method still shows relatively weaker performance for upper limb joints compared to lower limb joints.
-	Resolution Sensitivity: The accuracy of pose estimation varies with different pressure resolutions, indicating that selecting the optimal resolution is critical and application-specific.
-	Real-Time Applicability: The computational complexity of combining monocular vision with pressure data might affect real-time application, requiring further optimization for practical usages.
paper_summary: paper_summary: This paper proposes a  method to conduct 3D human pose estimation with a pressure mat and a monocular camera. Their designed system can be used in sports in local space. Furthermore, thy construct a new dataset for evaluating their model. Obvious enhancement can be found in their experiments.
strength_weakness: strength_weakness: 1. Dataset contribution should be encouraged. 
2. The system can be used in some real-world applications.  
3. This paper is easy to understand.  
4. The distance ambiguity in 3D pose estimation is a challenging problem.
5. The experiments show that their model can improve the performance.
limitations: limitations: 1. To verify the efficience of each component, more ablation experiments should be added.   
2. In the real-world applications, the position of pressure mat maynot be put in the place as the dataset. There maybe some offsets from the place in the dataset. Can this problem affect the model performance? 
3. I think that author should  also report MPJPE, which is applied more commonly.
4. It seems the dataset is not so challenging, due to the limited number of person, angle of view, and so on.
5. If the test-time adaption can be extend to multi-person scene, this work will be more valueable.