HmPEAR: A Dataset for Human Pose Estimation and Action Recognition

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-05 13:59:45

Abstract: We introduce HmPEAR, a novel dataset crafted for advancing research in 3D Human Pose Estimation (3D HPE) and Human Action Recognition (HAR), with a primary focus on outdoor environments. This dataset offers a synchronized collection of imagery, LiDAR point clouds, 3D human poses, and action categories. In total, the dataset encompasses over 300,000 frames collected from 10 distinct scenes and 25 diverse subjects. Among these, 250,000 frames of data contain 3D human pose annotations captured using an advanced motion capture system and further optimized for accuracy. Furthermore, the dataset annotates 40 types of daily human actions, resulting in over 6,000 action clips. Through extensive experimentation, we have demonstrated the quality of HmPEAR and highlighted the challenges it presents to current methodologies. Additionally, we propose straightforward baselines leveraging sequential images and point clouds for 3D HPE and HAR, which underscore the mutual reinforcement between them, highlighting the potential for cross-task synergies.

Main paper: 
 introduction: 
 3D Human Pose Estimation (3D HPE) serves as the foundation for understanding human kinematics, offering detailed insights into the spatial configurations of body joints. This technology is fundamental to Augmented and Virtual Reality, simulations, and other related fields. On the other hand, Human Action Recognition (HAR) explores the dynamics of human movement, providing an advanced understanding of human behavior and intentions. The integration of these two domains, 3D HPE and HAR, enables a more comprehensive and nuanced interpretation of human actions. Therefore, creating a dataset that captures both 3D human poses and action categories is essential for in-depth research in human-centric studies.Initial HPE efforts focused on 2D pose estimation [12, 16, 75] . However, the development of human body shape models [49, 65] has allowed the representation of 3D human meshes with minimal parameters, facilitating the advancement of HPE from 2D to 3D [4, 37, 41, 46] . Nevertheless, reliance on RGB data presents limitations, such as sensitivity to lighting conditions and a lack of depth accuracy. While depth sensors like Kinect have offered insights into 3D HPE [5, 83] , their application in outdoor or large-scale settings is limited. The rise of LiDAR, which, due to its precision and independence from lighting conditions, has been integrated into recent 3D HPE datasets [20, 27, 42, 77] , facilitating advances in algorithmic development for 3D HPE [19, 42] .RGB-based HAR has seen extensive exploration, with approaches ranging from two-stream 2D CNNs [23, 31, 34, 61] to 3D CNNs [43, 68, 72] and transformers [2, 7, 48] . The current trends in RGB-based models are twofold: leveraging larger datasets or more complex models [2, 7, 44, 67] , and pursuing multi-modal fusion to mitigate the shortcomings of single-modal approaches [24, 45, 56, 73] . Employing feature fusion methods to enhances the robustness of HAR, such as depth [22] and skeleton data [1, 11, 21, 25] . However, the majority of existing RGBD or RGB-Pose action recognition datasets neglect the intricate complexities inherent in dynamic, real-world scenarios. Previous efforts, such as those by Xu et al. [77] , have made strides by incorporating LiDAR point clouds, yet they fall short in providing detailed 3D human pose annotations. This gap highlights an urgent need for a comprehensive dataset that integrates LiDAR point clouds and detailed human poses for HAR.In this paper, we present a novel dataset, named HmPEAR, which integrates imagery and point cloud data for human 3D HPE and HAR. This dataset was captured using a high-precision 128-beam LiDAR and three high-resolution cameras (1920 × 1200 pixels). The data processing procedure involved manual tracking and further refinements. For the annotations of 3D human poses, we utilized professional motion capture (MoCap) equipment. Additionally, for action labels, we annotated them manually. To facilitate comprehensive evaluations for 3D HPE and HAR, we proposed three baseline models. Extensive experiments with proposed models and existing methods demonstrate the dataset's applicability and challenges. Our contributions are summarized as follows:• We introduce a novel dataset that includes imagery and Li-DAR point cloud with 3D human pose and action annotations. It bridges the gap between the fields of 3D HPE and HAR. • Through extensive experiments, we have validated the quality and supplementary value of our dataset. These experiments also illustrated the challenges that our dataset poses. • Additionally, we introduce a novel multi-task method that integrates 3D HPE and HAR, demonstrating their potential viability to complement each other. 
related works 2.1 3d human pose estimation datasets: 
 The Human3.6M [33] dataset represented a significant leap in RGB-based 3D Human Pose Estimation(3D HPE), offering extensive indoor scene data, while MPI-INF-3DHP [51] continued this trend, remaining limited to indoor settings. As for outdoor scenarios, 3DPW [70] introduced the first outdoor 3D HPE dataset with imagery, MuCo-3DHP [52] offered a collection of outdoor multiperson scenarios, featuring authentic interactions and occlusions.However, they both lacked depth data, which is essential for full 3D context understanding. BEHAVE [9] introduced depth data using RGBD cameras, highlighting the critical role of depth information in such contexts. LiDARHuman26M [42] made strides by incorporating accurate LiDAR point cloud data. However, its potential was limited due to lower quality imagery and lack of synchronization. SLOPER4D [20] addressed these issues by enhancing annotation quality through refined optimization techniques. 3D HPE datasets often present challenges, as they predominantly feature human actions that are either performed casually or dramatically. This characteristic may lead models to learn poses in isolation without completely understanding the nuanced action semantics associated with those poses. As a result, there is a potential gap in comprehending the contextual meanings and intricacies of human movements within the datasets. This underscores the need for more comprehensive and context-aware 3D HPE datasets.Traditional Human Action Recognition(HAR) datasets have primarily consisted of RGB images, with a subset providing auxiliary depth maps and skeletal sequences. Early datasets such as KTH [58] and Weizmann [10] were limited in dataset size and scope. In recent years, various types of HAR datasets have emerged. UCF101 [62] and HMDB51 [39] have introduced RGB videos from diverse sources, covering a wide variety of action categories. Kinetics-400 [14] and Kinetics-600 [13] were developed to meet the demand for extensive data for effective model training. AVA [32] focuses on complex action detection and temporal localization, addressing real-world challenges in understanding human actions. On the other hand, NTU RGB+D [47, 59] pioneered to combination of RGB videos with depth maps and 3D skeletons, enriching datasets and enabling fusion-based HAR methods. BABEL [54] provides the dense action labels that are precisely aligned with their movement spans in the Motion Capture sequence. HuCentLife [77] encompasses diverse daily-life scenarios centering on people, assigning corresponding action labels to individuals in each scenario, and offering multi-modal data including imagery and point cloud. In the aforementioned multi-modal datasets, some lack point clouds, while others don't provide detailed 3D human poses. To address these gaps, there is a clear demand for a dataset that combines high-quality imagery and point cloud data with detailed 3D human poses and explicit action annotations.Early 3D HPE efforts using RGB imagery for directly estimating joint locations [57, 66, 71] or body shapes [38, 69] . Advances in statistical human body shape models [49, 65] have facilitated the estimation of 3D human poses using a limited number of parameters [36, 37, 46, 53] . This enhances the robustness of joint location and shape predictions. Sophisticated optimization techniques have been introduced, such as HybriK [41] , which decomposes body model parameter estimation, and NIKI [40] , which leverages invertible networks to boost robustness and accuracy with RGB data. SPIN [37] uses the estimated 2D pose as initialization in SMPL iterative optimization to produce more accurate 3D poses and shapes. The incorporation of depth data [5] also helps the estimation. In outdoor scenes, LiDAR [20, 42] caters to the need for long-range and more accurate depth. In our work, we combine RGB and LiDAR point clouds, leveraging the estimation of human 3D poses with HAR tasks to enhance the result of 3D HPE.Initially, the field of HAR has focused on integrating both temporal and spatial aspects. Baccouche.et al [3] combined the capabilities of 3D CNN with Long Short-Term Memory (LSTM) for spatiotemporal feature learning. SlowFast [30] is an early adopter of a dual-stream network architecture, capable of simultaneously capturing spatial and temporal features. Transformer-based methods [7, 44, 67] leverage attention mechanisms to capture long-range dependencies in temporal sequences. Recent developments in RGB-based HAR have introduced increasingly sophisticated strategies [17, 76] , pushing the boundaries of learning-based HAR algorithms. State-of-the-art methods [1, 11, 25] tested on NTU RGB+D 120 [47, 59] suggest that combining RGB and pose information can yield promising results, which underscores the significance of pose data in such tasks. PoseConv3D [25] utilizes a 3D heatmap volume to represent human joints, while InfoGCN [17] employs attention-based graph convolution to capture the context-dependent intrinsic topology of human actions. Other approaches like STMT [82] and LocATe [63] directly leverage motion capture data to address the challenges in HAR. Recent LiDAR-based HAR methods have been developed to adapt to large-scale datasets and enhance accuracy. Among them, P4Transformer [28] employs the Transformer architecture to extract spatiotemporal features from continuous point clouds. PSTNet [29] was introduced to capture local structures and temporal dynamics. Mast-Pre [60] is presented as a self-supervised learning approach, leveraging spatiotemporal point tube masking and reconstruction tasks to glean structural information from point cloud videos. However, the absence of a comprehensive dataset restricts the development of unified models that are capable of capturing human motion and recognizing human actions in a more integrated manner.  
calibration: 
 We obtained the intrinsic parameters of the three cameras through Checkerboard Calibration [80, 81] . To determine the extrinsic parameters of different sensors, we followed the methodology outlined in Wen's work [74] . This involved using a terrestrial laser scanning (TLS) sensor to acquire a precise scene point cloud as a reference. Subsequently, we registered the scanned point cloud from our Li-DAR and the captured images to this reference to calculate their relative transformations. Finally, we calculated the extrinsic parameters between all sensors using the relative transformations.To ensure sensor synchronization, it is crucial for all devices to operate under a unified timing mechanism and to capture data at specified times. We utilize the hardware-based Precision Time Protocol (PTP) to synchronize the sensors and the AGX Xavier with the master clock. Specifically, the Auto66 is configured as the grandmaster clock, the AGX Xavier serves as a boundary clock, and all sensors are configured as slave clocks. Furthermore, we engineered a simple circuit that triggers the three cameras to capture images at a predefined LiDAR rotation angle. Analysis of the PTP output verifies that the synchronization discrepancy among sensors is maintained below one millisecond. For the synchronization of 3D human pose data, we record jumping timestamps at the beginning and the end of each sequence and manually align it with the other data by detecting the peaks. Although his manual approach may slightly compromise synchronization precision, all sequences are optimized following the SLOPER4D [20] to refine the human pose for each frame to minimize the errors.Fig. 1 illustrates a typical data collection scenario, where the subject performed actions with various positions and orientations. The distances from the capturing device to the subject typically range from 5 to 10 meters. To ensure realism, we did not impose restrictions on the details or duration of actions performed by the subjects.The ground of point clouds is eliminated by detecting the ground in the first frame and then applying K-Nearest Neighbors (K-NN) based abstraction to all frames. Subsequently, we employ the DB-SCAN [26] clustering algorithm to identify clusters within the remained point cloud. By selecting a single point within the cluster of the subject, we finally get the human point cloud. The 3D bounding box (bbox) is calculated based on the human point cloud and projected onto the images to get the corresponding 2D bbox. To ensure accuracy, all 2D bbox in the images are manually checked. All action clips were annotated manually and subsequently reviewed for validation. For discrete actions, such as clapping, annotators were required to record the start and end of the action, ensuring completeness of the action clip. For continuous actions, like running, annotators were instructed to limit the duration of action clips to under five seconds, preventing the risk of excessive length.The SMPL [49] EQUATIONwhere 𝑉 𝑒𝑟 ′ stands for visible vertices at the viewpoint of the LiDAR. 
hmpear dataset: 
 HmPEAR is a novel dataset designed for 3D HPE and HAR. It encompasses over 300,000 frames of imagery, LiDAR point cloud, and bounding boxes of the subject, with more than 250,000 frames of corresponding 3D human poses. The dataset incorporates 40 daily actions performed by 25 subjects, resulting in over 6,000 action clips. These actions cover a wide range of human activities, including basic whole-body movements such as standing, walking, sitting, and running, specific actions involving the legs, such as kicking with the left leg and jumping, hand-related gestures like clapping and waving, as well as interactive behaviors such as making phone calls, wearing hats, and carrying bags. Each action clip varies in duration from 1.5 to 5 seconds. The dataset comprises 10 different scenes under various lighting conditions. A subset of typical actions from our dataset is illustrated in Fig. 3 .Tab. 1 provides a comparison of our dataset with other relevant datasets. Unlike existing 3D HPE datasets, which lack action annotations, HmPEAR encompasses annotations of 40 daily actions. In contrast to existing HAR datasets, which are limited to smallscale or indoor scenarios, HmPEAR expands the scope to larger and outdoor environments. By providing LiDAR point clouds and 3D human pose annotations, HmPEAR enhances the dataset's utility and applicability in advanced 3D HPE and HAR research. 
methodology: 
 We propose simple baselines, as depicted in Fig. 4 and Fig. 5 , composed of a Multi-modality Encoder, a Temporal Encoder, and a Multi-task Header. The subsequent sections will provide detailed descriptions of these modules and the architecture of the baselines.As illustrated in Fig. 4 , we employ the backbone of HRNet [64] to extract features of varying resolutions from each image. The features are subsequently fused by upsampling and addition. PointNet++ [55] is used to extract point-wise features from the human point clouds. Then, an affine transformation, derived to crop the original image to the input image, is used to project the point cloud features to a new feature map. The imagery and point cloud features are concatenated and further processed through multiple convolutional layers within the Fused-encoder to generate frame-wise multi-modal features.Encoder. To leverage temporal information, the multimodal features are fed into a bidirectional GRU [18] , enabling the extraction of both frame-wise and segment-wise features. The frame-wise features are denoted as {𝐹 𝐺𝑅𝑈 , 𝐽 𝑖 ). The resulting features are then input into a Spatial-Temporal Graph Convolutional Network (ST-GCN) [79] to produce action labels, denoted as Â 𝑐𝑡 = ST-GCN({𝐹 𝐶𝑎𝑡 𝑖 }). The segmentwise features are utilized to predict the human shape parameters β = 𝐿𝑖𝑛𝑒𝑎𝑟 (𝐻 𝐺𝑅𝑈 ), and, along with the predicted human joints {𝐽 𝑖 }, are input into HybrIK [41] to estimate human poses. Similar to the SMPL model, HybrIK represents human motions using a limited set of parameters: M𝑖 = 𝐻𝑦𝑏𝑟𝐼𝐾 (𝛽, 𝐽 𝑖 , 𝜙 𝑖 ). The parameter 𝜙 𝑖 denotes the twist along the direction between connected joints.We evaluate three distinct models, including PEAR-Proj, as shown in Fig. 4 , along with two additional models presented in Fig. 5 . In PE-Proj, we exclude the action classification head, which is denoted as the 3D HPE module. In AR-Proj, we omit the 3D HPE component, with the 𝐻 𝐺𝑅𝑈 being directly input into a Linear layer for generating classification results, denoted as the simple Classifier. These models aim to investigate potential cross-task synergies between 3D HPE and HAR, thereby demonstrating the effectiveness of our approach and dataset. datasets. Given the sampling rate 𝑝 𝑎𝑐𝑡 , a clip of 𝐶 𝑎𝑐𝑡𝑖𝑜𝑛 = {𝑅𝐺𝐵, 𝑃𝑜𝑖𝑛𝑡𝐶𝑙𝑜𝑢𝑑, 𝑎𝑐𝑡𝑖𝑜𝑛} is selected with probability 𝑝 𝑎𝑐𝑡 , while a clip of 𝐶 𝑆𝑀𝑃𝐿 = {𝑅𝐺𝐵, 𝑃𝑜𝑖𝑛𝑡𝐶𝑙𝑜𝑢𝑑, 𝑆𝑀𝑃𝐿} is chosen with probability (1 -𝑝 𝑎𝑐𝑡 ). For clips of 𝐶 𝑆𝑀𝑃𝐿 , only the L 𝑝𝑜𝑠𝑒 is calculated, and for 𝐶 𝑎𝑐𝑡𝑖𝑜𝑛 clips, only the L 𝑎𝑐𝑡 is calculated.The overall loss function of our model is formulated as a weighted sum of 3D HPE loss L 𝑝𝑜𝑠𝑒 and HAR loss L 𝑎𝑐𝑡 , with the balance parameter 𝜂 as a hyperparameter.EQUATIONThe action classification loss is computed using a cross-entropy loss method, which is simple yet effective.EQUATIONThe 3D HPE loss is composed of the following items:EQUATIONwhere L 𝑗𝑡𝑠24 represents the loss associated with the 24 joints of the SMPL model, and L 𝑗𝑡𝑠17 corresponds to the loss for the 17 joints defined by H36M, these joints are regressed using a fixed regressor 𝑅 ℎ36𝑚 , given M𝑖 = 𝐻𝑦𝑏𝑟𝐼𝐾 (𝛽, 𝐽 24 𝑖 , 𝜙 𝑖 ), we obtain Ĵ 17 𝑖 = 𝑅 ℎ36𝑚 𝑉 𝑒𝑟 ( M𝑖 ). For L 𝑗𝑡𝑠24 and L 𝑗𝑡𝑠17 , we employ the Mean Squared Error(MSE) metric. Additionally, the human pose parameter losses, L 𝛽 and L 𝜃 , are computed using the Mean Absolute Error(MAE) metric.EQUATION 
experiments: 
 In this section, we detail the training settings and experimental results. We also provide detailed analyses of these experiments, aiming to assess the quality of our dataset and explore the cross-task synergies between 3D HPE and HAR.For comparison purposes, alongside HmPEAR, we employ 3DPW, SLOPER4D, COCO, H36M, and MPI-INF-3DHP [51] during the training of various methods. For 3DPW, COCO, H36M, AMASS [50] , MPI-INF-3DHP, and SLOPER4D, we adhere to the official splits of the training and testing sets. Regarding HmPEAR, we employed different training and testing settings for 3D HPE and HAR. Firstly, we selected five subjects from different scenes as the testing set, denoted as 𝑇 𝑡𝑒𝑠𝑡 , and used the remaining data as the training set, denoted as 𝑇 𝑡𝑟𝑎𝑖𝑛 . Secondly, all data in 𝑇 𝑡𝑒𝑠𝑡 and 𝑇 𝑡𝑟𝑎𝑖𝑛 with action labels were used as the testing and training sets for HAR. Finally, 2 out of 5 subjects from 𝑇 𝑡𝑒𝑠𝑡 and 8 out of 20 subjects from 𝑇 𝑡𝑟𝑎𝑖𝑛 were used as the testing and training sets for 3D HPE. This setting for 3D HPE resulted in a subset of our proposed dataset of similar size as SLOPER4D.For our proposed models, all images are cropped to a shape of 256x256 for input. The segment length, 𝑚, is set to 10. During training, we utilize a pre-trained HRNet. The initial learning rate is set to 2 × 10 -4 and is multiplied by a decay factor of 0.5 at epochs 2 and 5. We employ the Adam optimizer [35] for optimization. The training is conducted on two NVIDIA GeForce RTX 3090 GPUs for 15 epochs. For the evaluation of 3D HPE, we employ the Mean Per Joint Position Error (MPJPE) and the Procrustes-Aligned Mean Per Joint Position Error (PA-MPJPE) as metrics. To assess action classification, we utilize per-clip and per-segment accuracy measurements.6.2 Cross-Dataset Evaluation on 3D HPE 6.2.1 RGB based 3D HPE. We evaluate RGB-based 3D HPE methods, including VIBE and HybrIK, on 3DPW, SLOPER4D and our dataset(HmPEAR). The results are presented in Tab. 2. As VIBE requires AMASS for its motion discriminator, we trained all models from scratch with AMASS included. VIBE, trained on AMASS, HmPEAR, achieves better PA-MPJPE performance on SLOPER4D than the same model trained on AMASS, SLOPER4D. This underscores the quality and robustness of our proposed dataset. VIBE trained on {AMASS, HmPEAR} also yields better results when tested on 3DPW. However, this may be influenced by the domain gap between the datasets. Following the setting in HybrIK's original paper, we use all additional datasets including COCO, H36M, and MPI-INF-3DHP. Models trained on {+HmPEAR} demonstrate competitive performance.Point cloud based 3D HPE. We evaluate the Li-DARCap [42] on LiDARHuman26M(LH26m), SLOPER4D, and our HmPEAR, the results are shown at Tab. 3. It is important to note that the LH26m employs a solid-state LiDAR for point cloud data collection, while both the SLOPER4D and our dataset utilize a mechanical LiDAR. This distinction may account for the performance decline when models trained on the LH26m dataset are tested on others, and vice versa. When comparing models trained on different datasets, LiDARCap exhibits limited generalization capabilities. Thought, the model trained on HmPEAR shows relatively better performance.6.3 Evaluation of Proposed Models for 3D HPE 6.3.1 Multi-modality based 3D HPE. As shown in Tab. 4, when trained on the SLOPER4D dataset or our HmPEAR, PE-Proj demonstrated a degraded performance. This suggests that PE-Proj may be underperforming due to underfitting, as the SLOPER4D dataset or a subset of HmPEAR may not provide sufficient information for comprehensive learning. Notably, when SLOPER4D and our dataset were combined for training, PE-Proj exhibited improvement in performance and generalization across different testing sets. This underscores the supplementary value our dataset contributes to current 3D HPE datasets and the capabilities of the multi-modal model.To investigate the contributions of different modalities, we further set the feature in PE-Proj of imagery or point cloud to 0, denoted as input type of 'PC' and 'RGB' in Tab. 4. Neither the PE-Proj(PC) nor the PE-Proj(RGB) could achieve comparable performance as the multi-modal model. The PE-Proj (PC) showed a more serious performance degradation, which we suppose should be attributed to the sparsity of the projected point cloud features.For PEAR-Proj, we recorded the optimal models for 3D HPE and HAR, labeled as 'BestPE' and 'BestAR' respectively, as presented in Tab. 4. The PEAR-Proj, when trained on HmPEAR, demonstrates results comparable to those of PE-Proj trained on both SLOPER4D and HmPEAR. This highlights the effectiveness of the cross-task synergies between 3D HPE and HAR. We further visualize several 3D HPE results in Fig. 6 (please zoom in for better visualization). The human SMPL models colored yellow represent the ground truth, while the human SMPL models colored blue are predictions by the PEAR-Proj.We further evaluate several prevalent action recognition models on HmPEAR, including both RGB-based and point cloud (PC) based methods. All RGB-based methods, including I3D [14] , Slow-Fast [30] , TimeSformer [7] and UniFormer [44] were pre-trained on the Kinetics-400  
discussion 7.1 limitations: 
 It is important to acknowledge that the process of optimizing human poses and annotating actions in HmPEAR was resource-intensive. Consequently, we are unable to provide a large-scale dataset currently. Additionally, the age of subjects in our dataset falls within the range of 20 to 30, which may not provide sufficient diversity for training larger models. Furthermore, our dataset accounts for variations in time and lighting, but it does not yet include different weather conditions. In our future work, we plan to expand the dataset by incorporating a wider range of actions, more subjects, and diverse scenes with various weather conditions.In this work, we introduce HmPEAR, a dataset designed to bridge the fields of 3D HPE and HAR. Through cross-dataset experiments, we have demonstrated the quality and robustness of our dataset. Furthermore, we proposed baseline models that incorporate multimodal inputs and multi-task outputs. Comprehensive experiments and analyses showcased the ability of these models to leverage crosstask synergies between 3D HPE and HAR, offering a promising direction for future research on human pose and action analysis.  
 
 Appendix:  


Decision: This paper proposes a novel multimodal dataset crafted for advancing research in 3D Human Pose Estimation and Human Action Recognition, with a primary focus on outdoor environments.
The paper is clearly written, the contributed dataset is large-scale and it has been carefully designed, the cross data evaluation on pose estimation proposes competitive results. 
For all these reasons all reviewers agreed on accepting the paper and I also recommend acceptance.

Reviews: review: paper_summary: paper_summary: this paper introduces HmPEAR, a novel dataset crafted for advancing research in 3D Human Pose Estimation (3D HPE) and Human Action
Recognition (HAR), with a primary focus on outdoor environments.
strength_weakness: strength_weakness: 1. dataset contribution is good, compared to other point clouds based benchmark, the proposed dataset has larger scale
2. the paper is well-written and clear
limitations: limitations: 1. the proposed method is a bit complex, usually for a dataset paper, we want a simple yet efficient baseline, could author show a very naive version of method? (like point cloud encoder and keypoint regression decoder)
2. the results in Table 2 seems mean that the proposed dataset for training is not so useful, it makes contribution weak, could author explain more about the results?
paper_summary: paper_summary: This paper introduces a new dataset called HmPEAR to connect the field if 3D Human Pose Estimation (3D HPE) and Human Action Recognition (HAR). This dataset comprises imagery, LiDAR point cloud along with human pose and action annotations.  In addition, this paper introduces a mukti task combining the human pose estimation and human action recognition.
strength_weakness: strength_weakness: 1. The paper is well structured and well written.
2. Introduction of a novel dataset always benefits the community.
3. The cross data evaluation on pose estimation proposes competitive results.
limitations: limitations: 1. Baseline experiments seems limited.
2. Time/Frequency synchronisation needs to more clearly mention.
3. For the temporal encoders why not consider TCNs?
4. Did the authors consider using transformer encoders for the PEAR-Proj model?
paper_summary: paper_summary: This work proposes HmPEAR, a dataset for human pose estimation and action recognition, consisting of 300k frames collected from 10 scenes and 25 subjects. For each RGB video, there are corresponding LiDAR point clouds, 3D human poses, and action categories as annotations. The data is captured using one LiDAR and three RGB cameras with time synchronization. Bounding box annotations are obtained by performing DB-SCAN on the point cloud, action labels are manually annotated, and SMPL annotations are calculated using SLOPER4D. Cross-dataset experiments demonstrate that HmPEAR has better generalization capabilities for both RGB-based and LiDAR-based 3D pose estimation.
strength_weakness: strength_weakness: 1. The tools for data collection and the data annotation pipeline are professional.
2. The dataset contains the highest number of annotated SMPL frames compared to previous datasets.
3. The overall experiments demonstrate that models trained on HmPEAR exhibit good generalization.
4. The paper is well-written and easy to follow.
limitations: limitations: ## Dataset:
1. The number of total frames, scenes, action classes, and subjects is still limited compared to previous datasets. It would be beneficial to have more scenes and action classes in future versions of the dataset.
2. For the action class labels, it would be advantageous to include captions, such as "a person standing up then turns around." The authors could provide an official results of label-to-sentence augmentation (with LLM or human labor), for example, "carrying sth" can be augmented to many sentences, such as "a person is carrying an object." or "someone is holding and moving something." Such annotations can be helpful for CLIP-like models and text-to-motion tasks.
3. Will the data, data collection, and annotation pipeline be open-sourced?

## Evaluation:
1. In Table 2: RGB-based 3D pose estimation, the HybrIK evaluation results on the proposed dataset are missing(SLOPER4D and 3DPW), making it difficult to compare HmPEAR with SLOPER4D and 3DPW. It would be better to provide the results on HmPEAR instead of just citing from the previous paper.
paper_summary: paper_summary: The paper proposes a 3D human pose estimation and action recognition dataset . It further provides a baseline of 3D pose estimation and 3D human action recognition.
strength_weakness: strength_weakness: The paper provides an interesting dataset for a very challenging problem of 3D action recognition and pose estimation by providing a novel dataset and a baseline approach.
limitations: limitations: 1) The paper is missing multiple ablation studies:
- For example, Section 5.1.1 mentions the fusion of 2D image features and 3D point cloud features using additional fusion encoder. What is the intuition mechanism behind concatenating the features using fusion encoder. What is performance if the features are fused simple using concatenation or addition?

2) What are the reason behind using the architectures PointNet++ and HRNet, when better models for example:
- point cloud feature extraction: PointNeXt (NeurIPS’22)
- and pose-estimation: TransPose (ICCV’21) and PCT (CVPR’23)
are already available.

3) It is still unclear the need for the proposed dataset as it only contains 25 subjects, whereas a large scale dataset like HuCenLife (ICCV’23) containing 65k subjects while including both RGB frames and 3D point clouds sample. Wouldn’t it be a good idea to take an existing dataset and generate the annotations?

4) The idea of doing a joint pose estimation and action recognition seems to be not well motivated in the paper.

5) Table 1 is very confusing as it is not easy to distinguished the output modalities and the metadata. It will be nice to take inspiration from Table 1 of paper: “Human-centric Scene Understanding for 3D Large-scale Scenarios” to distinguish among different components of the proposed and the existing datasets.

6) The method section of the paper is not clearly understandable:
- What is the intuition behind solving the 3D action recognition and 3D pose estimation jointly.
- It will be very helpful to mention the output dimension of HRNet and PointNet. If the output from the PointNet is reduced to simple channel, then it seems to be only retaining the global features. However, the same features are later used for “Seg-wise feature”. A little detailed information in this case will be very helpful in understanding the main idea behind this architecture.


7) The presentation of paper can be improved:
- Paper contains multiple abbreviation, which makes it difficult to understand. Consider replacing some abbreviations with their full forms.
- SMPL annotation are mentioned in the first page of the paper. However, the idea of SMPL was not clear before section 3.7. It would be nice idea to include a brief explanation in the introduction.
- AMASS dataset appear for the first time in the results section. If that is one of the similar data then why is it not included in Table 1 comparison.
- Is there any order or sorting dataset in Table 1? If yes, it is not clear enough.

8) In the result section,
- paper is missing a brief explanation of the evaluation metrics.
- paper is missing comparison with the SOTA. Please refer to the paper: “WHAM: Reconstructing World-grounded Humans with Accurate 3D Motion”. There are multiple methods performing better than the proposed approach on 3DPW dataset. Examples:
- TRACE: 5D Temporal Regression of Avatars with Dynamic Cameras in 3D Environments. In IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), 2023
- Decoupling human and camera motion from videos in the wild. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.