Towards Practical Human Motion Prediction with LiDAR Point Clouds

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-02 05:46:08

Abstract: Human motion prediction is crucial for human-centric multimedia understanding and interacting. Current methods typically rely on ground truth human poses as observed input, which is not practical for real-world scenarios where only raw visual sensor data is available. To implement these methods in practice, a pre-phrase of pose estimation is essential. However, such two-stage approach often leads to performance degradation due to the accumulation of errors.
Moreover, reducing raw visual data to sparse keypoint representations significantly diminishes the density of information, resulting in the loss of fine-grained features. In this paper, we propose LiDAR-HMP, the first single-LiDAR-based 3D human motion prediction approach, which receives the raw LiDAR point cloud as input and forecasts future 3D human poses directly. Building upon our novel structure-aware body feature descriptor, LiDAR-HMP adaptively maps the observed motion manifold to future poses and effectively models the spatial-temporal correlations of human motions for further refinement of prediction results. Extensive experiments show that our method achieves state-of-the-art performance on two public benchmarks and demonstrates remarkable robustness and efficacy in real-world deployments.

Main paper: 
 introduction: 
 Bridging the gap between human actions and digital interactions, human motion prediction stands as a pivotal innovation in the ever-evolving field of multimedia. By accurately predicting future human movements through the analysis of past motion sequences, people are able to create multimedia content that is more intuitive, responsive, and engaging. For instance, fitness apps can offer personalized guidance and feedback in real-time based on the predicted future movements of their users. Assistive robots can predict human intentions and proactively handover objects to users as they reach out, instead of waiting until they stop moving.Current human motion prediction methods [1, 12, 28, 38, 44, 46, 47, 52] have achieved remarkable success by explicitly modeling the inherent body structural and dynamic features among sequential human skeletons. However, these approaches heavily rely on precise historical human poses, necessitating complex setups like multi-view cameras or densely deployed wearable IMUs, thus imposing substantial limitations on their practical application and deployment. In multimedia applications such as augmented reality(AR) environments, the system is expected to interpret human actions and gestures in real-time using their onboard sensors, with raw data often comprising images or point clouds from a single viewpoint. The dependency on the accuracy of past motion estimation techniques [17, 22, 36, 45, 50, 55] leads to cumulative errors as shown in the upper-right corner of Fig. 1 and the loss of fine-grained features present in the raw visual data. To advance practical human motion prediction, directly taking the raw visual data as input is a more viable strategy. Some studies [39, 51] have investigated video inputs to improve applicability, the absence of precise depth information and sensitivity to lighting conditions frequently results in ambiguous and inconsistent predictions in unconstrained environments.In recent years, LiDAR's long-range depth-sensing and lightinsensitive capabilities have established it as an indispensable sensor in the realm of autonomous driving [4, 23, 33, 43] and various human-centric applications [17, 21, 36, 37, 53] , such as 3D pose estimation, human motion capture, gait recognition, action recognition, etc. Studies in these areas have underscored the superiority of LiDAR over traditional cameras, highlighting its robustness and effectiveness due to its ability to provide precise 3D geometry and dynamic motion information about humans in free environment. Crucially, LiDAR is not affected by lighting conditions, presenting a significant advantage for real-world deployment. Based on this, we introduce LiDAR-HMP, the first LiDAR-based human motion prediction solution, which directly predicts future human poses from raw LiDAR point clouds and is more practical for real-world multimedia applications.Our approach is founded on three carefully crafted modules that effectively harness the valuable information contained within the raw point cloud input, enabling precise predictions of future human motion as shown in Fig. 1 . Firstly, the Structure-aware Body Feature Descriptor delves into the implicit semantics of human body structure entailed by the point cloud observation, amalgamating part-wise with global body features. This ensures a rich, detailed representation of human structure essential for accurate motion forecasting. Following this, our Adaptive Motion Latent Mapping module employs a set of learnable queries, each mapping to a future motion frame. These queries dynamically extract critical information from the observed point cloud frames, laying the foundation for precise motion prediction. Finally, the Spatial-Temporal Correlations Refinement module embeds coarse motion predictions and part-wise body features into a high-dimensional space. Leveraging spatial-temporal correlations among the predicted poses, it meticulously refines the predicted keypoints, enhancing the prediction accuracy. Together, these modules embody our design philosophy of capturing both the spatial structure and temporal dynamics of human motion, setting a new benchmark for LiDAR-based human motion prediction in real-world robotic applications.Notably, acknowledging the inherent unpredictability and potential for sudden changes in human movement, we extend our model to support diverse motion predictions [9, 24, 27, 32, 49] , thus broadening its applicability and enhancing predictive reliability and robustness. Our experimental results and visual analysis both indicate that diverse predictions not only enhance the robustness and reliability of forecasts but also offer invaluable insights for applications requiring an in-depth understanding of human dynamics. For instance, in interactive systems and autonomous navigation, the ability to predict a range of potential motions can significantly improve the decision-making process.We conduct experiments on both short-term and long-term human motion prediction on two public LiDAR-based human motion datasets, including LiDARHuman26M [17] and LIPD [53] . Our method significantly outperforms existing methods in terms of mean per joint position error (MPJPE) by a large margin, e.g. average 17.42𝑚𝑚 and 7.86𝑚𝑚 for short-term prediction, and average 11.62𝑚𝑚 and 9.12𝑚𝑚 for long-term prediction, respectively. We also evaluate the robustness and generalization ability of our method under long-range distance, occluded and noisy cases. In particular, we have deployed our method in real scenarios and achieved real-time human motion predictions, providing a feasible interface for further robotic applications. Our contributions can be summarized as follows:• We propose the first LiDAR-based method for practical 3D human motion prediction by fully utilizing the fine-grained motion details in raw LiDAR point clouds, which closely aligns with real-world application scenarios. • We present a structure-aware body feature descriptor that decouples the holistic human point cloud into distinct body parts based on the semantics of anatomical structure, enabling the capture of fine-grained dynamic motion details. • We map the past motion manifold to the future motion manifold adaptively, and fully leverage the spatial-temporal correlation of the motions to further refine the predicted results. • Our method achieves state-of-the-art performance on two public LiDAR-based human motion datasets. 
related work 2.1 human motion prediction: 
 Existing human motion prediction methods focus on forecasting future human motions based on past ground truth observations captured through multi-view cameras or dense IMUs, with specialized network structures to address the spatial and temporal dependencies in structured skeleton sequences. Several methods [11, 13, 16, 25, 31, 47] have explored modeling temporal dependencies in human motion prediction using RNN and TCN structures, but often overlook spatial relationships. Recent advancements [3, 8, 19, 42, 52] have seen GCNs achieving state-of-the-art results by learning spatial dependencies among joints with learnable weights. Building on the GCN framework, DMGNN [19] and MSR-GCN [8] introduced multi-scale body graphs to capture both local and global spatial features. PGBIG [28] extended this approach by incorporating temporal graph convolutions for extracting spatial-temporal features. SPGSN [18] introduced a graph scattering network to enhance the modeling of temporal dependencies through multiple graph spectrum bands. DMAG [12] uses frequency decomposition and feature aggregation respectively to encode the information. Beyond GCN-based approaches, transformer architectures [1, 2, 30] also have been utilized to model pair-wise spatialtemporal dependencies, indicating a broad interest in capturing complex spatial-temporal relationships in human motion prediction. AuxFormer [46] introduced a model learning framework with an auxiliary task that can recover corrupted coordinates depending on the rest coordinates. [44] synthesizes the modeling ability of the self-attention mechanism and the effectiveness of graph neural networks. However, these methods are primarily designed for inputs of ground truth observed skeletons and showed performance degradation with inputs of estimated skeletons from 3D pose estimation methods [14, 17, 22, 26, 34, 36, 45, 50, 55] due to accumulative errors, restricting their real-world applicability. Although some studies [39, 51] have explored the use of video inputs to enhance practical applicability, the absence of precise depth information and sensitivity to lighting changes frequently result in ambiguous and inconsistent predictions in uncontrolled settings.Due to the accurate depth sensing and light-insensitive ability in long-range scenes, LiDAR has emerged as a pivotal perception sensor for robots and autonomous driving [4, 15, 23, 33, 43, 54] . In recent years, numerous human-centric applications [5-7, 17, 21, 36, 37, 48, 53] , such as pose estimation, motion capture, action recognition, gait recognition, scene reconstruction, etc., have adopted LiDAR to expand usage scenarios and improve performance of solutions by utilizing accurate geometric characteristics of LiDAR point clouds. Especially, recent works [17, 36, 53] have already underscored the efficacy of single-LiDAR systems for human motion capture. Given LiDAR's success in large-scale human-related applications, we make the first attempt to leverage LiDAR for practical 3D human motion predictions by fully exploiting the dynamic spatial-temporal correlations among observed sequential LiDAR point clouds. 
methodology: 
 In this section, we introduce the details of our single-LiDAR-based 3D human motion prediction method LiDAR-HMP. After defining the problem in Sec. Contrary to the inherently structured skeleton-based human motion representations, the sparse and disordered nature of LiDAR point clouds pose substantial challenges for efficient feature extraction. Existing methods in human-focused research [36, 53] commonly adopt PointNet [35] to extract bodily features from point clouds, using max pooling to distill key features from the LiDAR data. While efficient and widely applied in existing LiDAR-based human analytics, such an approach overlooks the rich geometric and structural details in the 3D point cloud data, which are essential for deriving expressive features that accurately represent human body structure. To overcome these limitations, we introduce the Structure-aware Body Feature Descriptor. Our approach combines the holistic global feature of the entire body with detailed local features that capture the semantics of anatomical body parts, including the thigh, calf, arm, and more. This decomposition enables a nuanced and comprehensive representation of human motion, facilitating a deeper understanding of complex movements. Specifically, given a frame of point cloud observation of the past human motion 𝑃 𝑖 = {𝑝 1 , ..., 𝑝 𝑁 } ∈ P 1:𝑇 𝑜 , we segment the human body point cloud into 𝐾 semantically meaningful anatomical parts via a pre-trained human part segmentation model Φ(•). Details of the pre-trained human parsing model are illustrated in the supplementary materials. We retrieve the point-wise features 𝐹 𝑖 = {𝑓 1 , 𝑓 2 , . . . , 𝑓 𝑁 } from the PointNet encoder before the global max-pooling. These point-wise features are then scattered into 𝐾 bins corresponding to different body part segments. Within each bin, we perform max-pooling across the points to obtain part-wise feature 𝐻 𝑝𝑎𝑟𝑡 ∈ R 𝑇 𝑜 ×𝐾 ×𝑑 1 :EQUATIONWe then integrate the global body feature 𝐻 𝑔𝑙𝑜 yielded by the Point-Net encoder with the part-wise feature to obtain the structureaware body feature descriptor 𝐻 :EQUATIONwhere ⊕ denotes concatenation. The feature descriptor is further enhanced with a spatial-temporal transformer layer, which models the non-linear dependencies among the body parts across different frames, further refining the descriptor's ability to capture complex bodily dynamics. Building upon the expressive feature descriptor of the observed Li-DAR point cloud frames, we advance towards predicting the future 3D human motion from historical motion observations. Specifically, we initialize a set of learnable motion queries 𝐻 𝑞 ∈ R (𝑇 𝑜 +𝑇 𝑝 ) × (𝐾+1) ×𝑑 1 following a normal distribution N (0, 0.02), designated for each of the observed and predicted frames. These queries then selectively assimilate information from past frame descriptors within a transformer decoder layer to translate past frame features into a comprehensive feature set that encapsulates both past observations and future predictions, thereby laying the foundation for accurate human motion forecasting:EQUATIONwhere MHCA(• query, • key, • value ) represents the multi-head cross-attention layer [41] and 𝑊 𝑞 , 𝑊 𝑘 , 𝑊 𝑣 are the linear projection for query, key and value. To further reinforce the spatial-temporal consistency within the updated body features, we apply spatial and temporal transformer layers, progressively refining the predicted motion features 𝐹 into 𝐹 ′ . Finally, we regress the coarse 3D joints Ĵ𝑡 ∈ R 24×3 through the refined motion features 𝐹 ′ . The loss function can be formulated as below:EQUATION)Leveraging the aforementioned modules, we derive the initial predicted 3D skeletal joints. However, given the structured nature of human joint motion, it is crucial to explicitly model the spatialtemporal correlations among various human joints for accurate motion prediction. To address this, we employ a spatial-temporal transformer to integrate joint features, local semantic features, and global geometric features. This facilitates dynamic refinement of these correlations, effectively modeling the spatio-temporal contextual information. Specifically, we first embed the coarse 3D pose prediction Ĵ = { Ĵ1 , Ĵ2 , . . . , Ĵ 𝑇 } into joint-wise features using an Multilayer Perceptron(MLP) layer:EQUATION)where 𝑇 = 𝑇 𝑜 + 𝑇 𝑝 . Similarly, we transform the refined motion features 𝐹 ′ to the same dimension:EQUATION)These are concatenated to form tokens 𝑇 𝐾 = 𝑬 joint ⊕ 𝑬 𝐹 for the transformer layers. To explore implicit information within these tokens, we deploy a spatial transformer (STFormer) 𝐹 𝑆𝑇 𝐹 and a temporal transformer (TTFormer) 𝐹 𝑇𝑇 𝐹 to capture the dependencies within spatial and temporal dimensions, respectively. The STFormer aims to model the spatial dependencies among all tokens within a frame. We slice the token embedding 𝑇 𝐾 at the spatial dimension and encode the 𝑖 th token 𝑇 𝐾 𝑖 with a learnable spatial positional encoding to signify each token's relative spatial location before processing by STFormer. Utilizing the self-attention mechanism, STFormer models the dependencies of all tokens per frame, yielding spatially consecutive token-wise features as below:EQUATIONConversely, TTFormer is tasked with discerning the temporal correlations to ensure temporal consistency and motion accuracy. It regards each feature across the sequence as tokens, generating features 𝑇 𝐾 𝑡 ∈ R (𝐾+25) ×𝑑 2 with a total of 𝑇 tokens, where 𝑇 𝐾 𝑡 represents the 𝑡 th token of the token embedding 𝑇 𝐾 sliced in the temporal dimension. Additionally, we append a learnable temporal positional encoding to 𝑇 𝐾 𝑖 to mark each token's temporal position in the sequence. The TTFormer then encodes the temporally modeled motion features as:EQUATION)We adopt STFormer and TTFormer layers alternatively, which enables a thorough modeling of spatial-temporal correlations. STFormer layers effectively capture the complex spatial relationships present in the data, whereas TTFormer layers adeptly model the temporal dynamics of human motion. By focusing on both spatial and temporal aspects, essentially the "where" and the "when" of motion, our approach achieves a comprehensive analysis and synthesis, leading to more accurate and coherent motion predictions.In the network's final phase, we employ dual heads to decode human skeletons and point clouds concurrently, utilizing Multilayer Perceptrons (MLPs). The joint-wise regression head utilizes three MLP layers to decode the final joints Ĵ ′ 𝑡 ∈ R 24×3 . To supervise the training of this head, we apply an 𝐿 2 loss:EQUATION)For the point-wise regression head, three MLP layers are employed to decode each local-semantic feature into points corresponding to each human body part. With 𝐾 predefined body parts, we achieve the final point clouds representation P′ 𝑡 ∈ R 𝐾 ×32×3 . We compare the point cloud obtained from the network outputs P𝑡 = p ∈ R 3 , P𝑡 = 𝑁 , with P𝑡 with the ground truth point cloud 𝑃 𝑡 = 𝑝 ∈ R 3 , |𝑃 𝑡 | = 𝑀 by Chamfer Distance as follow:EQUATION)The overall loss is:EQUATIONGiven the highly subjective nature of human behaviour, motion prediction requires multiple potential future predictions. We extend our approach to diverse human motion prediction setting. Following the above framework, we implement 𝐾 = 4 multiple learnable motion queries in our adaptive motion intention prediction module.For each probable motion feature, they share all the network weight during the training process. We can regard each learnable query as a different motion mode that can map the past motion into multiple future motion latent space with various motion patterns.The "winner-take-all" (WTA) training strategy [10, 20] is employed, which only optimizes the best prediction with minimal average prediction error to the ground truth human motion. 
experiments: 
 In this section, we begin by introducing the datasets (Sec.4.1), implementation details (Sec.4.2). Subsequently, we conduct a qualitative and quantitative comparison between our method and current state-of-the-art (SOTA) methods on two publicly available LiDARbased human motion datasets (Sec.4.3), showcasing the superiority and generalization capabilities of our approach. Additionally, we conduct ablation studies to illustrate the superiority of our network design (Sec.4.4). Moreover, we analyze the robustness and generalization ability of our method (Sec.4.5). Finally, we present experimental results on diverse human motion predictions in Sec.4.6, highlighting its importance and utility in real-world scenarios, particularly in long-term prediction, where human behavior is highly unpredictable and emergent.LIPD [53] We build our network on PyTorch 2.1.0 and CUDA 12.1, trained over 100 epochs with batch size of 128, using an initial learning rate of 10 -4 . We set the hyper parameters of our model as {𝐾 = 9, 𝑑 1 = 1024, 𝑑 2 = 512} across all experiments. The process was run on a server equipped with two Intel(R) Xeon(R) E5-2678 CPUs and 4 NVIDIA RTX3090 GPUs. We resample each frame of input point clouds 𝑃 𝑖 to a fixed 𝑁 = 256 points by the farthest point sample algorithm, then we subtract the center coordinates of point clouds to normalize the input data. Following skeleton-based human motion prediction methods [18, 28] , our model takes 0.4s continuous point clouds as input(4 frames for LiDAR with 10fps), and we predict the future 0.4s(4 frames) and 1.0s(10 frames) human motion for short-term and long-term prediction, respectively. For compared methods, we keep the same training strategy as in the original paper.For training data, we take training set of LiDAR-related human motion dataset LIPD [53] , LiDARHuman26M [17] , and synthetic dataset of a subset of AMASS [29] , including ACCAD, BMLMovi, CMU following LIP [53] . We leverage Mean Per Joint Position Error (MPJPE) in millimeters as our evaluation metric, which is a common metric used for evaluating the accuracy of predicted 3D human poses against ground truth data.We evaluate LiDAR-HMP against several state-of-the-art methods [17, 28, 36, 40, 46, 47, 53] on two public datasets (LIPD [53] and LiDARHuman26M [17] ) to demonstrate its superiority in 3D human motion prediction task. The results for short-term and longterm human motion prediction are shown in Tab. 1 and Tab. 2, respectively.(1) Comparison with two-stage methods. Conventional methods for human motion prediction typically rely on historical ground truth skeletons as input. In contrast, our method utilizes historical LiDAR frames alone. We integrate results from state-of-theart LiDAR-based 3D human pose estimation methods into these skeleton-based approaches. Due to accumulative error and the loss of fine-grained features, two-stage methods often show degraded performance in real-world scenarios that depend solely on raw sensor data. In contrast, our approach demonstrates substantial improvements, achieving enhancements of average 17.42mm and 7.86mm in short-term human motion prediction, and average 11.62mm and 9.12mm in long-term prediction on the LIPD and Li-DARHuman26M datasets, respectively. This superior performance stems from our method's capability to fully capture the dynamic spatial-temporal motion features from raw LiDAR point clouds.Our approach not only bypasses the inaccuracies introduced by intermediate skeletons from 3D pose estimation methods, but also retains the fine-grained, point-level features from the raw LiDAR data.(2) Comparison with diffusion-based method. In addition, we also adapt the Motion Diffusion Model (MDM) [40] methodology for LiDAR-based human motion prediction by substituting its text encoder with a PointNet [35] encoder for another comparison. From Tab. 1 and Tab. 2, we can observe that diffusion-based methods obtain limited performance. This is mainly because the diffusion-based method relies on gradually denoising a signal towards generating a coherent output, may struggle with the variability and noise inherent in point cloud data. The nuances of human motion, particularly subtle movements are lost or inadequately captured during the diffusion process.Furthermore, visual comparisons in Fig. 3 highlight our method's superiority in predicting long-term complex motions. Such as predicting the motion of bending down and getting up in LIPD and the motion of the lowering left hand and raising the right hand in LiDARHuman26M, MDM and two-stage method can only predict short-term motions or remain consistent with historical motions. By effectively modeling dynamic spatial-temporal correlations among structure-aware motion features from raw visual input, our approach offers enhanced accuracy and applicability in real-world scenarios of 3D human motion prediction.(1) Ablation on overall network design. We assess the efficacy of our proposed modules through comprehensive ablation studies on LIPD [53] dataset, as detailed in Tab. 3. Notably, our baseline models focus solely on global-geometric feature extraction and incorporate the adaptive motion latent mapping module. Incorporating finegrained local-semantic features leads to average 2.65𝑚𝑚, 2.04𝑚𝑚 improvements on short-term and long-term prediction, respectively. The addition of our spatial-temporal correlations refinement module, which explicitly models the spatial-temporal relationships among distinct body joints and structure-aware motion features, further refines our 3D human motion predictions. Introducing an auxiliary points regression head also contributes to performance improvements by providing additional supervision, thereby enhancing the network's predictive capabilities.(2) Structure-aware body feature descriptor. The analysis presented in Tab. 4 underscores the importance of fine-grained localsemantic feature modeling, especially for capturing nuanced body movements. Our approach effectively leverages both holistic globalgeometric motion features and detailed local-semantic information, delivering more robust and precise human motion predictions.(3) Adaptive motion latent mapping. To validate the effectiveness of our adaptive motion latent mapping module, we conducted experiments by replacing the module's learnable motion query with a linear mapping in the temporal dimensions using an MLP. Additionally, we compare our method against a "static padding" approach, where future motions are extrapolated by replicating the last frame of observed motion at the input stage. As shown in Tab. 5, our approach achieves superior performance. This is attributed to our (1) Distance. To assess the generalization capability of our LiDAR-HMP across point clouds of varying sparsity and distances, we evaluate our method on the LIPD testing sets. Performance of the short-term motion prediction at 0.4s and long-term motion prediction at 1.0s in different distances is illustrated in Figure 4 . Despite the decreasing density of LiDAR human point clouds with increasing distance results in sparser representations, our method still maintains stable performance up to a long-range distance of 17 meters. This stability is crucial for large-scale applications such as autonomous driving and robot obstacle avoidance, where motion prediction at long ranges is essential. At about 20 meters, the point cloud may reduce to roughly 30 points, capturing only the basic human outline, which poses challenges for capturing detailed movements. Nonetheless, our method consistently delivers the best performance, even under these extreme conditions. Short-term(0.4s) MPJPE(mm) (2) Occlusion and noise. To evaluate the robustness of our algorithm against occlusions and noise, we conduct simulations on the LIPD dataset. We introduce random noise by adding 30 noise points around the human point clouds and apply noise to 20%, 40%, and 80% of the frames, respectively. For occlusion simulation, we randomly mask points within a cubic region measuring 0.4 meters on each side on the human body, affecting 20%, 40%, and 80% of the frames respectively. The results, as presented in Tab. 7 and Tab. 8, demonstrate that even under severe 80% noise or occlusion, the performance of our algorithm only slightly decreases, underscoring its robustness to noise and occlusions. More visualization results on noise and occlusion cases are detailed in our supplementary.(3) Real-world applications. Fig. 1 shows that our method is practical for in-the-wild scenarios, capturing human motion in real-world scenarios day and night with real-time performance about 40 fps. This strongly demonstrates the feasibility and superiority of our method in real-life applications.Given the inherent spontaneity and unpredictability of human behavior, deterministic action predictions, which provide only a single outcome, are not optimal for scenarios requiring the anticipation of multiple possible outcomes for accurate decision-making. To address this, we expand our framework to diverse human motion predictions, covering up to four potential future motions. Our comparative analysis of deterministic and diverse motion predictions, detailed in Table 6 , reveals that as the prediction time horizon increases, so does the uncertainty in human motion. This underscores the benefits of adopting diverse predictions, particularly for longterm forecasting. Visualizations of these diverse motion prediction results can be found in our supplementary materials. 
conclusion: 
 We introduce the first single-LiDAR-based method for practical 3D human motion prediction. Building upon our effective structureaware body feature descriptor, our approach adaptively maps the observed motion manifold to the future and models the spatialtemporal correlations of the human motion for further refinement. Additionally, we extend our method to support diverse predictions, accommodating multiple potential future motions for improved decision-making in real-world applications. Extensive experiments validate our method's superior performance and generalization capabilities in real-world human motion prediction scenarios. 
 
 Appendix:  


Decision: Reviewers' concerns are well addressed in the rebuttal. All reviewers agree to accept the paper. The AC recommends acceptance.

Reviews: review: paper_summary: paper_summary: Existing human motion prediction methods focus on forecasting future human movements based on past ground truth observations captured through multi-view cameras or dense IMUs, and use specialized network structures to address the spatial and temporal dependencies in structured skeleton sequences. This is often impractical for real scenarios where only raw visual sensor data is available. Some methods may experience performance degradation due to cumulative errors. Moreover, simplifying raw visual data into sparse key point representations significantly reduces information density, leading to the loss of fine-grained features. This paper proposes a single LiDAR-based 3D human motion prediction method that takes raw LiDAR point clouds as input and directly predicts future 3D human poses. The method consists of three parts: Structure-aware Body Feature Descriptor, Adaptive Motion Latent Mapping, and Spatial-Temporal Correlations Refinement. The first part addresses how to extract useful features related to human body structure from unordered LiDAR point clouds. The second part addresses how to predict future human motion based on historical observations. The third part addresses how to accurately simulate the spatial-temporal correlations between human joints. Finally, experiments were conducted on two public LiDAR datasets, proving the superior performance of LiDAR-HMP in both short-term and long-term human motion prediction.
strength_weakness: strength_weakness: 1. LiDAR-HMP directly utilizes raw LiDAR point cloud data for human motion prediction, eliminating the need for preliminary pose estimation steps, simplifying the prediction process, and reducing the accumulation of errors.
2. The Structure-aware Body Feature Descriptor can deeply comprehend the human body structure and extract expressive features, which helps in more accurately capturing human dynamics. Adaptive Motion Latent Mapping allows the method to dynamically extract information from historical data and adaptively predict future movements, enhancing the flexibility and accuracy of the predictions. The Spatial-Temporal Correlations Refinement module takes into account the spatial and temporal correlations between human joints, further improving the precision of the forecasts.
3. Experimental results show that LiDAR-HMP maintains effectiveness under varying lighting conditions and real-time requirements, demonstrating good robustness. Moreover, LiDAR-HMP has expanded support for diverse motion predictions, capable of providing multiple potential future motion forecasts, which is extremely useful for application scenarios that require consideration of various possible outcomes.
limitations: limitations: 1.	This article directly processes LiDAR point cloud data. When occluding the point cloud data, is the occlusion position random or targeted? Has the impact of occluding key body parts on the prediction results been considered?
2.	This paper directly processes LiDAR point cloud data, which can generate a substantial volume of data, requiring considerable computational power and storage space for processing and analysis.
3.	The paper mentions the deployment of the proposed scheme in real scenarios, but it does not provide a comparative analysis of actual performance metrics such as test time consumption and model file size, as seen in reference [28]. Complex models require significant computational resources, which may limit their application on devices with limited resources.
paper_summary: paper_summary: This paper proposed a single-LiDAR-based 3D motion prediction method, namely LiDAR-HMP, to directly predict the future 3D human pose from the raw LiDAR point cloud. To achieve this, a structure-aware body feature descriptor is proposed to encode the human body structure's global and local representation by leveraging a point encoder and human point cloud parser, respectively. After that, an adaptive motion latent mapping module is designed to regress coarse 3D human joints by querying the features from the past frame descriptors. Furthermore, a spatial-temporal transformer is proposed to refine the predictions using spatiotemporal contextual correlations.  Extensive experiments demonstrate the superiority of the proposed method in comparison with state-of-the-art on two benchmark datasets.
strength_weakness: strength_weakness: 1. The paper is well-written and easy to follow.
2. The proposed modules in the overall pipeline sound reasonable and the motivation is clear.
3. Comprehensive experiments have been performed and demonstrate the superiority of the proposed method.
limitations: limitations: I do not find obvious technical weaknesses in this paper, but only have a few questions:

1. As claimed in the paper, the proposed method is the first LiDAR-based method for 3D human motion prediction from raw LiDAR point clouds directly. Why are there few works to do so? What are the challenges behind this?

2. It seems the proposed method can predict the 24 joint points of the 3D human pose from the LiDAR point clouds. So how to get the ground truth 3D joints $J_t$ for calculating loss?

3. As mentioned in Sec. 4.2, the input point cloud $P_i$ of each frame is set to be a fixed number, i.e., $N=256$, by using the farthest point sample algorithm. How the number of points in each frame can impact the performance of the 3D pose estimation?

4. I am confused by the setting of the $K$ in the structure-aware body feature descriptor, which is the number of bins corresponding to different body part segments. So is it determined or predefined by the human parsing model? In addition, in Sec. 4.2, implementation details, $K$ is set to 9, but in Sec. 3.6, for diverse motion prediction extension, $K=4$. Why is that?

5. Furthermore, seems the $K$ also controls the size of the final point cloud predictions $\hat{P}_t^{\prime} \in \mathbb{R}^{K\times32\times3} $, does it mean for each bin, the prediction head will output 32 3D points? BTW, seems the notation $^{\prime}$ is a typo. 

6. Could authors elaborate more on the limitations of the proposed LiDAR-HMP and promising directions to enhance the performance in terms of, for example, scalability, flexibility, or applicability to other datasets or tasks?

7. Some typos, for example, in Sec. 4.6, line 910, “Table 6” should be “Tab. 6“.
paper_summary: paper_summary: This paper proposes a LiDAR-based model for 3D human motion prediction by fully utilizing the spatial-temporal information of point cloud frames. It achieves SoTA performances on two long-term motion prediction datasets.
strength_weakness: strength_weakness: 1. The paper proposes a novel LiDAR-based framework to predict human motion.
2. It exploits the potential of spatial-temporal information of point cloud frames to enhance the sequential prediction.
limitations: limitations: 1. Some parts of this paper are trivial: (1) the teaser image (Figure 1) provides abundant information to the audience and is easy to be confused. Additionally, your proposed work is a multi-stage framework as well (prediction + refinement), so it is better to clearly demonstrate the difference with the previous two-stage method at the left-upper corner sub-image; (2) the overall summary of "Section 4 EXPERIMENTS" (line 550-563) is useless.

2. The model designs for the spatial-temporal correlation (Adaptive Motion Latent Mapping and Spatial-temporal Correlations Refinement) are not clearly illustrated in Figure 2, although they are explained in detail in the manuscript.

3. The ablation study is conducted only on the LIPD dataset, not on the LiDARHuman26M dataset.
paper_summary: paper_summary: This paper presents a novel one stage point-cloud-based human motion prediction pipeline, which tries to address the problem of accumulation error observed in two stage pipelines. The experimental results seem to show the superiority of the proposed method.
strength_weakness: strength_weakness: + The motivation of this paper is clear. The two-stage methods may suffer the problem of accumulation error, so this paper proposes a one stage method.
+ The presentation of this paper is good. The writing and the presentation of figures are clear and food.
+ It seems that the proposed pipeline achieves great performance and can be embedded into a real time equipment.

The reviewer thinks it can be a good paper, and expects the authors to address the concerns. Although the current rating is not high, the reviewer has a strong willingness to give a WA rating after the authors address their concerns.
limitations: limitations: - The reviewer wants to know why previous methods follow the two-stage solution rather than the proposed one stage solution. It may be better to add the discussions about the challenges of one stage solution.
- What is the metric for evaluating the performance? The reviewer does not find the explanation for the metric used in the tables, and does not know how the data in the tables are computed. 
- As section 3.6 mentioned, the proposed pipeline can generate multiple predictions and uses WTA in the training stage. How to select the final prediction during the inference? How can the number of queries influences the final performance? The reviewer thinks these missing information is important.

To be honest, the presentation makes me feel that this is a good paper that should be published in the ACM MM conference. But some important information is missing, so the reviewer currently provides a BA rating. The reviewer really expects the missing information can be added, and will change the rating to WA.