Realistic Full-Body Motion Generation from Sparse Tracking with State Space Model

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-05 04:49:15

Abstract: In the domain of generative multimedia and interactive experiences, generating realistic and accurate full-body poses from sparse tracking is crucial for many real-world applications, while achieving sequence modeling and efficient motion generation remains challenging. Recently, state space models (SSMs) with efficient hardware-aware designs (i.e., Mamba) have shown great potential for sequence modeling, particularly in temporal contexts. However, processing motion data is still challenging for SSMs. Specifically, the sparsity of input conditions makes motion generation an ill-posed problem. Moreover, the complex structure of the human body further complicates this task. To address these issues, we present Motion Mamba Diffusion (MMD), a novel conditional diffusion model, which effectively utilizes the sequence modeling capability of SSMs and the robust generation ability of diffusion models to track full-body poses accurately. In particular, we design a bidirectional Temporal Mamba Module (TMM) to model motion sequence. Additionally, a Spatial Mamba Module (SMM) is further proposed for feature enhancement within a single frame. Extensive experiments on the large motion capture dataset (AMASS) demonstrate that our proposed approach outperforms the latest methods in terms of accuracy and smoothness and achieves new state-of-the-art performance. Moreover, our approach runs in real-time, making it ideal for employment in practical applications. The source code will be made public upon acceptance of this paper.

Main paper: 
 introduction: 
 With the recent popularity of generative multimedia and interactive experiences, tracking full-body human movement has become a highly demanded feature. However, there are no strong tracking signals for the entire human body in the typical AR/VR scenarios. Common VR systems are usually composed by Head Mounted Displays (HMD) and handheld controllers, in which Inertial Measurement Unit (IMU) sensors are utilized to track head and hands. While these devices can provide resourceful upper body motion information, the lower body is not tracked and must be estimated from the limited information provided by the upper body joints. The ill-posed intrinsic nature makes generating realistic full-body motion particularly challenging.Various methods have been proposed for this task, many of which are based on the Transformer architecture [46] . While Transformer models [6, 11, 21, 28, 38, 48, 57, 59 ] are renowned for their ability to capture long-range dependencies within a sequence, there remains a concern regarding their accurate understanding of temporal motion relationships among frames, primarily due to their It starts with Feature Mapping to map the noisy human poses and sparse tracking signals to the unified latent space for subsequent process. Then a U-shaped architecture is applied to process motion information across different timescales. In particular, we utilize the Motion Mamba Block (MMB) to construct it, which consists of two key components tailored for motion generation, i.e. Temporal Mamba Module (TMM) and Spatial Mamba Module (SMM). Specifically, TMM focuses on capturing the temporal dependencies inherent in motion sequences, while SMM is employed for feature enhancement within each frame.dependency on positional encoding. A recent work, AGRoL [12] , employs a MLP-based network for this task. Nevertheless, its reliance on 1D convolution to capture temporal relationships within motion sequences may limit its modeling capacity, thereby leading to insufficient performance. Recent advancements in State Space Models (SSMs) [17, 18, 32, 40, 47] , notably Mamba [14] , have demonstrated significant promise in sequence modeling with selective scanning mechanism, indicating a promising direction for building motion generation models based on this approach. However, it is not trivial to employ SSMs in this task. Generating human movements with high accuracy is of vital importance, yet the sparse nature of input conditions means this task is actually an ill-posed problem. Furthermore, the intricate structure of the human body also makes it very difficult to generate smooth and realistic movements.In light of these issues, we propose the Motion Mamba Diffusion (MMD), a novel conditional diffusion model, which generates human poses through iterative denoising, as illustrated in Figure 1 . It effectively leverages the robust generation capabilities of diffusion models in conjunction with the sequence modeling proficiency of SSMs. As shown in Figure 2 , the proposed MMD adopts a Ushaped architecture to process motion information across different timescales, and the Motion Mamba Block (MMB) is designed to construct it, comprising two key components tailored for motion generation. First, a bidirectional Temporal Mamba Module (TMM) is employed for motion sequence modeling of sparse inputs. This module excels at identifying temporal dependencies among frames, facilitating a thorough comprehension of the dynamics inherited in motion sequences. Second, we apply a Spatial Mamba Module (SMM) to enhance the motion features within a single frame. This module is dedicated to improving the understanding of the human body structure within each frame.The proposed MMD remarkably advances the performance of motion generation, notably improving the key metrics like MPJRE, MPJPE, and MPJVE by 13.53%, 14.55%, and 6.83%, respectively, compared to the previous state-of-the-art method AGRoL [12] on AMASS [31] . Overall, the contributions of this paper are summarized as follows:• A SSM-based framework, termed Motion Mamba Diffusion (MMD), is proposed for motion generation. It is a pioneering method for full-body pose estimation from sparse tracking signals with the incorporation of the selective scanning mechanism and denoising diffusion model. • Two innovative components tailored for motion generation, Temporal Mamba Module (TMM) and Spatial Mamba Module (SMM), are introduced to construct the proposed MMD. Specifically, TMM is employed for motion sequence modeling of sparse inputs, while SMM is utilized to enhance the motion features within a single frame. • Extensive experiments have been conducted to evaluate the proposed method, and the experimental results on the widely-used AMASS benchmark illustrate that our approach outperforms the state-of-the-art in terms of accuracy and smoothness. Further analysis reveals the contribution of each component to the performance improvement. Additionally, MMD runs in real-time, providing a crucial advancement for creating realistic virtual avatars in real-world applications. 
related work 2.1 motion generation from sparse tracking: 
 Full-body motion generation from sparse tracking signals of body joints has attracted significant interest in the research community. Many prior works, such as [23, 52] , have utilized up to 6 body-worn inertial sensors, typically distributed over the head, arms, pelvis, and legs. However, in many practical Head-Mounted Display (HMD) settings, only 3 tracking signals are available: the head and 2 wrists. CoolMoves [2] was the first to estimate full-body motion using only 3 tracking signals from headphones and handheld controllers.Nevertheless, the adopted KNN-based method interpolates poses from a smaller dataset with specific motion, raising concerns about its scalability to larger datasets with diverse subjects and activities.Recently, various learning-based methods are proposed for this task. For instance, AvatarPoser [24] employed a Transformer encoder to extract deep features from the input signals and decouples global motion to guide pose estimation. AGRoL [12] introduced an MLP-based architecture for motion reconstruction. Other methods approached sparse input body tracking as a synthesis problem. To that extent, VAE-HMD [10] proposed a Variational AutoEncoder (VAE) architecture, while FLAG [3] opted for a flow-based method.Emerging from the realm of image generation, denoising diffusion models [7, 13, 22, 27, 33, 43, 56] have garnered considerable attention due to their capability to yield high-quality results and adeptness at handling large datasets. These models acquire a probabilistic understanding over a denoising process applied to inputs, gradually refining Gaussian noise to the desired output. Furthermore, diffusion models excel in supporting conditional generation. For instance, ILVR [5] guided the generative process to generate superiorquality images based on a provided reference image. In a similar vein, GLIDE [35] explored diffusion models for text-conditional image synthesis. Furthermore, UniDiffuser [4] empowers a single model to accomplish multiple generation tasks with high quality.Recent advancements have extended diffusion models to motion synthesis. However, these endeavors have primarily focused on the text-to-motion [30, 39, 53] or audio-to-motion [8, 25, 51] tasks, with scant attention given to tracking the full body from sparse inputs. AGRoL [12] introduced the first diffusion model solely for motion reconstruction from sparse inputs. Nonetheless, there remains a concern regarding its precise comprehension of temporal sequence modeling, primarily stemming from the adopted MLP-based architecture. Our approach, Motion Mamba Diffusion, innovatively addresses this limitation, thereby leading to more accurate and smoother motion generation.Recently, State Space Models (SSMs) are introduced into deep learning [44] as state space transforming. Inspired by continuous state space models in control systems and benefiting from HiPPO initialization [15] , LSSL [18] demonstrates the potential to tackle longrange dependency problems. However, due to the prohibitive computation and memory demands, LSSL is practically unfeasible. To resolve this issue, S4 [17] proposes parameter normalization into a diagonal structure. Subsequently, various variants have emerged, encompassing complex-diagonal structures [16, 19] , diagonal decomposition coupled with low-rank operations [20] , and selection mechanisms [14] . These models are further integrated into larger representation models. Amid these advancements, Mamba [14] introduces the Selective Scan Space State Sequential Model (S6) Block, representing a promising innovation for addressing long sequence modeling across diverse tasks. Recent work has introduced Mamba into the field of computer vision [37, 42, 50, 54, 58] , while [55] presents a text-to-motion framework. 
preliminaries: 
 State Space Models (SSMs), especially with the advancements made by structured state space sequence models (S4) [17] and Mamba [14] , have showcased remarkable proficiency in sequence modeling. These models rely on a classical continuous system that maps a one-dimensional input function or sequence, denoted as 𝑥 (𝑡) ∈ R, through intermediate implicit states ℎ(𝑡) ∈ R 𝑁 to an output 𝑦 (𝑡) ∈ R. The aforementioned process can be formulated as a linear Ordinary Differential Equation (ODE):EQUATIONwhere A ∈ R 𝑁 ×𝑁 denotes the evolution matrix, while B ∈ R 𝑁 ×1 and C ∈ R 𝑁 ×1 represents the projection parameters. S4 and Mamba discretize this continuous system, making it more suitable for deep learning scenarios. Specifically, they introduce a timescale parameter ∆ and apply fixed discretization rules to transform A and B into discrete parameters Ā and B. Typically, zero-order hold (ZOH) is employed as the discretization rule, which can be defined as followsEQUATIONAfter discretization, SSM-based models can be computed through linear recurrence, described asEQUATIONAnother computational approach is global convolution, which can be formulated as followsEQUATIONwhere K ∈ R 𝐿 represents a structured convolutional kernel, and 𝐿 denotes the length of the input sequence 𝑥. 
the proposed method: 
 In this section, we initially introduce the formation of motion generation from sparse tracking, i.e., the positional and rotational data from a headset and two handheld controllers. Subsequently, the detailed architecture of the proposed MMD is described.Consider a sequence of human motion from sparse tracking c ∈ R 𝐿×𝐶 , where 𝐿 denotes the length of the sequence and 𝐶 represents the dimensions of the observed joint features. Our objective is to estimate the corresponding full-body poses x ∈ R 𝐿×𝐽 based on c, where 𝐽 denotes the dimensions of the full-body joint features. To represent human poses, we utilize the SMPL model parameters [29] and follow the practice of disregarding joints on the hands and face [10, 12, 24] .In our approach, we view the full-body pose estimation task as a conditional generation problem, where the sparse tracking serves as the conditioner. Specifically, a simple yet effective diffusion model detailed in Section 4.2 is proposed for motion generation. In the forward diffusion process, we gradually add Gaussian noise into the clean data until the output distribution is close to an isotropic Gaussian distribution. Taking a data pair (x 0 , c) in the dataset as a instance, the diffusion process from clean data x 0 to Gaussian noise x 𝑇 is defined asEQUATIONwhere x 𝑡 ∈ R 𝐿×𝐽 represents the motion at the 𝑡-th step and there are 𝑇 denoising steps in total. x 𝑇 tends to an isotropic Gaussian distribution when 𝑇 → ∞. The hyperparameters 𝛼 𝑡 ∈ (0, 1) are predefined positive constants and should gradually decrease to 0 at later steps.In the reverse diffusion process, the sampled Gaussian noise will be iteratively denoised to reconstruct a clean human motion sequence conditioned on the sparse tracking inputs c. To obtain the clean data x 0 , we learn a generative process 𝑝 𝜃 to predict and eliminate the noise added in the forward diffusion process. Specifically, the reverse process from Gaussian noise x 𝑇 to clean data x 0 is defined asEQUATIONwhere 𝜇 𝜃 (x 𝑡 , c, 𝑡) is a neural network parameterized by 𝜃 , which could be reformulated asEQUATIONand the variance 𝜎 2 𝑡 is a timestep dependent constant. The parametrization is 𝜎 2 𝑡 = 1-ᾱ𝑡 -1 1-ᾱ𝑡 (1 -𝛼 𝑡 ), where ᾱ𝑡 = 𝑡 𝑖=1 𝛼 𝑖 . Therefore, the model has to learn to predict the residual noise 𝜖 𝜃 (x 𝑡 , c, 𝑡). In our case, we follow the practice of [41] , directly predicting the clean body poses x 0 instead of the residual noise 𝜖 𝜃 . The objective function in training can be formulated asEQUATIONwhere 𝑝 motion is the distribution of the full-body poses in the dataset and x 0 denotes the output of the proposed diffusion model.As shown in Figure 2 , our Motion Mamba Diffusion (MMD) adopts a U-shaped architecture to process motion information across different time scales. This architecture is constructed using a meticulously designed Motion Mamba Block (MMB), which consists of two key components: Temporal Mamba Module (TMM) and Spatial Mamba Module (SMM). In particular, TMM specializes in modeling temporal dependencies among frames, enabling a thorough comprehension of the dynamics present in motion sequences. Meanwhile, SMM is employed to enhance the motion features within a single frame, which is dedicated to improving the understanding of the human body structure within each frame. The proposed components in our framework are introduced in detail as follows.Feature Mapping: At time step 𝑡, fully connected layers are first applied to map the noisy human poses x 𝑡 ∈ R 𝐿×𝐽 and the observed joint features c ∈ R 𝐿×𝐶 to a unified latent space. Then we can obtain the corresponding latent features f 𝑥 ∈ R 𝐿×𝐷 and sparse features f 𝑐 ∈ R 𝐿×𝐷 , where 𝐷 represents the dimension of the latent space. Subsequently, f 𝑥 and f 𝑐 are concatenated and input into a MLP layer for feature enhancement [12] . We denote the output motion features as f ∈ R 𝐿×𝐷 . Notably, the diffusion time step 𝑡 is also transformed into an embedding vector f 𝑡 ∈ R 1×𝐷 , which will be injected repetitively into the network.Temporal Mamba Module: To capture the temporal dependencies present in motion sequences, we propose the Temporal Mamba Module (TMM). It is worth noting that we employ a bidirectional strategy to enhance its temporal modeling capabilities, as shown in Figure 3 . F ′ : (B, L, D) ← Flip(F) if 𝑘 is backward else F ′ ← F 6:/* add the timestep embedding T */ 7:F ′ : (B, L, D) ← F ′ + T * 8:/* normalize the input features F ′ */ 9:F * : (B, L, D) ← Norm(F ′ )10:x : (B, L, E) ← Linear x (F * )11:z : (B, L, E) ← Linear z (F * )12:x ′ 𝑜 : (B, L, E) ← SiLU(Conv1d 𝑜 (x) )13:B 𝑜 : (B, L, N) ← Linear B 𝑜 (x ′ 𝑜 )14: In particular, we present the operations of TMM in Algorithm. 1. To operate over an input feature sequence F ∈ R 𝐵×𝐿×𝐷 of batch size 𝐵, a timestep embedding T ∈ R 𝐵×1×𝐷 will be first broadcast and added to guide the denoising process (For backward, we will first flip F along the dimension of the sequence length). Then a normalization layer is applied to normalize F. Next, we linearly project the normalized sequence to x and z with dimension size 𝐸. Subsequently, the 1-D convolution is applied to the x to obtain the x ′ 𝑜 . Then we linearly project the x ′ 𝑜 to the B 𝑜 , C 𝑜 , ∆ 𝑜 , respectively. ∆ 𝑜 is then utilized to transform Ā𝑜 , B𝑜 , respectively. In the end, we compute y forward and y backward with the SSM (For backward, we will flip the output along the dimension of the sequence length). The y forward and y backward will be added together to form the final output sequence O ∈ R 𝐵×𝐿×𝐷 . Note that 𝐸 represents the expanded state dimension, while 𝑁 denotes the SSM dimension. Spatial Mamba Module: This module is introduced to learn the spatial dependencies of the motion poses within a single frame. Similar to TMM, the bidirectional strategy is adopted to improve the comprehension of the human body structures. It is worth noting that SMM is structurally consistent with TMM. The difference is that it process features along the dimension of the latent space rather than the sequence length.C 𝑜 : (B, L, N) ← Linear C 𝑜 (x ′We present the process of TMM in Algorithm. 2. Specifically, given the output O of TMM, we first transpose it along the last two dimensions to get the O 𝑡 ∈ R 𝐵×𝐷 ×𝐿 . Correspondingly, we linearly project the timestep embedding T to T ′ with dimension size 𝐿 and then broadcast it. The subsequent calculation process follows a similar procedure to TMM. Note that we utilize the transpose x ′ 𝑜 : (B, D, E) ← SiLU(Conv1d 𝑜 (x) )16:B 𝑜 : (B, D, N) ← Linear B 𝑜 (x ′ 𝑜 )17: Training: At timestep 𝑡, we denote the output motion feature of our MMD as F 𝑡 . It will be passed through a fully connected layer to obtain the full-body motion x 𝑡 , which serves as the input for the next time step. This process will continue until we obtain the clean human poses x 0 . In training, the loss function adopted by our MMD is calculated in the same way as Equation 8.EQUATION 
experiments 5.1 datasets: 
 To verify the effectiveness of our method, we conduct extensive experiments on three subsets of AMASS [31] : CMU [1] , BMLr [45] , and HDM05 [34] for training and testing. The AMASS dataset is a large human motion database, which unifies different existing optical marker-based MoCap datasets and represents human poses with SMPL model parameters [29] . Following [24] , we split the three datasets into training and test sets with 90% and 10% of the data, respectively.Following previous works, the adopted metrics for evaluation can be divided into three categories: 1) Rotation-Oriented Metrics: Mean Per Joint Rotation Error (MPJRE) in degrees measures the Due to its simplicity and continuity, we represent the joint rotations by the 6D reparametrization. Therefore, for the observed joint features 𝒄 ∈ R 𝑁 ×𝐶 , which consists of the orientation, translation, orientation velocity and translation velocity of the head and hands in global coordinate system, 𝐶 = 18 × 3. For the sequences of fullbody poses 𝒙 ∈ R 𝑁 ×𝐷 , 𝐷 = 22 × 6. Unless otherwise stated, we set the length of motion sequence 𝑁 to 196 and the sampling steps to 8 for inference. All experiments were conducted using the PyTorch platform [36] and trained/tested on a single NVIDIA A100 GPU.In addition, our network used Adam optimizer [26] with a base learning rate of 3e-4.We compare Motion Mamba Diffusion (MMD) with the state-of-theart method (AGRoL) and other popular body tracking models. The quantitative experimental results are tabulated in Table 1 , where the best results are boldfaced. It can be seen that our proposed method showcases significant advancements, achieving superior performance across all metrics and outperforming the current state-ofthe-art technique. In particular, three key metrics from the AMASS dataset demonstrate MMD's superiority: a reduction in MPJRE by 13.53%, MPJPE by 14.55%, and MPJVE by 6.83%. Additionally, we measure the performance corresponding to different parts of the body. Specifically, MMD reduce the position error of the hands, upper body, lower body, and root by 39.69%, 19.35%, 13.16%, and 14.88%, respectively. Moreover, it is worth noting that our method exhibits a significant 22.70% improvement in Jitter, indicating its capability to generate smoother human motion, a crucial aspect for enhancing user experience and realism. The improvement in upper body and lower body is 44.39% and 8.42%, respectively.To further validate the performance of our proposed method, we conduct a 3-fold cross-dataset evaluation among different approaches. Specifically, we train our model on two subsets and then test on the remaining subset in a round-robin fashion. Table 2 presents the experimental results of various methods evaluated on the CMU, BMLrub, and HDM05 datasets. Our method achieves the best results across all evaluation metrics in all three datasets. Particularly noteworthy is our MMD, which significantly outperforms previous state-of-the-art methods in terms of MPJVE across all subsets, indicating smoother motions. Additionally, both the rotation error (MPJRE) and position error (MPJPE), representing prediction accuracy, are notably reduced.For visualization, we conducted qualitative experiments and the results are shown in Figure 4 , in which the extent of the error is indicated by the intensity in red. The results show that MMD generates more accurate human motion compared with AGRoL. For complex movements, our method can also produce plausible full-body predictions from merely three sparse tracking inputs. Additionally, two different motion sequences are presented for further comparison. As shown in Figure 5 , the results generated from our method exhibit higher smoothness and better consistency with the ground truth.In addition to delivering high-quality predictions, our MMD exhibits exceptional efficiency. It achieves an average processing time of only 0.57ms per pose when running on an A100 GPU, resulting in an impressive frame rate of 1754 frames per second (fps). This remarkable performance makes our method ideal for real-time avatar animation driven by head-mounted displays (HMDs) in practical applications.In this section, we present extensive ablation studies on our Motion Mamba Diffusion (MMD) architecture, aiming to assess the impact and effectiveness of each proposed component on the overall model performance. The specific details are as follows.The effectiveness of proposed modules. A series of ablation experiments were conducted to verify the effectiveness of our proposed components. As can be seen in Table 3 , the individual application of the Temporal Mamba Module (TMM) and Spatial Mamba Module (SMM) brings significant performance improvements, while the combination of the two modules can lead to better results. Notably, the first line represents that only the forward SSM is applied in the TMM, excluding the SMM, while the second line denotes that we utilize the bidirectional TMM. The number of blocks in the framework. We investigate the influence of varying the number of Motion Mamba Blocks (MMBs) in our MMD network. The number of MMBs in Figure 2 is denoted as [𝐸 1 , 𝐸 2 , 𝑀, 𝐷 1 , 𝐷 2 ] from left to right, respectively. As shown in Table 4 , the model's performance consistently improves as more blocks are added. Nevertheless, the performance gains start to plateau when too many blocks are used. Input Sequence Length. Our method takes a sequence of sparse tracking signals as input and predicts the full-body poses of the same length. In this ablation study, we evaluate the performance of five different input sizes: 64, 96, 144, 196, and 256. The results are summarized in Table 5 . As shown in Table 5 , the model achieves the best overall performance across three key metrics when the sequence length is set to 96. Therefore, we opt for this length for subsequent experiments. The number of sampling steps for inference. In Table 6 , we ablate the number of sampling steps during inference. It is worth noting that even when evaluating with merely 2 sampling steps, our model can achieve plausible performance. Meanwhile, we notice that the model tends to output worse results when too many sampling steps are used. To achieve the best overall performance, 5 is selected as our final sampling steps for inference.  
conclusion: 
 In this paper, we propose Motion Mamba Diffusion (MMD), a novel SSM-based framework for accurately estimating full-body motion from sparse tracking signals. It is a pioneering method, which effectively leverages the robust generation capabilities of diffusion models in conjunction with the sequence modeling proficiency of SSMs to generate realistic human poses. Specifically, we adopt a U-shaped architecture and introduce two innovative modules, Temporal Mamba Module (TMM) and Spatial Mamba Module (SMM), to model the temporal motion sequence and human body structure within each frame, respectively. Extensive experiments demonstrate that our method outperforms the latest approaches and achieves new state-of-the-art performance, showcasing its significant potential in various generative multimedia applications. 
 
 Appendix:  


Decision: This paper propose a novel state space approach to estimate human body pose from sparse tracking signals exploiting diffusion models and MAMBA. The experimental results are extensive. An interesting extension of this work should regard the evaluation of actual generative capability of the model. The rebuttal satisfied the reviewers which in general raised their scores.

Reviews: review: paper_summary: paper_summary: This paper introduces a SSM-based framework that enhanced the accuracy of full body motion generation from sparse tracking. The framework introduces two mamba module for modeling temporal relations and spatial relations.
strength_weakness: strength_weakness: 1: The introduction of new structure Mamba increases the accuracy of full body motion generation from sparse tracking.

2: The paper is well-structured and easy to read.
limitations: limitations: 1:Missing citation on motion generation from sparse tracking, especially works including spatial and temporal modeling, considering the contribution of this work. A lot of related works in this area should be included and analysized in related work:

Xiaozheng Zheng, Zhuo Su, Chao Wen, Zhou Xue, Xiaojie Jin. 2023. Realistic Full-Body Tracking from Sparse Observations via Joint-Level Modeling. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 14678-14688

Feiyu Yao, Zongkai Wu, and Li Yi. 2024. Full-Body Motion Reconstruction with Sparse Sensing from Graph Perspective. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 6612-6620.

Jose Luis Ponton, Haoran Yun, Andreas Aristidou, Carlos Andujar, and Nuria Pelechano. 2023. SparsePoser: Real-time Full-body Motion Reconstruction from Sparse Data. ACM Transactions on Graphics (TOG) 43, 1 (2023), 1–14.

Jiaman Li, C. Karen Liu, Jiajun Wu. 2023. Ego-Body Pose Estimation via Ego-Head Pose Estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 17142–17151.

2: More analysis is needed to prove the mamba modolue effectiveness. 
(1) The speed is slower. AGRoL produces 196 output frames in 35 ms on V100 (16/32GB), which is 0.17ms per pose, while this paper achieves 0.57ms per pose on A100 (40/80GB). Is this worth to trade computing time for precision improvement? A Comparison of computational efficiency is persuasive.
(2) Performance comparison on three subsets of AMASS is not enough. Related works with diffusion or VAE structrure (such as FLAG, AGRoL and HuMoR) use nearly all subsets of AMASS and even other dataset (AIST++) to prove the effectiveness of the method. Adding more experiments is persuasive.

I will change my rating to Accept if limitations 1 and limitations 2(2) are solved.
paper_summary: paper_summary: This paper studies full-body motion generation from sparse tracking. A state space model-based framework, named Motion Mamba Diffusion, is proposed to generate human poses by iterative denoising. This framework comprises two components: a Temporal Mamba Module to capture temporal dependencies inherent in motion sequences and a Spatial Mamba Module to enhance features within each frame. Experiments on the AMASS dataset show that the proposed method achieves good performance and runs in real time.
strength_weakness: strength_weakness: - This paper is well-written and easy to follow.

- The proposed model performs better than the previous methods on the AMASS dataset.
limitations: limitations: - Failure cases and limitation of the proposed method should be analyzed.

- This paper proposes bidirectional Temporal/Spatial Mamba Modules for motion generation. The related paper [1] also proposes a Hierarchical Temporal Mamba Block and a Bidirectional Spatial Mamba Block for motion generation. While [1] generates motion from text, this paper generates motion from sparse tracking. However, more discussions and comparisons of the two methods should be provided.


[1] Motion Mamba: Efficient and Long Sequence Motion Generation with Hierarchical and Bidirectional Selective SSM, arXiv preprint arXiv:2403.07487 (2024).
paper_summary: paper_summary: This paper proposes a motion Mamba diffusion model to estimate the corresponding full-body posture based on sparse tracking. This model utilizes diffusion models for motion generation and state space models for sequence modeling. This model generates motion through a motion Mamba block, which includes a bidirectional time Mamba module (TMM) to recognize temporal dependencies between frames and a spatial Mamba module (SMM) to enhance single-frame motion features.
strength_weakness: strength_weakness: 1. This paper addresses the problem of sequence modeling and motion generation.
2. This paper can generate realistic full-body movements.
limitations: limitations: 1. What is the difference between the proposed Mamba diffusion model and the general diffusion model? Is it simply introducing Mamba into the diffusion model?
2. What are the specific details of the proposed motion Mamba model?
3. The expression of SMM in line 516 of 4.2 is incorrect.
4. In the experimental section, which method did the data in lines 671 and 674 decrease compared to? It is not evident from the table data that the data described in the paper has been reduced. Where is the ambiguity of the data in lines 677 and 680 reflected?