StableMoFusion: Towards Robust and Efficient Diffusion-based Motion Generation Framework

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-09 08:27:12

Abstract: Thanks to the powerful generative capacity of diffusion models, recent years have witnessed rapid progress in human motion generation. 
Existing diffusion-based methods employ disparate network architectures and training strategies.
The effect of the design of each component is still unclear.
In addition, the iterative denoising process consumes considerable computational overhead, which is prohibitive for real-time scenarios such as virtual characters and humanoid robots.
For this reason, we first conduct a comprehensive investigation into network architectures, training strategies, and inference processs.
Based on the profound analysis, we tailor each component for efficient high-quality human motion generation.
Despite the promising performance, the tailored model still suffers from foot skating which is an ubiquitous issue in diffusion-based solutions.
To eliminate footskate, we identify foot-ground contact and correct foot motions along the denoising process.
By organically combining these well-designed components together, we present StableMoFusion, a robust and efficient framework for human motion generation. 
Extensive experimental results show that our StableMoFusion performs favorably against current state-of-the-art methods.

Main paper: 
 related work 2.1 motion diffusion generation: 
 In recent years, the application of diffusion models to human motion generation has led to significant improvements in the quality of generated motions. MotionDiffuse [36] softly fuses text features into diffusion-based motion generation through cross-attention. MDM [27] experimented with the separate Transformer encoder, decoder, GRU as denoising networks, respectively. PyhsDiff [34] incorporates physical constraints to generate more realistic motions; Prior MDM [23] uses diffusion priors to allow the model to be applied to specific generative tasks; MLD [3] utilizes the latent space of VAE to speed up diffusion generation; ReMoDiffuse [37] uses a retrieval mechanism to enhance the motion diffusion model. All of these methods use Transformer-based network structure, while MoFusion [4] and GMD [13] use Conv1D UNet for motion diffusion generation.Our work towards a more robust and efficient diffusion-based motion generation framework through a comprehensive investigation into network architectures, training strategies, and inference process. It also addresses the practical application challenges of long inference time and footskate phenomenon.To reduce the inference time with a trained network, there have been many advanced samplers to accelerate DDPM [8] .Song et al. [25] show that using Stochastic Differential Equation (SDE) for sampling has a marginally equivalent probability Ordinary Differential Equations (ODE). And then, DDIM [24] constructs a class of non-Markovian diffusion processes that realize skip-step sampling. PNDN [15] uses pseudo numerical to accelerate the deterministic sampling process. DEIS [38] and DPMSolver [16] improve upon DDIM by numerically approximating the score functions within each discretized time interval.Meanwhile, several work have focused on speeding up stochastic sampling. For example, Gotta Go Fast [11] utilizes adaptive step sizes to speed up SDE sampling, and Lu et al. [17] converts the higher-order ODE solver into an SDE sampler to address the instability issue.While these samplers have demonstrated efficacy in image generation, their impact on motion diffusion models remains unexplored. In this work, we evaluate them to find the most appropriate one for motion generation.In order to generate realistic motions in computer animation, various methods have been developed to improve footskate issue.Edge [28] embeds the foot contact term into the action representation for training and applies Contact Consistency Loss as a constraint to keep the physical plausibility of motion. RFC [33] , Drop [10] and Physdiff [34] uses reinforcement learning to constrain the physical states of actions, such as ground force reaction and collision situations to get a realism motion. UnderPressure [18] and GroundLink [7] respectively collect foot force datasets during motion. UnderPressure [18] also utilizes this dataset to train a network capable of predicting vertical ground reaction forces. Based on this, UnderPressure proposes a foot skating cleanup method. 
preliminaries: 
 The pipeline of Diffusion model [8] involve three interconnected processes: a forward process that gradually diffuses noise into sample, a reverse process that optimizes a network to eliminate the above perturbation, and an inference process that utilizes the trained network to iteratively denoise noisy sample.Specifically, a motion denoising network is first trained to predict the original motion 𝑥 0 from the noisy motion 𝑥 𝑡 : randomly select a ground-truth motion 𝑥 0 and a diffusion timestep 𝑡 ∼ 𝑈 [0,𝑇 ], 𝑇 being the maximum timestep. And then the noisy motion 𝑥 𝑡 after t-step diffusion is gained by Equation 1, where 𝜖 is a Gaussian noise.EQUATION√ ᾱ𝑡 and √ 1 -ᾱ𝑡 are the strengths of signal and noise, respectively. When √ ᾱ𝑡 is small enough, we can approximate 𝑥 𝑡 ∼ N (0, 𝐼 ).Next, given a motion-denoising model G 𝜃 (𝑥 𝑡 , 𝑡, 𝑐) for predicting the original sample, parameterized by 𝜃 , the optimization can be formulated as follows:min 𝜃 𝐸 𝑡 ∼𝑈 [0,𝑇 ],𝑥 0 ∼𝑝 𝑑𝑎𝑡𝑎 ||G 𝜃 (𝑥 𝑡 , 𝑡, 𝑐) -𝑥 0 || 2 2 (2)In the inference process, a trained motion-denoising network can progressively generate samples from noise with various samplers. For instance, DDPM [8] iteratively denoise the noisy data from 𝑡 to a previous timestep 𝑡 ′ , as shown in Algorithm 1.Given a text prompt 𝑐x 𝑡 ∼ N (0, I) for 𝑡 = 𝑇 to 1 do x 0 = G(x 𝑡 , 𝑡, 𝑐) 𝜖 ∼ N (0, 𝐼 ) if 𝑡 > 1, else 𝜖 = 0 x 𝑡 -1 = √ ᾱ𝑡 -1 𝛽 𝑡 1-ᾱ𝑡 x 0 + √ 𝛼 𝑡 (1-ᾱ𝑡 -1 ) 1-ᾱ𝑡x 𝑡 + β𝑡 𝜖 end for return x 0 
method: 
 Through comprehensive exploratory experiments conducted on diffusion-based motion generation, we propose a novel diffusion framework, named StableMoFusion, as illustrated in Figure 2 , to facilitate robust and efficient motion generation. This section begins with our investigation on the architecture of motion-denoising networks. Next, we discuss several training strategies pivotal for enhancing model performance in Section 4.2. Subsequently, we introduce our improvements in the inference process in Section 4.3, tailored to enable efficient inference. Lastly, we discuss and present a solution to the footskate issue in Section 4.4.Most existing work use Transformer [29] -based architectures as the motion-denoising network; however, it remains questionable whether these architectures are best for diffusion-based motion generation. In this subsection, we will present three new network architectures fine-tuned for the motion generation task: Conv1D UNet [4, 13] , Diffusion Transformer (DiT) [19] and the latest Retentive Network (RetNet) [26] .Baseline. We chose the Conv1D UNet with AdaGN [5] and skip connections in GMD [13] as the Conv1D UNet baseline and modify the structure to a canonical Unet structure, which consist of four downsampling stages. The motion length 𝑛 is successively reduced from 𝑁 to ⌊𝑁 /8⌋, and then the corresponding up-sampling phase is used to up-sample. There are two residual Conv1D blocks for each down-sampling or up-sampling stage, with a single block shown as Figure 3 (a) . Block Adjustment. We introduce Residual Linear Multi-Head Cross-Attention after each block to effectively integrate textual cues, and dropout is incorporated into the original Conv1D block to enhance model generalization, as shown in Figure 3 (b). In the baseline block, text prompts are encoded with timesteps and integrated into motion coding using a simple formula: 𝑥 • (1 + 𝑠𝑐𝑎𝑙𝑒) + 𝑠ℎ𝑖 𝑓 𝑡. However, this approach doesn't effectively incorporate textual cues into motion sequences because it applies uniform operations across the entire sequence. In diffusion pipelines, noise uniformly affects the entire sequence, resulting in consistent mappings between motion frames and timesteps. However, since each frame's motion corresponds to distinct textual cues, a straightforward "scale and shift" approach is insufficient for injecting textual information. Our solution employs an attention mechanism to dynamically focus each motion frame on its associated textual information. Residual connections help mitigate potential computation biases introduced by cross attention.GroupNorm Tweak. We rearranged the data before and after applying Group Normalization, as depicted in Figure 3 (b), to minimize the impact of padded data during network forward propagation. When testing the adapted Conv1D UNet on datasets like KIT-ML with varying sequence lengths, we noticed a significant performance drop. This suggests that the model struggles with datasets containing extensive padding. Further investigation revealed that implementing Group Normalization within the baseline block caused this issue. Since Conv1D operates along the temporal dimension, directly applying Group Normalization to the input disrupted the differentiation between padded and non-padded data, affecting loss computation and gradient descent.Baseline. To explore the effectiveness of the DiT structure for motion generation, we replace the Vision Transformer used for images in the DiT with self-attention used for motion data as the baseline, with the basic block structure shown in Figure 3 (c) . For text-to-motion generation, we embed text prompts via the CLIP [21] encoder and project them into token concatenated with motion embeddings for self-attention. It scales and shifts the motion embedding before and after each autoregressive computation using timestep, which ensures the motion denoising trajectory closely aligned with the timestep.Block Adjustment. We have also tried to incorporate Linear Multi-Head Cross-Attention into the DiT framework, as shown in Figure 3 (d). This adjustment allows for a more nuanced fusion of textual cues with motion dynamics than fusing all the text information into the one-dimensional text embedding in baseline, which enhances the coherence and relevance of generated motion sequences.Baseline. Our RetNet baseline follows a straightforward implementation similar to MDM, where the timesteps encoding is concatenated with the textual projection to form tokens, which are then fed along with motion embeddings into RetNet, with its basic block shown in Figure 3 (e) . RetNet incorporates a gated multi-scale retention mechanism, which enhances information retention and processing capabilities, thereby enabling nuanced comprehension and generation of motion sequences. Through our investigation, we aim to ascertain the feasibility of leveraging RetNet for motion generation tasks.Block Adjustment. To further integrate textual information, we also employ Linear Multi-Head Cross-Attention between retention and FFN, as shown in Figure 3 (f) . By segregating temporal and textual features, our approach aims to preserve the distinct characteristics of each modality and allow the model to independently learn and leverage relevant cues for motion generation. This separation enhances the model's interpretability and flexibility, enabling it to better capture the intricacies of both temporal dynamics and semantic context. Ultimately, we choose the Conv1D UNet with block adjustment and GroupNorm tweak as the motion-denoising model of StableMo-Fusion, as shown in Figure 2 . We call this network as CondUNet1D. Both DiT and RetNet use the idea of attention to activate the global receptive field in the temporal dimension, which benefits the modeling of long-range dependency. The receptive field of Conv1D UNet is mainly in the convolution kernel window, promoting a coherent and smooth transition between frames. We tend to prioritize smoother generation in current applications of motion generation.In our StableMoFusion, we set the base channel and channel multipliers of UNet to 512 and [2,2,2,2] respectively. For text encoder, we leverage pre-trained CLIP [21] token embeddings, augmenting them with four additional transformer encoder layers, the same as MotionDiffuse [36] , with a latent text dimension of 256. For timesteps encoder, it is implemented using position encoding and two linear layers, the same as StableDiffusion [22] , with a latent time dimension of 512.Recent research has shown that key factors in the training strategies of the diffusion model affect the learning pattern and its generative performance [2] . In this subsection, we will analyze the impact of two empirically valid training strategies on diffusion-based motion generation: exponential moving average and classifier-free guidance.Exponential Moving Average (EMA) calculates a weighted average of a series of model weights, giving more weight to recent data. Specifically, assume the weight of the model at time t as 𝜃 𝑡 , then the EMA formulated as: 𝑣 𝑡 = 𝛽 • 𝑣 𝑡 -1 + (1 -𝛽) • 𝜃 𝑡 , where 𝑣 𝑡 denotes the average of the network parameters for the first t iterations (𝑣 0 = 0), and 𝛽 is the weighted weight value.During the training of the motion-denoising network, the network parameters change with each iteration, and the motion modeling oscillates between text-motion consistency and motion quality. Therefore, the use of EMA can smooth out the change process of these parameters, reduce mutations and oscillations, and help to improve the stability ability of the motion-denoising model.To further improve the generation quality, we use Classifier-Free Guidance (CFG) to train the motion-denoising generative model. By training the model to learn both conditioned and unconditioned distributions (e.g., setting c = ∅ for 10% of the samples), CFG ensures that the models can effectively capture the underlying data distribution across various conditions. In inference, we can trade-off text-motion consistency and fidelity using s:𝐺 𝑠 (𝑥 𝑡 , 𝑡, 𝑐) = 𝐺 (𝑥 𝑡 , 𝑡, ∅) + 𝑠 • (𝐺 (𝑥 𝑡 , 𝑡, 𝑐) -𝐺 (𝑥 𝑡 , 𝑡, ∅)) (3)This ability to balance text-motion consistency and fidelity is crucial for producing varied yet realistic outputs, enhancing the overall quality of generated motion.Time-consuming inference time remains a major challenge for diffusion-based approaches. To address this problem, we improve inference speed by integrating four effecient and training-free tricks in the inference process: 1) efficient sampler, 2) embedded-text cache, 3) parallel CFG computation, and 4) low-precision inference.We integrate the SDE variant of second-order DPM-Solver++ sampler (SDE DPM-Solver++ 2M) into diffusion-based motion generation to reduce denoising iterations. DPM-Solver is a high-order solver for diffusion stochastic differential equations (SDEs), which implies additional noise will be introduced during the iterative sampling. Thereby, stochasticity of its sampling trajectories helps to reduce the cumulative error [32] , which is crucial for the realism of generated motion. In addition, we adopt the Karras Sigma [12] to set discrete timesteps. This method leverages the theory of constantvelocity thermal diffusion to determine optimal timesteps, thereby maximizing the efficiency of motion denoising within a given number of iterations.We integrate the Embedded-text Cache mechanism into the inference process to avoid redundant calculations. In diffusion-based motion generation, the text prompt remain unchanged across iterations, resulting in same embedded text in each computation of the denoising network. Specifically, we compute the text embedding initially and subsequently utilize the embedded text directly in each network forward, thereby reducing computational redundancy and speeding up inference.We implement the inference process of CFG in parallel to speed up the single iteration calculation while maintaining model generation performance. Due to the CFG mechanism Equation 3, in each iterative step during inference, it is necessary to execute a conditional and an unconditional denoising, respectively, using the trained motion network, and then sum up the results.We utilize half-precision floating point (FP16) computation during inference to accelerate processing. Newer hardware supports enhanced arithmetic logic units for lower-precision data types. By applying parameter quantization, we convert FP32 computations to lower-precision formats, effectively reducing computational demands, parameter size, and memory usage of the model. The key to eliminating foot skating is to first identify the foot joints and frame ranges where foot skating occurs, and then anchor those keypoints at their positions 𝑝 in the intermediate frames. We formulate this constraint as a loss term shown in Equation 4where 𝑗 indicates joint and 𝑓 is frame ranges.EQUATION𝐽 𝑠𝑘𝑎𝑡𝑖𝑛𝑔 contains all the joints where foot skating may occur, specifically including right ankle, right toes, left ankle and left toes. 𝐹 𝑠𝑘𝑎𝑡𝑖𝑛𝑔 is a collection of all frame ranges of the joint 𝑗. 𝑃 𝑗 means the positions of joint 𝑗. We incorporate Equation 4 to a gradient descent algorithm to correct foot skating motion.Following UnderPressure [18] , we use vertical ground reaction forces (vGRFs) to identity foot joint 𝑗 and its skating frames 𝑓 . The vGRFs predition model of UnderPressure 𝑉 23 requires motion of a predefined 23 joints skeleton 𝑆 23 , which is different from our motion data. In our work, we utilize HumanML3D [6] with 22 skeletal joints 𝑆 22 and KIT-ML [20] motion with 21 skeletal joints. The subsequent foot skating cleanup primarily focused on HumanML3D. We transferred the pre-trained weights of 𝑉 23 to our own model 𝑉 𝜃 22 using the constraints Equation 5, enabling us to directly predict the vertical ground reaction forces for HumanML3D motions. 𝑃 is keypoints of HumanML3D Drawing inspiration from UnderPressure [18] , we use foot contact loss 𝐿 𝑓 𝑜𝑜𝑡 to fix contact joints, pose loss 𝐿 𝑝𝑜𝑠𝑒 and trajectory loss 𝐿 𝑡𝑟𝑎 𝑗𝑒𝑐𝑡𝑜𝑟 𝑦 to to keep the semantic integrity of motion, vGRFs loss 𝐿 𝑣𝐺𝑅𝐹𝑠 to keep valid foot pose. Our supplementary material provides detailed definitions of these loss terms. The final definition of our loss function is as Equation 6 [18] where 𝜔 𝑞 , 𝜔 𝑓 , 𝜔 𝑡 , 𝜔 𝑣 are weights of its loss item. 𝑃 is keypoints of footskating motion and P is the result keypoints after footskate cleanup.Through our method, the footskate cleanup process can be generalized to various skeletal motions.In a few cases, motion corrected by Equation 6 may occurs unreasonable or unrealistic poses. The diffusion model trained on a large amount of motion data learns the prior knowledge of real motions and has the ability to correct the invalid motions.Therefore, we use our pretrained diffusion model to correct such cases. Motivated by OmniControl [31] and Physdiff [34] , we incorporates footskate cleaning method into the diffusion denoising process, denote as StableMoFusion * . 
experiments 5.1 dataset and evaluation metrics: 
 We use HumanML3D [6] and KIT-ML [20] dataset for our experiments. HumanML3D Dataset contains 14,646 motions and 44,970 motion annotations. KIT Motion Language Dataset contains 3,911 motions and 6,363 natural language annotations.The evaluation metrics can be summarized into four key aspects: 1) Motion Realism: Frechet Inception Distance (FID), which evaluates the similarity between generated and ground truth motion sequences using feature vectors extracted by a pre-trained motion encoder [6] . 2) Text match: R Precision calculates the average top-k accuracy of matching generated motions with textual descriptions using a pre-trained contrastive model [6] . 3) Generation diversity: Diversity measures the average joint differences across generated sequences from all test texts. Multi-Modality quantifies the diversity within motions generated for the same text. 4) Time costs: Average Inference Time per Sentence (AITS) [3] measures the inference efficiency of diffusion models in seconds, considering generation batch size as 1, without accounting for model or data loading time.In all of our experiments, FID and R Precision are the principal metrics we used to conduct our analysis and draw conclusions.For training, we use DDPM [8] with 𝑇 = 1, 000 denoising steps and variances 𝛽 𝑡 linearly from 0.0001 to 0.02 in the forward process. And we use AdamW with an initial learning rate of 0.0002 and a 0.01 weight decay to train the sample-prediction model for 50,000 iterations at batch size 64 on an RTX A100. Meanwhile, learning rate reduced by 0.9 per 5,000 steps. On gradient descent, clip the gradient norm to 1. For CFG, setting c = ∅ for 10% of the samples.For inference, we use the SDE variant of second-order DPM-Solver++ [17] with Karras Sigmas [12] in inference for sampling 10 steps. The scale for CFG is set to 2.5.We compare our StableMoFusion with several state-of-the-art models, including T2M [6] , MDM [27] , MLD [3] , MotionDiffuse [36] , T2M-GPT [35] , MotionGPT [9] , ReMoDiffuse [37] , M2DM [14] and fg-T2M [30] . on the HumanML3D [6] and KIT-ML [20] datasets in Table 2 and Table 3, Real 0.002 ±.000 0.511 ±.003 0.703 ±.003 0.797 ±.002 9.503 ±.065 -T2M [6] 1.067 ±.002 0.457 ±.002 0.639 ±.003 0.743 ±.003 9.188 ±.002 2.090 ±.083 MDM [27] 0.544 ±.044 0.320 ±.005 0.498 ±.004 0.611 ±.007 9.599 ±.086 2.799 ±.072 MLD [3] 0.473 ±.013 0.481 ±.003 0.673 ±.003 0.772 ±.002 9.724 ±.082 2.413 ±.079 MotionDiffuse [36] 0.630 ±.001 0.491 ±.001 0.681 ±.001 0.782 ±.001 9.410 ±.049 1.553 ±.042 GMD [13] 0.212 --0.670 9.440 -T2M-GPT [35] 0.116 ±.004 0.491 ±.003 0.680 ±.003 0.775 ±.002 9.761 ±.081 1.856 ±.011 MotionGPT [9] 0.232 ±.008 0.492 ±.003 0.681 ±.003 0.778 ±.002 9.528 ±.071 2.008 ±.084 ReMoDiffuse [37] 0.103 ±.004 0.510 ±.005 0.698 ±.006 0.795 ±.004 9.018 ±.075 1.795 ±.043 M2DM [14] 0.352 ±.005 0.497 ±.003 0.682 ±.002 0.763 ±.003 9.926 ±.073 3.587 ±.072 Fg-T2M [30] 0.243 ±.019 0.492 ±.002 0.683 2.770 ±.109 0.370 ±.005 0.569 ±.007 0.693 ±.007 10.91 ±.119 1.482 ±.065 MDM [27] 0.497 ±.021 0.164 ±.004 0.291 ±.004 0.396 ±.004 10.847 ±.109 1.907 ±.214 MLD [3] 0.404 ±.027 0.390 ±.008 0.609 ±.008 3.204 ±.027 10.80 ±.117 2.192 ±.071 MotionDiffuse [36] 1.954 ±.062 0.417 ±.004 0.621 ±.004 0.739 ±.004 11.10 ±.143 0.730 ±.013 T2M-GPT [35] 0.514 ±.029 0.416 ±.006 0.627 ±.006 0.745 ±.006 10.921 ±.108 1.570 ±.039 MotionGPT [9] 0.510 ±.016 0.366 ±.005 0.558 ±.004 0.680 ±.005 10.35 ±.084 2.328 ±.117 ReMoDiffuse [37] 0.155 ±.006 0.427 ±.014 0.641 ±.004 0.765 ±.055 10.80 ±.105 1.239 ±.028 M2DM [14] 0.515 ±.029 0.416 ±.004 0.628 ±.004 0.743 ±.004 11.417 ±.97 3.325 ±.37 Fg-T2M [30] 0.571 ±.047 0.418 ±.005 0.626 ±.004 0.745 ±.004 10.93 ±.083 1.019 ±.029StableMoFusion (Ours) 0.258 ±.029 0.445 ±.006 0.660 ±.005 0.782 ±.004 10.936 ±.077 1.362 ±.062their own paper and we run the evaluation 20 times and ± indicates the 95% confidence interval.Our method achieves the state-of-the-art results in FID and R Precision (top k) on the HumanML3D dataset, and also achieves good results on the KIT-ML dataset: the best R Precision (top k) and the second best FID. This demonstrates the ability of Stable-MoFusion to generate high-quality motions that align with the text prompts. On the other hand, while some methods excel in diversity and multi-modality, it's crucial to anchor these aspects with accuracy (R-precision) and precision (FID) to strengthen their persuasiveness. Otherwise, diversity or multimodality becomes meaningless if the generated motion is bad. Therefore, our Stable-MoFusion achieves advanced experimental results on two datasets and shows robustness in terms of model performance.StableMoFusion * focuses on the real effect of footskate cleanup. Therefore, the timestep to begin cleaning footskate during inference depends on the motion and thus the StableMoFusion * doesn't apply to the evaluation process of [6] . Figure 5 shows the visual results of our footskate cleanup method, StableMoFusion * . The red bounding box of footskate motion clearly has multiple foot outlines, whereas ours shows only one. The comparison graph shows the effectiveness of our method for cleaning footskate. Directly applying the footskate cleanup method of Un-derPressure [18] to our motion would result in motion distortion, while our method effectively avoids such deformation. In our supplementary material, we will further present a comparison between our method and the UnderPressure method by videos to illustrate it. We calculate AITS of StableMoFusion and ReMoDiffuse [37] with the test set of HumanML3D [6] on Tesla V100 as MLD [3] does, the other results of Figure 1 are borrowed from [3] . For instance, MDM [27] with CFG requires 24.74s for average inference; Motion-Diffuse [36] without CFG uses condition encoding cache and still requires 14.74s of average inference. While the MLD [3] reduces the average inference time to 0.217s by applying DDIM50 in latent space, we find this approach lacks the ability to edit and control motion by manipulating the model input.To tackle this, we employ 1) efficient sampler, 2) embedded-text cache, 3) parallel CFG computation, and 4) low-precision inference to reduce iteration counts and network latency. As shown in Figure 1 , our StableMoFusion significantly shortens the inference time and achieves higher performance within the original motion space.However, it remains incontrovertible that StableMoFusion's inference speed trails behind that of MLD, and fails to meet the industry's real-time standard with an average inference time of 0.5s. Thus, our future work will focus on acceleration: the inference time of StableMoFusion is currently tied to the computation of the network, and we will further investigate how to scale down the model and how to reduce single-step latency in inference.5.6.1 Network Architecture. We evaluate and compare all the architectures mentioned in Section 4.1 with the same training and inference pipeline. For a fair comparison, all methods use the real motion length from the ground truth to clip generated motion and seed(0) for one evaluation. As Table 4 show, each network enhancement in cross-attention has demonstrated performance enhancements, elucidating its pivotal role in augmenting model efficacy and effectiveness. Among them, Conv1D UNet achieves the best generation performance. And fine-tuning Conv1D UNet's GroupNorm can effectively improve its performance on the KIT-ML dataset, reducing the FID by about 64%. It also proves that the GoupNorm tweak on UNet is mainly useful for the dataset with dispersed length distributions, such as KIT-ML dataset. 5.6.2 Effectie Inference. By using the SDE variant of second-order DPM-Solver++ with Karras sigma, the inference process of diffusionbased motion generation is able to significantly reduce the minimum number of iterations required for generation from 1000 to 10 while enhancing the quality of generated motions, approximately 99% faster than the original inference process, as shown in Table 5 .The application of embedded text caching and parallel CFG further reduces the average inference time by about 0.3s and 0.15s, respectively. Our experiments also show that reducing the computational accuracy of the motion-denoising model by half, from FP32 to FP16, does not adversely affect the generation quality. This suggests that 32-bit precision is redundant for motion generation task.  
: 
 Table 1 : StableMoFusion achieves superior performance on motion generation compared to other state-of-the-art methods. Lower FID and higher R Precision mean, the better.FID↓ R Precision (top3)↑ MDM [27] 0.544 0.611 MLD [3] 0.473 0.772 MotionDiffuse [36] 0.630 0.782 ReMoDiffuse [37] 0.103 0.795 StableMoFusion (Ours) 0.098 0.841 Despite the notable progress made by diffusion-based motion generation methods, its development is still hindered by several fragmented and underexplored issues: 1) Lack of Systematic Analysis: these diffusion-based motion generation work usually employ different network architectures and training pipelines, which hinders cross-method integration and the adoption of advancements from related domains. 2) Long Inference Time: due to the timeconsuming iterative sampling process, most existing methods are impractical for applications with virtual characters and humanoid robots, where real-time responsiveness is crucial. 3) Footskate Issue: foot skating (footskate) in generated motions remains a major concern. This significantly undermines the quality of generated motions and limits their practical applicability.Therefore, in order to fill these research gaps and enhance the effectiveness and reliability of diffusion-based motion generation in practical applications, our study conducts a comprehensive and systematic investigation into network architectures, training strategies, and inference process. Our investigation is specifically directed towards text conditional motion generation, as text prompts are arguably the most promising format for practical application and the most convenient input modality among various conditional signals. Ultimately, we present a robust and efficient framework for diffusion-based motion generation, called StableMoFusion, as illustrated in Figure 2 .In StableMoFusion, we use Conv1D UNet with AdaGN and linear cross-attention as the motion-denoising network, and improve its generalization capability with GroupNorm tweak. During training, two effective strategies were employed to enhance the network's ability to generate motion. During inference, we use four trainingfree acceleration tricks to achieve efficient inference. Furthermore, we present a footskate cleanup method based on a mechanical model and optimization.Extensive experiments demonstrate that StableMoFusion achieves an excellent trade-off between text-motion consistency and motion quality compared to other state-of-the-art methods, as shown in Table 1 . Meanwhile, Stablemofusion's efficient inference process notably reduces the minimum number of iterations required for generation from 1000 to 10, as well as shorter inference times than methods of about the same performance, achieving an average inference time of 0.5 seconds on the Humanm3D test set, as shown in Figure 1 . In addition, our footskate cleanup method within diffusion framework sizably solves the foot skating problem of motion generation as shown in Section 5.4.Our major contributions can be summarized as follows:• We perform a systematic evaluation and analysis on the design of each component in the diffusion-based motion generation pipeline, including network architectures, training strategies, and inference process.  
 
 Appendix:  


Decision: The submitted paper presents a comprehensive and well-executed study on human motion generation using a diffusion-based method. The paper's strengths lie in its detailed comparative analysis, practical implications, and robust experimental validation. Despite a few areas that could benefit from further elaboration and clarification, the overall contribution and quality of the work warrant acceptance. 

Strengths:

The paper provides a thorough comparison study with quantitative results across numerous related methods. It evaluates performance on two different datasets and includes an ablation study for some model design choices, enhancing the credibility of the results.
The method addresses practical issues such as inference speed and perceptual problems like footskate reduction. Although the method for footskate reduction is inspired by a previous method, certain tweaks improve the results, demonstrating practical relevance and innovation.
The paper is well-written and well-motivated, with clear figures and a supplementary video that effectively present the findings.
The investigation into the design of each component in diffusion-based methods and the challenges in text-to-human motion generation is insightful and valuable.
The ablation study is convincing, demonstrating the importance of various design choices and the effectiveness of the proposed method. This is a significant strength, as it provides clear evidence of the method’s value.
The selection of three highly representative structures for comparison (Convolutional U-Net, DiT (Diffusion Transformer), and RetNet (Retentive Network)) is well thought out and valuable for the community. The comparison is clear and informative.
The footskate section, designed as a form of smoothing, appears to be effective and contributes positively to the quality of the generated motion.

Limitations:

The analysis of the model architecture, specifically the choice between transformer and U-Net models, could be more detailed. Providing more insight into why U-Net works well and comparing it with a transformer backbone would be beneficial. The current analysis is good but could be enhanced, potentially by revising Figure 3 or moving it to the supplementary section.
Including quantitative measures for footskate reduction would strengthen the paper. Specifically, evaluating the deviation in the position of foot joints and the impact on other joints when enforcing footskate loss would provide a more comprehensive analysis.
While the efficient diffusion framework for motion generation is emphasized, the backbone does not necessarily facilitate the inference process, and DPM-Solver++ is an existing method. Thus, the novelty might appear limited. Clarifying the unique contributions and integration would help.
The inference process of the proposed method is not illustrated in the paper. Including this, especially since Algorithm 1 only covers the vanilla DDPM, is important for understanding the efficient diffusion method proposed.
Providing a detailed description of where the results for each experimental subject in the ablation study were sourced from would improve transparency and replicability.


The paper presents a significant contribution to the field of text-to-human motion generation, with a well-motivated approach, thorough comparative analysis, and effective practical solutions. While there are some areas that could be further elaborated, the strengths and overall quality of the work strongly support its acceptance. The authors are encouraged to address the identified limitations in the final revision to further enhance the paper’s clarity and impact.

Reviews: review: paper_summary: paper_summary: This paper tackles the problem of text condtioned human motion generation using diffusion models. The authors conduct a study of relevant architechture details around designing of the overall diffusion-based pipeline, and motivate their changes in comparison to exisiting methods. The primary contribution is in analysis with previous works and introducing consistency terms to reduce the issue of foot skating in generated motions. Experiments include detailed comparisons and ablation study to motivate design decisions and improved performance both in terms of perceptual quality and inference speed.
strength_weakness: strength_weakness: - Detailed comparison study in experiment section with quantitative results across numerous related methods and, comparing results on two different datasets for evaluation. Inclusion of ablation study for some model design choices. 

- Method focuses on practical implications like inference speed and perceptual issues with footskate reduction. Althought the method for footskate reduction is inspired from a previous method, certain tweaks on it seem to improve the results. 

- The paper seems well-written and sufficiently well-motivated. The figures and supplmentary video were also clear in their presentation.
limitations: limitations: - In terms of analysis for the model architecture which is presented as a contribution, I feel that the study needs some tuning w.r.t choices, like for example transformer v/s unet model. Readers might appreciate if the authors can bring out details like why unet works well (compare with same model but transformer backbone). The current analysis in the paper is definitely not insufficient, but perhaps a different figure in place of Fig 3 could help (current Fig. 3 could go to supplementary as it seems more like an implementation detail).

- Some quantitive measures on the footskate reduction would help showcase the contribution from the method described in Section 4.4 and Appendix A. I assume identification of foot-skating and it's deviation from some ideal behaviour can be done offline? Results on the deviation in the position of foot joints could be a start perhaps.

- Another aspect of footskate reduction that might be worthwhile to consider (and evaluate) is the impact on other joints deviation (from an ideal / or some baseline generation) when enforcing footskate loss. From a couple of videos it seemed that the knees bend more / have a slight jerk to compensate for foot joints remaining static?
paper_summary: paper_summary: This paper is about the text-to-human motion generation task, which is based on the diffusion model. The author conducted a comprehensive investigation of the network architectures, training strategies, and inference processes. In addition, the author describes the challenges in this task, and tries to alleviate the challenges faced in the existing diffusion-based model for this task.
strength_weakness: strength_weakness: 1. Investigating the effect of the design of each component in diffusion-based methods.
2. Analysing the challenges in the existing text to human motion generation task.
limitations: limitations: 1. The novelty of this paper is limited. For example, the backbone of the diffusion model is generally tested in ablation studies. That is not necessarily a novelty to contribute to this paper.
2. The author mentioned that the proposed method accelerates the inference time of the diffusion model, however, the author only describes the algorithm of the vanilla DDPM in the paper, not listing the comparison or the improved part of the algorithm.
3. The foot skating phenomenon can be seen as a loss function for this task, and other papers also try to alleviate this issue in their papers, but do not consider this as a main contribution. So, the contribution is limited from this point.
paper_summary: paper_summary: The paper discusses a novel framework for generating human motions using diffusion models. It aims to address the challenges of computational overhead and the common issue of footskate in generated motions, which are prevalent in existing diffusion-based solutions. The authors conduct a comprehensive investigation into network architectures, training strategies, and inference processes to tailor components for efficient and high-quality human motion generation. They also present a method to eliminate footskate by identifying foot-ground contact and correcting foot motions during the denoising process. The framework, StableMoFusion, combines these components to achieve robust and efficient human motion generation, outperforming current state-of-the-art methods in terms of motion quality and inference efficiency.
strength_weakness: strength_weakness: 1. I am very pleased with the author's selection of three highly representative structures for comparison, as depicted in Figure 3: the Convolutional U-Net, DiT (Diffusion Transformer), and RetNet (Retentive Network). The comparison is very clear and also very valuable.

2. The ablation study is quite persuasive, and I am very convinced that this is an article worth reading by more people.

3. I interpret the design of the Footskate section as a kind of smoothing, which appears to be effective.
limitations: limitations: please provide a detailed description of where the results for each experimental subject in the ablation study were sourced from