FlexMotion: Lightweight, Physics-Aware, and Controllable Human Motion Generation

Authors: Arvin Tashakori, Arash Tashakori, Gongbo Yang, Z. Jane Wang, Peyman Servati
Primary area: generative models
Keywords: 3D human motion generation, diffusion models, conditional generation, physics aware

Abstract:
Lightweight, controllable, and physically plausible human motion synthesis is crucial for animation, virtual reality, robotics, and human-computer interaction applications. Existing methods often compromise between computational efficiency, physical realism, or spatial controllability. We propose FlexMotion, a novel framework that leverages a computationally lightweight diffusion model operating in the latent space, eliminating the need for physics simulators and enabling fast and efficient training. FlexMotion employs a multimodal pre-trained Transformer encoder-decoder, integrating joint locations, contact forces, joint actuations and muscle activations to ensure the physical plausibility of the generated motions. FlexMotion also introduces a plug-and-play module, which adds spatial controllability over a range of motion parameters (e.g., joint locations, joint actuations, contact forces, and muscle activations). Our framework achieves realistic motion generation with improved efficiency and control, setting a new benchmark for human motion synthesis. We evaluate FlexMotion on extended datasets and demonstrate its superior performance in terms of realism, physical plausibility, and controllability.

Decision: reject
Reviewer scores: 5, 3, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729720027693

thread_type:
official_review

<thread_object>
creator: reviewer_uppu

<content>
summary:
This paper introduces a human motion synthesis model that achieves realistic motion generation with high controllability. The model is trained using an augmented dataset, which utilizes OpenSim to enhance the biomechanical and physical fidelity of the original data. The training process involves three stages. In the first stage, an encoder and decoder are trained to map the motion into a latent space. In the second stage, a diffusion model is trained to generate latent variables from noise, allowing the decoder to reconstruct realistic motions. Lastly, a Spatial Controllability Module is trained to convert different user control inputs into motions. The proposed model outperforms various baselines on several evaluation metrics across different datasets.

strengths:
- The method appears solid and reliable.
- Extensive baseline comparisons and ablation studies demonstrate the results.
- The paper is well-structured and easy to follow

weaknesses:
- Different versions of the proposed methods need more elaboration. Adding this information to the appendix would help readers better understand the approach.
- Some animation visualizations that compares original sources motion, augmented motion and generated motion would be helpful.

questions:
- Are the source motions changed after data augmentation? I am curious about how the source data changes after augmentation compared to the original data.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730355552327

thread_type:
official_review

<thread_object>
creator: reviewer_uemu

<content>
summary:
The framework, FlexMotion, leverages a computationally efficient diffusion model in the latent space, eliminating the need for physics simulators and enabling fast training. It employs a multimodal pre-trained Transformer encoder-decoder that integrates various motion parameters like joint locations, contact forces, joint actuations, and muscle activations to ensure the physical plausibility of the generated motions. FlexMotion also introduces a plug-and-play module for spatial control over motion parameters, enhancing its applicability across different domains.

strengths:
1.The use of a diffusion model in the latent space is a novel approach that significantly reduces computational costs compared to traditional methods that rely on physics engines.

2.The integration of joint locations, contact forces, joint actuations, and muscle activations into a single framework is a comprehensive way to ensure physically plausible motion generation.

3.The plug-and-play module for spatial control over a range of motion parameters adds versatility to the framework, making it suitable for various applications.

4.The paper demonstrates superior performance in terms of realism, physical plausibility, and controllability over existing methods, as shown through evaluations on extended datasets.

5.FlexMotion's lightweight design and efficient training process make it suitable for real-time applications, which is a significant advantage over computationally intensive methods

weaknesses:
1. This paper does not provide any videos to show their qualitative results, which are important to prove their contribution and progress in this research area.

2. In this paper, the property of physics-aware motion is weak. Indeed, we need such properties on flat ground, but the physics-aware property also should work on some uneven terrains and human-human interactions.

3. This paper ignores some baselines, for example, the TL-control in ECCV 2024.

4. The physics-based results are still not as good as the simulation-based method, such as phydiff.

5. Some citation issues, for example the ``Adding conditional control to text-to-image diffusion models'' has two different reference.

questions:
1. The author should discuss more about how to obatin the physcis related inputs, for example torque.

2. The author should give more examples to show their method significantly superpass other methods.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730636442106

thread_type:
official_review

<thread_object>
creator: reviewer_s9g5

<content>
summary:
This paper presents a flexible system for diffusion-based generation of human movements that allows conditioning information in terms of text and various physical constraints. The system is trained in three stages. A transformer-based autoencoder is trained to represent motion-related quantities,  such as kinematic measures, forces, and muscle activation, in a more compact form. A diffusion model for motion generation is then trained in the latent space of the autoencoder, which is finally augmented with inputs from a controllability module to allow for more explicit physical constraints. 

The most important contribution of the paper is the physics-aware autoencoder and the fact that the diffusion model works in the latent space of the autoencoder. The controllability module is quite similar to models that have been used in many other contexts before.

strengths:
While previous approaches have tried to correct generated human movements to improve physical plausibility, often as a postprocessing step, the proposed system relies on the autoencoder’s decoder to produce motion-related quantities that are physically consistent. Two loss functions grounded in the physics of the musculoskeletal system are introduced to facilitate this. The experimental evaluation shows that the autoencoder with associated loss functions considerably improves physical plausibility, even without additional conditioning information.

The fact that the system is trained in stages makes it more modular, and possibly easier to reproduce and modify. This is illustrated by the fact that only a single consumer-grade GPU was used for training.

The paper is well-written and very easy to read and understand, which includes the illustrations. The summary of the experimental results could be improved though by focusing more on the most important observations, rather than commenting on many other less relevant details.

From the results, it appears that the proposed method is superior to all other tested methods in terms of both physical plausibility and speed. One should however keep in mind that not all other methods allow the same kind of conditioning information. However, even without conditioning information, the performance is competitive.

weaknesses:
Unfortunately, the experimental part is not as clear as it could have been. The reader needs more help to interpret the results, preferably with a story that focuses on the most important lessons learned. Possibly to give a complete picture, the two last pages of the paper contain a large number of detailed results that could be better integrated into a limited number of clear conclusions. 

The tables contain several experiments conducted with different conditions, but there ought to be a better explanation of these conditions. For example, does “1 Muscle” mean one muscle activation condition at one point in time or over the whole sequence? Given the results, both of these interpretations could be correct. Or is it for 20% of frames, since for “All Conditions” only up to 20% of frames are used?

The last ablation results in the appendix seem fishy indeed. It should be statistically impossible to get the numbers 2.345, 4.567, 5.678, and 6.789. Strangely, the same combination of numbers 0.198 and 0.298 occurs in two different experiments. The authors ought to clarify this and possibly adjust or remove the results of the last ablation. This review was done assuming that what is in the main paper is correct, while the results in the appendix are not yet complete. 

Minor issues: $\theta_{ctrl}$ in (13) should probably be $\theta_{total}$. FLOGs on line 478, should be FLOPs.

questions:
* How should the different conditions in the tables be interpreted?
* How would the system work if the diffusion model has not worked in the latent space of the autoencoder? Are there any such results in the tables to point at? 
* Why is PhysDiff only tested on HumanML3D? Given that this method seems to be the only other method that permits physical constraints, it would have been good to see results from PhysDiff for the other datasets.
* When do you get FID 0.298 on HumanML3D, when there is no conditional information, or when the Euler and muscle losses are dropped? Or the same number in both cases?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730700025334

thread_type:
official_review

<thread_object>
creator: reviewer_29dj

<content>
summary:
The paper's goal is to build a physics-aware human motion model. The physics-aware multimodal autoencoder is a transformer-based autoencoder mapping the concatenation of pose features and physics quantities to a latent representation, and then back to the original features. This autoencoder is trained with an L2 reconstruction loss and a physics constraint with the Euler-Lagrange equation as in Zhang et al. 2024b. A latent diffusion model with a series of transformers is then used to generate motions in the latent space from the autoencoder. The spatial controllability is introduced to this latent diffusion model in the same way as Zhang et al. 2023c., which is similar to ControlNet but with a copy of transformer blocks.

strengths:
Human motion understanding with physics awareness is an important problem.

The dataset with the OpenSim simulations can help follow-up research.

weaknesses:
Please provide videos for submissions related to motions. Qualitative evaluations are extremely important to support and explain quantitative results.

Applying OpenSim with muscle actuation to a large-scale kinematic motion dataset is not trivial. The authors provide no details on this. The appendix only has information about the OpenSim musculoskeletal model setup.

The formulation seems to have some flaws with missing details. Eq. 4 is missing the muscle activation term. It is not clear how the consistency between the OpenSim simulations and the Euler-Lagrange loss is guaranteed.

The technical novelty is questionable. Core components in this paper are straightforward applications of the previous methods, most notably Zhang et al. 2023b and 2023c but with the OpenSim data.

The authors do not discuss the sim2real problem at all. The actual dynamics in the kinematic motion captures of real people will have a significant gap from the simulations.

questions:
See weaknesses. I would especially like to see videos of the results if the authors are allowed to share new results during the discussion phase.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730742308857

thread_type:
official_review

<thread_object>
creator: reviewer_eiyc

<content>
summary:
FlexMotion incorporates a multimodal Transformer encoder-decoder that integrates joint locations, muscle activations, contact forces, and joint actuations. This ensures the generated motion aligns with human biomechanics without needing external physics simulators. 
By leveraging a diffusion model in latent space, the approach significantly reduces training and inference costs while maintaining performance.

FlexMotion includes a plug-and-play spatial controllability module, allowing precise control over motion parameters, such as joint trajectories and muscle activations. The model outperforms existing methods in physical realism, efficiency, and adaptability across datasets like HumanML3D, KIT-ML, and FLAG3D.

strengths:
FlexMotion represents a significant advancement in the field of human motion generation. This innovative framework blends a lightweight diffusion model with a Transformer encoder-decoder, allowing for efficient and realistic motion synthesis. Unlike previous methods that either rely on physics simulators or lack physical constraints, FlexMotion integrates these constraints directly into its latent space. This unique approach enables the generation of highly realistic and physically accurate human motion while maintaining computational efficiency.

The model's architecture is meticulously designed to achieve optimal performance. It employs a multimodal pre-trained Transformer to capture complex human motion patterns. Additionally, physics-aware constraints are incorporated to ensure that the generated motions adhere to the laws of physics. This combination empowers users with fine-grained control over the synthesized motion, allowing them to specify desired spatial, muscle, and joint actuation parameters.

To evaluate FlexMotion's capabilities, the authors conducted extensive experiments on various datasets, including HumanML3D, KIT-ML, and FLAG3D. The model consistently outperformed state-of-the-art methods in terms of realism, accuracy, and efficiency. Ablation studies further validated the effectiveness of the different components of the model, highlighting the importance of physical constraints and the modular controllability approach.

The implications of FlexMotion extend beyond animation and virtual reality. Its potential applications include robotics, human-computer interaction, and other fields that require realistic and controllable human motion. By addressing the limitations of existing methods, FlexMotion opens up new possibilities for creating more immersive and interactive experiences.

weaknesses:
FlexMotion is a novel framework that pushes the boundaries of human motion generation. By combining a lightweight diffusion model with a Transformer encoder-decoder, it achieves a balance between computational efficiency and physical accuracy. Unlike previous methods, FlexMotion directly embeds physical constraints into its latent space, resulting in more realistic and controllable motion synthesis.

One key limitation for FlexMotion is its adaptability to specific tasks or domains. and the model's dependency on pretrained weights. 
Furthermore, the paper could benefit from a more detailed analysis of the trade-offs between realism and physical accuracy. Exploring how adjustments to physical constraints impact the visual appeal of generated motions would provide valuable insights for users. 

Finally, addressing the lack of real-time testing and validation is crucial for practical applications. Demonstrating the model's performance in real-time scenarios would highlight its suitability for interactive applications like virtual reality and robotics. Additionally, providing more transparency in data augmentation and preprocessing methods would enhance reproducibility and facilitate further research.

questions:
1. Could you elaborate on how the data augmentation process using OpenSim was conducted?   

2. Which muscle activation, joint actuation, and contact force parameters were used, and how were these values calibrated for consistency across the different datasets?

3. Did you observe any cases where the model generated physically implausible or biomechanically inaccurate motions, even with the physics-based loss integration? If so, how frequently did these issues arise, and what measures did you implement to minimize such artifacts?
4. The paper describes the controllability module as a “plug-and-play” addition. How was the module trained or fine-tuned alongside the diffusion model, and what parameters or conditions proved most challenging for control?

5. How does FlexMotion handle motions with varying complexity (e.g., simple walking vs. complex actions like acrobatics)? Did you find any performance discrepancies or limitations in generating more complex motions?

6. Although the model is described as computationally efficient, were any real-time tests conducted to evaluate FlexMotion’s responsiveness? For example, does FlexMotion achieve real-time performance on consumer-grade GPUs or only on high-end systems?

7. Did you encounter any trade-offs between generating visually realistic (aesthetically pleasing) motions and maintaining physical plausibility? If so, how did you approach balancing these aspects, particularly in scenarios where users might prioritize one over the other?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732774790436

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We would like to thank all reviewers for their valuable feedback, constructive criticism, and insightful questions. We appreciate the positive comments on FlexMotion’s contributions to efficient, physics-aware human motion generation and its potential applications across various domains. Below, we summarize our main improvements and responses addressing the key points raised by all reviewers:

- Multiple reviewers emphasized the importance of qualitative video examples to showcase FlexMotion’s generated motions. In response, we have prepared a supplementary video demonstrating the model’s capabilities across different motion scenarios, including controlled parameters and complex movements.

- Reviewers noted that certain conditions, such as "1 Muscle" and "All Conditions," were not clearly defined in the tables and experimental results. We have revised the manuscript to clarify these experimental conditions, including a new paragraph with detailed explanations of each setup and condition for transparency.

- Several reviewers requested more details on our use of OpenSim for data augmentation. We have expanded the appendix to include comprehensive information on the augmentation process, calibration methods, and how biomechanical parameters were derived to ensure consistency across datasets.

- To address requests for real-time performance metrics, we have included preliminary benchmarks of FlexMotion’s responsiveness on consumer-grade GPUs. Our initial tests indicate promising efficiency, and we are actively optimizing the model for broader hardware compatibility.

- Recognizing the importance of FlexMotion's applicability to more varied environments, we have outlined our approach to extending physics-aware motion generation to uneven terrains and interactions. Additionally, we discuss future strategies to address the sim-to-real gap and evaluate real-world applicability, leveraging hybrid techniques and direct comparisons with motion capture data.

- To strengthen our comparative analysis, we have included TL-Control from ECCV 2024 as an additional baseline in our tables. This addition provides a broader evaluation of FlexMotion’s performance relative to other relevant models.

- We have corrected minor citation errors and numerical inconsistencies, particularly in the ablation results within the appendix, to ensure accuracy and clarity throughout the paper.

We hope that these improvements and clarifications address the reviewers' concerns comprehensively. Below, we provide detailed responses to each specific comment from each reviewer.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The submission introduces a system for controllable and physically plausible human motion synthesis.  Reviewers are overall lukewarm, with one reviewer arguing strongly for rejection due to the physically incorrect results in the video.  The AC read the submission, reviews, rebuttals, and author notes, and agreed with the overall sentiment of the reviewers that the submission is not ready to be published due to the underwhelming results.  The authors are encouraged to revise the method and manuscript for the next venue.