MotionDreamer: One-to-Many Motion Synthesis with Localized Generative Masked Transformer

Authors: Yilin Wang, chuan guo, Yuxuan Mu, Muhammad Gohar Javed, Xinxin Zuo, Juwei Lu, Hai Jiang, Li cheng
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: motion synthesis, generative masked modelling, vector quantization, single instance learning

Abstract:
Generative masked transformer have demonstrated remarkable success across various content generation tasks, primarily due to their ability to effectively model large-scale dataset distributions with high consistency. However, in the animation domain, large datasets are not always available. Applying generative masked modeling to generate diverse instances from a single MoCap reference may lead to overfitting, a challenge that remains unexplored. In this work, we present MotionDreamer, a localized masked modeling paradigm designed to learn motion internal patterns from a given motion with arbitrary topology and duration. By embedding the given motion into quantized tokens with a novel distribution regularization method, MotionDreamer constructs a robust and informative codebook for local motion patterns. Moreover, a sliding window local attention is introduced in our masked transformer, enabling the generation of natural yet diverse animations that closely resemble the reference motion patterns. As demonstrated through comprehensive experiments, MotionDreamer outperforms the state-of-the-art methods that are typically GAN or Diffusion-based in both faithfulness and diversity. Thanks to the consistency and robustness of quantization-based approach, MotionDreamer can also effectively perform downstream tasks such as temporal motion editing, crowd motion synthesis, and beat-aligned dance generation, all using a single reference motion. Our implementation, learned models and results are to be made publicly available upon paper acceptance.

Decision: accept (poster)
Reviewer scores: 8, 8, 3, 5

Reviews and discussion:
<thread>
creation date:
1730555319416

thread_type:
official_review

<thread_object>
creator: reviewer_fzyk

<content>
summary:
This paper proposes a generative masked model (MotionDreamer) for motion synthesis domain based on the motivation "large datasets are not always available". MotionDreamer designs a local masked modeling paradigm to learn motion internal patterns from a given motion with a key sliding window local attention module. Experiments demonstrate various motion tasks, including crowd motion synthesis, temporal editing and beat-aligned dance synthesis.

strengths:
- The paper is well-written and is easy to follow. 

- Qualitatively results look impressive. Ablation studies prove the effectiveness of the designs.

- The authors introduce various applications of motion synthesis and achieve great performance.

weaknesses:
- Generative masked modeling for motion synthesis has been investigated in many recent works, such as MMM, Momask, MotionGPT, as described in paper. On the other hand, single motion synthesis is also not new, such as SinMDM. It seems that MotionDreamer integrates the techniques of generative masked modeling into the task of single motion synthesis, including motion tokenization and codebook distribution regularization.

- Capturing local dependencies of motion features is quite important for single motion synthesis. SinMDM introduces QnA layers that allow local attention with a temporally narrow receptive field. And MotionDreamer incorporates sliding window local attention to achieve this. Besides, another key design of codebook distribution regularization has been explored in single image synthesis domain. Therefore, I think the technical contribution of the paper is somewhat limited.

-  In Ablation Studies (Sec. 4.4), it's better to use QnA layers in SinMDM to replace sliding window local attention and show the comparison of the performance. Due to the same task, the paper should demonstrate its unique effectiveness.

questions:
See Weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730712612257

thread_type:
official_review

<thread_object>
creator: reviewer_oiwr

<content>
summary:
This paper proposes a localized masked modeling paradigm designed to learn motion internal patterns from a given motion with arbitrary topology and duration. It introduces the codebook distribution regularization and sliding window local attention (SwinAttn) mechanism to avoid codebook collapse and overfitting.

strengths:
The codebook distribution regularization effectively prevents codebook collapse. The sliding window local attention mechanism in the Local-M transformer captures local dependencies and ensures smooth transitions across overlapping windows, enhancing the fidelity and diversity of generated motions.

weaknesses:
This paper applies some recent advancements from the image domain to motion synthesis, which is a commendable attempt. However, the contribution feels somewhat limited. As a work focused on enhancing the VQVAE structure for motion synthesis, the authors might benefit from incorporating insights and experiments from NCP, rather than simply adapting the network architecture.
# Added after AC's comment
I'm grateful for the feedback and suggestions. To clarify, the aspects of the contribution that feel limited include the novelty in adapting image domain advancements without significant modification or integration specific to motion synthesis. Additionally, while some metrics have been provided, they may not fully reflect the method's advantages. I believe visualizations of the latent space could be helpful in demonstrating its effectiveness. Regarding NCP, I apologize for the oversight—it refers to Neural Categorical Priors for Physics-Based Character Control. Thank you for the guidance.

questions:
In the final section of the paper, the authors mention several other applications. Could the authors elaborate on how the network architecture performs and what advantages it offers for these tasks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730732437378

thread_type:
official_review

<thread_object>
creator: reviewer_l4v3

<content>
summary:
The paper proposes Motiondreamer, a framework for one-to-many motion synthesis. The paper aims to learn the underlying motion pattern in a given sequence and leverage the motion pattern to generate natural and diverse motions.
The general idea of the paper is to first quantize a motion sequence into tokens, and then train a masked transformer to generate these motion tokens. 
There are two core technical contributions,  one lies in regularising motion tokenization to encourage more coverage, and the other one lies in injecting localality into the makes transformer to encourage motion decomposition.

strengths:
- The model is well-designed. Given the scarcity of motion data in this one-to-many setting, the inductive bias in the model design is particularly important. The overall framework follows a masked transformer architecture which is simple, but multiple practical techniques have been contributed to balance the motion faithfulness and the motion diversity, including
    - A KL term is added to the quantizer training besides the standard commitment loss to encourage coverage.
    - A sliding window attention that adds locality to the transformer to encourage diversity.
    - A differentiable dequantization loss that helps to compute loss on the motion space.
    
    All of them are well-motivated and make sense to me.
    
- The ablation study is very comprehensive. Although the above techniques, like KL regularisation, are quite standard and empirical, the paper provides detailed experiments to support their importance to the final results. I find those experiments convincing.
- The paper shows qualitative results on a webpage, which generally looks good to me. Compared to SinMDM, the proposed method reduces unnatural motions. Compared to SinMDM, the generated motion seems more diverse.

weaknesses:
- Table 1 and Figure 3 are inconsistent in the comparison between GenMM with Ours. Table 1 shows that GenMM has much better coverage than Ours. However, in Figure 3, the failure mode of GenMM is missing a part of the reference motion.
- The beat-aligned dance synthesis application does not convince me. The generated dances have a clear mismatch with the tempo. Most importantly, the application (including implementation details) is not well documented in the paper so I am not fully sure what is happening there, for example, what is a “light-weight encoder-decoder”? To my understanding, the beat-aligned dance is like an extension of the “subpart generation” task while keeping predefined keyposes on the music beats. That is not sufficient to produce a beat-aligned dance in my opinion.
- Some of the techniques could be a little niche to be presented as part of the method. For example, AttnFuse could better fit into implementation details.
- Minor
    - L281: by progressively fill → by progressively filling.

----
The overall framework makes sense to me. The proposed techniques are simple, but they are well-motivated and are supported by solid ablation experiments. I am convinced that they contribute to the overall performance. My concerns are about the comparison between GenMM and the dance generation application part. The presentation can be improved as well. I am slightly inclined to accept.

questions:
Please address the weaknesses above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730755227082

thread_type:
official_review

<thread_object>
creator: reviewer_rqv8

<content>
summary:
The paper introduces a novel method for intrinsic generation for motion, i.e., augmenting a single input motion sequence into multiple versions. The paper employs an underexplored generative approach - the masked transformer - and presents SOTA performance. To improve diversity and overall quality, the paper also offers a simple KL-divergence based loss to encourage full usage of the latent space, to mitigate mode collapse. The paper reports a complete evaluation, including comparisons, ablation studies, quantitative evaluation, qualitative evaluation and a user study.

strengths:
- Paper is well written and easy to follow.
- Proposed method is simple and elegant
- The mask transformer mechanism is a refreshing approach
- Codebook distribution regularization is interesting
- Evaluation is rigorous, including all conventional metrics

weaknesses:
- The application is a niche one 
- The method, being simple, is also limited in contribution
- Qualitative evaluation is limited

questions:
As stated above, I believe that paper presents interesting ideas, and I would like to see this player out there with the other methods. 

My main concern is the scope of impact. Perhaps these ideas can and should be examined in a wider scope:
- can the codebook regularization loss improve other VQ methods off the bat? 
- More ambitiously, can this method be combined with others, for example can the masked transformer be applied in the bottle neck of a diffusion process? or perhaps instead of the VAE?
- similarly, some more discussion regarding the regularization could benefit the paper. What other alternatives could have been employed?

In addition, I would have liked to see some more qualitative results 

Minor concerns:
- Swin is already rather widely adopted and known for shifting windows (Swin-Transformer), I recommend using another name for the sliding window approach (which I find simple and elegant for reducing the receptive field)
- "Crowd motion synthesis" can also be misinterpreted here, as an algorithm to avoid collisions, I would also find a different name here.
- I find figures 1 and 2 a bit exaggerated. The right side of Fig 2 depicts little information for its size. Figure 1 is also rather overwhelming, half of the amount of characters would probably go a long way.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732321739781

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to All the Reviewers (updated post discussion)

comment:
**We want to first express our big thanks to all reviewers for the thoughtful feedback, as well as for highlighting both the strengths and areas of improvement in our work.** We are particularly grateful for the encouraging comments on our framework, including the novel application of generative masked modeling for single motion synthesis (Reviewers rqV8, Fzyk), the well-motivated and elegantly designed technical contributions such as the codebook regularization loss and sliding window local attention (Reviewers L4V3, oiwR, Fzyk), and the rigorous evaluation through quantitative metrics, qualitative results, and ablation studies (Reviewers rqV8, L4V3, Fzyk).

***During the author-discussion phase***, we truly appreciate the acknowledgement from reviewers for our responses and updates on evaluation metrics consistency (Reviewer L4V3), implementation details for applications (Reviewer L4V3, oiwR), and effectiveness of our proposed sliding window local attention layer (Reviewer Fzyk). ***In our follow-up responses***, we've added additional experiments on latent space visualization as requested (Reviewer oiwR), and further illustrated the novel insights and effectiveness of our key contributions (Reviewer oiwR, Fzyk). 

---

In response to the reviewers’ suggestions and comments on further improvements to our work, we have carefully made individual responses to each reviewers correspondingly. To further address the reviewers' concerns, we have also updated our paper and supplementary materials to provide further clarification and demonstrations based on the comments. Below are the key updates:

### **Paper Updates:**

- Renamed **“SwinAttn”** to **“SlidAttn”** and **“crowd motion synthesis”** to **“crowd animation”** for clarity.
- Added clarifications on **Table 1 and Figure 3 consistency** in Section 4.2.
- Expanded **application implementation details** in Appendix A.4, supported by Figure 8.
- Added detailed comparisons of **SlidAttn and QnA layers** in Table 6, supported by analyses in Appendix A.6.
- Included discussions on the **broader impact of codebook regularization** and potential alternatives (e.g., entropy regularization) in Appendix A.7.

### **Visualization Video Demo Updates:**

- Added more results on **non-human skeleton samples** and qualitative ablations in the webpage (Appendix A.1).
- Highlighted **qualitative comparisons of QnA vs. SlidAttn generations** for the newly added ablation study (Appendix A.6 and Table 6).

### **Additional Results in Discussion Post Revision Deadline:**
- Latent space visualization shown and analyzed (see [follow-up response to reviewer oiwR](https://openreview.net/forum?id=d23EVDRJ6g&noteId=MRLUiLpqvu))

---
We would also like to take this opportunity to reiterate the novelty and contributions of MotionDreamer:

1. **Generative Masked Modeling for Single Motion Synthesis:**
MotionDreamer pioneers the application of generative masked modeling for single-instance learning in one-to-many motion synthesis. By modeling explicit categorical distributions of internal motion patterns, it achieves state-of-the-art performance in generating diverse and natural motions that faithfully preserve the reference motion’s internal patterns. *Supporting results: Table 1, Figure 3, and Appendix A.1.*

2. **Codebook Regularization for Single Motion Tokenization:**
To address the inherently imbalanced distribution of internal patterns in single motion sequences, the proposed KL-divergence-based regularization ensures balanced token utilization. This significantly enhances codebook utilization and representation fidelity, enabling robust tokenization for effective single-instance synthesis. *Supporting results: Table 2, Figure 3, Figure 5, and Appendix A.1.*

3. **Sliding Window Local Attention (SlidAttn):**
SlidAttn introduces a novel sliding window computation and aggregation paradigm, which explicitly operate overlapping local windows. This ensures smoother transitions and better correlation across internal motion patterns, which enhances motion fidelity and diversity while addressing overfitting to global patterns common in single-instance setting. *Supporting results: Table 1, Table 6, Appendix A.6, and Appendix A.1.*

4. **Flexible and Lightweight Framework:**
MotionDreamer’s lightweight and adaptable design supports various applications, including crowd animation, temporal editing, and beat-aligned dance synthesis, demonstrating versatility across diverse motion synthesis tasks. *Supporting results: Appendix A.4, Figure 8, and Appendix A.1.*

---

We hope that these updates and additional insights further strengthen our work and demonstrate its novelty and soundness. Thank you again for your valuable feedback and all the efforts.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733086039585

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear reviewers,

Thank you again for all your time and efforts in reviewing our work! We sincerely appreciate all the insightful comments and suggestions which we believe have helped us improve our work better. So far, we have made our responses to all the initial reviews as well as the follow up comments, and we hope that our responses and updates have answered your questions and addressed your concerns. 

*This is a kind reminder that the author-reviewer discussion phase will end in two days.* If our responses and updates have properly addressed your concerns, we would really appreciate it if you could kindly re-evaluate our work. Please do not hesitate to let us know if you have additional questions. We would be more than happy to open further discussion on them. 

Best,

Paper 9141 Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1733303081897

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Update on the general responses with summary post discussion

comment:
Dear reviewers,

We sincerely appreciate your precious time and efforts on the reviews and discussions throughout the weeks. As the end of the rebuttal phase is approaching, we have ***updated our [general response](https://openreview.net/forum?id=d23EVDRJ6g&noteId=uOLyuPigH7)*** by including the summary of author-reviewer discussion, and our highlighted contributions.

Thank you again for all the valuable feedback!

Best,

Paper 9141 Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces an approach to one-to-many motion synthesis using generative masked modeling. Reviewers generally appreciated the well-designed model, comprehensive evaluations, and practical applications. Strengths include the codebook regularization, sliding window local attention, and strong qualitative results. Some concerns were raised about the technical novelty and comparisons with existing methods like SinMDM. After revisions and clarifications, most concerns were addressed, with ratings improved to acceptance. The AC agreed with Reviewer L4V3 that the framework is well-designed, and its performance is supported by extensive experimental results, although some of the techniques are reminiscent of those applied in other domains. Based on the overall reviews, the paper is recommended for acceptance.