MotionDreamer: One-to-Many Motion Synthesis with Localized Generative Masked Transformer
Authors: Yilin Wang
chuan guo
Yuxuan Mu
Muhammad Gohar Javed
Xinxin Zuo
Juwei Lu
Hai Jiang
Li cheng
Venue: ICLR 2025 Poster (2025)
Keywords: motion synthesis
generative masked modelling
vector quantization
single instance learning
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We present MotionDreamer, a localized masked modeling paradigm designed to learn motion internal patterns from a given motion with arbitrary topology and duration.

Abstract: Generative masked transformer have demonstrated remarkable success across various content generation tasks, primarily due to their ability to effectively model large-scale dataset distributions with high consistency. However, in the animation domain, large datasets are not always available. Applying generative masked modeling to generate diverse instances from a single MoCap reference may lead to overfitting, a challenge that remains unexplored. In this work, we present MotionDreamer, a localized masked modeling paradigm designed to learn motion internal patterns from a given motion with arbitrary topology and duration. By embedding the given motion into quantized tokens with a novel distribution regularization method, MotionDreamer constructs a robust and informative codebook for local motion patterns. Moreover, a sliding window local attention is introduced in our masked transformer, enabling the generation of natural yet diverse animations that closely resemble the reference motion patterns. As demonstrated through comprehensive experiments, MotionDreamer outperforms the state-of-the-art methods that are typically GAN or Diffusion-based in both faithfulness and diversity. Thanks to the consistency and robustness of quantization-based approach, MotionDreamer can also effectively perform downstream tasks such as temporal motion editing, crowd motion synthesis, and beat-aligned dance generation, all using a single reference motion. Our implementation, learned models and results are to be made publicly available upon paper acceptance.

Published as a conference paper at ICLR 2025
MOTIONDREAMER: ONE-TO-MANY MOTION SYNTHE- SIS WITH LOCALIZED GENERATIVE MASKED TRANS- FORMER
Yilin Wang', Chuan Guo', Yuxuan Mu', Muhammad Gohar Javed', Xinxin Zuo",
Juwei Lut, Hai Jiang', Li Cheng!
University of Alberta 2Concordia University
Simon Fraser University
'Noah's Ark Lab, Huawei Canada
ABSTRACT
Generative masked transformers have demonstrated remarkable success across various content generation tasks, primarily due to their ability to effectively model large-scale dataset distributions with high consistency. However, in the anima- tion domain, large datasets are not always available. Applying generative masked modeling to generate diverse instances from a single MoCap reference may lead to overfitting, challenge that remains unexplored. In this work, we present Motion- Dreamer, a localized masked modeling paradigm designed to learn internal motion patterns from given motion with arbitrary topology and duration. By embedding the given motion into quantized tokens with a novel distribution regularization method, MotionDreamer constructs a robust and informative codebook for local motion patterns. Moreover, a sliding window local attention is introduced in our masked transformer, enabling the generation of natural yet diverse animations that closcly resemble the reference motion patterns. As demonstrated through com- prehensive experiments, MotionDreammrr outperforms the state-of-thee-rt meth- ods that are typically GAN or Diffusion-based in both faithfulness and diversity. Thanks the consistency and robustness of the quantization based approach, Mo- tionDreamer can also effectively perform downstream tasks such as temporal mo- tion editing, crowd animation, and beat-aligned dance generation, all using sin- gle reference motion. Visit project page: https//mmotoondrrueergggubbbbbbu
INTRODUCTION
Motions could be roughly interpreted as coherent and natural compositions of finite internal patterns. For example, a breaking can be performed by freestyle composition of breaking dance skills, such as baby freeze, helicopter, kick up, buck spin, etc. Learning these internal patterns from a single reference motion allows for the generation of diverse yet consistent motions that closely resemble the reference. This is particularly useful when data is scarce (e.g.. in the case of animal motion) or when the content needs to be constrained. Existing works (Li et al.] [202aa] Raab et al.] [024]
Li et al. 2023b) have attempted to address this using GAN (Goodfellow et al.] 2014 Denton al. 2015 Zhang et al. 2017) or Diffusion Model (Ho et al.) 2020) Rombach et al. 2022 Tevet et al. 2023). where the distribution of internal patterns is modeled implicitly, or through motion matching.
which matches and blends exemplar internal patterns without learning latent embedding. However, these approaches either struggle with limited expressiveness of internal patterns fail achieve diverse and natural synthesis.
Recent advances in motion synthesis have witnessed great success in the use of generative masked transformers (Zhang et al.) [223a] Jing et al. 2023] Guo et al. 2024 Pinyoanuntapong et al. 2024], largely due to their efficient embeddings and explicit distribution modeling. These learned models quantize motion into tokens and represent them with an explicit categorical distribution in the discrete space defined by the codebook. is done by randomly masking and predicting the specific masked tokens in context. However, they usually rely on large-scale datasets, applying global self-attention to learn the in-context distribution. When adapted to single or few sequences, these models tend to overfit to sequence wise global patterss instead of generating novel sequences based on the distribution of internal patterns. This results in model collapses, since the standard transformer layers attend all the tokens in sequence for positional encoding and self-attention.

---

Published as a conference paper at ICLR 2025
One Reference Motion
Many Synthesized Motions
Figure 1: Overview of the one-to-many motion synthesis. A single reference motion with arbitrary skeletons can be applied to generate natural and diverse novel motions while preserving the reference local motion patterns. Above shows the diverse generations from MotionDreammer of a girl doing breakdance (upper); a jaguar attacking (bottom).
In this paper, propose MotionDreamer, a generative masked transformer specifically designed for one-to-many motion synthesis, as presented Figure The key idea is to learn the explicit categor- ical distribution of the internal patterns by strategically narrowing the receptive field of transformer layers, as illustrated in Figure This achieved by our proposed localized generative masked trans- former (Local-M), which incorporates carefully designed sliding window local attention layer (Sli- dAttt) as the backbone. This structure focuses on capturing local dependencies of quantized motion tokens. The SlidAttn layer divides the input motion token sequence into overlapping local windows, where motion tokens are attended to using window-wise relative positional encoding (Shaw et al.] 2018). The resulting local attention scores then aggregated through overlap attention fusion, to better preserve cross-window coherence. During training, a quantized codebook is first optimized to embed internal patterns of a single reference motion. To prevent the under-utilization of code entries, a codebook distribution regularization technique is further introduced. Once the reference motion is represented with motion tokens from the codebook, Local-M is trained model the explicit cate- gorical distribution of internal patterns through generative masked modeling, where varying masked portions of motion tokens are progressively predicted. differentiable dequantization strategy is also integrated to facilitate the optimization in motion space, by applying the sparsemax activation function (artins & Astudillo 2016) instead the traditional argmax operation.
To summarize our main contributions: (1) We introduce MotionDreamerr novel localized gen- crative masked modeling paradigm for motion synthesis based on single reference motion. With just training motion sequence, it effectively addresses the key issues: a) codebook collapse, by promoting a highly utilized codebook through a distribution regularization technique, and b) overfitting, by shrinking the receptive field of transformer using the proposed sliding window local attention layer as the backbone. (2) Our Motion faithfully preserves the internal patterns of reference motion while notably diversifying the synthesized motions. It achieves state-of-the-an comprehensive performance improving the harmonic mean by 19%, and earning the highest score in perceptual assessments. (3) MotionDreamer is also shown work well in downstream applications of temporal editing, crowd animation and beat-aligned dance synthesis, by leveraging only a single reference motion. Our project page includes visualization demos and implementation codes.
2
RELATED WORKS
2.1 SINGLE INSTANCE SYNTHESIS
Single reference based motion synthesis presents as the extension single instance learning in the image domain (Rott Shaham al. 2019 Zhang et al. [2021] Hinz et al. 2021 Granot et al.) 2022) to motion domain. In image synthesis field, Rott Shaham et al. (2019) generates diverse yet remi- niscent results from single image based generative adversarial learning through the hierarchical
2

---

Published as a conference paper at ICLR 2025
image pyramid framework. On the other hand, Granot et al. (2022) argues that utilizing patch- nearest neighbor matching results in a more faithfll and robust synthesis based on single image. In the motion synthesis domain, Ganimator (Li et al. 2022a adapts the hierarchical GAN architec- ture from Rott Shaham et al. (2019) to learn and generate diverse motion sequences from a single motion instance. SinMDM (Raab et al.; 2024) approaches the hierarchical generation process by the diffusion model based on a light-weight UNet structure with local attention layers (Arar et al., 2022). Both methods model the internal patterns of the single motion implicitly in continuous latent space, which potentially induces more out-of-diste of-distribution during synthesis, leading to limited capability of representing internal patterns and unnatural artifacts in generated motions. Inspired by Granot et al. (2022). GenMM (Li et al.) 2023b) generates novel motions through multi-level match- ing and blending the reference motion patches simply based on nearest neighbour search. However, the diversity of GenMM (Li et al. 2023b) is limited as it basically shuffles and merges original motion patches from reference motion without learning the underlying distribution, and artifacts are more frequently observed in short and highly dynamic sequences. In this work, we propose novel localized generative masked transformer for this task, which presents an effective learning paradigm with strong expressiveness of local patterns as well as natural and diverse generations. We provide more insights about the mechanism in Section 3 and potentials in empirical results in Section
2.2 MOTION GENERATIVE TRANSFORMERS
Inspired by the impressive success of generative transformers in image synthesis (Ramesh et al.
2021 Chang et al., 2022, Li et al. 2023a; Yu et al.) 2023), several recent works (Zhang et al.,
2023a Jiang et al.j 2023; Lu et al. 2024| Guo et al., (2024] Pinyoanuntapong et al.] (2024) adapt the
approach text-driven human motion synthesis. [Zhang et al. (202aa) establishes a simple CNN-based VQ-VAE (van den Oord et al., 2017) for transforming hu- motion sequences into discrete motion token sequences, and a modified generative pretrained transformer(GPT) backbone is utilized for learning to generate novel human motions conditioned on text prompt. Jiang et al. (2023) learns mixed motion tokens and text tokens simultancously utilizing language model backbone Lu et al. 20244 leverages hierarchical VQ-VAE Razavi et al. (2019) and a hierarchical GPT for more precise whole-body motion generation. Guo al. (2024) and PPnyyoa nuntapong et al.] (2024) both adapt generative masked modeling to optimizing the transformer for enhanced semantics mapping between text and motion, where a scheduled varying portion of tokens masked for predictions during training iterations. Inspired by the effectiveness of explicit dis- tribution modeling for motion sequences shown in these works, our method employs the generative masked transformer for MotionDreamer and specifically adapts the framework to single instance based motion synthesis by shrinking the receptive field of transformer layers. With our proposed Dreaon we demonstrate strong capability of capturing fine-graine grained local motion features, and diversifying their combinations with plausible transitions for novel motions synthesis.
generative transformer framework
3 LOCALIZED MOTION GENERATIVE TRANSFORMER
Given a single reference motion mi.L of length L and arbitrary skeleton topology, the goal is to generate a generally novel motion sequence m. of arbitrary length L, that preserves the skeleton structure and underlying local patterns of the reference motion. As illustrated in Fig (a) and (b), we establish a localized generative masked transformer to model the explicit categorical distribution learned discrete latent space C. (namely codebook), from which sample
of internal patterns a tuple of embeddings CI.N, (namely motion tokens) in an auto-regressive manner to generate novel motion sequences. Our method incorporates two core components: codebook distribution regular- ization and Sliding Window Local Attention (SlidAttt). The codebook regularization mitigates the codebook collapse when training a single sequence, promoting more uniform and diverse use of code entries during tokenization, while SlidAttn ensures effective modeling of local transitions by operating tokens within overlapping windows.
3.1 SINGLE MOTION TOKENIZATION
The tokenization process encodes and maps the local motion segments in the original motion space M into motion tokens in a quantized discrete latent space C. During training, a finite codebook is preset to represent the motion token space: C = (ct) C Rd. where K is the number of
3

---

Published as a conference paper at ICLR 2025
Single
syothestood Hothes
Local-M Transformer
( Of Motieereer
D0) Expliit disribation moddlling for pnrenall
Figure 2: (a) Overview of MotionDreamee based localized generative masked transformer. The single reference motion mı:z is embedded as motion tokens c by optimizing a codebook through vector quantization, where a codebook distribution regularization loss Luken additionally intro- duced. The Local-M transformer learns the local dependencies of motion tokens through sliding window local attention (SlidAttn) layers. The SlidAttn layer attends tokens within each unfolded overlapping window for attention based on learnable query and relative positional embeddings. At- tention outputs are merged through overlap attention fusion (AttnFuse).
(b) Visualization of the explicit distribution modeling for internal patterns. MotionDreamer learns to express and diversify the combination of internal patterns with explicit categorical distribution of motion tokens, which is visualized multiple token candidates predicted by Local-M given previous generated ones.
code entries. The reference motion mI.L is encoded as a sequence of feature vectors through 1D convolutional encoder
i {21
NN} E(m).)
(1)
where = 4. and is the down-sampling factor of the encoder. Through vector quantization (VQ) (van den Oord et al. 2017), noted as Q(-), each of the feature vectors mapped the nearest code entry in codebook C, resulting in a sequence of motion tokens:
C1.N fci, C2
cN3 Q(ZI::C).
(2)
Finally, the motion token sequence is projected back to original motion space m through 1D convo- lutional decoder as the reconstructed motion sequence:
m = D(CiN).
3.1.1 CODEBOOK DISTRIBUTION REGULARIZATION
Due to the constraint of highly limited data and the imbalanced temporal distribution of internal pat- terns, training vector quantizer on single motion sequence is more likely struggle with issues such as codebook collapse. Although common strategies such as exponential moving average (EMA) and codebook reset (Razavi et al. 2019 Zhang et al., 2023a Guo et al., 2024 result in good construction at the training phase, we spot that the motion patterns are still often lost or blurred with unnatural transition between poses at the generation phase, shown in Figure This could be ascribed to the under-utilization of code entries, which may introduce invalid tokens that disturb the expression of learned patterns. order mitigate this, we propose minimize the KI diver- gence between predicted token distribution and pre-assumed prior token distribution (Zhang al.) 2023b) to encourage more uniform distribution the effective code entries:
K
log( 1/K
Lioken =
(L( prast P pasts Pprior)
Σ 1=1
(4)
Pk
where Pk is the posterior distribution of code entries approximated by the average of all the quantized one-hot vectors indicating the selected code entries. The overall optimization objective for training the single motion quantizer is then established
Lvo
Lree + BgG. + Lroken
(5)
4

---

Published as a conference paper at ICLR 2025
where Liee || ml11 1 is the motion reconstruction loss, and Cg = E the standard VQ commitment loss (sg(-) denotes the stop gradient operation) van den Oord et al. 2017). Broader impacts of Lunken on other VQ-based motion representation methods are discussed in Appendix
sل(fa))2 - is
3.2 LOCAL-MM TRANSFORMER
Local-M Transformer Pa models the explicit distribution of internal patterns through progressively predicting randomly masked motion tokens. Specifically, varying fraction of motion tokens C1:N are masked out iteration timestep. replaced with a special [MASK] token. The masking ratio is obtained through cosine scheduling function Ymm (1)) cos
[0, 1], where μ. ~ 2/(0, 1) is randomly sampled, following Chang et al. (2022). Our transformer p& is optimized by minimizing the negative log-likelihood of the target predictions:
mmssk
Σ
- llog Pol,""c."
[MASK]
where et is the token sequence after masking. To facilitate the optimization by directly minimizing the motion reconstruction loss in addition to the masked modeling loss, we integrate a differentiable dequantization strategy to enable gradient flow from decoded motion to Local-M transformer. By incorporating sparsemax(-) activation function (Martins & Astudillo, 2016), the module simulates the argmax selection of motion tokens in differentiable manner.
The overall loss function of Local-M transformer is:
LM = Lmass + Arec Lree
where Are the hyper-parametts that balance the reconstruction loss with mask modeling loss. free = |lm -mlli is computed where m is the motion decoded from the predicted token sequence.
3.2.1 SLIDING WINDOW LOCAL ATTENTION
In our problem setting, naively optimizing standard transformer blocks with Lmask leads to severe overfitting issue as the model fails to capture local dependencies of motion tokens given a single sequence. To address this problem, we introduce the SlidAttn layer. Each SlidAttn layer unfolds the motion token sequence into overlapping local windows, and computes attention within each local window, the output of which is aggregated through overlap attention fusion (AttnFuse). With SlidAttm, Pe approximates the joint distribution of motion tokens by modeling the local transitions in sliding windows.
SlidAttn Mechanism By unfolding the input motion token sequence C1.N with stride S into over- lapping local windows of size 2W + the key matrices K,- W w and value matrices Vi-Wat+ W in each window can be obtained. where = {W, W + S, W + 2S, W + BS}. Note that in our case, having the windows overlapped yet keeping a small overlap length erucial for learning smooth transitions across windows on top of local patterns within windows while preserving typi- cal internal patterns. In this paradigm, utilizing absolute positional encoding (Vaswani et al.) (017)) within local windows may induce severe boundary artifacts especially with only single sequence. To mitigate the issue, we employ window-wise relative positional encoding from Shaw et al. (2018) which ensures the attention mechanism to focus the relative distances instead of absolute positions. Furthermore, we introduce learnable query qi which allows the model to adaptively attend to important tokens across different windows. This encourages the model to better capture the fine-grained transitions and internal patterns that vary between different types of attending tokens within each local window
RelPos( (W adapted
motions. The attention computation shown in Figure 2
can be formulated as:
Attn; = softmax( q: Kw r VVW
Vily
where Kw and Vw are the abbreviation for K,- and V.- V.-wa+t: respectively.
AttnFuse Standard average pooling Vasswani et al. 2017 Beltagy et al. 2020) across the overlap- ping windows may require large amount of padding for the input token sequence in order to ensure consistent input-output sequence length, inducing indispensable artifacts. To address this, AttnFuse
5

---

Published as a conference paper at ICLR 2025
aligns the attention output Attn, of each window same the way the input sequence is unfolded with trivial padding (with padding size I window size < L/h), and blends the overlapped regions with average voting while straightly passing through the others, resulting in the final aggregated output. This not only mitigates the padding artifacts that noise the distribution of internal patterns, but also allows for preserving better coherence for generating smooth motions.
3.2.2 INFERENCE PROCESS
At the inference stage, we synthesize the novel motion m. of arbitrary length Lg using Local- M in a sliding window based auto-regressive manner, by progressively filling in a fully-masked template token sequence. The details of the inference process are elaborated in Appendix A.3
4 EXPERIMENT AND RESULTS
4.1 IMPLEMENTATION DETAILS
Dataset We collect 30 motion sequences from Míxamo (Mixamo 2023) with human-like skele- ton and 30 motion sequences with skeletons of animal and artist-crafted creatures from Truebone- ZOO (tudio 2023) dataset to form the SinMotion dataset used for evaluation. There are 30 long sequences (> 600 frames) and 30 short sequences (s 600 frames).
Evaluation Metrics We apply sets of metrics to measure the expressiveness of local motion patterns and synthesis diversity respectively following Li et al.] (2022a) and |Rabb et al. (2024): (1) Coverage(%) measures the percentage of local motion patterns presented in generated motions, which indicates the faithfulness or expressiveness of the internal patterns. (2) Global Diversity quantifies the overall variation in the generated motion sequenees based on patched nearest neighbor (Li et al. 2022a). It assesses the ability of the synthesis method to produce wide range of distinct motions that differ significantly from each other. (3) Local Diver- sity evaluates local frames diversity. (4) inter diversity measures the diversity between synthesized motions by averaging per-frame distances. (5) intra diversity diff measures the local pattern dis- tribution similarity between the reference motion and generated motions, where the statistic of the pattern distribution is approximated by internal sub-window distances. Computation details for (1), (2). (3) are presented in Appendix A.5
reference motions that are
As the evaluation results are comprehensively assessed by lots of metrics for different aspects, we also introduced Harmonic Mean following Raab et al. (2024) to combine the effects of all the met- rics. The Harmonic Mean is established as early as in van Rijsbergen (179)); Chinchor (192),, pro- viding more robust comparison in cases where metric values vary large range, and has been widely applied in many machine learning fields (Powers) 2011 Taha & Hanbury 2015 Chicco & Jurman 2020). It is formulated as HE- = H/(
zf), where H is the number of assessed metrics and x; the standardized score of the i-th metric. Given the nature of this task, we give the highest weight to metric (1) and lower weights to (2)-(5).
For implementation details of the model architecture, parameter settings and the parameter tuning ablations, please refer to Appendix A.2 and Appendix A.6
4.2 COMPARISON WITH STATE-OF-THE-AN METHODS
We compare our method with all the existing approaches for single reference motion synthesis on the collected SinMotion dataset. The compared framework includes GAN (Li et al. (022a). diffusion model (Raab et al., 2024) and non-parametric optimization method (Li et al. 2023b).
Quantitative Comparison For each reference motion in the SinMotion dataset, we randomly generate 20 samples with L, = L for measuring the metrics. Table presents the quantitative results of our method compared to the state-of-the-art methods. Our method achieves state-of-the- art results individual metrics (1)-(5), and the best results in the comprehensive metrics Harmonic Mean. GenMM (Li et al. 2023b) reaches higher coverage but with limited diversity. SinMDM (Raab et al.) 2024) presents good diversity and competitive overall results, while perceptual quality is traded off as typical artifacts can be frequently observed in the generated motions, illustrated in Figure 3 and video demos in supplementary materials.

---

Published as a conference paper at ICLR 2025
Table I: Quantitative comparison with state-of-the-ar methods on SinMotion dataset. Bold marks the best result, and underline notes the second best.
Method
Coverage (%)
Local
Intra Div.
Harmonic
Ganimator et 2022a
91.27
0.97
0.90
0.20
0.34
0.26
Brera
95.29
SinMDM
Raab et 2024
91.82
1.31
1.20
0.22
0.32
0.36
93.47
Reference
SinMDM
GenMM
Ours
time
Figure 3: Qualitative comparison "hiphop dance" sample from Mixamo. Pattern A and B refer to two difficult patterns presented the reference motion. Patterns that show up in generated motions framed out marked as either success or failure according to its quality.
Qualitative Comparison We select the top 2 difficult yet typical patterns in the "hiphop dance" sample by integrating opinions of dancers, and compare the visualization results of generated mo- tions from our method, SinMDM et al. 2024) and GenMM (Li et al. 2023b) by constraining Lg Figure 3 As visualized, the generation of both SinMDM (Raab et al. 2024) and GenMM al. 2023b) tend to lose track of pattern A, which is speedy and complex sub ----partt of person squatting down while spinning. GenMM (Li et al.) 2023b) fails to present pattern A in the generated sample, while SinMDM (Raab al. 2024) generates it with unnatural transition of sudden squatting down pose. The generation of our method succeeds to express pattern A and B as well as other local patterns, while getting generally distinct from the reference motion. It worth mention- ing that GenMM achieves higher quantitative results on coverage while the qualitative result reveals its lower fidelity in presenting highly dynamic and complex patterns compared to our method. More generated samples and diversity visualization can be found on our website linked in Appendix A.]
User Study To further assess the perceptual performance, we conduct user study among 20 participants. For each of the 3 reference motions, we present 3 randomly generated motions of each method. The participants are asked to assess each generated motion by (1) Coverage: it covers all the noticeable patterns in the reference motion; (2) Diversity: it is generally novel and distinguishable from the reference motion; (3) Naturalness: it is plausible and natural with smooth transition. The assessment is based 5 levels of scores from "I strongly disagree" to "5 strongly agree" with each of the above statements. Figure 4 presents the score distribution and averaged score of each method in each assessed aspect. As shown in Figure 4 method is scored highest in coverage and diversity, and closely comparable with GenMM (Li et al., 2023b) naturalness. Noted that GenMM (Li et al. 2023b) is non-parametric method that directly matches and blends the

---

Published as a conference paper at ICLR 2025
Sanrirry 111
Figure 4: Score distribution and average score results from user study. The score level ranges from of assessing Coverage, Diversity and Naturalness. The bars align with the right y-axis referring to percentage of votes of each method in each score level, and the horizontal lines align with left y-axis labeling the average score of each method.
Table Ablation study on VQ regulatization strategies on SinMotion dataset. Bold marks the best result.
Method
VQ Perplexity
Coverage (%)
Global Div. f
Local Div.
Inter Div, 1
Intra Div.
Harmonic Mean
Ours wio,, Casen
87.26
1,14
0.26
Ours
28.13
93.47
1,33
1,17
0.25
0.28
0.43
motion patches from the reference motion without extracting latent representation, which naturally attains good coverage and naturalness in most of the cases. Our method involves generating the motion token sequence in a discrete latent space and mapping back to the motion space, yet still reaches competitive perceptual naturalness GenMM (Li greatly surpassed SinMDM (Raab et al.) 2024) and GenMM (Li et al. (023b) in the user perceptual evaluation. It is worth mentioning that while our method gets close diversity measures compared to SinMDM (Raab et al. 2024) quantitatively, our superior perceptual performance is demonstrated in diversity as well other assessed aspects.
2023b). Moreover, our method has
4.3 ABLATION: CODEBOOK DISTRIBUTION REGULARIZATION
Mitigating the under-utilization of code entries during the single motion tokenization phase plays crucial role in preserving the local motion patterns. In this work, we introduce codebook dis- tribution regularization loss Likken based on KI. divergence in addition to the commonly used strategies (EMA and codebook reset) for this purpose. We run the training of single motion tokenization without Lioken and compare the results with our method quantitatively in Table 2 We introduce VQ perplexity to quantify the utilization of code entries, which is formulated as
VQ Perplexity = exp( Σk= Pt. log(ps)). As shown, the VQ perplexity and coverage drop dra- matically without Loken- To further demonstrate it perceptually, itative comparison of the expressiveness of local patterns. We select a local window from sample "house dancing". and retrieve the nearest neighbour local window in the generated motions from our method and ours without Lioken- The character in Figure colored according to the per-frame similarity score between the generated window and the reference window, with color closer to green representing higher similarity while color closer to orange referring to lower similarity. As visual- ized, training with Lioken contributes faithful expression the reference local motion patterns. Otherwise, the motion patterns get blurred with redundant poses and transitions.
visualize in Figure the qual-
Table 3: Ablation study on architecture of Local-M transformer on SinMotion dataset. Bold text marks the best result, and underline notes the second best.
SlidAttn Diff. dequant Coverage (%)
Div.
Local Div.
Inter Div.
Intra Div. J
Harmonic
98.82
0.03
0.04
0.02
0.08
0.07
99,26
0.01
0.01
0.03
0.03
90.42
1.38
1.21
0.20
0.30
0.35
93.47
1.33
1.17
0.25
0.28
0.43
8

---

Published as a conference paper at ICLR 2025
Reference
Ours
Ours w/o. Cuske
time
Figure 5: Ablation study codebook distribution regularization technique based optimizing Luken- Color closer to green representing higher per-fram similarity while color closer to orange referring to lower similarity.
Reference
Ours
Ours w/o. AttnFuse
time
Figure 6: Ablation study overlap attention fusion (AttnFuse). "Ours w/o. AttnFuse" refers to applying standard average pooling aggregation as the alternative baseline to AttnFuse. "backflip", "handstand" pattern and transition between two patterns marked. For generated motions, color closer to the pattern colors indicates higher per-frame similarity with the corresponding pattern, while color closer orange indicates lower similarity.
4.4 ABLATION: LOCAL-M TRANSFORMER
We conduct an ablation study the architecture of Local-M transformer in Table 3 The baseline backboned on standard transformer block optimized solely by generative masked modeling loss Lmask. As presented in Table 3 the standard transformer backbone fails to learn the local motion token transitions and instead overfits to the single reference motion. By replacing the standard trans- former blocks with SlidAttn blocks, the Local-M transformer is capable of generating diverse novel motions, yet with limited expressiveness of internal patterns. The coverage increases with differen- tiable dequantization enabling stronger constraints directly in motion space, which encourages precise representation motions. Integrating differentiable dequantization with SlidAttn during training results in the best overall performance according to the Harmonic Mean.
Overlap Attention Fusion Effectiveness The mechanism of SlidAttn is highlighted with the overlap attention fusion as an alternative the average pooling aggregation of standard local tention layers. In Figure (6, we particularly look into the effects of overlap attention fusion from qualitative results by comparing "Ours w/o. AttnFuse". We highlight the signature pattern "backilip", "handstand" and transitions between this two patterns. the generated motions of the two compared methods, the closer the color is to the pattern color, the higher per-fran frame similarity to the corresponding pattern is indicated, while color closer to orange indicates lower similarity to either pattern. We compare generated motions of similar motion tokens from the two methods they share the same VQ codebook. As presented in Figure 6 our method with overlap attention fusion achieves faithful expression of the patterns well seamless globally distinct re-arrangement of the two patterns by generating natu- ral transitions. Meanwhile, the baseline with average pooling aggregation fails to globally diversify the patterns with unnatural transitions and noisy motion patterns.
the standard average pooling aggregation which denoted as

---

Published as a conference paper at ICLR 2025
(8) Crowd Motion Synthesie
(c) Beat-aligned dance synthesis
Music
(b) Temporal Editing
iudport
Application visualization. (a) crowd animation: crowd doing warm-up. (b) Temporal Editing: floorcombo with the ending part fixed, different internal patterns generated. (c) Beat- aligned Dance Synthesis: the green boxes marks the keyframes align with the music beats.
Figure 7:
4.5
APPLICATION
Crowd Animation The unconditional generation of the one-to-many motion synthesis can effi- ciently provide a diverse characters with similar structure performing the same internal pat- terns in different ways, as shown in Figure
Temporal Editing The goal of temporal editing is to edit the reference motion by re-generating an arbitrary subpart with the learned local patterns, which can be realized through masking only the template tokens within the assigned editing subpart. As shown in Figure (b)) MotionDreamer can reproduce novel samples by fixing the end part of reference motion "floorcombo".
Beat-aligned Dance Synthesis Another interesting application we explore a variation from the music-to-danee tasks (Li et al. [021]] 2022b Tseng et al. 2023), the beat-aligned dance synthesis based on single instance, where the generated motions are ensured to preserve the keyframe poses on the music beats provided with the reference motion. To accommodate this task, we incorporate the beat features using librosa (McFee et al.) 2015) as an auxiliary temporal-aligned feature for the motion tokens employing additional pair of light-weight encoder-decoder. As visualized in Figure c). the synthesized dance clip can well align with the typical poses on the beats yet well diversifying them, and the dance patterns are also internally reminiscent yet generally diverse.
For implementation details please refer to Appendix A.4 and for video demos go to Appendix A.
CONCLUSION
In this work, we introduce MotionDreamerr novel one-to-many motion synthesis framework based on localized generative masked modeling. Our method leverages codebook distribution regulariza- tion to address the codebook collapse and under-utilization, ensuring a diverse representation of motion patterns even with single motion. Furthermore, we propose sliding window local atten- tion Atl mechanism that effectively captures local dependencies and smooth transitions across overlapping windows, significantly improving both the fidelity and diversity of generated motions. Through comprehensive experiments, we demonstrate that MotionDreamerr achieves state-of-the-an art performance, outperforming existing approaches in generating natural and diverse motions from single reference, and shows strong potential in practical applications such as temporal editing and beat-aligned dance synthesis. More future directions discussed in Appendix A.8
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Moab Arar, Ariel Shamir, and Amit H. Bermano. Learned queries for efficient local attention. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022.
Iz Beltagy, Matthew Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv:2004.05150, 2020.
Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Shar- ifi, Olivier Teboul, David Grangier, Marco Tagliasacchi, and Neil Zeghidour. Audiolm: lan- guage modeling approach to audio generation. In Proceedings of the 11th International Confer- ence on Learning Representations (ICLR), 2023.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman. Maskgit: Masked gener- ative image transformer. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2022.
Davide Chicco and Giovanni Jurman. The advantages of the matthews correlation coefficient (mcc) over fl score and accuracy in binary classification evaluation. BMC Genomics, 21(1):1-13, 2020.
Nancy Chinchor. Muc-4 evaluation metries. In Proceedings of the 4th Conference on Message Understanding, pp. 22-29. Association for Computational Linguistics, 1992.
Emily L Denton, Soumith Chintala, Arthur Szlam, and Rob Fergus. Deep generative image mod- els using a laplacian pyramid of adversarial networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems- Volume 2, pp. 1486-1494, 2015.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- 2014.
mation processing systems, pp.
Niv Granot, Ben Feinstein, Assaf Shocher, Shai Bagon, and Michal Irani. Drop the gan: In defense of patches nearest neighbors as single image generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13450-13459, June 2022.
Chuan Guo, Yuxuan Mu, Muhammad Gohar Javed, Sen Wang, and Li Cheng. Momask: Gener- ative masked modeling of 3d human motions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1900-1910, 2024.
Tobias Hinz, Matthew Fisher, Oliver Wang, and Stefan Wermter. Improved techniques for train- ing single-image gans. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 1300-1309, January 2021.
Jonathan Ho, Ajay Jain, and Picter Abbecl. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pp. 6840-6851, 2020.
Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. Motiongpt: Human motion as a foreign language. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Peizhuo Li, Kfir Aberman, Zihan Zhang, Rana Hanocka, and Olga Sorkine-Hornung, Ganimator: Neural motion synthesis from a single sequence. ACM Transactions on Graphics (TOG), 41()): 138, 2022a.
Siyao Li, Weijiang Yu, Tianpei Gu, Chunze Lin, Quan Wang, Chen Qian, Chen Change Loy, and Ziwei Liu. Bailando: 3d dance generation by actor-critic gpt with choreographic memory. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022b.
pp. 1281 1-12821,
Tianhong Li. Huiwen Chang, Shlok Mishra, Han Zhang, Dina Katabi, and Dilip Krishnan. Mage: Masked generative encoder to unify representation learning and image synthesis. P Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2142 2152, June 2023a. URL htt.hhtts:///// http::/:////pccccssss// theevf. com/ www/contentttcteeeeeoo wwntentttttt/cccco CVVR2023//
html/Ii MAGE MAsked Generative Bneodiiiieeeeeereeeeeeeee Encoder To _o___nify Representation
Learning maniing_aa and Twwaemmaaaaaeeeeeeeeeeeeeeee CVPR 2022022__eep_______2 html
11

---

Published as a conference paper at ICLR 2025
Weiyu Li, Xuelin Chen, Peizhuo Li, Olga Sorkine- Sorkine-Hornung, and Baoquan Chen. Example-based motion synthesis via generative motion matching. ACM Transactions on Graphics (TOG), 42(4), 2023b. doi: 10.1145/3592395
Yuming Li, Hang Zhou, Yinghao Huang, Xiaoyu Ma, Weiyu Xu, Yasutaka Mukai, Liang Wang, and Ziwei Li. Aist++: A large-scale dance motion dataset for music conditioned 3d dance genera- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2126-2136, 2021.
Shunlin Lu, Ling-Hao Chen, Ailing Zeng. Jing Lin, Ruimao Zhang, Lei Zhang, and Heung- Yeung Shum. Humantomato: Text-aligned whole-body motion generation. In Pruceedings of the Inter- national Conference on Machine Learning (ICML), 2024.
Julieta Martinez, Holger H Hoos, and James J Little. Stacked quantizers for compositional vector compression. In Proceedings of the 31st International Conference on Machine Learning (ICML), pp. 675-683, 2014.
André FT Martins and Ramón Fernandez Astudillo. From softmax to sparsemax: A sparse model of attention and multi-label classification. In Proceedings of the 33rd International Conference on International Conference on Machine Learning (ICML), volume 48, pp. 1614-1623, 2016.
Brian McFee, Colin Raffel, Dawen Liang, Daniel P.W. Ellis, Matt Mc Eric Battenberg, and Oriol Nieto. librosa: Audio and music signal analysis in python, 2015. URL https:
//librosa org/ Version 0.5.0.
Mixamo, Mixamo 3d characters and animations. https: http://www.iiixxm www.mo com 2023, Accessed:
2023-09-30.
Ekkasit Pinyoanuntapong. Pu Wang, Minwoo Lee, and Chen Chen. Mmm: Generative masked motion model. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
David M W Powers. Evaluation: From precision, recall and f-measure roc, informedness, marked- ness and correlation. Journal of Machine Learning Technologies, 2(1):37-63, 2011.
Sigal Raab, Inbal Leibovitch, Guy Tevet, Moab Arar, Amit H Bermano, and Daniel Cohen-Or. Single motion diffusion. The Twelfth International Conference on Learning Representations (ICLR), 2024.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. arXiv preprint arXiv:2102.11002) 2021.
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer: High- resolution image synthesis with latent diffusion models. arXiv preprint arXiv:21 /2. /0752, 2022.
Tamar Rott Shaham, Tali Dekel, and Tomer Michaeli. Singan: Learning generative model from single natural image. In Computer Vision (ICCV), JEEE International Conference on, 2019.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representa- tions. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pp. 464-468, 2018.
Truebones Motion Studio. Truebonezoo motion capture library. https://Eruebones. com. 2023. Accessed: 2023-09-30.
A Taha and Allan Hanbury. Metries for evaluating 3d medical image segmentation: analysis, selection, and tool. BMC Medical Imaging, 15(1):29, 2015.
Guy Tevet, Sigal Raab, Brian Gordon, Yoni Shafir, Daniel Cohen-Or, and Amit Haim Bermano. Hu- man motion diffusion model. In International Conference on Learning Representations (ICLR), 2023. Oral Presentation, Top 25% Paper.
12

---

Published as a conference paper at ICLR 2025
Jonathan Tseng, Rodrigo Castellon, and C. Karen Liu. Edge: Editable dance generation from mu- sic. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition 2023. doi: 10.1099CVVR527222223.0000
(CVPR), pp.
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learn- ing. arXiv preprint arXiv:1711.00937 2017.
Comelis J. van Rijsbergen. Information Retrieval. Hnnterworth-Heinemmane 1979.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In Advances Neural Infor- mation Processing Systems (NeurIPS), pp. 5998-6008, 2017.
Ivan Volkov. Homology-constrainee vector quantization entropy regularizer. arXiv preprint arXiv:2211.14363 2022.
Yiming Xiao, Kun Shu, Hong Zhang, Ben Yin, Weng S. Cheang, Hao Wang, and Jie Gao. Eggesture: Entropy-guided vector quantized variational autoencoder for co-speech gesture generation. In Proceedings of the ACM International Conference on Multimedia (MM), 2024.
Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming-Hsuan Yang. Yuan Hao, Irfan Essa, and Lu Jiang. MAGVIT: Masked gen- erative video transformer. Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, 2023.
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Sound- stream: An end-to-end neural audio codec. In Proceedings of the 39th International Conference on Machine Learning (ICML), pp. 19999--12333 2021.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaolei Huang, Dimitris N Metaxas, and David Ognibene. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp- 5907-5915, 2017.
Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Shaoli Huang, Yong Zhang, Hongwei Zhao, Hongtao Lu, and Xi Shen. T2m-gpt: Generating human motion from textual descriptions with discrete representations. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2023a.
Yulun Zhang, Songhua Liu, Yinqiang Shao, al. Regularized vector quantization for tokenized image synthesis. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10241-10251, 2023b.
Zicheng Zhang, Congying Han, and Tiande Guo. Exsingan: Learning an explainable gen- erative model from single image. Proceedings of the British Machine Vision Con- ference (BMVC), 2021. URL https:///ww waw.eelll-vvirraalceeeerora confereecee/pppes///aaa 0348.html
A APPENDIX
A.1
VIDEO DEMOS
The video version of visualization in the paper well as demos for more generation results and application results can be found on our project page link:
https::/motioondeeaee www.eaeeaaaar........i.ii....e wit.ub. 10/
13

---

Published as a conference paper at ICLR 2025
DUst
Single Motion--liee -irett Tokenization
Bas
09)
Local M
Local M
Geraral Systhoic Other Applications
RATOTA
Tumpara Bolting
(a) Inderence pipeline illustration
Single Motion Tolieniaation
synthesis
Figure 8: (a) Inference pipeline illustration. (b) Single Motion-Beat Tokenization for beat-aligned dance synthesis.
A.2 IMPLEMENTATION DETAILS
The proposed method and architecture have been majorly illustrated in Section our paper. The involved parameter setting of the proposed architecture of MotionDreamerr is presented in Table During training, we crop the single reference motion sequence into overlapping motion patches of length Tú and $p- At the single motion tokenization phase, encoder E, codehook C and decoder D are trained with learning rate (ri. Local-M transformer is trained with learning rate (r2. We select the parameter settings for training in Table based the best empirical results. All the reference motions are trained and evaluated on a single RTX2080Ti GPU.
Table 4: Parameter settings for architectures.
Table 5: Parameter settings for training and in-
Param.
Definition
Value
ference.
down-sampling factor tokenization
8
d
latent embedking fer Fg dimension
4096
Param.
Delinition
Value
de
transformer latent embedding dimension
384
length of motion patch for training VQ, Local-M
96, 128 (frames)
Ng
number of layers in number of layers in number of layers in Local-M
3
3,
stride of motion patches for training VQ. Local-M weighting factor É.
24, 32 (frames)
Np
3
Nar
3
weighting
weightnn dusing training Local-M
Cuses
K
number code entries in C local window size of SlidAttn layer window stride SlidArti layer
48
leaming rale training tekeniration
W
(tokens) 4 (tokens)
irs
learning rase for training Local M
2e
S
A.3 INFERENCE PROCESS ELABORATION
At the inference stage, shown in Figure g we synthesize the novel motion mi:L, of arbitrary length Ly localized auto-regressive strategy based on sliding windows. We first initialized blank template token sequence eç of length Lg with all the tokens masked. A synthesis window of size W, y is preset, and slides along the template token sequence with stride Sy for auto-regressive syn- thesis. At each window-step, the learned Local-M transformer progressively unmasks all the motion tokens within the synthesis window through iterative re- re-masking (IR), where tokens generated with low confidence are re-masked for updated generation in the next iteration. The stride of the sliding synthesis window ensures overlapping tokens from the previous window-step. These overlapping tokens remain un un-masked as condition tokens for generating other tokens in the new window-step After all the window-see steps, the completely generated token sequence Cy is de-quantieed and decoded to motion space, resulting in novel motion m...; Our best empirical selection for parameters is W Tp. Sg /
14

---

Published as a conference paper at ICLR 2025
A.4 APPLICATION IMPLEMENTATION ELABORATION
Crowd Animation The crowd animation shares exactly the same architecture, training and infer- ence pipelines what describe in the major contents. What we showcase here in this application is the way to use DremeeDr as an efficient tool for generating similar yet diverse motions for a crowd of characters with the same skeleton structures. More demos are shown in our project page link in A.
Temporal Editing As presented Figure 8 a), the template token sequence for temporal editing at the inference stage, noted as ca is constructed differently compared with either general synthe- sis elaborated in Appendix A..3 or other applications discussed in this section. By providing the sub-parts to generate with corresponding timestamps, the template sequence is constructed as the combination of encoded motion tokens for the fixed sub-parts and the [MASK] tokens for the sub- parts to be generated. The motion generated from cº retains exactly the same motion for the fixed sub-parts, while presenting diverse motions based on learned internal patterns for the others.
Beat-aligned Dance Synthesis The beat-aligned dance synthesis involves incorporating auxiliary beat features into the latent motion representation, ensuring that the synthesized motion aligns with the rhythm while maintaining natural transitions and variations. To approach this, the single motion- beat tokenization is proposed where we establish the lightweight encoder( E,, -eecodrrr (D₂) network based only NE (= ND) ID-convolutional layers for embedding the beat features, shown in Figure B)) Es, maps beat features, extracted from the paired music the using librosa (McFee et al.) 2015). to the discrete latent space shared with the motion features; D' decodes the quantized beat embeddings back to the original space for beat features. This lightweight architecture facilitates tempo alignment based on single-instance while minimizing computational overhead.
A.5 EVALUATION METRICS
The computations of our applied evaluation metrics follow Li et al. (022a) and Raab et al. (2024) Coverage The coverage of the reference motion m is measured based on all possible temporal windows given length Lgg which is written as WL.- Given generated motion m, a temporal window my E Wt would be labeled as covered if the distance measure for the nearest neighbor in vn is smaller than a threshold The distance metric is chosen as Frobenius norm on the local joint rotation matrices, and the window size L.- 30, defined as:
Cov(m, m) WI.e
Σ
I[NN(m..: m) < c).
m EWW.
Global Diversity Li et al. (2022a) proposes a distance measure for patched nearest neighbors (PNN) to measure the global diversity. The generated motion is segmented with length no shorter than threshold Tm for each sub-part, and the measure is oriented for finding an optimized segmen- tation for the motion with minimum averaged per-frame nearest neighbor cost. For more details, refer the readers to [i et al. (2022a)
Local Diversity every local window of the generated motion to its nearest neighbor in the reference motion:
The local diversity metric is based the local frame diversity by comparing
Diocal
Σ
NN m).
(10)
[
EW...
A.6 MORE ABLATION STUDIES
Table 6 shows the ablation study for our settings for the architecture and the key parameters.
Local Attention Mechanism We want to highlight the second part of Table 6 where we com- pare our proposed SlidAttn with other two alternatives for local attention layer, Local SASA and QnA. Compared to Local SASA, which is the most standard local attention layer which processes non-overlapping windows without learnable queries or positional encodings, SlidAttn significantly improves coverage and diversity, demonstrating the importance of overlapping windows and en- riched attention computation. QnA applied in SinMDM (Raab al. 2024) convolutional local
15

---

Published as a conference paper at ICLR 2025
Table 6: Ablation study for parameter settings. Bold text marks the best result.
Method
Coverage (%)
Global Div.
Local Div.
Inter Div.
Intra Div.
Harmonic Mean
K
32
84.23
1.08
1,02
0.16
0.30
0.24
K 48
93.47
1.33
1.17
0.25
0.28
0.43
Local SASA
74.23
1.13
1.10
0.15
0.31
0.23
Reah 2024 Arar 2022
1.37
1,20
0.26
0.32
SIGALE
93.47
1.33
1.17
0.25
0.28
0.43
SlidArtn w/o. learnable queries
71.42
1,48
1.37
0.20
0.36
0.23
SlidAttn w/o. SlidArm
89.84
1.05
1,01
0.22
0.30
0.22
93.47
1.33
1.17
0.25
0.28
0.43
1.24
0.20
IF 5
93.47
1.33
1.17
0.25
0.28
0.44
81.79
1.41
1.20
0.23
0.29
0.28
93.47
1.33
1.17
0.25
0.28
0.43
=
94.89
1.03
0.98
0.24
0.38
0.38
Table 7: Impact of codebook regularization loss other VQ methods for motion representation. Bold text marks the best result.
Method
R Precision Top 1
Perplexity"
MMM PPiycuauuntapong el al.
2024
0.503
1642.194
MMM (Pinyuuuntaponn et al. 2024
W7 Linken
0.572
1678.538
MoMask (Gno et al.
2024
0.504
368.914
MoMask Guo el al
2024
w/ Lyoken
0,504
372.702
attention layer modified from Arar et al. (2022), which incorporates learnable queries and relative positional encoding. Compared to QnA, which relies on the convolutional layers for overlapping window operation, SlidAttn reaches better overall performance with a dramatic rise coverage measures and competitive diversity results. This demonstrates that our proposed SlidAttn uniquely employs a more effective sliding window attention computation and aggregation paradigm for pro- cessing overlapping local windows under this framework.
Sliding Window Parameters The last part is worth highlighting as it presents the ablations for stride of the sliding windows in SlidAttn of Local-M (refer Section B) which demonstrates that having a relatively small overlap length (W- which attains good diversity while preserving the internal patterns from reference motion.
- S = 1) in our case results in the best Harmonic Mean,
A.7 DISCUSSION ON CODEBOOK REGULARIZATION
We also conduct additional experiments the broader impact of our codebook regularization loss on general motion tokenization trained on large datasets. We choose MMM (Pinyoanuntapong et al.) 2024) and MoMask (Guo et al.) 2024) to experiment with, they share similar framework but with different VQ architectures. MMM (Pinyoanuntapong et al., al., 2024) tokenizes the motions with a stan- dard VQ layer with large codebook, while MoMask (Guo al. 2024 incorporates residual VQ layers (Borsos et al. 2023 Martinez et al. 2014, Zeghidour et al. 2021 with a stack of codebooks for fine-grained motion representation. We apply our codebook regularization loss Lroken the two methods for training the motion tokenization, comparing 1) R Precision Top 1 for the reconstruc- tion performance the motion representation capacity of learned codebook(s); 2) perplexity for the codebook utilization.
As the results shown in Table 7 both R precision and perplexity have been significantly improved for MMM, indicating better and more effective use of the codebook to become more effective at representing motion data. For MoMask, the impact of Ltoken also evident. While the R pre- cision remains stable at 0.504, the perplexity improves from 368.99 688.914 to 372.702. This shows that the incorporation of Ctoken helps optimize the utilization of residual VQ layers without sacrificing the motion representation capacity. These results clearly demonstrate the effectiveness of our pro- posed Cuken in enhancing the quality and utilization codebooks across different VQ architectures, making it valuable addition to motion representation frameworks.
The improvements are particularly significant for methods like MMM, where the standard VQ lay- benefit the most from the regularization. For more advanced residual VQ architecture adapted
16

---

Published as a conference paper at ICLR 2025
by MoMask, the improvements in codebook utilization moderate, with no significant gains in motion representation capacity. This may be due to the nature of residual VQ layers, which obtain higher motion representation capacity by utilizing multiple codebooks to quantize the information loss from previous layers, rather than directly representing motion data. Consequently. higher code- book utilization alone does not significantly enhance motion representation capacity in this context.
There are also other alternatives regarding the codebook regularization for VQ-based representation, which be further explored in future work. For example, entropy regularization, which encour- ages entropy maximization of the token distribution, can promote codebook utilization, as applied in Volkov (2022); Xiao al. (2024). Stochastic sampling methods can also be incorporated such as Zhang et al. (2023b) by modifying sampling strategies to implicitly regularize the token utilization distribution across lcrge-scale datasets.
A.8 LIMITATIONS
While MotionDreamer demonstrates strong performance in generating diverse and natural motion sequences from a single reference, several limitations remain. First, the framework's 's performance highly dependent the quality of the reference motion. Second, due to the inherent nature of single instance based motion synthesis, the framework exhibits limited capacity to generalize across broader range of motion editing and conditional synthesis tasks, given the nature single instance based synthesis. Future work will focus on improving the model's extrapolation capabilities and enhancing its generalization both few-shot and large-scale datasets. particular, the development of a more flexible framework capable of distilling knowledge from diverse motion priors would allow the model to accommodate wider array of motion styles and patterns. Moreover, integrating more robust attention mechanism may enable the model to capture long-range dependencies and global patterns more effectively, thus extending its applicability broader set of motion editing and synthesis tasks.
17


Reviews:
[{"review_id": "dmW0mHi3tJ", "reviewer": "Reviewer_rqV8", "summary": "The paper introduces a novel method for intrinsic generation for motion, i.e., augmenting a single input motion sequence into multiple versions. The paper employs an underexplored generative approach - the masked transformer - and presents SOTA performance. To improve diversity and overall quality, the paper also offers a simple KL-divergence based loss to encourage full usage of the latent space, to mitigate mode collapse. The paper reports a complete evaluation, including comparisons, ablation studies, quantitative evaluation, qualitative evaluation and a user study.", "questions": "As stated above, I believe that paper presents interesting ideas, and I would like to see this player out there with the other methods. \n\nMy main concern is the scope of impact. Perhaps these ideas can and should be examined in a wider scope:\n- can the codebook regularization loss improve other VQ methods off the bat? \n- More ambitiously, can this method be combined with others, for example can the masked transformer be applied in the bottle neck of a diffusion process? or perhaps instead of the VAE?\n- similarly, some more discussion regarding the regularization could benefit the paper. What other alternatives could have been employed?\n\nIn addition, I would have liked to see some more qualitative results \n\nMinor concerns:\n- Swin is already rather widely adopted and known for shifting windows (Swin-Transformer), I recommend using another name for the sliding window approach (which I find simple and elegant for reducing the receptive field)\n- \"Crowd motion synthesis\" can also be misinterpreted here, as an algorithm to avoid collisions, I would also find a different name here.\n- I find figures 1 and 2 a bit exaggerated. The right side of Fig 2 depicts little information for its size. Figure 1 is also rather overwhelming, half of the amount of characters would probably go a long way.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- Paper is well written and easy to follow.\n- Proposed method is simple and elegant\n- The mask transformer mechanism is a refreshing approach\n- Codebook distribution regularization is interesting\n- Evaluation is rigorous, including all conventional metrics", "weaknesses": "- The application is a niche one \n- The method, being simple, is also limited in contribution\n- Qualitative evaluation is limited", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wkvTFIcLBA", "reviewer": "Reviewer_L4V3", "summary": "The paper proposes Motiondreamer, a framework for one-to-many motion synthesis. The paper aims to learn the underlying motion pattern in a given sequence and leverage the motion pattern to generate natural and diverse motions.\nThe general idea of the paper is to first quantize a motion sequence into tokens, and then train a masked transformer to generate these motion tokens. \nThere are two core technical contributions,  one lies in regularising motion tokenization to encourage more coverage, and the other one lies in injecting localality into the makes transformer to encourage motion decomposition.", "questions": "Please address the weaknesses above.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The model is well-designed. Given the scarcity of motion data in this one-to-many setting, the inductive bias in the model design is particularly important. The overall framework follows a masked transformer architecture which is simple, but multiple practical techniques have been contributed to balance the motion faithfulness and the motion diversity, including\n    - A KL term is added to the quantizer training besides the standard commitment loss to encourage coverage.\n    - A sliding window attention that adds locality to the transformer to encourage diversity.\n    - A differentiable dequantization loss that helps to compute loss on the motion space.\n    \n    All of them are well-motivated and make sense to me.\n    \n- The ablation study is very comprehensive. Although the above techniques, like KL regularisation, are quite standard and empirical, the paper provides detailed experiments to support their importance to the final results. I find those experiments convincing.\n- The paper shows qualitative results on a webpage, which generally looks good to me. Compared to SinMDM, the proposed method reduces unnatural motions. Compared to SinMDM, the generated motion seems more diverse.", "weaknesses": "- Table 1 and Figure 3 are inconsistent in the comparison between GenMM with Ours. Table 1 shows that GenMM has much better coverage than Ours. However, in Figure 3, the failure mode of GenMM is missing a part of the reference motion.\n- The beat-aligned dance synthesis application does not convince me. The generated dances have a clear mismatch with the tempo. Most importantly, the application (including implementation details) is not well documented in the paper so I am not fully sure what is happening there, for example, what is a “light-weight encoder-decoder”? To my understanding, the beat-aligned dance is like an extension of the “subpart generation” task while keeping predefined keyposes on the music beats. That is not sufficient to produce a beat-aligned dance in my opinion.\n- Some of the techniques could be a little niche to be presented as part of the method. For example, AttnFuse could better fit into implementation details.\n- Minor\n    - L281: by progressively fill → by progressively filling.\n\n----\nThe overall framework makes sense to me. The proposed techniques are simple, but they are well-motivated and are supported by solid ablation experiments. I am convinced that they contribute to the overall performance. My concerns are about the comparison between GenMM and the dance generation application part. The presentation can be improved as well. I am slightly inclined to accept.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B0aZlthOEi", "reviewer": "Reviewer_oiwR", "summary": "This paper proposes a localized masked modeling paradigm designed to learn motion internal patterns from a given motion with arbitrary topology and duration. It introduces the codebook distribution regularization and sliding window local attention (SwinAttn) mechanism to avoid codebook collapse and overfitting.", "questions": "In the final section of the paper, the authors mention several other applications. Could the authors elaborate on how the network architecture performs and what advantages it offers for these tasks?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The codebook distribution regularization effectively prevents codebook collapse. The sliding window local attention mechanism in the Local-M transformer captures local dependencies and ensures smooth transitions across overlapping windows, enhancing the fidelity and diversity of generated motions.", "weaknesses": "This paper applies some recent advancements from the image domain to motion synthesis, which is a commendable attempt. However, the contribution feels somewhat limited. As a work focused on enhancing the VQVAE structure for motion synthesis, the authors might benefit from incorporating insights and experiments from NCP, rather than simply adapting the network architecture.\n# Added after AC's comment\nI'm grateful for the feedback and suggestions. To clarify, the aspects of the contribution that feel limited include the novelty in adapting image domain advancements without significant modification or integration specific to motion synthesis. Additionally, while some metrics have been provided, they may not fully reflect the method's advantages. I believe visualizations of the latent space could be helpful in demonstrating its effectiveness. Regarding NCP, I apologize for the oversight—it refers to Neural Categorical Priors for Physics-Based Character Control. Thank you for the guidance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "isVhfztHPY", "reviewer": "Reviewer_Fzyk", "summary": "This paper proposes a generative masked model (MotionDreamer) for motion synthesis domain based on the motivation \"large datasets are not always available\". MotionDreamer designs a local masked modeling paradigm to learn motion internal patterns from a given motion with a key sliding window local attention module. Experiments demonstrate various motion tasks, including crowd motion synthesis, temporal editing and beat-aligned dance synthesis.", "questions": "See Weakness.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- The paper is well-written and is easy to follow. \n\n- Qualitatively results look impressive. Ablation studies prove the effectiveness of the designs.\n\n- The authors introduce various applications of motion synthesis and achieve great performance.", "weaknesses": "- Generative masked modeling for motion synthesis has been investigated in many recent works, such as MMM, Momask, MotionGPT, as described in paper. On the other hand, single motion synthesis is also not new, such as SinMDM. It seems that MotionDreamer integrates the techniques of generative masked modeling into the task of single motion synthesis, including motion tokenization and codebook distribution regularization.\n\n- Capturing local dependencies of motion features is quite important for single motion synthesis. SinMDM introduces QnA layers that allow local attention with a temporally narrow receptive field. And MotionDreamer incorporates sliding window local attention to achieve this. Besides, another key design of codebook distribution regularization has been explored in single image synthesis domain. Therefore, I think the technical contribution of the paper is somewhat limited.\n\n-  In Ablation Studies (Sec. 4.4), it's better to use QnA layers in SinMDM to replace sliding window local attention and show the comparison of the performance. Due to the same task, the paper should demonstrate its unique effectiveness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)