Human Motion Diffusion Model
Authors: Guy Tevet
Sigal Raab
Brian Gordon
Yoni Shafir
Daniel Cohen-Or
Amit Haim Bermano
Venue: ICLR 2023 notable top 25% (2023)

Abstract: Natural and expressive human motion generation is the holy grail of computer animation.
It is a challenging task, due to the diversity of possible motion, human perceptual sensitivity to it, and the difficulty of accurately describing it. Therefore, current generative solutions are either low-quality or limited in expressiveness. 
Diffusion models are promising candidates for the human motion domain since they
have already shown remarkable generative capabilities in other domains, and their many-to-many nature. 
In this paper, we introduce Motion Diffusion Model (MDM), a carefully adapted classifier-free diffusion-based generative model for human motion data.  MDM is transformer-based, combining insights from motion generation literature. 
A notable design-choice is that it predicts the sample itself rather than the noise in each step to facilitate the use of established geometric losses on the locations and velocities of the motion, such as the foot contact loss. As we demonstrate, MDM is a generic approach, enabling different modes of conditioning, and different generation tasks. We show that our model is trained with lightweight resources and yet achieves state-of-the-art results on leading benchmarks for text-to-motion, action-to-motion, and unconditioned motion generation. 

Published as a conference paper at ICLR 2023
HUMAN MOTION DIFFUSION MODEL
Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or and Amit H. Bermano Tel Aviv University, Israel gryytevetemiil.cau.uo
ABSTRACT
Natural and expressive human motion generation is the holy grail of computer animation. challenging task, due to the diversity of possible motion, hu- man perceptual sensitivity it, and the difficulty of accurately describing it. Therefore, current generative solutions are either low-quality or limited in ex- pressiveness. Diffusion models, which have already shown remarkable gener- ative capabilities in other domains, are promising candidates for human mo- tion due to their many-to-many nature, but they tend be resource hungry and hard to control. In this paper, we introduce Motion Diffusion Model (MDM), a carefully adapted classifier-free diffusion-basee bassdd generative model for the hu- man motion domain. MDM is transformer-bused combining insights from mo- tion generation literature. A notable design-choice the prediction of the sam- ple, rather than the noise, in each diffusion step. This facilitates the use of es- tablished geometric losses on the locations and velocities of the motion, such the foot contact loss. As demonstrate, MDM is generic approach, en- abling different modes of conditioning, and different generation tasks. We show that our model is trained with lightweight resources and yet achieves state-of- the-art results on leading benchmarks for text-to-motion and action-to-motion https::/guyter: github. 10 /md madpmmmmmapa///////
INTRODUCTION
Human motion generation is a fundamental task in computer animation, with applications spanning from gaming to robotics. It is challenging field, due to several reasons, including the vast span of possible motions, and the difficulty and cost of acquiring high quality data. For the recently emerging text-to-motion setting, where motion is generated from natural language, another inherent problem is data labeling. For example, the label "kick" could refer to a soccer kick, as well as a Karate one. At the same time, given specific kick there are many ways describe it, from how it is performed to the emotions it conveys, constituting a many-to-many problem. Current approaches have shown success in the field, demonstrating plausible mapping from text to motion (Petrovich et al. 2022 Tevet et al. 2022 Ahuja & Morency, 2019). All these approaches, however, still limit the learned distribution since they mainly employ auto-encoders or VAEs (Kingma & Welling) (013) (implying one-to-one mapping or a normal latent distribution respectively). In this aspect, diffusion models are a better candidate for human motion generation, as they are free from assumptions on the target distribution, and are known for expressing well the many-to-many distribution matching problem we have described.
Diffusion models (Sohl-Dicksted) al. 2015 Song & Ermon 2020 Ho et al. 2020) are generative approach that is gaining significant attention in the computer vision and graphics community. When trained for conditioned generation, recent diffusion models (Ramesh et al. 2022 Saharia et al.; 20226) have shown breakthroughs in terms of image quality and semantics. The competence of these models have also been shown for other domains, including videos (Ho al. 2022), and 3D point clouds (Luo Hu 2021). The problem with such models, however, is that they are notoriously resource demanding and challenging to control.
In this paper, we introduce Motion Diffusson Model (MDM) - a carefully adapted diffusion based generative model for the human motion domain. Being diffusion-based, MDM gains from the na-
Code he found at htt.tp/////ttiibbbcomo///////// thwb..comm//yylleeeeeetttttot fousinnnn

---

Published as a conference paper at ICLR 2023
"A person kicks with their left leg."
"A man runs to the right then runs to the left then back to the middle."
Our Motion Diffusion Model (MDM) reflects the many-to many-to-many nature of text-to-moet to-ootoon mapping by generating diverse motions given a text prompt. Our custom architecture and geometric losses help yielding high-quality motion. Darker color indicates later frames in the sequence.
Figure 1:
tive aforementioned many-to-mant expression of the domain, as evidenced by the resulting motion quality and diversity (Figure [... In addition, MDM combines insights already well established in the motion generation domain, helping it be significantly more lightweight and controllable.
First, instead of the ubiquitous U-net (Ronneberger et al. 2015) backbone, MDM is transformer- based. As we demonstrate, our architecture (Figure is lightweight and better fits the temporal and spatially irregular nature of motion data (represented as collection of joints). ume of motion generation research is devoted to learning using geometric losses (Kocabas et al.) 2020 Harvey et al., 2020 Aberman et al., 2020). Some, for example, regulate the velocity of the motion (Petrovich et al. (2021) to prevent jitter, specifically consider foot sliding using dedicated terms (Shi et al. 2020). Consistently with these works, we show that applying geometric losses in the diffusion setting improves generation.
large vol-
The MDM framework has a generic design enabling different forms of conditioning. We showcase three tasks: text-to-motion, iition-to-motion and unconditioned generation. We train the model in a classifier-free manner (Ho & Salimans] [002)), which enables trading-off diversity to fidelity, and sampling both conditionally and unconditionally from the same model. In the text-to-motion task, our model generates coherent motions (Figure T) that achieve state-of-ther oftthc-rrttttttt results on the Hu- manML3D (Guo al. 2022a) and KIT (Plappert al. 2016) benchmarks. Moreover, our user study shows that human evaluator prefer our generated motions over real motions 42% of the time (Figure aa)).. In action-to- tosoonn-t-moiion MDM outperforms the state-of-the of-the-art (Guo al., 2020 Petrovich al,, 2021 even though they were specifically designed for this task, on the common HumanAct12 (Guo ct al. 2020) and UESTC (Ji et al. 2018) benchmarks.
Lastly, we also demonstrate completion and editing. By adapting diffusion image-inpainting (oog al. 2020b Saharia al. 2022a). we set a motion prefix and suffix, and use our model to fill the gap. Doing under textual condition guides MDM to fill the gap with specific motion that still maintains the semantics of the original input. By performing inpainting in the joints space rather than temporally, also demonstrate the semantic editing of specific body parts, without changing the others (Figure 33)
Overall, introduce Motion Diffusion Model, motion framework that achieves state-of-the- tfftthe-art quality in several motion generation tasks, while requiring only about three days of training on a single mid-range GPU. It supports geometric losses, which non trivial to the diffusion setting.
2

---

Published as a conference paper at ICLR 2023
but are crucial to the motion domain, and offers the combination of state-of-the-an generative power with well thought-out domain knowledge.
2
RELATED WORK
2.1 HUMAN MOTION GENERATION
Neural motion generation, learned from motion capture data, can be conditioned by any signal that describes the motion. Many works use parts of the motion itself for guidance. Some predict motion
from its prefix poses (Fragkiadaki et al. 2015 Martinez et al.] 2017] Hernandez et al. 2019] Guo
al.
2022bb. Others (Harvey & Pal 2018 Kaufmann al. [2000] |020] Harvey et 2020 Duan al.
2021 solve in-berweening and super-resolution tasks using bi-directional GRU (Cho et al., 2014)
and Transformer (Vaswani al.] 2017) architectures. [Holden et al. (2016) use auto-encoder to learn motion latent representation, then utilize it to edit and control motion with spatial constraints such as root trajectory and bone lengths. Motion can be controlled with a high-level guidance given from action class (Guo et al., 20200 Petrovich et al., 2021 Cervantes et al. 2022). audio (Li et al. 2021 Aristidou et al., 2022) and natural language (Ahuja & Morencyj 2019 Petrovich et al., 2022]. In most cases authors suggests dedicated approach to map each conditioning domain into motion.
In recent years, the leading approach for the Text-to-Motion task is to learn a shared latent space for language and motion. JL2P (Ahuja & Morency 2019) learns the KIT motion-language dataset (Plappert et al., 2016) with an auto-encoder, limiting one-to-one mapping from text to mo- tion. TEMOS (Petrovich et al. 2022) and T2M (Guo et al., 2022a) suggest using VAE (Kingmal & Welling 2013) to map text prompt into a normal distribution in latent space. Recently, Mo- tionCLIP Tevet et al. 2022) leverages the shared text-image latent space learned by CLIP (Radford) et al.
2021 to expand text-to-motion out of the data limitations and enabled latent space editing.
The human motion manifold can also be learned without labels, shown by Hollen et al. (2016) V-Poser (Pavlakos et al., 2019), and more recently the dedicated MoDi architecture (Raab et al. We show that our model is capable for such an unsupervised setting as well.
2022
2.2 DIFFUSION GENERATIVE MODELS
Diffusion models (Sohl Dickstictt et al. 2013: Song & Ermon 2020) are class of neural generative models, based on the stochastic diffusion process as it modeled in Thermodynamics. In this set- ting, a sample from the data distribution is gradually noised by the diffusion process. Then, neural model learns the reverse process of gradually denoising the sample. Sampling the learned data dis- tribution is done by denoising a pure initial noise. [H et al. (2020) and Song al. (2020a) further developed the practices for image generation applications. For conditioned generation, Dhariwal & Nichol (2021). introduced classifier-guided diffusion, which was later adapted by GLIDE (Nichol et al.] (021) to enable conditioning over CLIP textual representations. The Classifier-Fres Guidance approach Ho & Salimans (2022) enables conditioning while trading-off fidelity and diversity, and achieves better results (Nichol et al.) 202T). In this paper, we implement text-to-motion by condi- tioning on CLIP in a classifier-free manner, similarly text-to-image (Ramesh et al.) [022]] Saharia iet al. 2022b). Local editing of images is typically defined as an inpainting problem. where a part of the image constant, and the inpainted part is denoised by the model, possibly under some con- dition (Song et al., 2020b; Saharia et al. 2022a). We adapt this technique to edit motion's 's specific body parts or temporal intervals in-betweening) according to an optional condition.
Closer to context, Gu et al. (0022) used the diffusion formulation to model the stochasticity of human trajectory prediction. More recently, concurrent to this work, Zhang et al. (2022) and Kim et al. (2022) have suggested diffusion models for motion generation. Our work requires significantly fewer GPU resources and makes design choices that enable geometric losses, which improve results.
3 MOTION DIFFUSION MODEL
An overview of our method described in Figure 2 Our goal to synthesize a human motion I::N of length N' given an arbitrary condition This condition can be any real-world signal that will dictate the synthesis, such as audio (Li et al. 202 Aristidou et al. (022). natural language dext-to-motion (Tevet et al. 2022 Guo al.; 2022a) or a discrete class cection-to-motion let al., 2020; Petrovich et al. 2021). In addition, unconditioned motion generation is also possible, which we denote the null condition c 0. The generated motion ..:- {xr N.N is sequences
(Guo
3

---

Published as a conference paper at ICLR 2023
promet
80
Linear
CLIP
Diffuse --
Diffuse 0-17-21 &
MLP
Transformer Encoder
MDM
MDM
MDM
Random
Sy
CT 1)
Linear
Atk
Linear
Figure 2: (Left) Motion Diffusion Model (MDM) overview. The model is fed a motion sequence 2 L:N of length N in noising step I, well itself and conditioning code c. C, CLIP (Rafford et al. 2021) based textual embedding in this case, is first randomly masked for classifier-fres free learning and then projected together with into the input token 2tk- In each sampling step, the transformer- encoder predicts the final clean motion ..... (Right) Sampling MDM. Given condition we sample random noise TT the dimensions of the desired motion, then iterate from 1. At each step MDM predicts the clean sample êo. and diffuses back to 21-1-
of human poses represented by either joint rotations or positions x² E RJDD, where is the number of joints and D is the dimension of the joint representation. MDM can accept motion represented by either locations, rotations, or both (see Section 4
Framework. Diffusion is modeled Markov noising process, (2!:N17 1==0 where zf:h is drawn from the data distribution and
( (1 o.).),
where az E (0. 1) constant hyper-parameters. When a is small enough, we can approximate JT L:N ~ ~O,, From here on we use Fe to denote the full sequence at noising step t.
In our context, conditioned motion synthesis models the distribution p(zolc) as the reversed diffu- sion process of gradually cleaning Zr. Instead of predicting as formulated by Ho et al. (2020), follow Ramesh et al. (2022) and
equivalent formulation predict the signal itself, i.e.,
to = G( t, c) with the simple objective (Ho et al. 2020).
Lsimple Ezor-q(oole),
- G( f, c
Geometric losses. In the motion domain, generative networks standardly regularized using geometric losses Petrovichh et al. (202T): (hhi et al. (2020). These losses enforce physical properties and prevent artifacts, encouraging natural and coherent motion. In this work experiment with three common geometric losses that regulate (1) positions (in case we predict rotations), (2) foot contact, and (3) velocities.
N
Cpos
[FK( - FK
(3)
N
N-1
Lisol
1
Σ
(FK FK(G))) f.l12.
(4)
N
N-1
Lvel
N [(x] - 21)) (6
(5)
In case we predict joint rotations, FK(.) denotes the forward kinematic function converting joint rotations into joint positions (otherwise, it denotes the identity function). fi E 13 is the binary foot contact mask for each frame i. Relevant only to feet, it indicates whether they touch the ground, and are set according to binary ground truth data (Shi et al., 2020). In essence, it mitigates the foot-sliding effect by nullifying velocities when touching the ground. Overall, our training loss is f. = Simple Apos Epus + Lvel Ctaat
4

---

Published as a conference paper at ICLR 2023
In-betweening:
In-betwennng:
Edit upper "Throw a ball"
"A person perfurns crazy dance noOVe."
"A person walking while raising
Figure 3: Editing applications. Light blue frames represent motion input and bronze frames are the generated motion. Motion in-betweening (left+center) can be performed conditioned on text or without condition by the same model. Specific body part editing using text is demonstrated on the right: the lower body joints are fixed to the input motion while the upper body altered to fit the input text prompt.
Model. Our model is illustrated in Figure 2 We implement G with a straightforward trans- former (Vaswani et al., (2177) encoder-only architectury The transformer architecture is temporally aware, enabling learning and generating variable- length motions. and is well-proven for the motion domain (Petrovich et al., 2021; Duan et al. [021] Aksan et al. (021). The noise time-step / and the condition code e are each projected to the transformer dimension by separate feed-forward net- works, then summed yield the token Zek- Each frame of the noised input z; is linearly projected into the transformer dimension and summed with a standard positional embedding. žek and the pro- jected frames are then fed to the encoder. Excluding the first output token (corresponding to 24k), the encoder result is projected back to the original motion dimensions, and serves as the prediction fo. We implement rext-to-motion by encoding the text prompt with CLIP (Radford et al.) 2021) text encoder, and action-to-motion with learned embeddings per class.
Sampling from p(zolc) is done in an iterative manner, according to Ho al. 2020) In every time step 1 we predict the clean sample èn = =G(z, f, c) and noise it back to 24-1- This is repeated from 7' until zo is achieved (Figure right.. We train our model G' using classifier-free guidance (Ho) & Salimans 2022). In practice, G learns both the conditioned and the unconditioned distributions by randomly setting c for 10% of the samples, such that G(z,, t, 0) approximates p(zg). Then, when sampling G we can trade-off diversity and fidelity by interpolating or even extrapolating the two variants using 8: Gz( L, c) = G(v,, I, 0) + 8 (G(a,,,,c) G(r,, ())).
Editing. We enable motion in in-betweening in the temporal domain, and body part editing in the spa- tial domain, by adapting diffusion inpainting to motion data. Editing is done only during sampling, without any training involved. Given subset of the motion sequence inputs, when sampling the model (Figure 2 right), at each iteration we overwrite so with the input part the motion. This encourages the generation to remain coherent to original input, while completing the missing parts. In the temporal setting, the prefix and suffix frames the motion sequence are the input, and we solve motion in betweening problem (Harvey et al. 2020). Editing can be done either condition- ally or unconditionally (by setting e = 0). In the spatial setting, we show that body parts can be re-synthesized according to condition e while keeping the rest intact, through the of the same completion technique.
EXPERIMENTS
We implement MDM for three motion generation tasks: Text-to-Motion(4.1) Piriiontto-Motion 4.2p and unconditioned generation(5.2 Each sub-section reviews the data and metrics of the used bench- marks, provides implementation details, and presents qualitative and quantitative results. Then, we show implementations of motion in- in-betweening (both conditioned and unconditioned) and body- part editing by adapting diffusion inpainting to motion (5.I). Our models have been trained with T = 1000 noising steps and cosine noise schedule. In Appendix A.I experiment with differ-
5

---

Published as a conference paper at ICLR 2023
ent values of T. All of them have been trained on single NVIDIA GeForce RTX 2080 Ti GPU for period of about 3 days.
4.1 TEXT-TO-MOTION
Text-to-motion is the task of generating motion given an input text prompt. The output motion is ex- pected to be both implementing the textual description, and valid sample from the data distribution (i.e. adhering to general human abilities and the rules of physics). In addition, for each text prompt, we also expect a distribution of motions matching it, rather than just a single result. We evaluate our model using two leading benchmarks KIT (Plappert et al. 2016) and HumanML3D (Guo et al., 2022a), over the set of metrics suggested by Guo et al. (2022a): R-precision and Multimodal- measure the relevancy of the generated motions to the input prompts, FID measures the dissimilar- ity between the generated and ground truth distributions (in latent space), Diversity measures the variability in the resulting motion distribution, and MultiModality is the average variance given single text prompt. For the full implementation of the metries, please refer to Guo et al. (2022a). We use HumanML3D as a platform to compare different backbones of our model, discovering that the diffusion framework is relatively agnostic to this attribute. In addition, we conduct user study comparing our model to current art and ground truth motions.
Data. HumanML3D is recent dataset, textually re-annotating motion capture from the AMASS (Mahmood et al. 2019) and HumanAct12 (Guo et al., 2020) collections. It contains 14, 616 motions annotated by 44, 970 textual descriptions. In addition, it suggests a redundant data repre- sentation including a concatenation of root velocity, joint positions, joint velocities, joint rotations and the foot contact binary labels. We also use in this section the same representation for the KIT dataset, brought by the same publishers. Although limited in the number (3, 911) and the diversity of samples, most of the rext-to-motion research is based on KIT, hence we view it as important to evaluate using it as well.
Implementation. In addition to our Transformer encoder-only backbone (Section 3 experiment MDM with three more backbones: (1) Transformer decoder injects 24k through the cross-attention layer, instead of as input token. (2) Transformer decoder + input token, where ztk: is injected both
ways, (3) GRU (Cho et al. 2014) concatenate 2zk to each input frame. and (4) a U-net adaptation for motion data (Table D. U-net is adapted to motion by replacing the 2D convolution filters with ID convolutions in the temporal axis, such that joints are considered as channels. This is due to the irregular behavior of the joint axis. Our models were trained with batch size 64, layers (except GRU that was optimal at 2), and latent dimension 512. To encode the text we use frozen CLIP- ViT-B/32 model. In addition, we experiment with replacing CLIP with Centence-BERY (Reimers & Gurevych 2019)- a BERT-based (Devlin et al. 2019) text encoder. The full details can be found in our published code and Appendix C Each model was trained for 500K steps, after which a checkpoint was chosen that minimizes the FID metric to be reported. Since foot contact and joint locations are explicitly represented in HumanML33,, we don't 't apply geometric losses in this section. We evaluate our models with guidance-scale 2.5 which provides a dnvversity-fielliii sweet spot (Figure 4
Quantitative evaluation. We evaluate and compare models to current art (JL2P Ahuja & Morency (2019), Text2Gisttree (Bhattacharya et al. 202T). and T2M (Guo et al. 2022a)) with the metrics suggested by Guoo et al. (2022a). As can be seen, MDM achieves state-of -he--e--r-ttrr art results in FID, Diversity, and MultiModality, indicating high diversity per input text prompt, and high-quality samples, as can also be seen qualitatively in Figure !
User study. We asked 31 users to choose between MDM and state-of-the-art works in a side-by-side view, with both samples generated from the same text prompt randomly sampled from the KIT test set. We repeated this process with 10 samples per model and 10 repetitions per sample. This user study enabled a comparison with the recent TEMOS model (Petrovich et al., 2022), which was not included in the HumanMLL3D benchmark. Fig. T shows that most of the time, MDM was preferred over the compared models, and even preferred over ground truth samples 42.3% of the cases. 4.2 ACTION-TO-MOTION
Action-to www.ttooomoooiiiooo is the task of generating motion given an input action class, represented by a scalar. The output motion should faithfully animate the input action, and at the same time be natural and reflect the distribution of the dataset on which the model is trained. Two dataset are commonly used to evaluate iition-to-motion models: HumanAct12 (Guo et al. 2020) and UESTC (Ji et al.) 2018)

---

Published as a conference paper at ICLR 2023
R Precision (top 3)1
Multimodal Dist.
Method
FIDJ
Diversity +
Multimodality!
Real
0.797#-002
0.002#-0000
2.974
9.503= 065
JL2P
0.486#.002
11.022-046
5.296+-0008
7.676= 058
Text2Gesture
0.345#.002
7.6644.030
#.000
6.409=
T2M
0.740 000
1.067#-002
3.340 ons
9.188= 009
2.090=.085
MDM (ours)
0.611#.007
0.544
5.5664.027
9.559 0:6 086
2.799#.072
+ scnt-BERT
0.609=-006
0.586-t.036
5.504=.03
9.666= 0035
2.707=1188
MDM (decoder)
0.608#.005
0.767=4085
5.507
9.176
2.927
+ imput token
#005
0.567=4.051
5.424 #t.02
9.425 = 060
2.834 095
MDM (U-net)
0.603#-006
1.137=
5.629tt0032
8.958 = 0008
2.636 214
MDM (GRU)
0.645
4.569
5.325 4:006
7.688 = 1.26/16
Table Quantitative results on the HumanML3D test set. All methods use the real motion length from the ground truth. "->" means results are better if the metric is closer the real distribution. We run all the evaluation 20 times (except MultiModality runs times) and + indicates the 95%
confidence interval. Bold indicates best result.
R Precision (top 3)*
Multimodal Dist
Method
FID
Diversity->
Multimodality
Real
0.779=-006
0.031#-004
2.788+.012
11.08=t000
JL2P
0.483++000
6.545=0072
5.147=.080
9.073#-100 100
Text2Gesture
0.338
12.12 18.
6.961
9.33
T2M
0.693#.007
2.770=t.109
3.401 =
10.91 #1119
1.482 065
MDM (ours)
0.396+.004
0.497#
9.191
10.847
1.907=
Table 2: Quantitative results on the KIT test set.
We evaluate our model using the of metrics suggested by Guo al. (2020), namely Fréchet In- ception Distance (FID), action recognition accuracy, diversity and multimodality. The combination of these metrics makes good measure of the realism and diversity of generated motions.
Data. HumanAct12 2 (Guo et al. 2020) offers approximately 1200 motion clips, organized into 12 action categories, with 47 to 218 samples per label. UESTC (Ji et al. 20188) consists of 40 action classes, 40 subjects and 25K samples, and is split to train and test. We adhere to the cross-subject testing protocol used by current works, with 225-345 samples per action class. For both datasets we the sequences provided by Petrovich al. 2021.
FID R-precision
59.4%
42. 3%
MDM vs.
MOM VS.
MDM vs.
MOM vs.
(a) KIT User Study
(b) Classifier-free scale sweep
Figure 4: (a) Text-to-motion rate of MDM over the compared model. MDM was preferred over the other models most of the time, and 42.3% of the cases even over ground truth samples. The dashed line marks 50%. (b) Guidance-scale sweep for HumanML3D dataset. FID (lower is better) and R-precision (higher is better) metrics function of the scale s, draws an accuracy-fide feceeacy-fidelity sweet spot around 2.5.
study for the KIT dataset. Each bar represents the preference

---

Published as a conference paper at ICLR 2023
Method
FID
Accuracyt
Diversity->
Multimodality
Real (INR)
0.020#t000
0.997#.001
6.850#.050
2.450=.040
Real (ours)
0.0500t0000
0.990#.000
6.880+.020
2.590=.010
Action2Motion 2020)
0.338
0.917#.003
6.879#.000
2.511 022
ACTOR (2021)
0. 1.1200.000
0.955 008
6.840=..030 030
2.530=.22 020
INR 2022)
0.088:t.004
0.973#.001
6.881#.04
2.569
MDM (ours)
0. 0.100#
0.990#.000
6.860#.050
2.520=.010
w/o foot contact
0.080=t.000
0.990=t000
6.810##0000
2.580=.010
Table Evaluation of action-to-motion on the HumanAct12 dataset. Our model leads the board in three out of four metrics. Ground-truth evaluation results are slightly different for each of the works, due to implementation differences, such as python package versions. sess the diversity and multimodality of each model using its own ground-trun truth results, they measured by their distance from GT. We show the GT metrics measured by our model and by the leading compared work, INR (Cervantes et al. 2022). Bold indicates best result, underline indicates second best, + indicates 95% confidence interval, indicates that closer to real is better.
important to as-
Method
FID +
FID +
Accuracy?
Diversity
Multimodality
Real
2.92#.26
2.79
0.98g: .001
33.34
14.16t.06
ACTOR (2021)
20.49 12.11
23. 23.43=2.20
0.911
31.96+..3
14.52
INR (2022 (hest variation)
9.55 0:
15.00
0.941
31.59
14.68
MDM (ours)
9.98+1.33
12.81 #1.46
0.950=t.00
33.02
14. 1...26+.12
w/o foot contact
9.69#-81
13.08=2.32
0.960=.000
33.10 29
14.06
Table 4: Evaluation of action-to-motion on the UESTC dataset. The performance improvement with our model shows clear gap from state-of-the-art. Bold indicates best result, underline indi- cates second best, + indicates 95% confidence interval, > indicates that closer to real is better.
Implementation.
The implementation presented in Figure2holes for all the variations of our work. In the case of action-to-motion, the only change would be the substitution of the text embedding by an action embedding. Since action is represented by a scalar, its embedding is fairly simple; each input action class scalar is converted into a learned embedding of the transformer dimension.
The experiments have been run with batch size 64, a latent dimension of 512, and an encoder- transformer architecture. Training on HumanAct12 and UESTC has been carried out for 750K and 2M steps respectively. In our tables we display the evaluation of the checkpoint that minimizes the FIID metric.
Quantitative evaluation. Tables and reflect MDM's performance on the HumanAct12 and UESTC datasets respectively. We conduct 20 evaluations, with 1000 samples in each, and report their average and 95% confidence interval. We test two variations, with and without foot contact loss. Full ablation study for geometric losses is presented Appendix A.2 Our model leads the board for both datasets. The variation with no foot contact loss attains slightly better results; never- theless, as shown in our supplementary video, the contribution foot contact loss to the quality of results is important, and without it witness artifacts such shakiness and unnatural gestures.
5
ADDITIONAL APPLICATIONS
5.1 MOTION EDITING
In this section we implement two motion editing applications in-betweening and body part edit- ing, both using the same approach in the temporal and spatial domains correspondingly. For in- betweening, we fix the first and last 25% of the motion, leaving the model to generate the remaining 50% in the middle. For body part editing, we fix the joints don't want to edit and leave the
8

---

Published as a conference paper at ICLR 2023
model to generate the rest. In particular, experiment with editing the upper body joints only. In figure 3 we show that in both cases, using the method described in Section generates smooth motions that adhere both to the fixed part of the motion and the condition (if one given).
Precisiont Recallf
Method
FID
KIDL
Diversityt
ACTOR (202))
48,80
0.53
0.72, 0.74
14.10
MoDi 2022)
13.03
0.12
0.71, 0.81
17.57
MDM (ours)
31.92
0.36
0.66, 0.62
17.00
Table 5: Evaluation of unconstrained synthesis on the HumanAct12 dataset. We test MDM in the challenging unconstrained setting, and compare with MoDi (Raab et al., [022), work that was specially designed for such setting. We demonstrate that in addition to being able to support any condition, we can achieve plausible results in the unconstrained setting. Bold indicates best result.
5.2 UNCONSTRAINED SYNTHESIS
The challenging task of unconstrained synthesis has been studied by only a few (Holden et al. 2016 Raab et al.] 2022). In the presence of data labeling, e-g., action classes or text description, the la- bels work as a supervising factor, and facilitate structured latent space for the training network. The lack of labeling make training more difficult. The human motion field possesses rich unlabeled datasets (Adobe Systems Inc., 2021). and the ability train top of them is advantage. Daring to test MDM in the challenging unconstrained setting, we follow (Rab et al. 2022) for eval-
uation. We the metries they suggest (FID, KID, precision/recall and multimodality). and run on an unconstrained version of the HumanAct12 (Guo et al.) 2020) dataset.
Data. Although annotated, we HumanAct12 (see Section 4.2) in an unconstrained fashion, ignoring its labels. The choice of HumanAct12 rather than a dataset with no labels (e.g.. Mix- amo (Adobe Systems Inc. 2021 is for compatibility with previous publications.
Implementation. Our model uses the same architecture for all forms of conditioning, as well as for the unconstrained setting. The only change to the structure shown in Figure 2 is the removal of the conditional input, such that Zez: is composed of the projection of only. To simulate an unconstrained behavior, ACTOR Petrovich et al. (202T) has been trained by (Raab al. 2022) with a labeling of one class to all motions.
Quantitative evaluation. The results of our evaluation are shown in table We demonstrate supe- riority over works that were not designed for an unconstrained setting, and get closer to MoDi (Raab) et al. 2022). MoDi is carefully molded for unconstrained settings, while our work can be applied to any (or no) constrain, and also provides editing capabilities.
DISCUSSION
We have presented MDM, a method that lends itself to various human motion generation tasks. MDM is untypical classifier-free diffusion model, featuring transformer-encoder backbone, and predicting the signal, rather than the noise. This yields both lightweight model, that is unburdening to train, and an accurate one, gaining much from the applicable geometric losses. Our experiments show superiority in conditioned generation, but also that this approach is not very sensitive to the choice of architecture.
notable limitation of the diffusion approach is the long inference time, requiring about 1000 forward passes for single result. Since our motion model is small anyway, using dimensions order of magnitude smaller than images, our inference time shifts from less than second to only about minute, which is an acceptable compromise. As diffusion models continue to evolve, besides better compute, in the future we would be interested in seeing how to incorporate better control into the generation process and widen the options for applications even further.
ACKNOWLEDGEMENTS
We thank Rinon Gal for his useful suggestions and references. This research was supported in part by the Israel Science Foundation (grants no. 2492/20 and 3441/21), Len Blavatnik and the Blavatnik family foundation, and The Tel Aviv University Innovation Laboratories (TILabs).

---

Published as a conference paper at ICLR 2023
REFERENCES
Kfir Aberman, Peizhuo Li, Dani Lischinski, Olga Sorkine- Hornung Daniel Cohen-Or, and Baoquan Chen. Skeleton-awe aware networks for deep motion retargeting. ACM Transactions on Graphics (TOG), 39(4):62-1, 2020.
Adobe Systems Inc. Mixamo, 2021. URL /ttthttps:////t wwwp:wwwwww.....mm.w... xamo com
mi
Accessed: 2021-12-25.
Chaitanya Ahuja and Louis-Philippy Morency. Language2pose: Natural language grounded pose forecasting. In 2019 International Conference 3D Vision (3DV), pp. 719-728. IEEE, 2019.
Emre Aksan, Manuel Kaufmann, Peng Cao, and Otmar Hilliges. A spatio-temporal transformer for 3d human motion prediction. In 2021 International Conference on 3D Vision (3DV), pp. 565-574. IEEE, 2021.
A Aristidou, Yiannakidis, K Aberman, Cohen-Or, A Shamir, and Chrysanthou. Rhythm is a dancer: Music-driven motion synthesis with global structure. IEEE Transactions Visualization and Computer Graphics, 2022.
Uttaran Bhattacharya, Nicholas Rewkowski, Abhishek Banerjee, Pooja Guhan, Aniket Bera, and Dinesh Manocha. Text2gestures: A transformer-based network for generating emotive body ges- tures for virtual agents. In 2021 IEEE Virtual Reality and 3D User Interfaces (VR), pp. 1-10. IEEE, 2021.
Pablo Cervantes, Yusuke Sekikawa, Ikuro Sato, and Koichi Shinoda. Implicit neural representations for variable length human motion generation. arXiv preprint arXiv:2203. 13694, 2022.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using encoder-decoder for statistical machine translation. arXiv preprint arXiv: 2014.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter the Association for Computational Linguisties: Human Language Technologies, Volume (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 00.18653/v1/N19-111 1423. URL https:
7/ 7/aclanthologe www.tthoooooyo.../111111 -erg/N19-1423
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis, Advances in Neural Information Processing Systems, 34:8780-8777 2021.
Yinglin Duan, Tianyang Shi. Zhengxia Zou, Yenan Lin, Zhehui Qian, Bohan Zhang, and Yi Yuan. Single-shot motion completion with transformer. arXiv preprint arXiv:2 103760776, 2021.
Katerina Fragkiadaki, Sergey Levine, Panna Felsen, and Jitendra Malik. Recurrent network models for human dynamies. In Proceedings of the IEEE international conference computer vision, pp. 4346-4354, 2015.
Tianpei Gu, Guangyi Chen, Junlong Li, Chunze Lin, Yongming Rao, Jie Zhou, and Jiwen Lu. Stochastic trajectory prediction via motion indeterminacy diffusion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 17113-17122. 2022.
Chuan Guo, Xinxin Zuo, Sen Wang, Shihao Zou, Qingyao Sun, Annan Deng, Minglun Gong, and
Li Cheng. Action2motion: Conditioned generation of 3d human motions. In Proceedings of the 28th ACM International Conference on Multimedia, pp. 2021-2029, 2020.
Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5152-5161, 2022a.
Wen Guo, Yuming Du, Xi Shen, Vincent Lepetit, Xavier Hameda-Pinndda and Francese Moreno- Noguer. Back to mlp: A simple baseline for human motion prediction. arXiv preprint arX15:2207.01567. 2022b.
10

---

Published as a conference paper at ICLR 2023
Félix G Harvey and Christopher Pal. Recurrent transition networks for character locomotion. In SIGGRAPH Asia 2018 Technical Briefs, pp. 1-4. 2018.
Félix Harvey, Mike Yurick, Derek Nowrourzezahrai, and Christopher Pal. Robust motion in- betweening. ACM Transactions on Graphics (TOG), 39(4):60-1, 2020.
Alejandro Hernandez, Jurgen Gall, and Francesc Moreno-Noguer. Human motion prediction via spatio-temporal inpainting. In Proceedings of the IEEE/CVF International Conference on Com- puter Vision, pp. 7134-7143, 2019.
Jonathan Ho and Tim Salimans. arXiv:2207.12598, 2022.
Classifier-free
diffusion guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. arXiv preprint arXiv:2204.03458, 2022.
Daniel Holden, Jun Saito, and Taku Komura. deep learning framework for character motion synthesis and editing. ACM Transactions on Graphics (TOG), 35(4):1-11, 2016.
Yanli Ji, Feixiang Xu, Yang Yang, Fumin Shen, Heng Tao Shen, and Wei-Shi Zheng. A large-scale rgb-d database for arbitrary-vieww human action recognition. Proceedings of the 26th ACM international Conference Multimedia, pp. 1510-1518, 2018.
Manuel Kaufmann, Emre Aksan, Jie Song, Fabrizio Pece, Remo Ziegler, and Otmar Hilliges. Con- volutional autoencoders for human motion infilling. In 2020 International Conference on 3D Vision (3DV), pp. 918-9277 IEEE, 2020.
Jihoon Kim, Jiseob Kim, and Sungjoon Choi. Flame: Free-form language- motion synthesis & editing, arXiv preprint arXiv:2209.000 00349, 2022.
Diederik P Kingma and Max Welling,
Auto-encoding variational bayes.
arXiv preprint
arXiv: 14,0 2013.
Muhammed Kocabas, Nikos Athanasiou, and Michael J Black. Vibe: Video inference for human body pose and shape estimation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 5253-5263, 2020.
Ruilong Li, Shan Yang, David A. Ross, and Angjoo Kanazawa. Ai choreographer: Music con- ditioned 3d dance generation with aist++. In The IEEE International Conference on Computer Vision (ICCV), 2021.
Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceed- ings of the IEEE/CV Conference on Computer Vision and Pattern Recognition, pp. 2837-2845, 2021.
Naureen Mahmood, Nima Ghorbani, Nikolaus F. Troje, Gerard Pons-Moll, and Michael J. Black. AMASS: Archive of motion capture as surface shapes. In International Conference on Computer Vision, pp. 5442-5451, October 2019.
Julieta Martinez, Michael Black, and Javier Romero. On human motion prediction using recur- rent neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2891-2900, 2017.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2 /2. 10741, 2021.
Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dim- itrios Tzionas, and Michael J. Black. Expressive body capture: 3D hands, face, and body from single image. Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 10975-10985, 2019.
11

---

Published as a conference paper at ICLR 2023
Mathis Petrovich, Michael J. Black, and Gül Varol. Action-condin conditoend 3D human motion synthesis with transformer VAE. In International Conference on Computer Vision (ICCV), pp. 10985- 10995, October 2021.
Mathis Petrovich, Michael J. Black, and Güi Varol. TEMOS: Generating diverse human motions from textual descriptions. In European Conference on Computer Vision (ECCV), 2022.
Matthias Plappert, Christian Mandery, and Tamim Asfour. The kit motion-language dataset. Big data, 4(4):236-252, 2016.
Sigal Raab, Inbal Leibovitch, Peizhuo Li. Kfir Aberman, Olga Sorkine- and Daniel Cohen- Or. Modi: Unconditional motion synthesis from diverse data, arXiv preprint arXiv:2206.0801 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. pp. 8748-8763. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
Nils Reimers and Iryna Gurevych. Sentence-bert. Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pp. 3982-3992, 2019.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234 241. Springer. 2015.
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: taage-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings, pp. 1-10, 2022a.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi. Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint 1 2022b.
Mingyi Shi, Kfir Aberman, Andreas Aristidou, Taku Komura, Dani Lischinski, Daniel Cohen-Or, and Baoquan Chen. Motionet: 3d human motion reconstruction from monocular video with skeleton consistency. ACM Transactions Graphics (TOG), 40(1):1-15, 2020.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learn- ing, pp. 2256-2265. PMLR, 2015.
Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500 2020a.
Yang Song and Stefano Ernon. Improved techniques for training score-basee based generative models. Advances in neural information processing systems, 33: 33:12438- 12438-12448, 2020.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020b.
Guy Tevet, Brian Gordon, Amir Hertz, Amit H Bermano, and Daniel Cohen Or. Motionclip: Ex- posing human motion generation to clip space. arXiv preprint arXiv:2203.0800 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
12

---

Published as a conference paper at ICLR 2023
Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondiffuse: Text-driven human motion generation with diffusion model. arXiv preprint 15000, 2022.
A ADDITIONAL EXPERIMENTS
A.1 DIFFUSION PARAMETERS
Learning MDM with different numbers of diffusion steps significantly affects performance and holds the potential to accelerate inference time. Table 6 shows optimal performance for T = 100, in addition, it enables accelerating inference by factor of 10 compared to T 1000, which is widely used for images.
Diffusion steps (T)
R Precision (top 3)1
Multimodal Dist
FID
Diversity +
Real
0.797#.002
0.002
2.974
9.503#
0.5744.000
1.881
3.816
9.369-t.058
100
0.640).0077
0.454
3.336
9.906#t0533
500
0.662 10622.000 007
0.553
+555
5.177
L.28
9.890
1000
0.611 1.00
0.544 1.044
3.566 L.27
9.559 1.088
Table 6: Diffusion steps (HumanML3D test set). We run all the evaluation 20 times. Bold indi- cates best result, underline indicates second best, + indicates 95% confidence interval, > indicates that closer real better.
A.2 GEOMETRIC LOSSES
We conduct a thorough experiment to evaluate the contribution of geometric losses with the Human- Act12 dataset. The results are presented in Table 7 For alignment with prior work, all metrics are calculated using the deep features of the action recognition network suggested by Guo et al. (2020). In general, MDM scores are too close to the real test distribution (i.e. the evaluation network fails to discriminate between the two). This means that quantitative results comparing the different variants MDM are too similar to evaluate. As result, we are not able to decide what combination of geo- metric losses is preferred. We leave for future work experimenting with a different, more expressive, evaluation network.
Method
FID4
Accuracyt
Diversity
Multimodality
Real
0.050
0.990#.000
6.880#.020
2.590#.010
MDM (ours)
0.100
0.990#.000
6.860 .50
2.520#.010
w/o foot contact
0.080 000
0.990=0000
6.810#.010
2.580+.010
w/o geometric losses
0.090-4-000
0.990
6.8204.020
2.550
foot contact only
(). L.000
0.990
6.860
2.520
velocity only
0.100=t.00m 000
0.990#.000
6.820 020
2.590
pose only
0.090
0.990#.00m
6.830 .200
2.570
Table 7: Geometric losses ablation study. (HumanAct12 dataset) the relative equals when the loss term is included, and 0 when is excluded.
13

---

Published as a conference paper at ICLR 2023
B EVALUATION METRICS.
For the completeness of our work, we describe here quantitative metrics used throughout the paper, they originally described and implemented by Guo et al. (0200) for action-to-motion and by Guo et al. (2022a) for text-to-motion.
B.1 ACTION-TO-MOTION
The following metrics are based RNN action recognition network as it was originally trained by Guo et al. (2020). We refer to as the evaluator network.
Frechet Inception Distance (FID). A widely used metric to evaluate the overall quality for gener- ation tasks. FID is calculated upon features extracted from 1,000 generated motion vs ground truth (real) taken from the test set. To adjust this metric to the motion domain, extract deep represen- tation of the motion with the evaluator network instead the inception neural network, originally used for images. A lower value implies better FID results.
Accuracy. We classify 1,000 generated motions using the evaluator network, than we calculate the overall recognition accuracy that indicates the correlation of the motion and its action type.
Diversity measures the variance of the generated motions across all action categories. We first randomly sample two subsets of the same size Sa out of set of all generated motions across all action categories denoted [Vi...., Vs. and {v' ,.., V's. The diversity of those sets of motions is defied
Diversity
Sa vi vz ll2
We use S2 = 200 for our experiments. The diversity value is considered better when closer to the diversity value the ground truth.
Multimodality measures the generated motions diversify within each action class. We randomly sample two subsets with size of the same motion class (Vo,ly Vo.t. and (Ve,1 v Ves,, The multimodality of all action classes C' is defined as.
Multimodality
V - vexll2
C S
0=1
We use S 20 for experiments.
B.2 TEXT-TO-MOTION
Originally suggested by Guo et al. (2022a), the following metrics are based on text feature extractor and motion feature extractor jointly trained under contrastive loss to produce geometrically close feature vectors for matched text-motion pairs, and vise versa.
R Precision. (top-3) For each generated motion, its ground truth text and randomly selected miss- matched descriptions from the test set. We calculate the euclidean distance between the motion feature and text feature of each description in the pool. We count the average accuracy at top- 3 places. the ground truth entry falling into the top-3 candidates, we treat it as True Positive retrieval. We use batch size 32 (i.e. 31 negative examples).
FID. Same as for action-to-motion, using the motion extractor as the evaluator network.
Multimodal Distance. We calculate the multimodal distance as the average Euclidean distance between the motion feature of each generated motion and the text feature of its corresponding de- scription in test set. lower value implies better multimodal distance.
Diversity. Same as for action-to-motie but with Sa = 300.
Multimodality. Same as for action-to-motion but with S., 10.
14

---

Published as a conference paper at ICLR 2023
C IMPLEMENTATION DETAILS
The full implementation of MDM can be found in our published code In addition, the followings the hyperparameters and model details for all of our experiments.
Diffusion framework. In all of our experiments, we used an implementation of DDPM (Ho et al., 2020 by Dhariwal & Nichol 2021 3 We use T 0000 diffusion steps, cosine noise scheduling (predefined sigmas). All other hyperparameters are according to the implementation defaults.
Transformer architecture. For our transformer architectures, used the PyTorch implementa- tion"] We used 8 transformer layers, attention heads, latent dimension d = 512, dropout 0.1, feed-forward size 1024 and gelu activations. The number of learned parameters for each model is stated in T
2014 with two layers and latent dimension 512. The number of learned parameters for each model is stated in Table 8
GRU architecture. We use the PyTorch implementation of GRU (Cho al.
Learning hyperparameters. For all of our experiments, we use batch size 64, learning rate 10-4,
Architecture
Parameters (-10%)
Transformer Encoder
17.88
Transformer Decoder
26.29
input token
26.29
U-net
23.47
GRU
4.47
Table 8: The number of Iearned parameters per architecture for the text-to-motion task. For the action-to-motie motoon task, there are additional 512 parameters per-class for the class embeddings mod- ule.
D USER STUDY
In Section 4. we conduct user study for the text-to-motion task. We asked 31 users to choose between MDM and state-of-the-an works in a side-by-side view, with both samples generated from the same text prompt randomly sampled from the KIT test set. We repeated this process with 10 samples per model and repetitions per sample. This user study enabled a comparison with the recent TEMOS model (Petrovich et al.) 2022). which was not included in the HumanMLL3D bench- mark. Fig. 4 shows that most of the time, MDM was preferred over the compared models, and even preferred over ground truth samples in 42.3% of the cases. This study designed mea- sure the precision of the models, i.e. which one better fits the input text. The exact phrasing of the question was "Which animation better fits the following description?". A sample question from this study is presented in Fig. 5)
15

---

Published as a conference paper at ICLR 2023
(5817) Which animation better fits follewing description?
Someone his left hand.
Someone stirring with his left hand.
1
2
1
2
Figure 5: An example question for our text-to-motion user study, using the Google Forms platform.
16


Reviews:
[{"review_id": "hDnr8qUG3M", "reviewer": "Reviewer_Eywo", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a diffusion based motion modeling method that uses transformers a backbone for the diffusion process. The authors train a general conditional motion generation method using classifier free guidance which allows them to use the same model for conditioned and unconditioned generation. In experiments, the authors perform favorably against other motion generation methods in the tasks for unconditional generation, text-conditioned generation, action-conditioned generation, in-betweening, and motion editing.", "summary_of_the_review": "All-in-all, this is a nice paper with good results and nice presentation. However, I am slightly learning towards acceptance due to the concerns presented in the weakness section, and lack of detail for reproducibility. Nevertheless, I am looking forward to the author's response and am willing to increase my score depending on it.", "strength_and_weaknesses": "Strengths:\n+ New diffusion architecture that combines transformers and diffusion\n+ Quantitatively and Qualitatively outperforms baselines in most of the evaluation tasks.\n+ Nice visualizations of the diffusion process and results\n\n\nWeaknesses:\n\n- Comparison between simply U-Net and the proposed backbone.\nGiven that the architectural novelty is the use of transformers instead of U-Net, I suggest the authors provide an ablation comparing the two design choices. The ablation should compare performance, inference time, and maybe any analysis showing differences of one versus the other. I feel like this is something critical that is missing from the paper, and would be good to have.\n\n\n- Inference time?\nGiven that diffusion models are known for being slow, it would be good to provide the time it takes for inference in comparison to other models. This would show how much room for improvement there is in this aspect of the method, and the tradeoffs of using the proposed method versus the performance gain over the baselines.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XxrfeoOUo3d", "reviewer": "Reviewer_Cmzb", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a diffusion model for synthesizing human motion parameters. The proposed model consists of a diffusion model adapted to the human motion data format, plus task-specific geometric losses. The proposed model is applied to multiple scenarios such as text-to-motion, action-to-motion, motion editing, and unconstrained synthesis. Extensive quantitative and qualitative evaluations show the promising performance gain achieved by the proposed model.", "summary_of_the_review": "Overall I think this is a strong paper, which presents the first steps of applying diffusion models to human motion synthesis. The presented method is technically sound and contains multiple insightful details. The paper's evaluation is comprehensive and its demonstration is well-executed. My concerns are minor and can be easily addressed. Therefore, my initial recommendation is to accept the submission.", "strength_and_weaknesses": "Paper strengths:\n- The paper presents a novel method, which is among the first to apply diffusion models to human motion synthesis. The method is well-thought, containing multiple non-trivial design choices to make the method work well on human motion data.\n- The paper is well-written and easy to follow. The demo is high-quality and well-made.\n- The experimental evaluations are thorough and convincingly demonstrate the effectiveness of the proposed model.\n\nPaper weaknesses:\n- Intuitively, the synthesized human motion should be varying-length. However, this does not seem to be addressed in the paper. I assume the clip length was fixed and sequences were temporally stretched/squeezed to meet the specified length. It would be great if additional discussions can be added.\n- It would be great to see a more detailed analysis of each particular geometric loss term and the influence of its weighting.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "NR8Becevw1", "reviewer": "Reviewer_W5bV", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a denoising diffusion model for both conditional and unconditional human motion (pose sequence) generation. Except for a couple of concurrent arXiv papers, this is the first time diffusion models have been applied to full-body human motion. The paper proposes a transformer-based model that can be conditioned on a text or action embedding in order to synthesize specific motions. The proposed approach makes several design decisions that make the diffusion model amenable to human motion including predicting clean samples rather than noise at each step and applying additional losses to improve motion quality. The model is evaluated on a large variety of tasks including text/action-conditioned generation, unconditioned generation, in-betweening, and motion editing, showing strong performance both qualitatively and quantitatively.", "summary_of_the_review": "Overall, the paper clearly introduces and thoroughly evaluates a strong diffusion model for human motion. Despite some small weaknesses, I think the qualitative results and flexibility of the model make it very useful for the community. Therefore, I recommend accept.", "strength_and_weaknesses": "\nStrengths:\n* The adaptation of the diffusion framework is done in a principled way, with reasonable design choices suitable for human motion modeling. For example, predicting the final clean motion from the network instead of the noise at each step is a subtle but important choice, which allows using geometry penalties during training and easy editing through overwriting at test time. Also using classifier-free guidance to control the influence of conditioning. Though no component is particularly novel (i.e. they are adapted from other domains), their combination gives a strong first pass at diffusion for human motion.\n* The approach is novel in the sense that it is the first diffusion model for full body motion (besides concurrent arXiv papers which show diffusion being used in a more limited capacity). \n* The model is demonstrated on a wide range of tasks (text-conditioned, action-conditioned, unconditioned, in-filling, editing). This large task variation really shows the capabilities of diffusion for human motion which may push the community to creatively continue in this direction. \n* Quantitively, the model is competitive or better than several relevant baselines across the evaluated tasks. Qualitatively, the results are quite convincing. Overall, the evaluation is quite thorough.\n* Code to be published.\n\nWeaknesses:\n* A couple components of the architecture were not clearly justified. First, the use of a transformer rather than the usual U-Net. The intro says the transformer “better fits the temporal and non-spatial nature of motion data” but a U-Net baseline is not compared against in experiments.  Second, why is CLIP is used rather than some other language-only model (as in TEMOS) to get the text embedding? Is the “visual” nature of the CLIP latent space actually helpful for human motion?\n* Sec 3 could use a bit more detailed intro to diffusion models for those not familiar, especially since this model is new to human motion.  For example, before Eq (2) it is stated that the model predicts the clean samples rather than the noise, but doesn’t explain that these are equivalent formulations.\n* Eq. 7 combines the clean motion predictions from the network for classifier-free guidance. This is a bit different from the original formulation of Ho & Salimans, which combines the noise prediction from the model. Are these formulations equivalent? (perhaps this doesn’t matter in practice).\n* For the user study in Sec 4.1. it’s not clear what the users were asked to evaluate. Is it the motion quality? Is it how well the motion matches the text? Ideally both should be evaluated.\n* Sampling from the diffusion model is quite slow (~1min for a sequence) as discussed in the conclusion. However, there has been much recent work on speeding up sampling lately, so this can be improved in future work.\n\nQuestions, comments, and clarifications (not considered in rating):\n* How many parameters are in the architecture? Is the transformer considered “lightweight” because it is faster or smaller than U-Net?\n* Intro: saying “non-spatial nature of motion data” is a bit confusing. Joints are positions which are by definition “spatial”. Maybe the authors meant the data is less structured, but even so joints are part of the kinematic tree structure.\n* Related work: Gu et al [Stochastic Trajectory Prediction via Motion Indeterminacy Diffusion, CVPR 2022] use a similar diffusion transformer for pedestrian trajectory prediction.\n* In practice, how long can generated sequences be before motion quality starts to degrade? Or can it truly handle “arbitrary” lengths?\n* If both joint positions and rotations are available in the pose, which are actually used as the output?\n* Since there are so many evalution metrics, it would be good to define them in the appendix for easy access.\n* It would be interesting to see how classifier guidance works for text and action-to-motion instead of having to re-train separately for each task. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "7bYOy0LUe9F", "reviewer": "Reviewer_Lxvz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes the Motion Diffusion Model (MDM) for multi-modality human motion generation, such as text-to-motion, action-to-motion, and unconditioned generation, etc. MDM is transformer-based and adopts an existing diffusion method that predicts the signal itself at each timestep instead of the noise so that geometric losses could be used. MDM achieves impressive results on several motion generation tasks, and requires lower computation resources at the same time.", "summary_of_the_review": "Overall, I lean to accept this paper at this time, since it is a simple and general framework with promising results in several motion generation tasks. However, I'm still confused about the necessity of introducing diffusion here, especially in predicting the signal itself. My rating may change depending on the experiment results I mentioned above.", "strength_and_weaknesses": "Pros\n- A simple and general framework, along with some good-quality generation results. \"simple\" is for the model simplicity, and \"general\" is for the ability to enable different forms of condition (text, action, editing...) \n- Lightweight compared to other multi-modality motion generation works.\n- Easy to read and understand.\n- Rich demos.\n\nCons\n\nThe necessity of the diffusion model is not clearly presented, except for adding geometric losses. The motivation for using diffusion models which predict the signal itself instead of the noise still needs a discussion. If the reason is that geometric losses could be added, given that we supervise each MDM with the GT $x_0$, it seems that there is no need to stack MDM for the diffusing process. \n\n- What will the result be when $T=1 or  2$ both for training and testing, compared to $T=1000$?  If there is just a small difference, it seems that the good performance is contributed to the transfomer encoder and CLIP, which has been widely used in many works.\n\n- Another concern is about the low diversity caused by predicting the signal at each step, which seems opposite to the original goal of diffusion models.\n \n- Have the authors tried to predict $\\epsilon$ and not use geometric losses in MDM? \n\nAblation experiments on diffusion models are appreciated and needed since they could help readers understand the motivation for introducing the diffusion model better. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%