Human Motion Diffusion as a Generative Prior
Authors: Yoni Shafir
Guy Tevet
Roy Kapon
Amit Haim Bermano
Venue: ICLR 2024 poster (2024)
Keywords: diffusion
prior
composition
generation
motion
generative
Primary area: generative models

Abstract: Recent work has demonstrated the significant potential of denoising diffusion models
for generating human motion, including text-to-motion capabilities.
However, these methods are restricted by the paucity of annotated motion data,
a focus on single-person motions, and a lack of detailed control.
In this paper, we introduce three forms of composition based on diffusion priors:
sequential, parallel, and model composition.
Using sequential composition, we tackle the challenge of long sequence
generation. We introduce DoubleTake, an inference-time method with which
we generate long animations consisting of sequences of prompted intervals
and their transitions, using a prior trained only for short clips.
Using parallel composition, we show promising steps toward two-person generation.
Beginning with two fixed priors as well as a few two-person training examples, we learn a slim
communication block, ComMDM, to coordinate interaction between the two resulting motions.
Lastly, using model composition, we first train individual priors
to complete motions that realize a prescribed motion for a given joint.
We then introduce DiffusionBlending, an interpolation mechanism to effectively blend several
such models to enable flexible and efficient fine-grained joint and trajectory-level control and editing.
We evaluate the composition methods using an off-the-shelf motion diffusion model,
and further compare the results to dedicated models trained for these specific tasks.

Published as a conference paper at ICLR 2024
HUMAN MOTION DIFFUSION AS A GENERATIVE PRIOR
Yonatan Shafir," Guy Tevet, Roy Kapon and Amit H. Bermano Tel Aviv University, Israel {Shafir2, (epeeieem@maii tau. ac.il
ABSTRACT
Recent work has demonstrated the significant potential of denoising diffusion models for generating human motion, including text-to-motion capabilities. How- ever, these methods are restricted by the paucity of annotated motion data, focus on single-person motions, and a lack of detailed control. In this paper, we intro- duce three forms of composition based on diffusion priors: sequential, parallel, and model composition. Using sequential composition, we tackle the challenge long sequence generation. We introduce DoubleTake, an inference-ting method with which we generate long animations consisting of sequences of prompted intervals and their transitions, using prior trained only for short clips. Using par- allel composition, we show promising steps toward two-person generation. Be- ginning with two fixed priors and a few two-person training examples, we learn a slim communication block, ComMDM, to coordinate interaction between the two resulting motions. Lastly, using model composition, we first train individ- ual priors to complete motions that realize a prescribed motion for given joint. We then introduce DiffusionBlending. an interpolation mechanism to effectively blend several such models to enable flexible and efficient fine- fine-grained joint and trajectory-level control and editing. We evaluate the composition methods using off-the-shell shelf motion diffusion model, and further compare the results to dedi- cated models trained for these specific tasks.
INTRODUCTION
Human Motion Generation has recently experienced tremendous leap forward. The recent elab- orate language models (Radford et al. 2021 Devlin et al.] 2019) and diffusion generation ap- proach (Sohl-Dickstein et al. 2015 Ho et 2020) have quickly found their way into the field, yielding motion generation models that produce diverse and high-quality sequences from text or other forms of control (Tevet al., 2023; 2022 Petrovich et al.) 2022] Guo et al., (022). In turn, these models have been already applied in the world of gaming, and hold the potential to open the field of character animation to novices and professionals alike.
However, the main problem the field of human motion generation has always struggled with and is still struggling with data. Motion data typically either acquired by elaborate motion capture settings (Joo et al.) 2015) or crafted by artists (Adobe Systems Inc.. (021). Both cases eventually lead to expensive and relatively small and homogeneous datasets (Punnakkal et al., 2021] Guo et al. 2022). For example, the datasets that current models are trained on, consist almost exclusively of short, single-person sequences. In the absence of data, tasks like multi-person interaction and long sequence generation are left behind, with poor generation quality.
In this paper, we show that pretrained diffusion- based motion generation models can be leveraged as priors for composition, allowing out-of-domain motion generation and efficient control. Contrary to the high data consumption reputation of diffusion models, we show three methods that overcome the cost barrier using the aforementioned prior, enabling non-trivial tasks in few-shot or even zero-shot settings.
In particular, we choose a pretrained Motion Diffusion Model (MDM) (Tevet et al. 2023 serve as the prior. MDM achieves state-of-the- art results in the text-to-motion and action to motion tasks for short single-person sequences, and has already been demonstrated to generalize well to condi-
"Equal contribution
Our code and trained models available httppp/////..ttccccc/o/oooor/trr

---

Published as a conference paper at ICLR 2024
"Walk" - "Sit while crossing lege" --
Capoeraa practice.
Loogg keep forward" - -TRurning in circle"
One is licking and the other is avoiding the kick."
We suggest three novel motion composition methods, all based on the recent Motion Diffusion Model (MDM). (Leff)) Sequential composition generating an arbitrary long motion with text control over each time interval. (Middle) Parallel composition generating two-person motion from text. different color represents different person both are generated simultaneously given the text prompt. (Right) Model composition achieving accurate and flexible control by blending models with different control signals here writing "hello" in mid-air.
Figure I:
tions from other domains (Teng et al. 2022). and to corrections performed between the sampling iterations (Yuan et al. 2022).
Using this prior, we demonstrate three forms of composition:
(1) Sequential composition, where short sequences concatenated create single long and coherent motion; (2) parallel composition, where two single motions are coordinated to perform together; and (3) model composition, where the motions generated by models with different control capabilities are blended together for composite control.
Our DoubleTake method (Figure [e-ff)) suggests sequential composition by carefully composing two generated motions in time, including the transition between them, and enables the efficient generation of long motion sequences in a zero-shot manner. Using it, we demonstrate 10-minute long fluent motions that were generated using a model that was trained only on up to 10 seconds long sequences (Guo al. 2022 Punnakkal al. 202I). In addition, due to the composite nature of the generation, Double allows individual control for each motion interval, while maintaining consistent motion and transitions. This result is fairly surprising considering that such transitions were not explicitly annotated in the training data. DoubleTake consists of two phases for every diffusion iteration in the first step, the individual motions, or intervals, are generated together in the same batch, each aware of the context of its neighboring intervals. Then, the second take refines the transitions between intervals to better match those generated in the previous phase.
For parallel composition, consider few-shot setting, and enable textually driven two-person motion generation for the first time [Tigure[I-Middee Using our prior-based approach, we demon- strate promising two-person motion generation using only few as dozen training examples. The key idea is that in order to learn human interactions, we only need to enable prior models to communicate with each other throughout the diffusion process. Hence, we learn slim communica- tion block, ComMDM, that passes a communication signal between the two frozen priors through intermediate activation maps.
Finally, we introduce novel control mechanism via model composition. We observe that the mo- tion inpainting process suggested by Tevet al. (2023 does not extend well to more elaborate yet important motion tasks such as trajectory and end-effector tracking. Hence, we first show that fine- tuning the prior for this task yields satisfying results while controlling even just a single end-effector. Then, introduce the DiffusionBlendinn technique, which generalizes classifier-free guidance (Ho) & Salimans 2022) to compose together different fine-tuned models and thus enables cross combi- nations of keypoints control on the generated motion. This enables surgical and flexible control for human motion that comprises a key capability for any animation system (Figure Right..
We demonstrate, both quantitatively and qualitatively, that these inexpensive composition methods extend more elaborately trained motion prior and outperform dedicated previous art in the respec- tive tasks (Wang et al. 2021 Athanasiou et al. 2022
2

---

Published as a conference paper at ICLR 2024
2 RELATED WORK
Motion Diffusion Models. Very recently, MDM (Tevet et al., 2023), MotionDiffuse (Zhang al. 2022), MoFusion (Dabral et al. 2023), and FLAME (Kim et al. 2022) successfully implemented motion generation neural models using the Denoising Diffusion Probabilistic Models (DDPM) (Ho) et al. 2020) setting, which was originally suggested for image generation. MDM enables both high- quality generation and generic conditioning that together comprise a good baseline for new motion generation tasks. EDGE (T seng et al.
2022) followed MDM by extending it for the music-to-motion task. SinMDM (Raab et al. 2023) adapted MDM to non-human motions using a single-sample learning scheme. PhysDiff Yuan et al. 2022) added to MDM pre- pre-trained physical model based on reinforcement learning which enforces physical constraints during the sampling process. These examples demonstrate the flexibility of MDM to novel tasks. In line with our motion control context, Jiang et al. (2022). Du et al. (2023) and Castillo et al. (2023) reconstruct full-body motion from a headset and hand controllers for Virtual Reality applications. While they focus on predicting the original motion, our approach models motion distribution based on one or a few joints.
Long-Sequence Motion Generation. Motion Graphs (Kovar et al., 2008 can synthesize long mo- tions via traversing discrete poses given a data corpus. This approach is limited to existing data and will fail to generalize for elaborate textual conditions. RNN-based motion generation tends to collapse into constant poses. Martinez et al. (2017) and (Zhou et al. (2018) overcome this issue by feeding the model with its own generated frames during training for the task of prefix completion. Yet, those methods are still limited to the relatively short sequences of the available data. More recently, several works suggested breaking the data limitation by regrerregess generating short sequences each one conditioned sitions were either learned according to a smoothness prior (Athanasiou et al.] 2022 Mao et al.] 2022] or from data (Athanasiou al./ 2022. Wang et al.] 2022), using the BABEL dataset (Pun- nakkal et al., 2021). which explicitly annotates transitions between actions. EDGE (Tseng et al.) 2022) suggested the unfolding method to generate long sequences with SLERP interpolating be- tween every two neighboring sequences. Contrarily, our Double Take suggests an unfolding method that leverages diffusion and blends the motion together at each denoising step. More recently, Dif- fCollage (Zhang et al. (023) suggested a diffusion baadsed solution for the task, utilizing a factor graph representation arranged as a linear chain.
textual prompt and the suffix of the previous sequence. Tran-
Multi-Person Motion Generation. Data scarcity is major obstacle for multi-person motion gen- eration, and the number of works is limited accordingly. MuPoTS-3D dataset (Mehta et al. 2018) includes 20 real-world multi-person sequences; CMU-Mocap (CMU) and 3DPW (Von Marcard ct al. 2018) includes 55 and 27 two-person motion sequences respectively. Yin et al. (2018) sug- gested overcoming the data barrier by exploiting 2D information. Recently, Song et al. (2022) contributed the synthetic multi-person GTA Combat dataset. None of the datasets is textually (or otherwise) annotated, hence, the recent MRT (Wang al., (021) and SoMoFormer (Vendrow al., 2022) models learned the unsupervised prefix completion task. Both learned motions under the DCT transform, which promotes smoothness and unrealistic motion, although improving L2 error mea- sures. Recently, DuMMF (Xu et al., 2022) presented a stochastic approach for multi-person motion completion. Concurrent to this work, InterGen (Liang et al. 2023) follow up and extend the two instances' communication principle presented in ComMDM and conducted a new benchmark.
Human Motion Priors. VPoser (Pavlakos et al., 2019) is human pose auto-encoder, trained on the AMASS motion capture dataset (Mahmood et al. 2019). It is used as prior for motion applications, such as motion demoising, fitting SMPL (Loper et al. 2015) model to joint location and as a pose code book for motion generation (Hong et | (022). More recently, Tiwari et al. (2022) showed that such prior can be learned as an implicit model. MoDi (Raab et al., (022) is an unsupervised motion generator, adapted from StyleGAN (Karras et al. 2019). Without further training, it enables latent space editing and motion interpolation. Contrary to those examples, MotionCLIP (Tevet et al. 2022) uses priors from the image and text domains to learn motion. It aligns the motion manifold with CLIP (Radford et al., 2021) latent space. This enables inheriting the knowledge learned by CLIP to generate motions out of the data limitations. In the diffusion context, MDM adapts diffusion image inpainting (Song et al., 20200: [Saharia et al. (022) for motion editing applications. In this work, we extend this principle by solving non-trivial motion tasks in few to zero-shot settings. More recently, MLD (Xin et al.) 2022) learned latent diffusion model, similar to LDM (Rombach et al., 2022b). which enables generating motion latent code instead of the motion itself, and lets a larger and pre-trriie pre-trained motion generator translate it into the physical space.
3

---

Published as a conference paper at ICLR 2024
Tokes #1 Handdehaika Censealion
Take #2 Transitinn Refrrment
MDM
bott Soltaame
Than
X
%x
Deep
"Long wap loweert
Unlolcing
DJ D
ComMDM
D D
bot orme
Thor
MDM
X
IN
L--1
Ln
kn=1
OUT
Take overview. We generate arbitrarily-long sequences with text and length con- trol per interval using fixed motion diffusion prior. At the first take, we generate each interval as sin- gle sample handshaking neighboring samples. At each denoising iteration, the handshakes are forced to be equal to eventually compose one long se- quence. To refine the transition between intervals, the second take partially noise the handshakes and clean them conditioned on the neighboring intervals using a soft mask. Solid frames mark generation or refinement; Dashed frames mark input motion to the
Figure 3:
Figure 4: ComMDM overview. Using two fixed MDM models, train slim commu- nication block (ComMDM) for two-person motion generation. ComMDM gets input the activations of transformer layer L, from both actors and outputs correction term which is added to the same activations. Op- tionally, ComMDM also predicts the initial poses D of the two persons. IN and OUT stand for the linear input and output layers of the transformer.
take.
METHOD
In this work, we use the recent Motion Diffusion Model (MDM) (Tevet al., 2023), pre-trained for the task of text-to-motion, to learn new generative tasks. We repre- sent Human Motion as a sequence of poses X x= {> N where x² RDD represent a single pose. Specifically, we use the SMPL (Loper et al., 2013) representation for
experiments with the BABEL (Punnakkal et al. (021])
Figure Soft blending overview. We allow b frames long linear masking be- tween Mhard to Mgoft such that during the Second take at every denoising step part of the originally generated motion (suffix or prefix) going through refine- ment to fit the transition.
dataset, including joint rotations and global positions on
top of a single human identity (3 = 0). For all other ex-
periments, we use the HumanMI.3D> (Guo et al.) 2022) representation, composed of joint positions, rotations, ve- locities, and foot contact information. MDM is a denois- ing diffusion model based on the DDPM (Ho et al. 2020) framework. It assumes T noising steps modeled by the stochastic process
(X, = NN -1.,)) a1)
(1)
for a noising step &. E T, were X7 -N is assumed. MDM models the denoising process: it predicts the clean motion X, given a noised motion X,, a noise step and textual condition encoded to CLIP (Radford et al.) 2021 space and represented by c. The model is learned with the MDM(Xx [, together with geometric losses that regulate the joint position, velocity and foot contact. Sampling novel motion from MDM is done in iterative manner, according to Ho et al. (2020). In every time step the clean sample X₂ predicted and noised back to X,-1- This is repeated from t. until X, is achieved.
standard Lximple E Xx,) (1).)
I| lt
In this Section, we present sequential composition with the DoubleTake method (3.I. which gen- eralizes MDM to generate motions arbitrary length without further training. through sequential composition. Then, we present parallel composition by employing slim communication layer,

---

Published as a conference paper at ICLR 2024
ComMDM (.2), trained with few as 10 interaction samples, for generating two-person motion. Lastly, we fine-tune MDM to control specific joints and present our model composition method, Dif- fusionBlending (3.3). that generalizes the classifier-free approach (Ho & Salimans 2022) to achieve fine-grained control over the body with any cross combination of joints to be controlled.
3.1 LONG SEQUENCES GENERATION
Our goal is to generate arbitrarily long motions, such that each time interval of the motion is poten- tially controlled with different text prompt and different sequence length. We want the transitions between intervals to be realistic and to semantically match the neighboring intervals. Since avail- able datasets are limited in motion length and often do not explicitly include transitions, we suggest approaching this task in a zero-shot manner, using a fixed generative prior that was trained with such short sequences,
We present DoubleTake (Figure 3 a two-stage inference-time process that suggests a parallel so- lution and generates the long motion in single batch. Typically, approaches that were designed specifically for this task (Athanasiou et al.] 2022; Mao et al., 2022] Wang et al. (022) generate each such interval conditioned on the fixed suffix the previous interval. In contrast, DaubleTake generates a prompted interval while observing both the previous and next intervals, which are gener- ated simultaneously. In the first take, we generate each interval as a different sample in the denoised batch, such that each one is conditioned on its own text prompt and maintains a handshake with its neighboring intervals through the denoising process. Handshake, т, is defined as a short (about a second long) prefix or suffix of the motion, such that the prefix of the current motion is forced to be equal to the suffix of the previous motion. Each interval maintains two such handshakes as demonstrated in Figure E] The handshake is maintained by simply overriding T with the frame-wise average of the relevant suffix and prefix at each denoising step. This allows our model to generate long sequences that depend on the past and future motions while being aware of the whole sequence during the generation of cach interval. The handshake length h = |T| can be arbitrarily defined by the user, also on a per-transition level. However, in practice, we find that the choice of one-second- long handshakes is robust throughout our experiments. Formally, handshakes are forced to be equal the end of each denoising iteration as follows:
T (1 a) St-1l h: +1++++000 ā @ S[ h]
where S; indicates the 7th sequence aj = j/h, : E j [0: h) and @ indicates element-wise
multiplication.
Looking the generated handshaked motion however, we observe visually displeasing results, as artifacts and inconsistencies occur in the transitions between semantically different motions (i.c. "Run" and then "Crawl"). Consequently we suggest adding the second take, applied on the output of the first take. In the second take, we reshape our batch as shown in Figure such that in each sample we get the transition sandwich (S,, T, S,+1). Now, we partially noise the sandwich T" noising steps and denoise it back to = 0 under our suggested seft-masking feature to refine transitions: In regular inpainting mask, the content is either taken completely from the input, or is completely generated. We suggest soft inpainting scheme, where each frame is assigned a soft mask value between 0 and 1 that dictates the amount of refinement the second take performs on top of the first take's result. To this end, define the masks Msoftt Mpard for the interval S and hanshake respectively, with a short, b frames long, linear transition between the mask values as demonstrated in Figure 2 Finally, we construct the long sequence by unfolding it, i.e. by reshaping each sequence and transition back to its linear place as demonstrated in Figure bottom.
3.2
GENERATION
Our goal is to simultancously generate motion of two people interacting with each other. The limited data availability dictates few-shot learning solution. Our key insight is that by dedicating fixed generator for each person in the scene the motion remains in the human motion distribution, and we only need to learn coordinate between the two. Hence, we introduce ComMDM (Figure 4 single-layer transformer model that is trained to coordinate between two instances of a fixed MDM (one for each person). ComMDM is placed after transformer layer n, gets as inputs the output activations of this layer from both models (O. (0).(0)) " (2.( and outputs a correction term for each of the two models (0), To further reduce the number of learned parameters, we exploit symmetry
5

---

Published as a conference paper at ICLR 2024
considerations and output only correction term, then the output to be corrected is entered first, such that the corrected output is Óz,(n) = 02((m) + Co some datasets, such HumanML3D, all motions are processed to start with the root at the origin and facing the same direction. Hence, naively using ComMDM on a model that was trained with such data will result in two people both being placed at the origin at the beginning of the motion. To mitigate that, ComMDM additionally learns D, the initial pose each person at the first frame as part of the diffuusio process. Hence, the full implementation of ComMDM is A0,(0)) Dj ComM (M , O.,-,((),, D2, t)-
O,-$((m). We note that in
We freeze the weights of the MDM instances and train only ComMDM with the Lsimple loss. We learn two motion tasks; For prefix completion, we use a fine-tuned version of MDM for prefix completion (See 3.3) and completely mask the textual condition. For the text-lo-motion task, we use a regular instance of MDM and mask the textual condition with a probability of 10% to support classifier free guidance.
3.3 FINE-TUNED MOTION CONTROL
Our goal is to generate full-body motion controlled by user-defined set of input features. These features can be root trajectory, single joint, or any combination them. We require a self-coherent generation that semantically adheres to the control signal. For instance, when specifying the root trajectory of a person to move backward, we expect the generated motion to have the legs adjusted to walking backward. As we show in Section 4.3 the motion in in-painting method suggested by Tevet iet al. (2023) fails to meet this requirement.
Single Control Fine-Tuning, Consequently, inspired by Rombach et al. 2022a), we introduce fine-tuning process to yield a model that adheres to the control features. In essence, our method works by masking out the noise applied to the ground-truth features we wish to control, during the forward pass of the diffusion process. This means that during training, the ground-truth control features propagate to the input of the model, and thus, the model learns to rely on these features when trying to reconstruct the rest of the features. Algorithm[|] (eerrtthm[]decrreee describes the fine-tuning process for trajectory control task. For sampling, we follow the core idea of the finetuning process: After we get the model's prediction of To. we inject the editing features into it. Then, in the forward process from the predicted 2g to -1, we mask out the noise in the control features to allow them to cleanly propagate into the model. Algorithm 2 defines this sampling process for trajectory control task. The fine-tuning stage requires less than 20K steps to generate visually pleasing results. It allows us to easily acquire dedicated model for given control task.
DiflusionBlendingg A fine-tuned model for every possible control task is sub-optimal. Hence, we suggest DiffusionBBlendnng, model composition method for using multiple models for composite control tasks. For instance, if we wish to dictate both the trajectory of the character and its left hand, can blend the model that was trained solely for trajectory control and the model that was trained only for the left hand.
To control cross combinations of the joints (i.e. both the root and the end effector as in Figure T we extend the core idea of the classifier-free approach (Ho & Salimans| 2022) and present Dif- fusionBlending. The classifier-free approach suggests interpolating or extrapolating between the conditioned model G' and the unconditioned model Go We argue that this idea can be generalized to any two "aligned" (see definition in (Wu et al. (021]) ) diffusion models G' and G that are condi- tioned on C, and с respectively. Then sampling with two conditions simultaneously is implemented as Go. (X,, L, Ca, (b)) = GO (Xe, L, ca) + 8- (G (Xe, I, ca) G ("(,, 4, (x))), with the scale parameter s trading- off the significance of the two control signals.
For the general case of N joints corresponding to N fine-tuned models {GT). the Diffusion- Blending will be defined, G!Ml(XX,
(Xe: ( = - 8n Ga (Xt,, Cn)), where ΣN 8n 1.
EXPERIMENTS
4.1 LONG SEQUENCES GENERATION
For long sequence generation with our DoubleTake method, we use fixed MDM (Tevet et al. 2023) trained on the HumanMI.33 (Guo et al. 2022) dataset, originally trained with up to 10 seconds long

---

Published as a conference paper at ICLR 2024
Mation
Transition
frames) Diversity++
Tranittion frames)
R-preeision
FID]
Divensity
MullMModa-DDstt
FID
FIDI
Diversity
Ground Truth
0.4. 10-3
8.51
3.57
10-3
0.9 10-m
TEACH
0.46
1.12
8.28
7.14
3.56
7.62
7,503
6.30
Double Take (ouns)
0.10
1.01
8.14
7.35
1.88
7.00
3.45
7.19
Emb
3.63
+ Trans. Emb geo losses
0.45
0.91
5.16
7,00
2.30
2.18
6.05
6.52
Table 1: Quantitative results on the BABEL test set. All methods use the real motion length from the ground truth. "-' means results are better if the metric is closer to the real distribution. We run all the evaluations times. Transition metrics were tested on two different lengths and it contains context from the suffix of the previous frame and the prefix of the next frame. tested on two different margin lengths since TEACH define transition of frames. Bold indicates best result, underline indicates second best result. R-precision reported is top-3.
motions. To compare with TEACH (Athanasiou et al. 2022). which was dedicatedly trained for this task, we train MDM for 1.25M steps on BABEL (Punnakkal al. [02T], the same dataset TEACH was trained on with the same hyperparameters suggested by [Tvvet et al. (023) on a single NVIDIA
GeForce RTX 2080 Ti GPU. For both datasets, we applied Double'Tak with a one-second-long transition length, T" 700, Mhard 0.85, Mzoft 0.1 and 6 10.
In both cases, we evaluate the generation using the evaluators and metrics suggested by Guo et al. (2022). In short, they learn text and motion encoders for the HumanML3D dataset as evaluators that map motion and text to the same latent space, then apply a set of metrics on the generated motions they are represented in this latent space. R-precision measures the proximity of the motion to the text it was conditioned on, FID measures the distance of the generated motion distribution to the ground truth distribution in latent space, Diversity measures the variance of generated motion in latent space, and MultiModel distance is the average 1.2 distance between the pairs of text and conditioned motion in latent space. For full details, we refer the reader the original paper. We further conducted user study comparing DoubleTake to TEACH. DoubleTTake was preferred in about 82 to 85% of the times. For more details please refer to Appendix C
Note that for the BABEL dataset, we trained the same evaluators following the setting defined by Guo et al. (2022). To provide a proper analysis, we generate a 32- 32-intervals long sequence, then apply HumanMI metrics on the intervals themselves, and once again for the transition. Note that the text-related metrics are not relevant for transitions.
Since the BABEL dataset annotates transitions as well, we suggest using our Transition Embedding: we choose to embed each frame with transition embedding signal, allowing the model better understand if the following frame belongs to transition or part of the motion. We then add this embedding to the frame's 's features. Addi- tionally, we choose to train model over the BABEL dataset with geometric losses as proposed in MDM. We note that whereas we do not apply any post-process to the motion, TEACH aligns the start of each interval to the end of the previous one and adds extra interpolation frames between the two. We observe that without this post-process TEACH produces poor transition, yet evalu- ated it with all the above to maintain fair conditions.
Take
Do.MeTake
Figure 5: DoubleTake transition re- finement. The second take refines the transitions generated in the first take to be more smooth and more realistic. Or- ange are subsequent transition frames and Blue context intervals.
Table presents quantitative results over the BABEL dataset, compared to TEACH. We evaluated the transi- tions with two variations the first with fair margins from the intervals (70 frames) and the other with minimal possible margins for both Taue and TEACH (30 frames which are 1 second). DoubleTake outperforms TEACH in terms of FID with all our methods. When considering transition evaluations the gaps in favor of DoubleTake larger. Figure (Appendix shows a qualitative comparison between the two approaches. Compar- isons to DiffCollage (Zhang al. 2023) were not performed since code is not available.
even
7

---

Published as a conference paper at ICLR 2024
Serice
Vesaatio
K pecisionn
FID
Diverstty
Gmoan listh
Specttrre
0.2
0,29
18"10"0
Fin
0.89
3.85
8:30
2.34
5.81
FIT)
0,05
1:28
136
Trajectory
Grnanl Truih
XID
Preestuned (Oun)
0.64
0.80
1-0-10
0.54
0:35
9:35
5005
9:76
Seorediokz.atty
050
1.0
1.00
1:00
9.31
933
5.57
511
3.21
7:0
6:55
8100
Left Wrist
XIDM
Fracstunal (Oum)
0.84
0.31
0.82
0:31
9:41
039
0158
9.24
5.00
1.30
6.75
Left Wrist
XDM
1115
5.3
Trajectory
DifusinBleninng (Oun)
0.07
0.22
0.33
038
1.36
9.81
5.0
2/0
Left Wrist
0.63
0.81
8.84
DuskieTike (Ma 0.50
022
0050
0.00
3.00
3775
6.00
5.06
120
K0
8m
Right DiflssonBlendnng (Ouns)
0.07
0.18
9.35
Table 2: Quantitative results on the Hu- manML3D test set. All methods use the real motion length from the ground truth. '-x' means results are better the metric is closer to the real distribution. We run all the evaluations 10 times. Bold indicates best result, underline indicates second best result. R- R-precision re- ported is top-3, Div. stands for diversity and M.- Dist for Multi-modal distance.
Table 3: Joints control with fine-tuned mod- els and DiffusionBlending. We compare our joints control method with the motion inpaint- ing method suggested by Tevet et al. (2023). We conduct the evaluation on HumanML3D test set. '+' sign represents a blending of two fine-tuned models with our DiffusionBlending method.
Table 2 presents ablations for the DoubleTake hyperparameters over the HumanML33 dataset. We show that our method using DoubleTTake soft masking, and one-second handshake size achieves the best results. Figure shows qualitatively how the second take refines the first take.
Root Error [m]
Joints Error [m]
2x
3s
MRT
0.28
0.49
0.62
0.32
0.53
0.65
ComMDM (ours)
0.24
0.38
0.43
0.32
0.44
0.19
MDM (na Com)
0.27
0.57
0.7
0.33
0.62
0.82
Com only
0.25
0.40
0.48
0.33
0.47
0.54
CoMMDM Zlayers
0.24
0.38
0.43
0.32
0.44
0.19
ComMDM
4layers
0.25
0.40
0.45
0.35
0.47
0.52
Intoraction
55,0%
ComMDM @ layerb
0.26
0.40
0.43
0.45
0.48
Ours vs.
Completion
61.0%
ComMDM 6 layerd
0.27
0.42
0.46
0.33
0.47
0.51
Quality
ComMDM @ layer2
0.26
0.40
0.45
0.33
0.46
0.50
ComMDM @ layerO
0.24
0.40
0.45
0.32
0.47
0.51
Interaction
Complction
Quality
79.2%
Table 4: CMU-mocap prefix completion L2 error.
Given a 1-second long prefix, all models predict 3- second long motion completion. We report the root and joints mean error for the first 1, 2. and 3 seconds. Bold indicates best result, underline indicates second best. We introduce two ablation studies, the first is for the number of layers constructing ComMDM (ours is 1), placed
Outs ys
Interaction 40.0%
Completion 36.2%
Truth
Quality
Figure 6: 3DPW two-person prefix completion user study. We asked users compare our ComMDM to the orig- inal MDM, MRT model, and ground truth in a side-by-side view. The dashed line marks 50%. ComMDM outper- forms both MRT and MDM in all three aspects of generation.
and the second is in which layer of MDM (ours is in the 8th). Observe that the communication block performs better when placed in higher layers of the transformer and constructed from fewer layers.
4.2
GENERATION
Due to the limited availability of data, we learn two-person motion in few-shot manner. We use fixed MDM trained on the HumanML3D dataset and learn a slim communication block, CoMMDM, as described in Section
Data. We train and evaluate ComMDM with the CMU-Mocap (CMU) and the 3DPW (Von Marcard al. (018) datasets, which contain 55 and 27 two-person motion sequences respectively. annotated with SMPL joints. We omit the 3DPW test set since it is noisy and does not include any meaningful human interaction. We augment the data by randomly mirroring and cropping each sequence. Then, process the data to the HumanML3D joint representation, for compatibility with the original MDM imput format. We train ComMDM for two different generation tasks, both with batch size 64 on single NVIDIA GeForce RTX 2080 Ti GPU.
Prefix completion. We follow MRT (Wang et al. 2021) and learn to complete 3 seconds of motion given I-second prefix. Table 4 presents the root and joints mean L.2 emror considering the ablation
8

---

Published as a conference paper at ICLR 2024
study presented, we placed the communication layer in the 8th and last layer of the transformer. We train ComMDM for 240K steps. We retrain MRT and observe that our data process alone improved the results originally reported by the authors. ComMDM outperforms MRT in terms of L2 error and generates visibly more meaningful completions, as presented in Figure 7 (Appendix). We further
conducted user study comparing ComMIDM to MDM, MRT, and ground truth data, according to the aspects of interaction level, completion of the prefix, and overall quality of the generated motion. 30 unique users were participating in the user study. Each model was compared through 10 randomly sampled prefixes and each such comparison was repeated by 10 unique users. The results (Figure () show that the motions generated by ComMDM were clearly preferred over MRT and MDM. Figure 11 (Appeendix) shows an example screenshot from this user study.
ComMDM
Text-to-Motion. We argue that prefix completion is a motion task that becomes irrelevant. is an explicit control signal that is both limiting the motion and giving a too-large hint for the generation. Additionally, reporting joint error promotes dull low-frqquency motion and discourages learning the distribution of motion given a condition. Hence, we make first step toward text control for two-person motion generation. Since no multi-person dataset is annotated with text, we contribute textual annotations for each training sample of both datasets, and train ComMDM for 100K steps. Figures| I and (Appendix) present diverse motion generation given unseen text prompts. We note that due the small number of samples, generalization is fairly limited interactions from the same type seen during training.
4.3 FINE-TUNED MOTION CONTROL
We compare our fine-tuned models and the DiffusionBlending sampling method with the original MDM model on various control tasks. For that sake, we sample text and control features according to each task from the HumanMI.3D test set. Motion is generated with the original MDM model by injecting the control features using the original inpainting method suggested by (Tevet et al., 2023). We then generate motions with the fine-tuned model that was trained for a specific control task, using our proposed inpainting method, All fine-tuned models were initialized with the same original MDM we compare with, and trained with our finetuning method for 80K steps, with a batch size of 64. Note that we consider the trajectory to be the angle of the character on 22 plane and its linear velocities in that plane (we don't 't include the vertical position). In the joint control tasks, we take the relative location of the joint with respect to the root location. For composite tasks such as left trajttory and left wrist+right foot, we apply DiffusionBlending method on the two corresponding fine-tuned models with equal weights (s = 0.5). All control experiments conducted on HumanML3D dataset, with text-conditioning and classifier-free guidance scale of 2.5. Quanti- tative results presented in Table 3 and qualitative results demonstrated in Figure 9 (Appendix). As can be seen, fine-tuning MDM is crucial for the control task, and produces high-quality results.
5 CONCLUSION
In this paper, we have shown that motion-based based prior can be employed for advanced motion gen- cration and control, using three novel composition methods. We have leveraged the diffusion ap- proach itself for the task, and have shown that it lends itself naturally to composition, enabling new tasks with little to no new data. Conceptually, we argue that the diffusion-based generative model can serve as a prior, or a proxy, to the human motion manifold, and thus the advanced techniques only need to address the integration between the parts being composed, relying on the fact that the generated motion is always projected back to the motion manifold. While promising, this initial approach is still in its infancy, and much can be further investigated. In long-sequence generation, for example, we are still limited to the quality of the initial model and the motion may suffer in- consistencies between distant intervals. In two-person motion generation, ComMDM does well at synchronizing motions between two priors, but only for interactions seen during training, lacking generalization. Based on the single-person synthesis case, we expect this approach as well to scale with larger datasets in the future. Nevertheless, two-person synthesis brings new challenges yet to be addressed. For example, future methods should allow for valid contacts between people. Lastly, we note the proposed techniques are not specifically designed for the motion domain. Hence per- haps the most promising avenue for future work is to adapt the techniques described in this paper to other fields of generation, as well as to investigate additional ways to combine the vast knowledge embedded in pretrained generative models for novel tasks.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
We extend our gratitude to Prof. Michiel Van de Panne for his invaluable guidance, and insightful suggestions, which have significantly enriched the quality and rigor of this paper. We thank Chuan Guo and Nikos Athanasiou for their technical support and useful advice. We thank Sigal Raab, Roy Hachnochi and Rinon Gal for the fruitful discussions. This research the Israel Science Foundation (grants no. 2492/20 and 3441/21), Len Blavatnik and the Blavatnik family foundation, and The Tel Aviv University Innovation Laboratories (TILabs). This work was supported by the Yandex Initiative in Machine Learning.
supported in part by
REFERENCES
Cmu graphics lab motion capture database, http://mocap.cs. www.ccccccceeeoomm
Adobe Systems Inc. Mixamo, 2021. https:////w
com
Accessed: 2021-12-25.
Nikos Athanasiou, Mathis Petrovich, Michael J. Black, and Gül Varol. TEACH: Temporal Action Compositions for 3D Humans. In International Conference on 3D Vision (3DV), 2022.
Angela Castillo, Maria Escobar, Guillaume Jeanneret, Albert Pumarola, Pablo Arbeláez, Ali Thabet, and Artsiom Sanakoyeu. Bodiffusion: Diffusing sparse observations for full-body human motion synthesis. arXiv preprint arXiv:2304.1113 11118, 2023.
Rishabh Dabral, Muhammad Hamza Mughal, Vladislav Golyanik, and Christian Theobalt. Mo- fusion: A framework for soooising-diffusion IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9760-9770, 2023.
motion synthesis. In Proceedings of the
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4171-4188 Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10. 10.1863//1/N11-142233 URL https: //aclaathology.ozg///
Yuming Du, Robin Kips, Albert Pumarola, Sebastian Starke, Ali Thabet, and Artsiom Sanakoyeu. Avatars grow legs: Generating smooth human motion from sparse tracking inputs with diffusion model. In Proceedings the the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 481-490, 2023.
Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang. Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5152-5161, 2022.
Jonathan Ho and Tim Salimans. 12Xi::2007.12598 2022.
Classifier-free diffusion guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Fangzhou Hong, Mingyuan Zhang, Liang Pan, Zhongang Cai, Lei Yang, and Ziwei Liu. Avatarclip: Zero-shot text-driven generation and animation of 3d avatars. ACM Transactions Graphics (TOG), 41(4):1-19, 2022. doi: 10. 00.114553522223.333000
Jiaxi Jiang, Paul Streli, Huajian Qiu, Andreas Fender, Larissa Laich, Patrick Snape, and Christian Holz. Avatarposer: Articulated full-body pose tracking from sparse motion sensing. In European Conference on Computer Vision, pp. 443 460. Springer, 2022.
Hanbyul Joo, Hao Liu, Lei Tan. Lin Gui, Bart Nabbe, lain Matthews, Takeo Kanade, Shohei Nobuhara, and Yaser Sheikh. Panoptic studio: massively multiview system for social motion capture. In The IEEE International Conference on Computer Vision (ICCV), 2015.
10

---

Published as a conference paper at ICLR 2024
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings the recognition, pp. 4401-4410, 2019.
the IEEE/CVF conference on computer vision and pattern
Jihoon Kim, Jiseob Kim, and Sungjoon Choi. Flame: Free-form language- motion synthesis & editing, arXiv preprint arXiv:2209.000) 0934999 2022.
Lucas Kovar, Michael Gleicher, and Frédéric Pighin. Motion graphs. In ACM SIGGRAPH 2008
classes, pp. 1-10. 2008.
Han Liang, Wengian Zhang, Wenxuan Li, Jingyi Yu, and Lan Xu. Intergen: Diffusion-based multi- human motion generation under complex interactions. arXiv preprint arXiv:2304.05664, 2023.
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi-person linear model. ACM transactions on graphics (TOG), 34(6):1-16, 2015,
Naureen Mahmood, Nima Ghorbani, Nikolaus F. Troje, Gerard Pons-Moll, and Michael J. Black. AMASS: Archive of motion capture surface shapes. In International Conference on Computer October 2019.
Vision, pp. 5442-5451,
Wei Mao, Miaomiao Liu, and Mathieu Salzmann. Weakly-supervisee action transition learning for stochastic human motion prediction. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8151-8166 2022.
Julieta Martinez, Michael Black, and Javier Romero. On human motion prediction using recur- rent neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2891-2900, 2017.
Dushyant Mehta, Oleksandr Sotnychenko, Franziska Mueller, Weipeng Xu, Srinath Sridhar, Gerard Pons-Moll, and Christian Thcobalt. Single-shot multi-person 3d pose estimation from monocular rgb. In 2018 International Conference on 3D Vision (3DV), pp. 120-130. IEEE, 2018
Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dim- itrios Tzionas, and Michael J. Black. Expressive body capture: 3D hands, face, and body from single image. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 10975-10985, 2019.
Mathis Petrovich, Michael J. Black, and Gül Varol. TEMOS: Generating diverse human motions from textual descriptions. In European Conference on Computer Vision (ECCV), 2022.
Abhinanda R. Punnakkal, Arjun Chandrasekaran, Nikos Athanasiou, Alejandra Quiros-Ramirez, Ramirez, and Michael J. Black. BABEL: Bodies, action and behavior with english labels. In Proceedings IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 722-731, June 2021.
Sigal Raab, Inbal Leibovitch, Peizhuo Li. Kfir Aberman, Olga Sorkine-Hornung Hornung, and Daniel Cohen- Or. Modi: Unconditional motion synthesis from diverse data, arXiv preprint arXiv:2206.0800 2022.
Sigal Raab, Inbal Leibovitch, Guy Tevet, Moab Arar, Amit H Bermano, and Daniel Cohen-Or. Single motion diffusion. arXiv preprint arXiv:2302.05905 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 9748-8763. PMLR, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), pp. 10684-10695, June 2022a.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CYF Con- ference Computer Vision and Pattern Recognition, pp. 10684-10695, 2022b.

---

Published as a conference paper at ICLR 2024
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to- maage-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings, pp. 1-10, 2022.
Jascha Sohl- Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learn- ing. pp. 2256-2265. PMLR, 2015.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:201 1. 13456, 2020.
Ziyang Song, Dongliang Wang, Nan Jiang, Zhicheng Fang, Chenjing Ding, Weihao Gan, and Wei Wu. Actformer: gan transformer framework towards general action-conditioned 3d human motion generation. arXiv preprint arXiv:2203.07706, 2022.
Guy Tevet, Brian Gordon, Amir Hertz, Amit H Bermano, and Daniel Cohen-Or. Motionclip: Ex- posing human motion generation to clip space. In Computer Vision-ECCV 2022: 17th Euro- pean Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXII, pp. 358-374. Springer, 2022.
Guy Tevet, Sigal Raab, Brian Gordon, Yoni Shatir, Daniel Cohen-or, and Amit Haim Bermano. Human motion diffusion model. In The Eleventh International Conference on Learning Repre- sentations, 2023. / httpa//wwwwe..tteefoooooooosssss
Garvita Tiwari, Dimitrije Antić, Jan Eric Lenssen, Nikolaos Sarafianos, Tony Tung, and Gerard Pons-Moll. Pose-ndf: Modeling human pose manifolds with neural distance fields. In European Conference Computer Vision, pp. 572-589. Springer, 2022.
Jonathan Tseng, Rodrigo Castellon, and Karen Liu. Edge: Editable dance generation from music. arXiv preprint arXiv:2211. 2022.
Edward Vendrow, Satyajit Kumar, Ehsan Adeli, and Hamid Rezatofighi. Somoformer: Multi-person pose forecasting with transformers. arXiv preprint arXiv:2208. 14023, 2022.
Timo Von Marcard, Roberto Henschel, Michael J Black, Bodo Rosenhahn, and Gerard Pons-Moll. Recovering accurate 3d human pose in the wild using imus and a moving camera. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 601-617, 2018
Jiashun Wang, Huazhe Xu, Medhini Narasimhan, and Xiaolong Wang. Multi-person 3d motion prediction with multi-range transformers. Advances in Neural Information Processing Systems, 34, 2021.
Weiqiang Wang, Xuefei Zhe, Huan Chen, Di Kang, Tingguang Li, Ruizhi Chen, and Linchao Bao. Neural marionette: A transformer-basee based multi-action human motion synthesis system. arXiv preprint arXiv:2209.1 2022.
Zongze Wu. Yotam Nitzan, Eli Shechtman, and Dani Lischinski. Stylealign: Analysis and applica- tions of aligned stylegan models. arXiv preprint arXiv:2110.11323; 2021.
Chen Xin, Biao Jiang, Wen Liu. Zilong Huang, Bin Fu, Tao Chen, Jingyi Yu, and Gang Yu. Execut- ing your commands via motion diffusion in latent space. arXiv, 2022.
Sirui Xu, Yu-Xiong Wang, and Liangyan Gui. Stochastic multi-person 3d motion forecasting. In The Eleventh International Conference on Learning Representations, 2022.
Kangxue Yin, Hui Huang, Edmond SL Ho, Hao Wang, Taku Komura, Daniel Cohen-Or, and Hao Zhang. A sampling approach to generating closely interacting 3d pose-pairs from 2d annotations. IEEE transactions visualization and computer graphics, 25(6):211722277, 2018.
Ye Yuan, Jiaming Song. Umar Iqbal, Arash Vahdat, and Jan Kautz. Physdiff: Shyyics-guidee human motion diffusion model. arXiv preprint (rxXi:22/2.025000 2022.
12

---

Published as a conference paper at ICLR 2024
Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondiffuse: Text-driven human motion generation with diffusion model. arXiv preprint arXiv:2208. 15001, 2022.
Qinsheng Zhang, Jiaming Song, Xun Huang, Yongxin Chen, and Ming yu Liu. Diffcollage: Parallel generation of large content with diffusion models. In CVPR, 2023.
Yi Zhou, Zimo Li, Shuangjiu Xiao, Chong He, Zeng Huang, and Hao Li. Auto-conditioned recur- rent networks for extended complex human motion synthesis. In International Conference on Learning Representations, 2018.
13

---

Published as a conference paper at ICLR 2024
APPENDIX
A ADDITIONAL RESULTS
show additional two-person motion generation results with ComMDM for the tasks of text-to-motion and prefix completion correspondingly. Figure [0 qualitatively compares Our
Figures g
Double algorithm TEACH (Athanasiou et al. 2022). Figure 9 demonstrates fine-tuning
motion control.
Hugging prefix completion
MRT [2021]
ComMDM (Ours)
"The plrying bankettell, with bell the other delendinn
Warm-up up profix completion
MRT (2021)
ComMDM (Ours)
people
Figure Two-Person Prefix Completion. MRT (Wang et al. 2021) tends to fixate on the prefix pose whereas our ComMDM pro- vides lively and semantically correct comple- tions. Blue figures are the input prefix frames, provided to both models. The red and orange figures are MRT and our completions corre- spondingly.
Figure 8: Two-Person Text-to-Motion. We use ComMDM to generate two- wwo-person interac- tions given an unseen text prompt describing it. Different color defines different character, both are generated simultaneously.
B FINE-TUNING ALGORIHHM
Algorithm Fine-tuning method
Algorithm 2 Sampling method
repeat
=0 =
q (xra)
for t t wet=T T....,00 do
U~inform [1, (1.....7)) T})
no (4) [trajectory] =given trajectory
(0, [trajectory] = 0
Original in-painting
> Our addition
N).,
Take gradient descent step on:
<[trajectory] =0
Our addition
Vel ée (Vārza vI aic,t)) I| until Converged
(t-1) 60 (Vorza + vI oee, t)
x
end for
C USER STUDYS
We conducted study of the two-person prefix completion task. Its details be found in Section 4.2 and the results are presented in Figure 6 Figure II presents a sample screenshot from the user study form.
14

---

Published as a conference paper at ICLR 2024
Trajactory control of walking circle
Nnw (asa)
Prictaned KDM (hun
walk jog forward wavy motion with hands cartwheel to the left
po09)
Hand
NOW (2622)
Fincure NDM KDrst
Double Telon (Cuns)
Fine-tuned Motion Control (unconditioned on text). We can see that MDM (Tevet et al.] 2023) generates motions that completely ignore the input features: jectory control MDM generates massive foot sliding, and in the hand control, the hand unre- alistically bends behind the back. Our finetuned models generate natural motions that semanti- cally and physically match the input features: In trajectory control we generate a walking mo- tion that fits the trajectory and in hand control, the model recognizes the swinging motion and generates a golf swing.
Figure 9:
DoubleTake compared to TEACH (Athanasiou al. 2022). We show that while our Double provides coherent motion with realistic transitions, TEACH gen- eration suffers from sliding.
Figure 10:
15

---

Published as a conference paper at ICLR 2024
GoT 1213
[ORANGE FRAMES ONLY) Which animation seems more
which animation do the orange frames continues the blue beginning
[ORANGE FRAMES ONLY) Which animation has better interaction
Figure 11: sample screenshot from the two-person prefix completion user study.
Quality
Transition Quality
85.0%
Motion Alignment
Figure 12:
Long sequences user study. We asked the users to compare DaubleTake to Athanaaooa et al. 2022). The dashed line marks 50%. DoubleTake outperforms TEACH in all three aspects of generation.
In addition, we conducted user study of the Long Sequences Generation task comparing Double- Take to TEACH (Athanasiou et al.] 2022). according to the aspects of (I) alignment with the text condition, (2) natural transitions, and (3) overall quality of the generated motion. 20 unique users were participating in the user study. The motions were generated randomly, sampling texts out of repeated by 10 unique users. The results (Figure [2]
the BABEL dataset. Each comparison show that our long sequences generated were clearly preferred by the users. Figure [3 presents sample screenshot from the user study form.
16

---

Published as a conference paper at ICLR 2024
2
Which zeiration signs bettr with the seat?
Which aiimatoon ownel, demonstites higher quilt??
Figure 13: A sample screenshot from the Long Sequence generation user study.
D ADDITIONAL PARAMETERS
The followings Table compares DoubleTake and TEACH (Athanasiou al. 2022 parameters and runtime.
number of
Attribute
DoubleTake
TEACH
Trainable Parameters
17.75M
28.84M
Inference time (10 seconds motion)
78 [s]
5.7 [s]
Inference time minutes motion)
15.7 [m]
2.5 [m]
Table 5: Attributes comparison between DoubleTake and TEACH
17


Reviews:
[{"review_id": "ywNt5HHxXt", "reviewer": "Reviewer_w1k5", "summary": "This paper presents off-the-shelf motion diffusion models as a priori for realizing three forms of synthesis tasks. doubleTake is used to generate long-term human motion. comMDM is used to generate two-person motion. DiffusionBlending is used to enable flexible and efficient fine-grained joint and trajectory level control and editing. Experimental results show that these low-cost composite methods generalize well-trained motion prior to different tasks and outperform previous specialized techniques in the respective tasks.", "questions": "1. The comparison with TEACH is not very fair either. Is it possible to have the same setting with TEACH but using DoubleMDM, e.g. noising the start and end intervals as diffusion inversions and then performing DoubleMDM with additional transition frames? In this case, you can use accuracy measurements instead of current metrics such as FID. This is because current metrics are more of a test of the quality of the movement and whether the movement is consistent with the text. It is also important to determine if the transition is good by measuring smoothness and accuracy.\n\nOverall, the author's response to the concerns is needed to make the final decision. I am also happy to increase the rating if my concerns are addressed.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Overall, the paper is well-written with a clear and well-motivated introduction.\n\nThe proposed method outperforms previous specialized techniques in the respective task. The experimental designs are comprehensive and show visually appealing results.", "weaknesses": "1. For the generation of long sequences, I don't think it makes sense to essentially generate each interval completely independently. A better option would be to use an autoregressive generation method similar to TEACH, but with the smarter option of combining each subsequence. Would it be possible to compare this with a scheme similar to EDGE [1]? Also, the paper admits that a comparison with DiffCollage is not possible due to a lack of publicly available code resources. Their implementation is simple. This may be optional, but would further support the paper.\n \n2. Regarding multiplayer motion generation, considering that MRT is an old paper, it could be considered to include a discussion and comparison with [1,2,3]. MRT focuses on deterministic prediction, and for the evaluation, is the prediction from ComMDM sampled once? A better comparison would be to use [2] as a baseline, to check the performance of diverse prediction and to measure diversity.\n\n[1] Tseng et al. EDGE. Edge: Editable dance generation from music. CVPR 2023\n\n[2] Xu et al. Stochastic Multi-Person 3D Motion Forecasting. ICLR 2023\n\n[3] Peng et al. Trajectory-Aware Body Interaction Transformer for Multi-Person Pose Forecasting. CVPR 2023\n\n[4] Liang et al. InterGen: Diffusion-based Multi-human Motion Generation under Complex Interactions. Arxiv 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oULgYxuA05", "reviewer": "Reviewer_ijG5", "summary": "In this paper, the authors address the challenges in human motion generation by introducing three new composition methods that use diffusion based generative models. The authors align these methods to 3 main challenges: long sequence generation, multi-person interactions, and controllable generation. The authors highlight that much of these problems arise from the lack of available data. The 3 proposed methods are sequential composition for long sequence generation, parallel composition for two-person motion generation, and model composition for fine-grained control and editing. The models involved in these methodologies are respectively DoubleTake for generating long motion sequences in a zero-shot manner, ComMDM to combine two frozen priors to enable two-person motion generation, and DiffusionBlending for flexible control of generated motion. The authors present several qualitative and quantitative results demonstrating the positive effects of their method.", "questions": "See weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I want to highlight the following strengths:\n- The main strength I see is that the methods presented work well without the need of generating more data (or consuming large amounts of unavailable data). This is a strong benefit, since the field of human motion generation is still lagging in terms of data availability. The authors demonstrate in all 3 cases that they can satisfy the task at hand requiring small amounts of extra data/training.\n- I see major novelty in the methods developed for long sequence generation and 2 person generation. Both methods have interesting new ways to combine different generations from diffusion models (one over time and one in space). Adding to it that the method doesn't require a lot of extra training, these proposed methods seem solid and novel to me.\n- The paper is well written, with good experiments on all fronts. The author's explanation of each method is easy to follow. I want to particularly highlight figures 3 and 4, where the choice of colors and graphics makes it very intuitive to understand.", "weaknesses": "My only concern is on the fine-tuned motion control part. The task seems very similar to controlled motion generation. In that case, there is a body of literature in this subject, many of which uses diffusion models for controlled motion generation. The authors failed to include these methods and compare against them. Of course these methods have different data requirements, but they seem to achieve the same goal. I put a list of these methods below. I ask the authors to explain why they did not include these methods in their comparisons? I'm still happy with the paper and I think the authors could make a case while still including these works, but I would like to hear from the authors on these choices.\n\n- Jiaxi Jiang, Paul Streli, Huajian Qiu, Andreas Fender, Larissa Laich, Patrick Snape, and Christian Holz. Avatarposer: Articulated full-body pose tracking from sparse motion sensing. ECCV, 2022.\n- Du, Y., Kips, R., Pumarola, A., Starke, S., Thabet, A., & Sanakoyeu, A. (2023). Avatars grow legs: Generating smooth human motion from sparse tracking inputs with diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 481-490).\n- Castillo, Angela, et al. \"BoDiffusion: Diffusing Sparse Observations for Full-Body Human Motion Synthesis.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HaPCuyfj8C", "reviewer": "Reviewer_xoQq", "summary": "This paper leverages a pretrained Motion Diffusion Model (MDM) as a generative prior and introduces three distinct techniques to enhance generation quality and enable new tasks: 1) DoubleTake is employed to reduce artifacts in long-range motion generation, enhancing the smoothness of transitions. 2) The introduction of ComMDM, a compact model integrated after the transformer layer in MDM, allows the model to handle two-person generation by freezing the original MDM and training this new module on a smaller data collection. 3) The authors also propose a fine-tuning method and DiffusionBlending to enhance controllability.", "questions": "Please kindly refer to the weaknesses mentioned above.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents compelling quantitative and qualitative results, setting a new state-of-the-art benchmark with a significant lead.\n\n2. This article broadens the scope of existing text-driven motion generation from three perspectives. The conclusions and experiments associated with these extensions are valuable contributions to the research community.\n\n3. The paper is well written, ensuring that its content is readily comprehensible to its readers.", "weaknesses": "1. The authors should conduct a user study to quantitatively compare the visual results of TEACH with the proposed method for long sequence generation. Model parameters and inference speed should also be provided for a more comprehensive performance comparison between the two.\n\n2. Dual-person motion generation lacks comparative experiments, for example, with InterGen \\[1\\]. This paper introduces the InterHuman benchmark, a large-scale dataset for dual-person motion, and provides more comparative references in the article\n\n\\[1\\] Han Liang, et al. InterGen: Diffusion-based Multi-human Motion Generation under Complex Interactions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2HoZlr3aDx", "reviewer": "Reviewer_1uD6", "summary": "This paper tackle the problem of motion generation. Given a pretrained diffusion based text-to-motion model i.e., MDM, this paper introduce three different way of adaptation to account for long sequence generation, multi-person motion generation and joint trajectory control.  For long sequence generation, two sequences are first generated individually and then blended with a fixed length overlapping. For two-person motion generation, a communication module (ComMDM) is trained to modify the intermediate features from two individual pre-trained MDM so as to coordinate the interaction between them. For trajectory control, several MDMs are frist finetuned to be given the trajectory of different joints. To achieve the control of multiple joints, the paper proposes to interpolate between two MDMs trained beforehand.", "questions": "For the DiffusionBlending, is the interpolation applied to the final motion or the intermediate feature as the CommMDM?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The motivation of limited long sequence data, multi-person interaction and detailed control is valid and convincing.\n- The proposed method is sound and interesting. \n- The experiments are comprehensive and the results on all three tasks are very good especially for the qualitative results.", "weaknesses": "- Despite the good results, technically, there are not much contributions. The main contribution is the way of using/finetuning a pre-trained motion diffusion model for three different tasks. However, in each of those tasks the adaptation is straightforward. They are either engineering ticks such as soft or hard masking or  the use of existing techniques such as the interpolation mechanism from Ho & Salimans, (2022).\n\n- The proposed ComMDM is constrained to two persons only. It is hard to scale to multiple persons.\n\n- It is also unclear how the DiffusionBlending can be used to blend more than 2 joints.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)