Multi-Track Timeline Control for Text-Driven 3D Human Motion Generation

Authors: alec  radford
jong wook kim
chris  hallacy
aditya  ramesh
gabriel  goh
sandhini  agarwal
girish  sastry
amanda  askell
pamela  mishkin
jack  clark
gretchen  krueger
mikhail  pavlov
scott  gray
chelsea  voss
mark  chen
ilya  sutskever
Venue: thecvf_com_CVPR_2024_Workshop_HuMoGen
Published: 2024-03-18 17:21:05

Abstract: Recent advances in generative modeling have led to promising progress on synthesizing 3D human motion from text, with methods that can generate character animations from short prompts and specified durations. However, using a single text prompt as input lacks the fine-grained control needed by animators, such as composing multiple actions and defining precise durations for parts of the motion. To address this, we introduce the new problem of timeline control for text-driven motion synthesis, which provides an intuitive, yet fine-grained, input interface for users. Instead of a single prompt, users can specify a multi-track timeline of multiple prompts organized in temporal intervals that may overlap. This enables specifying the exact timings of each action and composing multiple actions in sequence or at overlapping intervals. To generate composite animations from a multi-track timeline, we propose a new test-time denoising method. This method can be integrated with any pre-trained motion diffusion model to synthesize realistic motions that accurately reflect the timeline. At every step of denoising, our method processes each timeline interval (text prompt) individually, subsequently aggregating the predictions with consideration for the specific body parts engaged in each action. Experimental comparisons and ablations validate that our method produces realistic motions that respect the semantics and timing of given text prompts.

Main paper: 
 introduction: 
 Motivated by applications in video games, entertainment, and virtual avatar creation, recent work has demonstrated substantial progress in learning to generate 3D human motion [27, 37, 44, 60] . Generating motions from text descriptions is of particular interest; it has the potential to democratize animation with a natural language interface that is intuitive for beginner and expert users alike. To this end, several methods have been proposed that synthesize reasonable character animations given a single text prompt and fixed duration as input [38, 53, 65] .While these methods are a promising first step towards faster and more accessible animation interfaces, they lack the precise control that is crucial for many animators. Consider the input prompt (see Fig. 2d ): "A human walks in a circle clockwise, then sits, simultaneously raising their right hand towards the end of the walk, the hand raising halts midway through the sitting action." Due to a lack of representative training data, prior work struggles with such complex text prompts [38, 53] .Namely, the prompt includes temporal composition [4] where multiple actions are performed in sequence (e.g., walking then sitting), along with spatial composition [5] where several actions are performed simultaneously with differing body parts (e.g., walking while raising hand). Furthermore, such lengthy prompts quickly become unwieldy for the user and, despite their detailed descriptions, are still ambiguous with respect to the timing and duration of the constituent actions. insight is to stitch together such independent generations in both space and time before continuing to denoise. For spatial compositions, automatic body part associations [5] allow coherently concatenating predictions together. Score arithmetic [66] is used to ensure smooth transitions for temporal compositions. To further improve the performance of STMC, we introduce MDM-SMPL, which makes several improvements to prior motion diffusion models [53] , including directly using the SMPL [34] body representation.The performance of STMC on timeline control for text-driven motion synthesis is verified through comprehensive comparisons and a perceptual user study. In summary, the central contribution of this work consists of: (i) the new problem of multi-track timeline control for text-driven 3D human motion synthesis, and (ii) a novel test-time technique, STMC, that effectively structures the denoising process to ensure faithful execution of all prompts in a timeline. As a side contribution, (iii) we upgrade MDM to directly support the SMPL body representation instead of skeletons, and reduce runtime through fewer denoising steps. Code will be released upon publication. 
related work: 
 Human motion synthesis. A large body of work in both vision and graphics has been dedicated to generating 3D human motions [70] . This generation process can be unconditional [36, 56] or conditioned on actions [10, 17, 37] , music [32, 50, 52, 57] , speech [3, 69] , goals [30, 51, 60] , previous motion [13, 15, 44, 62] (i.e., future motion prediction), scenes/objects [21, 31, 58, 59] , and text [1, 2, 11, 16, 19, 29, 53, 65] . Technical approaches vary from early statistical models [8, 15] to modern generative models like VAEs [20, 37, 38] ,GANs [6, 12, 49, 61] , normalizing flows [22, 57] , and diffusion [11, 29, 30, 60, 68] . Our work is most related to recent text-conditioned diffusion models [53, 65] , however we solve a new problem where the model is conditioned on a timeline containing several text inputs instead of a single prompt.Motion composition. Due to the lack of training data, a particular challenge for action and text-conditioned motion generation is to synthesize compositional motions. Several works [4, 41, 66] focus on generating motions from a sequence of text prompts and durations, i.e., temporal compositions. TEACH [4] autoregressively generates one motion (per text prompt) at a time, conditioning the next motion in the sequence with the previous one. EMS [41] proposes a two-stage approach, by first generating each action separately and then merging them through a subsequent network. Diffusion models EDGE [54] and PriorMDM [48] ensure consistency between adjacent motions by enforcing temporal constraints at transitions. Our approach to temporal composition is based on DiffCollage [66] , which stitches motions (or images) together throughout the denoising process via score arithmetic at overlapping transitions.Other work generates motions from a set of texts to be executed at the same time, i.e., spatial (body-part) composition. SINC [5] labels ground truth motion capture (mocap) sequences with corresponding body parts by prompting GPT-3 [9] . These labels are used to create a synthetic dataset of motions stitched together from mocap sequences with compatible body parts, thereby improving performance of VAE-based 3D motion generation methods [38] for spatial composition.MotionDiffuse [65] proposes a noise interpolation method to control different body part motions separately. Our approach, STMC, takes inspiration from SINC [5] [3, 28, 54] , especially from text inputs [11, 14, 53, 65] . Some works focus on improving the controllability of motion diffusion models, e.g., by enabling temporal [48, 66] and spatial [65] composition of text prompts.Other controls such as following specific keyframe poses, joint trajectories, and waypoints have also been achieved using a mix of test-time diffusion guidance [28, 30, 45] , in-painting [48, 53] , and direct conditioning [60] . We focus on making text-tomotion generation more controllable by handling several text prompts in a fine-grained timeline format through a compositional denoising process. 
human motion synthesis from timelines: 
 We first formulate the new problem setup of multi-track timeline control (Sec. 3.1), then propose a motion denoising strategy to handle timeline inputs (Sec. 3.2 and Sec. 3.3), and finally summarize our improved diffusion model (Sec. 3.4).Inputs. As illustrated in Fig. 1 , the multi-track timeline enables users to define multiple intervals, each linked to a natural language prompt describing the desired human motion. For the jth prompt in the timeline, we represent its temporal interval as [a j ,b j ] and the corresponding prompt as C j . The intervals are arranged in a multi-track layout on the timeline, allowing for overlaps. Both the duration of each interval and of the overall timeline are variable, and users can add an arbitrary number of tracks (rows) to the timeline (although, in practice, a character can most often perform a handful of actions simultaneously).Outputs. The goal is to generate a 3D human motion that follows all the text instructions at the specified intervals. A human motion x lasting N timesteps is represented as a sequence of pose vectors x=(x 1 ,...,x N ) with each pose x i ∈R d . Several recent works [53, 65] Our generation method (Sec. 3.3) leverages a pre-trained motion 195 diffusion model such as MDM [53] or MotionDiffuse [65] 196 trained on single text prompts, which we briefly review here.These methods follow a denoising diffusion scheme and 198 synthesize animations through iterative denoising of a noisy 199 pose sequence. Given a clean motion x 0 , a Gaussian diffusion 200 process is employed to corrupt the data to be approximately 201 N (0,I). Each step of this process is given by: x 0 is tractable [25]:EQUATIONEQUATIONwhere explicit poses [53, 65] , rather than in a latent space [11] .EQUATIONEQUATIONMotion cropping and denoising. The input x t at denoising While SINC applies stitching only once, STMC does so at every step of denoising, encouraging a more coherent composition of movements by allowing the denoiser to correct any artifacts. This is possible because the denoiser outputs explicit human poses (i.e., we know which indices correspond to arms, legs, etc. within the pose vector), so we can extract body-part motions from separate crops and spatially combine them to obtain a composite motion. To achieve this, we first pre-process the input timeline to assign a text prompt to each body part at every timestep, thereby creating a separate motion timeline for every body part (see Fig. 3a ): left arm, right arm, torso, legs and head.As shown in Fig. 3a , each text prompt in the multi-track timeline is first annotated with a set of body parts involved in the motion. This can be done automatically by querying GPT-3 [9] as in SINC, or directly given by the user for additional creative control. Then, each text prompt is assigned to its annotated body parts within the corresponding time interval, which assumes that body parts at overlapping intervals are compatible (e.g., if a prompt is annotated with "legs", then no other prompt should involve legs throughout its entire interval). To fill in the To mitigate these potential artifacts, we apply DiffCollage [66] to each body-part motion. As shown in Fig. 3b , instead of directly denoising x aj:bj tfor each text prompt, we denoise an expanded time interval [a j -l,b j +l], where l is the desired over- Multi-track timeline (MTT) dataset. To properly evaluate our new task, we introduce a new challenging dataset of 500 multitrack timelines. Each timeline in the dataset is automatically constructed and contains three prompts on a two-track timeline (e.g., Fig. 2d ). To construct these timelines, we first manually collect a set of 60 texts covering a diverse set of "atomic" actions (e.g., "punch with the right hand", "jump forward", "run backwards", see Appendix C for the full list), and annotate the involved body parts for each text. To serve as ground truth for computing evaluation metrics (Sec. 4.2), we also select motion samples from AMASS that correspond to each text. Based on the atomic texts, we automatically generate timelines containing three prompts and two tracks (rows). For each timeline, the first track is filled with two consecutive prompts sampled from the set of texts and given randomized durations. A third random text with complementary body-part annotations is then placed in the second track at a random location in time.The main reasons for restricting the evaluation to three beyond three prompts can be found in the supplementary video. 
evaluation metrics: 
 Given the novelty of the task, identifying relevant metrics to evaluate different methods is crucial. Instead of relying on a single metric, we disentangle the evaluation of semantic correctness (how faithful individual motion crops are to the textual descriptions) from that of realism (e.g., temporal smoothness).Semantic metrics. Firstly, we evaluate the alignment between the generated motion and the text description within the specified intervals on the timeline, which we term "per-crop semantic correctness". To assess this, we utilize the recent text-to-motion retrieval model TMR We apply our STMC test-time approach on the pretrained diffusion models of MotionDiffuse [65] , MDM [53] , and MDM-SMPL (ours). For each denoiser, we establish several strong baselines by repurposing existing methods to the timeline-conditioned generation task for comparison. Results are shown in Tab. 1. Next to each method, the table indicates how many tracks the input timelines have (#tracks) and how many text prompts can be contained in a track (#crops). Next,we introduce each baseline and analyze results.Single-text input [53, 65] baseline. The simplest approach to condition motion diffusion on a timeline is to convert the timeline into a single text description, which aligns with the model's training input format (e.g., Fig. 2a ). Given that our timeline dataset is consistently comprised of three motions (A, B, and C), we formulate single-text prompts as follows: "A and then B while C". While timing information can be included in the prompt, e.g., "A for 4 seconds", this is out-of-distribution for models trained on HumanML3D, leading to worse results.This method parallels the baseline strategies of SINC [5] for spatial composition and TEACH [4] for temporal composition.As shown for each denoiser in Tab. 1, this approach is ineffective for both semantic correctness metrics and realism.Since these models cannot generate motions longer than 10 sec and there is no timing information in the prompt, for this experiment, outputs are limited to a maximum duration of 10 sec and semantic correctness metrics are reported over the entire duration of the motion rather than per-crop. The poor performance is a result of the models not being trained on the types of complex compositional prompts that result from collapsing the timeline to a single text description.DiffCollage [66] baseline. Instead of converting the multi-track timeline into a single prompt, one can collapse it into a single track timeline containing a series of consecutive text prompts, i.e., transform the problem to be one of temporal composition.DiffCollage can then be used to temporally compose the sequence of actions. For example, the timeline in Fig. 2d would be split into ["walking in a circle," "walking in a circle while raising the right hand," "sitting down while raising the right hand," "sitting down"]. Note that, unlike the single-text baseline, this splitting preserves the timings (#crops) in the timeline.While the DiffCollage baseline generally produces smooth transitions and reasonable FID scores, the semantic accuracy is consistently worse than STMC. This is due to the complex spatial compositions within the prompts after collapsing the timeline into a single track, which models trained on HumanML3D struggle with. In contrast, STMC uses body-part stitching throughout denoising to compose actions from simpler prompts.SINC [5] baseline. Rather than performing body-part stitching iteratively at every denoising step, an alternative approach is to stitch body motions together only once after all crops have finished the entire denoising process. This is most similar to SINC and forms the basis for two baselines that accept the full Our method STMC is compared to several strong baselines when using three different denoising models. The single-text and DiffCollage baselines struggle to handle complex compositional prompts that results from collapsing the timeline down to a single track. The SINC baselines produce reasonable semantic accuracy by denoising prompts independently as in STMC, but cause abrupt or unnatural transitions with higher transition distance (underlined) or FID. As discussed in Sec. 4.6, STMC is currently limited by the pre-trained diffusion model that it leverages for each motion crop; we expect improvements in these models to also boostEQUATIONSTMC. An additional experiment on varying the overlap length for temporal stitching can be found in Appendix E.We perform two separate user studies to compare STMC to SINC with Lerp and DiffCollage when using MDM. Fig. 4 CVPR We visualize the results of STMC with MDM-SMPL on several input timelines and color the bodies depending on their location in the timeline. We see that STMC is capable of generating realistic motions, which capture the semantics of the given text prompts with the desired timing and duration. In (a) and (c), STMC generates motions that precisely follow the instructions, controlling a single arm while still performing another action. The accurate timing of intervals is demonstrated in (b) where the arms are still up in the air when transitioning from "walking" to "jumping", which is difficult to achieve with alternative methods. In (c) and (d), we observe that STMC is capable of generating compositions that were not present in the ground truth data, such as "walking backwards while eating" or "walking while playing violin".#9 CVPR #9 CVPR 2024 Submissionshows results of both studies, measuring human preference for motion realism and semantic accuracy. On the left, STMC is preferred or similar to SINC 66% of the time for realism and 62% of the time for semantic accuracy, with 4.2 raters judging each video on average after filtering bad responses. Compared to DiffCollage on the right, our method is preferred or similar 68% of the time for realism and 70% for semantic accuracy, with 2.8 raters judging each video after filtering. This demonstrates that STMC improves the motion in ways that are discernible by humans but may not be fully captured in quantitative metrics.We visualize motions generated by STMC with MDM-SMPL in Figure 5 , given multi-track timelines as input from our MTT dataset. The coloring follows the input text, prioritizing the newest prompt when there is an overlap across tracks. These results show that STMC is capable of generating realistic motions for complex multi-prompt timelines, which follow the timing and duration of the given intervals. Please see the caption for full analysis of these examples, and we refer to the supplementary video for additional qualitative results and comparison to generated motions from baseline methods.While STMC expands the capabilities of pre-trained motion diffusion models to take a multi-track timeline as input, it is also limited by the models that it relies on. For example, Proceedings of the IEEE/CVF Conference on Computer Vision 
 
 Appendix:  


Decision: The paper addresses the task of combining temporal and spatial control for text-driven motion synthesis.

Pros:
* Well written
* a novel task definition
* A novel dataset
* Qualitative results generally exhibit high quality

Cons:
* Limited novelty
* Absence of quantitative assessment of individual sub-motions
* A small portion of the qualitative results does not match the text prompts (e.g., applause)

**Guidance to authors:** 
* Please ensure that the new dataset is made publicly available
* Please incorporate quantitative results concerning individual sub-motions

Reviews: review: title: title: This work addresses the problem of multi-prompt text-to-motion generation with temporal constraints using a pre-trained motion diffusion model. While the writing quality is decent and the proposed solution combines existing methods effectively, the significance of the work is limited due to unsatisfactory results.
main_review: main_review: **Summary**: This paper attempts to solve the problem of multi-prompt text2motion generation with temporal constraints per prompt using a pre-trained motion diffusion model. Similar to previous works, authors use the pre-trained motion diffusion model to generate a motion sequence per text prompt and its time-interval and then perform a form of stitching/blending to combine the generated motions to get a final cohesive motion sequence that adheres to all the provided conditions. Specifically, authors propose to use a temporal composition similar to DiffCollage and the spatial composition of SINC to combine the motions.

**Writing Quality**: Overall, the writing quality is decent, the problem definition seems clear and framework figures help clarify the method. However, I find the the method section a bit incomplete as certain details about different parts of the proposed solution are not clearly described. For instance, in section 3.2, it is said that body-part timeline conditioning is done following SINC but no further explanation is provided on how exactly SINC has done this in the method section.

**Results Quality**: I belive the weakest part of this paper is the results as it is very difficult to assess the significance of the proposed solution based on the provided experimental results. The quantitative experiments are missing relevant baselines such as PriorMDM which also performs motion stiching. Also, the final realism metrics are not satisfactory and show very minor improvement over the baselines. Qualitative results provided via the supplementary video also show defects. Firstly, some of the text prompts are completely ignored by the model. For instance, the very first motion in the video has "walking backwards" and "applaud" for one of the intervals and "appluad" is completely ignored. Since these two prompts are not inherently incompatible with each other, I do expect the generated motion to adhere to both of these conditions. As the main contribution of the paper is allowing for multiple prompts over a single time interval, these results are indicative of limitations in the model's ability to effectively incorporate multiple prompts within a single time interval, undermining the core contribution of the paper. Secondly, there are some artifacts in the provided results such as foot sliding and floating. Although physical artifacts are expected from motion diffusion models, the observed artifacts seem exacerbated by the addition of mutliple constraints.

**Clarity**: I strongly recommend improving the clarification of the method section in the later revisions. Apart from some missing details in the method section, I find the writing clear.

**Originality**: The problem of study is not necessarily original and the proposed solution is also not novel as it generates different motion sequences and applies a form of blending (as previously done before such as in PriorMDM) built on available methods applied to different problems (body-part composition of SINC and temporal composition of DiffCollage). However, I believe a simple and effective working solution is always appreciated, especially considering that it is not obvious how SINC and DiffCollage can be applied to this problem. Therefore, I believe authors have done a decent job making a working solution combining available methods and applying them to the problem of study.

**Significance**: Unfortunately, this work lacks significance as the provided results are not satisfactory.
title: title: Temporal and Spatial control for text-driven motion synthesis, proposing a test-time denoising method integrated with pre-trained motion diffusion models.
main_review: main_review: On the bright side, the exposition is clear, and the results surpass baselines both quantitatively and qualitatively. Additionally, the MTT dataset makes a significant contribution to the community.

On the downside, the paper lacks novelty as it primarily combines SINC and DiffCollage. Additionally, the absence of metrics for individual sub-motions raises concerns about potential degradation in synthesis quality compared to previous art. Lastly, while the overall quality is high, there are occasional inaccuracies, such as motions not matching text prompts (e.g., lack of applause when mentioned).

Overall, I am inclined to accept this paper as it meets the standards of a workshop.