T2LM: Long-Term 3D Human Motion Generation from Multiple Sentences

Authors: hyemin  ahn
timothy  ha
yunho  choi
hwiyeon  yoo
haoye  cai
chunyan  bai
yu-wing  tai
chi  -keung
yinglin  duan
tianyang  shi
zhengxia  zou
yenan  lin
zhehui  qian
bohan  zhang
yi  yuan
chuan  guo
xinxin  zuo
sen  wang
shihao  zou
annan  deng
minglun  gong
li  cheng
  ac
wei  ji
Venue: thecvf_com_CVPR_2024_Workshop_HuMoGen
Published: 2024-02-27 05:27:38

Abstract: In this paper, we address the challenging problem of long-term 3D human motion generation. Specifically, we aim to generate a long sequence of smoothly connected actions from a stream of multiple sentences (i.e., paragraph). Previous long-term motion generating approaches were mostly based on recurrent methods, using previously generated motion chunks as input for the next step. However, this approach has two drawbacks: 1) it relies on sequential datasets, which are expensive; 2) these methods yield unrealistic gaps between motions generated at each step. To address these issues, we introduce simple yet effective T2LM, a continuous long-term generation framework that can be trained without sequential data. T2LM comprises two components: a 1D-convolutional VQVAE, trained to compress motion to sequences of latent vectors, and a Transformer-based Text Encoder that predicts a latent sequence given an input text. At inference, a sequence of sentences is translated into a continuous stream of latent vectors. This is then decoded into a motion by the VQVAE decoder; the use of 1D convolutions with a local temporal receptive field avoids temporal inconsistencies between training and generated sequences. This simple constraint on the VQ-VAE allows it to be trained with short sequences only and produces smoother transitions. T2LM outperforms prior long-term generation models while overcoming the constraint of requiring sequential data; it is also competitive with SOTA single-action generation models.

Main paper: 
 related works: 
 Human motion synthesis. Human motion synthesis is naturally formulated as a generative modeling problem. In . Related to our works, Multi-Act [22] , ST2M [25] and TEACH [5] utilize a recurrent generation framework with past-conditional VAE to generate multiple actions sequentially. These require sequential training data [41] , an inherent limitation of the recurrent paradigm. DoubleTake, a part of PriorMDM [46] that utilizes MDM [51] as a generative prior, individually generates the actions and connects them with a diffusion model. 
method: 
 We now present in detail our T2LM approach. First, we explain how we compress human motion into a discrete space and reconstruct motion from it (Sec. 3.1). Second, we introduce a GPT-like autoregressive Text Encoder designed to map a given text to a sequence in the discrete latent space learned by the VQ-VAE (Sec. 3.2). Third, we discuss in Sec. 3.3 our procedure to generate long-term motion sequences corresponding to input text streams. We also include a desired length for each action in the stream. At train time, this is extracted from the data, while at inference, this can be either treated as an input or sampled from a prior.Motivation. Human motion is typically represented as a temporal sequence of 3D points -human meshes or skeletons -or a sequence of model parameters that produce such 3D representations [29, 35] . Plausible human motion usually represents a very small portion of these representation spaces, as evidenced by the fact that sequences of random samples do not produce any realistic motion. This has motivated methods that compress human motion into a discrete latent space and has shown to be beneficial for reconstruction and manipulation [31, 60] . In contrast to previous approaches [5, 38, 46, 51] , where a single latent represents the entire action available at each step, we design our approach so that each latent represents a fixed length of human motion. This enables continuous decoding of the semantics from textual descriptions without creating a duration mismatch between train and test sequences. We employ a 1D convolutional VQVAE to learn such a latent representation.Model. As depicted in Fig. 3 , our VQVAE consists of an Encoder E conv , a Decoder D conv , and a quantization module Q using a codebook V . The model is inspired by [31, 48, 60] 231 v i ∈ R d V . Specifically, each element z i of the latent vec- 232 tor sequence Z = E conv (X) = {z 1 , ..., z Tz } is quantizedẐ = Q(Z) := arg min vs i ||z i -v si || 2 i ∈ R Tz×d V (1) 237 X = D conv ( Ẑ) = D conv (Q(E conv (X))).(2)Eq. ( 2) is non-differentiable, and we handle it by the and codebook through optimization by following loss:L VQ =L recon (X, X) + ||sg [E conv (X)] -Ẑ|| 2 2 + β||sg Ẑ -E(X) -Ẑ|| 2 2 .(3)The term β||sg Ẑ -E conv (X) -Ẑ|| 2 2 , is referred to as a commitment loss, has shown to be necessary to stable training [53] . The reconstruction loss L recon consists of L1-loss on the parameter, reconstructed joint, and velocity.Product quantization. To enhance the flexibility of the discrete representations learned by the encoder E conv , we employ a product quantization. Each element z i within ) . . . , V (s 1 L ) . . . V (SZ = E conv (X) is divided into K chunks (z 1 i , ..., z K i ),EQUATIONFinally, using a 1D convolutional decoder D conv , we decode 331 these latent vectors to obtain the desired long-term motion: with lengths ranging from a minimum of 44 frames to a maximum of 250 frames.EQUATION 
evaluation metrics: 
 Sliding-scope and Transition-scope. Existing evaluation metrics for motion generation rely heavily on extracting features from the entire motion, making them dependent on motion length and inadequate for quantitatively assessing the quality of generated long-term motions. We propose two new evaluation criteria to address this limitation: FID and diversity within a Sliding-scope and Transition-scope.We use a fixed window of 80 frames for both scopes to extract subsets of long-term motions. We then measure FID and Diversity by comparing these subsets with sets extracted identically from the ground truth motion set. In Sliding-scope (SS-FID and SS-Div), we slide the window with a stride of 40 frames from the beginning to the end of the generated long-term motion to extract samples. In the Transition-scope (TS-FID and TS-Div), we extract samples centered around transitions in the generated long-term motion. The Sliding-scope provides an overall measure of how realistically the generated long-term motion represents We provide visualizations of generated long-term motions obtained with our method. The first, second, and third actions are rendered in blue, purple, and brown, respectively. This is a video figure that is best viewed by Adobe Reader.In this section, we compare the quality of motions generated with our T2LM to previous methods on the Hu-manML3D [15] and BABEL [41] datasets. Regarding the experiment on BABEL, we trained our model with individual actions and text annotations without using transitions. Our main comparison target on BABEL and Hu-manML3D is DoubleTake [46] , the only long-term generation method trained without sequential data. Furthermore, we also compare with TEACH [5] and MultiAct [22] on BABEL dataset. 1 Our straightforward approach outperforms previous long-term generation methods in both single-action and long-term generation despite not requiring any sequential data for training.Long-term generation. Tabs. 4 and 5 shows that our T2LM outperforms the main competing method, Double-Take [46] , in every criteria on both HumanML3D [15] and BABEL [41] . Regarding the Sliding-scope evaluation, our model demonstrates better overall quality of generated long-term motion compared to DoubleTake. Additionally, in the Transition-scope evaluation, our model produces more realistic transitions than those generated by Double-Take. When evaluating long-term generation on the BA-BEL dataset, our model outperforms MultiAct on SS-FID, SS-Div. and TS-FID metric. Our method also shows the better performance compared to TEACH on the SS-FID metric, indicating better overall quality. However, ours showed inferior performance in the Transition-scope evaluation. This can be attributed to the usage of transitions 1 ST2M is excluded from the comparison, since they do not use the 135-dimension representation as TEACH, DoubleTake and Ours. Instead, ST2M used 263-dimension representation. As a result, their quantitative evaluation lies on different dimension from TEACH, DoubleTake and Ours. (Quantitative scores of GT motions in [25] and [46] Specifically, our T2LM scored 14.1% higher Top-3 R-495 precision compared to DoubleTake [46] on HumanML3D.Moreover, we gained 16.2%, 15.9% and 12.9% Top-497 3 R-precision over MultiAct [22] , DoubleTake [46] and  
: 
 length conditional latent prediction model. Both are trained with single actions and accompanying texts. At inference time, a stream of input sentences and desired motion lengths is encoded into a stream of latent vectors. Finally, we continuously reconstruct the desired long-term motion with the 1D convolutional decoder.Our model has two key properties: First, it produces sequences of latent vectors, unlike approaches that encode the entire sequence into a single latent vector like Actor [37] .Second, we learn a prior over small chunks of motion, each encoded independently from the others, using a VQVAE encoder built from 1D convolutional layers with a local receptive field. This assumption, which departs from methods taking all past motion into account like PoseGPT [31] , is the simplest way to avoid any discrepancies between short training sequences and long sequences at inference time.These two key properties offer several advantages for long-term generations. First, it is possible to process a sequence of infinite length on the fly, as the cost of forwarding the model is linear in the size of the local receptive field [42] . This is in contrast with methods that employ a vanilla transformer architecture with a complexity that is quadratic in the sequence length. Thus, our model can process a continuous stream rather than a sequence of chunks that have to be later post-processed [5] . Secondly, using a sequence of latents with local receptive field allows to convey fine-grained semantics at the right temporal location. Empirically, we show that these simple changes lead to higher-quality actions compared to existing methods that generate variable-length actions with a single latent vector.Our experiments show that T2LM outperforms the state-of-the-art on long-term generation while matching or outperforming existing approaches for single-action when evaluated with FID scores and R-precision. We present two novel metrics aimed at evaluating the quantitative excellence of long-term motion more effectively: a) during transitions and b) along the sequence utilizing a sliding window approach.Our contributions are the following:• We propose a conceptually simple yet effective method T2LM for generating long-term human motions from a continuous stream of arbitrary-length text sequences.• We make two architectural design choices which together enable T2LM to generate smooth transitions and to be trained without any long-term sequential training data.• As a result, T2LM outperforms previous long-term generation methods while overcoming their limitations. We also match the performance of previous state-of-the-art single-action generation models. 
 
 Appendix:  


Decision: The paper addresses the task of long-motion synthesis. 

Pros: 
* Well written
* Thoroughly tested
* High qualitative results

Cons:
* Limited novelty
* Textual inaccuracies

**Guidance to authors:** Kindly address any textual inaccuracies highlighted by the reviewers, and correct any other typos or inaccuracies present in the paper.

Reviews: review: title: title: T2LM
main_review: main_review: T2LM is a method for generating long motions, described by a sequence of texts, each describing the motion of a single segment. The method is based on the GPT framework which is popular for motion generation and previously presented in the papers T2M, T2M-GPT, MotionGPT, and others. T2LM straightforwardly works at inference time by inputting the relevant text when its segment is generated.

Comments
- T2LM is generating the motions continuously without the need to stitch motion segments.
- Comprehensive evaluation - comparing to recent work, both diffusion and VAE-based solutions.
- T2LM outperforms recent works in terms of FID
- The novelty of this work is minimal, yet the method seems to work surprisingly well and is very relevant for this workshop.


Questions and suggestions for the next revision
- Figure 4 caption: test -> text (?)
- Figure 5 includes a single frame instead of motion, please fix that.
title: title: Text to Long-term motion generation. The model utilizies discrete diffusion using VQVAE and a text encoder.
main_review: main_review: Positively, the paper employs a sensible approach to quantization and mapping of latent representations, resulting in commendable qualitative outcomes. In addition the level of writing is high.

On the downside, the paper seems to extend motion techniques rooted in VQ-VAE, such as T2M-GPT. Its enhancements feel like engineering rather than innovations. In addition there are some inaccuracies that should be fixed for the final version: In Table 1, DoubleTake supports continuous generation just like T2LM, Equation 3 may contain errors, and there are some typos here and there.

Overall, I am inclined to accept this paper as it meets the standards of a workshop.