MoGenTS: Motion Generation based on Spatial-Temporal Joint Modeling
Authors: Weihao Yuan
Yisheng HE
Weichao Shen
Yuan Dong
Xiaodong Gu
Zilong Dong
Liefeng Bo
Qixing Huang
Venue: NeurIPS 2024 poster (2024)
Keywords: Motion Generation
Spation-Temporal
Joint
Primary area: generative_models
TL;DR: Human Motion Generation based on Spatial-Temporal Joint Modeling

Abstract: Motion generation from discrete quantization offers many advantages over continuous regression, but at the cost of inevitable approximation errors. Previous methods usually quantize the entire body pose into one code, which not only faces the difficulty in encoding all joints within one vector but also loses the spatial relationship between different joints. Differently, in this work we quantize each individual joint into one vector, which i) simplifies the quantization process as the complexity associated with a single joint is markedly lower than that of the entire pose; ii) maintains a spatial-temporal structure that preserves both the spatial relationships among joints and the temporal movement patterns; iii) yields a 2D token map, which enables the application of various 2D operations widely used in 2D images. Grounded in the 2D motion quantization, we build a spatial-temporal modeling framework, where 2D joint VQVAE, temporal-spatial 2D masking technique, and spatial-temporal 2D attention are proposed to take advantage of spatial-temporal signals among the 2D tokens. Extensive experiments demonstrate that our method significantly outperforms previous methods across different datasets, with a $26.6\%$ decrease of FID on HumanML3D and a $29.9\%$ decrease on KIT-ML.

MoGenTS: Motion Generation based on Spatial-Temporey Joint Modeling
Weihao Yuan' Yisheng He
Weichao Shen'
Yuan Dong'
Xiaodong Gu'
Zilong Dong't
Liefeng Bo!
Qixing Huang?
Alibaba Group
The University of Texas at Austin
Abstract
Motion generation from discrete quantization offers many advantages over continu- ous regression, but at the cost of inevitable approximation errors. Previous methods usually quantize the entire body pose into one code, which not only faces the diffi- culty in encoding all joints within one vector but also loses the spatial relationship between different joints. Differently, joint into vector, which i) simplifies the quantization process as the complexity associated with a single joint is markedly lower than that of the entire pose; ii) maintains temporall structure that preserves both the spatial relationships among joints and the temporal movement patterns; iii) yields 2D token map, which enables the application of various 2D operations widely used in 2D images. Grounded in the 2D motion quantization, we build spatial-temporal modeling framework, where 2D joint VQVAF, cemporal-spatial 2D masking technique, and spatial-temporal 2D attention are proposed to take advantage of spatial-tempont signals among the 2D tokens. Extensive experiments demonstrate that our method significantly outperforms previous methods across different datasets, with a 26.6% decrease of FID on HL33 and 29.9% decrease on KIT-ML.
this work we quantize each individual
Introduction
Human motion generation from textual prompts is fast fust-growing field in computer vision and is valuable for numerous applications like film making, the gaming industry, virtual reality, and robotics. Given text prompt describing the motion of person, the goal is to generate a sequence containing the positions of all joints in the human body at each moment corresponding to the text prompt.
Previous attempts to tackle this challenge generally proceeded in two directions. The first directly regresses the continuous human motions from the text inputs using methods such generative adversarial networks (GANs) [1 variational autoencoders (VAEs) 2 3 [5,, or recent diffusion
models 6
g
9
TO M. Although continuous regression has the advantage of directly optimizing
towards ground-truth data and does not lose the numerical precision, it struggles with the challenge of regressing continuous motion that encompasses intricate skeletal joint information and limited by the quality and scale of current -ootioo datasets. The second leverages the vector quantization (VQ) technique to convert continuous motion to discrete tokens, which transform the regression problem into a classification problem [12 13 14]
[6] [17]. In this way, the difficulty of motion generation could greatly reduced. In broader picture, approaching continuous regression problems using quantization techniques is becoming increasingly popular [18 [9] 20).. In this paper, we seek to push the limit of the second path by designing a novel representation and learning paradigm.
15]
"qqual Contribution t Corresponding Author
Project Page: httph::t///aet ttapss//aaggdd.... github. io/ so/mogents
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

"A person walks
Encoder
Joint VQ
Clip
Decoder
Spatial
Temp. Spa 20 Mask
Spa Temp 20 Attn
Temporal
Decoder
Masiced token
Poxition Embeedding
Quantization
Figure 1: Framework overview. (a) In motion quantization, human motion quantized into a spatial-tempor 2D token map by joint VQ-VAE. (b) In motion generation, a temporal-spatial 2D masking is performed to obtain a masked map, and then spatial-temporal 2D transformer is designed to infer the masked tokens.
Although recent methods have achieved impressive results by taking advantage of motion quanti- zation 13. 17 21], they share an inherent drawback. The VQ process inevitably introduces approximation errors, which impose undesirable limits the quality of the generated motions. Therefore, improving the accuracy of the VQ approximation a key point these methods. To this end, many techniques proposed, such residual VQ-VAE [17 and hierarchical VQ-VAE [22 to improve quantization precision and have obtained commendable results.
However, almost all previous methods quantize all joints of one frame into one vector and approximate this vector with one code from the codebook. This is not optimal since the whole-body pose contains much spatial information, i.e. the positions of all joints, such that quantizing the entire pose into one vector possesses two drawbacks. The first one that this makes the encoding process difficult, as each code within the codebook is tasked with encapsulating the comprehensive information of all joints, making the quantization fundamentally more complex. The second one lies in the loss of spatial relationships between the individual joints, hence the subsequent network could not capture and aggregate the spatial information.
To address these problems, we propose to quantize each joint rather than the whole-body pose into one vector. This brings three benefits. First, encoding at the joint level significantly simplifies the quantization process, as the complexity associated with representing the information of single joint is markedly lower than that of the entire pose. Second, with each joint encoded separately, the resulting tokens maintain spatial-temport distribution that preserves both the spatial relationships among joints and the temporal dynamics of their movements. Third, the spatial-temporal distribution of these tokens naturally organizes into 2D structure, akin to that of 2D images. This similarity enables the application of various 2D operations, such as convolution, 2D positional encoding, and 2D attention mechanisms, further enhancing the model's 's ability to interpret and generate human motion effectively.
In this paper, starting from the 2D motion quantization, we propose a spatial-tempont modeling framework for human motion generation. We employ spatial-temporal 2D joint VQVAE 23 to encode each joint across all frames into discrete codes drawn from codebook, which results in 2D tokens representing motion sequence, as shown in Figure [ Taking advantage of the 2D structure, both the encoder and decoder are equipped with 2D convolutional networks for efficient feature extraction, similar to 2D images. Then we perform the masked modeling technique following language tasks [24] and some prior motion generation works [1). However, unlike previous methods, we propose a temporal-spatial 2D masking strategy tailored for handling the 2D tokens. Then the randomly masked tokens are predicted by spatial-temporal 2D transformer conditioned on the text input. The spatial and temporal locations of different tokens are first encoded by 2D position encoding, after which the 2D tokens processed by both the spatial and temporal attention mechanisms. The spatial-tempora attention considers not only whether the generated motion conforms to the input text in terms of temporal sequence, but also whether the generated joints are sound in terms of spatial structure. Extensive experiments across different datasets demonstrate the efficacy of method in both motion quantization and motion generation. Compared to the previous SOTA method, the FID is decreased by 26.6% on HumanML.3D and 29.9% on KIT-MI..
The main contributions of this work summarized as follows:

---

We novelly quantize the human motion to spatial-temporad 2D tokens, where each joint is quantized an individual code of a VQ book. This not only makes the quantization task more tractable and reduces approximation errors, but also preserves crucial spatial information between individual joints.
The 2D motion quantization enables the deployment of 2D operations analogous to 2D images, such that we introduce 2D convolution, 2D position encoding, and 2D attention to enhance the motion auto-encoding and generation.
We propose temporal-spatial 2D masking strategy and perform attention in both the temporal and spatial dimensions, which ensures the quality of the motion in both the temporal movement and the spatial structure.
We outperform previous methods in both motion quantization and motion generation.
2
Related Work
2.1
Motion Generation from Continuous Regression
The initial methods devised for human motion generation were predominantly focused on directly regressing continuous motion values, rather straightforward approach without intermediary quanti-
zation 2 B 25 26
27 28 Numerous strategies have utilized the variational autoencoder (VAE)
framework to merge the latent embeddings of encoded text with those of encoded poses, subsequently
decoding this combined representation into motion predictions 22 26 5 Others have explored the potential of the recurrent network
25
29 30], generative adversarial network [28] 31 I. or
transformer network [32
33
34
35 to improve the quality of motion regression. Drawing on the
success of diffusion models, recent methods have begun introduce the diffusion process into motion diffusion [36 6 7
39 40), yielding commendable results. To improve the diffusion model in human motion generation, various mechanisms are proposed to be integrated into the denoising process. A hybrid retrieval mechanism considering both semantic and kinematic similarities is integrated to refine the denoising process in [TO]. Physical constraints 377 and guidance constraints [41] are also incorporated into the diffusion process for more reasonable motions. In addition, module of construction supported by linguistics structure that constructs accurate and complete language characteristics and a module progressive reasoning aware of the context are designed in (38).
10 37
38
While these methods hold the advantage of directly optimizing towards the ground truth with no information loss, they struggle with the challenge of regressing continuous motion that encompasses intricate skeletal joint information. Unlike the image tasks, there is no large-scale dataset for motion joints pre-training, therefore learning the continuous regression from scratch is not easy.
2.2 Motion Generation from Quantized Classification
To alleviate the difficulty of human motion generation, some methods quantize the continuous motion to discrete tokens and let networks predict the indices of the tokens, in which the original regression problem converted to a classification problem I12 13 input motion undergoes initial encoding via VQ- VAE 23 which generates motion tokens for subsequent prediction. The prediction of the token indices can be approached many ways. Drawing inspiration from advances in natural language processing, certain techniques adopt auto-regressive Others implement generative masked modeling strategies inspired by BERT [24), where tokens are randomly masked during training for the model to predict [17] 21. The discrete diffusion is also introduced to denoise discrete motion tokens 115 43). More recently, large language models (LLMs) have been used to take over the prediction process 212 AHl. Multimodal control signals are first quantized into discrete codes and
14
22 Typically, the
frameworks that predict tokens sequentially 13 14
then formulated in unified prompt instruction to ask the LLMs to generate the motion response [2].
Although all of these approaches reap the rewards of converting the task to quantized classifica- tion paradigm, they also grapple with the inherent approximation error from quantization. Many techniques like EMA [14], code reset [14, and residual structure 45 17 are introduced to improve quantization accuracy. However, the common practice of representing single token is fraught with difficulty and fails preserve the spatial relationships across joints. In contrast, our approach takes more granular path by quantizing each joint within the human skeleton
entire human pose with
3

---

CodeBook
33.55
Spattal
Temporal
Vector Map
Code Map
Encoder
Decoder
Figure 2: The structure of our spatial-temporal 2D Joint VQ-VAE for motion quantization.
independently.
This not only makes the quantization task more tractable and reduces approximation errors, but also preserves crucial spatial information between the joints.
Method
3.1 Problem Statement
Motion Quantization. Given a motion sequence {M] the objective construct an optimal codebook C = [= capable of accurately representing this sequence, so that the encoded motion features V could be substituted by the corresponding codes a; from the codebook with the minimum loss, resulting in a discrete token sequence {O}, representing the input motion.
Motion Generation. Given the textual prompt T, the task of our framework is to generate the indices of the tokens {0), so that the vectors corresponding these tokens could be decoded into motion sequence aligned with the input text.
3.2
Overview
The overview of our method is illustrated in Figure Our framework could be divided into two parts, motion quantization and motion generation. In the quantization phase, we first represent the input motion sequence in the joint-time structure, and encode it into 2D vector map. Subsequently, the vectors are quantized into codes of codebook, represented by the indices of the selected codebook entries, i.e., the joint tokens. Therefore, after the quantization, the input motion sequence is converted tokens arranged in a structure, where one dimension spatial while the other one is temporal. This results in 2D motion map, which is similar 2D image. In the generation phase, mask the 2D token map with a temporal-spatie
2D masking strategy, and then use a 2D transformer to predict the masked tokens, conditioned the CLIP 46 embedding T of the given text prompt. The 2D motion transformer considers both spatial attention and temporal attention between different 2D tokens. The 2D position embedding P is also used to convey the spatial and temporal locations of each token. Finally, the generated tokens are decoded back into a motion sequence that aligns with the given textual prompt, thus completing the process of text-driven motion generation.
3.3 Spatial-temporal 2D Joint Quantization Motion
Directly regressing the continuous motion sequence is easy. Thus, we perform motion quantization following previous methods on the whole pose 5 17 or body part 42 regression problem to classification problem, but differently, wholc-body pose. Therefore, we organize the motion sequence in a 2D structure and process it as 2D map, as illustrated Figure 2 For motion with sequence length and joint number J. employ a 2D convolutional network to encode the motion joints J
48) to convert the quantize each joint rather than the
" { I:=..... into 2D vectors
V (v/ w=l.... where each vector v? corresponds to time t and joint
[v/} = E (()
where E denotes the encoder network composed of two convolutional residual blocks, and j denotes 3D rotation angle of joint at time Then quantize the vector with codebook, i.c. replace

---

the vector v/ with its nearest code entry V in a preset codebook C = {c;
( - 2 ((v)),, Q vi C; where i = aar min ||c; v/ll2-
(2)
Here Q denotes the quantization process. After the quantization, the decoder 'D decodes the approxi- mate vectors A4 to get the original joint information, as
[ D ((v1)) D ((Q(E
(3)
Finally, the auto-encoding network is optimized by a loss considering both the vector approximation and the decoded joints, as
Lug -- - -ill +all? v/ll2.
(4)
where a denotes a weighting factor. We also employ the residual VQ- structure, exponential moving average, and codebook reset following previous methods [14 17.. but for simplicity only describe our method single-level quantization without incorporating a residual structure in this section.
3.4 Temporal-spatial 2D Token Masking
After quantization of the motion joints, we obtain 2D token map {0/] shown in Figure We follow language models 22 49 to perform random masking, but differently, here our tokens are in 2D structure and there are totally T tokens. In this case, performing a ID masking strategy ignoring the apatial-tempora relationship is not optimal, which applies the same masking strategy to all tokens. For example, it rarely appears that all joints in one frame are masked in the meantime. Networks trained in this way cannot work well in generating new motions, since the generation requires starting from an all- all-masked motion.
To solve this problem, we propose a 2D temporal-spatial masking strategy. We first perform the masking in only the temporal dimension and randomly mask the frames the sequence. Once one frame is masked, all joints in this frame are masked, as illustrated in Figure (a). After the temporal masking, we perform spatial masking do the second masking in the spatial dimension for all joints of an unmasked frame. All masked tokens are replaced by the Omask token, which denotes that this location is masked and predicted. Note that there is an underlying graph structure among the joints and that we have tried graph-based masking strategies. However, we find that the simple random masking strategy works the best. One interpretation is that it helps encode long-range correlations of joints in human motions, c.f. [50]
the remaining unmasked frames. Specifically, we randomly
The temporal and spatial masking strategies adopt the same mask ratio schedule following 51 This ratio is computed as
52
7(T) = cos( n
(5)
U(0, 1) is uniformly sampled, leading to mask ratio 7(7). Then according to this ratio, >(T) x T x tokens are randomly selected to masked in temporal masking, and (T) x J tokens randomly selected to be masked for one frame in the spatial masking. To add more disturbance to the masked prediction, the remasking mechanism in BERT [4] also employed: If a token 0f is selected to be masked, it would be replaced with Omask with an 80% probability, would be replaced with random token O with 10% probability, and remains unchanged with a 10% probability. After the temporal-spatial masking, we obtain a masked 2D token map for subsequent networks to predict.
where E [0, 1]. In training,
3.5 Spatial-temporal 2D Motion Generation
3.5.1 Spatial-Temporal Motion Transformer
Given a text input and a masked token map, we use transformer network to predict the tokens at the masked locations. There are three attentions performed in this transformer: spatial-temporal 2D) attention, spatial attention, and temporal attention, illustrated in Figure (b).
5

---

Spatial
Temporal Attn
Mosked token
Position Embedding
P
Embedeling
Temporal Masking
Spatial Masking
Spatial-temporal Masking
(b) Spatial-temporal Attention
Figure 3: The temporal-spatial masking strategy (a) and the spatial-temporal attention (b) for motion generation.
Spatial-Tempores 2D Attention. For the spatial-temporal attention, we first extract the text feature embedding using CLIP [46), resulting text token Otext. Then we add 2D position encoding [53] the 2D token map. The position encoding P/ is computed by the sinusoidal function in the spatial dimension j and in the temporal dimension t, respectively, so it provides both the spatial position and the temporal position information for the subsequent attention network. After the 2D position embedding. we flatten the 2D token map to the ID structure. Then we perform the ID attention in the flattened spatial-temport dimension, where the spatial and temporal relationship reserved by the 2D position encoding, as
As-1 = SootMaxxx K/vd+PP)
(6)
where Q, K, V (J]++1)xd are the query, key, and value matrices, d is the feature dimension, and +T' is the number of tokens. This bidimensional attention is performed in long sequence of JT tokens. Its strength is that is able learn patterns in the full spatial-temporal space. The limitation is that the mix of spatial and temporal information is too complex to learn. The resulting patterns inaccurate.
Joint Spatial Attention. Joint spatial attention is the one-dimensional attention in only the spatial dimension. As shown in Figure (b), we ignore the temporal dimension and rearrange the 2D tokens. The tokens of different time frames are regarded different batches, and then the attention is only calculated between different joints on each time batch, as
A, = SoftMax (Q. K./Vd P)V.,
where Qu, V, E R-xdd This spatial attention equally works on the joints of any time frame, so could capture the inner relationship between different joints without the influence of temporal information. The advantage of this attention module is that it is easier to learn. However, it only guarantees the rationality of the different joints one pose, regardless of whether the input text is or what the target motion is.
Joint Temporal Attention. Similar to spatial attention, we also perform one-dimensional attention only the temporal dimension. As shown Figure B)), we ignore the spatial dimension and rearrange the 2D tokens. The tokens of different joints are regarded as different batches, and then the attention only calculated between different times on each joint batch, as
A SoftMax(Q, K./Vd P)V,
where Q., Kg, Vg Txx Similar to joint spatial attention, this attention module is casier learn However, it mostly captures the change in motion of the joint and guarantees the rationality of the movement of one joint, such as motion smoothness.
x This temporal attention works equally on the time sequence of any joint.
Integration of attention modules. Our approach applies spatial-temporal 2D attention, joint spatial attention, and joint temporal attention a sequential manner. Here the spatial-temporal 2D attention provides the initialization, and joint spatial attention and joint-temporal attention offer regualarization.

---

3.5.2 Training and Inference.
Training. In the training, we use the spatial-temporal motion transformer to predict the probability logits of all C' tokens d in the masked positions. Then cross-entropy loss is computed between the prediction i and the target y, as
maask
ΣΣ Ye log(ū.) + (1 y4) log(1 (1 Ds).
(9)
mask 1=1
We also employ the residual structure following previous methods |7 and use residual spatial- temporal motion transformer to improve motion accuracy. The structure of the residual transformer is the same as that of the mask transformer, including temporal attention. In training, we randomly select one layer / from the residual layers learn. All tokens in the preceding layers [0 I] are summed as the input of the residual transformer. The prediction of the l-th layer tokens also optimized by the cross-entropy loss.
attention, spatial attention, and
Inference. In the inference, the generation starts from an empty token map, i.e., all tokens are masked. The prediction of the masked tokens is repeated by N iterations. In each iteration, we build the probability distribution of C' tokens from the SoftMax of the predicted logits, and sample the output from the distribution. In the next iteration, according to the schedule shown Section 3.4 with n/N, low-score tokens are selected to be masked and predicted again, until reaches N. Once the iterative masked generation is finished, the residual transformer progressively predicts the residual tokens of the residual VQ layers. Afterward, the base tokens plus the residual tokens are decoded by the VQVAE decoder to get the final generated motion.
Classifier Free Guidance. The classifier free guidance (CFG) technique [44 is utilized to incorpo- rate the text embeddings into the transformer architecture. During the training phase, the transformer undergoes unconditional training with probability of 10%. In the inference, CFG is applied the final linear projection layer just before the softmax operation. Here, the final logits, denoted as logits, derived by moving the conditional logits logits away from the unconditional logits logits using a guidance scale 8, as
logits = (1 + +) logits.co - logits
(10)
where is set to 4.
4 Experiments
4.1 Datasets
We evaluate our text-to-motion model on HumanML3D (5) and KIT-ML (55) datasets. HumanML3D consists of 14616 motion sequences and 44970 text descriptions, and KIT-ML consists of 3911 motions and 6278 texts. Following previous methods [5]. 23384/1460/4388 samples are used for train/validation/eest in HLmanML33, and 4888/300/830 are used for train/validation/test in KIT- ML. The motion pose extracted into the motion feature with dimensions of 263 and 251 for HumanML33 and KIT-ML respectively. The motion feature contains global information including the root velocity, root height, and foot contact, and local information including local joint position, velocity, and rotations in root space. The local joint information corresponds to 22 and 21 joints of SMPL. [56] for H33D and KIT-MI. respectively.
4.2 Implementation details
Our framework is trained on two NVIDIA A100 GPUs with Py PyTorc. The batch size is set to 256 and the learning rate set to 2e-4. To quantize the motion data into our 2D structure, we restructure the pose in the datasets to joint-based format, with the size of 12 x J. The data is then represented by the joint VQ codebook comprised 256 codes, each with dimension of 1024. Due to the global information included in the HumanML.3D data representation, we also incorporate global VQ codebook of the same size to encode the global information. For the residual structure, the number of residual layers is set to 5 following previous methods [17]. Both the encoder and decoder
additional
7

---

Methods
FID
Top1 f
Top2 f
Top3 1
MM-Dist
Diversity
Ground Truth
0.002
0.511 4.003
0.703#.000
0.7974.002
2.974 2
9.503 065
TEMOS
3.734#-028
0.424
0.612#-002
0.722#-000
3. 3.703=-000
8.973#-07 071
TM2T 13
1,501 1.501 2.01
0,424 4.003
0,618 0003
0.729 1.002
3.467=2.011
8, 8.589 8.589-..rg 1.89+..0r
T2M [5
1.0874.021
0.455-.000
0.455
4.003 003
0.636-.005
0.7364.000
3.3474
9.175
MDM
0.541
0.61 -07
5.566 t020
9.559 099
MLD 6
0.1733#-013
0.481
0.673
0.772 0002
3.196
9.724
MotionDiffuse
0.630
0.491 000
0.681 00001
0.782 0000
3.113 11.1113t.000 .001
9.410 049
PhysDiff 37
0,433
0.631
MotionGIT 12
0.567
3.775
9.006
T2M-GPT
2.111
0.492 003
0.679-t002
0.775#.002
3.121 3.121 2:000
9.761 2:081
M2DM 15
0.352 .00
0.497
0.6822t.002
0.763#-000
3.134
9.926
Fg-T2M 38
0.243#t002
0.492
0.683+.000
0.783#t
3.109
9.278
AtTT2M [6
0.112
0.499 003
0, 0.690 .022
0.786 002
3.038#.000 .07
9.700 090
Motionn
43
0.072=
0.515 003
0.706=20002
0.802#-002
2.941#.007
9.683#-102
ParCo
42
0.109
0.515 +.
0.706
0.801 0000
2.927
9.576 U88
MMM 21
0.080
0.504
0003
0.696
0.794 4000
2.998
9.411 450
MoMask
0.045+.002
0.521 4.00
0.713# .002
0.807
.000
2.958 .008
Ours
0.033 001
003 0.529
0.719 - 002
002 0.812
2.867
006
9.570
Ground Truth
0.031 0.004
0.424 +005 000
0.619
1019g.000 0D6
0.779 4000
7.88
11.080
TEMOS 4
3.7174.022
0.353
0.561
4.002
0.687 002
3.417
10.84
TM2T
3.599
0.280 005
0.463
1..-2000 006
0.5872.005
1.591 #.026
9.473#=117
T2M [5]
3.022
0.361 005
0.559
0007
0.681 000
3.488 + 4229
10.72
MDM
0.497=#.002
0.396 4.00
9.191 +.02
10.85 10.85 +00
MLD
6
0.404 0.404 +.00
0.390 4.08
0.609 +00s
0.734 #0007
3.204 +
10. 10.80 10180 +117
MitionDiffuse
1.954#
0.417
0.621 4.004
0.739 +
2.958
11.10
MotionGPT
12
0.597
3.394
10.54
T2M-GPT 14
0.514
0.416 000
0.627
0.745 4.006
3.007
10.86
M2DM 15
0.515 4.09
0.416 0.416+0004 0.0
0.628
0.743 4.004
3.015 +.107
11.417#
Fg-T2M
38
0.571 #0047
0.418 0.0
0.626 +004
0.745 4.004
3.114 #115
10.93
AttT2M
0.870#-088
0.413
0.63
0.751 #.006
3.039#.021
10.96
DiverseMotion
43
0.468#4000
0.416##000 00G
0.637#.00S
0.760=-011
2.892#t041
10.873#.100
ParCo
42
0.453:4.022
0,430 0.430 4.000
0.649
0.772 0.772
2.820 2..00.0022
10.95
MMM
21
0.429
0.381 4.0
0.590 .008
0.718 1.05
4.016 1.19
1.633..099
MoMask
17
0.204
0.433 007
0.656 0005
0.781 -00
2.779 .22
Ours
0.143#t000 004
0.445
0.671 000
0.797#
2.711 = 024
10.918 090
Table Evaluation on the HumanML3D dataset (upper half) and KIT-ML dataset (lower half).
constructed from convolutional residual blocks with downscale of 4. The transformers in our model are all set to have 6 layers, 6 heads, and 384 latent dimensions. The parameter a is set to and N is set to 10. More details are described in the supplementary material.
4.3 Evaluation
We evaluate both motion quantization and motion generation in our frame- work. The evaluation protocol strictly adheres to the standards established by previous methods [5] [44]]]]]] 17 (21],
HumanM1.3D
KIT-MI.
Methods
FID 4
MPJPF
FID 4
MPJPE J
TM2T 13
0.307
230.1
M2DM 15
0.063
0.413
T2M-GPT
14
0.070
58.0
0.472
using the same evaluator and metric calculation. Each experiment is re- peated 20 times and the results reported alongside a 95% statistical confidence interval.
MoMask
17
0.019
29.5 #000
0.112
37.2 2
Ours
0.005
13.8#.0
0.019
17.4
Table Evaluation motion quantization HumanML.3D dataset and KIT-ML dataset. MPJPE is measured millimeters.
Evaluation of Motion Quantization. The core idea of our framework starts from the spatial- temporal 2D joint quantization. Therefore, we first evaluate our VQVAE against that of previous quantization- based methods. For fair comparison, temporarily configure codebook to the

---

T2M-GP7
C Tia stop
mave out and back
right lenss to fell clbow tell knsse right clbore
haral owans look abWA resume narnigg
Krclesse back disen
X moave meny limos
MoMask
Conse esr hand
rin. hend over loek doom resume rurning
Waree and back
elhow
colease back down
X wave only once
Out
UnTON IN DINN
Cuassic left land
Bin
book
bend over Dolk doon resume nurning
down
right
cleac
warser our oud baok is serant bodi diss mileaun sevand
alov."
and sade dewe will aatigg rope.
Figure 4: Qualitative results the test set of HumanML3D. The color from light blue to dark blue indicates the motion sequence order. An arrow indicates this sequence is unfolded in the time axis.
same size as the previous methods i.c., 512 512. As reported in Table the FID and MPJPE accuracy of our method exceeds previous methods by large margin, in both the HumanMI33D and the KIT-MI datasets. This reveals that the difficulty of quantizing an individual joint much smaller than quantizing a whole-body pose.
Evaluation Motion Generation. We then compare our model previous including both the continuous regression-based methods and the discrete quantization- -based meth- ods. From the results reported in Table our method outperforms all previous methods on both the HumanMI.33 and the KIT-MI. datasets, which demonstrates the effectiveness of our method. Specifically, the FID is decreased by 26.6% on HumanML3D and decreased by 29.9% on KIT-ML. In addition, the R-precision even significantly surpasses the ground truth. From the qualitative results displayed Figure we also see that our method generates motions that are better matched with input texts compared to previous methods.
works,
4.4 Ablation Study
Joint 2D VQ. To verify the efficacy of the proposed framework, we first establish a base- line model incorporating a ID motion VQVAE and a ID token prediction transformer. Then we change the VQVAE to our 2D joint VQ- VAE, examining its impact on both motion auto- encoding and motion generation capabilities. According to the outcomes presented in Table 2 we observed a substantial enhancement in auto- encoding performance. However, as indicated in Table 3. the motion generation performance deteriorates. This decline can be attributed to the considerable increase in the number of tokens, which renders the ID masking strategy and ID transformer no longer adequate.
FID J
Top1 f
Baseline
0.108
0.501
+ 2D VQ
0.194
0.493
+ 2D Masking
0.054
0.516
2D Position Encoding
0.047
0.521
+ S&T Attn
0.033
0.529
Table Ablation study on HumanML3D dataset.
2D Masking. We then change the ID masking temporal-spatial 2D token masking, which leads significant performance improvement, as depicted Table 3 This demonstrates necessity of 2D masking strategy, as discussed in Section 3.4

---

"A
"A
welkss in
walks
walks in
countercio
countenclo
clwise
ekwise
ekwise
cinele
circl
circla
Original Generated
Temporal Editingg
Temporal-Spaaiae Editing
Figure Motion Editing. The edited regions are indicated green.
2D Position Encoding. With the 2D VQ. the attention is calculated between 2D tokens, but it is not a full version of the 2D attention. We upgrade this to spatial-temporal 2D attention by integrating the 2D position encoding, which provides both the spatial location and temporal location information. The performance improvement brought by this encoding is recorded in Table
Joint Spatial and Temporal Attention. As discussed in Section 3.5.1 the mix of spatial and temporal information is complex for the bidimensional attention performed on a long sequence. Here we inspect the benefits brought by the individual joint spatial and temporal attention mechanism. From the results in Table 3 this design further improves the performance of our framework.
4.5 Motion Editing / Inpainting
Due to generative mask modeling, our method is capable of not only generating motions from texts, but also editing a motion by masking the tokens of any locations in both the temporal and spatial dimensions. As shown in Figure 5 we first mask out the motion in the temporal dimension and then generate the motions corresponding to the masked locations according to a different text prompt, resulting in motion in which only the masked frames are edited. Subsequently, editing in the spatial dimension, which results in a motion where only the masked joints masked frames are edited. This editing mechanism could also be applied to inpainting motion sequence connecting two motion sequences by generating missing motions.
perform similar
5 Conclusion
This paper proposes to quantize each individual joint one vector, generating opatial-temponee 2D token mask for motion quantization, which reduces the approximation еттоr in quantization, reserves the spatial information between different joints for subsequent processing, and enables the 2D operators widely used in 2D images. Then the temporal-spatial 2D masking and spatial-temporal 2D attention are proposed to leverage the spatial-temporal information between joints for motion generation. Extensive experiments demonstrate the efficacy of the proposed method.
5.1 Limitations and future work.
Despite the improvement of our method in the quantization, there is still the ap- proximation error, which limits the motion generation. This should be further improved in the future network design, e.g., employing the coarse-to-fine technology. Another important point is the restricted size of the current human motion dataset. In the image pretraining, a much larger dataset is utilized to achieve decent quantization results. Similarly, large dataset for pretraining accurate human motion quantizer is also needed in future work.
Quantization.
Generation. We follow the mask-and- -endggnnrrttionn for motion generation in this work. The masking strategy used in the current framework is borrowed from Bert [24] On the contrary, some other methods follow the auto-regressive generation and also achieve good performance. In the future, one promising direction is combining these two technologies.
10

---

References
[1] Sergey Tulyakov, Ming- Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, Jun 2018.
[2] Anindita Ghosh, Noshaha Cheema, Cennet Oğuz, Christian Theobalt, and Philipp Slusallek. Synthesis of compositional animations from textual descriptions. Proceedings of the International Conference on Computer Vision, Jan 2021.
[3] Chaitanya Ahuja and Louis Phillppi Morency. Language2pose: Natural language grounded pose forecast- ing. In Proceedings of the International Conference on 3D Vision, Sep 2019.
[4] Mathis Petrovich, Michael Black, and Gül Varol. Temos: Generating diverse human motions from textual descriptions. In Proceedings of the European Conference on Computer Vision, pages 480-497. Springer, 2022.
[5] Chuan Guo, Shihao Zou, Xinxin Zuo. Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 3142-5157 IEEE, 2022.
[6] Xin Chen, Biao Jiang. Wen Liu, Zilong Huang, Bin Fu, Tao Chen, and Gang Yu. Executing your commands via motion diffusion in latent space. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 18000-18010. IEFF, 2023.
[7] Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit Haim Bermano. Human motion diffusion model. In Proceedings of the International Conference on Learning Representations. OpenReview.net, 2023.
[8] Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwci Liu. Motiondiffuse: Text-driven human motion generation with diftusion model. CoRR, abs/2208. 15001, 2022.
[9] Rishabh Dabral, Muhammad Hamza Mughal, Vladislav Golyanik, and Christian Thcobalt. Mofusion: A framework for -eeessingdiffusoo---er motion synthesis. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pages 9760-9770. IEEE, 2023.
[10] Mingyuan Zhang, Xinying Guo, Liang Pan, Zhongang Cai, Fangzhou Hong, Huirong Li, Lei Yang, and Ziwei Liu. Remoditfuse: eetrieval-augmeent motion diffusion model. In Proceedings of the International Conference on Computer Vision, pages 364 373. IEEE, 2023.
[11] José Sabeiro-Gomes, Tianhui Cai, Zoltán Ádám Milacski, Chen Wu, Aayush Prakash, Shingo Jason Takagi. Amaury Aubel, Dacil Kim, Alexandre Bernardino, and Fernando De Torre. Motiongpt: Human motion synthesis with improved diversity and realism via GPT-3 prompting. In IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5058-5000 5068. IEEE, 2024.
[12] Yaqi Zhang, Di Huang, Bin Liu, Shixiang Tang, Yan La, Lu Chen, Lei Bai, Qi Chu, Nenghai Yu, and Wanli Ouyang. Motiongpt: Finetuned Ilms are general-purpor purpose motion generators. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 7368-7376. 2024.
[13] Chuan Guo, Xinxin Zuo, Sen Wang. and Li Cheng. TM2T: stochastic and tokenized modeling for the reciprocal generation of 3d human motions and texts. In Proceedings of the European Conference on Computer Vision, volume 13695,, pages 580-597. Springer, 2022.
[14] Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Yong Zhang, Hongwei Zhao, Hongtao Lu, Xi Shen, and Shan Ying. Generating human motion from textual descriptions with discrete representations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 14730-14740.
IEEE, 2023.
[15] Hanyang Kong, Kehong Gong, Dongze Lian, Michael Bi Mi, and Xinchao Wang. Priority-centric human motion generation in discrete latent space. In Proceedings of the International Conference on Computer Vision, pages 14760-14770, IEBE, 2023.
[16] Chongyang Zhong, Lei Hu, Zihao Zhang, and Shihong Xia. Att22:: Text-driven human motion generation with melt--perspective attention mechanism. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, Frunce. October 1-6, 2023, pages 509-519, IEEE, 2023.
[17] Chuan Guo, Yuxuan Mu, Muhammad Gohar Javed, Sen Wang. and Li Cheng. Momask: Generative masked modeling of 3d buman motions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2024.
11

---

[18] Will Williams, Sam Ringer. Tom Ash, David MacLeod, Jamie Dougherty, and John Hughes. Hierarchical quantized autoencoders. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin, editors, Advances in Neural Information Processing Systems, 2020.
[19] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zcot text-to image generation. In Marina Mcila and Tong Zhang, editors, Proceedings of the International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8821-8831, PMIR, 2021.
[20] Sander Dieleman, Aäron van den Oord, and Karen Simonyan. The challenge of realistic music generation: modelling raw audio at scale. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicold Cesa- Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pages 8000-8010, 2018.
[21] Ekkasit Pinyoanuntapong, Pu Wang, Minwoo Lee, and Chen Chen. MMM: generative masked motion model. CoRR, abs/2312.03596, 2023.
[22] Shunlin Lu, Ling-Hao Chen, Ailing Zeng, Jing Lin, Ruimao Zhang, Lei Zhang, and Heung- Yeung Shum. Humantomato: Text-aligned whole-hody motion generation. CoRR, 10.1//27,, 2023.
[23] Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural diserete representation learning. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V.N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems, pages 6306-6315, 2017.
[24] Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pages 4171-4188 Association for Computational Linguisties, 2019.
[25] Angela Lin, Lemeng Wu, Rodolfo Corona, Kevin Tai, Qixing Huang, and Raymond J Mooney. Generating animated videos of human activities from natural language descriptions. Learning, 1(2018):, 2018.
[26] Nikos Athanasiou, Mathis Petrovich, Michael J. Black, and Gül Varol. TEACH: temporal action composi- tion for 3d humans. In Proceedings of the International Conference on 3D Vision, pages 414-423, IEEE, 2022.
[27] Daniel Holden, Jun Saito, and Taku Komura. A deep learning framework for character motion synthesis and editing, ACM Transactions on Graphics, 35(4):138:1-138: 2016.
[28] Haoye Cai, Chunyan Bai, Yu- Wing Tai, and Chi- Keung Tang. Deep vidco generation, prediction and completion human action sequences. Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, editors, Proceedings of the European Conference on Computer Vision, volume 1 11206 of Lecture Notes in Computer Science, pages 374-390. Springer, 2018.
[29] Matthias Plappert, Christian Mandery, and Tamim Asfour. Learning a bidirectional mapping between human bodle-body motion and natural language using deep recurrent neural networks. Robotics Auton. Syst., 109:13-26. 2018
[30] Yan Zhang, Michael Black, and Siyu Tang. Perpetual motion: Generating unbounded human motion. CoRR, abs/2007. .. 13886, 2020.
[31] Zhenyi Wang. Ping Yu, Yang Zhao, Ruiyi Zhang, Yufan Zhou, Junsong Yuan, and Changyou Chen. learning diverse stochastic human-action generators by learning smooth latent transitions. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 12281-12288 AAAI Press, 2020.
[32] Guy Tevet, Brian Gordon, Amir Hertz, Amit H. Bermano, and Daniel Cohen-Or. Motionclip: Exposing human motion generation to CLIP space. In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Gio- vanni Maria Farinella, and Tal Hassner, editors, Proceedings of the European Conference on Computer Vision, volume 13682 of Lecture Notes in Computer Science, pages 358-374. Springer, 2022.
[33] Junfan Lin, Jianlong Chang. Lingbo Liu, Guanbin Li, Liang Lin, Qi Tian, and Chang Wen Chen. Being comes from not being: Open-vocabulan text-to- to-motion generation with wordless training. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 23222 23231. IEEE, 2023.
12

---

[34] Uttaran Bhattacharya, Nicholas Rewkowski, Abhishek Banerjee, Pooja Guhan, Aniket Bera, and Dinesh Manocha. Text2gestures: transformer- network for generating emotive body gestures for virtual agents, In IEEE Virtual Reality and 3D User Interfaces, Mar 2021.
[35] Mathis Petrovich, Michael Black, and Gúl Varol. cendinncond 3d human motion synthesis with transformer VAE. In Proceedings of the International Conference on Computer Vision, pages (0965-10977) IEEE, 2021.
[36] Jihoon Kim, Jiseob Kim, and Sungjoon Choi. FLAME: free-form language- based motion synthesis & editing. In Brian Williams, Yiling Chen, and Jennifer Neville, editors, Proceedings of the AAAI Conference on Artificial Intelligence, pages 2255-8263. AAAI Press, 2023.
[37] Yuan, Jiaming Song. Umar lqbal, Arash Vahdat, and Jan Kautz. Physdiff: Physics-guided human motion diffusion model. In Proceedings of the International Conference on Computer Vision, pages 11964-19755 IEFE, 2023.
[38] Yin Wang, Zhiying Leng, Frederick W. B. Li. Shun-Cheng Wu, and Xiaohui Liang. Fg-12m: Fine-grained text-driven human motion generation via diffusion model. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pages 21978 21987. IEEE, 2023.
[39] Mingyuan Zhang, Huirong Li, Zhongang Cai, Jiawei Ren, Lei Yang. and Ziwei Liu. Finemogen: Fine- grained spatio-temporal motion generation and editing. Alice Oh, Tristan Naumann, Amir Gloherson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Newral Information Processing Systems, 2023.
[40] Zeyu Zhang, Akide Liu, Ian D. Reid, Richard Hartley, Bohan Zhuang, and Hao Tang. Motion mamba: Efficient and long sequence motion generation with hierarchical and bidirectional selective SSM. CoRR, abs/2403.07 04.0477777777 2024.
[41] Korrawe Karunratanakul, Konpat Preechakul, Supasom Suwajanakorn, and Siyu Tang. Guided motion diffusion for controllable human motion synthesis. In Proceedings of the International Conference on Computer Vision, pages 2151-2162. IEEE, 2023.
[42] Qiran Zou, Shangyuan Yuan, Shian Du, Yu Wang, Chang Liu, Yi Xu, Jie Chen, and Xiangyang Ji. Parco: Part-coordinating text-to-motion synthesis. CoRR, abs/2403. 18552, 2024.
[43] Yunhong Lou, Linchao Zhu, Yaxiong Wang, Xiaohan Wang, and Yi Yang. Diversemotion: Towards diverse human motion generation via discrete diffusion. CoRR, abs/2309.01372 2023.
[44] Zixiang Zhou, Yu Wan, and Baoyuan Wang. Avatargpt: All-in-one framework for motion understanding, planning, generation and beyond. CoRR, abs/2311.166668 2023.
[45] Julicta Martinez, Holger H. Hoos, and James J. Little. Stacked quantizers for compositional vector compression. CORR, abs/1411.22 2014.
[46] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8748-8763. PMLR, 2021.
[47] Zigang Geng, Chunyu Wang, Yixuan Wei, 7e Liu, Houqiang Li, and Han Hu. Human pose compositional tokens. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 660- 171. JEEE, 2023.
[48] Guénolé Fiche, Simon Leglaive, Xavier Alameda-Pinede Antonio Agudo, and Francesc Moreno-Noguer. VQ-HPS: human pose and shape estimation in a vector-quantized latent space. CoRR, abs/2312.08291, 2023.
[49] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition. pages 11305-11315. IEEE, 2022.
[50] Hyung-Gue Gun Chi, Myoung Hoon Ha, Seung- geun Chi, Sang Wan Lee, Qixing Huang. and Karthik Ramani. Infogen: Representation learning for human skeleton-basee action recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages IEEE, 2022.
13

---

[51] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings the IEEE Conference Computer Vision and Pattern Recognition, pages 11315-11325, 2022.
[52] Huiwen Chang. Han Zhang, Jarred Barber, Aaron Maschinot, José Lezama, Lu Jiang, Ming-Hsuan Yang. Kevin Patrick Murphy, William Freeman, Michael Rubinstein, Yuanzhen Li, and Dilip Krishnan. Muse: Text-to-image generation via masked generative transformers. In Proceedings of the International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 4055-4075. PMLR, 2023.
[53] Zelun Wang and Jyh-Char Charn Liu. Translating math formula images to latex sequences using deep neural networks with sequence-leve training, 2019.
[54] Jonathan Ho and Tim Salimans. Classificr-free free diffusion guidance. CoRR, wbwp2207.115999 2022.
[55] Matthias Plappert, Christian Mandery, and Tamim Asfour. The KIT motion-language dataset. Big Data, 4(4):236 252, 2016.
[56] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. SMPL: A skinned multi-person linear model. ACM Transactions on Graphics, 34(6):248:1 1444:11661 October 2015.
[57] Jing Lin, Ailing Zeng, Shunlin Lu, Yuanhao Cai, Ruimao Zhang, Haoqian Wang, and Lei Zhang. Motion-x: A large-scale 3d expressive whole-body human motion dataset. In Alice Oh, Tristan Naumann. Amir Globerson, Kate Sacnko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems, 2023.
14

---

A Appendix
A.1 Evaluation Metrics
We strictly follow previous methods 14, 17] to evaluate our model. We use the evaluator proposed in (5] calculate the text-motion align feature of the generated motion, ground-trum motion, and text as fgon fgr, fext Then we evaluate the features with the following metrics:
Frechet Inception Distance (FID) an objective metric calculating the distance between features extracted from the generated motion and the ground-truie motion, as
FID genn
Mer - Tr(Co0gen + Coug 2(Cougen
(11)
where Agen and Agg are the mean of fgen and fgt. Cov denotes the covariance matrix, and Tr denotes the trace of matrix. The features are extracted using the models following previous methods [5].
R-Precision (Top-1, Top-2, Top-3) measures the matching between the motion and the text. For each generated motion, ground-trum text description and 31 randomly selected mismatched descriptions from the test set form description pool. Then this metric is calculated by computing and ranking the Euclidean distances between the motion feature and the text feature of each description in the pool. Then the average accuracy is counted at Top-1, Top-2, and Top-3 places. The ground truth entry falling into the top-k candidates treated as successful retrieval.
MultiModal-Dis (MM-Dist) measures the distance between the text feature and the motion feature. This metric is calculated by the average Euclidean distance between these features, as
MM-Dist
|[gen
feet
(12)
Diversity measures variance of the whole motion sequences across the dataset. It randomly samples Naiver pairs of motion, where each pair is denoted by fill and fs22, Then the diversity is computed as
Diversity
Naver | pi.
(13)
Ndiver
where Ndiver is set 300.
A.2
Computational Overhead
We test the computational overhead of different methods on an NVIDIA 4090 GPU and report the average inference time per sentence in Table Although method does not achieve the shortest inference time, the increase in computational overhead is not significant. Overall, the computational overhead of our method is comparable to that of mainstream methods.
Method
Average Inference Time per Sentence
MLD
134 6327
MotionDiffuse
MDM
10416 ms
T2M-GPT
239 ms
MoMask
73 ms
Ours
181
Table 4: Computational overhead of different methods.
A.3
More Results
A.3.1
Motion Quantization Evaluation on More Datasets
To further assess the spatial-temporal 2D motion quantization of our method, we perform the evaluation on recent larger dataset, Motion-X [57]. This dataset collects 81.1K motion sequences from various public datasets and some self-captured videos. We convert their data to the 263-format
15

---

Methods
MPJPE
FID
Top1
Top2
Top3 t
MM-Dist
Diversity
Ground Truth
0.511 1.003
7703+.003
0.797
2.974 2.
9.503
MoMask 17
29.5 13.8
0.019
0.508=0008
003
0.701#-002
0.795=-00 002
2.999 006
9.565=
Ours
0.005 20050000 000
0.512 2. 002
0.704 4.00
0.798- .002
2.978
2.06
9.501 076
(a)
Methods
MPJPE +
FID 4
Top1
Top2 1
Top3 f
MM-Dist 4
Diversity
Ground Truth
0.424#-005
0.649 006
0.779 006
2.788+:012
11.080
MoMask 177
37.2
0.112 001
0.4174.006
0.645 # 007
0.769 0.769
2. 2.786
10.811 130
Ours
17.4
2.01
0.4234.004
0.6492.005 000
0.779
2.7574.012
11.056 2666..006 .69
(b)
Table 5: Evaluation of motion quantization on (a) Humanml3D dataset and (b) KIT-ML dataset. MPJPE is measured in millimeters.
Methods
MPJPE 4
FID 4
Top1 f
Top2 f
Top3 f
MM-Dist | Diversity
Ground Truth
0.420
0.631 =000 001
0.754
2.800
10.100 .83
MoMask 17
111 4.0
0.081
#0001
0.396 4.00
0.604 * 002
0.725 4.00
2.955 4.003
9.837 9.837 103
Ours
48.7
0.011
1000
0.417
0.627 t
4..50
2.832
10.113
Table 6: Evaluation of motion quantization Motion-X dataset. MPJPE measured in millimeters.
aligned with HumanML3D [5] following the official code of Motion-X. X. Since there is official motiinn-atchinn text-motion matching model, we only evaluate the motion quantization part of our framework. We follow [5] to train the text & motion feature extractors for evaluation. The results are reported in Table 6 Besides, the complete results of quantization evaluation on HumanML.3D and KIT-MI are also reported in Table 'he results indicate that method is capable of quantizing continuous motions with significantly reduced loss. This enhancement allows our transformer to work with motion tokens with smaller approximation errors, thus raising the upper limit of motion generation quality.
A.3.2 More Qualitative Results
We display more qualitative results in Figure 6 and Figure 7 From the generated motions, see that our method could generate complex and interesting motion sequences according to the text prompts.
16

---

POMGRR
MoMask
Ours
"A persor waflrr backwards then comnss forword
"A perzon dhrona zometiion thing hard with thetr right arm."
"A perzon doer swimming motion while standing"
permon lifts their right log and then left log.
Figure 6: More qualitative results of our method are presented. The color from light blue to dark blue indicates the motion sequence order. An arrow indicates this sequence unfolded in the time axis.
17

---

"A Figure walks in zig ang paliern.
"The person dribbling basketbull backwards.
person taking lagee diagonal step."
"A person walks forward their extended to the side.
"A person walks forward armss victory."
person farward left iog limping.
"A walks backwards and shen sups stairs bockwands as well.
"Walked right then up ladder, holding on with bosh arms then reached with right grob somerhing.
"While walking forward he appears be pushed, he rights himself and continus walking
"A person is performing bunges
"A person kicking left leg and then kicking right thom.
"A man ruises his right hand to this face then lowers back down.
Figure 7: More qualitative results of our method are presented. The color from light blue to dark blue indicates the motion sequence order. An arrow indicates this sequence unfolded in the time axis.
18

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction sections already reflect the paper's contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
19

---

Justification:
The limitations have been discussed in the conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have tried to include all the details and referenced work for reproduction. We will also release the code of our method.
Guidelines:
20

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
The code is not included for now. But we will release the code to the public soon.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
gablic//uides/Codeee.
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nis.cc/public/guust http://///giiiee//eeeuuubiiiiiii
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
21

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
These details are described in the experiment section and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
The error bars are reported in the experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
This is described in the experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
22

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: This work conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses such risks.
23

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The used assets are properly cited in the experiments section.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperssithcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?

---

Answer:
[NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "YfVJGSMUL4", "reviewer": "Reviewer_g7ot", "summary": "This paper proposes a novel approach to human motion generation by quantizing each joint into individual vectors, rather than encoding the entire body pose into one code. The key contributions are: (1) It quantizes each joint separately to preserve spatial relationships and simplify the encoding process. Then the motion sequence are organized into a 2D token map, akin to 2D images, allowing the use of 2D operations like convolution, positional encoding, and attention mechanisms. (2) It introduces a spatial-temporal 2D joint VQVAE to encode motion sequences into discrete codes and employs a temporal-spatial 2D masking strategy and a spatial-temporal 2D transformer to predict masked tokens.", "questions": "Please see the weaknesses. Overall, the idea is interesting. My major concern is the extra computational cost of this method, which could be much larger than previous methods with body-level VQ-VAE yet it is not investigated in the paper.", "limitations": "This paper has discusses the limitation of this paper: the approximation error in VQ-VAE and a larger dataset for training more accurate VQ-VAE. \nI think there is no potential negative societal impact in this paper.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a novel joint-level quantization approach, addressing the complexity and spatial information loss issues seen in whole-body pose quantization.By organizing motion sequences into a 2D token map, the method takes advantage of powerful 2D image processing techniques, enhancing feature extraction and motion generation.\n2. The integration of 2D joint VQVAE, temporal-spatial 2D masking, and spatial-temporal 2D attention forms a robust framework that effectively captures spatial-temporal dynamics in human motion.\n3. Extensive experiments demonstrate the method's efficacy, outperforming previous state-of-the-art methods on key datasets.\n4. The paper is well-written and easy to understand. The supp. mat. video provides comparisons with MoMask.", "weaknesses": "1. Although the overall idea of joint-level quantization is interesting, I still have the concern of computational overhead. While motion representation is typically lightweight, the use of 2D code maps and spatial-temporal attention can introduce significant computational overhead, similar to image data processing. It would be beneficial to compare the inference speed of mainstream methods (e.g., MoMask, T2M-GPT, MLD, etc) to show that the state-of-the-art performance is achieved with comparable computational costs.\n2. The experiments are conducted on relatively small datasets (HumanML3D and KIT-ML). To better validate the effectiveness of the proposed method, experiments on larger-scale datasets, such as Motion-X, would be advantageous.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback and constructive suggestions! We hope our responses adequately address the following questions raised about our work. Please let us know if there is anything we can clarify further.\n\n**1. Computational overhead of the proposed method.**\n\nThanks for this constructive advice. We test the computational overhead of different methods on an NVIDIA 4090 GPU and report the average inference time per sentence in the Table 1. \nAlthough our method does not achieve the shortest inference time, the increase in computational overhead is not significant.\nOverall, the computational overhead of our method is comparable to that of mainstream methods.\n\n| **Methods**          |     **Average Inference Time per Sentence**   | \n|-------------------|   :------------------:|\n| MLD     |  134 ms  |\n| MotionDiffuse     |  6327 ms  |\n| MDM     |  10416 ms  |\n| T2M-GPT     |  239 ms  |\n| MoMask     |  73 ms  |\n| Ours     |  181 ms  |\n\n\n\n**2. Experiments on Motion-X.**\n\nSorry for the lack of clarity. Actually, we have conducted the motion quantization experiments on the Motion-X dataset, as in Appendix A.3.1 or the below table. From the quantization results, our method also works well in the larger dataset, Motion-X.\nDue to the fact that there is no official text-motion-feature model, it is hard to evaluate the text-to-motion generation performance of our method. We have trained our own text-motion-feature model following HumanML3D, but it does not work well. We will clarify this in the revision, and also continue our attempts to evaluate our method on the Motion-X dataset.\n\n| **Methods** | **MPJPE** | **FID** | **Top1** | **Top2** | **Top3** | **MM-Dist** | **Diversity** |\n|-------------------|   :------------------:| :------------------:|  :------------------:| :------------------:|  :------------------:| :------------------:| :------------------:|\n| Ground Truth     |  -  | -  | 0.420 | 0.631 | 0.754 | 2.800 | 10.100\n| Momask     |  111  | 0.081  | 0.396 | 0.604 | 0.725 | 2.955 | 9.837\n| Ours     |  48.7  | 0.011  | 0.417 | 0.627 | 0.750 | 2.832 | 10.113", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jxqPz2WrEY", "reviewer": "Reviewer_SBXK", "summary": "This paper proposes an approach for text-conditioned motion generation. A common practice in this area is to use a quantized representation of human motion obtained with a VQ-VAE. However, most prior works represent the full body by a single token, which makes accurate reconstruction complicated.\n\nIn this work, the authors propose a new way of quantizing human motion: a single token is associated with a single joint. Then, the motion can be represented as a 2D grid of indices corresponding to spatial and temporal dimensions. Using this new representation, this paper proposes to generate motions using masked generative modeling.\n\nIn summary, the contributions of this paper are:\n- A new quantization of the human motion, representing each joint in a 2D map of tokens.\n- A masking strategy allowing the leverage of spatiotemporal information preserved by the proposed quantization.\n- A masked generative modeling strategy to generate new motions conditioned on text input.", "questions": "- How does the quantization work? I would like to understand if there is a spatial-temporal reduction, as the information in the paper seems contradictory.\n- From my understanding in Section 3.3, there is no information about which joint is processed in the encoder (so the encoder processes in the same manner every joint). Can the VQ-VAE be considered a quantization of the 3D space as a learned grid?\n- From Figure 2 and other explanations, the pose seems flattened to correspond to a column in the 2D map. How are the joints organized so that the spatial structure of the pose is preserved? Is there a topological ordering?", "limitations": "The section about the limitations is very short. I think that it could be improved by proposing more solutions to the quantization problem and other research perspectives. Otherwise, it sounds like the problem of motion generation is now completely solved.\n\nThe authors say that this work has no societal impact at all. I would agree that the impact on society is very limited.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "I would say that the main strength of this paper is not the novelty: masked generative modeling was already used for human motion generation [17]. However, this paper brings new components that make a lot of sense and seem to greatly impact the results. The bottleneck of prior works (1 pose = 1 token) is well-identified, and the proposed quantization strategy addresses this problem effectively by associating a token to each joint. In addition to improving the reconstruction after quantization, this representation proves useful as it preserves the spatiotemporal structure of the motion. Carefully designed operations (2D token masking, spatial-temporal motion Transformer) benefit from the proposed representation despite its higher dimension.\n\nAnother strength of this paper is the evaluation. The comparisons follow the standard procedures and seem totally fair to other methods. Providing confidence intervals by running experiments multiple times is an excellent practice. Even for the evaluation of the quantization in Table 2, I find it very good that the authors decreased the codebook size of the introduced model for fair comparison with other methods. In addition to the 2 datasets widely used for comparisons, the appendix provides results on numerous datasets, which is appreciated to evaluate the model's generalization capability. The ablations are also satisfying, as they allow to evaluate the impact of the main introduced components.", "weaknesses": "The main weakness of this paper is that it is difficult to understand the quantization of human motion:\n- L76 \"each joint is quantized to an individual code of a VQ book\": From my understanding, with the residual quantization, each joint is quantized to a sequence of indices; the final code is the sum of codes corresponding to those indices and associated codebooks.\n- Equation 1: Given L159, it seems that one joint in the input is converted to one token. Equation 1 suggests the same (the input of the encoder would be of dimension 3). And then L272, \"Both the encoder and decoder are constructed from 2 convolutional residual blocks with a downscale of 4,\" so I really do not understand at all. Is there a spatiotemporal reduction?\n- Equation 2: This does not correspond to residual quantization. Maybe it is meant to simplify the understanding, but I find it very confusing.\n\nGlobally, it is very difficult to understand how the method works until we reach section 3.5.2. For instance, until then, I did not understand the notion of a 2D map since the residual quantization would have made the grid 3-dimensional. I also wondered how the masking could encompass the depth of the quantization.\n\nOther minor issues include:\n- L125: It would be better to mention methods that represent a single pose (or human) with multiple tokens [a,b].\n- The presentation of Table 1 is not optimal. Giving the dataset in the table instead of the caption would be more clear (like in [17]). Also, why are there no bold results for diversity? It may look like this is because some other methods have better results.\n\n\n\n[a] Geng, Z., Wang, C., Wei, Y., Liu, Z., Li, H., & Hu, H. (2023). Human pose as compositional tokens. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 660-671).\n\n[b] Fiche, G., Leglaive, S., Alameda-Pineda, X., Agudo, A., & Moreno-Noguer, F. (2023). VQ-HPS: Human Pose and Shape Estimation in a Vector-Quantized Latent Space. arXiv preprint arXiv:2312.08291.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback and constructive suggestions! We hope our responses adequately address the following questions raised about our work. Please let us know if there is anything we can clarify further.\n\n**1. Clarification of the quantization.**\n\nSorry for the confusion caused. We will clarify this in the revision. In this paper, we do not intend to detail the residual quantization, since we simply follow established methods to employ this technique, as noted in L169. Thus for simplicity, we describe our method using only single-level quantization without incorporating a residual structure.\nTherefore,\n- L76: Here we only describe the single-level quantization without the residual structure.\n- Spatiotemporal reduction: In equation 1 within the method section, we describe our methodology that does not include spatiotemporal reduction. \nHowever, our experimental results indicate that appropriate spatiotemporal reduction can decrease computational load without significantly impacting performance.\nSo we perform a spatio-temporal reduction in the experiments and describe this in L272 in the implementation details.\n- Equation 2: Yes, here we only describe the single-level quantization without the residual structure.\n- Masking: The masking is only performed on the base-level. The prediction of the tokens in residual levels is based on the previous level without masking, following the previous method MoMask.\n\n**2. Encoder processes in the same manner every joint.**\n\nYes, the encoder processes every joint in the same manner. The 2D joint map, with dimensions T×J×D (where T represents the sequence length, J denotes the number of joints, and D indicates the feature dimension), is processed by the 2D convolutional networks, similar to image processing techniques.\nAfter the encoder, the quantization (either single-level or residual) begins to quantize the encoded vectors.\n\n\n**3. Spatial structure.**\n\nThe order of the joints is fixed in both training and inference, so we hope the network can learn the spatial structure of the flattened joints in the training, with the 2D positional encoding.\n\n**4. Minor issues and limitations.**\n\nWe will mention [a, b] in the revision.\n\nTable 1: Giving the dataset in the caption is due to space constraints. We will attempt to revise this. No bold results for diversity is because it is hard to say which one is better for this metric.\n\nLimitations: The brevity is due to the limited space. We will try to include a more extensive discussion on quantization in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xYzw7YYsdj", "reviewer": "Reviewer_1rhT", "summary": "This paper present a new method for text to motion generation. In this method the human motion is represented as 2D tokens in a codebook. This allow the authors to apply 2D operation on 3D motions and use a 2D masking strategy. The architecture is composed of a VAE to learn the codebook and of a Transformer to learn the relation between CLIP embedding of the text input and the corresponding codebook tokens using spatial-temporal attention. With this method the authors outperforms state of the art approaches quantitatively and qualitatively. An ablation study shows the effect of each component.", "questions": "It should be mentioned somewhere that j^i_t contains the 3 dimension of joint i.\nA user study would have been nice. Metrics are difficult to use on these more complex actions.\nIt might be better to mention clip in the overview instead of waiting fro section 3.5.", "limitations": "The limitations are very briefly addressed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is clear and detailed.\n\nThe mixed use of codebook, masking ans spatial-temporal transformer is interesting.\n\nThe method outperforms the state of the art quantitatively and qualitatively.\n\nThe ablation shows well the effect of each component.", "weaknesses": "The Figure 3 b is not very clear It seems that the CLIP embedding is concatenated to the flattened motion embedding and then positional encoding is applied while the text say that the CLIP embedding is added after positional encoding. I also don't understand why positional encoding is added twice, one on the flattened vector and one time on the matrix.\n\nAn ablation to show that this concatenation of token and text is the better than for example cross attention would have been welcome. Another interesting ablation would have been to see the performance of the model when computing the temporal and spatial attention in parallel instead of sequentially.\n\nIt is not clear whether P is added only during attention or also on the inputs like base transformer. There is also no explanation as to why add positional encoding after computing the attention matrix.\n\nOn several metric the ground truth is beaten but the paper does not provide explanation fro this.\n\nThe paper does not describe how FID features are extracted ?\n\nThe classifier free guidance should be mentioned in the main paper not just in the appendix. It is an important component.\n\nRegarding the motion editing figure : why is only one hand being raised with the Temporal-Spatial Editing while the temporal editing results shows both hands being raised. Since the plural is used in the prompt this would indicate that temporal editing is better.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback and constructive suggestions! We hope our responses adequately address the following questions raised about our work. Please let us know if there is anything we can clarify further.\n\n**1. Clarification of positional encoding.**\n\nSorry for the confusion caused. We will clarify this in the revision. \n\n- The positional encoding is performed after adding the CLIP embedding. We will revise the manuscript in the revision.\nThe reason why positional encoding is added twice is that we want to reinforce the network's awareness of the spatial-temporal 2D structure. Adding positional encoding once should also work. \n\n- The positional encoding P is added on the inputs like the base transformer. \n\n- Adding positional encoding after computing the attention matrix is because we want to reinforce the network's awareness of the spatial-temporal 2D structure before the next attention computing.\n\n\n**2. Ablation of cross attention of token and text and computing the temporal and spatial attention in parallel.**\n\nThanks for this constructive advice. We perform the ablation study as suggested and present the results in Table 2.\nFrom the results, we can see that the cross attention of token and text or computing the temporal and spatial attention in parallel performs similarly to our original framework.\n\n| **Setup**          |     **FID**   |      **Top1**   |\n|-------------------|   :------------------:| :------------------:|\n| Cross attention of token and text     |  0.034  | 0.521  |\n| Computing the temporal and spatial attention in parallel     |  0.035  | 0.522  |\n| Ours     |  0.033  | 0.529  |\n\n\n**3. Why ground truth is beaten.**\n\nThe ground truth is beaten on the metrics about the text-motion matching, so we think this is because the text label is not perfectly annotated.\nWe will add this explanation in the revision.\n\n\n**4. FID features extract.**\n\nFollowing previous methods, we use the same models proposed in T2M [5] to extract the FID features. We will add this description in the revision.\n\n**5. Classifier free guidance.**\n\nWe will mention this in the main paper in the revision.\n\n\n**6. Motion editing figure.**\n\nSorry for the misleading.\nIn Figure 5, the Temporal-Spatial Editing is just to show that we could edit only specific joints while keeping other joints fixed. This level of granular control is hard to realize in previous methods that encode all joints to one token.\n\n**7. User study.**\n\nThanks for this constructive advice. For the user study, we select 10 texts and display the generated results of T2M-GPT, MoMask, and Ours. Due to the limited time, we only collect the study from 14 users, resulting in 140 total votes. Of these, 102 votes favor our results, 33 votes prefer MoMask, and 5 votes prefer T2M-GPT, as is shown in Figure 1 in the uploaded PDF file.\n\n\n**8. Other questions.**\n\n- We will mention that $j^i_t$ denotes a 3D rotation of joint i on time t in the revision.\n- We will mention CLIP in the method overview in the revision.\n- Limitations: The brevity is due to the limited space. We will try to include a more extensive discussion in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to express our sincere gratitude to all the reviewers for their time and their valuable feedback. We deeply appreciate their recognition of our work, such as 

**Reviewer g7ot:**
"a novel approach",
"a robust framework that effectively captures spatial-temporal dynamics",
"Extensive experiments demonstrate the method's efficacy, outperforming previous state-of-the-art methods".

**Reviewer SBXK:**
"this paper brings new components that make a lot of sense and seem to greatly impact the results"

**Reviewer 1rhT:**
"The mixed use of codebook, masking and spatial-temporal transformer is interesting",
"The method outperforms the state of the art quantitatively and qualitatively"

In the following, we address each reviewer’s comments point by point, and attach some figures in the uploaded PDF file.
We hope our responses adequately address the questions raised about our work. Please let us know if there is anything else we can clarify further.

Meta-review: This paper presents work on 3d pose human motion generation.  The main contribution is quantization using body joints rather than entire poses.  This contribution is incorporated into a framework that includes masking and attention, leading to good quality results with respect to the state of the art.

The reviewers appreciated the exploration of joint-level encoding as a means to more effective quantization-based motion generation.  The overall framework is sensible, and the experimentation thorough.  Concerns were raised over computational costs as well as some details on clarity.  In the subsequent author response / discussion the computational costs and clarity issues were largely resolved.

Overall, the paper's core contributions are very focused on encoding for human motion generation.  The results are convincing and the paper is solidly executed.  I am not sure whether the claims regarding spatial relationships being lost in entire body pose encoding is accurate -- the whole pose is encoded, so spatial relationships are present; this seems more about compositionality.  However, this is a relatively minor point and overall the paper is a solid contribution and advances 3d human motion generation.  As such, it is recommended for acceptance in NeurIPS.
Decision: Accept (poster)