FineMoGen: Fine-Grained Spatio-Temporal Motion Generation and Editing
Authors: Mingyuan Zhang
Huirong Li
Zhongang Cai
Jiawei Ren
Lei Yang
Ziwei Liu
Venue: NeurIPS 2023 poster (2023)
Keywords: Motion Generation
Diffusion Model

Abstract: Text-driven motion generation has achieved substantial progress with the emergence of diffusion models. However, existing methods still struggle to generate complex motion sequences that correspond to fine-grained descriptions, depicting detailed and accurate spatio-temporal actions.This lack of fine controllability limits the usage of motion generation to a larger audience. To tackle these challenges, we present FineMoGen, a diffusion-based motion generation and editing framework that can synthesize fine-grained motions, with spatial-temporal composition to the user instructions. Specifically, FineMoGen builds upon diffusion model with a novel transformer architecture dubbed Spatio-Temporal Mixture Attention SAMI. SAMI optimizes the generation of the global attention template from two perspectives: 1) explicitly modeling the constraints of spatio-temporal composition; and 2) utilizing sparsely-activated mixture-of-experts to adaptively extract fine-grained features. To facilitate a large-scale study on this new fine-grained motion generation task, we contribute the HuMMan-MoGen dataset, which consists of 2,968 videos and 102,336 fine-grained spatio-temporal descriptions. Extensive experiments validate that FineMoGen  exhibits superior motion generation quality over state-of-the-art methods. Notably, FineMoGen further enables zero-shot motion editing capabilities with the aid of modern large language models (LLM), which faithfully manipulates motion sequences with fine-grained instructions.

FineMoGen: Fine-Grained Spatio-Temporal Motion Generation and Editing
Mingyuan Zhang
Huirong Li
Zhongang Cail.2 Jiawei Ren Lei Yang²
Ziwei Liu
S-Lab, Nanyang Technological University, 2 Sense'Time Research
Abstract
Text-driven motion generation has achieved substantial progress with the emer- gence of diffusion models. However, existing methods still struggle generate complex motion sequences that correspond to fine-grained descriptions, depicting detailed and accurate spatio-temporal actions. This lack of fine controllability limits the usage of motion generation to a larger audience. To tackle these challenges, we present FineMoGen, diffusion-basee motion generation and editing framework that can synthesize fine-grained motions, with spatial-temporal composition to the user instructions. Specifically, FineMoGen builds upon diffusion model with a novel transformer architecture dubbed Spatio-Temporal Mixture Attention (SAMI). SAMI optimizes the generation the global attention template from two perspec- tives: 1) explicitly modeling the constraints of spatio-temporal composition; and 2) utilizing sparsely-activate activated mixture-of -feeeeerss- to adaptively extract graiigri features. To facilitate a large-scale study on this new fine-grained motion genera- tion task, we contribute the HuMMMMnnMMoen dataset, which consists of 2,968 videos and 102,336 fine-grained spatio-temporal descriptions. Extensive experi- ments validate that FineMoGen exhibits superior motion generation quality over state-of-ther methods. Notably, FineMoGen further enables zero-shot motion editing capabilities with the aid of modern large language models (LLM), which faithfully manipulates motion sequences with fine-grained instructions. Project Page: https: /mingyunnnnnnnn zhang. github. io/projects /FineMoGen html.
Introduction
While traditional text-driven motion generation tasks have empowered models to generate motion sequences based on text, two issues still persist: 1) Users can hardly control the generated actions with fine-grained spatio-temporal details; 2) Users struggle to further edit the already generated animations. These two drawbacks significantly limit the application scenarios of these algorithms. Therefore, in this paper, we propose a new task, Fine-ge aene-grained Spatio-temporal Motion Generation and Editing. We anticipate that the algorithm framework derived from this task will better accept detailed commands from users and interactively optimize the generated action sequences according to users' further needs. This would thus allow text-driven motion generation technology to be utilized more widely by the general public.
Previous work has made some initial strides in fine-grained motion generation, but often ends up in one of two extremes: 1) it heavily relies on full supervision, producing good generation results but requiring extensive and detailed annotation []]]: 2) it fully depends zero-shot inference, allowing use in various scenarios, but the generated results often contain many artifacts [20, 18]. These work have not deeply explored the fine-grained correspondence between text and motion. This fine-grained relationship is represented spatially as the coordination of body parts and temporally as semantic consistency. With the help of such granular relationships, sequence of motions that is spatially reasonable and temporally semantically consistent can composed from the bottom up. To better
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Genrration
acoilen sle Juine
Hesd Prompt
Log Pouepo
Editing
camus
Are Froept
cnl
Figure 1: FineMoGen is a motion diffusion model that can accept fine-graines spatio-temports descriptions. The synthesized motion sequences natural and consistent with the given conditions. With the assistance of Large Language Model (LLM), users can interactively edit the generated sequence.
capture this correlation, we propose a new framework, FineMoGen. We explicitly model the influence of space and time in the attention mechanism we use, which not only improves the performance in full-supervise scenarios, but also mines spatio-temporal information from standard text-driven motion generation datasets, achieving better results in zeot scenarios. Specifically, we have made two modifications based on the Efficient Attention [16] used in previous work [20, 21J: 1) We delve deeper into the properties of Efficient Attention used therein, proposing a spatio-temporal decoupled method for explicit modelling. This not only supports both single text condition and fine-grained text condition formats during training. but also allows for the introduction of more conditions during testing; 2) We employ Sparsely-activated Mixture-ot coaee-ff---ptett to adaptively provide the most suitable information, thereby optimizing feature quality.
To thoroughly evaluate our proposed algorithm, we have conducted extensive experimental analysis on existing standard datasets such as HumanML33D [6]. KIT-ML [12], and BABEL [13]. However, the existing datasets only provide rough description of human motion a whole, without fine-grained descriptions of body parts and temporal decomposition. To promote fine-grained motion synthesis, we selected the comprehensive HuMMan [3] dataset and provided it with fine-grained text annotations. Specifically, we divided long motion sequence into several different motion stages. For each stage we provided an overall text description and detailed descriptions for seven different body parts. This dataset contains total of 2,968 videos and 102,336 text annotations, which can effectively support future research on fine-grained apatio-tempordl motion generation algorithms. Experimental results show that method has achieved state-of-the-an -ftthe-rrtttt levels these benchmarks. Visualization results further demonstrate that our method can generate more natural motion sequences that are more consistent with the given commands, whether in full-supervise scenarios zero-shot scenarios.
Building on the foundation of fine-graine -reiiga spatio-temporal motion generation, we further explored its integration with large language models (LLMs). Our framework allows users interact with the LLM using natural language instructions, which turn automatically modify the fine-grained descriptions. This enables our algorithm to adjust the content and style of the generated motion sequences according to user commands, enhancing interaction efficiency and extending application possibilities.
Our work has made the following three major contributions:
1. We have proposed the first framework capable of fine-graines motion generation and editing that applies to both zero-shot and fully-supervised scenarios.
2. We have delved deeply into the attention technology used in the motion diffusion model, proposing a apatio-tempora decoupled method for generating global templates, enhanced by the use of Mixture of Experts.
3. We have established a large-scale dataset with fine-grained spatio-temporal text annotations, HuMMan-MoGes to facilitate future research in this direction.

---

2 Related Works
2.1 Text-driven Motion Generation
Recent years have seen significant progress in text-driven motion generation. Traditional research in this field often requires the model to learn a multimodal joint space comprising both text descriptions and motion sequences. For instance, JL2P [1] integrates language and pose into joint embedding space. Ghosh et al. [5] strive to learn a joint-level mapping between language and pose, breaking down the pose embedding into two separate body part embeddings. MotionCLIP [17] also aligns text and motion in shared embedding space. By leveraging the shared text- text-image latent space learned by CLIP, MotionCLIP is capable of generating out-of-distribution and stylized motions. In the pursuit of generating motion sequences with high diversity, several works have introduced variational mechanisms. For instance, TEMOS [11] employs transformer-bet motion sequences conditioned on text descriptions. Guo et al. [6] propose an auto-regressive conditional VAE that generates a short clip a recursive manner. TM2T[8] leverages a neural model for machine translation to facilitate the mapping between text and motion. T2M-GPT[19] reframes the problem of text-driven motion generation as a next-index prediction task by mapping motion sequence to sequence of indices. Recently, diffusion- based generative models have shown impressive performance on leading benchmarks for soxttto-motin task. MotionDiffuse [20], MDM [18], FLAME [10] the first attempts to apply diffusion model into text-driven motion generation field. MLD [4] further exploit the usage of latent diffusion model and achieve a better performance. ReMoDiffuse [21] integrates the retrieval technique motion generation pipeline. The retrieved samples provide informative knowledge for the generation process and thus can generate more natural motion sequences.
VAEs to produce
2.2 Spatio-Temport Motion Composition
TEACH [2], based on the TEMOS [11] algorithm, extracts features from previously generated motion sequences and combines them with text features to obtain a latent code for motion reconstruction. This method is simple and effective, but it must be established under full-supervised scenario fit the adjacency relationship between motions in adjacent labels, limiting its usage in broader scenarios. Furthermore, all conditions are compressed into single latent code, which can easily lose low-level kinematic characteristics. On the other hand, diffusion model-based motion generative models can leverage the advantages of diffusion models for zero-shot spatio-temporal composition [20, 18, 15]. Going further, MofionDiffuse [20] proposes a smoothing term to make the generated motions appear more natural. PriorMDM [15] present a two-stage generation scheme as an inference trick to generate smoother motion sequence. However, due to the lack of effective supervised learning, these types of generated motions often contain many artifacts, such as abrupt changes in speed. These two predicaments mean that there is currently no appropriate method that can truly meet user needs. In this paper, we introduce novel attention mechanism that capable of not only uncovering spatio-temporal correlations in standard text-to-motion benchmarks but also more effectively learning transition schemes a fully-supervised context.
3 Methodology
In this section, we will thoroughly introduce our proposed method, FineMoGen. We first present our overall framework in Section 3.1. Then we define the problem of Fine-grained Spatio- namio-ae Generation and list the preliminaries used in our method, including the formulas for the diffusion model and the network architecture in Section 3.2. We then detail our proposed
Temporal MIxture Attention (SAMI), in Section 3.3. In Section 3.4, we explain how we have implemented zero-shot motion editing by leveraging the definition of fine-grained description. Finally, Section 3.5, we discuss the methods used in the training and testing phases.
3.1 Framework Overview
Figure 2 illustrates the entire process of our proposed FineMoGen, transformer-based motion diffusion model. As for fine-grained spatio-temporal motion generation, the fine-grained descriptions first passed through frozen-CLin model to generate text feature matrix. This feature matrix is
3

---

Mosion Editing
Motion Generation
Tanporal
Spocial
vami the person wove higher and for longer time
Text Encoder
Fine-gra Fine-grained Description
Text Feature Matrix
Inatracken
Concs:
Diffusion Model
LLM
co
Modified
Description
Figure 2: An overview of Fine.MoGen. As for the motion generation task, the fine-grained descrip- tions are first processed by a text encoder. A feature matrix can be acquired, and then sent to diffusion model-based motion generative network to generate corresponding motion sequence. For the editing purpose, I.L is used to interact with users and modify the fine-grate tne-grained description accordingly.
then fed into the transformer, where it influences the generation of various global templates within the SAMI mechanism. During the editing phase, users interact with a large language model (LLM) to efficiently modify the existing fine- fine-grained descriptions. After updating the conditions, FineMoGen generates new motion sequences that better align with the users' expectations.
3.2 Preliminaries
Similar to the standard text-driven motion generation task, the Fine- line-grained Spatio-temporal motion generation also demands the model to generate realistic action sequences consistent with the given text description. The difference lies in that the latter provides multiple fine- grained descriptions with their respective time ranges and body part scopes. Formally, in Fine-grained description matrix Text,,,, [1, Nr E Ns). where Nr and Ng indicate the number of stages and number of body parts divide. Accordingly, the motion sequences generate need to satisfy all constraints, ie.. the entire action sequence è can divided into Nr stages, with each stage containing Ng body parts. We expect that the motion sequence O, corresponding to the jth body part in the ith stage, aligns with the given text description Text,,j- Additionally, we aim for the generated motions to appear natural as whole. This means that we need to ensure smooth coordination among different body parts' movements and achieve seamless transitions between different time segments.
Problem Definition.
Spatio-temporal motion generation task, we define the annotation format
Diffusion Model. In line with the approach of ReMoDiffuse [21], we have developed transformer- based diffusion model specifically to tackle the fine-grained spatio-temporal motion generation task. The diffusion model integrates two key components for optimal generation quality: a diffusion process and a corresponding reverse process.
Diffusion Process is modeled as T 7'stte Markov chain, where the noised sequences XI...., are distortions of the real data X1 ~ q(xa). The intermediate sequence x; can represented by series of poses 0g IR,, , 2, .... Nm. where D is the dimensionality of the pose representation and Nm, is the number of frames. Each diffusion step inject Gaussian noises the data following the formulation: q(x, xx-) := N( vI -, B,I). An efficient approximation from Ho et al. [9] simplify the multi-step diffusion process by x, VaxXo + VI diee where a, := - 3
and ā, = II:m1 a...
Reverse Process stands as the counter-process tasked with eliminating the injected noise from the distorted motion sequences. We employ learnable deep learning model X, Sg(x,, c) to approximate the original sequence. Here, e signifies the provided condition. In the standard text-driven motion generation, is the singular text prompt given. However, for the fine-grained spatio-temporal motion generation task, e embodies the provided detailed linguistic constraints. The training objective for this network minimize the mean square error between the true original sequence Xo and the estimated one x0, which can be formulated as Ez Xx Sg (x,, t, c)}.

---

(VE))
0 1
(Yr33
Output
Sivkratze
Tarperal inspeedenne Medeling
N
ITN
(WL))
(Guj)
Sylustie
(Bay)
(V.p.
G Gi Gia Gia Clg
Giv, Gay Clap
Embbesding
Mal
Mart
Iupal
[Texte,]
X. (3g
Xx
Motion Transformer
b) SAMI
c) Temponil Independence Modelling
Figure 3: Architecutre of FineMoGen.In Figure a), our employed Motion Transformer first encodes the timestamp t, the noise-infused motion sequence x,, and descriptions Text,,; , into feature vectors, which undergo further processing by N layers. Central to these layers lies our proposed SAMI, which takes into account both spatial independence and temporal independence. In Figure (b)'s 's left segment, the feature fusion among different body parts is demonstrated. This fusion driven by a sequence of learnable parameters wijj and the enhanced feature Bk.j derived from MoE projection. The outcome of this spatial modeling is denoted as the feature vector Y k Figure b)'s right segment illustrates the process of temporal modeling. Initially, text features and motion features are concatenated and fed into MoE layers, yielding informative features Kij.A and Vi...b. These two feature vectors are then merged into global templates Gi For each time moment, the refined feature is obtained from these global templates through temporal independence modeling, as depicted in Figure c). Here, we generate Ng time-varying signals. After normalization by the time differences between vy and time anchors G we ultimately acquire the refined feature Y K,3 Upon summation with Y R, the final output of SAMI, denoted as Yk,i, is achieved.
Motion Transformer Network are frequently chosen as the foundation for estimators motion diffusion models. These networks consist of several identical blocks that sequentially process the input data as shown in Figure 3. For the provided linguistic prompts, a pre- trained CLIP model [14] is utilized to first extract informative and generalized features. These features are further refined by a series of trainable transformer layers to better align with the requirements of the task at hand. In line with ReMoDiffuse, each block in FineMoGen integrates an efficient mixed attention module and a ferd-forward network (FFN). This paper delves into the operational principles of the mixed attention mechanism and subsequently presents novel attention mechanism specifically designed for spatio-temporal motion composition.
3.3
Mixture Attention
In the attention module, we aim to accurately apply fine- ahe-ggained textual information to the corre- sponding body parts and time intervals. First we introduce the baseline attention mechanism we used and then illustrate how we modify it to achieve both spatial independence modelling and temporal independence modelling.
Baseline Mixed Attention. Here, we follow the design of attention in ReMoDiffuse [21] by incorporating efficient self-attention and efficient cross-attention [16] within the same module, forming a unique Mixed Efficient Attention module (MEA). Suppose the input motion feature sequence and text feature sequence are Xm. E NLN X, E N... respectively. N,, and N, are the lengths of motion and text sequences. I'm and L, are the dimensionality of them. The core idea of MEA is the usage of global templates instead of pair-wise attention. Formally, all elements in Xm and X, first produce a matrix V E (NNn+NN)) H L, by linear projections induced by the weight W. x attention mechanism and Lg is the dimensionality of global templates. Similarly, we get matrix and perform softmax operation on the last dimension, where N is the number of global templates in each head. These two matrix are multiplied together get all global templates G R xx,N This process can be written as:
W.Y E (H-LL) where H is the number of heads used in multi-head
K E Xx
V [W Xmi W.' Xz), K [ Xmi W.K. Xz), G Softmax(K)
(I)
5

---

where I indicates concatenation of two tensors. Matrix K can be regarded as a series of coefficients that K. shows the influence magnitude of kth element on G the jth global template in ith group. The features in V are summarized to H Ng groups with the normalized weights from K, results the global templates G.
Similar to standard self-attention, we also produce a matrix Q E RR XHNN to query on the global templates. Qu.i.i illustrates the significance of G. on the kth element. Here we also perform the Softmax operation on the last dimension. and then perform multiplication between Q and G to get the refined feature, as the output of this attention module. The process formulated as:
Q WoX,, xm: Y Softmax(Q)G.,
where Y RRNxxHXL is the output matrix.
Temporal Independence. An intuitive solution is to generate independent refined features for different time intervals and apply them to the corresponding motion segments. However, it lacks a direct mechanism to effectively integrate information across different time intervals and becomes challenging to generate satisfactory results when we want to generate a varying number of time intervals during testing. The core issue lies in the difficulty of describing the interactions between different time intervals in scalable manner. To tackle with this issue, we explicitly introduce the concept of time into this procedure. Formally, we propose the following approximation for temporal feature refinement:
No
Y = px(2x) = H (T)( (Tx)
j=1
where I& represents the time position of kth pose state in the motion sequence. (z) indicates the time-varied signal derive from the feature vector Gūj and ( denotes the relative significance of this template for the kth position. Consider the properties of motion generation task, we further construct G. (zr) and G (xx) as:
- 12/2
G)y(()= - G G + G (x (G G()()))))))
EiELLNN,) e (mx 1022
In this setup, the jth global template of the ith group is considered as a set of signals propagating outward from the temporal center G We perceive global template Gi, anchor with its initial state defined as G and velocity represented by G.-,- These three matrices are obtained from the original G through three separate linear projections.
Furthermore, to assimilate influences from all signals, we use the square of time difference as a measure to evaluate the importance of each global template and use a Softmax operation to normalize their weights. A direct advantage of this modeling approach is the ease in appending a new stage following the current one, by adding bias term to the Gi accordingly. Therefore, this method proves beneficial in both fully-supervise and zero-shot scenarios.
Spatial Independence. In conventional motion generative models [20, 11, 18], the raw motion vectors are directly inputted into a fully-connectee layer to generate latent representation: F (e), where θ signifies the raw data of a single motion sequence. F represents the latent representation following the linear projection performed by W.. This operation, while straightforward and effective, often overlooks the distinct interpretations of the motion representation. It also merges data from various body parts, which hampers body part-aware modeling. To overcome this, we manually divide the raw representation into Ns groups, each representing a commonly observed body part. In our work, we consider seven body parts: head, spine, left arm, right arm, left leg, right leg, and trajectory of the pelvis joint. We perform linear projection each part independently and concatenate them into the latent representation F' E N XN where Lp represents the dimensionality of each part. Given our aim for each global template group correspond exactly to one body part, we set Ns = Ng. Additionally, we utilize the Feed Forward Network (FFN) module to process each body part independently.
In addition, we enforce that features from each body part can only contribute to the corresponding group of global templates. In situations where our models are trained without detailed descriptions for each body part, text features from the overall sentence will contribute to each group. However, in the context of the Fine-grained Spatio-temporal Generation task, where we have multiple texts

---

describing different parts, each text feature sequence will only influence the corresponding body part. This group division approach ensures the preservation of spatial independence.
As a supplement, we introduce the spatial feature refinement here to allow the communication between different body parts in SAMI:
Ns Σ Bk.j Wisj
Y
(5)
where BR.j E R the projected feature from jth body part of kth element. Wa. E N is a learnable parameter to reflect the relative significance. The value of is shared to calculate Y for all and can be explained as a common knowledge to show the correlation between different body parts.
Cearsely-Activatte Mixture-of--xpert The design of spatio-tempora independence decouples various components of the entire sequence, compelling the model to focus more on local consistency. This introduces certain complexities the model's learning process. To address this, we attempt to broaden the overall network structure to enhance learning capability. Thus, we employ the Sparsely- activated Mixture-of-Expee technique, which significantly boosts learning effectiveness at the cost of marginal inference time increment. It can adaptively extract informative features from the motion sequence. Specifically, the linear projections we used for calculating BR. Qi.j. Kig from motion features are empowered by MoE. The formulation can be written as:
No.
MMOE (x)
Σ
W26(W32),
where Wi, 22, W3 are learnable parameters. T-Px(.) is a one-hot vector to set all elements to zero except for the largest l ones. Since text features unchanged during the denoising process, adaptive expression is unnecessary for this part.
Module Output. SAMI combines above-mentioned two refined features. The results can be formulated as Y = Y Y
3.4 Zero-shot Motion Editing
In Section 3.2, we defined fine-grained description format to precisely control the actions each body part should perform at each time interval. Based this, through a pretrained Large Language Model (LLM). We use this intermediate format as the current state corresponding to the action, and the user input natural instructions. The LLM will modify this intermediate format according to the input content to get new fine-grained description as shown in Figure 2. The motion generation model FineMoGen will then generate new motion sequence from scratch based on the modified description, serving as the result of the editing. Such a zero-shot pipeline, using the fine-grained description and LLM as medium, translates the user's natural instructions into content that the model can understand and accept, thereby achieving the effect of editing.
interactively edit with users
The teaser figure in Figure 1 provides an example of this procedure. begins with the user providing detailed description for two-stage animation. Subsequently, editing command, such as "I want the person to wave higher and for a longer duration," issued to modify the motion description for both arms in the second stage. Consequently, the motion in the second stage differs from the previous version. Additionally, the motion in the first stage may also differ from the previous one due to the re-generation process.
3.5 Training and Inference
In line with the motion diffusion model literature [21, 18], we randomly mask 10% of the text conditions to approximate p(Xo). Our training objective reduce the mean square error between the predicted initial sequence and the actual ground truth. During the training phase, we generally employ diffusion process comprising 1000 steps, while 50-step denoising process applied the inference stage.
7

---

Stage
Stage 2
Stage
Stage
Left & Right upper Himb: Raise the arms body sodes parallel the ground Left & Right alternately open the logs until the feet are 3-4 fact spart.
Head: Turn the right. Right limnk: Point right right Band right lanes
Head: Turn the froot. Jower limb: Point the right foot
Left upper limah: Put dowa the anis the of the body.
Shitt
Right lower limnh:
to the right unail the thigh parallel the ground.
the nghtt kneo and straightes
feet altemazely to move the legs inwards until they touch each other.
the right
Pelvis: Shifted downsard.
Shifled downward right.
Pelvis: Shifled upward left.
Shifled upward.
Figure 4: Examples in HuMMan-MoGed dataset. A motion sequence will be divided into multiple stages. As for each stages, we provide detailed annotation for different body parts.
4 Experiments
4.1 HuMMMan-MoGen Datasets
In addition to commonly used dataset, we re-label the HuMMan [3] dataset, serving as a large- scale benchmark to evaluate Fine-grained Spatio-Tempont Motion Generation quantitatively. The HuMMan [3] dataset includes total of 500 types of fitness movements, covering the majority of actions perform in our daily lives across various body parts. We selected 2,968 videos from the 160 types of actions in the HuMMan dataset to be annotated in detail. For each type of action, we first define several standard action phases based the specifications of that action. We provide detailed descriptions of the movements for each body part during each phase. Specifically, we annotated overall description, and seven more detailed annotations describe the head, torso, left arm, right arm, left leg, right leg, and trajectory of the pelvis joint. When annotating each video, divide the original video into multiple standard action phases. For each phase, we apply the pre-defined fine-grained textual descriptions to accurately describe the spatial and temporal aspects of the captured motion data. Figure 4 show an example of our annotated motion sequence. The total number of annotations reached 102,336. This fine-grained spatio-temport description provided by our data not available other datasets. The broad range actions covered by this dataset makes more challenging and valuable as a reference.
4.2 Standard Benchmarks and Evaluation Metries
Standard Benchmarks. Apart from our collected HuMMan-Mo dataset, we also evaluate our proposed pipeline on several leading benchmarks in text-driven motion generation tasks, the KIT-MI dataset[12], 12], the HumanML3D dataset[7] and the Babel dataset[13]. We use these three benchmarks for more comparison.
Evaluation Metrics. We employ the same quantitative evaluation measures as used in the literature, including Frechet Inception Distance (FID), Precision, Diversity, Multimodality, and Multi-Modaa Distance. The details of these metrics are discussed in supplementary materials.
4.3 Implementation Details
As for all four datasets, we choose almost the same hyperparameters, Regarding the motion encoder, we utilize 4-layer transformer, with latent dimension of 7 * 64. Here, corresponds to the number of body parts, and 64 represents the dimensionality for each part. For the text encoder, we build and apply a frozen text encoder found in the CLIP ViT-B/32, supplemented with two additional transformer encoder layers. In terms of the diffusion model, the variances, denoted as Br, are predefined to linearly spread from 0.0001 to 0.02, with the total number of noising steps set as T = 1000. We use the Adam optimizer train the model, initially setting the learning rate 0.0002.
8

---

Table I: Quantitative results on the HumanMMLD test set. "('()") indicates that the values better if the metric is larger (smaller). We run all the evaluations 20 times and report the average metric and 95% confidence interval is. The best result are in bold and the second best result are underlined.
Methods
Top
Precisiont Top
Top
FID
MM Disti
Diversity
MoltiModality
Real motions
0.511=:003 003
0.703=.003 Dos
0.797#.002 2002
0.002 000
2.974#.005
9.503 065
Guo et af. [6]
0.457= 002
0,639- m0s
0.7404-0mx 003
1.0672 002
3.340
9.1882 002
2,090 ead
T2M-GPT [19]
0.491 = 003
0,680 003
0.775 2:754.000 002
0.116 004
3.1184-011
9. 0.761 4: 001
1.856 o1
MDM [18]
0.611 007
0.544 ocd
5.566 -027
9.559 066
2.799 ora
MorionDiffuse [20]
0.491
0.681 0.681 #0003
0.782
0.630
3.113
9.410#.03
1.553 042
ReMoDiffuse [2]]
0.510
0. 698
0. 7979 0.795+-004
0.103
2.974
9.018
1.790
Ours
0.501
0.690
0. 0.784 0.78-120002
0.151
2.998 2.998
9. 9,263--094
2.696=110
Table 2: Quantitative results on the KIT-MI test set.
R Precision
Methods
FID
MM Dist
Diversity
MultiMedality!
Top
Top
Top
Real motions
0.424 005
0.649- D0G
0.779--00x 008
0.031 - 001
2.7884-012
11.08-
Gua af. [6]
0.370 005
0.569 007
0.6934 007
2.770 109
3.401 4.
10.91 TT8
1.482 065
T2M GPT [19]
0.416
0.627#
0.745*.00 000
0.514
3.007#.028
10.921 +
1.570
MDM 118]
0.396 0.39652.00
0.497=
9.191 20002
10.8471
1.9071
DifionDi [20]
0,417
0.621 904
0. 0.739 004
1.9544 ose
2.958 2.9584.000
11.10 143
0.730 013
ReMoDiffuse [21]
0.127#.024
0.6-11 0.6:11#0004 004
0. 0.765 t..5
0.155 1.655.000 vos
2.814 0322
10.80
105
1,239 028
Ours
0.432 +. 000
0.649 + 006
0.772
000
0.178 007
2.869#.014
10.85 115
1.877 093
This learning rate will gradually decay to 0.00002 in accordance with cosine learning rate scheduler. Training is performed using one Tesla V V100, with the batch size on single GPU set at 128.
As for the HumanML3D and KIT-ML datasets, we follow Guo et al [6]. where pose states mainly contain seven different parts: (r"a, ,r"x," ,"=,,""= rh .J" .J" J""). Here Y-axis is perpendicular to the ground. pma Fx 12 E R denotes the root joint's angular velocity along Y-axis, linear velocity along X-axis and Z-axis, respectively. T E I is the height of the root joint. j E TR-xx3 are the position and linear velocity of each joint, where J the number joints. j E R joint. Specifically, J is 22 in HumanML3D and 21 in KIT-ML. Tuning to the BABEL dataset and our proposed HaMMMan-MoGer we follow TEACH [2] and converts SMPL parameters to a 6D rotation representation together with root translation.
the 6D rotation of each
4.4 Quantitative Results
We begin by assessing our proposed method the standard task of text-driven motion generation. As depicted in Table and Table 2, FineMoGen yields results that are competitive with those from existing methods. It's worth noting that while FineMoGen isn't specifically designed for this task, it still achieves satisfactory performance in comparison to other methods.
Table 3 displays the performance of temporal composition on both the BABEL dataset and the MuMMMnn-MoGie dataset. Our FincMoGen significantly surpasses existing methods in terms of accuracy. This underscores FineMoGen's 's ability to synthesize motion sequences that are not only more natural but also more aligned with the provided text conditions.
4.5 Qualitative Results
Please kindly refer to the supplementary material for qualitative comparison and more demos about both fine- gre-grai motion generation and editing. From the demo video, we can observe the following: 1) Our method demonstrates a better understanding of the provided textual context compared to other approaches. Whether it's single-action generation or continuous action generation, our method generates motion sequences that are more consistent with the text. 2) Due to mathematically coherent temporal modeling, in continuous action generation, the transitions in our generated results are natural, with no apparent artifacts.

---

Table 3: Quantitative results of temporal composition the BABEL test set and HuMMan-
MoGen test set.
BABEL
HuMMan MoGien
Methods
Precisiont
FIDL
Diversity -> MullMMddllity1
Precisiont
FIDL
Diversity -
MultModalityy
0.62
3.57
0.49
0.007
2.15
TEACH [2]
0.46
1.12
8.28
7.14
0.38
1.59
7.04
6.37
PriorMDM [15]
0.43
1.04
8.14
7.39
0.31
1.43
6.89
6.45
Ours shot)
Ours (fully--uuent suprrviss))
0.56
0.73
7.94
7.17
0.43
1.21
6.71
6.28
Table 4: Ablation study on HuMMan-MoGen test set. All methods use zero-shot setting, it means that they are not trained on the temporal composition data.
Methods
Spatial Dependence
Temporal Dependence
MoE R Precision
FID
Diversity
MultiModalit"
Real Motions
0.49
0.007
7.28
2.15
Baselines
0.24
7.10
0.21
4.51
7:29
7.03
0.34
2.48
7.05
6.72
/
0.27
3.09
6.97
6.84
FineMotien
0.41
1.34
6.87
6.41
4.6
Ablation Study
Table 4 presents the ablation study conducted on the HuMMan-M dataset. For consistency, we consider only temporal composition in alignment with the results shown in Table 3. The outcomes suggest that both Temporal Dependence and MoE components contribute positively to the overall performance. Conversely, spatial dependence poses challenges for fitting the distribution. We concede that in a temporal composition setting, this module may not be necessary. We also provide ablation analysis on spatial combination in the supplementary material, which indicate consistent conclusions.
5 Conclusion and Limitation
In this work, we introduce the task of Fine-grained detailed textual descriptions allow users to control and edit the desired actions with greater flexibility, To address this task, we propose a new framework, FineMoGen, which decouples the influence of body parts and time on the feature refinement process the attention mechanism. This results in superior performance in both fully-supervised and zero-shot scenarios compared to previous works. Given the lack of text-motion datasets specifically designed for fine-grained descriptions, we have compiled dataset based the HuMMan dataset. This dataset includes a wide range of actions and an abundance of descriptions segmented by time and body part. These data enable us to analyze and test the effects of different algorithms more effectively.
Motion Generation, where more
Limitation. While this study proposes method for spatial decoupling, is still heavily reliant on specific fine-grained format, which to a certain extent limits its application scenarios. On the other hand, the efficacy of the motion editing component strongly tied the capability of the pre-trained Large Language Model (LLM). In practical use, it sometimes struggles to interpret user requirements accurately. Instruction tuning should be incorporated into the framework in the future, enabling the LLM to better understand instructions related to motion editing.
Boarder Impact. The ability generate fine-grained human motion not only enhances the productiv- ity of the animation industry, but also has potential benefits for helping medical research, improving training athletes and dancers, and providing more realistic interactive experiences in virtual reality. However, it may also be used for illegal activities such as generating fighting motions to create violent videos, or fraud and dissemination of false information.
Acknowledgment
This study is supported by the Ministry of Education, Singapore, under its MOE AcRF Tier 2 MMFF-22FP0221-000 NTU NAP, and under the RIE2020 Industry Alignment Fund Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry psrtner(s).
10

---

References
[1] Chaitanya Ahuja and Louis-Philippe Morency. Language?pose: Natural language grounded pose forecasting. In 2019 International Conference on 3D Vision (3DV), pages 719-728. IEEE, 2019.
[2] Nikos Athanasiou. Mathis Petrovich, Michael Black, and Gül Varol. Teach: Temporal action composition for 3d humans. In 2022 International Conference on 3D Vision (3DV), pages 414- IEFF, 2022.
[3] Zhongang Cai, Daxuan Ren, Ailing Zeng, Zhengyu Lin, Tao Yu, Wenjia Wang, Xiangyu Fan, Yang Gao, Yifan Yu, Liang Pan, Fangzhou Hong. Mingyuan Zhang, Chen Change Loy, Lei Yang, and Ziwei Liu. HuMMan: Multi-modal 4d human dataset for versatile sensing and modeling. In 17th European Conference on Computer Vision, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part VII, pages 557-577. Springer, 2022.
[4] Xin Chen, Biao Jiang, Wen Liu, Zilong Huang, Bin Fu, Tao Chen, and Gang Yu. Executing your commands via motion diffusion in latent space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18000-18010, 2023.
[5] Anindita Ghosh, Noshaba Cheema, Cennet Oguz, Christian Theobalt, and Philipp Slusallek. Syn- thesis of compositional animations from textual descriptions. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1396-1406, 2021.
[6] Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5152-5161, 2022.
[7] Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5152-5161, June 2022.
[8] Chuan Guo, Xinxin Zuo, Sen Wang, and Li Cheng. Tm2t: Stochastic and tokenized modeling for the reciprocal generation of 3d human motions and texts, arXiv preprint 2xXi:22077.01666 2022.
[9] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851 6851, 2020.
[10] Jihoon Kim, Jiseob Kim, and Sungjoon Choi. Flame: Free-form language- based motion synthesis & editing. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 8255-8263, 2023.
[11] Mathis Petrovich, Michael J Black, and Gül Varol. Temos: Generating diverse human motions from textual descriptions. In European Conference on Computer Vision, pages 480-497. Springer, 2022.
[12] Matthias Plappert, Christian Mandery, and Tamim Asfour. The kit lationn dataset. Big data, 4(4):236-252, 2016.
[13] Abhinanda R Punnakkal, Arjun Chandrasekaran, Nikos Athanasiou, Alejandra
and Michael Black. Babel: Bodies, action and behavior with english labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 722-731, 2021.
[14] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[15] Yonatan Shafir, Guy Tevet, Roy Kapon, and Amit H Bermano. Human motion diffusion as a generative prior arXiv preprint arXiv:2303.01413 2023.
[16] Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Efficient attention: Attention with linear complexities. In Proceedings of the IEEF/CVE winter conference on applications of computer vision, pages 3531-3539, 2021.

---

[17] Guy Tevet, Brian Gordon, Amir Hertz, Amit H Bermano, and Daniel Cohen-Or. Motionclip: Exposing human motion generation clip space. European Conference on Computer Vision, pages 358-374. Springer, 2022.
[18] Guy Tevet, Sigal Raab, Brian Gordon, Yoni Shafir, Daniel Cohen-or, and Amit Haim Bermano. Human motion diffusion model. In The Eleventh International Conference on Learning Repre- sentations, 2022.
[19] Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Shaoli Huang, Yong Zhang, Hongwei Zhao, Hongtao Lu, and Xi Shen. T2m-gpt: Generating human motion from textual descriptions with discrete representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2023.
[20] Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondiffuse: Text-driven human motion generation with diffusion model. arXiv preprint (rXiv:2208,15000 2022.
[21] Mingyuan Zhang, Xinying Guo, Liang Pan. Zhongang Cai, Fangzhou Hong, Huirong Li, Lei Yang, and Ziwei Liu. Remodiffuse: Retrievallaugmented motion diffusion model. arXiv preprint 20X4v:2304.01116, 2023.
12


Reviews:
[{"review_id": "lM4CFRJOnp", "reviewer": "Reviewer_C9Az", "summary": "The paper introduces a compositional text-to-motion synthesis approach that allows for control up to body part level. For fine-grained spatio-temporal editing, the motion of each body part from a predetermined set (e.g., head, arms, legs, etc.) can be described. Furthermore, for zero-shot motion control, the model has an interface with LLMs to support high-level motion descriptions, which are converted to part-specific commands. The paper introduces FineMoGen, a diffusion-based model with novel spatio-temporal mixture attention (SAMI). Specifically, each body part is represented with a spatial attention block and conditioned only on the corresponding part’s text feature to ensure spatial independence. Similarly, the temporal block aims to make smooth transitions between different stages (i.e., different atomic motions). The proposed model performs competitively on the ***atomic text-to-motion benchmarks such as HumanML3D and KIT-ML. In the ***compositional text-to-motion task, it performs quantitatively better than the baselines on the BABEL and the proposed HuMMan-MoGen datasets. There are major artifacts in the synthesized motions (e.g., implausible movements and foot skating), though. \n\n*** I distinguished two tasks by tagging them as \"atomic\" and \"compositional\". While the former focuses on generating a short motion snippet from a single instruction describing an atomic event, the latter also involves transitions between semantically different motion descriptions, requiring the composition of several instructions.\n", "questions": "I have listed my questions regarding the weaknesses of the paper above (Q1 and Q2). Here I am raising additional questions for further clarification.\n\n**Q3**) It is not clear to me how interactive editing works. Is the motion initialized with the output of the previous pass or generated from scratch as an additional stage? \n\n**Q4**) The supplementary video does not have any results for the atomic text-to-motion task. I am curious if the artifacts also exist in this task. Could the authors provide a qualitative comparison with the baselines?\n", "limitations": "Yes, the paper briefly mentions limitations and the broader impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "**originality**\nThe problem setting is interesting and practical. The proposed model is arguably novel. Ideas from the prior works are nicely extended. \n\n**quality**\nThe paper is well-motivated, and the concerns are addressed in the design of the proposed model. The decoupling of the spatial and temporal blocks and the corresponding inputs (i.e., part-specific text commands and descriptions for different stages, respectively) seems to be effective.\n\n**clarity**\nAlthough the presentation of the technical details could be improved, it is straightforward to get an overview of the motivation and the proposed approach. \n\n**significance**\nThe proposed model performs well in evaluations in both the atomic text-to-motion and compositional text-to-motion tasks by allowing for fine-grained editing. I believe this is an interesting direction and could be valuable for the community. \n", "weaknesses": "* The motion quality of the synthetic samples is one of the major weaknesses of this work. While the motion samples in the first part of the supplementary video (examples 1, 2, and 3) are okay, the rest, including the qualitative comparison and the editing sections, have major artifacts. Unfortunately, the inference settings (i.e., the dataset, type of text commands, etc.) are not provided. (**Q1**) Could the authors explain why this is the case? \n\n\n* (**Q2**) Similarly, the paper does not provide details on the underlying motion representation. Is it 3D positions or rotations? If the latter, which representation? What do the \"raw motion vectors\" look like? \n\n* I find it not straightforward to follow the technical details in Section 3.3. At least the transition from the \"Efficient Attention\" paragraph to Section 3.3 was not smooth. I think a revision would help. Moreover, an intuitive figure could be highly useful for conveying the message.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your recognition of our work’s strengths and insightful suggestions. Here we respond to each of your comments in detail below.\n\n> **Q1**: The inference setting is not clear. The qualitative results have artifacts.\n\n**A1**: The initial examples represent the ground truth data from our HuMMan-MoGen dataset, while the subsequent ones are all generated by algorithms applied to the BABEL dataset. It is acknowledged that generating fine-grained motion remains a challenging endeavor. Despite our method exhibiting a significant lead over existing approaches, the generated results are not flawless. We are committed to persistently pushing the boundaries of this task.\n\n> **Q2**: The details of motion representation should be clarified.\n\n**A2**: Thanks for your suggestions. As for the HumanML3D and KIT-ML datasets, we follow Guo et al[1], where pose states mainly contain seven different parts: $(r^{va},r^{vx}, r^{vz},r^h, \\mathbf{j}^p, \\mathbf{j}^v, \\mathbf{j}^r)$. Here Y-axis is perpendicular to the ground. $r^{va},r^{vx}, r^{vz} \\in \\mathbb{R}$ denotes the root joint's angular velocity along Y-axis, linear velocity along X-axis and Z-axis, respectively. $r^h \\in \\mathbb{R}$ is the height of the root joint. $\\mathbf{j}^p, \\mathbf{j}^v \\in \\mathbb{R}^{J \\times 3}$ are the position and linear velocity of each joint, where $J$ is the number of joints. $\\mathbf{j}^r \\in \\mathbb{R}^{J \\times 6}$ is the 6D rotation of each joint. Specifically, $J$ is 22 in HumanML3D and 21 in KIT-ML. Tuning to the BABEL dataset and our proposed HuMMan-MoGen, we follow TEACH[2] and converts SMPL parameters to a 6D rotation representation together with root translation. We will add these details in the final version.\n\n\\[1\\] Guo et al. Generating Diverse and Natural 3D Human Motions from Text\n\n\\[2\\] Athanasiou et al. TEACH: Temporal Action Composition for 3D Humans\n\n> **Q3**: How does interactive editing work is not clear.\n\n**A3**: Since the modified segments could potentially impact the continuity, after each description modification, we refrain from using the previous generated results. Instead, we regenerate a new sequence from scratch. We will add more implementation details about this part in the final version.\n\n> **Q4**: Qualitative results in the atomic text-to-motion generation task. \n\n**A4**: We provide the link for a new demo video to AC. We qualitatively compare our method with MDM, MotionDiffuse and ReMoDiffuse. Our results are similar to ReMoDiffuse and outperform the other two. We hope this video can address your concern.\n\n> **Q5**: Method illustration should be improved.\n\n**A5**: Thanks for your suggestion. We add a new figure in the attached PDF. We will carefully polish this part in the final version.\n"}, {"review_id": "EQA7PEJE6s", "reviewer": "Reviewer_DsXZ", "summary": "The paper targets an interesting problem of fine-grained spatial-temporal motion generation and editing. It introduces a new transformer structure with spatio-temporal mixturee attention. Also, a large scale dataset, called HuMMan-MoGen, has been introduced to train the proposed algorithm. Attractive experimental results have been reported in the paper. ", "questions": "Please address the questions in the weakness section. Also, will the dataset be public released together with the finegrained description?", "limitations": "The paper already discussed the limitations in the paper. ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. A new large-scale dataset with fine-grained spatio-temporal descriptions have been introduced in the paper. \n\n2. Attractive performance has been reported on the motion generation benchmark. \n\n3. The problem of fine-grained motion generation and edting would be helpful for the industry. ", "weaknesses": "1. The proposed HuMMan-MoGen dataset is originally from the HuMMan dataset. Is it possible to the dataset convering sufficient diversity of the finegrained motion clips?\n\n2. The annotations of different groups would make it easy to represent the motions. But some motions have been defined as a whole body rather than a combination of motions from different body parts. Is there any potential negative impact on this problem?\n\n3.  I would suggest to include the dataset contruction process in the paper rather than leaving it in the supplementary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your constructive and helpful suggestions. And we carefully reply to each of the reviewer's comments below. \n\n> **Q1**: Is HuMMan-MoGen diverse enough?\n\n**A1**: The HuMMan dataset comprises 500 actions, strategically curated to encompass foundational movements. From this collection, our proposed HuMMan-MoGen carefully selects the most representative 200 categories. These categories encompass a wide range of movement types for each body part. Notably, our dataset is intentionally structured to facilitate spatio-temporal composition, enabling models trained on it to generate an extensive variety of motion clips.\n\n\n> **Q2**:  Some motions have been defined as a whole body rather than a combination of motions from different body parts. Is there any potential negative impact on this problem?\n\n**A2**: Currently, as for each motion sequence, we provide an overall text description and detailed descriptions for seven different body parts. Taking running as an example, we will provide a holistic description such as 'a person is running', as well as finer-grained descriptions for each body part. These detailed descriptions illustrate how the limbs move and coordinate with each other during the running process. Therefore, for such comprehensive descriptions, fine-grained descriptions can serve as supplementary information, offering richer details for the model to reference.\n\n> **Q3**: The dataset construction process should be included in the paper.\n\n**A3**: Thanks for your suggestion. We will refine this part in our final version. \n\n> **Q4**: Will the dataset be public released together with the finegrained description?\n\n**A4**: Yes, we will publish all the SMPL sequences along with their corresponding fine-grained descriptions upon acceptance.\n"}, {"review_id": "4fxVgTorzL", "reviewer": "Reviewer_CFoF", "summary": "The paper proposes a method for diffusion-based motion generation and editing framework for synthesizing fine-grained motions. It takes fine-grained spatio-temporal description. The framework supports zero-shot and fully-supervised generation scenario. The utilize MOE for generating global template. They have also introduced a dataset with fine-grained text annotation. Their method is called spatio-temporal mixture attention (SAMI). To summarize, The fine-grained descriptions are first processed by frozen CLIP followed by two layers of learnable transformer encoders. Next, a text feature matrix is created, and then sent to a diffusion model-based motion generative network to generate corresponding motion sequence.    ", "questions": "Like most other generative methods, it also should have concern for its potential negative impacts. Can some sort of user attribution be incorporated in the model itself? ", "limitations": "The dependencies on the LLM might have two way effect. The natural description of the fine-grained motion may not be best utilized in this scenario. Like most other generative methods, it also should have concern for its potential negative impacts. Can some sort of user attribution be incorporated in the model itself? ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The zero-shot generation is encouraging. The results on HumanML3D and KIT-ML show improved and comparable results with existing methods. The superior temporal composition results show the effectiveness of SAMI. Ablation experiments are a plus. ", "weaknesses": "The dependencies on the LLM might have two way effect. The natural description of the fine-grained motion may not be best utilized in this scenario. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our sincere thanks for your insightful review and constructive suggestions. We reply to each of your comments here:\n\n> **Q1**: The natural description of the fine-grained motion may not be best utilized in some situations.\n\n**A1**: We concur that employing the generated fine-grained description directly for motion editing might not be the optimal approach in certain scenarios. For instance, a more effective strategy could involve dynamically selecting the appropriate level of detail to ensure enhanced precision during generation. We will try this direction in the future.\n\n> **Q2**: Can some sort of user attribution be incorporated in the model itself?\n\n**A2**: Taking advantage of the diversity within the HuMMan dataset, where each action involves participants of varying ages, genders, and body shapes, our trained models inherently avoid any inductive bias. Furthermore, we will also add a license to regulate users' usage methods in order to mitigate potential negative impacts that the model might bring about."}, {"review_id": "H829HLBnHx", "reviewer": "Reviewer_4S8a", "summary": "Existing methods often struggle to generate intricate motion sequences that align with fine-grained descriptions. Addressing this, the authors introduce FineMoGen, a diffusion-based motion generation and editing framework capable of synthesizing intricate, detailed motions that align with user instructions. FineMoGen employs a diffusion model coupled with a unique transformer architecture called Spatio-Temporal Mixture Attention (SAMI). This architecture optimizes global attention template creation by: 1) explicitly modeling the constraints of spatio-temporal composition, and 2) using a sparsely-activated mixture-of-experts to adaptively extract fine-grained features. In addition, the authors offer the HuMMan-MoGen dataset, designed to facilitate the study of fine-grained motion generation and editing. Extensive experimental validation suggests that FineMoGen outperforms state-of-the-art methods in motion generation quality.", "questions": "Please refer to weaknesses.", "limitations": "The authors mention the limitation and suggest a way to resolve it.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The authors' contribution of a novel dataset is significant and could stimulate further research in this field. A fine-grained labeled dataset extends the potential for human motion generation applications and aligns with intuitive human approaches to generating and editing human motion.\n2. The motivation behind the development of the Spatio-Temporal Mixture Attention model is robust and justified.", "weaknesses": "1. The relationship between SAMI and MEA is somewhat unclear, particularly from the perspective of notations. The association between F' and B is not well-explained.\n2. The factorization in Equation 4 could benefit from clearer explanation. The current description of G' as a time-varied signal seems insufficient, and the formulation of G* in Equation 4 lacks clarity, with minimal elaboration provided in the main paper.\n3. Figure 2 does not adequately represent the primary contributions of this paper. It solely incorporates the LLM module, which, while new, is not the main contribution. The SAMI and MoE, which are crucial to this paper, are omitted.\n4. The motivation behind the Sparse-activated Mixture-of-Expert is unclear. I failed to see the inherent correlation between the Sparse-activated Mixture-of-Expert and SAMI.\n5. On Line 214, there appears to be a typographical error - \"vGiven\" should likely be corrected.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the detailed feedback and comments. We respond to the concerns below:\n\n> **Q1**: The method illustration should be improved, such as the difference between MEA and SAMI, the usage of $G’$, etc.\n\n**A1**: Thanks for your suggestion. We add a new figure in the attached PDF, which clearly shows the detailed structure of our proposed SAMI architecture. We will carefully polish this part in the final version.\n\n> **Q2**: The motivation behind the Sparse-activated Mixture-of-Expert is unclear.\n\n**A2**: The design of spatio-temporal independence decouples various components of the entire sequence, compelling the model to focus more on local consistency. This introduces certain complexities to the model's learning process. To address this, we attempt to broaden the overall network structure to enhance learning capability. Thus, we employ the Sparse-activated Mixture-of-Expert technique, which significantly boosts learning effectiveness at the cost of marginal inference time increment. Our Ablation study corroborates the significance of this aspect. In the final version, we will provide additional elaboration on the motivation behind this approach.\n\n> **Q3**: There are some typos in the current version.\n\n**A3**: We will meticulously review the entire text and rectify any errors in the final version.\n"}]

Author rebuttal:
We express our gratitude to all the reviewers for their valuable and constructive feedback.  We are glad at reviewers’ recognition of our work's strengths: 

**1)** The problem setting is interesting and practical (R-C9Az, R-DsXZ, R-CFoF, R-4S8a).

**2)** The decoupling of spatial independence and temporal independence is interesting and achieves impressive results. (R-C9Az, R-DsXZ, R-CFoF, R-4S8a). 

**3)** The collected dataset is significant to the community (R-DsXZ, R-4S8a).

In the accompanying file, we have included a new figure to enhance the clarity of the architectural intricacies in our proposed method.


Meta-review: All reviewers find that the paper is well-motivated and the proposed method is shown to be effective. They appreciate the new large-scale dataset with fine-grained spatio-temporal descriptions that was introduced in the paper.

In the initial round of review, there are concerns about the motion quality of the synthetic samples, not sufficient details on the underlying motion representation, the dependencies on the LLM, unclear relationship between SAMI and MEA, and whether the "Sparse-activated" design is crucial. The authors addressed all these major concerns in the rebuttal period, which is acknowledged by all the reviewers. Therefore, we would like to suggest acceptance of the paper.

It is requested that the authors address all the requests in the ethics reviews. They are also suggested to include all the promised revisions during rebuttal period in the final version.
Decision: Accept (poster)