SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Mingyuan Zhang $^{1}$  Huirong Li $^{1}$  Zhongang Cai $^{1,2}$  Jiawei Ren $^{1}$  Lei Yang $^{2}$  Ziwei Liu $^{1}$
$^{1}$  S-Lab, Nanyang Technological University,  $^{2}$  SenseTime Research
Text-driven motion generation has achieved substantial progress with the emergence of diffusion models. However, existing methods still struggle to generate complex motion sequences that correspond to fine-grained descriptions, depicting detailed and accurate spatio-temporal actions. This lack of fine controllability limits the usage of motion generation to a larger audience. To tackle these challenges, we present FineMoGen, a diffusion-based motion generation and editing framework that can synthesize fine-grained motions, with spatial-temporal composition to the user instructions. Specifically, FineMoGen builds upon diffusion model with a novel transformer architecture dubbed Spatio-Temporal Mixture Attention (SAMI). SAMI optimizes the generation of the global attention template from two perspectives: 1) explicitly modeling the constraints of spatio-temporal composition; and 2) utilizing sparsely-activated mixture-of-experts to adaptively extract fine-grained features. To facilitate a large-scale study on this new fine-grained motion generation task, we contribute the HuMMan-MoGen dataset, which consists of 2,968 videos and 102,336 fine-grained spatio-temporal descriptions. Extensive experiments validate that FineMoGen exhibits superior motion generation quality over state-of-the-art methods. Notably, FineMoGen further enables zero-shot motion editing capabilities with the aid of modern large language models (LLM), which faithfully manipulates motion sequences with fine-grained instructions. Project Page: https://mingyuan-zhang.github.io/projects/FineMoGen.html.
While traditional text-driven motion generation tasks have empowered models to generate motion sequences based on text, two issues still persist: 1) Users can hardly control the generated actions with fine-grained spatio-temporal details; 2) Users struggle to further edit the already generated animations. These two drawbacks significantly limit the application scenarios of these algorithms. Therefore, in this paper, we propose a new task, Fine-grained Spatio-temporal Motion Generation and Editing. We anticipate that the new algorithm framework derived from this task will better accept detailed commands from users and interactively optimize the generated action sequences according to users' further needs. This would thus allow text-driven motion generation technology to be utilized more widely by the general public.
Previous work has made some initial strides in fine-grained motion generation, but often ends up in one of two extremes: 1) it heavily relies on full supervision, producing good generation results but requiring extensive and detailed annotation [11]; 2) it fully depends on zero-shot inference, allowing use in various scenarios, but the generated results often contain many artifacts [20, 18]. These work have not deeply explored the fine-grained correspondence between text and motion. This fine-grained relationship is represented spatially as the coordination of body parts and temporally as semantic consistency. With the help of such granular relationships, a sequence of motions that is spatially reasonable and temporally semantically consistent can be composed from the bottom up. To better
Generation
Figure 1: FineMoGen is a motion diffusion model that can accept fine-grained spatio-temporal descriptions. The synthesized motion sequences are natural and consistent with the given conditions. With the assistance of Large Language Model (LLM), users can interactively edit the generated sequence.
capture this correlation, we propose a new framework, FineMoGen. We explicitly model the influence of space and time in the attention mechanism we use, which not only improves the performance in full-supervise scenarios, but also mines spatio-temporal information from standard text-driven motion generation datasets, achieving better results in zero-shot scenarios. Specifically, we have made two modifications based on the Efficient Attention [16] used in previous work [20, 21]: 1) We delve deeper into the properties of Efficient Attention used therein, proposing a spatio-temporal decoupled method for explicit modelling. This not only supports both single text condition and fine-grained text condition formats during training, but also allows for the introduction of more conditions during testing; 2) We employ Sparsely-activated Mixture-of-Experts to adaptively provide the most suitable information, thereby optimizing feature quality.
To thoroughly evaluate our proposed algorithm, we have conducted extensive experimental analysis on existing standard datasets such as HumanML3D [6], KIT-ML [12], and BABEL [13]. However, the existing datasets only provide rough description of human motion as a whole, without fine-grained descriptions of body parts and temporal decomposition. To promote fine-grained motion synthesis, we selected the comprehensive HuMMan [3] dataset and provided it with fine-grained text annotations. Specifically, we divided a long motion sequence into several different motion stages. For each stage we provided an overall text description and detailed descriptions for seven different body parts. This dataset contains a total of 2,968 videos and 102,336 text annotations, which can effectively support future research on fine-grained spatio-temporal motion generation algorithms. Experimental results show that our method has achieved state-of-the-art levels on these benchmarks. Visualization results further demonstrate that our method can generate more natural motion sequences that are more consistent with the given commands, whether in full-supervise scenarios or zero-shot scenarios.
Building on the foundation of fine-grained spatio-temporal motion generation, we further explored its integration with large language models (LLMs). Our framework allows users to interact with the LLM using natural language instructions, which in turn automatically modify the fine-grained descriptions. This enables our algorithm to adjust the content and style of the generated motion sequences according to user commands, enhancing interaction efficiency and extending application possibilities.
Our work has made the following three major contributions:
1. We have proposed the first framework capable of fine-grained motion generation and editing that applies to both zero-shot and fully-supervised scenarios.
2. We have delved deeply into the attention technology used in the motion diffusion model, proposing a spatio-temporal decoupled method for generating global templates, enhanced by the use of Mixture of Experts.
3. We have established a large-scale dataset with fine-grained spatio-temporal text annotations, HuMMan-MoGen, to facilitate future research in this direction.
2.1 Text-driven Motion Generation
Recent years have seen significant progress in text-driven motion generation. Traditional research in this field often requires the model to learn a multimodal joint space comprising both text descriptions and motion sequences. For instance, JL2P [1] integrates language and pose into a joint embedding space. Ghosh et al. [5] strive to learn a joint-level mapping between language and pose, breaking down the pose embedding into two separate body part embeddings. MotionCLIP [17] also aligns text and motion in a shared embedding space. By leveraging the shared text-image latent space learned by CLIP, MotionCLIP is capable of generating out-of-distribution and stylized motions. In the pursuit of generating motion sequences with high diversity, several works have introduced variational mechanisms. For instance, TEMOS [11] employs transformer-based VAEs to produce motion sequences conditioned on text descriptions. Guo et al. [6] propose an auto-regressive conditional VAE that generates a short clip in a recursive manner. TM2T[8] leverages a neural model for machine translation to facilitate the mapping between text and motion. T2M-GPT[19] reframes the problem of text-driven motion generation as a next-index prediction task by mapping a motion sequence to a sequence of indices. Recently, diffusion-based generative models have shown impressive performance on leading benchmarks for text-to-motion task. MotionDiffuse [20], MDM [18], FLAME [10] are the first attempts to apply diffusion model into text-driven motion generation field. MLD [4] further exploit the usage of latent diffusion model and achieve a better performance. ReMoDiffuse [21] integrates the retrieval technique to motion generation pipeline. The retrieved samples provide informative knowledge for the generation process and thus can generate more natural motion sequences.
2.2 Spatio-Temporal Motion Composition
TEACH [2], based on the TEMOS [11] algorithm, extracts features from previously generated motion sequences and combines them with text features to obtain a latent code for motion reconstruction. This method is simple and effective, but it must be established under a full-supervised scenario to fit the adjacency relationship between motions in adjacent labels, limiting its usage in broader scenarios. Furthermore, all conditions are compressed into a single latent code, which can easily lose low-level kinematic characteristics. On the other hand, diffusion model-based motion generative models can leverage the advantages of diffusion models for zero-shot spatio-temporal composition [20, 18, 15]. Going further, MotionDiffuse [20] proposes a smoothing term to make the generated motions appear more natural. PriorMDM [15] present a two-stage generation scheme as an inference trick to generate smoother motion sequence. However, due to the lack of effective supervised learning, these types of generated motions often contain many artifacts, such as abrupt changes in speed. These two predicaments mean that there is currently no appropriate method that can truly meet user needs. In this paper, we introduce a novel attention mechanism that is capable of not only uncovering spatio-temporal correlations in standard text-to-motion benchmarks but also more effectively learning transition schemes in a fully-supervised context.
In this section, we will thoroughly introduce our proposed method, FineMoGen. We first present our overall framework in Section 3.1. Then we define the problem of Fine-grained Spatio-Temporal Generation and list the preliminaries used in our method, including the formulas for the diffusion model and the network architecture in Section 3.2. We then detail our proposed SpAtio-Temporal MIxture Attention (SAMI), in Section 3.3. In Section 3.4, we explain how we have implemented zero-shot motion editing by leveraging the definition of fine-grained description. Finally, in Section 3.5, we discuss the methods used in the training and testing phases.
Figure 2 illustrates the entire process of our proposed FineMoGen, a transformer-based motion diffusion model. As for fine-grained spatio-temporal motion generation, the fine-grained descriptions are first passed through a frozen-CLIP model to generate a text feature matrix. This feature matrix is
Figure 2: An overview of FineMoGen. As for the motion generation task, the fine-grained descriptions are first processed by a text encoder. A text feature matrix can be acquired, and then sent to a diffusion model-based motion generative network to generate corresponding motion sequence. For the editing purpose, LLM is used to interact with users and modify the fine-grained description accordingly.
then fed into the transformer, where it influences the generation of various global templates within the SAMI mechanism. During the editing phase, users interact with a large language model (LLM) to efficiently modify the existing fine-grained descriptions. After updating the conditions, FineMoGen generates new motion sequences that better align with the users' expectations.
Problem Definition. Similar to the standard text-driven motion generation task, the Fine-grained Spatio-temporal motion generation also demands the model to generate realistic action sequences consistent with the given text description. The difference lies in that the latter provides multiple fine-grained descriptions with their respective time ranges and body part scopes. Formally, in Fine-grained Spatio-temporal motion generation task, we define the annotation format as a description matrix  $\mathrm{Text}_{i,j}, i \in [1, N_T], j \in [1, N_S]$ , where  $N_T$  and  $N_S$  indicate the number of stages and number of body parts we divide. Accordingly, the motion sequences we generate need to satisfy all constraints, i.e., the entire action sequence  $\Theta$  can be divided into  $N_T$  stages, with each stage containing  $N_S$  body parts. We expect that the motion sequence  $\Theta_{i,j}$ , corresponding to the  $j$ th body part in the  $i$ th stage, aligns with the given text description  $\mathrm{Text}_{i,j}$ . Additionally, we aim for the generated motions to appear natural as a whole. This means that we need to ensure smooth coordination among different body parts' movements and achieve seamless transitions between different time segments.
Diffusion Model. In line with the approach of ReMoDiffuse [21], we have developed a transformer-based diffusion model specifically to tackle the fine-grained spatio-temporal motion generation task. The diffusion model integrates two key components for optimal generation quality: a diffusion process and a corresponding reverse process.
Diffusion Process is modeled as a  $T$ -step Markov chain, where the noised sequences  $\mathbf{x}_1,\dots ,\mathbf{x}_T$  are distortions of the real data  $\mathbf{x}_0\sim q(\mathbf{x}_0)$ . The intermediate sequence  $\mathbf{x}_t$  can be represented by a series of poses  $\theta_{i}\in \mathbb{R}^{D},i = 1,2,\ldots ,N_{m}$ , where  $D$  is the dimensionality of the pose representation and  $N_{m}$  is the number of frames. Each diffusion step inject Gaussian noises to the data following the formulation:  $q(\mathbf{x}_t|\mathbf{x}_{t - 1})\coloneqq \mathcal{N}(\mathbf{x}_t;\sqrt{1 - \beta_t}\mathbf{x}_{t - 1},\beta_t\mathbf{I})$ . An efficient approximation from Ho et al. [9] simplify the multi-step diffusion process by  $\mathbf{x}_t\coloneqq \sqrt{\bar{\alpha}_t}\mathbf{x}_0 + \sqrt{1 - \bar{\alpha}_t\epsilon}$ , where  $\alpha_{t}\coloneqq 1 - \beta_{t}$  and  $\bar{\alpha}_t\coloneqq \prod_{s = 1}^t\alpha_s$ .
Reverse Process stands as the counter-process, tasked with eliminating the injected noise from the distorted motion sequences. We employ a learnable deep learning model  $\bar{\mathbf{x}}_0 = S_{\theta}(\mathbf{x}_t,t,\mathbf{c})$  to approximate the original sequence. Here,  $\mathbf{c}$  signifies the provided condition. In the standard text-driven motion generation,  $\mathbf{c}$  is the singular text prompt given. However, for the fine-grained spatio-temporal motion generation task,  $\mathbf{c}$  embodies the provided detailed linguistic constraints. The training objective for this network is to minimize the mean square error between the true original sequence  $\mathbf{x}_0$  and the estimated one  $\bar{\mathbf{x}} 0$ , which can be formulated as  $\mathbb{E}_{x_0,\epsilon ,t}[\mathbf{x}_0 - S_\theta (\mathbf{x}_t,t,\mathbf{c})]$ .
Figure 3: Architecutre of FineMoGen. In Figure a), our employed Motion Transformer first encodes the timestamp  $t$ , the noise-infused motion sequence  $\mathbf{x}_t$ , and descriptions  $\mathrm{Text}_{i,j}$  into feature vectors, which undergo further processing by  $N$  layers. Central to these layers lies our proposed SAMI, which takes into account both spatial independence and temporal independence. In Figure (b)'s left segment, the feature fusion among different body parts is demonstrated. This fusion is driven by a sequence of learnable parameters  $\omega_{i,j}$  and the enhanced feature  $\mathbf{B}_{k,j}$  derived from MoE projection. The outcome of this spatial modeling is denoted as the feature vector  $\mathbf{Y}_{k,i}^S$ . Figure b)'s right segment illustrates the process of temporal modeling. Initially, text features and motion features are concatenated and fed into MoE layers, yielding informative features  $\mathbf{K}_{i,j,k}$  and  $\mathbf{V}_{i,j,k}$ . These two feature vectors are then merged into global templates  $\mathbf{G}_{i,j}$ . For each time moment, the refined feature is obtained from these global templates through temporal independence modeling, as depicted in Figure c). Here, we generate  $N_g$  time-varying signals. After normalization by the time differences between  $x_k$  and time anchors  $\mathbf{G}_{i,j}^t$ , we ultimately acquire the refined feature  $\mathbf{Y}_{k,i}^T$ . Upon summation with  $\mathbf{Y}_{k,i}^S$ , the final output of SAMI, denoted as  $\mathbf{Y}_{k,i}$ , is achieved.
Motion Transformer Network are frequently chosen as the foundation for estimators in motion diffusion models. These networks consist of several identical blocks that sequentially process the input data as shown in Figure 3. For the provided linguistic prompts, a pre-trained CLIP model [14] is utilized to first extract informative and generalized features. These features are further refined by a series of trainable transformer layers to better align with the requirements of the task at hand. In line with ReMoDiffuse, each block in FineMoGen integrates an efficient mixed attention module and a feed-forward network (FFN). This paper delves into the operational principles of the mixed attention mechanism and subsequently presents a novel attention mechanism specifically designed for spatio-temporal motion composition.
3.3 Spatio-Temporal Mixture Attention
In the attention module, we aim to accurately apply fine-grained textual information to the corresponding body parts and time intervals. First we introduce the baseline attention mechanism we used and then illustrate how we modify it to achieve both spatial independence modelling and temporal independence modelling.
Baseline Mixed Attention. Here, we follow the design of attention in ReMoDiffuse [21] by incorporating efficient self-attention and efficient cross-attention [16] within the same module, forming a unique Mixed Efficient Attention module (MEA). Suppose the input motion feature sequence and text feature sequence are  $\mathbf{X}_m\in \mathbb{R}^{N_m\times L_m}$ ,  $\mathbf{X}_t\in \mathbb{R}^{N_t\times L_t}$  respectively.  $N_{m}$  and  $N_{t}$  are the lengths of motion and text sequences.  $L_{m}$  and  $L_{t}$  are the dimensionality of them. The core idea of MEA is the usage of global templates instead of pair-wise attention. Formally, all elements in  $\mathbf{X}_{\mathbf{m}}$  and  $\mathbf{X}_{\mathbf{t}}$  first produce a matrix  $\mathbf{V}\in \mathbb{R}^{(N_m + N_t)\times H\times L_g}$  by linear projections induced by the weight  $W_{m}^{V}\in \mathbb{R}^{L_{m}\times (H\cdot L_{g})}$  or  $W_{t}^{V}\in \mathbb{R}^{L_{t}\times (H\cdot L_{g})}$ , where  $H$  is the number of heads used in multi-head attention mechanism and  $L_{g}$  is the dimensionality of global templates. Similarly, we get a matrix  $\mathbf{K}\in \mathbb{R}^{H\times N_g\times (N_m + N_t)}$  and perform softmax operation on the last dimension, where  $N_{g}$  is the number of global templates in each head. These two matrix are multiplied together to get all global templates  $\mathbf{G}\in \mathbb{R}^{H\times N_g\times L_g}$ . This process can be written as:
$$
\mathbf {V} = [ W _ {m} ^ {V} \mathbf {X _ {m}}; W _ {t} ^ {V} \mathbf {X _ {t}} ], \mathbf {K} = [ W _ {m} ^ {K} \mathbf {X _ {m}}; W _ {t} ^ {K} \mathbf {X _ {t}} ], \mathbf {G} = \mathrm {S o f t m a x} (\mathbf {K}) \mathbf {V}, \qquad (1)
$$
where  $[\cdot ;\cdot ]$  indicates a concatenation of two tensors. Matrix  $\mathbf{K}$  can be regarded as a series of coefficients that  $\mathbf{K}_{i,j,k}$  shows the influence magnitude of  $k$ th element on  $\mathbf{G}_{i,j}$ , the  $j$ th global template in  $i$ th group. The features in  $\mathbf{V}$  are summarized to  $H\times N_{g}$  groups with the normalized weights from  $\mathbf{K}$ , results in the global templates  $\mathbf{G}$ .
Similar to standard self-attention, we also produce a matrix  $\mathbf{Q} \in \mathbb{R}^{N_m \times H \times N_g}$  to query on the global templates.  $\mathbf{Q}_{k,i,j}$  illustrates the significance of  $\mathbf{G}_{i,j}$  on the  $k$ th element. Here we also perform the Softmax operation on the last dimension. and then perform multiplication between  $\mathbf{Q}$  and  $\mathbf{G}$  to get the refined feature, as the output of this attention module. The process is formulated as:
$$
\mathbf {Q} = W _ {m} ^ {Q} \mathbf {X} _ {\mathbf {m}}, \mathbf {Y} = \operatorname {S o f t m a x} (\mathbf {Q}) \mathbf {G}, \tag {2}
$$
where  $Y\in \mathbb{R}^{N_m\times H\times L_g}$  is the output matrix.
Temporal Independence. An intuitive solution is to generate independent refined features for different time intervals and apply them to the corresponding motion segments. However, it lacks a direct mechanism to effectively integrate information across different time intervals and it becomes challenging to generate satisfactory results when we want to generate a varying number of time intervals during testing. The core issue lies in the difficulty of describing the interactions between different time intervals in a scalable manner. To tackle with this issue, we explicitly introduce the concept of time into this procedure. Formally, we propose the following approximation for temporal feature refinement:
$$
\mathbf {Y} _ {k, i} ^ {T} \approx \mu_ {i} \left(x _ {k}\right) = \sum_ {j = 1} ^ {N _ {g}} \mathbf {G} _ {i, j} ^ {\prime} \left(x _ {k}\right) \cdot \mathbf {G} _ {i, j} ^ {*} \left(x _ {k}\right) \tag {3}
$$
where  $x_{k}$  represents the time position of  $k$ th pose state in the motion sequence.  $\mathbf{G}_{i,j}^{\prime}(x)$  indicates the time-varied signal we derive from the feature vector  $\mathbf{G}_{i,j}$  and  $\mathbf{G}_{i,j}^{*}(x_{k})$  denotes the relative significance of this template for the  $k$ th position. Consider the properties of motion generation task, we further construct  $\mathbf{G}_{i,j}^{\prime}(x)$  and  $\mathbf{G}_{i,j}^{*}(x_{k})$  as:
$$
\mathbf {G} _ {i, j} ^ {\prime} (x) = \mathbf {G} _ {i, j} ^ {s} + \mathbf {G} _ {i, j} ^ {v} \cdot (x - \mathbf {G} _ {i, j} ^ {t}), \mathbf {G} _ {i, j} ^ {*} (x _ {k}) = \frac {e ^ {- (x _ {k} - \mathbf {G} _ {i , j} ^ {t}) ^ {2} / \sigma^ {2}}}{\sum_ {l \in [ 1 , N _ {g} ]} e ^ {- (x _ {k} - \mathbf {G} _ {i , l} ^ {t}) ^ {2} / \sigma^ {2}}}, \tag {4}
$$
In this setup, the  $j$ th global template of the  $i$ th group is considered as a set of signals propagating outward from the temporal center  $\mathbf{G}_{i,j}^{t}$ . We perceive a global template  $\mathbf{G}_{i,j}$  as an anchor with its initial state defined as  $\mathbf{G}_{i,j}^{s}$  and velocity represented by  $\mathbf{G}_{i,j}^{v}$ . These three matrices are obtained from the original  $\mathbf{G}_{i,j}$  through three separate linear projections.
Furthermore, to assimilate influences from all signals, we use the square of time difference as a measure to evaluate the importance of each global template and use a Softmax operation to normalize their weights. A direct advantage of this modeling approach is the ease in appending a new stage following the current one, by adding a bias term to the  $\mathbf{G}_{i,j}^{t}$  accordingly. Therefore, this method proves beneficial in both fully-supervised and zero-shot scenarios.