SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jiawei Ren\*1 Mingyuan Zhang\*1 Cunjun Yu\*2 Xiao Ma3 Liang Pan1 Ziwei Liu
$^{1}$  S-Lab, Nanyang Technological University
3 Dyson Robot Learning Lab
Generating animation of physics-based characters with intuitive control has long been a desirable task with numerous applications. However, generating physically simulated animations that reflect high-level human instructions remains a difficult problem due to the complexity of physical environments and the richness of human language. In this paper, we present InsActor, a principled generative framework that leverages recent advancements in diffusion-based human motion models to produce instruction-driven animations of physics-based characters. Our framework empowers InsActor to capture complex relationships between high-level human instructions and character motions by employing diffusion policies for flexibly conditioned motion planning. To overcome invalid states and infeasible state transitions in planned motions, InsActor discovers low-level skills and maps plans to latent skill sequences in a compact latent space. Extensive experiments demonstrate that InsActor achieves state-of-the-art results on various tasks, including instruction-driven motion generation and instruction-driven waypoint heading. Notably, the ability of InsActor to generate physically simulated animations using high-level human instructions makes it a valuable tool, particularly in executing long-horizon tasks with a rich set of instructions. Our project page is available at jiawei-ren.github.io/projects/insactor/index.html
Generating life-like natural motions in a simulated environment has been the focus of physics-based character animation [24, 16]. To enable user interaction with the generated motion, various conditions such as waypoints have been introduced to control the generation process [40, 15]. In particular, human instructions, which have been widely adopted in text generation and image generation, have recently drawn attention in physics-simulated character animation [15]. The accessibility and versatility of human instructions open up new possibilities for downstream physics-based character applications.
Therefore, we investigate a novel task in this work: generating physically-simulated character animation from human instruction. The task is challenging for existing approaches. While motion tracking [4] is a common approach for character animation, it presents challenges when tracking novel motions generated from free-form human language. Recent advancements in language-conditioned controllers [15] have demonstrated the feasibility of managing characters using instructions, but they struggle with complex human commands. On the other hand, approaches utilizing conditional generative models to directly generate character actions [14] fall short of ensuring the accuracy necessary for continuous control.
To tackle this challenging task, we present InsActor, a framework that employs a hierarchical design for creating instruction-driven, physics-based characters. At the high level, InsActor generates motion
Figure 1: InsActor enables controlling physics-based characters with human instructions and intended target position. The figure illustrates this by depicting several "flags" on a 2D plane, each representing a relative target position such as (0,2) starting from the origin.
plans conditioned on human instructions. This approach enables the seamless integration of human commands, resulting in more coherent and intuitive animations. To accomplish this, InsActor utilizes a diffusion policy [23, 5] to generate actions in the joint space conditioned on human inputs. It allows flexible test-time conditioning, which can be leveraged to complete novel tasks like waypoint heading without task-specific training. However, the high-level diffusion policy alone does not guarantee valid states or feasible state transitions, making it insufficient for direct execution of the plans using inverse dynamics [1]. Therefore, at the low level, InsActor incorporates unsupervised skill discovery to handle state transitions between pairs of states, employing an encoder-decoder architecture. Given the state sequence in joint space from the high-level diffusion policy, the low-level policy first encodes it into a compact latent space to address any infeasible joint actions from the high-level diffusion policy. Each state transition pair is mapped to a skill embedding within this latent space. Subsequently, the decoder translates the embedding into the corresponding action. This hierarchical architecture effectively breaks down the complex task into two manageable tasks at different levels, offering enhanced flexibility, scalability, and adaptability compared to existing solutions.
Given that InsActor generates animations that inherently ensure physical plausibility, the primary evaluation criteria focus on two aspects: fidelity to human instructions and visual plausibility of the animations. Through comprehensive experiments assessing the quality of the generated animations, InsActor demonstrates its ability to produce visually captivating animations that faithfully adhere to instructions, while maintaining physical plausibility. Furthermore, thanks to the flexibility of the diffusion model, animations can be further customized by incorporating additional conditions, such as waypoints, as illustrated in Figure 1, showcasing the broad applicability of InsActor. In addition, InsActor also serves as an important baseline for language conditioned physics-based animation generation.
2.1 Human Motion Generation
Human motion generation aims to produce versatile and realistic human movements [12, 21, 22, 39]. Recent advancements have enabled more flexible control over motion generation [2, 10, 38, 9]. Among various methods, the diffusion model has emerged as a highly effective approach for generating language-conditioned human motion [36, 44]. However, ensuring physical plausibility, such as avoiding foot sliding, remains challenging due to the absence of physical priors and interaction with the environment [42, 29, 32]. Some recent efforts have attempted to address this issue by incorporating physical priors into the generation models [41], such as foot contact loss [36]. Despite this progress, these approaches still struggle to adapt to environmental changes and enable interaction with the environment. To tackle these limitations, we propose a general framework for generating long-horizon human animations that allow characters to interact with their environment and remain
Figure 2: The overall framework of InsActor. At the high level, the diffusion model generates state sequences from human instructions and waypoint conditions. At the low level, each state transition is encoded into a skill embedding in the latent space and decoded to an action.
robust to environmental changes. Our approach strives to bridge the gap between understanding complex high-level human instructions and generating physically-simulated character motions.
2.2 Language-Conditioned Control
Language-Conditioned Control aims to guide an agent's behavior using natural language, which has been extensively applied in physics-based animation and robot manipulation [35, 20, 19] to ensure compliance with physical constraints. However, traditional approaches often necessitate dedicated language modules to extract structured expressions from free-form human languages or rely on handcrafted rules for control [33, 43, 34]. Although recent attempts have trained data-driven controllers to generate actions directly from human instructions [18, 15], executing long-horizon tasks remains challenging due to the need to simultaneously understand environmental dynamics, comprehend high-level instructions, and generate highly accurate control. The diffusion model, considered one of the most expressive models, has been introduced to generate agent actions [14].
Nonetheless, current methods are unable to accurately control a humanoid character, as evidenced by our experiments. Recent work utilizing diffusion model to generate high-level pedestrian trajectories and ground the trajectories with a low level controller [30]. Compared with existing works, InsActor employs conditional motion generation to capture intricate relationships between high-level human instructions and character motions beyond pedestrian trajectories, and subsequently deploys a low-level skill discovery incorporating physical priors. This approach results in animations that are both visually striking and physically realistic.
3 The Task of Instruction-driven Physics-based Character Animation
We formulate the task as conditional imitation learning, in which we learn a goal-conditioned policy that outputs an action  $\mathbf{a} \in A$  based on the current state  $\mathbf{s} \in S$  and an additional condition  $\mathbf{c} \in C$  describing the desired character behavior. The environment dynamics are represented by the function  $\mathcal{T}: S \times A \to S$ .
The task state comprises the character's pose and velocity, including position  $\mathbf{p}$ , rotation  $\mathbf{q}$ , linear velocity  $\dot{\mathbf{p}}$ , and angular velocity  $\dot{\mathbf{q}}$  for all links in local coordinates. Consequently, the task state,  $\mathbf{s}$ , contains this information:  $\mathbf{s} := \{\mathbf{p}, \mathbf{q}, \dot{\mathbf{p}}, \dot{\mathbf{q}}\}$ . Following common practice, we employ PD controllers to drive the character. Given the current joint angle  $\mathbf{q}$ , angular velocity  $\dot{\mathbf{q}}$ , and target angle  $\tilde{\mathbf{q}}$ , the torque on the joint actuator is computed as  $k_{p}(\tilde{\mathbf{q}} - \mathbf{q}) + k_{d}(\tilde{\dot{\mathbf{q}}} - \dot{\mathbf{q}})$ , where  $\tilde{\dot{\mathbf{q}}} = 0$ ,  $k_{p}$  and  $k_{d}$  are manually specified PD controller gains. We maintain  $k_{p}$  and  $k_{d}$  identical to the PD controller used in DeepMimic [25]. The action involves generating target angles for all joints to control the character.
Our work addresses a realistic setting in which demonstration data consists solely of states without actions, a common scenario in motion datasets collected from real humans where obtaining actions is challenging [26, 3, 13]. To train our model, we use a dataset of trajectory-condition pairs  $\mathcal{D} = \{(\pmb{\tau}^i,\pmb{c}^i)\}_{i=1}^N$ , where  $\pmb{\tau} = \{\mathbf{s}_1^*,\dots,\mathbf{s}_L^*\}$  denotes a state-only demonstration generated by the expert of
length  $L$ . We use  $\mathbf{s}^*$  to denote states from the expert demonstration. For example, a human instruction can be "walk like a zombie" and the trajectory would be a state sequence describing the character's motion. Our objective is to learn a policy in conjunction with environment dynamics  $\mathcal{T}$ , which can replicate the expert's trajectory for given instruction  $\mathbf{c} \in C$ .
4 The Framework of InsActor
The proposed method, InsActor, employs a unified hierarchical approach for policy learning, as depicted in Figure 2. Initially, a diffusion policy interprets high-level human instructions, generating a sequence of actions in the joint space. In our particular case, the action in the joint space can be regarded as the state of the animated character. Subsequently, each pair of actions in the joint space are mapped into the corresponding skill embedding in the latent space, ensuring their plausibility while producing desired actions for character control in accordance with motion priors. Consequently, InsActor effectively learns intricate policies for animation generation that satisfy user specifications in a physically-simulated environment. The inference process of InsActor is detailed in Algorithm 1.
4.1 High-Level Diffusion Policy
For the high-level state diffusion policy, we treat the joint state of the character as its action. We follow the state-of-the-art approach of utilizing diffusion models to carry out conditional motion generation [44, 36]. We denote the human instruction as  $c$  and the state-only trajectory as  $\tau$ .
Trajectory Curation. In order to use large-scale datasets for motion generation in a physical simulator, it is necessary to retarget the motion database to a simulated character to obtain a collection of reference trajectories. Large-scale text-motion databases HumanML3D [8] and KIT-ML [26] use SMPL [17] sequences to represent motions. SMPL describes both the body shape and the body poses, where the body poses include pelvis location and rotation, the relative joint rotation of the 21 body joints. We build a simulated character to have the same skeleton as SMPL. We scale the simulated character to have a similar body size to a mean SMPL neutral shape. For retargeting, we directly copy the joint rotation angle, pelvis rotation, and translation to the simulated character. A vertical offset is applied to compensate for different floor heights.
Diffusion Models. Diffusion models [11] are probabilistic techniques used to remove Gaussian noise from data and generate a clean output. These models consist of two processes: the diffusion process and the reverse process. The diffusion process gradually adds Gaussian noise to the original data for a specified number of steps, denoted by  $T$ , until the distribution of the noise closely approximates a standard Gaussian distribution, denoted by  $\mathcal{N}(\mathbf{0},\mathbf{I})$ . This generates a sequence of noisy trajectories denoted by  $\pmb{\tau}_{1:T} = \{\pmb{\tau}_1,\dots,\pmb{\tau}_T\}$ . The original data is sampled from a conditional distribution,  $\pmb{\tau}_0\sim p(\pmb{\tau}_0\mid \pmb {c})$ , where  $c$  is the instruction. Assuming that the variance schedules are determined by  $\beta_{t}$ , the diffusion process is defined as:
$$
q \left(\boldsymbol {\tau} _ {1: T} \mid \boldsymbol {\tau} _ {0}\right) := \prod_ {t = 1} ^ {T} q \left(\boldsymbol {\tau} _ {t} \mid \boldsymbol {\tau} _ {t - 1}\right), \quad q \left(\boldsymbol {\tau} _ {t} \mid \boldsymbol {\tau} _ {t - 1}\right) := \mathcal {N} \left(\boldsymbol {\tau} _ {t}; \sqrt {1 - \beta_ {t}} \boldsymbol {\tau} _ {t - 1}, \beta_ {t} \mathbf {I}\right), \tag {1}
$$
where  $q(\pmb{\tau}_t|\pmb{\tau}_{t-1})$  is the conditional distribution of each step in the Markov chain. The parameter  $\beta_t$  controls the amount of noise added at each step  $t$ , with larger values resulting in more noise added. The reverse process in diffusion models is another Markov chain that predicts and removes the added noise using a learned denoising function. In particular, we encode language  $c$  into an encoded latent vector,  $\hat{c}$ , using the classical transformer [37] as the language encoder,  $\hat{c} = \mathcal{E}(c)$ . Thus, the reverse process starts with a distribution  $p(\pmb{\tau}_T) \coloneqq \mathcal{N}(\pmb{\tau}_T; \mathbf{0}, \mathbf{I})$  and is defined as:
$$
p \left(\boldsymbol {\tau} _ {0: T} | \hat {\boldsymbol {c}}\right) := p \left(\boldsymbol {\tau} _ {T}\right) \prod_ {t = 1} ^ {T} p \left(\boldsymbol {\tau} _ {t - 1} \mid \boldsymbol {\tau} _ {t}, \hat {\boldsymbol {c}}\right), \quad p \left(\boldsymbol {\tau} _ {t - 1} \mid \boldsymbol {\tau} _ {t}, \hat {\boldsymbol {c}}\right) := \mathcal {N} \left(\boldsymbol {\tau} _ {t - 1}; \mu \left(\boldsymbol {\tau} _ {t}, t, \hat {\boldsymbol {c}}\right), \Sigma \left(\boldsymbol {\tau} _ {t}, t, \hat {\boldsymbol {c}}\right)\right). \tag {2}
$$
Here,  $p(\pmb{\tau}_{t-1}|\pmb{\tau}_t,\hat{\pmb{c}})$  is the conditional distribution at each step in the reverse process. The mean and covariance of the Gaussian are represented by  $\mu$  and  $\Sigma$ , respectively. During training, steps  $t$  are uniformly sampled for each ground truth motion  $\pmb{\tau}_0$ , and a sample is generated from  $q(\pmb{\tau}_t|\pmb{\tau}_0)$ . Instead of predicting the noise term  $\epsilon$  [11], the model predicts the original data  $\pmb{\tau}_0$  directly which has
Algorithm 1 Inference of InsActor
1: Input: A instruction  $c$ , the diffusion model  $f_{\theta}$ , the skill encoder,  $q_{\phi}$  and decoder  $p_{\psi}$ . Diffusion steps  $T$ . A language encoder  $\mathcal{E}$ . A history  $o = \{\hat{\mathbf{s}}_1,\dots,\hat{\mathbf{s}}_l\}$ . A waypoint  $h$ . Animation length  $L$ .
2: Output: A physically simulated trajectory,  $\hat{\tau}$ .  
 $\triangleright$  Generate state sequence.
3:  $w \gets$  Initialize a plan from a Gaussian noise,  $w \sim \mathcal{N}(\mathbf{0},\mathbf{I})$
4:  $w \gets$  Apply the inpainting strategy in [14] for history  $o$  and waypoint  $h$  to guide diffusion.
5:  $\hat{c} \gets$  Encode the instruction  $\mathcal{E}(\pmb{c})$
6:  $\tau = \{\mathbf{s}_1,\dots .\mathbf{s}_L\} \gets$  Generate trajectory with the diffusion model  $f_{\theta}(w,T,\hat{c})$ $\triangleright$  Generate action sequence in a closed-loop manner.
7: for  $i = l, l + 1, \dots, (L - 1)$  do
8:  $z_{i}\gets$  Sample  $\mathbf{z}$  from  $q_{\phi}(\pmb {z}_i|\hat{\mathbf{s}}_i,\mathbf{s}_{i + 1})$
9:  $\pmb{a}_i \gets$  Sample action form  $p_{\psi}(\pmb{a}_i|\hat{\mathbf{s}}_i, \pmb{z}_i)$
0:  $\hat{\mathbf{s}}_{i + 1}\gets$  Get next state  $\mathcal{T}(\hat{\mathbf{s}}_{i + 1}|\hat{\mathbf{s}}_i,\pmb {a}_i)$
11: end for
12: Output  $\hat{\tau} = \{\hat{\mathbf{s}}_{l + 1},\dots,\hat{\mathbf{s}}_L\}$
the equivalent formulation [28, 36]. This is done by using a neural network  $f_{\theta}$  parameterized by  $\theta$  to predict  $\tau_0$  from the noisy trajectory  $\boldsymbol{\tau}_t$  and the condition  $\hat{c}$  at each step  $t$  in the denoise process. The model parameters are optimized by minimizing the mean squared error between the predicted and ground truth data using the loss function:
$$
\mathcal {L} _ {\text {P l a n}} = \mathrm {E} _ {t \in [ 1, T ], \boldsymbol {\tau} _ {0} \sim p (\boldsymbol {\tau} _ {0} | \boldsymbol {c})} [ \| \boldsymbol {\tau} _ {0} - f _ {\theta} (\boldsymbol {\tau} _ {t}, t, \hat {\boldsymbol {c}}) \| ], \tag {3}
$$
where  $p(\tau_0|c)$  is the conditional distribution of the ground truth data, and  $\| \cdot \|$  denotes the mean squared error. By directly predicting  $\tau_0$ , this formulation avoids repeatedly adding noise to  $\tau_0$  and is more computationally efficient.
Guided Diffusion. Diffusion models allow flexible test-time conditioning through guided sampling, for example, classifier-guided sampling [6]. Given an objective function as a condition, gradients can be computed to optimize the objective function and perturb the diffusion process. In particular, a simple yet effective inpainting strategy can be applied to introduce state conditions [14], which is useful to generate a plan that adheres to past histories or future goals. Concretely, the inpainting strategy formulates the state conditioning as a Dirac delta objective function. Optimizing the objective function is equivalent to directly presetting noisy conditioning states and inpainting the rest. We leverage the inpainting strategy to achieve waypoint heading and autoregressive generation.
Limitation. Despite the ability to model complex language-to-motion relations, motion diffusion models can generate inaccurate low-level details, which lead to physically implausible motions and artifacts like foot floating and foot penetration [36]. In the context of state diffusion, the diffuser-generated states can be invalid and the state transitions can be infeasible. Thus, direct tracking of the diffusion plan can be challenging.
4.2 Low-Level Skill Discovery
To tackle the aforementioned challenge, we employ low-level skill discovery to safeguard against unexpected states in poorly planned trajectories. Specifically, we train a Conditional Variational Autoencoder to map state transitions to a compact latent space in an unsupervised manner [40]. This approach benefits from a repertoire of learned skill embedding within a compact latent space, enabling superior interpolation and extrapolation. Consequently, the motions derived from the diffusion model can be executed by natural motion primitives.
Skill Discovery. Assuming the current state of the character is  $\hat{\mathbf{s}}_l$ , the first step in constructing a compact latent space for skill discovery is encoding the state transition in a given reference motion sequence,  $\tau_0 = \{\mathbf{s}_1^*, \dots, \mathbf{s}_L^*\}$ , into a latent variable,  $z$ , and we call this skill embedding. This variable represents a unique skill required to transition from  $\hat{\mathbf{s}}_l$  to  $\mathbf{s}_{l+1}^*$ . The neural network used to encode the skill embedding is referred to as the encoder,  $q_{\phi}$ , parameterized by  $\phi$ , which produces a Gaussian distribution:
$$
q _ {\phi} \left(\boldsymbol {z} _ {l} | \hat {\mathbf {s}} _ {l}, \mathbf {s} _ {l + 1} ^ {*}\right) := \mathcal {N} \left(\boldsymbol {z} _ {l}; \mu_ {\phi} \left(\hat {\mathbf {s}} _ {l}, \mathbf {s} _ {l + 1} ^ {*}\right), \Sigma_ {\phi} \left(\hat {\mathbf {s}} _ {l}, \mathbf {s} _ {l + 1} ^ {*}\right)\right), \tag {4}
$$
"A person picks up."
"A person doing martial arts."
"A person jumps."
"A person spins."
Figure 3: Qualitative results of InsActor with corresponding instructions. Top: only human instruction. Bottom: human instruction and waypoint target.
"A person raises arms and walks."
"A person walks like a zombie."
where  $\mu_{\phi}$  is the mean and  $\Sigma_{\phi}$  is the isotropic covariance matrix. Once we obtain the latent variable, a decoder,  $p_{\psi}(\mathbf{a}_l|\hat{\mathbf{s}}_l,\mathbf{z}_l)$ , parameterized by  $\psi$ , generates the corresponding actions,  $\mathbf{a}$ , by conditioning on the latent variable  $\mathbf{z}_l$  and the current state  $\hat{\mathbf{s}}_l$ :
$$
\boldsymbol {a} _ {l} \sim p _ {\psi} (\boldsymbol {a} _ {l} | \hat {\mathbf {s}} _ {l}, \boldsymbol {z} _ {l}) \tag {5}
$$
Subsequently, using the generated action, the character transitions into the new state,  $\hat{\mathbf{s}}_{l + 1}$ , via the transition function  $\mathcal{T}(\hat{\mathbf{s}}_{l + 1}|\hat{\mathbf{s}}_l,\pmb {a}_l)$ . By repeating this process, we can gather a generated trajectory, denoted as  $\hat{\tau} = \{\hat{\mathbf{s}}_1,\dots,\hat{\mathbf{s}}_L\}$ . The goal is to mimic the given trajectory  $\pmb{\tau}_{0}$  by performing the actions. Thus, to train the encoder and decoder, the main supervision signal is derived from the difference between the resulting trajectory  $\hat{\tau}$  and the reference motion,  $\pmb{\tau}_{0}$ .
**Training.** Our approach leverages differentiable physics to train the neural network end-to-end without the need for a separate world model [31]. This is achieved by implementing the physical laws of motion as differentiable functions, allowing the gradient to flow through them during backpropagation. Concretely, by executing action  $\mathbf{a}_l \sim p_\psi(\mathbf{a}_l | \hat{\mathbf{s}}_l, \mathbf{z}_l)$  at state  $\hat{\mathbf{s}}_l$ , the induced state  $\hat{\mathbf{s}}_{l+1}$  is differentiable with respect to the policy parameter  $\psi$  and  $\phi$ . Thus, directly minimizing the difference between the predicted state  $\hat{\mathbf{s}}$  and the  $\mathbf{s}^*$  gives an efficient and effective way of training an imitation learning policy [31]. The Brax [7] simulator is used due to its efficiency and easy parallelization, allowing for efficient skill discovery. It also ensures that the learned skill is trained on the actual physical environment, rather than a simplified model of it, leading to a more accurate and robust representation.
Thus, the encoder-decoder is trained with an objective that minimizes the discrepancy between resulting trajectories and Kullback-Leibler divergence between the encoded latent variable and the prior distribution, which is a standard Gaussian,
$$
\mathcal {L} _ {\text {S k i l l}} = \left\| \boldsymbol {\tau} _ {0} - \hat {\boldsymbol {\tau}} \right\| + \lambda D _ {\mathrm {K L}} \left(q _ {\phi} \left(\boldsymbol {z} \mid \mathbf {s}, \mathbf {s} ^ {\prime}\right) \| \mathcal {N} (\mathbf {0}, \mathbf {I})\right), \tag {6}
$$