Title: InsActor: Instruction-driven Physics-based Characters
Venue: NeurIPS 2023
Accepted: yes
Keywords: Physics-based Animation; Human Motion Generation
Abstract: Generating animation of physics-based characters with intuitive control has long been a desirable task with numerous applications. However, generating physically simulated animations that reflect high-level human instructions remains a difficult problem due to the complexity of physical environments and the richness of human language. 
In this paper, we present $\textbf{InsActor}$, a principled generative framework that leverages recent advancements in diffusion-based human motion models to produce instruction-driven animations of physics-based characters.
Our framework empowers InsActor to capture complex relationships between high-level human instructions and character motions by employing diffusion policies for flexibly conditioned motion planning.
To overcome invalid states and infeasible state transitions in planned motions, InsActor discovers low-level skills and maps plans to latent skill sequences in a compact latent space. 
Extensive experiments demonstrate that InsActor achieves state-of-the-art results on various tasks, including instruction-driven motion generation and instruction-driven waypoint heading. Notably, the ability of InsActor to generate physically simulated animations using high-level human instructions makes it a valuable tool, particularly in executing long-horizon tasks with a rich set of instructions. Our project page is available at [jiawei-ren.github.io/projects/insactor/index.html](https://jiawei-ren.github.io/projects/insactor/index.html)

InsActor: Instruction-driven Physics-based Characters

Abstract

Generating animation of physics-based characters with intuitive control has long been a desirable task with numerous applications. However, generating physically simulated animations that reflect high-level human instructions remains a difficult problem due to the complexity of physical environments and the richness of human language. In this paper, we present InsActor , a principled generative framework that leverages recent advancements in diffusion-based human motion models to produce instruction-driven animations of physics-based characters. Our framework empowers InsActor to capture complex relationships between high-level human instructions and character motions by employing diffusion policies for flexibly conditioned motion planning. To overcome invalid states and infeasible state transitions in planned motions, InsActor discovers low-level skills and maps plans to latent skill sequences in a compact latent space. Extensive experiments demonstrate that InsActor achieves state-of-the-art results on various tasks, including instruction-driven motion generation and instruction-driven waypoint heading. Notably, the ability of InsActor to generate physically simulated animations using high-level human instructions makes it a valuable tool, particularly in executing long-horizon tasks with a rich set of instructions. Our project page is available at jiawei-ren.github.io/projects/insactor/index.html

1 Introduction

Generating life-like natural motions in a simulated environment has been the focus of physics-based character animation [24, 16]. To enable user interaction with the generated motion, various conditions such as waypoints have been introduced to control the generation process [40, 15]. In particular, human instructions, which have been widely adopted in text generation and image generation, have recently drawn attention in physics-simulated character animation [15]. The accessibility and versatility of human instructions open up new possibilities for downstream physics-based character applications.

Therefore, we investigate a novel task in this work: generating physically-simulated character animation from human instruction. The task is challenging for existing approaches. While motion tracking [4] is a common approach for character animation, it presents challenges when tracking novel motions generated from free-form human language. Recent advancements in language-conditioned controllers [15] have demonstrated the feasibility of managing characters using instructions, but they struggle with complex human commands. On the other hand, approaches utilizing conditional generative models to directly generate character actions [14] fall short of ensuring the accuracy necessary for continuous control.

To tackle this challenging task, we present InsActor , a framework that employs a hierarchical design for creating instruction-driven, physics-based characters. At the high level, InsActor generates motion

'it then walks like

a zombie to

🚩

'

'kick!'

'now it walks

to

🚩

'

'a person deep squats

and jumps to

🚩

'

plans conditioned on human instructions. This approach enables the seamless integration of human commands, resulting in more coherent and intuitive animations. To accomplish this, InsActor utilizes a diffusion policy [23, 5] to generate actions in the joint space conditioned on human inputs. It allows flexible test-time conditioning, which can be leveraged to complete novel tasks like waypoint heading without task-specific training. However, the high-level diffusion policy alone does not guarantee valid states or feasible state transitions, making it insufficient for direct execution of the plans using inverse dynamics [1]. Therefore, at the low level, InsActor incorporates unsupervised skill discovery to handle state transitions between pairs of states, employing an encoder-decoder architecture. Given the state sequence in joint space from the high-level diffusion policy, the low-level policy first encodes it into a compact latent space to address any infeasible joint actions from the high-level diffusion policy. Each state transition pair is mapped to a skill embedding within this latent space. Subsequently, the decoder translates the embedding into the corresponding action. This hierarchical architecture effectively breaks down the complex task into two manageable tasks at different levels, offering enhanced flexibility, scalability, and adaptability compared to existing solutions.

Given that InsActor generates animations that inherently ensure physical plausibility, the primary evaluation criteria focus on two aspects: fidelity to human instructions and visual plausibility of the animations. Through comprehensive experiments assessing the quality of the generated animations, InsActor demonstrates its ability to produce visually captivating animations that faithfully adhere to instructions, while maintaining physical plausibility. Furthermore, thanks to the flexibility of the diffusion model, animations can be further customized by incorporating additional conditions, such as waypoints, as illustrated in Figure 1, showcasing the broad applicability of InsActor. In addition, InsActor also serves as an important baseline for language conditioned physics-based animation generation.

2 Related Works

2.1 Human Motion Generation

Human motion generation aims to produce versatile and realistic human movements [12, 21, 22, 39]. Recent advancements have enabled more flexible control over motion generation [2, 10, 38, 9]. Among various methods, the diffusion model has emerged as a highly effective approach for generating language-conditioned human motion [36, 44]. However, ensuring physical plausibility, such as avoiding foot sliding, remains challenging due to the absence of physical priors and interaction with the environment [42, 29, 32]. Some recent efforts have attempted to address this issue by incorporating physical priors into the generation models [41], such as foot contact loss [36]. Despite this progress, these approaches still struggle to adapt to environmental changes and enable interaction with the environment. To tackle these limitations, we propose a general framework for generating long-horizon human animations that allow characters to interact with their environment and remain

Diffuser

𝑠

!"#

$

𝑠

!"#

$%#

𝑠

!"&

$

𝑠

!"&

$%#

𝑠

'

𝑠

'

$%#

…

…

Instruction

🗣

'A person deep

squats and jumps.'

Denoised Plan

Waypoint

🚩

History

Condition

Guide

𝑠

#

𝑠

!

Skills

𝑠

!

𝑠

!"#

(

…

𝑝

)

𝑎 𝑠, 𝑧

Decoder

𝑠

!"#

𝑠

!"&

(

𝒵

𝑇	×

…

…

robust to environmental changes. Our approach strives to bridge the gap between understanding complex high-level human instructions and generating physically-simulated character motions.

2.2 Language-Conditioned Control

Language-Conditioned Control aims to guide an agent's behavior using natural language, which has been extensively applied in physics-based animation and robot manipulation [35, 20, 19] to ensure compliance with physical constraints. However, traditional approaches often necessitate dedicated language modules to extract structured expressions from free-form human languages or rely on handcrafted rules for control [33, 43, 34]. Although recent attempts have trained data-driven controllers to generate actions directly from human instructions [18, 15], executing long-horizon tasks remains challenging due to the need to simultaneously understand environmental dynamics, comprehend high-level instructions, and generate highly accurate control. The diffusion model, considered one of the most expressive models, has been introduced to generate agent actions [14].

Nonetheless, current methods are unable to accurately control a humanoid character, as evidenced by our experiments. Recent work utilizing diffusion model to generate high-level pedestrian trajectories and ground the trajectories with a low level controller [30]. Compared with existing works, InsActor employs conditional motion generation to capture intricate relationships between high-level human instructions and character motions beyond pedestrian trajectories, and subsequently deploys a lowlevel skill discovery incorporating physical priors. This approach results in animations that are both visually striking and physically realistic.

3 The Task of Instruction-driven Physics-based Character Animation

We formulate the task as conditional imitation learning, in which we learn a goal-conditioned policy that outputs an action a ∈ A based on the current state s ∈ S and an additional condition c ∈ C describing the desired character behavior. The environment dynamics are represented by the function T : S × A → S .

The task state comprises the character's pose and velocity, including position p , rotation q , linear velocity ˙ p , and angular velocity ˙ q for all links in local coordinates. Consequently, the task state, s , contains this information: s := { p , q , ˙ p , ˙ q } . Following common practice, we employ PD controllers to drive the character. Given the current joint angle q , angular velocity ˙ q , and target angle ˜ q , the torque on the joint actuator is computed as k p ( ˜ q -q ) + k d ( ˜ ˙ q -˙ q ) , where ˜ ˙ q = 0 , k p and k d are manually specified PD controller gains. We maintain k p and k d identical to the PD controller used in DeepMimic [25]. The action involves generating target angles for all joints to control the character.

Our work addresses a realistic setting in which demonstration data consists solely of states without actions, a common scenario in motion datasets collected from real humans where obtaining actions is challenging [26, 3, 13]. To train our model, we use a dataset of trajectory-condition pairs D = { ( τ i , c i ) } N i =1 , where τ = { s ∗ 1 , ..., s ∗ L } denotes a state-only demonstration generated by the expert of

length L . We use s ∗ to denote states from the expert demonstration. For example, a human instruction can be 'walk like a zombie' and the trajectory would be a state sequence describing the character's motion. Our objective is to learn a policy in conjunction with environment dynamics T , which can replicate the expert's trajectory for given instruction c ∈ C .

4 The Framework of InsActor

The proposed method, InsActor, employs a unified hierarchical approach for policy learning, as depicted in Figure 2. Initially, a diffusion policy interprets high-level human instructions, generating a sequence of actions in the joint space. In our particular case, the action in the joint space can be regarded as the state of the animated character. Subsequently, each pair of actions in the joint space are mapped into the corresponding skill embedding in the latent space, ensuring their plausibility while producing desired actions for character control in accordance with motion priors. Consequently, InsActor effectively learns intricate policies for animation generation that satisfy user specifications in a physically-simulated environment. The inference process of InsActor is detailed in Algorithm 1.

4.1 High-Level Diffusion Policy

For the high-level state diffusion policy, we treat the joint state of the character as its action. We follow the state-of-the-art approach of utilizing diffusion models to carry out conditional motion generation [44, 36]. We denote the human instruction as c and the state-only trajectory as τ .

Trajectory Curation. In order to use large-scale datasets for motion generation in a physical simulator, it is necessary to retarget the motion database to a simulated character to obtain a collection of reference trajectories. Large-scale text-motion databases HumanML3D [8] and KIT-ML [26] use SMPL [17] sequences to represent motions. SMPL describes both the body shape and the body poses, where the body poses include pelvis location and rotation, the relative joint rotation of the 21 body joints. We build a simulated character to have the same skeleton as SMPL. We scale the simulated character to have a similar body size to a mean SMPL neutral shape. For retargeting, we directly copy the joint rotation angle, pelvis rotation, and translation to the simulated character. A vertical offset is applied to compensate for different floor heights.

Diffusion Models. Diffusion models [11] are probabilistic techniques used to remove Gaussian noise from data and generate a clean output. These models consist of two processes: the diffusion process and the reverse process. The diffusion process gradually adds Gaussian noise to the original data for a specified number of steps, denoted by T , until the distribution of the noise closely approximates a standard Gaussian distribution, denoted by N ( 0 , I ) . This generates a sequence of noisy trajectories denoted by τ 1: T = { τ 1 , ..., τ T } . The original data is sampled from a conditional distribution, τ 0 ∼ p ( τ 0 | c ) , where c is the instruction. Assuming that the variance schedules are determined by β t , the diffusion process is defined as:

where q ( τ t | τ t -1 ) is the conditional distribution of each step in the Markov chain. The parameter β t controls the amount of noise added at each step t , with larger values resulting in more noise added. The reverse process in diffusion models is another Markov chain that predicts and removes the added noise using a learned denoising function. In particular, we encode language c into an encoded latent vector, ˆ c , using the classical transformer [37] as the language encoder, ˆ c = E ( c ) . Thus, the reverse process starts with a distribution p ( τ T ) := N ( τ T ; 0 , I ) and is defined as:

Here, p ( τ t -1 | τ t , ˆ c ) is the conditional distribution at each step in the reverse process. The mean and covariance of the Gaussian are represented by µ and Σ , respectively. During training, steps t are uniformly sampled for each ground truth motion τ 0 , and a sample is generated from q ( τ t | τ 0 ) . Instead of predicting the noise term ϵ [11], the model predicts the original data τ 0 directly which has

Algorithm 1 Inference of InsActor

Input: A instruction c , the diffusion model f θ , the skill encoder, q ϕ and decoder p ψ . Diffusion steps T . A language encoder E . A history o = { ˆ s , ..., ˆ s } . A waypoint h . Animation length L .

1: 1 l 2: Output: A physically simulated trajectory, ˆ τ .

▷ Generate state sequence.

3: w ← Initialize a plan from a Gaussian noise, w ∼ N ( 0 , I )

4: w ← Apply the inpainting strategy in [14] for history o and waypoint h to guide diffusion.

5: ˆ c ← Encode the instruction E ( c )

6: τ = { s 1 , .... s L } ← Generate trajectory with the diffusion model f θ ( w,T, ˆ c ▷ Generate action sequence in a closed-loop manner.

7: for i = l, l +1 , ..., ( L -1) do

8: z i ← Sample z from q ϕ ( z i | ˆ s i , s i +1 )

9: a i ← Sample action form p ψ ( a i | ˆ s i , z i )

10: ˆ s i +1 ← Get next state T (ˆ s i +1 | ˆ s i , a i )

11: end for

12: Output ˆ τ = { ˆ s l +1 , .... ˆ s L }

the equivalent formulation [28, 36]. This is done by using a neural network f θ parameterized by θ to predict τ 0 from the noisy trajectory τ t and the condition ˆ c at each step t in the denoise process. The model parameters are optimized by minimizing the mean squared error between the predicted and ground truth data using the loss function:

where p ( τ 0 | c ) is the conditional distribution of the ground truth data, and ∥ · ∥ denotes the mean squared error. By directly predicting τ 0 , this formulation avoids repeatedly adding noise to τ 0 and is more computationally efficient.

Guided Diffusion. Diffusion models allow flexible test-time conditioning through guided sampling, for example, classifier-guided sampling [6]. Given an objective function as a condition, gradients can be computed to optimize the objective function and perturb the diffusion process. In particular, a simple yet effective inpainting strategy can be applied to introduce state conditions [14], which is useful to generate a plan that adheres to past histories or future goals. Concretely, the inpainting strategy formulates the state conditioning as a Dirac delta objective function. Optimizing the objective function is equivalent to directly presetting noisy conditioning states and inpainting the rest. We leverage the inpainting strategy to achieve waypoint heading and autoregressive generation.

Limitation. Despite the ability to model complex language-to-motion relations, motion diffusion models can generate inaccurate low-level details, which lead to physically implausible motions and artifacts like foot floating and foot penetration [36]. In the context of state diffusion, the diffusergenerated states can be invalid and the state transitions can be infeasible. Thus, direct tracking of the diffusion plan can be challenging.

4.2 Low-Level Skill Discovery

To tackle the aforementioned challenge, we employ low-level skill discovery to safeguard against unexpected states in poorly planned trajectories. Specifically, we train a Conditional Variational Autoencoder to map state transitions to a compact latent space in an unsupervised manner [40]. This approach benefits from a repertoire of learned skill embedding within a compact latent space, enabling superior interpolation and extrapolation. Consequently, the motions derived from the diffusion model can be executed by natural motion primitives.

Skill Discovery. Assuming the current state of the character is ˆ s l , the first step in constructing a compact latent space for skill discovery is encoding the state transition in a given reference motion sequence, τ 0 = { s ∗ 1 , ..., s ∗ L } , into a latent variable, z , and we call this skill embedding. This variable represents a unique skill required to transition from ˆ s l to s ∗ l +1 . The neural network used to encode the skill embedding is referred to as the encoder, q ϕ , parameterized by ϕ , which produces a Gaussian distribution:

)

where µ ϕ is the mean and Σ ϕ is the isotropic covariance matrix. Once we obtain the latent variable, a decoder, p ψ ( a l | ˆ s l , z l ) , parameterized by ψ , generates the corresponding actions, a , by conditioning on the latent variable z l and the current state ˆ s l :

Subsequently, using the generated action, the character transitions into the new state, ˆ s l +1 , via the transition function T (ˆ s l +1 | ˆ s l , a l ) . By repeating this process, we can gather a generated trajectory, denoted as ˆ τ = { ˆ s 1 , ..., ˆ s L } . The goal is to mimic the given trajectory τ 0 by performing the actions. Thus, to train the encoder and decoder, the main supervision signal is derived from the difference between the resulting trajectory ˆ τ and the reference motion, τ 0 .

Training. Our approach leverages differentiable physics to train the neural network end-to-end without the need for a separate world model [31]. This is achieved by implementing the physical laws of motion as differentiable functions, allowing the gradient to flow through them during backpropagation. Concretely, by executing action a l ∼ p ψ ( a l | ˆ s l , z l ) at state ˆ s l , the induced state ˆ s l +1 is differentiable with respect to the policy parameter ψ and ϕ . Thus, directly minimizing the difference between the predicted state ˆ s and the s ∗ gives an efficient and effective way of training an imitation learning policy [31]. The Brax [7] simulator is used due to its efficiency and easy parallelization, allowing for efficient skill discovery. It also ensures that the learned skill is trained on the actual physical environment, rather than a simplified model of it, leading to a more accurate and robust representation.

Thus, the encoder-decoder is trained with an objective that minimizes the discrepancy between resulting trajectories and Kullback-Leibler divergence between the encoded latent variable and the prior distribution, which is a standard Gaussian,

where ∥ · ∥ denotes the mean squared error and ( s , s ′ ) is a pair of states before and after transition. The latter term encourages the latent variables to be similar to the prior distribution, ensuring the compactness of the latent space. λ is the weight factor that controls the compactness. During inference, we map the generated state sequence from the diffusion model to the skill space to control the character.

5 Experiments

The goal of our experiment is to evaluate the effectiveness and robustness of InsActor in generating physically-simulated and visually-natural character animations based on high-level human instructions. Specifically, we aim to investigate 1 ) whether InsActor can generate animations that adhere to human instructions while being robust to physical perturbations, 2 ) whether InsActor can accomplish waypoint heading while being faithful to the language descriptions, and 3 ) the impact of several design choices, including the hierarchical design and the weight factor for skill space compactness.

5.1 Implementation Details

To implement the experiment, we use Brax [7] to build the environment and design a simulated character based on DeepMimic [25]. The character has 13 links and 34 degrees of freedom, weighs 45kg, and is 1.62m tall. Contact is applied to all links with the floor. For details of neural network architecture and training, we refer readers to the supplementary materials.

5.2 Evaluation Protocols

Datasets. We use two large scale text-motion datasets, KIT-ML [26] and HumanML3D [8], for training and evaluation. KIT-ML has 3,911 motion sequences and 6,353 sequence-level language descriptions, HumanML3D provides 44,970 annotations on 14,616 motion sequences. We adopt the original train/test splits in the two datasets.

Metrics. We employ the following evaluation metrics:

R Precision : For every pair of generated sequence and instruction, we randomly pick 31 additional instructions from the test set. Using a trained contrastive model, we then compute the average top-k accuracy.

Frechet Inception Distance (FID): We use a pre-trained motion encoder to extract features from both the generated animations and ground truth motion sequences. The FID is then calculated between these two distributions to assess their similarity.

Multimodal Distance: With the help of a pre-trained contrastive model, we compute the disparity between the text feature derived from the given instruction and the motion feature from the produced animation. We refer to this as the multimodal distance.

Diversity: To gauge diversity, we randomly divide the generated animations for all test texts into pairs. The average joint differences within each pair are then computed as the metric for diversity.

Success Rate: For waypoint heading tasks, we compute the Euclidean distance between the final horizontal position of the character pelvis and the target horizontal position. If the distance is less than 0.5 m, we deem it a success. We perform each evaluation three times and report the statistical interval with 95% confidence.

5.3 Comparative Studies for Instruction-driven Character Animation

Comparison Methods. We compare InsActor with two baseline approaches: 1) DReCon [4]. We adapted the responsive controller framework from DReCon [4]. We use the diffuser as a kinematic controller and train a target-state tracking policy. The baseline can also be viewed as a Decision Diffuser [1] with a long planning horizon, where a diffuser plans the future states and a tracking policy solves the inverse dynamics. 2) PADL [15]: We adapt the language-conditioned control policy in PADL [15], where language instructions are encoded by a pretrained cross-modal text encoder [27] and input to a control policy that directly predict actions. It is also a commonly used learning paradigm in conditional imitation learning [34]. Since the two baselines have no publicly available implementations, we reproduce them and train the policies with DiffMimic [31].

Settings. We utilize two different settings to assess InsActor's robustness. In the first setting, we evaluate the models in a clean, structured environment devoid of any perturbation. In the second setting, we introduce perturbations by spawning a 2kg box to hit the character every 1 second, thereby evaluating whether the humanoid character can still adhere to human instructions even when the environment changes.

Results. We present the results in Table 1 and Table 2 and qualitative results in Figure 3. Compared to the dataset used in PADL that consists of 131 motion sequences and 256 language captions, our benchmark dataset is two orders larger, where the language-conditioned single-step policy used in PADL has difficulty to scaling up . In particular, the inferior performance in language-motion matching metrics suggests that a single-step policy fails to understand unseen instructions and model the many-to-many instruction-motion relation. Compared to PADL, DReCon shows a better result in language-motion matching thanks to the high-level motion planning. However, unlike Motion Matching used in DReCon that produces high-quality kinematic motions, the diffuser generates invalid states and infeasible state transitions, which fails DReCon's tracking policy and results in a low FID . In comparison, InsActor significantly outperforms the two baselines on all metrics. Moreover, the experiment reveals that environmental perturbations do not significantly impair the performance of InsActor, showcasing InsActor's robustness.

5.4 Instruction-driven Waypoint Heading

Waypoint Heading. Thanks to the flexibility of the diffusion model, InsActor can readily accomplish the waypoint heading task, a common task in physics-based character animation [40, 15]. This task necessitates the simulated character to move toward a target location while complying with human instructions. For instance, a human instruction might be, 'walk like a zombie.' In this case, the character should navigate toward the target position while mimicking the movements of a zombie.

Guided Diffusion. We accomplish this using guided diffusion. Concretely, we adopt the inpainting strategy in Diffuser [14]. Prior to denoising, we replace the Gaussian noise in the first and last 25% frames with the noisy states of the character standing at the starting position and target position respectively.

Results. We conduct this experiment with the model trained on HumanML3D. We contrast InsActor with DReCon and two InsActor variants: 1) InsActor without language condition, and 2) InsActor without targeting. Our experimental results demonstrate that InsActor can effectively accomplish the waypoint heading task by leveraging guided diffusion. The model trained on HumanML3D is capable of moving toward the target position while following the given human instructions, as evidenced by a low FID score and high precision. Although adding the targeting position condition to the diffusion process slightly compromises the quality of the generated animation, the outcome is still satisfactory. Moreover, the success rate of reaching the target position is high, underscoring the effectiveness of guided diffusion. Comparing InsActor with its two variants highlights the importance of both the language condition and the targeting in accomplishing the task. Comparing InsActor with DReCon shows the importance of skill mapping, particularly when more infeasible state transitions are introduced by the waypoint guiding. Without skill mapping, DReCon only has a 38.0% success rate, which drops drastically from the 90.7% success rate of InsActor.

Multiple Waypoints. Multiple waypoints allow users to interactively instruct the character. We achieve this by autoregressively conditioning the diffusion process to the history motion, where a qualitative result is shown in Figure 4. Concretely, we inpaint the first 25% with the latest history state sequences. We show qualitative results for multiple-waypoint following in Figure 1 and more in the supplementary materials.

5.5 Ablation Studies

Hierarchical Design. To understand the importance of the hierarchical design in this task, we performed an ablation study on its structure. We compared our approach to two baselines: 1 ) A policy with only a high-level policy, wherein the diffusion model directly outputs the skills, analogous to the Diffuser approach [14]; 2 ) A low-level policy that directly predicts single-step skills. We show the results in Table 4. By leveraging skills, the low-level policy improves from PADL but still grapples with comprehending the instructions due to the absence of language understanding. Conversely, without the low-level policy, the skills generated directly by the diffusion model are of poor precision. Although the use of skills safeguards the motions to be natural and score high in FID, the error accumulation deviates the plan from the language description and results in a low R-precision. The experimental results underscore the efficacy of the hierarchical design of InsActor.

Weight Factor. InsActor learns a compact latent space for skill discovery to overcome infeasible plans generated by the diffusion model. We conduct an ablation study on the weight factor, λ , which controls the compactness of the skill space. Our findings suggest that a higher weight factor results in a more compact latent space, however, it also curtails the instruction-motion alignment. Conversely, a lower weight factor permits a greater diversity in motion generation, but it might also lead to less plausible and inconsistent motions. Hence, it is vital to find a balance between these two factors to optimize performance for the specific task at hand.

6 Conclusion

In conclusion, we have introduced InsActor, a principled framework for physics-based character animation generation from human instructions. By utilizing a diffusion model to interpret language instructions into motion plans and mapping them to latent skill vectors, InsActor can generate flexible physics-based animations with various and mixed conditions including waypoints. We hope InsActor would serve as an important baseline for future development of instruction-driven physics-based animation. While InsActor is capable of generating such animations, there are crucial but exciting challenges ahead. One limitation is the computational complexity of the diffusion model, which may pose challenges for scaling up the approach to more complex environments and larger datasets. Additionally, the current version of InsActor assumes access to expert demonstrations for training, which may limit its applicability in real-world scenarios where such data may not be readily available. Furthermore, while InsActor is capable of generating physically-reliable and visually-plausible animations, there is still room for improvement in terms of the quality and diversity of generated animations. There are mainly two aspects for future development, improving the quality of the differentiable physics for more realistic simulations and enhancing the expressiveness and diversity of the diffusion model to generate more complex and creative animations. Besides them, extending InsActor to accommodate different human body shapes and morphologies is also an interesting direction.

From a societal perspective, the application of InsActor may lead to ethical concerns related to how it might be used. For instance, InsActor could be exploited to create deceptive or harmful content. This underscores the importance of using InsActor responsibly.

Acknowledgment

This research is supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: AISG2-PhD-2021-08-018), NTU NAP, MOE AcRF Tier 2 (T2EP20221-0012), and under the RIE2020 Industry Alignment Fund - Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s).

References

Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision-making? arXiv preprint arXiv:2211.15657 , 2022.

Sadegh Aliakbarian, Fatemeh Sadat Saleh, Mathieu Salzmann, Lars Petersson, and Stephen Gould. A stochastic conditioning scheme for diverse human motion prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5223-5232, 2020.

10

4

10

3

10

2

10

1

10

0

0.3

0.4

0.5

0.6

R Precision (Top 3)

R Precision (Top 3)

FID

0.5

1.0

1.5

2.0

FID

Mykhaylo Andriluka, Umar Iqbal, Eldar Insafutdinov, Leonid Pishchulin, Anton Milan, Juergen Gall, and Bernt Schiele. Posetrack: A benchmark for human pose estimation and tracking. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5167-5176, 2018.

Kevin Bergamin, Simon Clavet, Daniel Holden, and James Richard Forbes. Drecon: Data-driven responsive control of physics-based characters. ACM Trans. Graph. , 38(6), November 2019.

Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems (RSS) , 2023.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems , 34, 2021.

C Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax-a differentiable physics engine for large scale rigid body simulation. arXiv preprint arXiv:2106.13281 , 2021.

Chuan Guo, Xinxin Zuo, Sen Wang, Shihao Zou, Qingyao Sun, Annan Deng, Minglun Gong, and Li Cheng. Action2motion: Conditioned generation of 3d human motions. In Proceedings of the 28th ACM International Conference on Multimedia , pages 2021-2029, 2020.

Félix G Harvey, Mike Yurick, Derek Nowrouzezahrai, and Christopher Pal. Robust motion in-betweening. ACM Transactions on Graphics (TOG) , 39(4):60-1, 2020.

Gustav Eje Henter, Simon Alexanderson, and Jonas Beskow. Moglow: Probabilistic and controllable motion synthesis using normalising flows. ACM Transactions on Graphics (TOG) , 39(6):1-14, 2020.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020.

Leslie Ikemoto, Okan Arikan, and David Forsyth. Generalizing motion edits with gaussian processes. ACM Transactions on Graphics (TOG) , 28(1):1-12, 2009.

Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3.6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE transactions on pattern analysis and machine intelligence , 36(7):1325-1339, 2013.

Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning , 2022.

Jordan Juravsky, Yunrong Guo, Sanja Fidler, and Xue Bin Peng. Padl: Language-directed physics-based character control. Association for Computing Machinery, 2022.

Libin Liu, Michiel van de Panne, and KangKang Yin. Guided learning of control graphs for physics-based characters. ACM Transactions on Graphics , 35(3), 2016.

Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi-person linear model. ACM transactions on graphics (TOG) , 34(6):1-16, 2015.

Corey Lynch and Pierre Sermanet. Language conditioned imitation learning over unstructured data. Robotics: Science and Systems , 2021.

Oier Mees and Wolfram Burgard. Composing pick-and-place tasks by grounding language. In International Symposium on Experimental Robotics , 2021.

Oier Mees, Alp Emek, Johan Vertens, and Wolfram Burgard. Learning object placements for relational instructions by hallucinating scene representations. In 2020 IEEE International Conference on Robotics and Automation (ICRA) , pages 94-100, 2020.

Jianyuan Min and Jinxiang Chai. Motion graphs++ a compact generative model for semantic motion analysis and synthesis. ACM Transactions on Graphics (TOG) , 31(6):1-12, 2012.

Dirk Ormoneit, Michael J Black, Trevor Hastie, and Hedvig Kjellström. Representing cyclic human motion using functional analysis. Image and Vision Computing , 23(14):1264-1276, 2005.

Tim Pearce, Tabish Rashid, Anssi Kanervisto, Dave Bignell, Mingfei Sun, Raluca Georgescu, Sergio Valcarcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann, and Sam Devlin. Imitating human behaviour with diffusion models. In The Eleventh International Conference on Learning Representations , 2023.

Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne. Deepmimic: Exampleguided deep reinforcement learning of physics-based character skills. ACM Trans. Graph. , 37(4):143:1-143:14, July 2018.

Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel Van de Panne. Deepmimic: Exampleguided deep reinforcement learning of physics-based character skills. ACM Transactions On Graphics (TOG) , 2018.

Matthias Plappert, Christian Mandery, and Tamim Asfour. The kit motion-language dataset. Big data , 4(4):236-252, 2016.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. CoRR , abs/2103.00020, 2021.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 2022.

Davis Rempe, Leonidas J. Guibas, Aaron Hertzmann, Bryan Russell, Ruben Villegas, and Jimei Yang. Contact and human dynamics from monocular video. In Proceedings of the European Conference on Computer Vision (ECCV) , 2020.

Davis Rempe, Zhengyi Luo, Xue Bin Peng, Ye Yuan, Kris Kitani, Karsten Kreis, Sanja Fidler, and Or Litany. Trace and pace: Controllable pedestrian animation via guided trajectory diffusion. In Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Jiawei Ren, Cunjun Yu, Siwei Chen, Xiao Ma, Liang Pan, and Ziwei Liu. Diffmimic: Efficient motion mimicking with differentiable physics. ICLR , 2022.

Soshi Shimada, Vladislav Golyanik, Weipeng Xu, and Christian Theobalt. Physcap: Physically plausible monocular 3d motion capture in real time. ACM Trans. Graph. , 39(6), nov 2020.

Mohit Shridhar and David Hsu. Interactive visual grounding of referring expressions for human-robot interaction. In Proceedings of Robotics: Science and Systems , 2018.

Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In Proceedings of the 5th Conference on Robot Learning (CoRL) , 2021.

Simon Stepputtis, Joseph Campbell, Mariano Phielipp, Stefan Lee, Chitta Baral, and Heni Ben Amor. Language-conditioned imitation learning for robot manipulation tasks. In Proceedings of the 34th International Conference on Neural Information Processing Systems , 2020.

Guy Tevet, Sigal Raab, Brian Gordon, Yoni Shafir, Daniel Cohen-or, and Amit Haim Bermano. Human motion diffusion model. In The Eleventh International Conference on Learning Representations , 2023.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Zhenyi Wang, Ping Yu, Yang Zhao, Ruiyi Zhang, Yufan Zhou, Junsong Yuan, and Changyou Chen. Learning diverse stochastic human-action generators by learning smooth latent transitions. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 12281-12288, 2020.

Xinchen Yan, Akash Rastogi, Ruben Villegas, Kalyan Sunkavalli, Eli Shechtman, Sunil Hadap, Ersin Yumer, and Honglak Lee. Mt-vae: Learning motion transformations to generate multimodal human dynamics. In Proceedings of the European conference on computer vision (ECCV) , pages 265-281, 2018.

Heyuan Yao, Zhenhua Song, Baoquan Chen, and Libin Liu. Controlvae: Model-based learning of generative controllers for physics-based characters. 41(6), 2022.

Ye Yuan, Jiaming Song, Umar Iqbal, Arash Vahdat, and Jan Kautz. Physdiff: Physics-guided human motion diffusion model. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2023.

Ye Yuan, Shih-En Wei, Tomas Simon, Kris Kitani, and Jason Saragih. Simpoe: Simulated character control for 3d human pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2021.

Hanbo Zhang, Yunfan Lu, Cunjun Yu, David Hsu, Xuguang Lan, and Nanning Zheng. Invigorate: Interactive visual grounding and grasping in clutter. Proceedings of Robotics: Science and Systems , abs/2108.11092, 2021.

Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondiffuse: Text-driven human motion generation with diffusion model. arXiv preprint arXiv:2208.15001 , 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes InsActor, a framework for instruction-driven character animation. Given human instruction and/or way points, InsActor first leverages a diffusion model to generate state sequences of the character. Next, it uses a skill embedding model to convert the state sequence into physically plausible trajectories with state and actions.\n\nSTRENGTHS:\n1. The paper presents the motivations, setups, and methods quite well. The diffusion model for state sequence generation and conversion from state sequence to physical trajectories are described in sufficient details.\n\n2. The paper provides sufficient experiments and comparisons to show it out-performs the baseline method. The hierarchical design is also justified with ablation studies.\n\nWEAKNESSES:\n1. The works appears to have limited novelty, as it is somewhat a straightforward combination of character motion synthesis with diffusion methods such as [13] and a low-level trajectory optimization framework.\n\n2. This work seems to miss a comparison with a straightforward method for the low-level policy, since the second stage can be formulated with a standard trajectory optimization problem. Since Brax is also a differentiable simulator, trajectory optimization should be easy to set up and in principle, it can be used to obtain an optimal solution in terms of distances between the states generated from the first stage and states from a physically plausible trajectory. I think this work could include results and analysis from trajectory optimization or explain why it is not feasible here.\n\nQUESTIONS:\nSuggestion on writing:\n- line 92: I believe angular velocity $q$ of a link is not the derivative of rotation, and the symbol $q$ clashes with the notation of joint position $q$. This description needs to be clarified to avoid confusion.\n\nLIMITATIONS:\nThe paper has provided adequate discussion of limitation.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of generating text-conditioned character animation that is physics-based. It proposes a two-stage approach that first generates a kinematic motion conditioned on text using diffusion, and then tracks this motion with a physics-based motion VAE in the learned latent space. Experiments on the common KIT-ML and HumanML3D benchmarks show improved performance over prior work in physics-based motion from text, and show the ability to specify goal waypoints for the motion to hit while following human text prompts.\n\nSTRENGTHS:\nPhysics-driven text-to-motion is an important problem and conditioning on free-form text input has not really been tackled in the literature, so the paper is novel in that respect. Diffusion has shown promising results recently, but these are all kinematic.\n\nThe proposed idea of using high-level diffusion followed by physics-based tracking is simple and solid. It would make a good first baseline for future work in this area. \n\nTechnically, InsActor uses a physics-based motion VAE to track kinematic motion which is novel, and it uses differentiable physics to train rather than a learned world model or RL. If this approach really does work to track general motions in the HumanML3D dataset, it would contribute an alternative to recent RL approaches that can be difficult to generalize. \n\nThe proposed method is evaluated on HumanML3D and KIT, which are the most relevant benchmarks for the text-to-motion task. Also, the DReCon baseline in Tables 1 and 2 is an important baseline that uses the alternative state-based approach to tracking.\n\nThe supplementary video and figures are visually pleasing, and I appreciate that the supp video is extensive and shows many results. The shown demo is also cool, and demonstrates fast generation capabilities (relative to other diffusion approaches).\n\nWEAKNESSES:\nTo better motivate the need for physics in text-to-motion, there should be a comparison between the proposed InsActor and a state-of-the-art kinematic diffusion model like MDM [33] added to Tables 1 and 2. Currently some numbers in these tables, e.g. FID and diversity, are worse than those reported in MDM, and it’s not clear why that’s the case since the high-level planner in InsActor is very similar. Does this high-level planner perform worse than MDM? Or does the physics-based motion tracking somehow have a large effect on motion quality and diversity? The high-level policy ablation reported in Table 4 takes a step in this direction, but it is trained on rollouts from the motion VAE instead of directly on mocap data as done in MDM and other text-to-motion diffusion models.\n\nLooking at video results at 0:46 and 01:54, I don’t think the high-level diffusion planner is on par with recent models like MDM. Both with and without waypoint guidance there are some significant artifacts like jittering and skating, some of which seem to be affecting the final motion from InsActor (e.g. some noisy popping of limbs and unnatural sliding). I understand this planner is not necessarily the main contribution, but I think poor kinematic motions from the planner undermines the comparison to the target-state tracking policy DReCon, which may perform better when operating on, e.g., outputs from MDM that better reflect realistic motion. Since the low-level motion VAE model for tracking (Sec 4.2) is a key contribution of the work, it’s very important to justify that it is necessary by showing that the DReCon baseline is still inferior when operating on more reasonable kinematic inputs.\n\nThe methods Sec 4 is missing some details that could improve understanding and reproducibility: \n* The tasks states described in L92 are all local, so how is the global root trajectory modeled in the diffusion Sec 4.1? \n* L154: if the pose state is in the local frame, how is inpainting performed to ensure motion meets a global target waypoint?\n* What is the architecture of the diffusion model? Is it using 1D convolutions as in Diffuser or a transformer as in other human motion diffusion models? $\\mu$ and $\\Sigma$ in Eqn 2 are never defined. In general, I’m wondering why not use a SOTA motion diffusion model out-of-the-box for this high-level planning component?\n* L185: is the low-level motion VAE trained directly on outputs of the diffusion model or on mocap data from the dataset? If on mocap, why is the encoder (Eqn 4) expected to produce reasonable results when operating on noisy and unrealistic pose transitions? \n* Similarly, Sec 5.3 shows robustness to perturbations from boxes, but is this kind of perturbation seen in training of the low-level policy too? If not, how does this robustness arise without using RL for training (i.e. without some exploration). \n\nAn evaluation on the low-level tracking component by itself would be very helpful. E.g. reporting tracking errors for the latent policy from InsActor compared to DReCon for both motion-captured and diffusion-generated motions. The current metrics in Tables 1 and 2 were designed for kinematic text-to-motion models, and I would think are mostly influenced by the diffusion planner which is the same for InsActor and DReCon, so a tracking-only evaluation could help parse the difference in performance. There are also open-source RL physics-based trackers that may be worth considering, e.g. [Luo et al., Dynamics-Regulated Kinematic Policy for Egocentric Pose Estimation, NeurIPS 2021].\n\nQUESTIONS:\nOverall, I think a general physics-based text-to-motion model is an important and novel direction, and the hierarchical approach of diffusion planning with physics-based tracker could be a strong baseline going forward. But I’m mainly concerned that the quality of the output from the high-level diffusion planner is compromising the comparison to DReCon and therefore the need for a latent motion VAE tracker has not been fully justified. I would really like to see how InsActor performs when plugging in a SOTA diffusion model like MDM [33] as the planner. Moreover, an evaluation of standalone tracking performance would make the comparison between the two tracking approaches (latent vs state-based) much more clear. \n\nSome other comments and suggestions that didn’t have an influence on my rating: \n* The related work (Sec 2) is missing relevant physics-based human animation methods and a discussion of why they are difficult to scale up to the general text-to-motion task. E.g. [Peng et al., ASE: Large-Scale Reusable Adversarial Skill Embeddings for Physically Simulated Characters, SIGGRAPH 2022] [Won et al., A Scalable Approach to Control Diverse Behaviors for Physically Simulated Characters, SIGGRAPH 2020], etc..\n* Concurrent work PhysDiff [Yuan et al., arxiv 2022] is an alternative approach to adding physicality to text-to-motion diffusion and could be discussed in future revisions of the paper. Also related, concurrent work Trace and Pace [Rempe et al., CVPR 2023] gives controllability over physics based characters with guidance of a diffusion planner.\n\n===================== After Rebuttal ============================\n\nAfter considering other reviews and discussions with authors and between reviewers, I have decided to slightly raise my score and am leaning towards accept. I think the paper lays out a compelling kinematic diffusion + physics-based tracking idea that can serve as a baseline and inspire improvement in each component of the system including differentiable simulation, motion diffusion, and physics-based tracking. The evaluations show that this hierarchical approach works better than state+action diffusion, and that tracking with a latent model is more robust than direct state-based tracking for planned motions from MotionDiffuse.\n\nHowever, I am still quite concerned about the qualitative results and would really encourage the authors to update the paper text to discuss these qualitative issues such that future work can pursue important directions (e.g., the choice of Brax and differentiable simulation in general rather than RL, and the noisy plans from MotionDiffuse especially in the waypoint setting). It would also be good to clarify why in Table 2 of the rebuttal doc, motion quality (FID) drops significantly from kinematic “Planner” (b) output to full physics-based InsActor (c), indicating that adding physics-based tracking is not necessarily improving motion realism despite it being physically constrained (unlike in PhysDiff). I encourage the authors to show some video results of the planner output vs InsActor on regular text-to-motion (not the waypoint setting) to demonstrate the difference and potential advantages/disadvantages of using the latent tracking technique vs RL and more reliable simulators as in DReCon. \n\n==========================================================\n\nLIMITATIONS:\nLimitations are sufficiently discussed in Sec 6.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes InsActor, a physics-based character control framework that enables controlling agents to follow high-level instructions. It employs a high-level diffusion motion planner and a low-level character controller to achieve mapping between text-based instructions and physics-based motion control. By creating waypoints and desired state transitions, the diffusion motion planner specifies the motion plan that the low-level motion controller follows. The low-level motion controller consists of a pre-trained VAE encoder to translate diffusion state into actionable latents for control.\n\nSTRENGTHS:\n- I find the idea of using a higher-level motion diffusion model to specify waypoints and states, and then using a lower-level motion controller to follow them, intuitive and easy to understand. This formulation is flexible and creates the ability to have both high-level control of motion through text and slightly lower-level control through waypoints.\n- The low-level skill discovery module based on diffMimic itself seems like a significant contribution, as it has the potential to be used in many tasks and enables character control. Using a VAE to translate invalid state transitions into actionable latent codes for stable control has great potential in other downstream tasks.\n- Experiments show that the methods outperform the in-house implementation of the state-of-the-art methods PADL and DReCon.\n\nWEAKNESSES:\n- The main weakness I find in this work is its qualitative results. The simulated character is jittery, floaty, and appears to have foot sliding, which is unexpected for a physics-based method. For instance, at the 1:43 mark of the video, InsActor's feet quickly shuffle in a way that should be impossible in a physics simulator. The character seems overall drunk when walking, and there are occasional high-frequency jitters in the root (or camera?), for instance, at 00:48. I am not sure what could have caused this issue: is it the policy? or the setting/fidelity of the Brax simulator?\n- Similar to the previous point, the visual results of the implemented PADL and DReCon are far inferior to their original shown results. The movement is unnatural and jittery. I understand that there is no official implementation provided, but similar scenarios could be recreated in InsActor to create visual comparisons. PADL's generated motion is stable and physically realistic, unlike the ones shown in the provided video. As motion generation is largely visual, the provided quantitative results do not really provide that much insight into the real performance of the method.\n- Visualization should be provided for the high level diffusion planer. Since states are directly generated by the planner, they should be visualized and compared with existing models such as motion diffusion model (MDM). The lower level controller, combined with the latent skill decoder $p_\\psi$, forms an imitator that follows specified states. How well does the diffusion model generates the state and how “invalid” are they?\n- The claim of \"long, unstructured human commands\" is overstated. The tested text instructions are still short, clear, and close to the ones in the dataset.\n- There are many missing details about the performance of the lower-level skill discovery module. If the whole HumanML3D dataset is used for training, how well can the skill embedding encapsulate the motion described in the dataset?\n\nQUESTIONS:\nMy main concern is the fidelity of the generated motion. Why does the character appear to be floaty and jittery? Is it the learned policy or the simulator settings?\n\n--- \nAfter rebuttal, my main concern about the fidelity of the generated motion returned. However, after discussion with other reviewers and the authors, I believe that the this issue could be a problem of the underlying simulator and should not undermine the overall contribution of the framework. Thus, I raise my score to borderline accept. \n\n---\n\nLIMITATIONS:\nLimitations are addressed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an approach for generating physically plausible human motion from open text prompts. The approach combines high-level trajectory generation with guided diffusion and a low-level skill model encoded in a latent space to correct for physical plausibility during execution in simulation with PD control. The proposed approach is compared against existing motion generation approaches with a set of quantitative metrics and qualitative evaluation.\n\nSTRENGTHS:\nOverall this work soundly presents an improvement to SoTA in the space of human motion generation from text prompts. The use of guided diffusion to produce high-level trajectories is not surprising, nor is its effectiveness when combined with low-level skill models as a method of correcting physical implausibilities and errors. The paper is well written and clearly describes the architecture, training procedure, and use of guidance for waypoint following. Evaluation is sound and clearly demonstrates the strengths of the approach. I appreciate the use of multiple quantitative metrics and stochastic evaluation of many samples. This work benefits from the existence and availability of its component parts (Brax, ControlVAE, motion diffusion models), but connects them to achieve fairly good results. I’m always happy to see motion generation connected to physics simulation instead of remaining kinematic. The results of this work seem fairly robust to perturbation, which is a good signal for the usefulness of future iterations of the approach.\n\nWEAKNESSES:\nWhile the resulting motion is fairly good, it isn’t yet achieving the level of fidelity to make it generally useful in generative contexts (e.g. game or animation characters).  \nThis work achieves its success from connecting other existing techniques, though the overall system is a sound contribution.\nThe stack is trained considering only the character’s joint space and a specific character’s physical constraints.\n\nQUESTIONS:\nShould cite Trace and Pace: Controllable Pedestrian Animation via Guided Trajectory Diffusion for similar architecture (guided-diffusion trajectories + low-level physical controller) https://arxiv.org/abs/2304.01893\nTable 3 ordering of metrics is different from Metrics text section. Swap Multimodal Distance and FID in one or the other.\nIs it feasible to generalize this stack to different body shapes, dynamics, to include objects or other kinds of waypoints?\nWould it be possible/better to have some model-based or explicit low-level skill experts for specific sub-tasks (e.g. locomotion) to achieve higher fidelity in addition to general/open motion skills?\n\nLIMITATIONS:\nDiscussion of limitations was pretty good and I appreciated the limitation notes throughout the paper. I did not see any mention of generalizing to other character morphologies (even different human body shapes). I currently believe this work is using a single embodiment as a strong assumption for achieving accuracy and physical plausibility. I would be interested to hearing more about the feasibility of generalization for applications.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an approach for generating animations using a language-conditioned neural network in particular a diffusion model. The underlying goal is to generate physics based human character animation in an easy and intuitive fashion. The approach leverages a diffusion model to generate high-level states of the animated character which are then refined in a lower level module. This module is realized via an auto encoder which generates skill embedding representing the state transition in a compact fashion. A decoder is then used to synthesized the final action of the agent. The approach also uses an inpainting strategy to ensure that the agent goes through critical phases or points, i.e., this seems similar to the difference of keyframes and in-betweens in traditional animation. The paper compares the introduced approach to other recent methods on the topic, e.g., PADL and concludes among other things that more faithful animations (wrt to the original language query) are generated. The contributions of the paper is the combination of Diffusion and autoencoder models for motion generation, the inpainting strategy and waypoint system.\n\nSTRENGTHS:\nThe paper addresses a really interesting and important problem that is very exciting. It is also one of the first papers to use diffusion models in this context and the results seem promising. The combination of in-painting with diffusion is also very clever (but details are missing). A BIG plus compared to other similar systems is that it does not require RL. For example, PADL requires multiple policies to be learned and later combined with each policy requiring 7 simulated years to be trained! That does not seem to be the case here since this is a purely supervised approach - even though substantial computation is required for all deep learning of course.\nI really appreciated the use of contrastive models to evaluate the faithfulness of generated models. Diversity was also an interesting measure since in animation it is critical that the character shows a range of motions. Generally the paper is written in a way that can be followed (with caveats see below). \n\n** Update after rebuttal **\nI have increased my score to \"weak accept\". I appreciate the details provided by the authors - this helped better contextualize and understand the paper. What is clear from the rebuttal process is that it is hard to judge the visual fidelity alone, despite the substantial effort the authors put into the evaluation section. Especially the choice of visualizing the resulting animations via retargeting to a robot (in Blender) may have negatively impacted the reviewers' opinion. The animations in the latter part of the video are actually more convincing in my opinion. However, as pointed out before, the animations are not well-adapted to the environment, i.e., foot skating, actions on objects, etc. In future submissions on the topic, I strongly advise addressing this topic even if only partially.\n\nWEAKNESSES:\nA major question that the paper is not addressing is the generation of actions on objects and surroundings. A critical element of any character animation is to be able to specify an object acted upon, e.g., \"push the red object on the table\". That is not addressed at all in this paper, while it is addressed on other papers in robotics (the manipulation papers) but also animation (PADL). In PADL, a module specifically identifies the target object and animations are synthesized accordingly. In the robotics papers [Lynch2020, Stepputtis2020] the actions of the agents are also conditioned on an image of the surroundings and target objects are visually identified in the image. Potentially that could be included into InsActor through the waypoint system (maybe?), but it is so far not addressed in the paper. Personally, this is the biggest flaw of the paper, since at the moment it would not be appealing to me or many others to use this system.\nAlong these lines, the presented example results are not very convincing with regards to the complexity of the generated motions. \n\nAnother weakness of the paper is that it seems important details are left out or not well justified. For example, the waypoint encoding is interesting but should be explained with an example. I am particularly unsure what this would entail for the user - do we have to place the character at those waypoints and set all of its joint values? This would require the user to actually become an animator and generate keyframes. That could be a serious limitation if true. Right now I am assuming we only need to position the basic character at the waypoint (i.e. modify position and orientation of agent) not moving limbs (i.e. modify joint angles). The paper states \"...replace the Gaussian noise in the first and last 25% frames with the noisy states of the character standing at the starting position and target position respectively.\"\n\nPersonally, I would really have liked to see a long sequence in which the agent performs a number of different actions in a row. At Siggraph they often have these cute teaser figures in which a humanoid walks, crouches, jumps etc. over a period of time.\n\nQUESTIONS:\n- What exactly is the transition function? I am assuming that is the differential simulator, right?\n\nLIMITATIONS:\nAs described above, the lack interaction with the environment and linguistic goals is a major limitation that has not been brought up in the paper. The system is not able to execute commands like \"lift the green cup\" or \"move to the lion statue\". That substantially limits the applications of the approach. It is also surprising that the authors mention problems like foot skating at the beginning but do not describe how that is resolved later in the paper. While I understand that in a physics-based system the dynamics of the environment can make some of these considerations obsolete, there is still room for broken animations if this not adequately addressed. Hence, I would ask the authors to address this point in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-28"}