InsActor: Instruction-driven Physics-based Characters
Authors: Jiawei Ren
Mingyuan Zhang
Cunjun Yu
Xiao Ma
Liang Pan
Ziwei Liu
Venue: NeurIPS 2023 poster (2023)
Keywords: Physics-based Animation; Human Motion Generation
TL;DR: We generate motions for physically-based characters from human instructions.

Abstract: Generating animation of physics-based characters with intuitive control has long been a desirable task with numerous applications. However, generating physically simulated animations that reflect high-level human instructions remains a difficult problem due to the complexity of physical environments and the richness of human language. 
In this paper, we present $\textbf{InsActor}$, a principled generative framework that leverages recent advancements in diffusion-based human motion models to produce instruction-driven animations of physics-based characters.
Our framework empowers InsActor to capture complex relationships between high-level human instructions and character motions by employing diffusion policies for flexibly conditioned motion planning.
To overcome invalid states and infeasible state transitions in planned motions, InsActor discovers low-level skills and maps plans to latent skill sequences in a compact latent space. 
Extensive experiments demonstrate that InsActor achieves state-of-the-art results on various tasks, including instruction-driven motion generation and instruction-driven waypoint heading. Notably, the ability of InsActor to generate physically simulated animations using high-level human instructions makes it a valuable tool, particularly in executing long-horizon tasks with a rich set of instructions. Our project page is available at [jiawei-ren.github.io/projects/insactor/index.html](https://jiawei-ren.github.io/projects/insactor/index.html)

InsActor: Instruction-driven Physics-based Characters
Jiawei Ren"!
Mingyuan Zhang"!
Cunjun Yu
Xiao Ma²
Liang Pan'
Ziwei Liu
S-Lab, Nanyang Technological University
National University of Singapore
3 Dyson Robot Learning Lab
Abstract
Generating animation of physics-based characters with intuitive control has long been desirable task with numerous applications. However, generating physically simulated animations that reflect high-level human instructions remains difficult problem due to the complexity of physical environments and the richness of human language. In this paper, we present InsActor. a principled generative framework that leverages recent advancements in diffusion-basee human motion models to produce instruction-drive work empowers InsActor to capture complex relationships between high-level human instructions and character motions by employing diffusion policies for flexibly conditioned motion planning. To overcome invalid states and infeasible state transitions in planned motions, InsActor discovers low-level skills and maps plans to latent skill sequences in compact latent space. Extensive experiments demonstrate that InsActor achieves state-of-the results on various tasks, includ- ing instruction -erivvvin-- motion generation and instruction-diie driven waypoint heading. Notably, the ability of InsActor to generate physically simulated animations using high-level human instructions makes it valuable tool, particularly in executing long-hond long-horizon tasks with a rich set of instructions. Our project page is available at
animations of physics-based characters. Our frame-
1 Introduction
Generating life-like natural motions in a simulated environment has been the focus of thysics-basee character animation [6). To enable interaction with the generated motion, various conditions such as waypoints have been introduced to control the generation process [40] [5]. In particular, human instructions, which have been widely adopted in text generation and image generation, have recently drawn attention in physics-simulates character animation (15). The accessibility and versatility of human instructions open up possibilities for downstream physics-based character applications.
Therefore, we investigate novel task in this work: generating physically-simulace character animation from human instruction. The task is challenging for existing approaches. While motion tracking [4] is common approach for character animation. presents challenges when tracking novel motions generated from free-form human language. Recent advancements in language-conditonn controllers [15] have demonstrated the feasibility of managing characters using instructions, but they struggle with complex human commands. On the other hand, approaches utilizing conditional generative models to directly generate character actions [14] fall short of ensuring the accuracy necessary for continuous control.
To tackle this challenging task, we present InsActor, a framework that employs a hierarchical design for creating instruction-driven, physics-based characters. At the high level, InsActor generates motion
"Equal contribution
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

"a person deep squats and jumps to
"now it walks
"it then walks like a zombie to
"kick!" "
Figure InsActor enables controlling physics- characters with human instructions and intended target position. The figure illustrates this by depicting several "flags" on a 2D plane, each representing a relative target position such as (0,2) starting from the origin.
plans conditioned human instructions. This approach enables the seamless integration of human commands, resulting in more coherent and intuitive animations. To accomplish this, InsActor utilizes diffusion policy (23]5) to generate actions in the joint space conditioned human inputs. It allows flexible test-time conditioning, which can be leveraged to complete novel tasks like waypoint heading without task-specific training. However, the high-level diffusion policy alone does guarantee valid states or feasible state transitions, making insufficient for direct execution of the plans using inverse dynamics []. Therefore, the low level, InsActor incorporates unsupervised skill discovery to handle state transitions between pairs of states, employing an encoder-decoder architecture. Given the state sequence in joint space from the high-level diffusion policy, the low-level policy first encodes into a compact latent space to address any infeasible joint actions from the high-level diffusion policy. Each state transition pair mapped the decoder translates the embedding into the corresponding action. This hierarchical architecture effectively breaks down the complex task into two manageable tasks at different levels, offering enhanced flexibility, scalability, and adaptability compared to existing solutions.
skill embedding within this latent space. Subsequently,
Given that InsActor generates animations that inherently ensure physical plausibility, the primary evaluation criteria focus on two aspects: fidelity to human instructions and visual plausibility of the animations. Through comprehensive experiments assessing the quality of the generated animations, InsActor demonstrates its ability to produce visually captivating animations that faithfully adhere instructions, while maintaining physical plausibility. Furthermore, thanks the flexibility of the diffusion model, animations can be further customized by incorporating additional conditions, such as waypoints, as illustrated in Figure showcasing the broad applicability of InsActor. In addition. InsActor also serves as an important baseline for language conditioned physics-baas based animation generation.
2
Related Works
2.1
Human Motion Generation
Human motion generation aims produce versatile and realistic human movements 22
22
39]. Recent advancements have enabled more flexible control over motion generation 2 10 38
D] Among various methods, the diffusion model has emerged as highly effective approach for generating language-one cotraage-oodiiionedd human motion 36 44. However, ensuring physical plausibility. such as avoiding foot sliding, remains challenging due to the absence of physical priors and interaction
with the environment 42
29 32]. Some recent efforts have attempted to address this issue by
incorporating physical priors into the generation models [41, such as foot contact loss (36). Despite
this progress, these approaches still struggle to adapt to environmental changes and enable interaction with the environment. To tackle these limitations, we propose general framework for generating long-horizon human animations that allow characters to interact with their environment and remain

---

Guide
Waypoint
History
S[+1
1+2
Decoder
"A person deep squats and jumps."
$1
St
si+
Si+2
St.
Pip (als, 2)
St
S1+1
Instruction
Diffuser
Tx
Condition
Denoised Plan
Z
Skills
Figure 2: The overall framework of InsActor. At the high level, the diffusion model generates state sequences from human instructions and waypoint conditions. At the low level, each state transition is encoded into skill embedding in the latent space and decoded to an action.
robust to environmental changes. Our approach strives bridge the gap between understanding complex high-level human instructions and generating physically-simut character motions.
2.2 Language-Condittoned Control
Language-Corn Conditioned Control aims to guide an agent's behavior using natural language, which has been extensively applied in physics-based animation and robot manipulation [35] 20 [9] to ensure compliance with physical constraints. However, traditional approaches often necessitate dedicated language modules to extract structured expressions from free-form human languages or
rely on handcrafted rules for control 33
4
34
Although recent attempts have trained data-driven
controllers to generate actions directly from human instructions [I] 15], executing long-horizon
tasks remains challenging due to the need simultancously understand environmental dynamics, comprehend high-level instructions, and generate highly accurate control. The diffusion model, considered one of the most expressive models, has been introduced to generate agent actions [14].
Nonetheless, current methods are unable to accurately control humanoid character, evidenced by our experiments. Recent work utilizing diffusion model to generate high-level pedestrian trajectories and ground the trajectories with low level controller [30]. Compared with existing works, Ins.Actor employs conditional motion generation to capture intricate relationships between high-level human instructions and character motions beyond pedestrian trajectories, and subsequently deploys low- level skill discovery incorporating physical priors. This approach results in animations that are both visually striking and physically realistic.
3 The Task of Instruction-driven Physics- Character Animation
We formulate the task as conditional imitation learning, in which we learn goal-conditioned policy that outputs action A based the current state S and an additional condition C describing the desired character behavior. The environment dynamics are represented by the function S x A S.
The task state comprises the character's pose and velocity, including position p. rotation q, linear velocity p. and angular velocity & for all links in local coordinates. Consequently, the task state, contains this information: s = (p, q, P, q). Following common practice, we employ PD controllers to drive the character. Given the current joint angle q. angular velocity å. and target angle q. the torque the joint actuator is computed as kp q) + kā q). where q 0, kp and ka manually specified PD controller gains. We maintain kp and ky identical to the PD controller used in DeepMimic (23]. The action involves generating target angles for all joints to control the character.
Our work addresses realistic setting in which demonstration data consists solely states without actions, a common scenario in motion datasets collected from real humans where obtaining actions challenging 26 [3]. To train our model, we use a dataset of trajectory-conditi pairs {(T', ? where T {s].... ..., ...,...,, s denotes state-only demonstration generated by the expert of

---

length L. We use s" to denote states from the expert demonstration. For example, a human instruction can be "walk like zombie" and the trajectory would be state sequence describing the character's motion. Our objective is to learn policy in conjunction with environment dynamics T, which can replicate the expert's trajectory for given instruction с E C.
The Framework of InsActor
The proposed method, InsActor, employs unified hierarchical approach for policy learning, as depicted in Figure 2 Initially, a diffusion policy interprets high-level human instructions, generating sequence of actions in the joint space. In our particular case, the action in the joint space can be regarded as the state of the animated character. Subsequently, each pair of actions in the joint space mapped into the corresponding skill embedding in the latent space, ensuring their plausibility while producing desired actions for character control in accordance with motion priors. Consequently, InsActor effectively learns intricate policies for animation generation that satisfy user specifications in physically-simun simullated environment. The inference process of InsActor is detailed in Algorithm
4.1 High-Level Diffusion Policy
For the high-level state diffusion policy, treat the joint state of the character its action. We follow the state-of-the-art approach of utilizing diffusion models to carry out conditional motion generation [44 36). We denote the human instruction and the state-only trajectory as
Trajectory Curation. order to use large-scale datasets for motion generation a physical simulator, it is necessary retarget the motion database to simulated character to obtain collection of reference trajectories. Large-scale text-motion databases HumanML3D [8] and KIT-ML [26] use SMPL 17 sequences represent motions. SMPL describes both the body shape and the body poses, where the body poses include pelvis location and rotation, the relative joint rotation of the 21 body joints. We build simulated character to have the same skeleton as SMPL. We scale the simulated character to have similar body size to a mean SMPL neutral shape. For retargeting, we directly copy the joint rotation angle, pelvis rotation, and translation to the simulated character. A vertical offset is applied to compensate for different floor heights.
probabilistic techniques used remove Gaussian noise from data and generate a clean output. These models consist two processes: the diffusion process and the reverse process. The diffusion process gradually adds Gaussian noise to the original data for a specified number of steps, denoted by T. until the distribution of the noise closely approximates standard Gaussian distribution, denoted by (O I). This generates sequence of noisy trajectories denoted by TI.T = (TI,... Tr). The original data is sampled from a conditional distribution, To P(To c), where is the instruction. Assuming that the variance schedules determined by Be, the diffusion process is defined as:
Diffusion Models. Diffusion models
T a(TITTTT)) = II ((TTTTIL ((T((T=-)) = N(T: BITT-1, t=1
where q(Te/T(-1) is the conditional distribution of each step in the Markov chain. The parameter 3 controls the amount of noise added at each step f, with larger values resulting in noise added. The reverse process in diffusion models another Markov chain that predicts and removes the added noise using learned denoising function. In particular, we encode language into encoded latent vector, é, using the classical transformer (37] as the language encoder, è = E(c). Thus, the reverse process starts with a distribution p(Tr) N 0, I) and defined as:
((TorT()) = p(TT) TT) P(T = C,
Here, P(T:-I/TE.. é) the conditional distribution at each step the reverse process. The mean and covariance of the Gaussian are represented by and Σ, respectively. During training, steps / are uniformly sampled for each ground truth motion To, and a sample is generated from g(T./To). Instead of predicting the noise term (TT) the model predicts the original data directly which has

---

Algorithm 1 Inference of InsActor
1: Input: A instruction c, the diffusion model fo. the skill encoder, lo and decoder pa. Diffusion steps T. A language encoder &. A history [ {81,-.., St
A waypoint h. Animation length L.
2: Output: A physically simulated trajectory. 4.
Generate state sequence.
3: w f Initialize a plan from a Gaussian noise, ~ (0,
4: W Apply the inpainting strategy in [4 for history o and waypoint guide diffusion.
5: è < Encode the instruction (c)
6: T {s1. .5 - Generate trajectory with the diffusion model fo(w, w, 2)
b Generate action sequence in a closed-loop manner.
7: for = 1,1 I.... (L 1) do
8:
< Sample from go (2:(S:, Si+1)
9: a; f Sample action form Pa(aa (a; (Si, Zi)
10:
S,+1 + Get next state T( [8,, a,)
11: end for
12: Output f ($141, 8L}
the equivalent formulation 28 36 This is done by using neural network fo parameterized by 0 to predict To from the noisy trajectory т, and the condition at each step in the denoise process. The model parameters optimized by minimizing the mean squared error between the predicted and ground truth data using the loss function:
Plan Ese(I, ( To (T Il].
where p(Tolc) the conditional distribution of the ground truth data, and || || denotes the mean squared error. By directly predicting this formulation avoids repeatedly adding noise to and is more computationally efficient.
Guided Diffusion. Diffusion models allow flexible test-time conditioning through guided sampling, for example, classifier-guided sampling [6) Given objective function condition, gradients can be computed to optimize the objective function and perturb the diffusion process. In particular, simple yet effective inpainting strategy can be applied introduce state conditions [14), which is useful to generate a plan that adheres to past histories or future goals. Concretely, the inpainting strategy formulates the state conditioning as a Dirac delta objective function. Optimizing the objective function is equivalent to directly presetting noisy conditioning states and inpainting the rest. We leverage the inpainting strategy to achieve waypoint heading and autoregressive generation.
Limitation. Despite the ability to model complex language-to-motinn relations, motion diffusion models can generate inaccurate low-level details, which lead physically implausible motions and artifacts like foot floating and foot penetration [36]. In the context of state diffusion, the diffuser- generated states can be invalid and the state transitions can be infeasible. Thus, direct tracking of the diffusion plan can be challenging.
Low-Level Skill Discovery
To tackle the aforementioned challenge, employ low-level skill discovery to safeguard against unexpected states in poorly planned trajectories. Specifically, we train a Conditional Variational Autoencoder to map state transitions to compact latent space in unsupervised manner [40] This approach benefits from a repertoire of learned skill embedding within compact latent space, enabling superior interpolation and extrapolation. Consequently, the motions derived from the diffusion model can be exccuted by natural motion primitives.
Skill Discovery. Assuming the current state of the character is š) the first step in constructing compact latent space for skill discovery encoding the state transition in a given reference motion sequence, .... s represents a unique skill required to transition from & to 8/41. The neural network used to encode the skill embedding is referred to as the encoder, 4o. parameterized by é. which produces Gaussian distribution:
into latent variable, 2, and we call this skill embedding. This variable
qs ( S1, s = (Z1;; (S 1i+)) ((r.s)
(4)
5

---

"A person picks up."
"A person doing martial arts."
"A person raises arms and walks."
"A person walks like zombie." Figure 3: Qualitative results of InsActor with corresponding instructions. Top: only human instruction. Bottom: human instruction and waypoint target.
"A person jumps."
"A person spins."
where Me is the mean and Eg the isotropic covariance matrix. Once obtain the latent variable, decoder, Pi [sr zt). parameterized by vh, generates the corresponding actions, a, by conditioning on the latent variable z and the current state Sy:
a1 Pa St, 21)
Subsequently, using the generated action, the character transitions into the new state, $1+1. via the transition function T ($ [S;, at). By repeating this process, we can gather generated trajectory, denoted as + {$1 .... $ )- The goal is mimic the given trajectory To by performing the actions. Thus, to train the encoder and decoder, the main supervision signal is derived from the difference between the resulting trajectory + and the reference motion, Tu-
Training. Our approach leverages differentiable physics train the neural network end-to-end without the need for a separate world model [1]. This is achieved by implementing the physical laws of motion differentiable functions, allowing the gradient to flow through them during backpropagation. Concretely, by executing action a; ~ Pip (ar|Sr, zt) at state št, the induced state 81+1 is differentiable with respect the policy parameter vi and Thus, directly minimizing the difference between the predicted state & and the s* gives an efficient and effective way of training an imitation learning policy (3I] The Brax 77] simulator is used due its efficiency and easy parallelization, allowing for efficient skill discovery. It also ensures that the learned skill is trained on the actual physical environment, rather than simplified model of it, leading more accurate and robust representation.
Thus, the incoder-decoder is trained with an objective that minimizes the discrepancy between resulting trajectories and Heilbackk-eibble divergence between the encoded latent variable and the prior distribution, which standard Gaussian,
Ssaal =| To + || (2/sss) (o,,
(6)
and (s, is pair of states before and after transition. The latter term encourages the latent variables to be similar to the prior distribution, ensuring the compactness of the latent space. À is the weight factor that controls the compactness. During inference, we map the generated state sequence from the diffusion model to the skill space to control the character.
where || denotes the mean squared
Experiments
The goal of our experiment evaluate the effectiveness and robustness InsActor in generating physically-simulate and visually-natura character animations based on high-level human instruc- tions. Specifically, we aim to investigate /) whether InsActor can generate animations that adhere to human instructions while being robust to physical perturbations, 2) whether InsActor can accomplish waypoint heading while being faithful the language descriptions, and 3) the impact of several design choices, including the hierarchical design and the weight factor for skill space compactness.

---

Table 1: Quantitative results on the KIT-ML test set. f: with perturbation.
R Precisiont
Methods
Multimodal Dist.
FID
Diversity?
Top
Top
Top
DReCon 4
0.243.1.000 000
0.42011.02 021
0.5221.039
2.31011.09
1.0551.112
4.259.1.014
PADL 15
0.091#.008
0.172+0008
0.242#.015
3.482 0338
3.889++.044
2.940=.031
InsActor (Ours)
0.352.1.013 013
0.55011.010
0.64811015
1808 027
0.786:1.055
4.392-1077
DReCon
0,253: 013
0.3844.006
0.447=.006
2.7644 003
1.973:.110
4.252
PADLI hS
0.100:1.012
0.1581.011
0.2171.015
3.783:1.099
4.706:1.
3. 3.168..065
InsActor (Ours)
0.323=20133
0.49
0.599#.008
2.147#.061
1.043#0091
4.359=0073
Table 2: Quantitative results on the HumanML3D test set. f: with perturbation.
R Precisionf
Methods
Multimodal Dist.
FID
Diversity?
Top 1
Top 2
Top 3
DReCon
0.265 .077
0.391
0.470.1.001
2.570.L.00%
1.244 .000
4.070.L.062
PADL 15
0.144 003
0.22711012
0,29711018
3.349:1.00
2.162:4: 022
3.73611.091
InsActor (Ours)
0.331#.000
0.497:1013
0.5981.001
1.9711:004
0.566:t.023
4.165:t.066
DReCon 4
0.233 000
0.352 001
0,424+000
2.850=.002
1.829+ 0002
4.008
PADL 15
0.117= 005
0.192#.000
0,254+0000
3.660=.040
2.964+1115
3.849+.159
InsActor (Ours)
0.455.1.006
0.5461.003
2.203.1.006
0.694.1.005
4.212.1.154
5.1
Implementation Details
To implement the experiment, character based on Deep.Mmmic [25]. The character has 13 links and 34 degrees of freedom, weighs 45kg, and is 1.62m tall. Contact is applied to all links with the floor. For details of neural network architecture and training, we refer readers to the supplementary materials.
Brax 7 to build the environment and design simulated
5.2 Evaluation Protocols
Datasets. We use two large scale text-motion datasets, KIT-MI. 26 and HumanMI.3D ( for training and evaluation. KIT-MI. has 3,911 motion sequences and 6,353 sequence-level language descriptions, HumanML3D provides 44,970 annotations on 14,616 motion sequences. We adopt the original train/test splits in the two datasets.
Metrics.
We employ the following evaluation metrics:
1. R Precision:
For every pair of generated sequence and instruction, we randomly pick 31 additional instructions from the test set. Using a trained contrastive model, we then compute the average top-k accuracy.
2. Frechet Inception Distance (FID): We use pre-trained motion encoder to extract features from both the generated animations and ground truth motion sequences. The FID is then calculated between these two distributions to assess their similarity.
3. Multimodal Distance: With the help of a pre-trained contrastive model, we compute the disparity between the text feature derived from the given instruction and the motion feature from the produced animation. We refer this as the multimodal distance.
4. Diversity: To gauge diversity, randomly divide the generated animations for all test texts into pairs. The average joint differences within each pair are then computed as the metric for diversity.
5. Success Rate: For waypoint heading tasks, we compute the Euclidean distance between the final horizontal position the character pelvis and the target horizontal position. If the distance is less than 0.5 m, deem it success. We perform each evaluation three times and report the statistical interval with 95% confidence.
7

---

Table 3: Quantitative results for the waypoint heading task. Evaluated HumanML3D. We set the start point at (0,0) and the waypoint uniformly sampled from a 6x6 square centered at (0,0). is considered successful waypoint heading if the final position is less than 0.5m away from the waypoint. L: Langauge. W: Waypoint.
R Precision
Method
Multimodal Dist.
FID
Diversity
Success Rate
Top
DReCon
v
D. 0000
4.192:t.019
8.607
2.5831.
0.380:t.002
InsActor (Ours)
0.08911.001
4.106:1.001
3.0411.101
3.137.1.029
1.352.002
InsActor (Ours)
x
0.598#.001
1.971#.004
0.566#.023
4.165#.076
0.081#.004
InsActor (Ours)
0.3881.003
2.7534.000
2.5274.015
3.285=.034
0.9071.002
Table 4: Ablation on hierarchical design. Evaluated on the KIT-ML test set.
R Precisionf
High
Low
MultiModal Dist.
FID
Diversity?
Top I
Top 2
Top 3
X
0.2641.011
0.398:10116
0.460:2.018
2.692:1.034
1.501:1.095
4.370:1.06
0.068#.011
0.145+.030
0.188#.024
3.707+0096
L.106#0093
4.148+t008
0.3522.013
0.550.1.010
0.648.1.015
1.8081.027
0.7861.055
4.392.1.07
5.3 Comparative Studies for Instruction-drivet Character Animation
Comparison Methods. We compare InsActor with two baseline approaches: /) DReCon 4 We adapted the responsive controller framework from DReCon [4. We use the diffuser as a kinematic controller and train target-state tracking policy. The baseline can also be viewed as a Decision Diffuser [ with long planning horizon, where a diffuser plans the future states and a tracking policy solves the inverse dynamics. 2) PADL [15): We adapt the language-condttioned control policy in PADL [[5., where language instructions are encoded by a pretrained cross-modal text encoder (27) and input to a control policy that directly predict actions. It is also a commonly used learning paradigm in conditional imitation learning [34]. Since the two basclines have no publicly available implementations, we reproduce them and train the policies with DiffMimic (31
Settings. We utilize two different settings to assess InsActor's robustness. In the first setting, evaluate the models in a clean, structured environment devoid of any perturbation. In the second setting, we introduce perturbations by spawning 2kg box to hit the character every 1 second, thereby evaluating whether the humanoid character can still adhere to human instructions even when the environment changes.
and qualitative results in Figure Compared Results. We present the results in Table 1 and Table and
to the dataset used in PADL that consists of 131 motion sequences and 256 language captions, our benchmark dataset is two orders larger, where the in PADL has difficulty to scaling up. In particular, the inferior performance in language-motion matching metries suggests that a single-step policy fails to understand unseen instructions and model the many-to-many instruction-motion relation. Compared to PADL, DReCon shows better result in language-more motion matching thanks to the high-level motion planning. However, unlike Motion Matching used in DReCon that produces high-quality kinematic motions, the diffuser generates invalid states and infeasible state transitions, which fails DReCon's tracking policy and results in low FID. In comparison, InsActor significantly outperforms the two baselines on all metrics. Moreover, the experiment reveals that environmental perturbations do not significantly impair the performance of InsActor, showcasing InsActor's 's robustness.
contiaiiiooddeioneeed single-step policy used
8

---

"A person crouches." "A person kicks." Figure 4: Qualitative results of InsActor with history conditioning. Generation is conditioned on the second human instruction and history motion.
"A person crouches."
"A person runs."
5.4 Instruction-driven Waypoint Heading
Waypoint Heading. Thanks to the flexibility of the diffusion model, InsActor readily accom- plish the waypoint heading task, a common task in physics-based character animation 40 15]. This task necessitates the simulated character move toward target location while complying with human instructions. For instance, a human instruction might be, "walk like a zombie." In this case, the character should navigate toward the target position while mimicking the movements of zombie.
Guided Diffusion. We accomplish this using guided diffusion. Concretely, we adopt the inpainting strategy in Diffuser [44. Prior to denoising, replace the Gaussian noise in the first and last 25% frames with the noisy states of the character standing at the starting position and target position respectively.
Results. We conduct this experiment with the model trained on H3manML.3D. We contrast Insoctor with DReCon and two InsActor variants: /) InsActor without language condition, and 2) InsActor without targeting. Our experimental results demonstrate that InsActor can effectively accomplish the waypoint heading task by leveraging guided diffusion. The model trained on HumanMI.3D capable of moving toward the target position while following the given human instructions, as evidenced by a low FID score and high precision. Although adding the targeting position condition to the diffusion process slightly compromises the quality of the generated animation, the outcome still satisfactory. Moreover, the success rate of reaching the target position is high, underscoring the effectiveness of guided diffusion. Comparing InsActor with its two variants highlights the importance of both the language condition and the targeting in accomplishing the task. Comparing InsActor with DReCon shows the importance of skill mapping, particularly when more infeasible state transitions introduced by the waypoint guiding. Without skill mapping, DReCon only has 38.0% success rate, which drops drastically from the 90.7% success rate of InsActor.
Multiple Waypoints. Multiple waypoints allow users to interactively instruct the character. We achieve this by autoregressively conditioning the diffusion process to the history motion, where qualitative result is shown in Figure 4 Concretely, we inpaint the first 25% with the latest history state sequences. We show qualitative results for multiple-waypoin following in Figure and more in the supplementary materials.
5.5 Ablation Studies
Hierarchical Design. To understand the importance of the hierarchical design in this task, we performed an ablation study on its structure. We compared approach two baselines: 1) A policy with only a high-level policy, wherein the diffusion model directly outputs the skills, analogous to the Diffuser approach [14); 2) A low-level policy that directly predicts single-step skills. We show the results in Table By leveraging skills, the low-level policy improves from PADL but still grapples with comprehending the instructions due to the absence of language understanding. Conversely, without the low-level policy, the skills generated directly by the diffusion model are of poor precision. Although the use of skills safeguards the motions to be natural and score high in FID, the error accumulation deviates the plan from the language description and results in a low R-precision. The experimental results underscore the efficacy of the hierarchical design of InsActor.

---

Weight Factor. InsActor learns a compact la- tent space for skill discovery to overcome infea- sible plans generated by the diffusion model. We conduct an ablation study on the weight factor, A, which controls the compactness of the skill space. Our findings suggest that a higher weight factor results in more compact latent space, however, it also curtails the instruction-motiin alignment. Conversely, lower weight factor permits greater diversity motion generation, but might also lead to less plausible and in- consistent motions. Hence, is vital to find balance between these two factors optimize performance for the specific task at hand.
R Precision (Top 3)
FID
R Procision (Top 3)
-20
0.6
0.5
1.5
0.4
1.0
0.3
0.5
10
10"
Figure 5: Ablation study the weight factor X. Evaluated on the KIT-ML dataset.
Conclusion
In conclusion, we have introduced InsActor, a principled framework for physics-based character animation generation from human instructions. By utilizing a diffusion model to interpret language instructions into motion plans and mapping them to latent skill vectors, InsActor can generate flexible physics-based animations with various and mixed conditions including waypoints. We hope InsActor would serve as an important baseline for future development of instruction-drive animation. While InsActor capable of generating such animations, there crucial but exciting challenges ahead. One limitation is the computational complexity of the diffusion model, which may pose challenges for scaling up the approach to more complex environments and larger datasets. Additionally, the current version of InsActor assumes access to expert demonstrations for training, which may limit its applicability in real- wol-w scenarios where such data may not be readily available. Furthermore, while InsActor is capable of generating physically-reliabl and visually-plausible animations, there is still room for improvement in terms of the quality and diversity of generated animations. There are mainly two aspects for future development, improving the quality of the differentiable physics for more realistic simulations and enhancing the expressiveness and diversity of the diffusion model to generate more complex and creative animations. Besides them, extending InsActor to accommodate different human body shapes and morphologies is also an interesting direction.
physics-based
From a societal perspective, the application of InsActor may lead ethical concerns related how might be used. For instance, InsActor could be exploited to create deceptive or harmful content. This underscores the importance of using InsActor responsibly.
Acknowledgment
This research supported by the National Research Foundation, Singapore under its Singa- pore Programme (AISG Award No: AISC2-PHD-2021--88-0118 NTU NAP, MOE AcRF Tier 2 (T2EP20221-0012) and under the RIE2020 Industry Alignment Fund Industry Collaboration Projects (IAF-ICP) Funding Initiative, well cash and in-kind contribution from the industry partner(s).
References
[1] Anurag Ajay, Yilun Du Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision-maan making? arXiv preprint arXiv:2211.15673 2022.
[2] Sadegh Aliakbarian, Fatemeh Sadat Saleh, Mathieu Salzmann, Lars Petersson, and Stephen Gould. A stochastic conditioning scheme for diverse human motion prediction. In Proceddings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 5223-5232.
2020.
10

---

[3] Mykhaylo Andriluka, Umar Iqbal, Eldar Insafutdinov, Leonid Pishchulin, Anton Milan, Juergen Gall, and Bernt Schiele. Posetrack: benchmark for human pose estimation and tracking. In Proceedings of the IEEE conference computer vision and puttern recognition, pages 5167-5176, 2018.
[4] Kevin Bergamin, Simon Clavet, Daniel Holden, and James Richard Forbes. Drecon: Data-driven responsive control of physics-based characters. ACM Trans. Gruph, 38(6), November 2019.
[5] Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems (RSS), 2023.
[6] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances Neural Information Processing Systems, 34, 2021.
[7] Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax-a differentiable physics engine for large scale rigid body simulation. arXiv preprint arXiv:2 13281, 2021.
[8] Chuan Guo, Xinxin Zuo, Sen Wang, Shihao Zou, Qingyao Sun, Annan Deng, Minglun Gong, and Li Cheng. Action2motion: Conditioned generation of 3d human motions. In Proceedings of the 28th ACM International Conference on Multimedia,, pages 2021-2029, 2020.
[9] Félix G Harvey, Mike Yurick, Derek Nowrouzezahrai, and Christopher Pal. Robust motion in-betweening. ACM Transactions Graphics (TOG), 39(4):60-1, 2020.
[10) Gustav Eje Henter, Simon Alexanderson, and Jonas Beskow. Moglow: Probabilistic and controllable motion synthesis using normalising flows. ACM Transactions on Graphics (TOG), 39(6):114, 2020.
[11] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances Neural Information Processing Systems, 33:6840-6851, 6851, 2020.
[12] Leslie Ikemoto, Okan Arikan, and David Forsyth. Generalizing motion edits with gaussian processes, ACM Transactions Graphics (TOG), 28(1):1-112 2009.
[13] Catalin lonescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3. 6.m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE transactions pattern analysis and machine intelligence, 36(7):1325-1339, 2013.
[14] Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning, 2022.
[15] Jordan Juravsky, Yunrong Guo, Sanja Fidler, and Xue Bin Peng. Padl: Language-directed physics-based character control. Association for Computing Machinery, 2022.
[16] Libin Liu, Michiel van de Panne, and KangKang Yin. Guided learning of control graphs for physics-based characters. ACM Transactions on Graphics, 35(3), 2016.
[17] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons- Pons-Moll, and Michael J Black. Smpl: A skinned multi-person linear model. ACM transactions on graphics (TOG), 34(6):1-16. 2015.
[18) Corey Lynch and Pierre Sermanet. Language conditioned imitation learning over unstructured data. Robotics: Science and Systems, 2021.
[19] Oier Mees and Wolfram Burgard. Composing pick-and-place tasks by grounding language. In International Symposium on Experimental Robotics, 2021.
[20] Oier Mees, Alp Emek, Johan Vertens, and Wolfram Burgard. Learning object placements for relational instructions by hallucinating scene representations. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 94-100, 2020.
[21] Jianyuan Min and Jinxiang Chai. Motion graphs++ a compact generative model for semantic motion analysis and synthesis. ACM Transactions Graphics (TOG), 316))1112, 2012.
11

---

[22] Dirk Ormoneit, Michael Black, Trevor Hastie, and Hedvig Kjellström. Representing cyclic human motion using functional analysis. Image and Vision Computing, 23(14):1264-1276 2005.
[23] Tim Pearce, Tabish Rashid, Anssi Kanervisto, Dave Bignell, Mingfei Sun, Raluca Georgescu, Sergio Valcarcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann, and Sam Devlin. Imitating human behaviour with diffusion models. In The Eleventh International Conference on Learning Representations, 2023.
[24] Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne. Deepmimic: Example- guided deep reinforcement learning of physics-based character skills. ACM Trans. Graph., 3 (144):44::1-43::: July 2018.
[25] Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel Van de Panne. Deepmimic: Example- guided deep reinforcement learning of physics-based character skills. ACM Transactions On Graphics (TOG), 2018.
[26] Matthias Plappert, Christian Mandery, and Tamim Asfour. The kit motion-language dataset. Big data, 444):236-252, 2016.
[27] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. CORR, abs/2103.00020, 2021.
[28] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
[29] Davis Rempe, Leonidas J. Guibas, Aaron Hertzmann, Bryan Russell, Ruben Villegas, and Jimei Yang. Contact and human dynamics from monocular video. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
[30] Davis Rempe, Zhengyi Luo, Xue Bin Peng, Ye Yuan, Kris Kitani, Karsten Kreis, Sanja Fidler, and Or Litany. Trace and pace: Controllable pedestrian animation via guided trajectory diffusion. In Conference on Computer Vision and Puttern Recognition (CVPR), 2023.
[31] Jiawei Ren, Cunjun Yu, Siwei Chen, Xiao Ma, Liang Pan, and Ziwei Liu. Diffmmimic: Efficient motion mimicking with differentiable physics. ICLR, 2022.
[32] Soshi Shimada, Vladislav Golyaník, Weipeng Xu, and Christian Theobalt. Physcap: Physically plausible monocular 3d motion capture in real time. ACM Trans. Graph, 39(6), nov 2020.
[33] Mohit Shridhar and David Hsu. Interactive visual grounding of referring expressions for human-robot interaction. In Proceedings of Robotics: Science and Systems, 2018.
[34] Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In Proceedings of the 5th Conference on Robot Learning (CoRL), 2021.
[35] Simon Stepputtis, Joseph Campbell, Mariano Phielipp, Stefan Lee, Chitta Baral, and Heni Ben Amor. Language-condittoned imitation learning for robot manipulation tasks. In Proceedings of the 34th International Conference on Neural Information Processing Systems, 2020.
[36] Guy Tevet, Sigal Raab, Brian Gordon, Yoni Shafir, Daniel Cohen-or, and Amit Haim Bermano. Human motion diffusion model. In The Eleventh International Conference on Learning Repre- sentations, 2023.
[37] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez. Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[38] Zhenyi Wang, Ping Yu, Yang Zhao, Ruiyi Zhang, Yufan Zhou, Junsong Yuan, and Changyou Chen. Learning diverse stochastic numan-action generators by learning smooth latent transitions. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 12281-12288.
2020.
12

---

[39] Xinchen Yan, Akash Rastogi, Ruben Villegas, Kalyan Sunkavalli, Eli Shechtman, Sunil Hadap, Ersin Yumer, and Honglak Lee. Mt-vae: Learning motion transformations generate mul- timodal human dynamics. In Proceedings of the European conference computer vision (ECCV), pages 265-281, 2018.
[40] Heyuan Yao, Zhenhua Song, Baoquan Chen, and Libin Liu. Controlvae: Model-based learning of generative controllers for physics-basee characters. 41(6), 2022.
[41] Ye Yuan, Jiaming Song, Umar lqbal, Arash Vahdat, and Jan Kautz. Physdiff: Physics-guise guided human motion diffusion model. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
[42] Ye Yuan, Shih-En Wei, Tomas Simon, Kris Kitani, and Jason Saragih. Simpoe: Simulated character control for 3d human pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[43] Hanbo Zhang, Yunfan Lu, Cunjun Yu, David Hsu, Xuguang Lan, and Nanning Zheng. Invigorate: Interactive visual grounding and grasping in clutter. Proceedings of Robotics: Science and Systems, abs/2108. 2021.
[44] Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondiffuse: Text-driven human motion generation with diffusion model. arXiv preprint arXiv: 15001, 2022.
13


Reviews:
[{"review_id": "KNQHOwGTa0", "reviewer": "Reviewer_ndRp", "summary": "The paper presents an approach for generating animations using a language-conditioned neural network in particular a diffusion model. The underlying goal is to generate physics based human character animation in an easy and intuitive fashion. The approach leverages a diffusion model to generate high-level states of the animated character which are then refined in a lower level module. This module is realized via an auto encoder which generates skill embedding representing the state transition in a compact fashion. A decoder is then used to synthesized the final action of the agent. The approach also uses an inpainting strategy to ensure that the agent goes through critical phases or points, i.e., this seems similar to the difference of keyframes and in-betweens in traditional animation. The paper compares the introduced approach to other recent methods on the topic, e.g., PADL and concludes among other things that more faithful animations (wrt to the original language query) are generated. The contributions of the paper is the combination of Diffusion and autoencoder models for motion generation, the inpainting strategy and waypoint system.", "questions": "- What exactly is the transition function? I am assuming that is the differential simulator, right?", "limitations": "As described above, the lack interaction with the environment and linguistic goals is a major limitation that has not been brought up in the paper. The system is not able to execute commands like \"lift the green cup\" or \"move to the lion statue\". That substantially limits the applications of the approach. It is also surprising that the authors mention problems like foot skating at the beginning but do not describe how that is resolved later in the paper. While I understand that in a physics-based system the dynamics of the environment can make some of these considerations obsolete, there is still room for broken animations if this not adequately addressed. Hence, I would ask the authors to address this point in the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper addresses a really interesting and important problem that is very exciting. It is also one of the first papers to use diffusion models in this context and the results seem promising. The combination of in-painting with diffusion is also very clever (but details are missing). A BIG plus compared to other similar systems is that it does not require RL. For example, PADL requires multiple policies to be learned and later combined with each policy requiring 7 simulated years to be trained! That does not seem to be the case here since this is a purely supervised approach - even though substantial computation is required for all deep learning of course.\nI really appreciated the use of contrastive models to evaluate the faithfulness of generated models. Diversity was also an interesting measure since in animation it is critical that the character shows a range of motions. Generally the paper is written in a way that can be followed (with caveats see below). \n\n** Update after rebuttal **\nI have increased my score to \"weak accept\". I appreciate the details provided by the authors - this helped better contextualize and understand the paper. What is clear from the rebuttal process is that it is hard to judge the visual fidelity alone, despite the substantial effort the authors put into the evaluation section. Especially the choice of visualizing the resulting animations via retargeting to a robot (in Blender) may have negatively impacted the reviewers' opinion. The animations in the latter part of the video are actually more convincing in my opinion. However, as pointed out before, the animations are not well-adapted to the environment, i.e., foot skating, actions on objects, etc. In future submissions on the topic, I strongly advise addressing this topic even if only partially.", "weaknesses": "A major question that the paper is not addressing is the generation of actions on objects and surroundings. A critical element of any character animation is to be able to specify an object acted upon, e.g., \"push the red object on the table\". That is not addressed at all in this paper, while it is addressed on other papers in robotics (the manipulation papers) but also animation (PADL). In PADL, a module specifically identifies the target object and animations are synthesized accordingly. In the robotics papers [Lynch2020, Stepputtis2020] the actions of the agents are also conditioned on an image of the surroundings and target objects are visually identified in the image. Potentially that could be included into InsActor through the waypoint system (maybe?), but it is so far not addressed in the paper. Personally, this is the biggest flaw of the paper, since at the moment it would not be appealing to me or many others to use this system.\nAlong these lines, the presented example results are not very convincing with regards to the complexity of the generated motions. \n\nAnother weakness of the paper is that it seems important details are left out or not well justified. For example, the waypoint encoding is interesting but should be explained with an example. I am particularly unsure what this would entail for the user - do we have to place the character at those waypoints and set all of its joint values? This would require the user to actually become an animator and generate keyframes. That could be a serious limitation if true. Right now I am assuming we only need to position the basic character at the waypoint (i.e. modify position and orientation of agent) not moving limbs (i.e. modify joint angles). The paper states \"...replace the Gaussian noise in the first and last 25% frames with the noisy states of the character standing at the starting position and target position respectively.\"\n\nPersonally, I would really have liked to see a long sequence in which the agent performs a number of different actions in a row. At Siggraph they often have these cute teaser figures in which a humanoid walks, crouches, jumps etc. over a period of time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for comprehensive comments. We would like to address your concern as follows:\n\n> Q1: A major question that the paper is not addressing is the generation of actions on objects and surroundings. Potentially that could be included into InsActor through the waypoint system (maybe?)\n\nWe agree that enabling character-object interaction is important in animation generation. Although it is a more challenging setting, we believe that our framework will be compatible with the setting by additionally including objects into the states. The high-level controller can generate motion plans conditioned on object states and the low-level controller will execute the interaction. For example, given the information of a chair, a plan for a character to sit on the chair will be generated and executed without specifically training for the task. However, extending our current waypoint encoding system to encode object information including different types, sizes, and 3D locations is highly non-trivial. In addition, the current motion data scale for human-scene interactions may not match the complexity of the problem. Therefore, we do not consider object interactions in this work. Nonetheless, we do notice recent works like [A] that show promising results on physics-based character-object interaction, and view it as a vital future work. We believe our approach will serve as an important baseline that is extendable to character-scene joint modeling and expandable with additional human-scene motion data.\n\n> Q2: Another weakness of the paper is that it seems important details are left out or not well justified. For example, the waypoint encoding is interesting but should be explained with an example.\n\t\nThis is a great point for us to clarify! As described in the main paper, the general idea of waypoint conditioning is to encode a sequence of mean poses at the starting and the ending locations. Specifically, we first set the positions and rotations of all body links to zero, which will be a mean pose after denormalization. And then, we add a normalized position offset to move the mean pose to the starting point and the ending point. For both starting and ending, we pad the mean pose for 25% of the sequence length. In the diffusion sampling process, we fix the noise-injected mean poses following the inpainting technique. All the rest of the plan, including velocity and angular velocity in the beginning and ending state, will be generated by the diffusion planner. We will add an illustrative figure with a concrete example in future versions.\n\n> Q3: Personally, I would really have liked to see a long sequence in which the agent performs a number of different actions in a row. At Siggraph they often have these cute teaser figures in which a humanoid walks, crouches, jumps etc. over a period of time.\n\t\nThanks for the great suggestion! InsActor is able to perform different actions in a row with the combination of language condition and waypoint encoding. Figure 1 included in the paper is achieved by doing so. Concretely, Fig 1. includes four text prompts which instruct the character to sequentially crouch, jump, walk, walk like a zombie to a waypoint, and finally end with a kick. We have also shown the corresponding animation in the supplementary material. \n\n\n> Q4: What exactly is the transition function? I am assuming that is the differential simulator, right?\n\nYes, the transition function is the differentiable simulator/dynamics.\n\nAdditional References:\n\n[A] Hassan et al., “Synthesizing Physical Character-Scene Interactions”, SIGGRAPH, 2023"}, {"review_id": "QKoEBeiQjl", "reviewer": "Reviewer_Mekg", "summary": "The paper proposes an approach for generating physically plausible human motion from open text prompts. The approach combines high-level trajectory generation with guided diffusion and a low-level skill model encoded in a latent space to correct for physical plausibility during execution in simulation with PD control. The proposed approach is compared against existing motion generation approaches with a set of quantitative metrics and qualitative evaluation. ", "questions": "Should cite Trace and Pace: Controllable Pedestrian Animation via Guided Trajectory Diffusion for similar architecture (guided-diffusion trajectories + low-level physical controller) https://arxiv.org/abs/2304.01893\nTable 3 ordering of metrics is different from Metrics text section. Swap Multimodal Distance and FID in one or the other.\nIs it feasible to generalize this stack to different body shapes, dynamics, to include objects or other kinds of waypoints?\nWould it be possible/better to have some model-based or explicit low-level skill experts for specific sub-tasks (e.g. locomotion) to achieve higher fidelity in addition to general/open motion skills?\n", "limitations": "Discussion of limitations was pretty good and I appreciated the limitation notes throughout the paper. I did not see any mention of generalizing to other character morphologies (even different human body shapes). I currently believe this work is using a single embodiment as a strong assumption for achieving accuracy and physical plausibility. I would be interested to hearing more about the feasibility of generalization for applications.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall this work soundly presents an improvement to SoTA in the space of human motion generation from text prompts. The use of guided diffusion to produce high-level trajectories is not surprising, nor is its effectiveness when combined with low-level skill models as a method of correcting physical implausibilities and errors. The paper is well written and clearly describes the architecture, training procedure, and use of guidance for waypoint following. Evaluation is sound and clearly demonstrates the strengths of the approach. I appreciate the use of multiple quantitative metrics and stochastic evaluation of many samples. This work benefits from the existence and availability of its component parts (Brax, ControlVAE, motion diffusion models), but connects them to achieve fairly good results. I’m always happy to see motion generation connected to physics simulation instead of remaining kinematic. The results of this work seem fairly robust to perturbation, which is a good signal for the usefulness of future iterations of the approach.\n", "weaknesses": "While the resulting motion is fairly good, it isn’t yet achieving the level of fidelity to make it generally useful in generative contexts (e.g. game or animation characters).  \nThis work achieves its success from connecting other existing techniques, though the overall system is a sound contribution.\nThe stack is trained considering only the character’s joint space and a specific character’s physical constraints. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for constructive comments. We would like to address your concern as follows:\n\n> Q1: While the resulting motion is fairly good, it isn’t yet achieving the level of fidelity to make it generally useful in generative contexts (e.g. game or animation characters)\n\nThis is a great point for us to elaborate more! As our approach delivers the first large-scale instruction driven physics-based character baseline, we also identify several aspects to improve the motion fidelity:\n1. Simulation quality. Although all of our results are from physics simulation, the differentiable simulator may not generate high-fidelity motions. We note that the Brax engine has made approximations to achieve the differentiability. As a result, our resulting motion may have some artifacts as observed. This is a known issue in the state-of-the-art motion tracker DiffMimic that we use.  Nonetheless, we believe that this can be resolved by future efforts in developing a better differentiable physics engine.\n2. Data Scale. Being a data-driven approach, we recognize that the data scale is an important bottleneck. Despite using the largest text-to-motion dataset HumanML3D, it is far smaller than billion-level text-to-image datasets. We believe that our approach serves as the first  scalable instruction-driven physics-based character animation baseline, and will continuously improve when more data is getting available.\n\n> Q2: This work achieves its success from connecting other existing techniques, though the overall system is a sound contribution. The stack is trained considering only the character’s joint space and a specific character’s physical constraints.\n\t\nAction generation in the joint space is the commonly adopted protocol in prior works[22,A] as they are general enough to achieve various kinds of motions. We use the same protocol in order to align with previous works.\n\n> Q3: Should cite Trace and Pace: Controllable Pedestrian Animation via Guided Trajectory Diffusion for similar architecture.\n\t\nThanks for bringing this paper to our attention! This is an important related work for InsActor we shall discuss. We will add it into the manuscript in the final version.\n\n> Q4: Swap Multimodal Distance and FID in one or the other. Is it feasible to generalize this stack to different body shapes, dynamics, to include objects or other kinds of waypoints? \n\nWe believe that our framework can be extended to include objects into the states. The high-level controller can generate motion plans conditioned on object states and the low-level controller will execute the interaction. For example, given a chair, a plan for a character to sit on the chair will be generated and executed without specifically training for the task. However, extending our current waypoint encoding system to encode object information including different types, sizes, and 3D locations is highly non-trivial. In addition, the current motion data scale for human-scene interactions may not match the complexity of the problem. Therefore, we do not consider object interactions in this work. Nonetheless, we do notice recent works like [A] that show promising results on physics-based character-object interaction, and view it as a vital future work. We believe our approach will serve as an important baseline that is extendable to character-scene joint modeling and expandable with additional human-scene motion data.\n\nWe answer the question regarding different body shapes in Q6.\n\n> Q5: Would it be possible/better to have some model-based or explicit low-level skill experts for specific sub-tasks (e.g. locomotion) to achieve higher fidelity in addition to general/open motion skills?\n\nWe agree that a single policy network for all motion skills on a large dataset isn't optimal. Previous motion tracking works have demonstrated that an MoE ensemble enhances model capacity and captures dynamic moves more effectively[B,C]. In these works,  the motion dataset is categorized into groups, and an expert policy is trained for each group. A higher-level policy can then determine the best expert policy deployment. Although expert ensemble is a promising approach to achieve high  motion fidelity, implementing an expert system from scratch requires large engineering efforts and could take a long time for training[C]. Since this work is less focused on motion tracking but more on language understanding, we choose DiffMimic which can be trained efficiently as shown in Figure 1 in the attached PDF.\n\n> Q6: I did not see any mention of generalizing to other character morphologies (even different human body shapes).\n\n1. Different Morphology: This direction is intriguing but remains challenging due to current data limitations. Effective high-level planner training relies on ample text-motion pairs. However, data for non-human character morphologies is scarce, hindering high-level planner training. Cross-morphology transfer attempts[d] are intricate due to dataset motion diversity, constituting a significant independent contribution.\n\n2. Varied Human Body: Body variation is an interesting topic and there are promising ways to achieve this level of generalization. For example, we can train the diffusion model with SMPL shape conditioning at the high level, and train a shape-invariant controller following [D]. However, as the first work to consider language-guided physical motion generation, our emphasis is on language understanding. We leave it for future study. \n\nWe identify them as crucial future work and will add a discussion in future versions.\n\nAdditional References:\n\n[A] Peng et al. “AMP: Adversarial Motion Priors for Stylized Physics-Based Character Control\n” TOG 2021\n\n[B] Wagener et al. “MoCapAct: A Multi-Task Dataset for Simulated Humanoid Control.”NeurIPS 2022.\n\n[C] Won et al., “A Scalable Approach to Control Diverse Behaviors for Physically Simulated Characters”, SIGGRAPH 2022\n\n[D] Won et al., “Learning Body Shape Variation in Physics-based Characters”, TOG 2019\n"}, {"review_id": "3Y8WYe95TP", "reviewer": "Reviewer_K1zb", "summary": "This work proposes InsActor, a physics-based character control framework that enables controlling agents to follow high-level instructions. It employs a high-level diffusion motion planner and a low-level character controller to achieve mapping between text-based instructions and physics-based motion control. By creating waypoints and desired state transitions, the diffusion motion planner specifies the motion plan that the low-level motion controller follows. The low-level motion controller consists of a pre-trained VAE encoder to translate diffusion state into actionable latents for control.", "questions": "My main concern is the fidelity of the generated motion. Why does the character appear to be floaty and jittery? Is it the learned policy or the simulator settings?\n\n--- \nAfter rebuttal, my main concern about the fidelity of the generated motion returned. However, after discussion with other reviewers and the authors, I believe that the this issue could be a problem of the underlying simulator and should not undermine the overall contribution of the framework. Thus, I raise my score to borderline accept. \n\n---", "limitations": "Limitations are addressed.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- I find the idea of using a higher-level motion diffusion model to specify waypoints and states, and then using a lower-level motion controller to follow them, intuitive and easy to understand. This formulation is flexible and creates the ability to have both high-level control of motion through text and slightly lower-level control through waypoints.\n- The low-level skill discovery module based on diffMimic itself seems like a significant contribution, as it has the potential to be used in many tasks and enables character control. Using a VAE to translate invalid state transitions into actionable latent codes for stable control has great potential in other downstream tasks.\n- Experiments show that the methods outperform the in-house implementation of the state-of-the-art methods PADL and DReCon.", "weaknesses": "- The main weakness I find in this work is its qualitative results. The simulated character is jittery, floaty, and appears to have foot sliding, which is unexpected for a physics-based method. For instance, at the 1:43 mark of the video, InsActor's feet quickly shuffle in a way that should be impossible in a physics simulator. The character seems overall drunk when walking, and there are occasional high-frequency jitters in the root (or camera?), for instance, at 00:48. I am not sure what could have caused this issue: is it the policy? or the setting/fidelity of the Brax simulator?\n- Similar to the previous point, the visual results of the implemented PADL and DReCon are far inferior to their original shown results. The movement is unnatural and jittery. I understand that there is no official implementation provided, but similar scenarios could be recreated in InsActor to create visual comparisons. PADL's generated motion is stable and physically realistic, unlike the ones shown in the provided video. As motion generation is largely visual, the provided quantitative results do not really provide that much insight into the real performance of the method.\n- Visualization should be provided for the high level diffusion planer. Since states are directly generated by the planner, they should be visualized and compared with existing models such as motion diffusion model (MDM). The lower level controller, combined with the latent skill decoder $p_\\psi$, forms an imitator that follows specified states. How well does the diffusion model generates the state and how “invalid” are they?\n- The claim of \"long, unstructured human commands\" is overstated. The tested text instructions are still short, clear, and close to the ones in the dataset.\n- There are many missing details about the performance of the lower-level skill discovery module. If the whole HumanML3D dataset is used for training, how well can the skill embedding encapsulate the motion described in the dataset?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for helpful comments. We would like to address your concern as follows:\n\n> Q1: The main weakness I find in this work is its qualitative results. The simulated character is jittery, floaty, and appears to have foot sliding, which is unexpected for a physics-based method.\n\nOur  results are directly exported from the Brax simulation engine, which conform to the physic-based character animation settings. However, we do observe some artifacts in our qualitative results. They may come from multiple sources:\n\n1. Imperfect motion retargeting. For the main results in the paper and the teaser video, we retarget the simulated motion to a character (a white Y-Bot from Adobe Mixamo) for better visual appearance. However, to preserve the motion content, we copy the rotation angles and add a vertical offset, which can not fully address the difference in bone lengths and introduces additional physics implausibility like foot floating and foot penetration.\n2. Brax simulator smoothens the contact to allow differentiability, which is a known issue [28]. This approximation could lead to possible foot sliding or wrong collision. Brax is relatively new and differentiable physics simulation is still in rapid development, therefore we believe that the issue can be addressed by a better DPS.\n3. Imperfect motion planning and motion tracking policy. Although we are using a state-of-the-art text-to-motion diffusion generator (MotionDiffuse), the generated motion plan can contain jittery and impossible state transitions. For motion tracking, although we used the state-of-the-art motion tracking algorithm (DiffMimic), tracking a very large motion database is non-trivial and could lead to performance degradation. \n\n> Q2:  Similar to the previous point, the visual results of the implemented PADL and DReCon are far inferior to their original shown results. \n\t\nOur task is much more challenging than DReCon and PADL. DReCon tracks 10-minute motion captures, PADL tracks 9-minute captures, while our model uses HumanML3D, a dataset with 28.59 hours of diverse motions. This large dataset presents significant tracking challenges. Since neither PADL nor DReCon have official implementations, to compare fairly, we reproduce them using DiffMimic. Our tracker achieves low pose tracking error on motion datasets for high-quality reproduction. We refer the reviewer to Table 2 in the uploaded PDF file for more details.\n\n> Q3: Visualization should be provided for the high level diffusion planner.  How well does the diffusion model generate the state and how “invalid” are they?\n\nWe have provided some plan visualizations in the supplementary videos 00:45 and 01:54. Quantitatively, Table 2 in the appended PDF shows that our planner achieves strong generation performance. Examples of \"invalid\" state transitions are demonstrated in our supplementary video at 00:45, revealing artifacts like floating, foot sliding, and jittering. These issues even signify in the waypoint heading task as demonstrated in our supplementary video at 01:54. As a result, directly tracking these plans can easily lead to failure, as shown in the alongside DReCon results, whereas InsActor overcomes this with skill embeddings. Notably, physics-plausibility in motion generation, addressed in PhysDiff[A], aligns with our findings as MDM lacks it. Incorporating physics-based optimization enhances MDM's scores on physical plausibility, supporting our observation of infeasible state transitions.\n\n> Q4: The claim of \"long, unstructured human commands\" is overstated. The tested text instructions are still short, clear, and close to the ones in the dataset.\n\nOur claim regarding previous works' inability to handle \"long, unstructured human commands\" refers to longer and more complex language we address. For example, PADL processes sentences like \"jump and swing sword down\" and \"shield charge forward,\" using Multiple Choice Questions to structure input. In contrast, InsActor directly interprets intricate language, such as \"the person raises their left foot up to their knee and then kicks their right foot out, then returns foot to their knee,\" as demonstrated in our supplementary video. Additionally, InsActor comprehends more general human language due to the larger motion database size. While PADL operates on 131 individual clips totaling around 9 minutes, InsActor employs the HumanML3D dataset with 14,616 motions spanning 28.59 hours. \n\nWe acknowledge potential misunderstanding arising from \"long, unstructured human commands\" phrasing and will address this in our next version. Nevertheless, leveraging our extensive text-motion dataset, we anticipate our language understanding capability to improve with more descriptive datasets in the future.\n\n> Q5: There are many missing details about the performance of the lower-level skill discovery module. If the whole HumanML3D dataset is used for training, how well can the skill embedding encapsulate the motion described in the dataset?\n\nIn the general response, we show that the tracking error is low and is on par with previous motion tracking works. In addition, the FID is 0.086 when tracking ground truth motions. We agree that using a single policy network to encapsulate all motion skills on a large motion dataset can be suboptimal. As previous works in motion tracking have shown[B], an MoE ensemble can largely improve the model capacity and better capture dynamic moves like break dancing. However, adopting the system to skill embedding is non-trivial and is out of the scope of this paper that focuses on understanding human language instruction. Nonetheless, we believe that a stronger skill embedding module is an important next step and will be a crucial addition to our framework. \n\nAdditional References:\n\n[A] Yuan et al., “PhysDiff: Physics-Guided Human Motion Diffusion Model”, ICCV 2023\n\n[B] Won et al., “A Scalable Approach to Control Diverse Behaviors for Physically Simulated Characters”, SIGGRAPH 2022\n"}, {"review_id": "DkDJVwQSMg", "reviewer": "Reviewer_s1C4", "summary": "This paper tackles the problem of generating text-conditioned character animation that is physics-based. It proposes a two-stage approach that first generates a kinematic motion conditioned on text using diffusion, and then tracks this motion with a physics-based motion VAE in the learned latent space. Experiments on the common KIT-ML and HumanML3D benchmarks show improved performance over prior work in physics-based motion from text, and show the ability to specify goal waypoints for the motion to hit while following human text prompts.", "questions": "Overall, I think a general physics-based text-to-motion model is an important and novel direction, and the hierarchical approach of diffusion planning with physics-based tracker could be a strong baseline going forward. But I’m mainly concerned that the quality of the output from the high-level diffusion planner is compromising the comparison to DReCon and therefore the need for a latent motion VAE tracker has not been fully justified. I would really like to see how InsActor performs when plugging in a SOTA diffusion model like MDM [33] as the planner. Moreover, an evaluation of standalone tracking performance would make the comparison between the two tracking approaches (latent vs state-based) much more clear. \n\nSome other comments and suggestions that didn’t have an influence on my rating: \n* The related work (Sec 2) is missing relevant physics-based human animation methods and a discussion of why they are difficult to scale up to the general text-to-motion task. E.g. [Peng et al., ASE: Large-Scale Reusable Adversarial Skill Embeddings for Physically Simulated Characters, SIGGRAPH 2022] [Won et al., A Scalable Approach to Control Diverse Behaviors for Physically Simulated Characters, SIGGRAPH 2020], etc..\n* Concurrent work PhysDiff [Yuan et al., arxiv 2022] is an alternative approach to adding physicality to text-to-motion diffusion and could be discussed in future revisions of the paper. Also related, concurrent work Trace and Pace [Rempe et al., CVPR 2023] gives controllability over physics based characters with guidance of a diffusion planner.\n\n===================== After Rebuttal ============================\n\nAfter considering other reviews and discussions with authors and between reviewers, I have decided to slightly raise my score and am leaning towards accept. I think the paper lays out a compelling kinematic diffusion + physics-based tracking idea that can serve as a baseline and inspire improvement in each component of the system including differentiable simulation, motion diffusion, and physics-based tracking. The evaluations show that this hierarchical approach works better than state+action diffusion, and that tracking with a latent model is more robust than direct state-based tracking for planned motions from MotionDiffuse.\n\nHowever, I am still quite concerned about the qualitative results and would really encourage the authors to update the paper text to discuss these qualitative issues such that future work can pursue important directions (e.g., the choice of Brax and differentiable simulation in general rather than RL, and the noisy plans from MotionDiffuse especially in the waypoint setting). It would also be good to clarify why in Table 2 of the rebuttal doc, motion quality (FID) drops significantly from kinematic “Planner” (b) output to full physics-based InsActor (c), indicating that adding physics-based tracking is not necessarily improving motion realism despite it being physically constrained (unlike in PhysDiff). I encourage the authors to show some video results of the planner output vs InsActor on regular text-to-motion (not the waypoint setting) to demonstrate the difference and potential advantages/disadvantages of using the latent tracking technique vs RL and more reliable simulators as in DReCon. \n\n==========================================================", "limitations": "Limitations are sufficiently discussed in Sec 6.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Physics-driven text-to-motion is an important problem and conditioning on free-form text input has not really been tackled in the literature, so the paper is novel in that respect. Diffusion has shown promising results recently, but these are all kinematic.\n\nThe proposed idea of using high-level diffusion followed by physics-based tracking is simple and solid. It would make a good first baseline for future work in this area. \n\nTechnically, InsActor uses a physics-based motion VAE to track kinematic motion which is novel, and it uses differentiable physics to train rather than a learned world model or RL. If this approach really does work to track general motions in the HumanML3D dataset, it would contribute an alternative to recent RL approaches that can be difficult to generalize. \n\nThe proposed method is evaluated on HumanML3D and KIT, which are the most relevant benchmarks for the text-to-motion task. Also, the DReCon baseline in Tables 1 and 2 is an important baseline that uses the alternative state-based approach to tracking.\n\nThe supplementary video and figures are visually pleasing, and I appreciate that the supp video is extensive and shows many results. The shown demo is also cool, and demonstrates fast generation capabilities (relative to other diffusion approaches).", "weaknesses": "To better motivate the need for physics in text-to-motion, there should be a comparison between the proposed InsActor and a state-of-the-art kinematic diffusion model like MDM [33] added to Tables 1 and 2. Currently some numbers in these tables, e.g. FID and diversity, are worse than those reported in MDM, and it’s not clear why that’s the case since the high-level planner in InsActor is very similar. Does this high-level planner perform worse than MDM? Or does the physics-based motion tracking somehow have a large effect on motion quality and diversity? The high-level policy ablation reported in Table 4 takes a step in this direction, but it is trained on rollouts from the motion VAE instead of directly on mocap data as done in MDM and other text-to-motion diffusion models.\n\nLooking at video results at 0:46 and 01:54, I don’t think the high-level diffusion planner is on par with recent models like MDM. Both with and without waypoint guidance there are some significant artifacts like jittering and skating, some of which seem to be affecting the final motion from InsActor (e.g. some noisy popping of limbs and unnatural sliding). I understand this planner is not necessarily the main contribution, but I think poor kinematic motions from the planner undermines the comparison to the target-state tracking policy DReCon, which may perform better when operating on, e.g., outputs from MDM that better reflect realistic motion. Since the low-level motion VAE model for tracking (Sec 4.2) is a key contribution of the work, it’s very important to justify that it is necessary by showing that the DReCon baseline is still inferior when operating on more reasonable kinematic inputs.\n\nThe methods Sec 4 is missing some details that could improve understanding and reproducibility: \n* The tasks states described in L92 are all local, so how is the global root trajectory modeled in the diffusion Sec 4.1? \n* L154: if the pose state is in the local frame, how is inpainting performed to ensure motion meets a global target waypoint?\n* What is the architecture of the diffusion model? Is it using 1D convolutions as in Diffuser or a transformer as in other human motion diffusion models? $\\mu$ and $\\Sigma$ in Eqn 2 are never defined. In general, I’m wondering why not use a SOTA motion diffusion model out-of-the-box for this high-level planning component?\n* L185: is the low-level motion VAE trained directly on outputs of the diffusion model or on mocap data from the dataset? If on mocap, why is the encoder (Eqn 4) expected to produce reasonable results when operating on noisy and unrealistic pose transitions? \n* Similarly, Sec 5.3 shows robustness to perturbations from boxes, but is this kind of perturbation seen in training of the low-level policy too? If not, how does this robustness arise without using RL for training (i.e. without some exploration). \n\nAn evaluation on the low-level tracking component by itself would be very helpful. E.g. reporting tracking errors for the latent policy from InsActor compared to DReCon for both motion-captured and diffusion-generated motions. The current metrics in Tables 1 and 2 were designed for kinematic text-to-motion models, and I would think are mostly influenced by the diffusion planner which is the same for InsActor and DReCon, so a tracking-only evaluation could help parse the difference in performance. There are also open-source RL physics-based trackers that may be worth considering, e.g. [Luo et al., Dynamics-Regulated Kinematic Policy for Egocentric Pose Estimation, NeurIPS 2021].", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed comments. Due to the character limit, we would like to address your major concerns here. For more detailed questions (e.g. tracking details), we are glad to discuss them during the discussion phase.\n\n> Q1:  To better motivate the need for physics in text-to-motion, there should be a comparison between the proposed InsActor and a state-of-the-art kinematic diffusion model like MDM [33] added to Tables 1 and 2. Why FID and diversity are worse than those reported in MDM? The high-level policy ablation reported in Table 4 takes a step in this direction, but it is trained on rollouts from the motion VAE instead of directly on mocap data as done in MDM and other text-to-motion diffusion models.\n\nThis is a great point for us to clarify! Unfortunately, since MDM has a different output domain than our state trajectories, we can not directly compare MDM in Table 1/2. However, we would like to clarify that our high-level planner is adapted from MotionDiffuse[A], a state of the art text-to-motion model that is on par with MDM. Quantitatively, Table 2 in the appended PDF shows that our planner achieves strong generation performance. \n\nAs we report in the general response, there is indeed a gap between the kinematic and physically simulated setting, which demonstrates the challenge of this new task. The gap is led by the fact that the kinematic motion generator is not physically-plausible, which can not be reflected on the previous evaluation metrics like FID. This issue has also been concurrently observed in a recent work PhysDiff[B], which proposed a trajectory optimization approach to fix the generated motion. Different from PhysDiff which is in the kinematic setting, our character moves in a simulated environment and the physical plausibility is more rigorously ensured. In addition, motion generation models tend to produce sequences that appear appealing but are physically implausible. In the process of grounding the state sequence in physics, certain compromises are made on visual aesthetics in order to adhere to physical realities. \n\nThe \"high-level\" ablation study in Table 4 refers to the framework proposed in the Diffuser[13], where a generative model predicts both state sequence and action sequence. This study is to show the importance of our low-level skill embedding, and does not reflect the capability of our high level planner.\n\n> Q2: Looking at video results at 0:46 and 01:54, I don’t think the high-level diffusion planner is on par with recent models like MDM. Since the low-level motion VAE model for tracking (Sec 4.2) is a key contribution of the work, it’s very important to justify that it is necessary by showing that the DReCon baseline is still inferior when operating on more reasonable kinematic inputs.\n\nQualitatively, we do notice that our generated plans have more jittering than motions generated by either MDM or MotionDiffuse. This could be caused by the fact that MotionDiffuse uses temporal smoothing in the visualization but we did not smooth our plans in our visualization. However, we show in Table 1 in the appended PDF that plan smoothing has a minimal effect on the tracking result.\n\nIn terms of the concern that a poor kinematic motion undermines the comparison to DReCon, we would like to highlight that making an executable plan can be very difficult for generative planners. This has also been discussed by a recent work PhysDiff[B], which shows that the physics plausibility generated by MDM has a large room for improvement. Particularly, we observe that the invalid state transition increases when under the waypoint conditioned setting. The mismatch between global motion and locomotion makes it extremely difficult for DReCon-like trackers to track. Note that DReCon has only been verified on a small scale 10-minute motion database with high quality motion data generated by Motion Matching.\n\n\n> Q3: The methods Sec 4 is missing some details that could improve understanding and reproducibility: An evaluation on the low-level tracking component by itself would be very helpful. There are also open-source RL physics-based trackers that may be worth considering, e.g. [Luo et al., Dynamics-Regulated Kinematic Policy for Egocentric Pose Estimation, NeurIPS 2021].\n\nWe thank the reviewer for pointing it out. Due to the character limit, we will address your main concern by providing an evaluation of the tracking module and will add more details regarding the tracking module in future iterations or the discussion period.\n\nAs shown in Figure 1 in the appended PDF, our motion tracker achieves excellent tracking performance with a pose error smaller than 0.05m. We also show in Table 2 in the appended PDF that our implemented DReCon trackers achieves 0.086 FID tracking the test dataset. Although motion tracking is not our main contribution, we highlight that our low-level policy is trained on a very large motion database and achieving the tracking performance is challenging.\n\nThanks for bringing the work by Luo et al. to our attention! However, although the work provides an RL-based motion tracker, its setting is a simplified version of ours. Concretely, in their setting, there is a residual force, i.e., hand of god, which is an external force acted on the character. Although the low-level control can be simplified and the motion quality can be improved with additional external force (hand of god), the generated motion is not technically physically plausible due to the invisible force. On the other hand, in our setting,  all forces are produced by the character itself, which follows the standard physics-based character animation setting and hence is more challenging.\n \nAdditional References:\n\n[A] Zhang et al., “MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model”, ArXiv, 2022. \n\n[B] Yuan et al., “PhysDiff: Physics-Guided Human Motion Diffusion Model”, ICCV, 2023.\n "}, {"review_id": "Kt8jY55wMF", "reviewer": "Reviewer_7eMQ", "summary": "This work proposes InsActor, a framework for instruction-driven character animation. Given human instruction and/or way points, InsActor first leverages a diffusion model to generate state sequences of the character. Next, it uses a skill embedding model to convert the state sequence into physically plausible trajectories with state and actions. ", "questions": "Suggestion on writing:\n- line 92: I believe angular velocity $q$ of a link is not the derivative of rotation, and the symbol $q$ clashes with the notation of joint position $q$. This description needs to be clarified to avoid confusion.", "limitations": "The paper has provided adequate discussion of limitation.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. The paper presents the motivations, setups, and methods quite well. The diffusion model for state sequence generation and conversion from state sequence to physical trajectories are described in sufficient details.\n\n2. The paper provides sufficient experiments and comparisons to show it out-performs the baseline method. The hierarchical design is also justified with ablation studies.", "weaknesses": "1. The works appears to have limited novelty, as it is somewhat a straightforward combination of character motion synthesis with diffusion methods such as [13] and a low-level trajectory optimization framework.\n\n2. This work seems to miss a comparison with a straightforward method for the low-level policy, since the second stage can be formulated with a standard trajectory optimization problem. Since Brax is also a differentiable simulator, trajectory optimization should be easy to set up and in principle, it can be used to obtain an optimal solution in terms of distances between the states generated from the first stage and states from a physically plausible trajectory. I think this work could include results and analysis from trajectory optimization or explain why it is not feasible here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful comments. We would like to address your concern as follows:\n\n> Q1: The work appears to have limited novelty, as it is somewhat a straightforward combination of character motion synthesis with diffusion methods such as [13] and a low-level trajectory optimization framework.\n\nThis is a great point for us to clarify! We would like to highlight that this paper is more focused on studying a scalable framework for the new task of text-to-motion generation, instead of any individual component. We have conducted rigorous evaluation and fair comparison on various different design choices using state-the-art motion planning algorithm (MotionDiffuse) and a state-of-the-art tracking algorithm (DiffMimic) on a largest text-to-motion dataset available to identify the best framework, in which we believe is our core contribution in this work. Also, we believe that our framework will benefit from a better motion generation model and motion tracker in the future.\n\nThere are two main points we would like to highlight of InsActor:\n\n1. Scalability: Building a scalable system to achieve human instructed animation generation is highly non-trivial. Compared with previous works, InsActor understands more general human language in terms of the motion database size, where most related work PADL works on 131 individual clips, for a total of approximately 9 minutes, and InsActor works on HumanML3D dataset with  14,616 motions for a total of 28.59 hours (before filtering out physically impossible motions like sit on a chair). Training a general low-level policy handling various motions is extremely difficult as it may take up to hours to train a low-level policy to perform one single motion, e.g. backflip. InsActor is the first system to work on such a large-scale amount of data.\n\n2. Thoroughness: As the first attempt for large scale physically-aware language guided animation generation, InsActor evaluates and benchmark various alternatives to gain insight to better push the task forward. With extensive experiments, we show that directly outputting actions and state sequences like Diffuser[13] or Decision Diffuser[1] using state of the art generative models may fail in this task, which requires accurate continuous control. Instead, a proper decomposition of the task should be performed in order to derive an efficient solution.  \n\nThus, InsActor is not merely a combination but an important baseline for future work as also acknowledged by reviewer s1C4. InsActor is scalable as the data scale grows and provides insights to the problem of language guided animation generation.\n\n\n> Q2: This work seems to miss a comparison with a straightforward method for the low-level policy, since the second stage can be formulated with a standard trajectory optimization problem. Since Brax is also a differentiable simulator, trajectory optimization should be easy to set up and in principle, it can be used to obtain an optimal solution in terms of distances between the states generated from the first stage and states from a physically plausible trajectory. I think this work could include results and analysis from trajectory optimization or explain why it is not feasible here.\n\t\nWe chose not to use trajectory optimization for several reasons:\n\n1. Computational Efficiency: While employing standard trajectory optimization alongside the differentiable simulator is possible, optimizing trajectories for long motion sequences can be time-consuming. Each generated trajectory necessitates optimization, leading to potential delays. In contrast, the utilization of a trained policy enables fast inference. Given the emphasis on engaging with human users within our work's scope, we chose not to adopt trajectory optimization due to computational efficiency.\n\n2. Robustness: The application of trajectory optimization methods such as [A] typically yields a single trajectory. However, our envisioned use cases for InsActor involve animations and gaming scenarios. In these dynamic environments, robustness is important. Inherent noise in state estimation or random perturbations like pushing in the environment affecting characters are pervasive. Consequently, the efficacy and robustness of the low level policy becomes pivotal in ensuring InsActor's robustness during real-world deployment. Simple trajectory optimization is unable to handle such cases. In addition, we attach the tracking error of the learned low-level policy in the PDF file, which is low and is on par with previous motion tracking works.\n\n3. Flexibility: A learned policy can adapt to various situations and environments. Once trained, a policy can generalize its behavior to new situations that it hasn't explicitly encountered during training. This is important for future development of InsActor like enabling object interaction and accommodating more intricate animations. The current framework of InsActor enables seamless integration of object interaction into the framework with the low level tracking module. \n\nIn summary, the design is mainly due to the consideration of the application scenario and use cases. Nevertheless, we agree with the reviewer that including such comparison would make the experiment results more thorough and we will add them in the final version due to time limit during the rebuttal period.\n\t\nAdditional References:\n\n[A] Gärtner et al., “Differentiable Dynamics for Articulated 3d Human Motion Reconstruction”, CVPR 2022."}]

Author rebuttal:
We thank reviewers for the encouragement and insightful feedback. We are glad that the reviewers found

* the problem “interesting”(R-ndRp) and “important” (R-s1C4); and
* the result “good” (R-Mekg) and “promising”(R-ndRp) and InsActor has the potential to be used in many tasks (R-K1zb); and
* the evaluation informative (R-ndRP, R-Mekg, R-K1zb, R-7eMQ); and
* the hierarchical design of InsActor “clever” (R-ndRP), “intuitive”(R-K1zb), “simple and solid” (R-s1C4).

We execute additional experiments to address reviewers' concerns. We refer the reviewers to the uploaded PDF file for more details. Experiments include:

1. Comparison between our diffusion planner and MDM. We show that our planner is built on a state-of-the-art text-to-motion generator.
2. More ablations on planning and tracking. We further verifies the performance of our planner,  show that there is a motion quality gap between plan and plan tracking, and proves the effectiveness of the tracker in our DReCon baseline.
3. Quantitative performance of low-level control. We show the training curve of our skill mapping module, which converges to a sufficiently low pose error.

We would like to address some common issues as follows:
1. Clarification of high level planner:
    * We adapted the high level planner from MotionDiffuse [40], which is on par with MDM on standard text-to-motion generation benchmarks.
    * Some visual artifacts in plans are caused by not using temporal smoothing for visualization, which has minimal impact on the final results.
    * We emphasize that in the waypoint heading setting, generating an executable plan is particularly more challenging and a compact skill mapping will be necessary.
2. Generalization of the method:
    * Generalization to human-object interaction. We acknowledge the significance of character-object interaction in animation generation. Our approach serves as an important baseline, expandable with additional object/scene modeling. We view human-object interactions as vital future work.
    * Generalization to other morphologies. Effective high-level planner training relies on ample text-motion pairs. However, data for non-human character morphologies is limited, thus, transferring to different morphologies is non trivial.
3. Tracking module details:
    * The tracking error of the low-level policy is low and is on par with previous motion tracking works.
    * Compared with standard trajectory optimization, the design of InsActor is advantageous considering computational efficiency, robustness during deployment, and flexibility for future developments.

While we agree with the importance of some limitations reviewers raised (e.g. lack of object interaction), we hope our rebuttal highlights why addressing these limitations constitute full, separate contributions themselves (e.g.,  requiring more data). We kindly ask the reviewers to let us know if further clarification or information is needed. 


Meta-review: The paper describes a framework for generating physically plausible human character animations from free-form text prompts. The two-stage framework first employs a diffusion model to produce high-level kinematic motion for the animated character and then utilizes a physics-based motion VAE to track this motion in a latent skill space. Experiments on the KIT-ML and HumanML3D benchmarks demonstrate that taking a hierarchical approach achieves performance gains over contemporary baselines (e.g., PADL).

The paper was evaluated by five reviewers who discussed the paper at-length with the authors and among themselves. Several reviewers found the paper to be well written and the proposed framework to be intuitive and clever. They also appreciate the focus on generating motions that are physically realizable as opposed to only considering the kinematics of the motion, which yields sound improvements over the previous state-of-the-art. However, there is also consensus among the reviewers who are concerned about the qualitative nature of the results, which demonstrate notable artifacts in the rendered motion. These concerns are particularly notable given that generating animations that are consistent with physics is a fundamental focus of the paper. There was an extensive discussion about this issue among the reviewers and between the reviewers and authors, who attribute the artifacts to issues with the differentiable Brax simulator and claim that it is secondary to the paper's primary contributions. Neither the reviewers nor the AC found these arguments to be compelling or consistent with the claims made in the paper (and its title). That said, there was agreement that the value of the paper is not necessarily in a usable system for language-enabled character animation, but rather in establishing a baseline that will foster future work in the different components of the system including differentiable simulation, motion diffusion, and physics-based tracking. The authors are strongly encouraged to temper claims about the physical plausibility of the rendered animations and to be clear about the limitations of the system, which go beyond attributing inconsistencies to issues with the chosen simulator.
Decision: Accept (poster)