SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\|\cdot\|$  denotes the mean squared error and  $(\mathbf{s},\mathbf{s}^{\prime})$  is a pair of states before and after transition. The latter term encourages the latent variables to be similar to the prior distribution, ensuring the compactness of the latent space.  $\lambda$  is the weight factor that controls the compactness. During inference, we map the generated state sequence from the diffusion model to the skill space to control the character.
The goal of our experiment is to evaluate the effectiveness and robustness of InsActor in generating physically-simulated and visually-natural character animations based on high-level human instructions. Specifically, we aim to investigate 1) whether InsActor can generate animations that adhere to human instructions while being robust to physical perturbations, 2) whether InsActor can accomplish waypoint heading while being faithful to the language descriptions, and 3) the impact of several design choices, including the hierarchical design and the weight factor for skill space compactness.
Table 1: Quantitative results on the KIT-ML test set. †: with perturbation.
Methods	R Precision↑	Multimodal Dist↓	FID↓	Diversity↑
Top 1	Top 2	Top 3
DReCon [4]	0.243±.000	0.420±.021	0.522±.039	2.310±.097	1.055±.162	4.259±.014
PADL [15]	0.091±.003	0.172±.008	0.242±.015	3.482±.038	3.889±.104	2.940±.031
InsActor (Ours)	0.352±.013	0.550±.010	0.648±.015	1.808±.027	0.786±.055	4.392±.071
DReCon†[4]	0.253±.013	0.384±.006	0.447±.006	2.764±.003	1.973±.100	4.252±.040
PADL†[15]	0.100±.012	0.158±.011	0.217±.015	3.783±.069	4.706±.298	3.168±.065
InsActor (Ours)†	0.323±.013	0.496±.017	0.599±.008	2.147±.061	1.043±.091	4.359±.073
Table 2: Quantitative results on the HumanML3D test set. †: with perturbation.
Methods	R Precision↑	Multimodal Dist↓	FID↓	Diversity↑
Top 1	Top 2	Top 3
DReCon [4]	0.265±.007	0.391±.004	0.470±.001	2.570±.002	1.244±.040	4.070±.062
PADL [15]	0.144±.003	0.227±.012	0.297±.018	3.349±.030	2.162±.022	3.736±.091
InsActor (Ours)	0.331±.000	0.497±.015	0.598±.001	1.971±.004	0.566±.023	4.165±.076
DReCon†[4]	0.233±.000	0.352±.001	0.424±.004	2.850±.002	1.829±.002	4.008±.147
PADL†[15]	0.117±.005	0.192±.003	0.254±.000	3.660±.040	2.964±.115	3.849±.159
InsActor (Ours)†	0.312±.001	0.455±.006	0.546±.003	2.203±.006	0.694±.005	4.212±.154
To implement the experiment, we use Brax [7] to build the environment and design a simulated character based on DeepMimic [25]. The character has 13 links and 34 degrees of freedom, weighs  $45\mathrm{kg}$ , and is  $1.62\mathrm{m}$  tall. Contact is applied to all links with the floor. For details of neural network architecture and training, we refer readers to the supplementary materials.
5.2 Evaluation Protocols
Datasets. We use two large scale text-motion datasets, KIT-ML [26] and HumanML3D [8], for training and evaluation. KIT-ML has 3,911 motion sequences and 6,353 sequence-level language descriptions, HumanML3D provides 44,970 annotations on 14,616 motion sequences. We adopt the original train/test splits in the two datasets.
Metrics. We employ the following evaluation metrics:
1.  $R$  Precision: For every pair of generated sequence and instruction, we randomly pick 31 additional instructions from the test set. Using a trained contrastive model, we then compute the average top-k accuracy.
2. Frechet Inception Distance (FID): We use a pre-trained motion encoder to extract features from both the generated animations and ground truth motion sequences. The FID is then calculated between these two distributions to assess their similarity.
3. Multimodal Distance: With the help of a pre-trained contrastive model, we compute the disparity between the text feature derived from the given instruction and the motion feature from the produced animation. We refer to this as the multimodal distance.
4. Diversity: To gauge diversity, we randomly divide the generated animations for all test texts into pairs. The average joint differences within each pair are then computed as the metric for diversity.
5. Success Rate: For waypoint heading tasks, we compute the Euclidean distance between the final horizontal position of the character pelvis and the target horizontal position. If the distance is less than  $0.5\mathrm{m}$ , we deem it a success. We perform each evaluation three times and report the statistical interval with  $95\%$  confidence.
Table 3: Quantitative results for the waypoint heading task. Evaluated on HumanML3D. We set the start point at (0,0) and the waypoint uniformly sampled from a 6x6 square centered at (0,0). It is considered a successful waypoint heading if the final position is less than  $0.5\mathrm{m}$  away from the waypoint. L: Langauge. W: Waypoint.
Method	L	W	R Precision↑ Top 3	Multimodal Dist↓	FID↓	Diversity↑	Success Rate↑
DReCon [4]	✓	✓	0.178±.000	4.192±.019	8.607±.114	2.583±.157	0.380±.002
InsActor (Ours)	×	✓	0.089±.001	4.106±.001	3.041±.101	3.137±.029	0.935±.002
InsActor (Ours)	✓	×	0.598±.001	1.971±.004	0.566±.023	4.165±.076	0.081±.004
InsActor (Ours)	✓	✓	0.388±.003	2.753±.009	2.527±.015	3.285±.034	0.907±.002
Table 4: Ablation on hierarchical design. Evaluated on the KIT-ML test set.
High	Low	R Precision↑	MultiModal Dist↓	FID↓	Diversity↑
Top 1	Top 2	Top 3
×	✓	0.264±.011	0.398±.016	0.460±.018	2.692±.034	1.501±.095	4.370±.066
✓	×	0.068±.011	0.145±.030	0.188±.024	3.707±.096	1.106±.093	4.148±.098
✓	✓	0.352±.013	0.550±.010	0.648±.015	1.808±.027	0.786±.055	4.392±.071
5.3 Comparative Studies for Instruction-driven Character Animation
Comparison Methods. We compare InsActor with two baseline approaches: 1) DReCon [4]. We adapted the responsive controller framework from DReCon [4]. We use the diffuser as a kinematic controller and train a target-state tracking policy. The baseline can also be viewed as a Decision Diffuser [1] with a long planning horizon, where a diffuser plans the future states and a tracking policy solves the inverse dynamics. 2) PADL [15]: We adapt the language-conditioned control policy in PADL [15], where language instructions are encoded by a pretrained cross-modal text encoder [27] and input to a control policy that directly predict actions. It is also a commonly used learning paradigm in conditional imitation learning [34]. Since the two baselines have no publicly available implementations, we reproduce them and train the policies with DiffMimic [31].
Settings. We utilize two different settings to assess InsActor's robustness. In the first setting, we evaluate the models in a clean, structured environment devoid of any perturbation. In the second setting, we introduce perturbations by spawning a  $2\mathrm{kg}$  box to hit the character every 1 second, thereby evaluating whether the humanoid character can still adhere to human instructions even when the environment changes.
Results. We present the results in Table 1 and Table 2 and qualitative results in Figure 3. Compared to the dataset used in PADL that consists of 131 motion sequences and 256 language captions, our benchmark dataset is two orders larger, where the language-conditioned single-step policy used in PADL has difficulty to scaling up. In particular, the inferior performance in language-motion matching metrics suggests that a single-step policy fails to understand unseen instructions and model the many-to-many instruction-motion relation. Compared to PADL, DReCon shows a better result in language-motion matching thanks to the high-level motion planning. However, unlike Motion Matching used in DReCon that produces high-quality kinematic motions, the diffuser generates invalid states and infeasible state transitions, which fails DReCon's tracking policy and results in a low FID. In comparison, InsActor significantly outperforms the two baselines on all metrics. Moreover, the experiment reveals that environmental perturbations do not significantly impair the performance of InsActor, showcasing InsActor's robustness.
Figure 4: Qualitative results of InsActor with history conditioning. Generation is conditioned on the second human instruction and history motion.
5.4 Instruction-driven Waypoint Heading
Waypoint Heading. Thanks to the flexibility of the diffusion model, InsActor can readily accomplish the waypoint heading task, a common task in physics-based character animation [40, 15]. This task necessitates the simulated character to move toward a target location while complying with human instructions. For instance, a human instruction might be, "walk like a zombie." In this case, the character should navigate toward the target position while mimicking the movements of a zombie.
Guided Diffusion. We accomplish this using guided diffusion. Concretely, we adopt the inpainting strategy in Diffuser [14]. Prior to denoising, we replace the Gaussian noise in the first and last  $25\%$  frames with the noisy states of the character standing at the starting position and target position respectively.
Results. We conduct this experiment with the model trained on HumanML3D. We contrast InsActor with DReCon and two InsActor variants: 1) InsActor without language condition, and 2) InsActor without targeting. Our experimental results demonstrate that InsActor can effectively accomplish the waypoint heading task by leveraging guided diffusion. The model trained on HumanML3D is capable of moving toward the target position while following the given human instructions, as evidenced by a low FID score and high precision. Although adding the targeting position condition to the diffusion process slightly compromises the quality of the generated animation, the outcome is still satisfactory. Moreover, the success rate of reaching the target position is high, underscoring the effectiveness of guided diffusion. Comparing InsActor with its two variants highlights the importance of both the language condition and the targeting in accomplishing the task. Comparing InsActor with DReCon shows the importance of skill mapping, particularly when more infeasible state transitions are introduced by the waypoint guiding. Without skill mapping, DReCon only has a  $38.0\%$  success rate, which drops drastically from the  $90.7\%$  success rate of InsActor.
Multiple Waypoints. Multiple waypoints allow users to interactively instruct the character. We achieve this by autoregressively conditioning the diffusion process to the history motion, where a qualitative result is shown in Figure 4. Concretely, we inpaint the first  $25\%$  with the latest history state sequences. We show qualitative results for multiple-waypoint following in Figure 1 and more in the supplementary materials.
Hierarchical Design. To understand the importance of the hierarchical design in this task, we performed an ablation study on its structure. We compared our approach to two baselines: 1) A policy with only a high-level policy, wherein the diffusion model directly outputs the skills, analogous to the Diffuser approach [14]; 2) A low-level policy that directly predicts single-step skills. We show the results in Table 4. By leveraging skills, the low-level policy improves from PADL but still grapples with comprehending the instructions due to the absence of language understanding. Conversely, without the low-level policy, the skills generated directly by the diffusion model are of poor precision. Although the use of skills safeguards the motions to be natural and score high in FID, the error accumulation deviates the plan from the language description and results in a low R-precision. The experimental results underscore the efficacy of the hierarchical design of InsActor.
Weight Factor. InsActor learns a compact latent space for skill discovery to overcome infeasible plans generated by the diffusion model. We conduct an ablation study on the weight factor,  $\lambda$ , which controls the compactness of the skill space. Our findings suggest that a higher weight factor results in a more compact latent space, however, it also curtails the instruction-motion alignment. Conversely, a lower weight factor permits a greater diversity in motion generation, but it might also lead to less plausible and inconsistent motions. Hence, it is vital to find a balance between these two factors to optimize performance for the specific task at hand.
Figure 5: Ablation study on the weight factor  $\lambda$ . Evaluated on the KIT-ML dataset.
In conclusion, we have introduced InsActor, a principled framework for physics-based character animation generation from human instructions. By utilizing a diffusion model to interpret language instructions into motion plans and mapping them to latent skill vectors, InsActor can generate flexible physics-based animations with various and mixed conditions including waypoints. We hope InsActor would serve as an important baseline for future development of instruction-driven physics-based animation. While InsActor is capable of generating such animations, there are crucial but exciting challenges ahead. One limitation is the computational complexity of the diffusion model, which may pose challenges for scaling up the approach to more complex environments and larger datasets. Additionally, the current version of InsActor assumes access to expert demonstrations for training, which may limit its applicability in real-world scenarios where such data may not be readily available. Furthermore, while InsActor is capable of generating physically-reliable and visually-plausible animations, there is still room for improvement in terms of the quality and diversity of generated animations. There are mainly two aspects for future development, improving the quality of the differentiable physics for more realistic simulations and enhancing the expressiveness and diversity of the diffusion model to generate more complex and creative animations. Besides them, extending InsActor to accommodate different human body shapes and morphologies is also an interesting direction.
From a societal perspective, the application of InsActor may lead to ethical concerns related to how it might be used. For instance, InsActor could be exploited to create deceptive or harmful content. This underscores the importance of using InsActor responsibly.
This research is supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: AISG2-PhD-2021-08-018), NTU NAP, MOE AcRF Tier 2 (T2EP20221-0012), and under the RIE2020 Industry Alignment Fund - Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s).
[1] Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision-making? arXiv preprint arXiv:2211.15657, 2022.
[2] Sadegh Aliakbarian, Fatemeh Sadat Saleh, Mathieu Salzmann, Lars Petersson, and Stephen Gould. A stochastic conditioning scheme for diverse human motion prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5223-5232, 2020.
[39] Xinchen Yan, Akash Rastogi, Ruben Villegas, Kalyan Sunkavalli, Eli Shechtman, Sunil Hadap, Ersin Yumer, and Honglak Lee. Mt-vae: Learning motion transformations to generate multimodal human dynamics. In Proceedings of the European conference on computer vision (ECCV), pages 265-281, 2018.
[40] Heyuan Yao, Zhenhua Song, Baoquan Chen, and Libin Liu. Controlvae: Model-based learning of generative controllers for physics-based characters. 41(6), 2022.
[41] Ye Yuan, Jiaming Song, Umar Iqbal, Arash Vahdat, and Jan Kautz. Physdiff: Physics-guided human motion diffusion model. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
[42] Ye Yuan, Shih-En Wei, Tomas Simon, Kris Kitani, and Jason Saragih. Simpoe: Simulated character control for 3d human pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[43] Hanbo Zhang, Yunfan Lu, Cunjun Yu, David Hsu, Xuguang Lan, and Nanning Zheng. Invigorate: Interactive visual grounding and grasping in clutter. Proceedings of Robotics: Science and Systems, abs/2108.11092, 2021.
[44] Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondiffuse: Text-driven human motion generation with diffusion model. arXiv preprint arXiv:2208.15001, 2022.