InsActor: Instruction-driven Physics-based Characters
Authors: name: Jiawei Ren, name: Mingyuan Zhang, name: Cunjun Yu, name: Xiao Ma, name: Liang Pan, name: Ziwei Liu

Generating animation of physics-based characters with intuitive control has
long been a desirable task with numerous applications. However, generating
physically simulated animations that reflect high-level human instructions
remains a difficult problem due to the complexity of physical environments and
the richness of human language. In this paper, we present InsActor, a
principled generative framework that leverages recent advancements in
diffusion-based human motion models to produce instruction-driven animations of
physics-based characters. Our framework empowers InsActor to capture complex
relationships between high-level human instructions and character motions by
employing diffusion policies for flexibly conditioned motion planning. To
overcome invalid states and infeasible state transitions in planned motions,
InsActor discovers low-level skills and maps plans to latent skill sequences in
a compact latent space. Extensive experiments demonstrate that InsActor
achieves state-of-the-art results on various tasks, including
instruction-driven motion generation and instruction-driven waypoint heading.
Notably, the ability of InsActor to generate physically simulated animations
using high-level human instructions makes it a valuable tool, particularly in
executing long-horizon tasks with a rich set of instructions.