Precise Action-to-Video Generation Through Visual Action Prompts
Authors: name: Yuang Wang, name: Chao Wen, name: Haoyu Guo, name: Sida Peng, name: Minghan Qin, name: Hujun Bao, name: Xiaowei Zhou, name: Ruizhen Hu

We present visual action prompts, a unified action representation for
action-to-video generation of complex high-DoF interactions while maintaining
transferable visual dynamics across domains. Action-driven video generation
faces a precision-generality trade-off: existing methods using text, primitive
actions, or coarse masks offer generality but lack precision, while
agent-centric action signals provide precision at the cost of cross-domain
transferability. To balance action precision and dynamic transferability, we
propose to "render" actions into precise visual prompts as domain-agnostic
representations that preserve both geometric precision and cross-domain
adaptability for complex actions; specifically, we choose visual skeletons for
their generality and accessibility. We propose robust pipelines to construct
skeletons from two interaction-rich data sources - human-object interactions
(HOI) and dexterous robotic manipulation - enabling cross-domain training of
action-driven generative models. By integrating visual skeletons into
pretrained video generation models via lightweight fine-tuning, we enable
precise action control of complex interaction while preserving the learning of
cross-domain dynamics. Experiments on EgoVid, RT-1 and DROID demonstrate the
effectiveness of our proposed approach. Project page:
https://zju3dv.github.io/VAP/.