Video-As-Prompt: Unified Semantic Control for Video Generation
Authors: name: Yuxuan Bian, name: Xin Chen, name: Zenan Li, name: Tiancheng Zhi, name: Shen Sang, name: Linjie Luo, name: Qiang Xu
Organization: ByteDance

Unified, generalizable semantic control in video generation remains a
critical open challenge. Existing methods either introduce artifacts by
enforcing inappropriate pixel-wise priors from structure-based controls, or
rely on non-generalizable, condition-specific finetuning or task-specific
architectures. We introduce Video-As-Prompt (VAP), a new paradigm that reframes
this problem as in-context generation. VAP leverages a reference video as a
direct semantic prompt, guiding a frozen Video Diffusion Transformer (DiT) via
a plug-and-play Mixture-of-Transformers (MoT) expert. This architecture
prevents catastrophic forgetting and is guided by a temporally biased position
embedding that eliminates spurious mapping priors for robust context retrieval.
To power this approach and catalyze future research, we built VAP-Data, the
largest dataset for semantic-controlled video generation with over 100K paired
videos across 100 semantic conditions. As a single unified model, VAP sets a
new state-of-the-art for open-source methods, achieving a 38.7% user preference
rate that rivals leading condition-specific commercial models. VAP's strong
zero-shot generalization and support for various downstream applications mark a
significant advance toward general-purpose, controllable video generation.