Video-As-Prompt: Unified Semantic Control for Video Generation
Authors: Yuxuan Bian
Xin Chen
Zenan Li
Tiancheng Zhi
Shen Sang
Linjie Luo
Qiang Xu
Venue: ICLR 2026 Poster (2026)
Keywords: Video Generation
Controllable Video Generation
Video Dataset
Primary area: generative models
TL;DR: A Unified Semantic-Controlled Video Generation Framework based on Video Prompts and In-Context Control.

Abstract: Unified, generalizable semantic control in video generation remains a critical open challenge. Existing methods either introduce artifacts by enforcing inappropriate pixel-wise priors from structure-based controls, or rely on non-generalizable, condition-specific finetuning or task-specific architectures. We introduce Video-As-Prompt (VAP), a new paradigm that reframes this problem as in-context generation. VAP leverages a reference video as a direct semantic prompt, guiding a frozen Video Diffusion Transformer (DiT) via a plug-and-play Mixture-of-Transformers (MoT) expert. This architecture prevents catastrophic forgetting and is guided by a temporally biased position embedding that eliminates spurious mapping priors for robust context retrieval. To power this approach and catalyze future research, we built VAP-Data, the largest dataset for this task with over 100K paired videos across 100 semantic conditions. As a single unified model, VAP sets a new state-of-the-art for open-source methods, achieving a 38.7\% user preference rate that rivals leading condition-specific commercial models. VAP's strong zero-shot generalization and support for various applications mark a significant advance toward general-purpose, controllable video generation.

Published as a conference paper at ICLR 2026
VIDEO-AS-PROME UNIFIED SEMANTIC CONTROL FOR VIDEO GENERATION
Yuxuan Bian 1.2 Xin Chen Zenan Li", Tiancheng Zhi Shen Sang", Linjie Luo?t, Qiang Xu! 'The Chinese University of Hong Kong (CUHK) "ByteDance Shhezzhnn Loop Area Institute {yxbian, qxu} ese. cuhk edu.hk hk fchris. chenl, linjie. Luo)@bytedance. edance. com
Concept-Guided Video Generation (Transformation)
Imoge
Video
Generated Video
hum Ladudu...
Condept-Guided Video Generation (Interaction)
paper man....
Style-Guided Video Generation
becomes Ghibli style...
Motion-Guided Video Generation (Non-Human)
expann like balloon
Mation-Guided Video Generation (Human)
..akkke dance...
Camera-Guided Video Generation
Hitchcock comern zogm
Zero-Shot Performance with Unseen Semantic-Control
crumble info clods...
Figure 1: Video-As-Prompt (VAP) is unified semantic-controlled video generation frame- work: it treats reference videos with wanted semantics video prompts and controls generation via a plug-and-play in-context Mixture-of- Trans-oorrrfsffrerrrss expert. Row 1-6: reference videos used as prompts for diverse semantic- contttic-oottoleed tasks (concept, style, motion, camera). Row 7: zero-shot results from VAP when given an unseen semantic, demonstrating strong generalization. We strongly encourage readers to view our project page for better visualization.
ABSTRACT
Unified, generalizable semantic control in video generation remains a critical open challenge. Existing methods either introduce artifacts by enforcing inappropri- ate pixel-wise priors from structure- based controls, or rely on non-generalizable, condition-speeific finetuning task-specific architectures. We introduce Video- As-Prompt (VAP), new paradigm that reframes this problem as in-context gen- eration. VAP leverages a reference video as direct semantic prompt, guid- ing a frozen Video Diffusion Transformer (DiT) via a plug-and and-play Mixture- Tranrsssfores (MoT) expert. This architecture prevents catastrophie forgetting and is guided by temporally biased position embedding that eliminates spurious mapping priors for robust context retrieval. To power this approach and catalyze future research, we built VAP-Data. the largest dataset for this task with over
This work done when Yuxuan Bian intern ByteDance. Project Lead.
OOr project page is wtttcpp::/httteednc...tiiiit wtw.ttuu....vvieoooooooooooo Prompt/

---

Published as a conference paper at ICLR 2026
100K paired videos across 100 semantic conditions. As single unified model, VAP sets a new state-of state-of-the-art for open-source methods, achieving a 38.7% % user preference rate that rivals leading condition- strong zero-shot generalization and support for various applications mark a signif- icant advance toward ceneera-ppuposse controllable video generation.
commercial models. VAP's
INTRODUCTION
While unified struccurre-controlled video generation (Jiang et al. 2025) under pixel-aligned condi-
tions (e.g., depth (Gao al. 2025a), pose (Hu 2024), mask (Bian al. 2025b). optical flow (Tin)
et al. 2025) is well studied, semantic-controlees generation Enneration-lackiing a pixel-aligned condition
(e.g., concept (Liu al., 2025). style (Ye al. 2025), motion (Zhang et al. 2025b). camera (Bai
et al. 2025aa) to the target video-remain fragmented without unified and generalizable frame- work, limiting applications in visual effects (Macklin et al. 2014)) video stylization (Jamriška al.
2019). motion imitation (Wang al. (025), and camera control (Bai ct al. 2025b).
Migrating current unified struccurr-coontrolled methods (Jiang et al., 2025 Zhang et al. 2023) of- ten causes artifacts because they enforce inappropriate pixel-wise mapping priors from structure- based control abilities (see Fig. 2
(a)). Other semantic-controlled methods fall into two groups:
(1) Condition-Specifie Overfit (see Fig. _ (b)): methods (Liu et al.
2025 Civitai 2025 finetune backbones (Yang et al. [0224] Wan et al. 2025) or LoRAs (Hu et al.
2022) for each semantic con-
dition (e-g., Ghibli style, Hitchcock camera zoom), which is costly; (2) Task-Specific Design (see
Fig. 2 (c)): methods (Ye et al.) 2025 Bai et al. 2025a Zhang et al. 20255) craft task-specific mod- ules or inference strategies for condition type (e.g.. style, camera), often encoding videos with the same semantics to specially designed space and guiding generation. While effective, these condition/task specific approaches hinder unified model and limit zero-shot generalizability.
However, recent image generation (Tan et al. 2025 and dtructue--cotrolle video generation Ju al.) 2025) show that Diffusion Transformers (DiTs) support strong in-context control, motivating unified framework for in-context semantic-controlled video generation. As shown in Fig. 2 (d), rather than assuming pixel-wise correspondence (Jiang et al.] 2025), training per-condition mod- els (Liu et al. 2025) using task-specific designs (Ye et al. (025). we treat the video of the wanted semantics as a reference video prompt and guide generation via in-context control. This formula- tion removes the inappropriate pixel-wise mapping prior from structure-basse controls, avoids per- condition training or per-task model designs, and enables a single unified model to handle diverse semantic controls and generalize in zero-shot manner unseen semantics (see Fig,
We present Video-As- AssProtptt (VAP), the first unified framework for samantic-controllee vidco gen- eration under non-pixel-aligned conditions, by treating reference video with the wanted semantics as a video prompt and using plug-and-play in-context control. As shown in Fig. (d), VAP adopts a plug-and-play
Dixture-of-Traasfoomme (MoTs) design (Liang et al., 2024) to augment any frozen Video Diffusion Transformer (Peebles & Xie (023) with a trainable parallel expert for interpret- ing the video prompt and guiding the generation, preventing catastrophic forgetting and enabling in-context control. The expert (for the reference prompt) and the frozen backbone (for target gen- eration) run independent feed-forward and layer-norm paths and communicate via full attention for synchronous layer-wise reference guidance. For robust context retrieval, we adopt temporally bi- ased Rotary Position Embedding (RoPE) that places the reference before the current video along the temporal axis while keeping spatial fixed; this removes the nonexistent mixel-mapping prior from shared RoPE, matches the temporal order expected by in-context generation, and preserves spatial consistency the model can exploit spatial semantic changes of the reference video prompt.
Existing datasets (Jiang et al., 2025 Ju et al., 2025) lack focus on semantic-controlle video gener- ation. We introduce VAP-Data, the largest dataset date, with over 100K curated samples across 100 semantic conditions, providing a robust data foundation for unified semantic-controlled video generation. Extensive experiments show that VAP, unified model for diverse semantic conditions (S) and downstream generation tasks (Sec, B). produces coherent, semantically aligned videos, achieves a 38.7% user preference rate competitive with leading closed-source commercial models, surpasses condition-specifee methods, and exhibits zero-shot generalization (Fig.
2

---

Published as a conference paper at ICLR 2026
concfurr-Controlles. Video Generation
Semantic-Controlle Video Generation
Atel-Aligeed Condition
Reference Image
Genersted
Reference
Reference Imrge
Generched
(depth, optical Rlo
Video
Video
Video
Pori-Aigred Condition
Reference Image
Nose
Reference Smage
Noiee Reference Referesce Image
Nobe
Referesce Video
Reference brage
Noles
Video
Loyer
Loyer
Laper
Layer
Expert Leyer 1269)
In-Cathext
Artertian
DT Leyer Xarv)
Speckic Madale
10-
10-
(a) Residual Addition
(b) Concilion-Spccfiic Overfit
(c) Tesk-Spacific Design
(4) Videg-As-PPromp (ours)
Figure 2: Controllable Video Generation Paradigms. Structure-Controlled Video Generation (a). The condition is pixel-aligned with the target video. Most works inject conditions (e.g. depth, optical flow) into a DiT via an extra branch using (a) residual addition, leveraging pixel-wise align- ment. Semantic-Controlled Video Generation (b, c, d). The condition and target video share the same semantics. Most works use (b) Condition-ppecffic Overfi or (c) Task-Specific Design: fine- tuning per semantic or adding task-specific modules. (d) Video-as-Prooe arooptt::: We use a reference video with the same semantics prompts and adopt plug-and-plas in-context control framework built on mixture-of miixure-offtransformer traansffrorrrs to achieve unified semantic-controlled video generation.
Our contributions are highlighted follows:
We present VAP, a unified semantic-controlled video generation paradigm, treating reference video with the wanted semantics video prompt for generalizable in-context control.
We propose a plug-and-play in-context video generation framework built on the mixture-of- transformers architecture that prevents catastrophic forgetting, supports various downstream tasks, and delivers strong zero-shot generalization to unseen semantic condition categories.
We construct and release VAP-Dala, the largest dataset for semantic-controlled video generation, with over 100K curated paired samples across 100 condition categories.
RELATED WORKS
2.1
VIDEO GENERATION
Video generation has progressed from early GAN-based models Vondrick et al. 2016 Skorokhodov
al. 2022) modern diffusion models (Brooks et al.) [024]] Gao et al. (025b)) Leveraging the
scalability of diffusion transformers (DiTs) (Peebles & Xiel 2023), research has moved from convo-
lutional architectures (Singer et al. 2022 Blattmann et al. 2023 Girdhar et al. 2023 Chen et al.
2024b Zhang et al.] 2024) to vransformer-based ones (Gupta et al. [023; Menapace et al. 2024
Brooks et al. 2024 Polyak et al. 2024 Kong et al.
2024] Wan et al. 2025] Gao et al. 2025b)
The standard pipeline encodes Gaussian noise into latent space with a VAE (Kingma & Welling
2013). splits the latents into patches, processes the patches with a DiT, and decodes to pixel space to
produce high-quality, smooth videos. However, pre-trained DiT's typically support only text prompts or fist/llat-frrme control (Wan et al. (2025] Gao et al. (2025b). To enable finer, user-defined control,
many methods add task-specific modules pre-trained DiTs (Jiang al. 2025 Bian al. 2025b)
or design special inference (Zhang et al. 2025b; Wang et al. 2025 for new controllable video tasks.
2.2 CONTROLLABLE VIDEO GENERATION
In general, controllable video generation can be categorized into Structure-Controllee Video Gen- eration and Semantic- Semantic-Controllee Controlled Video Generation (see top of Fig. 2). The former (Jiang et al. 2025 Bian al. 2025b) is driven by pixel-aligned conditions (e.g., depth, pose, mask. optical flow). while the latter (Zhang et al., 2025b] Ye et al. (025) focuses on generation based on semantic conditions without pixel mapping prior (e.g., concept, style, motion, camera).
3

---

Published as a conference paper at ICLR 2026
Concept
Style
Camera
Motion
Ladudu-Ma
Chibi
Hitchtock
Squah
Minecroft
Camic
Earth Zoom Out
Expansion
Enfily Transformation
Non-Human Motion
Enhity Inheraction
Human Motion
Couple Arrival
Simpsons Comic
Zoom In
Split Dance
Liqui
Megic
Left
(a) Overview Semantic Conditions (100) Four
Categories VAP-Data
cloning
wittermeoon
aade
tal
dizzydizzzi ZZydizzy ZYS
bloge
pegic
hitche Bek Zoom ceitinn
zoon
ja
wan.an.cmm comic
Bunch
EArE ZOOB out
expans
sin
simps ons
comi C
Ao oo oomm Loom S soul depa
D binhhaan hove we
rasutoya
move an iggtt yow Kinzs
Tuzzyfuzzy
move
iacnasa
Zongzz
covér ed liquid mal Tongei
move
left
Overview of Reference Images VAP-Data
Word Cloud of Semantic Conditions
Figure 3: Overview of Our Proposed VAP-Data. (a) 100 semantic conditions across 4 categories: concept, style, camera, and motion; (b) diverse reference images, including animals, humans, ob- jects, and scenes, with multiple variants; and (c) a word cloud of the semantic conditions.
Strueture-Controllee Video Generation In structure-controlled video generation, condition videos (e.g., depth, pose) are typically pixel-aligned with the target videos, so control signals are mostly modeled with an additional adapter/branch and injected via residual addition to exploit this mapping prior (Zhang et al.) [2023] Mou et al.] [024] Bian al.] [2025a] Lin al., (004)). as shown
in Fig. 2 (a). Common conditions include Trajectory (Wang et al. 2024| Zhou et al. 2024|] Guo
al.
2025a Gu et al. 2025). Pose (Ju et al.]
[023 Lei et al. 2024 Hu 2024), Depth (Esser al., Gao et al. 2025a). Optical flow (Jin et al. [025 Koroglu et al. (025), and Mask (Bian et al.,
2023
2025b
Yang et 20.5)) Recent works (Ju et al. [2025] Jiang et al.] 2025) further enable all-in-one
structure-controlleed generation by treating these inputs as a unified, pixel sxxl-nl spatial condition.
Controll-d Video Generation Semantic-controlle video generation handles condi- tions which lack pixel-wise correspondence with target videos (see Fig. []. including Concept (Liu
ct al.
2025 Hsu al., 2024 Pika 2025) Pix Verse 2025 Vidu 2025 Kling 2025) (e.g., turning an
object into Ladudu taking it like paper man), Stylization (Huang al. 2025 Ye et al. 2025
Camera Movement (He et al. [2024, Bai et al.) 2025alb) b), and Motion (Zhao et al. 2024 Yatim
ltt al.
2024 Pondaven et al. (0255) Zhang et al. 2025b] Wang et al. 2025), where the reference and
target share motion but differ in layout or skeleton. As shown in Fig. (b) and (c), prior methods fall into Condition-Specific Overfit (Liu et al. 2025 Civitai 2025). which fine-tune DiT back- bones or LoRAs for each semantic condition; and Task- Pask-Speciffe Design (Ye et al. 2025 Bai et al.)
2025a Zhang et al. 2025b) Wang et al. 2025), which add task-specific modules inference strate- gies for a class of semantic conditions (e.g., style, motion, camera). Above approaches fit narrow distributions but are not unified and generalizable; they require per-condition retraining or per-task designs and lack zero-shot generalization. A concurrent work (Mao et al. (0255) adopts LoRA A mixture-of-experts for unified generation across multiple semantic conditions, but it still learns each condition by overfitting subsets of parameters and fails to generalize to unseen ones. This raises key question: How can we build a unified semantic-controlled video generation framework?
Inspired by in-context learning (Huang al. 2024 Bian et al. 2024 Tan et al. 2025 Guo et al. 20256 Ju et al. 2025). we propose Video-As As As-Prompt (VAP), which treats videos with the wanted semantics as unified in-context prompts to guide generation. By casting the task an in-context gen- eration with reference video prompts, VAP. to our knowledge, is the first to unify multiple semantic- controlled tasks without task-specific designs, while achieving strong zero-shot abilities.
4

---

Published as a conference paper at ICLR 2026
Reference
Torget Caption
Nolay target
Tokenization
Caption
Tekens
First Frome
Takans
lakers
First Frome
fpkees
Menccalf.
els.
uoder
Video Caption
VAE Fnceder
Nolte
Text Enceder
Tokenization
RoPE (blased)
In-Context Expert
Any Pretrained
Video DiT
Caption
Video
Nokey Target Video
Temporally Binsde RoPE
FAN [Nom)
QXV
In-Context Full Attention
FFN [Nom)
QKV
Mixture-of-Tranete of-Transformer Blocks
Haght
Referenc
Schen
Yarget Vidas
...
Nf layers
... layers
Time
QAV
In-Context Atfention
GXV
Decoder
Text Tabara
FFN (Morm]
(Nom)
Figure 4: Overview of Video-As-Prompt. The reference video (with the wanted seman- ties), target video, and their first frames (reference images) are encoded into latents by the VAE and, together with captions (see top right), form an in-context token sequence Reefte fiidoo Tartert, Tar Tidoo (see middle). First frame tokens are concatenated with video tokens. We add temporal bias A to RoPE to avoid nonexistent pixel-aligned priors from the origi- nal shared RoPE (see bottom right). The reference video and captions act as the prompts and are fed into trainable DiT Expert Transformer (see left), which exchanges information bidirectionally with the pre-trained DiT via full attention at each layer, enabling plug-and sae--add-plyy in-context generation.
METHODS
VAP supports unified semantic-controller video generation under various semantic conditions (e.g., concept, style, motion, and camera). Our insight is to use videos with the wanted semantics as uni- fied prompts to guide generation across tasks, avoiding per-condition fine-tuning or per-task designs. Although we study limited set of conditions, the method extends to others without major structural changes and shows promising generalizability for different semantic conditions (see Sec. A), various downstream tasks (see Sec. B) and unseen semantics in training (see Fig.
3.1 PRELIMINARY
Video diffusion models (Brooks al. 2024 Gao et al. 2025b) learn the conditional distribution p(x C') of video x given conditions C. Using Flow Matching (Lipman et al. (022) for illustration, noise sample Xo _N, 1) is denoised to X1 along the path X, (xg
-minn X
where Tmnn 10 and [0, 1]. The model u trained predict the velocity V, dx. which
simplifies to: Vi = dxs X1 (1 Jmin )xo. We optimize u with parameters + by minimizing
dt
the mean squared error loss L between the ground-truth velocity and the model prediction:
Ex C [lue (Xt, t, C) (x1 - -m X
During inference, the model first samples Gaussian noise Xo ~ (O001) and then uses an ODE solver with discrete of N denoising timesteps produce X1-
3.2 REFERENCE VIDEOS AS
PROMPTS
Semantic-controlledd video generation spans diverse condition types (e.g., concept, style, motion, camera). dineture-conttolle
methods assume pixel-wise alignment between condition and tar- get (Zhang et al. 2023) Jiang et al. (025); injecting a semantic- same but pixel-misaligned video con- dition via residual addition yields copy-and-paste artifacts (see Fig. (a)). Prior semantic-controlled video generation works partially tackle this by using per-condition fine-tuning or per-task designs, treating tasks in isolation. In contrast, VAP employs reference videos video prompts, which share the same semantics as the targets, independent of task category. unifying heterogeneous conditions in one model. Formally, let U'_1 C₁ denote condition types with conditions C, (total m); prior methods often fine-tune n (per-task) or up to m (per-condition) models, whereas we train single unified model ue that jointly learns p(x c) for any C. We evaluate four represen-
5

---

Published as a conference paper at ICLR 2026
Reference Video
Residaal
Concal Finhune
Reference Image
Shared RoPt
Ours
Caphion
-A young woran compleiees
siggl, continuuus 360- degree spin place..
Figure Motivation. Ablation visualizations (Semantic: Spin 360") on structure designs of VAP.
tative types-concepe concept (Co,), style (Cs), motion (Cm), and camera (Cea) for distinct task definitions and data. Our VAP-Data follows this taxonomy, overviews appear in Fig. 3
Concept-Guided Generation: Videos sharing concept, such as entity transformation (e.g..
person becomes Ladudu doll) or interaction (e.g., an Al lover approaches the target).
Style-Guided Generation: Videos in reference style (e.g., Ghibli, Minecraft).
Motion-Guided Generation: Videos following reference motion, including non-human motion (e.g. objects expand like balloons) and human motion (e.g., shake it dance).
Camera-Guided Generation: Videos following reference camera motion, from basic translations (up, down, left, right, zoom in/out) to the Hitchcock dolly zoom.
We also input captions (Preft Ptar)) of the reference video and target video aid in finding and transferring the shared mentioned semantic control signals (e.g., "cover liquid metal" in Fig. 6
Discussion.
Thus uh learns conditional distribution p(x Cen C. C.n: Coa: Pref, (tef:Prrr) Ptar).
3.3
PLUG-AND-PLAY IN-CONTEXT CONTROL
Our model takes four primary inputs: a reference video (providing the semantics), a reference im- aggB (providing the initial appearance and subject), captions (helping find the target semantic), and noise (inference) or noisy target video (training). We first encode the reference video c E Raxhxuxe
and the target video X E IR" x [Bnxxxxxx wxxx into latents è E Rn xk' xw" xd and x E IR' xh' xw' xd by VAE. original temporal/spatial sizes; n', h', w/ are latent sizes. With ne text tokens
Here and h ta. ty E RR xd naive baseline to finetune the DiT on the concatenated sequence [ta, 2, t,. x following in-context structure-controlleed generation (Ju et al. 2025). This often leads to catastrophic forgetting with limited data (Fig. (b). Tab. 2 because (1) DiT's I''s are pre-trained only for generation, in-context conditioning, and (2) our reference/target pairs lack pixel-aligned priors, making se- mantic in-context generation much harder. To stabilize training, we adopt Mixture-of Transformers (MoT) (Liang et al., 2024): a frozen Video Diffusion Transformer (DiT) plus trainable parallel expert transformer initialized from the backbone. The expert consumes [ta, c). while the frozen DiT processes [/,,, x] (see Fig. 4). Each keeps its own query, key, value projections, feed-forward layers, and norms; at each layer, concatenate Q/K/V and run full attention for two-way information fu- sion. This shapes references into prompts conditioned on the current generation and routes guidance into the frozen DiT. With MoT, preserve the backbone's 's generation ability, boost the training stability, and achieve plug-and and play in-context control independent of DiT architecture (see Tab.
3.4 TEMPORALLYY BIASED ROTARY POSITION EMBEDDING
Similar to observations on Rotary Position Embedding (RoPE) (S et al. 2024) in in-context image generation (Tan al. 2025). find that sharing position embedding between the reference condi- tion and the target video is suboptimal: it imposes false pixel-level spatiotemporal mapping prior, making the model assume a nonexistent mapping between the reference and the target videos, and perform unsatisfactorily (see artifacts in Fig. 5 (c)). Accordingly, we shift the reference prompt's temporal indices by fixed offset A, placing them before all noisy video tokens while keeping spa- tial indices unchanged (see right bottom of Fig. 4. This removes the spurious prior, matches the temporal order expected by in-context generation, and leads to improved performance (see Tab. 2
"hhe first frame of the reference vidco is also injected for inheriting the Image-to- Video backhome ability. Without loss of generality, we assume and video are jointly modeled with full attention.

---

Published as a conference paper at ICLR 2026
Table 1: Qualitative Comparison. We compare against the SOTA structurr-conntrllled generation method VACE (Jiang et al., 2025), the base video DiT model CogVideoX-12V (Yang et al.) 2024), the condition-specife variant Cog VideoX-122 (LoRA) (Hu et al. and the closed-source com- mercial models Kling/Vidu (Kling) 2025 Vidu 2025). Overall, VAP delivers performance compa- rable to the closed-source models and, on average, surpasses the other open-source baselines, as a unified and generalizable model. Red stands for the best, Blue stands for the second best.
Metries
Text
Overall Quality
Semantic
User Study
Model
Score?
Motion Smoothness]
Dynamic Degree/ Aesthetic Quality"
Aligument Score"
Preference Rate (%)
Structure- Controllel Methods
VACE (Original)
5.88
97.60
68.75
53.90
35.38
0.6%
VACE (Depth)
22.64
97.65
75,00
56.03
43.35
0.7%
VACE (Optical Flow)
22.65
97.56
79.17
57.34
46.71
1.8%
Backbone and Coudition-Secilin Methods
Cog VideooX112
22.82
98.48
72.92
56.75
26.01
6.9%
Cog VideeeX1- (LoRA))
23.59
98.34
70.83
54.23
68.60
13.1%
Kling/ Vidu
79.17
59.16
74.02
Ours
Video-As-Prompt
24.13
98.59
77.08
$7.71
70.44
38.7%
We fine-ture LoRA Cog ViV for each semantic condition in the benchmork and report avenge performance.
: Kling and provide dedicated interfaces each semantic condition; We report the proforence agarepating wins comparisons. Each average human preferences received correpponding method.
them specific.
4
EXPERIMENTS
4.1
IMPLEMENTATION DETAILS
We train VAP on CogVideoX to evaluate effectiveness across DiT architectures For faimesss match parameter counts: Cog SigVidoo--22-551 it is a distributed copy spanning 4 of layers; both are about 5B parameters. Following pre-trained DiTs, we resize videos to x 720 (832) and sample 49 frames at 16 fps. We use A with learning rate x 10-5 and train for 200 steps on 48 NVIDIA A100s (Details are in Sec. C.I.
Yang et al. 2024 and Wan2.1-12V-144 (Wan et al. 2025)
the in-context DiT expert is a full copy of the original; on 1-12.1-11V-144
4.2 METRICS
We evaluate 5 metrics across three aspects: text alignment, video quality, and semantic alignment. Following prior work (Liu al., 2025| Diang et al., (2025). we measure text alignment with CLIP similarity (Radford et al.] 1 and assess video quality using motion smoothness (Radford et al.) 2021], dynamic degree (Teed & Deng, 2020), and aesthetic quality (Schuhmann et al. (022). We also introduce a semantic-alignmenn
score that measures consistency between the reference and gen- crated videos; we submit each vidco pair and detailed evaluation rules to Gemini-2.5-pr (Comanici
ct al. 2025) for automatic scoring (Details are in Sec.
4.3 DATASET
Semantic-controlle video generation requires paired reference and target videos sharing the same con-pixel-alignee semantic controls (e.g., concept, style, motion, camera). Prior work mostly lies on a few manually collected videos tailored to specific semantic conditions (Liu et al., 2025), limiting the emergence of unified models. To address this, collect 2K high-quality reference im- ages from Pexels (Pexels 2025) spanning men, women, children, animals, objects, landscapes, and multi-subject cases. We then use Image-to-Video visual-effects templates from commercial mod- els (VIDU (Vidu 2025) and Kling (Kling 2025)) and community LoRAs (Civital 2025) to create paired videos by matching each image all compatible templates (some restrict subject categories). Overall, we obtain VAP-Data, a semantic- controllee paired dataset with over 100K samples across 100 semantic conditions- the largest resource (see Sec. 3.2 and Fig. 3). For evaluation, we evenly sampled 24 semantic conditions from 4 categories (concept, style, motion, camera) in the test subset, with 2 samples each. Detailed information and limitations are in Sec.
"As Wan2.1 is more resource-intensive, results are reported Cog VidcoX unless otherwise noted.
7

---

Published as a conference paper at ICLR 2026
(onggea0
LOngina
VICE Shephi
vIce looph
VACE
Vlaw)
Cagrisen-LLAA
Capitbak-
King/Vdd
Hald
VAP Jouns
Styte
Figure 6: Qualitative comparison with VACE (Jiang et al. 2025), Cog VideoX (12V) Yang et al., 2024 Cog Vog (I2V) and commercial models (K KKling, 2025 Vidu 2025); VACH(*) uses
a form condition (top left). More visualizations are in See. A and supplementary.
4.4 COMPARISON WITH PREVIOUS METHODS
We evaluate VAP against: (1) the state-of-the-art (SOTA) structure-controllee video generation method VACE (Jiang al. 2025) under multiple structure conditions (e.g., original reference video,
depth, optical flow); (2) condition specific methods, where we train a LoRA (Hu et al.) (022) for
each semantic condition specific models (Bai et al., 2025a Ye et al., and report averaged performance; (3) state-of-
common community practice often reported to match or surpass task-
the-art closed-source commercial models, including Kling (Kling 2025 and Vidu (Vidu 2025).
Quantitative Comparison. For the SOTA structure-contrroleed method VACE (Jiagg et al. 2025
the model conditions on a video and same-size mask indicating edit (1) vs. fixed (0) regions. Fol- lowing VACE, we use the reference video, its depth, and its optical flow as video conditions, setting mask to I so the model follows rather than copies them. Overall, VACE performs worst, as expected when structure-controllled methods applied directly semantic-controllet generation. This is because VACE assumes pixel-wise mapping between the condition and the output (e.g., video and its depth), which breaks under semantic control and copies unwanted appearance layout from the reference. As control moves from raw video, depth to optical flow, appearance detail decreases, and metrics improve, confirming that the pixel-wise prior ill-suited for semantic- controlled generation. Driving a pre-trained DiT (Cog VideoX-I2V) with captions carrying semantic cues yields decent video quality but weak semantic alignment, since many semantics are hard to express with coarse text. Common LoRA fine-tuning often obtains strong semantic alignment by overfitting specific condition: it harms base quality (vs. the Cog VideoX-12V row), needs a sepa- rate model per condition, and fails to generalize to unseen references. By contrast, VAP outperforms open-source baselines on most metrics, achieves performance comparable to commercial models, and, for the first time, provides unified model for semantic-controlle video generation.
User Study We conducted a user study with 20 randomly selected video-generation researchers to evaluate video quality and semantic alignment. In each trial, raters compared different method outputs shown with a semantic-contre reference video and chose the better result for (i) semantic alignment and (ii) overall quality. We report the preference rate- the normalized share of selections across all comparisons, totaling 100% Tab. [ VAP and Kling/Vidu (commercial, closed-source, task-specific) achieve the overall highest preference rate, while VAP works unified model.
8

---

Published as a conference paper at ICLR 2026
Reference Video
Reference Image
Generated Video
Crumble
Levilate
Malt
Figure 7: Zero-Shot Performance. Given semantic conditions unseen in VAP-Data (left column), VAP still transfers the abstract semantic pattern to the reference image in zero-shot manner.
Qualitative Comparison In Fig. VAP yields better temporal coherence, visual quality, and se- mantic consistency than structure controlled baselines (Jiang et al.) 2025). DiT backbones, and condition -peecff finetuning (Hu et al. (2022), and matches condition-specific commercial models Kling (Kling 2025) and Vidu (Vidu 2025). VACE's pixel-mapping bias treats the semantic refer- ence video as pixel-aligned, causing appearance/layout copying (e.g., the frog stands like the dog; the Statue of Liberty imitates a sheep); this artifact weakens when the reference is replaced by depth and then optical flow, which progressively remove appearance details. LoRA fine-tuning improves semantic alignment without copy artifacts but requires a separate model per condition and lacks zero-shot generalization. In contrast, VAP uses single model that treats all semantic conditions as a unified reference-video prompt, enabling unified semantic-controlle generation.
Zero-Shot Generation By treating all semantic conditions unified video prompts, VAP sup- ports diverse semantic-controlled generation tasks; moreover, when given an unseen semantic refer- ence (Liu et al.) 2025) that doesn't 't belong VAP-Data (see Fig. 77. the in-context ability learned from video-as- video-as-prompt data enables VAP to perform zero-shot generation guided by new references.
4.5 ABLATION STUDY
In-Context Generation Structure. We train 4 VAP variants to test the effectiveness of our mixture- of-transforrers (MoTs) adoption: A1. Single-Branch Finetuning ug: expand pre-trained DiT input sequence to [Refteat Refvideo: Tartezt: Tarvideol and finetune the full model; A2. Single- Branch LoRA Finetuning u same as Al but freeze the backbone and train only the LoRA layers; A3. Unidirectional Cross-Attn uuc: freeze the pre-trained DIT, add new branch with the same weights, and inject its features via layer-wise cross-attention; and A4. Unidirectional Addition ug same A3 but inject features via residual addition. We evaluate the same benchmark of VAP-Data. Results in Tab. show: A1. MoT boosts performance by preserving the base DiT's gen- erative ability, solving the catastrophic forgetting while enabling plug-and-play in-context control. A2. LoRA helps retain the backbone's 's ability, but its limited capacity struggles with complex in- context generation, yielding suboptimal results. A3. Layer-wise bidirectional information exchange in MoT lets the reference video- prompt representation adapt synchronously to the target tokens, improving semantic alignment. A4. Even with retraining, txxel-topiixel mapping, mismatching cemantic-controll generation and degrading performance.
methods rely on rigid
Position Embedding Designs. To validate the effectiveness of our temporally-bing biased RoPE, we evaluate two variants. (1) u/g: applying identical RoPE to both the reference and target videos, which enforces an unrealistic pixel-wise alignment prior and leads to degraded performance; (2)u in addition to introducing temporal bias A, following in-context image generation (Tan et al.) 2025), we add a width bias by placing the reference video to the left of the target video. Experiments show that this increases the difficulty of spatial referencing and results in performance degradation.
Scalability, As shown in the scalability section, VAP improves across all metrics as training data grows, demonstrating strong scalability. This follows from unified design that treats reference videos as prompts without task-specific modifications, together with the MoT framework, which preserves the backbone's generative capacity while enabling plug-and-play in-context generation.
9

---

Published as a conference paper at ICLR 2026
Table Ablation Study. We ablate on the in-context generation structure designs, temporal-biased RoPE, the scalability, and the transferability across different DiT structures. The last row is our default model (VAP), which uses MoT structure, temporal- -iased RoPE, 100K training pairs, and Cog VideoX-I2V-SB. Red stands for the best, Blue stands for the second best.
Metries
Text CLIP Score
Overall Quality
Semantic Aligument Score
Variant
Motion Smoothness
Dynamic Degree
Aesthetic Quality
In-Context Generation Structure
(Unidir- Cinss-Atn)
23.03
97.97
70.83
56.93
68.74
M43
Sangll-Branch-LoRR.
23.12
98.25
72.92
57.19
69.08
Mgg
(Unidir-Cross-fe Cross-Attn)
22.96
97.94
66.67
56.88
67.16
143
Addii--d
22.37
97.63
62.50
56.91
55.99
Position Embeedding Design
MdJ (ldenttical PE)
23.17
98.49
70.83
$7.09
68.98
wg (Nog. shift
57.31
69.05
Scalability
Me (1K) (10K)
22.84
92.12
60.42
56.77
63.91
22.87
94.89
64.58
56.79
66.28
Mg (50K)
23.29
96.72
70.83
56.82
68.23
DiT Strueture
Whe (1
23.93
97.87
79.17
$8.09
70.23
Ours
Mg (VAP)
24.13
77.08
57.71
70.44
Notation. ue (our VAP puramteetieed by B). s fin-contcat single brunch finctuning). (in-contest single branch LoRA finctuning), uc (unidince- injection), (unidirectional residual addition). (identical position embedding in reference and target), (iemporal shift negative tomporallwiht
tional
position embedding). (Wan2.1 Scale, (u())) indicates the number of training puirs (M e (IK, 10K, 50K, 100K); )). Our final version uses 100K training pains.
backbone).
DIT Structure. To test transferability, we equip Wan2. 1-12-11BB with VAP equal in parameter counts to Cog VideoX-I2V 12V-5B version (evenly inserted across 4 layers; = 5B), from Wan2.1 's stronger base iasprvves dynamic degree and aesthetic score but, because the only in-context interaction, yields slightly worse reference alignment than VAP on Cog VideoX.
We also ablate the in-context expert transformer layer distribution of VAP, and the video-prompt representation. Further experiment details are in Sec. due to page limits.
CONCLUSION
Video-As-Prompt (VAP) a unified, semantic-controllet video generation framework that treats ref- erence videos as prompts and enables gelug-and-pla in-context control via pixel-wise priors) expert. VAP overcomes limits of duructure-controlle methods (e.g., inappropriate and tesk/condition-speciii control and zero-shot generalizability. We build VAP-Data, the largest semantic-controlle generation dataset, and show in extensive experiments that VAP achieves state-of the-art among open-source models, comparable performance to commercial models, and strong generalization.
designs (e.g.. non-generalizable models), providing scalable semantic
video
Limitations and Future Works. Despite strong performance, some limitations need further study: (1) We experimented on our large-scale VAP-Data, yet the semantic conditions in VAP-Data are rel- atively limited, synthetic, and derived from other generative models, which may inherit the specific stylistic biases, artifacts, and conceptual limitations of the source templates (see Sec. DD. We leave the construction of larger-scale, real, semantic- controlle- video data future work. (2) VAP uses reference video, reference caption, and a target caption to guide semantic control. To stay close to original DiT distribution, we employ standard video descriptions as captions; however, inaceu- rate semantics descriptions or large subject mismatch can degrade generation quality (see Sec. F. Instruction-style captions (e.g., "please follow the Ghibli style in the reference video") may more effectively capture the intended semantics and improve control.
ACKNOWLEDGMENTS
This project was supported in part by the Innovation and Technology Fund Hong Kong S.A.R.
10

---

Published as a conference paper at ICLR 2026
6 ETHICS STATEMENT
Scope and intended use (research-only). VAP targets semantic-controlled video generation for research, education, and creative prototyping, where reference video and optional caption steer concep/ixlee/ootioncaaeee
is not intended for surveillance, impersonation, political persuasion, or other high-risk deployments. We will accompany any artifact release with research-only license and an acceptable-use policy (AUP) that explicitly prohibits abusive or unlawful scenarios.
Misuse risks and technical/operationaal mitigations. Potential misuses include identity imper- sonation, "deepfake" content, targeted harassment, deceptive political messaging, and generation of sexualized or violent media. Our mitigations include: (i) research-only release; (ii) default content filters blocking clearly harmful categories (e.g., sexual content, explicit violence, hate symbols).
REFERENCES
Jianhong Bai, Menghan Xia, Xiao Fu, Xintao Wang. Lianrui Mu, Jinwen Cao, Zuozhu Liu, Haoji Hu, Xiang Bai, Pengfei Wan, al. Recammaster: coneraecontrolled generative rendering from single video. In ICCV, 2025a.
Jianhong Bai, Menghan Xia, Xintao Wang. Ziyang Yuan, Zuozhu Liu, Haoji Hu, Pengfei Wan, and Di ZHANG. Syncammaster: Synchronizing multi-camera video generation from diverse viewpoints. In ICLR, 2025b.
Yuxuan Bian, Xuan Ju, Jiangtong Li, Zhijian Xu, Dawei Cheng, and Qiang Xu. Multi patch predic- tion: adapting language models for time series representation learning. In ICMI., 2024.
Yuxuan Bian, Ailing Zeng, Xuan Ju, Xian Liu, Zhaoyang Zhang, Wei Liu, and Qiang Xu. Motion- craft: Crafting whole-body motion with plug-and and-play multimodal controls. In AAAI, 2025a.
Yuxuan Bian, Zhaoyang Zhang, Xuan Ju, Mingdeng Cao, Liangbin Xie, Ying Shan, and Qiang Xu. Videopainter: Any-length video inpainting and editing with plug-and-play context control. In SIGGRAPH, 2025b.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In CVPR, 2023.
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng. Ricky Wang, and Aditya Ramesh. Video generation models as world simulators, 2024. URL https: com/ research/
video-gereration--nnn --s---mmmolllss-ww-rro------ d imulators
Boyuan Chen, Diego Martí Monsó, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitz- mann. Diffusson forcing: Next-token prediction meets full-sequence diffusion. NeurIPS, 2024a.
Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for nigh-quality video diffusion models, 2024b.
Civitai. Civitai. https://civitai: com/ 2025.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.06261. 2025.
Tri Dao. FlashAttention-2:: Faster attention with better parallelism and work partitioning. In ICLR, 2024.
Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germani- dis. Structure and content-guided video synthesis with diffusson models. In ICCV, 2023.
Gongfan Fang, Kunjun Li. Xinyin Ma, and Xinchao Wang. Tinyfusion: Diffusion transformers learned shallow. In CVPR, 2025.
11

---

Published as a conference paper at ICLR 2026
Jianxiong Gao, Zhaoxi Chen, Xian Liu, Jianfeng Feng, Chenyang Si, Yanwei Fu, Yu Qiao, and Ziwei Liu. Longvie: Multimodal-guinn -uidde controllable ultra-long video generation. arXiv preprint (.Xiv:25088033644, 2025a.
Yu Gao, Haoyuan Guo, Tuyen Hoang. Weilin Huang. Lu Jiang, Fangyuan Kong, Huixia Li, Jiashi Li, Liang Li, Xiaojie Li, et al. Seedance 1.0: Exploring the boundaries of video generation models. arXiv preprint urXiv:2506.09/ 13, 2025b.
Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Duval, Samaneh Azadi, Sai Saketh Ramb- hatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra. Emu video: Factorizing text-to-video generation by explicit image conditioning. arXiv preprint arXiv:2311.10700 2023.
Zekai Gu, Rui Yan, Jiahao Lu, Peng Li, Zhiyang Dou, Chenyang Si, Zhen Dong, Qifeng Liu, Cheng Lin, Ziwei Liu, et al. Diffusion as shader: 3d-aware video diffusion for versatile video generation control. In SIGGRAPH, 2025.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Sparsectrl: Adding sparse controls to text-to-video diffusion models. In ECCV, 2025a.
Yuwei Guo, Ceyuan Yang, Ziyan Yang, Zhibei Ma, Zhijie Lin, Zhenheng Yang, Dahua Lin, and Lu Jiang. Long context tuning for video generation. arXiv preprint Stti::2503.15889 2025b.
Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang, and José Lezama. Photorealistic video generation with diffusion models. arXiv preprint arXhv:2312.06662 2023.
Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cameractrl: Enabling camera control for text-to-video generation. arXiv preprint 24110:2404.02201, 2024.
Hao- Yu Hsu, Zhi-Hao Lin, Albert Zhai, Hongchi Xia, and Shenlong Wang. Autovfx: Physically realistic video editing from natural language instructions. arXiv preprint arXhv:2411.02394 2024.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 2022.
Li Hu. Animate anyone: Consistent and controllable image-to-video synthesis for character anima- tion. In CVPR, 2024.
Lianghua Huang, Wei Wang, Zhi-Fan Wu, Yupeng Shi, Huanzhang Dou, Chen Liang, Yutong Feng, Yu Liu, and Jingren Zhou. In-context lora for diffusion transformers. arXiv preprint 21X11:2410.23775. 2024.
Yuzhou Huang, Ziyang Yuan, Quande Liu, Qiulin Wang. Xintao Wang, Ruimao Zhang, Pengfei Wan, Di Zhang, and Kun Gai. Conceptmaster: Multi-concept video customization diffusion transformer models without test-time tuning. arXiv preprint arXiv:2501.04695 2025.
Ondrej Jamriška, Šárka Sochorová, Ondfej Texler, Michal Lukáë, Jakub Fišer, Jingwan Lu, Eli Shechtman, and Daniel Sýkora. Stylizing video by example. ACM Transactions on Graphics, 2019.
Zeyinzi Jiang, Zhen Han, Chaojie Mao, Jingfeng Zhang, Yulin Pan, and Yu Liu. Vace: All-in-one video creation and editing. arXiv preprint 2055:22503.07598, 2025.
Wonjoon Jin, Qi Dai, Chong Luo, Seung-Hwan Baek, and Sunghyun Cho. Flovd: Optical flow meets video diffusion model for enhanced camera--ontrolled video synthesis. In CVPR, 2025.
Xuan Ju, Ailing Zeng. Chenchen Zhao, Jianan Wang. Lei Zhang, and Qiang Xu. Humansd: A native Akeleton-guides diffusion model for human image generation. In ICCV, 2023.
Xuan Ju, Weicai Ye. Quande Liu, Qiulin Wang, Xintao Wang, Pengfei Wan, Di Zhang, Kun Gai. and Qiang Xu. Fulldit: Multi-task video generative foundation model with full attention. In ICCV, 2025.
12

---

Published as a conference paper at ICLR 2026
Diederik P. Kingma and Max Welling arxiv:1312.611, 2013.
Auto-encoding variational bayes.
arXiv preprint
Kling. Kling:
Image-to-video special effects.
https://klinges kuaishou com/
[aagg--owww..ee//ppccaaaaaefee/s 2025.
Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. Xrviv preprint arXiv:2412.03603, 2024.
Mathis Koroglu, Hugo Diselles-Dupre Guillaume Jeanneret, and Matthieu Cord. Onlyflow: Optical flow based motion conditioning for video diffusion models. In CVPR, 2025.
Guojun Lei, Chi Wang, Hong Li, Rong Zhang, Yikai Wang, and Weiwei Xu. Animateanything: Consistent and controllable animation for video generation. arXiv preprint arXiv:2411.1088 2024.
Weixin Liang. Lili Yu, Liang Luo, Srinivasan Iyer, Ning Dong, Chunting Zhou, Gargi Ghosh, Mike Lewis, Wen-tau Yih, Luke Zettlemoyer, al. architecture for mudti-modal foundation models. arXiv preprint arXiv:241 2222441114999 2024.
A sparse and scalable
Han Lin, Jaemin Cho, Abhay Zala, and Mohit Bansal. Cirl-adapeer: An efficient and versatile framework for adapting diverse controls to any diffusion model. arXiv preprint arXiv:2404.0990 2024.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamen Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXi::2210.02747, 2022.
Xinyu Liu, Ailing Zeng, Wei Xue, Harry Yang, Wenhan Luo, Qifeng Liu, and Yike Guo. Vfx creator: Animated visual effect generation with controllable diffusion transformer. arXiv preprint arXiv:2502.05979, 2025.
Miles Macklin, Matthias Müller, Nuttapong Chentanez, and Tae- Yong Kim. Unified particle physics for real-time applications. ACM Transactions on Graphics, 2014.
Fangyuan Mao, Aiming Hao, Jintao Chen, Dongxia Liu, Xiaokun Feng, Jiashu Zhu, Meiqi Wu, Chubin Chen, Jiahong Wu, and Xiangxiang Chu. Omni-effects: Unified and spatially- visual effects generation. arXiv preprint wwXi::2508.07981, 2025.
Willi Menapace, Aliaksandr Siarohin, Ivan Skorokhodov, Ekaterina Deyneka, Tsai-Shien Chen, Anil Kag, Yuwei Fang, Aleksei Stoliar, Elisa Ricci, Jian Ren, et al. Snap video: Scaled spa- tiotemporal transformers for vexdeo synthesis. arXiv preprint arXiv:2402.14797, 2024.
Chong Mou, Xintao Wang. Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In AAAI, 2024.
OpenAl. Introducing GPT-5, 2025.
URL htt.p:://openal.commmnneee
Introducing-gpt--3
William Peebles and Saining Xic. Scalable diffusion models with transformers. In ICCV, 2023.
Pexels. Pexels. https:
2025.
Pika. Pika: Ai video generation effects. http:s///ppkk..attpikafffee... 2025.
Pix Verse. Pixverse: Ai video creation platform. htttps://aap..... htttp::/eppppixvesssaaaaaa
2025.
Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih-Yao Ma, Ching- Yao Chuang, et al. Movie gen: A cast of media founda- tion models. arXiv preprint ArXiv:2410.133 13720, 2024.
Alexander Pondaven, Aliaksandr Siarohin, Sergey Tulyakov, Philip Torr, and Fabio Pizzati. Video motion transfer with diffusion transformers. In CVPR, 2025.
13

---

Published as a conference paper at ICLR 2026
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. NeurIPS, 2022.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang. Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209. 14792, 2022.
Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny. Stylegan-v: continuous video generator with the price, image quality and perks of stylegan2. In CVPR, 2022.
Kiwhan Song, Boyuan Chen, Max Simchowitz, Yilun Du, Russ Tedrake, and Vincent Sitzmann. History-guided video diffusion. In ICML, 2024.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputing, 2024.
Zhenxiong Tan, Songhua Liu, Xingyi Yang, Qiaochu Xue, and Xinchao Wang. Ominicontrol: Min- imal and universal control for diffusion transformer. In ICCV, 2025.
Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In ECCV, 2020.
Vidu. Vidu: Ai video templates.
2025.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. NerulPS, 2016.
Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.2033
2025.
Luozhou Wang. Ziyang Mai, Guibao Shen, Yixun Liang, Xin Tao, Pengfei Wan, Di Zhang, Yijun Li, and Ying-Cong Chen. Motion inversion for video customization. In SIGGRAPH, 2025.
Zhouxia Wang, Ziyang Yuan, Xintao Wang, Yaowei Li, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionctrl: A unified and flexible motion controller for video generation. In SIGGRAPH, 2024.
Enze Xie, Junsong Chen, Yuyang Zhao, Jincheng Yu, Ligeng Zhu, Chengyue Wu, Yujun Lin, Zhekai Zhang, Muyang Li, Junyu Chen, al. Sana 1.5: Efficient scaling of training-time and inference- time compute in linear diffusion transformer. arXiv preprint 2rXiv:2501.18427 2025.
Shiyuan Yang. Zheng Gu, Liang Hou, Xin Tao, Pengfei Wan, Xiaodong Chen, and Jing Liao. Mtv- inpaint: Multi-task long video inpainting. arXiv preprint arXiv:2503.11412. 2025.
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, al. Cogvideox: Text-to-video diffusion models with expert transformer. In ICLR, 2024.
Danah Yatim, Rafail Fridman, Omer Bar-Tal, Yoni Kasten, and Tali Dekel. Space-time diffusion features for zero-shot text-driven motion transfer. In CVPR, 2024.
Zixuan Ye, Huijuan Huang, Xintao Wang, Pengfei Wan, Di Zhang, and Wenhan Luo. Stylemaster: Stylize your video with artistic generation and translation. In CVPR, 2025.
David Junhao Zhang, Dongxu Li, Hung Le, Mike Zheng Shou, Caiming Xiong, and Doyen Sahoo. Moonshot: Towards controllable video generation and editing with multimodal conditions. arXiv preprint 200007,4 2024.
14

---

Published as a conference paper at ICLR 2026
Jintao Zhang, Jia Wei, Pengle Zhang. Jun Zhu, and Jianfei Chen. Sageattention: Accurate 8-bit attention for plug-and-play inference acceleration. In ICLR, 2025a.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. ICCV, 2023.
Shiyi Zhang, Junhao Zhuang, Zhaoyang Zhang, Ying Shan, and Yansong Tang. Flexiact: Towards flexible action control in heterogeneous scenarios. In SIGGRAPH, 2025b.
Rui Zhao, Yuchao Gu, Jay Zhangjic Wu, David Junhao Zhang, Jia- Wei Liu, Weijia Wu, Jussi Keppo, and Mike Zheng Shou. Motiondirector: Motion customization text-to-video diffusion models. In ECCV, 2024.
Haitao Zhou, Chuang Wang, Rui Nie, Jinxiao Lin, Dongdong Yu, Qian Yu, and Changhu Wang. Trackgo: A flexible and efficient method for controllable video generation. arXiv preprint arXiv:2408. 11475, 2024.
15

---

Published as a conference paper at ICLR 2026
In the appendix, provide more qualitative results (Sec. A downstream application demonstra- tion (Sec. B more implementation details (Sec. including the hyperparameters and the seman- tic aligmment score metric. Then, illustrate more dataset details and limitations of our VAP- Data (Sec. D). Furthermore, we discuss more about the influence of reference video quality, caption quality, and multiple reference videos (See. [.. And conduct more ablation about VAP (Sec. Finally, we illustrate the use of large language models (Sec. G
A GALLERY
To further demonstrate our VAP's performance, we provide more semantic-controlled generation cases in Fig. Fig. 9 for better visualization.
We strongly encourage readers to view our webpage in the supplementary
Reference Video
Reference Image
Generated Video
Turn info Ladudu
Turn into Lodudu
Turn into Loduda
Turn Into a MineCroft Toy
MineCroft Toy
Toy
Squid
Fuzzy
info Musting Man
Figure
Lover
Lover
Covered Liquid Matol
Covered Liquid Metol
Wrap
Zangzi
Figure 8: Additional visualizations of VAP, including entity transformation and entity interaction in concept semantic categories.
16

---

Published as a conference paper at ICLR 2026
Reference Video
Reference Image
Generated Video
Style
Ghibli Style
Ghibli Style
Simpsons Style
Simpsons Style
Blooming Style
Blooming style
Expantion
ZE
Expansion
Squish
Squish
Hitcheock Zoom
Hitchrock Zoom
Earth Zoom Out
Figure 9: Additional visualizations of VAP, including style semantic categories, motion semantic
categories (Non-Human Motion and Human Motion), and camera semantic categories.
17

---

Published as a conference paper at ICLR 2026
Reference Video
Reference Image
Generated Video
+
Ghibli Style
Lodudu
Mecha X
Poper
Gender Swap
Cloning
Zoom Out
chcook
Figure 10: Given different reference videos (with different semantics) and the same reference image, our VAP consistently generates a new video for each semantic.
B APPLICATION
Our Video-as-Prompp (VAP' model supports the following downstream applications by disentan- gling a semantic concept from a source video and applying it to a new subject:
1. Given different reference videos (with different semantics) and the same reference image, our VAP consistently generates new video for each semantic (Fig. 10);
2. Given different reference videos (with the same semantics) and the same reference im- age, VAP consistently generates the target video aligned with the provided semantics (Fig.
Given one reference video and different reference images, our VAP transfers the same se- mantics from the reference video to each image and generates the corresponding videos (Fig. 12);
4. Beyond video prompts, VAP allows for fine-grained adjustments using modified text prompts, by fixing the reference inputs and only changing single word in the prompt (e.g., black to white). VAP can precisely edit attributes the generated output while preserving identity and motion (Fig. 13
C IMPLEMENTATION DETAILS
C.I HYPERPARAMETERS
In Tab. summarize hyperparameters for two VAP variants based CogVideoX-5B (Yang et al. 2024) and Wan2.1-14B (Wan et al. (025). respectively, showing transfcrability across differ- DiT architectures.
18

---

Published as a conference paper at ICLR 2026
Reference Video
Reference Image
Generated Video
+
Ghibli Style
+
Style
Ghibli Style
Expand 6 Bolloon
Expand like Balloon
Expand
Figure 11: Given different reference videos (with the same semantics) and the same reference image, our VAP consistently generates the target video aligned with the provided semantics
Reference Video
Reference Image
Generated Video
+
Style
+
Ghibli Style
Ghibli
Expand like Balloon
//
+
Expand like Balloon
Expand like Ballcon
Given one reference video and different reference images, our VAP transfers the same semantics from the reference video to each image and generates the corresponding videos.
Figure 12:
C.2 METRICS
As stated in Sec. 4.2 standard video-quality metries (e.g., CLIP score (Radford et al., 2021), aes- thetic score (Schuhmann et al. 2022 do not reliably capture adherence specific semantic dition, so we introduce a semantic-alignme semantic condition and the generated video; submit each (reference, generation) pair and the evaluation rules to Gemini-2.5-pp (Comanici et al. 2025) for automatic scoring.
score that measures consistency between the reference
The evaluation rules pair general template with key criteria for each semantic; for each case, provide the template, the criteria for the current semantic (see Tab. 4 the reference video condition, and the generated video to the VLM, which scores them under these rules.
19

---

Published as a conference paper at ICLR 2026
Reference Video
Reference Image
Generated Video
+
into block Ledudu
+
Turn Ladudu
Turn into golden Ladudu
Turn info green Ladudu
Turn Info purple Ladudu
Turn Info
Figure 13: Given fixed reference video and a reference image, our VAP preserves semantics and identity while using msor-modified prompt to adjust fine-graine fine-grained attributes.
Table 3: Hyperparameter selection for CogWideoX-I2V-5BB-oa based and Wan2.1-I2V-14B-batt VAP.
Model
Hyperparameter
Cog
Wan2. 1-22-bbadd
Batch Size / GPU
1/1
1/2
Accumulate Step
Optimizer
Adam W
Adam W
Weight Decay
0.0001
0.0001
Learning Rate
0.00001
0.00001
Learning Rate Schedule
constant with warmup
constant with warmup
WarmUp Steps
1000
1000
Training Steps
20,000
20,000
Resolution
480p
480p
Prediction Type
Velocity
Flow Matching 40
Num Layers
42
[0, 1, 2, 3.
4, 5, 6, 7, 8, 9, 10,
11, 12,
13,
14,
15, 16, 17, 18, 19, 20,
21, 22,
[0, 8, 12, 16, 20, 24, 28, 32, 36]
MoT Layers
23,
24,
25,
26
27, 28, 29, 30. 31, 32,
33.
34,
35,
36,
37, 38, 39, 40, 41]
Pre-trained Model
VigViiexX-12V-511 Cog
Wan2.1-12V-145
Sampler
VideoX DDIM 50
Flow Euler 30
Sample Steps
Guide Scale
6.0
5.0 "420s
Generation Speed (1 A100)
"540s
Device
A100 x48
A100x48
Training Strategy
FSDP DDP / BFloat16
FSDP DDP Parallel BFloat16
To validate the stability of the semantic alignment score, we conduct the same evaluation experiment with another state-of-the-ar vision lanuage model GPT-5 (OpenAl 2025); its scores match closely Gemini-2.55Pro (Comanici et al. 2025) and follow the trends of human preference rate in our user study (see Tab. 5). confirming the validity of the metric. This further verifies the effectiveness and validity of proposed semantic alignment score.
20

---

Published as a conference paper at ICLR 2026
Table 4: Prompt components for the semantic-aligmment metrics. We provide the general tem- plate and the specific criteria of "Ghibli Style" as an example.
Category
Content
General Template
You are an expert judge for reference based semantic video generation.
INPUTS
REFERENCE video: the target semantic to imitate. TEST video: a new output conditioned on NEW reference image. Human criteria (treat ground truth suceess checklist; overrides defaults conflict): {criteria]
REGIME DECISION
Classify the semantics into one of: A) ID-TRANSFORM (identity-chagiing): the main subject/object changes semantic class material/state. Layout and identity may legitimately change consequence of the transformation. B) stylization, camera motion (pan/zoom), mild geometry exaggeration, lighting changes, human motion, etc. The main subject class/identity should remain the same.
ID-TRANSFORM andentity-preserviii
If the REFERENCE: elcarly shows class/state change, choose A. Otherwise, choose B. When uncertain, choose B.
EVALUATION
1) SEMANTIC MATCH (0-60) Regime A (ID-TRANSFORM); How strongly and accurately does TEST reproduce the REFERENCE'S target state/look/behavior on the correct regions? Is the touree-target mapping consistent (same parts transform to corresponding target parts)? Does the transformed state resemble the REFERENCE target, not generic filter? Regime (NON-ID- DONN-I--RANSSS TRANSFORM); Does TEST replicate the specific semantic (style, camera motion, geometric exaggeration) while keeping the subject recognizable and aligned to the intended scope?
2) IDENTITY / LLAOUUT CORRESPONDENCE. (0-20) Regime A: Reward semantic correspondence rather than identical identity; coarse scene continuity is preserved unless the REFERENCE implies re-layout. Regime Main subject identity stays intact flcc///oly/llohhes/eeaues)) and coarse spatial layout remains consistent (no unintended subject swaps/teleports) 3) TEMPORAL QUALITY and PRAANSOORMATION CONTINUITY (0-20) Check sossst-ssstiii--ffottt completeness of the transformation implied by the REFERENCE. Avoid pop-in/out. Motion is smooth, minimal flicker, and the background is reasonably stable. No frozen loops unless
REFERENCE loops.
HARD FAIL CAP (force FINAL |= 20 if any true) REFERENCE shows an ID- TRANSFORM. but TEST lacks the transformation, targets the wrong class/material, or completes 770% of the transformation timeline. Severe identity loss in Regime (unrecognizable face/hody, unintended person/object swap).- Gross broken anatomy (detached/missing limbs, implausible face mash) is not required by the semantics. Extreme temporal instability unreadable corruption (heavy strobe, tearing, tiling). - Hallucinated intrusive objects that block the subject or derail the semantics. OUTPUT (exactly ONE line of JSON; integer only) ("score": 1-100] Regime: NON-IP CON-ID-10 TRANSFORM (identity-preeerving stylization).
Semantic Criteria
Semantic: Ghibli-styl stylization the overall look gradually transitions hand-drawn, soft, film-like Ghibli aesthetic across the whole frame.
Identity preservation: The main subject remains recognizable;
pppeeaaance/roppottoon///e colors are largely maintained (stylistic simplification and brush-like textures allowed).
Motion allowance: Light natural motion is allowed (e.g.. slight subject scene movement) without disrupting effect consistency.
Exclusions: No identity swaps, major re-layout, or gross anatomy distortions unless explicitly implied by the reference.
21

---

Published as a conference paper at ICLR 2026
Table 5: Semantic alignment score metric and user preference. Columns are models; rows are semantic alignment score evaluated by Gemini-2.5-Pro (Comanici et al., 2025), semantic alignment score evaluated by GPT-5 (OpenA (025). and human preference rate results of our user study.
Metric
VACC:irriinal VACE (Depth) VACE (Optical Flow) CoVVidenX-I2V Cag VVV (LoItA) Kling Vidu Video-AA Prommpp (VAP)
Alignment Score Sormini-25--Poo
St
41:35
46 71
25.04
68
24:00
70:44
Alignment Score (GPT-5)]
32.53
10.41
45.00
28, 16
65.91
71
70.26
Preference Rale (S.)
0.6S
0.7%
1.8
6.9%
13.1%
38.25
38.7%
D DATASET
D.I DATASET DETAILS
In-context learning requires vast amounts of example pairs, which simply do not exist for semantic video tasks. Filming 100k real-world pairs is nearly impossible for a research exploration. Our
solution was to bootstrap it. We curated thousands of high-quality real images and then used the
existing "z00 of specialist models" (commercial APIs (Kling) 2025 Vidu 2025; Pix Verse (225))
and LoRAs (Hu et al.) [2022] Civitai 2025 ) powerful, automated engine to create our paired dataset, VAP-Data. As shown in Sec. 3.2 and Fig. 3 VAP-Data is the largest semantic-controlledd
paired dataset date, with over 100K samples across 100 semantic conditions, covering primary categories: concept (entity transformation and interaction), style, motion (human and non-human), and camera movement. The detailed distribution of semantic conditions is provided in Tab. 6
Crucially, VAP-Data is more than just a dataset; it's proof of a concept. We show that we can train single generalist model (VAP) to learn the unified underlying principle of semantic control by showing it various examples from disparate specialist models.
For evaluation, we evenly sampled 24 semantic conditions from 4 categories (concept, style, motion, camera) in VAP-Data test subset, with 2 samples each, totaling 48 test samples.
D.2 DATASET LIMITATIONS
Even though our VAP-Data is the largest semantic-controit video generation dataset, it still has limitations. As noted in Sec, 4.3
VAP-Data was created using visual effects templates from commer-
cial models (vidu (Vidu 20255. 5, Kling KKling 2025)) and community LoRAs (Hu et al. 2022: Wan
al. 2025 Yang al. 2024 Civitai 2025). Thus, the dataset is synthetic and derived from other
generative models, leading to VAP may inherit the specific stylistic biases, artifacts, and conceptual limitations of the source templates (e.g.. if the source models are poor at generating hands, VAP will likely not learn to generate hands well from this data). Building a large, real-world, semantic- controlled video dataset would help address this issue, but it beyond this paper's main focus; we leave it for future work.
Nevertheless, zero-shot experiments in Sec. 4.4 and downstream tasks in Sec. show that VAP gen- eralizes to unseen semantic conditions (Liu et al., (025) (e.g., crumble, dissolve, levitate, melt) and across tasks, including using different reference videos to prompt single reference image under different semantic conditions using the same reference videos to prompt different reference im- ages under a fixed semantic condition. These results demonstrate the generality of VAP and we hope they inspire advances in controllable video generation; broader data collection is left to future work. Additional visualizations are available on the supplementary webpage.
E LIMITATION ANALYSIS
E. INFLUENCE OF REFERENCE VIDEO AND CAPTION
VAP learns in-context generation from large paired video-caption data: given captions for a ref- erence and target video, the shared semantic attributes in both captions aid in transferring the semantic properties of the reference video to the target video. Specifically, when both captions men- tion the same concept (e.g., "molten metal pours over the target..."" in similar way. VAP retrieves the relevant semantics from the reference prompt and applies to the target. The reason why we use standard video-deseription captions (e.g." A static Grogu is centered... A viscous, reflective
22

---

Published as a conference paper at ICLR 2026
Table 6: Dataset statisties by 4 primary semantic categories. We reorganize the dataset into primary categories: Concept (merging entity transformation and interaction), Style, Motion (cover- ing human and non-human motion transfer), and Camera Movement. For each primary category, report its subcategory (if any), the alphabetical semantic condition subset list (names come from commercial models API definition (Kling 2025] Vidu 2025). and community visual effects LoRA
definition (Civitai 2025). see Sec. 4.3) and the total number of videos.
Primary Category
Total Videos
Subcategory
Subset (alphabetical)
Concept (n=56)
Entity Transformation (n=24)
captain america, cartoon doll, mushrooms, fairy me, fishermen, fuzzyfuzzy, gender swap, get thin- ner. hair color change, hair swap, ladudu me, mecha x. minecraft, monalisa, muscling, pet to human. sexy me, squid game, style me, super saiyan, toy me, venom, vip, zen
17k
Entity Interaction (m=21)
aliens coming, child memory. christmas, cloning, couple arrival, couple drop, couple walk, covered liquid metal, drive yacht, emoji figure, gun shooting, jump to pool, love drop, nap me, punch hit, selfie with younger self, slice therapy, soul depart, water- melon hit, zongzi drop, zongzi wrap
20k
Style (a=11)
Stylization (n=11)
american comic, bjd, bloom magic, bloombloom. clayshot, ghibli, irasutoya. jojo, painting, sakura season, simpsons comic
15k
Motion (n=41)
Human Motion Transfer
break glass, crying, cute bangs, emotionlab, fly- ing, hip twist, laughing, live memory, live photo, belly dance, pet finger, shake dance, shake it down, split stance human, split stance pet, walk for- ward
10k
(m=16)
Non- nun-human Motion Transfer
auto spin. balloon flyaway, crush. decapitate, dizzy- dizzy, expansion, explode, grow wings, head to bal- loon, paperman, paper fall, petal scattered, pinch, rotate, spin360, squish
19k
(n=16)
Camera (n=12)
Camera Movement Control (n=12)
dolly effect, earth out, hitchcock zoom, 19k down, move left, move right, move up, orbit, orbit dolly, orbit dolly fast, zoom in, zoom out
Subset counts (n) are reported per subcategory and are alphabetically sorted within each subcategory. Overall subsets across all primary categories: 100. Overall videos across all categories: >> 100k.
gold liquid appears on the forehead metal begins pour over her face from above... "), is match the pre-training data distribution, Consequently, performance depends on caption quality and on structural similarity between the main subjects: stable when caption styles align and subjects are similar, but degrades when descrip- vs. "...A
"A young woman stands still... A thick, reflective liquid
tions diverge (e.g., "...A viscous, reflective gold liquid appears on the forehead viscous, reflective rose-gold water pours over the snail ..."! ") or when subjects differ markedly (e.g., Grogu vs. snail). As shown in Fig. [4] the bad caption mislabels "water" instead of the intended "liquid metal"; the good reference subject (the young woman) is structurally closer to Grogu, while the snail differs greatly and its semantic signal is weak (the liquid metal and shell have similar colors), yielding poorer alignment and less appealing visuals for the bad reference case.
We empirically observe that Aisoo-As-Proomin typically:
First, relies on the most critical shared semantics the reference and target captions to establish coarse-level "semantic anchor" for the concept to be transferred.
23

---

Published as a conference paper at ICLR 2026
Reference Image
Video
Genzration Renulf
siill. A lhick, relsel live ligut matul begine to pour over from sheve
enall raal stone vitcoue roflaction rose rsee-ged wallr pours over
dialic Brage cerferedl.. visoos refheclive gold lipiid appears
Figure 14: Limitation visualization. VAP transfers semantics reliably when the semantic descrip- tion of reference caption aligns with that of the target caption and subject structure aligns with the target: aligned descriptions ("gold liquid" and "liquid metal") and similar subject structures (Grogu and young woman) yield good results (top). Mislabeled semantic descriptions ("water" vs. "liquid metal"), or large subject mismatch (Grogu snail), reduce alignment and visual quality (bottom).
Reference Video
Ground Truth
Video
Failure Result
Reference Video 3
Figure 15: Failure case of multi-reference prompting. Left: three reference videos with divergent structure and similar semanties (human, spider, flatfish). Right: Ground truth is on top. Using three (bottom) spuriously transfers unwanted appearance cues (e.g., fish shape and spider-like legs) onto the dog. We attribute this leakage generic captions that lack explicit referent; stronger multi- reference control or instruction-style captions could mitigate it.
2. Then, it leverages the reference video fill in the finer-graine details (e.g., stylistic at- tributes, motion patterns, or camera speed).
When captions are noisy, under-specified, or semantically misaligned, this "anchor" becomes am- biguous. This ambiguity directly impairs the model's semantic selection and leads to the observed performance degradation.
E.2 INFLUENCE OF MULTIPLE REFERENCE VIDEOS
We examine how the number of video prompts affects performance by supplying 1-3 semantically matched reference videos during training and testing. Empirically, results are similar to using single reference. However, with multiple references, the model may blend unwanted visual details across videos, as shown in Fig. 15 We hypothesize this stems from general-pe purposse captions, which lack explicit semantic referents. When the three references differ in structure (human, spider, flatfish) and in semantic realization (e.g., reference clearly shows "AI Lover Drop"; reference weakest, with a flatfish swimming up
2 introduces falling spider without hug; reference instead of falling), the model mixes semanties from reference I with appearance from reference (spider legs) and contours from reference 3 (fish shape). A more effective multi-reference control mechanism (e.g., tailored RoPE for multi-reference conditions)-or that specifies the intended referent may mitigate this issue. A full study of model and caption design for multi-reference training is beyond this work and left for future research.
instruction-style caption
E.3 EFFICIENCY
Like prior plug-and-play methods Zhang et al. 2023: Jiang et al., 2025). our approach avoids re- training pre-trained video diffusion transformers at pre-training scale, but the added parameters introduce extra inference cost-higher memory use and longer runtime. Specifically, the impact varies with the distribution MoT layers in VAP; as shown Tab.3 inference time roughly doubles average, mainly due to additional in-context computation. Given the strong plug-and and-play unified semantic control in-context generation and the fact that we avoid retraining the backbone, this
overhead is acceptable. Performance optimizations (e.g., sparse attention (Dao 2024
Zhang al.
24

---

Published as a conference paper at ICLR 2026
2025a) and pruning (Fang al. 2025: Xie et al. 2025) orthogonal and beyond the scope of this work; we leave them to future work.
F ABLATION STUDY
Table Ablation Study. We verify the effectiveness of our MoT structure, temporal-biasee RoPE, the scalability, and the transferability in different DiTs. The bottom row reports our full model.
Metries
Text
Overall Quality Dynamic Degree
Reference Aligument Score
Variant
CLIP Score
Motion Smoothness
Aesthetic Quality
(Single Branch)
23.03
97.97
70.83
56.93
68.74
(Single-Brancde (trach-LLoRA)
23.12
98.25
72.92
$7.19
69.28
Unidii-Cross-Am)
22.96
97.94
66.67
56.88
67.16
(Unidir-Aee Addii-nn
22.37
97.63
62,50
56.91
$3.99
Position Embedding Design
(Identical PE)
23.17
98.49
70.83
57.09
68.98
(Neg. shift in T, W)
23.45
98.53
72.92
57.31
69.05
Scalahility
(1K)
22.84
92.12
60.42
56.77
63.91
Nig [10K)
22.87
94.89
64.58
56.79
66.28
(50K)
23.29
96.72
70.83
56.82
68.23
N/ (100K)
24.13
77.08
57.71
70.44
DiT Structure
(Wan2.1-2V-14BB)
23.93
97.87
79.17
58.09
70.23
In-Context Expert Transformer Layer Distribution
(Cass)
24.05
98.52
75.00
57.58
70.22
Hig (Caas (l03N.))
23.72
98.19
70.83
56.71
69.61
Crrosaal) Mgg (Cisstla
23.90
98.41
75.00
57.18
69. 90.94
23.96
98.33
72.92
57.06
70.02
Video Prompt Representation
(noisy reference)
23.98
98.41
75.00
57.42
70.18
Ours
Mi (VAP)
57.71
70.44
Notation. uy (our VAP parameterized by 61. iin-cootext single branch finetuning), al (in context single branch LoRA finetuning), uc (unidircc- (unidiractional residual addition) (idenrical position emhedding reference target), (tempora shift negative temporal/witth shifts position embedding).
rional
(Wan2.1
backfone)) nuref (noisy reference prompts).
Mo'l layers, (e(C) activates MoT' blocks on layer index set & S [Mi] = (1..., Ni] of the backhonee with N) Transformer layers. We instantiate Case (1,, (1,2,... [0.5.N:]. fer tee [1,N) Ni,, fiat, ((Nnar) [1,3,.... [0.5N:]). and Cas- (1,3,,,... S...., Ni).
Scale, (M(M) indicates the number of video training pairs used (M e [1K, 10K, 50K, 100K))
In-context Generation Structure. We train 4 VAP variants to test the effectiveness of our mixture- of-traasformers (MoTs) adoption: A1. Single-Branch Finetuning u expand pre-trained DiT input sequence to RRefteut= Reduid Tariezt. Tarvideo] and finetune the full model; A2. Single- Branch LoRA Finetuning ug same as Al but freeze the backbone and train only the LoRA layers; A3. Unidirectional Cross-Attn uz: freeze the pre-trained DIT, add new branch with the same weights, and inject its features via layer-wise cross-attention; and A4, Unidirectional Addition u same A3 but inject features via residual addition. We evaluate on the same benchmark of VAP-Data. Results in Tab. 2 show: A1. MoT boosts performance by preserving the base DiT's generative ability while enabling plug-and-play in-context control. A2. LoRA helps retain the backbone's 's ability, but its limited capacity struggles with complex in-context generation, yielding suboptimal results. A3. Layer-wise bidirectional information exchange in MoT lets the reference piteopprompt representation adapt synchronously to the target tokens, improving semantic alignment. A4. Even with new data, residual-addition methods rely on rigid pixel-to-pixe mapping, mismatching semantic-controlled generation and degrading performance.
Position Embedding Designs. To validate the effectiveness of our temporally-biased RoPE, we evaluate two variants. (1) vdy applying identical RoPE to both the reference and target videos, which enforces an unrealistic pixel-wise alignment prior and leads to degraded performance; (2)ua: in addition to introducing temporal bias A, following in-context image generation (Tan et al.) 2025). we add a width bias by placing the reference video to the left of the target video. Experiments show that this increases the difficulty of spatial referencing and results in performance degradation.
25

---

Published as a conference paper at ICLR 2026
Scalability, As shown in Tab. 2 VAP improves across all metrics as training data grows, demonstrat- ing strong scalability. This follows from our unified design that treats reference videos as prompts without task-specific modifications, together with the MoT framework, which preserves the back- bone's generative capacity while enabling plug-and-play in-context generation.
DiT Structure. To test transferabilityy we equip Wan2.1-12V-144 with VAP equal in parameter counts to Vog version (evenly inserted across 1 layers; = 5B), which- from Wan2.1's stronger base-improovs dynamic degree and aesthetic score but, because the only in-context interaction, yields slightly worse reference alignment than VAP on Cog VideoX.
Layer Distribution We analyze how different layer distributions affect our in-context DiT Expert. (1)ue( ( Chest-batr): initializing and copying from the first half of the pre- trained DiT: (2)ug (iirstlaas:: from the first and last layers; (3)ue(Cula [|o.NN] 1): from the odd layers of the first half; and (4)ue (Coda): from all odd layers. The results show that balanced feature interaction improves generation quality (we( (Crrstlass) outperforms we halss-harr, and we (Cau) out- performs ue (Codd [ J)). However, while reducing layers can improve training and inference efficiency, it inevitably harms certain aspects of performance (uo(VAP) outperforms (o
Coeture-of-Transfrooe
Video Prompt Representation Inspired by Diffusion Forcing (Chen et al. 2024a Song et al. 2024
Guo et al. 2025b), we study video prompt representation by injecting noise into it. However, this often leads to severe artifacts. The core reason is that, unlike long-video generation in Diffusion Forcing, where copy-paste or overly static results common, our reference videos already differ significantly in appearance and layout from the target videos. Thus, adding noise to the video prompt corrupts the contextual information and degrades generation quality.
G USE OF LARGE LANGUAGE MODELS (LLMS)
Scope of use. We used a large language model (LLM) only for writing polish, including gram- mar correction, phrasing refinement, and improvements to clarity and readability. The LLM did not contribute to research ideation, problem formulation, method design, experimental setup, re- sult selection, interpretation, or drafting of technical content (theorems, algorithms, proofs, metrics, or analyses). All technical claims, experiments, figures, tables, and conclusions were conceived, implemented, and verified by the authors.
Process and safeguards. LLM assistance was applied post hoc to author-writter passages to im- prove presentation quality, without introducing new technical material or references. We reviewed each edited passage for accuracy and faithfulness to the original meaning and ran standard pla- giarism checks. No proprietary or sensitive data were disclosed to the LLM beyond non-sensitive manuscript text; when necessary, potentially identifying details were redacted.
No material impact on research outcomes. The use of LLMs had no bearing on the research ideas, empirical results, evaluation protocols, conclusions reported in this paper.
26


Reviews:
[{"review_id": "5LuewnXpz2", "reviewer": "Reviewer_YyML", "summary": "This paper addresses the significant challenge of achieving unified, generalizable semantic control in video generation. Current methods often fail by enforcing inappropriate pixel-wise priors from structure-based controls or by relying on non-generalizable, condition-specific finetuning and task-specific architectures.\n\nThe authors introduce Video-As-Prompt (VAP), a new paradigm that reframes this problem as in-context generation. VAP leverages a reference video as a direct semantic prompt. Its architecture augments a frozen base Video DiT with a plug-and-play MoT expert. This design prevents catastrophic forgetting, the trainable expert processes the reference prompt while the frozen backbone handles the target generation, with both communicating via full attention at each layer. A key component is a temporally biased RoPE, which breaks the model's default assumption of a pixel-aligned mapping between the prompt and the target video.\n\nTo train this model, the authors created and released VAP-Data, the largest dataset for this task, containing over 100,000 paired videos across 100 semantic conditions. As a single unified model, VAP sets a new state-of-the-art for open-source methods, achieves user preference rates comparable to leading commercial models, and demonstrates strong zero-shot generalization to unseen semantic concepts.", "questions": "The multi-reference failure case suggests the model struggles to disentangle pure semantics from appearance/layout when the reference videos are structurally diverse, leading to \"leakage\" of features like \"spider legs\" and \"fish shape\". You hypothesize this stems from using generic video captions. Do you believe training with more explicit \"instruction-style\" captions would be sufficient to solve this, or does this failure point to a need for a more explicit architectural component to enforce disentanglement?\nRegarding the synthetic VAP-Data, your zero-shot results are promising. However, one could argue the model is learning a \"meta-task\" of how to follow VFX-style instructions from the synthetic data, rather than a truly general semantic understanding. How does VAP perform on zero-shot tasks that are semantically distinct from the VFX-style domain?\nThe temporally biased RoPE is a key innovation. You mention shifting the reference's temporal indices by a fixed offset $\\Delta$. How was this offset value chosen, and how sensitive is the model's performance to this hyperparameter? Furthermore, does the MoT expert learn to attend to the entire reference video prompt equally at each step of the target generation, or does it learn a dynamic temporal correspondence?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Unified and Generalizable Model: VAP is, to the authors' knowledge, the first framework to successfully unify a diverse set of semantic controls into a single model without requiring per-task modules or finetuning.\n\nStrong Zero-Shot Performance: The model shows a strong ability to generalize to semantic conditions that were not included in the VAP-Data, such as \"crumble\" and \"dissolve\". This indicates it is learning a generalizable concept of semantic transfer.\n\nVAP-Data: The paper introduces the largest-ever dataset for semantic-controlled video generation. This dataset, and the bootstrapping method used to create it, will be highly valuable for future research.\n\nAblation Studies: The paper provides a very strong set of ablations that convincingly justify the architectural design. The comparison clearly shows the superiority of the MoT design over finetuning, LoRA, unidirectional cross-attention, and residual addition. The ablation on RoPE design also confirms the authors' hypothesis .", "weaknesses": "he paper is strong and transparently discusses its own limitations.\nSynthetic Data Limitations: The primary weakness, is that VAP-Data is entirely synthetic, generated using other models. The paper notes this means VAP may inherit the \"stylistic biases, artifacts, and conceptual limitations\" (e.g., bad hands) of these source models\nDependence on Caption Quality: Performance relies on well-aligned semantic descriptions in the reference and target captions. The authors show that mislabeled captions or a large mismatch in subject structure can significantly degrade generation quality.\nInference Cost: The plug-and-play MoT expert, while effective, adds significant computational overhead. The paper notes it roughly doubles inference time and increases memory usage.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vnkyc8kIzK", "reviewer": "Reviewer_1zha", "summary": "This paper proposes Video-As-Prompt (VAP), a unified framework for semantic-controlled video generation. Instead of pixel-aligned control (e.g., depth, pose), the method uses reference videos as prompts to transfer concepts, styles, motions, and camera behaviors. A Mixture-of-Transformers (MoT) design augments a frozen video DiT with a parallel expert transformer, enabling plug-and-play in-context conditioning while avoiding catastrophic forgetting. A temporally-biased RoPE removes false spatial correspondences between reference and target tokens. The authors also introduce VAP-Data, a 100K-pair synthetic benchmark across 100 semantic categories. Experiments show strong semantic alignment, competitive with commercial models, and notable zero-shot generalization to unseen effects.", "questions": "- How does the method perform on noisy, compressed, or hand-held reference videos? Any robustness evaluation or failure cases that could be shared?\n\n- The paper mentions instruction-style prompting as potentially beneficial. Have the authors quantified improvements when using such synthetic instructions versus descriptive captions?\n\n- Is the MoT design amenable to parameter-efficient variants (e.g., partial expert layers or shared attention blocks)? If so, did the authors explore such configurations?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The paper introduces a clean and unified perspective by treating a reference video as a semantic prompt, avoiding the fragmented control paradigms (e.g., pose/depth-specific pipelines) used in prior work.\n\n+ The MoT structure and temporally-biased RoPE are well-motivated and demonstrated to be effective in preventing forgetting and cross-token interference; ablations support the design choices.\n\n+ Strong qualitative performance across multiple semantic axes (concept, style, motion, camera intent), with notable zero-shot generalization and performance competitive with proprietary systems.", "weaknesses": "- The training data is largely synthetic and template-driven, which may limit generalization to real-world video distributions; robustness to natural, diverse videos is not extensively evaluated.\n\n- The MoT architecture increases compute cost and memory footprint, making the method relatively heavy compared to lightweight or plug-in control modules.\n\n- The method assumes reasonably descriptive captions for reference and target videos; behavior under noisy or under-specified captions remains insufficiently analyzed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tQeV4lZDQB", "reviewer": "Reviewer_KiRJ", "summary": "This paper presents Video-As-Prompt (VAP), a unified framework that treats a reference video as a prompt for semantically controlled video generation. It enables multiple controls such as concept, style, motion, and camera within a single model, achieving strong generalization and competitive performance.", "questions": "Refer to Major part in Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper proposes a unified “video-as-prompt” paradigm that reframes semantic control in video generation as an in-context learning problem, offering a clear conceptual advance over task-specific approaches.\n2. The temporally biased RoPE effectively avoids pixel-level copying between reference and target videos, leading to more robust semantic alignment.\n3. The Mixture-of-Transformers design enables plug-and-play integration with existing video diffusion transformers while preventing catastrophic forgetting.", "weaknesses": "Major:\n1. The paper lacks a theoretical analysis explaining why in-context learning via Mixture-of-Transformers effectively transfers semantic patterns.\n2. The proposed temporally biased RoPE is only justified empirically, without an ablation or analytical study on the optimal bias magnitude.\n3. The inference cost roughly doubles due to the dual-transformer structure, yet efficiency and scalability trade-offs are not thoroughly studied.\n\nMinor:\n1. The semantic diversity in VAP-Data is constrained to four categories, leaving out high-level semantics such as narrative or causal events; but the constructed dataset is the largest to date for semantically controlled video generation and provides a valuable foundation for future research.\n2.  The work does not analyze failure cases quantitatively, leaving unclear when and why semantic leakage or identity loss occurs.\n\nSome Typo issues:\n1. Line 131: Add a space in \"architectures(Singer et al., 2022; …)\" to \"architectures (Singer et al., 2022; …)\".\n2. Line 157: Add a space in \"Concept(Liu et al., 2025; …)\" to \"Concept (Liu et al., 2025; …)\".\n3. Line 160: Add spaces in \"Camera Movement(He et al., 2024; …)\" and \"Motion(Zhao et al., 2024; …)\" to \"Camera Movement (He et al., 2024; …)\" and \"Motion (Zhao et al., 2024; …)\".\n4. Lines 239 and 390: Ensure consistent usage of “fine-tuning” instead of “finetuning”.\n5. Line 300: Add a space in \"480×720(832)\" to \"480×720 (832)\".\n6. Line 321: Add a space in \"resource(see Sec. 3.2 …)\" to \"resource (see Sec. 3.2 …)\".\n7. Line 1026: Correct the typo \"metics\" to \"metrics\".\n8. Line 1240: Correct capitalization \"LoRa\" to \"LoRA\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4BWbnYsXpt", "reviewer": "Reviewer_sfsG", "summary": "The paper presents a unified framework for efficiently compressing large-scale video datasets into compact synthetic ones while preserving both spatial and temporal dynamics. Unlike prior two-stage video distillation methods such as VDSD and IDTD, which are computationally heavy and fail to model motion coherence, the proposed approach introduces a uni-level optimization framework enhanced by a Temporal Saliency-Guided Filter (TSGF). This filter leverages inter-frame differences to guide the distillation process, adaptively constraining optimization to retain key motion cues and suppress redundant frames, and further employs temporally guided augmentation to enhance diversity without breaking temporal continuity. Extensive experiments on benchmarks including MiniUCF, HMDB51, Kinetics-400, and SSv2 demonstrate state-of-the-art performance and efficiency gains, achieving significant improvements under extreme compression while maintaining strong temporal consistency, thus offering a scalable and effective solution for video dataset distillation.", "questions": "See Weakness.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.\tThe paper introduces a uni-level video dataset distillation framework that eliminates the complexity of prior bi-level methods such as VDSD and IDTD. This design effectively reduces computational time and memory consumption while maintaining high performance.\n2.\tA novel TSGF module is proposed to guide the distillation process by computing inter-frame differences, which helps preserve and enhance temporal coherence. This ensures that essential motion information is retained throughout the optimization process.\n3.\tThe authors further design a temporal-aware data augmentation technique that enhances the diversity of synthetic videos without disrupting their temporal consistency.\n4.\tThe framework demonstrates robustness across datasets of different scales and domains, offering a scalable and effective paradigm for video dataset distillation.", "weaknesses": "1.\tSome figures (e.g., Fig. 1, Tab. 2) contain typographical errors and could benefit from clearer annotations or more consistent formatting to enhance readability.\n2.\tThe Temporal Saliency-Guided Filter (TSGF) relies on simple inter-frame differences to estimate temporal importance, which may be sensitive to noise, static scenes, or camera motion. A discussion of robustness or potential failure cases would strengthen the work.\n3.\tThe reference format are inconsistencies.\n4.\tAlthough the paper claims lower computational cost, it does not provide detailed runtime or memory comparisons under consistent hardware conditions. Quantitative evidence for scalability would make the efficiency claim more convincing.\n5.\tThe claimed efficiency of the uni-level framework may be overstated, as bi-level methods can reuse pre-trained image models or distilled datasets, potentially achieving lower cost and stronger spatial representations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U3wa0RTuyh", "reviewer": "Reviewer_v4tD", "summary": "The paper introduces Video-As-Prompt (VAP), a unified framework for semantic-controlled video generation. Specifically, VAP treats the input video as a semantic prompt, which is used to guide a frozen Video DiT by a plug-and-play Mixture-of-Transformers (MoT) expert. VAP includes three key innovations: 1) in-context MoT for bidirectional information fusion between reference and target videos; 2) a temporally biased RoPE to remove false pixel-wise alignment priors and improve temporal coherence; 3) VAP-Data, a large-scale dataset with over 100K paired videos across 100 semantic conditions (e.g., concept, style, motion, and camera). The experiments show that VAP achieves a 38.7% user preference rate, achieving comparable results with commercial models (e.g., Kling and Vidu), and outperforming existing open-source models.", "questions": "- Could the authors provide more samples using out-of-distribution or real-world videos as the reference semantic control?\n- Could the authors explain why the proposed modules only provide marginal improvements over the baseline model?\n- Could the authors provide the discussion and potential solution regarding the performance drop when reference and target captions/objects mismatch?\n- Since the collection of VAP-Data is claimed as one major contribution, will the dataset be released to the community?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The most impressive strength of the work is the wide coverage of diverse semantic control tasks, including the control on concept, style, motion, and camera, under a single framework. To the best of my knowledge, VAP is the first model to achieve this level of task unification for IV2V generation which can substantially enhance the community interest.\n- While the proposed architecture looks like ControlNet (i.e., a trainable copy alongside of a frozen model), the proper adaption to this task (as shown in Figure 2) sounds novel and makes sense to me. Since the main goal is similar (i.e., ControlNet handle 2D spatial control from reference images while VAP extends to 3D temporal control from reference videos), but the proper adaption could make the model better generalize to different tasks.\n- The proposed temporally biased RoPE is also elegant and sounds well-motivated to me as it can avoid 1-to-1 positional mapping between reference and target frames which would otherwise cause false pixel matching and copy-paste artifacts.\n- I have checked the generated video samples provided in the supplementary material and they look high-quality. In addition, based on the quantitative results provided in Table 1, the model matches or even exceeds commercial models (e.g., Kling and Vidu) on several metrics, which show a promising and strong performance.\n- The paper is well written and easy to follow. The figures are well-plotted and informative which can make readers quickly understand the core ideas.", "weaknesses": "- My major concern is the construction of VAP-Data, which is entirely synthesized from the existing models or checkpoints (such as Kling, Vidu and the LoRAs from the community). As a result, VAP may overfit to the semantic controls which have already been supported by the off-the-self models. Under this perspective, the only contribution of the paper becomes an architecture design to unify the existing capabilities instead of the generalization to other novel semantic control. It could be insufficient for an ICLR-level contribution.\n- While the authors claim VAP can achieve zero-shot generalization in Figure 7, the demonstrated effects (such as “crumble,” “dissolve,” “levitate,” and “melt”) are already supported by the existing models (e.g., Pika, Kling). It is unclear how truly novel these conditions are for the zero-shot evaluation.\n- The real-world evaluation is also limited. To verify the generalization of the model, the authors can consider selecting some famous movie/TV shows/meme clips with interesting semantic control as the references. Such effects could serve as stronger out-of-distribution and real-world references, which can more robustly evaluate the generalizability of the model.\n- While the authors provide extensive ablation studies in Table 2, the impact of each proposed component seems very marginal. Could the author provide more theoretical analysis on this? Otherwise, it is unconvincing that the architecture of VAP is promising.\n- While I appreciate the authors provide thorough limitation analysis in Appendix E, the performance degradation looks significant when the reference and target captions/objects mismatch. It could significantly limit practical usage, since the consistency of reference and target captions/objects cannot always be guaranteed. Could the authors explain the potential reason and solution for this issue? Could re-captioning videos or randomly combining videos with different objects but same effect mitigate this issue?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)