UniVideo: Unified Understanding, Generation, and Editing for Videos
Authors: Cong Wei
Quande Liu
Zixuan Ye
Qiulin Wang
Xintao Wang
Pengfei Wan
Kun Gai
Wenhu Chen
Venue: ICLR 2026 Poster (2026)
Keywords: diffusion;multimodal generation;image generation;video generation;image editing;video editing
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: a unified multimodal models that can handle diverse multimodal tasks

Abstract: Unified multimodal models have shown promising results in multimodal content generation and editing, but remain largely limited to the image domain. In this work, we present UniVideo, a versatile framework that extends unified modelling to the video domain. UniVideo adopts a dual-stream design, combining a Multimodal Large Language Model (MLLM) for instruction understanding with a Multimodal DiT (MMDiT) for video generation. This design preserves the MLLM's original text generation capabilities, enables accurate interpretation of complex multimodal instructions, and maintains visual consistency in the generated content. Built on this architecture, UniVideo unifies diverse video generation and editing tasks under a single multimodal instruction paradigm and is jointly trained across them. Extensive experiments demonstrate that UniVideo matches or surpasses state-of-the-art task-specific baselines in text/image-to-video generation, in-context video generation and in-context video editing. Notably, the unified design of UniVideo enables two forms of generalization. First, UniVideo supports task composition, such as combining editing with style transfer, by integrating multiple capabilities within a single instruction. Second, even without explicit training on free-form video editing, UniVideo transfers its editing capability from large-scale image editing data to this setting, handling unseen instructions such as changing the environment or altering materials within a video. Beyond these core capabilities, UniVideo also supports generation with thinking, where the MLLM interprets complex prompts
and guides the MMDiT during synthesis. To foster future research, we released our model and code.

UnIVIdEO: UNIFIED UNDERSTANDING, GENERA- TION, AND EDITING FOR VIDEOS
Cong Wei
Quande Liu
Zixuan Ye²
Qiulin Wang²
Xintao Wang*
Pengfei Wan
Kun Gaf
Wenhu Chen
University of Waterloo
2Kling Team, Kuaishou Technology
ABSTRACT
Unified multimodal models have shown promising results in multimodal content understanding and generation but remain largely limited the image domain. In this work, we present UniVideo, a versatile framework that extends unified modeling the video domain. UniVideo adopts dual-stream design, com- bining a Multimodal Large Language Model (MILM) for visual understanding with Multimodal DiT (MMDiT) for visual generation. This design preserves the MLLM's original text generation capabilities, enables accurate interpretation of complex multimodal instructions, and maintains visual consistency in the gen- erated content. Built this architecture, UniVided Video unifies diverse video gen- eration and editing tasks under a single multimodal instruction paradigm and is jointly trained across them. Extensive experiments demonstrate that UniVide deo matches or surpasses state-of-the-art task-specific baselines in visual understand- ing, text/image-to-vide
generation, in-context video generation and in-context video editing. Notably, the unified design of UniVideo enables two forms of generalization. First, UniVideo supports task composition, such as combin- ing editing with style transfer, by integrating multiple capabilities within single instruction. Second, even without explicit training on free-form video editing, UniVideo transfers its editing capability from large-scale image editing data this setting, handling unseen instructions, such as changing the environment or al- tering materials within a video. Beyond these core capabilities, UniVide deo also supports generation with thinking, where the MLLM interprets complex prompts and guides the MMDiT during synthesis. To foster future research, released our model and code at https:/ggituub.oo//iinnn Teamm Video!
INTRODUCTION
A long-term goal of multimodal Al assistants to build models that can seamlessly understand diverse inputs across modalities and generate outputs in kind, enabling natural communication through language, images, and video demonstrations.
Recent advances in unified models suggest that this vision is increasingly attainable. Prior work
(Shi et al. 2024a; Pan et al., 20255 Sun et al. 2023] Team 2024 Tong et al. 2024; Wang et al. 2024b Deng et al. 2025 Wu et al., 2025b, Ma et al. 2025bb Xie et al. 2024 2025 Zhou et al., 2024) has demonstrated promising results in text-image understanding and generation by jointly optimizing these capabilities within unified systems. More recently, models such Google Nano banana and GPT-image-1 have pushed this paradigm further by integrating computer vision, image manipulation. and multimodal reasoning into single framework, marking shift from specialized endality generators toward powerful unified systems.
Despite this progress, unified understanding -entiittionnnnnonn models remain limited to text and image (Lin et al., (0225] Wu et al.) 2025c), leaving video largely underexplored. Existing video generation models primarily address single text-to-video task and rely on text encoders to process instructions
WWok done during an internship at Kling Team, Kuaishou Technology.
Cooresponning authors.

---

Text Videol Image Video Generation
Video Understanding
The ridveless poen
"Rase prosdd adeaaled capice siday"
uo Magle dortec
Context Generation
"Cenanae video mas drowed vitran bongg
The tuan lodda La cocas lake.
Pllaii weal denctioo priches.
"Coee video rees regrosed
Theacting the sacase of cinglitods abbratoy.."
consosatioe.
Generation with Thinking
"Geaerae video loned on the ponpee"
The generated video ahould fndd shows man a green sutt sis in meaddu of yullow frweess Nust brown monkey congeng tightl to ropa comes
rppliing the
The sequence opens with dynamio iow-angle trecking shot Suckdenly explosion erupts Aeeal Lamborgnin)
pronpe"
Context Editing
video with the sas informe image"
Replica the Sockman mide seperna inage
Free-form Editing
"Mide the
"Change the of the sitzo nisecratt
Figure I: UniVideo unified system that can understand multi-modal instructions and gener- ate multi-modal content. More videos are available on https:/ccongeil1333333 2330.githu..oo Uni Video
(Wn al. 2025 Ju et 2025b Polyak et al. 2024 Kong 2024 restricting their ability to understand and reason over multimodal instructions (Hu et 20.4)) Meanwhile, video editing methods typically employ task-specific modules or pipelines (Ku al. 2024 Jiang et al. 2025 Ye et al., (0255), which makes it difficult to scale across diverse tasks. Consequently, due to the lack of unified modeling, advanced capabilities such as multimodal prompting, in-context video generation, and sophisticated free-form editing remain beyond the reach of any single model.
Motivated by these limitations, we present UniVideo -a unified framework for understanding, generation, and editing in the video domain. UniVideo bridges this gap by enabling multimodal instruction following and delivering robust performance across diverse video tasks.
To build UniVideo, we propose two-stream design, where MLLM serves as the understand- ing branch and an MMDiT backbone (Esser et al. 2024) serves as the generation branch. While prior work such Qwen-Image (Wu et al. 2025a) explores similar idea in the image domain, model generalizes this design to video. Both streams now receive image and video instructions: the understanding branch through semantic encoder, and the generation branch through VAE-based
2

---

Understanding
Generation
MMDiT
MLLM
Self Attention
Generate
MLP
VAE
video of the san holding the cat in front of the house
Noloy
Inages
Video
Text
Images
Figure 2: Model architecture. UniVidec is dual-stream model consisting of an MLLM for understanding and an MMDiT module for generation. While prior work such as Owen-Image and OmniGen2, explores similar idea in image domain, our model generalizes this design video.
encoders. In contrast, prior unified models such as GPT-image- (Lin et al. 2025) rely exclusively on semantic encoders, which often struggle to capture fine-grained visual details. Similarly, bot- tlenecked approaches using learnable query tokens (Tong et al. 2024 Pan et al., 2025) compress inputs into fixed set of tokens, creating severe capacity bottleneck when instructions contain videos. As a result, both approaches fall short in supporting in-context video generation. Our design preserves the multimodal reasoning capabilities of the MLLM while enabling the model to handle diverse video tasks with multimodal inputs. Moreover, it ensures cross-stream consistency, which is crucial for precise editing and for maintaining subject identity in in-context generation.
Based on this unified architecture, we train UniVided across a wide spectrum of tasks, including -imag,, text-to-video, image-to-video, in-context video generation, in-context video editing, and image editing, As a unified system, UniVideo not only understands multimodal instructions and distinguishes between tasks but also achieves improvements methods. Thanks to unified training, UniVi dedo generalizes to novel task compositions unseen dur- ing training, such as deleting one identity while swapping another within a single instruction. More importantly, although Un V deo is not trained free-form video editing data, it demonstrates generalization ability transfer from image editing to free-form video editing (e.g., changing object materials modifying weather conditions), highlighting the effectiveness of our unified video un- derstanding and generation framework.
state-of-the-an task-specific
Furthermore, UniVider Vio retains the strong visual understanding and text generation capability of its underlying frozen MILM. By leveraging the MLLM's autoregressive reasoning and language generation abilities, UniVider Video can effectively interpret ambiguous and complex multimodal in- structions that require joint vision-language understanding, such as reasoning over visual prompts before performing generation. Since Un v deo's 's text generation ability originates from frozen MILM,, Un Univicn deo should be regarded as a post-trained unified multimodal generative system (Wu al. 2025c Pan al. (025)) capable of producing images, videos, and text, rather than unified model trained from scratch Ma et al. 2025b Deng et al. 2025))
Our key contributions
I) We introduce UniVideo, multimodal generative model that unifies understanding, generation, and editing videos within a single framework. To build UniVideo. we propose dual-stream architecture that combines the multimodal reasoning capabilities of the MLLM with the generation strengths of the MMDIT. Unlike prior task-specific or modality-restricct approaches, Un v deo can interpret multimodal instructions, distinguish between diverse tasks, and achieve state-of-the-an performance across a wide range of benchmarks.
2) We systematically study the key design choices that enable this unified framework, including connector architectures, generator designs, and multimodal conditioning strategies, and provide em- pirical evidence for their effectiveness.
3) We demonstrate that Un Vi deo generalizes to unseen tasks and novel task compositions without ad hoc designs, highlighting the benefits of unified framework.
3

---

MMDiT
MLLM
Self Attention
"Generate cense capt ion that reflects the visual image"
"The video begins with dynasic low-angle tracking shot skiniing-
MLP
VAE
"Then susdenly, sleek sports car bursts into frane, its glossy black body accented with sharp neon-green details_"
<irage
<Teage
<Ieage 4>
Taneer
Figure 3: Generation with thinking. UniVideo leverages the MLLM stream to understand and interpret user intent from complex multimodal prompts that cannot be handled by the DiT alone. For example, users can provide diagrams or visual annotations to guide video generation without writing dense textual prompts.
METHOD
2.1 MODEL ARCHITECTURE
As demonstrated in Figure 2] UniVideo consists of two main components: multimodal large language model (MLLM) and a multimodal DiT (MM-DiT). The MLLM handles visual-textual understanding, taking text, image, and video inputs and optionally producing text responses. The MM-DiT focuses on visual generation with two branches: one incorporates high-level semantic information from the MLLM, while the other integrates fine-grained reconstruction signals from VAE. Specifically, we extract the last-layer hidden states of the MLLM, which encode rich semantic features of the multimodal input. These are aligned to the input space of the MM-DiT via a trainable connector and fed into its understanding stream. In parallel, visual signals are encoded by the VAE and passed into the MM-DiT generation stream to preserve fine details. This design enables strong semantic grounding together with high-fidelity visual detail, which is especially important for video generation. We provide a model design analysis in subsection D.I.
editing and
2.2 UNIFYING MULTIPLE TASKS
We unify diverse multimodal tasks through natural language instructions, as illustrated in Figure I For texi-to-video (T2V), the text input is processed by the MLLM, while the noisy video is fed into the MM-DiT. For image-to-vided (12V), both the image and text are processed by the MLLM, whereas the image and noisy video are provided the MM-DiT. For in-context video generation (MultilD2V) and in-context video editing (ID-V2V), multiple visual conditions are often available, such as several reference images together with a reference video. Each visual signal encoded with the VAE, padded to uniform shape, concatenated along the temporal axis, and then processed with self-attention. Unlike prior approaches that introduce task-specific bias embeddings (Ye et al. 2025b) context adapter modules (Jiang et al. 2025), we avoid task-specific customization. To help the MM-DiT distinguish between condition latents and noisy video latents, we apply 3D positional embeddings, which preserve the spatial indices across frames while incrementing only the temporal dimension. In practice, we find this strategy more effective than Qwen2-VL's MRoPE (Wang et al., 2024ah, which offsets all axes whenever new visual input is introduced.
2.3 GENERATION WITH THINKING
UniVideo leverages its MLLM branch to interpret unconventional or hand-crafted prompts, as illustrated in Figure 3 and Figure 6 For example, users may provide input image with man- ual annotations, which the MLLM translates into a structured plan and dense prompt tokens that guide video generation. Unlike agent-based approaches that invoke multiple downstream genera- tors, UniVides offers a more simplified design: the MMDiT directly integrates embeddings from the dense prompt tokens produced by the MI..M.
4

---

Two dressed in sharp, tactical agent attire, advance purposefully towards the camera. Each man holds the distinctive ray gun.
In Context Genaration
VΑCE
A man dressed in vibrant Bawaiian shirt with a colorful floral pattern, sits beach an ice cream cone, taking bite.
lounge
perches.
Kling 1.6
Add the hat from the reference image to the video.
Context Editing
Hi
PIKA
Let the woman have the hair style in the reference image
UNIC(with mask)
UniVideo
Figure 4: Qualitative comparison of Uni Univideo with SoTA Task Specific Experts on In Context
Generation and In Context Editing tasks.
2.4 TRAINING STRATEGY
Stage 1. Connector alignment between MLLM and MMDIT. In this stage, we train only the MLP connector while keeping both the MLLM and MMDiT frozen. Training is performed on pretraining samples across text-to-image (T2I) and text-to-video (T2V) generation tasks, as well as an image- reconstruction task which only images from the text-to-image dataset are fed into the MILM and the MMDiT reconstructs the image using visual features from the MILMM. After this stage, UniVideo can generate images and videos conditioned on text or image inputs from the MLLM.
Stage 2. Fine-tuning on T21/T2V. In this stage, we keep the MLLM frozen and fine-tune the connector and MMDiT on small-scale, high-quality T21/T2V samples. After this stage, UniVideo achieves performance comparable the MMDiT backbone that uses its text encoder.
Stage 3. Multi-task Training. Finally, we extend training to include in-context generation (multi- (D-to-vicoo), in-context video editing (modifying the input video based reference image, such as ID swapping, ID addition, ID deletion, or style transfer), image editing and image-to-video tasks, alongside the previous text-to-image (T2I) and text-to-video (T2V) tasks. We keep the MILM frozen and only train the connector and MMDiT. This stage enables UniVideo to unify a broad range of video generation and editing tasks under multimodal instruction. Details of training setting is provided in Appendix Table 5
EXPERIMENTS
In this section, we first describe the implementation details of UniVided in subsectton 3.1 Then, we present the main results in |subsection 3.2] We conduct a comprehensive benchmark of UniVideo with SoTA methods across a broad spectrum of video understanding and generation tasks. Our results show that UniVideo's strong unified capabilities across all settings. Next, demonstrate the zero-shot generalization ability of UniVideo and analyze the visual prompt un- derstanding ability in iusubsection 3.3 Finally, we validate the design choices of Uni anivided through ablation studies in subsection 3.4] We discuss the design choices for aligning the MLLM and the Diffusion generator in Appendix subsection D.1
5

---

Table 1: Quantitative comparison on Visual Understanding and Video Generation. Best results are shown in bold, and second-best are underlined. *Understanding results for UniVideo are reported using its MLLM backbone (Qwen-2.5VL-7B).
Model
Understanding
Video Generation
MMB
MMMU
MM-Vet
Vbench T2V
Video Understanding Model
LLaVA-1.5 Liu et al. 2024a
36.4
67.8
36.3
X
LLVA-NeXI al. 2024b
79.3
51.1
57.4
Video Generation Model
(1g Yang et al. 2024
x
81.61
I2VGen-XI. al. 2023c
X
Kong KKong Step-Video- (o2VViiooo(2))MM of al. 2025a
HunnyannVideof (2V)
2024
83.24
X
81.83
Wan2.1(T2V) Wan et al.
2025
X
84.70
Unified Understanding & Generation Model
Emu3 (Wang et
2024b
58.5
31.6
37.2
80.96
TokenFlow> Tow-Xl XW- Qu ct al.
2025
76.8
43.2
48.2
X
Janus
Wu et al.
2025b
69.4
30.5
34.3
JanusFlow (Ma ct al. 2025b
74.9
29.3
30,9
X
OmniGen2
Wu et
2025c
79.1
53.1
61.8
Show-o Xic al. 2024
26.7
BAGEL
(Deng et al.
2025
85.0
55.3
67.2
X
Show-o2
Xie et al.
2025
79.3
48.9
56.6
81.34
UniVideo
83.5
58.6
66.6
83.48
3.1 IMPLEMENTATION DETAILS
We adopt Awen2.5VL-7B Bai 2025b as the MILM backbone and Hunyuan Video-T2V 13B (Kong et al.) 2024) as the MMDiT backbone. The original Hunyuan Video use two text en- coders; we remove them and instead use qwen2.5VI. as the unified multi-modal embedder. To align feature dimensions between qwen2.5VL. and Hunyuan we apply an MLP with a 4x expan- sion. Additional details are provided the Appendix.
3.2
MAIN RESULTS
3.2.1
VISUAL UNDERSTANDING AND GENERATION
UniVideo 's visual understanding is powered by frozen pretrained MILM. Freezing the MLLM preserves its strong native understanding ability and prevents performance degradation from joint training with generative tasks. As shown in Table I UniVideo achieves competitive scores of 83.5 MMBench (Liu al. 2024e). 58.6 on MMMU Yue al. 2024). and 66.6 MM- Vet Yu et al. 2023) for understanding tasks. At the same time, retains strong generation ability, supporting both 12V and T2V within single unified model. In contrast, baseline models rely different vari- ants for different tasks, whereas Un iinivice deo reaches performance comparable to the Hunyuan Video backbone on the VBench(Huang al. 2024) benchmarks.
3.2.2 IN-CONTENT VIDEO GENERATION
Benchmark:
Following FulIDiT (Ju et al. 2025b) and OmniGen2 [Wu et al. (025c), construct test set covering both single-ID and multi- ID video generation scenarios. In the single-ID setting, subject may have multiple reference images (e.g., different viewpoints of person object). In the multi-ID setting, the references include 2-4 distinct identities. Details are provided in the Appendix.
Metrics: We conduct both human evaluations and automatic metric assessments. For human eval- uation. we follow the protocols of Instruct- Imagen (Hu et al.) (024a) and OmniGen2 (Wu et al.) 2025ch perform systematic study. Each sample is rated by least three annotators on (i) subject
Instruct-Imagen

---

Table 2: Quantitative comparison on In-Context Generation. Human evaluation includes Subject Consistency (SC), Prompt Following (PF), and Overall Video Quality (VQ). Automatic metrics measure Smoothness and Aesthetics. Best results in bold, second-best underlined.
Single Reference
Multi Reference (22)
Model
SCT
PFt
VQf
Smooth
Aesth
SCT
PFf
VQt
Smooth
Aesth
VACE
0.31
0.65
0.42
0.922
5,426
0.48
0.53
0.48
0.530
5.941
Kling 1.6
0.68
0.95
0.88
0.938
5,896
0.73
0.45
0.95
0.916
6.034
Pika2.2
0.45
0.43
0.15
0.928
5.125
0.71
0.48
0.43
0.898
5.176
UniVideo
0.88
0.93
0.95
0.943
5.740
0.81
0.75
0.85
0.942
6.128
consistency (SC), (ii) prompt following (PF), and (iii) overall video quality (VQ). Scores in each category are drawn from {0, 0.5. 1), where indicates inconsistency or extremely poor quality, and indicates full consistency or high quality. For automatic evaluation, we adopt three metrics from VBench (Huang et al. 2024): smoothness, and aesthetics.
Baselines: We compare UniVi deo with the state-of-the-art open-source model VACE, given the scarcity of video models capable of in-context generation. We also include commercial baselines such as Pika2.2 and K.ingl.6.
UniVide achieves superior com- petitive performance across all metrics compared to the baselines. Additional results are shown in Figure 4, and more examples available our project website. Notably, baseline models often struggle with complex instructions involving multiple identities (e.g., when the number of reference images 4), whereas Un Video accurately follow instructions while preserving identity.
Results:
Quantitative comparisons presented in Table 2
Change of the dancer's to green
Unseen Tasks and Instructions
Make the and man dancing
Transform landscape into vibrant autumn
Change the material of the chocolate
Figure 5: Zero-Shot Generalization. We demonstrate two type of generalization. Un V dideo was not trained on General Free-form Video Editing data. It transfers this ability from diverse image editing data the video domain through joint training with in-context video generation and editing data (limited to ID deletion, swapping, addition, and stylization), enabling it to handle previously unseen video editing instructions. (ii) UniVi deo can also generalize to novel task compositions, even though it was not explicitly trained on such compositions.
3.2.3 IN-CONTEXT VIDEO EDITING
Benchmark: Following UNIC Ye et al. 2025b), construct test set covering four editing types: swap, delete, addition, and style transfer. Each example consists source video and a reference image, together with natural language instruction. Further details are provided in the Appendix.

---

Table 3: Quantitative comparison with task-specific expert models on In-Context Video Editing. Our model is the only mask-free approach, capable of performing edits solely based on instructions without requiring explicit mask inputs to indicate editing regions. Despite this more challenging setting, it achieves superior competitive performance across all metrics compared to state-of-the- art task-specific expert baselines. Best scores are shown bold, and second-best are underlined.
In Context Insert
Model
Identity
Alignment CLIP-scoret
Video Quality
CLIP-I1
DINO-I
Smoothness
Aesthetic
VACE
0,513
0.105
0.103
0.947
5.693
UNIC
0.598
0.245
0.216
0.961
5.627
Klingl.6
0.632
0.287
0,246
0.993
5.798
Píka2.2
0.692
0.399
0.253
0.951
5,591
UniVideo (Mask Free)
0.693
0.398
0.259
0.943
6.031
Context Swap
Model
Identity
Alignment CIIIP-ccoref
Video Quality
CLIP-T
DINO-I
Smoothnesst
Aesthetic
VACE
0.703
0.391
0.218
0,960
5.961
UNIC
0.725
0.429
0.242
0,971
6.056
Kling1.6
0.707
0.437
0.211
0.995
6.042
Pika2.2
0.704
0.406
0.211
0.967
5.097
Any
0.605
0.229
0.218
0.917
4.842
UniVideo (Mask Free)
0.728
0.427
0.244
0.973
6.190
In Context Delete
Model
Video Reconstruction
Alignment CLIP-scoret
Video Quality
PSNR1
RefVideo-CUi
Smoothnesst
Aesthetic
VACE
20.601
0.874
0.206
0.968
5.637
UNIC
19.171
0.817
0.217
0.970
5.493
Klingg.6
15.476
0.888
0.208
0.998
4.965
Any V2V
19.504
0.869
0,205
0.964
5.325
VideoPainter
22.987
0.920
0,212
0.957
5.403
Uni UniVideo (Mask Free)
17.980
0.888
0.214
0.971
5.498
In Context Stylization
Model
Style & Content
Alignment CLIP-scoret
Video Quality
CSD-Scoref
ArtFID]
Smoothnesst
Aesthetic
Any
0.207
43.299
0.195
0,937
4,640
StyleMaster
0.306
38.213
0.188
0.952
5.121
UNIC UniVideo (Mask Free)
0.197
36.198
0.215
0.932
5.045
0.228
37.877
0.226
0.963
6.281
We adopt the evaluation protocol of UNIC eY al. 20256) and conduct automatic metric assessments. Specifically, we use CLIP-I and DINO-I to measure identity consistency, and CLIP- Score to measure prompt following.
Metrics:
We compare inivideo with state-of-the-arr task-specific expert models, including UNIC, V2V, and VideoPainter. We also evaluate against commercial models such as Pika2.2 and Klingl.6. Note that all baseline models require explicit mask inputs to localize editing regions and guide generation, whereas UniVideo operates without masks.
Baselines:
Although UniVideo is evaluated under the more challenging mask-free setting, it still achieves superior or competitive performance across all metrics compared to the basclines. Additional results shown in Figure 4 and further examples are provided on our project website. UniVideo can accurately follow instructions while preserving the identity of the reference images.
Results: Quantitative comparisons are presented in Table 3
8

---

3.3 MODEL ANALYSIS
3.3.1 ZERO SHOT GENERALIZATION
We observed type of generalization ability of UniVideo. Although the training data of UniVideo does not include general free-form video editing tasks, it transfers this ability from diverse image editing data and in-context video editing data (limited ID deletion, swapping, ad- dition, and stylization) to the video domain, enabling it to handle free-form video editing instruc- g.... changing material or environment). Surprisingly, we find that UniVideo can perform tasks such as changing materials of character. We also observe that UniVideo is capable of han- dling task compositions. can combine in-context editing with style transfer, or perform multiple edits simultaneously (e.g., deleting one identity while adding another). Demonstrations in Figure 5
Prompting
Figure Qualitative results of generation with thinking. We illustrate two types of visual prompts: in the first three examples, annotations are drawn on canvas, while in the last exam- plc, the annotation is drawn directly on an input image.
3.3.2 GENERATION WITH THINKING
We demonstrate Un Vi deo's generation-wiih--hiiniii capability in Figure 6 focusing on visual prompt understanding that requires structured multimodal reasoning. We consider two representa- tive types of visual prompts. In the first setting, users draw reference images and story plans on canvas. Here, the model can interpret the plan and generate corresponding videos. In the second setting, annotations are drawn directly on an input image, which the model treats as an I2V task; in this case, CniVideo can interpret the motion These results highlight the advantages of Un Uniiidd video in handling complex multimodal instructions. Although the qualitative results are obtained in zero-shot setting, future end-to-end training task-specific data may further improve performance.
events described by the visual prompt.
3.4 ABLATION STUDY
Our ablation studies address the following two questions: (i) Does multi-task learning enhance per- formance compared with single-task learning? ? (ii) our two-branch design effective? Specifically, should visual embeddings be streamed to both the MLLM and MMDiT branches?
We conduct human evaluations on In-Context Video Editing and In- Context Video Generation, using the same evaluation protocol in insuubsubecttinn 3.2.2 (i) To study multi-task learning, compare Uni Video with single-task baseline. The single-task baseline shares the same architecture as Un i Vi but requires independent model for each task and has access only to task-specific

---

Table 4: Ablation study across single-task model, UniVideo, and UniVideo w/ Visual for MMDiT across different In-Context tasks.
Single-task
UniVideo
UniVidec wio Visual for MMDIT
PFt
SCT
VQt
PFT
SCT
VQt
PP
SCT
VQT
singleid multiid
0.85
0.83
0.93
0.93
0.88
0.95
0.75
0.32
0.86
In-context generation
0.75
0.79
0.73
0.75
0.81
0.85
0.81
0.23
0.83
insert
0.81
0.85
0.86
0.92
0.92
0.91
0.68
0.18
0.75
In-context editing
swap
0.53
0.78
0.68
0.91
0.85
0.85
0.63
0.15
0.62
delete
0.32
0.42
0.89
0.52
0.58
0.92
0.21
0.13
0.63
stylization
0.56
0.43
0.63
0.79
0.64
0.64
0.86
0.11
0.57
Average
0.64
0.67
0.79
0.80
0.78
0.85
0.66
0.18
0.71
demonstrate the effectiveness of multi-task learning, especially for the editing task, where UniVida Video benefits from large-scale image editing data during joint learning. (ii) To evaluate the impact of streaming visual inputs, compare UniVider Video with variant that share the same architecture: w/o visual for MMDiT: visual inputs are fed only to the MLLM branch. As shown in Table feeding visual inputs exclusively the MILM results in dramatic drop in identity preservation.
data. Results in Table 4
RELATED WORK
Unified Multimodal Understanding and Generation. Recent progress in multimodal generation has been driven primarily by the text and image domains, spanning autoregressive modeling. diffu- sion-autoregression hybrids, and LLM-based regression approaches(Sun et al. 2024a Team 2024
Sun et al. 2024b Wang et al. 2024b). While these advances demonstrate strong capabilities in images, unified approaches beyond the image domain remain limited. We instead present a unified
video model. A full discussion of prior multimodal works is provided in Appendix /ubsecction C.]
Image/Video Generation and Editing. Diffusion models have achieved remarkable success in image and video SonthesissRombaee et al. 2022 Pt 2023 Esser et 2024| Ramesh al.
2021 Saharia et al. (2022) and unified image editing systems (Xiao et al. [025 Tan et al.) 2024 Chen al. 2025d). In contrast, the video domain remains dominated by single-task frameworks. Attempts at unification( Ku et al. 2024,] Ju et al., 20255 Jiang et al. (025) still require task-specifie
pipelines or modules. We bridge this gap by unifying diverse video tasks under single framework.
Extended related work in Appendix (subsection C.2
CONCLUSION
We introduce UniVideo, a unified multimodal generative model for video understanding, gener- ation, and editing. By integrating an MLLM for semantic understanding with an MMDiT for gen- eration, UniVideo combines strong multimodal reasoning with fine-grained visual consistency. can interpret multimodal instructions and handle diverse tasks effectively. Our experiments show that Cnivideo not only matches or outperforms task-specific baselines across tex/image-to-viiee video editing, and in-context generation, but also generalizes to unseen tasks and novel task com- positions-capallee capabilities that specialized pipelines struggle to achieve. Beyond robust performance, UniVideo can also support visual prompting understanding, underscoring the advantages of uni- fied modeling over fragmented approaches. Looking forward, UniVideo opens new directions for multimodal research, advancing us toward assistants that can naturally communicate through language, images, and video.
REFERENCES
Jianhong Bai, Menghan Xia, Xiao Fu, Xintao Wang. Lianrui Mu, Jinwen Cao, Zuozhu Liu, Haoji Hu, Xiang Bai, Pengfei Wan, et al. Recammaster: Camera--contoolld generative rendering from single video. arXiv preprint arXiv:2503. 1647, 2025a.
10

---

Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025b.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arX11:231..5117, 2023a.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 22563-225777 2023b.
Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 13392-1842, 2023.
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, et al. Video generation models world simulators. OpenAl Blog, 1:8, 2024.
Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang. Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafterl: Open diffusion models for high- quality video generation. arXiv preprint arXiv:2310. 19512, 2023.
Jiuhai Chen, Zhiyang Xu, Xichen Pan, Yushi Hu, Can Qin, Tom Goldstein, Lifu Huang, Tianyi Zhou, Saining Xie, Silvio Savarese, et al. Blip3-o: family of fully open unified multimodal archtet-rc training and dataset. arXiv preprint arXiv:2505.09550 2025a.
Junying Chen, Zhenyang Cai, Pengeheng Chen, Shunian Chen, Ke Ji, Xidong Wang, Yunjin Yang, and Benyou Wang. Sharegpt-4o-image: Aligning multimodal models with gpt-4o-level image generation. arXiv preprint arXiv:2506. 15000 2025b.
Shoufa Chen, Chongjian Ge, Yuqi Zhang, Yida Zhang, Fengda Zhu, Hao Yang, Hongxiang Hao, Hui Wu, Zhichao Lai, Yifci Hu, Ting-Che Lin, Shilong Zhang, Fu Li, Chuan Li, Xing Wang, Yanghua Peng. Peize Sun, Ping Luo, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Xiaobing Liu. Goku: Flow based video generative foundation models. arXiv preprint arXiv:2502. 04896, 2025c.
Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Yuwei Fang, Ivan Skorokhodov, Jun- Yan Zhu, Kfir Aberman, Ming-Hsuan Yang, and Sergey Tulyakov. Videoalchemy: Open-set person- alization in video generation. 2024.
Xi Chen, Zhifei Zhang, He Zhang, Yuqian Zhou, Soo Ye Kim, Qing Liu, Yijun Li, Jianming Zhang. Nanxuan Zhao, Yilin Wang, et al. Unireal: Universal image generation and editing via learning real-world dynamics. In Proceedings of Computer Vision and Pattern Recognition Conference, pp. 12501-12511, 2025d.
Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, et Emerging properties in unified multimodal pretraining. arXiv preprint 1iXX::2505.14653 2025.
Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, et al. Dreamlim: Synergistic multimodal comprehension and creation. arXiv preprint arXiv:2309. 1499, 2023.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. Forty-first international conference on machine learning, 2024.
Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models with unified multi multi-granularigg comprehension and generation. arXiv preprint Atl::2404.4396 2024.
11

---

Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object- foouuc- framework for evaluating ext-to--mmae alignment. Advances in Neural Information Processing Systems, 36: 22132-52152, 2023.
Agrim Gupta, Linxi Fan, Surya Ganguli, and Li Fei-Fei. Metamorph: Learning universal controllers with transformers. arXiv preprint 200.v:2203.11931, 2022.
Hexiang Hu, Kelvin CK Chan, Yu-Chuan Su, Wenhu Chen, Yandong Li, Kihyuk Sohn, Yang Zhao, Xue Ben, Boqing Gong, William Cohen, et al. Instruct-imagen: Image generation with multi- modal instruction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4754-4763. 2024a.
Jiahao Hu, Tianxiong Zhong, Xuebo Wang, Boyuan Jiang, Xingye Tian, Fei Yang, Pengfei Wan, and Di Zhang. Vivid-10m: A dataset and baseline for versatile and interactive video local editing. arXiv preprint arXiv:2411.15260 2024b.
Jiehui Huang, Yuechen Zhang, Xu He, Yuan Gao, Zhi Cen, Bin Xia, Yan Zhou, Xin Tao, Pengfei Wan, and Jiaya Jia. Unityvideo: Unified multi-modal multi-task learning for enhancing world- aware video generation. arXiv preprint arXiv:2512.0788 2025a.
Jiehui Huang, Xiao Dong, Wenhui Song, Zheng Chong, Zhenchao Tang, Jun Zhou, Yuhao Cheng, Long Chen, Hanhui Li, Yiqiang Yan, et al. Consistentid: Portrait generation with multimodal fine- grained identity preserving. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026.
Yuzhou Huang. Ziyang Yuan. Quande Liu, Qiulin Wang. Xintao Wang, Ruimao Zhang, Pengfei Wan, Di Zhang, and Kun Gai. Conceptmaster: Multi-concept video customization on diffusion transformer models without test-time tuning. arXiv preprint arXiv:2501.04664 2025b.
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianx- ing Wu, Qingyang Jin, Nattapol Chanpaisit, al. Vbench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21807-218 21818, 18, 2024.
Zeyinzi Jiang, Zhen Han, Chaojie Mao, Jingfeng Zhang, Yulin Pan, and Yu Liu. Vace: All-in-one video creation and editing. arXiv preprint arXiv:2503.07598, 2025.
Xuan Ju, Tianyu Wang, Yuqian Zhou, He Zhang, Qing Liu, Nanxuan Zhao, Zhifei Zhang, Yijun Li, Yuanhao Cai, Shaoteng Liu, Daniil Pakhomov, Zhe Lin, Soo Ye Kim, and Qiang Xu. Edit- verse: Unifying image and video editing and generation with in-context learning. arXiv preprint arXiv:2509.20366 2025a. URL
abs/
20360
Xuan Ju, Weicai Ye, Quande Liu, Qiulin Wang, Xintao Wang, Pengfei Wan, Di Zhang, Kun Gai, and Qiang Xu. Fulldit: Multi-task video generative foundation model with full attention. arXiv preprint 2rXiv22503.1990 2025b.
Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprini arXiv:2412.03603, 2024.
Max Ku, Cong Wei, Weiming Ren, Harry Yang, and Wenhu Chen. Anyv2v: A tuning free frame- work for any video-to-video editing tasks. arXiv preprini arXiv:2403. 14468, 2024.
Black Forest Labs, Stephen Batifol, Andreas Blattmann, Frederic Boesel, Saksham Consul, Cyril Diagne, Tim Dockhorn, Jack English, Zion English, Patrick Esser, et al. Flux. kontext: Flow matching for in-context image generation and editing in latent space. arXiv preprint arXiv:2506. 15742, 2025.
Feng Liang, Haoyu Ma. Zecheng He, Tingbo Hou, Ji Hou, Kunpeng Li, Xiaoliang Dai, Felix Juefei- Xu, Samaneh Azadi, Animesh Sinha, et al. Movie weaver: Tuning- firing-free multi-concept video per- sonalization with anchored prompts. In Proceedings of the Computer Vision and Pattern Recog- nition Conference, pp. 13146 1314-13156, 2025.
12

---

Chao Liao, Liyang Liu, Xun Wang. Zhengxiong Luo, Xinyu Zhang, Wenliang Zhao, Jie Wu, Liang Li, Zhi Tian, and Weilin Huang. Mogao: An omni foundation model for interleaved multi-modal generation. arXiv preprint arX5v:2505.05472, 2025.
Jun Hao Liew, Hanshu Yan, Jianfeng Zhang, Zhongcong Xu, and Jiashi Feng. Magicedit: High- fidelity and temporally coherent video editing. arXiv preprint arXiv:2308, 14749, 2023.
Bin Lin, Zongjian Li, Xinhua Cheng, Yuwei Niu, Yang Ye, Xianyi He, Shenghai Yuan, Wangbo Yu, Shaodong Wang, Yunyang Ge, et al. Uniworld: High-resolution semantic encoders for unified visual understanding and generation. arXiv preprint arXiv:2506.03147, 2025.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee, Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 06296-26306, 2024a.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang. Sheng Shen, and Yong Jae Lee. Llavanext: Improved reasoning, ocr, and world knowledge, 2024b.
Lijie Liu, Tianxiang Ma, Bingchuan Li, Zhuowei Chen, Jiawei Liu, Gen Li, Siyu Zhou, Qian He, and Xinglong Wu. Phantom: Subject-consistnt video generation via cross-modal alignment. arXiv preprint 2rXi::2502. 1079, 2025a.
Shaoteng Liu, Yuechen Zhang. Wenbo Li, Zhe Lin, and Jiaya Jia. Video-p2p: Video editing with cross-attention control. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1999, 2024c.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang. Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European conference on computer vision, pp. 38-55. Springer, 2024d.
Shiyu Liu, Yucheng Han, Peng Xing, Fukun Yin, Rui Wang, Wei Cheng, Jiaqi Liao, Yingming Wang, Honghao Fu, Chunrui Han, et al. Steplx-edit: A practical framework for general image editing, arXiv preprint arXiv:2504. 17761, 2025b.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, al. Mmbench: your multi-modal model an all-around player? In European conference computer vision, pp. 216-233. Springer, 2024c.
Zhiheng Liu, Weiming Ren, Haozhe Liu, Zijian Zhou, Shoufa Chen, Haonan Qiu, Xiaoke Huang. Zhaochong An, Fanny Yang. Aditya Patel, et al. Tuna: Taming unified visual representations for native unified multimodal models. arXiv preprint arXhv:2512.02014. 2025c.
Jiabin Luo, Junhui Lin, Zeyu Zhang, Biao Wu, Meng Fang, Ling Chen, and Hao Tang. Univid: The open-source unified video model. arXiv preprint 250Kv:::5092222000 2025.
Guoqing Ma, Haoyang Huang, Kun Yan, Liangyu Chen, Nan Duan, Shengming Yin, Changyi Wan, Ranchen Ming, Xiaoniu Song, Xing Chen, et al. Step-video-t2v technical report: The practice, challenges, and future of video foundation model. arXiv preprint arXiv:2502.10244: 2025a.
Yiyang Ma, Xingchao Liu, Xiaokang Chen, Wen Liu, Chengyue Wu, Zhiyu Wu, Zizheng Pan, Zhenda Xie, Haowei Zhang, Xingkai Yu, et al. Janusflow: Harmonizing autoregression and rec- tified flow for unified multimodal understanding and generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 7739-7751, 2025b.
Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi. and Ying Shan. 12i-adapter: Learning adapters to dig out more controllable ability for text-to-iman diffusion models. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pp. 4296-
4304, 2024.
Xichen Pan, Satya Narayan Shukla, Aashu Singh. Zhuokai Zhao, Shlok Kumar Mishra, Jialiang Wang, Zhiyang Xu, Jiuhai Chen, Kunpeng Li, Felix Juefei-Xu, et al. Transfer between modalities with metaqueries. arXiv preprint arXiv:2504.0665 002.0000000666... 2025.
13

---

Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 2023.
Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih- Ma, Ching Yao Chuang, et al. Movie gen: A cast media founda- tion models. arXiv preprini arXiv:2410.133: 13720, 2024.
Liao Qu, Huichao Zhang, Yiheng Liu, Xu Wang. Yi Jiang, Yiming Gao, Hu Ye, Daniel K Du, Ze- huan Yuan, and Xinglong Wu. Tokenflow: Unified image tokenizer for multimodal understanding and generation. In Proceedings of the Computer Vision and Puttern Recognition Conference, pp.
2545-2555, 2025.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ily Sutskever. Zero-shot text-to-image generation. In International conference on machine learning, pp. 8821-8831. Pmlr, 2021.
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10699 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural informa- tion processing systems, 35:36479-36494, 2022.
Shelly Sheynin, Adam Polyak, Uriel Singer, Yuval Kirstain, Amit Zohar, Oron Ashual, Devi Parikh, and Yaniv Taigman. Emu edit: Precise image editing via recognition and generation tasks. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8871- 1- 8879, 2024.
Weijia Shi, Xiaochuang Han, Chunting Zhou, Weixin Liang. Xi Victoria Lin, Luke Zettlemoyer, and Lili Yu. Lmfusion: Adapting pretrained language models for multimodal generation. arXiv preprint arXiv:2412.15188 2024a.
Yichun Shi, Peng Wang. and Weilin Huang. Seededit: Align image re-generation to image editing. arXiv preprint arXiv:2411.06683 2024b.
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint 2.Xi::2406.06555. 2024a.
Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Emu: Generative pretraining in multimodality. arXiv preprint arXiv:237..02222 2023.
Quan Sun, Yufeng Cui. Xiaosong Zhang, Fan Zhang, Qiying Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 14499, 2024b.
Zhenxiong Tan, Songhua Liu, Xingyi Yang, Qiaochu Xue, and Xinchao Wang. Ominicontrol: Min- imal and universal control for diffusion transformer. arXiv preprint arX1v:2411.15098, 2024.
Zhiyu Tan, Hao Yang, Luozheng Qin, Jia Gong, Mengping Yang, and Hao Li. Omni-video: Democ- ratizing unified video understanding and generation. arXiv preprint arXiv:2507.06/ 19, 2025.
14

---

Chameleon Team. Chameleon: Mixed-mode early-fusion foundation models. arXiv preprint arXiv:2405.09818. 2024.
Shengbang Tong,
David Fan, Jiachen Zhu, Yunyang Xiong, Xinlei Chen, Koustuv Sinha, Michael Rabbat, Yann LeCun, Saining Xie, and Zhwang Liu. Metamorph: Multimodal understanding and generation via instruction tuning. arXiv preprint arXiv:2412.14164 2024.
Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang. et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314 2025.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024a.
Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need.
arXiv preprint arXiv:2409. 18869, 2024b.
Zhouxia Wang, Ziyang Yuan, Xintao Wang, Yaowei Li, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionetrl: A unified and flexible motion controller for video generation. In ACM SIGGRAPH 2024 Conference Papers, pp. 1-11. 2024c.
Cong Wei, Zheyang Xiong, Weiming Ren, Xeron Du, Ge Zhang, and Wenhu Chen. Omniedit: Building image editing generalist models through specialist supervision. In The Thirteenth Inter- national Conference Learning Representations, 2024.
Chenfei Wu, Jiahao Li, Jingren Zhou, Junyang Lin, Kaiyuan Gao, Kun Yan, Sheng-ming Yin, Shuai Bai, Xiao Xu, Yilei Chen, et al. Qwen-image technical report. arXiv preprint Siti:22008.022244 2025a.
Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingehao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, al. Janus: Decoupling visual encoding for unified multimodal understanding and generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 22966-12977 2025b.
Chenyuan Wu, Pengfei Zheng, Ruiran Yan, Shitao Xiao, Xin Luo, Yueze Wang, Wanli Li, Xiyan Jiang, Yexin Liu, Junjie Zhou, al. Omnigen2: Exploration advanced multimodal generation. arXiv preprint arXiv:2506.1885 2025c.
Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multi- modal 1lm. In Forty-first International Conference Machine Learning, 2024.
Shitao Xiao, Yueze Wang, Junjie Zhou, Huaying Yuan, Xingrun Xing, Ruiran Yan, Chaofan Li, Shuting Wang, Tiejun Huang, and Zheng Liu. Omnigen: Unified image generation. In Proceed- ings of the Computer Vision and Pattern Recognition Conference, pp. 13294-13304, 2025.
Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408. 125228 2024.
Jinheng Xie, Zhenheng Yang, and Mike Zheng Shou. Show-o2: Improved native unified multimodal models. arXiv preprint arXiv:2506. 15564, 2025.
Zhuoyi Yang. Jiayan Teng. Wendi Zheng. Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint 1=XXix22408006777 2024.
Yang Ye, Xianyi He, Zongjian Li, Bin Lin, Shenghai Yuan, Zhiyuan Yan, Bohan Hou, and Li Yuan. Imgedit: A unified image editing dataset and benchmark. arXiv preprint 2rXi:25505.20275 2025a.
15

---

Zixuan Ye, Xuanhua He. Quande Liu, Qiulin Wang, Xintao Wang, Pengfei Wan, Di Zhang, Kun Gai, Qifeng Chen, and Wenhan Luo. Unic: Unified in-context video editing. arXiv preprint 0442022506..4221,, 2025b.
Zixuan Ye, Huijuan Huang, Xintao Wang, Pengfei Wan, Di Zhang, and Wenhan Luo. Stylemaster: Stylize your video with artistic generation and translation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 2630-2640, 2025c.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308. 02490, 2023.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, al. Mmmu: A massive multi-discipline multi- modal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9556-9567, 2024.
Kai Zhang, Lingho Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually annotated dataset for instruction-guiine image editing. Advances in Neural Information Processing Systems, 36:31428-31 2023a.
Lvmin Zhang, Anyi Rao, and Mancesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF international conference computer vision, pp. 3836-3847, 2023b.
Shiwei Zhang, Jiayu Wang, Yingya Zhang, Kang Zhao, Hangjie Yuan, Zhiwu Qin, Xiang Wang, Deli Zhao, and Jingren Zhou. 12vgen-xl: High-quality image-to-video synthesis via cascaded diffusion models. arXiv preprint arXiv:2311.08 1.04145 2023c.
Haozhe Zhao, Xiaojian Shawn Ma, Liang Chen, Shuzheng Si, Rujie Wu, Kaikai An, Peiyu Yu, Minjia Zhang, Qing Li, and Baobao Chang. Ultraedit: Instruction-based fine-grained image editing scale. Advances in Neural Information Processing Systems, 37:3058-3093, 2024.
Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model. arXiv preprint 1xXx:24408.11333 2024.
16

---

A
APPENDIX
Appendix contains the following sections:
Statement for Large Language Models
Extended Related Work
Additional Experiment and Analysis
Training Details
Limitation and Future Work
Training Dataset Construction
Evaluation Benchmark
B STATEMENT FOR LARGE LANGUAGE MODELS
We use large language models (LLMs) in this paper solely for grammar correction and text re- finement. They are not employed for generating original content or contributing to the conceptual development of the ideas presented.
C EXTENDED RELATED WORK
C.I UNIFIED MULTIMODAL UNDERSTANDING AND GENERATION
Recent progress in multimodal generation has been driven primarily by the text and image domains. Autoregressive models such as LlamaGen, Chameleon, Emu2, and Emu3 (Sun et al., 2024a Team 2024 Sun al. 2024b Wang et al. 2024b) adopt discrete token prediction. Hybrid approaches like Show-o, Transfusion, and DreamL (Xie et al.) 2024] Zhou et al., 2024: Dong et al. 2023) in- tegrate autoregression with diffusion for image synthesis. Regression- or instruction tuning based methods, including SEED-X, -. Janus, MetaMorph, Next-gpt and OmniGen2 (Ge et al., [0244] Wu jet al. 2025b Gupta et al. 2022 Wu et al., 2024; (025c), adapt LLMs for image feature prediction and controllable generation. Efficiency-oriented designs such as LMFusion and MetaQueries (Shi jet al. 2024a Pan et al., 2025) freeze MLLMs and add lightweight modules or learnable queries, while large-scale pretraining efforts like Show-o2, BLIP3-o. MoGao, and BAGEL (Xie al. 2025
Chen et al. 2025a Liao et al. 2025 Deng et al. 2025 Liu et al. 2025c demonstrate strong gen- eralization interleaved multimodal data. Despite these advances. most works remain centered image understanding and generation. In contrast, we move beyond the image domain by presenting unified video model. The most related works to ours are Omni-Video and UniVid (Tan et al., 2025 Luo et al. 2023). which primarily focus on the basic text-to-video generation task. However, these approaches do not investigate the potential benefits of unified architecture such as how unification can enhance compositional generalization in tasks like in-context editing and in-context generation. In contrast, our work explicitly demonstrates that a unified framework leads to stronger generalization to unseen tasks, highlighting the advantages of architectural unification across diverse understanding and generation scenarios.
C.2 IMAGE/VIDEO GENERATION AND EDITING.
Diffusion models have achieved remarkable success in high-fidelity image synthesis, with systems
like Stable Diffusion, DALL-E, and Imagen(Rombll (Robaac et al. 2022; Podell et al.) 2023 Esser et al.
2024 Ramesh et al., 2021; Saharia al. (022) establishing strong text-to-image capabilities and recent video dif diffusion Moodels(Blattann et al. 2023b Polyak et al. [024]] Chen et al. 2025c 2023
Yang et al. 2024 Blattmann al. 2023aa Kong et al. 2024 Brooks et al. 2024 Ma et al. 2025a) enabling scalable video generation. To improve controllability, models including ControlNet, T2I- Adapter(Zhang et al. 2023b Mou et al. 2024 introduce extermal condition modules, while edit- ing frameworks like InstructPix2Pix. EMU-Edit (Brooks et al., 2023 Sheynin al. 2024) support instruction-driven refinement. Recently, unified image generation has emerged, with OmniGen, OmniControl, and UniReal (Xiao et al. 2025 Tan al. 2024 Chen et al. 2025d) expanding from
17

---

generation to interence-guiders editing. General editing methods (Wei et al. 2024 Zhao al. 2024 Liu et al., 20256 Shi et al.
2024b; Zhang et al. 2023a) further highlight this trend. contrast, the
video doman remams dominated by single-task task frameworks such Video-P2P, MagicEdit, Mo-
tionCtrl (Liu et al. 2024c Liew et al. 2023 Wang et al. 2024c| Liu et al. 2025a Bai et al., 2025a
Huang al. 2025a 2026). Video Alchemist and Movie Weaver (Liang et al., 2025 Chen al,
2024) are dedicated to in-context generation. Attempts at unification include Any et al, 2024). which requires task-specific pipelines, EditVerse et al.) (025a), which can not perform the visual understanding task. VACE(Jiang et al.) (025). which relies on heavy adapter designs, FullDiT(Ju et al., 2025b), which supports multi-condition video generation but lacks editing, and YN et al.] 2025b), which unifies tasks but depends on task-specific condition bias, limiting scalability. Yet, compared to images, unified and flexible video generation and editing remains far less explored. Our work bridges this gap by unifying diverse video tasks under multimodal instruction framework. We provide the model capabilities comparison in Table
Table 5: Training hyperparameters across different stages. Stage 1: Connector alignment, Stage 2: Fine-tuning, Stage 3: Multi-task training.
Stages
Hyperparameters
Stage
Stage 2 (Fine-tuning)
Stage (Multi-task)
(Connector Alignment)
Learning rate
1 10-1
2.0 x 10-5
2.0 x 10-3
LR scheduler
Constant
Constant
Constant
Weight decay
0.0
0.0
0.0
Gradient norm clip
1.0
1.0
1.0
Optimizer
AdamW (31 0.9, 32 0.95, c 1.0 x 10-15)
Warm-up steps
50
50
50
Training steps
15K
5K
15K
EMA ratio
0.9999
0.9999
Gen resolution (min, max)
(240, 480)
(480, 854)
(480, 854)
Gen frames (min, max)
(1, 1)
(1, 129)
(1, 129)
Und resolution (min, max) Und frames (min, max)
(240, 480)
(480, 854)
(480, 854)
(1, 1)
(1,4)
(1, 4)
Diffusion timestep shift
5.0
5.0
5.0
D ADDITIONAL EXPERIMENT AND ANALYSIS
D.1 MODEL DESIGN
Our model design study addresses the following question: What is the most effective approach for aligning pretrained MLLM with diffusion generator during Stage I training?
We investigate three design choices for aligning the pretrained MLLM with the diffusion generator in Stage 1. Throughout this stage, the MILM remains frozen, while we vary the connector and DiT architectures across three variants as shown in Figure
DIT. The first variant adopts cross-attention-basee DiT for text conditioning, where we replace its original text encoder with an MLP connector that projects the final hidden states from the MILM into the DiT text embedding space. Both the MLP and DIT trained.
(a)
(b) Cross-atted attention DiT with Learnable query. Building upon (a), we use a learnable query mecha- nism following Pan al. (2025). Specifically, we extract the final hidden states of learnable queries from the MLLM, which are then passed through an MLP layer and used to replace the original text conditioning in the DIT's cross-attention module. We test two variants: (1) jointly training the
18

---

Table 6: Evaluation of text-to-image generation ability Geneval (Cosh et al. the methods using LLM rewriter
2023 benchmark. refer
Type
Method
Single obj.
Two obj.
Counting
Colors
Position
Color attr.
Overall
SDv2.1
0.44
0.07
0.50
Image Model
SDXL.
0.98
0.74
0.39
0.85
0.15
0.23
0.55
[E-XL
0.97
0.74
0.66
0.81
0.13
0.35
0.61
LUMINA-Nes
0.92
0.48
0.70
0.09
0.46
SD3-medium
0.99
0.94
0.72
0.89
0.33
0.60
0.74
FLUX. 1-Uev
0.99
0.81
0.79
0.74
0.20
0.47
0.67
NOVA
0.99
0.91
0.62
0.85
0.33
0.56
0.71
OmniGen
0.98
0.84
0.66
0.74
0.40
0.43
0.68
TokenFlow
0.60
0.81
0.16
0.55
Janus
0.97
0.68
0.30
0.84
0.46
0.42
0.61
Janus Pro
0.99
0.89
0.59
0.90
0.79
0.66
0.80
Unified Model
Emu3-Gen
0.99
0.81
0.42
0.80
0.49
0.45
0.66
MctaQuery-XLi
0,80
BLIP3-o' 4B
0.81
BIIP3-nn 8B
0.84
BAGEL
0.99
0.94
0.81
0.88
0.64
0.63
0.82
UniWorld-Vi
0.99
0.93
0.79
0.89
0.49
0.70
0.80
OmniGen2
0.95
0.64
0.88
0.55
0.76
0.80
UniVideo
0.93
0.63
0.23
0.76
0.21
0.41
0.53
UniVideo
0.98
0.85
0.36
0,90
0.45
0.64
0.69
Table 7: Comparison results on LngEdii-Bennhl (Ye et al. 2025a and MEdit-Bench--E (Liu al. 2025b) (SC: Semantic Consistency, PQ: Perceptual Quality, O: Overall). "Overall" is calculated averaging all scores across tasks. The best results highlighted in bold, and the second-best results underlined.
ImgEdit- Bench
GEdit-Bench
Model
Add
Adj.
Ext.
Rep.
Rm.
Bg.
Sty.
Hyb.
Act.
Overall
G-SC
G-PQ
G-O
GPT-4o
4.61
4.33
2.90
4.35
3.66
4.57
4.93
3.96
4.89
4.20
7.85
7.62
7.53
MagicBrush
2.84
1.58
1.51
1.97
1.58
1.75
2.38
1.62
1.22
1.90
4.68
5,66
4.52
Instruct-P2P
2.45
1.83
1.44
2.01
1.50
1.44
3.55
1.20
1.46
1.88
3.58
5.49
3.68
AnyEdit
3.18
2.95
1.88
2.47
2.23
2.24
2.85
1.56
2.65
2.45
3.18
5.82
3.21
OmniGen
3.47
3.04
1.71
2.94
2.43
3.21
4.19
2.24
3.38
2.96
5.96
5.89
5.06
3.58
3.39
1.73
3.15
2.93
3.84
2.04
3.05
6.85
StepiX-Fdit
3.88
3.14
1.76
3.40
4.63
2.64
3.06
7.09
6.76
6.70
BAGEL
3.56
3.31
1.70
3.30
2.62
3.24
4.49
2.38
4.17
3.20
7.36
6.83
6.52
UniWorld-V1
3.82
3.64
2.27
3.47
3.24
2.99
4.21
2.96
2.74
3.26
4.93
7.43
4.85
OmniGen2
3.57
3.06
1.77
3.74
3.20
3.57
4.81
2.52
4.68
3.44
7.16
6.77
6.41
Univideo
4.22
3.90
1.66
3.96
3.73
4.11
4.54
3.55
4.81
3.83
7.08
7.08
6.41
learnable queries, MLP layer, and DiT (as in Pan al. (025 ); and (2) training only the learnable queries and MLP while keeping the DiT frozen.
(c) Uni deo architecture. The main difference this variant lies in its use of MMDiT, which em- ploys self-attention for joint text-video interaction instead of cross-attention. We replace MMDiT's original text encoder with an MLP connector that projects the final hidden states from the MLLM into the MMDIT's text embedding space. Only the MLP layer is trained, while both the MILM and MMDiT remain frozen.
For the cross-atrention variants, 2025), originally based on a text encoder(Raffel et al.] 2020), which we replace with Qwen2.5- VL. For UniVideo. we follow the implementation details described in subsectionn 3.1 All variants are trained for 15K steps, and the qualitative results are presented Figure
internal model with architecture similar to WWn et al.)
Our findings show that the cross-attention variants require unfreezing the DiT generator achieve effective alignment with the MLLM, as evidenced by the comparison between (b.2) and (b.1). Nevertheless, after unfreezing, variants (a) and (b.1) exhibit limited text-following abil- ity poy-pariiullarly for compositional object prompts. In contrast, the UniVidec architecture achieves efficient and robust alignment by training only the MI.P connector.
19

---

DiT
DIT
Aficution
Attention
MLLM
MLLM
Text
Text
Noisy Video Token
Learnable Query
Noisy Video Token
(a) Cross Attention
(b) Cross Atlention with Learnable Query
MMDiT
MLLM
Self Attention
Text
Noisy Video Taken
(c) Self Attention
Figure 7: Three design choices for aligning the MLLM with the diffuusio generator in Stage training. We keep the MLLM fixed and vary connector and DiT architecture across three variants: (a) the DiT uses cross-altention for text conditioning, where we replace its original text encoder with an ML.P layer that aligns the final hidden states from the MLLM; (b) building upon (a), we introduce a learnable query design and extract the final hidden states from these learnable queries; and (c) our Uni Video architecture employs an MMDiT design that leverages self-attention for text conditioning.
(A) Cross Allention
Query
(b2) Cress Allentiu with Leamablee Usery (DIT Frooes)
(UnlVaden)
demaseer modke by
Daguliippett partt costload.
smarhh seand sisqpill nereca trangall aranedin drrlgg
Figure 8: Qualitative comparison of design choices for aligning the MLLM with the diffusion generator in Stage training. In all settings, the MLLM is kept frozen. (a) Cross-Attention DIT: we train the MLP connector and DiT; (b.1) Cross-Attention DiT with Leurnable Query: following (Pan et al. (2025), we train the learnable query tokens, MLP connector, and DiT; (b.2) similar (b.1), but the DIT is frozen while only the learnable query tokens and MLP connector are trained; (c) UniVideo (MMDiT): only MLP connector trained, with all other components frozen. All variants are trained for 15K steps. Among all variants, UniVideo (MMDIT) demonstrates the best prompt alignment.
We study two UniVided variants in Stage 3 training. The MLLM is kept frozen, while we vary the connector design across two variants. illustrated in Figure 9
(a) Uni Video (hidden). this variant, we extract the final-layer hidden states of all image, video, and text tokens produced by the MLLM. These token representations used inputs the under- standing branch of MMDiT. During training, only the MLP connector and the MMDiT are updated, while the MILM remains frozen.
(b) Uni Video (query). This variant adopts a learnable query mechanism following Pan et al. 2025 Specifically, we extract the final hidden states of the learnable query tokens from the MLL.M,
20

---

MMDIT
MMDIT
MLLLM
MILM
Self Atteuties
f
Irage Video
Text
Image Vadeo
Text
Image Video
Tokin
Query/h
NoLay
(8) UniVideo (hidden)
UniVadeo (query)
Figure 9: Two design variants for multimodal conditioning in Stage 3 multi-task training. We study two multimodal conditioning strategies: (a) extracting the final hidden states of image, video, and text tokens produced by the MLLM; and (b) adopting learnable query design and using the final hidden states the learnable queries together with the final hidden states of the text tokens.
together with the final hidden states of the text tokens. In this setting, we train the learnable queries, the MLP connector, and the MMDiT, while keeping the MLLM frozen.
Uni Vi deo (query) can be more computationally efficient at inference time due its fixed number of query tokens. By default, use 512 learnable queries, which reduces the number of condi- tioning tokens compared to using all the multimodal hiddens from the MLLM. This efficiency gain particularly heneficial for tasks where video inputs dominate the MMDiT conditioning, such as video editing.
During training, however, the UniVich Video (query) variant requires backpropagation through the MLLM computation graph in order to optimize the learnable queries, which incurs additional mem- ory overhead. In contrast, the Uni Vi deo (hidden) variant does not require gradient flow through the MLLM and is therefore more memory-efficient during training.
D.2 IMAGE GENERATION TASKS.
Image Generation. We evaluate UniVideo on text-to-image generation using the Geneval benchmark (Ghosh al. 2023). Methods marked with i denote approaches that employ an LLM- based prompt rewriter. Despite being an image-video generalist model and using only small proportion of '2I samples during Stage training, Un V deo retains competitive text-to-image performance and outperforms many specialized text-to-image models and unified text-image multi- modal models Table 6
Image Editing. We further evaluate UniVide on image editing tasks using ImgEdit-Bench (Ye et al. 2025a) and Bedit-Bench-EN (Liu et al. (0255). We report Semantic Consistency (SC) as the primary evaluation metric. Despite being an image-video generalist model, UniVi univideo achieves strong editing performance and surpasses several specialized image editing models Table 7
D.3 VIDEO GENERATION TASKS.
For the text-to-video generation task, we use the prompt suite provided in VBench Huang et al.) 2024), which contains 946 prompts covering 16 dimensions, including subject consistency, back- ground consistency, aesthetic quality, imaging quality, object class, multiple objects, color, spatial relationship, scene, temporal style, overall consistency, human action, temporal flickering, motion smoothness, dynamic degree, appearance style. Following VBench, obtain the semantic score and the quality score based on the scores on these dimensions. The results are shown in Table 9
E TRAINING DETAILS
We adopt Swee2.5VV-77B (Bai et al. 2025b) as the MLLM backbone and Hunyuan Video-T2V 13B (Kong et al.) 2024) as the MMDiT backbone. The original Hunyuan Video also uses CLIP as its text encoder; remove it and instead employ qwen2.5VL as the unified multimodal embedder. The released Hunyuan Video checkpoint is a CFG-distilled model, whose distillation embeddings discard to simplify the training. To align feature dimensions between gwen2.5VL and Hunyuan- Video, we apply an MLP with a expansion. We report training configurations, hyperparameters in Table
21

---

F LIMITATION AND FUTURE WORK
Our model is trained on diverse tasks with multimodal instructions. While we do not observe task confusion, it sometimes fails to strictly follow editing instructions, occasionally over-editing unre- lated regions. Due to backbone limitations, the model also struggles to fully preserve the motion of original videos, indicating the need for stronger video backbones. Moreover, although UniVida idoo generalizes to free-form video editing, its success rate remains lower than in image editing, under- scoring the greater difficulty of video editing. Future work could explore large-scale video editing datasets and improved backbones for motion fidelity. Additionally, as UniVideo represents an as- sembled multimodal generative system capable of producing images, videos, and text, future work could aim to develop a native multimodal video model trained end-to-end.
G TRAINING DATASET CONSTRUCTION
This section details the construction of our datasets.
G.1 ID-RELATED TASKS
For in-context video generation, which requires identity annotations, we follow the data creation pipeline of ConceptMaster (Huang et al. 2025b). We first extract keyframes from each video and then use Qwen2.55VI--7B (Bai al. 20255) to identify the primary subjects in the video. The model is prompted to focus on semantically meaningful objects and ignore irrelevant background elements. Based on the subject tags generated by the Qwen2.5-VL-7B (Bai al., (2025b), we obtain subject bounding boxes on the first frame with Grounding DINO (Liu et al.) 2024d), We filter out videos with target areas that are either too small or too large. The lower bound is 10% of the frame and the upper bound is 60% of the frame. We then use apply SAM2 (Ravi et al.] (024) to obtain object segmentation masks from the source video. To further filter out object tracks that are not consistently visible (e.g., those that are too small in most frames or segmented unreliably), we compute a visibility consistency score. For each track, we count the number of frames in which the object's 's mask area exceeds a preset area threshold and divide this by the total number of frames in the track. Frames where the object is too small or poorly segmented do not contribute to the score. A higher score indicates that the subject remains clearly visible for most of the video. We discard tracks whose visibility consistency score falls below a predefined threshold. After this stage, we get sources videos and subject masks.
As demonstrated in Figure 10 to build in-context video tasks, we leverage an inpainter model.
For the object swap task, the inpainter is instructed fill the masked region using the text tags predicted by Qwen2.5-VIL (Bai et al., 2025b). To construct training pairs for this task, we use the inpainted video together with the subject image as the input, and the original video as the target.
For the object removal and addition tasks, we do not provide explicit textual instructions to the inpainter. Instead, the model fills the masked region based solely on the surrounding visual context, effectively removing the target object while preserving the background. For the addition task, we construct training pairs by using the inpainted video and the subject image as input, with the original video as the target. For the deletion task, we use the original video as the input and the inpainted video as the target.
To construct editing instructions for each pair of data, we employ 1iven2.5-VL-222 (Bai et al., 2025b) to generate precise editing instructions based on the first frame of the input video and and the first frame of the target video.
The inpainter is built on 1 B 1B-parameter model with an architecture similar Wan2.1 Wan et al., 2025). which employs cross- cross-attention modules for text conditioning and self-attention for visual tokens. We select and copy an interleaved half of the Transformer blocks from the original DiT to form the control net. While the original DiT processes noisy video tokens together with text tokens, the newly added control blocks operate on the masked video, the corresponding masks, and the text tokens. The output of each control block is injected back into the DiT as an additive control signal.
To train the video inpainter, we use the open source dataset VIVID- (Hu et al. 2024b), which provides source video and object mask for inpainter training.
22

---

Source video
Subject Imuge
Masked Video
Inpainted Video
Input
Turget
In Context Swap
In Context Addition
Context Deletion
Figure 10: In-Context task dataset construction examples. The top section illustrates our pipeline: first extract the subject image from the initial frame, then apply SAM2 (Ravi et al.) (2024) to obtaín video masks, and subsequently perform video inpainting based on these masks. The bottom section shows how we group the resulting images and videos into input-target pairs to form dataset.
After constructing the dataset, conduct human filtering stage ensure the final quality of all edited videos. Annotators are provided with both the source video and the edited video and evaluate each sample solely based on three criteria: video quality, instruction following, and consistency with the source video(degree of overedit).
For object removal and addition tasks, sample is accepted only if the edit satisfies all three di- mensions: (1) high video quality, meaning the edited region is clear and artifact-free; (2) correct execution of the instruction, such fully removing appropriately adding the target object; and
23

---

Annotator Instructions
You are given two inputs:
Source video
Edited video
A sample should be accepted only if it satisfies all three dimensions:
1. Video Quality
The edited region is clear, stable, and free of severe blur.
No obvious artifacts such as texture duplication, holes, melting shapes, or structural collapse.
Motion is temporally consistent with strong flicker jitter.
2. Instruction Following
The edit correctly follows the given instruction (e.g., object removal, addition, or swap).
3. Consistency With the Source Video
No unintended changes over-editing outside the target region.
The edited content matches the original motion, lighting, and scene dynamics across frames.
Figure 11: Annotator instruction used for human filtering of in context task video data.
(3) consistency with the original video, ensuring natural backgrounds and no over-editing beyond the target region. Any sample exhibiting artifacts, partial edits, or temporal flicker is rejected.
For object swap tasks, annotators apply the same three metrics. A sample is accepted only if (1) the edited content is visually stable and free of distortions, (2) the swap operation correctly follows the instruction, and (3) the resulting video remains consistent with the original motion, lighting, and scene dynamics. Samples containing structural distortions, unnatural textures, or temporal incon- sistency are rejected. Identity verification unnecessary, the source video already defines the intended target appearance.
G.2 STYLIZATION
Following UNIC (Ye et al. 2025b), Text-to-Video (T2V) models are capable of generating stylized videos with high visual quality and strong fidelity to given reference style image. Instead of directly stylizing an existing real video, we leverage this capability to first produce a high-quality stylized video using T2V model. We then convert this stylized video into a realistic counterpart using a stylized-to-real ControlNet Video DiT model.
The input to the ControlNet is gray tile signal. Specifically, downsample the video spatially by a factor of 8 and then upsample it by the same factor to remove high- frequency details, producing low-fidelity tile image. We further discard the color information by converting this tile image into grayscale. This results in structural guidance signal that preserves spatial layout while suppressing style and texture.
Similar to StyleMaster (ee et al. 2025c). the ControlNet is built on 1B-parameter DiT architecture similar to 1 (Wan al./ 2025). which combines cross-attention for text conditioning with self-attention over visual tokens. We construct the ControlNet by copying interleaved half of the Transformer blocks from the original DIT. While the original DiT processes noisy video tokens alongside text tokens, the ControlNet blocks operate the gray tile signal together with the text tokens. The output of each ControlNet block is injected back into the DiT through additive residual connections.
24

---

Table 8: Model capabilities across understanding, generation, editing, and in-context generation.
Vindicates support; Xindicates not supported. The last row is highlighted.
Model
Understanding
Image Gen.
Video Gen.
Image Edit.
Video Edit. In-context Video Gen.
LLaVA-1.5
SD3-medium
RLUX.I-dev
X
X
QwenImage
/
X
Hunyuan Video
Show-o
Janus-Pro
X
X
Emu3
X
/
X
BLIP3-o
X
X
BAGEL
OmniGen2
VACE
X
UniVideo
We train the stylized-to-real ControlNet using 10K video pairs in which both the input and target videos are real. During training, the model therefore learns real-to-real reconstruction task. Since the control signal (the gray tile) preserves only coarse spatial structure while discarding color, de- tails, and style, the model learns generate realistie content guided only by spatial layout. At inference time, the model can effectively perform stylized-to-res mapping because the stylized in- put video is also converted into a gray-tile signal, which contains only spatial layout information and thus matches the training distribution.
G.3 IMAGE EDITING, TEXT-TO-VIDEO AND TEXT-TO-IMAGE
We leverage state-of-the-art image-editing models such as FLUX.1 Kontext (Labs et al. 2025) to construct a diverse collection edited images. We further incorporate high-quality open-source datasets, including OmniEdit (Wei et al.) 2024), ImgEdit (Ye et al. (025a). and ShareGPT-4o- Image (Chen et al., 20256). Following OmniEEdi, we apply an additional VLM-based filtering stage the curated image-editing dataset. Each (source, edited) pair is evaluated using Qwen2.5-VI., which assigns 0-10 scores along three core dimensions:
Image Quality: the edited region must be sharp and visually stable, with no artifacts such as duplicated textures, holes, melting shapes, unnatural boundaries, or structural distor- tions.
Instruction Following: the edit must correctly execute the given instruction (e.g., object removal, addition, swap), without partial or incorrect modifications.
Consistency With the Source Image(degree overedit): no unintended changes or over- editing may occur outside the target region, and the edited content must remain coherent with the original scene's lighting, colors, and geometry.
Samples falling below threshold on any dimension are discarded. After filtering, we retain approxi- mately 500K high-quality edited samples.
For text-to-image and text-to-video generation tasks, we utilize additional internal datasets.
H EVALUATION BENCHMARK
H.1 IN-CONTEXT VIDEO GENERATION
For the in-context video generation, construct test set consisting of 20 cases, evenly split between single-ID and multi-ID scenarios. For each case, we collect ID images and carefully design prompts ensure reasonable evaluation. As shown in Fig. [2 build ID pool with diverse images, ranging from cartoons to real-world subjects, including humans, animals, and common objects. We then select ID images from this pool and design appropriate prompts for them.
25

---

Table 9: T2V Generation Performance on VBench
Model
Quality
Subject
Background
Temporal
Dynamic
Aesthetic
Imuging
V5.
85.03
77.01
98.00
97.41
99.19
98.02
57.15
69.48
68.61
MiniMax-Vidoo Video-0
84.85
77.65
97.51
97.05
99.10
99.22
64.91
63.U3
67.17
Kling 1.6
85.00
76.99
97.40
96. 84
98.64
99.13
62.22
64.81
69.70
I-n2.1-T2V.1.33
HinyoanVideo
85.09
75.82
97.37
97.76
99.44
98.99
70.83
60.36
67.56
Vehitect-2.0 (VEahaner))
83.54
77.06
96.83
96.66
98.57
98.98
63,89
60.41
65.35
96.72
Cog V.de
82.78
79.76
96.87
97.35
98.88
98.31
50,93
62.79
65.02
Ours
84.36
79.96
96.57
96.66
99.29
99.15
49.72
68.68
67.32
Model Name
Object
Multi-Obi
Action
Color
Spatial
Soone
Appearance
Temporal
Overall
MiniMa, -iddee---VV Video-01
87.83
76.04
92.40
90.36
75.50
50.68
20.06
25.63
27.10
Kling
Wan2.1-122-1.2 1-12B-1
88.81
74.83
94,00
89.20
73.04
41.96
21.81
23.13
25.50
HunyuanVideo
Gen-3
87.81
53.64
96.40
85.95
65.09
$4.57
24.31
24.71
26.11
Vchitect-2.0 (VEnhaneer)
56.57
TUNA
95.41
92.31
97.50
87.67
78.12
$8.59
23.18
24.68
27.71
Cog Vidco
87.47
97.20
87.55
Ours
94.59
82.06
97.40
82.83
76.43
48.74
24.92
25.06
26.38
Chouse IDs
Design Prompt
A man with
red backpack.
Figure Construction pipeline in-context video generation test set.
The single-ID examples are shown in Fig. 13
The single can have either one ID image, as shown by the cat example, or multiple shots of the same ID, as demonstrated by the human example.
IDs
Prompt
Panoramic shot, man leaning against tree, playing a beautiful melody the guitar his hand. His smile chords harmonious camera slowly moves around the mon.
Wu kong, elnd in ornate golden armor adorned with intricate red black patterns, strides confidently through brightly modern supermarket.
Figure 13: Example of single-ID test case in in-context video generation test set.
26

---

As shown in Fig. 14 in the multiple-ID scenarios, the number of IDs in ranges from 2 to 4, with larger numbers leading to higher difficulty. Our prompts focus on the interaction between these ID images and describe the relationships among them. For example, in the first case, the prompt describes a woman sitting on the sofa beside the bag, which connects the woman, sofa, and bag provided in the ID images. In the second case, the relationship between the two characters is described as Psyduck riding Pikachu.
IDs
Prompt
man dressed vibrant Hawallan shirt with colerted
Burel patiern, sits benck lounge chaic. Ou his shoulder, Piánchu with small detective hat perches. The man halds
taking bitee
middle-aged brown stand amidst bustling, state-of- the laberatory, engrosed in deep conversation They move purpose and precision, vecasionally pointing comples equipment data dloplays, their body langenge convering focused. shared atteation. Their expressios are mix Intense concentration and intellectnal
men,
gray
curtunity, digniffing moment of profound, high-stalker stalee scientific or technological disrmsion. The camera diowly artilis the men as they talk, emphasiring the intricate and dynamic envirusment of their eetting.
Figure 14: Example of multi-ID test case in in-context video generation test set.
H.2 IN-CONTEXT VIDEO EDITING
For the in-context video editing, we evaluate on the UNICBench Ye al. (2025b) across four tasks: ID Insertion, ID Swap, ID Deletion, and Stylization. Since setting differs from other video editing models (which may require masks to indicate the edited area, while ours uses instructions instead), demonstrate in detail how derive our inputs from the existing video editing bench- mark.
First, shown in Fig. 15 for ID insertion, the elements in UNICBench consist of reference video, reference ID, and a caption for the target video. The goal of ID insertion is to naturally integrate new objects elements from the reference ID into the target video. Here replace the caption with more direct instruction.
octopus the odge the sea. The actopus has orange-yellon body with elearly visible suckers on its teatacles.
Elements UNICBench
Inputs of UniVideo
octopus from edge
Figure 15; Example of ID insertion test case.
For ID swap, the elements in UNICBench consist of a reference video, mask, reference ID, and caption for the target video. The goal of ID swap is replace specific elements in the target video with corresponding elements from the reference ID while preserving the original video's 's context and motion. In our setting, don't need mask to indicate the editing area; instead, we more
27

---

convenient instruction- approach. For example, in Fig. I6 simply the instruction "Use the man's ' face in the reference image to replace the man's 's face in the video."
incluling black i-sirri blue jesm, sorakers, seen walking on paved path in park witch his winged cow guide compuniion.
UNICRocc
Isputs of UniViden
Use winged con in the reference
Image replace the dog in the video.
Figure 16: Example of ID swap test case.
For ID deletion, UNICBench provides reference video, mask, and caption for the target video. ID deletion aims to naturally remove specified objects or elements from the video while maintaining visual consistency and filling the removed areas with appropriate background content, While current video editing methods use masks to specify the object for removal, our approach simplifies this through text instructions. As demonstrated in Fig. 17 "Delete the computer in the video."
we use straightforward prompts such as
man light gray suit and yellow die seated
office desk, while a woman blazer with a black collar stands beside him, helding a gles wallr.
UNICBouch
white
Imputs of Unit Tileo
Delete the compater the viden.
Figure 17: Example of ID deletion test case.
For stylization, the existing elements in UNICBench include style reference image, target caption, and reference video. The purpose of stylization to transform the visual appearance of the target video to match the artistic style of the reference image while preserving the original video's 's content and motion dynamics. We standardize the instruction format to "Transform the video into the style of the reference image," as shown in Fig. 18
28

---

A woman with long hair glasses stands near a river.
Elements UNICBench
Inputs UniVideo
Transform the video of into style of the reference image.
Figure 18: Example of stylization test case.
29


Reviews:
[{"review_id": "FCfcZGPIhO", "reviewer": "Reviewer_zojR", "summary": "The paper is cleanly structured and easy to read. Qualitative results are abundant and illustrative, and the in-context editing/generation comparisons are clearly presented. At a high level, the work’s strengths are system-level integration and breadth of tasks. The level of novelty is average but acceptable.", "questions": "1. The workflow diagram (Fig. 3) isn’t very clear. Is it correct to interpret that the user feeds an interleaved text–image instruction to the MLLM, and then the MLLM’s output is concatenated with the VAE features of the conditioning images as the input? Also, why is the noise term missing in Fig. 3? It feels like this should be made consistent with Fig. 2.\n2. Maybe compare to more unified understanding & generation model", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. A pragmatic dual-stream unification (frozen MLLM for understanding + MMDiT for generation) that feeds visual inputs to both streams, with ablations showing why both sides need visuals for identity preservation and semantics.\n2. Strong mask-free in-context editing/generation results despite baselines using masks; the qualitative figures and automatic metrics make the claim legible and practically relevant.\n3. Transparent training recipe and task coverage, including staged training, freezing choices, connector design, and explicit mixing ratios. It is useful for reproduction and future baselines.", "weaknesses": "1. Thinking Mode is not well and fully discussed(How much does it benefit?).\n2. What is the benefit of making it a single model rather than using a workflow or agent? Qualitative analysis?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lOxqSLuj4s", "reviewer": "Reviewer_2yjX", "summary": "The paper presents a system for joint image/video understanding, generation and editing. It consists on two streams: understanding and generation. The understanding stream is the frozen Qwen-VL2.5-7B. The generation stream is initialized from HunyuanVideo-T2V-13B. The training is split into 3 stages: 1) training the connector between MLLM and MMDiT; 2) fine-tuning MMDiT for base tasks (T2I + T2V); 3) multi-task fine-tuning of MMDiT. The results look very good visually, and the model was even shown to zero-shot generalize to novel tasks (e.g. free-form video editing). The authors also perform a thorough quantitative evaluation demonstrating the superiority of the method over existing baselines.", "questions": "Could you please include more dataset details (as specified in the previous section)?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- It is a quite elegant architecture which i think among the early works for this paradigm shift in generative modeling of concatenating everything into a single sequence and modeling. While it does not serve as a good reference for ablations, but it's a good proof of concept to convince the community that this paradigm works.\n- The results are really good, especially given the little amount of GPUs used to fine-tune the model. Also, the data size is affordable to collect for non-bigtech companies (e.g. startups)\n- The paper reads well and the illustrations are good. The submission also includes many qualitatives which are very easy to view on the website.", "weaknesses": "- From my perspective, the main weakness is the lack of rich ablations, that could make the submission to be a good reference for follow up works. For example, is it necessary to do all the 3 stages sequentially or we can train jointly? Is the diffusion schedule the same for all the modalities? How much improvement can we get by making it different? Would it help to do some dropout on some input modalities in a task? Is it possible to replace full attention with cheaper attention variants in-between the modalities? Also some profiling results would be interesting to see, e.g. how much compute is spent on each modality in each component (is MLLM heavy?) And so on.\n- Some important details are missing from the submission, mainly related to data curation. It would be fine to omit them for a technical report, but it is not good practice to omit them for an academic submission. For example, how exactly was stylized video transformed into a real one (appendix F.2)? How was the video inpainting model training (appendix F.1)? Was the dataset post-curated with human annotators? If so, what were the instructions for them?\n\nSmall writing comments:\n- typo: \"generate an video\" => \"generate a video\" (in figures)\n- typo: \"source open source\" on line 907\n- I would suggest re-coloring Figure 3 to display the frozen MLLM as blue, and trainable DiT as red, as common in the prior literature.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gA7RbeJKke", "reviewer": "Reviewer_p7Fb", "summary": "The work proposes a method to imbue a pretrained video generator with multimodal input understanding. The work builds on the observation that text embedding can be replaced with embeddings produced by a frozen MLLM, making it possible for the downstream video generator to leverage MLLM capabilities such as thinking and understanding of multimodal inputs. The authors first curate datasets comprising a variety of multimodal tasks such as image editing, image to video, style transfer, object swapping, addition, deletion. Then a pretrained Hunyuan model is adapted to receive QwenVL2.5-7B multimodal inputs in 3 stages, by first training a connector MLP, then fine-tuning the video generator on T2I and T2V, and finally training on the full range of tasks. The resulting model produces convincing results in a variety of tasks and shows ability to generalize outside of the set of training tasks. Quantitative evaluation shows generally better or comparable scores with respect to VACE and commercial models.", "questions": "I hope the authors could clarify the following questions\n- Table 5 reports that \"w/o visual for MLLM\" achieves an overall score that is very close to VOGUE. While the exact setting is slightly unclear (are we just removing visual tokens from the MLLM output or is the MLLM not receiving visual tokens at all?) LL440-LL441 suggests that no visual token is processed by the MLLM in this setting. While a small gap is present in the prompt following (PF) metric, this raises the question of whether an MLLM is needed at all. If only text tokens are processed by the MLLM in the performed ablation, wouldn't the original text encoder produce similar performance without the need for an MLLM? This is an important point to clarify as otherwise the role of the MLLM is unclear, and a simpler framework with equal capabilities could be constructed by means of the collected dataset only without any MLLM.\n- Table 2 reports a series of Understanding metrics. Do I understand correctly that these metrics are computed using the frozen MLLM model alone without involvement of the video generator? If so, such metrics are less interesting to show. The claim that the model can perform \"Understanding\" in a unified way with generation and editing in this case would be problematic as the \"Understanding\" part is completely offloaded to a pretrained and frozen external model, which is not a unified approach. In a truly unified approach, the MM-DiT itself would output text tokens for understanding tasks when necessary.\n\nMinor:\n- Could the authors clarify more how the dataset ratios shown in Table 1 were computed? Can the authors offer insights into how optimal ratios should be derived?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The task of creating unified video generators capable of tackling multiple visual tasks from T2V to video editing is of high relevance\n- Qualitative results are convincing. The model seems to be able to tackle a range of video generation tasks from object insertion to instruction based editing with convincing quality\n- Ablations are insightful and validate that 1) joint training on all tasks reinforces model performance in all tasks, 2) relying on MLLM visual embeddings is insufficient, requiring visual tokens being passed to the video generation backbone directly\n- The authors provide complete training and dataset details \n- Authors commit to public release of model and code", "weaknesses": "- Ablation on MLLM visual embeddings raises doubts of how much the introduction of the MLLM contributes to model performance with respect to the newly collected data. See questions.\n\nMinor:\n- Minor amount of typos, especially lack of space before citations (LL254, LL258)\n- LL52, LL423-LL427 the claim appears inaccurate. Veo 3 was previously found to support this feature (https://www.reddit.com/r/singularity/comments/1m9b0bq/googles_new_feature_in_veo_3_you_can_now_draw/) (https://www.youtube.com/watch?v=KNGMBRyGcDo). I suggest removing the claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VlARFwpTr1", "reviewer": "Reviewer_dJ5L", "summary": "This paper introduces VOGUE, a unified framework for video understanding, generating, and editing from a single multimodal prompt. To achieve this, VOGUE includes a MLLM for complex semantic and instruction understanding, which is followed by a MMDiT for high-fidelity video generation. The authors demonstrate that VOGUE is able to understand complex multimodal instructions and perform diverse tasks such as T2V, I2V, in-context generation, in-context editing, and even zero-shot generalization. The experiments also show that VOGUE outperforms SOTA task-specific methods on multiple benchmarks like VBench, UNICBench, and human evaluations.", "questions": "- Could the authors explain the theoretical motivation for the proposed dual-stream architecture and include ablation studies to verify the design choice of separating the understanding and generation streams?\n- Could the authors elaborate on the dataset curation pipeline?\n- For the visual prompting task, could the authors describe whether the model can handle OOD or ambiguous prompts, and provide some failure cases to help check the generality of this task?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The most impressive strength of the work is the wide coverage on diverse tasks, including video understanding, T2V, I2V, in-context video generation, and in-context video editing, under a single framework. As far as I understand, VOGUE is the first video generation model that achieves this level of task unification which can substantially enhance the community interest.\n\n- I also like the experiments where the authors show the model can achieve some zero-shot generation tasks which are not explicitly covered during training. For example, it can transfer the editing abilities to the unseen tasks composition (such as combining style transfer with object deletion). This highlights the advantages to leverage the powerful and frozen MLLM.\n\n- The experiments include the comparisons with existing models on eight tasks. It is also impressive that such a general-purposed model can achieve a similar or comparable performance with the tasks-specific models.\n\n- The authors also provide thorough ablation studies, including 1) the necessity of multi-task learning (compared to single-task learning) and 2) dual-stream model architecture, which are insightful.\n\n- The authors promise that they will release the model checkpoint and code, which can significantly increase the reproducibility and also benefit the further research in unified video generation and editing.\n\n- The paper is well written and easy to follow. The figures are well-plotted and informative which can make readers quickly understand the core ideas.", "weaknesses": "- My major concern is the incremental technical contribution. The dual-stream architecture that separately processes multimodal instructions and visual inputs has been explored in some existing models such as FLUX, which also employs two streams to process text and image inputs. Also, using MLLM embeddings to condition the diffusion process has also been explored in Qwen-Image. Therefore, while the model design is effective, it feels more like an integration of existing ideas and does not look insightful to me.\n- It is unclear why the dual-stream architecture improves generalization compared to single-stream model design. For example, models like FullDiT [1] and Qwen-Image achieve great performance using full self-attention without separating understanding and generation streams. Could the authors provide the motivation behind this architecture design?\n- The paper provides limited information about the construction of the training dataset. I hypothesize that learning such a general-purpose model requires high-quality, large-scale, and carefully-curated datasets for training. However, the authors only give a brief summary in Appendix F without describing more details such as the data filtering or annotation. It raises concerns about how the model learns to cover complex tasks such as visual prompt understanding.\n- Following the above concerns, while Figure 6 shows several types of visual prompts, it is unclear how the model handles out-of-distribution or ambiguous prompts.\n- There are some missing citations and comparisons with existing multi-subject video personalization models, such as Video Alchemist [2] and Movie Weaver [3], which can also be formulated as in-context generation.\n\n[1] \"FullDiT: Multi-Task Video Generative Foundation Model with Full Attention\", ICCV 2025\n\n[2] \"Multi-subject Open-set Personalization in Video Generation\", CVPR 2025\n\n[3] \"Movie Weaver: Tuning-Free Multi-Concept Video Personalization with Anchored Prompts\", CVPR 2025", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)