OmniVCus: Feedforward Subject-driven Video Customization with Multimodal Control Conditions
Authors: Yuanhao Cai
He Zhang
Xi Chen
Jinbo Xing
Yiwei Hu
Yuqian Zhou
Kai Zhang
Zhifei Zhang
Soo Ye Kim
Tianyu Wang
Yulun Zhang
Xiaokang Yang
Zhe Lin
Alan Yuille
Venue: NeurIPS 2025 poster ()
Keywords: Video Generation
Multimodal Generation
Subject-driven Video Customization
3D/4D Generation
Flow Matching
Transformer
Primary area: applications
TL;DR: A data construction pipeline and a DiT framework for subject-driven video customization under multimodal control conditions

Abstract: Existing feedforward subject-driven video customization methods mainly study single-subject scenarios due to the difficulty of constructing multi-subject training data pairs. Another challenging problem that how to use the signals such as depth, mask, camera, and text prompts to control and edit the subject in the customized video is still less explored. In this paper, we first propose a data construction pipeline, VideoCus-Factory, to produce training data pairs for multi-subject customization from raw videos without labels and control signals such as depth-to-video and mask-to-video pairs. Based on our constructed data, we develop an Image-Video Transfer Mixed (IVTM) training with image editing data to enable instructive editing for the subject in the customized video. Then we propose a diffusion Transformer framework, OmniVCus, with two embedding mechanisms, Lottery Embedding (LE) and Temporally Aligned Embedding (TAE). LE enables inference with more subjects by using the training subjects to activate more frame embeddings. TAE encourages the generation process to extract guidance from temporally aligned control signals by assigning the same frame embeddings to the control and noise tokens. Experiments demonstrate that our method significantly surpasses state-of-the-art methods in both quantitative and qualitative evaluations. Project page is at https://caiyuanhao1998.github.io/project/OmniVCus/

OmniVCus: Feedforward Subject-rriven Video Customization with Multimodal Control Conditions
Yuanhao Cai', He Zhang?, Xi Chen Jinbo Xing", Yiwei Hu², Yuqian Zhou² Kai Zhang". Zhifei Zhang". Soo Ye Kim Tianyu Wang", Yulun Zhang".. Xiaokang Yang". Zhe Lin 2, Alan Yuille' Johns Hopkins University, Adobe Research, The University of Hong Kong, The Chinese University of Hong Kong," Shanghai Jiao Tong University
Abstract
Existing feedforward subject-driven video customization methods mainly study single-subject scenarios due to the difficulty of constructing multi-subject train- ing data pairs. Another challenging problem that how to use the signals such depth, mask, camera, and text prompts to control and edit the subject in the customized video is still less explored. In this paper, we first propose a data construction pipeline, VideoCus-Factory, subject customization from raw videos without labels and control signals such as depth-to-video and mask-to-vidoo pairs. Based an Image-Vine Video Transfer Mixed (IVTM) training with image editing data to enable instructive editing for the subject in the customized video. Then we propose diffusion Transformer framework. OmniVCus. with two embedding mechanisms, Lottery Embedding (LE) and Temporally Aligned Embedding (TAE). LE enables in- ference with more subjects by using the training subjects to activate more frame em- beddings. TAE encourages the generation process extract guidance from tempo- rally aligned control signals by assigning the same frame embeddings to the control and noise tokens. Experiments demonstrate that our method significantly surpasses state-of- ofte-of-thearr art methods in both quantitative and qualitative evaluations. Project page is at https: http://aaauaaaaaaaaee 19988. github. io/ io/proje htopprojecc jett/ /ent///m/iiii//mi VCus/
produce training data pairs for multi-
constructed data, develop
Introduction
Text-to-video diffusion generation models [1-3] have achieved great success in creating high-quality videos from user-provided text prompts. These advancements spark increasing interest in subject- driven video customization that aims create video for specific identities in user-provins usev-provided images.
Current subject-driven video customization methods are mainly divided into two categories: tuning- based and feedforward methods. Tuning-based solutions [4-6] are time-consuming. They fine-tune adapters [7, 17,8)/LoRAs [9] attached to a pre-trained video diffusion model each time for one inference. In contrast, feedforward methods [10-14] integrate the visual embeddings of subjects into diffusion models during training in a data-driven manner to enable video customization without test-time tuning. Despite progress on feedforward methods, there are still some challenges as follows:
(i) Existing methods mainly study single-subject customization due to the difficulty of constructing multi-subject data pairs. Some works have explored multi-subject data construction but still have limitations. For instance, ConceptMaster [11] constructs closed-set data pairs with limited subject categories, which degrades the model's generalization ability. (ii) As the subjects in each training video are always limited, how to enable inference with more subjects is important but under-explored. (iii) How add control conditions of different modalities to subject-driven customization is also less studied. These conditions include textual instructions to edit the subject in the generated video, camera trajectory move the viewpoint, segmentation mask sequence, depth map, and so on. The data- data-preparation pipelines of previous methods also neglect to produce control signals for the subjects.
*Corresponiing Authors
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Input
de Neww
Input
Busject
[a] Bingle Customization
Subject
[b) Instuctive Ediiin Bubject Customization
A Avemmn PTCC navengand vorer of RVGE walling
#101
Teme har the diild IMF1 Boingher tomevert
Sunject
(e) Doxble-oubeec bijcct Customizaiio
Subject Cam
(d) Camera-contbraled Subject Customization
Acag of numningty axtiee crecs EMCC and pappl flovrr
Litc-ie hoet of church B9C2
A man of 9411 wsig ligttp as I4S2 th table with crangeo of 1M64 undi scccau IMG3
Busject
(61) Zero-shot Multple--uttee suttect Du Dustomizaaion
Subject
(62) Zero-shot Multple-suiject Customization
Bus
Aeev HAUT stoe table tonn arem
hyplose the clir the warsaval (MG) with og as
Subject Mask
i Masik-ooritdddd deed Bukjjest Costomiaation
Subject - Depth
tgl Depli-contrrleed Bubjact Customizaiion In onorge from MOS sang cipaad momiite shoesins
Acrath i MMG BIrOWIC
IG11
Bubject
Mask
Masic cont oleed Bubjoct Customization Texture Transfer
Subject Daoth
(1) Dapth-conoooled Subject Customizttonn
Figure I: In (a) and (c), our method can change the pose and action of subjects. (b) The instructive editing texts are purple. In (cl) and (c2), our method trained with two subjects but can compose more subjects inference. (d), (f), and (g) are the results under different controls. the challenging cases of (g), (h), and (i), the subjects are unaligned with the mask or depth, but our method can adapt the shape or transfer the texture of the subjects.
To cope with these problems, we firstly propose a data construction pipeline, EeteoCus-Factory to produce training data pairs for multi-subject customization from raw videos without any labels. Our VideoCus-Factory first selects frame from video and uses multimodal large language model to caption the frame and detect the subjects. Then we perform subject filtering, data augmentation, and random background placement to prevent the leakage of subject size, position, and background. This improves the variation and grounding ability of the training model and enables the inference of subject images with background. Based Mixed (IVTM) training with image instructive editing data to enable instructive editing effect for the subject in the customized video. Besides, our VideoCus-Factory such as depth-to-video and mask-to-video to control the subject-driven customization. Secondly, we propose DiT-based framework, VCni to train videos, and control signals are patchified, encoded, and concatenated into long ID token to input into VmniVCus. In particular, we design two embeddings in Omni VCus. As the number of subjects in a training video is limited, we propose a Lottery Embedding (LE) to enable customization with more subjects in inference than those used in training. The core idea of LE is to use limited number of training subjects to activate the frame embeddings of more subjects. To enable more effecctive control effect of conditions, we propose Temporally Aligned Embeddings (TAE). Our TAE assigns the same frame embeddings to the noise tokens and temporally aligned control tokens with dense semantic information, such as mask and depth. For the sparse viewpoint control signal without semantic information, TAE feeds them into a multi-layer perceptron (MLP) and then add them to the noise tokens reduce the token length and computational complexity. Benefit from the constructed data and proposed techniques, our method can flexibly compose multimodal conditions to control the video customization, as shown in Fig. 7. nutshell, our contributions can be summarized as:
constructed data, we develop Image- Video Transfer
also generate signal data pairs
constructed data. The input images,
(i) We design a data construction pipeline, VideoCus-Factory, to produce training data pairs and control signals for subject-driven video customization from only raw videos. Based on our data, we develop an IVTM training strategy to enable instructive editing effect of the subject in the vidco.
(ii) We propose a new method, OmniVCus, with two embedding designs, LE and TAE, for subject- driven video customization. LE enables video customization with more subjects inference than training. TAE enables more effective control of temporally aligned signals to the customized video.
(iii) Experiments show that our method significantly outperforms state-of-the-art (SOTA) methods in quantitative metrics while yielding more visually favorable, editable, and controllable results.

---

Mask
Mast-conttddlld Customzattion
Mulli-sujedt
(i)
Subject Customization
8000
Awomen IMGI losking ins on open gerato wat onage (UGG and nucabbraa 1MGG
Mult-cutjecc- Depth Sequence
(10 Depth controlled Subject Custemization
Figure 2: Our method can flexibly compose different conditions to control multi-subject video customization.
2 Related Work
Text-to- Video (T2V) Diffusion Models [1-3, 15-28] have witnessed significant progress in recent
years. Preliminary T2V diffusion models are mainly based on stable diffusion [29], which formulates
the diffusion process in latent space and uses a U-shaped convolutional neural network (CNN) [30] as the denoiser. Yet, the model capacity of CNN limited for large-scale training. Thus, a later work DiT [31] employs Transformer [32] to replace U-Net in diffusion. These DiT-based methods [2,33-43] show very impressive performance in video generation and flexibility to add control conditions just by extending the input ID tokens. This work exploits the advancement of the DiT framework to explore its potential subject-driven video customization under different modalities of control signals.
Subject-driven Video Customization approaches are mainly divided into two categories: tuning- based [4,5,44 42,42] and feedforward methods [12, 14,5-600.. Prior tuning-basee methods are typically focused on single-subject scenarios. For instance, Dream Video [4] fine-tunes an identity adapter and combines textual inversion to customize video for subject. Videomage [52] employs subject and motion LoRAs to capture personalized content for multi-subject customization. These methods require a long time for inference. To avoid test-time tuning, later works [53] develop feedforward solutions. Some recent efforts [11, 54] such ConceptMaster [11] and Video Alchemist [12] try to construct data pairs for multiple subjects but their data pipelines still have limitations. Plus, how to add control to subject-driven video customization is still under-explored.
Method
3.1 VideoCus-Factory
Our data construction pipeline, VideoCus Factory, is depicted in Fig. VideoCus-Factory can produce training data pairs for multi-subject video customization from raw videos without any labels.
Video Captioning. For video sequence, we first randomly select a frame and then use the multi- modal large language model, Kosmos-2 [61], to caption it and detect the subjects in the frame. Take the video in Fig. 3 as example, Kosmos-2 outputs the caption "An image of bride and groom walking away from car and looking back at it", list of subjects la bride". "groom" "car"), the starting and ending positions of the subjects in the caption
[[12, 19]. [24, 29], [48, 53]], and bounding boxes the detected subjects. We modify the caption by removing the prefix "An image of" and plug in image labels such as IMG1 and IMG2 corresponding to the subjects with their positions in the caption.
Subject Filtering. We feed the detected bboxes and raw video into SAM2 [62] to track and segment the subjects. Then filter out the failure segmentation cases by thresholding the average segmentation values across frames. For example, in Fig. 3, "a car" is not segmented in some frames. Thus, filter it out. We also filter out some word clouds and large background without identity.
Data Augmentation. If we directly use the images of segmented subjects to train generation model, the scales, positions, and poses of subjects are leaked during training. As result, the model tends to learn image animation with less variation. addition, since the segmented subjects do not have background like ConceptMaster [11], directly training with this data degrades the model's grounding ability, limits its application as it needs to crop out the subject first, and easily leads to the effect. To handle these problems, center position, and augment their colors. Then we randomly select an image, which may also be pure white, and place it the background for the subject to derive the input images of training data.
randomly rotate the subjects, rescale them, move them to the
Control Signals. To add control conditions, VideoCus-Factory also constructs control signal training data pairs. As shown in Fig. 3, the mask sequence of the subject and the raw vidco with the modified

---

Video Death
Captar
Anything
Dopth Sequence
Faeibee of will Coster 102 198 (6.53)
Raw Video
car
Subject Filtering
Belest
Kosmoe-2
SAM-22
Row Video
Selected Frame
Bounding Booss
Aeta gueente webhiinghokuus au
Basic Training Data Pars
Meaks
Mulipication
Caption
Buckgrourd
Augrwotation
'groom"
Mask Seqquence Bubject
Triining Imagas Sutjects
Augneented Sudjects
Sutjects
Our data construction pipeline VideoCus-Pactor uses Kosmos-2 [61] caption the raw video and detect the subjects. Then we use SAM-2 [62] to segment and filter the detected subjects to derive the training input images. VideoCus Factory also constructs control data pairs such as mask-to-videe and depth- tideo.
Figure 3:
caption without image labels form the data pairs for mask-to-vidd control data. The depth sequence predicted by the video-depth-anythhin [63] and the raw video with the same caption form the depth- to vodvideo control data pairs. Note that the mask-to-video and depth-to-video data are not paired with the subject-driven customization data in training. Our model can flexibly compose them in inference.
3.2 OmniVCus
As shown in Fig. 4, OmniVCus is a DiT-based framework. It can be mixed trained with different tasks, including single-/double-subjjed customization, sopth-/maas-too-videe to-ima po-image/-vviees and image instructive editing. Texts, images, and videos are patchified, encoded into the latent space, concatenated with the noise tokens, and fed into the full-attention DiT. OmniVCus can flexibly compose tokens of different signals to control and edit the subject in the customized video. We notice that the condition frame tokens are mainly divided into two parts: the image tokens containing the subjects, and the other temporally aligned tokens of control signals. To handle these two types of tokens, we design Lottery Embedding (I.E) and Temporally Aligned Embedding (TAE).
cext-to-multiview, text-
Lottery Embedding. As the subjects in each training sample are limited, it is important to enable customization with more subjects inference than training. To this end, Lottery Embedding (LE) uses the limited subjects in the training samples to activate more frame embeddings, as shown in Fig. 4 (a). Denote the max number of subjects
training sample as K and the number of subjects we aim to compose as M (MM > K). Then LE randomly selects a set S of K numbers from [1, M] as
S Unif({ ( (AS (1. M) |A| KKK K)),
(1)
where Unif denotes the uniform distribution. Since the Transformer is unordered, we need to create an order for it the frame embedding and match the order of the image index label. Thus, we sort
in ascending order to derive S, and then assign the elements of Sy as frame embeddings to the reshaped, undergo MLP, and then added
input images of subjects. These frame embeddings to the tokens of the corresponding subject images. By our LE, we can activate more frame position embeddings during training and enable zero-shot mure-subject video customization in inference.
Temporally Aligned Embedding. mask are temporally aligned with the generated video. Thus, better direct the Transformer model to extract guidance from these control signals, TAE assigns the same frame embeddings to the control tokens and noise tokens. In Fig. 4 (b), denote the length of the generated video as N. Then the frame position embeddings for control tokens and noise tokens are {M 1, M + 2, .M + N To distinguish the control signals and noise, we only add the timestep embedding to the noise tokens.
notice that the control signals, such as camera, depth, and
In particular, the depth and mask sequences are structural signals containing fine- mne-grained spatial and semantic information. Hence, we use a 3D- VAE to encode them to preserve these information. In contrast, the camera signals not contain such information. On the other hand, the computational complexity of Transformer is quadratic to the length of input tokens. Thus, we integrate the camera signals into noise instead concatenating them. Specifically, to enhance the control of camera signals and spatially align them to the noise tokens, adopt the pixel-aligned ray embeddings,

---

Instructive Prompt
IMO1 IPS52 IMDS IMB5
PHD1 IM65
Task driven Customization
InpucTed UNG
Tral qualederss
(a) Lattery Embedding
Imagia Encoder
Tank Instructive Editing Customization
Diffusion Transformer
Text Male
Images of Subjects
Me MA
Full FullAttmton Block
Full Atzemtonnnn Blook
Full Arl.Attenoo Blook
3D VAE Enooder
F Cll.Anseniio
Task Mask cont--ooeeeeeed Customization
Texe Adde corcico wug wwwplpapeeeeeeeoooooooooooo
Mask
Depth
Structure Control
Decuoka
HO
Itat Caram
Task Camera-contron controlled Customization
AMPP
(D/eepporally Aligned Emibadiing g
Fomm Embrarring
Camera Trajectory
Camera Control
Fooo Weghtt
Figure 4: OmniVCus is DiT architecture that can compose different input signals to customize a vidco. (a) LE enables more-subject customization in inference by activating more frame embeddings with training subjects. (b) TAE extracts the guidance from control signals by aligning the frame embeddings of condition and noise tokens.
plücker coordinates, parameterized as r = (o x d, d), where o and d are the position and direction of the ray landing on a pixel. Then the plücker coordinates are patchified and undergo an MLP to add with the noise. Denote the input structure control tokens as Tip the noise tokens as TH and the tokens of plücker coordinates as Té. Then the mapping function of TAE, STAF, is formulated as
(Tout, Tout) = STAR (Tén Tim T sct
T + MLP,
+ (LP + MLP,(() + MLPP. (T.))
where ps is the frame position, is the timestep, and MLP,, MLP,, MLP, three MLPs. The output tokens Tout and Toue are then concatenated with other tokens to be fed into the DiT model.
Image-Video Transfer Mixed Training. Due to the lack of training data pairs for subject-driven customization with instructive editing. We develop an Image- Video Transfer Mixed (IVTM) training strategy to enable the editing effect for the subject of interest without constructing new data pairs. As the image instructive editing training data sufficient, our goal is to transfer the editing effect from image to video. To this end, need construct common task image and video the bridge for the knowledge transfer of instructive editing from image-to-image to image-to-video. In our IVTM training, this common task pair is single-subject image and video customization. For the image customization, we select the caption frame in Fig. 3 with the processed image of subject as the training pairs. In Fig. 4, we align the frame position embeddings of the input images of the image instructive editing and single-subject image/video customization in our IVTM training for better transferring. This frame embedding is also assigned by our LE in Eq. (1) with K
allow better composing instructive editing with multi-subject customization. In inference, we compose the prompts of image instructive editing and subject-driven customization to activate the editing effect.
Training Objective. Our model is mixed trained with different tasks using the flo--matching loss [64, 65]. Specifically, denoting the ground-truth video latents X and noise as X ~ (0, 1), the noisy input is produced by linear interpolation X = [X] + (1 x)x at timestep L. The model = X X Then the overall training objective is formulated as
ne predicts the velocity V
Dt x
[|v' (x ,t Ctsee Camgs Cdepthy Cmasky Ctras))l2]
(3) where Ctzt, Cangs Cdepth, Cmask, and Ciraj denote the input texts, images, depths, masks, and cameras. As different training tasks are not paired with each other, some input conditions are missing in different training samples. But model can flexibly compose different input signals in inference.
min Ex
4 Experiment
4.1 Experimental Settings
Dataset. For subject-driven video customization, depth-to-video, and mask-to-video generation, we our CalcoCus-Factory to create 1.2M, 1.4M, and 1.66 training data pairs. The data for these
5

---

IMG1
The woman in IMG1 is talking to a man on a street
Subject
VideoBooth
DresmVideo
SkyAeels--2
OmniGen 1on2.1-12V
Our Method
Figure 5: Visual comparison of sugle-subject video customization with state-of of the-art algorithms. Our method can change the pose and viewpoint of the subject while keeping the identity such as the hair, jacket, and sweater.
Methods
Subject
CLIPP-T
CLIP-1
DINO-1
Consistency
Dynamic
Methods
Alignment
Meniily
Quality
VideoReath [53]
Single
0.2541
0.5891
0.3033
0.9593
0.4287
VideoBoooh [53]
91.9
94.6
DreamVidzo [+]
Single
0.2799
0.6214
0.3792
0.9609
0.4696
Vidam [4]
89.2
89.2
97.3
WanZ. 1-12V [1]
Single
0.2785
0,6119
0:4203
0.9154
0.5310
Wan2.1-12V [i]
83.8
36.4
73.0
SkyRoels [54]
Single
0.2820
0.6609
0,4612
0.9797
0.5238
SkyReels [54]
75.7
81.1
70.3
Ours
Single
0.3293
0.1144
0.2255
0.9928
0.5541
Ours
SkyReels [54)
Multiple
0.2785
0.6429
0,4107
0.9710
0,5892
SkyReels [54]
73.0
78.4
67.6
Ours
Multiple
0.3264
0.6672
0.1965
0.9908
0,6878
Ours
(a) Comparison of subject driven video customization.
(b) User preference (%) of our method.
Methods
CLIP-T
CLIP-I
DING-1
Consistency
Dynamic
Methods
Aligament
Idaatity
Quality
VideoBoooh [53]
0.2453
0.5935
0,2989
0,9570
0,5825
VideoBooth [53]
94.6
94.6
91.9
DrcamVidoo [4]
0.2642
0.6116
0,3511
0.9604
0,5760
DisunVidco [4]
91.9
97,3
94.6
Wan2. [1]
0,2690
0.6208
0.3722
0.6157
Wan2. 1-12V [I]
78.4
81.1
67.6
SkyReels [54]
0.2761
0.6368
0,4259
0.9635
0.5904
SkyReels (54]
73.0
75.7
64.9
Ours
0.3126
0.7061
0.4942
0,9915
0.6226
Ours
(c) Comparison of instructive editing for subject customization.
(d) User preference (%) of our method.
Methods
CLIP-T
CLIP-I
DINO-1
Consistency
Dynamic
Methods
Alignment
Identity
Quality
Motionctrl [66]
0,2984
0,5215
0,2066
0.9857
0,4272
Motionetrl [66)
91.9
86.8
78.4
Cameractd [67]
0,2909
0.5163
0.1982
0,9711
0,5845
Cameractdl [67]
70.3
89.2
91.9
Cam12V [68]
0,2871
0.5365
0,2248
0.9660
0.5623
Cam12V [68]
73.0
83.8
81.1
Ours
0.3104
0.6751
0.5233
0.9911
0.6204
Ours
(c) Comparison of
subject customization.
(f) User preference (%) of method.
Table 1: Quantitative results and user study of sfate-of-the-ae subject-driven video customization methods.
three tasks not paired with each other, and every input video sequence our VideoCus-Factory is randomly selected from our internal video data pool. For subject-driven video customization, the scale factor randomly selected from 0.7 1.3 and the color augmentation includes brightness scaling (0.9~1.1), linear contrast adjustment (0.9~>.1)), saturation scaling (0.9~~1)) 1.1), and hue shift t-10~~09% For text-to-mulliview. and short text prompts as the training samples. We adopt the OmniEdit [70] as the image instructive editing dataset containing 1.2M data pairs. Besides, we also fine-tune the model with text-to-image (~300M) and text-to- video (~1M) data. In evaluation, we collect 1 112 samples for single-subject customization and instructive editing customization, 76/74/56 samples for huubll-/trille/quadrup subject customization, and 112 samples for camera-controlled subject-driven video customization.
select 320K samples from Objaverse [69] labeled with long
Implementation Details. Our model is fine-tuned from T2V model with 5B parameters for 100K steps in total at a batch size of 356 on 64 A 100 GPUs for 5 days. We adopt the AdamW optimizer [71] (8) = 0.9, 82- = 0.95) with weight decay of 1. The learning rate linearly warmed up to 1e-5 with 2K iterations and decays to lc using cosine annealing [72]. The spatial resolution of training images and videos is set 512x512 for text-to-mulliview and image instructive editing and 384 X80 for other tasks. The frame number and fps of the training video are set to 64 and 24. We use five metrics for evaluation. (1) CLIP-T computes the average cosine similarity between CLIP [73] image embeddings of all generated frames and their text embedding. We remove the image index label and adapt the text prompts after instructive editing when computing CLIP-T. (2) CLIP-I calculates the average cosine similarity between the CLIP image embeddings of all generated images and the target images. (3) DINO-I [74] also measures the visual similarity between the generated and target subjects using ViTS/16 DINO [75]. (4) Temporal Consistency computes the CLIP image embeddings

---

IMG1
Remove the blazer of the man IMG1 standing in a summer park
Subject
OmniGen- VideoRooth
OmniGan DreamViden
Omniien SkyReals-A7
Method Figure 6: Visual comparison instructive editing subject-driven video customization with SOTA methods. VideoBooth, Dream Video, and SkyReels-A2 first use OmniGen to edit the subject and then customize the video. Wan2. 1-12V [1] uses OmniGen to cdit and customize the subject and then animate the image to derive the vidco.
OmniGen, Wan2.1-12V 1-12v
dog IMG1 playing with yellow toy of IMG2 living
IMG1
1M02
orange IMG1, grapes IMG2, IMG3 falling into the
IMG1
IMG2
woman IMG1 doing squat bird IMG3 flying front as IMG2
Subject
SkyRools-A2
Our
Figure 7: Visual comparison of multi-subject vidco customization with the SOTA method SkyReels-A2 [54].
and averages the cosine similarity between every pair of consecutive frames. (5) Dynamic Degree is computed as the optical flow predicted by RAFT [76] magnitude between consecutive frames.
4.2 Main Results
Composing Different Control Conditions. As shown in Fig. and 7, our model can flexibly compose different control signals. (i) In Fig. 1 (b), benefit from IVTM training, model can modify the subject and transfer its style to sketch. (ii) Benefit from LE, our model trained with 2 subjects but compose 4 subjects in inference, as shown in Fig. (cl) and (c2). (iii) In Fig. (f) and (g),
model can change the pose and action of subjects following the mask or depth. (g) is a hard case where the depth is from man but model can fill the depth with the woman while keeping the face identity and swapping her shirt with a suit following the instruction. In harder cases where the subjects severely unaligned with the mask or depth, subjects. In Fig. 1 (h), the model transfers the appearance of church to the mask of house. In Fig. ()), the texture of orange transferred to the depth of strawberry. (iv) Even for more challenging multi-subject cases under different control signals in Fig. 7, our model can still robustly handle them.
model can still transfer the texture of
Comparison with SOTA Methods. (i) We compare OmniVCCu with 4 SOTA methods including an 12V method (Wan2.1-12V [1]), two single-subjed video customization methods (Dream Video [4] and VideoBooth [53]). and multi-subject video customization method (SkyReels-A2 [54]) on subject-driven video customization without and with instructive editing in Tab. 1a and Ic In Tab. la, Wan2.1-12V [1] uses the SOTA image customization model OmniGen [77] to first customize the subject and then animate it. The results for multi-subject customization are averaged on double- and triple-subject customization, as Sky SkyReels-A2 can support at most three subjects. In 1c. OmniGen is first used to instructively edit the subject for all compared basclines as they show limitations in editing the subject itself. Our OmniVCus significantly outperforms previous methods in all tracks in both Tab. 1a and Ic, suggesting its advantages in identity preserving and high-quality video generation. Fig. 6, and 7 show the visual comparisons of single-subject customization without and with instructive editing and multi multi-subject customization. In Fig. 5, our method can better change the pose of the woman while keeping the identity such as the hair, jacket, and sweater. Fig. 6, our method can better follow the text to remove the blazer while keeping the man's identity and customize the background. In Fig. 7, SkyReels-A2 misses some subjects and tends animate the image. In contrast, our method can better compose all the subjects and follow the texts to customize the video.
7

---

IMG1
A waman in a mask as IMG1 holding a book and sitting an couch in living room
Subject
CmnnGGn Motionctrl
OmniGen Cameractrl
Canners Trajectory
OmniGen Cami2V
Our Method
Figure 8: Comparison of cameral-controoled subject-driven video customization. Motionetrl, Cameractrl, and Caml2V first employ OmniGen to customize the image of subject and then animate it following the camera.
CLIPT
DINO-1
Consistency
Dynamic
Embedding
CLIP-T
DING-1
Consariency
Dynamic
Baseline
0.2175
0,2405
0.9588
0.3759
Naive
0.2618
0.2947
0.9511
0.4948
+ SSubject Filtering
0
0,5053
0.9617
0.3826
Add to Noise
0.1222
0.1680
0,9319
0,5437
Data Augementation
0,3293
0.5215
0.9288
0:5541
Our TAE
0,3054
0,3794
0.9909
0.4966
(a) Ablation of VideoCus FaltoCus-Factory data construction
(b) Ablation of our Temporally Alignment Embedding
Emehedding
JP-P-T
Consistency
Dynamic
Training Method
Mixal
Direct Mixed
Our
weo LE
0,2105
0.3361
0.9702
0.6913
CLIP-T
0.2137
0,2585
0,3126
with LE
0.2728
0.4163
0,9810
0.6806
User Pref. (%)
91.9
75.7
(c) Ablation study of our Lottery Embedding mechanism
(d) Ablation of our mixed training for editing effect
Table 2: Ablation study. (a) is conducted on single-subject customization. (b) is done on depth-controlled tomization, (c) is done on multi-subject customization. (d) done on instructive editing subject customization.
(ii) Tab. 1e compares OmniVCus with three SOTA camera-controlled I2V methods (Motionctrl [66]. Cameractrl [67], and Cam12V [68]). They use OmniGen to customize the subject first and then animate Our method surpasses the recent best method Cam12V by 0.2985 in DINO-I. Fig. 8 shows the visual results. Our method can better keep the identity of woman and follow the camera trajectory.
(iil) We also conduct user study with 37 participants on single-subject video customization without and with instructive editing and camera-controlled subject-driven video customization in Tab. 1b, 1d, and If. Each participant views the videos generated by OmniVCus and a random competing method, along with the images of the subject, text prompts, and control signals. The participants are asked three questions: 1) Which video aligns better with the customization prompt/editing iestruction/camert trajectory? 2) Which video better keeps the identity of the subject? 3) Which video has better quality? As reported in Tab. 1b, 1d, and If, reports the user preference (%) of our method over competing entries. Our method outperforms all SOTA methods by a large margin.
4.3 Ablation Study
VideoCus-Factoryy We conduct experiments on single-subject video customization to study the steps of our VideoCus-Factory in Tab. 2a. We remove the subject filtering and data augmentation including random background placement from VideoCus-Factory as the bascline. When we apply the subject filtering, the DINO-I score significantly improves by 0.2648 because the left training video data after filtering can keep the subject in all frames. Subsequently, when using the data augmentation, the CLIP-T and dynamic degree gain by 0.0862 and 0.1715. This is because the segmented subject without data augmentation leaks the position, scale, and background in the training process. As result, the customized videos have less variation and suffer from the copy- copy-paste issue. As shown in Fig. 9 (a), with our data augmentation, the dog can change its pose in the customized video.
Temporally Aligned Embedding. We conduct experiments study the effect depth-controlled subject video customization in Tab. 2b. We compare our TAE with two options: 1) Naive method that assigns different frame embeddings to the tokens of depth (length N) and noise (length N) from M + 1 to M + 2NV and the timestep embedding is added to all tokens. 2) Adding the depth to the noise after undergoing an MLP. As listed in Tab. 2b, the add-to-noise method causes model collapse because the fine- tne-grained spatial information in depth tokens is corrupted by the noise. Our TAE surpasses the naive embedding by a large margin in CLIP-T and DINO-I scores. As shown in Fig. 9 (c1) and (c2), our TAE can help the model better follow the guidance of depth to generate higher-quality video with fewer artifacts. Besides, TAE can help the control signals to be better composed with other tokens. The naive embedding fails to customize the bottle of the beer from text tokens and keep the identity of the boy from image tokens. In contrast, TAE can preserve the identity and follow the prompts to customize the scenario. Fig. 9 (dl) and (d2) compare the naive embedding
8

---

oudject
wb Dita Augmenaztion
with Data Augnentation
Sudjec
Camera
(o1) Narve Viteppant tokan
"Tmppors Aliyeed Einoedking Viewpoint Control
Analysis Augmeriation VideoCup Factory
Analysis Dife Mechanisms
A IMS1 boolig hes open veft pereprttooann with torntt and bottle beer
Subject
Dopth
(61) Natro Frome Embedding Dopth controled Customization
Dopth
[c2] Depth Customizaton
Apunplinal Sozking
Oriihal Video
a Mask Suquence
Subjec!
(61) Nabra Emtedding for Misk-cortrclldd Teclure Transfer
(4Z) Our TAE Misk-cortroled Teclure Transfer
A Apzgga MGG coanntrra PIG3 ander orages MICH athwwartooingg
Buljeot
wb LoteryErnboddin
WithLotery Embodding
Our Mixod Training (1) Analyis Miledd Trwining for Instructive Editin Efficd
Bubject
Direct Maod Training
(o)Amalysii using the Lodery Embodding MuB subject Customization
Figure 9: Visual analysis. (a) Using the data augmentation in our VideoCus- TideoCus-Factory and action of subject in the customized video. (h) Our TAE can hetter control the viewpoint. (cl) and (c2) show that our TAE can improve the consistency. video quality, and guidance of depth. (dl) and (d2) show that our TAE can better handle the case that the subject is not aligned with the mask by transferring the texture. (e) Using our LE better preserves the identity in zero-shot multi-subject customization. (l) studies the effect of IVTM training.
vary the scale, pose,
and TAE on mask-controlled customization. When the subject is unaligned with the mask of the black cylinder, the naive embedding customizes low-quality video without the pumpkin texture while generating undesired black edge. In contrast, TAE can better transfer the texture with less artifacts.
We also compare the camera embedding in TAE with the naive method that directly concatenates the viewpoint tokens into the overall long 1D tokens. This naive method leads to an increase in training time by 10% as the computational complexity of self-attention is quadratic to the length of input tokens. As shown in Fig. (b1) and (b2), our TAE can control the viewpoint rotation more effectively.
Lottery Embedding. We conduct experiments on triple- and quadruple-subject customization that do appear in the training data study the effect of our LE on zero-shot more-subject customization. The averaged results are reported in Tab. 2c. When using our LE, the CLIP-T and DINO-I scores are significantly improved by 0.0623 and 0.0799. We also conduct visual analysis in Fig. 9 (c), without using LE, the customized video misses the subject of orange and mistakenly extracts the tomato from IMG3 instead of the desired subject, the cucumber. In contrast, using our LE can accurately compose the four subjects with proper scales and physically consistent motion in the customized video.
Image-Vinn Video Transfer Mixed Training. We conduct experiments the instructive editing single- subject customization in Tab. 2d to study the effect of our IVTM training strategy. CLIP-T score and the user preference percentage of IVTM training are reported. Our IVTM performs much better than no mixed training and direct mixed training with the OmniEdit dataset. We observe that the direct mixed training still not enable some hard instructive editing categories such as removal, color change, style transfer, etc. Fig. 9 (f) shows an example of style transfer. The direct mixed training can not follow the editing instruction while our IVTM can customize the church in an oil painting style.
5 Conclusion
In this paper, we focus on studying the subject-driven video customization with different control conditions feedforward manner. We first propose data construction pipeline, VideoCus-Factory, to produce training data pairs. Our IideoCus-Factory can also create depth-to-vided and mask-to- video control signal data. Subsequently, we present a DiT-based framework, Omni OCni, with two embedding mechanisms, LE and TAE. LE enables more-subject video customization in inference by using limited training subjects to activate more frame position embeddings. TAE enhances the control effect of temporally aligned signals by assigning the same frame embeddings to the control tokens and noise tokens. Experiments show that our method outperforms SOTA algorithms both quantitative and gualitative evaluations while achieving more flexible control and editing effects.

---

A building fire
4D-fy
Our Method
A clown fish swimmiing through the coral reaf
MAV3D
Our Method
Figure 10: Text-to-4D generation comparison with 4D-fy [78] (upper) and MAV3D [79] (lower).
6 Limitations
The main limitation of our method is that it takes a long time (about three months) to create data. Meanwhile, the labeling quality also constrained by the capability of the used foundational models. Specifically, the quality of the caption label and detection bounding boxes is constrained by the model capability of Kosmos-2 [61]. The quality of mask-to-video and depth-to-video control data pairs is constrained by SAM2 [62] and Sideo-Deth-AAnyee capability of these foundational models improves, the quality data labeling will be better.
[63]. However, we believe that as the
Broader Impact
Subject-driven video customization is an important and challenging topic in computer vision. It enables creators to quickly generate edit video content based on specific subjects (people, objects scenes), giving rise variety of applications: from short video clips of few seconds social media, to special effects synthesis at the film industry level, to personalized narratives and immersive experiences in advertising, education and digital cultural heritage. This technology not only lowers the threshold for high-quality content production, unleashes the creative potential of small and medium-sizze sized studios and independent bloggers, but also brings higher efficiency and more flexible iteration space to the traditional film and television production process.
Until now, subject-driven video customization techniques have no negative social impact yet. Our proposed method does not present any negative foreseeable societal consequences, either.
Model Emergence Capability: Text-to-4D
As aforementioned, model mixed trained with customization data (dynamic) and text-to-3D data (static) can perform text-to-4D generation. Please note that the 4D here refers to dynamic multi-view. Our model does not directly contain 4D representation.
We compare our method with two SOTA text-to-4D generation methods: MAV3D [79] and 4D-fy [78] in Fig. 10. Our method can generate more 3D-consistent novel views with higher dynamic degree.
Please refer our project page for more video dynamic generation results.
Acknowledgement
This work was supported by the office of Naval Research with award N000142412696 and in part by Adobe Inc.
References
[1] A. Wang. B. Ai. B. Wen, C. Mao, C.-W. Xie, D. Chen, F. Yu. H. Zhao, J. Yang. J. Zeng. et al. "Wan: Open and advanced large-scale video generative models." arXiv preprint vrXiv:2503.20314 2025.
10

---

[2] Z. Yang, J. Teng, W. Zheng, M. Ding, S. Huang. J. Xu, Y. Yang, W. Hong. X. Zhang. G. Feng, et al., "Cogvideox: Text-to-video diffusion models with an expert transformer," in ICLR, 2025.
[3] T. Brooks, B. Peebles, C. Holmes, W. DePue, Y. Guo, L. Jing, D. Schnurr, J. Taylor, T. Luhman, E. Luhman,
C. Ng. R. Wang, and A. Ramesh, "Video generation models as world simulators." Sora. Accessed Mar., 2024 [Online].
[4] Y. Wei, S. Zhang. 7- Qing, H. Yuan, Z. Liu, Y. Liu, Y. Zhang, J. Zhou, and H. Shan, "Dreamvideo: Composing your dream videos with customized subject and motion." in CVPR, 2024.
[5] E. Molad, E. Horwitz, D. Valevski, A. R. Acha, Y. Matias, Y. Pritch, Y. Leviathan, and Y. Hoshen, "Dreamix: Video diffusion models are general video editors," arXiv preprint arXiv:2302.0122 2023.
[6] Y. Wei, S. Zhang. H. Yuan, B. Gong. L. Tang, X. Wang. H. Qiu, Li, S. Tan, Zhang, al., "Dreamre- lation: Relation-centric video customization," " arXiv preprint arXiv:250..0600, 2025.
[7] L. Zhang, A. Rao, and M. Agrawala, "Adding conditional control to text-to-image diffusion models," in ICCV, 2023.
[8] C. Mou, X. Wang, L. Xie, Y. Wu, J. Zhang, Z. Qi, and Y. Shan, "TZi-adapter: Learning adapters to dig out more controllable ability for ext-to-image diffusion models," in AAAI, 2024.
[9] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhut Y. Li, S. Wang, L. Wang, W. Chen, et al., "Lora: Low-rank adaptation of large language models.." in ICLR, 2022.
[10] Y. Wei, Zhang, H. Yuan, X. Wang, H. Qiu, R. Zhao, Y. Feng, F. Liu, Z. Huang, J. Ye, et al., "Dreamvideo-2: Zero-shot subject-driven video customization with precise motion control," arXiv preprint UrXiv:2410.138300 2024.
[11] Y. Huang, Z. Yuan, Q. Liu, Q. Wang, X. Wang. R. Zhang, P. Wan, D. Zhang, and K. Gai, "Conceptmaster: Multi-concept video customization on diffusion transformer models without test-time tuning," arXiv preprint Xr0:
2025.
[12] T.S.. Chen, A. Siarohin, W. Menapace, Y. Fang, K. Lee, I. Skorokhodov, K. Aberman, J.-Y. Zhu, M.-H. Yang, and S. Tulyakov. "Multi-subject open-set personalization in video generation." in CVPR, 2025.
[13] T. Hu, Z. Yu, Z. Zhou, S. Liang, Y. Zhou, Q. Lin, and Lu, "Hunyyancustom:: A multimodal-driven architecture for customized video generation," " arXiv preprini WwXiv:2505.04512, 2025.
[14] X. Ju, W. Ye, Q. Liu, Q. Wang, X. Wang, P. Wan, D. Zhang, K. Gai, and Q. Xu, "Fulldit: Multi-task video generative foundation model with full attention," arXiv preprint arXiv:2503. 00.225533.997.... 2025.
[15] X. Chen, 7. Liu, M. Chen, Y. Feng, Y. Liu, Y. Shen, and H. Zhao, "Livephoto: Real image animation with text-guided motion control," in ECCV, 2024.
[16] Xing, M. Xia, Y. Zhang, H. Chen, W. Yu, H. Liu, G. Liu, X. Wang, Y. Shan, and T.-T. Wong. "Dynami- crafter: Animating open-domain images with vidco diffusion priors," in ECCV, 2024.
[17] Wang, H. Yuan, D. Chen, Y. Zhang, X. Wang, and S. Zhang, "Modelscope text-to- video technical report," arXiv preprint arXiv:2308.065: 06571, 2023.
[18] A. Blattmann, R. Rombach, H. Ling. T. Dockhorn, S. W. Kim, S. Fidler, and K. Kreis, "Align your latents: High High-resolution video synthesis with latent diffusion models," CVPR, 2023.
[19] R. Girdhar, M. Singh, A. Brown, Q. Duval, S. Azadi, S. Rambhatla, A. Shah, X. Yin, D. Parikh, and
I. Misra, "Emu video: Factorizing text-to-video generation by explicit image conditioning," arXiv preprint
1rXiv:2311.10709, 2023.
[20] Z. Luo, D. Chen, Y. Zhang, Y. Huang, L. Wang, Y. Shen, D. Zhao, J. Zhou, and T. Tan, "Videofusion: Decomposed diffusion models for high-quality video generation," in CVPR, 2023.
[21] A. Mahapatra, A. Siarohin, H.-Y. Lee, S. Tulyakov, and J.-Y. Zhu, "Text-guided synthesis of culerian cinemagraphs," ACM TOG, 2023.
[22] U. Singer, A. Polyak, Hayes, X. Yin, J. An, S. Zhang, Q. Hu, H. Yang. O. Ashual, O. Gafni, et al., "Make-a-video: Text-tovvideo generation without text-video data," arXiv preprint arXiv:2209. 2022.
[23] D. Zhou, W. Wang, H. Yan, W. Lv,, Y. Zhu, and J. Feng, "Magicvideo: Efficient video generation with latent diffusion models," arXiv preprint arXiv:2211. 2022.
[24] Y. He, T. Yang, Y. Zhang, Y. Shan, and Q. Chen, "Latent video diffusion models for high-fidelity long video generation," " arXiv preprint arXiv:2211.3322. 2022.
[25] A. Blattmann, T. Dockhom, S. Kulal, D. Mendelevitch, M. Kilian, D. Lorenz, Y. Levi, Z. English, V. Voleti, A. Letts, et al., "Stable video diffusion: Scaling latent vidco diffusion models to large datasets," arXiv preprint arXiv:211..1522. 2023.
[26] J. Ho, T. Salimans, A. Gritsenko, W. Chan, M. Norouzi, and J. Fleet, "Video diffusion models," NeurIPS, 2022.
11

---

[27] S. Zhang, J. Wang, Y. Zhang, K. Zhao, H. Yuan. Z. Qin. X. Wang, D. Zhao, and J. Zhou. "I2vgen-xl: High-quulity image-to-video synthesis via cascaded diffusion models," " arXiv preprint Sedi::2311.04111 2023.
[28] Y. Wang, X. Chen, X. Ma, S. Zhou, Z. Huang. Y. Wang, C. Yang. Y. He, J. Yu, P. Yang, et al., "Lavie: High-quality video generation with cascaded latent diffusion models," IJCV, 2025.
[29] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. "High-resolution image synthesis with latent difffusion models," in CVPR, 2022.
[30] O. Ronneberger, P. Fischer, and T. Brox, "U-net: Convolutional networks for biomedical image segmenta- tion," in MICCAI, 2015.
[31] W. Peebles and S. Xie, "Scalable diffusion models with transformers," in ICCV, 2023.
[32] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L.. Jones, A. N. Gomez, f... Kaiser, and I. Polosukhin, "Attention is all you need," in Advances in neural information processing systems, 2017.
[33] T. Brooks, B. Peebles, C. Holmes, W. DePue, Y. Guo. L. Jing, D. Schnurr, J. Taylor, T. Luhman, E. Luhman, et al., "Vidco generation models world simulators," " OpenAl Blog, 2024.
[34] W. Hong, M. Ding, W. Zheng. X. Liu, and J. Tang, "Cogvideo: Lcage-scale pretraining for text-to-video generation via transformers." " in ICLR, 2023.
[35] W. Menapace, A. Siarohin, I. Skorokhodov, E. Deyncka, T.-S. Chen, A. Kag, Y. Fang, A. Stoliar, E. Ricci, J. Ren, et al., "Snap video: Scaled spatiotemporal transformers for text-to-video synthesis"" in CVPR, 2024.
[36] H. Lu, G. Yang. N. Fei, Y. Huo, Z. Lu, P. Luo, and M. Ding, "Vdt: General-purpose video diffusion transformers via mask modeling." in ICLR, 2024.
[37] X. Ma, Y. Wang, G. Jia, X. Chen, Z. Liu, Y.-F Li, C. Chen, and Y. Qiao, "Latte: Latent diffusion transformer for vidco generation," TMLR, 2024.
[38] P. Gao, L. Zhuo, D. Liu, R. Du, X. Luo, Qiu, Zhang, Lin, R. Huang, Geng, et al., "Lumina-12x: Transforming text into any modality, resolution, and duration via flow-based large diffusion transformers," arXiv preprint arXiv:2405,0m 05945, 2024.
[39] A. Polyak, A. Zohar, A. Brown, A. Tjandra, A. Sinha, A. Lee, A. Vyas, B. Shi, C.-Y. C.-Y. Chuang, et al., "Movie gen: A cast of media foundation models," arXiv preprint 2111:24100 2024.
[40] A. Gupta, L. Yu, K. Sohn, X. Gu, M. Hahn, F.-F. Li, I. Essa, L, Jiang, and J. Lezama, "Photorealistic vidco generation with diffusion models," in ECCV, 2024.
[41] D. Kondratyuk, L. Yu, X. Gu, J. Lezama, J. Huang, G. Schindler, R. Hornung, V. Birodkar, J. Yan, M.-C. Chiu, et al. "Videopoet: A large language model for zero-shot video generation." in ICML, 2024.
[42] J. Xing, I.. Mai, C. Ham, J. Huang, A. Mahapatra, C.-W. Fu, T.-T. Wong, and F. Liu, "Motioncanvas: Cinematic shot design with controllable image-to-video generation," arXiv preprint arXiv:2502.04229, 2025.
[43] S. Liu, T. Wang. J.H Wang, Q. Liu. Z. Zhang, J.-Y. Lee, Y. B. Yu, Z. Lin, S. Y. Kim, et al., "Generative vidco propagation," arXiv preprint 4rXiv:2412.1976/ 2024.
[44] X. He, Q. Liu, S. Qian, X. Wang, T. Hu, K. Cao, K. Yan, and J. Zhang, "Id-animator: Zero-shot identity- preserving human video generation," " arXiv preprint arXiv:2404. /5575, 2024.
[45] H. Ye, J. Zhang, S. Liu, X. Han, and W. Yang. "Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models," Xrviv preprint 2xX8v:2308.067.. 2023.
[46] Y. Huang, Y. Qin, S. Lu, X. Wang, R. Huang, Y. Shan, and R. Zhang, "Story3d-agent: Exploring 3d
storytelling visualization with large language models," arXiv preprini arXiv:2408. 11800, 2024.
[47] T. Wu, Y. Zhang, X. Wang, X. Zhou, G. Zheng, Z. Qi, Y. Shan, and X. Li, "Customerafter: Customized vidco generation with preserving motion and concept composition abilitics," in AAAI, 2025.
[48] J. Materzyúskka, J. Sivic, E. Shechtman, A. Torralba, Zhang, and B. Russell, "Newmove:
Customizing
text-to -iiieoo models with novel motions," in ACCV, 2024.
[49] 7. Ma, D. Zhou, C.-H. Ych, X.-S. Wang, X. Li, H. Yang, 7. Dong, K. Keutzer, and Feng, "Magic-me: Identity-speciff video customized diffusion," " arXiv preprint arXiv:2402.099: 2024.
[50] F. Long. Z. Qiu, T. Yao, and T. Mei, "Videodrafter: Content-consistnt multi-scene video generation with 1lm," CORR, 2024.
[51] Y. Fang, W. Menapace, A. Siarohin, T.-S. Chen, K.-C Wang, I. Skorokhodov, G. Neubig, and S. Tulyakov, "Vimi: Grounding video generation through multi-modal instruction," in EMNLP, 2024.
[52] C.-P. Huang, Y.S. Wu, H.-K. Chung. K.PP. Chang. F.-E. Yang, and Y.-C F. Wang. "Videomage: Multi- subject and motion customization of text-to-video diffusion models," in CVPR, 2025.
12

---

[53] Y. Jiang, T. Wu, S. Yang, C. Si, D. Lin, Y. Qiao, C. C. Loy, and Z. Liu, "Videobooth: Diffusion-based video generation with image prompis," " in CVPR, 2024.
[54] Z. Fei, D. Li, D. Qiu, J. Wang, Y. Dou, R. Wang, J. Xu, M. Fan, G. Chen, Y. Li, et al., "Skyreels-a2: Compose anything in video diffusion transformers"" arXiv preprint arXi:2204.02436, 2025.
[55] Y. Zhou, D. Zhou, M.-M. Cheng, J. Feng, and Q. Hou, "Storydiffusion: Consistent self-attention for
long-range image and video generation." in NewrIPS, 2024.
[56] D. J. Zhang, D. Li, H. Le, M. Z. Shou, C. Xiong, and D. Sahoo, "Moonshot: Towards controllable vidco generation and editing with multimodal conditions," IJCV, 2025.
[57] Z. Wang, A. Li, L. Zhu, Y. Guo, Q Dou, and Z. Li, "Customvideo: Customizing text-to-video generation with multiple subjects," arXiv preprint arXiv:2401. 009922999999...... 2024.
[58] T-S. Chen, C. H. Lin, H.-Y. Tseng, T.-Y. Lin, and M.-H. Yang, "Motion-conditionee diffusion model for controllable vidco synthesis," arXiv preprint arXiv:2304 14404, 2023.
[59] D. Kim, Zhang, W. Jin, Cho, Q. Dai, J. Park, and Luo, "Subjecc-driven vidco generation via disentangled identity and motion," " arXiv preprint arXiv:2504.17816 2025.
[60] Y. Zhou, R. Zhang, J. Gu, N. 7hao, J. Shi, and T. Sun, "Sugar: Subject-driven video customization in zero-shot manner," arXiv preprint arXiv:2412. 10533, 2024.
[61] Z. Peng, W. Wang, L. Dong, Y. Hao, S. Huang, S. Ma, and E. Wei, "Kosmos-2: Grounding multimodal large language models to the world," in ICLR, 2023.
[62] N. Ravi, Gabeur, Y.-T. Hu, R. Hu, C. Ryali, Ma, H. Khedr, R. Rüdle, Rolland, L.. Gustafson, et al.,
"Sam 2: Segment anything in images and videos," " in ICLR, 2025.
[63] S. Chen, H. Guo, S. Zhu, F. Zhang. Z. Huang, J. Feng, and B. Kang, "Video depth anything: Consistent depth estimation for super-long videos." in CVPR, 2025.
[64] Y. Lipman. R. T. Chen, H. Ben-Hamm,, M. Nickel, and M. Le, "Flow matching for generative modeling," in ICLR, 2023.
[65] X. Liu, X. Zhang, J. Ma, J. Peng, et al., "Instaflow: One step is enough for high-quality biffusion-based text-to-inn to-image generation," in ICLR, 2024.
[66] Z. Wang, Z. Yuan, X. Wang. Y. Li. T. Chen, M. Xia, P. Luo, and Shan, "Motionctrl: A unified and flexible motion controller for vidco generation," in ACM SIGGRAPH, 2024.
[67] H. He, Y. Xu, Y. Guo, G. Wetzstein, B. Dai, H. Li, and C. Yang, "Cameractrl: Enabling camera control for text-to-vidde -vdeo generation," ICLR, 2025.
[68] G. Zheng, T. Li, R. Jiang, Y. Lu, T. Wu, and X. Li, "Cami2y: Camera-comtrolled image-to-video diffusion model," arXiv preprint arXiv:24 10. 15957, 2024.
[69] M. Deitke, D. Schwenk, J. Salvador, L. Weihs, O. Michel, E. VanderBilt, L. Schmidt, K. Ehsani, A. Kemb- havi, and A. Farhadi, "Objaverse: A universe of annotated 3d objects," in CVPR, 2023.
[70] C. Wei, Z. Xiong, W. Ren, X. Du, G. Zhang, and W. Chen, "Omniedit: Building image editing generalist models through specialist supervision," in ICLR, 2025.
[71] T. Loshchilov and F. Hutter, "Decoupled weight decay regularization," in ICLR, 2019.
[72]
Loshchilov and F. Hutter, "Sgdr: Stochastic gradient descent with warm restarts," in ICLR, 2017.
[73] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin,
J. Clark, al., "Learning transferable visual models from natural language supervision," in ICML,, 2022.
[74] N. Ruiz, Y. Li, V. Jampani, Y. Pritch, M. Rubinstein, and K. Aberman, "Dreambooth: Fine tuning
text-to-image diffusion models for subject-driven generation." in CVPR, 2023.
[75] M. Caron, H. Touvron, I. Misra, H. Jégou, Mairal, P. Bojanowski, and A. Joulin, "Emerging properties
in self-supervised vision transformers," in ICCV, 2021.
[76] Z. Teed and J. Deng. "Raft: Recurrent all- pairs field transforms for optical flow," in ECCV, Springer, 2020.
[77] S. Xiao, Y. Wang. J. Zhou, H. Yuan, X. Xing. R. Yan, C. Li, S. Wang. T. Huang, and Z. Liu, "Omnigen: Unified image generation." arXiv preprint arXiv:2409. 11340, 2024.
[78] S. Bahmani, I. Skorokhodov, V. Rong, G. Weizstein, L. Guibas, P. Wonka, Tulyakov, Park, A. Tagliasacchi, and D. B. Lindell, "4d-fy: Text-to-4d generation using hybrid score distillation sampling." in CVPR, 2024.
[79] U. Singer. S. Sheynin, A. Polyak, Ashual, I. Makarov, F. Kokkinos, N. Goyal, A. Vedaldi, D. Parikh. Johnson, et al., "Text-to-4d dynamic scene generation," ICMI, 2023.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please refer to the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer to Sec. for details.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
his paper does not include any theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Please refer to Sec. and Sec. 4.1 for the details of our method and experimen- tal settings.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer:
[No]
Justification: Code and data are the assets of the company. We will apply for approval to release the code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.. ec/
ooiicyyyyy)).. for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: cenollc/ggidess..
/ /nips...
sssissssn ssissssliccyyyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer Sec. 4.1 for details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Please refer to Sec. for details.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to Sec. 4.1 for details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https https://neeet httppp::///uripp/// cc/public/tthiicauiieee
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer to Sec. for more details.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have carefully credited all previous works we used in the paper. The license and terms are properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: the paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: LLM is not important component of our method. The declaration is not required.
Guidelines:
The answer NA means that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https htt/e//rps..cc/oofeeeeeeeeeeeeee for what should or should be described.
ct /
20


Reviews:
[{"review_id": "Qhl7EOyyhC", "reviewer": "Reviewer_uLej", "summary": "This paper addresses the limitations of existing feedforward subject-driven video customization methods, which struggle with multi-subject scenarios and lack fine-grained control. First, the authors propose a data pipeline to create multi-subject training data with control signals, and an IVTM training strategy to enable instructive editing capability transferred from image-only data. The proposed model, OmniVCus, equipped with Lottery Embedding and Temporally Aligned Embedding, extends to more subjects and improves control alignment.", "questions": "* The Image-Video Transfer Mixed Training section is somewhat unclear. A more detailed explanation would help clarify how the image-level instructive editing knowledge is transferred to the video domain.\n* How many sampling steps are used during inference for both the proposed method and the baselines? Also, are all methods built upon the same foundation model to ensure a fair comparison?\n* Why is text-to-multiview data used, as mentioned in Line 184? Its role is not clearly explained in the paper.\n* Additional suggestion: There is a relevant work [1] on tuning-based multi-subject video customization. Although a direct comparison may not be feasible due to the lack of released code, it is still suggested to cite it in the related work to help strengthen the completeness of the literature review.\n\n[1] Huang, Chi-Pin, et al. \"Videomage: Multi-subject and motion customization of text-to-video diffusion models.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\nThe setting of the paper is novel. It extends beyond prior work that can only achieve text+image to video generation by incorporating diverse control signals and enabling the instructive editing capability. This method expands the flexibility and applicability of subject-driven video customization. The project page demo provided in the supplementary material is impressive, especially the illustrations of emergent Text-to-4D capability. The paper is well-written and easy to follow. The conducted experiment is solid, with comparisons against state-of-the-art baselines on subject customization, instructive editing, and camera-controlled tasks, respectively.\n\n**Weaknesses**\n\nThe proposed method is relatively straightforward, and the ideas of Lottery Embedding and Temporally Aligned Embedding are not particularly novel. Additionally, the Image-Video Transfer Mixed Training section is a little bit unclear and would benefit from a more detailed explanation.", "was_revised": true, "final_justification": "After reading the authors' rebuttal, I've decided to keep my score at 5. Their explanations of Lottery Embedding and Temporally Aligned Embedding are clear and make sense, especially in addressing the scalability challenges of multi-subject customization and improving control effects. The clarification of the Image-Video Transfer Mixed (IVTM) training also helped me better understand how the model transfers editing ability from image to video tasks.\n\nWhile the method itself is relatively simple and doesn't involve particularly novel techniques, the overall design is thoughtful and works well in practice. The authors also give clear answers regarding sampling steps, model differences, and the role of text-to-multiview data. The paper is overall good, and the response helped confirm the strength and usefulness of the proposed approach.", "rebuttal": "&nbsp;\n## Response to Reviewer uLej\n&nbsp;\n\n`Q-1:` Questions about the Lottery Embedding (LE) and Temporally Aligned Embedding (TAE)\n\n`A-1:` (i) The motivation of designing LE is to enable zero-shot more-subject customization in inference while using training samples with limited subjects. This is a very important yet less explored problem, as constructing multi-subject-to-video training data pairs is very challenging and hard to scale up. Previous methods usually assign a fixed number of frame position embeddings to the subjects and infer with the same number of subjects as used during training. As a result, when performing zero-shot more-subject customization, the additional frame embeddings have not been seen during training, making the model easily fail to insert some subjects, as shown in Fig. 9 (e).\n\nTo address this issue, our LE uses a limited number of subjects to randomly activate more frame embeddings in training. As these additional embeddings have already been learned in training, they can be directly assigned to more subjects without extra efforts in inference to enable zero-shot more-subject video customization. As shown in Tab.2 (c), using our LE can significantly improve the CLIP-T and DINO-I scores by 30% and 24%. As compared in Fig.9 (e), using our LE can better preserve the subjects in the customized videos than the naive embeddings used by previous methods.\n\n(ii) Our TAE is designed to better handle the control signals of depth, mask, and camera by leveraging their properties that are temporally aligned with the video noise. Previous methods usually indiscriminately concatenate the tokens of all control signals (depth and camera) and assign different frame position embeddings to the control signals and the video noise. This naive embedding scheme does not fully exploit the per-frame correlation between the control signal and video noise to guide the video customization. Besides, concatenating camera poses may easily result in conflicts with depth/mask to degrade control effect and unnecessary computational complexity, as the camera signal does not contain fine-grained information.\n\nTo cope with these problems, TAE assigns the same frame position embeddings to the video structure control signals (depth/mask) and the video noise, and only adds the timestep embeddings to the video noise to distinguish them. This key clue directs the model to extract the per-frame guidance, learn the semantics, and enhance the control effects. As listed in Tab.2 (b), using TAE can significantly improve the CLIP-T and DINO-I scores by 17% and 29%. As compared in Fig.9 (c1), (c2), (d1), and (d2), using TAE can better preserve the identity of the subject than the naive embedding scheme and even has the emergent capability to transfer the texture when the subject is not aligned with the mask.\n\nBesides, we observe that the camera signals do not contain fine-grained spatial and semantic information. Thus, to avoid the conflicts with depth/mask and reduce the computational complexity, our TAE first adopts the Plücker coordinates to encode the camera poses into pixel-aligned ray embeddings and then integrates these ray embeddings into noise tokens by MLPs. As shown in Line 269-272 and Fig.9 (b), using our TAE can reduce 10% training time and control the camera movement more effectively.\n\n&nbsp;\n&nbsp;\n\n`Q-2:` Explanation of the Image-Video Transfer Mixed (IVTM) Training\n\n`A-2:` As analyzed in Line 156 - 168, the core idea of our IVTM training is to build a common task pair on image-level and video-level as the bridge to transfer the instructive editing knowledge from image to video. In particular, we set this common task pair as single-subject image and video customization. We choose this task pair because single-subject image customization has the same input/output format (image-to-image) with image instructive editing, and single-subject video customization has the same input/output format (image-to-video) with instructive editing subject-driven video customization. Benefit from the same input/output format, mixed training this task pair with image instructive editing dataset can transfer the knowledge and enable zero-shot instructive editing subject-driven video customization.\n\nTo better transfer the knowledge, we align the frame position embeddings, as shown in the green small cubes in Fig.4, of the input images of the task pair and image instructive editing.\n\nTo enable the instructive editing effect in multi-subject video customization, we assign the above frame position embeddings by our Lottery Embedding mechanism in Eq.(1) with $K = 1$, also shown in Fig.4 (a). Through our LE mechanism, more frame position embeddings are activated with the instructive editing knowledge transfer during the training process. Thus, we can directly impose editing instruction on multiple subjects in inference, as shown in Fig. 2 (i).\n\n&nbsp;\n&nbsp;\n\n`Q-3:` Questions about the inference sampling steps and foundation model.\n\n`A-3:` The sampling steps are listed in the following table.\n\n| Method | VideoBooth | DreamVideo | Wan2.1-I2V | SkyReels | Motionctrl | Cameractrl | CamI2V | Ours |\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n| Sampling Steps | 250 | 50 | 50 | 50 | 25 | 25 | 25 | 50 |\n| foundation model | Stable-Diffusion-v1.4-1B | Stable-Diffusion-v1.5-1.4B | Wan2.1-14B | Wan2.1-14B | Stable-Video-Diffusion-1.5B | Stable-Video-Diffusion-xt-2.3B | DynamiCrafter-1.4B | Internal-5B |\n\nDifferent methods are built upon different foundation models, as shown in the above table. Because the training code and data of different methods are not all publicly available, it is not feasible to implement different methods on the same foundation model.\n\nHowever, compared to the SOTA methods, SkyReels and Wan2.1-I2V equipped with OmniGen, our method uses a smaller model (5B vs 14B) but yields better customization results and more unified control effects under multi-modal conditions, as shown in Fig. 1, 5, 6, and 7.\n\n&nbsp;\n&nbsp;\n\n`Q-4:` Why using the text-to-multiview data?\n\n`A-4:` The text-to-multiview dataset provides camera-to-video data pairs. We mixed train our model with the text-to-multiview data to enable camera-controlled subject-driven video customization. As shown in Fig.4, the text-to-3D dataset provides control conditions of camera poses, and some other datasets provide the control signals of reference images of subjects during training. Although these control conditions are not matched to the same sample in the training process, our method can flexibly compose them (texts, camera poses, and images of subjects) in inference to enable camera-controlled subject-driven video customization, as shown in Fig.1 (d), Fig.2 (ii), and Fig.8\n\n&nbsp;\n&nbsp;\n\n`Q-5:` Cite the relevant work VideoMage\n\n`A-5:` Thanks for reminding us. VideoMage is a great work. We will not only cite it but also discuss it in detail to highlight its contribution and increase its impact."}, {"review_id": "1XxWH1oKUY", "reviewer": "Reviewer_oD5B", "summary": "This paper introduces OmniVCus, a feedforward framework for subject-driven video customization that supports multi-subject scenarios and multimodal control conditions, such as depth maps, segmentation masks, camera trajectories, and textual prompts. The authors propose a data construction pipeline, VideoCus-Factory, to generate training data pairs for multi-subject customization from raw videos without requiring labeled data. The framework employs a Diffusion Transformer (DiT) architecture with two key embedding mechanisms: Lottery Embedding (LE) for enabling inference with more subjects than seen in training, and Temporally Aligned Embedding (TAE) for improved control signal integration. The method is trained using an Image-Video Transfer Mixed (IVTM) strategy, combining image editing and video customization data to enable instructive editing. Experimental results demonstrate superior performance over state-of-the-art methods in both quantitative metrics and qualitative evaluations, particularly in multi-subject customization and flexible control.", "questions": "1. Regarding the unintended editing effects, such as the hairstyle changes in Figure 1(g), can the authors clarify the specific control conditions or training data limitations that led to these issues?\n2. The paper does not discuss the computational cost of training or inference for OmniVCus. Could the authors provide details on the training time, hardware requirements, and inference latency, particularly for multi-subject scenarios?\n3. The IVTM training strategy is a key component, but its impact is only briefly analyzed in Table 2d. Could the authors provide a more detailed ablation study, including the effect of varying the ratio of image-to-video data?", "limitations": "The authors have discussed the limitations.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* The paper presents innovations in subject-driven video customization data construction. The VideoCus-Factory pipeline is a novel approach to constructing multi-subject training data without reliance on labeled video data, addressing a critical gap in the field.\n* The ability to handle multi-subject customization and incorporate diverse control conditions is a substantial advancement for real-world applications。\n* The framework’s feedforward approach avoids the computational overhead of test-time tuning, making it practical for deployment.\n* The paper is well-structured, with clear descriptions of the VideoCus-Factory pipeline, the OmniVCus framework, and the experimental setup.\n\nWeaknesses:\n* The fidelity of human subject customization is limited in some cases. For instance, as shown in Figure 1(g), the hairstyle of a character is altered during editing, resulting in unintended changes that deviate from the desired customization. This suggests that the model struggles with fine-grained identity preservation.\n* While the paper is generally clear, some technical aspects of the OmniVCus framework, such as the precise implementation of LE and TAE, lack sufficient detail. For example, the mechanism by which LE activates additional frame embeddings for unseen subjects during inference is not fully explained.\n* The paper does not adequately discuss the computational efficiency or scalability of the proposed framework. The training complexity of IVTM and the resource demands of the DiT architecture are not detailed.", "was_revised": true, "final_justification": "The authors‘ responses have clarified many of the concerns I raised in the initial review.\n\nThe explanation about the misalignment between the depth map and the subject identity is reasonable and demonstrates that the model has been designed to generalize in such challenging cases. The clarification that identity preservation is better when control signals are aligned is helpful. However, it would be beneficial if future versions of the paper could emphasize this trade-off more explicitly, and perhaps suggest ways to detect or mitigate misalignment during deployment.\n\nConsidering the overall quality of the paper, I will maintain my positive score.", "rebuttal": "&nbsp;\n## Response to Reviewer oD5B\n&nbsp;\n\n`Q-1:` Questions about the hairstyle changes of human subject customization.\n\n`A-1:` The change of hairstyle in Fig.1 (g) is due to the misalignment between the given subject and the depth control signal. In particular, the given subject is a woman with long hair. However, the depth sequence is estimated from a man with short hair. The depth signal controls the structure of the customized video and restricts the shape of the human subject's hair. As a result, when filling the woman into the depth of the man, our model keeps her face identity and changes her hairstyle to fit the shape of the depth map. This is a challenging case to test whether the model can handle subject and control signals that are not perfectly aligned, thereby evaluating its customization ability.\n\nSimilarly, when the given subject is severely misaligned with the depth/mask control signal, our model can still transfer the texture of the subject and change its shape to fill in the semantically appropriate position, as shown in Fig.1 (h) and (i). This texture transfer effect is a feature of our well-designed frame embedding mechanism and flexible framework, as previous frame embedding method performs poorly in these challenging cases, as compared in Fig.9 (d1) and (d2).\n\nOn the other hand, when the subject is well aligned with the given mask or depth, the fine-grained identity of the given subject can be better preserved. As shown in Fig.1 (f) and Fig.9 (c2), the hairstyle of the boy does not change in the customized videos because the mask and depth are all estimated from a short-haired man. In addition, when not using depth/mask control signals, the customized videos are not restricted, and the fine-grained identity of the subject can also be better preserved. As shown in Fig. 5, 6, 7, and 8, the hairstyle of the human subjects does not change because there is no mask/depth to restrict the shape of the human's hair.\n\n&nbsp;\n&nbsp;\n\n`Q-2:` Details about LE and TAE.\n\n`A-2:` We will release the code of LE and TAE for their precise implementation.\n\n(i) Additional frame embeddings for more subjects are activated during the training process. Previous methods usually assign as many frame position embeddings as the number of subjects in the training data sample. As a result, when performing zero-shot more-subject customization, the additional frame position embeddings have not been seen during training, making the model easily fail to insert some subjects, as shown in Fig.9 (e).\n\nIn contrast, the core insight of our LE is to use a limited number of subjects in the training data to activate more frame position embeddings. As described in Line 127 - 137, LE randomly selects $K$ out of $M$ frame position embeddings ($M$ > $K$, $M$ = 4, $K$ = 2). Thus, these additional frame position embeddings have already been learned during the training process and can be directly assigned to more subjects in inference to enable zero-shot more-subject video customization without extra effort. As compared in Tab.2 (c), using our LE can significantly improve the CLIP-T and DINO-I scores by 30% and 24%.\n\n(ii) The technical details of TAE are described in Line 138 - 155. The key idea of TAE is to leverage the temporally aligned properties of the input conditions (depth, mask, and camera) to extract the per-frame guidance, learn the semantics, and enhance the control effects.\n\nFor the depth/mask signals that control the video frame structure, TAE assigns the same frame position embeddings to them and the video noise, while only adding the timestep embedding to the noise for distinction. We notice that the camera condition does not contain fine-grained spatial and semantic information. Thus, to avoid conflicts with depth/mask and reduce the computational complexity, TAE adopts the Plücker coordinates to encode the camera poses into pixel-aligned ray embeddings and then integrates these ray embeddings into video noise by MLPs.\n\nAs shown in Tab.2 (b), Fig.9 (b), (c), (d), and Line 269-272, using TAE can better preserve subject identity, reduce training time, enhance control effect, and enable the model's emergent capability of texture transfer.\n\n&nbsp;\n&nbsp;\n\n`Q-3:` Training time, hardware requirements, inference latency, and scalability\n\n`A-3:` (i) As described in Line 191, we use 64 A100 GPUs to train our OmniVCus for 5 days. \n\n(ii) The inference time (m = minutes, s = seconds) for different tasks on a single A100 GPU at a spatial resolution of 384$\\times$640 for 64 frames at 24 fps are shown in the following table:\n\n| Num of Subjects | Only Subject-driven | Instructive Editing | Camera-controlled | Depth-controlled | Mask-controlled |\n|:-:|:-:|:-:|:-:|:-:|:-:|\n| 1 | 3m 51s | 3m 51s | 3m 53s | 7m 11s | 7m 11s |\n| 2 | 3m 57s | 3m 57s | 3m 59s | 7m 23s | 7m 23s |\n| 3 | 4m 4s | 4m 4s | 4m 7s | 7m 36s | 7m 36s |\n| 4 | 4m 11s | 4m 11s | 4m 14s | 7m 49s | 7m 49s |\n\n(iii) We show the ablation study about the scalability of batch size and spatial resolution on the single-subject video customization task in the following two tables.\n\n| Batch Size | 89 | 178 | 267 | 356 |\n|:-:|:-:|:-:|:-:|:-:|\n| CLIP-T | 0.2726 | 0.3008 | 0.3145 | 0.3293 |\n| DINO-I | 0.4481 | 0.4830 | 0.5027 | 0.5215 |\n| Consistency | 0.9535 | 0.9867 | 0.9913 | 0.9928 |\n| Dynamic | 0.4794 | 0.5206 | 0.5563 | 0.5541 |\n\n| Spatial Size | 96$\\times$160 | 192$\\times$320 | 288$\\times$480 | 384$\\times$640 |\n|:-:|:-:|:-:|:-:|:-:|\n| CLIP-T | 0.3017 | 0.3169 | 0.3240 | 0.3293 |\n| DINO-I | 0.4985 | 0.5087 | 0.5157 | 0.5215 |\n| Consistency | 0.9892 | 0.9910 | 0.9922 | 0.9928 |\n| Dynamic | 0.5428 | 0.5693 | 0.5516 | 0.5541 |\n\nWhen we fix the spatial size at 384$\\times$640 and increase the batch size, or fix the batch size at 356 and increase the spatial size, the video generation quality is gradually improved.\n\n&nbsp;\n&nbsp;\n\n`Q-4:` More detailed ablation study of varying the ratio of image-to-video data\n\n`A-4:` The ablation study of varying the training data ratio between the task pair:  single-subject image customization and single-subject video customization is shown in the following table. IVTM yields the best instructive editing performance (the highest CLIP-T score) when the ratio of image-to-video data is 4:1.\n\n| Image : Video | 2:1 | 3:1 | 4:1 | 5:1 | 6:1 |\n|:-:|:-:|:-:|:-:|:-:|:-:|\n| CLIP-T | 0.2714 | 0.2955 | 0.3126 | 0.3007 | 0.3082 |"}, {"review_id": "Yv9e67HXtr", "reviewer": "Reviewer_UWzw", "summary": "The paper presents a method for incorporating multiple different conditionings into video diffusion model, the conditioning are the following: (i) subject images from which particular objects will be extracted, (ii) pixel-aligned conditionings such as masks and depth, (iii) camera poses. For each of these 3 embeddings different conditioning mechanism is used: Lottery Embedding for (i), Temporally Aligned Embedding for (ii) and direct summation of MLP processed plucker embedding for camera.", "questions": "- What is the main difference of this method compared to SkyReal A2, VideoAlcmemist, VACE{1} and FullDiT{2}.\n- Can the authors provide evaluation on SkyReal A2 benchmark and VACE benchmark?\n\n{1} VACE: All-in-One Video Creation and Editing\n\n{2} FullDiT: Multi-Task Video Generative Foundation Model with Full Attention", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 1, "originality": 1, "strengths_and_weaknesses": "**Strengts:**\n- The paper and method is fairly clear.\n- The quantitative evaluation seems to show better results compared to new methods, such as SkyReels.\n\n**Weaknesses:**\n- The modifications proposed in this paper is quite insignificant, the data collection pipeline is quite similar to SkyReels and VideoAlchemist, mentioned in this paper. The only difference is the better filtering based on SAM2 mask tracking and data augmentation based on pasting the images onto the random background. The difference in conditioning mechanisms is also quite minor.\n\n- It is not clear which evaluation prompts are used. It well maybe that the prompts favor the proposed method, for example prompts from the training dataset or prompts very similar to them. Why not use SkyReals-A2 benchmark or VideoAlcmemist benchmark for evalutions?\n\n- It is not clear if the baseline models are utilized correctly. There is only one SkyReals-A2 example, and the first frame looks like a white segmented-out woman. So my guess is that segmented out frame is provided as image for i2v conditioning not as a reference image. In Fig. 6, the first frame is completely broken which also suggest that the model is used incorrectly. \n\n- The generation quality is far from being perfect, the qualitative results of the proposed method either suffer from the copy-pasting issue (a1 last row, a3 second row first sample and forth row)  or have severe face flickering artifacts (a3 first row first sample, a3 second row second sample).", "was_revised": true, "final_justification": "The authors provided detailed rebuttal, which partially addressed some concerns. I still believe evaluation of SkyReels-A2 in Tab. 1 is not correct. Given other reviews and detailed rebuttal, I decided to change the score.", "rebuttal": "&nbsp;\n## Response to Reviewer UWzw\n&nbsp;\n\n`Q-1:` Comparison with concurrent works SkyReels-A2, VideoAlchemist, VACE, and FullDiT\n\n`A-1:` Our work is different in:\n\n(i) Motivation and Research Scope\n\nWe propose a unified framework to comprehensively study subject-driven video customization under different control conditions including depth, mask, camera, and editing instruction. In contrast, SkyReels-A2 and VideoAlchemist only study the vanilla subject-driven customization without adding any other control signals. VACE and FullDiT are general video generation and editing frameworks. For subject-driven customization, VACE does not study camera control while FullDiT does not study mask control. In addition, three important problems in subject-driven customization are not explored by these four works: zero-shot more-subject customization, texture transfer, and instructive editing. Our work aims to fill these research gaps.\n\n(ii) Method Details\n\nFirstly, towards zero-shot more subject customization, we propose Lottery Embedding (LE). Previous methods usually assign a fixed number of frame position embeddings to the subjects and infer with the same number of subjects as used in training. Thus, when performing zero-shot more-subject customization, the additional frame embeddings have not been seen during training, making the model easily fail to insert some subjects, as shown in Fig.9 (e). To address this issue, our LE uses a limited number of subjects in the training data to activate more frame position embeddings. Thus, the additional frame embeddings have already been learned in training and can be directly assigned to more subjects in inference. As compared in Tab.2 (c), using LE can improve CLIP-T and DINO-I by 30% and 24%.\n\nSecondly, we design a TAE to better handle the control signals by leveraging their temporally aligned properties. VACE and FullDiT directly assign different frame position embeddings to mask/depth and noise. The per-frame correlation between the control signal and noise is not fully exploited. In contrast, our TAE assigns the same frame position embeddings to the control signals (depth/mask) and video noise, and adds the timestep embeddings to the noise to distinguish them. This key clue directs the model to extract the per-frame guidance. In Tab.2 (b), using TAE improves CLIP-T and DINO-I by 17% and 29%. As compared in Fig.9 (c) and (d), using TAE can better preserve the subject identity and has the emergent capability to transfer the texture when the subject is not aligned with the mask. Besides, for the camera signal, FullDiT concatenates it to the input tokens. This may lead to conflicts with depth/mask and unnecessary computational complexity, as the camera signal does not contain fine-grained information. To tackle these issues, our TAE first encodes camera poses into ray embeddings and then integrates them into noise tokens by MLPs. See Fig.9 (b) and Line 269-272, using TAE can reduce 10% training time and control the camera movement more effectively.\n\nThirdly, to enable instructive editing effect in video customization, we design an Image Video Transfer Mixed (IVTM) training strategy. In Line 156 - 168, our IVTM builds a common task pair on image- and video-level as the bridge to transfer the instructive editing knowledge from image to video. See Tab.2 (d) and Fig.9 (f), using IVTM improves CLIP-T by 21% and enables the editing effect of style transfer, which is not supported by the four works.\n\n(iii) Comparison on data pipeline and conditioning method with SkyReels and VideoAlchemist\n\nOur data collection pipeline is different as it has subject filtering, data augmentation, and control signal construction. Our subject filtering adopts SAM2 tracking to filter out failure segmented subjects, which is very important to improve the identity preservation in model training. In data augmentation, we rescale, re-center, rotate the subject, and change its color and background to avoid the information leakage of the subject, which encourages the trained model to have more variation in customized videos. Besides, our data pipeline also constructs high-quality control signals such as depth and mask. See Tab.2 (a), using our subject filtering and data augmentation significantly improves CLIP-T and DINO-I. Besides, training with our data can yield mask-/depth-controlled effects, as shown in Fig.1.\n\n(iv) Performance Comparison\n\nWe compare with FullDiT, VideoAlchemist, SkyReels, and VACE in the following table, `A-2` table, and `A-5` tables. Our method outperforms the four methods on the metrics of identity preservation (CLIP-I, DINO-I, Subj-S) and text following (CLIP-T, Text-S).\n\n|Method|CLIP-T|DINO-I|CLIP-I|Smoothness|Dynamic|Aesthetic|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|FullDiT|18.64|46.22|68.59|94.95|16.68|5.46|\n|Ours|19.72|50.06|71.83|95.30|15.29|5.61|\n\nSkyReels, VACE, and FullDiT are arxiv papers. VideoAlchemist is published by CVPR 2025, later than 1st March. According to NeurIPS policy, these four papers are concurrent works.\n\n&nbsp;\n\n`Q-2:` Questions about the evaluation prompts\n\n`A-2:` Our evaluation prompts are hand-written and fundamentally different from the training prompts generated by Kosmos-2. There is no overlap in texts, images, or videos between the testing and training sets. Besides, to ensure all baseline models perform at their best, we use their own data preprocessing pipelines to adapt the testing images and text prompts.\n\nThe benchmarks of SkyReels-A2 and VideoAlcmenist only have reference images of subjects and do not support comprehensive evaluation for video customization under different control conditions such as editing instruction, camera pose, depth, and mask. In addition, their benchmarks have at most two subjects and a background for each sample. It is not sufficient for us to study zero-shot more-subject customization. Besides, fixing the third image as the background reference is also not flexible for evaluation.\n\nThus, we collect our dataset for more comprehensive and flexible evaluation. We will also release our testing set for further research.\n\nBesides, we also test on the benchmarks of VideoAlcmemist and SkyReels-A2 in the following table and `A-5` first table. Our method performs better on the metrics of identity preservation (Subj-S, ID consistency) and text following (Text-S, Image-Text Similarity).\n\n|Method|Text-S|Vid-S|Subj-S|Dync-D|\n|:-:|:-:|:-:|:-:|:-:|\n|VideoAlchemist|0.254|0.780|0.570|0.506|\n|Ours|0.273|0.791|0.585|0.499|\n\n&nbsp;\n\n`Q-3:` Questions about how we use the baseline models\n\n`A-3:` For each baseline model, we strictly follow the instructions in its official repositories to use its code. In addition, to ensure all baseline models perform at their best, we use their own data preprocessing pipelines to adapt the testing images and text prompts to match their input requirements.\n\nSkyReels-A2 sometimes generates artifacts, flickering, cropping issues, and even corrupted early frames in customized videos when handling challenging cases. This is why the first frame of Skyreels-A2 in Fig.5 loses part of background and the first frame is broken in Fig.6. Please note that we use the original image as the reference image instead of using the segmented out image. SkyReels-A2 can only perform reference-to-video. It can not perform I2V so it does not have I2V conditioning. We have double-checked by using both inference scripts and gradio demo scripts to make sure the generation results are the same.\n\nIn addition, there are also some cases (e.g., Fig.7) where SkyReels-A2 does not break early frames but still performs worse than our method. \n\n&nbsp;\n\n`Q-4:` Questions about our generation results\n\n`A-4:` (i) Although in some cases the subjects look similar in the given image and customized video, they still have some variations. In a1 last row, the face gesture and action of the boy change. The woman also changes from kneeling for picnics to standing to mop the floor. In a3 second row first sample, the man's eyes and the clothes color change. His hand also changes to hold a cup to drink coffee, while the pose of the cup also changes. In a3 forth row, the man's hand movements and clothes, and computer pose have all changed.\n\nIn most cases, our customized video has a large variation on the subjects instead of simply copy-pasting them, such as a1 first row and a3 last row.\n\n(ii) a3 first row first sample and second row second sample are two challenging cases where the people's faces are very small. In the first sample, the old man's face is defocused in the reference image. As a result, his face is blurry in the customized video. In the second sample, the reference image IMG2 only shows the woman's side face. Thus, when generating the frontal face, there is a chance of producing minor flickering artifacts. However, towards the same woman subject, a2 last row shows a good customized video with her frontal face.\n\n&nbsp;\n\n`Q-5:` Evaluation on SkyReels-A2 and VACE benchmarks\n\n`A-5:` The results on SkyReels-A2 and VACE benchmarks are shown in the following tables. Our method outperforms SkyReels-A2 and VACE on the metrics of identity preservation (ID, object, and subject consistency) and text following (image-text similarity). The gaps on other metrics such as image quality mainly stem from the performance gaps between our 5B base model and Wan2.1-14B model.\n\n|Method|ID Consistency|Object Consistency|Background Consistency|Image Quality|Asthetic Quality|Motion Smoothness|Dynamic Degree|Image-Text Similarity|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|SkyReels-A2|0.398|0.809|0.677|0.683|0.579|0.891|1.000|28.188|\n|Ours|0.403|0.815|0.664|0.581|0.552|0.877|0.963|28.426|\n\n|Method|Asthetic Quality|Background Consistency|Dynamic Degree|Image Quality|Motion Smoothness|Overall Consistency|Subject Consistency|Temporal Flickering|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|VACE|63.25|98.03|30.00|72.29|99.51|25.85|98.54|99.15|\n|Ours|61.90|96.28|45.06|68.33|99.17|26.42|98.81|96.04|"}, {"review_id": "rK3NUfQxJQ", "reviewer": "Reviewer_sS6b", "summary": "This paper presents OmniVCus, a comprehensive framework for subject-driven video customization that addresses several major limitations of prior work. The key problems it tackles are the difficulty of creating multi-subject training data and the lack of fine-grained control over the subject's actions and attributes. The authors first introduce \"VideoCus-Factory,\" a data pipeline that automatically generates multi-subject training pairs, along with control signals like depth and masks, from unlabeled raw videos. They then propose the OmniVCus model, which uses two novel embedding techniques: \"Lottery Embedding\" to enable generating videos with more subjects than seen during training, and \"Temporally Aligned Embedding\" to better integrate control signals. Finally, they use a mixed training strategy to transfer knowledge from image-editing datasets, allowing users to edit subjects in the final video with text instructions.", "questions": "The \"Lottery Embedding\" concept for zero-shot generation of more subjects is very clever. Is there a practical upper limit to this capability? For instance, if the model is trained with a maximum of K=2 subjects, how well does the quality hold up when trying to generate M=5 or M=10 subjects at inference time?\n\nYour \"VideoCus-Factory\" data pipeline is a core part of this work. How sensitive is it to potential failures from the upstream models it relies on, such as SAM-2 failing to track a subject correctly? How effectively does your subject filtering step handle this kind of noise?\n\nThe model demonstrates an impressive ability to compose multiple control signals. Have you observed any negative interference between different controls? For example, does applying a very dynamic camera motion make it more difficult for the model to also follow a precise text-based edit to a subject's clothing?\n\nThe texture transfer ability shown in Figure 1(h) and 1(i) is very interesting. Was this an emergent capability of the model, or was it explicitly trained for? It seems to prioritize the source texture over the target object's identity when the control signal is misaligned.", "limitations": "Yes", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths:**\n\nThe paper successfully tackles multiple hard problems at once: multi-subject data creation, zero-shot subject composition, and multi-modal controllability (text, mask, depth, camera).\n\nThe proposed \"VideoCus-Factory\" data pipeline is a major contribution, as it removes the dependency on manually labeled data for this complex task.\n\nThe novel technical ideas, particularly \"Lottery Embedding\" for scaling the number of subjects and \"Temporally Aligned Embedding\" for control, are clever and effective.\n\nThe range of capabilities demonstrated in the qualitative results is extremely impressive, showing a single model that can handle many different types of user requests.\n\n**Weaknesses:**\n\nThe training process is very complex and computationally expensive, relying on a mix of several large datasets and significant GPU resources, which may make the results difficult to reproduce.\n\nThe data pipeline's quality is dependent on the performance of several other large pre-trained models (Kosmos-2, SAM-2), meaning errors from those models can propagate into the training data.\n\nThe user study, while a good inclusion, was conducted with a relatively small number of participants (37).", "was_revised": true, "final_justification": "The rebuttal effectively addressed major concerns: the user study was expanded to 114 participants with strong results, subject filtering mitigates upstream model errors, and the single-stage training process is comparable to prior work with code to be released. While computational cost and some control-signal interference remain, these are acceptable limitations. The contributions remain technically solid and impactful, justifying the increased score.", "rebuttal": "&nbsp;\n## Response to Reviewer sS6b\n&nbsp;\n\n`Q-1:` Questions about the complex training process\n\n`A-1:` As our goal is to achieve subject-driven video customization with multimodal control conditions in a unified framework, it is necessary and unavoidable to construct multiple large datasets with different control conditions and use sufficient GPUs to train a model with large capacity through enough iterations of training data. Other works also require complex training process with significant GPU resources. For example, FullDiT [15] requires 64 GPUs to train a DiT model in four stages by gradually adding conditions of text, camera, identity, and depth from four datasets. In contrast, our model can be trained in one stage.\n\nBesides, we will release the code for data construction and training configuration for better reproducibility.\n\n&nbsp;\n&nbsp;\n\n`Q-2:` Questions about the error propagation from the large pre-trained models to the training data\n\n`A-2:` The training data pairs for subject-driven video customization are hard to collect. Using manual annotation is time-consuming, expensive, and even unaffordable. Thus, many researchers choose to use large pre-trained models for data construction. As a result, it is difficult to completely avoid model errors. However, our data pipeline has a subject filtering process to alleviate the model error propagation. Please refer to `A-5` about the effectiveness of our subject filtering.\n\n&nbsp;\n&nbsp;\n\n`Q-3:` Small number of participants in the user study\n\n`A-3:` Following your suggestion, we increased the number of participants in the user study from 37 to 114. The updated user preference (%) of our method over other competitors are listed below:\n\n(i) Single-subject and multi-subject video customization\n\n| Method | Alignment | Identity | Quality |\n|:-:|:-:|:-:|:-:|\n| VideoBooth | 92.1 | 94.7 | 95.6 |\n| DreamVideo | 90.4 | 89.5 | 93.0 |\n| Wan2.1-I2V† | 85.1 | 87.7 | 71.9 |\n| SkyReels | 74.6 | 80.7 | 71.1 |\n| SkyReels* | 72.8 | 77.2 | 69.3 |\n\n\\* = multi-subject video customization\n\n(ii) instructive editing for subject-driven video customization\n\n| Method | Alignment | Identity | Quality |\n|:-:|:-:|:-:|:-:|\n| VideoBooth | 96.5 | 93.9 | 92.1 |\n| DreamVideo | 91.2 | 98.2 | 97.4 |\n| Wan2.1-I2V | 83.3 | 84.2 | 70.2 |\n| SkyReels | 73.7 | 75.4 | 68.4 |\n\n(iii) camera-controlled subject customization\n\n| Method | Alignment | Identity | Quality |\n|:-:|:-:|:-:|:-:|\n| Motionctrl | 87.7 | 88.6 | 78.1 |\n| Cameractrl | 69.3 | 90.4 | 83.3 |\n| CamI2V | 71.9 | 82.5 | 79.8 |\n\nOur method still outperforms other competitors by a large margin. We will update the results in the revision.\n\n&nbsp;\n&nbsp;\n\n`Q-4:` The upper limit of zero-shot generation of more subjects enabled by the Lottery Embedding\n\n`A-4:` According to our observation, when $M=5$, our method can still stably insert the subjects into the customized videos but often suffers from poor harmonization performance and low physics consistency. However, our method can still generate acceptable results with a low success rate (5.3%). When $M\\geq6$, our method often misses some input subjects in the customized videos, and the quality of the generated video is pretty poor. When $M=10$, our method cannot survive in this challenging setting. \n\nThus, the upper limit for LE would be $M=5$, as you found.\n\n&nbsp;\n&nbsp;\n\n`Q-5:` How sensitive is the \"VideoCus-Factory\" to potential failures from the upstream models?\n\n`A-5:` We input 100 samples with varying levels of errors from upstream models into our subject filtering step. 89 of them are filtered out. All the samples where SAM-2 tracking fails have some frames where the subjects are mostly or completely unsegmented. These cases are all filtered out. Among the remaining 11 samples, 7 contain only small errors in the segmentation maps of individual frames — for example, some small parts of the subject are not segmented out. These small errors have little impact on model training. The remaining 4 samples have some frames with larger segmented background areas. However, these over-segmented errors may not be in the reference frame to affect the input images of subjects during training. Thus, the negative impacts could be smaller.\n\n&nbsp;\n&nbsp;\n\n`Q-6:` Is there any negative interference between different controls?\n\n`A-6:` Yes. But the combined control of camera motion and text-based editing of a subject's clothing performs well because these two control signals do not conflict with each other. The camera movement does not affect the clothing of the subject. As shown in Fig.4, the camera embeddings are integrated into the video noise tokens, which do not have any overlap with the text tokens.\n\nHowever, composing the control signals of mask and depth, mask and camera, and depth and camera may have negative interference. Firstly, the mask and depth both control the spatial structure of the generated frames. As shown in Fig.4, mask and depth share the same frame position embeddings. So, simultaneously applying these two signals may fail to control the video, e.g., the subjects are not filled in the foreground segmentation or semantically appropriate positions. In addition, although the camera signal does not have conlicts with depth and mask in model conditioning, the sequence of depth or mask maps already implicitly controls the camera movement of the generated video. Thus, simultaneously applying camera motion and depth/mask signals may also have negative interference, e.g., the camera pose motion loses its control effect.\n\n&nbsp;\n&nbsp;\n\n`Q-7:` Questions about the texture transfer ability\n\n`A-7:` The texture transfer ability is an emergent capability of our model, as we do not have any training data pairs to support texture transfer. \n\nThis ability stems from our well-designed embedding mechanism and flexible framework.\n\nFirstly, our TAE assigns the same frame position embeddings to the video structure signals (depth and mask) and the video noise, while only adding the timestep embedding to the video noise to distinguish them. This key clue directs the DiT model to extract the per-frame guidance, learn the semantics, and enhance the control effects. As compared in Fig.9 (d1) and (d2), our TAE can better transfer the texture than the naive embedding. \n\nSecondly, our OmniVCus takes the concatenated tokens of all control signals as input. This flexible framework can enable some zero-shot capabilities, such as depth- or mask-controlled subject-driven video customization during inference, by directly composing different control signals. However, the mixed training data only consists of depth-to-video, mask-to-video, and subject-to-video data pairs, each containing only a subset of the control conditions, i.e., only depth, mask, or reference images. Please note that the subject and depth/mask are from different samples and do not match each other in the training data. This unpaired training strategy also allows our method to flexibly change reference images of subjects under depth or mask control signals to achieve the texture transfer effect."}]

Author rebuttal:

Meta-review: This paper deals with the challenging multiple-subject video customization task with versatile control conditions. It brings new contributions in regard to dataset construction and effective embedding mechanisms, leading to superior performance in quantitative and qualitative evaluations. Reviewers are positive on the paper's novelties, clarity, and performance gain. Yet, reviewer oD5B raised a concern related to alignment that might affect usage in practice, and reviewer UWzw wondered about the potential unfair comparison with SkyReels-A2. In spite of that, both reviewers rated Weak Accept, and the authors promised to carefully improve the final paper to address these concerns. On the basis of the author–reviewer discussion and the reviewers' recommendations, the AC believes this paper deserves acceptance as a poster.
Decision: Accept (poster)