Title: OmniVCus: Feedforward Subject-driven Video Customization with Multimodal Control Conditions
Venue: NeurIPS 2025
Accepted: yes
Keywords: Video Generation

Multimodal Generation

Subject-driven Video Customization

3D/4D Generation

Flow Matching

Transformer
Abstract: Existing feedforward subject-driven video customization methods mainly study single-subject scenarios due to the difficulty of constructing multi-subject training data pairs. Another challenging problem that how to use the signals such as depth, mask, camera, and text prompts to control and edit the subject in the customized video is still less explored. In this paper, we first propose a data construction pipeline, VideoCus-Factory, to produce training data pairs for multi-subject customization from raw videos without labels and control signals such as depth-to-video and mask-to-video pairs. Based on our constructed data, we develop an Image-Video Transfer Mixed (IVTM) training with image editing data to enable instructive editing for the subject in the customized video. Then we propose a diffusion Transformer framework, OmniVCus, with two embedding mechanisms, Lottery Embedding (LE) and Temporally Aligned Embedding (TAE). LE enables inference with more subjects by using the training subjects to activate more frame embeddings. TAE encourages the generation process to extract guidance from temporally aligned control signals by assigning the same frame embeddings to the control and noise tokens. Experiments demonstrate that our method significantly surpasses state-of-the-art methods in both quantitative and qualitative evaluations. Project page is at https://caiyuanhao1998.github.io/project/OmniVCus/


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents OmniVCus, a comprehensive framework for subject-driven video customization that addresses several major limitations of prior work. The key problems it tackles are the difficulty of creating multi-subject training data and the lack of fine-grained control over the subject's actions and attributes. The authors first introduce \"VideoCus-Factory,\" a data pipeline that automatically generates multi-subject training pairs, along with control signals like depth and masks, from unlabeled raw videos. They then propose the OmniVCus model, which uses two novel embedding techniques: \"Lottery Embedding\" to enable generating videos with more subjects than seen during training, and \"Temporally Aligned Embedding\" to better integrate control signals. Finally, they use a mixed training strategy to transfer knowledge from image-editing datasets, allowing users to edit subjects in the final video with text instructions.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\nThe paper successfully tackles multiple hard problems at once: multi-subject data creation, zero-shot subject composition, and multi-modal controllability (text, mask, depth, camera).\n\nThe proposed \"VideoCus-Factory\" data pipeline is a major contribution, as it removes the dependency on manually labeled data for this complex task.\n\nThe novel technical ideas, particularly \"Lottery Embedding\" for scaling the number of subjects and \"Temporally Aligned Embedding\" for control, are clever and effective.\n\nThe range of capabilities demonstrated in the qualitative results is extremely impressive, showing a single model that can handle many different types of user requests.\n\n**Weaknesses:**\n\nThe training process is very complex and computationally expensive, relying on a mix of several large datasets and significant GPU resources, which may make the results difficult to reproduce.\n\nThe data pipeline's quality is dependent on the performance of several other large pre-trained models (Kosmos-2, SAM-2), meaning errors from those models can propagate into the training data.\n\nThe user study, while a good inclusion, was conducted with a relatively small number of participants (37).\n\nQUESTIONS:\nThe \"Lottery Embedding\" concept for zero-shot generation of more subjects is very clever. Is there a practical upper limit to this capability? For instance, if the model is trained with a maximum of K=2 subjects, how well does the quality hold up when trying to generate M=5 or M=10 subjects at inference time?\n\nYour \"VideoCus-Factory\" data pipeline is a core part of this work. How sensitive is it to potential failures from the upstream models it relies on, such as SAM-2 failing to track a subject correctly? How effectively does your subject filtering step handle this kind of noise?\n\nThe model demonstrates an impressive ability to compose multiple control signals. Have you observed any negative interference between different controls? For example, does applying a very dynamic camera motion make it more difficult for the model to also follow a precise text-based edit to a subject's clothing?\n\nThe texture transfer ability shown in Figure 1(h) and 1(i) is very interesting. Was this an emergent capability of the model, or was it explicitly trained for? It seems to prioritize the source texture over the target object's identity when the control signal is misaligned.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a method for incorporating multiple different conditionings into video diffusion model, the conditioning are the following: (i) subject images from which particular objects will be extracted, (ii) pixel-aligned conditionings such as masks and depth, (iii) camera poses. For each of these 3 embeddings different conditioning mechanism is used: Lottery Embedding for (i), Temporally Aligned Embedding for (ii) and direct summation of MLP processed plucker embedding for camera.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengts:**\n- The paper and method is fairly clear.\n- The quantitative evaluation seems to show better results compared to new methods, such as SkyReels.\n\n**Weaknesses:**\n- The modifications proposed in this paper is quite insignificant, the data collection pipeline is quite similar to SkyReels and VideoAlchemist, mentioned in this paper. The only difference is the better filtering based on SAM2 mask tracking and data augmentation based on pasting the images onto the random background. The difference in conditioning mechanisms is also quite minor.\n\n- It is not clear which evaluation prompts are used. It well maybe that the prompts favor the proposed method, for example prompts from the training dataset or prompts very similar to them. Why not use SkyReals-A2 benchmark or VideoAlcmemist benchmark for evalutions?\n\n- It is not clear if the baseline models are utilized correctly. There is only one SkyReals-A2 example, and the first frame looks like a white segmented-out woman. So my guess is that segmented out frame is provided as image for i2v conditioning not as a reference image. In Fig. 6, the first frame is completely broken which also suggest that the model is used incorrectly. \n\n- The generation quality is far from being perfect, the qualitative results of the proposed method either suffer from the copy-pasting issue (a1 last row, a3 second row first sample and forth row)  or have severe face flickering artifacts (a3 first row first sample, a3 second row second sample).\n\nQUESTIONS:\n- What is the main difference of this method compared to SkyReal A2, VideoAlcmemist, VACE{1} and FullDiT{2}.\n- Can the authors provide evaluation on SkyReal A2 benchmark and VACE benchmark?\n\n{1} VACE: All-in-One Video Creation and Editing\n\n{2} FullDiT: Multi-Task Video Generative Foundation Model with Full Attention\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 1, "ORIGINALITY": 1, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces OmniVCus, a feedforward framework for subject-driven video customization that supports multi-subject scenarios and multimodal control conditions, such as depth maps, segmentation masks, camera trajectories, and textual prompts. The authors propose a data construction pipeline, VideoCus-Factory, to generate training data pairs for multi-subject customization from raw videos without requiring labeled data. The framework employs a Diffusion Transformer (DiT) architecture with two key embedding mechanisms: Lottery Embedding (LE) for enabling inference with more subjects than seen in training, and Temporally Aligned Embedding (TAE) for improved control signal integration. The method is trained using an Image-Video Transfer Mixed (IVTM) strategy, combining image editing and video customization data to enable instructive editing. Experimental results demonstrate superior performance over state-of-the-art methods in both quantitative metrics and qualitative evaluations, particularly in multi-subject customization and flexible control.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* The paper presents innovations in subject-driven video customization data construction. The VideoCus-Factory pipeline is a novel approach to constructing multi-subject training data without reliance on labeled video data, addressing a critical gap in the field.\n* The ability to handle multi-subject customization and incorporate diverse control conditions is a substantial advancement for real-world applications。\n* The framework’s feedforward approach avoids the computational overhead of test-time tuning, making it practical for deployment.\n* The paper is well-structured, with clear descriptions of the VideoCus-Factory pipeline, the OmniVCus framework, and the experimental setup.\n\nWeaknesses:\n* The fidelity of human subject customization is limited in some cases. For instance, as shown in Figure 1(g), the hairstyle of a character is altered during editing, resulting in unintended changes that deviate from the desired customization. This suggests that the model struggles with fine-grained identity preservation.\n* While the paper is generally clear, some technical aspects of the OmniVCus framework, such as the precise implementation of LE and TAE, lack sufficient detail. For example, the mechanism by which LE activates additional frame embeddings for unseen subjects during inference is not fully explained.\n* The paper does not adequately discuss the computational efficiency or scalability of the proposed framework. The training complexity of IVTM and the resource demands of the DiT architecture are not detailed.\n\nQUESTIONS:\n1. Regarding the unintended editing effects, such as the hairstyle changes in Figure 1(g), can the authors clarify the specific control conditions or training data limitations that led to these issues?\n2. The paper does not discuss the computational cost of training or inference for OmniVCus. Could the authors provide details on the training time, hardware requirements, and inference latency, particularly for multi-subject scenarios?\n3. The IVTM training strategy is a key component, but its impact is only briefly analyzed in Table 2d. Could the authors provide a more detailed ablation study, including the effect of varying the ratio of image-to-video data?\n\nLIMITATIONS:\nThe authors have discussed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the limitations of existing feedforward subject-driven video customization methods, which struggle with multi-subject scenarios and lack fine-grained control. First, the authors propose a data pipeline to create multi-subject training data with control signals, and an IVTM training strategy to enable instructive editing capability transferred from image-only data. The proposed model, OmniVCus, equipped with Lottery Embedding and Temporally Aligned Embedding, extends to more subjects and improves control alignment.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nThe setting of the paper is novel. It extends beyond prior work that can only achieve text+image to video generation by incorporating diverse control signals and enabling the instructive editing capability. This method expands the flexibility and applicability of subject-driven video customization. The project page demo provided in the supplementary material is impressive, especially the illustrations of emergent Text-to-4D capability. The paper is well-written and easy to follow. The conducted experiment is solid, with comparisons against state-of-the-art baselines on subject customization, instructive editing, and camera-controlled tasks, respectively.\n\n**Weaknesses**\n\nThe proposed method is relatively straightforward, and the ideas of Lottery Embedding and Temporally Aligned Embedding are not particularly novel. Additionally, the Image-Video Transfer Mixed Training section is a little bit unclear and would benefit from a more detailed explanation.\n\nQUESTIONS:\n* The Image-Video Transfer Mixed Training section is somewhat unclear. A more detailed explanation would help clarify how the image-level instructive editing knowledge is transferred to the video domain.\n* How many sampling steps are used during inference for both the proposed method and the baselines? Also, are all methods built upon the same foundation model to ensure a fair comparison?\n* Why is text-to-multiview data used, as mentioned in Line 184? Its role is not clearly explained in the paper.\n* Additional suggestion: There is a relevant work [1] on tuning-based multi-subject video customization. Although a direct comparison may not be feasible due to the lack of released code, it is still suggested to cite it in the related work to help strengthen the completeness of the literature review.\n\n[1] Huang, Chi-Pin, et al. \"Videomage: Multi-subject and motion customization of text-to-video diffusion models.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}