UniVid: Unifying Vision Tasks with Pre-trained Video Generation Models
Authors: name: Lan Chen; name: Yuchao Gu; name: Qi Mao
Published: 2025-09-26 01:43:40

Abstract: Large language models, trained on extensive corpora, successfully unify
diverse linguistic tasks within a single generative framework. Inspired by
this, recent works like Large Vision Model (LVM) extend this paradigm to vision
by organizing tasks into sequential visual sentences, where visual prompts
serve as the context to guide outputs. However, such modeling requires
task-specific pre-training across modalities and sources, which is costly and
limits scalability to unseen tasks. Given that pre-trained video generation
models inherently capture temporal sequence dependencies, we explore a more
unified and scalable alternative: can a pre-trained video generation model
adapt to diverse image and video tasks? To answer this, we propose UniVid, a
framework that fine-tunes a video diffusion transformer to handle various
vision tasks without task-specific modifications. Tasks are represented as
visual sentences, where the context sequence defines both the task and the
expected output modality. We evaluate the generalization of UniVid from two
perspectives: (1) cross-modal inference with contexts composed of both images
and videos, extending beyond LVM's uni-modal setting; (2) cross-source tasks
from natural to annotated data, without multi-source pre-training. Despite
being trained solely on natural video data, UniVid generalizes well in both
settings. Notably, understanding and generation tasks can easily switch by
simply reversing the visual sentence order in this paradigm. These findings
highlight the potential of pre-trained video generation models to serve as a
scalable and unified foundation for vision modeling. Our code will be released
at https://github.com/CUC-MIPG/UniVid.