Probabilistic Adaptation of Text-to-Video Models
Authors: name: Mengjiao Yang; name: Yilun Du; name: Bo Dai; name: Dale Schuurmans; name: Joshua B. Tenenbaum; name: Pieter Abbeel
Published: 2023-06-02 19:00:17

Abstract: Large text-to-video models trained on internet-scale data have demonstrated
exceptional capabilities in generating high-fidelity videos from arbitrary
textual descriptions. However, adapting these models to tasks with limited
domain-specific data, such as animation or robotics videos, poses a significant
computational challenge, since finetuning a pretrained large model can be
prohibitively expensive. Inspired by how a small modifiable component (e.g.,
prompts, prefix-tuning) can adapt a large language model to perform new tasks
without requiring access to the model weights, we investigate how to adapt a
large pretrained text-to-video model to a variety of downstream domains and
tasks without finetuning. In answering this question, we propose Video Adapter,
which leverages the score function of a large pretrained video diffusion model
as a probabilistic prior to guide the generation of a task-specific small video
model. Our experiments show that Video Adapter is capable of incorporating the
broad knowledge and preserving the high fidelity of a large pretrained video
model in a task-specific small video model that is able to generate
high-quality yet specialized videos on a variety of tasks such as animation,
egocentric modeling, and modeling of simulated and real-world robotics data.
More videos can be found on the website https://video-adapter.github.io/.