Probabilistic Adaptation of Black-Box Text-to-Video Models
Authors: Sherry Yang
Yilun Du
Bo Dai
Dale Schuurmans
Joshua B. Tenenbaum
Pieter Abbeel
Venue: ICLR 2024 poster (2024)
Keywords: Controllable video generation
probabilistic adaptation
video diffusion
black box adaptation
Primary area: generative models
TL;DR: We propose Video Adapter, a framework for generating personalized, stylized, and domain-specific videos by leveraging large pretrained text-to-video diffusion models as a probabilistic prior without requiring access to pretrained model weights.

Abstract: Large text-to-video models trained on internet-scale data have demonstrated exceptional capabilities in generating high-fidelity videos from arbitrary textual descriptions. However, similar to proprietary language models, large text-to-video models are often black boxes whose weight parameters are not publicly available, posing a significant challenge to adapting these models to specific domains such as robotics, animation, and personalized stylization. Inspired by how a large language model can be prompted to perform new tasks without access to the model weights, we investigate how to adapt a black-box pretrained text-to-video model to a variety of downstream domains without weight access to the pretrained model. In answering this question, we propose \emph{\methodname}, which leverages the score function of a large pretrained video diffusion model as a probabilistic prior to guide the generation of a task-specific small video model. Our experiments show that, by incorporating broad knowledge and fidelity of the pretrained model probabilistically, a small model with as few as 1.25% parameters of the pretrained model can generate high-quality yet domain-specific videos for a variety of downstream domains such as animation, egocentric modeling, and modeling of simulated and real-world robotics data. As large text-to-video models starting to become available as a service similar to large language models, we advocate for private institutions to expose scores of video diffusion models as outputs in addition to generated videos to allow flexible adaptation of large pretrained text-to-video models by the general public.

Published as a conference paper at ICLR 2024
PROBABILISTIC ADAPTATION OF BLACK-BOX SEXT-TO-VIDEE MODELS
Sherry Yang*.1.2 Yilun Du Bo Dai',
Dale Schuurmans Joshua B. Tenenbaum Pieter Abbeel GGoogle DeepMind, UC Berkeley, MIT, *University of Alberta video-adapter. .er..ggtiie github.
ABSTRACT
Large text-to-video models trained on internet-scale data have demonstrated ex- ceptional capabilities in generating high-fidelity videos from arbitrary textual descriptions. However, similar to proprietary language models, large text-to- video models are often black boxes whose weight parameters are not publicly available, posing significant challenge to adapting these models to specific domains such as robotics, animation, and personalized stylization. Inspired by how a large lan- guage model can be prompted perform new tasks without access to the model weights, we investigate how to adapt a black-box pretrained text-to-vide -iiddo model to a variety downstream domains without weight access to the pretrained model. In answering this question, we propose Video Adapter, which leverages the score function of large pretrained video diffusion model as a probabilistic prior to guide the generation of a task- small video model. Our experiments show that, by incorporating broad knowledge and fidelity of the pretrained model probabilis- tically, small model with few 1.25% parameters of the pretrained model can generate high-quality yet domain-specifiic videos for a variety of downstream domains such as animation, egocentric modeling, and modeling of simulated and real-world robotics data. As large text-to-video models are starting to become available as a service, similar large language models, we advocate for private enterprises to expose the scores of video diffusion models as outputs in addition to generated videos allow flexible adaptation of large pretrained text-to- video models by the general public. See website at htts::/viieo-aaaappee adapter.itthub.ii.
INTRODUCTION
Large text-to-vides models with billions of parameters trained on internet-scale data have become capable of generating highly realistic videos from general text descriptions (Ho et al., 2022: Hong et al., 2022; Singer et al., 2022). When models are used for specific domains such as generating video plans for roboties (Du et al., 2023b) and self-driving cars (Santana & Hotz, 2016), videos for animation (Wang et al., 2019), videos with customizable styles similar to those common in text-to-image (Wang et al., 2019; Ramesh et al., 2022; Liu et al., 2022; Gal et al., 2022; Ruiz et al., 2022; Zhang & Agrawala, 2023), a pretrained text-to-video model requires task-specific adaptation. Efficient and effective adaptation of text-to-video models is what stands in the way from expanding their current application of these models in media and entertainment to their potential to solve real-world problems by modeling real-world physics and dynamics in problem-specific settings.
Unfortunately, similar to state-of- ofte------herrrr language models (OpenAI, 2023; Anil et al., 2023), pre- trained text-to-video models are black boxes to the general public; one can use them to generate videos, but not finetune them to solve domain-specifee tasks, as the parameters of pretrained text-to- video models are not publicly available (Ho et al., 2022; Villegas al., 2022; Blattmann et al., 2023). This rules out direct applications of efficient finetuning from text-to-iin -inttto-mmaee... such LoRA in stable diffusion (Smith et al., 2023), DreamBooth (Ruiz et al., 2023), and ControlNet (Zhang & Agrawala, 2023), which require access to the pretrained model weights. While some finetuning- free techniques can control image generation by manipulating visual or textual features (Ramesh et al., 2022; Gal al,, 2022), is not clear how to manipulate these features for video generation, these features
"Equal contribution. syilundu @ii..edu>.
Correspondence to: Sherry Yang <sherryy beerre@@errrreee...... Yilun Du

---

Published as a conference paper at ICLR 2024
Egocentric
manipulation:
Puts weod table
Egocentric
manipulation: Scrubs hands sand
Egocentric
navigation:
Walks forward
Egocentric
navigation: Valks few stop
Stylization Animation
Stylization Science fiction
Robotics
Simulated
Robotics
Real world
Figure I: Video Adapter Generated Videos. Video Adapter capable of flexible generation of diverse videos with distinct styles including vidcos with manipulation and navigation hased egocentric motions, videos with personalized styles such as animation and science fictions, and simulated and real robotic videos.
would then have to capture complex temporal information extracted from networks with orders of magnitude larger sizes than text-to-image (Blattmann et al., 2023).
Inspired by finetuning-fre -ree adaptation of language models through in-context learning (Brown et al., 2020) and sophisticated prompting (Wei et al., 2022), which essentially modify the prior distribution of pretrained langauge models perform specific tasks, ask the natural question of whether it is possible to modify the prior distribution of pretrained text-to-video models to achieve downstream tasks without finetuning the pretrained model. Intuitively, even though the exact video statistics in downstream task differ from the pretraining videos, certain video properties such as dynamics of the world and semanties of objects from the large pretrained model are still tremendously helpful the generation of downstream videos. This suggests that large pretrained video model could be used as knowledge prior guide the generation of task-specific videos while maintaining broad properties such temporal consistency and object permanence.
To this end, propose Video Adapter, probabilistic approach for exploiting black-box video diffusion model to guide the generation of task and domain specific videos. By factoring the domain- specific video distribution into pretrained prior and small trainable component, can preserve desirable characteristics of the pretrained model (i.e., temporal consistency and object permanence) in generating specialized videos, effectively adapting the black-box pretrained model without requiring access to the pretrained model weights. One limitation of Video Adapter is that it requires scores of the black-box video diffusion model as outputs, but we note that this is hard avoid effectively use the broad knowledge of pretrained models. Therefore, we advocate for proprietary
wants to
2

---

Published as a conference paper at ICLR 2024
text-to-video APIs to expose diffusion scores additional outputs to broaden the applications of large video diffusion models.
We evaluate Video Adapter on a variety of tasks and domains as illustrated in Figure 1. Quantitatively, Video Adapter achieves better FVD and Inception Scores than the pretrained video model or the task-specific small models in generating domain specific videos for robotics (Ebert et al., 2021) and egocentric movements (Grauman et al., 2022). Qualitatively, we show that Video Adapter can generate stylized videos such as sci-fi and animation, and further enable domain randomization in robotics (Tobin et al., 2017) for bridging sim-to-real (Zhao et al., 2020) through randomized stylisation of lighting and distractors.
PRELIMINARIES
We first introduce relevant background information on denoising diffusion probabilistic models (DDPMs) and discuss their connection to Energy-Based Models (EBMs). We will then use this connection to EBMs to convert black-box video diffusion models to probabilistic priors.
Denoising Diffusion Probabilistic Models. Denoising diffusion probablistic models (Sohl- Dickstein et al., 2015; Ho al., 2020) are class of probabilistic generative models where the [ _H) E X is formed by iterative denoising. Given video sampled from a video distribution p(T), a randomly sampled Gaussian noise variable e ~ N I). and set of T different noise levels 3,, denoising model Eg is trained to denoise the noise corrupted
generation of video
video J at each specified noise level t E [1,
co (o(VV B + Bic, 1)
MSS =
Given this learned denoising function, new videos may be generated from the diffusion model by initializing video sample Tr noise level T from a Gaussian (N I). This sample Ty is then iteratively denoised following the expression:
(1) where Yt is the step size of denoising, a; is linear decay on the currently denoised sample, and oe is a time varying noise that depends on de and "/t- The final sample To after rounds denoising corresponds to the final generated video.
Tt- (T, (T, +))
(0,
Energy-Based Models View DDPMs. The denoising function eg estimates the score (Vincent, 2011; Song & Ermon, 2019; Liu et al., 2022) of an underlying (unnormalized) EBM probability distribution (LeCun et al., 2006; Du & Mordatch, 2019) characterizing the noise perturbed data. Therefore, a diffusion model corresponds to an EBM, ps (T) x e -Bo(T), where the denoising function is given by e(Tr,() Eot)..) The sampling procedure in a diffusion model corresponds to the Langevin sampling procedure on an EBM (see derivation in Appendix A):
(2) This equivalence of diffusion models and EBMs allows us to consider sampling from the product of two different diffusion models p1 (2))2(T), such that each diffusion model corresponds to EBM, e (r) and e Ea(r), and the product is given by e (r) = = (E)((())+ In particular, we can sample from this distribution also by using Langevin sampling:
Tt-1 az(7; v (T)) E), <~ {~NN(0, o1.)
T (T E. + E), E~ (0,
(3)
which corresponds to the sampling procedure using denoising functions
T--1 ax(TT (50, (Tr, 1) + <a, (T,(TT, ()) + +), (~
(4)
Below we will illustrate how this factored EBM parameterization of diffusion model can allow black-box pretrained model to be leveraged as a probabilistic prior.
3 PROBABILISTIC ADAPTATION OF BLACK-BoL -Box TEXT-TO- VIDEO MODELS
To explain how a black-box text- coxticonditonee video diffusion model can be effectively used as probabilistic prior for video generation, we will first introduce the functional form of probabolistic adaptation in Section 3. and then discuss how the probabilistic composition can be implemented with diffusion models in Section 3.2. To generate high-quality videos, we also explain in Section 3.3 how the underlying probabilistic composition can be sharpened to generate low temperature samples.
3.1 BLACK- Box TEXT-TO- TO- VIDEO MODELS AS PROBABILISTIC PRIORS
Black-box text-to- video models were pretrained massive datasets consisting of millions of videos, and are therefore able to capture a powerful prior Ppretrined (7 (T) text) on the natural distribution of videos Given smaller task-specific dataset of video-text pairs, D D ((T0, texto), ..., (Tn: text,,)) )}
3

---

Published as a conference paper at ICLR 2024
Training
Sampling
LAION
Pretrain
Pretruined
Pretrained
Pretrained Text-to-Video
YouTube
(5.68)
Eg + Aesttriined
Domain
Small Texdel--iideo
Specitic"
Small
Small
Figure 2: Video Adapter Framework. Video Adapter only requires training small domain -eeeciii---iiieeee text-to to video model with orders of magnitude fewer parameters than a large video model pretrained from internet data. During sampling, Video Adapter composes the scores of the pretrained and the domain specific video models, achieving high-quality and flexible video synthesis.
how can one leverage the powerful prior captured by pretrained video diffusion model to synthesize videos similar to those in Dadup? One approach is to directly finetune the weights of Ppretrained 7 text) using D Adtppt= but Ppeetrained (tet)) has billions of parameters whose weights are often proprietary to private enterprises. Similar challenges with large language models have led to prompting and in-context learning. Analogously, we propose Video Adapter as finetuning-free method to adapt pretrained video diffusion a new dataset of videos Dadapt through probabilistic composition. Specifically, given Adapt. we learn separate small video diffusion model pe pe(T text) to represent the distribution of videos in Aalpt We then adapt Ppretrained (tex) Dddapi by constructing a product distribution Pproduce (T [tex)) in the form (see adaptation to multiple domains in Appendix D):
Pproducc(1 (T (Tetex
Ppretrained (T|text) (F((ttext)
(5)
Product Distribution
Pretmined
Small Video Model
By fixing the pretrained model Ppeetrained (T (texx), we train the small video model po(r)text) via maximum likelihood estimation on Datapt- This allows po(T (tex)) to exhibit high likelihood across videos in Addppt but because Pe(r]text) is small model trained on less diverse data, it can exhibit erroneously high likelihood across many unrealistic videos. The product distribution Pprodue (T text) removes unrealistic videos by downweighting any videos that are not likely under the pretrained prior, enabling one to generate videos in the style of D Adapt that are realistic under Ppretrained (T text).
3.2 IMPLEMENTING PROBABILISTIC ADAPTATION
To adapt the black-box model (T from Equation 5, as well as to sample from it, we exploit the EBM interpretation of diffusion models discussed in Section 2. Based on the EBM interpretation, the pretrained diffusion model Ppretraimed text) corresponds to an EBM e E /(eex) while the smaller video model pe (T(tee))) parameterizes an EBM -E((]t) The product distribution then corresponds to:
Pprodues ( (T|txxt) X Pprettiieed text)) xx e - (Epramina (rl [ext))+ Eo(((ttt)) = e - E (xxx)) which specifies new EBM E' ET( from the sum of energy functions of the component models. Substituting this EBM into Equation 3, we sample from the product distribution Pproduca (txt) through the diffusion sampling procedure:
T ( (T. -V, (EVee(EEptinaa [tex) + (T((tttext)) + E), E~ (=,(00,0))
which corresponds to sampling from Equation according to
Tt-1 = are "(ppretrannedd (t. t/text) + <o(Tt, + +), E~ {~N(0,0;11)
Thus, to probabilistically adapt the pretrained black-box model new dataset D DAdapt. we can use the standard diffusion sampling procedure, but change the denoising prediction to the sum of predictions from both the black-box pretrained model and the task-specific small model. To control the strength of the pretrained prior in final video generation, we can introduce weight term À to scale the pretrained distribution P Pfrettained ( (ext),,, which corresponds to scaling the denoised prediction from Eprctrained (Te. f[text) by a scalar X
E(Tr, /(text) = EØ(TI, l|text) + A (T., & l[tet)).
The combined model can be further refined by integrating multiple steps MCMC sampling between each diffusion noise distribution similar to (Du et al., 2023a; Sjöberg et al., 2023). Note that the prior strength À can be tunable or time-dependent, which we found to be useful in practice (Appendix B). 3.3 ADAPTING Low TEMPERATURE SAMPLING
In practice, text conditioning in the denoising model ((7e,t\text) from Equation 6 are often parametrized using classifier-free guidance (Ho & Salimans, 2022) to generate sharp images or videos conditioned text while avoiding distractions from spurious likelihood modes of diffusion
4

---

Published as a conference paper at ICLR 2024
Algorithm 1 Sampling algorithm of Video Adapter
Input: Pretrained black-bor model Epratained f/text), inverse temperature w, prior strength Initialize sample 77 ~ ~N(O,T for to = port=T...... T...., do
Euxt +- <o(T,, f/jext) + A (T, t|text) (T.
I/ Compute score using text-conditioned prior. I/ Compute unconditional score. Compute weight for low temperature sampling. I/ Run diffusion sampling (can use other samplers).
Eat <- www
7t-1 = ddpm sampLe(7,,
end for
models. This corresponds to sampling from the modified probability distribution:
p((]text)
d (F((ext) x p(r)
o p(T))(tex()>)",
p(T)
where w corresponds to the classifier free guidance strength, typically chosen to be significantly larger than 1. By upweighting the expression p(text|r) via the inverse temperature w, the modified distribution (T (tex) above can generate lower temperature video samples conditioned on the text.
It appears straightforward to similarly construct low temperature samples when adapting the black-box model by sampling from the distribution
Pig Prroduce ( (txtext) Prretrained (T(tett) (r(ex),,
(7)
<lg but using the classifier-free distribution P'pretrained (T (Tete as the probabilistic prior is now problematic,
since classifier-fi free guidance has restricted P Ppeetrained P (T|text) to very few high probability modes which might be incompatible with Adapt- To effectively leverage a broad probabilistic prior while simultaneously generating low temperature samples emulating Dadapt. we propose to first construct new text-conditioned video distribution following Section 3.1:
Pproducx text) ε Pprctrained (T [tex) po (T|tt)).
We can then use the density ratio of this composed text-conditioned distribution with the unconditional video density (T( learned on D Addap to construct new implicit classifier Pproucr (T text)) By increasing the inverse temperature w on this implicit classifier, we can generate low temperature and high quality video samples conditioned on a given text by sampling from the modified distribution:
Ppreccuer ttxt) = (T( Peroduct (T text)
W
PT(T)
which corresponds sampling from a modified denoising function:
(T, l/text) = <0(T, + w(eo(T, L. l[text) + prpetrained (text)) - <o(T, ())
We quantitatively and qualitatively ablate the effect of this denoising function in Figure 6 and Table 2, showing that this variant leads better blending of styles between models. The overall pseudocode for the proposed approach with classifier-free guidance is given in Algorithm 1.
EXPERIMENTS
In this section, we illustrate how black-box pretrained text-to-video model can deliver a rich set of downstream capabilities when combined with a task-specific video model. In particular, leveraging high quality and broad probabilistic prior enables (1) controllable video synthesis from edge- only inputs. (2) high-quality video modeling that outperforms both the pretrained model and the task-specific video model, and (3) domain randomization and data augmentation for robotics. See experiment details and additional experimental results in Appendix B and in supplementary material.
4.1 ADAPTING TO SPECIFIC VIDEO DOMAINS
Setup. We first demonstrate that the probabilistic prior in Video Adapter can be used to adapt and modify the styles of videos. We curate two adaptation datasets D Dddapts one with an "animation" " style and the other with a "scifi" style, where videos containing relevant keywords in their descriptions are grouped together to form Addpu A black-box large video diffusion model with 5.6B parameters was pretrained mapping Sobel edges to all videos, and two task-specific small models with 330M parameters were trained to map Sobel edges to DAdapi videos.
Stylizing Video Generation. In Figure 4 and Figure 5, we demonstrate how the pretrained prior can adapt the animation and scifi models to alternative styles while maintaining the original animation and scifi contents. These results show that Video Adapter can effectively combine rich knowledge of
See video visualizations the supplementary zip file.
5

---

Published as a conference paper at ICLR 2024
Yelee Kecpee
dak Acull
Tolae Kurple
Dryt
Figure 4: Video Adapter enables stylization of Animation Specific Model. Video Adapter enables large pretrained model to adapt and change style a small animation style model.
Video Aduee Sadl
Wilee Aiouen
Vil koul
-aee VEa TV:
thut New
Figure 5: Video Adapter enables stylization of SciFi Specific Model. Video Adapter enables large pretrained model to adapt and change style a small Scifi animation style model.
styles from the black-box model, such as "digital art", "outdoor video", "storybook illustration", with the animation content the small model, thereby achieving flexible stylization.
Specific Animation Style. We further trained a small video model on an "animation" style of a particular artist. In Figure 3, we illustrate how the pretrained prior can maintain the anime content while changing the styles such as back- ground color.
Analysis. In Figure 6, we change the mag- nitude of the weight on the pretrained prior, and compare Video Adapter with directly in- terpolating the classifier-free scores between the pretrained and adapter models (as in Equa- tion 7). We find that Video Adapter main- tains the adapter style more accurately, whereas classifier-fres free score interpolation collapses to the teacher style with intermediate interpolation, leading erratic artifacts.
(Dua
Figure Instance Specific Stylization. Video Adapter enables the stylization of video model trained on single animation style
4.2 HIGH-QUe QUALITY EFFICIENT VIDEO MODELING
Setup. To demonstrate Video Adapter's ability in adapting the black-box pretrained model to domains that are not a part of pretraining, we consider adapting to Ego4D (Grauman et al., 2022) and Bridge Data (Ebert et al., 2021). These adaptations are nontrivial, as Ego4D consists mostly egocentric videos that are not commonly found the internet. Similarly, the Bridge Data consists of task-specific videos of a WidowX250 robot that out the distribution of the pretraining data. For Ego4D, we take subset of the original dataset consisting of 97k text- video pairs and split them into train (90%) and test (10%) to form D Addge- For the Bridge Data, we take the entire dataset consisting of 7.2k text-video pairs and use the same train-test split to form D Dddapt

---

Published as a conference paper at ICLR 2024
Video
Eiges
Srall
Woght 95
Cresse Hose
Woggh:
Figure 6: Analysis of Video Adapter. As adaptation weight increases, Video Adapter modifies the style as instructed (left), whereas directly mixing two classifier-free guidance scores fails to adapt the video (right).
Bridge
Ego4D
Model
FVD 4
FID
Param (B))
FVD 4
TS
Param (B) +
Small (S)
186.8
38.8
0.07
228.3
2.28
0.07
Small (S) Pretrained
177.4
37.6
0.07
156.3
2.82
0.07
Small (L)
152.5
30.1
0.14
65.1
3.31
2.8
Small (L) + Pretrained
148.1
29.5
0.14
52.5
3.53
2.8
Pretrained
350.1
42.6
5.6
91.7
3.12
5.6
Pretrained Finetune
321.0
39.4
5.6
75.5
3.33
5.6
Table Video Modeling Quantitative Performance Video Adapter (Small + Pretrained) achieves better FVD, FID, and Inception Scores than both the pretrained model, pretrained model finetuned for equivalent number of TPU hours, and the task-specifte small model with parameters as fewer as 1% of the pretrained model.
For the pretrained model, we use the 5.6B base model pretrained on generic internet videos from (Ho et al., 2022). For the task-specific small model, we downscale the video diffusion model from (Ho et al., 2022) by factor of 80, 40, and to create a diverse set small models to be trained on task-specific DAdai Table shows the number of parameters of pretrained and small video models. Both the pretrained model and the small models are trained to generate subsequent frames conditioned on the first frame.
Quantitative Results. Table shows the quantitative performance of Video Adapter under different video modeling metrics. On the Bridge Data, training a small model with parameters equivalent to 1.25% of the pretrained video model (first row) already achieves better metrics than the pretrained model. However, Video Adapter incorporating the pretrained model as a probablistic prior is able to further improve the metrics of the small model (second row). On Ego4D, due the complexity of the egocentric videos, the smallest model with 1.25% of the pretrained vidco model can no longer achieve performance better than the pretrained model (first row), but incorporating the pretrained model during sampling still improves performance (second row). After increasing the size of the small model, Video Adapter is able to achive better metrics than both the pretrained and task-specific model. We further compare Video Adapter to finetuning the pretrained model for an equivalent number of TPU hours (see Appendix C), and show that Video Adapter achieves better performance than full tuning. Note that only compare to full tuming out of curiosity opposed to benchmarking, the motivation of this work is the lack weight access to the black-box pretrained models.
Qualitative Results. Figure and Figure 8 show the gener- ated videos on Bridge Data and Ego4D. On the Bridge Data in Figure 7, the pretrained model produces videos that do not cor- respond to the task described by the text (there is robot arm movements in the generated video). The task-specific small model produces videos with unrealistic movements that tele- port the robot arm. Video Adapter, on the other hand, produces videos with realistic movements that complete the task.
Model
FVD J FID 1
CFG Mix 167.4
33.1
Small (L)
152.5
30.1
Video Adapter
148.1
29.5
Table 2: Ablations. Video Adapter im-
proves the underlying video modeling performance of models on while directly mixing classificr-free scores (CFG Mix) hurts performance.
On Ego4D in Figure 8, the pretrained model produces high quality videos that contain little egocentric movement (first row), as the pretraining data mostly consists of generic videos from the internet that are not egocentric. The task-specific small model trained Ego4D, on the other hand, produces videos with egocentric movement but of low quality (second row) due to limited model capacity. Video Adapter combines the best of both and generates high-quality egocentric videos (third row).
7

---

Published as a conference paper at ICLR 2024
Pur pot pan in sink.
Pur eggplant into
Pretrained
Small
Video
Adepter
Put pan from drying rack into sink.
Put corn into bowl.
Pretroined
Small
Vidro
Adepter
Figure 7: Video Adapter Bridge Data. The pretrained model (first row) produces videos that are high-quality but generally static and fail complete the task. The small (L) model (sccond row) produces low-quality videos with unrealistic arm movements. Video Adapter (third row) produces high-quality videos and successfully completes the task. Removes hand cable bicycle's scal.
around.
Pretmiined
Video Adepter
fabeic.
Pretrained
Small
Video
Adepter
Figure 8: Video Adapter on Ego4D, The pretrained model (first row) produces high- quality but nearly static videos that do not reflect the egocentric nature. The small (L) model (second row) produces low-quality videos but with more egocentric movements. Video Adapter (third row) produces high-quality and egocentric videos.
Ablations. In Table 2, we report generative modeling performance of the small model on Bridge interpolation between the classifier-free scores of pretrained and
either using Video Adapter, small models. We find that Video Adapter improves performance, while interpolation between classifier-free scores hurts performance.
4.3 SIM-TO-REAL VIDEO AUGMENTATION
Setup. One important application of controllable video synthesis is to render realistic robotic videos from simulation with variety of data augmentations so that policies trained on the augmented observations are more likely to generalize well real-world settings (Zhao et al., 2020). To demon- strate Video Adapter's capability in supporting sim-to-real transfer, we train task-specific small edge-to-real model on 160k real robot trajectories of the LanguageTable dataset (Lynch et al., 2022), generating videos of execution conditioned
the Sobel edges of the real videos. Similarly, train another small edge-to-sing timm model on 160k simulated robot videos. Note that the simulated and real robotics data not paired (paired sim-to-real data are hard to find) but connected through We again leverage the edge-conditioned large model pretrained on internet data for style specification.
8

---

Published as a conference paper at ICLR 2024
Move the rod block towards the blue block.
Push the green star into the rod circle.
Simulated roborics
Real robotics
Adapted
('colorful' "dark")
Figure 9: Video Adapter on sim-to-real transfer. First row: simulated videos of execution plans generated by Video Adapter. Second row: real videos execution plans generated by Video Adapter. Third row: real videos of execution plans generated by Video Adapter with data augmentation.
Adapted Videos. Figure 9 shows the generated robotic videos from Video Adapter. Video Adapter can effectively generate paired simulated and real robotic videos that complete task described by language prompt. and further generate videos with various data augmentation styles that can be utilized to train policies with better sim-to-real transfer abilities through techniques similar to domain randomization (Tobin et al., 2017).
5 RELATED WORK
Text-to- Video Synthesis. Following the recent success of text-to text-to-image models (Nichol et al., 2021;
Balaji et al., 2022; Ramesh et al., 2022; Rombach al., 2022; Saharia al., 2022b; Yu al., 2022; Chang et al., 2023), large text-to-video models with autoregressive (Hong et al., 2022; Villegas et al., 2022; Wu al., 2022a; 2021) and diffusion (Ho et al., 2022; Singer et al., 2022; Blattmann et al., 2023; Zhou et al., 2022; Esser et al., 2023) architectures have been developed, often by extending existing text-to-image models. Unfortunately, the model weights of large text-to-video models are generally not publically available, preventing downstream adaptations of these models.
Adapting Pretrained Models Adapting pretrained models for customized editing, inpainting, and stylization has been extensively studied in -ottt and image-to-image translation models (Gal et al., 2022; Hertz et al., 2022; Kawar et al., 2022; Li et al., 2022; Lugmayr et al., 2022; Meng et al., 2021; Ruiz et al., 2022; Saharia et al., 2022a; Sasaki et al., 2021; Su al., 2022). In text-to-video, most existing work either leverages text prompts (Molad et al., 2023; Esser et al., 2023), finetunes a pretrained model on stylized data (Wu al., 2022b), or performing light training on a copy of the pretrained video model similar to ControlNet (Dhesikan & Rajmohan, 2023). Text-only adaption can be unreliable, whereas finetuning, prefix-tuning (Li & Liang, 2021). low-rank adaptation (Hu et al., 2021), I), and ControlNet all require access to the pretrained model weights, which are often not available for text-to-video models. Black-box adaptation has been applied extensively language models (Brown et al., 2020; Wei et al., 2022; Liu et al., 2023), and large video models will soon face the same problem.
Compositional Generative Models. The techniques in this paper are further related to existing work on compositional generative modeling (Liu et al., 2022; Nie et al., 2021; Du et al., 2020; 2023a; 2021; Liu al., 2021; Wang et al., 2023; Wu et al., 2022c; Deng et al., 2020; Urain et al., 2021; Gkanatsios et al., 2023; Gandikota et al., 2023; Po & Wetzstein, 2023). where different generative models probabilistically combined to jointly generate outputs. In (Du et al., 2020), approach to combine different probability distributions using EBMs is introduced. Most similar in spirit to this work, (Deng et al., 2020) composes pretrained language model with small EBM to improve language generation. However, different from this work, the small EBM is used to improve to global consistency of the language model, whereas we aim to use a small model to probabilistically adapt to a large pretrained video model to separate domains.
LIMITATION AND CONCLUSION
As video foundation models become more powerful but remain proprietary, black-box adaptation of these models is inevitible. We have proposed Video Adapter for leveraging black-box text-to-video models as probabilistic priors for guiding generation of specific videos. One limitation of Video Adapter is it still requires training a small domain domain-specific model, so adaptation is not completely training free. Another limitation is Video Adapter requires diffusion scores from pretrained black-box models. We advocate future video diffusion models to make scores as a part of the output to improve accessibility of these models.

---

Published as a conference paper at ICLR 2024
REFERENCES
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, al. Palm 2 technical report. arXiv preprint arXiv:2305. 10403, 2023.
Yogesh Balaji, Seungjun Nah, Xun Huang. Arash Vahdat, Jiaming Song, Karsten Kreis, Miik Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. ediffi: text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.01. 324, 2022.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. arXiv preprini arXiv:2304.08818 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint arXiv:2301.0077
2023.
Yuntian Deng, Anton Bakhtin, Myle Ott, Arthur Szlam, and Marc' Aurelio Ranzato. Residual energy-based models for text generation. arXiv preprint arXiv:2004.1171: 2020.
Rohan Dhesikan and Vignesh Rajmohan. Sketching the future (stf): Applying conditional control techniques text-to-video models. arXiv preprint arXiv2230.005845, 2023.
Yilun Du and Igor Mordatch. Implicit generation and generalization in energy- based models. arXiv preprint arXiv: DeXiiv/9903.. /90.:0990300089 2019.
Yilun Du, Shuang Li, and Igor Mordatch. Compositional visual generation with energy based models. Advances in Neural Information Processing Systems, 33:6637-666 6647, 2020.
Yilun Du, Shuang Li, Yash Sharma, B. Joshua Tenenbaum, and Igor Mordatch. Unsupervised learning of compositional energy concepts. In Advances in Neural Information Processing Systems, 2021.
Yilun Du, Conor Durkan, Robin Strudel, Joshua B Tenenbaum, Sander Dicleman, Rob Fergus, Jascha Sohl-Dickstein, Arnaud Doucet, and Will Grathwohl. Reduce, reuse, recycle: Compositional generation with energy-based based diffusion models and meme. arXiv preprint arXiv:2302.11552 2023a.
Yilun Du, Mengjiao Yang. Bo Dai, Hanjun Dai, Ofir Nachum, Joshua B Tenenbaum, Dale Schuur- mans, and Pieter Abbeel. Learning universal policies via text-guided video generation, 2023. URL
htts::/arxivv. org/abbs2300.00111, 2023b.
Frederik Ebert, Yanlai Yang. Karl Schmeckpeper, Bernadette Bucher, Georgios Georgakis, Kostas Daniilidis, Chelsea Finn, and Sergey Levine. Bridge data: Boosting generalization of robotic skills with cross-domain datasets. arXiv preprint 1xxiv:2109.13399 2021.
Ahmed El Alaoui, Andrea Montanari, and Mark Sellke. Sampling from the Betriington-kirppatte gibbs measure via algorithmic stochastic localization. 2022 IEEE 63rd Annual Symposium on Foundations of Computer Science (FOCS), pp. 323-334 IEEE, 2022.
Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Ger- manidis. Structure and content-guided video synthesis with diffusion models. arXiv preprint arXiv:2302.030/1, 2023.
Rinon Gal, Yuval Alaluf. Yuval Atzmon, Or Patashnik. Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.01615 2022.
Rohit Gandikota, Joanna Materzynska, Jaden Fiotto Kaatto-a and David Bau. Erasing concepts from diffusion models. arXiv preprint arX1v:2303.07335, 2023.
10

---

Published as a conference paper at ICLR 2024
Nikolaos Gkanatsios, Ayush Jain, Zhou Xian, Yunchu Zhang, Christopher Atkeson, and Katerina Fragkiadaki. Energy-based models as zero-shot planners for compositional scene rearrangement. arXiv preprint arXiv:2304.14391, 2023.
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 18995-19012, 2022.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen- Or. Prompt- to-prompt image editing with cross attention control. arXiv preprint Sit::2008.016226, 2022.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6855 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.020:
2022.
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205. 2022.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0960 09685. 2021.
Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. arXiv preprint arXiv:2201.117933 2022.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Huang. A tutorial on energy-based learning. Predicting structured data, 1(0), 2006.
Haoying Li, Yifan Yang, Meng Chang, Shiqi Chen, Huajun Feng. Zhihai Xu, Qi Li, and Yueting Chen. Srdiff: Single image super-resolution with diffusion probabilistic models. Neurocomputing, 479:47-59, 2022.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.0011 2021.
Nan Liu, Shuang Li, Yilun Du Josh Tenenbaum, and Antonio Torralba. Learning compose visual relations. Advances in Neural Information Processing Systems, 14:23166-2333 278, 2021.
Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable diffusion models. arXiv preprint 2rXiv:2206.01714, 2022.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-35, 2023.
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Lue Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEFF/CV Conference on Computer Vision and Pattern Recognition, pp- 11461-11471, 2022.
Corey Lynch, Ayzaan Wahid, Jonathan Tompson, Tianli Ding, James Betker, Robert Baruch, Travis Armstrong, and Pete Florence. Interactive language: Talking to robots in real time. arXiv preprint arXiv:2210.06407 2022.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In International Conference Learning Representations, 2021.
11

---

Published as a conference paper at ICLR 2024
Eyal Molad, Eliahu Horwitz, Dani Valevski, Alex Rav Acha, Yossi Matias, Yael Pritch, Yaniv Leviathan, and Yedid Hoshen. Dreamix: Video diffusion models are general video editors. arXiv preprint WtXiv:2302.003222 2023.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2 10741, 2021.
Weili Nie, Arash Vahdat, and Anima Anandkumar. Controllable and compositional generation with latent-space energy-based models. Advances in Neural Information Processing Systems, 34, 2021.
OpenAl. Gpt-4 technical report, 2023.
Ryan Po and Gordon Wetzstein. Compositional 3d scene generation using locally conditioned diffusion, arXiv preprint arXiv:2303. 12218, 2023.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, (1(1):5485-5555 2020.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani. Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. arXiv preprint arXiv:2208. /224 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22500-22510, 2023.
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings, pp. 1-10, 2022a.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479 94449,,, 2022b.
Eder Santana and George Hotz. Learning driving simulator. arXiv preprint arXiv: 1608.0000 2016.
Saced Saremi and Aapo Hyvarinen. Neural empirical bayes. arXiv preprint UrXiv:1903002333 2019.
Saced Saremi, Arash Mehrjou, Bernhard Schölkopf, and Aapo Hyvärinen. Deep energy estimator networks. arXiv preprint arXiv:1805.08300 2018
Hiroshi Sasaki, Chris G Willcocks, and Toby P Breckon. Unit-ddpm: Unpaired image translation with denoising diffusion probabilistic models. arXiv preprint 1r4iv:2104.05358 2021.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint 2rXiv:2209. 14792, 2022.
Anders Sjöberg, Jakob Lindqvist, Magnus Önnheim, Mats Jirstrand, and Lennart Svensson. Memc-correction of score-based diffusion models for model composition. arXiv preprint 1401 2023.
12

---

Published as a conference paper at ICLR 2024
James Seale Smith, Yen-Chang Hsu, Lingyu Zhang, Ting Hua, Zsolt Kira, Yilin Shen, and Hongxia Jin. Continual diffusion: Continual customization text-to- to-taggeaaee diffusion with c-lora. arXiv preprint ar A.X1v:2334.000222 2023.
Jascha Sohl- Dicksteei, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamies. In International Conference Machine Learning, pp. 2256-2265. PMLR, 2015.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Xuan Su, Jiaming Song, Chenlin Meng, and Stefano Ermon. Dual diffusion implicit bridges for image- to-image translation. In The Eleventh International Conference on Learning Representations, 2022.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 JEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 23-30. IEEE, 2017.
Julen Urain, Anqi Li, Puze Liu, Carlo D' D'Eramo, and Jan Peters. Composable energy policies for reactive motion generation and reinforcement learning. arXiv preprint arXiv:2105.04902 2021.
Ruben Villegas, Mohammad Babacizadeh, Picter-Jan Kindermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual description. arXiv preprint 2rXiv:2210002399 2022.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computa- fion, 23(7):1661-1674, 2011.
He Wang. Sören Pirk, Ersin Yumer, Vladimir Kim, Ozan Sener, Srinath Sridhar, and Leonidas Guibas. Learning generative model for multi-step human-object interactions from videos. In Computer Graphics Forum, volume 38, pp. 367-378. Wiley Online Library, 2019.
Zihao Wang, Lin Gui, Jeffrey Negrea, and Victor Veitch. Concept algebra for text-controlled vision models. arXiv preprint arXiv:2302.0362. 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Lc, and Denny Zhou, Chain of thought prompting elicits reasoning in large language models. arXiv preprint mrXi::2201.11933, 2022.
Chenfei Wu, Lun Huang, Qianxi Zhang, Binyang Li, Lei Ji, Fan Yang, Guillermo Sapiro, and Nan Duan. Godiva: Generating open-domain videos from natural descriptions. arXiv preprint arXiv:2104. 14806, 2021.
Chenfei Wu, Jian Liang, Lei Ji, Fan Yang. Yuejian Fang, Daxin Jiang, and Nan Duan. Nüwa: Visual synthesis pre-training for neural visual world creation. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 27- 2022, Proceedings, Part XVI, pp. 720-736. Springer, 2022a.
Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Weixian Lei, Yuchao Gu, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. arXiv preprint arXiv:2212. 1565, 2022b.
Tailin Wu, Megan Tjandrasuwita, Zhengxuan Wu, Xuelin Yang, Kevin Liu, Rok Sosic, and Jure Leskovec. Zeroc: A neuro-symbolic model for zhro-shot concept recognition and acquisition at inference time. Advances in Neural Information Processing Systems, 35:9828-9844 2022c.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content- rich text-to-image generation. arXiv preprint arXiv:2206. 10789, 2022.
13

---

Published as a conference paper at ICLR 2024
Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models, 2023.
Wenshuai Zhao, Jorge Peña Queralta, and Tomi Westerlund. Sim-to-real transfer in deep reinforce- ment learning for robotics: survey. In 2020 IEEE symposium series on computational intelligence (SSCI), pp. 737-744. IEEE, 2020.
Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.11018, 2022.
14

---

Published as a conference paper at ICLR 2024
Appendix
In the Appendix we provided detail derivation of connection between diffusion models and EBMs in Section A. We further provide additional experimental details in Section B. Finally, provide comparison with using the same computational budget to finetune the existing large pretrained model in Section C.
A CONNECTION BETWEEN DIFFUSION AND EBM
The sampling procedure a diffusion model corresponds to the Langevin sampling procedure on an EBM. To see this, we consider perturbing sample - (T- from target distribution p(rf-1) with Gaussian noise, i.e.,
- +5,
& &~ N (0. 1
which corresponds to the transition operator
-
T(-'|='-1) x exp
20
where the joint distribution of 7' and -' 7' is
1| -'-1
p(r 1 x exp
1b (+(011
2
We can express the Bayes estimator of 7'-1 given the perturbed observation r² as
m(7") / po( pp'd"= r' + of log (7')
(8)
Proof By the property of Gaussian distribution, we have
a o' = (x') (x x').
(9)
Therefore, we have
p(x)dxx = 1 (1---))( 2') oV logg(p() = = (4',2))
VVa f p(x)))
dr = (p - - (10)
dx - x' =E
(11)
(x²)
Thus, we can represent the perturbed data with an EBM p(r*) x exp (Eg (r,,))), and learn the parameters through regression (Vincent, 2011; Saremi et al., 2018; Saremi & Hyvarinen, 2019; Song Ermon, 2019), which leads the optimal solution
min E, (ND.ENN(0.033))
|| ml m1
Er, V0. ||| Ea (f² + e, a,).))
(12)
whch also corresponds to the denoising diffusion training objective. Once have the trained Eg (r*),
then recover the sample -- according the denoising
sampling procedure
1-| a' m'm((7)) + a'E -' (r', o.)) - & N~N (0,of1)
(13)
which corresponds to the sampling via stochastic localization (El Alaoui et al., 2022) and Equation 2 the main paper.
B EXPERIMENTAL DETAILS
B. EXPERIMENT DETAILS
Dataset The large pretrained model is trained on 14 million video-text pairs plus 60 million image- text pairs, and with the LAION-400M image-text dataset. The images are spatially resized to 24x40 and videos using anti-aliased bi-linear resizing. We use different frame rate for different types of videos for best visualization results. For the Bridge (Ebert et al., 2021) we directly use the released opensource dataset. For Ego4D (Grauman et al., 2022) data, take small portion of the released dataset. For Anime and Sci-Fi style, we curate two separates datasets with their respective keywords. The keywords used for filtering data for Anime style (in small letter) "disney", "cartoon"; "anime".
15

---

Published as a conference paper at ICLR 2024
"animation", "comic", "pixar", "animated", "fantasy". The keywords used for filtering data for Sci-Fi style are "science fiction", "sci-fi", "scifi", "astronaut", "alien", "NASA", "interstellar". For the the Case Closed animation (also named Detective
animation with a particular artist style, Conan). For the Language Table dataset, we used the data from (Lynch et al., 2022).
Dataset
Pretrain
Bridge
Ego4D
Anime
Sci-Fi
Case Closed LangTable Sim LangTable Real
Train
474M
2.3k
97k
0.6M
21k
0.16M
0.16M
Table 3: Training data size. Number of text-video or text-image pairs used for training the pretrained large model and each of the small model. Training data for particular styles can be magnitude smaller than the pretraining dataset.
Architecture. To pretrain the large model, we use the same pretraining dataset, base architecture, and training setup as (Ho et al., 2022), with modifications of first-frame conditioning for Bridge and Ego4D, and edge conditioning for stylisation and sim-to-real. Specifically, the large model architecture consists of video U-Net with 3 residual blocks of 1024 base channels and channel multiplier [1, 2, 4], attention resolutions [6, 12, 24], attention head dimension 64, and conditioning embedding dimension 1024. To support first frame conditioning, we replicate the first frame across all future frame indices, and concatenate the replicated first frame channel-wise to the noisy data following (Du et al., 2023b). To support edge conditioning, we run a sobel edge detector and use gradient approximations in the x-direction the conditional video, and concatenates these edge frames with noisy data similar to first-frame conditioning. The large model consists of 5.6 billion parameters in total. For the set of small models for adaptation, Ego41) Small (L.) has 512 base channels in each of the residual blocks. Ego4D Small (S) and Bridge Small (S) have single residual block with 32 base channels. Bridge Small (L) has single residual block with 64 base channels. The set of stylisation models (animation, sci-fi, and particular anime style) have 3 residual blocks and 256 base channels. For illustrating the generated videos at higher resolution, we train two additional spatial super resolution models 24x40 + 48x80 (1.4B) and 48x80 - 192x320 (1.2B). We additionally use T5-XXL (Raffel et al., 2020) to process input text prompts which consists of 4.6 billion parameters, which omit from the parameter count all large and small models require text embeddings.
Training and Evaluation. We train each of our video diffusion models for 2M steps using batch size 2048 with learning rate le-4 and 10k linear warmup steps. The large 5.6B pretrained model requires 512 TPU-v4 chips, whereas various small models require anywhere between 8 and 256 TPU-v4 chips depending on the size. use noise schedule log SNR with range [-20, 20]. We use 128 samples and 1024 samples to compute the FVD, FID, and Inception Scores metric on Bridge and Ego4dD, respectively.
Sampling. All diffusion models are trained with 1000 timesteps of sampling. To generate videos, we combined scores from both pretrained models and adapter models for all timesteps except the last 100 timesteps. The last 100 timesteps capture high frequency information in an image, and we found better image quality if we did not combine scores in these timesteps. For simplicity, we use a pretrained neural strength of 0.2 for Ego4D and 0.1 for Bridge, and 0.4 for all animation datasets, but found additional gains when using large neural strength at earlier timesteps and smaller ones later.
16

---

Published as a conference paper at ICLR 2024
C COMPARISON TO FINETUNING
To illustrate the computational efficiency of Video Adapter, we further compare video modeling metrics of Video Adapter to finetuning the pretrained model for equivalent number of TPU time. Specifically, the pretrained model requires 512 TPU-v4 chips whereas the small model on Bridge data requires 8 TPU-v4 chips. The small Bridge model requires 100k steps to reach convergence, and hence we finetune the pretrained model for 100,000/ / 64 = 1,560 steps. Video Adapter achieves better FVD and FID than finetuning the pretrained model for an equal number of TPU steps as shown in Table 1.
D COMPOSING MULTIPLE FACTORS
Given set of N separate datasets {Dihi=I:N specifying a set of N different styles of videos, we can also apply our probabilistic adaptation framework across these N models by learning N separate distributions (F:(T [ext)] We can directly sample from the product distribution
Pproduct Pproduca(T (T|text)
Peretrained (T text)
II (T ttex)) 1=1:N
(14)
Distribution
Pretmnined
N Different Video Models
which now assigns high likelihood to videos that exhibit each of the composed styles. We can directly sample from this composed model using the modified composite denoising function
N
Co(T, f(tett) = co(T, () + w
(c,(T, ftext) + prettaiied (T, ttxttt - < ()),
i=1
which simply corresponds to using weighted average over the predictions of each of the N models.
E COMPARISON TO PARAMETER EFFICIENT FINETUNING
The problem setting of Video Adapter that pretrained model weights are not accessible. This scenario is common in large language models (e.g., GPT-4, Bard), and large text-to-video models are heading the same direction. Parameter efficient finetuning (e.g., LoRA, null-text inversion, prefix- tuning) requires access to pretrained model weights, whereas Video Adapter does not. Therefore Video Adapter is not comparable to parameter efficient finetuning.
Nevertheless, we conducted comparisons to LoRA and null-text inversion out of curiosity (prefix- tuning is omitted since it has only been applied to language models). For LoRA, we use rank and rank 64 to compare to the smaller and larger task-specific VideoAdapter model. For null-text inversion, we use an unconditional null embedding of size [64, 4096) (the same dimension as the original text embeddings). We report the video modeling metrics in Table 4.
Bridge
Ego4D
Model
FVD 1
FID J
FVD
IS f
Small (S)
186.8
38.8
228.3
2.28
Small (S) + Pretrained
177.4
37.6
156.3
2.82
Small (I.)
152.5
30.1
65.1
3.31
Small (L) Pretrained
148.1
29.5
52.5
3,53
LoRAA-Rakk
170.2
32.2
74.5
3,4
LoRA-Rank64
165.5
31.6
50.3
3.5
Null-text inversion
288.8
40.2
90.2
3.1
Table 4: Comparison of Video Adapter to parameter efficient finetuning under fixed compute budget. Video Adapter performs close MERA-Rank64 on Ego4D and better than parameter efficient finetuning on Bridge.
We observe that LoRA-Rank performs slightly better than Video Adapter (small). However, In the LoRA-Rankl case, LoRA still performs worse than training a small domain specific model. this case, Video Adapter can simply use the small model without the pretrained prior. In comparison, we found LoRA-Rankóó leads to mixed results when compared to Video Adapter (large), i.c., LoRA outperforms Video Adapter Ego4D but not Bridge data. We found that null-text inversion performs the worst, potentially due to limited flexibility of null-emheddings during finetuning.
Our results illustrate that Video Adapter, despite requiring only black-bo adaptation without access to pretrained model weights performs better than Null-text inversion and very comparably to LoRA finetuning (with pretrained model weights).
17


Reviews:
[{"review_id": "lqvwVRvrwV", "reviewer": "Reviewer_Z2pV", "summary": "The Video Adapter framework addresses the challenge of adapting black-box, large text-to-video models to specific domains such as robotics and animation, without access to the models' weight parameters. This innovative approach, inspired by the prompting mechanism in large language models, uses the score function of a pretrained video diffusion model as a probabilistic prior, guiding the generation of a smaller, task-specific video model. Remarkably, this smaller model, with only 1.25% of the parameters of the original, can generate high-quality and domain-specific videos across diverse applications. The authors advocate for private enterprises to provide access to the scores of video diffusion models, enhancing the adaptability of large pretrained text-to-video models, with video demos available in the supplementary material.", "questions": "Please check weakness for details.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1. The tackled problem is important and practical.\n2. The proposed idea is intuitive.\n3. Some of the results look interesting.", "weaknesses": "1. The current manuscript could really use a better position to avoid over-claiming.\n\na. Currently the paper highlights the ability of adapting black-box pretrained large text-video model. However, it relaxes the assumption of obtaining intermediate generation results of the black-box model, which is not realistic, and even contradicts with the term \"black-box\". It is really inappropriate to just use an existing term in a completely different setting.\n\nb. The current manuscript also highlights parameter efficiency. However, the current evidence is not enough to really claim this. There has been some many parameter efficient fine-tuning techniques and the comparison with such existing method is really needed to claim superiority from this part. Specifically, comparisons with LoRA and other SOTA PEFT methods should be at least provided to help the audience really know the performance of existing methods.\n\nc. The current setup is actually not that highlighting efficiency. The largest small model can even take half of the computation of the large model. It would be much more rigorous if the authors can set a computation budget and compare methods under a fixed budget.\n\n2. Evaluation of the video generation. From qualitative inspect, I don't really think Video Adapter in Figure 8 is really better. Is there any case study further examining the human preference and the score obtained from automatic evaluation, especially on datasets like Ego4D?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OGFgSdzrj3", "reviewer": "Reviewer_yyCL", "summary": "The paper proposes an adapter-based video-generation model training methodology. They use a large pre-trained videogen model and use the output scores from the model (at each step of denoising in diffusion models) to help train a smaller model for a specific task.\nThe black box large pre-trained model has been used as a prior to learn a distribution of new stylistic videos in a finetuning-free, probabilistic composition framework. The small model is learned via MLE and the black box model EBM interpretation is used to sample from it. To probabilistically adopt the pre trained black-box model to new data, the denoising prediction is changed to the sum of predictions from both the black-box pre-trained model and the task-specific small model. The combined model is further refined by MCMC sampling. \n The authors show successful experiments on controlled video generation, egocentric videos, and realistic robotic videos from simulation.", "questions": "The description of “Controllable video synthesis” is not very clear in terms of inputs. The text description as inputs is not provided very clearly hence someone without knowing the problem statement of this particular problem has difficulty in understanding what are the inputs.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors tackle a novel problem of building proprietary video models which has not been addressed before in the video generation domain. The videos provided as supplementary material look great.\n2. The paper is well-written and self-explanatory with well-cited results and related work", "weaknesses": "The paper is based on the fact that there will be more closed source companies willing to open source their diffusion scores.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iuysOVRvXO", "reviewer": "Reviewer_nNvZ", "summary": "This paper proposed a new method called Video Adapter to adapt a large pretrained text-to-video diffusion model for new transfer and enviroment. Video Adapter trained a small domain-specific model to provide the score to guide the inversion trajectory of large-scale pretrained model. The proposed method was successful in many downstream tasks including animation, style transfer and egocentric modeling.", "questions": "1. The prior strength weight is constant during the sampling. It would be interesting to see if that the prior strength weight is changing with t could be helpful.\n\nI would increase my rating if my concerns are properly addressed.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. This paper is easy to follow, and the overall writing is good.\n2. The proposed method, Video Adapter, is a simple yet effective way to guide the inversion trajectory to adapt for new scenarios.", "weaknesses": "1. My major concern is the lack of comparison with other baseline models like Lora, null-text inversion and prefix-tuning. Surely, these methods I mentioned above were designed for image adaptation but, correct me if I am wrong, it seems that they can be easily transplanted to the video diffusion model to serve as baselines models. Authors should explain more about why such comparison is not necessary.\n\n2.In table 1, small+pretrained demonstrate the best quantitative performance on both Bridge and Ego4D. However, these numbers are very close to only small variant. Considering the poor generative quality of small model in Figure 7 and 8, it is questionable whether small+pretrained can generate high-quality videos consistently.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YXtu8zFco3", "reviewer": "Reviewer_4TFV", "summary": "This paper investigates how to adapt a pre-trained text-to-video model to a specific domain without modifying the pretrained model. The authors propose Video Adapter, which trained a task-specific small video model and leverage pre-trained models to guide generation during inference.", "questions": "Method in Sec 3 is not limit to text-to-video generation and should also be applicable to text-to-image generation. Have you considered other use cases?\n\nOnly two models (a pretrained model and a domain-specific model) are involved in derivation in Sec. 3. If we have more than 1 domain-specific model, can we easily extend this method to multiple domain-specific models?\n\nI know the assumption here is that pretrained models are not accessible during training. But what if we can access the pre-trained models during training, will it outperform LoRA and prefix-finetuning?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Since this framework only requires training a small domain-specific text-to-video model, the domain-specific model can be adapted to any pretrained models as long as their input and output definitions are same.\n\nExperiments clearly demonstrate the effectiveness of the proposed method.", "weaknesses": "The authors assume pretrained model weights are not accessible. Table 1 and figure 9 verifies the authors’ claim. However, it is not clear how does the model perform compared to other methods, such as LoRA and prefix-tuning if pretrained model weights are accessible.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)