Solving New Tasks by Adapting Internet Video Knowledge
Authors: Calvin Luo
Zilai Zeng
Yilun Du
Chen Sun
Venue: ICLR 2025 Poster (2025)
Keywords: Text-Conditioned Generalization
Video Diffusion
Adaptation
Planning
Policy Learning
Primary area: reinforcement learning
TL;DR: We compare techniques for adapting large-scale video generative models to in-domain robotic data, and demonstrate that it facilitates text-conditioned generalization to novel tasks.

Abstract: Video generative models demonstrate great promise in robotics by serving as visual planners or as policy supervisors.  When pretrained on internet-scale data, such video models intimately understand alignment with natural language, and can thus facilitate generalization to novel downstream behavior through text-conditioning.  However, they may not be sensitive to the specificities of the particular environment the agent inhabits.  On the other hand, training video models on in-domain examples of robotic behavior naturally encodes environment-specific intricacies, but the scale of available demonstrations may not be sufficient to support generalization to unseen tasks via natural language specification.  In this work, we investigate different adaptation techniques that integrate in-domain information with large-scale pretrained video models, and explore the extent to which they enable novel text-conditioned generalization for robotic tasks, while also considering their independent data and resource considerations.  We successfully demonstrate across robotic environments that adapting powerful video models with small scales of example data can successfully facilitate generalization to novel behaviors.  In particular, we present a novel adaptation strategy, termed *Inverse Probabilistic Adaptation*, that not only consistently achieves strong generalization performance across robotic tasks and settings, but also exhibits robustness to the quality of adaptation data, successfully solving novel tasks even when only suboptimal in-domain demonstrations are available.

Published as a conference paper at ICLR 2025
SOLVING NEW TASKS BY ADAPTING INTERNET VIDEO KNOWLEDGE
Calvin Luo".! Zilai Zeng" " Yilun Du², Chen Sun'
'rrwnn University, 'Harvard University
ABSTRACT
Video generative models demonstrate great promise robotics by serving as vi- sual planners policy supervisors. When pretrained internet-scale data, such video models intimately understand alignment with natural language, and thus facilitate generalization novel downstream behavior through text- conditioning. However, they may not be sensitive to the specificities the par- ticular environment the agent inhabits. On the other hand, training video mod- els on in-domain examples of robotic behavior naturally encodes environment- specific intricacies, but the scale of available demonstrations may not be sufficient to support generalization to unseen tasks via natural language specification. In this work, we investigate different adaptation techniques that integrate in-domain information with large-scale pretrained video models, and explore the extent to which they enable novel coxtcconttioned generalization for robotic tasks, while also considering their independent data and resource considerations. We success- fully demonstrate across robotic environments that adapting powerful video mod- els with small scales of example data can successfully facilitate generalization to novel behaviors. In particular, we present a novel adaptation strategy, termed In- verse Probabilistic Adaptation, that not only consistently achieves strong general- ization performance across robotic tasks and settings, but also exhibits robustness the quality of adaptation data, successfully solving novel tasks even when only suboptimal in-domain demonstrations are available.
INTRODUCTION
Recent advancements in video generative models have made promising their application to inter- active problems and decision-making. Such video models trained explicitly on in-domain demon- strations have demonstrated accurate encoding of environment-sseccfic visual details and dynamics, and have been popular choices to utilize for robotic learning (Du et al., 2024b; Huang et al., 2023; Yang et al., 2023b; Ko et al., 2024; Liang et al., 2024). When optimized on expert video demonstra- tions, their encoded understanding of expert behavior can be directly used to supervise the learning of high-performing policies (Huang al., 2023; Escontrela et al., 2024), and applied as performant visual planners (Du et al., 2024a) in robotic settings. However, for arbitrary robotic environments, there is usually a severe difference in scale of tractably available expert demonstration data, espe- cially with associated text labelling, in comparison with general internet-scal scale datasets of videos paired with natural language. As result, such in-domain video generative models usually suffer from weaker generalization capability, across novel text specifications and motions of interest.
Instead of training directly on in-domain demonstrations, stronger generalization performance can be obtained by using text-to-video models pretrained on internet-scale data. Having summarized powerful priors over visual styles, natural motion, and alignment with natural language from large- scale data, such models can be leveraged to supervise policies that behave flexibly conditioned on text across multiple environment visual styles without modification (Luo et al., 2024). However, in- teraction is often performed in fixed environment with specific visual characteristics and potentially unique interaction dynamics, which general cetrromment-gnnossi video model may not inher- ently understand or respect. Thus, directly applying large-scale pretrained video generative model
*: Equal contribution. Correspondence to: calvin Iuo @ brown.od. edu and zilai zeng @ wing@rrwwnndu.

---

Published as a conference paper at ICLR 2025
Large-Scele Pretreined Texb-toVidee Model
Large-Scale Pretrained Text-to-Video Model
Large-Scale Pretrained Text-to-Vider Model
deo Midil
Subject Customization
Probabilistic Adaptation
Direct Finetuning
Adaptation Techniques. We explore how in-domain information can be integrated into large-scale text-to-video models through three different adaptation techniques: Subject Customiza- tion, Probabilistic Adaptation, and Direct Finetuning. Subject Customization only modifies the image and text encoder, rather than the motion module, and is lightweight in terms of data require- ments: it only utilizes pairs of static images and text annotated with a special identifier. Probabilistic Adaptation learns small in-domain model from paired video data, which is then used through score composition with large-scale video model that is kept frozen. The small in-domain model can be flexibly parameterized to consider available training resources. Direct Finetuning seeks update the motion module of the large-scale pretrained video model with in in-domain paired video data.
Figure 1:
without modification comes with a potential drawback; they may not understand the intricacies of particular environments of interest to successfully facilitate high-performance behavior within them.
These considerations naturally motivate the investigation of ways to mutually cover the independent deficiencies of each approach. In this work, we perform thorough study on novel task generaliza- tion via adapting internet video knowledge; we seek to illuminate how in-domain information can be best integrated into large-scale pretrained text-to-video models, such that powerful zero-shot text- conditioned generalization capabilities enabled while considering environment-spece -specfff knowl- edge pertaining to visual styles and interaction dynamics. We compare the downstream robotic per- formance of multiple adaptation techniques and contrast their respective requirements in-domain data samples, which range from utilizing only a few still-frames of the agent to text-labelled video demonstrations, and training resources, which span from direct finetuning of the large-scale video model to utilizing it only for inference without any updates. Broadly, we provide this study of adap- tation for facilitating action prediction (Adapt2Act) valuable insight the practitioner interested in balancing performance with resource availability.
We perform standardized evaluations across both robotic manipulation tasks (Yu et al., 2020) and continuous control (Tassa et al., 2018), and demonstrate that adapted video generative models are able to successfully act as accurate video planners for novel rext-conditioned specifications across a variety of robotic tasks, and can also supervise the learning of novel text-conditioned policies. Fur- thermore, in this work we propose a novel adaptation technique termed Inverse Probubilistic Adap- tation, which we highlight achieves consistently strong generalization performance across robotic environments and downstream evaluation approaches. We also discover that even when only subop- timal in domain demonstrations are provided, it can still effectively leverage web-scale priors and text conditioning to generate coherent vidco plans and successfully solve novel tasks. Visualizations and code are provided at aiffusions supervpeti rvpieeiii i on g thub. 1o/ 2actt
RELATED WORK
Adaptation Techniques for Diffusion Models. Although many large-scale pretrained text-to-video
models (Ho et al., 2022; Guo al., 2023; Ramesh al., 2022; Xing et al., 2023; Villegas et al., 2022: Singer et al., 2022; Khachatryan et al., 2023) have demonstrated strong capabilities of synthesizing high-quality videos following the given prompts, it is often desirable to perform adaptation for specialized tasks, such customizing video generation to specific subjects or styles.
DreamBooth (Ruiz et al., 2023) finetunes text-to-image diffusion models to connect a unique identi- fier to subject of interest. using few images of that specific subject. The subject will be implanted into the output space of the diffusion model after finetuning, enabling novel view synthesis with the subject via prompting with its corresponding identifier. In Video (Wei al., 2024), this
2

---

Published as a conference paper at ICLR 2025
"I "[tezt prompt)"
" (tezt prompt)"
Adapted
Adapted Video Generative
Video Generative
Model
Model
(1:+27)
((t--2)
((2+22)
Visual Planning
Policy Supervision
Figure 2: Downstream Task Evaluation. We identify downstream robotic task performance as a way to achieve standardized, quantitative comparisons across adaptation techniques. We evaluate how adapted video models can enable text-conditioned generalization via two approaches: visual planning and policy supervision. For visual planning, the adapted video model synthesizes a text- conditioned video plan into the future, which then converted into actions to follow through a separately trained inverse dynamics model. In policy supervision, the adapted video model is used discriminative manner to evaluate frames achieved by the policy; these are converted into text- conditioned rewards, which the policy is optimized to maximize.
idea is extended to facilitate novel video generation with respect particular subject of interest. learns subject customization for a pretrained video diffusion model through a few provided static images, which is achieved by combining textual inversion with finetuning identity adapter.
Prior work on turge-to-small adaptation of video models, through composing predicted scores, has demonstrated successful transfer of artistic styles while maintaining powerful text-conditionin be- havior (Yang et al., 2023a). In this work, we evaluate this approach to explore the degree to which in-domain environment dynamics and notions of expert behaviors similarly generalize through adap- tation with large-scale pretrained video models, conditioned flexibly on natural language. Further- more, we propose and evaluate novel probabilistic adaptation technique, which performs score composition in an inverted manner from that presented in (Yang et al., 2023a).
Prior adaptation works mostly seek improvements over visual quality and its related metries, such as FID (Heusel et al., 2017) and FVD (Thomas et al., 2018). Here, we focus on new application domain to evaluate adaptation: robotic task performance. We study how the cext-conditioning capa- bilities of large-scale pretrained video generative models can be combined with environment-specift information to deliver further improvements on text-conditioned task generalization.
Video Models for Decision Making, A large body of recent work has explored how video models may be used for decision making (Yang al., 2024; McCarthy et al., 2024). One line of work ex- plores how video generative models can provide rewards, particularly through pixel interface (Ser- manet et al., 2016; Ma et al., 2022). In VIPER (Escontrela et al., 2024), video model trained on expert demonstrations that solve the task of interest; it is then utilized to provide dense rewards to supervise downstream policies by evaluating the likelihood of achieved frames during interac- tion. Similarly, expert demonstrations are also used in Diffusion-Rewann (Huang et al., 2023), but a diffusion model is trained instead. Rewards are once again provided through achieved frames, but through novel cross-entropy computation. In Video-TADPoLe (Luo et al., 2024), a large- scale pretrained video diffusion model is used to provide text-conditioned rewards through achieved unvironment-renders frames. In this work, we also seek to use video generative models as super- visors for policy learning, but we treat it as a method to evaluate the efficacy of different techniques for adapting large-scale pretrained video models to in-domain data.
A separate line of work utilizes video models as mixel-based planners (Ko et al., 2024; Du et al., 2024a;b; Ajay et al., 2023; Wen et al., 2023; Liang et al., 2024; Yang al., 2023b; Zhou et al., 2024b; Wang et al., 2024a; Zhou et al., 2024a). In such works, the video model can be directly used to generate a visual plan to solve task, which can be converted into actions using an inverse dynamics model (Du et al., 2024a) or through dense 3D correspondences (Ko et al., 2024). Alternatively, the video model can also be used as a visual dynamics model as part of a more complex planning routine (Ajay al., 2023; Du et al., 2024b), to form more complex, long horizon video plans. We utilize vidco models as visual planners for robotic tasks to understand the quality of different adaptation techniques in integrating in-domain data into large-scale pretrained text-to-video models.
3

---

Published as a conference paper at ICLR 2025
3 METHOD
Video models pretrained on internet-scale data exhibit strong zero-shot generalization capabilities across diverse visual scenarios, which make them attractive to leverage for downstream robotic tasks. However, the general nature of their pretraining may not inherently enable them to understand domain-specific nuances of the environment within which we would like to learn robotic behavior. We investigate how this can be addressed by integrating in-domain information into large-scale text-to-video models; in Section 3.1 we describe three adaptation approaches of interest, each with separate requirements on in-domain training data and optimization cost. Then, in Section 3.2, we describe two techniques through which we can evaluate the novel task generalization capabilities of these adapted video models in standardized manner.
3.1 ADAPTATION TECHNIQUES
We aim to adapt pretrained large video models to in-domain data, while maintaining and general- izing their internet-scale knowledge to solving downstream robotic tasks. We utilize an Animatel-- iff (Guo et al., 2023) checkpoint as our large video model, which is designed to effectively animate pretrained text-to-image diffusion models such as StableDiffusion (Rombach et al., 2022). At its core, AnimateDiff features motion module pretrained large-scale video dataset, providing powerful motion priors to guide video generation. We investigate three different techniques for in-domain adaptation: direct finetuning, subject customization, and probabilistic adaptation.
3.1.1 DIRECT FINETUNING
Directly finetuning a pnnerally-pettrained text-to-video model is one of the most straightforward ways to mitigate potential domain gaps. Given a video To sampled from in-domain data distribution p(To), a randomly sampled Gaussian noise c ~ (0, I) and a schedule of noise levels B, that are indexed by timestep E [0, 7 text-conditioned video diffusion model can be trained or finetuned as denoising function E-(-) by optimizing the denoising objective (Ho al., 2020) below:
Ldenoise (0) = selle - es (T é text)]l
in which 5; is a noise corrupted video obtained by perturbing To with sampled Gaussian noise e and noise level 1, and training is done with text-conditioning dropout to implement classifier-fres guidance (Ho & Salimans, 2022). AnimateDifff is implemented as a trained motion module built on top of pretrained StableDiffusion components; in study we keep these reused parts unchanged and only adjust the motion module. Direct finetuning involves additional training with labelled pairs of video demonstrations collected from the domain of interest, allowing the model to update the parameters of its motion module to arbitrary extents and shift its output space towards the target domain. However, direct finetuning of large video models may cause issues such as model collapse or catastrophic forgetting to emerge, especially when demonstration samples are limited.
3.1.2 SUBJECT CUSTOMIZATION
Performing direct finetuning on pretrained large video models can often be a computationally ex- pensive endeavour. Furthermore, requires labelled in-domain video demonstrations, the ready availability of which cannot always be assumed in adaptation scenarios of interest for downstream robotic tasks. We therefore investigate cheaper alternatives for adaptation with respect to data and optimization cost. Customized generation (Ruiz et al., 2023; Gal et al., 2023; Wei et al., 2024) has been widely used for synthesizing subjects and scenes that accommodate user preferences, utilizing only few static images of a subject. In this work, we also explore how this technique can be used to inject subject-centric information into pretrained video models, potentially enabling them to better supervise robotic task behavior. We use DreamBooth (Ruiz al., 2023) to customize the generation process due to its simplicity and data efficiency. This method binds a unique text identifier to a specific subject, examples of which provided using still images paired with text captions without motion information, enabling novel view synthesis of the subject contextualized in different scenes.
Following DreamBooth (Ruiz al., 2023), we design special prompts by using rare token (e.g. "[D]") as the unique identifier for each in-domain subject (e.g. "a photo of (D) robot arm"). We fine- tune SiffluDi with DreamBooth on static images of the subject paired with this special text
4

---

Published as a conference paper at ICLR 2025
prompt. We then instantiate AnimateDiff with the CeeamBooth-fintutree U-Net and text encoder for subject-infomin video generation. Unlike direct finetuning which requires labelled video data, this approach performs few-shot customization just through the use of static images; the adaptation is possible without requiring expert video demonstrations, when only still observations of the envi- ronment and its subject are available. Since this adaptation technique will not expose any subject motions to the video model, it also allows us to study whether the pretrained video model can di- rectly transfer its motion prior, which is obtained from domain-agnostic pre-training, onto arbitrary in-domain subjects of interest and facilitate generalization over downstream robotic tasks.
3.1.3 PROBABILISTIC ADAPTATION
Under some circumstances, the large-scale pretrained model is available for inference, but adjusting it in any way may be feasible desirable. In such scenarios, consider Probabilistic Adapta- tion (Yang et al., 2023a), where small sample of demonstrations is utilized to train an in-domain video model, which can be flexibly parameterized to accommodate available modeling resource constraints. Adaptation is then performed by combining the predicted scores from the pretrained, frozen large-scale video model (pretrained Tt, t text) with those of the lightweight domain-spectie -pecifc video model co (Te, È text) during inference. We use low-temperature sampling (Yang et al., 2023a) to compute the adapted score following the denoising function below:
ē= ( 1) +a (T(( (
text) Ereetrained (Te, text) cT(TT;
where 7 is the prior strength, and is the guidance scale of text-conditioning. This method only requires the training of a small component with limited in-domain data, and allows the pretrained large video model to serve probabilistic prior which guides the generation process of the domain- specific model through score composition.
Moreover, extend Equation 2 to its inverse version, in which the adaptation direction between co(T,, è text) and Cpretrained (Te, è text) is inverted:
Einv Ppretrained (Tt, 1) + a Epretmined / text) + (To / text) - *pretrained (T,, (T+, 0))
In inverse probabilistic adaptation, the pretrained video model controls the generation process, while consulting the small model for domain-speciin information. Both probabilistic adaptation formula- tions allow more flexible and low-cost adaptation the video space compared to direct finetuning: empirically, find that one direction may work better than the other in certain circumstances.
3.2 EVALUATING TASK GENERALIZATION CAPABILITIES OF VIDEO MODELS
To measure the quality of adaptation, samples from the adapted video models can be judged with respect in-domain examples in terms of Fréchet Video Distance (FVD) scores (Yang et al., 2023a; Thomas et al., 2018). However, beyond simply assessing surface-level visual style, we propose further evaluating adaptation quality via their ability facilitate downstream robotic performance. For tasks with predefined evaluation schemes, this provides a quantifiable metric in terms of achieved performance and success, and can deliver additional insights into the capabilities of video models beyond appealing visual content generation. In this work consider two approaches, depicted in Figure 2, for applying video models to decision making - visual planning and policy supervision. Under both scenarios, we measure what degree downstream robotic performance and text- conditioned generalization may be enabled via different adaptation techniques.
3.2.1 VIDEO MODELS AS VISUAL PLANNERS
One intuitive approach to use video generative models for decision-making is to use synthesized videos as visual plan of the future, which then can be converted into executable actions. Concretely, a planning model learns p(s' 8), for state 8 and subsequent state s': in the case of visual planning, takes the form of RGB frames and p(s' 8) is video generative model. Synthesized video plans are then translated into actions to execute in the environment through a separately learned inverse dynamics model p(a s. s'). Applying text-guided video generation for task planning has been successfully applied in prior work (Du et al., 2024a;b; Ajay et al., 2023). However, as the performance of video planning can be highly dependent both the visual quality of the imagined
5

---

Published as a conference paper at ICLR 2025
plan and the robustness of the inverse dynamics model, prior work has only utilized video models trained on in-domain demonstrations.
In this work we identify and exploit the fact that the planning model p(s' s) does not require any action information in its training, and is therefore a promising component through which information from large-scale data and pretraining can be introduced. Through the lens of visual planning, explicitly examine integrating large-scale video information into the planner p(s' s) by directly leveraging video generative model pretrained on internet-scale data.
A set of action-labeled trajectories is still required to train an inverse dynamics model; in this work we also utilize small set of such demonstration data temper the large-scale pretrained video model to generate coherent in-domain visual plans while preserving priors summarized from large-scale pretraining. We therefore investigate whether environment-specifee dynamics through cheap adaptation to in-domain data while leveraging pre- served conditinit generalization capabilities facilitate performant in-domain visual plan- ning even for novel tasks of interest unseen during adaptation.
models can capture
3.2.2 VIDEO MODELS AS POLICY SUPERVISORS
In addition to their use as planners, video generative models can be used as policy supervisors. In this approach, the video model is utilized evaluate frames achieved by the agent during interaction in discriminative manner; these signals can then be converted into rewards with which to optimize the policy. While many prior works require video models that are trained solely on anvironment-specinf demonstrations of expert quality (Huang et al., 2023; Escontrela et al., 2023), with rewards computed against summarized notion of "expertness". here we seek to extract accurate text-conditioned rewards from text-to-video models. Video-TADPole (Luo et al., 2024) measures text-alignment of robotic trajectories by noise-corrupting achieved pixel observations, and evaluating how likely a large-scale pretrained text-to-video model would reconstruct the video interactions conditioned on the provided natural language prompt (additional details provided in Appendix C). We train a text-conditioned policy by maximizing cumulative learning. Successful optimization of such a policy enables us to evaluate the ability of adapted large- scale text-to-video models in facilitating novel task generalization, specified by natural language.
rewards through reinforcement
After adapting to limited data samples, synthesizing coherent high-quality in-domain videos can still be challenging for video models. This may pose issues in visual planning, where the generated plan must appear sufficiently in-domain for an inverse dynamics model to be able to accurately translate into meaningful actions. On the contrary, video models do not necessarily need the ability create high-quality in-domain videos from scratch to behave as effective policy supervisors; they simply need to be able to critique the quality of achieved in-domain frames. Thus, expressing adapted knowledge through rewards may allow the detachment of downstream policy performance from de- mands on video generation quality. Conversely, potential drawback of this approach in comparison to visual planning is the high variance commonly observed in the policy learning process.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP AND EVALUATION
We evaluate to what degree adapted video models can facilitate downstream robotic behavior generalization across a variety of environments and tasks, spanning robotic manipulation to continuous control. We focus the bulk of our explorations on MetaWorld-v2 (Yu et al., 2020), which offers a suite of robotic manipulation tasks with different levels of complexity. This benchmark allows us to thoroughly assess the generalization capabilities of adaptation methods across wide selection of tasks. To study the effectiveness of adaptation techniques in a low data regime, we curate a small dataset of in-domain examples from 7 MetaWorld tasks (denoted with an asterisk in Table A1) to adapt pretrained video models. For each task, we utilize 25 expert videos for direct finetuning and probabilistic adaptation, while sampling small set of non-consecutive observations for subject customization. During inference, we evaluate the adapted video models on tasks, 7 of which are novel tasks that are not exposed during adaptation (denoted with asterisk in Table A1).
Benchmarks:

---

Published as a conference paper at ICLR 2025
Egisode Return
AnimuteDifff
In-Domain-Only
Finetuning
Subject Customization
Prob. Adogtation
Adiptation
Humanoid Walking
145.8 1 48.2
24 4.400 0.3
111.5 A 106,4
176,7 +: 42.7
188 0.2
92.6 4: 51,1
Dog Walking
60.2 8.8
76.2 + 29.5
44,6 44.3
117.9 49.7
0.9
88.7 + 9.0
Overall
103
39.3
78.1
146.3
6.5
90.7
Table I: Policy Supervision Continuous Control. We report ground-truth episode return achieved by policies optimized using the listed adapted video models, aggregated over seeds. We observe that direct finetuning produces marginal improvement over vanilla AnimateDiff model, and surprisingly, despite adaptation on just static images, subject customization is able to substan- tially improve continuous locomotion performance over the base pretrained video model.
Figure 3: Novel Text-Conditioned Generalization. the top row, we visualize free-form video generation from directly finetuned AnimateDiff model for the novel text prompt "a dog jumping". This was a behavior unseen during adaptation. When using this adapted video model for policy supervision, we showcase that it can successfully supervise a downstream Dog agent to behave according to novel text specifications in a zero-shot manner (policy rollout shown bottom row).
Additionally, we extend our evaluation to Humanoid and Dog environments from the DeepMind Control Suite (Tassa et al., 2018). We select "Dog walking" and "Humanoid walking", which offer quantitative evaluation through ground-truth rewards, as tasks where we collect 20 demonstrations for adaptation. Following Video-TADPoLe (Luo al., 2024), we evaluate the adapted models on walking as well as other behavior achievement tasks specified by novel text prompts. We provide detailed list text prompts used for both adaptation and task evaluation in Table Al.
Implementation details of adaptation: In our experiments, we use AnimateDiff (Guo et al., 2023) (~1.5B parameters) as our pretrained text-to-video model, which combines StableDiffusson with a motion module pretrained on WebVid-10M (Bain et al., 202 for high-quality video generation. To perform direct finetuning on AnimateDiff, we follow the training pipeline provided by the authors and only update the parameters of its motion module with small in-domain dataset. For subject customization, we utilize 20 static images to finetune StableDiffusion with DreamBoott for each adopt DreamBooth LoRA (Hu et al., 2022) for lower memory usage
environment. In addition, and better training efficiency. In probabilistic adaptation, we implement our small in-domain video model based AVDC (Ko al., 2024), text-to-video model that diffuses over pixel space; im- plemented using ~ 109M parameters, this is comparable in size to that of the small models used in prior work (Yang et al., 2023a). To enable direct score composition between the in-domain model and AnimateDifff, we modify the AVDC model to diffuse over the same latent space used by Sta- bleDiffusion. We include detailed hyperparameters for in-domain model training Appendix
Evaluation metrics:
For robotic manipulation tasks in World we report the "success rate", computed as the proportion evaluation rollouts which the agent successfully completes the given task. For Dog and Humanoid evaluation, we follow the setup in Video Tideo- (Luo et al., 2024) to report quantitative performance for walking tasks, which have ground- -ruth reward func- tions, while providing qualitative results for novel behavior achievement where ground- reward functions are unavailable. As in prior work (Yang et al., 2023a), we also use FVD (Thomas et al., 2018) to measure the visual quality of videos generated by adapted video models.
4.2 POLICY SUPERVISION
We implement video models as policy supervisors following the setup in Video-TADPole TADPoLe (Luo et al., 2024). We reuse the same TDMPC (Hansen et al., 2022) backbone for policy optimization, along with the same hyperparameter settings and training steps, which tabulated in Section D.

---

Published as a conference paper at ICLR 2025
Success Rate (5) w/
Door Close*
Door Open
Window Close
Window Open
Drawer Close
In-Domain-
100.0 + 0.0
31.1 44.0
0.0 0.0
33.3 + 47.1
74.4 36.2 98.9 +
Vanilla AnimateDiff
100.0 + 0.0
0.0 0.0
33.3 47.1
31.1 : 44.0
Direct Finetuning
100.0 1 0.0
0.0 0.0
0.0 1 0.0
47.8 1 41.4
95.6 1 7.7
Subject Customization
100.0 = 0.0
0.0 0.0
100.0 + 0.0
60.0 + 42.5
100.0 0.0
Prob. Adaptation
95.6 + 7.7
30.0 = 52.0
33.3 = 57.7
100.0 + 0.0
100.0 0.0
Inverse Prob. Adaptation
97.8 1 3.8
65.6 56.8
98.9 1.9
98.9 1.9
100.0 0.0
Success Rate (%) w/
Drawer Open
Coffee Push"
Soccer
Button Press
Overall
Hi-Domain-Only
0.0 + 0.0
0.0 + 0.0
0.0 0.0
33.3 = 47.1
30.2
Vanilla AnimateDiff
33.3 + 47.1
28.9 + 23.2
0.0 0.0
33,3 = 47.1
39.8
Direct Finetuning
0.0 + 0.0
30.0 = 26.0
5.6 = 9.6
0.0 = 0.0
31.0
Subject Customization
0.0 1 0.0
15.6 1 22.0
20.0 1 17.8
0,0 1 0.0
44.0
Prob. Adaptation
0.0 + 0.0
1.1 + 1.9
2.2 + 3.8
0.0 + 0.0
40.2
Inverse Prob. Adaptation
66.7 + 57.7
13.3 + 23.0
6.7 = 6.7
66.7 = 57.7
68.3
Policy Supervision on MetaWorke We report the mean success rate across 9 manipulation tasks in MetaWorld, over 3 seeds. "x" denotes seen tasks during adaptation. We observe that inverse probabilistic adaptation achieves the highest overall performance, both in averaged success rate over the entire task suite, as well as successful generalization to the highest number of novel tasks. Subject customization also achieves surprisingly high aggregate success given its cheap data cost.
Table 2:
Continuous Control: In Table 1, we report walking performance as evaluated by the ground-truth reward function across all adaptation techniques, in comparison to Video-TADPoLe using default AnimateDiff. In terms of length of 4 for both Humanoid and Dog experiments. We also list noise levels used for different adaptation techniques in Table A5. These settings were discovered in an offline manner using policy discrimination, which is described in Appendix E. Utilized text prompts can be found in Table Al.
parameters, we utilize context window size of 8, stride
We first observe that Direct Finetuning shows slight decrease compared to vanilla AnimateD- iff, demonstrating the challenges of capturing complex in-domain dynamics through small-scale finetuning. We also observe that the small in-domain model performs poorly on walking tasks. Consequently, both probabilistic adaptation and its inverse further show degradation on average per- formance when integrating vanilla AnimateDiff with the in-domain model. However, Inverse Proba- bilistic Adaptation not only surpasses the in-domain model on Humanoid Walking by a large margin, but also achieves stronger Dog Walking performance than both pretrained and in-domain models. We conjecture that while our small in-domain model has difficulty modeling Humanoid and Dog motions, increasing in-domain model capacity and applying proper adaptation with pretrained mod- els yields improvements through Video-TADPole performs better on default walking behavior across both environments, with significant improvement in the case of Dog. This is striking result, as Subject Customization only utilizes static images of the scene for adaptation and reuses the default motion module pretrained from AnimateDiff.
We further discover that Subject Customization
However, for novel contiiondt generalization to new poses, we find that direct finetuning per- forms the best. For a novel text prompt and motion, such as "a dog jumping". a directly finetuned video model is able to supervise the learning of an associated policy. We provide visual of the achieved policy rollout in Figure 3. These results indicate that for continuous locomotion settings, direct finetuning may be the best balance in terms of preserving performance but also enabling in- teresting text-conditioned generalization; subject customization, on the other hand, is low-cost yet performant approach consider. Finally, Inverse Probabilistic Adaptation enables competitive continuous control performance without finetuning the large pretrained model.
Robotic Manipulation: In Table 2, we report the average success rate on Meta World tasks across adaptation techniques, using a standardized context window of 8, stride length of 4, and noise level of 700. We discover that inverse probabilistic adaptation has the best performance. It is able to achieve non-zero success rate over all 9 tasks, with the highest average success rate of 68.3%. By default, utilizing vanilla AnimateDiff through Video-TADPoLe able to achieve decent per- formance, highlighting the default text-conditioned generalization capabilities of large pretrained models. We also believe that integrating it with an in-domain model that supervises the motion of the particular dynamics of the environment, enables better generalization more challenging tasks (e.g. Door Open). However, the default probabilistic adaptation formulation may heavily rely on the in-domain text-conditioned score, which may be inaccurate when handling novel task prompts
8

---

Published as a conference paper at ICLR 2025
Success Rate (%) w/
Door Close"
Door Open
Window Close
Window Open
Drawer Close
In- In-Domain-Onff
93.3 = 14,9
1:0::00 0.0
53.3 = 29.8
6.7 + 14.9
20.0 + 29.8
Vanilla AnimateDiff
100.0 + 0.0
0.0
13.3 18.3
40.0 27.9
46.7 : 29.8
Direct Finetuning
100.0 1: 0.0
0.0 0.0
0.0 = 0.0
6.7 1: 14.9
80.0 1 29.8
Subject Customization
100.0 0.0
0.0:0000
0.0 = 0.0
20.0 + 29.8
25.0 + 31,9
Proh Adaptation
100.0 0.0
0.0
73.3 27.9
13.3 = 18.3
40.0 43.5
Inverse Prob. Adaptation
100.0 + 0.0
0.0 0.0
53.3 = 18.3
0.0 = 0.0
53.3 + 38.0
Success Rate (%) w/
Drawer Open
Coffee Push
Succer
Button Press
Overall
In Domain Only
0.0 0.0
0.0 0.0
0.0 + 0.0
40.0 14,9
23.7
Vanilla AnimateDiff
0.0 0.0
0.0 + 0.0
0.0 0.0
0.0 = 0.0
22.2
Direct Finetuning
0.0 0.0
13.3 18.2
6.7 14.9
0.0
23.0
Subject Customization
0.0 0.0
13.3 + 29.8
0.0
0.0
17.6
Prob. Adaptation
6.7 = 14.9
6.7 + 14,9
0.0
33.3 = 23.6
30.4
Inverse Prub. Adaptation
0.0
0.0 0.0
0.0 0.0
26.7 27,9
25.9
Table 3: Visual Planning on MetaWorld. We report the mean success rate via visual planning across 9 tasks, aggregated over 5 seeds each. We discover that both probabilistic adaptation and its inverse are able to act as performant visual planners, and substantially improve over vanilla Ani- mateDiff; notably, probabilistic adaptation achieves success more unseen tasks than alternatives.
due to its small training scale. We therefore hypothesize that this explains why inverse probabilistic adaptation is more performant; it may be more robust to novel text-conditioning, as more weight put on leveraging textual priors from the pretrained model.
4.3 VISUAL PLANNING
We implement video models as visual planners following the framework in UniPi (Du et al., 2024a). To generate plan, synthesize a sequence of 8 future frames conditioned on both the current visual observation from the environment and the text prompt specifying the task. This is then trans- lated into executable action sequence via inverse dynamics model. To mitigate the potential error accumulation problem, we evaluate our visual planner in a closed-loop manner, in which we only execute the first inferred action for every environment step. We provide detailed hyperparame- ters for video planning, and the implementation of the inverse dynamics model, in Appendix D.
We evaluate visual planners with different adaptation techniques across selected tasks in MetaWorld, of which 7 are unseen during adaptation, and report the success rates in Table Among all evaluated adaptation techniques, we observe that probabilistic adaptation and its inverse version achieve the highest overall performance, with probabilistic adaptation achieving the highest number of non-zero performance on novel tasks. We find that using only static images for adaptation through subject customization fails to produce useful in-domain visual plans. Meanwhile, direct finetuning only brings marginal improvement over Vanilla AnimateDiff and performs on par with the small in-domain model, struggling to generalize well on unseen tasks due to the small scale of demonstration data being considered. For such settings with cheap amounts of in-domain data, discover that probabilistic adaptation and its inverse appear comparatively more promising.
Robotic Manipulation:
Additional Metrics:
Visual planning relies on synthesizing coherent high-quality video plans that can be accurately interpreted by the inverse dynamics model. Therefore, we may be interested in measuring the quality of visual plans created by adapted video models in a free-form manner, in other words, generated from pure noise. In Table 4, we report FVD scores of free-form generation for both seen and unseen tasks in MetaWorld benchmark. In each setup, we generate 1,000 synthetic videos over 7 robotic manipulation tasks. We observe that probabilistic adaptation and its inverse have strong FVD scores compared to other adaptation techniques. At the same time, high FVD scores achieved by direct finetuning fails to translate to high task success rate through visual plan- ning: this suggests that while its plans are of high visual quality, they may not necessarily model the correct motions for solving tasks in a text-conditioned manner. This further suggests that be- yond just gauging visual metrics alone, using downstream robotic tasks can provide meaningful and additional insights to evaluate adaptation performance of video models.
Studying Data Quality: In probabilistic adaptation, our in-domain model is trained solely on lim- ited expert demonstrations, which can still be prohibitively expensive to collect in some scenarios. By combining with pretrained video models through adaptation, provide further investigation on whether the knowledge (c.g. motion priors) obtained from large-scale pretraining can bridge the gap between the suboptimality of in-domain data and task evaluation performance.

---

Published as a conference paper at ICLR 2025
FVD Scures (MelaWorld))
Vanilla AnimateDift
Domaoinn Onlyy Direct Finetuning Subject Custonnization
Adaptation Inverse Adaptation
Seen
4625
2987.2
946-D
2212.8
848.3
92866
Unseen
4409.7
3080.7
915.0
2316.0
1237.6
1250.3
Table 4: FVD Scores with Free-form Generation. We report FVD scores for videos of MetaWorld tasks, produced by Free-form Generation via the video generative models of interest. This is com- puted for both seen and unseen task sets, each with 7 tasks, aggregated over 1000 synthetic videos.
Success Rate (%) w/
Door Close"
Door Open
Window Close
Window Open
Drawer Close
In-Domain-Only
93.3 + 14.9
0.0 + 0.0
40.0 + 27.9
0.0 + 0.0
33.3 + 23.6
Prob. Adaptation Inverse Prob. Adaptation
100.0 1 0.0
0.0 + 0.0
60.0 : 36.5
13.3 + 18.3
46.7 1: 29.8
93.3 + 14.9
0.0 = 0.0
53.3 = 29.8
0.0 = 0.0
93.3 + 14.9
Success Rate (%) w/
Drawer Open
Coffee Push"
Soccer
Button Press
Overall
In- Ht--Domain-Oni Only
0.0 0.0
0.0 0.0
0.0 0.0
13.3 + 18.3
20.0
Prob, Aduptation
6.7 # 14.9
0.0 = 0.0
0.0 0.0
0.0 = 0.0
25.2
Inverse Prob. Adaptation
6.7 1. 14,9
0.0 0.0
0.0 1 0.0
0.0 1 0.0
27.4
Table 5: Probabilistic Adaptation with Suboptimal Data for Video Planning. We report the mean success rate via visual planning across 9 tasks, aggregated over seeds each. Compared to results in Table 3, we discover that the overall performance of the in-domain model and probabilistic adap- tation is severely impacted by the suboptimality of the training data. In contrast, the performance of inverse probabilistic adaptation remains robust under the suboptimal setup.
In Table 5, perform planning with probabilistic adaptation and its inverse, where the available adaptation data is produced by a suboptimal agent. The suboptimal agent takes an expert action only 30% of the time, and random action 70% of the time. In consistency with the previous setup, we collect 25 (now suboptimal) demonstrations from the same 7 tasks denoted with asterisks in Table AI. We observe that the performance of probabilistic adaptation decreases when integrating with a suboptimal in-domain model. Surprisingly, the overall average task success rate remains robust for inverse probabilistic adaptation. Overall, both adaptation techniques outperform the in- domain only baseline. This is promising sign that in adapting large-scale text-to-video models for robotic downstream tasks, expert demonstrations may not be explicitly needed. This potentially opens up opportunities for applying large-scale video models to novel robotic tasks, where only random or suboptimal demonstrations are tractably available.
5 CONCLUSION AND FUTURE WORK
In this work, we investigate how internet video knowledge can be adapted with a small amount of in- domain video demonstrations to achieve novel task generalization for downstream robotics, which we generally acronymize as Adapt2Act. We explore several methods through which internet-scale video models may be adapted to model the appearance and dynamics of novel environments, and subsequently utilized to perform new behaviors or accomplish unseen tasks conditioned on natu- ral language. Our considered adaptation techniques vary their data and resource requirements. We have conducted extensive evaluations on MetaWorld and DeepMind Control Suite tasks under both policy supervision and visual planning setups. In particular, our proposed Inverse Probabilistic Adaptation approach demonstrates strong generalization capabilities across different task settings, and remains robust when only suboptimal demonstrations are available. Our findings highlight the promise of leveraging internet-scale text-to-video priors to model domain-speciff robotic behaviors through data-efficient adaptation, which enables further advancements on text-conditioned general- ization for embodied intelligence.
Limitations and Future Work. When the same adapted model is utilized to solve the tasks un- der different setups, performance differences that are difficult to explain may arise. For example, while inverse probabilistic adaptation outperforms its inverse version significantly under the policy supervision setup, both achieve similar success rates under the video planning setup. Additionally, a natural next step to further study is the use of suboptimal data, which can be collected cheaply by running random policies tasks of interest, or iteratively augmented from on-policy observations.
10

---

Published as a conference paper at ICLR 2025
Acknowledgments. This work is supported by Samsung and NASA. Our research was conducted using computational resources at the Center for Computation and Visualization at Brown University. Chen would like to thank Mia for inspiration.
REPRODUCIBILITY
All adaptation techniques in our work are implemented using available open-sourced components. publicly available checkpoints for pretrained large models,
As mentioned in Section 4.1, such as AnimateDiff (Guo et al., 2023) as well as StableDiffusion (Rombach et al., 2022). We uti- lize DreamBooth (Ruiz al., 2023) for subject customization. Furthermore, reuse the codebase provided by the authors of AVDC (Ko et al., 2024) for in-domain model training, with minimal ad- justments to enable the latent diffusion. For policy learning, we follow Video-TADPol (Luo al., 2024) framework, which itself is built off of publicly available AnimateDiff and TDMPC (Hansen al., 2022). Furthermore, we release detailed hyperparameter settings and implementation details in Appendix D, as well as elaborate on techniques on how they were discovered or selected in Ap- pendix E. We believe that the simplicity of our approach, along with the utilization of open-sourced checkpoints, makes this work highly reproducible. We also commit to open-sourcing our code, to support further reproducibility efforts in the community.
REFERENCES
Anurag Ajay, Seungwook Han, Yilun Du, Shuang Li, Abhi Gupta, Tommi Jaakkola, Josh Tenen- baum, Leslie Kaelbling, Akash Srivastava, and Pulkit Agrawal. Compositional foundation models for hierarchical planning. In Conference on Neural Information Processing Systems (NeurIPS), 2023. 3,5
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: joint video and image encoder for end-to-end retrieval. In IEEE International Conference on Computer Vision (ICCV), 2021.7 7
Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation. Advances Neural Information Processing Systems, 36, 2024a. 3,
Yilun Du, Sherry Yang. Pete Florence, Fei Xia, Ayzaan Wahid, brian ichter, Pierre Sermanet, Tianhe Yu, Pieter Abbeel, Joshua B. Tenenbaum, Leslie Pack Kaelbling, Andy Zeng, and Jonathan Tomp- son. Video language planning. In International Conference on Learning Representations (ICLR), 2024b. 3,5
Kolve Eric, Mottaghi Roozbeh, Han Winson, VanderBilt Eli, Weihs Luca, Herrasti Alvaro, Deitke Matt, Ehsani Kiana, Gordon Daniel, Zhu Yuke, Kembhavi Aniruddha, Gupta Abhinav, and Farhadi Ali. A12-THOR: an interactive 3d environment for visual AI. arXiv, 1712.05474, 2017. 19
Alejandro Escontrela, Ademi Adeniji, Wilson Yan, Ajay Jain, Xue Bin Peng, Ken Goldberg, Young- woon Lee, Danijar Hafner, and Pieter Abbeel. Video prediction models as rewards for reinforce- ment learning. In Conference on Neural Information Processing Systems (NeurIPS), 2023. 6
Alejandro Escontrela, Ademi Adeniji, Wilson Yan, Ajay Jain, Xue Bin Peng, Ken Goldberg, Young- woon Lee, Danijar Hafner, and Pieter Abbeel. Video prediction models as rewards for reinforce- ment learning. Advances in Neural Information Processing Systems, 36, 2024. 1,3
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-or. An image is worth one word: Personalizing text-to-image generation using textual inversion. In International Conference on Learning Representations (ICLR), 2023.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang. Yu Qiao, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint 0rx2i:2307.044255. 2023. 2, 11
11

---

Published as a conference paper at ICLR 2025
Nicklas Hansen, Xiaolong Wang, and Hao Su. Temporal difference learning for model predictive control. arXiv preprint arXiv:2203.0495: 2022. 7, 11,
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.
Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho and Tim Salimans. arXiv:2207.12599 2022. 4
Classifier-free
diffusion guidance.
arXiv preprint
Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Conference on Neural Information Processing Systems (NeurIPS), 2020.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. Advances Neural Information Processing Systems, 35:8633- 8646, 2022. 2
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: low-rank adaptation of large language models. In International Con- ference Leurning Representations (ICLR), 2022. 7
Tao Huang, Guangqi Jiang, Yanjie Ze, and Huazhe Xu. Diffusion reward: Learning rewards via conditional video diffusion. arXiv preprint 2xX1:2332..4134, 2023. 1,3, 6, 14
Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text-to-image diffusion models are zero-shot video generators. arXiv preprint arXiv:2303.13493 2023. 2
Po-Chen Ko, Jiayuan Mao, Yilun Du, Shao-Hua Sun, and Joshua B. Tenenbaum. Learning to act from actionless videos through dense correspondences. In International Conference on Learning Representations (ICLR), 2024. 3, 7, 11, 19
Junbang Liang, Ruoshi Liu, Ege Ozguroglu, Sruthi Sudhakar, Achal Dave, Pavel Tokmakov, Shuran Song, and Carl Vondrick. Dreamitate: Real-world visuomotor policy learning via video genera- tion. arXiv preprint ar 2406 /6862, 2024. 1,
Calvin Luo, Mandy He, Zilai Zeng, and Chen Sun. Text-aware diffusion for policy learning. In NeurlPS, 2024. 1, 3, 6, 7, 11, 14, 16
Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. Vip: Towards universal visual reward and representation via value-implicit pre-training. arXiv preprint wtt::2220000000000 2022. 3
Arjun Majumdar, Karmesh Yadav, Sergio Arnaud, Jason Ma, Claire Chen, Sncha Silwal, Aryan Jain, Vincent-Pierre Berges, Tingfan Wu, Jay Vakil, Pieter Abbeel, Jitendra Malik, Dhruv Batra, Yixin Lin, Oleksandr Maksymets, Aravind Rajeswaran, and Franziska Meier. Where are we in the search for artificial visual cortex for embodied intelligence? In Conference on Neural Information Processing Systems (NeurIPS), 2023. 17
Robert McCarthy, Daniel CH Tan, Dominik Schmidt, Fernando Acero, Nathan Herr, Yilun Du, Thomas G Thuruthel, and Zhibin Li. Towards generalist robot learning from internet video: A survey. arXiv preprint 2rXiv:2404.1960 2024. 3
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06115, 2022. 2
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022. 4,
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.
Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023. 2, 4, 11
12

---

Published as a conference paper at ICLR 2025
Pierre Sermanet, Kelvin Xu, and Sergey Levine. Unsupervised perceptual rewards for imitation learning. arXiv preprint
2016.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint (tt1:22009.14992 2022.
Jiaming Song, Chenlin Meng. and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations (ICLR), 2021. 16
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmalcki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint 1ef.::1800000000 2018. 7,
Unterthiner Thomas, van Steenkiste Sjoerd, Kurach Karol, Marinier Raphael, Michalski Marcin, and Gelly Sylvain. Towards accurate generative models of video: A metric & challenges. arXiv, 181 1812.01717, 2018. 5,
Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual description. arXiv preprint arXiv:2210002399, 2022. 2
Boyang Wang, Nikhil Sridhar, Chao Feng, Mark Van der Merwe, Adam Fishman, Nima Fazeli, and Jeong Joon Park. This&that: Language-gesturr controlled video generation for robot planning. arXiv preprint arXiv:2407.055300 2024a.
Fu- Yun Wang, Zhaoyang Huang, Weikang Bian, Xiaoyu Shi, Kegiang Sun, Guanglu Song, Yu Liu, and Hongsheng Li. AnimateLCCM: computation-efficient personalized style video generation without personalized video data. arXiv, 2402.00769, 2024b. 18
Yujie Wei, Shiwei Zhang, Zhiwu Qing, Hangjie Yuan, Zhiheng Liu, Yu Liu, Yingya Zhang, Jingren Zhou, and Hongming Shan. Dreamvideo: Composing your dream videos with customized sub- ject and motion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2024. 2,
Chuan Wen, Xingyu Lin, John So, Kai Chen, Qi Dou, Yang Gao, and Pieter Abbeel. Any-point trajectory modeling for policy learning. arXiv preprint arXiv:2401.0002: 2023. 3
Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Xintao Wang, Tien-Tsin Wong, and Ying Shan. Dynamicrafter: Animating open-domain images with video diffusion priors. arXiv preprint UrXiv:2210.112190 2023." 2
Mengjiao Yang, Yilun Du, Bo Dai, Dale Schuurmans, Joshua B Tenenbaum, and Pieter Abbeel. Probabilistic adaptation of text-to-video models. arXiv preprint arXiv:2306.01872, 2023a. 3, 5,
Mengjiao Yang, Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Pieter Abbeel. Learning interactive real-world simulators. arXiv preprint arXiv:2310.06114 2023b. 1,
Sherry Yang, Jacob Walker, Jack Parker-Holder, Yilun Du, Jake Bruce, Andre Barreto, Pieter Abbeel, and Dale Schuurmans. Video as the new language for real-world decision making. arXiv preprint arXiv:2402.171339 2024. 3
Tianhe Yu, Deirdre Quillen, Zhampeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi task and meta reinforcement learning.
In Conference on robot learning, pp. 1094-1100. PMLR, 2020. 2,
Enshen Zhou, Yiran Qin, Zhenfei Yin, Yuzhou Huang, Ruimao Zhang, Lu Sheng, Yu Qiao, and Jing Shao. Minedreamer: Learning to follow instructions via chain-of-imagination for simulated- world control. arXiv preprint arXiv:2403. 12037, 2024a. 3
Siyuan Zhou, Yilun Du, Jiaben Chen, Yandong Li, Dit- Yan Yeung, and Chuang Gan. Robodreamer: Learning compositional world models for robot imagination. arXiv preprint arXiv:2404.12377, 2024b.
13

---

Published as a conference paper at ICLR 2025
A
TEXT PROMPTS
Below listed the text prompts used for adaptation and task evaluation.
Task
In-Domain Prompts
AnimateDiff Prompts
DreamBooth Identifier
Dog Walking
dog/phuraoh hound walking
dog/pharaoh hound walking
[D) dog
Humanoid Walking
a(n) humanoiddaatiin figure walking
a(n) bumannid/actioo figure walking
a [D) action figure
Assembly"
assembly dial turn
robot arm placing a ring over peg
Dial Turn'
mbot tuming dial
Reach"
reach
robot arm reaching red sphere
Pog Unplug Side'
peg unplug
robot arm unplugging a gray peg
Lever Pull"
lever pull
rubot arm pulling a lever
Coffee
coffee push
arm pushing
cup towards coffee machine
Door Close"
door close
rubot closing door
[D]
Door Open
door open
robot arm opening a door
Window Close
window close
robot closing window
Window Open
window open
robot arm opening a window
Drawer Close
drawer close
robot arm closing drawer
Drawer Open
drawer open soccer
rubot arm open a drawer
Soccer
rbot arm pushing soccer ball
Button Press
button press
rubot pushing butlon
Table A1: Task-Promp Pairs. We include comprehensive list of tasks and their text prompts for adaptation and evaluation. "+" denotes tasks seen during adaptation.
Task
In- Do-aimi Prompts
AnimateDiff Prompts
Spatula in Kitchen"
spatula
find the spatula find the toaster find the painting find the blinds find toilet paper find the pillow find the desk lamp find the mirror find laptop
Toaster in Kitchen"
toaster
Painting in Living Room"
painting
Blinds Bedroom"
blinds
ToiletPaper in Bathroom" toilet paper
Pillow Living Room
pillow
DeskLamp in Living Room
desk lamp
Mirror in Bedroom
mirror
Laptop in Bedroom
laptop
Table A2: Task-Prompt Pairs for iTHOR. We include comprehensive list of iTHOR tasks and their text prompts for adaptation and evaluation. "*" denotes tasks seen during adaptation.
B CONTINUED DENOISING
In diffusion-basse based policy supervision, rewards are extracted from the procedure of corrupting frames achieved by the policy with some level of Gaussian noise and then making denoising predictions us- ing the video model (Huang et al., 2023; Luo al., 2024). For additional insight, we propose visualization technique called continued denoising, and report FVD scores for videos generated in such a manner. continued denoising, rather than extracting a scalar from components of the de- noising prediction as in Ticoo-TDDPLee, we treat the noised video as an initialization and iteratively continue sampling produce final clean video prediction tthss, "continuing" the denoising pro- cedure. In our experiments we perform continued denoising conditioned on a desired text prompt, noise level of 700, total frame length of 16, and 10 denoising steps.
As mentioned in Section 3.2.2, policy supervision does not necessarily require strong free-form generation of in-domain videos; rather it evaluates observed frames achieved by following the cur- rent policy. For qualitative purposes, continued denoising provides evaluation of achieved frames is done (examples in Figure AI), as well as a sanity check on the integration of in-domain information through adaptation. Furthermore, it enables quantitative com- parison through FVD scores, which provides an idea on the capability of adapted video models to reconstruct in-domain-lik videos conditioned on text. is intuitive hypothesize that lower FVD score correlates with better in-domain adaptation, as it understands how to accurately complete the provided in-domain frames from a heavy noise corruption.
visual sense of how this
14

---

Published as a conference paper at ICLR 2025
In Table A3, report the FVD scores for the set of seen and unseen tasks that evaluated in free-form generation experiments. We discover that the lowest FVD score for continued denois- ing is achieved by the in-domain model, which is unsuprissing examples. The next-best FVD scores are achieved by probabilistic adaptation and its inverse. This significant because it supports the finding that with adaptation, generalization to unseen tasks is possible, and suggests that accurate domain-specific rewards can be supplied through policy super- vision. Indeed, this aligns with result in Table 2, where Inverse Probabilistic Adaptation achieves the best overall task performance through policy supervision.
was explicitly trained on such
IVD Scores (Mota World)
AnimatelDiff In-Deeainn-Qaly
Direct Fineraninng Subjeett Custoriization
Adaptation
Inverse Adaptitionn
Seen Unseen
270044
602.8
1004.6
1078.9
622.6
627.4
2613.2
610.1
978.5
17118
630.6
681.8
Table A3: FVD Scores with Continued Denoising. We report FVD scores for videos of Meta- World tasks, produced by Continued Denoising via the video generative models of interest. This is computed for both seen and unseen task sets, each with 7 tasks, aggregating results over 1000 synthetic videos.
Figure Al: Continued Demoising. We visualize frames from a task unseen during adaptation, corrupted with a level of Gaussian noise (top row). We then show the result of continued denoising using an inverse probabilistic adaptation model verify it can visually generalize to fill in novel in-domain information. Despite not having seen button, it is able to reconstruct it conditioned on text. This figure is for intuition; in practice, a much higher noise level is used, shown Figure A2.
Figure A2: Continued Denoising (in practice). In practice, an aggressive level of Gaussian cor- ruption usually used on achieved frames for reward computation (700 for MetaWorld), However, because to the human eye this may look virtually indistinguishable from pure noise, we supply an illustrative example Figure A1 using a noise level of 400. Here, we showcase visuals of the same unseen task corrupted with practical noise level of 700. We then show the result of contin- ued denoising to visually verify the model integrates adapted in-domain information successfully. When performing continued denoising from such high corruption, conditioned "a robot arm pushing a button", it is therefore quite surprising the level of detail with which the adapted text-to-video model is able to reconstruct novel in-domain features such the button which it has not even seen during adaptation. The resulting continued denoising video can also be evaluated against in-domain examples via FVD for further insights.
the text prompt
15

---

Published as a conference paper at ICLR 2025
C VIDEO-TADPOLE REWARD COMPUTATION
Video-TADPoLa (Luo et al., 2024) rewards are densely computed for a trajectory achieved by a policy, in terms of their rendered frames. For arbitrary start index and end index j inclusive of the trajectory, for < j. let O[++1: denote the associated sequence of rendered frames. Video- TADPoLe then utilizes a source noise vector éy ~ NN(; 0, 0,1)-++1) of the same dimensionality as Gaussian coruptiio to produce noisy observation 1-j+-- Then, Video- Padoo computes batch of alignment reward terms through one inference step of the texi-to-video diffusion model as:
align [e:]
& (0p+111111 Enoise: v) EΦ( (111+111111 t 112
and batch of reconstruction reward terms
[ Enoise) cull2 2 ( Enoie: y) -oll
For provided context window of size Video-TADPol calculates the reward each timestep / utilizing each context window that involves achieved observation 0c+1:
align symlog (w] II
11) + symlog (w2 r Itt--+ 11)
ne
stride term can be used to make this computation tractable across long trajectories, where the context window skips timesteps before computing sequence of Video-TADPoLe rewards again. The context window n, stride 8, and noise level meise are hyperparameters to be set by the user; in practice, good settings for such hyperparameters can be found in offline manner through policy discrimination (Section E).
D IMPLEMENTATION DETAILS
We include the default hyperparameters from the TD-MPC implementation Table A9 for com- pleteness. We do not modify the default recommended settings for both Humanoid and Dog envi- ronments, as well as the Meta-Worll World experiments.
Hyperparameter
Value
Training Objective
pred_noise
Number of Training Steps
60000
Loss Type
L2
Learning Rate
le-4
Beta Schedule
Linear schedule (0.0085, 0.012)
Timesteps
1000
EMA Decay
0.99
EMA Update Steps
10
Table A4: Hyperparameters for In-Domain Model Training.
Visual Planning Hyperparameters: generate video plan with adapted video models, we per- form DDIM (Song al., 2021) sampling for 25 steps. We use 7.5 as the text-conditioning guidance scale for directly finetuned AnimateDiff, and use 2.5 for other adaptation techniques. Additionally, use 0.1 as the prior strength for probabilistic adaptaion and 0.5 for its inverse version.
Noise Level
Humanoid Walking
Dog Walking
In-Domain Only
600
600
Direct Finetuning
700
700
Subject Customization
500
600
Prob. Adaptation
700
700
Inverse Prob. Adaptation
600
500
Table A5: VideoTADPoLe Noise Levels for Deep Control.
16

---

Published as a conference paper at ICLR 2025
Inverse Dynamics: We employ small MLP network our inverse dynamics model. The model takes in the embeddings two consecutive video frames, which are extracted using VC-1 (Majum- dar et al., 2023), and predicts the action that enables the transition between the provided frames. We train the inverse dynamics model on a dataset comprising a mixture of expert and suboptimal trajectories rendered from the environment, using the same set of tasks and data volumn used for adaptation. For faimess, we reuse the same dynamics model across all adaptation techniques during evaluation. We provide the detailed hyperparameters inverse dynamics training Table A6.
Hyperparameter
Value
Input Dimension
1536
Output Dimension
4
Training Epochs
20
Learning Rate
3c-5
Optimizer
AdamW
Table A6: Hyperparamters Inverse Dynamics Model Training
Hyperparameter
Value
Component
# Parameters (Millions)
Discount factor
0.99
Seed steps
5, 000
VAE (Encoder)
34.16
Replay huffer
Unlimited
VAE (Decoder)
49.49
Sampling technique
PER (a 0.6,3 0.4)
Planning borizon (H)
U-Net
865.91
Initial parumeters (a", (0))
(0.2)
Text Encoder
340.39
Population size
512
Elite fraction Iterations
12 (Humanoid)
Table A7: StableDiffusion Components. For completeness, list sizes of the components of the StableDiffusion v2.1 checkpoint used ViDeo-TADPoLe experiments. The checkpoint used purely for inference, and is not modified updated in any way. Note that the VAE De- coder is not utilized in our framework.
(Dog)
Policy fraction Number of particles
5%
Momentum coefficient Temperature (r) MLP hidden size
0.5
MLP activation
ELU
Latent dimension
100 (Humanoid, Dog)
Learning rale
3e-4 (Dog)
1c-3 (Humanoid)
Optimizer
Adam (9: 0.9, 0.099)
Temporal coefticient (A)
0.5
Reward coefficient (o,)
Value loss coefficient (cy)
0.1
Consistency loss coefficient (cy)
Component
# Parameters (Millions)
Exploration schodule
0.05 (25k steps)
Planning borizon schedule
1 1-+ 5 (25k steps)
Batch
(Dog)
VAE (Encoder)
34.16
512 (Humanoid)
VAE (Decoder)
49.49
Momentum coefficient (C)
0.99
U-Net
1312.73
Steps per gradient update
6 update frequency
2
Text Encoder
123.06
Table A8: AnimateDiff' Components. For completeness, we list sizes of the components of the AnimateDiff checkpoint used in Video- TADPoLe experiments. The checkpoint is used purely inference, and not modified or up- dated in any way. Note that the VAE Decoder is not utilized in our framework.
Table A9; TD-MPC hyperparameters. We use the official implementation TD- MPC (Hansen al., 2022) with adjustments to the hyperparameters, but list it below for completeness. We set the number of training steps to 2 million for continuous control ex- periments using TD-MPC, and 700k steps for MetaWorld experiments.
E POLICY DISCRIMINATION
Rather than performing expensive sweep over Video- TADPoLe hyperparameters directly by launching policy supervision experiments across each adapted video model technique, which can be expensive, look for offiin method to determine reasonable hyperparameter settings. For
17

---

Published as a conference paper at ICLR 2025
each environment, we therefore utilize example expert quality demonstration video well as example poor quality demonstration video (with arbitrary quality levels in-between, if available). Then, we can perform search over TADooe parameters by computing Video-TADPoL rewards for these trajectories using an adapted video model, conditioned on the task-relevant text prompt, with respect to different context window, stride, and noise level settings. We seek parameter settings that, through the adapted video model's Video-TA TADPoLe reward computation, can correctly distinguish between the expert, text-aligned video demonstration from the poor, text-unaligned video demonstration; this can be done by comparing the predicted Video-TAVe TADPoLe rewards. Once iden- tified in this offline manner, we can subsequently use the discovered settings of context window, stride, and noise level for learning text-conditioned policies. In practice, we have found that these settings can be reused for novel text-conditioning within the same environment without issue.
F POLICY SUPERVISION WITH ADDITIONAL PRETRAINED VIDEO MODELS
Success Rate (%) w/
Door Close'
Door Open
Window Close
Window Open
Drawer Close
In-Domain-Oni
100.0 = 0.0
31.1 + 44.0
0.0 = 0.0
33.3 = 47.1
74.4 + 36.2
Vanilla AnimateLCM
100.0 = 0.0
0.0 = 0.0
98.9 : 1.9
33,3 = 29.1
100.0 + 0.0
Prob. Adaptation
100.0 = 0.0
0.0 0.0
66.7 57.7
0.0 0.0
100.0 0.0
Inverse Prob. Adaptation
100.0 1 0.0
100.0 0.0
100.0 1 0.0
94.4 9.6
100.0 0.0
Success Rate (5) w/
Drawer Open
Coffee Push"
Soceer
Button Press
Overall
In- Domain-Only
0.0 + 0.0
0.0 + 0.0
0.0 + 0.0
33.3 + 47.1
30.2
Vanilla AnimateLCCM
0.0 + 0.0
5.6 = 9.6
0.0 0.0
0.0 = 0.0
37.5
Prob. Adaptation
0.0 + 0.0
32.2 + 28.0
4.4 = 7.7
0.0 + 0.0
33.7
Inverse Prob. Adaptation
16.7 + 29.0
41.1 + 15.0
4.4 = 5.1
30.0 = 52.0
65.2
Table A10: Policy Learning on Meta MetaWorld with AnimateLCM. We report the mean success rate across 9 manipulation tasks in MetaWorld, aggregated over 3 seeds.
We also provide policy supervision results on MetaWorld with AnimateL.CM (Wang al., 2024b) in Table A10. Similar to AnfmateDiff, vanilla AnimateLCCM is also able to achieve decent suc- cess rates through Video-TADPoLe. Furthermore, we discover that inverse probabilistic adaptation consistently achieves the best performance with both AnimateDiff and AnimateLCM. With Ani- MateLCM, inverse probabilistic adaptation obtains the highest overall success rate of 65.2%. 2, sur- passing all other evaluated video models and adaptation techniques, with non-zero success rates across all evaluated tasks.
G VISUAL PLANNING WITH ADDITIONAL PRETRAINED VIDEO MODELS
Success Rate (%) w/
Door Close"
Door Open
Window Close
Window Open
Drawer Close
In-Domain-One Only
93.3 1 14,9
0.0 1 0.0
53,3 1 29.8
6.7 1 14.9
20.0 1L 29.8 40.0 + 27.9
Vanilla AnimateL.CM
100.0 + 0.0
0.0 + 0.0
0.0 + 0.0
20.0 + 18.3
Prob. Aduptation
100.0 = 0.0
0.0 0.0
53.3 38.0
0.0
53.3 29.8
Inverse Prob. Adaptation
100.0 = 0.0
0.0 = 0.0
40.0 = 14.9
0.0 = 0.0
93.3 + 14.9
Success Rate (%) w/
Drawer Open
Coffee Push"
Soccer
Button Press
Overall
In-Domain-Only
0.0 + 0.0
0.0 0.0
0.0 0.0 0.0
40.0 + 14.9
23.7
Vanilla AnimateLCCM
0.0 0.0
0.0 0.0
0.0
0.0
17.8
Prob. Adaptation
0.0 + 0.0
0.0 0.0
0.0 = 0.0
6.7 + 14.9
23.7
Inverse Prob. Adaptation
0.0 + 0.0
0.0 0.0
6.7 + 14.9
26.7 + 27.9
29.6
Table A11: Visual Planning on MetaWorld with AnimateLCM. We report the mean success rate across 9 manipulation tasks in MetaWorld. Each table entry shows the average success rate aggregated from 5 seeds.
We provide visual planning results on MetaWorld with an additional video diffusion model, Ani- mateLCM (Wang et al., 2024b), in Table 111. We observe both probabilistic adaptation and its in- verse version bring improvements in overall success rate compared to Vanilla AnimateLCM. Specif- ically, inverse probabilistic adaptation achieves the best overall performance and outperforms the
18

---

Published as a conference paper at ICLR 2025
in-domain-only baseline by 24.9%, reconfirming the efficacy of adaptation in improving in-domain task performance. This further demonstrates that adaptation as an approach can be applied flexibly across different backbone text-to-video models for successful downstream robotic applications.
H VISUAL PLANNING FOR OBJECT NAVIGATION IN ITHOR ENVIRONMENTS
Success
Sponola Kittloo"
Toaster Kltchon
Painting Room"
Blunds Redroom"
ToiletPaper Bathroon"
In-Domain-Chly
=
Prob. Adiptation
20.0 29.8
60.0 = 27.9
0.0 0.0
26.7 36.5
73.3 14.9
Inverse Prob. Adaptation
13.3 18.3
33.3 = 23.6
0.0 0.0
33.3 33.3
40.0 14.9
Sucpess Rate (5) w/
Pillow in Liinng Rovm
Deskt amp living Room
Minor in Bedroom
Taptop in Redromm
Overall
In-Domain-Culy
6.7 + 14.9
6.7 14.9
0.0 0.0
26.7 27.9
15.6
Prol, Adiptation
13.3
133 18.3
0.0
$3.3
Inverse Proh. Adaptition
+ 14.9
1333 18.3
Table A12: Visual Planning on ITHOR. We report the mean success rate across 9 object navigation tasks in iTHOR. Each table entry shows the average success rate aggregated from 5 seeds. "*" denotes seen tasks during adaptation.
We provide additional experimentation of adaptation techniques on iTHOR (Eric et al., 2017). in which a mobile robotic agent is asked to perform egocentric navigation to specified target object in different scenes. This benchmark poses challenges of navigating in partially observable settings and allows us to further evaluate adaptation methods on in-domain video generation from egocentric views. To perform adaptation, we reuse the video dataset provided by AVDC (Ko et al., 2024), which spans 12 target objects and includes 25 successful navigation trajectories for each object. We report the success rates of visual planning across navigation tasks in Table A12, in which 4 tasks are unseen during adaptation. We provide detailed list of iTHOR tasks along with their corresponding text prompts in Table A2. In Table A12, we again observe that the overall performance of both probabilistic adaptation and its inverse outperform that of in-domain-only baseline by large margin, highlighting that the internet knowledge of pretrained video models can be effectively utilized for various downstream robotic applications through proper adaptation. This result further highlights how adaptation can be flexibly applied across varied robotic settings.
I STEP COUNTS TO TASK SUCCESS IN CLOSE-LOOP VISUAL PLANNING
Step Count
Door Close'
Door Open
Window Close
Window Open
Drawer Close
In-Domain-Oni
80.0
176.0
344.0
25.3
Vanilla DiimateDiff
122.3
323.0
217.3
160,9
Direct Finetuning
96.0
159.2
333.3
63.8
Subject Customization
150.5
297.3
238.7
Prob. Adaptation
75.4
171.5
312.0
31.0
Inverse Prob. Adaptation
87.0
222.6
35.8
Step Count w/
Drawer Open
Coffee Push
Soccer
Button Press
Hi-Domain-Only
204.0
Vanilla AnimateDiff
Direct Finetuning
Subject Customization
155.0
Prob. Adaptation
244.0
$2.0
183.2
Inverse Proh. Adaptation
144.0
192.0
Table A13: Step Counts of Visual Planning on MetaWorld World. We report the average number of taken steps in successful evaluation rollouts across 9 manipulation tasks in Meta MorlW. Unsuccess- ful rollouts are omitted. We observed that probabilistic adaptation in general achieves task success using fewer number of steps.
19


Reviews:
[{"review_id": "SnBS5ivyN8", "reviewer": "Reviewer_DMT5", "summary": "This work investigates how to adapt large-scale, pre-trained video models to solve novel, text-conditioned robotic tasks in specific environments. The authors explore three adaptation techniques: direct ft, subject customization, and probabilistic adaptation. It further introduces inverse probabilistic adaptation. The model is evaluated on metaworld and deepmind control suite.", "questions": "My suggestion is to conduct more experiments, such as testing model generalization ability.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) The motivation is straightforward. \n2) The paper is well written, it does lots of controlled experiments to discuss which adaption techniques are better.", "weaknesses": "The scope of this paper is limited. While visual planning and goal-conditioned imitation/reinforcement learning (IL/RL) are indeed significant topics in robotics, the approach to generating visual targets appears straightforward.\n\nThe experimental results lack novelty, as it is unsurprising that a web-scale, pre-trained text-to-image (T2I) or video generator would produce superior images or videos.\n\nFurthermore, this work does not directly address whether improvements in image quality contribute to enhanced performance or generalization in control tasks. For example, while Table 2 shows that inverse probability achieves the highest success rate, its FVD score in Table 3 is higher than others, raising questions about the relationship between quality and control success. Additionally, the study does not include generalization tests for control tasks. Although the paper aims to address new tasks, no genuinely novel tasks are explored.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "in8pkVL9dT", "reviewer": "Reviewer_NuKB", "summary": "The paper presents a study on how to incorporate pre-trained, language conditioned video generation models into robot learning. The paper considers three adaptation strategies for using video generation towards policy learning: finetuning the majority of the video generation model on expert demonstrations (most expensive for data and compute), subject customization using a few static images from the target domain, and probabilistic adaptation which trains a small in-domain model supervised by the large-scale video generation model. The paper further considers two options for downstream robotic task evaluation: video planning by predicted a plan to follow into the future and policy supervision by using the adapted video model to synthesize rewards for the policy. The different adaptation and evaluation techniques are benchmarked on the MetaWorld-v2 and DeepMind Control Suite simulations. Probabilistic adaptation and subject customization show promise in this domain when combined with the AnimatedDiff video generation model.", "questions": "1. Is there a reason batch-balanced co-training was not studied as an alternative adaptation technique? Co-training has seen decent success in robot learning in recent years [2]? \n2. Why is inverse probabilistic adaptation good fro MetaWorld-v2 but so much worse for the DeepMind Control Suite?\n\n[2] Khazatsky, Alexander, et al. \"DROID: A large-scale in-the-wild robot manipulation dataset.\" RSS, 2024.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The general problem of incorporating large-scale video generation into robot policy learning is of high interest to the robotics community. Particularly, with the rise in popularity of World Models in robotics, this paper's study is quite timely.\n* The paper is well written and fairly easy to follow.\n* I found the discussion of positives and negatives of the policy supervision evaluation approach to be insightful in Sec. 3.2.2.\n* The experiments are fairly extensive in the simulated environments and transparent regarding underperformance in certain domains.\n* The limitation section is thoughtfully constructed.", "weaknesses": "* The majority of the assumption in the paper seems to be that video models provide strong motion priors and just require adaptation to the downstream domain visually (please correct me if I'm wrong) e.g., through static images in subject customization. However, from my experience large-scale video generation models still really struggle with the dynamics of the environment. This even seems to come across in the MetaWorld-v2 benchmark where a lot of the tasks achieve 0% or very low success rate.\n* All the results are based on a single video generation model: AnimateDiff. It would have been more compelling to show results on a few models, in case the findings are particular to the specific chosen model.\n* Some of the language is quite repetitive throughout the paper (e.g., last sentence of 3.1.1 and first sentence of 3.1.2).\n* I am not sure I entirely agree with the premise that it is impossible/highly challenging to obtain in-domain video demonstrations for a task. ~100 demonstrations are often assumed for imitation learning papers like Diffusion Policy [1], particularly if high success rates are desired on dexterous tasks. This relates to the low success numbers seen in the MetaWorld-v2 tables, particularly for tasks unseen during adaptation. Optimization compute cost seems like potentially a bigger concern.\n* Since a large focus of the paper is evaluating video generation adaptation in robotics tasks, it feels like real-world robot experiments should be included in the analysis. In simulation, adaptation may be more focused on the appearence of the simulated environment than the physics modeling of the language-specified task. In the real-world, modeling the physics of the interactions between the robot and the environment in real-time is the challenge.\n* The data budgets for the expert demonstrations is quite low and appears rather arbitrary (25 for MetaWorld-v2 and 6 and 17 for the DeepMind Control Suite). It would be helpful to ground these design choices in recent literature. \n* Standard error is only present for the first column of Table 1. It should be included across the considered methods.\n* The very poor performance of probabilistic adaptation in Table 1 is attributed to the small capacity of the in-domain model used. However, this should have been explored to understand the potential of this method in the DeepMind domain.\n* The results in Figure 3 were not particularly convincing to me for the dog jumping case. It seems the video generation model was able to generate a partially reasonable but not entirely accurate jumping dog sequence. The resulting simulated behavior appears to be a degenerated version of that suboptimal supervision. Although I do see that there is potential there for simulating unseen behaviors zero-shot.\n* Showcasing results where all methods get 0% (or close to 0^%) success rate for about half the tasks is not very meaningful as a takeaway. It seems like these experiments should have been iterated on futher. Likely including additional expert demonstrations would help the success rate.\n\nSome typos and points of confusion are listed below:\n1. Line 018 - video modeling ... [encodes].\n2. The score in Sec. 3.1.3 should be formally defined.\n3. The FVD acronym should be defined once in line 239.\n\n[1] Chi, Cheng, et al. \"Diffusion policy: Visuomotor policy learning via action diffusion.\" The International Journal of Robotics Research, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QTQfWBICiY", "reviewer": "Reviewer_fa6P", "summary": "This paper investigates three adaptation techniques for generative video models, and proposes two evaluation metrics that go beyond traditional visual similarity scores in an attempt to more precisely test the effect of different adaptation techniques of video models on downstream performance for robotic tasks. The proposed adaptation techniques require small amounts of training data, and in some cases (subject customization) only static paired data. The proposed evaluation methods are by using the adapted video model as either a visual planner or policy supervisor, each with its own strengths as noted at the end of the methods section. Experiments are performed on a dog/humanoid and robot arm dataset 2 tasks and 16 tasks respectively. The authors finally observe that domain specific fine-tuning (for the dog/humanoid) and inverse probabilistic adaptation (for the robot arm) achieve the best performance. \n\nOverall this paper explores an important topic that will have many implications when using generative video models for robotic tasks.", "questions": "Questions that should be addressed:\n1. I am unsure if (inverse) probabilistic adaptation has been explored before at all in the robotics tasks context. Maybe make this more explicit in the ‘adaptation techniques for diffusion models’ related works section. \n2. Eq 1 & 2 introduces notation but many of the variables (theta, t, tao) are not defined. Please state what each variable means.\n3. Also, please clarify what e_theta is in Eq 1&2. Is this indicating the denoising unet for the AnimatedDiff model?  \n4. It would be good to clarify what ‘out of domain’ means in these adaptation experiments. It seems the robot arm and its dynamics must stay fixed for subject customization since the visual token is encoding this specific robot arm right? What about for (inverse) probabilistic adaptation, are there any such constraints on the domain you are adapting to? \n5. Sec 3.2 does not have details about how the reward is calculated using the adapted video model, all details are in Appendix B. Maybe it is because VideoTADPoLe is unknown to me, but I expect this to be the case for many readers. So please give atleast a brief description in the main paper and then you can point the readers to Appendix B.  \n6. Please include the exact details of the ‘small dataset’ mentioned in line 312. I believe is important and should not just be kept in supplemental because the paper focuses on how to adapt with small # of examples. \n7. Please show a qualitative example from the Subject Customization for Figure 3 so that the readers  can see the qualitative comparisons as described in the paragraphs in Sec 4.2. \n8. What is ‘high variance’ referring to in L285? Variance in the output of a video model planner?\n9. AnimateDiff produces very short horizon videos (in terms of time into the future). Therefore, completing a task seems to require lots of closed-loop rollouts. A mention of the number of steps to complete the task might be good to include in each type of experiment.\n\nMinor Comments:\n1. It is unclear whether AnimateDiff is the right model for this work. Why did the authors not choose to just use a video-model itself such as StableDiffusionVideo which has stronger priors about long-term motions? Some mention of this would be good to include.\n2. The ‘Studying Data Quality’ section is interesting, but very few details are mentioned in the main paper, with most left to the supplementary section. If you would like to keep this section in the main paper, I would suggest at least including what a ‘suboptimal dataset’ means. Otherwise there is very little a reader can take away from this section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Several robot arm tasks in MetaWorld environment and an additional environment are proposed as an evaluation scheme\n2. The work is well motivated and tackles a nuanced but important question that is often glossed over in other policy-focused works.\n3. Implementation details are very clearly stated and discussed so readers have a full picture\n4. The subject customization technique is unique and novel, and I think the inverse probabilistic adaptation method is original (but please clarify this too (ask mentioned in Question #1 below) \n5. Very interesting observations regarding visual appearance vs task success rate are made, motivating the need to perform real world experiments such as those done in this paper to really understand what the right way to train/fine-tune/adapt video models for robotic tasks.", "weaknesses": "1. Quantitative results are not presented for the out-of-domain Humanoid and Dog environments. Table 1 seems to be for in-domain dataset, and Figure 3 shows qualitative results, but quantitative results for the jump task would be good to include in Table 1. \n\n2. Two different adaptation techniques are concluded to work the best for the two different environments. (fine tuning for the dog/human environment, and inverse probabilistic adaptation for the robotic manipulation environment). This raises 2 questions: first, why do the ‘best methods’ differ in these two environments? And second, what are readers supposed to take away when working with a new different environment? A discussion, about knowing what technique to use based on the environment and why, would be good to include.  \n\n3. Why do many adaptation methods completely fail for several tasks when using Video Planning (Table 4)? Especially the Soccer task? Is this due to poor inverse dynamics model or the inability to perform good video adaptation? There should be an explanation of why this is happening, otherwise it's hard to be convinced this is a valid experiment since the point of failure is unclear. \n\n4. Subject customization is interesting, but it is unclear if motion dynamics in a new environment can be learned. The training data only includes static images paired with text, so when the dynamics vary greatly, will this method fail in comparison to direct fine-tuning/probabilistic adaptation? A good test of this would be to change the dynamics in an out-of-domain environment (something like the gravity constant if you can control that in the dog/humanoid world) and compare the 3 adaptation methods. I expect that static transfer cannot capture dynamics differences. If this is a tough experiment to run but there is a different way to prove the soundness of this adaptation method given different dynamics, please go ahead and present that, this was just one possible idea.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VOfibiM6rC", "reviewer": "Reviewer_yUek", "summary": "This paper explores a range of domain adaptation techniques and incorporates them into large-scale, pre-trained video models for domain-specific robotic tasks. Notably, the authors extend the motion module, originally introduced in AnimateDiff, to enhance the animation capabilities of the video model for these specific tasks. Building on a vanilla video model, the authors examine various domain adaptation strategies, including direct fine-tuning, subject-specific customization, and probabilistic adaptation. A key innovation in this work is the redesign of the probabilistic adaptation approach to an inverse version, which demonstrates superior performance across both visual planning and policy learning tasks. The inverse approach yields significant improvements, achieving a success rate of 52.6% in policy learning and 28.4% in video planning, along with enhanced visual quality in the MetaWorld environment.", "questions": "While the authors provide a comparative analysis with baseline methods, the paper still lacks some thorough examination of adaptation techniques to demonstrate the significant improvement of video adaption to in-domain video generation and robot manipulation tasks. Additionally, some baseline methods were not accurately reproduced during the evaluation, raising concerns about the validity of the results.\n\nGiven these issues, I am inclined to give the rating below the acceptance threshold. However, I will reassess my rating after reviewing feedback from other reviewers and the authors' responses. I am willing to raise my score if the authors address these concerns.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This work explores an efficient strategy for adapting internet video knowledge to in-domain simulation environments.\n2. The experimental design is thorough, encompassing evaluations of policy learning and video planning across both seen and unseen tasks. \n3. Part of the experimental results look convincing, effectively demonstrating the performance of the proposed adaptation strategies.", "weaknesses": "1. The results in Table 1 show that subject customization achieves significant improvement over both the direct fine-tuning approach and vanilla AnimateDiff. However, probabilistic adaptation underperforms compared to the other three settings, exhibiting a notable discrepancy in return values. Although the authors assert that probabilistic adaptation focuses on in-domain estimation and struggles to learn effective policies in the Humanoid and Dog environments, I am concerned that the evaluation protocol may not be optimal when using policy discrimination for this setting. To my knowledge, probabilistic adaptation adopts score composition from both the pre-trained model and the domain-specific model, introducing a distribution shift during sampling. I recommend that the authors tune hyper-parameters for probabilistic adaptation methods by re-evaluating policy discrimination to minimize model prediction biases. Additionally, it would be helpful to see a case study that further examines the return values for probabilistic adaptation/inverse probabilistic adaptation, similar to the results presented in the Policy Discrimination section of the project website.\n\n\n2. The results in Table 2 show that the inverse probabilistic adaptation model outperforms other adaptation methods in policy learning success rate on MetaWorld. Could the authors also provide the average returns for each baseline method?\n\n3. Evaluation of Video Quality: The quantitative results in Table 3 are not entirely convincing, as only two evaluation tasks (coffee push and button press) are considered, which may lead to evaluation bias. Please provide additional video quality analysis across all tasks to present a more comprehensive assessment of the overall results.  \n \n4. Based on the qualitative results from both the project website and the paper, it does not appear that the in-domain-only model performs worse than probabilistic adaptation or inverse probabilistic adaptation in task-level motion; the primary differences seem to be at the pixel-level generation. Another concern is that the reproduced results of the in-domain-only model exhibit lower video quality compared to the qualitative results presented in the AVDC paper (synthesized videos in MetaWorld). This discrepancy could lead to an unfair comparison. Could the authors explain their implementation process in detail, noting any deviations from the AVDC paper that might account for this discrepancy in video quality? A thorough review of the reproduction process would help ensure an accurate assessment of the model's performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)