Solving New Tasks by Adapting Internet Video Knowledge

Authors: Calvin Luo, Zilai Zeng, Yilun Du, Chen Sun
Primary area: reinforcement learning
Keywords: Text-Conditioned Generalization, Video Diffusion, Adaptation, Planning, Policy Learning

Abstract:
Video generative models demonstrate great promise in robotics by serving as visual planners or as policy supervisors.  When pretrained on internet-scale data, such video models intimately understand alignment with natural language, and can thus facilitate generalization to novel downstream behavior through text-conditioning.  However, they may not be sensitive to the specificities of the particular environment the agent inhabits.  On the other hand, training video models on in-domain examples of robotic behavior naturally encodes environment-specific intricacies, but the scale of available demonstrations may not be sufficient to support generalization to unseen tasks via natural language specification.  In this work, we investigate different adaptation techniques that integrate in-domain information with large-scale pretrained video models, and explore the extent to which they enable novel text-conditioned generalization for robotic tasks, while also considering their independent data and resource considerations.  We successfully demonstrate across robotic environments that adapting powerful video models with small scales of example data can successfully facilitate generalization to novel behaviors.  In particular, we present a novel adaptation strategy, termed *Inverse Probabilistic Adaptation*, that not only consistently achieves strong generalization performance across robotic tasks and settings, but also exhibits robustness to the quality of adaptation data, successfully solving novel tasks even when only suboptimal in-domain demonstrations are available.

Decision: accept (poster)
Reviewer scores: 5, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729952378123

thread_type:
official_review

<thread_object>
creator: reviewer_yuek

<content>
summary:
This paper explores a range of domain adaptation techniques and incorporates them into large-scale, pre-trained video models for domain-specific robotic tasks. Notably, the authors extend the motion module, originally introduced in AnimateDiff, to enhance the animation capabilities of the video model for these specific tasks. Building on a vanilla video model, the authors examine various domain adaptation strategies, including direct fine-tuning, subject-specific customization, and probabilistic adaptation. A key innovation in this work is the redesign of the probabilistic adaptation approach to an inverse version, which demonstrates superior performance across both visual planning and policy learning tasks. The inverse approach yields significant improvements, achieving a success rate of 52.6% in policy learning and 28.4% in video planning, along with enhanced visual quality in the MetaWorld environment.

strengths:
1. This work explores an efficient strategy for adapting internet video knowledge to in-domain simulation environments.
2. The experimental design is thorough, encompassing evaluations of policy learning and video planning across both seen and unseen tasks. 
3. Part of the experimental results look convincing, effectively demonstrating the performance of the proposed adaptation strategies.

weaknesses:
1. The results in Table 1 show that subject customization achieves significant improvement over both the direct fine-tuning approach and vanilla AnimateDiff. However, probabilistic adaptation underperforms compared to the other three settings, exhibiting a notable discrepancy in return values. Although the authors assert that probabilistic adaptation focuses on in-domain estimation and struggles to learn effective policies in the Humanoid and Dog environments, I am concerned that the evaluation protocol may not be optimal when using policy discrimination for this setting. To my knowledge, probabilistic adaptation adopts score composition from both the pre-trained model and the domain-specific model, introducing a distribution shift during sampling. I recommend that the authors tune hyper-parameters for probabilistic adaptation methods by re-evaluating policy discrimination to minimize model prediction biases. Additionally, it would be helpful to see a case study that further examines the return values for probabilistic adaptation/inverse probabilistic adaptation, similar to the results presented in the Policy Discrimination section of the project website.


2. The results in Table 2 show that the inverse probabilistic adaptation model outperforms other adaptation methods in policy learning success rate on MetaWorld. Could the authors also provide the average returns for each baseline method?

3. Evaluation of Video Quality: The quantitative results in Table 3 are not entirely convincing, as only two evaluation tasks (coffee push and button press) are considered, which may lead to evaluation bias. Please provide additional video quality analysis across all tasks to present a more comprehensive assessment of the overall results.  
 
4. Based on the qualitative results from both the project website and the paper, it does not appear that the in-domain-only model performs worse than probabilistic adaptation or inverse probabilistic adaptation in task-level motion; the primary differences seem to be at the pixel-level generation. Another concern is that the reproduced results of the in-domain-only model exhibit lower video quality compared to the qualitative results presented in the AVDC paper (synthesized videos in MetaWorld). This discrepancy could lead to an unfair comparison. Could the authors explain their implementation process in detail, noting any deviations from the AVDC paper that might account for this discrepancy in video quality? A thorough review of the reproduction process would help ensure an accurate assessment of the model's performance.

questions:
While the authors provide a comparative analysis with baseline methods, the paper still lacks some thorough examination of adaptation techniques to demonstrate the significant improvement of video adaption to in-domain video generation and robot manipulation tasks. Additionally, some baseline methods were not accurately reproduced during the evaluation, raising concerns about the validity of the results.

Given these issues, I am inclined to give the rating below the acceptance threshold. However, I will reassess my rating after reviewing feedback from other reviewers and the authors' responses. I am willing to raise my score if the authors address these concerns.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730646608560

thread_type:
official_review

<thread_object>
creator: reviewer_fa6p

<content>
summary:
This paper investigates three adaptation techniques for generative video models, and proposes two evaluation metrics that go beyond traditional visual similarity scores in an attempt to more precisely test the effect of different adaptation techniques of video models on downstream performance for robotic tasks. The proposed adaptation techniques require small amounts of training data, and in some cases (subject customization) only static paired data. The proposed evaluation methods are by using the adapted video model as either a visual planner or policy supervisor, each with its own strengths as noted at the end of the methods section. Experiments are performed on a dog/humanoid and robot arm dataset 2 tasks and 16 tasks respectively. The authors finally observe that domain specific fine-tuning (for the dog/humanoid) and inverse probabilistic adaptation (for the robot arm) achieve the best performance. 

Overall this paper explores an important topic that will have many implications when using generative video models for robotic tasks.

strengths:
1. Several robot arm tasks in MetaWorld environment and an additional environment are proposed as an evaluation scheme
2. The work is well motivated and tackles a nuanced but important question that is often glossed over in other policy-focused works.
3. Implementation details are very clearly stated and discussed so readers have a full picture
4. The subject customization technique is unique and novel, and I think the inverse probabilistic adaptation method is original (but please clarify this too (ask mentioned in Question #1 below) 
5. Very interesting observations regarding visual appearance vs task success rate are made, motivating the need to perform real world experiments such as those done in this paper to really understand what the right way to train/fine-tune/adapt video models for robotic tasks.

weaknesses:
1. Quantitative results are not presented for the out-of-domain Humanoid and Dog environments. Table 1 seems to be for in-domain dataset, and Figure 3 shows qualitative results, but quantitative results for the jump task would be good to include in Table 1. 

2. Two different adaptation techniques are concluded to work the best for the two different environments. (fine tuning for the dog/human environment, and inverse probabilistic adaptation for the robotic manipulation environment). This raises 2 questions: first, why do the ‘best methods’ differ in these two environments? And second, what are readers supposed to take away when working with a new different environment? A discussion, about knowing what technique to use based on the environment and why, would be good to include.  

3. Why do many adaptation methods completely fail for several tasks when using Video Planning (Table 4)? Especially the Soccer task? Is this due to poor inverse dynamics model or the inability to perform good video adaptation? There should be an explanation of why this is happening, otherwise it's hard to be convinced this is a valid experiment since the point of failure is unclear. 

4. Subject customization is interesting, but it is unclear if motion dynamics in a new environment can be learned. The training data only includes static images paired with text, so when the dynamics vary greatly, will this method fail in comparison to direct fine-tuning/probabilistic adaptation? A good test of this would be to change the dynamics in an out-of-domain environment (something like the gravity constant if you can control that in the dog/humanoid world) and compare the 3 adaptation methods. I expect that static transfer cannot capture dynamics differences. If this is a tough experiment to run but there is a different way to prove the soundness of this adaptation method given different dynamics, please go ahead and present that, this was just one possible idea.

questions:
Questions that should be addressed:
1. I am unsure if (inverse) probabilistic adaptation has been explored before at all in the robotics tasks context. Maybe make this more explicit in the ‘adaptation techniques for diffusion models’ related works section. 
2. Eq 1 & 2 introduces notation but many of the variables (theta, t, tao) are not defined. Please state what each variable means.
3. Also, please clarify what e_theta is in Eq 1&2. Is this indicating the denoising unet for the AnimatedDiff model?  
4. It would be good to clarify what ‘out of domain’ means in these adaptation experiments. It seems the robot arm and its dynamics must stay fixed for subject customization since the visual token is encoding this specific robot arm right? What about for (inverse) probabilistic adaptation, are there any such constraints on the domain you are adapting to? 
5. Sec 3.2 does not have details about how the reward is calculated using the adapted video model, all details are in Appendix B. Maybe it is because VideoTADPoLe is unknown to me, but I expect this to be the case for many readers. So please give atleast a brief description in the main paper and then you can point the readers to Appendix B.  
6. Please include the exact details of the ‘small dataset’ mentioned in line 312. I believe is important and should not just be kept in supplemental because the paper focuses on how to adapt with small # of examples. 
7. Please show a qualitative example from the Subject Customization for Figure 3 so that the readers  can see the qualitative comparisons as described in the paragraphs in Sec 4.2. 
8. What is ‘high variance’ referring to in L285? Variance in the output of a video model planner?
9. AnimateDiff produces very short horizon videos (in terms of time into the future). Therefore, completing a task seems to require lots of closed-loop rollouts. A mention of the number of steps to complete the task might be good to include in each type of experiment.

Minor Comments:
1. It is unclear whether AnimateDiff is the right model for this work. Why did the authors not choose to just use a video-model itself such as StableDiffusionVideo which has stronger priors about long-term motions? Some mention of this would be good to include.
2. The ‘Studying Data Quality’ section is interesting, but very few details are mentioned in the main paper, with most left to the supplementary section. If you would like to keep this section in the main paper, I would suggest at least including what a ‘suboptimal dataset’ means. Otherwise there is very little a reader can take away from this section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730672932114

thread_type:
official_review

<thread_object>
creator: reviewer_nukb

<content>
summary:
The paper presents a study on how to incorporate pre-trained, language conditioned video generation models into robot learning. The paper considers three adaptation strategies for using video generation towards policy learning: finetuning the majority of the video generation model on expert demonstrations (most expensive for data and compute), subject customization using a few static images from the target domain, and probabilistic adaptation which trains a small in-domain model supervised by the large-scale video generation model. The paper further considers two options for downstream robotic task evaluation: video planning by predicted a plan to follow into the future and policy supervision by using the adapted video model to synthesize rewards for the policy. The different adaptation and evaluation techniques are benchmarked on the MetaWorld-v2 and DeepMind Control Suite simulations. Probabilistic adaptation and subject customization show promise in this domain when combined with the AnimatedDiff video generation model.

strengths:
* The general problem of incorporating large-scale video generation into robot policy learning is of high interest to the robotics community. Particularly, with the rise in popularity of World Models in robotics, this paper's study is quite timely.
* The paper is well written and fairly easy to follow.
* I found the discussion of positives and negatives of the policy supervision evaluation approach to be insightful in Sec. 3.2.2.
* The experiments are fairly extensive in the simulated environments and transparent regarding underperformance in certain domains.
* The limitation section is thoughtfully constructed.

weaknesses:
* The majority of the assumption in the paper seems to be that video models provide strong motion priors and just require adaptation to the downstream domain visually (please correct me if I'm wrong) e.g., through static images in subject customization. However, from my experience large-scale video generation models still really struggle with the dynamics of the environment. This even seems to come across in the MetaWorld-v2 benchmark where a lot of the tasks achieve 0% or very low success rate.
* All the results are based on a single video generation model: AnimateDiff. It would have been more compelling to show results on a few models, in case the findings are particular to the specific chosen model.
* Some of the language is quite repetitive throughout the paper (e.g., last sentence of 3.1.1 and first sentence of 3.1.2).
* I am not sure I entirely agree with the premise that it is impossible/highly challenging to obtain in-domain video demonstrations for a task. ~100 demonstrations are often assumed for imitation learning papers like Diffusion Policy [1], particularly if high success rates are desired on dexterous tasks. This relates to the low success numbers seen in the MetaWorld-v2 tables, particularly for tasks unseen during adaptation. Optimization compute cost seems like potentially a bigger concern.
* Since a large focus of the paper is evaluating video generation adaptation in robotics tasks, it feels like real-world robot experiments should be included in the analysis. In simulation, adaptation may be more focused on the appearence of the simulated environment than the physics modeling of the language-specified task. In the real-world, modeling the physics of the interactions between the robot and the environment in real-time is the challenge.
* The data budgets for the expert demonstrations is quite low and appears rather arbitrary (25 for MetaWorld-v2 and 6 and 17 for the DeepMind Control Suite). It would be helpful to ground these design choices in recent literature. 
* Standard error is only present for the first column of Table 1. It should be included across the considered methods.
* The very poor performance of probabilistic adaptation in Table 1 is attributed to the small capacity of the in-domain model used. However, this should have been explored to understand the potential of this method in the DeepMind domain.
* The results in Figure 3 were not particularly convincing to me for the dog jumping case. It seems the video generation model was able to generate a partially reasonable but not entirely accurate jumping dog sequence. The resulting simulated behavior appears to be a degenerated version of that suboptimal supervision. Although I do see that there is potential there for simulating unseen behaviors zero-shot.
* Showcasing results where all methods get 0% (or close to 0^%) success rate for about half the tasks is not very meaningful as a takeaway. It seems like these experiments should have been iterated on futher. Likely including additional expert demonstrations would help the success rate.

Some typos and points of confusion are listed below:
1. Line 018 - video modeling ... [encodes].
2. The score in Sec. 3.1.3 should be formally defined.
3. The FVD acronym should be defined once in line 239.

[1] Chi, Cheng, et al. "Diffusion policy: Visuomotor policy learning via action diffusion." The International Journal of Robotics Research, 2023.

questions:
1. Is there a reason batch-balanced co-training was not studied as an alternative adaptation technique? Co-training has seen decent success in robot learning in recent years [2]? 
2. Why is inverse probabilistic adaptation good fro MetaWorld-v2 but so much worse for the DeepMind Control Suite?

[2] Khazatsky, Alexander, et al. "DROID: A large-scale in-the-wild robot manipulation dataset." RSS, 2024.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730831282632

thread_type:
official_review

<thread_object>
creator: reviewer_dmt5

<content>
summary:
This work investigates how to adapt large-scale, pre-trained video models to solve novel, text-conditioned robotic tasks in specific environments. The authors explore three adaptation techniques: direct ft, subject customization, and probabilistic adaptation. It further introduces inverse probabilistic adaptation. The model is evaluated on metaworld and deepmind control suite.

strengths:
1) The motivation is straightforward. 
2) The paper is well written, it does lots of controlled experiments to discuss which adaption techniques are better.

weaknesses:
The scope of this paper is limited. While visual planning and goal-conditioned imitation/reinforcement learning (IL/RL) are indeed significant topics in robotics, the approach to generating visual targets appears straightforward.

The experimental results lack novelty, as it is unsurprising that a web-scale, pre-trained text-to-image (T2I) or video generator would produce superior images or videos.

Furthermore, this work does not directly address whether improvements in image quality contribute to enhanced performance or generalization in control tasks. For example, while Table 2 shows that inverse probability achieves the highest success rate, its FVD score in Table 3 is higher than others, raising questions about the relationship between quality and control success. Additionally, the study does not include generalization tests for control tasks. Although the paper aims to address new tasks, no genuinely novel tasks are explored.

questions:
My suggestion is to conduct more experiments, such as testing model generalization ability.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732153323732

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We thank all reviewers for their constructive feedback - we are glad that our work was recognized as “well-motivated” and “insightful”, and our experimental design and execution were recognized as “extensive” and “thorough”.

We have since updated the paper to reflect the comments raised by the reviewers, notably for notation clarification and dataset details.  We also evaluate using an additional backbone text-to-video model, AnimateLCM [1], which we report in Appendix F, Table A9.  We discover that inverse probabilistic adaptation achieves the best overall performance and outperforms both using an in-domain model and the large-scale AnimateLCM model by themselves, reconfirming the efficacy of adaptation in improving in-domain task performance.  This further demonstrates that adaptation as an approach can be applied flexibly across different backbone text-to-video models for successful downstream robotic applications.

Furthermore, we include new experimental results for a novel environment and task setting - navigation in iTHOR [2].  We note that the navigation task is a distinct challenge from the previously investigated approaches of continuous locomotion (DMC) and robotic manipulation (MetaWorld), as the videos are now egocentric in nature and successful task performance relies on handling partial observability.  In Appendix G, Table A10, we show that across 9 object navigation tasks, the overall performance of both probabilistic adaptation and its inverse outperform that of in-domain-only baseline by a large margin, highlighting that the internet knowledge of pretrained video models can be effectively utilized for various downstream robotic applications through proper adaptation.  We provide the table in Markdown format below, in which unseen tasks are denoted in italic format:

| Success Rate (%) w/  	| Spatula  in Kitchen  | Toaster  in Kitchen | Painting  in Living Room | Blinds  in Bedroom | ToiletPaper  in Bathroom | *Pillow  in Living Room* | *DeskLamp in Living* Room | *Mirror  in Bedroom* | *Laptop  in Bedroom* | Overall |
|--------------------------|:--------------------:|:-------------------:|:------------------------:|:------------------:|:------------------------:|:----------------------:|:-----------------------:|:------------------:|:------------------:|:-------:|
| In-Domain-Only       	|  	13.3 ± 29.8 	| 	33.3 ± 33.3 	|         	0        	| 	13.3 ± 29.8	|    	40.0 ± 36.5   	|   	6.67 ± 14.9  	|   	6.67 ± 14.9   	|      	0     	| 	26.7 ± 27.9	|   15.5  |
| Prob. Adaptation     	| 	20.0 ± 29.8  	| 	60.0 ± 27.9 	|         	0        	| 	26.7 ± 36.5	|   	73.3 ± 14.9    	|   	13.3 ± 18.3  	|   	13.3 ± 18.3   	|      	0     	| 	53.3 ± 29.8	|   **28.9**  |
| Inverse Prob. Adaptation |  	13.3 ± 18.3 	| 	33.3 ± 23.6 	|         	0        	| 	33.3 ± 33.3	|    	40.0 ± 14.9   	|   	6.67 ± 14.9  	|   	13.3 ± 18.3   	| 	6.67 ± 14.9	| 	60.0 ± 27.9	|   23.0  |


We thank the reviewers for their time and consideration, and welcome further discussion,

The Authors
  
  
  


[1] Wang et al. AnimateLCM: Computation-Efficient Personalized Style Video Generation without Personalized Video Data. arXiv:2402.00769 2024.

[2]  Kolve et al. AI2-THOR: An Interactive 3D Environment for Visual AI. arXiv:1712.05474, 2017.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732573051765

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response - Additional Experimental Results

comment:
We have updated the draft with additional experimentation results.  In Appendix I, Table A12, we include quantitative policy supervision results on MetaWorld through adaptation with another video model backbone, AnimateLCM.  This complements our previous experiment of utilizing adapted AnimateLCM for visual planning (Appendix F, Table A9).  We discover that despite the change in text-to-video model backbone, inverse probabilistic adaptation consistently achieves the best success-rate performance and relative performance trends across methods are maintained.  With AnimateLCM, inverse probabilistic adaptation obtains the highest overall success rate of 65.2%, surpassing all other evaluated video models and adaptation techniques, with non-zero success rates across **all** evaluated tasks.

We invite the reviewers to engage with our updated draft and results, as well as our clarification responses, and look forward to their additional feedback and discussions regarding our work.

The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper investigates techniques for adapting large-scale internet video models to facilitate novel robotic behaviors through text conditioning, earning a recommendation for acceptance from all reviewers after discussion. The work demonstrates how video models pretrained on internet-scale data can be effectively adapted using limited in-domain examples to enable text-conditioned generalization for robotic tasks, evaluated through both visual planning and policy supervision approaches.

The reviewers consistently praised the comprehensive experimental evaluation across different environments (DeepMind Control, MetaWorld, iTHOR), the thorough analysis of different adaptation techniques, and the paper's strong motivation given the growing importance of video models in robotics. Initial concerns focused on the inconsistency of best-performing methods across environments, lack of clarity around implementation details, and questions about baseline comparisons.

The authors provided extensive responses and additional experiments that satisfied most reviewer concerns. They added evaluations with a new video model (AnimateLCM), expanded results on iTHOR navigation tasks, clarified implementation details particularly around latent-space modeling, and demonstrated more comprehensive FVD evaluations across all tasks. The authors also effectively explained the challenging nature of their experimental setup compared to prior work, particularly for MetaWorld tasks.

While some minor concerns remain about training configurations for latent-space modeling and direct fine-tuning performance on seen tasks, the overall consensus is that this work makes a valuable contribution by thoroughly evaluating how internet-scale video knowledge can be adapted for robotic tasks.