Self-Improving Loops for Visual Robotic Planning
Authors: Calvin Luo
Zilai Zeng
Mingxi Jia
Yilun Du
Chen Sun
Venue: ICLR 2026 Poster (2026)
Keywords: visual planning
self-improvement
video models
Primary area: applications to robotics, autonomy, planning
TL;DR: We propose SILVR: a robust, sample-efficient framework that iteratively improves performance on novel robotic tasks through visual planning.

Abstract: Video generative models trained on expert demonstrations have been utilized as performant text-conditioned visual planners for solving robotic tasks. However, generalization to unseen tasks remains a challenge. Whereas improved generalization may be facilitated by leveraging learned prior knowledge from additional pre-collected offline data sources, such as web-scale video datasets, in the era of experience we aim to design agents that can continuously improve in an online manner from self-collected behaviors.  In this work we thus propose the Self-Improving Loops for Visual Robotic Planning (SILVR), where an in-domain video model iteratively updates itself on self-produced trajectories, and steadily improves its performance for a specified task of interest.  We apply SILVR to a diverse suite of MetaWorld tasks, as well as two manipulation tasks on a real robot arm, and find that performance improvements continuously emerge over multiple iterations for novel tasks unseen during initial in-domain video model training.  We demonstrate that SILVR is robust in the absence of human-provided ground-truth reward functions or expert-quality demonstrations, and is preferable to alternate approaches that utilize online experience in terms of performance and sample efficiency.

Published as a conference paper at ICLR 2026
SELF-IMPROVING LoOPS FOR VISUAL ROBOTIC PLANNING
Calvin Luo" Zilai Zeng" " Minnxi Jia', Yilun Du², Chen Sun
Brown University, 'Harvard University
ABSTRACT
Video generative models trained on expert demonstrations have been utilized as performant text-conditioned visual planners for solving robotic tasks. However. generalization to unseen tasks remains a challenge. Whereas improved gener- alization may be facilitated by leveraging learned prior knowledge from addi- tional pre-collected offline data sources, such as web-scale video datasets, in the era of experience we aim to design agents that can continuously improve in online manner from self-collected behaviors. In this work we thus pro- pose the Self-Improving Loops for Visual Robotie Planning (SILVR), where an in in-domain video model iteratively updates itself self- trajectories, and steadily improves its performance for a specified task of interest. We ap- ply SILVR to diverse suite of World tasks, well tasks on a real robot arm, and find that performance improvements continuously emerge multiple iterations for novel tasks unseen during initial in-domain video model training. We demonstrate that SILVR is robust in the absence of human-provided ground-truth reward functions or expert-quality demonstrations, and is preferable to alternate approaches that utilize online experience in terms of performance and sample efficiency. Visualizations and code are provided diffusio wer.fusionssupee supeeiiiissss..nn github. io /o/silvr...
manipulation
INTRODUCTION
Advancements video generative modeling capabilities have directly led to their increased utiliza- tion visual planners for robotic applications (Du et al., 2024b; Yang et al., 2023b; Ko et al., 2024; Liang et al., 2024). The synthesized visual plan, in the form of video frames generated with text conditioning, can be translated into executable actions via inverse dynamics models (IDMs). Intu- itively, the data on which the video generative models and the IDMs are trained can greatly impact robotic performance and generalization. When explicitly optimized on in-domain examples of ex- pert behavior, such visual planners are able to synthesize successful plans for solving demonstrated tasks in a robust manner. However, for arbitrary robotic behaviors, expert-quality demonstrations may not be readily available, and collection may be prohibitively expensive. It is thus worthwhile to investigate how visual planning can automatically adapt and generalize novel tasks of interest.
Recent work has investigated how base generalization performance for visual planning can be im- proved by integrating knowledge from large-scale datasets of text and video collected from the inter- net. Adapt2Act (Luo et al., 2025) creates a powerful, generalizable, text-conditioned visual planner by combining large-scale model pretrained on web-scale video data with video model trained on small set of in-domain demonstrations via score composition. At high level, the adapted video model draws upon large-scale motion priors and powerful zero-shot text conditioning capabilities from the web-pretrained video model to facilitate generalization. Simultaneously. it can leverage the in-domain video model to better generate visual plans that respect the anvironment-speciffe visual characteristics and dynamics of the robotic setting. The result is adapted video model that can generate aedommin-aapparing plans for novel, unseen tasks conditioned on natural language.
Despite extending the amount of data utilized for visual planning internet-level, the model still only has access to purely offline data. In the era of experience, we aim to design agents that can continuously improve from self-collected behaviors and feedback. In such a way, the agent can break free beyond the limits of offline data and learn by itself to refine performance on a specified task
*: Equal contribution. Correspondence to: calvin Iuo @ browh.owoeeoo edu and zilai zeng @ brown.edu.

---

Published as a conference paper at ICLR 2026
Pre-training
Self-Improving Loops for Visual Robotic Planning (SILVR)
Interne-Scale Vilcos
(4) Adoptation Pitstoning
Co
Plub
lef?
"Paak she red cup"
() Adoptation
Fieaerol
Video Prior
General Text Video Sgeneral
In-Domain Vadlo
Inverse Dynamies fuuloo,o.u
Fihering
EnV Visual Plonner
(2) Vinull Plan Rollont
Figure I: SILVR Framework. SILVR has access to two pretrained video generative models (left): one pretrained generally on internet-scale data and another pretrained on general set of in-domain demonstrations. By default, SILVR uses the in in-domain video model as a visual planner, which when utilized interact with the environment, is able to achieve successful trajectories even for initially unseen tasks. These trajectories are then iteratively fed back to finetune the in-domain model (right), thus improving the overall quality of future visual planning as whole through self-collected online experience. SILVR can optionally incorporate internet-scale pretrained video models as prior, which particularly improves performance in the case real-world robotic experiments.
of interest. We therefore propose Self-Improving Loops for Visual Robotic Planning (SILVR), where the generated video iteratively self-improves with online experience, particularly with respect to behaviors previously unseen in the initial dataset of environment demonstrations. As shown in Figure 1, each loop adapts the video generative model (and optionally, the IDM) with environment- grounded data collected by the robotic agents following their own-generated visual plans.
SILVR utilizes sparse reward signal filter online experience for further finetuning of the visual planner, however, it is quite natural to consider alternative methods beyond visual planning or direct finetuning of the generative model. In our experiments we demonstrate that visual planning supe- rior to direct behavior cloning in both initial generalization performance as well as self-improvemmnt capability, given the same amount of offline data and online experience. Furthermore, the final re- sulting visual planner can be distilled into a lightweight policy via behavior cloning for both fast and performant decision- making.. We further showcase how SILVR is more sample efficient than rein- forcement learning finetuning approaches, making it more applicable SILVR is also robust to the quality of the sparse reward signal; rather than requiring a human-defined ground-truth success function, demonstrate that iterative improvements still arise when utilizing a pretrained vision-language model (VLM) to score experience based on the task descriptions.
weal-world robotic settings.
To summarize, SILVR enables superior sample-efficient seff-improvemeent over initially unseen tasks through visual planning, naturally integrating internet-scale pretrained video priors over text- alignment and motion when necessary, in comparison regular cetion-prediction policies. During deployment, a policy can be "distilled" from SILVR's components for fast inference, showcasing how SILVR facilitates effective generalization and self-improvement for robotic tasks without sac- rificing final execution speed.
We perform extensive evaluations of SILVR on the MetaWorld task suite, focusing on novel tasks unseen during initial training of the in-domain video model. We discover that the success rate of following visual plans synthesized through SILVR indeed continuously improves, by as much as 285% over 10 iterations. We also apply SILVR to a real-world robot arm for two distinct manipula- tion tasks: selecting and pushing colored object, and selecting and opening a colored drawer. We show how SILVR also naturally enables the incorporation of priors from internet-scale pretrained video generative models to facilitate task and visual generalization in real-world visual settings and dynamics. We demonstrate that performance for color combinations unseen during the initial offline training improves over multiple iterations through SILVR.
2 RELATED WORK
Video Generation for Decision Making. Recent advances in video models have achieved unprece- dented visual quality and physical fidelity for video synthesis (Guo et al., 2023; Yang al., 2024b;
2

---

Published as a conference paper at ICLR 2026
Brooks et al., 2024; Veo-Team al., 2024; Wang al., 2025). This has demonstrated promise in summarizing world dynamics through videos (Yang et al., 2024a; Bruce et al., 2024) and has inspired the application of video models to solving decision-making problems (Escontrela et al., 2023; Du et al., 2024a; Yang et al., 2023b; McCarthy et al., 2024; Liang et 2024). Prior works have utilized video generative models as reward functions (Luo et al., 2024; Escontrela et al., 2023; Huang et al., 2023), dynamics models (Yang et al., 2023b; Bruce et al., 2024; Valevski et al., 2024), and pixel-based planners (Ko et al., 2024; Ajay et al., 2023; Du al., 2024b; Zhou et al., 2024). As in UniPi (Du et al., 2024b), we employ video models to predict cext-conditioned visual plans that depict future outcomes, which are subsequently translated into actions via inverse dynamics. While the performance of such visual planners may often be limited by their offline pretraining data, our approach allows iterative improvement by learning from online environment interactions.
Self-Improvin Generative Models. Continuously improving by learning from self- aeff-prouued cu- mulative experience is an essential capability of intelligent agents. Prior work has demonstrated the effectiveness of improving LLMs with their self-generated outputs (Yu et al., 2024; Tian et al., 2024; Huang et al., 2022), where the LLM can serve as its own reward function (Yuan et al., 2024) for preference optimization data synthesizer (Patel et al., 2024) for supervised finetuning. However, a similar self-improvement recipe for video generation models remains underexplored. Most rele- vant to our work, VideoAgent (Soni et al., 2024) refines video generation through self-conditioning consistency and feedback from a VLM, and collects the successful plan rollouts for finetuning. We instead base our improvement loop on self-adaptation, where we leverage internet-scale video priors to synthesize improved visual plans for tasks unseen during initial in-domain training. Furthermore, our approach can still achieve celf-improvement even with an initial model trained on suboptimal data and a notable relaxation on filtering requirements for finetuning data.
Reinforcement Learning Finetuning of Behavior Cloning Policies. Behavior cloning approaches such as Diffusion Policy (Chi et al., 2023), which implements a policy as diffusion model trained on offline-collectdd experience, are a performant approach for decision-making. There have been numerous approaches for finetuning pretrained diffusion policies with respect to online experience and rewards. ResIP (Ankile et al., 2025) utilizes a frozen diffusion policy model to propose action predictions, and learns policy top using reinforcement learning that transforms it into more curate action to perform in the environment. DPPO (Ren et al., 2024) treats the sampling procedure of diffusion policy an internal Markov Decision Process, and explicitly finetunes the weights of the pretrained diffusion policy with respect to achieved rewards. DSRL (Wagenmaker et al., 2025) utilizes frozen diffusion policy in deterministic fashion to map noise samples action samples, and learns a noise selector through reinforcement learning. In this work, we show that SILVR is more sample efficient than reinforcement learning finetuning of behavior cloning policies, and can achieve faster iterative improvements with respect to online experience.
3 METHOD
We introduce the Self-Improving Loop for Visual Robotic Planning (SILVR), in which video gen- crative model initially trained on a general set of in-domain demonstrations iteratively improves its visual planning performance for a particular task of interest in a self-adaptive manner. In Section 3.2, we describe how a small in-domain video model can be integrated with a generally pretrained text- ti-video model to produce strong, generalizable in-domain visual planner for real-world visual settings. Finally, in Section 3.3, we demonstrate how SILVR bootstraps an in-domain video model into high-performing visual planner for solving novel robotic control task through iteratively fine-tuning on celf-collected experience.
3.1
VIDEO MODELS AS VISUAL PLANNERS
Synthesizing visual plan in imagination and then executing it by converting it into actions is an intuitive and effective way to utilize video generative models for decision making. Prior work has applied text-guided video generation successfully for task planning (Du et al., 2024a;b; Ajay et al., 2023; Luo et al., 2025), across a variety of robot configurations and environment settings.
the UniPi framework (Du et al., 2024a), in which text-to-video model used to synthesize a rext-conditioned sequence of future frames as a task plan. To physically realize the plan, we use a separately trained inverse dynamics model (IDM) to
Specifically, base our implementation
3

---

Published as a conference paper at ICLR 2026
Algorithm Self- lelf-lmpoo Loops for Visual Robotic Planning (SILVR)
Initial in-domain video model cø, Inverse dynamics model JIDM. Frozen internet- pretrained video prior Egeneral, Number of iterations K, Number of rollouts per iteration N, Envi- ronment env, Task prompt g, In-domain initial training data Pini Data filter fr Output: Sclf improvedd in in-domain video model ég, Optional distilled policy
Input:
1: Eg +-+-- E0
2: D <- Dini or Φ
Initialize finetuning data with Dini or an empty set
3: for i = -1,,.., K do
4:
Daett + Φ
5:
č Adaptation(eo. Cyeneral: 9)
6:
for j 1, 1,.... N do
7:
resere))
8:
Deett t Diett U UiViissaaLPPaminnngloooootttttn fIDM))
9: end for
10: D - D Usell
Finetune in-domain model &g with accumulated data D
fwu can be optionally finetuned
12: end for
Optionally distill Eg into a lightweight policy
14: return ép.
translate consecutive pairs of visual frames into executable robotic actions, which are then directly performed in interaction with the environment. Visual planning offers the practitioner flexible com- putational tradeoffs; at high level, replanning often incurs high computational cost but generally increases accurate plan following, whereas replanning infrequently is cheap but may suffer from error compounding. In this work, we focus on how such video generative model can generalize and self- adlf- novel task of interest through online self-collected experience.
3.2 INVERSE PROBABILISTIC ADAPTATION
Prior work (Luo al., 2025) has investigated how in-domain demonstration data can best be in- tegrated with large-scale pretrained video models for generalizable visual planning; in this work we leverage similar insights to successfully integrate on-the-fly experience into visual planners for iterative self- ielroemproveent.. Inverse Probabilistic Adaptation (Luo et al., 2025; Yang et al., 2023a) (IPA) a training-free approach that adapts generally pretrained text-to-video models for domain- specific video generation. To perform adaptation, the score predicted by an in-domain video model <e trained on small sample demonstrations is composed with the score prediction a web-scale pretrained model Eganeral during the sampling procedure, depicted in the function below:
Eny Egeneral (Te, t) (genneral (Te. text) (Co text) Cgeneral (Te, 8)
(1)
where y is the prior strength. and a is the guidance scale of conectoodiiiiingg. Intuitively, the small in-domain text-to-video model serves as probabilistic knowledge prior that guides the generation process of the small in-domain model during sampling. Prior work (Luo et al., 2025) has found that visual planner constructed through IPA exhibits both strong generalization capability and in-domain understanding; it is able to synthesize performant visual plans that appear in-domain even for novel tasks unseen during video model training. This may stem from the fact that IPA utilizes the large- scale pretrained model, which inherently has stronger cott-onditioned generalization, as the main denoiser. While Luo al. (2025) based their conclusions experiments in simulated environments, we believe the true promise of web-scale pretrained models lies in their powerful prior for real-world generalization scenarios, demonstrated by our Panda arm object manipulation evaluations.
3.3 SELF-IMPROVING LOOPS FOR VISUAL ROBOTIC PLANNING
For visual planning approaches, task performance is fundamentally fixed function of the video models used, and by extension, the data observed. Even when utilizing IPA, which can improve text-conditioned generalization novel tasks by effectively increasing the amount of offline data
4

---

Published as a conference paper at ICLR 2026
utilized to internet-scale, performance is set after adaptation. As result, in this paper. we wish to design agents that can not only leverage offline data as a helpful prior for generalization, but also extend beyond it to continuously improve from self-collected online experience data.
We therefore propose Self-Improo leproving Loops for Visual Robotic Planning (SILVR), a framework that combines offline data with online experience create visual planner that iteratively improves for a particular task of interest. SILVR is initialized with an in-domain video model cg pre-trained on set of task demonstrations within the environment. In each iteration, the in-domain video model is optionally integrated with a large-scale pretrained video model Egenenl through IPA. The video model is utilized as a visual planner interact with the environment and solve tasks not necessarily observed in the initial training stage; in SILVR, the trajectories collected through this interaction are used for further finetuning of the in-domain video model (as shown Algorithm 1). As the in-domain model adapts to its own self-collected experience from deployment on novel task, it improves its ability to solve that particular task over time. In this way, SILVR iteratively bootstraps an in-domain video model into a strong visual planner for novel task of interest through a self- adapting improvement cycle.
We demonstrate that the visual planning approach enables strong self-improvement in a virtu- ous loop in a sample efficient manner, compared to reinforcement learning finetuning of behav- ior cloning models. We further stress-test framework through ablations, and demonstrate that ground-truth human-defined reward functions can be replaced with an automated VLM success sig- nal without inhibiting iterative eef--imprveement from occurring, and that SILVR can handle sub- optimal initial data quality We find that SILVR is a robust approach for iteratively adapting to a task through effective utilization of both offline data and online experience, and reduces requirements on human-supplied components both in terms of feedback and demonstration quality.
Whereas visual planning demonstrates strong self-improvement capabilities, and can flexibly inte- grate in benefits from large-scale pretrained video models, it can be slow in execution compared to direct policy approaches. However, after applying SILVR, the final video planning components can be distilled into a lightweight policy through behavior cloning for future inference. We demonstrate in our experiments that such final distilled policy achieves higher performance than applying self- improvement techniques to such a policy from the start, and can even demonstrate slight improve- ment over the final performance of the visual planner teacher. Thus, SILVR enables a successful balance between visual planning for improved and sample-efficien utilization of online experience, and a lightweight distilled policy for reactive inference during deployment.
4 EXPERIMENTS
We investigate how SILVR can improve an in-domain video model initially trained on a limited set of demonstrations and tasks to further solve novel robotic control tasks through self-collected experience. We focus on two main robot settings to evaluate SILVR: the MetaWorld-v2 (Yu et al., 2020) simulated environment, and a real-world Franka Emika Panda robot arm. We describe our experimental setup for each environment, as well as different design decisions considered.
4.1 EXPERIMENTAL SETUP AND EVALUATION
Synthetic Environment: MetaWorld encompasses a wide selection of tasks, allowing us to thor- oughly assess visual planning performance trends through SILVR for many choices of held-out novel tasks. Furthermore, MetaWorld provides ground- truth success evaluations, enabling strictly quanti- tative comparisons on task performance as well as iterative improvement abilities. For MetaWorld experiments, we first collect 25 demonstrations from 8 different tasks (denoted with an asterisk in Table A2) for initial in-domain video model and inverse dynamics model training. Subsequently, instantiate SILVR with both models on 12 unseen tasks for self-imprrvemmen. In each SILVR iteration, we collect 30 trajectories rendered from the environment via visual planning, and finetune both in-domain video model and inverse dynamics model using filtered successful data. Due the sim-to-real gap. we disable the use of internet-scale video prior in Meta World by default.
Real-World Environment: Deploying SILVR robot arm in the real world demonstrates the practicality of the approach, as well as tests robustness to real-world confounding factors such as lighting conditions. In one experiment, we utilize Franka Emika Panda robot arm for the task
5

---

Published as a conference paper at ICLR 2026
Iteration
2
4
DSRL (w/ GT Filter)
(11.7)
(..6))
(+0.9)
7.5 (33.4)
7.7 (+3,4)
BCIL (w/ GT Filter)
5.6
12.3 (+2.3)
(+1.6)
23.3 (=0.4)
23.2 (4009)
SILVR (w/ GT Filter)
14.7 (+-0.6)
27.7 (7
33.5
(42.2)
43.5 (+22.6)
44.2 (#:4.5)
SILVR (w/ VLM Filter)
17.0 (:0.6)
24.4 (=0.))
28.7 (+2.8)
34,4 (#1.4)
38.4 (#1.3)
SILVR-Distiled DP
44.2 (+4.5) - 49.2 (+3.4)
Table 1: SILVR Results on MetaWorld. We report the average performance over 12 unseen Meta- World tasks for SILVR and all baseline methods, each aggregated over three seeds. We also provide the performance of diffusion policy distilled from the video model from the last SILVR iteration, denoted as "SitlR-Distilled DP". SILVR outperforms all baselines by large margin since Itera- tion Furthermore, SELVR-Distiledd DP achieves the best overall performance.
"Push Orange Cup"
"Open Yellow Drawer"
"Close Drawer"
Iteration 0
Iteration
Figure 2: Qualitative results visual plans improvement. We illustrate visual plans for a variety of tasks and settings at Iteration 0 (top) and Iteration 2 (bottom) with random initial object locations. Although the visual plan Iteration 0 renders blurry objects and fails to complete the specified tasks, our approach synthesizes the correct visual plan (with slight color drift) after two SILVR iterations.
of pushing cups specified by a user-provided text prompt. In contrast to the Meta setups, where each task of interest has its own distinct visual setting, we construct the cup experiment as a consistent scene setting of 3 differently colored cups (Figure 1). Success is then measured in terms of whether the robot arm can accurately locate specified color cup and push it forward. To test generalization, conditioned on natural language, we evaluate successful planning and execution performance on unseen cup colors. In practice, we use a set of four colors (red, green, blue, pink) for in-domain training and two novel colors for testing generalization (orange, purple). This translates to 12 possible unique tasks formed from combinations the seen colors, and we train our in-domain video model with 10 human- temo--etleeperrteedd demonstrations of each for total of 120 training videos. Then, generalization evaluation calculated as an average over 5 rollouts for every possible pair combination of the seen color set combined with the novel color, for a total of 30 videos. For both novel colors, we initialize SILVR using the same pretrained in-domain video model. In each SILVR iteration, we combine self-collected data with the initial demonstrations for in-domain finetuning.
In a second real-robot experiment, we utilize the Panda arm to select and open drawer specified via a user-prue eser-provided text prompt. The scene is constructed as two distinctly colored closed drawers, where the robot is prompted with one particular color and expected to open its corresponding drawer. We use set of three colors (red, green, blue) for in-domain training and one novel color (yellow) for testing generalization. With 24 possible drawer placement combinations for each ordered pair of seen colors, of which there are six, this amounts to a total of 1 144 human-teke ceman-teleoperated demonstration training videos. Consistent with the cup pushing experiment, for evaluation; therefore, performance is calculated as an average over 12 rollouts for every possible pairing of the novel color with
use half the possible combinations
color, for total of 36 self-collected trajectories per iteration.
For both real-robot experiments, success is judged by a human for evaluation. The same success signal is also used to perform data filtering the rollouts. We study the impact of data filtering in Section 4.5, and enable adaptation with internet video priors through IPA by default.
Implementation Details: We implement our in-domain video model based AVDC (Ko al., 2024), with an added cross-attention layer to each level of the denoising U-Net to further improve text-conditioning capabilities. We train in-domain video models to predict 8 future frames condi-

---

Published as a conference paper at ICLR 2026
MetaWorld (Sim)
Push Cup (Real)
Open Drawer (Real)
455
Succoss Rate (%)
Success Rate (96
Success Rate (%)
757
277
322
500
50
600
809
129
10
324
123
2
2
Iteration
lteration
Iteration
SILVR (w/o Video Prior)
SILVR (w/ AnimateDiff)
BCIL
Figure 3: SILVR Results in comparison to Behavior Cloning Improvement Loop (BCIL). We report the average performance over 12 unseen Meta tasks, as well as novel pushing and drawer opening tasks for Panda arm experiments across several iterations of self-improvement (x-axis). Numbers in the graph correspond to success rate achieved (y-axis).
Metal (10 Iterations)
SILVR (Iteration 9)
SLLVR-Distilld DP (Lteration 9*)
Aig Taak Performance
262
90-
77A
Iteration
Deer Chose
Peoee Close
Honde Pess
Window Open
Place Stde
Figure 4: SILVR results on MetaWorld for 10 iterations. We report effects of training SILVR on an extended amount of iterations. On the left plot, we show that performance continues to monotonically increase, but with diminishing improvements and effective saturation past iteration 5. On the middle and right plots we visualize a comparison between the final iteration visual planner against its distilled student BC policy from the visual planner across 6 tasks, where observe that certain tasks actually improve after distillation.
tioned on the current observation and task prompt, with frame skip of 1 for MetaWorld and 16 for real-robot experiments. For the large-scale pretrained text-to-video model, we use AnimateD-- Wed-idd10n (Bain et al., 2021). Each iteration of SILVR finetunes the in-domain video model for 10,000 steps with learning rate of le-6 on MetaWorld and le-5 on Panda Arm drawer opening tasks, and 8,000 steps with learn- ing rate of 2e-5 on Panda Arm pushing tasks. We investigate two IDM implementations; one that follows prior work (Luo et al., 2025) that implements the IDM as MLP network that takes in the outputs of a VC-1 (Majumdar et al., 2023) encoder model, finetuned on in-domain demonstrations. Whereas the MI.P-IDM was sufficient for real-world experiments, we found improved performance when using Diffusion- IDM (DIDM), which is implemented as Diffusion Policy (Chi et al., 2023) with an additional goal frame provided as conditioning. for simulated settings. Additional details on IDM design and hyperparameters are provided in Section B. For MetaWorld experiments, the DIDM is iteratively finetuned on the online collected experience along with the video model.
iff (Guo al., 2023) (~2B parameters), which is pretrained
4.2 SILVR VIA FILTERED FINETUNING
We report incremental visual planning results for MetaWorld through five SILVR iterations, against two self-improving baseline methods built off of Diffusion Policy (DP) (Chi et al., 2023): DSRL (Wagenmaker et al., 2025) and Behavior Cloning Improvement Loop (denoted as "BCIL"). We initialize DSRL and BCIL with Diffusion Policy trained on the same in-domain data as used for in-domain video model and inverse dynamics model pre-training in SILVR. We utilize the ground- truth task success signal to filter the same amount self-collected data per iteration and finetune models with only successful trajectories.
7

---

Published as a conference paper at ICLR 2026
Success Rate (8)
304 763
Success Rate
Iteration
Iteration
Ground-Truthh( (GT)
GPTS
GeminiQ 5-PPo
No Filneing
Ground-Trath (GT)
No Filwiing
(a) Meta World (over 12 unseen tasks)
(b) Panda Arm Cup Push (Orange)
Figure 5: Ablations on data filtering. We compare the effect filtering has on success rate (y- axis) across iterations of finetuning (x-axis), on both MetaWorka (5a) and Panda arm (5b) setups. On MetaWorld (left plot), we further report accuracy when filtering is performed by VLM. We observe SILVR consistently improves task even without access ground-truth filtering signals.
In Table 1, the performance is averaged over 12 unseen MetaWorld tasks and aggregated over seeds. Despite the initial in-domain data being shared across all methods, observe that the visual planning approach achieves better performance than DP-hased approaches at Iteration O, laying solid foundations for subsequent improvement. Compared DP, which learns to map ob- servations to actions directly, visual planning decouples dynamics modeling from action prediction. We hypothesize that the separately learned environment visual dynamics easier transfer when solving novel task, leading to stronger base generalization performance through visual planning. While DSRL fails to improve and BCIL quickly saturates at a low success rate after several itera- tions, SILVR continuously improves and consistently outperforms the baselines by large margin, demonstrating its high sample efficiency.
4.3 SILVR WITH INTERNET- SCALE VIDEO PRIOR
One important observation from Table is that the generalization capability of the visual planner in solving novel tasks has a fundamental impact on self-improving dynamics. When the task of interest is exceptionally challenging or involves confounding factors, the capacity of the in-domain video model alone might not be sufficient to elicit self-improving behaviors. In the real-world exper- imental setup, we adapt our in-ommiin model with an dnternet-pretriinee video prior, AnimateDiff, to further strengthen the zero-shot generalization and adaptability of the visual planner. In Figure 2, we qualitatively illustrate the improvements of generated visual plans after two SILVR iterations in combination with AnimateDiff. Without observing any demonstrations of the specified tasks at Iteration O, the visual planner can synthesize plans with blurry objects where the robot arm exccutes the task incorrectly. On the other hand, two iterations of SILVR not only improve the clarity of the visual plans, but also demonstrate successful task completion behaviors the same initial layout.
In the middle plot of Figure 3, report the average performance of pushing cups in two unseen colors, orange and purple, aggregated over 30 rollouts per iteration. We discover that SILVR consis- tently improves over iterations when adapting with AnimateDiff. In the rightmost plot of Figure 3, we provide the SILVR results across iterations on opening a novel colored drawer, averaged over 36 rollouts per iteration, and find that SILVR bootstraps initial visual planning performance with the help of the internet video prior. However, in both real-world experiments, the visual planner struggles to improve or even continuously deteriorates without video prior. This highlights the im- portance of internet video prior in self-improvie behavior under real-world setups with increased visual complexity and task difficulty. simulated environments like Meta World, we observe that the self-im imprrviniinn trend occurs regardless of whether internet video priors are utilized or not, indicating that the benefits of utilizing internet priors may diminish when there a substantial sim-to-real gap.
Consistent with our findings MetaWorld, action- predictive behavior cloning has a lower base gen- eralization performance and slower eeff-improvement trend compared with SILVR on real robots, as shown in the BCIL curves of the middle and right plots in Figure 3. This highlights key benefit of SILVR: its ability to seamlessly utilize internet-pretraiigg video prior for improved text-conditioned generalization and sample-efficien online improvement in real-world robotic settings.
8

---

Published as a conference paper at ICLR 2026
4.4 SILVR SATURATION AND DISTILLATION
To further understand the limits of self-improving behavior over iterations, we provide MetaWorld results for 10 SILVR iterations in Figure 4. We find that SILVR saturates at Iteration 5 with marginal gains in the following iterations. We hypothesize that this may potentially arise from discovered local minima task-specific strategy, where similar experiences are collected until saturation. We believe that a possible mitigation is to introduce the notion of "exploration" into the visual planning framework to avoid "unimodal" behavior. Such research may look into how to extract out more diverse plans from the video planner by exploiting the stochastic nature of visual generative models. We leave this investigation as promising future work.
While visual planning approaches demonstrate strong performance in task generalization, their in- ference speed is bottlenecked by the video generation process, which can be prohibitively expensive for downstream applications. To mitigate this, distill the video model from the last SILVR itera- tion into a lightweight diffusion policy, As shown in Figure 4, the SlLst-dis diffusion policy at Iteration 9 significantly outperforms the BCIL baseline and achieves the best overall performance. A slight performance increase after distillation a trend consistently observed across iterations, such demonstrated in Iteration 4 of Table 1. This further demonstrates that SILVR not only cels in sample-efficient task adaptation, but also supports high inference efficiency for downstream deployment via distillation.
4.5 IMPACT OF DATA FILTERING SIGNALS ON SILVR
While utilizing self-collected data is promising approach for scalable self-improvement, filtering collected experience often requires some level of human intervention, whether through manually de- termining successful trajectories or designing heuristic for quality control. We therefore investigate how different filtering techniques affect SILVR performance, or if SILVR is robust to such design decisions. For both MetaWorld and Panda Arm settings, we compare between using ground-truth or human-evaluated notion of success to filter what trajectories the in-domain model is finetuned on, against not using any filtering at all and utilizing all achieved trajectories irregardless of out- come. While ground-truth success signals can often be inaccessible, we investigate whether the current state-of-thh-ar VLMs, GPT-5 and Gemini-2.5-Pro, can provide useful task success signals and serve as robust alternative to ground-truth signals on MetaWorld.
In Figure 5a, we observe that both GPT-5 and Gemini-2.5-PP can still enable self-improving be- havior across SILVR iterations when serving as task success judge, in which Gemini achieves the best performance among all VLM filters. We also discover that without any data filtering, the im- provement over each SILVR iteration appears to be marginal compared to the filtered setup. On the other hand, in Figure 5b, for the Panda arm, we observe that no filtering still facilitates continuous improvement over every iteration through SILVR when adapted with the internet video prior. This is an encouraging finding, as it suggests that even for settings where manual curation of experience expensive, self-improvement can still occur. We attribute this property to score composition (Luo et al., 2025; Yang et al., 2023a); suboptimal demonstrations can still communicate useful informa- tion the in-domain model, such as visuals, valid motions, and interaction dynamics of the specific deployment environment that when combined with an internet-pretrane a final composed output plan can be both performant as well as appearing in-domain. As such, the success rate may still improve from iteration to iteration without filtering, as the in-domain model improves its modeling of environment dynamics and visuals, even from suboptimal self-collected experience over iterations.
video model can result in
4.6 SILVR WITH SUBOPTIMAL DATA
Visual planners are usually trained explicitly on expert in-domain demonstrations, which commu- nicate not only wwwtrrommnttspeccine visual characteristics, physics, and interaction dynamics to the generative model during optimization, but also notion of success and optimal behavior. However, for arbitrary environments, such expert-quality in-domain data can be expensive to collect and curate at scale. On the other hand, suboptimal demonstration data, such as utilizing random actions during the collection procedure, may generally be cheaper to gather; however, training on a large dataset of low-quality data may not result in a performant visual planning model capable of generating plans worth following. A natural question is how robust SILVR is to initialization data, or whether a performant video planner can still be created when only suboptimal demonstrations available.

---

Published as a conference paper at ICLR 2026
Wordd
Expert Init. (Iteration 4)
Suboptimal Init. (lleration 4)
Arg Took Performance
S8V5 Bldeet
25
Iteration
Fracce Close
Texcst Open
Pate-kkle
Diawenn C Kese
Roach Haal
Button Press Hill
Figure 6: Ablation studies on in-domain data quality. On the left plot, we report SILVR perfor- mance averaged over 12 novel tasks and 3 seeds when the provided initial demonstrations of seen tasks expert-quality or suboptimal. We find that SILVR successfully self-improves despite sub- optimal data initialization. In the middle and rightmost plots, we visualize tasks that have the most distinctive performance between expert and suboptimal SILVR initializations at the final iteration.
In our setting, we construct suboptimal data as simulated trajectories where 70% of the time a ran- dom action is selected and 30% an expert action is utilized. As a consequence of this interaction procedure, the resulting trajectories are unable to successfully solve complex tasks. In MetaWorkd find that SILVR still demonstrates continuously improving behavior when initialized with subop- timal data, as shown in the leftmost plot of Figure 6, highlighting the robustness of SILVR to initial data quality. Additionally, suboptimal setup, and report their performance on Iteration 4 in the middle and rightmost plots of Figure 6. We find that Faucet Open, Faucet Close and Plate Slide benefit most from initialization with expert demonstrations, indicating that the skills acquired from seen tasks can be essentially useful when solving these novel tasks. Meanwhile, Drawer Close, Reach Wall, Button Press Wall can benefit more from random exploration than expert actions from specific seen tasks, leading to stronger performance when the in-domain model initially trained on suboptimal demonstrations.
select 6 tasks whose performance differs most between the expert and
CONCLUSION AND FUTURE WORK
Conclusion. We present SILVR, a self- oof--imprveeentt loop for solving novel robotic tasks via visual planning. Initialized from an in in-domain video model pretrained on a small general set of demonstra- tions, SILVR iteratively updates the visual planner for novel task of interest through self-collected online experience. Compared to equivalent behavior cloning setups, or utilizing online experience through reinforcement learning finetuning, find that SILVR achieves superior self-improvemenn capabilitics and demonstrates sample efficiency. We successfully apply SILVR as a self-improving visual planner not only for synthetic environments, but also on robot arms deployed in the real world.
Limitations and Future Work. SILVR implicitly assumes that the initial in-domain model, option- ally adapted with preerree- video model, achieves reasonable success rate collect online experience and achieve self-im reff-improveent. This assumption may not hold when the novel task is too challenging. Additionally, the choice of preerrne- video model can pose trade-off on between video quality and computation cost. Whereas in this work we choose AnimateDifi (Guo al., 2023) as a large-scale pretrained video model with both reasonable generation quality as well as computational efficiency, more recent video generative models with enhanced visual quality can be further explored for improved visual planning performance on downstream robotic tasks.
Whereas SILVR achieves the best self rel--improvemett performance with some initial success rate on the novel task, the cold start problem may pose challenges. Just as exploration can help address the cold start problem for standard reinforcement learning, we believe that investigating how to improve exploration for the visual planning framework in a principled manner is a promising future direction.
Acknowledgments. This work is partially supported by Samsung and the U.S. National Science Foundation under Cooperative Agreement No. 2433429, Our research was conducted using compu- tational resources at the Center for Computation and Visualization at Brown University. We would like to thank Professors George Konidaris and Stefanie Tellex for their generous support for our real-robot experiments. We would also like to thank Shijie Wang and Zitian Tang for their help with the experiments. Calvin would like to thank Dennis Lee and Ally Koo, as well as Kayan Shih and her family, for all their kindness and hospitality during the paper writing process.
10

---

Published as a conference paper at ICLR 2026
6 REPRODUCIBILITY STATEMENT
To support reproducibility, all codebases utilized in this paper were modified versions of publicly available repositories. Furthermore, provide extensive details of hyperparameters and settings both in the main body of the text as well as in the Appendix.
REFERENCES
Anurag Ajay. Seungwook Han, Yilun Du, Shuang Li, Abhi Gupta, Tommi Jaakkola, Josh Tenen- baum, Leslie Kaelbling, Akash Srivastava, and Pulkit Agrawal. Compositional foundation models for hierarchical planning. In Conference Neural Information Processing Systems (NeurIPS), 2023. 3
Lars Ankile, Anthony Simeonov, Idan Shenfeld, Marcel Tome, and Pulkit Agrawal. From imitation to cettiemment-rssidue rl for precise assembly. In 2025 IEEE Interaational Conference on Robotics and Automation (ICRA), pp. 01-08. IEFF, 2025. 3
Max Bain, Arsha Nagrani, Gäl Varol, and Andrew Zisserman. Frozen in time: joint video and image encoder for end-to-end retrieval. In IEEE International Conference Computer Vision (ICCV), 2021. 7
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang. and Aditya Ramesh. Video generation models world simulators. OpenAl Blog, 2024. URL https: wwwpai..nnoooooooeoo..o.. wwwoonnnmoelseeeee world- Seiid-sime simulators.
Jake Bruce, Michael Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, Yusuf Aytar, Sarah Bechtle, Feryal M. P. Behbahani, Stephanie Chan, Nicolas Manfred Otto Heess, Lucy Gonzalez, Simon Osindero, Sherjil Ozair, Scott Reed, Jingwei Zhang, Konrad Zolna, Jeff Clune, Nando de Freitas, Satinder Singh, and Tim Rocktaschel. Genie: Generative interactive environments. arXiv preprint arXiv:2402. 15391, 2024. 3
Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics:
Science and Systems (RSS), 2023. 3, 7, 15
Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation. Advances in Neural Information Processing Systems, 36, 2024a. 3
Yilun Du, Sherry Yang, Pete Florence, Fei Xia, Ayzaan Wahid, brian ichter, Pierre Sermanet, Tianhe Yu, Pieter Abbeel, Joshua Tenenbaum, Leslie Pack Kaelbling, Andy Zeng, and Jonathan Tomp- son. Video language planning. In International Conference on Learning Representations (ICLR), 2024b. 3,3
Alejandro Escontrela, Ademi Adeniji, Wilson Yan, Ajay Jain, Xue Bin Peng, Ken Goldberg, Young- woon Lee, Danijar Hafner, and Pieter Abbeel. Video prediction models as rewards for reinforce- ment learning. In Conference on Neural Information Processing Systems (NeurIPS), 2023. 3
Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725. 2023. 2, 10, 15
Yuwei Guo, Ceyuan Yang. Anyi Rao, Maneesh Agrawala, Dahua Lin, and Bo Dai. SparseCtrl: adding sparse controls texi-to-video diffusion models. In European Conference on Computer Vision (ECCV), 2024. 15
Jiaxin Huang, S. Gu, Le Hou, Yuexin Wu, Xuezhi Wang. Hongkun Yu, and Jiawei Han. Large language models can self-improve. In Conference on Empirical Methods in Natural Language Processing, 2022.
11

---

Published as a conference paper at ICLR 2026
Tao Huang, Guangqi Jiang, Yanjie Ze, and Huazhe Xu. Diffusion reward: Learning rewards via conditional video diffusion. arXiv preprint arXiv:2312.1413,, 2023. 3
Po-Chen Ko, Jiayuan Mao, Yilun Du, Shao-Hua Sun, and Joshua B. Tenenbaum. Learning to act from actionless videos through dense correspondences. In International Conference on Learning Representations (ICLR), 2024. 1, 3, 6, 15, 16
Junbang Liang, Ruoshi Liu, Ege Ozguroglu, Sruthi Sudhakar, Achal Dave, Pavel Tokmakov, Shuran Song, and Carl Vondrick. Dreamitate: Real-world visuomotor policy learning via video genera- tion. arXiv preprint arXiv:2406. /6862, 2024. 1,
Calvin Luo, Mandy He, Zilai Zeng, and Chen Sun. Text-aware diffusion for policy learning. In Advances in Neural Information Processing Systems, volume 37, 2024. 3
Calvin Luo, Zilai Zeng, Yilun Du, and Chen Sun. Solving new tasks by adapting internet video knowledge. In The Thirteenth International Conference on Learning Representations, 2025. URL https:://nn http:///ppeeeeeeeeeeeeeeee review net/ / forum? d==011BR4 j.Y. 1, 3, 4, 7,9, 14, 15, 16
Arjun Majumdar, Karmesh Yadav, Sergio Arnaud, Jason Ma, Claire Chen, Sneha Silwal, Aryan Jain, Vincent-Pierre Berges, Tingfan Wu, Jay Vakil, Pieter Abbeel, Jitendra Malik, Dhruy Batra, Yixin Lin, Oleksandr Maksymets, Aravind Rajeswaran, and Franziska Meier. Where are we in the search for an artificial visual cortex for embodied intelligence? In Conference on Neural Information Processing Systems (NeurlPS), 2023. 7, 14
Robert McCarthy, Daniel C.H. Tan, Dominik Schmidt, Fernando Acero, Nathan Herr, Yilun Du, Thomas George Thuruthel, and Zhibin Li. Towards generalist robot learning from internet video: A survey. arXiv preprint arXiv:2404. 19664, 2024.
Ajay Patel, Markus Hofmarcher, Claudiu Leoveanu-Condreh, Marius-Constane Dinu, Chris Callison- Callison-Burch, and Sepp Hochreiter. Large language models can self-improve web agent tasks. arXiv, 2405.20309, 2024. 3
Allen Ren, Justin Lidard, Lars L. Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majum- dar, Benjamin Burchfiel, Hongkai Dai, and Max Simehowitz. Diffusion policy policy optimiza- tion. arXiv preprint arXiv:2409.0000 00588, 2024. 3
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- lional Conference on Learning Representations (ICLR), 2021.
Achint Soni, Sreyas Venkataraman, Abhranil Chandra, Sebastian Fischmeister, Percy Liang, Bo Dai, and Sherry Yang. Videoagent: Self-improving video generation. arXiv preprint arXiv:2410.10070 2024.
Ye Tian, Baolin Peng, Linfeng Song, Lifeng Jin, Dian Yu, Lei Han, Haitao Mi, and Dong Yu. Toward self- ielf-immrovementt of LLMs via imagination, searching, and criticizing. In Conference on Neural Information Processing Systems (NeurIPS), 2024. 3
Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. Diffusion models are real-time game engines, arXiv preprint arXiv:2408. 14837, 2024.7
Veo-Team, :, Agrim Gupta, Ali Razavi, Andeep Toor, Ankush Gupta, Dumitru Erhan, Eleni Shaw, Eric Lau, Frank Belletti, Gabe Barth-Maron, Gregory Shaw, Hakan Erdogan, Hakim Sidahmed, Henna Nandwani, Hernan Moraldo, Hyunjik Kim, Irina Blok, Jeff Donahue, José Lezama, Kory Mathewson, Kurtis David, Matthieu Kim Lorrain, Mare van Zee, Medhini Narasimhan, Miaosen Wang, Mohammad Babaeizadeh, Nelly Papalampidi, Nick Pezzotti, Nilpa Jha, Parker Barnes, Pieter-Jan Kindermans, Rachel Hornung, Ruben Villegas, Ryan Poplin, Salah Zaiem, Sander Dieleman, Sayna Ebrahimi, Scott Wisdom, Serena Zhang, Shlomi Fruchter, Signe Norly, Weizhe Hua, Xinchen Yan, Yuqing Du, and Yutian Chen. Veo 2024. URL. https://p./ deepmi nd. nd. google e /ttchhccoollll logies///aeoooo..o... peries/veel /vee/////// /eeo 2
Andrew Wagenmaker, Mitsuhiko Nakamoto, Yunchu Zhang, Seohong Park, Waleed Yagoub, Anusha Nagabandi, Abhishek Gupta, and Sergey Levine. Steering your diffusion policy with latent space reinforcement learning. arXiv preprint arXiv:2506. 15799, 2025. 3, 7,
12

---

Published as a conference paper at ICLR 2026
Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang. Jianyuan Zeng, Jiayu Wang, Jingfeng Zhang, Jingren Zhou, Jinkai Wang. Jixuan Chen. Kai Zhu, Kang Zhao, Keyu Yan. Lianghua Huang, Mengyang Feng, Ningyi Zhang, Pan- deng Li, Pingyu Wu, Ruihang Chu, Ruili Feng, Shiwei 7hang, Siyang Sun, Tao Fang, Tianxing Wang, Tianyi Gui, Tingyu Weng. Tong Shen, Wei Lin, Wei Wang. Wei Wang, Wenmeng Zhou, Wente Wang, Wenting Shen, Wenyuan Yu, Xianzhong Shi, Xiaoming Huang, Xin Xu, Yan Kou, Yangyu Lv, Yifei Li, Yijing Liu, Yiming Wang, Yingya Zhang, Yitong Huang, Yong Li, You Wu, Yu Liu, Yulin Pan, Yun Zheng, Yuntao Hong, Yupeng Shi, Yutong Feng, Zeyinzi Jiang, Zhen Han, Zhi-Fan Wu, and Ziyu Liu. Wan: Open and advanced large-scale scale video generative models. arXiv preprint arXiv:2503.203:
2025. 3
Mengjiao Yang. Yilun Du, Bo Dai, Dale Schuurmans, Joshua B Tenenbaum, and Picter Abbeel. Probabilistic adaptation of text-to-video models. arXiv preprint arXiv:2306.01872. 2023a. 4, 9
Mengjiao Yang. Yilun Du Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Pieter Abbeel. Learning interactive real-world simulators. arXiv preprint arXiv:23100061114 2023b. 1, 3
Sherry Yang, Jacob Walker, Jack Parker-Holder, Yilun Du, Jake Bruce, Andre Barreto, Pieter Abbeel, and Dale Schuurmans. Video as the new language for real-world decision making. arXiv preprint arXiv:2402. 17139, 2024a. 3
Zhuoyi Yang. Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models 2024b. 2
with expert transformer. arXiv preprint arXiv:2408.060)
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-worid: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning. pp. 1094-1100, PMLR, 2020.
Xiao Yu, Baolin Peng, Michel Galley, Jianfeng Gao, and Zhou Yu. Teaching language models to self-improve through interactive demonstrations. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.). Proceedings of the 2024 Conference of the North American Chapter of the As- sociation for Computational Linguistics: Human Language Technologies (Volume 1: Long Pa- pers), pp. 5127-5149, Mexico City, Mexico, June 2024. Association for Computational Linguis- tics. doi: 10.18653/v1/0244 naacl-long..287. URL https: /ttp//aaaaaa// /t/////aaaaatttotoyaaeeeaeee org/2022/ phacl-1ong.2877
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason E Weston. Self-rewarding language models. In International Conference on Machine Learning (ICML), 2024. 3
Siyuan Zhou, Yilun Du, Jiaben Chen, Yandong Li, D. Y. Yeung, and Chuang Gan. Robodreamer: Learning compositional world models for robot imagination. arXiv preprint arXiv:2404.12377, 2024. 3
13

---

Published as a conference paper at ICLR 2026
A TASKS AND TEXT PROMPTS
Below we list the tasks and associated text prompts used for evaluating SILVR. Tasks with demon- strations seen during training of the in-domain model are denoted with asterisk.
Task
In-Domain Model Prompts
Internet- Doamnn Model Prompts
Coffee Button"
coflee button
robot arm pressing the collee machine button
Door Open"
door open
a rohot arm reaching door handle and pulling it to open the door robot opening drawer pulling its white handle backward a robot arm unplugging a peg by palling it from the right to the left sliding a plate from the left into the
Drawer Open"
drawer open peg unplug side
Pog Unplug Side
Plate Slide Side"
plate slide
right side
Push
push
robot arm pushing object forward to green sphere
Reach"
reach
a. robot arm reaching red sphere
Sweep"
sweep
robot moving object to the left side of the table a rohot arm pushing a dooor to close it robot closing window
Door Close
door cloxe
Window Close
window close
pulling handle from the left to the right a robot arm opening a window by pushing its handle from the right to the left arm closing drawer pushing white handle forward
Window Open
window open
Drawer Close
drawer close
Foucet Close
faucet close
robot arm pulling faucet counterclockwise
Faucel Open
faucet open
a robot arm pushing a faucet clockwise rohot pressing down handle
Handle Press
handle
Handle Press Side
handle press side
a robot arm pressing down a handle on the side robot turning dial counterclockwise a robot arm sliding a plate forward into the net arm reaching loward red sphere over a wall robot arm reaching over wall to press a button
Dial Tum
dial
Plate Slide
plate slide
Reach Wall
reach wall
Button Press Wall
batton press wall
Push Red Cup"
red
robol arm pushing the red cup robot arm pushing the blue cup robot arm pushing the green cup a robot arm pushing the pank cup robol pushing the orange cup a robot arm pushing the purple cup
Blue Cup'
Push Green Cup"
green
Push Pink Cup"
pink
Push Orange Cup
orange
Push Purple Cup
purple
Open Red Drawer'
red
a rohot arm opening the red drawer robot opening the green drawer a robot arm opening the blue drawer a arm opening the yellow drawer
Open Green Drawer'
green
Open Blue Drawer"
blue
Open Yellow
yellow
Table A2: Task-Prompt Pairs. We include comprehensive list of tasks and their text prompts for in-domain training and evaluation. "*" denotes tasks seen during initial training of the in-domain model. We also provide the prompts used interface with the internetpreerainn text-to-video model during adaptation with IPA.
B IMPLEMENTATION DETAILS
We provide detailed architecture configurations of the models used in SILVR, and their relevant hyperparameter settings below.
Following prior work (Luo al., 2025), we design one choice of inverse dynamics model as small MLP network built on top of a pretrained pixel-based repre- sentation network. The MLP-IDM takes as input the embeddings of two video frames, which are extracted using VC-1 (Majumdar et al., 2023), and outputs a prediction of the action that enables the transition between the provided frames.
MLP Inverse Dynamies Model:
For the Panda arm experiments, the MLP-IDM is tasked with predicting the end effector position of the last frame provided. This then executed in the physical environment through inverse kinemat- ics. Furthermore, the two video frames have frameskip of 16; the frequency which the camera is queried for trajectories is so high such that two temporally consecutive frames is not more sub- stantially meaningful than just observing the last frame. For Meta World experiments, the two vidco frames are consecutive, and thus have a frameskip of
The total parameter count of the MLP-IDM used in worldw experimentation is 85.81M. Of these, 85.80M parameters are inherited from VC-1 whereas our MLP-IDM design contributes only an additional 10759 parameters due the additional MLP on top.
In our real-world experiments, we reuse the same MLP-IDM for all tasks within the same envi- ronments, and do not perform any finetuning during the SILVR iterations with subsequently self-
14

---

Published as a conference paper at ICLR 2026
collected data. In such way, the MLP-IDM is trained set of seen tasks, but applied to novel task without further modification. The subsequent success on such novel tasks therefore highlights not only the robustness of the MLP-IDM learned, but also the visual quality of the synthesized visual plans. The detailed hyperparameters MIP-IDM training are provided in Table A3.
Hyperparameter
Value
Input Dimension
1536
Output Dimension (Panda)
7
Training Epochs
20
Learning Rate
1c-5
Optimizer
AdamW
Table A3: Hyperparameters of MLP Inverse Dynamics Model Training. We list the relevant hyperparameters of training the MLP inverse dynamics model.
Diffusion Inverse Dynamics Model (DIDM): Whereas the MLP-IDM does continue to facilitate self-improvement for MetaWorld experiments through SILVR, we find that the most performant implementation for our simulated experiments was a Diffusion Inverse Dynamics Model (DIDM), The DIDM is built off the UNet implementation of Diffusion Policy (Chi al., 2023); it is modified to take in not only the current frame but also a frame 9 timesteps into the future and outputs an action chunks of size 8. We further implement the DIDM to operate directly in StableDiffusion latent space, where each frame is of dimension (64,64.4), rather than RGB space of (512.512.3) for further speed efficiency. The DIDM initially trained on the 8 seen-task set (with 25 initial demonstrations per task) for 200 epochs. with learning rate of le-4, and batch size of 128. At each SILVR iteration on World, the DIDM is further finetuned on the 30 collected demonstrations for 30 epochs, reusing Ir of 1e-4, and with batch size of 30.
Hyperparameter
Value
Input Dimension
32768
Output Dimension (MelaWorld)
4
Training Epochs
200
Learning Rate
le-4
Optimizer
AdamW
Table A4: Hyperparameters of Diffusion Inverse Dynamics Model Training, We list the relevant hyperparameters of training the diffusion inverse dynamics model.
In-Domain Model: We reuse the implementation of small-scale diffusion model that conditions on both natural language and an initial pixel frame from (Ko et al., 2024). To improve text-conditioned capabilities of the model, add additional Cross-Attention layer every level of the U-Net, which attends to the CLIP-encoded text prompt. Specifically, we instantiate UNet with 3 ResNet blocks for Meta World settings and ResNet blocks for Panda arm tasks. We report the detailed list of model parameters in Table A5. In total, the in-domain model consists of 179.91M parameters MetaWorld World and 156.58M parameters for Real-World experiments. We perform initial in-domain
for Meta training for 70K training steps on MetaWorld World and 88K steps on Panda, with a batch size of 8 and learning rate of 2e-5. each SILVR iteration, we finetune the in-domain video model for 10K steps with with batch size of 4 and learning rate of le-6 on MetaWorld. On Panda Arm, we finetune for 8,000 steps with batch size of 8 and a learning rate of 2e-5 on Cup Pushing and for 10,000 steps with batch size of 8 and learning rate of le-5 Drawer Opening. All experiments are performed on a single NVIDIA A6000 or RTX3090 GPU.
Internet-Domain Model: Following Adapt2Act (Luo et al., 2025), we employ AnimateDiff' (Guo et al., 2023) as the frozen anternet-pretraing video model for inverse probabilistic adaptation. Addi- video generation. Model
tionally, SparseCtrl (Guo et al., 2024) enable maage-conditionn components and their parameter counts are listed in Table A6. In total, AnimateDiff consists of 2.005B parameters.
15

---

Published as a conference paper at ICLR 2026
Component
# Parameters (Millions)
U-Net (MetaWorld)
116.71
U-Net (Panda Arm)
93.38
Text Encoder
pattcchhhh)
63.2
Table A5: In-Domain Model Components. SILVR relies on a small in-domain text-to-video model, which base our implementation off of prior work (Ko al., 2024). We list the size of the components of the model architecture used.
Component
# Parameters (Millions)
VAE (Encoder)
34.16
VAE (Decoder)
49.49
U-Net
1302.16
Text Encoder
123.06
ControlNet 496.73
Table A6: AnimateDiff Components. IPA relies internet-scale text-to-video model; in this work we use AnimateDiff. We thus list the size of components of the AnimateDiff checkpoint used. The checkpoint used purely for inference, and is not modified updated in any way. Note that the VAE Decoder is not utilized in our framework.
Visual Planning Hyperparameters: In visual planning, we predict future frames conditioned on the current observation and task prompt. We follow (Luo et al., 2025) to perform DDIM (Song al., 2021) sampling for 25 steps to synthesize visual plans. in which the text-conditioning guidance scale set 2.5 for MetaWorld experiments and 7.0 for Panda Arm Pushing. We use 0.5 as the prior strength for inverse probabilistic adaptation.
Visual planning provides the user control over the quality of execution against the speed. In our experiments, each visual plan consists of 9 frames, including one current observation and eight future frames, and can be translated into actions. By performing open-looy control, we execute all 8 actions from single visual plan sequentially in the environment without any re-planning. While synthesizing visual plan can often involve multiple sampling steps and thus be time-cor consuminn open-loop control greatly improves the interaction efficiency. However, since open-loop control does not adjust the control actions based on the feedback from the environment, the subsequent actions from the plan might become suboptimal to the latest states and cause error accumulation. To mitigate this issue, closed-loop control adjusts the action for every interaction step. Specifically, we execute only the first action from the plan, and perform re-planning based on the new observation received from the environment. Although this control style allows us to interact most reliably, it incurs a large computational overhead due to frequent re-planning. To balance the execution quality and efficiency, we can flexibly choose control loop between the two extremes of open-loop and closed-loop For example, we execute half of the plan (e.g. 4 actions) before re-planning, which we reference as semi-open-loop control.
Choices of Control Loop:
To achieve the best execution speed, we employ open-loop control in Panda Arm Pushing and Drawer Opening tasks, in which we discover that visual plans can be performed decently, with neg- ligible deviation in the real execution. For MetaWorld experiments, we utilize semi-open-loop control to balance performance and efficiency.
Behavior Cloning Improvement Loop: Diffusion Policy initially trained on the same data used in SILVR for 150 epochs with learning rate lc-4 and batch size 64. For each iteration, deploy the diffusion policy to interact with the environment and collect 30 task demonstrations. The policy will then be fine- tuned the successful data filtered by ground-trute task success signals for 50 epochs with batch size 30 and learning rate le-4.
Diffusion Policy: After applying SILVR. optionally distill the visual planning components into a diffusion policy. This has the benefit of lightweight, fast inference for final deployment, while still leveraging the self-improving benefits of visual models during the training
16

---

Published as a conference paper at ICLR 2026
process, thus balancing both worlds. The architecture of the DP is consistent with that used in the DSRL and BCIL experiments. During distillation, the visual planner teacher model first collects 120 demonstrations from the environment. Then, DP is trained for 300 epochs with batch size of 64 and a learning rate of le-4 on only these 120 demonstrations collected by the teacher.
In essence, SILVR can be thought of composed of two systems. The slower system is the video planner approach, where the advantage appears in greater autonomous improvement capabilities. The faster system is the distilled diffusion policy, which although has weaker self-improvement, can capture the current performance of the slower system and be deployed with fast inference.
DSRL Implementation: We utilize the open-source implementation of DSRL (Wagenmaker al,, 2025) for our experiments. We maintain the vast majority of parameters from the default setting, but make sure to collect 30 demonstrations for each iteration as in SILVR. When optimizing over the collected experience, we utilize a comparable amount of gradient steps with the amount found in the default settings of DSRL; we use a batch size of 256 with 60000 gradient steps per update. We find that given the same amount of experience and a sufficiently large update budget, DSRL is unable to match the iterative performance improvements SILVR. This highlights the sample inefficiency of reinforcement learning, which may need extra experience to successfully bootstrap value function for the novel task of interest.
We performed a light ablation, comparing a roughly equivalent amount of gradient updates with SILVR (150 gradient steps per update) against the amount of updates used across multiple standard robotic tasks provided by the DSRL publicly available codebase, and found no significant difference in improvement trend. This suggests that the bottleneck for self-improvemmnt through DSRL is not update computation, but experience collection; on the other hand, SILVR demonstrates significantly better sample efficiency for iterative self-improvement.
Iteration
2
4
DSRL (150 Updates)
10.1 (#0.1)
8.9 (+0.6)
8.6 (+1.0)
9.4 (=0.1)
8.3 (+0.2)
DSRL (60000 Updates)
9.4 (+1.7)
8.3 (+1.6)
7.4 (10.9)
7.5 (13.4)
7.7 (+3.4)
Table A7: DSRL Performance Ablation over Different Update Rates. We report mean success rate and standard deviation across three iterations for DSRL with different numbers of updates per iteration. We show that the bottleneck for DSRL is not gradient updates, but collected experience.
C METAWORLD TASK PERFORMANCE DECOMPOSITION
Iteration
Rutton Press Whll
(0x-0.0)
0.0(+00))
0-00000))
00000000)
0.0-0.00
(.0000.0)
(-6.0)
(-0.0)
(-000)0)
(-000
Tum
4.1.)
25.67
(1188))
233 (L208)
1..00
25.6 22.2)
28.9 (.299
36.7 (1318)
36.7 (1.28)
34.(11322)
Close
(26.9)
(7.662
58916225.)
72.2 (-14)
78.9 (2126))
85.6(+5.3)
Close
(=1.9)
(27.7)
(26.7)
200
27.8 (0.))
28.9 2110 (213.)
(-6.)
27
4-200-20.00
(-19)
(+133)
13.3-173))
Open
6.7(1).7)
(1184)
(11000)
778811022)
(L..)
(11022))
(18.8)
(19.5
3.3)
(1..8)
PresSide
(2-51)))1
31.1 (#12.6)
(=1000
722 (=13.5
(10
(00
(-
66.7 (-6.7)
68.9 (5:84)
65.6(=11.7)
Plase Slide
(--.))
(67(-16677
(-17)))
(-158)
(-11-1117)
(-0
(-0
62.2 (+184)
71.1 (+110)))0)
(+1001
Resch Wall
(0.0(-).8)
(2.9(+1.))
(-9.)
300(-67)
278 (-51)
(-130)
(-3.)
24. (+8.4)
26.7 (+120)
(+.0
Window Close
3.3...3.3
1000110.0
2.360))
(L155)
167 (1533)
200
21.1 (L135)
200 (1100)
34.4 (1135)
25.6 (L8.4)
Window Open
(2.61)))1)
(=1144
(617
(4.1
71.1 (=13.)
74.777771
(=117
75.3 (28.8)
74.4 (2:100)
AVG
(L0.6)
(11.)
332.2
433 (L2.5)
(L4.)
C1..0
$1.3
(2.0
(15:2)
53,1 (L3.8)
Table A8: MetaWorld Task Performance. We provide a detailed list of task performance for the leftmost plot in Figure 4. We report the mean success rate and standard deviation aggregated over 3 seeds each, across 10 iterations. We reiterate that none of these 12 tasks had been seen a priori during initial in-domain video model training.
D GENERALIZATION CAPABILITIES BETWEEN VISUAL PLANNING AND BC
Our results suggest that visual planning has some default generalization and self-improvement ben- efits for decision-making compared to using direct action-prediction BC policies. We hypothesize
17

---

Published as a conference paper at ICLR 2026
that modeling consistent visual motions extract more training signal from the provided data than modeling action sequences directly, as there is more supervisory training signal from pixels than Low-dimensional actions. Indeed, shown in the MetaWorld portion of Figure 2, for initial iter- ation 0 on a novel task concerning an unseen object, the visual planner still manages to generate coherent motions for the robot even if it blurs out the specific novel object interaction. These coherent motions, even if the specific object interaction is not modeled correctly (or even blurred) initially, can still be accurately translated by the IDM into meaningful robot actions: thus visual planning may have additional generalization benefits. On the other hand, the basic BC policy does model visual details but predicts sequence of actions entirely from the conditioning frame; having overfit to its training set, when faced with novel object in the scene it may predict highly suboptimal actions, thus leading to poorer generalization.
E FULL METAWORLD SUBOPTIMAL RESULTS
As mentioned in Section 4.6, evaluate SILVR 12 unseen MetaWorld tasks with suboptimal initial in-domain data. We provide a detailed task performance breakdown Figure A1 and Ta- ble A9. We observe that most tasks, well average performance, exhibit improving trend overall across iterations, demonstrating the robustness of SILVR to initial data quality.
SILVR (Expert)
SILVE (Suboptima))
Average Task Performance
s0x Di
Success (%)
Success Rate (%)
Success Ittte (%)
0
2
Iteration
Iteration
Iteration
Bution Press Wall
Dooor Close
Bancet Cloose
Handle Preex
Plate Slide
Window Close
Dial Ham
Drawer Close
Open
Biandle Prees Stde
Reach Wall
Window Open
Figure A1: SILVR Curves with suboptimal in-domain data. For each task, we plot the mean success rate across iterations, aggregated over seeds.
Iteration
4
Button Press Wall
2.2 (=:19)
2.2 (+3.8)
11.1 (=:0.2)
36.7 (1:321)
(=2008)
Dial Turn
2.2 (+1.9)
(+1226)
31.1 (+13.5)
28.9 (+-3.44)
(-11.7)
Door Close
5.6 (+3.8)
(+11
(47.77 (48.8)
(0.0
Drawer Close
14.4 (1:6.)
(120.9)
46.7 (1:0.33
57.8 (1:2.11)
(4.4
Faucet Close
0.0
0.0.0))))
0,0
0.0
1.1.9
Faucet Open
0+000))0)
1.1.9
(+1.9
(+8.4)
(=16.8)))
Handle Press
(15.88)
34.69)
(115.0)
75.6.9)
(-219)
Handle Press Side
24199)
45.6
(1102)
(1.004)
76.7 (-5.8)
Plate Slide
0.0
0.000))0)
0.0
0.0
2-1.9)
Reach Wall
21.61)
32.29
(47.77
50..77
52.2 (=6.9)
Window Close
2.2
(.4.9)9))
22.2
24.4 (4-1.77)
30.0 (=23.3)
Window Open
4.1.9)
11.1 (=6.9)
16.7 (#12.00
12.2 (#669)
37.8 (=22.7)
AVG
9.503)))
2+.9
292.))
38.1 (+4.9)
41.0 (+4.4)
Table A9: SILVR Performance with Suboptimal Initial Data. For each task, we report the mean success rate and standard deviation aggregated over 3 seeds, across 5 iterations.
Additionally, we provide BCIL and DSRL performance with suboptimal in-domain data Ta- ble A10. Surprisingly, we observe that BCIL benefits more from suboptimal initialization. We hypothesize that the exploration brought by initial random actions is crucial for bootstrapping the Diffusion Policy performance in BCII..
18

---

Published as a conference paper at ICLR 2026
Iteration
4
DSRL
7.4 (+3.8)
7.1 (#0.6)
8.6 (#1.3)
7.8 (+1.0)
8.1 (+0.8)
BCIL
8.1 (=1.0)
21.6 (+0.9)
29.1 (+2.3)
37.2 (+6.1)
39.6 (+5.3)
Table 10: Baseline Performance with Suboptimal Initial Data. We report the mean success rate and standard deviation across 12 unseen tasks, aggregated over 3 seeds each.
F METAWORLD COMPONENT ABLATIONS
Iteration
4
Finetuning All (SILVR)
14.7 (+0.6)
27.7 (+1.9)
33.5 (+2.2)
43.5 (+2.6)
44.2 (+4.5)
No IDM Finetuning
14.1 (+1.6)
22.2 (#3.1)
24.6 (+2.5)
24.8 (+1.7)
26.8 (#1.9)
Only IDM Finetuning
15.0 (+18)
(43.2)
27.7 (+3.5)
26.9 (+4.1)
29.8 (#3.4)
Table A11: Component Update Ablations. report the mean success rate and standard deviation across 12 unseen tasks, aggregated over 3 seeds each.
We investigate the importance finetuning different visual planning components with respect to online experience on final task performance in the MetaWorld suite. We find that in contrast with the real-world experiments, keeping the IDM frozen for the MetaWorld suite struggles to adjust to novel objects and motions, resulting in modest performance gains even when the video model improves. Similarly, we have found that when the IDM is finetuned but the video model not, there indeed some performance gains most likely from better translation of the base generalization ability of the video model; but performance also saturates quickly. Thus, under situations with heavy novelty such across objects and motions, updating both components with respect online experience is critical, Mcanwhile, in real-world experiments, we observed that the initially trained IDMs are inherently robust to novel tasks, in which many motions learned from seen tasks can be reused, and can be utilized directly to great effect without additional finetuning.
19

---

Published as a conference paper at ICLR 2026
G ADDITIONAL PLAN VISUALIZATIONS
We show additional visual plans for SILVR, across multiple environments and tasks, along with their execution results.
G.1 SILVR WITH GROUND-TRUNE FILTERING
Visual plans and their executions for SILVR with Ground-Tran filtering are illustrated below.
Iteration
Visual Plan
Execution
Iteration
Visual Plan
Execution
Figure A2: SILVR on Drawer Close with Ground-Trule filtering.
Iteration
Visual Plan
Execution
Iteration
Visual Plan
Execution
Figure A3: SILVR on Window Close with Ground-Truth filtering.
20

---

Published as a conference paper at ICLR 2026
Iteration 0
Visual Plan
Execution
Iteration 2
Visual Plan
Execution
Figure A4: SILVR Orange Cup Pushing (Red/Pink/Orange) with Ground -Tuth filtering.
Iteration 0
Visual Plan
Execution
Iteration 2
Visual Plan
Execution
Figure A5: SILVR Orange Cup Pushing (Red/Green/Orangg) with Ground-Trun Truth filtering.
21

---

Published as a conference paper at ICLR 2026
Iteration 0
Visual Plan
Execution
Iteration 2
Visual Plan
Execution
Figure A6: SILVR Purple Cup Pushing (Blue/Pink/Purple) with Ground-Trus Truth filtering.
Iteration 0
Visual Plan
Execution
Iteration 2
Visual Plan
Execution
Figure A7: SILVR on Purple Cup Pushing (Ree/GrrennPurplee) with Ground-Truth filtering.
22

---

Published as a conference paper at ICLR 2026
Iteration 0
Visual Plan
Execution
Iteration 2
Visual Plan
Execution
Figure A8: SILVR Yellow Drawer Opening (Vellow/Green) with Ground-Trus filtering.
Iteration 0
Visual Plan
Execution
Iteration 2
Visual Plan
Execution
Figure A9: SILVR Yellow Drawer Opening (Yellow/Blue) with Ground-Trute filtering.
23

---

Published as a conference paper at ICLR 2026
G.2 SILVR WITHOUT FILTERING
Visual plans and their executions for SILVR without filtering are illustrated below.
Iteration 0
Visual Plan
Execution
Iteration 2
Visual Plan
Execution
Figure A10: SILVR on Drawer Close without filtering.
Iteration 0
Visual Plan
Execution
Iteration 2
Visual Plan
Execution
Figure A11: SILVR on Orange Cup Pushing (Blue/Pink/Orangee without filtering.
24

---

Published as a conference paper at ICLR 2026
G.3 CORRECTIVE INFLUENCE OF INTERNET-SCALE VIDEO MODEL ADAPTATION
In-Domain Only
Visual Plan
Execution
Adapted w/ AnimateDiff
Visual Plan
Execution
Figure A12: Push Orange Cup
In-Domain Only
Visual Plan
Execution
Adapted w/ AnimateDiff
Visual Plan
Execution
Figure A13: Push Purple Cup
25

---

Published as a conference paper at ICLR 2026
In-Domain Only
Visual Plan
Execution
Adapted w/ AnimateDiff
Visual Plan
Execution
Figure A14: Open Yellow Drawer
In-Domain Only
Visual Plan
Execution
Adapted w/ AnimateDiff
Visual Plan
Execution
Figure A15: Drawer Close
26

---

Published as a conference paper at ICLR 2026
G.4 LONG-HORIZON TASK EVALUATION
Whereas the experiments thus far have demonstrated how SILVR can iteratively improve perfor- mance single tasks, natural question consider how applicable it is to complex tasks. We note that many long-horizon tasks can be broken down into atomic tasks with shorter horizons, for which SILVR can already demonstrate clear and robust seff-improvement behaviors, We thus investigate how SILVR. by learning individual abilities, naturally facilitates the direct iterative im- provement of sequential long-horizon tasks where success rate depends on the full completion of behaviors in a specific order.
In particular, we construct an experiment of pushing cups in an order specified by natural language. For example, in Figure A16 the prompt is "Push Cups in the order of Red, Orange, Green", where success is only satisfied if done so the correct order. We evaluate SILVR checkpoints automati- cally using VLM, implemented as Gemini 2.5, to determine what atomic skill (e.g.. pushing the cup of a specific color) to currently execute from the instruction sequence and when to switch to the subsequent one by querying if the visual execution of the current atomic skill was successful.
We find that over successive iterations, despite only applying SILVR to individual atomic tasks, the performance the overall long-horizon tasks improves. This suggests that for arbitrary com- plex long-horizer nogg-horzzon tasks of interest, SILVR can be combined with an efficient subtask segmentation mechanism to improve individual performance atomic behaviors and thus improve the final over- all performance on long-horizon sequential compositions.
Iteration
Visual Plan
Execution
Iteration
Visual Plan
Execution
Iteration
Visual Plan
Execution
Figure A16: Push Cups in the order of Red, Orange, and Green. The final-iteratio visual planner completed the long- horizon task, whereas those from the first two iterations failed.
27

---

Published as a conference paper at ICLR 2026
Iteration
Visual Plan
Execution
Iteration
Visual Plan
Execution
Iteration 2
Visual Plan
Execution
Figure A17: Push Cups in the order of Blue, Purple, and Green. The visual planners of the last two iterations completed the long-horizon task, whereas the one from the first iteration failed.
G.5 VISUALIZING FAILURE CASES OF FINAL-ITERATION VISUAL PLANNERS
Below we visualize the failure cases from the final-iteration visual planners across both real-world and simulation task settings. Most real-world failures arise from semantically incorrect visual plans, in which the robot arm attempts to push the cup (Figure A18 and A19) or open the drawer (Fig- ure A20) of a wrong color, whereas in simulation, we observe a mixture of execution (Figure A21) and semantic (Figure A22) errors, both of which can contribute to unsuccessful task completion.
Iteration 2
Visual Plan
Execution
Figure A18: Push the Orange Cup (Iteration 2)
28

---

Published as a conference paper at ICLR 2026
Iteration 2
Visual Plan
Execution
Figure A19: Push the Purple Cup (Iteration 2)
Iteration 2
Visual Plan
Execution
Figure A20: Open the Yellow Drawer (Iteration 2)
Iteration 9
Visual Plan
Execution
Figure A21: Plate Slide (Iteration 9)
Iteration 9
Visual Plan
Execution
Figure A22: Button Press Wall (Iteration 9)
29


Reviews:
[{"review_id": "HhRKuFZsTj", "reviewer": "Reviewer_dWn2", "summary": "The paper studies how to make video-based visual planners automatically adapt and generalize to novel robotic tasks by learning from self-collected online experience. The proposed method (SILVR) creates an iterative loop: adapt an in-domain text-to-video model with an internet-scale prior, roll out generated visual plans via an inverse dynamics model, and use the filtered successful trajectories to update both the in-domain video model and the inverse dynamics model. Experiments in both the MetaWorld simulation and the real world demonstrate that SILVR can improve success rates and outperforms other online improvement baselines.", "questions": "Please refer to the weakness section. In addition, I have one minor question:\n- Besides successful trajectories, have the authors considered using failed rollouts to update the IDM as well? I suspect this could be particularly useful when collecting complete successful trajectories is difficult (e.g., in long-horizon tasks).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Using online interaction to improve visual planning is a novel and interesting research problem.\n- The approach is intuitive and performs well in practice.\n- Thorough experiments in both simulation and on real robots. The training tasks and test tasks are separated clearly, allowing a strict assessment of how well the method adapts to unseen tasks.", "weaknesses": "- I treat the method as two complementary parts: (a) improve the visual planner by successful trajectories rolled out with the help of a good IDM, and (b) improve the IDM with meaningful online behavior collected under the guidance of a good visual planner. Though the paper is mostly presented from perspective (a), it would be valuable to do some ablations to understand whether the empirical gains come mainly from (a), (b), or their combination. My feeling is that the answer may differ across settings. On the real robot, the training and test tasks use the same motions, so the IDM might easily learn to infer actions by focusing on the robot's visual differences. Whereas MetaWorld test tasks require novel motion patterns, which means improvements to the IDM might matter more.\n- I am also curious about the planning performance of AnimateDiff in the real-robot experiments. Does this experiment simply \"distill\" the capabilities of the general model into the in-domain model, or can the final improved in-domain model actually surpass the general model?\n\nUnderstanding the above questions would further strengthen the paper’s arguments. However, I believe the current version is already sufficiently complete to merit acceptance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YFp89N8ZG7", "reviewer": "Reviewer_rmg3", "summary": "This paper proposes to iteratively update a text-&-image conditioned video model that generates visual imaginations of robot trajectories via online RL (instantiated as filter-BC), towards constructing a self-improving robot policy. The policy is formulated as a high-level video generator and an inverse dynamics model that can predict actions given consecutive frames. In Meta-World and two real-world experiments, the authors explore running filter-BC to improve the high-level policy, where in each iteration data is collected by the current combined policy, a success detector is called (either human ground truth, or the authors also explore using a VLM success detector) to label the self-collected data, then the video model is trained on the successful split of this data. The authors run 10 iterations of self-improvement in Meta-World, observing an average performance increase from 15% success rate to 56%, and run 2 iterations of self-improvement on two real-world tasks, observing an improvement from 50% to between 60 and 70%. For real-world experiments they find that their approach can generalize/perform better if their video model is combined with frozen pre-trained video model using a CFG-like score composition. They also run a few ablations, finding that with a VLM success detector that may not be as accurate, performance of their method still improves, and their method can also work with no success detector at all, and when the BC pre-training data is suboptimal. They only compare to baseline approaches on Meta-World (not the real-world), but compare to DSRL and regular filter-BC (without a video model), and find that their approach does better. The authors also explore distilling the two-level policy into a one-level policy after RL training.", "questions": "(1) The IDM model is updated alongside the high-level policy only for the MetaWorld experiments (as stated in line 347). Why not also for the real-world experiments?\n\n(2) Instead of doing score composition do combine the pre-trained video model and in-domain one at inference time, can the pre-trained one be finetuned on in-domain robotic data, and then passed to the self-improvement process? \n\n(3) For score composition to work, does the internet pre-trained model need to generate videos that look sufficiently similar to the in-domain model? It would seem that if the pre-trained model generates very different videos, score composition would not work well. How do you assess whether the required similarity is achieved?\n\n(4) DSRL is used as a baseline in Table 1, but DSRL doesn't have a concept of \"iterations\" of self-improvement in the same manner that the proposed approach does. Rather a gradient step can be taken any time new observations are added to the replay buffer. Can you explain then how DSRL results were mapped to iterations 1 through 4?\n\n(5) What causes the approach's performance to go down (as opposed to up) when just the in-domain model is used in the real-world experiments? What do the authors think specifically prevents the method from working in this case?\n\n(6) How can success rate go up without filtering in Figure 5? What is the natural success rate percentage of the data being collected?\n\n(7) How do baseline approaches (e.g., filter BC) perform in the real-world? I noticed the real-world experiments do not have any baseline approaches.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "(1) As far as I'm aware the idea of doing self-improvement of a high-level video generation policy in the robotic setting has not been explored yet, making this work novel in this regard.\n\n(2) The performance of the approach is shown to be better than two relevant self-improvement baselines, DSRL and regular filter-BC, which is a promising result. \n\n(3) The authors ablate many components of their approach, making the experimentation fairly thorough", "weaknesses": "(1) While self-improvement is obtained, success rate on both MetaWorld and the real-world seems to cap out at around 60 to 70%, and more self-improvement iterations do not improve performance. Intuitively it would seem that there should be no cap on performance. Why does the approach not seem to be able to improve past this success rate?\n\n(2) The authors do not make a convincing argument about why filter BC of a video generative high-level policy should be better than filter-BC of a regular single-level policy. *This leaves the reader unclear about what they should take away from the paper.* On line 362 the authors mention that \"we hypothesize that the separately learned environment visual dynamics is easier to transfer when solving a novel task, leading to stronger base generalization performance through visual planning\". Why should a visual dynamics model generalize better than a direct action prediction policy? Even if it does generalize better, the better generalization should only affect initial performance. Why is the proposed approach able to *improve* better than the baseline?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1Xojx5HPh6", "reviewer": "Reviewer_UiiZ", "summary": "In this work, the authors propose a method named Self-Improving Loops for Visual Robotic Planning (SILVR), which iteratively updates in-domain video model on self-produced trajectories to improve its performance for a specified task of interest. The authors thoroughly demonstrate how SILVR establishes connection between in-domain video model and high-performing visual planner for specified novel robotic control task. Extensive evaluations on the MetaWorld task suite and real-world robot arm verify the effectiveness of SILVR.", "questions": "See weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed SILVR is straightforward and easy to follow, leveraging iterative fine-tuning of a visual planner on self-collected experience without complex reward engineering.\n2. The paper provides extensive experiments across both simulated (MetaWorld) and real-world (Franka Panda arm) settings, demonstrating broad applicability and robustness.\n3. SILVR shows significant performance improvement on unseen tasks (up to 285% in MetaWorld) and outperforms reinforcement learning and behavior cloning baselines in sample efficiency.\n4. The method is robust to suboptimal initial data, does not require ground-truth rewards (works with VLM-based filtering), and can effectively incorporate internet-scale video priors for real-world generalization.", "weaknesses": "Since that I am not an expert in this field, here are a few potential weaknesses I observed from the paper:\n1. While the method is well-executed and effective, the core idea of a self-improving loop—fine-tuning a model on its own successful outputs—is a well-established concept in other areas like large language models. Applying this established concept to visual planning, while practical, may be perceived as a solid incremental advancement rather than a foundational shift in methodology.\n2.  The real-robot experiments (pushing a colored cup, opening a colored drawer), while important for demonstrating real-world applicability, are relatively simple table-top manipulations. It remains unclear how well SILVR would scale to tasks with more complex dynamics, longer time horizons, or tasks requiring intricate multi-step reasoning and tool use.\n3. The self-improving loop implicitly assumes the initial model can achieve a non-trivial success rate to collect useful data. For tasks that are too distinct from the initial training distribution, the model might fail completely at iteration zero, preventing any improvement and creating a \"cold-start\" problem that SILVR fails to address.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XGW2qjKnYH", "reviewer": "Reviewer_2K1s", "summary": "The paper proposes **SILVR (Self-Improving Loop for Visual Robotic Planning)**, a framework that treats a video generative model as a visual planner and **iteratively improves it with self-collected experience**. Concretely, an in-domain video model produces future-frame plans given a text prompt; a separately trained **inverse dynamics model (IDM)** converts consecutive planned frames into executable actions that are run in the environment, after which successful rollouts are filtered and used to **finetune both the in-domain video model and the IDM**—then the loop repeats.", "questions": "1. Why do some tasks trained from suboptimal initial data outperform the expert-initialized setting?\n2. Why does the SILVR-distilled Diffusion Policy (DP) achieve a higher success rate than the final SILVR visual planner?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "1. The authors propose a simulator-driven self-optimization approach for video generation, enabling the success rate on corresponding tasks to increase with additional iterations.\n2. They evaluate SILVR’s generalization and robustness across multiple environments. In simpler settings, SILVR can accomplish the target tasks to a certain extent and shows a degree of robustness and generalization.", "weaknesses": "1. Requires an initially solvable task; limited gains when the starting success rate is very low.\n2. Real-world gains depend on an internet video prior and are modest/unstable without it, with few reported iterations.\n3. Evaluation focuses on short-horizon/simple tasks, leaving feasibility on harder or long-horizon tasks unclear.\n4. Minor presentation issue: Some figures are hard to read in the PDF due to lack of detail explaination for some signals.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)