Self-Improving Embodied Foundation Models
Authors: Seyed Kamyar Seyed Ghasemipour
Ayzaan Wahid
Jonathan Tompson
Pannag R Sanketi
Igor Mordatch
Venue: NeurIPS 2025 poster ()
Keywords: Robotics
Foundation Models
Self-Improvement
Reinforcement Learning
Primary area: reinforcement_learning
TL;DR: We present a reward‑engineering‑free, online Self‑Improvement procedure that enables robotic foundation models to sample-efficiently improve their policies, and autonomously practice and acquire skills generalizing far beyond their imitation data.

Abstract: Foundation models trained on web-scale data have revolutionized robotics, but their application to low-level control remains largely limited to behavioral cloning. Drawing inspiration from the success of the reinforcement learning stage in fine-tuning large language models, we propose a two-stage post-training approach for robotics. The first stage, Supervised Fine-Tuning (SFT), fine-tunes pretrained foundation models using both: a) behavioral cloning, and b) steps-to-go prediction objectives. In the second stage, Self-Improvement, steps-to-go prediction enables the extraction of a well-shaped reward function and a robust success detector, enabling a fleet of robots to autonomously practice downstream tasks with minimal human supervision. Through extensive experiments on real-world and simulated robot embodiments, our novel post-training recipe unveils significant results on Embodied Foundation Models. First, we demonstrate that the combination of SFT and Self-Improvement is significantly more sample-efficient than scaling imitation data collection for supervised learning, and that it leads to policies with significantly higher success rates. Further ablations highlight that the combination of web-scale pretraining and Self-Improvement is the key to this sample-efficiency. Next, we demonstrate that our proposed combination uniquely unlocks a capability that current methods cannot achieve: autonomously practicing and acquiring novel skills that generalize far beyond the behaviors observed in the imitation learning datasets used during training. These findings highlight the transformative potential of combining pretrained foundation models with online Self-Improvement to enable autonomous skill acquisition in robotics.

Self-Improving Embodied Foundation Models
Seyed Kamyar Seyed Ghasemipour Generalist
kamyarogeneralistai..
Ayzaan Wahid & Jonathan Tompson & Pannag Sanketi' & Igor Mordatch
Google DeepMind
{ayzaan, tompson, psanketi, imordatcch&@googlecco
Abstract
Foundation models trained on web-scale data have revolutionized robotics, but their application to low-level control remains largely limited to behavioral cloning. Drawing inspiration from the success of the reinforcement learning stage in fine- tning large language models, propose two-stage post-training approach for robotics. The first stage, Supervised Fine-Tuning (SFT), fine-tunes pretrained foundation models using both: a) behavioral cloning, and b) steps-to-go prediction objectives. In the second stage, Self- Self-Improvement steps-to-go prediction enables the extraction of a well-shaped reward function and robust success detector, enabling fleet of robots to autonomously practice downstream tasks with minimal human supervision. Through extensive experiments on real-world and simulated robot embodiments, our novel post-training recipe unveils significant results on Embodied Foundation Models. First, we demonstrate that the combination SFT and Self- Beff-Improvement is significantly more sample-efficient than scaling imitation data collection for supervised learning, and that leads to policies with significantly higher success rates. Further ablations highlight that the combination web-scale pretraining and Self-Improvement is the key to this sample-efficiency. Next, we demonstrate that our proposed combination uniquely unlocks a capability that current methods cannot achieve: autonomously practicing and acquiring novel skills that generalize far beyond the behaviors observed in the imitation learning datasets used during training. These findings highlight the transformative potential of combining pretrained foundation models with online Self- Self- lolf-lmmrovement to enable autonomous skill acquisition in robotics.
Introduction
Recent works have demonstrated that foundation models can be effectively fine-tuned to directly act as low-level robot policies 99] 45 39 31
21 77 and that they inherit significant generalization and robustness capabilities due to the web-scale pretraining of the foundation models from which they were derived. Thus far the training regime for Embodied Foundation Models (EFMs) has been limited behavioral cloning (i.e. supervised learning) 41 45 39 31 21] 77. In contrast, from the literature on large language models (LLLMs) we observe that after the initial pretraining, post-training for downstream tasks is typically divided into two stages: 1) Supervised Fine-Tuning (SFT), followed by 2) Reinforcement Learning (RL). RL-tuning of LLMs has been shown to markedly, and rapidly, improve downstream task performance beyond SFT 47 40 and has become critical stage in the training recipe of foundation models (25
50 20
Foundiig Member of Technical Staff at Generalist AI. Project completed April 2024 Google DeepMind.
Equal supervision.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Pretrained
Stage 1: Supervised Training (Offline)
Stage 2:
Online Self-improvement
Multimodal Foundation
Data: Robot imitation Learning Datasets P
Data: On-Robat Policy Rollouts =
Rollouts (1 Human Monitoring Multiple Self-Improving Robots)
Data: Internet
Behavioral Cloning Objective
EFM Policy &
Success Detector
Embodied Foundation Model (EFM)
move the blue cube to circle?"
"42"
Steps-To-Go Objective
Embodied Foundation Model (EFM)
Self-Improvemeet Reward Labelling & Updates
"How many steps move the blue cube the red circle?"
EFM
Updated EFM Policy
Reward
Figure 1:
Overview of our proposed two-stage fine-tuning approach. Stage 1 (Supervised Fine-Tuning):
Starting from pretrained multimodal foundation model, using rohat imitation learning datasets with fine-tune EFMs with a) behavioral cloning, and b) steps-to-go prediction objectives. Stage 2 (Self-Improvement): Self-predicter rewards and success detection enable fleet of rohots to autonomously practice downstream tasks with minimal human supervision. Leveraging these self- aef-predieted signals, online reinforcement learning rapidly improves policies and enables the acquisition of novel outtof-distribution tasks.
Despite the unique algorithmic and engineering challenges of investigating RL-tuning of foundation models in the context of real-world robotics, the aforementioned sample-efficiency and performance gains from the LLM literature strongly motivate its investigation. In this work we directly tackle these challenges and design a two-stage framework inspired by LLM post-training processes: In Stage (SFT), given an imitation learning dataset we fine-tune EFMs using
1 "Supervised Fine-Tuuing two objectives: a) behavioral cloning, and b) predicting the number of "steps-to-go desired goals. In Stage 2 "Self- Implovement" we leverage the model's 's steps-to-go predictions to extract a well-shaped reward function as well as a robust success detector. These key components enable one human operator to monitor multiple robots they autonomously practice downstream tasks. Critically, our data-driven reward design eliminates the need for ground-truth rewards, and leverages the robustness and generalization properties of the underlying foundation models.
accomplish
Through extensive experiments on two robot embodiments, LanguageTable (34] and Aloha [57 33.
in the real-world and simulations, we demonstrate the surprising efficacy of our novel post-training framework. First, we demonstrate that not only does Self-Improvement robustly improve policy performance beyond behavioral cloning, but the combination of SFT and Self- Seff-Improvement is significantly more sample-effin efficient than scaling imitation data collection for supervised learning alone. As example, on the Language Table domain 34 10% additional robot time in the form of Self-Improvement increases policy success rates from 45% + 75%. In constrast, increasing the amount of robot imitation data by 8x leads meager 45% 60% improvement. Further ablations highlight the key role of foundation model pretraining in enabling this sample-efficiency and robustness.
Excitingly, our novel combination of online Delf-Imprrvemmnt and web-scale pretraining also unlocks unique capability not afforded by prior methods: enabling robots to autonomously practice and acquire new skills. contrast to prior works that have demonstrated semantic generalization such as executing the same aik--nd--pace motions in new contexts 99] we show that this combination enables behavioral generalization that extends far beyond the imitation data used in Stage 1. Our work highlights the transformative potential of combining pretrained foundation models with online Self-Imprrvemmet unlock autonomous skill acquisition robotics. Our project website can be found at: self-fimrooooe improving efms. www.eef.ss..sub....e...eeeee.eae github.15 io

---

2 Methodology
Our focus in this work to investigate the efficacy of RL post-training for embodied foundation models in the context of robotics. However, a critical challenge of reinforcement learning for robotics, and in particular for manipulation tasks, is the problem of reward engineering. Designing effective reward functions requires repeated trial-and-error iterations of training policies and patching reward definitions to mitigate unintended outcomes. Furthermore, even with a perfect definition, measuring rewards in the real-world requires significant engineering effort. Thus, as move towards future where we train robots to accomplish increasingly broad sets tasks, manual reward design becomes untenable for real-world robotics.
We overcome this obstacle via learning data-driven reward functions that also inherit robustness and generalization properties from the web-scale pretraining of the underlying foundation models. Our proposed post-training framework composed two stages: 1) Supervised Fine-Tuning (SFT) wherein we fine-tune EFMs using behavioral cloning as well as "steps-to-go" prediction objectives, and 2) Self-Improvement (Online RL) wherein EFMs autonomously practice downstream tasks and rapidly improve via optimizing self oeffppreditte rewards.
2.1 Stage 1: Supervised Fine-Tuning (SFT)
The first stage of our framework is the Supervised Fine- Tame-TTnnng (SFT) stage. Let DD denote an imitation learning dataset. We assume that we can sample tuples (or, at,gee) D, where o, and a, denote the observation and action at a timestep respectively, and gr denotes a goal, event, or outcome that occurs in the future (i.e. & (') within the same trajectory (ot, a.). This assumption subsumes most imitation learning datasets, including hindsight-relabelle, as well as single-task dataset We make assumptions regarding the optimality of the trajectories in the dataset. Given D, we initialize the EFM using pre-trained foundation model and perform supervised fine-tuning using the following objectives:
(ECM) = =- -EE ar.ge )/DD log Paction (a: Ot, ge))
Caupssto-go (EFM)
E (o
RFM log Pseeps to-go (t' - or 9v)
CBC denotes a goal conditioned behavioral cloning loss, where we maximize the log- log-likelihood of a dataset action conditioned the observation and the goal. The second objective, Lsseps to teaches the EFM to predict how many timesteps away the policy is from accomplishing the intended goal, given the current observations. This objective plays critical role in enabling the second post-training stage, Self- leff--lmpoveeeenf
2.2 Stage 2: Self-Improvement
In Stage 2, our goal is fine-tune the EFM on downstream tasks using online RL in order rapidly improve policy performance. As we will see later on in our experiments (Sections 4.3.1 and 4.3. downstream tasks may even be significantly different than those that appeared in the dataset used for Stage training.
Reward Function Let,
d(o.g) = E (stepss-toos steps-to-go
denote the expected value of "steps-to-go" in order to accomplish goal g given observation o, as predicted by the model. The reward function we use for online RL fine-tuning is defined as follows,
r(or, ag (1+1.9) d(or, d(o++1:9)
"s an example, we can treat single-task datasets a goal-conding condicondi dataset where all episodes share the same goal, and that goal is accomplised at the last timestep of every episode.
"In Stage we can include additional auxiliary supervised objectives as well. As an example, in our experiments with the LanguageTable domain, conditioned on the first and last image of an episode we ask the model predict the instruction that was executed in that episode.
3

---

Algorithm 1: Self-Improvement
Input:
Initialize the policy JEMM from a Stage checkpoint. Initialize and freeze a separate
Stage I checkpoint for reward computation and success detection.
while true do
Initialize empty replay buffer;
while replay buffer smaller than N x B do
Sample instruction gi
Execute current policy Paction (ax (o: and end the episode if one of the following conditions is met:
The success detector indicates success: success (r: 9)
The maximum episode length is reached
The human operator manually terminates the episode
Compute Monte Carlo returns using Equation Place (ot, 9, R2) tuples in the replay buffer;
2
R₂ El 7 r(o,, ar, 01t1,g))
end
Perform N policy updates using REINFORCE loss end
c. Rs log partica Matiolarllgg ())
Intuitively, this reward function predicts how much closer the robot got towards accomplishing goal g after taking action a₁ As the reward function derived from d(o, g). which is function of the EFM itself, we refer to our RL fine-tuning process as SSeff-Improvement The choice of using the expected value in Equation iis for simplicity and alignment with the notion of value function in RL (Section G.. We leave investigations of alternative definitions such as CVaR 4 for risk-aware policies, or distributional RL [5], to future work.
Success Detection It is important for robot episodes to terminate upon successfully accomplishing the intended goal. Otherwise, significant portion of the collected data will include the robot resting in successful state. In settings where we do not have a ground-trus truth success detector, as in real-world experiments, we use the following success indicator derived from the model,
success(o, g) == L[d(o,g) < s]
with being a very small number of timesteps We found this formulation of success detection to be very robust even in low data regimes, and significantly more reliable than explicitly including success detection binary classification objective in Stage 1.
Self-Improvement With the above reward function and success detector in place, we can perform online RL fine-tuning of the EFM on desired downstream tasks. We take a frozen Stage I checkpoint for reward function computation and success detection, and initialize the Stage 2 policy from a Stage 1 checkpoint as well [ Each iteration of our Self- rell-Improveeent loop proceeds as follows: Using instruction g. executing the
the current policy we collect set of robot trajectories by sampling robot policy, and terminating the episode when either 1) the success detector indicates success (i.e. success(o, g) = 1), 2) pre-specified maximum episode length is reached, 3) human operator manually terminates an episode (for example
robot station gets into a bad configuration). Subsequently, for each timestep in the collected trajectories we compute the Monte Carlo returns R, E xi r(ot. at, 0++1,9) g) and place elements (0t, Ot, 9, RR) in replay buffer Once enough data has been collected, we perform policy updates using the REINFORCE loss,
Rr log Prstion (a, loe,
sampling minibatches from the replay buffer without replacemen After N updates, the remaining
items in the replay buffer are cleared out and the next iteration begins. Algorithm above presents psuedocode of our proposed Stage Self-Improvement procedure. Although sample-efficiency is
$ Note that these checkpoints are not necessarily identical. For a discussion on checkpoint selection we refer the interested reader to Appendix [
We use 0.9, 5e-2. Please refer to Appendix C for further discussion.

---

Bisepp-to-ggo
12
20
30
Episode Step
An example trajectory from the Aloha Single Insertion Task and a plot representing IE jsstess-togoo under the model's prediction (i.e. d(o,g) g)). Key moments: 1) Model helieves the episode is about to complete successfully, 2) Policy accidentally drops the peg and d(o, g) increases, 3) Policy regrasps the peg from bad angle not suitable for insertion so d(o. g) remains high, 4) Policy drops the peg. providing an opportunity to regrasp correctly which reduces d(o, g). 5) Policy is pushing the peg inside and d(o, 9) marks that the policy is about to succeed, 6) The right hand knocks the socket out of the left hand's 's grip which increases d(o,g). g).
Figure 2:
key consideration of our work, we chose to perform on on-policy RL without data reuse. On-policy methods enjoy better training stability, and using REINFORCE specifically obviates the need for training value functions. These choices eliminate two vertices of the deadly triad [5]], Off-Policy Learning and Bootstrapping. In Section G we discuss how our choice of reward function leads to well-shaped objective that reduces the need for baselines in the REINFORCE estimator. We leave the investigation of alternative RL algorithms, including off- aff-policy methods, to future work.
3 Intuition on Reward Function
Visual Intuition We can begin to build our intuition regarding the efficacy of steps-to-go prediction by visualizing model predictions on domains of interest. Figures and visualizations the Aloha Single Insertion task. In this task, the left arm must first pick up a blue socket, after which the right arm must pick up the red peg and fully insert it into the socket. The captions in these figures walk the reader through the level intricate details that be can learned during Stage training from the steps-to-go objective. We also encourage readers to visit our supplementary materials website to view additional visualizations such as videos, including on the Language Table domain.
Due to space limitations, we continue our discussion in Appendix G First we discuss the mathe- matical intuition behind why our algorithmic choices above lead well-shaped reward function. Subsequently, we discuss the python notebook in supplementary material, and demonstrate the efficacy of Self-Improvement on a pedagogical 2D pointmass domain.
Experiments
In experiments seek to validate proposed Self-Improvement framework and answer the following questions:
Q1: Does Self-Improvement improve performance downstream tasks beyond the super- vised learning stage?
Q2: Is the combination of supervised learning and Self-Improvement more sample-efficient than supervised learning alone?
Q3: Is reel--mprovveett, which depends RL, reliable and reproducible enough to be employed in real-world robotics?
Q4: What is the contribution of pretraining to our half--mproeeeentt procedure?
Q5: Does web-scale foundation model pretraining enable Self' Gelf-Improvement on tasks that generalize beyond what was seen in the imitation dataset?
We study these questions using the Language Table [3 and Aloha [57 3 robot embodiments, with experiments in both simulation and the real-world. Throughout this work we use the PaLI 3 billion parameter vision-language model [13 [4] as our base pretrained foundation model. The inputs to our PaI.I EFM are images alongside text sequence representing relevant information such as the instruction, auxiliary information (c.g. joint positions), and whether to predict actions steps-to-go.
5

---

Simulated Tane
Real LanguageTable
Aloha Single-Insertion
Real2Sim Lang
Banana Table
1.0
6.00
0.50
0.65
0.00
0.75
24
0.25
0.70
0.55
Success Rate
0.65
0.45
0.7
0.75
0.75
0.6
6.70
0.35
0.70
0.50
0.5
0.5
30%
126 22R
28%
95%
5226
25%
25%
50%
85%
5003
7505
55000
12200
13500
56.
3%
2%
24
4%
5%
84
Num Black2Block Episodes
Num Block2Block Episodes
Num Episodes
Num Block28lock Episodes
Hours
Figure 3: Stage Self-Improvement Results. Orange: Stage behavioral cloning policies (oquivalent to RT-2 bascline D Blue: Policics after Stage 2 online Sclf- Impr-veprovttttt with a minimal amount of additional episodes. Results in simulated and real Language Table, as well as the Aloha domain, demonstrate that our proposed two- stage post-training approach achieves higher success rates significantly more sample-efficiently than supervised learning alone. Our Real2Sim Language Table, and in particular BananaTable results, demonstrate that novel combination of online Self- self-Improvement and web-scale pre- traiiinnn enables policies to rapidly acquire novel skills far outside the Stage imitation learning dataset. Variations across random seeds are small, highlighting the robustness of our approach. Values above are averaged across 3 seeds (unaggregated results in Figures I0 & [T]. While Stage Language'Tabl Table datasets contain varied tasks, for fairness the x-axes in the Language Table plots above count the number of Block2Block episodes (as percentage of the total number of Block2Block episodes in the full imitation learning datasct).
The output is sequence of tokens. To employ PalI models as policies, we follow the RT-2 9 policy parameterization and predict tokenized actions. Thus, our Stage 1 behavioral cloning policies are exactly equivalent RT-2 policies which will serve as key baselines. For full details and visualizations regarding models, environments, tokenization, and training we refer readers to Appendix A
4.1
Self-Improvement is Effective, Robust, and More Efficient Than SFT Alone
4.1.1 Simulated LanguageTable
The dataset we use to train Stage policies for the simulated LanguageTabble domain the one provided by the original work [34]. This dataset consists of 181,020 human-generann generated trajectories, with 78.623 unique instructions describing the goals of the trajectories. We subsample this dataset to create 3 new datasets 10%, 20%, and 80% of the original size. For each dataset size, following Stage 1 training we perform Stage fine-tuning with 3 seeds to validate the reliability of our Self-Improvement procedure. We perform Stage 2 fine-tuning on the Block2Block subset of tasks (e.g. "move the blue moon to the red pentagon" " We stop Stage training when policy success rates appear to plateau.
Results B plot) presents our results on the simulated Language Tabble Table domain, where orange markers represent BC policy performance after Stage (equivalent to RT-2), and blue markers represent policy performance after Stage 2 Self-Improvement. As can be observed, across all dataset sizes (10% 20%, 80%), our proposed Infroooppoee procedure leads very significant increase in success rates (minimum 1.5x performance boost), with incredible sample-efficiency in terms of number of episodes (less than 2% extra episodes collected in the Self-Improvement stage). Of particular note. Self- Improeeproeeenttt with 1% additional episodes on top of the 10% dataset size leads to policies that significantly outperform BC policies trained on 20% and 80% dataset sizes. In Appendix K we also show that relf-Imprrvement is robust and reproducible across random seeds.
4.1.2 Real-World LanguageTable
The significant sample-efficiency and robustness of our results above suggest that our Self- Improvement procedure may indeed be applicable for real-world robotics. We apply our proposed approach to the real-world LanguageTable domain, in two settings of using 20% and 80% of the imitation learning dataset [34 As in the simulated setting, we perform Stage 2 fine-tuning on the
"Our analysis in Appendix L shows that Block2Block instructions make up 44% and 49% of the instructions in the simulated and real Table datasets respectively. *We run the 80% data experiment once using 3 robot stations, and run the 20% data experiment twice, once with 3 and once with 4 robot stations.

---

Block2Block subset of tasks. Given that instruction sampling, reward labeling, and success detection are entirely automated processes, during cell-Improvement a single human operator is able to monitor our full fleet of LanguageTaale robol stations. The sole responsibility of the human operator to reset a station if block falls off the table, or if station has not been shuffled for 5 minutes. Each experiments is run for approximately 20 hours. For details on the real-world Language experimentation protocol we refer the interested reader to Appendix
Results Figure B (second plot) presents our results. For both the 20% and 80% data settings, our Stage 2 Self-Improvement procedure improves policy success rate from ~62- 63% up to ~87%-88% with only ~3% additional Block2Block episodes collected. To put this into perspective, this means that with a total amount of experience equivalent to 20% (imitation dataset size) + 3% (Seff-Improvement episodes), we obtain policies that far exceed BC (RT-2) policies trained with 80% imitation dataset size! Furthermore, as opposed to the 1-to-1 tuman-to-robot ratio needed for teleop imitation data collection, celf-Improvement requires only a fraction of the human effort due to the 1-to-many human-to-robot ratio enabled by our proposed approach.
4.1.3 Simulated Aloha Single Insertion Task
We also validate our proposed fine-tuning framework on a second robot embodiment, the bimanual We design and collect data for a bimanual insertion task, where the left gripper must pick up a socket, and the right gripper must pick up a peg and insert it into the socket. Figure 7 presents visualization of this task, with videos available on our supplementary materials website. Due to the much more complex observations, 70-dim action space, and much smaller imitation datasets, this presents a challenging setting for further validation of our proposed approach. For details on the environment and data collection process, we refer readers to Appendix D.3 We create imitation dataset sizes of 5K, 10K, and 15K episodes. We apply our two-stage fine-tuning on 5K and 10K dataset sizes, and report results for supervised learning on the 15K dataset as well better situate the numbers. The differences in methodology compared to experiments LanguageTTable domain are the following: 1) Checkpoint selection for Stage 2 policy initialization (Appendix B). 2) We noticed that the success condition (the peg fully reaching the end of the socket) is not observable from the cameras, so we add a small positive constant to the reward function when the success condition is met.
Aloha manipulation platform (57 3
the
Results Figure (middle) presents results. As can be seen, for both dataset sizes Self- Improvement significantly improves policy success rates. As before, we also notice significant sample-efficiency gains where policies trained with 5K (imitation) 2.5K (Self- SelffImmprovement) episodes outperform policies trained with 10K imitation episodes (i.e. RT-2), and rival the success rate of those trained with 15K imitation episodes.
A1 Self-Improvement significantly improves policy performance beyond the supervised learning stage.
A2 The combination of supervised learning + Self-Improvement is much more sample- efficient than supervised learning alone. A3 Self-Improvement is robust and effective for real-world robot learning.
4.2 Importance of Foundation Model Pretraining
It is critical to study to what extent the success of our proposed reff-Improvement procedure is afforded by the web-scale pretraining of the PaL. [33 [3 vision-language foundation model we start from. To ablate the effect of the multimodal knowledge embedded into PaII, run our proposed two-stage fine-tuning process starting from alternative variations of the PaI.I model:
Scratch: where we use the PaLI architecture but with randomly initialized parameters.
Uni-PaLI: where the PaLI parameters are initialized from vision model and language model, each pretrained separately, unimodally, without any joint multimodal vision- -anguage fine-tuning. For details please refer to Section A.1
We compare these variations using an identical setup as Section 4.1.T on the Simulated LanguageTTable domain. Despite our best efforts and very long training runs, we observed that Stage BC policies
7

---

Ablations on Simulated LanguageTable
Self-mmprovemen LanguageTable Real2Sim Domain Transfer
1.0
Stage
0.9
Stage Scretch Reward
0.8
Stage w/ Pat Reward
Uni-FaU Reward
0.6
R07L 5m Data Pei Th
0.7
success Rate
Success
0.6
0.3
0.2
Stage PaLl Policy, Uni PaUl Reward Stage PaLl Puricy, FuLl Howard
0.2
20% Dalaset
0.0%
0.5%5
1.0%
1.5%
2.0%
2.5%
Stage 2 Episodes las % Lanquagetable sim dataset)
Figure 4: Left Ablation results demonstrate the critical role of the web-scale pretraining of foundation models for enabling effective Stage training, in particular in the small dataset size regime. Right "Success Rate" plots during Stage Self- self-lmmprveemnt on Language Table Real2Sim domain transfer task. Reward labels from the PaI.I model lead significantly faster Self-Improvement comparison to the Uni-Pal.I model.
derived from the Scratch and Uni-PaLI variations very significantly underperformed PaL. BC policies. Hence, we focus ablations on the Self- lef--Improvement stage, where initialize policies from PaLI Stage checkpoints, and use Scratch or Uni-PaLI checkpoints for reward computation.
Results Figure (left) presents our results. There a clear ordering in performance: PaLI reward models are best, followed by Uni-PaLL and then Scratch. Scratch reward models lead high variance results across random seeds, and struggle to provide any meaningful improvements in low-data (10% & 20%) regimes. While better than Scratch, Uni-Pat reward models perform significantly worse than PaI.I reward models across the board, with the gaps more pronounced at lower data settings. In fact, Self-Imppovement with the PaI.I II reward model in the 20% dataset size regime leads to better policies than Sepoolmtrovement with the Uni-Pal.I reward model in the 80% regime! These results clearly demonstrate the immense value of multimodal pretraining for Self- Self-Improvement.
A4: Multimodal pretraining leads to significantly better Self-Improve policies, and is key enabler of sample-efficiency.
4.3 Generalization
The novel combination of our proposed online Self- Self-Improvement process and the use of pretrained multimodal foundation models unlocks a unique capability: enabling policies to practice novel tasks that generalize beyond what was covered by the Stage imitation learning datasets. this section we present results for two increasingly difficult forms of generalization.
4.3.1 Domain Transfer Between Simulation and Real
Starting with a simpler form of generalization, in this section we investigate domain transfer between simulation and real. Sim2Real is important class of approaches that can significantly reduce the amount of real-world experience needed to train performant robot policies, and has been successfully applied in many settings (3 investigate the inverse problem, Real2Sim transfer, on the LanguageTable Table domain. We train Stage I models using 80% of the in the simulated Language Tanbl environment. Similar to our ablation in Section 4.2 we also train Stage 2 models using the Uni-PaLI reward model variant highlight the role of foundation model pretraining in enabling domain transfer.
30). To make experimentation simpler, in this section we
LanguageTTable dataset, and perform Stage Self-Imprrvement
Results Figure 4 (right) presents our results. With only 3% extra episodes in the target domain our Self-Improvemmet procedure improves policy performance from ~22% to 559%. This performance is equivalent to BC policies trained with 80% of target domain's 's imitation dataset. Additionally, Figure (right) demonstrates that the Uni-PaLI reward model leads to significantly slower Self-Improvement procedure, highlighting the key advantage of pretraining. Given our strong real-world T.anguageTablle results in section 4.1.2 we expect our Real2Sim results to be strongly indicative of Sim2Real transfer well.
(simulated LanguageTTabl

---

Before Stage Fine-Tuning (Instruction: Move the banana the left center the board)
After Stage 2 Fine-Tuning (Instruction: Move the banana to the top center of the board)
LanguageTable Robot View
BananaTable Robot View
Figure 5:
Strong Generalization to BananaTTable. Top Before Stage 2 fine-tuning on the Banana domain, the policy struggles to effectively maneuver banana across the table due to the difficult geometry. Bottom Left After Stage tine-tuning policies visibly more proficient the BananaTable task (videos on supplementary wehsite). Bottom Right Prior Stage Banana fine-tuning, the policy and reward models have never seen the BananaTTable task, creating very challenging generalization problem.
4.3.2 Strong Generalization to Learning Novel Skills
Moving towards stronger form of generalization, we investigate whether Self- Ielffllmpoveeeettt with pretrained foundation models enables policies to practice and acquire novel behavioral skills beyond those observed in the imitation learning datasets used in Stage 1. Starting from policy and reward model trained with the real-world Language'aabe Table datase we perform Self- self-immrovement on a new task we dub "Banana'Tabl Table" (Figure { In this task we replace the Language' Table blocks with single prosthetic banana and request policies to push the banana to various locations on the board (e.g. "move the banana to the left center of the board"). The Language Table dataset contains no bananas, nor any episodes where the blocks are not on the table. Thus we are solely relying on the generalization abilities of the underlying PaLI foundation model from which the policy, reward function, and success detector are derived.
In contrast to prior works that have demonstrated semantic generalization abilities of robot foundation models (e.g. executing the same pick and place motions in novel contexts RT-2 ()), transfer the BananaTabh Table task requires behavioral generalization, necessitating policies to learn new skills. As an example, due its elongated geometry, inaccurate pushing of the banana results in it rotating around itself instead of moving in the intended direction (Figure 5 top).
Results Within ~88 hours of Self- telf-Improvement using 2 robot stations, the policy success rate improves from ~63% to ~85%. Beyond the quantitative results, videos on our supplementary wehsite (as well as Figure 5) demonstrate that the policy becomes visibly more proficient at accomplishing the BananaTable tasks, as picks up on effective strategies for moving the banana around the table. After Self- Improvemtteett,, the policy learns to push from either the middle or the tips rather than the rest of the banana to prevent from rotating around itself.
A5: The novel combination of our proposed online Self- Leff-Improvement procedure and web-scale pretrained foundation models enables policies to rapidly acquire new skills that generalize far beyond the tasks covered by the imitation learning datasets they are provided.
5
Related Works
Due to space limitations, here we include a more concise overview of related works. For a more detailed discussion, please refer to [ppendix[]
Embodied Foundation Models A number of prior works have leveraged pretrained multimodal foundation models robot policies, by incorporating action prediction heads. Brohan et al. 9 discretize continuous robot actions and map them onto language model token spaces. Other exam- ples architectures include diffusion models [39] (54) (building diffusion policies [15] (2)), flow matching 77 (29) (building on (33), ), and regression 31. Critically, these prior works only leverage supervised learning, and the delineation between pretraining and post-training the task and data
WW initialize the BananaTable Self- Belf-Improvement procedure using the reward model and Self leff-Improeed
policy from the 80%-dat experiment in Section 4.1.2 4.1.2

---

mixture used. To the best of our knowledge, our work is the first to move past supervised learning for training robot foundation models. The key contribution of our work to present ceneral-purpor soward-engineering
online Seff-Improvement procedure that not only leads to rapid policy improvement, but enables acquisition of novel hehaviors outside of what the models have seen in their training data. This form of behavioral generalization is only achievable through online post-training.
Improving Robot Policies Without Ground-Trun Rewards A key obstacle of pererae-purpose Self- seff-Improvement through RL is that commonly do not have access to ground-trute reward functions and success metrics, either due to the challenge of designing one (reward engineering), or difficulty in measuring them in the real world (reward instrumentation). A line of prior work, dubbed CCode-As-Rewandde ,, leverages L.Ms to write code for reward engineering [36] [56] 53 Such approaches have number of downsides that make them impractical for general-purpose robot there exists rich literature
learning in the real world (Appendix D. Aside from Code-As-Reward
on data-driven reward functions. An important class of works 66 35 46 12] learn latent observation representations on top of which reward and value functions can be defined (e.g. via L2 distance in la- tent space to target goals). Other approaches include creative relabeling techniques (32
and RL objectives 22 In comparison, the key advantage of our proposed approach is the straightforward integration with web-scale pretrained foundation models. The closest related works to ours are those based on learning distances in timesteps (26 (27) Aside from key differences in settings, our focus in this work is on post-training Embodied Foundation Models and generalization, demonstrating that steps-to-go policy improvement is a viable path towards general-purpose Self-Improving robot policies. Additionally, we present steps-to-go thresholding as a novel path towards obtaining robust open-ended success detectors, which have typically been trained via binary classification [9]. Lastly, Yang et al. (55) extend our approach to real-world simulators built on generative video foundation models, and Ma et al. 37 present an extension to an in-context learning formulation.
Future Work and Limitations
Due to space limitations, here include more concise overview of future work and limitations. For a more detailed discussion, please refer to Appendix []] Our work has clearly demonstrated the immense potential of the novel combination of pretrained multimodal foundation models with online Self-Improvement. This combination not only enables very efficient policy improvement on real-world robots, but also unlocks strong generalization capabilities and autonomous acquisition of new skills. There still exist, however, many important avenues for future work. Algorithmic: In this work we leveraged on-policy REINFORCE for simplicity which does not reuse any collected data during Self- reff-lmprovemet. Off-policy methods have the potential to even more substantially boost sample-efficiency. Skill-Chaining: How can Seff-Improvement and our proposed success detector be extended to enable skill-chaining and solving long-horizon tasks? Robustness: Since imitation datasets typically contain expert behavior, they do not contain many forms of failure cases and recovery behaviors. How can our post- post-training approach be extended to ensure the reliability of reward models and success detectors on out-of-distribution failure cases? We hope that the strong results presented in this work motivate investigation of these fruitful research avenues.
7 Conclusion
Drawing inspiration from the success of the reinforcement learning stage in fine-tuning large language models, in this work we proposed a two-stage post-training approach for Embodied Foundation Models. The first stage, Supervised Fine-Tuning (SFT), fine-tunes pretrained multimodal foundation models using two objectives: a) behavioral cloning, and b) steps-to-go prediction. The second stage, online Self- infrommprovettt, leverages steps-to-go prediction for RL post-training. With no task-specific reward engineering and minimal human supervision. this stage enables a fleet robots to autonmously practice downstream tasks and aquire new skills. Through extensive experiments in the real-workl and simulated domains we have demonstrated that Self-Improo telf-Improvement significantly improves policy performance beyond the supervised learning stage, and that the combination of supervised learning and Self- Improvemet is much more sample-efie efficl-effcc than supervised learning alone. We then showed that combining web-scale pretraining with Sepfo-mmrove leads to significantly better Self-Improved policies, and is key enabler of sample-efficienyy. Finally. we demonstrated that this novel combination uniquely unlocks a capability not possible by current methods: autonomously aquiring new skills that generalize far beyond the tasks covered in the imitation learning datasets. These findings highlight the transformative potential of combining pretrained foundation models with online self-Improvement to enable autonomous skill acquisition in robotics.
10

---

References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidr, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08777. 2023.
[2] Ilge Akkaya, Marcin Andrychowicz. Maciek Chociej. Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's s cube with robot hand. arXiv preprint arXiv:1910.07113 2019.
[3] Jorge Aldaco, Travis Armstrong. Robert Baruch, Jeff Bingham, Sanky Chan, Kenneth Draper, Debidatta Dwibedi, Chelsea Finn, Pete Florence, Spencer Goodrich, et al. Aloha 2: An enhanced low-cost hardware for bimanual teleoperation. arXiv preprint arXiv:2405.0224 02292, 2024.
[4] Gordon Alexander and Alexandre M Baptista. A comparison of var and ovar constraints on portfolio selection with the mean-variance model. Management science, 50(9):1261-1273, 2004.
[5] Marc G. Bellemare, Will Dabney, and Mark Rowland. Distributional Reinforcement Learning. http://www.itttrbbeoo.oo wwwssstriuuiiona--tttt
MIT Press, 2023. http:///ww..iistribbbbiiii
[6] Chethan Bhateja, Derek Guo, Dibya Ghosh, Anikait Singh, Manan Tomar, Quan Vuong, Yevgen Chebotar, Sergey Levine, and Aviral Kumar. Robotic offline from internet videos via value- function pre-training. arXiv preprint arXiv:23009 13041, 2023.
[7] Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. To: A vision- saiion-laguuae-actinnn flow model for general robot control. arXiv preprint arXiv:2410.24100 2024.
[8] Konstantinos Bousmalis, Giulia Vezzani, Dushyant Rao, Coline Devin, Alex Lee, Maria Bauza, Todor Davchev, Yuxiang Zhou, Agrim Gupta, Akhil Raju, al. Robocat: A self- improving foundation agent for robotic manipulation. arXiv preprint arXiv:2306. 17706, 2023.
[9] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choro- manski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision- languag-- action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15816 2023.
[10] Albin Cassirer, Gabriel Barth-Maron, Eugene Brevdo, Sabela Ramos. Toby Boyd, Thibault Sottiaux, and Manuel Kroiss. Reverb: framework for experience replay, 2021.
[11] Yevgen Chebotar, Karol Hausman, Yao Lu, Ted Xiao, Dmitry Kalashnikov, Jake Varley, Alex Irpan, Benjamin Eysenbach, Ryan Julian, Chelsea Finn, al. Actionable models: Unsupervised offline reinforcement learning of robotic skills. arXiv preprint a7Xiv:2104.007749 2021.
[12] Annie S Chen, Suraj Nair, and Chelsea Finn. Learning generalizable robotic reward functions from" " in-the-wild" human videos. arXiv preprint arXiv:2103.168177 2021.
[13] Xi Chen. Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski. Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, al, Pali: A jointly-scaled multilingual language-image model. arXiv preprint 22Xiv:2209.06777 2022.
[14] Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Carlos Riquelme Ruiz, Sebastian Goodman. Xiao Wang, Yi Tay, et al. Pali-x: On scaling up multilingual vision and language model. arXiv preprint arXiv:2305.18565 2023.
[15] Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusson policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.041377 2023.
[16] Cheng Chi, Zhenjia Xu, Chuer Pan, Eric Cousineau, Benjamin Burchfiel, Siyuan Feng. Russ Tedrake, and Shuran Song. Universal manipulation interface: In-the-wild robot teaching without in in-the-wild robots. arXiv preprint 2rXiv:2402.10329 2024.

---

[17]
Open X-Emboddmmnt Collaboration, Abby O'Neill, Abdul Rehman, Abhiram Maddukuri, Ab- hishek Gupta, Abhishek Padalkar, Abraham Lee, Acom Pooley. Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, Albert Tung, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anchit Gupta, Andrew Wang, Andrey Kolobov, Anikait Singh, Animesh Garg, Aniruddha Kembhavi, Annie Xie, Anthony Brohan, Antonin Raffin, Archit Sharma, Arefeh Yavary, Arhan Jain, Ashwin Balakrishna, Ayzaan Wahid, Ben Burgess-Limens
Beomjoon Kim, Bernhard Schölkopf, Blake Wulfe, Brian Ichter, Cewu Lu, Charles Xu, Charlotte Le, Chelsea Finn, Chen Wang. Chenfeng Xu, Cheng Chi, Chenguang Huang, Christine Chan, Christopher Agia, Chuer Pan, Chuyuan Fu, Coline Devin, Danfei Xu, Daniel Morton, Danny Driess, Daphne Chen, Deepak Pathak, Dhruv Shah, Dieter Büchler, Dinesh Jayaraman, Dmitry Kalashnikov. Dorsa Sadigh, Edward Johns, Ethan Foster, Fangchen Liu, Federico Ceola, Fei Xia, Feiyu Zhao, Fe- lipe Vieira Frujeri, Freek Stulp, Gaoyue Zhou, Gaurav S. Sukhatme, Gautam Salhotra, Ge Yan, Gilbert Feng, Giulio Schiavi, Glen Berseth, Gregory Kahn, Guangwen Yang. Guanzhi Wang, Hao Su, Hao-Shu Fang, Haochen Shi, Henghui Bao, Heni Ben Amor, Henrik I Christensen, Hiroki Furuta, Homer Walke, Hongjie Fang. Huy Ha, Igor Mordatch, Ilija Radosavovic, Isabel Leal, Jacky Liang. Jad Abou-Chakra, Jachyung Kim, Jaimyn Drake, Jan Peters, Jan Schneider, Jasmine Hsu, Jeannette Bohg, Jeffrey Bingham, Jeffrey Wu, Jensen Gao, Jiaheng Hu, Jiajun Wu, Jialin Wu, Jiankai Sun, Jianlan Luo, Jiayuan Gu, Jie Tan, Jihoon Oh, Jimmy Wu, Jingpei Lu, Jingyun Yang, Jitendra Malik, João Silvério. Joey Hejna, Jonathan Booher, Jonathan Tompson. Jonathan Yang, Jordi Salvador. Joseph J. Lim, Junhyek Han, Kaiyuan Wang, Kanishka Rao, Karl Pertsch. Karol Hausman, Keegan Go. Keerthana Gopalakrishnan, Ken Goldberg, Kendra Byrne Kenneth Oslund, Kento Kawaharazuka, Kevin Black, Kevin Lin, Kevin Zhang, Kiana Ehsani, Kiran Lekkala, Kirsty Ellis, Krishan Rana, Krishnan Srinivasan, Kuan Fang, Kunal Pratap Singh, Kuo-Hao Zeng. Kyle Hatch, Kyle Hsu, Laurent Itti, Lawrence Yunliang Chen, Lerrel Pinto, Li Fei-Fei, Liam Tan, Linxi "Jim" Fan, Lionel Ott, Lisa Lee, Luca Weihs, Magnum Chen, Marion Lepert, Marius Memmel, Masayoshi Tomizuka, Masha Itkina, Mateo Guaman Castro, Max Spero, Maximilian Du, Michael Ahn, Michael C. Yip, Mingtong Zhang, Mingyu Ding, Minho Heo, Mohan Kumar Srirama, Mohit Sharma, Moo Jin Kim, Naoaki Kanazawa, Nicklas Hansen, Nicolas Heess, Nikhil Joshi, Niko Suenderhauf, Ning Liu, Norman Di Palo, Nur Muhammad Mahi Shafiullah, Oier Mees, Oliver Kroemer, Osbert Bastani, Pannag R Sanketi, Patrick "Tree" Miller, Patrick Yin. Paul Wohlhart, Peng Xu, Peter David Fagan, Peter Mitrano, Pierre Sermanet, Pieter Abbeel, Priya Sundaresan, Qiuyu Chen, Quan Vuong, Rafael Rafailov, Ran Tian, Ria Doshi, Roberto Mart' -art in, Rohan Baijal, Rosario Scalise, Rose Hendrix, Roy Lin, Runjia Qian, Ruohan Zhang, Russell Mendonca, Rutav Shah, Ryan Hoque, Ryan Julian, Samuel Bustamante, Sean Kirmani, Sergey Levine, Shan Lin, Sherry Moore, Shikhar Bahl, Shivin Dass, Shubham Sonawani, Shuran Song, Sichun Xu, Siddhant Haldar, Siddharth Karamcheti, Simeon Adebola, Simon Guist, Soroush Nasiriany. Stefan Schaal, Stefan Welker, Stephen Tian, Subramanian Ramamoorthy, Sudeep Dasari, Suneel Belkhale, Sungjae Park, Suraj Nair, Suvir Mirchandani, Takayuki Osa, Tanmay Gupta, Tatsuya Harada, Tatsuya Mat- sushima, Ted Xiao, Thomas Kollar, Tianhe Yu, Tianli Ding, Todor Davchev, Tony Z. Zhao, Travis Armstrong, Trevor Darrell, Trinity Chung, Vidhi Jain, Vincent Vanhoucke, Wei Zhan, Wenxuan Zhou, Wolfram Burgard, Xi Chen, Xiangyu Chen, Xiaolong Wang, Xinghao Zhu, Xinyang Geng, Xiyuan Liu, Xu Liangwei, Xuanlin Li, Yansong Pang. Yao Lu, Yecheng Ja- son Ma, Yejin Kim, Yevgen Chebotar, Yifan Zhou, Yifeng Zhu, Yilin Wu, Ying Xu, Yixuan Wang, Yonatan Bisk, Yongqiang Dou, Yoonyoung Cho. Youngwoon Lee, Yuchen Cui, Yue Cao, Yuah-Hua Wu, Yujin Tang, Yuke Zhu, Yunchu Zhang, Yunfan Jiang, Yunshuang Li, Yunzhu Li, Yusuke Iwasawa, Yutaka Matsuo, Zehan Ma, Zhuo Xu, Zichen Jeff Cui, Zichen Zhang, Zipeng Fu, and Zipeng Lin. Open X-Embodiment: Robotic learning datasets and RT-X models. https://arxis org/ abs/2310 08864 2023.
[18) Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong. Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arX.v.2303.03338 2023.
[19] Yuqing Du, Ksenia Konyushkova, Misha Denil, Akhil Raju, Jessica Landon, Felix Hill, Nando de Freitas, and Serkan Cabi. Vision-language models as success detectors. arXiv preprint arX1v:2303.07280, 2023.
[20] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Ilama 3 herd
12

---

of models. arXiv preprint arXiv:2407.21783 2024.
[21] Zane Durante, Bidipta Sarkar, Ran Gong, Rohan Taori, Yusuke Noda, Paul Tang, Ehsan Adeli, Shrinidhi Kowshika Lakshmikanth, Kevin Schulman, Aroold Milstein, et al. An interactive agent foundation model. arXiv preprint (.XXi::2402.05922, 2024.
[22] Benjamin Eysenbach, Tianjun Zhang, Sergey Levine, and Russ R Salakhutdinov. Contrastive learning as conl-conditooned reinforcement learning. Advances in Neural Information Process- ing Systems, 35:35603-35620 2022.
[23] Jesse Farebrother, Jordi Orbay, Quan Vuong, Adrien Ali Taiga, Yevgen Chebotar, Ted Xiao, Alex Irpan, Sergey Levine, Pablo Samuel Castro, Aleksandra Faust, et al. Stop regressing: Training value functions via classification for scalable deep rl. arXiv preprint arXiv:2403.03950, 2024.
[24] Raj Ghugare, Matthieu Geist, Glen Berseth, and Benjamin Eysenbach. Closing the gap between td learning and supervised learning a generalisation point of view. arXiv preprint MiXiv:2401.11237 2024.
[25] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint 1xXi::5501.12448, 2025.
[26] Kristian Hartikainen, Xinyang Geng, Tuomas Haarnoja, and Sergey Levine. Dynamical distance learning for semi-supervised and unsupervised skill discovery. arXiv preprini arXiv: 99990000022 2019.
[27] Joey Hejna, Jensen Gao, and Dorsa Sadigh. Distance weighted supervised learning for offline interaction data. In International Conference Machine Learning, pp. 12882-12906. PMLR, 2023.
[28] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-688 6851, 2020.
[29] Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn. Niccolo Fusai, et al. To.5: a vision- language-action model with open-world generalization. arXiv preprint arXi::2504116544 2025.
[30] Satoshi Kataoka, Youngseog Chung, Seyed Kamyar Seyed Ghasemipour, Pannag Sanketi, Shixiang Shane Gu, and Igor Mordatch. Bi-manual block assembly via sim-to-real reinforcement learning. arXiv preprint arXiv:2303. 14870, 2023.
[31] Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-actinn model. arXiv preprint 2XXx:22406.09246, 2024.
[32] Aviral Kumar, Anikait Singh, Frederik Ebert, Mitsuhiko Nakamoto, Yanlai Yang, Chelsea Finn, and Sergey Levine. Pre-training for robots: Offline enables learning new tasks from handful 2022.
of trials. arXiv preprint
[33] Yaron Lipman, Ricky TQ Chen, Heli Ben- Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022.
[34] Corey Lynch, Ayzaan Wahid, Jonathan Tompson, Tianli Ding, James Betker, Robert Baruch, Travis Armstrong, and Pete Florence. Interactive language: Talking to robots in real time, IEEE Robotics and Automation Letlers, 2023.
[35] Yecheng Jason Ma. Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. Vip: Towards universal visual reward and representation via value-implicit pre-training. arXiv preprint arXiv:2210.000330 2022.
[36] Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Fureka: Human-level reward design via coding large language models. arXiv preprint arXiv:2310.12931, 2023.
13

---

[37] Yecheng Jason Ma, Joey Hejna, Chuyuan Fu, Dhruv Shah, Jacky Liang, Zhuo Xu, Sean Kirmani, Peng Xu, Danny Driess, Ted Xiao, et al. Vision language models are in-context value learners. In The Thirteenth International Conference on Learning Representations, 2024.
[38] Andrew Y Ng. Daishi Harada, and Stuart Russell. Policy invariance under reward transforma- tions: Theory and application to reward shaping. In Ieml, volume 99, pp. 287, 1999.
[39] Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Charles Xu, Jianlan Luo, Tobias Kreiman, You Liang Tan, Lawrence Yun- liang Chen, Pannag Sanketi, Quan Vuong, Ted Xiao, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. Octo: An open-source generalist robot policy. In Proceedings of Robotics: Science and Systems, Delft, Netherlands, 2024.
[40] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 11-27730-277444 2022.
[41] Abhishek Padalkar, Acorn Pooley, Ajinkya Jain, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anikait Singh, Anthony Brohan, et Open x-embodiment: Robotic learning datasets and rt-x models. arXiv preprint arXiv:2310.0886. 2023.
[42] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International conference on machine learning, pp. 4055-4064. PMLR, 2018.
[43] Lerrel Pinto, Marein Andrychowicz, Peter Welinder, Wojciech Zaremba, and Pieter Abbeel. Asymmetric actor critic for image-based robot learning. arXiv preprint arXiv:1710.0653 2017.
[44] Kanishka Rao, Chris Harris, Alex Irpan, Sergey Levine, Julian Ibarz, and Mohi Khansari. RI- cyclegan: Reinforcement learning aware simulation-to- ro-rl.... In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11157-11166, 2020.
[45] Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.0611 2022.
[46] Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE international conference on robotics and automation (ICRA), pp. 1134-114 1141. IEEE, 2018.
[47] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-300 -0001, 2020.
[48] Jie Tan, Tingnan Zhang, Erwin Coumans, Atil Iscen, Yunfci Bai, Danijar Hafner, Steven Bohez, and Vincent Vanhoucke. Sim-to-real: Learning agile locomotion for quadruped robots. arXiv preprint arXiv: 10332, 2018.
[49] Gemini Robotics Team, Saminda Abeyruwan, Joshua Ainslie, Baptisee Alayrac Montser- rat Gonzalez Arenas, Travis Armstrong, Ashwin Balakrishna, Robert Baruch, Maria Bauza, Michiel Blokzijl, et al. Gemini robotics: Bringing ai into the physical world. arXiv preprint arXiv:2503.20024 2025.
[50] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre. Morgane Rivière, Mihir Sanjay Kale, Juliette Love, al. Gemma: Open models based on gemini research and technology. arXiv preprint 2xXiv:2403.08295 2024.
[51] Hado Van Hasselt, Yotam Doron, Florian Strub, Matteo Hessel, Nicolas Sonnerat, and Joseph Modayil. Deep reinforcement learning and the deadly triad. arXiv preprint arXiy: 1 02668, 2018.
14

---

[52] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[53] David Venuto, Sami Nur Islam, Martin Klissarov, Doina Precup, Sherry Yang, and Ankit Anand. Code as reward: Empowering reinforcement learning with vlms. arXiv preprint 4rX1v:2402.04764, 2024.
[54] Junjie Wen, Yichen Zhu, Jinming Li, Minjie Zhu, Kun Wu, Zhiyuan Xu, Ning Liu, Ran Cheng, Chaomin Shen, Yaxin Peng, et al. Tinyvla: Towards fast, data-efficient lasson-language-cction models for robotic manipulation. arXiv preprint arXiv:2409. 12514, 2024.
[55] Mengjiao Yang, Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Pieter Abbeel. Learning interactive real-world simulators. arXiv preprint 4rXiv:2310.06114, 2023.
[56] Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Sean Kirmani, Kuang-Huel Lee, Montse Gonzalez Arenas, Hao- Lewis Chiang, Tom Erez, Leonard Hasenclever, Jan Humplik, et al. Language to rewards for robotic skill synthesis. arXiv preprint arXiv:2306.000 086477, 2023.
[57] Tony Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn, Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304. 13705, 2023.
[58] Tony Z. Zhao, Jonathan Tompson, Danny Driess, Pete Florence, Kamyar Ghasemipour, Chelsea Finn, and Ayzaan Wahid. Aloha unleashed: A simple recipe for robot dexterity. arXiv preprint 2xX1:24400.13166 2024.
15

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We provide extensive experiments for each claim made, and answer all of the key questions laid out in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
16

---

Justification:
Please refer to Section 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sele--peeiiiainnn. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA].
Justification: Our work does not provide any major theoretical results. Our discussion on Mathematical Intuition discusses simple algebraic manipulations.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We present our method in full detail, with further experimental details in the Appendix. We also include a Colab notebook as a pedagogical implementation of our algorithm.
17

---

Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-sonnee open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We are unable to opensource our code. However, we include a Colab notebook as a pedagogical implementation of our algorithm. The datasets we use for imitation learning are existing opensourced datasets.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. www.iiiccguieeeeeee
ssionPolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
18

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide extensive details in the main text as well as Appendix sections.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core paper to level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All experiments in the simulated domains were run with a minimum of random seeds. Aside from Figure all plots (including the ones in the Appendix) contain either raw data or error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide discussion in Appendix A.4
Guidelines:
19

---

The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
We have read the NeurIPS Code of Ethics and conform the code.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification:
We include discussion in Appendix O
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
20

---

Justification:
We do not release any models or datasets.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: provide citations for data, model, and simulation sources. Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL..
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, thcors thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification: In supplementary materials include a Colab notebook with pedagogi- cal implementation of our proposed method. We discuss the setup in our manuscript.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
21

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We do not have crowdsourcing or experiments with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
We do have experiments requiring such approvals.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: Pretrained multimodal foundation models are a key component our work and provide detailed discussions in manuscript.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our L.LM policy hhcppp//nearrrjsccc/oofeeeeeeeee for what should or should not be described.
22

---

A Implementation Details
A.1 Background
PaLi Vision-Language Foundation Model Our investigations in this work are independent of the choice of underlying foundation model used. Throughout this work we use the billion parameter PaLI-3B [13 [4] vision-language model as the base pretrained foundation model that we fine-tune for robotics tasks, PaLI model receives as input one or more images alongside text, and provides text output. At a high level, the PaLI architecture is comprised of two components: 1) Vision Transformer (ViT) 42). and 2) an encoder-decoder Transformer (52). Input images are processed by the VIT into sequence of "visual tokens". The sequence of visual tokens is concatenated with the tokenized text input and fed into the Transformer which outputs text tokens. The weights of the PaL. model are initialized from Transformer language model and VIT vision model that pretrained separately in a unimodal fashion. Following this initialization, the model is fine-tuned with a variety of rision-languag training objectives to obtain a multimodal foundation model. For further details regarding the PaLI model, we refer the interested reader to 13 14.
RT-2 Brohan et al. [9] introduce model family, dubbed RT-2, that enables vision-languee foundation models (VLMs) to directly produce low-level robot actions for closed-loop control. The two VLMS considered in that work are PaI.I 13 14 and MM- [1], both of which take images alongside text as input, and provide output in the form of text tokens. To enable these VLMSs to act as robot policies, continuous robot actions are discretized and mapped onto the text token space of the VLMs. Given image and text inputs, the VLMs are fine-tuned via behavioral cloning (i.e. supervised learning) predict the tokenized robot actions. While the methods we present this work are independent of the choice of underlying model, throughout this work our robot policy architectures are equivalent RT-2 using the PaLI VLM.
As we use the RT-2 policy representation, we also decided to model steps-to-ge to-soo--o predictions by discretizing the range of possible number of steps, and mapping them onto the PaLI VLM token space.
A.2 Environments, Tasks, and Tokenization
For details about the environments and tasks used in this work, please refer to Appendix D For details about tokenization, please refer to Appendix E
A.3 Training Details
Stage 1 (Supervised Fine-Tuning! During the supervised training stage we uniformly distribute each training batch amongst the objectives used in Stage 1. For all domains considered in this work this includes the a) behavioral cloning, and b) steps-to-go prediction objectives. In the real-world and simulated Language Table domain experiments, we have an additional objective e) predicting the episode instruction given the first and last frame of an episode. We did not ablate the value of incorporating this objective during training. We used batch size 128 during this stage, used the AdamW optimizer, and trained the entire Pal.I model (i.e. kept no component frozen).
Stage 2 (Self- (eero--prro During this stage we used batch size 64 to require less real-world rollouts for given number of desired training steps. We kept the VIT portion of the model frozen, intuitively believing that the model has already learned visual features for the task, and that freezing the ViT may potentially help with model stability. We did not ablate this decision. We used the same A optimizer as in Stage The algorithm box in Section 2.2 presents the psuedocode for our proposed Stage 2 Self- eeprovmmpeeeenntt procedure. In each RL loop, we collect enough robot trajectories to perform 16 model update steps (N = 16). Intuitively, decreasing
reduces off off-policiness of the RL updates, while increasing N improves the diversity of data in the replay buffer due to the larger number of trajectories being collected before performing N RL updates.
A.4 Compute Resources
Stage 1 (SFT) training was done using one of the following configurations, interchangeably:
23

---

64 TPUv4 (2x4x4)
128 TPUv3
For Stage (Self-Imppnl (Self-Improvement) used:
Half of SFT stage resources for the learner job (since we used half batch size)
TPUv4 (2x2x1) for the reward model
4 TPUv4 (2x2x1) for the success detector
B
Checkpoint Selection for Stage 2 Initialization
The frozen checkpoint used for reward computation and success detection is not necessarily identical to the checkpoint used for policy initialization since the best performance for steps-to-go and behavioral cloning (BC) objectives can happen at different points over the course of Stage training. For the most part throughout this work took the checkpoints the best validation loss for the corresponding objective. An exception to this was how we chose the policy initialization checkpoint in the Aloha domain. We observed that at the best validation loss (~5K-I0K steps into training) the BC policy did not have a reasonable success rate. Allowing the model to continue training for much longer and overfitting the validation loss (~ 1000K000KK steps into training) improved the policy success rate substantially.
C REINFORCE Multiplicative Constant
We perform policy updates using the REINFORCE loss,
EFA -c- Rr log (astion (axlor,))
In simulation experiments we found that using a small positive multiplicative factor in the RE- INFORCE loss plays significant role in ensuring the model trains stably. Note that this is not equivalent to scaling the learning rate due to interactions with regularizers such as weight decay. Throughout this work we use 5e-2. We did not perform any careful tuning of and chose its approximate scale using the following intuition: Let y denote the discount factor being used. If we assume the policy gets N steps closer to the goal after every timestep, we have,
N
R Σ N
1
Intuitively, we would like to make the weights on the log probability fall approximately into the range =.2 We use 7 = 0.9 throughout this work,
-1 to 1 (i.e. -c- R. E [-1, 1J). Thus we have and hypothetically assume that the range of N is approximately [-2, 2] (e.g. we believe the Stage 2
policy can become twice as efficient as the BC policy 10) This results in our choice of c = 5e-2.
D Environments and Tasks
D.1
LanguageTable
shows the real- world and simulated LanguageTTable environments used in this work. The Language Table domain (34] has 2D action space representing delta movement in the x-y plane. The dataset we used in Stage (SFT) are the ones provided by the original work (3 introducing this domain. This dataset consists of 18 1,020 human- human-generann generated trajectories, with 78,623 unique instruc- tions describing the goals of the trajectories. The tasks we perform Stage 2 (Self-Improvemeng on the Block2Block subset of tasks which contain instructions of the form "move the blue cube to the green star". As noted in Appendix L. for the simulated and real dataset respectively, 47% and 49% of the instructions fall under the Block2Block tasks. The two images given to PaLI represent the current and previous frame as viewed by the LanguageTabll robot camera.
Figure 6
Note that this is approximate intuitive guess and does not need to be precise. A poor guess simply affects the scale of the loss and does not constrain policy learning in any manner.
24

---

EA5E67
Figure 6: Language Table Environments. Left The four Language Table robot stations used for our real-world experiments. Right, Top Camera view of the real-world Language'Table robot station. Right, Bottom Camera view of the simulated LanguageTable robot station.
D.2 SananaTable
In the BananaTable task we remove all blocks from the LanguageTabll stations and replace them with a single banana. The instructions for the BananaTable task have the form, "x the banana to the Y of the table.", where set of verbs synonomous with pushing, and Y is one of left, top left, top center, top right, right, bottom right, bottom, bottom left, center.
D.3 Aloha
The Aloha domain [57 is bimanual robot station with 14 degrees of freedom and controlled via joint position commands. As opposed the default of predicting 50Hz actions, we predict 10Hz actions. A common design choice in the Aloha domain is to train policies to predict N actions into the future. This is commonly referred as action-chunking (57]. or action horizon [1]]. We use N = which results in action space that is 70-dimensional (14 x 5). During rollouts, we execute the full action-chunk. In the Aloha domain, as input we also provide the model with the current joint positions, i.e. we append 14 tokens to the input text instructions, where each token represents number from 0-255. For details on tokenization, please refer to Appendix
The Aloha environment has 4 cameras (Figure right). To turn them into two images to pass to our PaLI models, we vertically stack two images into one image with black buffer in between. We stack the top and table view images to form the first image, and stack the left and right wrist view to form the second image. We add small black band between the stacked views inside each image in hopes of better delineating them. Since we pass 224 224 images to PaI.I, this means that each Aloha camera view appears with an effective resolution of about x 100. This is significantly less resolution than the typ- ical Aloha resolution 480 640 (58).
Figure 7: The four camera views in the simulated Aloha Bingle-Insertie task.
We designed and collected data for a bimanual insertion task, where the left gripper must pick up a socket, and the right gripper must pick up a peg and insert that peg into the socket. We collected 800 demonstrations using a VR headset to display the Mujoco simulation, and using the real-world Aloha leader robots to control the virtual robots. the 800 demonstrations and used the model to generate 3 datasets of size, 5K, 10K, and 15K. Note that these datasets only contain successful rollouts, and max episode lengths was chosen generously (1500 steps) to allow for recovery from mistakes.
then trained small diffusion policy (15) on
25

---

Critical to successful PaLI policies was to employ semi-globar action representations as in Chi et al. [Tó], as well training Stage 1 (SFT) far beyond the point at which the best validation loss was obtained for the behavioral cloning loss (Appendix B
E
Tokenization
E.1 Real/Sim LanguageTabl Table & BananaTabhe
We use the same tokenization approach for the real-world and simulated LanguageTabll, as well as the Banana Table domains.
Action Tokenization As noted in Appendix D.I. the above domains have 2D continuous action space. We represent Language actions a sequence tokens:
1. token for +/-
2. token representing number in the range [0, 10]
3. token for +/-
4. token representing a number in the range [0, 10]
The continuous 2D actions are binned to fall into this representation.
Steps-to-go Tokenization We computed the upper percentile of episode lengths the imitation dataset be 100 steps. We discretized the range from 0 100 steps into 50 bins, and represented each bin using single token.
E.2 Aloha
Action Tokenization As discussed in D.3 our action space is 5 x 14 dimensions. We represent each dimension with token, meaning the model outputs tokens. Each token represents a number from 0-255. The continuous Aloha actions which are in the range [-1, 1] are discretized and binned into these 256 bins.
Steps-to-go Tokenization The upper bound episode lengths in the imitation dataset was 1500 steps. Since we train policies with action-chunk [57] and execute the full action-chunk during rollouts, this reduces the maximum episode length to 300 steps. We represent the range from 0 to 300 using a single token per number.
Joints Tokenization In the Aloha domain, input also provide the model with the current joint positions. We append 14 tokens to the input text instructions, where each token represents a number from 0-255. The continuous Aloha joints which in the range [-1, 1] discretized and binned into these 256 bins, an identical manner as the actions.
Real-World Language Table Experimentation Procedure
For all real-world experiments, human was responsible for monitoring all robots and performing resets. They did not provide any form of labels or success indicators to the models. Operators were instructed perform resets either when block drops off the table, or if a station has not been shuffled and reset in the past 3-5 minutes of operation.
26

---

Table
Right Wrist
Table View
Left Wrist
Figure 8: The two figures above demonstrate the intricate level of detail that a model learns from the steps-to-go prediction objective in Stage 1. Each figure captures an interesting moment in an Aloha Single Insertion task rollout. Each is comprised of 5 consecutive frames, where below each frame visualize the probability distribution the model's prediction for steps-to-go until success. The x-axis represents the number of steps-to- go, and the y-axis represents the probability mass. Top In the first frame, the policy is about to successfully insert the peg and complete the task, so the model predicts that with high likelihood the policy will succeed soon. However, in the next frame the policy lets go of the peg too soon and the peg is about to fall. Thus the predicted steps-to-go widens drastically into multimodal distribution. considering the spectrum of possibilities from quick recovery to longer recovery times. the policy recovers in the fourth prediction narrows back to unimodal distribution, with high likelihood of success in the near horizon. Bottom In the first two frames the policy is track to successfully complete the task, so the model predicts that with high likelihood the policy will succeed soon. However, in the third frame the socket begins to slip out of the left gripper. Despite this slippage being barely visible from the left wrist camera, and visible in any of the other camera views, the model immediately picks up on this event and predictions widen significantly with multiple modes. Specifically, the model places some probability mass on an immediate save, and distributes the rest of the probability mass over a range of possible recovery times. In the fourth and fifth frames the socket fully slips of the gripper, the model removes the probability mass the immediate outcome.
fifth frames, model's
G Intuition on Reward Function (Continued)
Visual Intuition (Continued) We invite readers to study Figure g which demonstrates the intricate level of detail that a model learns from the steps-to-go prediction objective Stage 1.
Mathematical Intuition Let μu denote the policy corresponding to the imitation learning dataset D (e.g. if the dataset collected via tele-operation, would represent the "human policy"). Consider the reward function -1 satisfies 9) that is 0 when the goal is satisfied, and -1 elsewhere. Let Vи denote the undiscounted value function of policy μ for this reward function. We have,
T
(o = E,,
Σ imt
o; satisfies g)
E
1- steps-to-go
=: d(o,,
27

---

Dataset Trajectories
Behavioral Cloning Policy (Stage 1)
Self-Improved Policy (Stage 2)
Figure 9: Pointmass Navigation Domain. Sample trajectories from the imitation learning dataset, as well as
BC (Stage and Self-Improved (Stage 2) policies.
Substituting V in Equation we obtain,
r(oe, Gt, 0++1-9) V" (0++11.)) Vi(or, = (1 V" (7(0++1.))
( V"(0++1,9) (0z,
core reward
reward shaping
(5)
where is the discount factor used in the Stage 2 RL updates. We that r(oe, de, 0 g) is implicitly a shaped reward function (38 that provides higher rewards in states where μ knows how leads to policies that achieve intended goals more efficiently than the dataset policy p., while being implicitly
to perform well (i.e. core reward (1 7)- (*(0041, g) is high). Thus, Self-
regularized to stay close regions of the state space where μ is proficient!
Using Equation
5 to simplify the telescoping sum in the Monte Carlo returns we have,
R. x r(oi, ai 0++1,9) = [11 -1- (1)-)) Σ i-t (o(++11.9)
Vi(oi,
baseline
The baseline (o, g) leads to lower variance estimates that are particularly useful in our case of using the REINFORCE estimator. When 7 close O, we have R, = V/ V"(o++1.9) which is closely similar to single-step policy improvement for the - 1l [o, satisfies g) reward. As Y - R₁ encourages policies traverse trajectories along which the states have high value under the dataset policy (i.e. high V*).
V" g)
Pointmass Navigation Domain In our supplementary materials website we include a self-contained python notebook implementing Self- celf-Improvement on a pointmass navigation domain. In each episode the pointmass starts in random position, and the goal is for the pointmass to reach different randomly sampled target position. We create purposely sub-optimal imitation learning dataset for this task, where using a PD-controller we navigate position. We then execute our proposed fine-tuning tuning procedure on this imitation dataset using MLP policy and steps-to-go prediction models. Figure I shows sample trajectories from the dataset, as well as BC (Stage 1) and Self-Improved (Stage 2) policies. As anticipated, BC policies mimic the sub-optimalities of the dataset. However, in the second stage, and without access to ground-truth rewards, our proposed Self-Improvement procedure very rapidly brings policies close to optimality. For reproduction using our self-contained Colab notebook, as well as videos visualizing trajectories and steps-to-go predictions, please refer our supplementary materials website.
waypoints before heading the goal
H Future Work and Limitations
Episode Boundaries & Skill-Chaining The steps-to-go auxiliary loss that underpins our Self- Improvement stage naturally lends itself to hierarchical control. By explicitly annotating the start
28

---

and termination of sub-skills, the same progress estimator can be reused to produce dense sub-task rewards, enabling Long-horizon skill-chaining reminiscent of option- based RL. At inference time, a high-level planner whether finite-state machine, a human tele-operator, model - can invoke the learned sub-policies and rely on the steps-to-ge -o predictions to decide when to transition to the next skill. The chief obstacle is scalable episode and skill boundary annotation: manual labeling is prohibitively expensive, calling for creative strategies - such as those leveraging existing multi-modal foundation models that recover consistent boundaries from raw interaction logs. Exploring such automated segmentation is an exciting avenue for future research.
reasoning foundation
Reward Models Because reward inference does not have real-time requirements in our framework, latency constraints are minimal; we can therefore allocate far larger models - or even iterative, chain-of-thought style reasoning 225 to obtain higher-fidelity labels. A key challenge of learning reward models from imitation- based datasets is handling settings where failure states fall outside the support of the datasets, and the absence of recovery trajectories for those out-of-distribttion (OOD) states. More expressive steps-to- coppstt-goo estimators leveraging broader data sources robotics or otherwise - could recognise these OOD states and either assign appropriate shaped rewards or trigger& switch to a recovery skill, thereby improving robustness at deployment. Another potentially promising avenue handle OOD states is to collect dataset of robot rollouts using robot policies. and labeling those trajectories for training the steps-to-go go estimator.
Embodied Foundation Models Our study fine-tunes general- purpose vision-languan backbones that were never exposed to robotics data during pretraining. As larger multimodal corpora of robot experience become available, it will be crucial to design pretraining curricula that endow Embodied Foundation Models with strong priors for physical reasoning, while preserving their broad visual- semantic knowledge [29 49). On the post-training side, as opposed to post-training for specific downstream tasks done in our work, a ceneral-purpose post-training stage analogous language models could render the resulting policies effective in a purely zero-shot manner, reducing or even eliminating the need for task-specific downstream fine-tuning. As an encouraging sign, preliminary evidence from our Language Tabgle experiments hints that strengthening one task (Block2Block) via Self- Self-Improvement can noticeably boost success rates for other instructions (e.g. moving block to specific locations).
RL Algorithms For simplicity and stability we elected use on-policy REINFORCE with no data reuse. This choice eliminates two vertices of the deadly triad [51], Bootstrapping and Off-Policy Learning. However, this choice forgoes the data-reuse benefits of modern off-policy algorithms. Investigating off- off-policy variants that scale to large models [23 stands to further curb robot-hour requirements. Theoretical investigations into our choice of reward function and policy update proce- dure are also promising avenues for future work. As discussed in Section our
algorithm implicitly regularizes policies to remain close to the behvaior cloned policy, but with a distincly different mechanism than the often-used KI regularization approach (25). Our approach is more broadly applicable to any value function, not just steps-to-go. Lastly, we observe that pushing Self-Improvement beyond its performance peak can degrade success rates, suggesting that better stopping criteria or adaptive regularisers are required to prevent over-optimisation of the shaped reward. Theoretical investigations into our choice of reward function may also uncover the causes of such degradation.
Summary Our proposed Self- Imlroemppo recipe already unlocks substantial gains, with minimal reward engineering and a single human supervising multiple robots. However, addressing the challenges above essential for scaling towards general-purposs autonomous skill acquisition. We look forward to future research efforts that extend our groundwork to long-horizon tasks, richer reward inference, domain-aligned pretraining, and more sample-efficient reinforcement learning.
I
Related Works
Embodied Foundation Models A number of prior works have leveraged pretrained multimodal
foundation models robot policies. Driess et al. [18] demonstrate how separately pretrained vision and language foundation models can be co-trained to create multimodal foundation models. They highlight how these multimodal models can learn to query low-level robot controllers towards
29

---

accomplishing high-level objectives. Building on this direction Brohan et al. [9] discretize continuous robot actions and map them onto language model token spaces. This method. dubbed RT-2, enables pretrained vision-language foundation models (VLMs) to be fine-tuned as robot policies. This approach was further validated by applying to the Open X Embodiment [77] dataset containing over IM robot trajectories from 21 institutions, and is the policy architecture we base our work off of. Since RT-2, a variety of works have extended pretrained VLMs by incorporating action prediction heads. Example action head architectures include diffusion models [39 [54] (building on Chi et al. [15], Ho
et al. 2S), flow matching 17 29] (building on Lipman et al. 331 ), and LI regression [3.. Critically,
in prior works the delineation between pretraining and post-training is the task and data mixture used to perform offline supervised mie-tuning. To the best of our knowledge, our work is the first to move past supervised learning for training robot foundation models. The key contribution of our work is to present a general-purpose, only leads to rapid policy improvement, but enables acquisition of novel behaviors outside of what the models have seen in their training data. This form of behavioral generalization is only possible through an online post-training mechanism.
reward-engineeringfeee free, online seff-Improvement procedure that not
Improving Robot Policies Without Ground-Trum Rewards Our goal is to design methods that enable generalist robot foundation models to autonomously become proficient on any downstream task. A key obstacle of general-purpose Self-Improvement through reinforcement learning is that commonly we do not have access to ground-truth reward functions and success metrics, either due to the challenge of designing one (reward engineering), or difficulty in measuring them in the real world (reward instrumentation). A line of prior work, that we dub "Code-As -eee--s--Reeereee leverages LLMs write code for reward engineering [36 56 53 Policies are then trained with the designed reward, and the success rates and other feedback are provided to the LLMS for improving the reward function. Such approaches have number of downsides that make them impractical for general-purpose robot learning in the real world: 1) It is notoriously difficult to arrive at intended policies through reward engineering, in particular for dexterous manipulation tasks, 2) The iteration loop of training a policy with a given reward and patching the reward function based on outcomes is impractical for real-world roboties, 3) The variables needed in such reward functions require bespoke instrumentation to be measurable outside of simulation domains, 4) We still require a ground-truth success detector to provide feedback to the LLM designing the rewards. Aside from Gode-As-Rewarde there exists a rich literature on obtaining data-drived reward functions. Such approaches forego manual reward engineering and instead design expressive representations that scale with increasing data. An important class of works 6 35 46 12 learn latent observation representations on top of which rewards and value functions can be defined (e.g. via L.2 distance in latent space to target goals). Kumar et al. [32] use heuristic of labeling imitation learning datasets with +1 rewards near successful states, and 0 elsewhere. They demonstrate that offline and online RL, using the combination of target task and pre-existing data, can be used to sample-efficiently improve robot policy performance. Eysenbach et al. [22] design a contrastive learning objective that corresponds to form of goal-conditioned Q-Learning. Chebotar et al. [IT demonstrate that offline goal-conditioned RL with relabeled goals can lead to sample-efficicee tasks. In comparison to the above, the key advantage of our proposed approach is the straightforward integration with web-scale pretrained foundation models. RoboCat
downstream fine-tuning for new
(8] trains a large behavioral cloning (BC) Transformer with similar architecture as Gato [45 on a diverse set of robotics tasks. They demonstrate that policy performance can be improved by rolling out BC policies, hindsight relabeling episodes with accomplished goals, and adding the trajectories back into the imitation dataset. However, it is important to note that using hindsight relabeled supervised learning as a policy improvement procedure can have important failure cases 24 The closest related works to ours those based on learning distances in terms of timesteps. Hartikainen et al. 26 present an iterative procedure where 1) policies are rolled out to collect trajectories. 2) steps-to-ge prediction between states is updated through supervised learning, and 3) the negative distance reward function is used for updating policies through RL. In an offline setting, Hejna et al. (27 also learn steps-to-go between states using supervised learning, estimate shortest paths between states and goals, and use weighted behavioral cloning to obtain improved policies. Aside from important differences in settings, our focus in this work is on foundation models and generalization, demonstrating that steps-to-go policy improvement is a viable path towards ceneral-purpose Self-Improving robot policies. In addition to providing a dense reward signal, our work presents steps-to-go thresholding as novel path towards obtaining robust open-ended success detectors, which have typically been trained via binary classification [19]. Lastly, we highlight two extensions of our work. Yang al. [55] demonstrate that
30

---

our approach is effective in real-world simulators built on generative video foundation models. Ma al. [37 present an extension our work. They demonstrate that the long context capabilities of state- of-the-art foundation models enables in-context steps-to-go prediction, which can be used for offline RI., success detection, and dataset filtering, without necessitating foundation model fine-tuning.
J Future Work and Limitations
Episode Boundaries & Skill-Chaining The steps-to-go auxiliary loss that underpins our Self- Improvement stage naturally lends itself to hierarchical control. By explicitly annotating the start and termination of sub-skills, the same progress estimator can be reused to produce dense sub-task rewards, enabling long-horizon skill-chaining reminiscent of option-baset RL. At inference time, high-level planner whether finite-state machine, a human tele-operator, or a reasoning foundation model - can invoke the learned sub-policies and rely on the steps-to-go predictions to decide when to transition to the next skill. The chief obstacle is scalable episode and skill boundary annotation: manual labeling is prohibitively expensive, calling for creative strategies such those leveraging existing multi-modal foundation models that recover consistent boundaries from raw interaction logs. Exploring such automated segmentation is exciting avenue for future research.
Reward Models Because reward inference does not have real-time requirements in our framework, latency constraints are minimal; we can therefore allocate far larger models or even iterative, -hain-of-thought style reasoning [25) to obtain higher-fidelity labels. A key challenge of learning reward models from imitation-bue based datasets is handling settings where failure states fall outside the support of the datasets, and the absence of recovery trajectories for those out-of-distribution (OOD) states. More expressive steps-to-ge estimators leveraging broader data sources robotics or otherwise could recognise these OOD states and either assign appropriate shaped rewards or trigger& switch to a recovery skill, thereby improving robustness at deployment. Another potentially promising avenue to handle OOD states is to collect dataset of robot rollouts using robot policies, and labeling those trajectories for training the steps-to-go estimator.
Embodied Foundation Models
Our study fine-tunes general-purpose vision- backbones that were never exposed to robotics data during pretraining. As larger multimodal corpora of robot experience become available, it will be crucial to design pretraining curricula that endow Embodied Foundation Models with strong priors for physical reasoning, while preserving their broad visual- On the post-training side, as opposed post-training for specific downstream tasks as done in our work, a general-purpose post-training stage analogous to language models could render the resulting policies effective in a purely zero-shot manner, reducing or even eliminating the need for task uusk-specific downstream fine-tuning. As an encouraging sign, preliminary evidence from our Language Table experiments hints that strengthening one task (Block2Block) via Self-Improvement can noticeably boost success rates for other instructions (c.g. moving a block to specific locations).
semantic knowledge (29 49
RL Algorithms For simplicity and stability we elected to use on-policy REINFORCE with no data reuse. This choice climinates two vertices the deadly triad (51], Bootstrapping and Off-Policy Learning. However, this choice forgoes the data-reuse benefits of modern off off-policy algorithms. Investigating off-policy variants that scale to large models (23 stands to further curb robot-hour requirements. Theoretical investigations into choice of reward function and policy update proce- dure are also promising avenues for future work. As discussed in Section G our self-Immrrvement algorithm implicitly regularizes policies to remain close the behvaior cloned policy, but with distincly different mechanism than the often-used KL regularization approach (25). Our approach is more broadly applicable to any value function, not just steps-to-go. Lastly, we observe that pushing self-Immrovemmnt beyond its performance peak can degrade success rates, suggesting that better stopping criteria or adaptive regularisers are required to prevent over-optimisation of the shaped reward. Theoretical investigations into our choice of reward function may also uncover the causes of such degradation.
Summary
Our proposed self-Improvement recipe already unlocks substantial gains, with minimal reward engineering and a single human supervising multiple robots. However, addressing the challenges above is essential for scaling towards general-purpose autonomous skill acquisition. We
31

---

look forward to future research efforts that extend our groundwork to long-horizon tasks, richer reward inference, domain -alinned pretraining, and more sample-efficient reinforcement learning.
K
Additional Plots
Self-mmprovement on Real-World LanguageTable
0.90
0.85
65-
Stage 3. robett Stage
Stage robets Stage
Mean Epizodte Length
60-
Rate
50-
0.55
Stage 1. robots Stage
20%
Stage 1. robets Stage
0.50
20% bata Stage 1. robots Stage
0.6%
0.8%
0.65%
Stage 2 Episodes (es % LanguageTab dataset)
Stage 2 Episodes (as LanguageTablc dataset)
Figure 10:
cell-mmproveeett "Success Rate" plots during Stage 2 Self- Self-Improvement on the real-world LanguageTable domain. We conducted real-world experiment 3 times: 1) 80% imitation dataset size in Stage 1, Stage fine-tuned on 3 robots simultaneously. 2) 20% data in Stage 1, Stage with 3 robots, 3) 20% data in Stage 1, Stage 2 with 4 robots. In all Stage experiments single human monitored and performed periodic resets all robots. Each experiment took approximately
hours (4 hours 5 days). The x-axis the plots aboves shows the amount of extra episodes collected during the Stage online Self-Improvement process, as percentage of the total number of Block2Block episodes in the Lanugage Table dataset.
Sclf Improvement Archa Single Insurtion "tak
Selfiipproemmet on Simulated Language
Stage 2i car Sage
2.8
Sucoess Rate
0.5
2.5
Rwward
Stape
NJU Mai: Roward
20% Stage
22
20% Paul Pakkyy PaL Reward
60% RMU Palicy
Reword
50% LAWA
100% LiWe
10% PNU Pakac Uni Roward
20% LAWA
Paul Palicy
Resand
Patkcy
0%
10%
201e
30%
40%
50%
69%
70%
80%
90%
100%
Total Episodes in Stage Stage 2 las S Language Tabyle Dalaset!
580
TMO
5000
15730
1500 3583
Left Results and ablations on the simulated LanguageTTable domain. We emphasize to the reader that while it appears that the Stage and Stage 2 plots have identical x-axis values, there is no bug in the plot and they are in fact different. The Stage Self-Improvement process is simply sample-efficient that the difference in x-axis is not observable in this plot. The first plot in Figure in the main presents different difference more apparent. Right "Success Rate" and "Mean Episode Length" plots during Stage 2 Self- Improveppte on the Aloha Single Insertion Task (5K and 10K data settings, 3 random seeds cach setting). Similar to the discussion in Section[4.] the above plots demonstrate that the combination of SFT + Self- Improvement is more sample-efficient than allocating the full robot time budget to collecting imitation data for SFT.
Figure 11:
view that makes
32

---

L Computing Percentage of Block2Block Instruction in LanguageTable
To get sense of the percentage of the LanguageTable datasets corresponding to Block2Block tasks, we used Gemini 1.5 Pro and its structured outputs feature to label LanguageTable instructions as being Block2Block or not. For both the simulation and real datasets randomly sampled N 5000 of the instructions in the dataset, and used the following prompt to classify them as Block2Block. Using the structured outputs feature of Gemini enables to enforce the LLM responses to be either Yes or No.
You are language model with expertise in determining the structure and type of robotic instructions. Your task is identify whether given instruction is "block to block" type instruction or not. A "block to block" instruction involves moving pushing one block towards another specific block on a board. This does not include separating two blocks, putting one block in between two blocks, putting block near a group of blocks.
Please analyze the following instruction and respond with either "yes" or "no" based on whether it fits the definition of a "block to block" instruction: Examples where the answer "yes":
"push the red circle towards the blue triangle"
"push blue cube the right of green cube"
"oove the red moon towards the bottom left side of the red pentagon"
push red pentagon into the green cube"
placce the yellow star to the left of the red moon"'
"push the green cube vertically below the yellow pentagon"
drag green star into blue cube"
"slide the red star at the bottom right of the green star"
Examples where the answer is "no":
"push the blue cube in between yellow star and green star"
"push the red crescent away from the blue crescent"
"place the arm the left of the yellow star"
"move the blue crescent to the center of the board"
"ddjstt the group of blocks to form circle"
"separate the green star and the blue cube"
"push blue moon along with yellow star to the left side"
"move yellow star and blue moon together slightly the top side of the board"
"place the blue cube at the top center"
"slide blue cube bit right"
The instruction for you to label as "yes" or "no" is:
For the simulated and real dataset respectively. 47% and 49% of the instructions were labeled as Block2Block. Using Hoeffding's Inequality can see that with N 5000, these estimates are within 2.8% emror margin with 99.9% confidence.
M Interesting Observations & Incomplete Experiments
In this section we note interesting observations, and highlight results of experiments that we were unable to complete due a change of institution affiliation of the lead author.
Positive Transfer Self-Improvement Section 4. we note that we perform Self-Improvement on the Block2Block subset of the LanguageTable tasks. We have observed through real-world
33

---

Figure 12: Real-world replica of our simulated Aloha dingle-Insertin task
rollouts that Self- seff-Imprvvement not only significantly improves performance on Block2Block tasks, but actually improves the model across all LanguageTable tasks.
All-Instructions Experiment in the Real-World Language Domain In the real-world Language Table domain we conducted an experiment where after Self-Improvement Block2Block tasks, we ran self-Improvement on all possible instructions. Prior to conducting this experiment, we trained separate PaI.I model produce goal instruction given the current image. We then started the all-instructions Seff-Improvemnet by mixing instructions at 50-50 ratio between Block2Block and model- instructions. Over time decayed the rate of Block2Block instructions down to zero. We noticed clear improvement in success rate metrics for all tasks. However, we noticed that for portions of the training time some human operators had improperly up the robot stations. Thus, we decided to not include these results in our paper. We were unable to retry this experiment due to time constraints.
Real-Worle Aloha Single-Insertion We recreated our simulated Aloha Single- task in the real world by 3D printing identical assets, and collecting an imitation dataset using Aloha teleoperation. We then trained the Stage (SFT) model with an identical setup (Appendix D.3) and tokenization (Appendix E.2) as the simulated Aloha task. We also built second variant of our real-world Self- reff-lmproeement infrastructure better suited to higher rate control (Appendix N.2). We verified that our Stage BC policies demonstrated reasonable success rates, but we were unable to complete full Self-Improvement experiment before we had to conclude our project.
N Infrastructure Overview
Below discuss two variations of the infrastructure implemented for online Self- colf-Improvement. All experiments presented in this work were conducted using Version 1, which proved effective for both simulated and real-world environments operating at control frequencies up to 5Hz.
However, when we attempted extend our approach to the WorldWorld Aloha Single-Inserng (Appendix[M] MM.. we discovered that the remote action inference server design of Version could not reliably support the 10Hz control frequency required by the Aloha platform. This limitation motivated the development of Version 2, which performs policy inference locally on the actor machines to eliminate network latency. We successfully verified that our Stage behavioral cloning policies performed well on the real robot and briefly tested the Version 2 infrastructure, but time constraints
task
34

---

prevented us from completing full Self-Improvement experiment for the Aloha Single- Single-Insertion Insertion task in the real world.
N.1 Version 1: Non-Local Policy
Our first infrastructure implementation for Stage 2 Self- Iepr---oooeeenttt employs a distributed architec- ture designed to handle both simulated and real-world robot environments. This version separates the computational components from the actors (robots) and uses a client-server model for inference. The system comprises the following key components:
Dual-Purpose TPU Nodes For Learning and Action Inference The same set of TPUs serves dual functionality, alternating between serving as learner nodes and action inference servers depending on the current phase of the RL iteration. During the data collection phase, these TPUs operate as action inference servers, processing observation- -oal pairs (0t, g) and returning actions a; ~ P (4el Ot, g). During the learning phase, they switch performing policy updates using the data from the replay buffer.
Steps-to-Go Inference Nodes A separate set of inference nodes is dedicated to computing the steps-to-go predictions d( g) from Equation These nodes host frozen Stage checkpoints and respond queries with steps-to-ge topp-to-go predictions. The recipients of these predictions then use them for two purposes: (1) computing rewards using Equation 2 by taking differences between consecutive predictions for episode labeling, or (2) evaluating the success condition success g) [[d(o, S during episode execution to determine termination.
Actors and Environment Loop The actors, which can be either real-world robot stations or simulated environments, execute the following environment loop. At the start of each episode, a goal instruction g is sampled. The actor then iteratively: (1) obtains the current observation o, from the environment, (2) queries the action inference server to obtain action dz and (3) executes the action in the environment.
In parallel, a background process continuously queries the steps-to-go inference nodes with the most recent observation to check the success condition. This asynchronous design accounts for inference latency-the success condition may not be evaluated for every single observation due to query time, but rather checks the most recent available observation. The episode terminates when either the success threshold is met, the maximum episode length reached, or a human operator manually intervenes (particularly important for real-world experiments).
Once an episode completes, still within the environment loop, each observation is labeled with its steps-to-go prediction by querying the steps-to-ge inference nodes. These predictions are used to compute rewards according to Equation 2 by taking differences between consecutive predic-
tions: r(o,, 0t, 0++1-9) = d(or,g) g) d(0++1..)) The Monte Carlo returns R. = E/
r(o;, a,, (,91 are then calculated, and the labeled tuples (ot, ae. 9, R2)) are sent to the replay buffer server.
Replay Buffer Server The replay buffer is implemented as standalone server using Google DeepMind's Reverb [10], which provides efficient distributed data storage and sampling. This server receives labeled experience tuples from all actors and maintains them for training. The learner nodes sample minibatches from this server during the learning phase.
Central Coordinator A central controller orchestrates the entire system, monitoring the replay buffer server's size and coordinating phase transitions. When the replay buffer accumulates sufficient data for N policy updates with batch size B (i.c.. N x B samples), the controller signals all actors to pause data collection and instructs the TPU nodes to switch from inference mode learning mode. After completing the N RL policy updates, the replay buffer is cleared, and the system transitions back data collection with the updated policy.
N.2 Version 2: Local Policy
Our second infrastructure implementation maintains the same overall architecture as Version 1, with one critical difference: policy inference is performed locally the actor machines rather than
35

---

through remote inference servers. This design reduces network latency during episode execution and decouples the learning infrastructure from the inference workload.
Key Architectural Change In this version, the TPU nodes are dedicated exclusively to learning. After completing N RL policy updates, the updated model weights are distributed to all actor machines, where they are loaded for local inference. Each actor machine maintains its own copy of policy model and performs action inference locally during data collection.
Actor Machines with Local Inference The actors now handle both environment interaction and policy inference. During episode execution, when an action is needed, the actor queries its local policy model rather than making network request to remote server. This eliminates the latency associated with network communication for action inference, which can be particularly beneficial in real-world robotics settings where low-latency control is critical. The background process for success detection remains unchanged-it continues to query the steps-to-go inference nodes with the most recent observation to check the success condition and determine episode termination. Similarly, after episode completion, the actors still query the steps-to-go inference nodes to label each observation for reward computation, exactly in Version 1.
Weight Distribution After each learning iteration, the central coordinator instructs the learner nodes to broadcast the updated model weights to all actor machines. This weight synchronization ensures that all actors use the same policy version during the subsequent data collection phase. The actors pause briefly load the new weights before resuming episode collection.
Remaining Infrastructure All other components remain identical to Version 1:
The steps-to-go inference nodes continue operate separate servers, providing predictions for reward computation and success detection
The replay buffer server, implemented using Google DeepMind's Reverb 10] continues to centrally manage experience storage and sampling
The central coordinator maintains its role in orchestrating phase transitions and monitoring the replay buffer size
Episode labeling with rewards and Monte Carlo returns computation remains unchanged
This local inference approach trades increased memory usage on actor machines (each must hold the full policy model) for reduced inference latency and decreased network traffic during data collection. The design is particularly advantageous when actor machines have sufficient computational resources or when network reliability a concern.
Broader Impacts
In this work present novel Self-Improvement approach that enables Embodied Foundation Models to sample-efficienlly improve their task performance, as well as autonomously practice and acquire novel skills. Enabling autonomous Seff-Improvement may have significant positive and negative societal impacts that should be carefully considered.
Potential Positive Impacts Our approach fundamentally improves policy success rates and can unlock new capabilities. This enables robotics to be applicable new domains and industries that previously lacked robotic solutions. We also demonstrated that the combination of SFT and Self-Improvement significantly more sample-efficient that supervised learning alone. This sample- efficiency directly translates to more effective use of resources.
Potential Negative Impacts More effective robotic solutions may lead to job loss. Thus, careful attention opportunity creation is key for the long term stability of society. Enabling robots to more efficiently acquire a broader skillset may also enable unacceptable applications of robots by bad actors.
36


Reviews:
[{"review_id": "wiK6IOBYSh", "reviewer": "Reviewer_UpTX", "summary": "This paper presents a two-stage framework for post-training embodied foundation models (EFMs): (1) Supervised Fine-Tuning (SFT) using behavioral cloning and steps-to-go prediction; and (2) a novel Self-Improvement phase that leverages the learned steps-to-go estimates to define a shaped reward function and a robust success detector for online reinforcement learning. The method is tested on two robot platforms—LanguageTable and Aloha—both in simulation and real-world environments. The authors show significant gains in sample efficiency, performance robustness, and skill generalization beyond imitation data. Notably, the method enables autonomous acquisition of novel behaviors such as BananaTable manipulation.", "questions": "1.\tHave you explored off-policy RL algorithms (e.g., SAC, TD3) using the same steps-to-go reward function? What challenges did you face?\n\n2.\tHow robust is the success detector threshold (e.g., s = 3) across domains or tasks? Could an adaptive threshold further improve learning?\n\n3.\tIs the PaLI reward model fine-tuned or frozen during RL training? Would co-training improve performance or degrade stability?\n\n4.\tIn the BananaTable task, does the agent develop new motor primitives or just repurpose existing skills in novel contexts?\n\n**Minors:**\n\n-\tLine 98: in RL (Section ??).", "limitations": "1.\tThe method depends on on-policy RL with no data reuse, which may limit scalability to more complex tasks.\n\n2.\tThe frozen reward model from SFT assumes prediction accuracy remains consistent across domains; this may fail under strong domain shifts.\n\n3.\tThe steps-to-go abstraction implicitly assumes task progress can be estimated linearly or monotonically, which may not hold in deceptive or delayed-reward settings.\n\n4.\tWhile BananaTable and insertion tasks are insightful, broader application across locomotion, dexterous manipulation, or non-prehensile control would strengthen generality claims.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**:\n\n- The paper presents a clean and conceptually grounded two-stage pipeline for post-training embodied foundation models (EFMs). By combining supervised fine-tuning (SFT) with a self-supervised online reinforcement learning phase (Self-Improvement), the authors draw a compelling parallel to the LLM post-training recipe (SFT + RLHF). This adaptation to robotics is well-motivated and could establish a new foundation for scalable robot policy training without explicit reward functions.\n\n- The proposed reward shaping mechanism—using the decrease in predicted steps-to-go as a proxy for task progress—is both elegant and practical. It removes the need for manually engineered rewards or hand-crafted success detectors. The success indicator derived from steps-to-go thresholding (e.g., d(o, g) ≤ s) is a clever way to detect task completion robustly in real-world conditions where ground-truth success metrics are often unavailable.\n\n- The experiments are impressively comprehensive, spanning simulation and real-world settings, two distinct robot platforms (LanguageTable and Aloha), and a diverse range of training set sizes and generalization scenarios. The authors go beyond mere benchmarks: they conduct ablations on reward model quality (PaLI vs. Uni-PaLI vs. Scratch), show robustness across random seeds, and highlight domain transfer through Real2Sim and BananaTable evaluations.\n\n**Weaknesses**:\n\n- The reinforcement learning component relies on a straightforward on-policy REINFORCE algorithm. While this choice offers training stability and avoids the need for value estimation, it misses out on potential gains in sample efficiency from off-policy algorithms (e.g., SAC, TD3, or even Q-learning variants). The absence of any baseline comparisons using such alternatives limits our understanding of the true potential of the proposed reward signal under modern RL schemes.\n\n- The method is primarily empirical, with no theoretical analysis of convergence, stability, or generalization guarantees. For example, the steps-to-go reward assumes monotonic progress toward the goal, but this may not hold in multi-modal or deceptive tasks. More discussion on the theoretical assumptions behind using steps-to-go as a proxy for value could strengthen the paper’s foundation.\n\n- While the BananaTable task offers strong qualitative evidence of generalization, the paper lacks formal metrics or benchmarks to quantify generalization gaps (e.g., success rates under zero-shot task variations, trajectory diversity metrics, or skill composition tests). The evaluation focuses on success rate improvements within the same task class, which may not fully capture the policy’s ability to generalize to novel skill compositions or long-horizon dependencies.", "was_revised": true, "final_justification": "The author's rebuttal has addressed most of my concerns.", "rebuttal": "Thank you for your detailed and insightful review of our work! We hope our responses below address your questions. We look forward to a productive author-reviewer discussion period!\n\n**On-Policy REINFORCE**\\\nOur work introduces many novel components into a full framework for post-training embodied foundation models, with emphasis on reliability in order to be successful in real-world experiments. This, in this work from the beginning we used on-policy REINFORCE and did not attempt any off-policy methods. This was motivated by a number of factors:\n- On-Policy methods are known to be more stable in general. On-Policy REINFORCE specifically removes two vertices of the deadly triad: off-policy learning & bootstrapping\nTD-Learning with large models has significant stability issues, and designing stable methods is an active area of research [1]\nThe current choices for reinforcement learning with Large Language Models (LLMs) are based on on-policy methods as well (e.g. DeepSeek-R1 [2]). In fact [2] introduced the GRPO on-policy RL algorithm which does not learn a value function either. GRPO is rapidly becoming one of the most popular algorithms used in training LLMs.\n\nThe fact that our results demonstrates such significant sample-efficiency with on-policy REINFORCE and no data reuse demonstrates that any advancements in off-policy training methods for large foundation models stands to improve our results.\n\n**Monotonic Progress Assumption & Delayed Rewards Settings**\\\nWhile it is accurate that we do not provide convergence, stability, or generalization guarantees, it is **not the case** that the steps-to-go reward model assumes monotonic progress towards the goal. In our work, we are learning the **distribution** over steps-to-go which is very multimodal due to mistakes, and many different ways of accomplishing the same goal:\nIn LanguageTable the human demonstrations are not perfectly efficient and often make mistakes\nIn the Aloha data there are mistakes such as a) peg being dropped, b) a socket being dropped, c) accidental drops or mistakes during insertion.earning steps-to-go from such data is in fact a **significant feature** of our work. This enables the model to learn the **distribution** over steps-to-go.\n\nIn our supplementary videos website (link at the end of Section 1 of our manuscript), the top 2 videos in the **Aloha** section demonstrate this ability to learn the multimodal distribution. In these videos:\nTop: Shows the camera views\nMiddle plot: Expectation of the steps to go distribution across time. x-axis is time, y-axis is expected value of steps-to-go prediction.\nBottom plot: Shows the predicted probability distribution over steps-to-go at that moment in time. x-axis is steps-to-go, y-axis is probability.\nWe encourage you to focus on the bottom plot while viewing the videos. You can observe that at critical times (e.g. about to pick a peg, about to do an insertion, etc.) the model makes very multimodal predictions, and once the outcome is clear (e.g. successful picked up the peg) the prediction become more confident and uni-modal.\n\nLearning steps-to-go from such data is in fact a **significant feature** of our work. As discussed in Section 2.2, learning the distribution over steps-to-go opens up many avenues of future work such as learning risk-aware policies using CVaR, or distributional reinforcement learning.\n\nWith respect to “delayed reward settings”, steps-to-go prediction also leads to reward densification. As an example, in the LanguageTable Block2Block task, a standard reward function would the negative L2 distance from block A to block B. However, this reward would not provide any signal which the arm is moving toward block A. Instead, steps-to-go prediction provides dense reward signals that accurately provide signal to the arm moving the correct position to effectively push block A. Videos demonstrating such predictions can be viewed in our supplementary videos website (link at the end of Section 1 of our manuscript) where we plot the expected value of steps-to-go predictions alongside LanguageTable trajectories.\n\n**Skill Composition Generalization**\\\nThe focus of our work is to introduce a very effective framework for post-training embodied foundation model for a downstream task class. Thus, in the BananaTable task the notion of strong generalization that we focus on is the ability to learn a novel behavior, namely how to handle the physics of a new object. As discussed in Section 4.3.2, in the BananaTable task the success  rate improves from 63% to 85%, and the videos in our supplementary videos website (link at the end of Section 1 of our manuscript) demonstrate that this is a significant generalization gap to bridge, as the BC models are very inefficient at accomplishing this task.\n\nIn terms of skill composition and long-horizon tasks, as discussed in our Future Works section, we believe our success detector based on steps-to-go prediction can be effectively leveraged to perform skill-chaining: once the success detector indicates that the sub-task is completed, the policy moves onto the next sub-task.\n\n**Robustness of Success Detector Threshold**\\\nWe chose this threshold via analyzing the expected value of steps-to-go predictions on a small subset of the dataset. We have not investigated adaptive thresholding.\n\n**Frozen Reward Model**\\\nPer our discussion in Section 2.2, we keep the reward model frozen during RL training, mimicking default RL paradigms where reward functions typically do not change during training. For our specific choice of reward model, due to the connection to Value Functions and baselines (Appendix F), there may be interesting connections to value function training and off-policy RL. In turn, updating the reward model might suffer from similar instability issues as value functions (discussed earlier above).\n\n**BananaTable New Skills**\\\nDuring post-training the policy learns to significantly update its motor primitive and learn how to effectively manipulate a banana. The success rate improvement from 63% to 85% (Section 4.3.2), does not sufficiently capture the extent to which the policies become better at this task. We strongly encourage viewing the videos in our supplementary videos website (link at the end of Section 1 of our manuscript). Before post-training the policy is very inefficient, often making the banana rotate about itself instead of moving in the intended direction. After post-training, the policy can much more steadily push the banana in the right direction.\n\n[1] Stop Regressing: Training Value Functions via Classification for Scalable Deep RL\n[2] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning"}, {"review_id": "AyvRtcmkKO", "reviewer": "Reviewer_E1GX", "summary": "This paper proposes a two-stage post-training pipeline for embodied foundation models, using a steps-to-go reward which represents the model's estimate of how many time steps remain until task completion and serves as an implicit measure of progress. First, a supervised fine-tuning stage adapts a pretrained vision-language backbone model with a dual loss that combines behavioral cloning and steps-to-go prediction. Then, the trained steps-to-go head is frozen, and its output is reused as both a shaped reward and a success detector, enabling efficient on-policy reinforcement learning on real or simulated robots with minimal human oversight.", "questions": "- How does the method account for diversity in action trajectories that achieve the same semantic goal but follow different physical paths? For example, consider a task where the robot must move an object from point A to point B. One trajectory might approach the goal directly in a straight line, while another might take a longer but equally valid path around an obstacle or due to environmental constraints. Since the steps-to-go reward is based on minimizing the number of steps, does the method implicitly favor shorter trajectories? If so, could this discourage exploration and lead to a collapse toward only the shortest, most direct behaviors, thereby reducing the diversity and robustness of learned policies?\n- If more computational resources were available, would it be beneficial to freeze the entire Stage 1 checkpoint (not just the steps-to-go head) during Stage 2 RL fine-tuning? Could this help stabilize training or improve sample efficiency?", "limitations": "Yes, the authors have adequately addressed the limitations of their work. However, it would be beneficial to clearly discuss scenarios where the application of the steps-to-go reward might be challenging.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "> ### Strenghts\n- The proposed steps-to-go reward is conceptually simple yet effective.\n- The framework is validated across multiple environments: Language Table (real-world and simulator), Banana Table (real-world), and Aloha single insertion.\n- Figure 2 provides a clear visual demonstration of how steps-to-go functions in real environments, enhancing intuitiveness.\n- In Appendix, The framework demonstrates compatibility with other vision-language backbones, suggesting broader applicability.\n\n---\n\n> ### Weaknesses\n- The primary contribution of the paper lies in demonstrating the efficacy of the steps-to-go reward design. However the paper lacks thorough comparison or discussion with other established reward design approaches.\n- Prior work on mitigating sparse rewards, such as subgoal-based methods or goal-state similarity-based reward relabeling, is not sufficiently acknowledged or contrasted. A deeper analysis would help clarify when and why steps-to-go provides a significant advantage.\n- The generalization capability of the method may be sensitive to the sampling frequency of robot’s trajectory data. For instance, even for trajectories that perform the same behavior, the number of steps required to reach the goal can vary depending on the frame rate. In practical settings involving mixed sampling frequencies (for example, 60 fps and 30 fps), the paper provides limited discussion on how steps-to-go predictions are normalized or calibrated. This raises concerns about the scalability and robustness of the framework.\n- The novel skill acquisition results in Section 4.3.2 are intriguing but may be slightly overstated. The instruction format “move the X to the Y” is already present in both the Language Table and Banana Table datasets, which weakens the claim of strong generalization to novel skills. Moreover, it remains unclear how the model generalizes to previously unseen object properties.", "was_revised": true, "final_justification": "Most of my concerns have now been resolved, and I appreciate how clearly authors articulated the key contributions. In particular, authors detailed explanation regarding the design of a general-purpose reward model, which minimizes the need for task-specific engineering, and interpretation of the experimental videos were convincing enough to change evaluation.", "rebuttal": "Thank you for your detailed and insightful review of our work! We hope our responses below address your questions. We look forward to a productive author-reviewer discussion period!\n\n**Comparisons to Other Reward Models**\\\nRelated works, existing reward models fall short in one way or another\nOur reward model is general, very straightforward to incorporate into foundation model training\nBeing able to demonstrate real-world results in such a setting is an incredible undertaking, requiring massive infrastructure efforts. Thus the fact that such an approach work so well is a very significant for the robotics community.\n\n**Comparisons to Other Reward Models & Discussion of Prior Works**\\\nOur objective in designing our reward model was to have a) a very general purpose reward model that requires minimal task-specific engineering, and b) is very straightforward to incorporate into large scale training of foundation models. Existing reward models fall short in one way or another. Due to space limitation we present a condensed form of our Related Works discussion in the main manuscript, but provided a much more extensive discussion in Appendix L. Specifically for the forms of reward modeling you mentioned:\nLines 1144-1146: we discussed goal-state similarity-based reward\nLines 1151-1152: we discussed an example goal relabeling based method\nLines 1158-1167: we discussed timestep distance based methods\n\nWe would like to also emphasize that being able to demonstrate 1) real-world robotics results, with 2) foundation models, 3) trained with on-robot online RL, 4) with a general-purpose learned reward, is a very large undertaking that pushes the boundaries of current robotics methodologies. We believe that demonstrating the fact that such an approach can result in such significant improvements, sample-efficiency, and generalization is a very significant contribution to the robotics community in its own right.\n\n**Mixed Sampling Frequencies**\\\nIn this work we did not study mixed sampling frequencies as it is not a standard variation that is typically studies, but it could indeed by a realistic setting that may occur when training multi-embodiment foundation models. Our initial idea for how to address this setting is to perform steps-to-go prediction in terms of wall-clock-time, instead of “environment steps” which would be FPS-dependent.\n\n**Novel Skill Acquisition**\\\nDuring post-training the policy learns to significantly update its motor primitive and learn how to effectively manipulate a banana. The success rate improvement from 63% to 85% (Section 4.3.2), on its own, does not sufficiently capture the extent to which the policies become better at this task. We strongly encourage viewing the videos in our supplementary videos website (link at the end of Section 1 of our manuscript).\n\nThe video of the policy before post-training shows that the policy is very inefficient, often making the banana rotate around itself instead of moving in the intended direction. The video after post-training, shows that the policy has learned how to deal with the novel physics of the banana, and steadily push the banana in the right direction.\n\nThe mechanism that enables this generalization is as follows:\nBefore Stage 1, the models are initialize from a PaLI vision-language model (VLM) that is pretrained on the web-scale data\nAfter Stage 1 on LanguageTable data, the reward model understands what the tasking of “move the X to the Y” means for the blocks, but because it starts from the PaLI VLM, it generalizes its understanding of “move the X to the Y” from blocks to arbitrary objects such as bananas.\nDuring Post-Training, this enables to policy to practice manipulating the banana and learning how deal with the physics of this new object’s properties\nThis is one of the key contributions of our work. This result is uniquely unlocked by our proposed novel combination of pretrained foundation mode + post-training approach.\n\n**Diversity in Action Trajectories**\\\nIn our work, we are learning the **distribution** over steps-to-go which is very multimodal due to mistakes, and many different ways of accomplishing the same goal:\nIn LanguageTable the human demonstrations are not perfectly efficient and often make mistakes\nIn the Aloha data there are mistakes such as a) peg being dropped, b) a socket being dropped, c) accidental drops or mistakes during insertion.earning steps-to-go from such data is in fact a **significant feature** of our work. This enables the model to learn the **distribution** over steps-to-go.\n\nIn our supplementary videos website (link at the end of Section 1 of our manuscript), the top 2 videos in the **Aloha** section demonstrate this ability to learn the multimodal distribution. In these videos:\nTop: Shows the camera views\nMiddle plot: Expectation of the steps to go distribution across time. x-axis is time, y-axis is expected value of steps-to-go prediction.\nBottom plot: Shows the predicted probability distribution over steps-to-go at that moment in time. x-axis is steps-to-go, y-axis is probability.\nWe encourage you to focus on the bottom plot while viewing the videos. You can observe that at critical times (e.g. about to pick a peg, about to do an insertion, etc.) the model makes very multimodal predictions, and once the outcome is clear (e.g. successful picked up the peg) the prediction become more confident and uni-modal.\n\nThus from the perspective of learning the multimodal steps-to-go distribution, our method in its current form already accomplishes this desiderata. Learning steps-to-go from such data is in fact a **significant feature** of our work. As discussed in Section 2.2, learning the distribution over steps-to-go opens up many avenues of future work such as learning risk-aware policies using CVaR, or distributional reinforcement learning.\n\nWe believe the second component to your question, “does the method implicitly favor shorter trajectories” is independent of our proposed approach and relates to the specific choice of RL algorithm that is used for optimizing the reward function. In our work we used on-policy REINFORCE which will favor shorter trajectories. In contrast, if for example we decided to use Maximum Entropy RL (Max-Ent RL) [1], we could tune the extent to which the policy optimizes for shorter trajectories: In Max-Ent RL, the goal is to learn a policy whose trajectories $\\tau$ have the probability distribution $$p(\\tau) \\propto \\exp\\left(\\frac{\\text{returns}}{\\text{temperature}}\\right)$$. Thus by tuning the temperature parameter, we can tune the extent to which the policy cares about optimizing for shorter trajectories.\n\n**Freezing Entire Stage 1 Checkpoint**\\\nWould you be able to clarify your question please? During Stage 2 we have one frozen checkpoint acting as reward model and success detector, and a non-frozen checkpoint which is the policy and is being updated during Stage 2. If we freeze everything, which component would be updated during Stage 2 training to improve the policy?\n\n**Limitations**\\\nIn Appendix M we have provided an extensive discussion on Limitation and avenues of Future Work, discussing important challenging cases that we anticipate.\n\n[1] Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review"}, {"review_id": "4Y1WIqDIKG", "reviewer": "Reviewer_vuwt", "summary": "The authors propose to fine-tune foundation models for robotics by introducing a RL phase after training, similar to LLMs. In order to obtain a suitable reward function from offline imitation learning, the method trains the foundation model offline on imitation learning data with two objectives:\n- A behavior cloning objective for learning actions given state and goal\n- A \"steps-to-go\" objective, which lets the model predict the remaining length of the trajectory until the goal condition is met.\n\nDuring the reinforcement learning \"self-improvement\" phase, the steps-to-go prediction serves as a reward function. If an action reduces the number of predicted steps, i.e. is an action towards the goal, the reward is positiv. if it increases the number of predicted steps, the action was counter to the goal and the reward is negativ. This objective can also serve as an automatic success detector for real world RL, when the steps-to-go prediction achieves a sufficiently small value.\n\nThe models are evaluated regarding success improvement and data efficiency in simulation and real-robot tasks, namely LanguageTable and Aloha setups.", "questions": "## See Weaknesses.\n\n### Additionally, for my understanding:\nFor the reward function / steps-to-go prediction. I am wondering about a case, where the robot does something which does not change the steps-to-go prediction, e.g., moving on a circle around an object. The reward signal in this case would be 0. Would it make sense to include a small penalty for such cases, for example $r = d_t - d_{t+1} - 0.1$ ?\n\nFigure 2, step 15 (2nd red circle) seems to be close to this edge case.", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strength\n- Elegant reward function / model design\n- solid theoretical analysis\n- experiments include both sim and real-robot and support the claims regarding data, human-effort efficiency and data generalization capabilities\n- well written\n\n## Weakness\n- as mentioned in the limitations, the current experimental evaluation is on relatively simple tasks when compared to CALVIN or LIBERO benchmarks, which require long horizon manipulation, chaining of multiple skills and handling of distractor objects or \"unseen\" objects hidden inside a desk drawer. It is unclear how this approach would perform in these settings.\n- The aloha peg insertion experiment requires an adaptation of the reward function, as the steps-to-go function is unable to detect a successful peg insertion from cameras alone. I believe this is an important limitation which should be communicated more clearly. Additionally, it is unclear to me how the success is detected to add the constant reward.\n\n### Minor Comment\n- I believe Figure 3 could be easier to read, if all the subplots would use the same Y-Axis scaling.", "was_revised": true, "final_justification": "I appreciate the authors detailed responses in the discussion phase. I am convinced that this is elegant work with potentially high impact for robot learning.", "rebuttal": "Thank you for your detailed and insightful review of our work! We hope our responses below address your questions. We look forward to a productive author-reviewer discussion period!\n\n**Skill-Chaining**\\\nThe focus of our work is to introduce a very effective framework for post-training embodied foundation models for a downstream skill. We did not consider tasks comprised of a sequence of subskills.\n\nAs discussed in our Future Works section, we believe our success detector based on steps-to-go prediction can be effectively leveraged to address skill-chaining and long-horizon tasks: during rollouts, once the success detector indicates that a sub-task is completed, the policy moves onto the next sub-task. This extension of our approach can enable simultaneous RL training on all the subskills needed for a desired long-horizon task.\n\n**Unseen Objects**\\\nIn our work we present a novel combination of web-scale pretrained foundation models + an effective on-robot online RL post-training approach. We believe that this combination can address unseen objects in two ways:\n\n**Semantic:** When the difference between objects is mostly perceptual and semantic (i.e. similar physical properties), the generalization inherited from the foundation models can address the gap. This was indeed observed in RT-2 [1] which trained robot policies through behavioral cloning on top of a pretrained vision-language foundation model (equivalent to our Stage 1 policy).\n\n**Novel Behavior:** The more challenging approach is learning to handle unseen objects which also have unseen properties. This is one of the key contributions of our work. Our approach uniquely unlocks this form of generalization as observed in our BananaTable experiments (Section 4.3.2). The mechanism that enables this generalization is as follows:\nBefore Stage 1, the models are initialized from a PaLI vision-language model (VLM) that is pretrained on the web-scale data\nAfter Stage 1 on LanguageTable data, the reward model understands what the task of “move the X to the Y” means for the blocks, but because it starts from the PaLI VLM, it generalizes its understanding of “move the X to the Y” from blocks to arbitrary objects such as bananas.\nDuring Post-Training, this enables to policy to practice manipulating the banana and learning how deal with the physics of this new object’s properties\n\n**Aloha Peg Success**\\\nThank you for this note. We will update the manuscript to more clearly communicate this change. As to the specifics of success detection in this setting, since this was a simulation domain we were able to detect the success condition from the simulation state.\n\nWe would also like to provide some additional context as to why this success signal was needed. We had designed this task to trigger the success when the peg was fully pushed into socket. Specifically, the mesh of the peg would have to come into contact with the inside wall at the end of the socket. Without the success bonus, the policy would still learn to successfully accomplish the peg insertion task, however, it would not try to push the peg all the way back to the end of the socket, and therefore not trigger the success. Upon closer inspection we saw that with the Aloha camera views, and the image resolutions we were using, it was extremely difficult to tell the difference between when the peg was fully pushed in vs. “almost fully” pushed in. Hence the reward model would not be able to incentivize fully pushing the peg in. The success bonus made it so that the policy would learn to push the peg further and trigger the success mechanism.\n\nIn addition to communicating the success bonus more clearly in the main text, we will include these notes in our Appendix as well.\n\n**Reward Signal 0**\\\nIt is correct that if the robot executes an action that does not change the reward function the reward would be 0. However, if the robot had executed a better action and made progress towards the goal, it would have received a positive reward (and conversely a worse move would have received a negative reward). Thus the robot is still incentivized to choose actions that make progress towards the desired goal.\n\n[1] RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control"}, {"review_id": "VMyFIjMZln", "reviewer": "Reviewer_DXir", "summary": "The paper presents a two-stage approach for fine-tuning a vision language model to generate actions for robotic applications. In the first stage, the VLM is fine-tuned using both a behavioral cloning loss and a step-to-go loss. The BC loss aims at cloning actions in the trajectories while the step-to-go loss predicts the number of steps required to solve the task (i.e., steps to reach the goal). The underlying assumption is that each trajectory terminates in a goal. The second phase leverages policy gradient (in particular REINFORCE) on top of a reward function based on the step-to-go prediction to improve task success using online interactions. The authors showed that this approach is sample efficient and leads to significant performance improvements over the two considered domains.", "questions": "- My main concern is about the dataset assumption \"all episodes in the dataset end in a state where the episode goal is accomplished\". This seems quite a strong assumption. Could you clarify what your meaning of a goal is? In principle, every state is a goal. Are you simply considering that if a trajectory ends in goal g, the intent of the trajectory was to reach goal g? Or do you want something stronger, such as that you cannot have failures in the dataset? If so, why do you need that? How could you remove such assumption?\n\n- Another main limitation is that this work is specifically designed for goal-reaching tasks. How would you extend your approach beyond this class of problems?\n\n- Modern VLA has moved from action tokenization to direct prediction (e.g., using flow-matching). Any reason for using action tokenization in this work? What do you expect to change in that case?\n\n- Do you test multiple realizations of the 10%, 20% and 80% datasets or you consider only one realization? You mentioned that you considered 3 random seeds, could you explain the protocol? Did you repeat each phase 3 times? I could not find the unaggregate results in appendix. Could you add standard deviation (or any other metric) in Figure 3?\n\n- Figure 2, could you make clear what is the value of the predicted steps-to-go in moment 1? It seems to be very close to the condition `d<=3` that you use to terminate the episode? This makes me wonder that a slightly wrong hyper-parameter may result in an overall failure of the approach.\n\n- Appendix needs quite a bit of work. There are several sections not finished. Double check the main paper for missing references (e.g. in line 98).\n\n- Could you clarify the input to the VLM? Do you use a single image? Do you stack images to have information about velocity, etc? Do you use also state-based information?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- Overall I found the paper quite interesting and ready to read\n- The approach is interesting and results are promising\n\nWeaknesses\n- See section below", "was_revised": true, "final_justification": "The authors addressed my concerns and I decided to increase the score. I think the paper is novel and interesting for the community.", "rebuttal": "Thank you for your detailed and insightful review of our work! We hope our responses below address your questions. We look forward to a productive author-reviewer discussion period!\n\n**Goals & Dataset Assumption**\\\nThank you for this insight! Indeed we meant that “if a trajectory ends in goal $g$, the intent of the trajectory was to reach goal $g$”, with no further assumptions. This definition, however, only captures a subset of the settings we can address, and in fact our approach is more broadly applicable. More carefully stated, what we care about is that we can sample tuples $(o_t, a_t, g, t_g)$ where $o_t$ is the observation, $a_t$ is the associated action in the dataset, $g$ is a future goal that is accomplished at timestep $t_g$ in the trajectory where $(o_t, a_t)$ appeared. This statement subsumes the domains we considered in our work (LanguageTable with language goals, as well as the single-task domains), while incorporating other scenarios such as the one you pointed out, “In principle, every state is a goal”. We will make this update to our main manuscript.\n\n**Non-Goal-Reaching Tasks**\\\nIn this work we did not consider non-goal-reaching tasks. One potential avenue towards addressing such domains may come from literature connecting discounted RL in MDPs with Stochastic Shortest Paths (SSP). Specifically, there exists a construction that reduces any $γ$-discounted, bounded-reward MDP into an SSP [1]. Since SSP is a goal-reaching task with reward 1 in the goal state and 0 elsewhere, we may be able to apply our proposed approach.\n\n**Action Prediction**\\\nOur approach does not make any particular assumptions about the action prediction approach. One consideration when leveraging diffusion/flow-matching policies is that due to intractable log-likelihood, we would need to replace on-policy REINFORCE with a different RL algorithm. Recently there has been an explosion of RL methods designed for such policy formulations. More broadly, however, our general framework, reward model, and success detector do not need to change.\n\n**Dataset Sizes**\\\nWe create one realization of each dataset size, then train Stage 1 once for each dataset size, and subsequently train Stage 2 with 3 random seeds. The unaggregated results appear in Appendix H, Figure 10, Left. We will update Figure 3 with error bars.\n\n**Success Detection Threshold**\\\nIn simulated domains since we have access to the simulation state, we do not use the threshold success detector. We only use thresholding success detection for real-world experiments. Thus moment 1 in Figure 2 would not trigger the success condition. For each real-world domain, we decided the threshold by visualizing model predictions on various trajectories. The videos in the LanguageTable section of our supplementary videos website (link at the end of Section 1 of our manuscript) visualize many real-world trajectories while plotting the expected value of steps-to-go across time. As can be seen, the thresholding mechanism is quite reliable.\n\nFurthermore, so long as the threshold value is not too high (i.e. not too lenient) it would not have a large detrimental impact. If the threshold is too low, the main effect is that the episodes will contain a bit more amount of states near the goal.\n\n**Appendix Cleanup**\\\nThank you for the note! In Appendix K we plan on including a diagram depicting the infrastructure we built for on-robot on-policy RL with large foundation models.\n\n**Observations**\\\nFor the LanguageTable and BananaTable domains we include the current and previous image frames, and the input text represents the goal instruction. The output is a discretized action for the x direction and a discretized action for the y direction.\n\nIn Aloha, there are 4 cameras, but to mitigate training compute requirements, we a) stack the top and table view images into one image, and b) stack the left and right wrist view into one image as well, and pass these two images to the PaLI model. The input text includes discretized joint angle information. The output is an action chunk of 5 Aloha actions, with each dimension of each action being represented by a single token. Thus the output is 5 x 14 tokens.\n\nFor details we invite you to refer to Appendices D & E.\n\n[1] https://www.cse.iitm.ac.in/~prashla/notes/rlbook/discounted-cost-mdps.html#equivalence-of-discounted-mdp-and-ssp"}]

Author rebuttal:

Meta-review: This paper develops a new two-stage framework for post-training embodied foundation models: supervised fine-tuning with behavioral cloning and steps-to-go prediction, followed by a self-improvement phase that uses the learned steps-to-go estimates to shape rewards and detect task success for online reinforcement learning. Tested on LanguageTable and Aloha robots in simulation and the real world, the method improves sample efficiency, robustness, and skill generalization beyond imitation data, and enables autonomous learning of novel behaviors such as BananaTable manipulation. It reached a consensus that the approach proposed is novel, conceptually grounded and simple, and the experiments are comprehensive. The overall writing and presentation are also clear. I suggest the authors incorporate the feedback from the reviewers in preparing the camera-ready version.
Decision: Accept (poster)