Self-Improving Vision-Language-Action Models with Data Generation via Residual RL
Authors: Wenli Xiao
Haotian Lin
Andy Peng
Haoru Xue
Tairan He
Zhengyi Luo
Yuqi Xie
Fengyuan Hu
Linxi Fan
Guanya Shi
Yuke Zhu
Venue: ICLR 2026 Poster (2026)
Keywords: VLA
Robot Foundation Model
Robot Learning
Reinforcement Learning
Primary area: applications to robotics, autonomy, planning

Abstract: Supervised fine-tuning (SFT) has become the de facto post-training strategy for large vision-language-action (VLA) models, but its reliance on costly human demonstrations limits scalability and generalization. We propose Probe, Learn, Distill (PLD), a plug-and-play framework that improves VLAs through residual reinforcement learning and distribution-aware data collection. In Stage 1 (specialist acquisition), we freeze the VLA backbone and train lightweight residual actors via off-policy RL. These specialists take over in states where the base policy fails, thereby probing failure regions of the generalist. In Stage 2 (data collection), we employ a hybrid rollout scheme that biases residual interventions toward states frequently visited by the base policy, aligning collected trajectories with the generalist’s deployment distribution while capturing recovery behaviors. In Stage 3 (fine-tuning), these curated trajectories are distilled back into the generalist with standard SFT, applicable to both flow-matching and autoregressive heads. We evaluate PLD across diverse settings: it achieves a near-saturated 99% task success rate on the LIBERO benchmark, delivers over 50% performance gains in SimplerEnv, and demonstrates practicality on real-world Franka arm manipulation tasks. We further provide ablations showing that residual policy probing and distribution-aware replay are key to collecting deployment-aligned data that improves VLAs’ capabilities on both seen and unseen tasks. Our results demonstrate that RL-generated, policy-aligned data can surpass teleoperation-only demonstrations, offering a scalable path toward self-improving VLA models.

Published as a conference paper at ICLR 2026
SELF-IMPROVING VISION-LANGUAGE-ACCIII MOD- ELS WITH DATA GENERATION VIA RESIDUAL RL
Wenli Xiao 21, Haotian , Andy Peng" 3, Haoru Xue Tairan He Yuqi Xie', Fengyuan Hu' Jimmy Wu', Zhengyi Luo', Linxi "Jim" Fan' 1, Guanya Shi,, Yuke Zhu 41 NVIIDA, PCMU, SUC Berkeley, 'UT Austin, +GEAR Team Leads, 'Equal Contributions
ABSTRACT
Supervised fine-tuning (SFT) has become the de facto post-training strategy for large vision-anguagge-attion (VLA) models, but its reliance on costly human demonstrations limits scalability and generalization. We propose Probe, Learn, Distill (PLD), a three-stage plug-and-play framework that improves VLAs through residual reinforcement learning (RL) and distribution-avv -aware aware data collection. In Stage (specialist acquisition), we freeze the VLA backbone and train lightweight residual actors via off- off-policy RL. These specialists take over in states where the base policy fails, thereby probing failure regions of the VLA generalist. In Stage 2 (data collection),
employ a hybrid rollout scheme that biases residual inter- ventions toward states frequently visited by the base policy, aligning collected trajectories with the generalist's ' deployment distribution while capturing recovery behaviors. Stage 3 (fine-tuning), these curated trajectories are distilled back into the generalist with standard SFT, applicable both flow-matching and autoregres- sive heads. We evaluate PLD across diverse settings: it achieves a near-saturated 99% task success rate the LIBERO benchmark, delivers over 50% performance gains in SimplerEnv, and demonstrates 100% success rate on real-world Franka and YAM arm dexterous manipulation tasks. We further provide ablations showing that residual policy probing and distribution- lecting deployment-aliee aligned data that improves VLAs' capabilities both seen and unseen tasks. Our results demonstrate that RL-generated, policy-aligned data can surpass teleoperation-onl demonstrations, offering scalable path toward self- seff-improving VLA models.
replay are key to col-
Picc
Insertinn
GPU Insertion (Srape
Q: CPU lnce rion Brage
Pug in veloos Des
GFG Insortin (Stoge 3)
We demonstrate the performance of PLD on several real-world challenging manipulation tasks. The robot successfully picks up diverse objects and performs peg insertion on the Franka arm. Additionally, we deploy PLD on YAM bi-manual settings, showing that the PLD policy continuously performs GPU insertion and unplugging cycles nonstop for 1 hour without human intervention. Check videos at https: / /www. wenl ixiao. com/ self improve VLA-PIA
Figure 1:

---

Published as a conference paper at ICLR 2026
Policy Rollout
W SFT
W SFT
Ta930
Unseen Task Performance
Unseen Task Performance
Unseen Task Performance
80%
80%
30%
60%
60%
60%
Teaaso
Surcess
30%
30%
10%
10%
10%
Tocko
Seen lask Performence
Seen lask Performance
Seon Task Performance
Airaaning/Eslllutt
Seen
Rase Policy Roliout Data
Human Data
(Ours)
Unseen Task Performance
Base Palicy Roliout Dalaa
Human Data
PLD Dota (Cuns)
Figure 2: Synergetic effect of PLD data. We fine-tune To on subsets of LIBERO-90 with varying task coverage ratios, where each ratio (10-80%) indicates the fraction of distinct task instances included in training relative the full 90-task distribution. For each ratio, we randomly sample disjoint subsets of tasks and report the average results. The x-axis thus represents the degree of task coverage (not the number of trajectories), while the evaluation is always conducted on all 90 tasks. We compare different data formulations: PLD data yields the highest in-distribution performance while retaining the cross-task generalization property of high-quality human data. further enables modest-level zero-shot transfer even when trained on only 10% of tasks (24.4% SR on unseen tasks), whereas the VLA fine-tuned on base-policy rollout data (0-1 REINFORCE) underperforms and fails to generalize. (Success rate numbers reported in Table 2.)
INTRODUCTION
Supervised fine-tuning (SFT) has become the standard post-training paradigm for large language models (LLMs): after broad pre-training, models are adapted to downstream applications by training on curated instruction-response pairs, yielding many improvements in language following, safety, and generalization (Ouyang et al., 2022; Team, 2025). Inspired by these successes, the same recipe is now being applied to robot foundation models, particularly rision-language -aatuuggge-ttion action (VLA) policies, where large, heterogeneous robotics and vision- lsion-language datasets provide the base initialization. and SFT specializes models to specific tasks and embodiments (O'Neill et al., 2024; Team al., 2024; Kim et al., 2024; 2025; Black et al., 2024; Bjorck et al., 2025). However, transferring this paradigm unique challenge. Collecting high-quality robot demonstrations is both costly and labor-intensive, making large-scale datasets much harder to obtain. Even when such data are available, they are often collected through teleoperation pipelines that are decoupled from the deployed VLA policy, leaving critical coverage gaps: human operators must manually anticipate and correct failure modes, but their demonstrations rarely reflect the actual distribution of states the policy will encounter at deployment. As result, while SFT reliably improves performance on the tasks trained on, much less understood about whether these gains transfer to new tasks and environments.
from language roboties
These challenges raise the following question: Can VLA models improve themselves using RL- curated data with minimal human effort? Specifically, can this self-curated training match or surpass fine-tuning on numan-exper (oracle) teleoperation data, both in-distribution and out-of-distributtin Our central observation is that data collection should not be agnostic to the base policy: the data- collecting policy and the generalist must interact, so that exploration leverages the generalist's prior knowledge and collected data remain aligned with its trajectory distribution. A natural way in-
2

---

Published as a conference paper at ICLR 2026
stantiate this idea is to employ reinforcement learning (RL) to acquire task-specific specialists that guide data collection. However, applying RL in this setting is hindered by two key challenges. Sparse reward signals in language-conditione manipulation tasks render RL unstable and sample- inefficient. Moreover, training task-specific experts independently from the generalist introduces distributional mismatch, and once these experts converge, their behavior often lacks the diversity needed to provide robust coverage for SFT.
Motivated by these challenges, we introduce PLD, a three-stage post-training pipeline. Stage 1: Online specialist acquisition. We freeze the VLA backbone and train several lightweight residual actors for multiple tasks via sample-efficient pollcy RL, enabling them to "take over" the base policy at arbitrary states and achieve above 99% task success. Stage 2: Automatic data collection. We propose hybrid rollout scheme that biases residual takeovers toward states frequently visited by the base model, mitigating distribution shift while capturing recovery behaviors. Stage 3: Su- pervised fine-tuning. The collected data for multiple tasks are distilled back into the base model through SFT, process agnostic to VLA architectures, supporting both flow-matching and autore- gressive action heads (Black et al., 2024; Kim et. 2024). An overview of our pipeline can be found efficiently acquire task-specific RL experts through VLA-guide
in Figure 3. With PLD, exploration. Consequently, the VLA further improves using the PLD data, achieving performance above 99% on the LIBERO benchmark.
This paper makes the following contributions: 1) Autonomous post-training recipe. We propose post-training pipeline that enables VLA models improve autonomously without relying on additional oracle demonstrations. Our method achieves near-saturated 99% success rates on the LIBERO benchmark, and delivers over 50% performance gains in SimplerEnv, underscoring both its effectiveness on seen tasks and its ability to generalize to unseen ones. 2) Systematic study of RL RL-generated data. We analyze the key components of automatic data collection most beneficial for SFT, and conduct extensive experiments in simulation and on real robot hardware to exam- ine how Rl-generated data influences generalization to unseen tasks. 3) Comprehensive empirical validation. We provide large-scale ablations of our design choices. Besides, we showcase >99% success rate on Franka Arm and YAM arm dexterous manipulation tasks. Achieving continuous GPU insertion and unplugging operating for I hour without human intervention, offering potential for data-efficient post-training of robot foundation models.
2 PRELIMINARIES
2.1 TASK FORMULATION
We study snnguage-conditionne manipulation with sparse binary rewards using Lasion-La Action (VLA) models as the base policy class. We assume a partially observed control process with horizon T, where an episode terminates and resets on task success with a restricted time limit. After each episode, a reward E {0, 1} is assigned. Let g denote the language prompt of goal speci- fication, and let o, denote partial observations comprising robot proprioception (e.g., joint angle) and RGB images. The policy consumes (or, g) and outputs a 7-DoF action (6-Dof delta pose and 1-DoF continuous gripper command), which we express as at = Do(hg(o,,g) where hg is a vision-language backbone and D is an action head. Consistent with recent VIA models, D& is in- stantiated by one of three common families: (i) diffusion or flow-based action head for continuous control (Team et al., 2024; Black al., 2024), or (ii) a discrete action tokenizer for autoregressive decoding (Kim et al., 2024; Pertsch et al., 2025).
2.2 SUPERVISED FINE-TUNING
Given VLA policy and a demonstration data set D = {(o, It. a:) of observations o,, goal specifi- cations gt. and expert actions a, SFT adapts the policy by maximizing the likelihood of conditional action. Letting 2; (o,.gt), 9t), the canonical objective is behavior cloning (BC) loss. In contemporary VLA systems, the loss instantiation depends on the action head architecture. Auto-regressive/toknn heads (Kim al., 2024; Pertsch al., 2025) train with sequence NLL over action tokens ui.k:
LAR (0) Ex K] [logrdd po (u [)ck,))]
3

---

Published as a conference paper at ICLR 2026
Residual Policy Leamiing
Hybrid data collection
Base policy SFT
N SOL
[PAA]] [the] [red]
VLLA
VLA
instruction
Side Cant
Whet Carn
Propriception
FLD Detmet
Cept
Deploy Generalist
Reward
VLA
O2
0/1
7
poliy
Diverse Man pulasio Tasks
Figure 3: An overview of PLD. Our pipeline consists of three stages: 1) learning specialist residual policy for each task via online off-policy RL, with efficient exploration guided by frozen VLA generalist; 2) Automatic generation of hybrid trajectories by having the VLA rollout for the first t steps and let the specialist takeover to generate recovery data; 3) Supervised fine-tuning using collected multi-task PLD data; 4) Deploy the fine-tuned generalist diverse manipulation tasks in zero-shot.
With recent work improving efficiency via action chunking and parallel decoding, and continuous action parameterization trained by an & regression objective (Kim et al., 2025). Diffusion heads model a conditional denoising process for actions and train via score-matching MSE:
(o) = (,,,,((),) [||c cs (a, (noisy) 22, xr.t)
enabling iterative sampling inference (Team al., 2024; Chi al., 2024). Flow-matching heads learn continuous velocity field to transport prior to the action distribution, trained with an L2 flow-matching loss, and are often paired with VLM backbones for semantically grounded con- trol (Black et al., 2024; Intelligence et al., 2025). Across these heads, SFT remains the standard mechanism to specialize a generalist policies to new embodiments and tasks using modest labeled robot data (Kim et al., 2024; 2025).
2.3 GOAL-CONDITIONED RI.
We model continuous control as an MDP (Bellman, 1957) M= (S, p., with state space S, action space A, transition dynamics p(s' s, s, initial-state distribution Po, reward function and discount E (0, 1]. In goal-conditioned settings, each task is specified by goal variable g drawn from p(g); the reward becomes goal goal-dependent
A - R. and the policy S x A(4), is written as (aa g). is convenient to view GCRL an augmented MDP on x with stationary goals:
i((5' g) (8,g), = pls' s, 1lg =9}-
Under the infinite-horizon setting, the RL objective is
J ( Eysp(g) Bsg~po- a)-n(())).g) sesrrp(()ss..a))) E o² r(s g)).
(I)
We consider sparse binary reward setting, i.e., r(s, g) = [d(o(s). < defined via success predicate over a goal-relevant representation ((x), metric d, and tolerance & > 0.
3 METHODS
Method Overview We study the synergy between the data produced by our method when a mod- generalist VLA serves as the prior policy. The premise is that, if exploit the base policy's
4

---

Published as a conference paper at ICLR 2026
prior correctly, it can both solve hard tasks quickly and explore efficiently. While recent work plores direct RL fine-tuning of expressive policy class (Mark et al., 2024; Dong et al., 2025b), such formulation can be resource-intensive even for single-task tuning: e.g., LpenVLAAOFF requires per-GPU memory up to ~62.5 GB for LIBERO training at batch size 8 (Kim et al., 2025). Mean- while, it remains unclear whether these approaches scale gracefully to multi-task fine-tuning under heterogeneous setups. We therefore opt for a decoupled pipeline. We freeze the base policy mi and learn lightweight residual action policy 56 with sample-efficient off-policy RL (Gaussian policy parameterization). We then collect expert data by letting the residual "take over" after specified steps of "base policy probing". Finally, we distill these skills back into the base model via SFT and deploy the generalist on various manipulation tasks. We provide an overview of PLD in Figure 3.
3.1 DATA EFFICIENT RL VIA POLICY PRIOR WARM-START
Building upon the previous success of sample-efficient RL with prior data (Ball et al., 2023), we consider off-policy actor-critic framework and maintain two separate buffers for offline and online experience replay. We first fill the offline buffer with successful rollouts Boffe = [TI, .2.... from the base policy Th. This process serves attempts. During training, the offline and online experiences will be replayed symmetrically; for example, mini-batches consist of equal samples from both buffers, ensuring that the value function is constantly trained on high-value state-action pairs.
importance sampling to preserve only successful
In practice, train task-specific residual action module (-(-,s, ag) conditioned on as Tb- We use iTs to explore near the base policy behavior, actively searching for more optimal solutions guided by the Q-function. To modulate exploration and avoid deviating drastically from TT, during the initial phase, the delta action's 's magnitude is scaled down to [- where & E [o, 1] is tuned by scheduler. This design choice is two-fold: First, although unable perfectly generalize to an unseen manipulation task or scenario, the base policy can make reasonable attempts to solve the task, serving as useful initialization for exploration. Moreover, directly training the expressive foundation policy (e.g.. flow action heads) to maximize the Q-value can be extremely difficult (Mark et al., 2024). In contrast, a residual Gaussian policy can be easily trained through any off-the-shelf off-policy RL algorithm.
Alongside T&s action value function Qa acquired through policy iteration and TD-Jearning (Sutton & Barto, 2018) as in Eq. equation 2, where (-|ss) mu(-1s) a.) is the combined policy. (st: t r(s,a) +. (s,, Qtarget ($t+1. 1(+1)], as + as
(2)
To stabilize off-policy learning and mitigate forgetting, we introduce warm-up stage using solely Tr for data collection akin to (Zhou et al., 2024b). Mcanwhile, the Q-function is initialized by a conservative objective such Cal-QL (Nakamoto et al,, 2024). Importantly, we do not explicitly enforce behavior constraints to policy loss, such that the resulting expert म is less influenced by either data quality or base policy performance.
3.2 BOOTSTRAPPING RL SPECIALIST FOR SCALABLE DATA GENERATION
We then turn to the question of how to collect demonstration data using RL specialists. Data col- lected through RL experts is highly optimal, with consistent behavior and nearly no hesitation, demonstrating smooth solutions that finish tasks with a shorter horizon. However, such a narrow distribution of unimodal expert behavior may leave out-of-distribution and failure states underrepre- sented. Thus, scaling purely expert data may not result in performance gain, but instead risks the generalist overfitting on these data and harming both robustness and generalization (As discussed in the following section).
To mitigate this issue, we propose a hybrid data collection scheme that incorporates base-policy initialization: We first rollout the base policy for random steps, then let the learned residual RL. policy take over, resulting in demonstration trajectories Tdemo = {(s1, as.1), (st-1, ab,t-1) U ((1,, db,t + dz).... } that contain the behavior of the expert recovering from potential suboptimal region. We refer to this procedure as base policy probing. Accordingly, we boost the robustness of the RL expert by training the RL expert
initial state distribution 8o po given by random steps of base policy probing. The probing step only serves as state initialization and will not be added to the replay buffer. The details of PLD are summarized in Algorithm
5

---

Published as a conference paper at ICLR 2026
Language Prompt: Open the bottom drawer
Becover Ratio: Ofe
Recover Ratio: 20%
Recover Ra 40%
Figure 4: Visualization of Data diversity. We visualize PLD data with different base policy ini- tialization probing horizons. Increasing probing horizon yields longer episodes and greater diversity among successful trials. This broader data support leads to improved fine-tuning performance, which eventually saturates. (As the saturation curve shown in Figure 13).
4 EXPERIMENTS
In this section, we systematically evaluate the effectiveness of PLD. We first demonstrate the effi- ciency of PLD-RL in solving sparse-reward manipulation tasks, which serves as the cornerstone of our pipeline. Then we focus on study 1) How does the probing mechanism of PLD benefit VLA SFT; 2) How does PLD data compare with other sources of demonstrations. Finally, we investigate the key factors of our pipeline and how they contribute to improving the performance of VLA.
We consider simulation as proxy to real-world performance, and evaluate methods across two widely adopted simulation benchmarks, including LIBERO (Liu al., 2023), SimplerEnv (Li et al., 2024). LIBERO is lifelong learning benchmark focused anguage-ggiide manipulation tasks. It comprises 130 tasks grouped into four suites that stress object distribution, spatial arrange- ment, task goals, and their mixture. SimplerEnv is robotics manipulation benchmark that aims for high sim-to-real correlation.
In the following sections, will be mainly analyzing different data sources: PLD data PLD Human data Humman RL expert data (RL expert rollout w/o base policy probing) DRL, and base- policy rollout data (Selective successful rollouts, also referred to as "self-bootstrap data") D Policy Unless stated otherwise, all methods use identical data volume, training budgets, augmentation, and hyperparameters across architectures; the default base policy used is mo (Black al., 2024).
4.1 EFFECTIVENESS AND EFFICIENCY OF LEARNING RL SPECIALIST
In this section, we seek answers to the following questions: Does PLD benefit from both policy guidance and hybrid online learning? We compare state-of-the-art methods that leverage policy priors and data priors: WSRL (Zhou et al., 2024b) (offline initialization only); RLPD (Ball et al., 2023) (No base policy guidance). For the pre-training stage, we collect dataset of 50 trajectories per task, containing only the successful trials of the same base policy (To). and using Cal-QL (Nakamoto al,, 2024) the default pre-training algorithm. Subsequently, we retain these data for methods with online hybrid data replay. We plot the training curve of 250k steps of online interaction, showing mean rollout performance and 95% CIs (confidence levels) across seeds in Figure
PLD outperforms baseline methods by large margin across 8 tasks on LIBERO-90, indicating that PLD effectively exploits the VLA policy prior and yields pronounced sample efficiency at low in- teraction budgets. In terms of asymptotic performance, PLD can achieve over 95% performance every task that we report to fine-tune performance (over 120 manipulation tasks). Notably, we ob- serve initial performance drop for PLD. This phenomenon implies the initial phase of exploration, where the residual policy starts to diverge from the base policy and visits potentially suboptimal states. Ablation study PLD-RL's design choice be found in the Section E.2.

---

Published as a conference paper at ICLR 2026
PLD (Ours) Ino base model)
WSRL-
Task 1
Tkik
Task 17
Task 21
Success Rate
0.5
0.0
10ok
200k
200R
10tte
200k
100%
200k
33
1.0
1.0
Success Rate
0.0
0.0
0.0
0.0
L00K
200k
100h
200k
100k
200k
100k
200k
Actor Steps
Actor Steps
Actor Steps
Actor Steps
Figure 5: Benchmarking Sample-Efficient RL Performance. We compare PLD with RL baseline algorithms that either leverage policy prior or data prior. We report mean rollout performance (Av- erage return calculated within sliding window of 100 episodes) and 95% CIs for seeds across manipulation tasks selected from LIBERO-90.
Table 1: Performance on LIBERO benchmark of VL.A models fine-tuned on PLD data.
Open
Model
Spatial
Object
Goal
Ayg
Spatial
Object
Goal
Avg
Baseline (SFT/OFT)
95.2
97.6
87.4
93.4
92.9
99.1
83.25
91.8
w/ PID)
97.7
98.5
95.3
97.2
99.5
99.1
98,9
99.2
A
+2.5
+0.9
+7.9
+3.8
+6.6
+0.0
+15.7
+7.4
4.2 IN-DISTRIBUTION PERFORMANCE
In this section, we investigate how effectively the proposed pipeline enhances the performance of the VLA. We evaluate in-distribution fine-tuning on the LIBERO benchmark using three subsets, each consisting of 10 ingguage-conditoon
tasks: LIBERO-ORE -bject- LIBERO-Spaticl and LIBERO- Goal. We additionally report results on a custom suite that consists of 4 tasks from SimplerEnv. To demonstrate architecture architecture-agnosiiiiss agnosticism, we instantiate the base VLA with (i) Open VLA (autoregres- sive action tokens) (Kim et al., 2024) and (ii) TD (flow-matching action head) (Black et al., 2024). Since VLA models mainly trained real-world datasets that cannot work out of the box simulation benchmarks, we leverage their official checkpoints for model fine-tuning on each bench- mark as the baseline. At test time, each policy is evaluated 50 episodes per task, and we report the mean success rate per suite and average over the benchmark. Table and Table list the perfor- mance gain achieved by further applying our method. Across all suites and both architectures. PLD data yields consistent absolute gains over human-only SFT while requiring no additional human demonstrations. We observe that larger PLD datasets monotonically improve in-distribution success and that the distilled generalist notably surpasses the average specialist, indicating effective transfer of task-specific competence into the base VLA.
4.3 GENERALIZATION
Generalization to Unseen tasks To study the synergetic effect of PLD data, we examine whether PLD data improves zero-shot performance on unseen tasks in the LIBERO benchmark (Liu et al., 2023). Concretely, we fine-tune яo via SFT using data drawn from the disjoint coverage subsets of LIBERO-90 in proportions 10.1, 0.3, 0.6, 0.8, 1.0): for each coverage level, we randomly sample tasks to form a new subset in distribution and then evaluate all tasks the suite. We sampled subsets for each coverage level to provide more unified result. We consider three different data sources: (i) Ours DPDD (ii) human expert data Humman and (iii) relf-bootstrapping data DDo rollout (SFT on this variant corresponds to 0-1 REINFORCE (Shenfeld et al., 2025)). We visualize the result in Figure 2. Across coverage levels, an fine-tuned on D attains the strongest in in-distribution performance and maintains robust zero-shot transfer unseen tasks; human data-only SFT achieves

---

Published as a conference paper at ICLR 2026
Training
Testing Eni
Lierr-90
LibereCocal-
open the
Libere Goal
open the top drawer and put the bowl inside
lbeeo-Coall
cream cheese in the bowl
Task 2. put the black bow in the top drawer of cabinet
Task the bottom drawer of the cabinet
Task
open drawer of cabinet put the bowl
100 Task
Task 6
Task
Task
500 for Libero Sm
125
X2x
200
125
103
100
scale training (1
bero--Goal dalaset
Policy Rollout
(wlo probing)
probing)
Figure 6: Few-shot generalization. Scaling in-distribution (LIBERO-goal) PLD yields better few- shot performance on new tasks (LIBERO-90).
Ostasct State Coverage
Fallure Made MLA with HuHaan Dats
Fofgut
Mode
Stack there forever
Figure 7: Visualization of foreen receve and recovery behavior in the real-world.
RLD
approximately similar level of zero-shot generalization the same training budget but lags in-distribution tasks; mo self-bootstrapping rollout data underperforms in-distribution and fails to generalize dittr--distiibution tasks.
Generalization to Out-of-domain We study few-shot generalization for tasks with different goals, layouts, and backgrounds. We first collect PLD data of varying scales set on source tasks (LIBERO-Goal) and evaluate the fine-tuning performance on targel tasks (LIBERO-90). Specifi- cally, the VIA is fine-tuned on PLD data of source tasks plus a small number of oracle demos of target tasks. To analyze transfer by skill family, we select tasks from LIBERO-goal goal and LIBERO-90 that have high semantic correlation to form a set of source/target tasks. We scaled the size of [DPLD] from 50 to 500 trajectories and compared against DLL and BSS under the same data and training budget. As shown in Figure 6, we observe monotonic improvements in SFT performance as the data scales from 50 500 trajectories.
4.4 REAL-WORLD PERFORMANCE
We evaluate our approach 7-DoF Franka Emika Panda arm in the real world, considering two sets of canonical manipulation tasks: pick-and-place and peg insertion, illustrated in Figure 18. Unlike prior works (Luo et al., 2025; Zhou et al., 2024b), do not restrict task randomization, making real-world reinforcement learning particularly challenging. more detailed experimental setup is provided in Section G.I.
8

---

Published as a conference paper at ICLR 2026
Data collection and policy training. We first collected 200 teleoperated trajectories to perform supervised fine-tuning (SFT) of the base policy To- Using this initialization, we trained T and To- -o-LLP without human interventions. Both policies reached 100% success on the two tasks within hours of training. We then leveraged the learned expert policies to autonomously collect 200 successful demonstrations each, forming datasets DPLD and DRLPD, which were subsequently used to further SFT To, yielding +DPLD, DDlluman, and +DRLPD
Performance and failure modes. Across 30 randomized trials per task, all methods achieved perfect success on peg insertion (30/30), demonstrating robust reactive skills. In cube pick-up, however, + +DRLPD and +DHummn succeeded in only 16/30 and 10/30 trials, respectively, + maintained 30/30. Figure illustrates typical failure: policies trained on DRLPD Dumman often pushed the cube into the upper-left corner, where the gripper became stuck. By contrast, +DPLD was reliably recovered by repositioning the cube before grasping. Distribution analysis confirms that neither human demonstrations nor RL rollouts visited such corner states, whereas PLD explicitly probed the base policy and generated diverse trajectories that captured these cases. This explains its robustness and highlights its potential as a self-improving data flywheel.
Robustness for long-horizon tasks. To evaluate the robustness of PLD in executing long-horizon and dexterous manipulation tasks, we set up two 6-DoF YAM robot arms developed by 12RT- Robotics (2025). We consider an industrial insertion task lask-specfficallly inserting micro graphics card into a motherboard. To enable fully autonomous operation without human intervention or re- setting, we decompose the task into four stages: Stage I: Pick up the GPU from the table and insert into slot 1. Stage Move the GPU from slot 1 to slot 3. Stage Firmly insert the GPU into slot 3. Stage 4: Unplug the GPU from slot 3 and place it back on the table. A reward classifier is trained to govern the state machine that coordinates these stages. After at most 8 hours of training for each subtask and distilling the learned skills into generalist VLA policy (implementation described in Section F.4), the system can continuously perform the full task loop without human assistance for at least I hour. As shown in the video, although the one-shot success rate for each stage is not 100%, the system is capable of recovering from failures, keeping the data flywheel running autonomously.
4.5 How DOES PLD WORK?
We take a deeper look at the underlying reason for PLD data's bonus in generalization. As shown in Figure 9, we plot 50 trajectories for each method (task description: "open middle drawer of the middle cabinet"). RI. expert provides optimal and concentrated solutions to the task, but lacks diversity and diverges far from the behavior of the base policy, while PLD data clustered near the trials of the base policy and contain various recovery behaviors. Based on empirical observation, we hypothesize that due to the base policy probing, PLD data provides solution that is biased towards the base policy, thus fine-tuning forgets less of the base model's generalizability. This resembles observations in LLM fine-tune (Shenfeld et al., 2025), where the KL-divergence can serve as an indicator of forgetting. Meanwhile, large data coverage also benefits robustness in sequential decision making (Kelly al., 2019).
5 RELATED WORKS
5.1 ROBOTICS FOUNDATION MODELS
Following the success of large language models and vision language models (Brown et al., 2020; Touvron et al., 2023; Chen et al., 2022), recent works on robotics foundation models turned to similar transformer-basee
architecture with aggressive data scaling. This inspired earlier works in VLAs such as RT-1, RT-2, and Open etc. (Brohan et al., 2023b;a; Kim et al., 2024; Xue et al., 2025). Meanwhile, diffusion- from generative modeling techniques (Ho et al., 2020), demonstrating smooth and accurate action generation. This has led to more recent VIA architectures to date, such as Octo (Team et al,, 2024), OpenVLAA-OFT (Kim et al., 2025), GROOT (Bjorck et al., 2025), and the m-series of models (Black et al., 2024; Intelligence et al., 2025; Pertsch et al., 2025). The VLA training procedure is
action generation, explored in Chi et al. (2024), takes motivation
'Generalization results reported Section G.
9

---

Published as a conference paper at ICLR 2026
typically analogous to VLM training. First, model weights are initialized from the respective VLM backbones (Kim et al., 2024; Black et al,, 2024). Then, the model is supervised with next-token- prediction tasks on diverse pretraining datasets, spanning across multi-modal web data (Intelligence et al,, 2025) such as COCO (Chen et al,, 2015) and VQAv2 (Goyal et al,, 2017). and robotics- specifie, cross-embodiment data (Khazatsky et al., 2025; O'Neill et al,, 2024). Finally, supervised fine-tuning is conducted on a small set of high-quality telcoperation data collected from the target robot deployment platform performing the target tasks.
5.2 SAMPLE-RFFICIEN RL WITH DATA AND POLICY PRIORS
Sample and exploration efficiency have been long-standing problem in RL, especially in sparse- reward settings. Recent works have explored leveraging offline data to improve sample efficiency. Offiin-to-online transfer (Vecerik et al., 2017; Nair et al., 2020; Kostrikov et al., 2021; Nakamoto et al,, 2024; Zhou et al., 2024b; Li et al., 2025) considers two-stage pipeline that first initializes policy or critic using pessimism or constrained objective in offline RL Levine et al. (2020) and follows with an online fine-tuning phase to have new data collected and alleviate distributional shift; Hybrid RL (Song et al., 2022; 2024; Ball al., 2023) considers online RL with access to an offline dataset. Given expert demonstration, one can either continuously replay this data to ensure high- value state visitation Ball et al. (2023) or to guide exploration Dong et al. (2025a). Data prior can also guide reset-free real-world learning (Walke et al., 2023; Sharma et al., 2023). Another line of work assumes access to policy prior, such as pre-trained generalist. Ye et al. (2023); Chen et al. (2025;; Jülg et al. (2025) leverage foundation policy to guide RL through an auxiliary behavior regularization objective. Action editing is another efficient way to improve upon the policy prior. ResiP (Ankile et al., 2025) considers learning residual policy through PPO (Schulman et al,, 2017), while EXPO (Dong et al., 2025b) considers an off- off-policy solution and co-trains the base policy during the process. Our work leverages suboptimal base policy to achieve non-zero success rate for warm -ttrt-stating exploration, but does not require access to oracle demos or a human expert for further intervention.
5.3
VLA POSTIIRAINING
The prevailing large-scale recipe for VLA post-training is to pretrain on diverse, heterogeneous robot data and then fine-tune on task-specific demonstrations (Zhou al.,, 2024a; Black et al,, 2024). For example, Black et al. (2024) performs supervised post-training on carefully curated task- targeted corpus, with per-task coverage ranging from a few to over 100 hours of teleoperation. Because such post-training data are expensive to acquire, the authors note that most diversity must come from the pretraining limited coverage at adaptation time. To enable self-immproeement, prior work has explored scaling high-quality data via online RI. specialists (Ball et al., 2023). However, these pipelines often require substantial human intervention and collect data a large way agnostic to the generalist's s behavior, restricting scalability. Other lines investigate on-policy RL for post-training (Lu et al., 2025; Tan et al, 2025), or optimize single-task fine-tuning at the expense of generalization (Chen et al., 2025). Our work jointly targets these limitations by seeking a post-training pipeline that reduces human effort, aligns data collection with the generalist's 's state distribution, and remains sufficiently sample efficient for real-world systems.
crstuuu--unddrsooriii a key limitation of pure SFT: data scarcity and
CONCLUSIONS
We presented PLD- -a three-stage post-training pipeline that enables VLA models to improve au- tonomously without relying on additional oracle human demonstrations. PLD couples frozen VLA generalist with lightweight residual RL specialists to warm-start exploration and distills curated suc- cesses back into the base model with standard SFT. Across large-scale simulation experiments and real-world deployment, PLD improves without additional human demonstration, achieving near- saturated ~99% success on LIBERO, >50% gains in SimplerEnv, and robust real-world perfor- mance. Ablations identify residual policy probing and distribution-avan -aware replay as key to stability, sample efficiency, and generalization. We consider PLD as practical step toward autonomous, scalable post-training and a foundation for future work on multi-embodiment transfer, continual on-robot learning, and safety-constrained data collection.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Lars Ankile, Anthony Simeonov, Idan Shenfeld, Marcel Torne, and Pulkit Agrawal. From imitation to reffinement-resiuuse rl for precise assembly. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pp. 01-08. IEEE, 2025.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.064450 2016.
Philip Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learn- ing with offline data. In International Conference on Machine Learning, pp. 1577-1594, PMLR, 2023.
Richard Bellman. A markovian decision process. Journal of mathematics and mechanics, pp. 679- 684, 1957.
Johan Bjorek, Fernando Castañeda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, al. GrOOt nl: An open foundation model for generalist humanoid robots. arXiv preprint 2rXiv:2503.14735 2025.
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. To: sisio--lnguagg-actinn flow model for general robot control. arXiv preprint arXiv:2410.24164 2024.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Miine and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http:
//qithub. com/ j ax aa-mm/l jax.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choro- manski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexander Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Lisa Lee, Tsang-Wer Wei Edward Lee, Sergey Levine, Yao Lu, Hen- ryk Michalewski, Igor Mordatch, Karl Pertsch, Kanishka Rao, Krista Reymann, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Pierre Sermanet, Jaspiar Singh, Anikait Singh, Radu Soricut, Huong Tran, Vincent Vanhoucke, Quan Vuong, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. Rt- models transfer web knowledge to robotic control, 2023a. URL
2: Vision-laaguuge-actteo https: / arxivv orgg abss
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalash- nikov, Yuheng Kuang. Isabel Leal, Guang-Huer Lee, Sergey Levine, Yao Lu, Utsav Malla, Deek- sha Manjunath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton Tan, Huong Tran, Vin- cent Vanhoucke, Steve Vega, Quan Vuong, Fci Xia, Ted Xiao, Peng Xu, Sichun Xu. Tianhe Yu, and Brianna Zitkovich. Rt-1: Robotics transformer for real-world control at scale, 2023b. URL https: //traaivvvvv/////////a/eeeeea/wa www.vviiog/.v.oooo.o..o..ooooore alss /2212 06817.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam. Girish Sastry. Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., neurips oo wappppppr.... files/ www.s//papee//22222/ pappx/202c/
2020. URL httpp::s//ppooceeeeee 96741 42 .ddf.
11

---

Published as a conference paper at ICLR 2026
Xi Chen, Xiao Wang. Soravit Changpinyo, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly-scaled multilingual eeguage-imagge model. arXiv preprint arXhv:2209.0679M 2022.
Xinlei Chen, Hao Fang. Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollar, and C. Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server, 2015. URL
https:///are org/ alss/ 1504... 00025.
Yuhui Chen, Shuai Tian, Shugao Liu, Yingting Zhou, Haoran Li, and Dongbin Zhao. Con- fft: A reinforced fine-tuning method for vla models via consistency policy. arXiv preprint arXiv:2502.055500 2025.
Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion, 2024. URL
https: arxiv org/ abs 2303. 04137.
Perry Dong, Alec M Lessing, Annie Chen, and Chelsea Finn. Reinforcement learning via implicit imitation guidance. arXiv preprint arXiv:2506.07505, 2025a.
Perry Dong, Qiyang Li, Dorsa Sadigh, and Chelsea Finn. Expo: Stable reinforcement learning with expressive policies. arXiv preprint arXiv:2507.07996, 2025b.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 2020.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference on machine learning, pp. 1587-1596. PMLR, 2018.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6904-6913, 2017.
Siddhant Haldar, Jyothish Pari, Anant Rai, and Lerrel Pinto. Teach robot to fish: Versatile imitation from one minute of demonstrations. arXiv preprint arXiv:2303.01497, 2023.
Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models, 2020. URL
https: Marxivv .aw.vorrg/ao abs / /2006 11239.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. URL https:
//arxiv. orr.viorrr/aaaae
waarggabba /aas/ 2106 09665.
I2RT-Robotics.
Yam - 6-dof robotic arm.
https://i2rts wwt.:/ttt.cooo.o.ooooommooooooot htto///potcss////oooooooo
yam- manimulatorooooooo 2025.
Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, et al. [0.55:: a vision-language- action model with open- open-world generalization. arXiv preprint arXiv:2504. 16054, 2025.
Tobias Jülg, Wolfram Burgard, and Florian Walter. Refined policy distillation: From vla generalists to rl experts. arXiv preprint arXiv:2503.0m 05833, 2025.
Michael Kelly, Chelsea Sidrane, Katherine Prigs--Campbell, and Mykel J Kochenderfer. Hg- dagger: Interactive imitation learning with human experts. In 2019 International Conference on Robotics and Automation (ICRA), pp. 8077-8083. IEEE, 2019.
Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, Peter David Fagan, Joey Hejna, Masha Itkina, Marion Lepert, Yecheng Jason Ma, Patrick Tree Miller, Jimmy Wu, Suneel Belkhale, Shivin Dass, Huy Ha, Arhan Jain, Abraham Lee, Young- woon Lee, Marius Memmel, Sungjae Park, Ilija Radosavovic, Kaiyuan Wang, Albert Zhan, Kevin Black, Cheng Chi, Kyle Beltran Hatch, Shan Lin, Jingpei Lu, Jean Mercat, Abdul Rehman, Pannag Sanketi, Archit Sharma, Cody Simpson, Quan Vuong, Homer Rich Walke, Blake
12

---

Published as a conference paper at ICLR 2026
Wulfe, Ted Xiao, Jonathan Heewon Yang. Arefeh Yavary, Tony Z. Zhao, Christopher Agia, Ro- han Baijal, Mateo Guaman Castro, Daphne Chen, Qiuyu Chen, Trinity Chung, Jaimyn Drake, Ethan Paul Foster, Jensen Gao, Vitor Guizilini, David Antonio Herera, Minho Heo, Kyle Hsu, Jiaheng Hu, Muhammad Zubair Irshad, Donovon Jackson, Charlotte Le, Yunshuang Li, Kevin Lin, Roy Lin, Zehan Ma, Abhiram Maddukuri, Suvir Mirchandani, Daniel Morton, Tony Nguyen, Abigail O'Neill, Rosario Scalise, Derick Seale, Victor Son, Stephen Tian, Emi Tran, An- drew E. Wang, Yilin Wu, Annie Xie, Jingyun Yang, Patrick Yin, Yunchu Zhang, Osbert Bastani, Glen Berseth, Jeannette Bohg. Ken Goldberg, Abhinav Gupta, Abhishek Gupta, Dinesh Jayara- man, Joseph Lim, Jitendra Malik, Roberto Martín MartinnMartin, Subramanian Ramamoorthy, Dorsa Sadigh, Shuran Song, Jiajun Wu, Michael C. Yip, Yuke Zhu, Thomas Kollar, Sergey Levine, and Chelsea Finn. Droid: A large-scale in-the-wild robot manipulation dataset, 2025. URL https: /tts://arxiiii org/ http//as//// 2203/2403.02 12945.
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-acttoon model. arXiv preprint (Bl::2006.09244, 2024.
Moo Jin Kim, Chelsea Finn, and Percy Liang. Fine-tuning vision- aision-anggaag--ctto models: Opti- mizing speed and success, arXiv preprint arXiv:2502.19645, 2025.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. arXiv preprint arXiv:2110.06166 2021.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. reinforcement learning. Advances in neural information processing systems, 33:11 19111999191,, 2020.
Conservative q-learning for offline
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives open problems. arXiv preprint arXiv:2005.01643 2020.
Qiyang Li, Zhiyuan Zhou, and Sergey Levine. Reinforcement learning with action chunking. arXiv preprint (tt.::2507.099999 2025.
Xuanlin Li, Kyle Hsu, Jiayuan Gu, Karl Pertsch, Oier Mees, Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sieh, Sean Kirmani, et al. Evaluating real-world robot manipulation policies in simulation. arXiv preprint 2rXiv:2405.05941, 2024.
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: arXiv: 0wt..0299999911 2015.
Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems, 36:44776-44791, 2023.
Guanxing Lu, Wenkai Guo, Chubin Zhang, Yuheng Zhou, Haonan Jiang, Zifeng Gao, Yansong Tang, and Ziwei Wang. Vla-rl: Towards masterful and general robotic manipulation with scalable reinforcement learning. arXiv preprint trXiv:250S.18719 2025.
Jianlan Luo, Zheyuan Hu, Charles Xu, You Liang Tan, Jacob Berg. Archit Sharma, Stefan Schaal, Chelsea Finn, Abhishek Gupta, and Sergey Levine. Serl: A software suite for sample-efficient robotic reinforcement learning. In 2024 IEEE International Conference on Robotics and Automa- tion (ICRA), pp. (1961-16969, IEEF, 2024.
Jianlan Luo, Charles Xu, Jeffrey Wu, and Sergey Levine. Precise and dexterous robotic manipulation via human-in-the- -hote--oop reinforcement learning. Science Robotics, 10(105):cads5033, 2025.
Max Sobol Mark, Tian Gao, Georgia Gabriela Sampaio. Mohan Kumar Srirama, Archit Sharma, Chelsea Finn, and Aviral Kumar. Policy agnostic rl: Offline rl and online rl fine-tuning of any class and backbone. arXiv preprint wrXiv:2412.06685, 2024.
Ashvín Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online rein- forcement learning with offline datasets. arXiv preprint arXiv:2006.09355 2020.
13

---

Published as a conference paper at ICLR 2026
Mitsuhiko Nakamoto, Simon Zhai, Anikait Singh. Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-ql: Calibrated offline rl pre-training for efficient online fine- tuning. Advances in Neural Information Processing Systems, 36, 2024.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022.
URL https: //tp//axii// orgg abs /2203.222.. 02155.
Abby O'Neill, Abdul Rehman, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acom Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, et al. Open x-embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration O. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 6892-6903. IEEE, 2024.
Karl Pertsch, Kyle Stachowicz, Brian Ichter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, and Sergey Levine. Fast: Efficient action tokenization for vision-language-action models. arXiv preprint arXiv:2501.09747, 2025.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Archit Sharma, Ahmed M Ahmed, Rehaan Ahmad, and Chelsea Finn. Self-improving robots: End- to-end autonomous visuomotor reinforcement learning. arXiv preprint arXiv:2303.01488 2023.
Idan Shenfeld, Jyothish Pari, and Pulkit Agrawal. RI's razor: Why online reinforcement learning forgets less. arXiv preprint arXiv:2509.042) 2025.
Yuda Song, Yifei Zhou, Ayush Sekhari, J Andrew Bagnell, Akshay Krishnamurthy, and Wen Sun. Hybrid rl: Using both offline and online data can make efficient. arXiv preprint arXiv:2210.06718, 2022.
Yuda Song, J Andrew Bagnell, and Aarti Singh. Hybrid reinforcement learning from offline obser- vation alone. arXiv preprini 2rXiv:2406.07253, 2024.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Shuhan Tan, Kairan Dou, Yue Zhao, and Philipp Krähenbühl. Interactive post-training for vision- language-action models. arXiv preprint arXiv:2505. 17016, 2025.
Gemini Team. Gemini: family of highly capable multimodal models, 2025. URL https:
//arxiv. orgg abs 221222111222 11805.
Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al. Octo: An open-source generalist robot policy. arXiv preprint arXiv:2405. /22 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux. Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kamhadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. URL https: : / wawaiiivv......./ www.o/////oo.. abs////////// 2207/ 09288.
Ikechukwu Uchendu, Ted Xiao, Yao Lu, Banghua Zhu, Mengyuan Yan, Joséphine Simon, Matthew Bennice, Chuyuan Fu, Cong Ma, Jiantao Jiao, et al. Jump-start reinforcement learning. In Inter- national Conference Machine Learning, pp. 34556-34583. PMLR, 2023.
14

---

Published as a conference paper at ICLR 2026
Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nico- las Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller. Leveraging demonstra- tions for deep reinforcement learning robotics problems with sparse rewards. arXiv preprint orXivv/707.000
2017.
Homer Rich Walke, Jonathan Heewon Yang, Albert Yu, Aviral Kumar, Jedrzej Orbik, Avi Singh, and Sergey Levine. Don't start from scratch: Leveraging prior data to automate robotic reinforcement learning. In Conference on Robot Learning, pp. 1652-1662. PMLR, 2023.
Haoru Xue, Xiaoyu Huang, Dantong Niu, Qiayuan Liao, Thomas Kragerud, Jan Tommy Grav- dahl, Xue Bin Peng, Guanya Shi, Trevor Darrell, Koushil Sreenath, and Shankar Sastry. Le- verb: Humanoid whole-body control with latent vision- vision-language instruction, 2025. URL hotps:
//arxiv. org/ abs 25006250600 13751.
Weirui Ye, Yunsheng Zhang, Haoyang Weng, Xianfan Gu, Shengjie Wang, Tong Zhang, Mengchen Wang, Pieter Abbeel, and Yang Gao. Reinforcement learning with foundation priors: Let the embodied agent efficiently learn on its own. arXiv preprint arXiv:2310.02633. 2023.
Kelin Yu, Yunhai Han, Qixian Wang, Vaibhav Saxena, Danfei Xu, and Ye Zhao. Mimictouch: Lever-
aging multi-modal human tactile demonstrations contact-rich manipulation. arXiv preprint arXiv:2310.1695 16917, 2023.
Zhiyuan Zhou, Pranav Atreya, Abraham Lee, Homer Walke, Oier Mees, and Sergey Levine. Au- tonomous improvement of instruction following skills via foundation models. arXiv preprint arXivv2407.20663, 2024a.
Zhiyuan Zhou, Andy Peng, Qiyang Li, Sergey Levine, and Aviral Kumar. Efficient online reinforce- ment learning fine-tuning need not retain offline data, arXiv preprint arXiv:2412.07772, 2024b.
15

---

Published as a conference paper at ICLR 2026
A LARGE LANGUAGE MODEL USAGE STATEMENT
We used the large language model ChatGPT only for writing assistance, including grammar correc- tion, wording improvements, and minor stylistic edits draft text. The model was not used for research ideation, data collection, dataset labeling, code generation, experiment design, or analysis. All technical content was authored and verified by the human authors. We accept full responsibility for all content in this paper.
B EXTENSIVE RELATED WORKS
Robotics Foundation Models Following the success of large language models and vision lan- guage models (Brown al,, 2020; Touvron et al., 2023; Chen et al., 2022), recent works on robotics foundation models turned to a similar transformer-bee based architecture with aggressive data scaling. This inspired earlier works in VLAs such as RT-1, RT-2, and VLe etc. (Brohan et al., 2023b;a; Kim et al., 2024; Xue et al., 2025). Meanwhile, diffusion- biaruddoo---eeddddd action generation, explored in Chi al. (2024), takes motivation from generative modeling techniques (Ho et al., 2020), demonstrating smooth and accurate action generation. This has led to more recent VLA architectures to date, such as Octo (Team et al., 2024), OpenVLA-OFT (Kim et al,, 2025), GROOT (Bjorck et al., 2025), and the x-series of models (Black et al., 2024; Intelligence et al., 2025; Pertsch et al., 2025). The VLA training procedure is typically analogous to VLM training. First, model weights are initialized from the respective VI.M backbones (Kim et al., 2024; Black et al,, 2024). Then, the model is supervised with pext-token-prediction tasks on diverse pretraining datasets, spanning across multi-modal web data (Intelligence et al., 2025) such COCO (Chen al,, 2015) and VQAv2 (Goyal et al., 2017), and robotics-ppecific, cross-embont mooos-embodiment data (Khazatsky et al., 2025; O'Neill et al,, 2024). Finally, supervised fine-tuning is conducted on a small set of high-quality teleoperation data collected from the target robot deployment platform performing the target tasks.
Sample-efficient RL with Data and Policy priors Sample and exploration efficiency have been long-standing problem in RL, especially in sparse-reward settings. Recent works have explored leveraging offline data to improve sample efficiency. Offline-to-olinee transfer ( (Vecerik et al., 2017: Nair et al., 2020; Kostrikov et al., 2021; Nakamoto et al, 2024; Zhou et al., 2024b; Li et al., 2025) considers a two-stage pipeline that first initializes policy or critic using pessimism or constrained objective in offline RL Levine et al. (2020) and follows with online fine-tuning phase to have new data collected and alleviate distributional shift; Hybrid RL (Song et al., 2022; 2024; Ball et al,, 2023) considers online RL with access an offline dataset. Given expert demonstration, one can either continuously replay this data to ensure high-value state visitation Ball et al. (2023) or to guide exploration Dong et al. (2025a). Data prior can also guide reset-free real-world learning (Walke et al., 2023; Sharma et al., 2023). Another line of work assumes access to policy prior, such as a pre-trained generalist. Ye et al. (2023); Chen et al. (2025); Jülg et al. (2025) leverage foundation policy to guide RL through an auxiliary behavior regularization objective, Action editing is another efficient way to improve upon the policy prior. ResiP (Ankile et al., 2025) considers learning a residual policy through PPO (Schulman et al., 2017), while EXPO (Dong et et.,, 2025b) considers an off-policy solution and co-trains the base policy during the process. Our work leverages a suboptimal base policy to achieve non-zero success rate for warm-starting exploration, but does not require access to oracle demos or a human expert for further intervention.
VLA post-training The prevailing large-scale recipe for VLA post-training is to pretrain on di- verse, heterogeneous robot data and then fine-tune on task-specific demonstrations (Zhou et al., 2024a; Black al., 2024). For example, Black al. (2024) performs supervised post-training on a carefully curated task-targeted corpus, with per-task coverage ranging from a few to over 100 hours of teleoperation. Because such post-training data are expensive to acquire, the authors suuuur--nddrrcoriin a key limitation
note that most diversity must come from the pretraining of pure SFT: data scarcity and limited coverage at adaptation time. To enable self-improvement, prior work has explored scaling high-quality data via online RL specialists (Ball et al., 2023). How- ever, these pipelines often require substantial human intervention and collect data in a large way agnostic to the generalist's behavior, restricting scalability. Other lines investigate on- on-policy RL for post-training (Lu et al., 2025; Tan et al., 2025), or optimize single-task fine-tuning at the ex- pense of generalization (Chen et al., 2025). Our work jointly targets these limitations by seeking
16

---

Published as a conference paper at ICLR 2026
post-training pipeline that reduces human effort, aligns data collection with the generalist's state distribution, and remains sufficiently sample efficient for real-world systems.
RL with Residual Actor Residual actors are lightweight policies that output adjustments on top
of the base policy's 's action. Compared to the expressive base policy, the residual actor reduces optimization complexity, circumventing problems regarding computation inefficiency or unstable training dynamics of large models or flow-based policies (Mark et al., 2024; Tan et al., 2025).
Residual RL has been leveraged address different challenges in robotics. (Haldar al., 2023) Consider data-efficient imitation refinement to adapt the base policy across robot morphologies and new object configurations; (Yu al., 2023) bridges embodiment or modality gaps, such as human- hand to robot-gripper transfer or tactile/vision modality mismatches; (Ankile et al., 2025) scales to long-horizon or precision tasks, augmenting a frozen action chunking policy with a residual policy to address the distribution shifts and the lack of closed-loop corrective control. They include the residual policy as a core component at test time, and the training process limited to simulation and requires sim2real transfer. (Dong et al,, 2025b) leverages residual actor address the challenge of fine-tuning expressive policies. The small Gaussian policy can be seamlessly optimized using existing off-policy RL maximize the Q function. Then expressive base policy match the edit policy's 's behavior using the imitation objective. In contrast, one of the core motivations for adopting residual RL in this work is enhance sample efficiency. Residual policy with controlled scale enables exploration that starts from the base policy's 's trails. Along with other design choices, our method can be directly adopted for real-world learning on physical robot hardware. Also, we consider augmenting the base policy by distilling expert behavior.
17

---

Published as a conference paper at ICLR 2026
C ALGORITHM
Algorithm PLD with base-policy initialization
Require:
Ts. nTs Qω. Qn', Botfines Bonline
Initialization
Collect successful trajectories of Tii Dossline (Ti, 72: nnJ Initialize online buffer Duntine = Ø
Initialize the critic network Qs: Qei with Cal-QL on foffeine Randomly initialize residual policy network n6
RL training
Freeze Tb, denote F-|ss) as the combined policy (base plus residual) for each RL step do
if collect data then if Warm up step then base model rollout a (-)se(-(8) else
sample action ā T(-ss)
end if
Environment step: r, s', done env.step(a) Add (s, r, s') to buffer Dontine-
end if
Equally sample data from online and offline buffer: Dontine U Doffeine Calculate TD target by bootstrapping
Update Q₂ by equation 2
Update कs by maximizing the SAC target
Polyak update ø' pe' + (1 p) end for #Base policy SFT
For each task, we collect hybrid behavior dataset DSFT:
abases
t < T base
(s:) =
Abase + 0g;
1 2 T base
for each SFT step do update T6 by BC objective. end for Return Tb
D MORE RESULTS
Table 2: LIBERO-90 90 Success rate for To SFT with different dataset.
PLD Data
Base Policy Rollout Data
Human Data
Ratio
Overall SR
Seen/Unseen SR
Seen/Unnsen
Ratio
Overall
Seen/Unseen
0.1
0.314
01244
0.1
0.103
0.056
0.523
0.1
0.272
0.214
0.3
0.470
0,259
0.068
0.198
0.3
0.419
0.240
0.6
0.637
0.872
0.283
0.328
0.506
0.062
0.6
0.611
0.286
0.829
0.745
0.864
0.344
0.423
0.8
0.694
0.803
0.031
0.871
0.815
N/A
1.0
0.488
N/A
18

---

Published as a conference paper at ICLR 2026
Table 3: Evaluate PLD SimplerEnv
Model
WidowX Piek Eggplant
WidowX Pick Carrot
Google Open Drawer
Google Coke Can
Avg
Octo-SFT
43.3
92.5
85.7
71.8
ours
99.3
95.5
96.6
+32.3
+50.6
+6.8
49.8
+24.9
LIERO-Long
Success Rate
1.0
Human data (oracle)
Bootstrapping data
PLD rollout
Figure 8: Short-to-long generalization. Zero-shot evaluation on LIBERO-10 long horizon tasks.
D.1 OCTO SFT FOR SIMPLERENV TASKS
In addition to the in-distribution results on LIBERO, we provide results on SimplerEnv in Table 3. Similar to the training and evaluation protocol on LIBERO, first train task-specific residual RL specialists and fine-tune VLA on PLD data for all four tasks.
D.2 GENERALIZE TO LONG-HORIZON TASK
We assess skill composition on LIBERO-100 by fine-tuning the base VLAA on LIBERO-90 (source) and evaluating zero-shot the held-out LIBERO- 10 long-horizon tasks (target). To construct PLD data, we first train residual RL specialists independently on each LIBERO-90 task, then aggregate their successful rollouts. As shown in Figure 8, the fine-tuning of the data PLD exceeds the tuning of the data from the baseline policy roll-out (self-bootstrapped)) but still falls short of the performance achieved with demonstrations by human experts.
1.2
Postion
Prodioo
vo
6.2 Position
-0.2
Posttion
Postioon
.0.55
x Aanlan
Pesstton
Position
Base Policy Rollout Data
RL Rollout Data
PLD Rollout Data (Ours)
Figure 9: Visualization of different data sources. We plot 50 trajectories for each method (task prompt: "open middle drawer of the middle cabinet"). RL expert data of high quality but lacks diversity and diverges far from base policy behavior, while PLD data aligns better with the base policy and contains diverse recovery behavior.
19

---

Published as a conference paper at ICLR 2026
E IMPLEMENTATION
E. RL BASELINES
To ensure apple-to-apple comparison in Section 4.1, implement these baselines based on the SERL Luo et al. (2024) framework and adapt them to fit in the settings of our study. We provide a detailed explanation of baseline formulation and our implementation.
RLPD RLPD (Ball et al, 2023) proposed a hybrid RL pipeline that leverages offline data to foster learning in challenging sparse reward settings. During training, it equally draws samples from both the online and offline buffer. It also uses LayerNorm to deal with the Q-value blow-ups common when querying OOD actions under high up-to-date (UTD) ratio. We refer to the implementation in the SERL software for both simulation and real deployment.
WSRL In the original paper (Zhou et al,, 2024b), WSRL uses Cal-QL (Nakamoto et al., 2024) to pre-train both the action and critic during the offline phase. For the online phase, it discards offline data and warms up the replay buffer with 50k steps of pre-trained policy rollouts. We did not provide a large dataset that contains diverse behavior on the D4RL (Fu et al., 2020) benchmark. Rather, we use the same procedure as PLD to collect successful trajectories from the base model. We implement WSRL under the SERL framework, as the UTD is no longer fixed to 4. We the WSRL baseline as an ablation study of the warm-up online exploration using the base policy, and offline data retention through hybrid data replay.
JSRL. Jump-start RL (Uchendu et al., 2023) is a meta-algorithm using an existing guide policy to "rolling-in". The key mechanism is to shape the initial-state distribution for the learner: JSRL. repeatedly resets episodes from states that the guide visits (a curriculum from easy/near-god states to harder/far-from-pe states), making difficult tasks learnable with fewer trials. It leverages the guide policy for data collection, without directly imitating its actions. JSRL is agnostic to the underlying RL backbone. practice, we choose SAC to learn the exploration policy. Since JSRL only leverages the policy prior (VLA policy in practice) to warm-up exploration during online interaction, we use as an ablation of the hybrid experience replay mechanism.
Cal-QL Calibrated Q-learning (Nakamoto et al., 2024) addresses the underestimation issue of CQL (Kumar al., 2020), thereby significantly improving fine-tuning performance in the offline-to- online setting. It learns conservative value function that underestimates the value of OOD actions, while ensuring the values are within a reasonable scale. In practice, it bounds the conservative Q function by the value of the behavior policy μ (policy corresponds to the offline dataset D). The modified Q-learning objective is the following:
1 2
min (Ea~D,ovx (ax V*(s)))
E [(Qa(s,a) B ((sss)))))
Where Q is the target Q-value function and the second term corresponds to minimizing TD-error Lil- licrap et al. (2015).
Implicit Q-Learning (IQL). IQL an offline RL method that avoids querying the critic function on ditto--distribution queries while still improving over the behavior policy (Kostrikov et al., 2021). )). The key step is to fit state value V& by expectile regression over the actions of the dataset and then bootstrap Qe toward this value. Let ö(s, a) = Qo(s, a) - (s(s) and define the expectile loss (5) |n f5 < 0} 82 with r E (0.5, 1). IQL alternates
(V) min (s C, Qo a) Vo(s))],
(3)
min E(o.d (8) (r (s')))
(4)
(policy) max E B exp( Qu(o.a) log L (a) $)).
(5)
3
@
which realizes policy improvement without nuttof-distributtion action queries (the policy step duces to advantage-weightee regression) (Kostrikov et al., 2021). In our comparison to Cal- QL (Nakamoto al,, 2024) critic-initialization baseline, we consider simplified version of
20

---

Published as a conference paper at ICLR 2026
Dana
Toee
Trick
Trak 17
1.0
14
0.3
0.6
0.5
0.0
0.0
044
0.4
02
1588
100K
1936
100K
150%
100%
backk 33
Teac 34
Taank 45
Taac
1.9
1.0
1.0
0.8
66
0.8
0.5
0.
0.5
Adter
Figure 10: Base policy ablation. Mean and 95% CIs of rollout performance across 6 seeds.
IQL that directly regresses Qs toward n-step return R. Eitr-1 expectile regression:
11 7 7 + Vii using
min (e
(fr(Qo(st,a)) Blecaa)-oDGG((((s,,, RR)]
Unless otherwise noted, set 17 0.7, value shown effectively propagate high-value signals in the IQL paper.
E.2 DESIGN CHOICES OF PLD
In this section, provide detailed study of design choices that make PLD data efficient and achieve high convergence performance. We evaluate all algorithms on the selected 8 LIBERO-90 tasks.
Sensitivity to Base Model Performance. We evaluate how the quality of the base model influ- ences the efficacy of the RL phase in PLD. Using a subset 8 tasks from LIBERO-90, we fine-tune the initial policy using 1, 10, and 50 demonstrations, yielding three distinct baselines: xn PLD to each baseline. For the experimental setup, we perform a hyperparameter sweep over action scales (0.01, 0.1, 0.5. 1.0] across 6 random seeds, reporting the average performance of the best configuration. As shown in Figure 10, residual RL is highly effective when initialized with competent policy, boosting success rates to 99% provided the base model achieves at least 80% success. Conversely, the method struggles when the base model is too weak; for example, with no demo residual RL fails to converge on out of 8 tasks.
demo
10 demo and 50 demo We then apply Phase
Sensitivity to the initialization horizon We choose task 0-9 from LIBERO-90, change the steps used to initialize the random sample, initiating steps Thuse [0, a:T) rollout the base policy. E [0.0, 0.2, 0.4, 0.6, 0.8]. As a increases, the average episode length of successful trajectories increases, indicating a detour required correct the suboptimal behavior of the base policy. As demonstrated in Figure 13, performance plateaus at a = 0.6 and drops as increases further. This consistent with our analysis that SFT benefits from the data diversity.
Reward shaping We empirically analyze the impact of naive reward shaping. Specifically, we consider a step-wise survival cost as reward bias as in prior works Luo al. (2024). As shown in Section E.2, adding slight reward bias has little impact, but it could increase convergence speed in 2 out of tasks; However, A large bias could significantly hinder performance. For the major results reported in the main paper, we do not apply reward shaping.
Action scale One core component residential policies is the scale of exploration. To avoid un- learning results from diverging too far from the base policy, delta actions are usually scaled down
21

---

Published as a conference paper at ICLR 2026
Bies - -10
Blos -0.1
Slas -0.01
Blas 0.0
Task
Task 2
Task 17
Task 21
1.0-
1.0
1.0
1.0
Success Rate
0.3
0.5
3.5
0.5
0.0
0.0
3.0
0.0
30k
100k
150k
700k
50k
100k
150k
200k
S0k
100k
153k 2006
50c
100k
150k 200k
Task
Task 34
Task 46
Task 47
1.0
1.0
1.0
1.0
Success Rate
0.3
3.5
0.0
3.0
30k
100k
150k
200k
50k:
10ok.
150k
200k
50k
100k
15sk 2036
50k.
109k.
150k 200k
Actor Steps
Actor Steps
Actor Steps
Actor Steps
Figure 11: Reward bias ablation. Mean and 95% CIs of rollout performance across 3 seeds.
Action 0.0001
Action Scale 0.02
Action Scale 0.2 Action Scale
Action Scale
Action Scale 0.001
Action Scale
Task
Task 2
Task 17
Task 21
1.0
1.0
1.0
1.0
Success Rate
0.0
0.0
0.0
0.0
100k
200k
300k
100g
200k
300k
100k
200k
300%
200c
200k
300k
Task 33
Task 34
Task 46
Task 47
1.0
1.01
1.0
1.0
Success Rate
0.5
0.0
0.0
0.0
0.0
100k
230k
300k
0
100k
200k
300k
100k
200k Actor Steps
300K
100c
200k
300k
Actor Steps
Actor Steps
Actor Steps
Figure 12: Action scale ablation Mean and 95% CIs of rollout performance across 3 seeds.
and bounded within range of [-C. (Ankile al., 2025; Dong al., 2025b). Figure 12 compares different residual action scales. Setting & too large at the start can degrade early performance: up- dates deviate excessively from the base policy, inducing unstable exploration, while small & will lead to insufficient exploration and lower asymptotic performance. We argue that needs to be care- fully tuned enable exploration while minimizing performance drop. For single-arm manipulation, suggest & 0.5 a good choice for LIBERO and & = 0.1 for SimplerEnv.
Critic pre-training While warm-start through pre-training the critic is beneficial to asymptotic performance and prevents initial performance drop, the careful selection of the pre-training method
22

---

Published as a conference paper at ICLR 2026
100%
+4. +4.3%
Success Rate
+3.0%
95%
40.5%
Init ratio Ot
150
0.2
Figure 13: Ablation of Probing Horizon. With a measuring the percentage of base policy probing during data collection, we see fine-tune performance plateau at a 0.6. Performance drops mono- tonically as a increases further.
CaloL (a-0.0)
CalOL (a=1.0)
COL la-0.51
(T=0.8)
CalQL (a=0.5)
COL (a=0.0)
COL (a=1.0)
Task
Task 2
Task
Task 21
Success Rate
0.5
0.5
0.5
0.5
0.0
0.0
0.0
0.0
Task
Task 34
Task
Task 47
1.0
Success Rate
0.5
0.5
0.5
0.0
0.0
0.0
100k 130k 200k
100k
150K
200k
50k
100k
150k
100k
150k
200k
Actor Steps
Actor Steps
Actor Steps
Actor Steps
Figure 14: Offline pre-training ablation. seeds.
Mean and 95% Cls of rollout performance across
could be important as well. We compare using CQL, Cal-QL, and IQL to the pre-training method. We consider using only trajectories of successful trials of the base policy, while the standard offline RI. benchmark tends to have far larger data volume (Fu et al., 2020). In Figure 14, online performance using the Cal-QL pre-trained critic consistently better and is robust to the conserva- tive coefficients a. CQL demonstrates the worst performance with severe forgetting issue, which aligns with the previous study (Nakamoto et al., 2024).
Update frequency the SERL pipeline, data collection and policy learning run asynchronously and periodically exchange network parameters and online data. We ablate the update frequency- the number of gradient steps performed by the leamer between parameter synchronizations with the data-collection aitor-sweeppng from 1 to 500. As shown in Figure 15, overall performance is largely insensitive to this hyperparameter, indicating robustness across a wide range of synchroniza- tion cadences.
23

---

Published as a conference paper at ICLR 2026
Update Freq
Update Freq
Update Freq Update Freq 100
Update freq
Update Freq 5
Update Freq
Task 1
Task 2
Task 17
1.0
1.0
Success Rate
0.5
0.5
0.5
0.0
0.0
0.0
100k
150k
200k
50k
100k
150k
200k
100k
150k
200k
Task 21
Task 33
Task 46
1.0
1.0
Success Rate
0.5
0.5
0.5
0.0
0.0
0.0
100k
150K
50k
100k
150K
200k
100k
150K 200k
Actor Steps
Actor Steps
Actor Steps
Figure 15: Update frequency ablation. Mean and 95% Cls of rollout performance across seeds.
OTF 1
OTF -10
OTF
OTF 5
OTF 20
Task
Task 2
Task
Task 21
1.0
Success Rate
0.3
0.5
0.3
0.5
150k
150k
Task
Task 34
Task
Task
1.0
1.0
1.0
Success Rate
0.5
0.5
0.5
0.5
Actor Steps
Actor Steps
Actor Steps
Actor Steps
Figure 16: On-the-fly Policy Ablation. Mean and 95% Cls of rollout performance across seeds.
On-the-Fly Policy On-the-fly (OTF) policy is introduced in (Dong et al., 2025b) to more effec- tively maximize the value function. It samples multiple actions and backs up the maximum Q value during TD learning. We adopt OTF to PLD while only sampling multiple actions from the residual policy n& and conditioned on a fixed base action. We compare different sample sizes in Figure 16. We found that OTF can improve sample efficiency, and a larger sample size (> 20) shows significant performance gain. But empirically, the asymptotic performance will eventually be similar. We use OTF= by default.
JSRL We further provide results, including JSRL Uchendu et al. (2023) in Figure 17. We modify the original implementation by opting for linear scheduler. JSRL demonstrates high data efficiency in general, but could fail to converge on some tasks. While PLD can reliably provide solutions for all tasks.
24

---

Published as a conference paper at ICLR 2026
(Ours)
RrJ5RL
Task
Task
Task
Task 21
Success Rate
0.5
0.0
0.0
0.0
0.0
100k
200k
100k
200k
100k
200k
120k
200k
Task
Task
Success Rate
0.5
0.5
0.5
0.5
0.0
0.0
0.0
100k
100k
200k
100k
Actor Steps
Actor Steps
Actor Steps
Actor Steps
Figure 17: Compared with JSRL. Mean and 95% Cls of rollout performance across 3 seeds.
F IMPLEMENTATION DETAILS
F.1 RL ALGORITHM
To ensure apples-to-apples comparisons, all baselines in Section 4.1 and Section E.2 use the same network architecture- an 3-layer MLP Gaussian policy and Clipped Double Q-networks (CDQ) Fu- jimoto al. (2018) with LayerNorm (Ba al., 2016). Both actor and critic use pre-trained ResNetV1-10 encoder to extract visual information. We present a detailed hyperparameter setting in Table 4.
F.2 COMPUTATIONAL COST ANALYSIS
We analyze the computational resources required for the Residual RL phase (Phase 1) of PLD, conducted on NVIDIA L40 GPUs for simulation and an RTX 4090 for real-world experiments. A key advantage of PLD its resource efficiency: since the large VLA base policy remains frozen and we only optimize a lightweight residual MIP, the GPU memory footprint is significantly reduced compared to full fine-tuning. As shown in Table 5, training occupies a peak of only ~5GB VRAM per task. To further optimize GPU usage, we offload the experience replay buffer to System RAM (up to 100GB per task). This low GPU footprint enables linear scalability for multi-task learning: for example, we successfully parallelized the LIBERO-99 experiment by distributing 90 tasks across a cluster node with 90 L40 GPUs and 10TB of CPU memory.
F.3 SFT
We employ LoRA (Hu et al., 2021)
(221)ep to fine-tune our VLA base models (Open and mp) efficiently. All SFT experiments are conducted on node with 8x NVIDIA L40 GPUs. We use a LoRA rank of and apply the default hyperparameters provided by the respective open-source codebases for both =n and VLen
F.4 INFERENCE AND ACTION CHUNKING.
We adopt my model and Jax (Bradbury et al., 2018) implementation for real-world experiments. To handle
control real-world tasks, we utilize temporal action chunking.
For standard Franka manipulation tasks, we chunk size of k 20 and execution step of H = (inferencing every steps to predict the next 20).
For the YAM arm GPU insertion task, we increase the temporal context, using a chunk size of k 26 and execution step of H 15.
25

---

Published as a conference paper at ICLR 2026
Table 4: RL. hyperparameter setrings. We share the same setting across all tasks.
Hyperparameter
Value
Training
Batch
Buffer capacity
250000
Discount factor (7)
0.99
Gradient clipping norm
1.0
Learning rate
x 10"4
Optimizer
AdamW
Reward bias
Residual Policy
Target entropy
Initial temperature (r)
1.0
Action scale (E)
0.5
Critic
Q functions ensemble
2
Target update rate
0.005
Architecture
Visual Encoder
ResNety
Hidden layer dimension
256
Latent space dimension
256
Q function dropout
0.0
Activation
Tanh
Normalization
LayerNorm
Table 5: Computational Cost Analysis. Resources reported are per single task.
Setting
Hardware
Peak VRAM
System RAM
Training Time
LIBERO (Sim)
NVIDIA L40
~ GB
100 GB
4-6 Hours
Real World
NVIDIA RTX 4090
~ GB
1-00 GB
2-8 Hours
G REAL-WORLD EXPERIMENTS
G.1 EXPERIMENT SETUP
We deploy PLD on 7-DoF Franka Emika Panda with end-effector delta pose control at 20 Hz. The robot is equipped with one wrist-mounted camera, one side-view camera, and proprioceptive sensing as inputs. For each task, we pretrain an independent binary reward classifier by collecting small-scale dataset of success and failure states. The model structure follows the setup in (Luo et al., 2025), which uses a pretrained ResNet-10 and 3-layer MLP model. We ensure the trained classifier using augmented false positive samples until it achieves 99% success rate for each task. Due to the 3D printed desk, we don't need to reset the environment for the pick-cube task. PLD performs auto-reset, residual RI. training, and SFT automatically without human supervision. For
26

---

Published as a conference paper at ICLR 2026
Q: Pick up the blue cube
Q: Plug in the yellow peg
Figure 18: Franka Panda real-world setup for manipulation tasks.
the peg-insertion task (depicted in Figure 19), humans need randomly move the position of the hole to increase diversity.
G.2 GENERALIZATION PERFORMANCE
We perform SFT To on Pick Up Blue Cube (Clean Env) and Peg Insertion data, and evaluate the fine-tuned policy on Pick Up Blue Cube (Cluttered Env) and Pick Up Red Cube (Cluttered Env) tasks. The results in Table show that VLA SFT on PLD data achieves hetter generalization performance compared to human teleoperation data.
Table 6: Comparison of PLD vs Human Data on real-world tasks (success rate).
SR Dataset
PLD Data
Human Data
Pick Up Blue cube (cluttered Env)
28/30 (93.3%)
12/30 (40.0%)
Pick Up Red cube (cluttered Env)
20/30 (66.7%)
10/30 (33.3%)
Peg Insertion
30/30 (100.0%)
30/30 (100.0%)
27

---

Published as a conference paper at ICLR 2026
SFT Model to perform the following tasks
Pick Up the Red Cube
Up Blue Cube
Plug in the Peg
Figure 19: Real-worid Generalization Performance. We evaluate one model's multi-task perfor-
mance three language-conditioned manipulation tasks, including pick-and-place and peg inser- tion.
28


Reviews:
[{"review_id": "fggJQUI9FU", "reviewer": "Reviewer_F1ZL", "summary": "This paper proposes a three-stage self-improving framework named PLD (Probe, Learn, Distill) to provide high-quality data to finetune Vision-Language-Action (VLA) models. Its core idea is to freeze the VLA base model and train lightweight residual reinforcement learning (RL) policies to probe the failure regions of the base model; then adopt a hybrid data collection scheme of to align residual interventions with the deployment distribution of the base policy; and finally distill the collected high-quality trajectories back into the base model via SFT. Experiments validate the effectiveness of PLD on the LIBERO and SimplerEnv simulation benchmarks as well as real-world robotic tasks.", "questions": "The paper states that PLD supports multi-task training. In multi-task scenarios, are residual policies trained independently for each task, or do they share partial parameters to improve efficiency? If trained independently, will PLD’s storage costs (e.g., parameters of residual policies for each task) increase significantly as the number of tasks grows? Are there any parameter sharing or model compression schemes available?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper addresses two core pain points of VLA models in SFT: \"data dependence\" and \"distribution mismatch\". PLD combines residual RL with distribution-aware data collection. It avoids the high resource consumption of directly fine-tuning large VLA models and solves the poor generalization issue of pure RL expert data, presenting a novel and logically consistent approach. \n2. The effectiveness of PLD is well demonstrated through multi-environment (LIBERO, SimplerEnv, real robots), multi-model ($\\pi_0$, OpenVLA), and multi-baseline comparisons. Additionally, PLD demonstrates better generalization ability than human data in zero-shot and few-shot experimental tasks. \n3. Quantitative analyses are conducted on the impact of key parameters, such as the action scale of the residual policy, the probing ratio of the hybrid rollout strategy, and the pre-training method for the Q-network. This provides reusable parameter selection guidelines for subsequent research.", "weaknesses": "1.  Direct comparisons with recent residual RL methods (e.g., EXPO[1], ResiP[2]) are missing. It is recommended to supplement comparative experiments or discussions on this aspect. \n2. Current tasks (e.g., object grasping, peg insertion) are relatively simple. It is suggested to validate the method on more complex long-horizon tasks such as cloth folding. \n3. The paper mentions that PLD trains lightweight residual policies but does not discuss costs such as training time and GPU memory usage. It is recommended to supplement a computational cost table to clarify PLD’s advantages in \"resource efficiency\", especially its scalability in multi-task training scenarios.\n\n[1] Dong P, Li Q, Sadigh D, et al. Expo: Stable reinforcement learning with expressive policies[J]. arXiv preprint, 2025.\n\n[2] Ankile L, Simeonov A, Shenfeld I, et al. From imitation to refinement-residual rl for precise assembly[C]//ICRA, 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "npdxaRErRb", "reviewer": "Reviewer_wXQ9", "summary": "A paper for enriching data in finetuning VLA using residual RL. This framework utilize residual RL to generate more diverse and successful trajectory for tuning VLA. To evaluate the performance, this paper conduct comprehensive experiments on the libero benchmark and two real-world experiments.", "questions": "1. Residual policy can be different policy architecture other than base policy. In simulator, the residual policy can be simple state-based policy, which might be trained more efficiently. Why still using VLA architecture?\n\n2. For the policy warmstart with expert data, the residual action will always be 0 action. Could you give some more analysis on why this kind of warm start can work? It only warm start the critic or also warm start the residual actor?\n\n3. For those baselins like WSRL, the policies are learned from scratch but not doing residual rl base on the pretrained policy, right? I concern that it might not be a fair comparison.\n\n4. WSRL only uses online data but not offline data after warm starting the Q and show that it's better than off-policy learning. Since this paper utilize similar strategy to warmstart Q using only base policy, why still use offline data? \n\n5. Why don't choose offline residual RL [2] but online RL?\n\n6. I might not an expert in VLM tuning. Since this method still need to do SFT for tuning the VLA, why it requires less computation resource than doing online RL for tuning the VLA directly?\n\n[1]. Xu et al., Compliant Residual DAgger: Improving Real-World Contact-Rich Manipulation with Human Corrections, NeurIPS 2025", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Using residual RL to enrich the data for VLA finetuning is promising.\n\n2. The proposed pipeline is well-designed, which includes base policy probing, warm-start, success classifier, and SFT for policy distillation.\n\n3. The experiments in libero benchmark and real-world RL is comprehensive.", "weaknesses": "1. Using residual RL to finetune and resolve different gaps is not a pretty novel idea, and this paper didn't disscuss those related work accross different areas in details. For example, residual RL has been used in data efficient learning [1], bridge human2robot embodiment gap [2, 3], real2sim2real transferring for data scaling [4], peg insertion [2, 5], and dexterous manipulations [3, 4]. Disscussing those related work and highlight the contribution might be important.\n\n2. Only binary reward has been used for reward shaping, where it will limit the learning efficiency. Althought it's hard to design dense reward for visual policy or in the real world, there are some related work using VLM [6], generated videos [7], and object-centric representations [8] to do reward shaping.\n\n3. Compared with offline RL or SFT, real-world RL required more human effort to reset and might have unexpected behaviors for some challenging tasks.\n\n4. The paper shows that the control frequency is 20 Hz. For pi0, it can reach high control frequency because of its open-loop action chunking, and its inference frequency is low. Is the residual RL tuning each action of the action chunking and resulting in a close-loop control? However, after SFT, the VLA policy still should be low frequency inference for open-loop action chunking but not close-loop control? For more challenging task, it might require close-loop policy.\n\n5. In the previous real-world paper HIL-SERL [9], they point out that using residual RL to refine a bad BC base policy will result in bad performance. Could you show more insight about this problem? To this end, together with the concern 4, I would be worried about this method's performance in more challenging task, where the policy requires close-loop control, and the VLA base policy might have bad performance.\n\n6. No figure to show the entire pipeline.\n\n[1]. Haldar et al., Teach a Robot to FISH: Versatile Imitation from One Minute of Demonstrations RSS 2023\n\n[2]. Yu et al., MimicTouch: Leveraging Multi-modal Human Tactile Demonstrations for Contact-rich Manipulation, CoRL 2024\n\n[3]. Guzey et al., HuDOR: Bridging the Human to Robot Dexterity Gap through Object-Oriented Rewards, ICRA 2025\n\n[4]. Wan et al., LodeStar Icon LodeStar: Long-horizon Dexterity via Synthetic Data Augmentation from Human Demonstrations, CoRL 2025\n\n[5]. Ankile et al., From Imitation to Refinement – Residual RL for Precise Visual Assembly, ICRA 2025\n\n[6]. Wang et al., RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedback, ICML 2024\n\n[7]. Huang et al., Diffusion Reward: Learning Rewards via Conditional Video Diffusion, ECCV 2024\n\n[8]. Yu et al., GenFlowRL: Shaping Rewards with Generative Object-Centric Flow in Visual Reinforcement Learning, ICCV 2025\n\n[9]. Luo et al., Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning, Science Robotics", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Tju4w8PF3e", "reviewer": "Reviewer_yzFS", "summary": "This paper proposes PLD, a three-stage post-training framework for vision-language-action (VLA) models that reduces dependence on human demonstrations. Stage 1 freezes the VLA and trains lightweight residual RL specialists to improve recovery from failure states. Stage 2 uses a hybrid rollout that starts with the base policy and then hands control to the residual policy, collecting realistic recovery trajectories. Stage 3 distills this data back into the base model via supervised fine-tuning. Experiments on LIBERO, SimplerEnv, and a real Franka arm show strong success rates and generalization, demonstrating a scalable path to improving robotic foundation models with minimal human supervision.", "questions": "- The real-world results depend on a set of 200 teleoperated trajectories to initialize the base policy. How does the performance of PLD scale with fewer (or noisier) demonstrations, and would the method still succeed if starting from a weaker base policy or no demonstrations at all?\n\n- PLD trains residual experts for multiple tasks, and then distills them back into the base VLA model. Could the authors elaborate on why this distillation step (collect trajectories) is necessary compared to directly deploying the VLA + the residual experts? What specific limitations does PLD address that simple residual policies cannot?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Presents a novel and practical three-stage framework that combines RL and SFT for post-training VLAs without relying on costly human demonstrations.\n\n- Effectively addresses scalability challenges in robot learning by improving data diversity (failure recovery trajectories) and reducing dependence on human demonstration data.\n\n- Provides robust evidence through extensive experiments on LIBERO, SimplerEnv, real-world Franka setups, and systematic ablation studies.\n\n- The paper is well-written and clearly organized, making the methodology and key insights easy to understand.\n\n- Clearly identifies the core challenges preventing direct transfer of LLM post-training successes to VLA models, providing both conceptual and empirical justification for the proposed solution.", "weaknesses": "- The real-world experiments focus on short-horizon tabletop manipulation where the base policy is already strong. It remains unclear how PLD performs when initial success rates are low or when recovery requires multi-step planning. Including a few such cases or a discussion of failure modes would better support claims of scalability.\n\n- When applying PLD in the real world, the trade-off between safe operation and sufficient state coverage raises concerns about the method’s ability to handle more challenging, long-horizon, or highly exploratory tasks where recovery behaviors are harder to discover.\n\n- The approach trains multiple task-specific residual RL actors, but the paper does not clarify how many specialists are required as task counts grow, or whether skills can be shared across tasks with related dynamics. Providing guidelines or experiments on task grouping would strengthen the method’s practicality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UkmN95OgkB", "reviewer": "Reviewer_BWyM", "summary": "The paper introduces PLD (Probe–Learn–Distill), a three-stage self-improvement framework for robotic foundation models. PLD enables a VLA model to autonomously refine itself without relying on human demonstrations. It first learns task-specific residual policies via reinforcement learning, then uses a mixed strategy to collect diverse, high-quality data, and finally distills this knowledge back into the base model through supervised fine-tuning. The method achieves near-perfect success rates in both simulation (LIBERO, SimplerEnv) and real-world experiments on the Franka Panda arm, demonstrating strong efficiency, generalization, and practicality.", "questions": "1. How would PLD handle long-horizon or multi-stage tasks requiring sequential planning?\n2. Could the authors provide ablation studies like removing failure/recovery data from PLD?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "PLD provides a practical approach for VLA models to achieve self-improvement without additional human demonstrations. The data produced by PLD not only aligns with the behavioral distribution of the base model, but also includes recovery behaviors that are typically scarce in human demonstration datasets. Through experiments in both simulated environments and real-world robotic settings, the paper shows that fine-tuning with such self-generated data can effectively enhance the success rate, robustness, and generalization ability of VLA models. This offers a promising solution to the long-standing challenge of high data collection costs in robot learning.", "weaknesses": "1. The evaluation focuses mainly on short-horizon, simple manipulation tasks. The approach has not been tested on long-horizon or temporally dependent tasks.\n2. The human data only includes successful trajectories, while PLD’s automatically collected data includes failure and recovery cases, which clearly increase diversity. This makes it hard to determine whether the improvement comes from PLD’s three-stage structure or simply from having more varied data. Including ablation studies that remove failure/recovery data from PLD would make the comparison more fair and convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)