RFS: Reinforcement learning with Residual flow steering for dexterous manipulation
Authors: Entong Su
Tyler Westenbroek
Anusha Nagabandi
Abhishek Gupta
Venue: ICLR 2026 Poster (2026)
Keywords: Robotics
reinforcement learning
sim-to-real
Primary area: applications to robotics, autonomy, planning

Abstract: Imitation learning has emerged as an effective approach for bootstrapping sequential decision-making in robotics, achieving strong performance even in high-dimensional dexterous manipulation tasks. Recent behavior cloning methods further leverage expressive generative models, such as diffusion models and flow matching, to represent multimodal action distributions.
However, policies pretrained in this manner often exhibit limited generalization and require additional fine-tuning to achieve robust performance at deployment time. Such adaptation must preserve the global exploration benefits of pretraining while enabling rapid correction of local execution errors. We propose Residual Flow Steering (RFS), a data-efficient reinforcement learning framework for adapting pretrained generative policies. RFS steers a pretrained flow-matching policy by jointly optimizing a residual action and a latent noise distribution, enabling complementary forms of exploration: local refinement through residual corrections and global exploration through latent-space modulation. This design allows efficient adaptation while retaining the expressive structure of the pretrained policy.
We demonstrate the effectiveness of RFS on dexterous manipulation tasks, showing efficient fine-tuning both in simulation and in real-world settings when adapting pretrained base policies. Project website: https://weirdlabuw.github.io/rfs/

Published as a conference paper at ICLR 2026
RFS: REINFORCEMENT LEARNING WITH RESIDUAL FLOW STEERING FOR DEXTEROUS MANIPULATION
Entong Su', Tyler Westenbroek' Anusha Nagabandi", Abhishek Gupta'
University Washington, Paul G. Allen School of Computer Science & Engineering 2 Amazonn Frontier AI & Robotics
ABSTRACT
Imitation learning has emerged as an effective approach for bootstrapping se- quential decision-making in robotics, achieving strong performance even in high- dimensional dexterous manipulation tasks. Recent behavior cloning methods fur- ther leverage expressive generative models, such diffusion models and flow matching, to represent multimodal action distributions. However, policies pre- trained in this manner often exhibit limited generalization and require additional fine-tuning to achieve robust performance at deployment time. Such adaptation must preserve the global exploration benefits of pretraining while enabling rapid correction of local execution errors. We propose Residual Flow Steering (RFS), data-efficient reinforcement learning framework for adapting pretrained genera- tive policies. RFS steers pretrained flow-matching policy by jointly optimizing a residual action and a latent noise distribution, enabling complementary forms of exploration: local refinement through residual corrections and global explo- ration through latent-space modulation. This design allows efficient adaptation while retaining the expressive structure of the pretrained policy. We demonstrate the effectiveness of RFS on dexterous manipulation tasks, showing efficient fine- tuning both in simulation and in real- world settings when adapting pretrained base policies. Project website: https//weerrllabuw wwwp:/lwwwww.ggbbboo.ooooo github.io/ffs/
INTRODUCTION
Imitation learning from human demonstrations has proven effective for learning berfprerforinn policies robotics and sequential decision making (Zhao et al., [023] 2025| Chi et al.; 2023) Hus- sein et al., 2017; Ho and Ermon 2016). Recent progress has been driven by generative imitation
learning, which leverages expressive generative models such as diffusion models (Ho et al., 2020)
and flow models (Lipman et al. 2022 capture the distribution of human demonstrations. These approaches are particularly well suited to the inherent multimodality of human behavior (Chi et al., 2023 Black al. 2024). enabling robust empirical scaling complex manipulation tasks (Team) jet al. 2025). However, imitation learning alone often fails to generalize to all test-time scenar- (Team et al.) 2025). leaving substantial headroom for policy improvement through test-time fine-tuning toward production-level performance.
While supervised fine-tuning (Ouyang et al. 2022 Black et al. 2024 has proven effective for decision-making systems, it depends on high-quality, curated expert data. Instead, we seek methods that avoid this dependency by leveraging cheaper, suboptimal, or self-collected data. Reinforcement learning (RL) and offline reinforcement learning (offline RL) (Levine et al.) (2020) provide princi- pled framework for behavior improvement, as agents optimize reward objectives rather than relying on expert demonstrations. Although many methods have been proposed to combine imitation learn- ing pretraining with reinforcement learning fine-tuning (Rajeswaran et al., 201 Sa Hu et al.) 2024 Nair et al. 2018 Nakamoto et al.) 2024). these approaches typically rely on closed-form likeli- hoods (Lillicrap et al. 2015) and reparameterization (Wang et al. 2019). making them incompatible with expressive generative architectures such diffusion and flow models. Moreover, key chal- lenge in this setting is balancing adaptation to new tasks with the preservation of knowledge acquired during pretraining.

---

Published as a conference paper at ICLR 2026
We begin by viewing residual RL and diffusion steering instances of a broader class of policy- modulation methods, which adapt pretrained generative imitation-leariin policies by modulating their imputs or outputs rather than updating model parameters. Residual policy learning (Ankile)
et al. 2024b) enables local action refinements but struggles to induce global behavioral changes, whereas diffusion steering (Wagenmaker et al. 2025) allows global modulation but offers limited fine-grained control in off-distribution states. These complementary limitations motivate a unified Drspective.Motivatee by this perspective, we introduce Residual Flow Steering (RFS), policy- modulation framework for adapting pretrained generative policies using reinforcement learning. RFS jointly modulates the initial latent noise and applies an affine residual correction to the pol- icy output, enabling global semantic adaptation alongside local dexterous refinement.
We instantiate RFS for multi-finger manipulation, developing controllers that achieve robust multi- object grasping in simulation and transfer effectively to the real world. These controllers can be further improved using limited human demonstrations, enabling data-efficient adaptation to real- world dynamies while preserving precision and dexterity.The contributions of this work are: (I) We introduce residual flow steering (RFS), an efficient framework for adapting pretrained flow- matching policies. (2) RFS enables effective simulation pretraining from human demonstrations, capturing complex multi-finger coordination behaviors. (3) RFS supports real-world fine-tuning via offline reinforcement learning, outperforming bootstrapped imitation-learnine baselines. (4) We conduct extensive ablations to identify the key components of RFS in both simulation and real-world settings.
2 RELATED WORK
Reinforcement Learning with Demonstrations. Imitation learning (IL) is commonly used to boot- strap reinforcement learning (RL) for complex robotic tasks by constraining early exploration and providing strong initial behaviors. Prior work incorporates demonstrations via imitation losses,
play buffers, or reset strategies (Nair et al. 2018 Qin et al. [022 Xiao et al. 2222a Hu et al. 2024)) while other approaches combine demonstration data with online rollouts improve exploration effi
ciency (Ball et al., 2023 Hester et al.) 2017 Ankile et al., 2025 Song et al.] 2023). Demonstration-
guided RL has also been applied to high-DoF dexterous manipulation (Rajeswaran et al., 2018b)
Haldar et al. [2223 Wang et al. 2024). demonstrating that while demonstrations accelerate learning, they remain insuffcient for achieving the fine ane-grained precision required in challenging dexterous
tasks.
Offine-to-online fine-tuning. Offline-to-onlin fine-tuning is widely used improve sample effi ciency in robotic RL: policy is first pretrained on demonstrations and then refined with limited in- teraction. Lightweight offline-to-onlne methods (Kostrikov et al. 2021 Nair al. 2021 Nakamoto et al. 2024) are popular; however, because they operate directly in action space, they often struggle with distribution shift and the fine-grained corrections required for dexterous control. Recent work applies RL to diffusion policies (Ren al. 2024bb Fang et al. 2025 Davey and Zheng 2025) and
how-matching policies (Zhagg et al., 20256 McAllister et al. Pork]Parkk Park et al. 2025); however, the
iterative refinement structure of generative models and the high dimensionality of action-chunked policies make stable RL fine-tuning particularly challenging and prone to instability.
Policy Modulation. Policy modulation methods adapt pretrained policies by modifying either their outputs or their inputs. Output modulation adjusts a base policy's 's actions using residual poli- cies (Ankile al. [2024a] Zeng al. 22200] Silver et al. [2019] Alakuijala
al.) [022]] Davchev cet al. 2022]] Carvalho et al. 2022; Yuan et al. 22224, Xiao et al. 2025b). enabling effective local corrections but unable induce global behavioral changes when the base policy is limited. Input modulation methods, such as DSRL (Wagenmaker et al.) (025), instead adjust the latent noise of diffusion policies to modify high-level behavior; however, they remain constrained to the demon- stration manifold, limiting off- exploration and fine-grained refinement.
3 BACKGROUND
3.1 FLOW MATCHING
Flow matching (Lipman et al. 2023 Liu 2022) is a generative modeling framework that learns a time-dependent velocity field that transports base distribution Po to target distribution Pi. A
2

---

Published as a conference paper at ICLR 2026
sample evolves according the ODE dag ey (2t, t), and the objective is to align the pushforward of Po with P1 while avoiding ODE backpropagation and likelihood computation by supervising conditional paths rather than full marginals (Lipman et al. 2023). Training draws 210 Po x P1
samples t. 1~2 (0, 1]. forms the interpolation Zt = -1-14 1)zo + Laı, and regresses the predicted velocity toward the straight- path target 8* (2t, () x1 Zip
( = Ezo -o, 21 p1m e- 0o (a), () - v" t)
Given trained velocity field no, samples from p1 obtained via Euler integration (Lipman al. 2023 yielding straightforward sampling path:
x+1 .* + At vo(x", tx), x. ~ Pu, xx 21.
(2)
The framework becomes conditional by learning uo ( t, with an external variable In
setting, an expert dataset DD {si, ax) is used to train a conditional policy velocity field. The corresponding training objective (Eq. 3) learns conditional velocity field vo ());
[(0) Eao (.a))_De=<((000 (-..j-D.ttt/..o]llt(((aa. (aa,,,
ao)1? az
t)ao ta.
Integrating the learned field (a s) (Eq. 2 produces actions from the policy, enabling expres- sive multimodal behavior while providing natural control points for downstream adaptation. For notational convenience, we use Ta and to interchangeably for flow-based policies.
3.2 REINFORCHMENT LEARNING
For adaptation with reinforcement learning (RL), we consider Markov decision process (MDP) M = (S, .A, 'T ,,, 7),, where and A denote the state and action spaces, T(s' 8, a.) is the transition kernel, is the reward function, and y [0, 1) is the discount factor. The goal is learn stochastic policy Th (a s) that maximizes the expected discounted return (0) Er-p, [S:. r'r(st, st, az)]. The choice RL algorithm
central our method; instead, focus on two policy-modulation paradigms that underlie our approach.
Output modulation. central example residual reinforcement learning. in which small residual
policy T.. (a a is trained via RL to correct errors in a pretrained base policy we se(a | s) (Ankile al. 2024b Silver et al. 2019). Action execution is modified via an additive correction, as shown
in
Input modulation. complementary line of work modulates the imputs generative policies. Diffusion steering (Wagenmaker et al. (2025) adapts pretrained diffusion models by learning a policy over the initial latent noise rather than sampling it from a fixed Gaussian. Action generation performed by integrating the flow-matching dynamics in Eq. 2 which maps initial latent noise ao to an action via a denoising process. Diffusion steering optimizes Eq, 5 by adjusting ag to steer the generative model's behavior through this denoising process.
8C
max E eowpol(s), s'mp(e')s,a) tas a NT ap (a/s), 100 (alss)
Σ
rr((st, de)
max E
Pompols), ('mp(s((ssa) Ougipa (Rofs), anppolals.ap)
Σ
oc
)rt(())
(5)
(4)
4 RESIDUAL FLOW STEERING: METHOD AND APPLICATIONS
We propose Residual Flow Steering (RFS), unified framework that combines latent-noise steering for global adaptation with residual actions for precise local corrections on top of a pretrained base policy. This formulation enables efficient learning in high-dexterity tasks and supports robust sim- to-real transfer.
4.1 UNIFIED POLICY MODULATION FRAMEWORK
4.1.1 POLICY MODULATION ALGORITHMS FOR ADAPTING PRETRAINED GENERATIVE MODELS
3

---

Published as a conference paper at ICLR 2026
Since residual RL and DSRL rep- resent output and input modula- tion, respectively, it is natural to view them under a unified policy- modulation perspective. To make
Wo
ai
06 + dg
RFS
TFM
Env
Wo, TRFS
Qyy
G Or
this connection explicit, we examine the structure of residual RL (Ankile et al. 2024b Silver et al.) 2019) and lotent-noise steering (Wagenmaker al. 2025 Du and Song) 20255
Demo Daia Manifold
Residual Action Space
Optimall Action Basin
Residual Flow Steering (RFS). Given state s, the RFS policy rREs outputs a latent flow variable wi and a residual action a,, which jointly steer pretrained base pol- icy 7FM to produce the final action as + Or. RFS enables
Figure 1:
given a pretrained generative pol-
icy (0,, s) and a denoising-basee
both global mode shifting and fine-grained residual correc- tion, allowing the policy to expand beyond the demonstra- tion data manifold.
sampling function a ~ De do, vg)
with do (N, Z). We place the two objectives side by side:
max E
(s)
Σ Ert(s)a))) 1=0
E sor-po(s ppp(s11s.0
Σ En'r(s(,ar) 1=
d'mppee (s,0)
a= (s,00000
tmg, a,
Gp (
ag = Des.(,,..
(6)
Both objectives modulate the behavior the base policy Up fi-hither by adjusting the latent noise or applying an affine action correction. More generally, policy modulation for generative pol- icy (a i, 8) optimizes parametric input (g) or output (f) transformations without modifying the underlying parameters 0 as formalized in Eq.
4.1.2 RESIDUAL FLOW STEERING FOR JOINT LOCAL AND GLOBAL ADAPTATION
We instantiate the modulation functions f and Residual Flow Steering (RFS). which unifies residual RL. for output modulation (Silver et al., 2019) Zhang et al. 2020) with diffusion-steering
RI for input modulation (Wagenmaker al. (025). Input modulation adjusts the latent noise of fow-matching policy for global bchavioral variation via ag = TH s), while output modulation applies local refinements through the affine correction RFS combines these mechanisms through a unified modulation policy TRES( a, s), jointly producing ag for global steering and a, for precise local adjustments. Here, ap shapes the generative model's 's overall behavior, while a, compensates for off- off-manifold requirements or imperfections in base policy Vg. The resulting objective in Eq. is compatible with standard algorithms, and the next subsection instantiates RFS for dexterous manipulation in simulation and the real world.
Des(s, ags 0&) + a, with a, TT s).
E. pgg) s'mp(s'|asn [s.a] a= (a).s); ag (s) a,--De((s,oo
Σ ('r(se,
max E Fans
Rg po)pp(s), s'mp(ss Eapps'lannn [x.a)
Σ Ertrsshaa))
t==
ay wiass (no,a.)s) as Des(s,0g.0))
Lt=
a=a,
4.2 INSTANTIATING RFS FOR DEXTEROUS MANIPULATION
We instantiate RFS for dexterous manipulation through a simulation-to-ret pipeline that enables online fine-tuning simulation and offline fine-tuning using limited real-world data, applied to state- and vision-basce flow-matching policies, respectively, enabling broad exploration and data- efficient adaptation (Fig. 2). We demonstrate the approach across several dexterous manipulation tasks.
4.2. SIMULATION TRAINING: DATA GENERATION AND ONLINE RL
Dexterous manipulation poses key challenge in effective exploration digeedimensional action spaces. We collect a small set VR-teleoperated demonstrations entirely in simulation to train a base generative policy eo using flow-matching objective (Lipman al. 2022). Although imperfect
4

---

Published as a conference paper at ICLR 2026
VR Data Collection and Buse Policy Training
Corrective Action Collection
Pilter Rewen
Emiaabl Gree
Base Policy
Mipllceeeet
Human Intervention
π S Online Finetuning in Simualtion and Distillation
Grasp
Zeno-soot Transter
Mesidual Comecttin
TRFS Deployment After Offline Finetuning
Grap
Pick & Paot
Disillation
Pick & Plaze
Packing
Puch-t--Gaap
Polacy
Figure 2: Overview the sim-to-real Residual Flow Steering (RFS) pipeline. (1) VR teleop- eration used to collect demonstrations across multiple manipulation tasks to train task-specific flow-matching base policies.
(2) In simulation, the RFS policy TRFS is fine-tuned on top of each base policy and distilled into task-specific visuomotor policies to improve sim-to-real transfer. (3) During zero-shot real-world deployment, human corrective actions correct execution failures such unstable grasps and misplacement. (4) These corrected transitions used for offline fine-tuning of tris on a Franka-Leap Hand system, improving real-world grasping and pick-and-place perfor- mance.
in isolation, this policy captures coordinated hand-arm behaviors and provides a strong initialization for reinforcement learning. We then train the RFS policy TRES with online PPO to optimize grasp success and stability.
The trained RFS policy is used to generate additional simulation demonstrations with access to privileged low-level state information s (e.g., object poses and velocities), which are distilled into conditioned visuomotor policies via a standard student -eanhe framework (Chen et al.,
2021.
4.2.2 REAL-WORLD FINE-TUNING: OFFLINE RFS
Direct transfer of the distilled visuomotor policy uo (a Ope- Spro: t) to the real world often fails under novel objects varied initial conditions. To bridge this gap, we apply an offine RFS fine-tuning stage using a small human-corrected dataset. During data collection, the base pol- icy executes rollouts while human provides corrective actions at visited states, yielding D {((o,s) s), (Ahuman, ab. ao), (d,, s'). Residual actions are defined as a, Ahuman ag producing the RFS-ready dataset DRIS = {((o,)) s), (ag. a,), (o', s"), r)). We then train TREs a, Ope, 8pm with offline reinforcement learning (Levine et al. 2020). using TD3+BC (Fujimoto and Gu 2021 for actor actor-critic updates with imitation regularization.
Critie Update. Given DRES we apply a standard TD-learning critic update in Eq. 10 The critic conditioned the combined actions and a' rather than the decomposed components (ag, ay); Section 5.2 provides an empirical justification for this design choice.
min (ao,s) (o'")")))))) d. (:ps .'')
[ -
-
@
(10)
as ausPushioo,,oo ao. a=ab-ar oe Pushio' dors)) a'=a,ta
Actor Update. Following TD3+BC (Fujimoto and Gu 2021), the actor maximizes the critic while applying BC regularizer on the residual actions:
arg ((loos)) ())))) [Q(o, 8, ABc ||ā, a, 1111
Tass
anp (o..)) Po āā
(11)

---

Published as a conference paper at ICLR 2026
5 EXPERIMENTS
We structure our experiments around the following research questions: Q1: Does residual refine- ment improve success in dexterous manipulation tasks in simulation? Q2: Which design choices are most effective across both simulation and real-world settings? Q3: How does latent-noise mod- ulation influence real-world adaptation performance? As outlined in Section 4.2 we first evaluate RFS for simulation data generation and then for real-world fine-tuning. Our experiments focus pri- marily dexterous grasping with multifingered hand, though the method generalizes to broader manipulation tasks.
5.1
RFS FOR SIMULATION DATA GENERATION AND ONLINE RL TRAINING
5.1.1
SETUP
Simulation Objects
Real Objects
We evaluate RFS on six simulated tasks: grasp- ing pick-and-place, non-prehensile push- to-grasp, long-horizon packing, and high- precision stacking and pouring (Fig. +. Us- ing Apple Vision Pro AR, we collect approx- imately 400 demonstrations per task and pre- train a generative base policy Ug [z, t, 8) using flow matching (Lipman et al. (022)). Base- policy success rates are reported in Table [ Although the pretrained policy achieves only moderate success, it provides strong motion priors that RFS further refines via PPO (Schul- man et al. 20176) (Sec. 4.2.Tp.
Figure 3: Simulation and real objects used for dexterous grasping and pick & place.
5.1.2 BASELINES AND ABLATIONS
Baselines. We compare RFS against broad suite of reinforcement-learmin baselines commonly used for simulation-driven dexterous manipulation. All demonstration-basel baselines use the same offline data as RFS. We group these methods into four categories based on their underlying adaptation strategies: (1) Diffusion/Flow RL Fine-tuning. DPPO (Ren et al. 2024a) and Rein- Flow (Zhang et al., 2025a), which fine-tune demmonstrationppettaanee diffusion or flow-matching policies through reinforcement learning with environment interaction. (2) Offline-to-Onl
RL.
IQL (Kostrikov et al. 2021), AWAC (Nair et al.) 2021). and Flow Q-Learning (Park et al.) 2025), which are mitiiliizdd from offline demonstrations and subsequently improved via online interaction. (3) RL with Demonstrations, RLPD (Ball et al. 2023) and IBRL (Hu et al. 2024), which incorpo- rate demonstration data directly into policy updates.
Ablations. To isolate the contributions of both residual refinement and latent-noise steering, we conduct targeted ablations. (1) DSRL (Wagenmaker et al., 22025:: steering the latent noise of the base policy without residual actions. (2) State-of-the-Art Residual RL; strong residual-learning
baselines, including Policy Decorator Yuan et al. 2024) and ResiP (Ankile al. 2024a).
5.1.3 EMPIRICAL RESULTS IN SIMULATION
Comparison to Baselines: We summarize the performance of all baselines in Table I and analyze
their behavior as follows:
(1) SerfusionnFFlow-asssee RL Fine-Tuning, DPPO (Ren et al. 2024a) underperforms across all tasks, with all success rates below 0.50. ReinFlow MaanFlooaaag et al. (2025a) shows modest improvements (0.58 grasp, 0.46 pick & place, 0.39 packing), but fails on push-to-get push-to-grasp, stacking, and pouring, In contrast, restricting RL updates to the initial noise and residual terms (RFS) yields more stable learning, achieving success rates least 0.35 higher across all tasks than applying RL over the full denoising trajectory.
(2) Offline-to-Olline RL. IQL (Kostrikov et al. 2021 is the strongest offline-to-onlin baseline (0.69 grasp, 0.56 pick-and aande,lace 0.63 stack, 0.59 pour). but struggles on packing (0.20) and push- to-grasp (0.26). AWAC (Nair et al. 2021) and FlowQ (Park et al. 2025) collapse multiple tasks.

---

Published as a conference paper at ICLR 2026
Overall, affline-to-onlle RL methods lack directed exploration, whereas RFS leverages base-policy pretraining to achieve consistently higher success rates across all tasks.
(3) RL with Demonstrations. RLPD (Ball et al., 2023) performs well grasping (0.54) and pick-and-place (0.76), but collapses on tasks requiring long-horizon reasoning (c.g.. packing) or high-precision manipulation (e.g., stacking and pouring). IBRL (Hu et al. 2024) succeeds only grasping, packing, and pouring. In contrast, RFS consistently improves over the base policy across all tasks.
Comparison to Ablations. We ad- ditionally evaluate strong ablations and related adaptation strategies (Ta- ble [I.
(1) Residual RL methods. Pol- icy Decorator (Yuan et al., (024) and ResiP (Ankile et al.J (2024a) out- perform earlier baselines, but their gains remain largely local, achiev- ing moderate success on grasping (~ 0.60) and pick-and-place (~ 0.50).
Performance degrades on sequential and high-precision tasks, including packing (0.40), push-to-grasp (0.15),
stacking (0.30), and pouring (0.20).
In contrast, RFS achieves ~ 0.80
success on packing and push-to-grasp
and ~ 0.90 on the remaining tasks.
(2) DSRL (Wagemmaker 2025). DSRL is the strongest base- line, achieving 0.74 (grasp), 0.70 ppick-and-place). 0.64 (packing), and 0.43 gusssp)--raasp, but only 0.15 on stacking and 0.26 pouring. Be- cause its corrections are constrained to the demonstration distribution, DSRL struggles to adapt to challenging or out-of-distribution states.
al.
Figure 4: Representative rollouts for the dexterous manipu-
lation tasks. From top to bottom: Packing, Push-to-Grapp. Packing and Stacking.
Overall By jointly modulating the input and output of the base policy, RFS preserves base-policy performance while consistently improving upon it, achieving the highest success rates across all tasks: 0.89 (Grasping), 0.94 (Pick & Place), 0.78 (Packing), 0.72 Mush-to-Grasph 0.95 (Stacking), and 0.87 (Pouring), with an average success rate of 0.87.
Camppey
Graping
Base Polley
Flo Marting
1553
D:40m
0.367
0:30
0.111
0.05
0.15
(1:50
Finstuing
ReidFi
0.564 0.04
0462
0.000=000
0:02)
0.33
Offlieeto-Chllter
101 Kourttas
DARD 80054
0.500
0.301
8.06
0.367 1.0.00
BA28 0.07
0.633
0.305
0:488
0050
0/486
8.00
1,40
010 0.0
010 0.00
0.155 01007
RL. with Deasestrations
RD Bak
0.547 1.54:5:0000
0.763 0.00
0:00-1000
0012 0:024
0.53 0-133
0:68 0.15
0.345 0.07
Reiddeal RL the-Att
Polcy
Poomata Ntauutl 2024
0.52 0 0.09
0.314 0.160
0,535
0.176:2:000 0.05
0.00=00.0 0.00
0.17 0.35
0.286 0.066
kroP Artll It moreiy
0.6 0199
0.57
1001-
0.303
8.008
0.165 0.18
857 0.85
0134
0.12
8/155
0100
DIRE. Wagromaker 2016
0:99 0.012
0.639
1:35 0.15
0.266
0.000
0.483 0.51
RES (Ous)
0,899 = 0.806
0,33 0.012
0,751
= 0,19
0.721 = 0.022
0351 0.006
0,873 0.000
0.561 0.816
Table 1: Success rates across tasks and RL. methods.
5.2 REAL-WORLD OFFLINE FINETUNING
Using data generated by RFS in simulation with access to privileged state information, we distill a point cloud-bace based policy via a standard student-teacher framework (Chen et al. 202I) for real- world deployment.
7

---

Published as a conference paper at ICLR 2026
We then evaluate offline RFS for adapting the distilled policy in real-world grasping and pick-and- place tasks with both seen and unseen objects (Fig. Fig. Experiments are conducted on a Franka arm equipped with LEAP hand under Cartesian impedance control at Hz. We evaluate seven objects, including two shared with simulation and five novel deformable objects. Compared rigid simulation assets, real-world objects exhibit varying compliance, contact dynamics, and appearance, introducing a sim-to-real gap that often causes pretrained policies to fail.
For offline RL fine-tuning, we collect 50 human corrective demonstrations using a SpaceMouse (Appendix A.5.I). I) Rewards are defined programmatically by segmenting and tracking objects with SAM2 Ravi et 2024) and extracting their centroids (Appendix A.5.2)
5.2.1 BASELINES AND ABLATIONS
Baselines. We evaluate RFS with offline RL (Section 4.2.2) as data-efficient approach for fine- tuning simulation-trained policies. We compare against three adaptation strategies: (1) Zero- Shot Transfer, which deploys the distilled simulation policy without fine-tuning; (2) BC Fine- tuning, which uses 50 real-world demonstrations to further train the distilled policy via flow match- ing: (3) Co-Training, which fine-tunes the distilled policy by mixing 50 real-world demonstra- tions with additional simulated rollouts under the fow-matching objective. These comparisons demonstrate that offline RL fine-tuning yields substantially stronger adaptation than standard flow- matching- based updates.
Critic Design Choices: We additionally study alternative critic architectures in the offline-RL setting, comparing: (1) Q(a,, o). which conditions only on residual actions and observations; (2) Q(a,,, which conditions on both residual and base-policy actions; (3) Q(a + a,, o) (Ours), which conditions solely the final executed action. This isolates the effect of critic input structure on offline-RL stability and performance.
Ablations: We further analyze offline RL using only residual actions (Residual RL) and only latent- noise steering (DSRL) to quantify their individual contributions.
5.3 EMPIRICAL RESULTS IN THE REAL WORLD
Comparison Supervised Fine-Tuning As shown in Fig. 5 and Fig. 6 the zero- shot policy performs poorly. exhibiting translational and rotational offsets, insuf- ficient finger closure during grasping, and mistimed or misplaced actions in pick- and-place, highlighting significant sim- to-real gap. While co-training substan- tially improves performance on known ob- jects (83.3% grasping and 60% pick-and- place success), generalization to unseen objects remains limited. The noisy and in- consistent nature of human-collected data introduces instability and rollout failures, underscoring the difficulty of generalizing novel objects.
Method
Pick-and-Place
Grasping
0-shot
50.0 + 0.0
43.3 + 6.2
Co-training
60.0 + 0.0
83.3 + 0.0
BC
40.0 + 0.0
73.3 0.0
Residual RL
50.0 + 0.0
80.0 + 0.0
DSRL RFS (Ours)
80.0 + 0.0
70.0 + 0.3
90.0 + 0.0
80.0 + 2.9
Table 2: Performance seen objects. Values report mean with 95% confidence interval trials.
Pick & Place
Grasp
RFS (Ours)
70.0
74,0
DSRL
Comparison to Variants of Offline RL. As discussed in Sec. 4.2.2] TD3+BC con- ditions the critic on the actor's output, al- lowing different Q-function parameteriza- tions. Conditioning the critic on either Q(ar. o) or Q((a,, as], o) fails to produce consistent grasp poses precise place- ment, yielding zero success on both grasp- In contrast, con-
Residual RL
36.0
45.0
BC.
23.0
35.0
Co-training
47.5
0-shot
40.0
30.0
Figure 5: Real-world evaluation results on unseen objects.
Bars show mean success rates for Pick-and- Place (left) and Grasping (right), with horizontal error bars denoting the 95% confidence interval. Numeric annotations report mean success rate (495% CI).
ing and pick-and-place ditioning on the executed action Q(a, + ar, used by both Residual RI. and
8

---

Published as a conference paper at ICLR 2026
O-shot Transfer
For away
Loose grasp
Unstabl grasp
Mistimed
Misplaced
Thase
1)
Residual Correction
RFS
Ubasc
base
ill noiss
Opes
IT RFS
Figure 6: Common failure modes in real-world manipulation and RFS corrections. We illustrate five representative failure cases of zero-shot sim-to-real transfer, including early hand closure, loose or unstable grasp poses, and mistimed or misplaced actions during bock-and-place The top row shows failures produced by the zero-shot policy, while the bottom row shows the corrected outcomes after applying Residual Flow Steering (RFS). By jointly correcting action timing, grasp pose, and target location, RFS enables more stable and successful task execution.
RFS (Fig. 5 achieves substantially higher performance across all baselines. These results indi- cate that directly coupling residual actions with the executed base actions is critical for stable and effective offline RL adaptation.
Comparison to Ablations of RFS. We evaluate RFS variants that use only residual actions (T(a, o)) or only latent-noise steering via DSRL (T o)). As shown in Fig. 5 and Fig. 6 the full model [(a, ar o) achieves the strongest performance, particularly grasping and pick-and-ples tasks. Qualitatively, latent-noise steering facilitates global exploration, while residual actions enable precise local refinement, making their combination more effective than either component alone.
unseen objects in both
Comparison Direct Real-Data Fine-Tuning. Fine-tuning RFS on a policy trained solely on real demonstrations yields only modest gains (70% > 73% on seen objects and 35% + 50% on unseen objects for grasping, with little improvement for pick-and-place), reflecting the limited coverage of the real-data distribution. In contrast, simulation pretraining provides broad coverage over grasps, poses, and perturbations, enabling adaptation behaviors that cannot be learned from real data alone. As result, fine-tuning cumulation-pretraine policy with a small number of real demonstrations yields substantially larger improvements and stronger generalization.
CONCLUSION
We introduced Residual Flow Steering (RFS), a reinforcement learning framework that unifies la- tent noise steering for global exploration with residual corrections for local refinement. Across both simulation and real-world dexterous manipulation tasks, RFS consistently outperforms con- ventional baselines-includiing residual-only methods and diffusion- effective simulation pretraining and data-efficien fine-tuning in the real world. Despite these strong results, several limitations remain and point to directions for future work. First, our current imple- mentation primarily relies on point cloud observations, which provide limited semantic and con- textual understanding. As a result, performance can degrade cluttered or semantically rich en- vironments that require higher-level reasoning. Second, real-world adaptation in framework is currently restricted to offline fine-tuning. This prevents the policy from updating online in response dynamically changing conditions, limiting its responsiveness and robustness during real-world deployment.
steering- enabling
ACKNOWLEDDMENTS
This work was supported by Amazon FAR (Frontier AI & Robotics).

---

Published as a conference paper at ICLR 2026
REFERENCES
Minttu Alakuijala, Gabriel Dulac-Amold, Julien Mairal, Jean Ponce, and Cordelia Schmid. Resid- ual reinforcement learning from demonstrations, 2021. URL https://azxiie wwwpwiiwww....aa../////////
2106. 08050
Lars Ankile, Anthony Simeonov, Idan Shenfeld, Marcel Torne, and Pulkit Agrawal. From imitation to refinement residual rl for precise assembly, 2024a. URL htttps://rxiv..or///a
2407.1667
Lars Ankile, Anthony Simeonov, Idan Shenfeld, Marcel Torn,, and Pulkit Agrawal. From imitation to refinement residual RL for precise visual assembly. CoRR, abs/2407.16677,, 2024b. doi: 10. 48350/ARXIVV2407. 16677. https:/
http://.....5500/1..2222201000 16677
Lars Ankile, Zhenyu Jiang, Rocky Duan, Guanya Shi, Pieter Abbeel, and Anusha Nagabandi. Resid- ual off- policy rl for finetuning behavior cloning policies, 2025. URL https://axxi..orr
abs/ 220//259.... 10099.9933 19301
Philip J. Ball, Laura Smith, llya Kostrikov, and Sergey Levine. Efficient online reinforcement learn- ing with offline data, 2023. URL
2302. 02948.
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky. A vision- language-action flow model for general robot control. arXiv preprint arXiv:2410.24164, 2024. URL httthhttp://aaat
otg/ abs/ 2410 24164
Joao Carvalho, Dorothea Koert, Marek Daniv, and Jan Peters. Residual robot learning for object- centric probabilistic movement primitives, 2022. URL 03918
Tao Chen, Jie Xu, and Pulkit Agrawal. A system for general in-hand object re-orientation. In Alek- sandra Faust, David Hsu, and Gerhard Neumann, editors, Conference on Robot Learning, 8-11 November 2021, London, UK, volume 164 of Proceedings of Machine Learning Research, pages 297-307. PMLR, 2021. URL https::/proceeddiiittt html
www.pppww........e..o...ep chen22a.
Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems (RSS), 2023.
Todor Davchev, Kevin Sebastian Luck, Michael Burke, Franziska Meier, Stefan Schaal, and Sub- ramanian Ramamoorthy. Residual learning from demonstration: Adapting dmps for contact-rich manipulation. IEEE Robotics and Automation Letters, 7(2) 4495, April 2022. ISSN 2377- 3774. doi: 3150024
URL http:/ dx doi www.//////.....///......1 2022 http:::://..........o////11111/1
Ashley Davey and Harry Zheng. Convergence of proximal policy gradient method for problems with control dependent diffusion coefficients, 2025. URL
18379
Maximilian Du and Shuran Song. Dynaguide: Steering diffusion polices with active dynamic guid- ance, 2025. URL hittps://arxiv wwwpwvoorrggab.... /ttp///t////aavrrr.r///////////0
Linjiajie Fang, Ruoxue Liu, Jing Zhang, Wenjia Wang, and Bing- Yi Jing. Diffusion actor-critic:
Formulating constrained policy iteration as diffusson noise regression for offline reinforcement learning, 2025. http::
org/ aww./b//a00 20555
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach offline reinforcement learning. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021.
Siddhant Haldar, Jyothish Pari, Anant Rai, and Lerrel Pinto. Teach robot to fish: Versatile imitation from one minute of demonstrations, 2023. URL htttss://aaattt httpps//aanxxv.ooor///////3 01497
10

---

Published as a conference paper at ICLR 2026
Todd Hester, Matej Vecerik, Olivier Pietquin, Mare Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Gabriel Dulac- Dulac-Arnold Ian Osband, John Agapiou, Joel Z. Leibo, and Audrumas Gruslys. Deep q-learning from demonstrations, 2017. URL https: //axiv..
org/abs/170ce 03732
Jonathan ing-
Ho and Stefano Ermon. In Advances in Neural Information Processing Systems (NeurlPS), vol-
Generative adversarial imitation learn-
ume
6391 gwweratieeeeeeeeeeeeee seee-avversaarrr wwwiimmmiiiionn. lneeminnggg nips. cc cc/paper/
29, pages 4565-4573, 2016. URL
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 6840-6851, 2020. URL
https: https::/arxiise httpss:/aarxvv.ooo
abs 2006. 11239
Hengyuan Hu, Suvir Mirchandani, and Dorsa Sadigh. Imitation bootstrapped reinforcement learn- ing, 2024.
abs/ 22111 02100119
Ahmed Hussein, Mohamed Medhat Gaber, Eyad Flyan, and Chrisina Jayne. Imitation learning: A survey of learning methods. ACM Comput. Surv,, 50(2), April 2017. ISSN 0360-0300, doi: 10. 10.1145/305491.
org http://1..1111333330 http://///.cc...rr/000..1111////
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning, 2021. URL /ttpptt/stxxxi......./ org. abaaabss2111 2210. 06169
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tu- torial, review, and perspectives on open problems, 2020. URL
2005. 01643]
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Manfred Otto Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. CoRR, abs//1509022 02977, 2015. URL
org/
Corpus ID: 16:2666666663
Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022. URL https://arxiv.
brg/abs/2u0 2aab0/2211.. 02747
Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian NickeeiieianNNNek.,.i for generative modeling, 2023. URI. htl.:tttps////xivooooo
Nickel, and Matt Le. Flow matching
.00747
Qiang Liu. Rectified flow: A marginal preserving approach optimal transport, 2022. URL https::/arxxv..rr//// 2tt.a////229..55.2..
David McAllister, Songwei Ge, Brent Yi, Chung Min Kim, Ethan Weber, Hongsuk Choi, Haiwen Feng, and Angjoo Kanazawa. Flow matching policy gradients, 2025. /ttps:t::://axxv
org/aba/2500 2atp.////5022233333
Mayank Mittal, Calvin Yu, Qinxi Yu, Jingzhou Liu, Nikita Rudin, David Hoeller, Jia Lin Yuan, Ritvik Singh, Yunrong Guo, Hammad Mazhar, Ajay Mandlekar, Buck Babich, Gavriel State, Marco Hutter, and Animesh Garg, Orbit: A unified simulation framework for interactive robot learning environments. IEEE Robotics and Automation Letters, 86):3740-3747, 2023. doi:
10.11091LRA.203:227000
Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Over- coming exploration in reinforcement learning with demonstrations, 2018. URL https://
arxiv.org/abe mwxtv.org//bb/1100.. /
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online inforcement learning with offline datasets, 2021. URL httph::///// httpp:////axiivooo//b////0000
0':59
Mitsuhiko Nakamoto, Yuexiang Zhai, Anikait Singh, Max Sobol Mark, Yi Ma. Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-ql: Calibrated offline rl pre- pre-training for efficient online fine- tuning, 2024. URL Rtll.httss///axiiii /2303 054799057999)

---

Published as a conference paper at ICLR 2026
NVIDIA, i, Mayank Mittal, Pascal Roth, James Tigue, Antoine Richard, Octi Zhang, Peter Du, Antonio Serrano-Muñon Xinjie Yao, René Zurbrügg, Nikita Rudin, Lukasz Wawrzymiak, Mi- lad Rakhsha, Alain Denzler, Eric Heiden, Ales Borovicka, Ossama Ahmed, Iretiayo Akinola, Abrar Anwar, Mark T. Carlson, Ji Yuan Feng, Animesh Garg, Renato Gasoto, Lionel Gulich, Yijie Guo, M. Gussert, Alex Hansen, Mihir Kulkarni, Chenran Li, Wei Liu, Viktor Makoviy- chuk, Grzegorz Malczyk, Hammad Mazhar, Masoud Moghani, Adithyavairavan Murali, Michael Noseworthy, Alexander Poddubny, Nathan Ratliff, Welf Rehberg, Clemens Schwarke, Ritvik Singh, James Latham Smith, Bingjie Tang, Ruchik Thaker, Matthew Trepte, Karl Van Wyk, Fangzhou Yu, Alex Millane, Vikram Ramasamy, Remo Steiner, Sangeeta Subramanian, Clemens Volk, CY Chen, Neel Jawale, Ashwin Varghese Kuruttukulam, Michael A. Lin, Ajay Mandlekar, Karsten Patzwaldt, John Welsh, Huihua Zhao, Fatima Anes, Jean-Francois Lafleche, Nicolas Moënne-Loccoz, Soowan Park, Rob Stepinski, Dirk Van Gelder, Chris Amevor, Jan Carius, Jumyung Chang, Anka He Chen, Pablo de Heras Cicchomski, Gilles Davict, Mohammad Mo- hajerani, Julia von Muralt, Viktor Reutskyy. Michael Sauter, Simon Schirm, Eric L. Shi, Pierre Terdiman, Kenny Vilella, Tobias Widmer, Gordon Yeoman, Tiffany Chen, Sergey Grizan, Cathy Li, Lotus Li, Connor Smith, Rafael Wiltz, Kostas Alexis, Yan Chang, David Chu, Linxi "Jim" Fan, Farbod Farshidian, Ankur Handa, Spencer Huang, Marco Hutter, Yashraj Narang, Soha Pouya, Shiwei Sheng, Yuke Zhu, Miles Macklin, Adam Moravanszky, Philipp Reist, Yunrong Guo, David Hoeller, and Gavriel State. Isaac lab: A gpu-accelerated simulation framework for multi-modal robot learning. 2025. URL https: / bttts::/arxiv.... httaaaaiw....aaa/////aaaa aaaaaassss551 2511. 04831
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkinn Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. arXiv preprint ar wtt.:22220.222... 2022. URL httph:tt////ttt pettps://arxiv..oo http:/awwwvv..aaaaaaaaoooo /bs/////////////aee//g
Seohong Park, Qiyang Li, and Sergey Levine. Flow q-learning, 2025. URL https://arxiv
org/aba 2502. 02538
Younghyo Park and Pulkit Agrawal. Using apple vision pro to train and control robots, 2024. URL
https: //tpt/ggitttbbbb com http://pwobbbblll///ssoosooooro.
Charles R. Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. Pointnet: Deep learning on point
sets for 3d classification and segmentation, 2017. Utt//tttpss//axxv.oor//bbb//////
00593
Yuzhe Qin, Yuah-Hua Wu, Shaowei Liu, Hanwen Jiang, Ruihan Yang, Yang Fu, and Xiaolong Wang. Dexmv: Imitation learning for dexterous manipulation from human videos, 2022. URL
https: org/ alss /tabs/2110a 210.21111..0 05877
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations, 2018a. URL http:httt///axiiieee wttp://wwwwwg//////o/oa// 10087
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations, 2018b. URL http::tts//// //axiv. org/ abs/ 1109/7799..0 10087
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Va- sudev Alwala, Nicolas Carion, Chao- Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Fe- ichtenhofer. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024. URL https: arxiv.
abs
Allen Z. Ren, Justin Lidard, Lars I. Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majum- dar, Benjamin Burchfiel, Hongkai Dai, and Max Simchowitz. Diffusion policy policy optimiza- tion, 2024a. URL htLhttps:: http://aaavvvvv..//.a/ wwwp:wwwwwwrr...oooooo...oo.oo. abs////////// 220.//0009988888888
Allen Z. Ren, Justin Lidard, Lars L. Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majum- dar, Benjamin Burchfiel, Hongkai Dai, and Max Simchowitz. Diffusion policy policy optimiza- tion, 2024b. htR..ttps:/// /
abs /2409 00588
12

---

Published as a conference paper at ICLR 2026
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, /ww. org abs
0637
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017b. URLptttp::/aazii.ooo
http/ab//1100 1707 063477
Tom Silver, Kelsey Allen, Josh Tenenbaum, and Leslie Kaelbling. Residual policy learning, 2019.
URL httpthtss//axii..ooct wttp://r////////1..10030303 06298
Yuda Song, Yifei Zhou, Ayush Sekhari, Andrew Bagnell, Akshay Krishnamurthy, and Wen Sun. Hybrid rl: Using both offline and online data can make rl efficient, 2023. URL https://
mexiv.org/o abs/ 22102221.222 06718
TRI LBM Team, Jose Barreiros, Andrew Beaulicu, Aditya Bhat, Rick Cory, Eric Cousineau, Hongkai Dai, Ching-Hsin Fang, Kunimatsu Hashimoto, Muhammad Zubair Irshad, Masha Itk- ina, Naveen Kuppuswamy. Kuan-Hui Lee, Katherine Liu, Dale McConachie, Ian McMahon, Haruki Nishimura, Calder Phillips-Graffhe Charles Richter, Paarth Shah, Krishnan Srinivasan, Blake Wulfe, Chen Xu, Mengchao Zhang. Alex Alspach, Maya Angeles, Kushal Arora, Vi- tor Campagnolo Guizilini, Alejandro Castro, Dian Chen, Ting-Sheng Chu, Sam Creasey, Sean Curtis, Richard Denitto, Emma Dixon, Eric Dusel, Matthew Ferreira, Aimee Goncalves, Grant Gould, Damrong Guoy, Swati Gupta, Xuchen Han, Kyle Hatch, Brendan Hathaway, Allison Henry, Hillel Hochsztein, Phoebe Horgan, Shun Iwase, Donovon Jackson, Siddharth Karamcheti, Sedrick Keh, Joseph Masterjohn, Jean Mercat, Patrick Miller, Paul Mitiguy, Tony Nguyen, Jeremy Nimmer, Yuki Noguchi, Reko Ong, Aykut Onol, Owen Pfannenstichl, Richard Poyner, Leticia Priebe Mendes Rocha, Gordon Richardson, Christopher Rodriguez, Derick Seale, Michael Sher- man, Mariah Smith-Jones, David Tago, Pavel Tokmakov, Matthew Tran, Basile Van Hoorick, Igor Vasiljevic, Sergey Zakharov, Mark Zolotas, Rares Ambrus, and Kerri Fetzer-Borelli. A care- ful examination of large behavior models for multitask dexterous manipulation. arXiv preprint arXiv:2507.05331, 2025.
/enttps://ariive http://aawvvoooaaaaa//./o wttp:/wwwwww//a//////////0/ 0
Andrew Wagenmaker, Mitsuhiko Nakamoto, Yunchu Zhang, Seohong Park, Waleed Yagoub, Anusha Nagabandi, Abhishek Gupta, and Sergey Levine. Steering your diffusion policy with latent space reinforcement learning, 2025. URL httph::tp///// httt:p:://arxvv.orrte
/206. 15799
Huan Wang, Stephan Zheng, Caiming Xiong, and Richard Socher. On the generalization gap in reparameterizaable reinforcement learning. 2019. URL
12654
Jun Wang, Ying Yuan, Haichuan Che, Haozhi Qi, Yi Ma, Jitendra Malik, and Xiaolong Wang. Lessons from learning spin "pens", 2024. URL http:://axxxl..ot///bb///2
18902
Wenli Xiao, Haotian Lin, Andy Peng, Haoru Xue, Tairan He, Yuqi Xie, Fengyuan Hu, Jimmy Wu, Zhengyi Luo, Linxi "Jim" Fan, Guanya Shi, and Yuke Zhu. Self-improving vision-language- action models with data generation via residual 2025a. URL htttttt///axxii>> www.:/owrrrreeee /bs/ 2511.00091
Wenli Xiao, Haotian Lin, Andy Peng, Haoru Xue, Tairan He, Yuqi Xie, Fengyuan Hu, Jimmy Wu, Zhengyi Luo, Linxi "Jim" " Fan, Guanya Shi, and Yuke Zhu. Self-improving vision-language- action models with data generation via residual rl, 2025b.
22511.0000
Xiu Yuan, Tongzhou Mu, Stone Tao, Yunhao Fang, Mengke Zhang, and Hao Su. Policy decorator: Model-agnostic online refinement for large policy model, 2024. (tR.:httss://arxiv.ooo abs 2tt./2222.13633
Andy Zeng. Shuran Song, Johnny Lee, Alberto Rodriguez, and Thomas Funkhouser. Tossingbot: Learning to throw arbitrary objects with residual physics, 2020.
abs/ abs//1903.1 11239
Shangtong Zhang, Wendelin Bochmer, and Shimon Whiteson. Deep residual reinforcement learn- ing, 2020. URL U htttp:://arxiv..ot// /1w.a//1999 /090.. 01072
13

---

Published as a conference paper at ICLR 2026
Tonghe Zhang, Chao Yu, Sichang Su, and Yu Wang. Reinflow: Fine-tuning flow matching pol-
icy with online reinforcement learning, 2025a. URL
22094
Tonghe Zhang, Chao Yu, Sichang Su, and Yu Wang. Reinflow: Fine-tuning flow matching pol- icy with online reinforcement learning, 2025b. URL httpps//aax....orr//bb////////// 22094
Tony Z. Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. In Proceedings of Robotics: Science and Systems (RSS), ALOHA: Action Chunking with
2023. URL httpa////aaii..ooaa///////////// /20p//224..oo 13705 Transformers (ACT).
Tony Z. Zhao, Jonathan Tompson, Danny Driess, Pete Florence, Seyed Kamyar Seyed Ghasemipour, Chelsea Finn, and Ayzaan Wahid. Aloha unleashed: A simple recipe for robot dexterity. In Pro- ceedings of The 8th Conference Robot Learning (CoRL), volume 270 of Proceedings of Ma- chine Learning Research, pages 1910-1924. PMLR, 2025. URL https: / proceedings. mlr. wwwps/eesssvv..... 7 zhao25bbbb. html
14

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 THE USE OF LARGE LANGUAGE MODELS
In this paper, we primarily LLMs for language polishing, making our writing concise and accessible. We provide raw drafts and ask the LLM to refine them. We also use LLMs for guidance on figure preparation, such as creating illustrations in Inkscape and adjusting figure size and font in Matplotlib, During rebuttal, we also used an LLM to polish the writing, making the language more concise and clear based on reviewer feedback.
A.2 SIMULATION SETUP AND TRAINING
A.2.1 REINFORCEMENT LEARNING DESIGN CHOICES
To enhance robustness and mitigate controller misalignment for sim-to-real transfer, introduce the following design choices:
Simulator: All of our simulated tasks are built IsaacLab (NVIDIA al. 2025)
Teleoperation: We further developed the teleoperation system in IsaacLab (NVIDIA et al. 2025) make more suitable for dexterous hand manipulation.
External Disturbances: Apply random external disturbances to each joint at intervals 2-5 steps, encouraging the policy to recover from perturbations.
Observation Space: In the simulation RL training, we use state- based information, including the robot proprioception, the current object pose, the target object pose, and binary contact signals between each fingertip and the object.
Data Collection: We the Vision Pro for data collection, leveraging application built IsaacLab Mittal et al. (2023).
A.3 PPO HYPERPARAMETERS
We train the Residual Flow Steering (RFS) policy using Proximal Policy Optimization (PPO) (Schul- man et al. (017a) during online fine-tuning in simulation. The same set of hyperparameters is used across all tasks.
Policy and Value Networks. Both the RFS policy TRFS (4o, ar |) and the value function (ss) are parameterized by multilayer perceptrons with three hidden layers of width 256. 128,62866 and ReLU activations. The policy outputs both the latent noise steering variable wo for flow matching base policy and the residual action a,..
Table 3: PPO hyperparameters used for online RFS training simulation.
Hyperparameter
Value
Discount factor ?
0.99
GAE parameter X
0.95
Policy learning rate
3 10-4
Value learning rate
10-3
Optimizer
Adam
Clip range
0.2
Value loss coefficient
0.5
Max gradient norm
1.0
Minibatch size
1024
A.3.1 POLICY DISTILLATION
To improve sim-to-real transfer, we use point clouds as visual observations. To mitigate errors from camera calibration and hardware variations, collect simulation data from multiple camera
15

---

Published as a conference paper at ICLR 2026
viewpoints and calibrate all point clouds to the robot base frame. We further inject random noise into the camera extrinsics during data collection and into the point clouds during training to improve robustness. In simulation, collect 1,000 demonstrations for each of the grasping and pick-and- place tasks to train the policy.
A.4 REAL WORLD EXPERIMENT DETAILS
A.4.1 REAL ROBOT EVALUATION
For real-robot evaluation, we tested our policy within a 30 x 50 cm region located 0.35-0.65 m from the robot base and spanning -0.25 0.25 m horizontally. We evaluated known objects with 20 trials each, and seven unknown objects with 10 trials each.
A.5 TELEOPERATION DATA COLLECTION
For real-world telcoperation data collection in the co-training setup, we developed Vision Pro application based on Park and Agrawal (2024) that supports teleoperation a controlled frequency of 10Hz.
A.5.1 OFFLINE DATA COLLECTION
When the hand is within ~ cm of the table- most failures occur (Fig. 6). we enable human intervention via a SpaceMouse (Fig.3) (8. Rather than granting full manual control, which would shift the policy distribution, we compute residual corrections as bounded deltas from the base policy out- put, conditioned on the current observation. Specifically, the CFM module predicts the next action, and we take the difference between this rollout and the operator's 's input. Residuals constrained to 11.5 cm in Cartesian translation and <0.05 rad for finger motion, applied uniformly across all joints. In practice, corrections were limited Cartesian translation and minor finger adjustments, with the SpaceMouse z-axis mapped to finger motion.
A.5.2 REWARD FUNCTION DESIGN
In the real world, the reward is constructed using SAM2 (Ravi et al., 2024) to track the object in image space and extract its point cloud, from which the object center computed. The reward consists of two terms: (1) the distance between the object center and the palm, obtained via forward kinematics, and (2) the object's lifting height, which encourages stable grasp execution. Figure illustrates the reward trend.
A.6 TD3+BC HYPERPARAMETERS
We fine-tune the Residual Flow Steering (RFS) policy in the real world using the TD3+BC al- gorithm (Fujimoto and Gu 202T). operating in an offline setting with fixed dataset of human- corrected transitions. Observations are represented as point clouds, and the perception backbone adapted from PointNet (Qi et al. 2017). The TD3+BC hyperparameters used in all real-world experiments are summarized in Table
Table 4: TD3+BC hyperparameters used for offline RFS fine-tuning in woal-wrldd experiments.
Hyperparameter
Value
Actor learning rate
3 10 4
Critic learning rate
3 10-1
Policy update delay
10
Target policy noise std
0.2
Target noise clip
0.5
Behavior cloning weight Agc
0.2
Batch size
512
Replay buffer
Fixed offline dataset
16

---

Published as a conference paper at ICLR 2026
Helpful chicken expert
Leap hand V
Space mouse
Figure 7: Reward progression over time in real-world ex- periments.
Figure Offline real-world data collection setup using the Leap Hand VI and space mouse interface.
B SIMULATION BASELINES AND CODEBASES
We compare RFS against diverse set of simulation baselines spanning diffusion/flow RL fine- tuning, offline-to-online RL, RL with demonstrations, and the SOTA residual RL methods. For each baseline, use the open-source implementations and follow their recommended training protocols.
DPPO (Ren et al.
2024a):
https: / com/ ccwp//rommpprinccttoo//oggo
ReinFlow (Zhang et al 2025a):
pettps:://githubbb.mm//eeee
Flow Q.Leernnnnng (FQQ) and IQL PPar et al. 2025):
AWAC et al. 2021
https::/github. com/ wow//oosttoooo/////o....
RLPD and IBRL (Hu et al. 2024h:
attps://git com htt.:auuuccccceeaaaaaaaaaaaaaaaa
bengyuan-hu/ihe
Policy Decorator Yuaan al.
2024)
https:://iithb.. com http:///oooggggoo//oooooooeoeoee
ReSIP (Ankile et al. 2024a):
https://github com/
All baselines are trained using the same simulator, observation modalities, and task specifications RFS to ensure a fair comparison.
17


Reviews:
[{"review_id": "uRKJ0Hrjjs", "reviewer": "Reviewer_22ER", "summary": "The paper proposes Residual Flow Steering (RFS), an RL fine‑tuning scheme for flow‑matching / diffusion policies that jointly (i) steers the initial latent noise for global changes and (ii) applies a residual action for local corrections—without modifying base policy parameters. The method is instantiated for dexterous grasping: pretrain in simulation (Leap Hand + Franka), distill a visuomotor policy, then fine‑tune in the real world with offline RL (TD3+BC). Experiments show improved success over action‑representation baselines and ablations (residual‑only or steering‑only) in both pinch and power grasps.", "questions": "Why is “changing the action representation” (absolute/relative joints, PCA, VQ‑VAE) considered a primary baseline for the first task? What does that comparison communicate about the contribution of the paper?\n\nCould you add head‑to‑head comparisons against other RL fine‑tuning methods for diffusion/flow policies to strengthen the claim that RFS is preferable beyond its own ablations?\n\nDo you have preliminary results (even in sim) on more complex tasks (e.g., non‑prehensile skills or multi‑stage manipulation) to support generalization claims?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "RFS is a clean unification of residual RL (output modulation) and latent steering (input modulation), formalized via a modulation policy.\n\nStrong, consistent improvements over baselines and over residual‑only / steering‑only ablations in simulation and better real‑world success vs. zero‑shot and supervised fine‑tuning.\n\nThe paper is well written. The introduction and the method are easy to follow and communicate the contributions and the implementation well.", "weaknesses": "Limited novelty. The algorithm mainly combines two widely used adaptation strategies (residual action learning and latent steering).\n\nBaseline coverage. Real‑world evaluation lacks comparisons to other RL fine‑tuning approaches for diffusion/flow policies (e.g., recent flow‑RL fine‑tuners); most comparisons are ablations or action‑space baselines.\n\nTask scope. Validation centers on grasping; the paper claims broader applicability, but no additional manipulation tasks are shown.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Yisnvz4yMP", "reviewer": "Reviewer_yEFi", "summary": "The paper considers the problem of fine-tuning pretrained generative (flow matching) policies for reinforcement learning. The contribution of this work is to introduce a general framework that incorporates two types of preexisting fine-tuning methods: input modulation (flow steering) and output modulation (residual learning). The authors propose a specific instance of this framework: Residual Flow Matching (RFS) which learns both an initial latent noise distribution and a policy that outputs residual actions, which is added to the push forward action produced by the pretrained model (pushed forward from the learned noise). The authors conduct simulated and real-life experiments in the dextrous manipulation robotics domain, and the experiments demonstrate RFS achieving favorable performance.", "questions": "Besides addressing my points in the weaknesses section, please make the following changes:\n1. The notation for math in the paper is a bit cluttered. Specifically, please do not use the symbol $a$ whenever you write $a=a_r+a_b$, as we also have $\\pi(a|s)$ (for example in Equation 4).\n2. Grammatical mistakes:\n\na. In the abstract the sentence \"Doing so allows policies to perform both local (residual actions) and\nglobal exploration (latent noise), data-efficient adaptation.\" seems grammatically incorrect.\n\nb. Please revise In table 1: absolute joint psose -> absolute joint pose.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper integrates two previous methods (flow steering, residual learning) with complementary benefits and drawbacks to get the best features of both.\n2. The framework the paper introduces is broad enough to be applicable to many important reinforcement learning applications, even outside the domain of manipulation, especially given the wide adoption of generative policies in various RL applications.", "weaknesses": "1. **Some Baseline choices lack motivation/clarity:**  I do not understand the use of the VQ-VAE and PCA baselines in Section 6.1.2/Table 1/Fig. 4. From my understanding, both are methods to get different state-action representations, whereas the focus of the paper is finetuning. If my understanding is correct, these results are comparing PPO finetuned with RFS with PPO trained for two separate state-action representations (made by PCA, VQ-VAE). Why is such a comparison meaningful?\n\n2. **Seeds for experiments:** For the simulation experiments, it is really not valid to make claims based on only 3 random seeds, especially given that the paper is mostly empirical in nature. \n \n3. Unless I am mistaken, residual RL is not a baseline for `w/ hand` in Table 1. Given that the authors propose a method that is essential residual RL+flow steering, I feel residual RL should be included for the `w/ hand` comparisons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lr7XIgTA18", "reviewer": "Reviewer_oCuu", "summary": "The paper focuses on offline-to-online reinforcement learning (RL) setting.\nSpecifically, the paper proposes to adapt generative policies with reinforcement learning by unifying residual policy learning and diffusion steering into a new class of algorithms, residual flow steering (RFS).\nThe paper illustrates, through simulated and real-life experiments, that RFS outperforms compared algorithms.", "questions": "See above.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The unification between residual RL and latent-noise steering is interesting, and can open up avenues for various choices of $f$ and $g$.\n- In the offline RL setting, RFS appears to generalize to unknown objects better in the real-life setting. The extra robustness experiments are helpful in demonstrating RFS' benefits.", "weaknesses": "- In section 5.2, the paper proposes to collect extra human correction data $a$---this seems to be a strong limitation, similar to applying the DAgger algorithm. It is possible that I have totally misunderstood this process:\n\t- The whole trajectory $((o_1, s_1), (o_2, s_2), \\dots)$ is generated using the correction action $a$, as opposed to below.\n\t- First sample the trajectory $((o_1, s_1), (o_2, s_2), \\dots)$ using the base policy actions, then obtain the correction actions based on the already collected trajectory.\n\t- Nevertheless, perhaps the paper can clarify this on lines 296-298.\n- Experiments\n\t- The experimental setting is confusing. In section 5.2 it is mentioned that the RFS is finetuned via offline RL, but in the experiment, specifically section 6.1.2, the setting is now in the online RL setting.\n\t\t- In 6.1.2, does the learner obtain any human-correction data for finetuning, in addition to the PPO trajectories?\n\t\t- Secondly, if the setting is originally offline-to-online RL, I think it's unfair to compare the learning curve of Tabula-rasa RL, and instead it should be compared against algorithms such as Cal-QL, AWAC-like algorithms, or RLPD.\n\t\t- Instead, the rationale on the chosen baselines on action-space reduction and action codebooks are unclear. Is the intuition to reduce the sample efficiency through easier exploration with smaller action space?\n\t- Likewise, for section 6, the choice of compared algorithms can be strengthened with offline RL algorithms like CQL and IQL.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PnCNomufca", "reviewer": "Reviewer_kxpu", "summary": "This paper introduces Residual Flow Steering (RFS), a novel reinforcement learning framework for fine-tuning flow-matching policies in dexterous manipulation.\nRFS adapts a pre-trained policy by learning to modulate both the initial latent noise (for global exploration) and output a residual action (for local refinement).\nThe method is evaluated in both simulation and real-world settings, demonstrating improved performance over several baselines. The core idea is intuitive and addresses a relevant challenge in policy adaptation. However, the paper could be strengthened by a more comprehensive related work section, clearer methodological explanations, and more competitive baseline comparisons.", "questions": "1. In section 5.2, it is mentioned that base policy actions $a_b$ and human corrections $a$ are recorded and transformations are applied to them to obtain $(a_0, a_r)$ for training. Why is the initial latent noise $a_0$ not recorded directly?\n2. The RFS policy $\\pi_{RFS}$ outputs $a_0$, which is passed through the \"Push\" function (an ODE solver involving multiple evaluations of $v_\\theta$). Could you clarify if gradients are backpropagated through the entire \"Push\" operation?\nIf so, there will be a huge computational cost for the gradient calculation during training and it is unstable. If not, how are the gradients for $a_0$ obtained?\nBy the way, could you mention the number of steps used for the \"Push\" operation?\n3. The policy $\\pi_{RFS}$ is trained to output both $a_0$ and $a_r$. I am curious about the results if we separate the current loss into two terms. For example, detaching $a_r$ when calculating gradients for the $a_0$, or vice-versa, to see if it stabilizes training or improves performance?\n4. The process of using RFS for simulation data generation could be explained more clearly.\nI got the following questions when reading this part:\nFirst, are VR-teleoperated demonstrations real-world data or simulation data?\nSecond, the high-level policy $\\pi_{RFS}$ is trained with the same demonstrations using an offline RL method or trained in simulation using online RL method?\nI can find the answer in Section 6.1: simulation data, online RL method (PPO) while it will be easier for the reader to understand if the Section 5.1 can be rephrased.\n5. What is the specific advantage of applying RFS to create a simulation policy first, rather than directly applying RFS to the original base policy $v_{\\theta}(a_t, s, t)$ using human-collected real-world data?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper proposes a novel RL method for adapting a policy pre-trained with flow-matching to dexterous manipuation tasks.\nThe adaptation is achieved by training a policy to output an initial noise and a residual action, which correspond to the input and output of the flow policy.\n2. The overall ideas and intuition of the method are clear.\n3. The paper demonstrate the performance of the proposed method with both simulation and real-world experiments.", "weaknesses": "1. A more comprehensive related work section will enhance the paper.\nFor example, recent papers on residual RL are not included, such as policy decorator[1].\nAnd some baselines compared in this paper are not introduced in the related work.\n2. Some details about the method are not clearly explained. Please check the questions in the following section.\n3. The chosen baselines, while reasonable, do not fully demonstrate the advantage of RFS over the state-of-the-art.\nA more compelling comparison would be against: (1) Recent RL methods specifically designed for fine-tuning diffusion/flow policies (beyond the ablated DSRL) (2) State-of-the-art residual RL methods that finetune a base policy, to better isolate the contribution of the combined approach.\n\nMinor:\n1. The title \"Imitation-Bootstrapped Reinforcement Learning\" in related work might be slightly misleading. As defined in works like IBRL[2], the term has a specific meaning. The papers [3][4] cited under this heading are more broadly categorized as \"RL with Demonstration\". It would be helpful to refine this terminology for precision.\n\n[1] Zhiyuan Yuan, et al. \"Policy Decorator: Model-Agnostic Online Refinement for Large Policy Model.\" ICLR 2025.\n\n[2] Hengyuan Hu, et al. \"Imitation Bootstrapped Reinforcement Learning.\" RSS 2024.\n\n[3] Ashvin Nair, et al. \"Overcoming exploration in reinforcement learning with demonstrations.\" ICRA 2018.\n\n[4] Aravind Rajeswaran, et al. \"Learning complex dexterous manipulation with deep reinforcement learning and demonstrations.\" RSS 2018.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)