Rapidly Adapting Policies to the Real-World via Simulation-Guided Fine-Tuning
Authors: Patrick Yin
Tyler Westenbroek
Ching-An Cheng
Andrey Kolobov
Abhishek Gupta
Venue: ICLR 2025 Poster (2025)
Keywords: Robot Learning
Reinforcement Learning
Fine-Tuning
Primary area: applications to robotics, autonomy, planning
TL;DR: We use value functions trained in simulation to guide efficient exploration for efficient real-world finetuning, with robot hardware and theoretical results

Abstract: Robot learning requires a considerable amount of high-quality data to realize the promise of generalization. However, large data sets are costly to collect in the real world. Physics simulators can cheaply generate vast data sets with broad coverage over states, actions, and environments. However, physics engines are fundamentally misspecified approximations to reality. This makes direct zero-shot transfer from simulation to reality challenging, especially in tasks where precise and force-sensitive manipulation is necessary. Thus, fine-tuning these policies with small real-world data sets is an appealing pathway for scaling robot learning. However, current reinforcement learning fine-tuning frameworks leverage general, unstructured exploration strategies which are too inefficient to make real-world adaptation practical. This paper introduces the \emph{Simulation-Guided Fine-tuning} (SGFT) framework, which demonstrates how to extract structural priors from physics simulators to substantially accelerate real-world adaptation. Specifically, our approach uses a value function learned in simulation to guide real-world exploration. We demonstrate this approach across five real-world dexterous manipulation tasks where zero-shot sim-to-real transfer fails. We further demonstrate our framework substantially outperforms baseline fine-tuning methods, requiring up to an order of magnitude fewer real-world samples and succeeding at difficult tasks where prior approaches fail entirely. Last but not least, we provide theoretical justification for this new paradigm which underpins how SGFT can rapidly learn high-performance policies in the face of large sim-to-real dynamics gaps.

Published as a conference paper at ICLR 2025
RAPIDLY ADAPTING POLICIES TO THE REAL WORLD VIA SIMULATIONGGUIDED FINE-TUNING
Patrick Yin". Tyler Westenbroek +1
Ching-an Cheng", Andrey Kobolov² & Abhishek Gupta'
RI
Figure 1:
Five dynamic, contact-rich manipulation tasks hammering (top left), insertion (right), and three pushing (bottom left) tasks solved in the real world using SGFT.
ABSTRACT
Robot learning requires a considerable amount of high-quality data to realize the promise of generalization. However, large data sets costly to collect in the real world. Physics simulators can cheaply generate vast data sets with broad cov- fun-
erage over states, actions, and environments. However, physics engines damentally misspecified approximations to reality. This makes direct zero-shot transfer from simulation to reality challenging, especially in tasks where precise and force-sensitive manipulation is necessary. Thus, fine-tuning these policies with small real-worid data sets an appealing pathway for scaling robot learn- ing. However, current reinforcement learning fine-tuning frameworks leverage general, unstructured exploration strategies which are too inefficient to make real- world adaptation practical. This paper introduces the Simulation- Guided Fine- tuning (SGFT) framework, which demonstrates how to extract structural priors from physics simulators to substantially accelerate real-world adaptation. Specifi- cally, our approach uses a value function learned in simulation to guide real-world exploration. We demonstrate this approach across five real-world dexterous nipulation tasks where zero-shot sim-to-real transfer fails. We further demonstrate framework substantially outperforms baseline fine-tuning methods, requiring to an order of magnitude fewer real-world samples and succeeding at difficult tasks where prior approaches fail entirely. Last but not least, provide theoret- ical justification for this new paradigm which underpins how SGFT can rapidly learn nigh-performance policies in the face large sim-to-real dynamics gaps. Project webpage: weirdlabuw, weeidlabu.ggihhub.ooggg
"qqual contribution 1. University of Washington Mierosoft Research

---

Published as a conference paper at ICLR 2025
1 INTRODUCTION
Robot learning offers a pathway to building robust, general-purpose robotic agents that can rapidly adapt their behavior to new environments and tasks. This shifts the burden from designing task- specific controllers by hand to the problem of collecting large behavioral datasets with sufficient coverage. This raises a fundamental question: how do we cheaply obtain and leverage such datasets at scale? Real-world data collection via teleoperation (Walke et al., 2023] team 2024) can gen- erate high-quality trajectories but scales linearly with human effort. Even with community-drin teleoperation (Mandlekar et al. 2018 Collaboration 2024), current robotics datasets are orders of magnitude smaller than those powering vision and language applications.
Massively parallelized physics simulation (Todorov et al., 2012 Makoviychuk et al. 2021 can cheaply generate vast synthetic robotics data sets. Indeed, generating data sets with extensive cov- erage over environments, states, and actions can be largely automated using techniques such as automatic scene generation (Chen et al., 2024 Deitke et al.) (022), dynamics randomization (Peng) jet al. 2018 Andrychowicz et al. 2020), and search algorithms such as reinforcement learning.
Unfortunately, cemuulation-geerrate data is not a silver bullet, as it provides cheap but ultimately off- off-domain data. Namely, simulators are fundamentally misspecified approximations to reality. This is highlighted in tasks like hammering in nail, where the modeling of high-impaet, deformable con- tact remains an open problem (Acosta et al.) 2022 Levy et al.] (024). In these regimes, no choice of parameters for the physics simulator accurately capture the real-world dynamics. This gap persists despite efforts towards improving existing physics simulators with system identification (Memmel et al. 2024 Huang et al., 2023: Levy et al., 2024). Thus, despite impressive performance for many tasks, methods that transfer policies from simulation to reality zero-shot (Kumar et al., 2021 Lee et al. 2020 Peng et al. 2018 Andrychowicz et al.] 2020) run into failure modes when they en- counter novel dynamics not covered by the simulator (Smith et al.) 2022b).
The question becomes: can inaccurate simulation models be useful in the face of fandamen- tal misspecifications? A natural technique is fine-tune policies pre-trained in simulator using real-world experience (Smith et al. 20226] Zhang et al. (003).. This approach can overcome mis- specification by training directly on data from the target domain. However, existing approaches typically employ the unstructured exploration strategies used by standard, general reinforcement learning algorithms Haarnoja et al. (2018). As result, current RL fine-tuning frameworks remain too ineffiienef for real-world deployment. We argue the following: despite getting the finer details wrong, physics simulators capture the rough structure of real-world dynamics well enough to transfer targeted exploration and adaptation strategies from simulation to reality.
This motivates the Eimulation-Guidee Fine-Tuning (SGFT) framework, which uses value func- tion Vsim learned in the simulator to transfer behavioral priors from simulation to reality. Our key insight is that the ordering defined by Vsim captures successful behaviors such as reaching towards and object, picking it, and moving it to a desired position - which are invariant across simulation and reality, even if the low-level dynamies diverge substantially in the two domains. In more detail, SGFT departs from standard finetuning strategies by 1) using Vsim to synthesize dense rewards to guide targeted real-world exploration and 2) shortening the learning horizon when adapting in the real-world. Prior approaches [Smith et al. (2022b); Zhang et al. (2023) optimize the same infinite- horizon objective in both simulation and reality, and thus only use the simulator to initialize real- world learning. These approaches often suffer from catastrophic forgetting Wołczyk et al. (2024), where there is a substantial drop in policy performance early in the fine-tuning phase. Incorporating Vsin into the reward enables the agent to retain structural information about the optimal behaviors learned in simulation throughout the fine-tuning tne-ttunng processes, while shorter learning horizons are well- known to yield more trackable policy optimization problems Westenbroek et al. (2022); Cheng al. 2011 Altogether, SGFT provides a stronger learning signal which enables base policy optimiza- tion algorithms to consistently and rapidly improve real-world performance.
Although SGFT is a very general framework for sim-to-real fine-tuning, we place a special emphasis on implementations which use sample efficient model-based reinforcement learning algo- rithms Janner et al. 2019);; Hansen et al. (2024). Model-based approaches typically struggle with long-horizar tasks due to compounding errors in model predictions Janner et al. (2019), but our Morizon-ssortenn strategy conveniently side-steps this challenge, fully unlocking the performance benefits promised by generative world models. We outline contributions follows:
2

---

Published as a conference paper at ICLR 2025
Large Simulated Datacet
Train value function
Efficient Real World Learning
F, = r, + YV (5 1 V Vimm(s) (s,)
V
sim
V AiM
Small Real World Dataset
Train dynamios model
P
Figure 2: Depiction of a model-hased instantiation of SGFT. While prior approaches optimize the same chal- lenging infinite-horizon objective during simulation pretraining and real- woal-world line-tuning, SGFT modifies the fine-tuning objective by a) using the value function learned in simulation Vaim reshape rewards and guide efficient yeal-world exploration b) shortening the search horizon to make real-world learning more tractable. For the model-based instantiation depicted (and outlined in Section 4.2). this amounts to branching short, exploratory model-based rollouts from rcal-world trajectories to scarch for sequences of actions which will improve policy performancc. Intuitively, SGPT uses the simulator to approximately bootstrap long-horizon rewards (via Veim), while small amounts of real-world data used search for short sequences of optimal actions under real-world dynamics.
1. We introduce the SGFT framework, which requires modifying only a few lines of code in existing sim-to-real RL finetuning frameworks /mmthh et al. (2022b); /Zhang al. (2019) and can be built on top of any base policy optimization algorithm.
2. We implement two model-based instantiations of SGFT and demonstrate through five contact-rich, sim-to-real robot manipulation experiments that SGFT' provides substantial sample complexity gains over existing fine-tuning methods.
3. We demonstrate theoretically how a) SGFT can learn highly performant policies, despite the bias introduced by the SGFT objective and b) how this enables SGFT to overcome compounding errors which plague standard MBRL approaches.
Together, these insights underscore how SGFT effectively leverages plentiful off-domain data from a simulator alongside small real-works data sets to substantially accelerate real-world fine-tuning.
2 RELATED WORK
-ottiattiiont---aiiii Transfer. Two main approaches have been proposed for overcoming the dynamics gap between simulation and reality: 1) adapting simulation parameters real-world data (Chebotar et al. 2019) Ramos et al.) 2019) Memmel et al. (2024) and 2) learning adaptive or robust
policies to account for uncertain real-world dynamies (Qi et al. 2022 Kumar al. 202 Yu et al., 2017). However, these approaches still display failure modes in regimes where simulation and
reality diverge substantially (Smith et al. (2022a) namely, where no set of simulation parameters closely match the real-world dynamics.
Adapting Policies with Real-World Data. Many RL approaches have focused the general fine-tuning problem (Rajeswaran et al.) 2018 Nair et al. 2020) Kostrikov et al. 2021 Hu et al.) 2023 Nakamoto et al.) 2024). These works initialize policies, Q-functions, and replay butfers from offline data and continue training them with standard RL methods. A number of works have specif- ically considered mixing simulated and real data during policy optimization either through co- training (Tome et al. (2024). simply initializing the replay-buffer with simulation data (Smith et al.)
2022a Ball al. (023), or adaptively sampling the simulated dataset and up-wei up-weighting transitions that approximately match the true dynamics (Eysenbach et al. 2020 Liu et al.) 2022. Xu et al. 2023) Niu et al. 2022). However, recent work Zhou et al. (2024) has demonstrated that there are minimal benefits to sampling off-domain samples when adapting online in new environments,
this can bias learned policies towards sub-optimal solutions. In contrast to these prior approaches which primarily simulated experience initialize real-world learning we focus distilling effective exploration strategies from simulation, using Vsim to guide real-world learning. We demonstrate theoretically that this approach to transfer leads to low bias in the policies learned by SGFT.
3

---

Published as a conference paper at ICLR 2025
Model-Based Reinforcement Learning. A significant body of work on model-based RL learns
a generative dynamics models to accelerate policy optimization (Sutton 1991 Wang & Ba 2019
Janner et al.] 2019: Yu et al.] 2020; Kidambi et al. 2020, Ebert et al., 2018: Zhang et al. 2019).
In principle, a model enables the agent to make predictions about states and actions not contained new situations. However, the central challenge
in the training data, enabling rapid adaptation for model-based methods is that small inaccuracies in predictive models can quickly compound over time, leading to large model-bias and a drop in controller performance. An effective critic can be used to shorten search horizons (Hansen et al. 2024 Bhardwaj et al. [0200] Hafner et al.) 2019 Dadbabaie et al., 20011 Grune& & Rantzer] 2008) yielding easter decision decision-making problems, but learning such critic from scratch can still require large amounts of on-task data. We demonstrate
that, for many real-world continuous control problems, critics learned entirely simulation can be robustly transferred to the real-world and substantially accelerate model-based learning.
Reward Design in Reinforcement Learning. A significant component of our methodology is learn- ing dense shaped reward in simulation to guide real-world fine-tuning. Prior techniques have tried to infer rewards from expert demos (Ziebart al., 20088] Ho & Ermon 2016). success examples (Fu
et al. 2018 Li et al. 2021). LLMs (Ma et 2003] Yu et 20023) and heuristics (Margolis et al.
2024 Berner al., 2019). We rely on simulation to provide reward supervision Westenbrock et al.
(2022) using the PBRS formalism (Ng et al., 1999). This effectively encodes information about the
dynamics and optimal behaviors in the simulator, enabling us to shorten the learning horizon and
improve sample efficiency Cheng et al. (2021); Westenbroek et al. (2022).
PRELIMINARIES
Let S and A be state and action spaces. Our goal is to control a real-world system defined by an unknown Markovian dynamics s' ~ (-ea a). where s, E S are states and a E A is an action. The usual formalism for solving tasks with RL is to define a Markov Decision Process of the form M, = (S, A. Preat, Preat r; 77 with initial real-world state distribution Preat reward function r, and discount factor 7 E [0, 1). Given policy n, we let (5) denote the distribution over trajectories 7 = (9o, 40 81, ai....) generated by applying т starting at initial state so. Defining the value function under as V (s) Es,rdd (s) 1. (s)),, our objective is to find i f sup, Egrpp (s)). We define the optimal value function as (s) == sup, V (s).
Unfortunately, obtaining good approximation to Treal using only real-world data is often im- practical. Thus, many approaches leverage an approximate simulation environment ~ Pesm (s, a) and solve an approximate MDP of the form Maam == (S, A. Psim: Patme T, 7) to train a policy Teim meant to approximate म real We let Vsim denote Tsim 's value function with respect to Msim- Here, Paim is the distribution over initial conditions in the simulator.
4 SIMULATION-GUDDE
FINE-TUNING
We build our framework around the following intuition: even when the finer details of preaf and psim differ substantially, we can often assume that Teim captures the rough motions needed to complete a task in the real world (such as swinging a hammer towards a nail). Specifically, we hypothesize that the ordering defined by Vsim captures these behaviors in a form that can be robustly transferred from simulation to reality and used to guide efficient real-world adaptation.
4.1 SIMULATION- GUIDED FINE-TUNING
When fine-tuning sii to the real world we propose a) reshaping the original reward function ac- cording to r(s) - F(s, s') = rss) + +Vii - Vsim (s) and b) shortening the search horizon to more tractable H-step objective E F(st, The reshaped objective is an instance of the Potential-Basee Reward Shaping (PRBS) formalism (Ng et al. 1999). where Vsam is used as the potential function. This reshaping approach is typically applied to infinit-hhrrizon objectives, where it can be shown that the optimal policy under F(s, s') is the same as the optimal policy under the original reward r(s) (Ng et al. 1999). contrast, the finite horizon search problem biases the ob- jective towards behaviors which were successful in the simulator. Indeed, by telescoping out terms we can rewrite our policy optimization objective as:
H
R (8) = E " Vsim (8H) +
Σ 1=0
>'r(s)) - Veim so ar ~ TH.((1ss)
(I)
4

---

Published as a conference paper at ICLR 2025
(s) sup VH" (s), (s. : (s))(ss)) (sV r(s)]-
Here TH H H........ - denotes H-step dependent policies, where TH the policy applied at time t. We emphasize that these H H-step returns are optimized over the real-world dynamics. We propose learning policy which optimizes these H- H-step returns from each state:
TH (-ss) t sup QH (s,
x
What behavior does this objective elicit? For each H the H-step Bellman equation dictates that the optimal policy Treal for the original MDP Mrrea can be found by solving:
H-1
real (-1s) sup E
H1 V Viea (5H)
E xr((s)) V Vieat( so =,a n T(11s) t=0
Thus, [ effectively uses V Viim as surrogate for Vireal Namely, ① uses the simulator to bootstrap long-horizon returns, while real-world interaction is optimized only over short trajectory segments. In the extreme case where H namely, the policy search problem will be reduced to contextual bandit problem. As we take H + 00, TH will optimize purely real long-horizon returns and thus recover the behavior of real"
मi will greedily attempt to increase Vsim at each timestep;
We are particularly interested in optimizing this objective with small values of H. This provides an ideal separation between what learned using cheap, plentiful interactions with Psim and what is learned with more costly interactions with Preal. In simulation, we can easily generate enough data to explore many paths through the state space and discover which motions lead to higher returns (e.g. swinging hammer towards a nail). This information is distilled into Vsim during the learning process, which defines an ordering over which states are more desirable to reach H steps in the future. By optimizing the reshaped objective Equation (T) for small values of H, only need to learn short sequences actions which move the real-world system to states where Vaim higher. Intuitively, (I) learns where to go with large amounts of simulated data and how to get there with small amounts of real-world data, efficiently adapting म sim to the real-world dynamcis.
Connections to the MPC Literature: Note that the - Vaai (80) term in ① does not depend on the choice of policy, and thus does not affect the choice of optimal policy. Thus, (I) is equivalent to the planning objective used by model predictive control (MPC) methods (Jadbabaie et al. 2001 Hansen et 2024, Sun et al.) 2018: Bhardwaj et al., 2020) with H-step look-ahead and a terminal reward of V si (assuming oracle access to the real-world dynamics). Of course, we cannot calculate the optimal MPC policy " directly because we do not know Preat, and thus seek to approximate its behavior by learning from real-world interactions.
The Simulation-Cuided Fine- Tuning Training Loop. We pro- pose the general Sinulation-Gtided Fine-Tuuning (SGFT) framework in the pseudo-code in Algorithm [ SGFT fine-tunes Taim to succeed under the real-world dynamics by iteratively 1) unrolling the current policy to collect transitions from Preat and 2) using the current dataset D of transitions to approximately optimize π <- max Q' (s, T(;)))
Algorithm iituulatioo-Guided Fine-tuning ( (SGFT)
Require: Pretrained policy Taim and value function Vssm π f- siim D +
2: for each iteration k do
for time step t= 1, ....,, do
4:
a, T--sst)
5:
Observe the state 8141 and the reward r..
6:
+ +- r + Veam ($1+1) Veim(s))
7:
DU {(5,, FI, S141)
at each state 8y the agent has visited. 8:
end for
By optimizing ( SGFT optimizes policies towards the actions taken by The Note that this framework can be built on top of any base policy optimization method; however, over the next two sections will argue that this framework is particularly beneficial when paired with mosel-based search strategies.
9: Approx. optimize π <- max, Qn π ((s)))
using transitions in D, at all observed states 8y-
10: end for
4.2 LEVERAGING SHORT MODEL ROLL-OUTS
Learning a generative model و for Preat with the real-world data set D enables an agent to generate synthetic rollouts and reason about trajectories not contained in the data set. In principle, this should
5

---

Published as a conference paper at ICLR 2025
substantially accelerate the learning of effective policies. The central challenge for model-basee reinforcement learning is that small errors in p can quickly compound over multiple steps, degrading the quality of the predictions Janner et al. (2019). As a consequence, learning model accurate enough to solve long-horizon problems can often take as much data as solving the task with modern model-free methods (Chen et al., 202T: Hiraoka et al. [021]. By bootstrapping Vsim in simulation, where data is plentiful, the SGFT framework enables agents to act effectively over long horizons using only short, local H[-step predictions about the real-world dynamies. This side-steps the core challenge for model- -ased methods, and enables extremely efficient real-world learning. We now outline how the SGFT can be built on top of two predominant classes of MBRL algorithms.
Notation. In what follows, we will use 'hat' notation denote H-step
Algorithm Dyna-SGFT
returns of the form ① under the transitions generated by the model p rather than the real dynamics Preal- Namely, V is the H-step value of policy म under ÿ: Vi and Qj the optimal values. Similarly, H de- notes the optimal H-step policy un- der the model dynamics. Note that this simply the MPC policy gen-
Require: Pretrained policy Tsim and value function Vaim
1: π Raim
2: for each iteration k do
3: Generate rollout {(5,, re, under
4:
F < Ti + (8t+1) 1)- Vsim (8t)
5:
D <- DU (8t, at, Tt, 8e+1)
6:
Fit generative model á with D.
for G policy updates do
Generate synthetic branched rollouts D under n. Approx. optimize ग + max, Q²H (s. (ss))) Vsy DD using augmented dataset D U D
erated by using ý and optimizing T with á substituted for Preat.
9:
10: end for
Improved Sample Efficiency with Data Augmentation (Algorithm 2 The generative model 인 can be used for data augmentation by generat- ing dataset of synthetic rollouts D to supplement the real-world dataset
11: end for
Algorithm 3 MPC-SGFT
Require: Pretrained value Vaim and initialized model p.
I: for each iteration k do
D (Janner et al., 2019; Sutton [990; Gu et al.) 20166. The combined dataset can then be fed to any policy
2:
Generate rollout {(s,, Te, (= using and trajectory optimization to calculate Aij
Te,
3:
F. <- Pt + Vein (St+1) Vsim (st)
optimization strategy, such as generie
4:
D (St, az Ft, $-+1)}
We con- sider state-of-the-art Dyna-style algo- rithms (Janner et al.] 2219),, which, in our context, branch H-step rollouts from states the agent has visited previously in the real-world. As Algorithm[ljshowe after each data-collection phase, this approach updates ğ and then repeatedly a) generates a dataset of synthetic H H-step rollouts under the current policy π starting from states in D, and b) approximately solves π < max; Qi (s, (s)) at the observed real-world states using the augmented dataset D UUD DD and base model-free method such as SAC (Haarnoja et al. 2018).
model-free algorithms.
5:
Fit generative model ý with D.
6: end for
Online Planning (Algorithm 3) The most straightforward way to approximate the behavior of TH is simply to apply the MPC controllr ä generated using the current best guess for the dynamics p. Algorithm 3 provides general pseudocode for this approach. which iteratively 1) rolls out * (which is calculated using online optimization and ò (Williams et al. 2017)) then 2) updates the model on the current dataset of transitions D. This high-level approach encompasses a wide array of methods proposed in the literature, e.g., Ebert et al.] (2018) and iZhang et al. (2019). For the experiments in 6 we implemented this approach using the TDMPC-2 (Hansen et al. 2024) algorithm, which additionally learns policy prior to accelerate the trajectory optimization.
THEORETICAL ANALYSIS
The preceding discussions have covered how the dense rewards and horizon shortening strategy employed by SGFT can lead to efficient real-world adaptation. However this leaves the outstanding question: how does the bias introduced by the reward shaping and horizon shortening affect policy performance? ? Longer prediction horizons decrease the bias of the objective by relying more heavily on real returns, but lead less sample efficient adaptation Cheng et al.) (202): Westenbroek et al.

---

Published as a conference paper at ICLR 2025
Given these competing challenges, can expect SGFT to simultancously learn high- performing policies and adapt rapidly in the real-world?
2022).
We introduce a novel geometric analysis which demonstrates that SGFT can achieve both goals, even when there is a large gap between Psim and Preat- Specifically, we introduce mild technical conditions on the structure of Psim, Vsim and Preal which ensure that iir is nearly optimal for Mreal, even when short prediction horizons H are used. Our analysis builds on the following insight: even when there is large gap in the magnitude |Peim Preat (so between simulation and reality, it is reasonable to assume that Vsim still defines a reasonable ordering over desirable states under the real-world dynamics Preat. Namely, we argue that Vaem can capture the structure of motions that complete the desired task (such as reaching towards and object, picking it up, and moving it to desired location), even if the low-level sequences of actions needed to realize these motions differs substantially between simulation and reality (due e.g. to difficult-o--mode contact dynamics). We use the following formal definition to capture this intuition, which is from Cheng let al. (2019) and shares strong connections to Lyapunov Westenbroek et al. (2022); Grune & Rantzer (2008) and Dissipation Brogliato et al. theories from dynamical systems and control:
Definition 1. We say that Vsin is improvable with respect to Mreal for each s E S we have:
max E. (opraaaa ViVssi Vsim (5) -r(s).
To unpack why this definition is useful, note that Vsim is by definition improv- able with respect to Main- Indeed, by the temporal difference equation we have Ex'-paam ( s,a), amtaim (-1s))77 (eemm
Vscm (s) -r(s). Namely, Veim is constructed so that polices can greedily increase Vsim under Paim at each step and be guaranteed to reach desirable states in the long run, namely, states which correspond to successfully completing the task. Defini- tion Jensurss that this condition then holds under the real-world dynamics; this requirement ensures that the ordering defined by Vaim encodes feasible motions that solve the task in the real-world. This enables policy optimization algorithms to greedily follow Vaim at each step while ensuring that the resulting behavior successfully completes the task. We use the following pedagogical example to investigate why this is reasonable property assume for continuous control problems:
Pedagogical Example. Consider the following case where the real and simulated dynamics are both deterministic, namely, s' = Preat a) and s' = Peim a). Specifically, consider the case where s = (81, 82) ES c R2, a E = IR, and the dynamics are given by:
Paim (s, a) =
s1]
s1
+ At
$2 f sin(s)) +
$2
82
Preat ())
s1
+ At
82 I sin(s) + e(s), 92)
$2
These are the equations of motion for simple pendulum (Wang al. (022) under an Euler dis- cretization with time step At, where 81 is the angle of the arm, $2 is the angular velocity, is the torque applied by the motor, g is the gravitational constant, and / is the length of the arm. The real- world dynamics contains unmodeled terms e($1,8)) which might correspond to complex frictional or damping effects. Consider the policy for the real world given by (seal(s) = Taim (s) - e(s1. $2) and observe that Psim (s, (s)) = (seat(s, Treat (s)). Even though the difference in transition dynamics e(81, 82) might be quite large, the set of feasible next states the same for the two envi- ronments. That is, the space of feasible motions in the two MDPs are identical, even though it takes substantially different policies to realize these motions.
Geometric Insight. More broadly, if for each & there exists some a such that Preat (-l, a) (eim Tasm (s)), then Vaim is improvable with respect to Mreal- This follows from the fact that Vsim is improvable under Psim by definition (see above discussion) and the fact that there exists actions which exactly match the state transitions in simulation and reality. More generally, it is reasonable to expect that Psim approximately captures the geometry of what motions are feasible under Preat. even the actions required to realize the motions differ substantially in the two MDPs Thus it is reasonable to assume Vsim is improvable. This intuition is highlighted by our real-world examples in cases where we use SGFT with a prediction horizon of H = 1. In these cases the learned policy is able to greedily follow Vsan at each state and reach the goal, even in the face of large dynamics gaps. Relatedly, work on Lyapunov theory Westenbroek et al.) (2022) has demonstrated theoretically that value functions naturally robust dynamics shifts under mild conditions.
7

---

Published as a conference paper at ICLR 2025
Main Theoretical Result. Before presenting our main theoretical result, we provide useful point of comparison from the literature Bhardwaj et al. (2020). When translated to our setting, Bhard- waj et al., 2020 Theorem 3.1) assumes that V(s, a) we have a) the value gap between simulation and reality is bounded by |Vsin (s)
Vieaa (s) < and b) that a generative model ý is used for
planning wherein [|Ø(-|s, a. Preat (-1s, a < a. Recalling that #1 is the controller synthesized
using the model p. (Bhardwaj ct al. 2020 Theorem 3.1) bounds the suboptimality for the model-
S O (12aHH ( -I ) To understand the bound, first set a 0 so that there is no modeling error (as in the case of model-free instantiations of SGFT). In this regime we are incentivized to increase H, as this will decrease the 1-,,0 c term and improve performance. However this term scales very poorly for long-horizon problems where ? ~ 1, and may need large values of H to obtain near optimal policies. When a > O, the situation becomes even more challenging. Increasing И will increase the - term, reflecting how longer pre- diction horizons propagate model-basee errors and increase suboptimality. Thus, without additional structural assumptions, we cannot conclude that SGFT can learn performnre controllers when H is small. The following result demonstrates this is possible when Vsim is improvable:
based controller as Vica(()) V
(8)
real
Theorem 1. Assumes that ((s a) have a) the value gap between simulation and reality is bounded by (i)) - (s)] < c aand b) if a generative model P is used by SGFT then [pp(-|s, a) - Preat(-1s, a)Ilı < a. Further suppose that Vsim is improvable with respect Mreat Then for H sufficiently small and each 8 E S we have:
(ient(s) -Viaa S O (1 1I-
(3)
where #1 is the policy learned by SGFT.
See Appendix A for proof. This result demonstrates that when Vism is improvable with respect to MMreal SGFT can obtain nearly optimal policies, even when H is small. When compared to (Bhardwaj et al. 2020 Theorem 3.1), which assumes uniform worst-case bounds on [V - Vreeall, when Vsim improvable we gain a substantial factor of I- when bounding the effects of errors in Veim- In short, this result demonstrates that even mild structural consistency between Paim and Preat is enough to ensure that SGFT provides effective guidance towards performant policies when H is small. In the case of MBRL (a 0), this is especially important as keeping H small combats compounding errors in the dynamics model. Our proof technique adapts ideas from the theoretical control literature Grune & Rantzer (2008); Westenbroek al. (2022) to the more general setting we consider, and can be seen as a model- analogy to the results from Cheng et al. (2021).
EXPERIMENTS
We answer the following: (1) Can SGFT facilitate rapid online fine-tuning for dynamic, contact- rich manipulation tasks? (2) Does SGF7' improve the sample efficiency of fine-tuning compared to
baselines? (3) Can SGFT learn successful policies where prior methods fail entirely?
6.1 METHODS EVALUATED
SGFT Instantiations. We implement concrete instantiations of the general Dyna-SGFT and MPC-SGFT frameworks sketched in Algorithms 2 and 3 SGFT-SAC fits model to real world transitions to perform data augmentation and uses SAC as a base model-free policy optimization algorithm. We use H 1 in all our experiments. Crucially, we set the 'done' flag to true at the end of each rollout- this ensures SAC does not bootstrap its own critic from the real-world data and only uses Vsam to bootstrap long-horizon returns. SGFT-TDMPC-2 uses TDMPC-2 Hansen et al. (2024) as a backbone. The base method learns a critic, a policy, and an approximate dynamics model through interactions with the environment. When acting in the world, the MPC controller solves online planning problems using the approximate model, the critic as a terminal reward, and uses the policy prior to seed an MPPI planner Williams et al. (2017). To integrate this method with SGFT, when transferring to the real world we simply freeze the critic learned in simulation and use the reshaped objective in Equation II as the online planning objective. For our experiments, we use H = 4 and the default hyperparameters reported in Hansen et al. (2024).
We note that Bhardwaj et al. 2020) focuses on general MPC problems where some approximation V m Vitat is used as the terminal cost for the planning objective. However, despite the difference in settings, the structure of the underlying H-step objective we consider is identical much of the MPC literature.
8

---

Published as a conference paper at ICLR 2025
Hammering Success Rate
Inseriion Success Rave
1.0
08
SOFT-TOMPCZ (Ours)
Doeran Handomization
05
SOFT-SAC (Ouns)
Recument Pollcy
Rewrneee
6.
PaRS SAG
TOMPC2
Doe Randomiation
ASID
0-4
Asymmmaric Actar-Crtic:
DRDPO
Rumnn
IOL
DOREAMON
20
100
125
190
175
200
Number Real World Rollouts
Number of Wool Roliouts
Puck Pushing Success Rate
Buil Pusliing Successs Rele
Clith Pashinn Success Rale
Furning Averagg Succooen Rale
1.0
0.5
0.0
0.3
04
0.8
0.4
Poarring
02-
100
00
10
Number RRoll-Worrd Rollouts
Number of Worl Rollouts
Number of Wood-Worddddd Roilouts
Figure 4: Real-world success rates during the course online fine-tuning. We plot task success rates We see that SGFT yields significant
over number of fine-tuning rollouts for the tasks described in Sec. 6 improvements in success and efficiency.
Baseline Fine-tuning Methods. We also fine-tune polices using the original aafinite-horizon ob- jective for Myeat, including SAC [Haarnoja et al. (2018). TDMPC-2 Hansen et al. (2024). IQL. Kostrikov et al. (202I). PBRS fine-tunes the policy using the reshaped reward, an infinite horizon, and SAC (Haamoja et al. (2188) does not use horizon shortening. RLPD fine-tunes a fresh policy to solve the original MDP Mreal using RLPD (Ball al. 2023). These algorithm cover state-of-tha art model-based, model-free, and offtine-tor Ceffine-to-online adaptation algorithms.
Baseline Sim-to-Real Methods. Our Domain Randomization baseline refers to policies trained with extensive domain randomization in simulation and transferred directly to the real world. These policies rely only on the previous observation. Recurrent Policy + Domain Randomization uses policies conditioned on histories of observations, similar methods such as Kumar al.) (202T). Asymmetric Actor-Critic (Pinto et al.) (2018) uses policies trained in simulation with critic con- ditioned on privileged information. DOREAMON (Tiboni et al.) (023b) is a recently proposed transfer method which automatically generates curricula to enable robust transfer. ASID (Memmell et al, 2024) and DROPO (Tiboni et al., 2023a) identify simulation parameters with small real-world data sets. These methods serve as state-offthe-art baselines for sim-to-real transfer methods.
6.2 SIM-TO- REAL EVALUATIONS
We test each method on five real-world manip- ulation tasks illustrated in Figure and two ad- ditional real-world deformable object pushing tasks illustrated in Figure [ demonstrating that both the SGFT-SAC and SGFT-TDMPCC in- stantiations of SGFT excel learning policies with minimal roal-world data.
Simmlntion
Real World
Hammering is highly dynamic task involving force and contact dynamics that are impractical to precisely model in simulation. In our setting, the robot is tasked with hammering a nail board. The nail has high, variable dry friction along its shaft. In order to hammer the nail into the board, the robot must hit the nail with high force repeatedly. The dynamics are inherently misspecified between simulation and reality here due to the infeasibility of accurately modeling the properties of the nail and its contact interaction with the hammer and board.
Figure 3: Sim-to-Real Setup Simulation setup for pre- training (top) and execution of real-world finc-tuning (bottom) of real-world hammering (left), insertion (middle), and pushing (right).
Insertion (Heo al. 2023) involves the robot grasping a table leg and accurately inserting it into a table hole. The contact dynamics between the leg and the table differ between simulation and real-world conditions. In the simulation, the robot successfully completes the task by wiggling the

---

Published as a conference paper at ICLR 2025
leg into the hole, but in the real world this precise motion becomes challenging due inherent noise in the real-world observations as well as contact discrepancies between the leg and the table hole. Puck Pushing requires pushing a puck of unknown mass and friction forward to the edge of the table without it falling off the edge. Here, the underlying feedback controller of the real world robot inherently behaves differently from simulation. Additionally, retrieving and processing sensor information from cameras incurs variable amounts of latency. As a result, the controller executes each commanded action for variable amounts of time. These factors all contribute to the sim-to-real dynamics shift, requiring real-world fine-tuning to reconcile.
Deformable Object Pushing is similar to puck pushing but involves pushing deformable objects which are challenging to model in simulation. As result, we choose to model the object as puck in simulation, making the simulation dynamics fundamentally misspecified compared to the real world. This example exemplifies how value functions trained under highly different dynamics in simulation can still provide useful behavioral priors for weal-world exploration. We include two deformable pushing tasks towel and a squishy toy ball.
Each task is evaluated on a physical setup using a Franka FR3 operating with either Cartesian position control or joint position control at 5Hz. We compute object positions by color-thresholding pointclouds or by Aruco marker tracking, although this approach could easily be upgraded. Further details on reward functions, robot setups and environments can be found in Appendix B
Analysis. The results of real-world fine-tuning on these five tasks are presented in Figure 4 For all five tasks, zero-shot performance is quite poor due to the dynamics sim-to-real gap. The poor performance of direct sim-to-real transfer methods such as Domain Randomization, Recurrent Pol- icy + Domain Randomization (Kumar et al.] (021)). Asymmetric actor-Critic (Pinto et al. 2018))
ASID (Memmel et al., 2024). DROPO (Tiboni al. 2023a), and DOREAMON (Tibomi et al. 2023b) highlight that these gaps are due to more than parameter misidentification or poor policy training in simulation, but rather stem from fundamental simulator misspecification.
The second class of comparison methods includes offline pretraining with online fine-tuning techniques like IQL (Kostrikov al. (021), SAC (Haarnoja et al.] 2018), and RLPD (Ball al., 2023). Whether model-free or model-based, the SGFT finetuning methods (ours) substantially out- perform these techniques in terms of efficiency and asymptotic performance. Moreover, they prevent catastrophic forgetting, wherein finetuning leads to periods of sharp degradation in the policies ef- fectiveness. This suggests that simulation can offer more guidance during real- world policy search than just network weight initialization and/or replay buffer data initialization for subsequent fine- tuning. Our full system consistently leads to significant improvement from fine-tuning, achieving 100% success for hammering and pushing within an hour of fine-tuning and 70% success for insert- ing within two hours of fine-tuning. The fact that SGFT outperforms both TD-MPC2 (Hansen et al., 2024] and PBRS- rollouts and value-driven reward shaping. Last but not least, note that SGFT offers improvements on top of both SAC and TDMPC2, showing the generality of the proposed paradigm.
suggests that efficient fine-tuning requires combination of both short model
In the Appendix we also include a) set of sim-to-sim transfer benchmarks and ablations (Appendix E) b) visualizations of transferred value functions (Appendix D
LIMITATIONS AND FUTURE WORK
We present SGFT, a general framework for efficient sim-to-real fine-tuning with existing RL algo- rithms. The key idea to leverage value functions learned in simulation to provide guidance for real-world exploration. In the future it will be essential to scale methods to work directly from per- ceptual inputs. Calculating dense rewards from raw visual inputs is challenging, and represents an important limitation of the current instantiation of SGFT. Future work will investigate which visual modalities lead to robust sim-to-real transfer. Moreover, despite the sample complexity gains SGFT affords existing RL algorithms, we believe new base adaptation algorithms designed specifically for low-data regimes will be needed to make truly autonomous real world adaptation practical. For all methods tested, we found that tuning for real-world performance was consuminn, Thus, future work will investigate using large simulated data sets to distill novel policy optimization algorithms that can be transferred to the real-world with less effort.
10

---

Published as a conference paper at ICLR 2025
8 ACKNOWLEDGMENTS
We would like to thank Marius Memmel, Vidyaaranya Macha, Yunchu Zhang, Octi Zhang, and Arhan Jain for their invaluable engineering help. We would like to thank Lars Lien Ankile and Marcel Torne for their fruitful engineering discussions and constructive feedbacks. Patrick Yin is supported by sponsored research with Hitachi and the Amazon Science Hub.
REFERENCES
Brian Acosta, William Yang. and Michael Posa. Validating robotics simulators on real-world im- pacts. IEEE Robotics and Automation Letters, 7(3):6471-64777 2022.
OpenAl:
Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020.
Philip Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learn- ing with offline data. In International Conference on Machine Learning, pp. 1577-1594. PMLR, 2023.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Christopher Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique Pondé Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota with large scale deep reinforcement learning. CORR, abs/1912.06680, 2019. URL http: //t:xiva..iv..... org/ abs 1 1912. 1912.06680
Mohak Bhardwaj. Sanjiban Choudhury, and Byron Boots. Blending mpc & value function approxi- mation for efficient reinforcement learning. arXiv preprint urXiv:2012.05909 2020.
Bernard Brogliato, Rogelio Lozano, Bernhard Maschke, Olav Egeland, et al. Dissipative systems analysis and control.
Yevgen Chebotar, Ankur Handa, Viktor Makoviychuk, Miles Macklin, Jan Issac, Nathan D. Ratliff, and Dieter Fox. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. In ICRA, 2019.
Xinyue Chen, Che Wang, Zijian Zhou, and Keith Ross. Randomized ensembled double q-learning: Learning fast without model. arXiv preprint arXiv:2101.05982, 2021.
Zoey Chen, Aaron Walsman, Marius Memmel, Kaichun Mo. Alex Fang, Karthikeya Vemuri, Alan Wu, Dieter Fox. and Abhishek Gupta. Urdformer: A pipeline for constructing articulated simula- tion environments from real-world images. arXiv preprint 1 2024.
Ching-An Cheng, Xinyan Yan, Nathan Ratliff, and Byron Boots Predictor-corrector policy opti- mization. In International Conference on Machine Learning, pp. 1151-1161. PMLR, 2019.
Ching-An Cheng, Andrey Kolobov, and Adith Swaminathan. Heuristic-guided reinforcement learn- ing. Advances in Neural Information Processing Systems, 34:13550-13563, 2021.
Open-X- Dren-X-Embodimnn Collaboration. Open x-embodiment: Robotic learning datasets and rt-x mod- cls. ICRA, 2024.
Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Kiana Ehsani, Jordi Salvador, Winson Han, Eric Kolve, Aniruddha Kembhavi, and Roozbeh Mottaghi. ProcTHOR: Large-scale embodied AI using procedural generation. In Advances in Neural Infor- mation Processing Systems 35: Anmual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA. USA, November 28 December 9, 2022, 2022. URL http:ttt////peeeeaiiiie ptppppprrrraippppppppaa.eeeee

---

Published as a conference paper at ICLR 2025
Yuqing Du, Olivia Watkins, Trevor Darrell, Pieter Abbeel, and Deepak Pathak. Auto-tuned sim-to- real transfer. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp- 1290-1296. IEEE, 2021.
Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine. Visual fore- sight: Model-based deep reinforcement learning for vision-based robotic control. arXiv preprint 1812.008100 2018.
Benjamin Eysenbach, Swapnil Asawa, Shreyas Chaudhari, Sergey Levine, and Ruslan Salakhutdi- nov. Off-dynamics reinforcement learning: Training for transfer with domain classifiers, arXiv preprint 200.v.2000..99111 2020.
Justin Fu, Avi Singh, Dibya Ghosh, Larry Yang, and Sergey Levine. Variational inverse control with events: A general framework for data-driven reward definition. NeurlPS, 2018.
Lars Grune and Anders Rantzer. On the infinite horizon performance of receding horizon controllers. IEEE Transactions on Automatic Control, 53(9):2100-2111 2008
Shixiang Gu, Timothy Lillicrap, Ilya Sutskever, and Sergey Levine. Continuous deep q-learning with model-based acceleration. In International conference on machine learning, pp. 1829-2838.. PMLR, 2016.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In ICML, 2018. URL
httt::/pprocedding .aaes.mm...... mmlr press vv80 Ba1////hhaaeee hww.aaaaannnaaaaaaaaaaaaaaaaa abb...11..... htt.mmmmbm
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. International conference on machine learning, pp. 2555-2565. PMLR, 2019.
Nicklas Hansen, Hao Su, and Xiaolong Wang. TD-MPC2: Scalable, robust world models for contin- uous control. In The Twelfth International Conference on Learning Representations, 2024. URL
net www.weww.ouweeeeee/e///e wtt.orommmoxxxcstttttt
Minho Heo, Youngwoon Lee, Doohyun Lee, and Joseph J. Lim. Furniturebench: Reproducible real-world benchmark for long-horizon complex manipulation, 2023. URL httptt/s::/:/xxiv
org/aba/ alt/a/abb222..... 2305. 12821
Takuya Hiraoka, Takahisa Imagawa, Taisei Hashimoto, Takashi Onishi, and Yoshimasa Tsu- ruoka. Dropout g-functions for doubly efficient reinforcement learning. arXiv preprint arXiv:2110.02034, 2021.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
Hengyuan Hu, Suvir Mirchandani, and Dorsa Sadigh. Imitation bootstrapped reinforcement learn- ing. arXiv preprint arXiv:2311.021990 2023.
Peide Huang, Xilun Zhang, Ziang Cao, Shiqi Liu, Mengdi Xu, Wenhao Ding, Jonathan Francis, Bingqing Chen, and Ding Zhao. What went wrong? closing the sim-to-real gap via differentiable causal discovery. In Jie Tan, Mare Toussaint, and Kourosh Darvish (eds.), Conference on Robot Learning. CoRL 2023, 6-9 November 2023, Atlanta, GA, USA, volume 229 of Proceedings of Machine Learning Research, pp. 734-760 760. PMLR, 2023. URL htttps:/pproceediiii mlr. wttess//229hhaagggrr
html
Ali Jadbabaie, Jie Yu, and John Hauser. Unconstrained receding-horizon control of nonlinear systems. IEEE Transactions on Automatic Control, 46(5):776-783, 2001.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. Advances in neural information processing systems, 32, 2019.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. Advances in neural information processing systems, 33: 21810-21823, 2020.
12

---

Published as a conference paper at ICLR 2025
Ily Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. International Conference on Learning Representations, 2021.
Ashish Kumar, Zipeng Fu, Deepak Pathak, and Jitendra Malik. RMA: rapid motor adaptation for
legged robots. In RSS, 2021.
Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning quadrupedal locomotion over challenging terrain. Science robotics, 5(47)xeabc5986, 2020.
Jacob Levy, Tyler Westenbroek, and David Fridovich-Kar Keil. Learning to walk from three minutes of data with semi-structured dynamics models. In 8th Annual Conference on Robot Learning, 2024.
URL httphhttts///ppeeeeeeee htt.://wwwwweeeeeeeeeeeeeeeeeeee forum? wwmmmmdd=evviii 1=
Kevin Li, Abhishek Gupta, Ashwin Reddy, Vitchyr H Pong, Aurick Zhou, Justin Yu, and Sergey Levine. Mural: Meta-learning uncertainty-aware rewards for outcome-driven reinforcement learning. ICMI,, 2021.
Jinxin Liu, Hongyin Zhang, and Donglin Wang. Dara: Dynamics-aware reward augmentation in offline reinforcement learning. arXiv preprint arXiv:2203.06666 2022.
Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayara- man, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Fureka: Human-level reward design via coding large language models. arXiv preprint arXiv:2310.12931 2023.
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, and Gavriel State. Isaac gym: High performance gpu- spu-based physics simulation for robot learning. In NeurIPS-2021 Datasets and Benchmarks Track, 2021.
Ajay Mandlekar, Yuke Zhu, Animesh Garg, Jonathan Booher, Max Spero, Albert Tung, Julian Gao, John Emmons, Anchit Gupta, Emre Orbay, al. Roboturk: crowdsourcing platform for robotic skill learning through imitation. In Conference on Robot Learning, 2018.
Gabriel B. Margolis, Ge Yang, Kartik Paigwar, Tao Chen, and Pulkit Agrawal. Rapid locomo- tion via reinforcement learning. Int. J. Robotics Res., 43(4):572-587 2024. doi: 10.1177/ 02783649231224053.
Marius Memmel, Andrew Wagenmaker, Chuning Zhu, Patrick Yin, Dieter Fox, and Abhishek Gupta. ASID: active exploration for system identification in robotic manipulation. CoRR, abs/2404, 12308, 2024.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online rein- forcement learning with offline datasets. arXiv preprint (xXi::2006009359, 2020.
Mitsuhiko Nakamoto, Simon Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-ql: Calibrated offline rl pre-training for efficient online fine- tuning. Advances in Neural Information Processing Systems, 36, 2024.
Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. Icml, volume 99, pp. 278-287, 1999.
Haoyi Niu, Yiwen Qiu, Ming Li, Guyue Zhou, Jianming Hu, Xianyuan Zhan, et al. When to trust your simulator: Dynamics-aware hybrid offline-and-online reinforcement learning. Advances in Neural Information Processing Systems, 35:36599-3666 36612, 2022.
Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference robotics and automation (ICRA), pp. 3803-3810. IEEE, 2018.
Lerrel Pinto, Marcin Andrychowicz, Peter Welinder, Wojeiech Zaremba, and Pieter Abbeel. Asym- metric actor critic for image-bace based robot learning. In 14th Robotics: Science and Systems, RSS 2018. MIT Press Journals, 2018.
13

---

Published as a conference paper at ICLR 2025
Haozhi Qi. Ashish Kumar, Roberto Calandra, Yi Ma, and Jitendra Malik. In-hand object rotation via rapid motor adaptation. In CoRL, 2022. URL https://ppxoeeedngg mlr press/ v205/qi23a.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning Complex Dexterous Manipulation with Deep Reinforce- ment Learning and Demonstrations. In Proceedings of Robotics: Science and Systems (RSS), 2018.
Fabio Ramos, Rafael Possas, and Dieter Fox. Bayessim: Adaptive domain randomization via prob- abilistic inference for robotics simulators. In RSS, 2019.
Laura Smith, llya Kostrikov, and Sergey Levine. A walk in the park: Learning to walk in 20 minutes with model-free reinforcement learning. arXiv preprint arXiv:2208. 078660 2022a.
Laura M. Smith, J. Chase Kew, Xue Bin Peng. Sehoon Ha, Jie Tan, and Sergey Levine. Legged robots that keep learning: Fine-tuning locomotion policies in the real world. In ICRA, 2022b.
Wen Sun, Andrew Bagnell, and Byron Boots. Truncated horizon policy search: Combining rein- forcement learning & imitation learning. arXiv preprint arXiv.1805.112400 2018.
Richard Sutton. Integrated architectures for learning, planning, and reacting based on approxi- mating dynamic programming. In Machine learning proceedings 1990, pp. 216-224. Elsevier, 1990.
Richard S. Sutton. Dyna, an integrated architecture for learning, planning, and reacting, SIGART Bull., 2(4):160-163, jul 1991. ISSN 0163-5719. doi: 10.1145/1223444122377. URL https:
//601.org/10.1111522234
DROID Collaboration team. Droid: A large-scale in-the-wild robot manipulation dataset. In
Robotics Science and Systems (RSS), 2024.
Gabriele Tiboni, Karol Arndt, and Ville Kyrki. Dropo: Sim-to-real transfer with offline domain randomization. Robotics and Autonomous Systems, 166:104432, 2023a.
Gabriele Tiboni, Pascal Klink, Jan Peters, Tatiana Tommasi, Carlo D' D'Eramo, and Georgia Chal- vatzaki. Domain randomization via entropy maximization. arXiv preprint A.Niv:33/1008880 2023b.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In IROS, 2012.
Marcel Torne Anthony Simeonov, Zechu Li, April Chan, Tao Chen, Abhishek Gupta, and Pulkit Agrawal. Reconciling reality through simulation: A real-to-sim-to-real approach for robust ma- nipulation. CoRR, abs/2403.0300 2024. doi: 10.45500ARX1.24000.09999 URL https //doi. 0ww.4445555Xi. 2403. 2403.03949
Homer Walke, Kevin Black, Abraham Lee, Moo Jin Kim, Max Du, Chongyi Zheng, Tony Zhao, Philippe Hansen-Estruch, Quan Vuong, Andre He, Vivek Myers, Kuan Fang, Chelsea Finn, and Sergey Levine. Bridgedata v2: dataset for robot learning at scale. In Conference on Robot Learning (CoRL), 2023.
Jin Wang, Qilong Xue, Lixin Li, Baolin Liu, Leilei Huang, and Yang Chen. Dynamic analysis of simple pendulum model under variable damping. Alexandria Engineering Journal, 61(12): 10563-10575, 2022.
Tingwu Wang and Jimmy Ba. Exploring model-based planning with policy networks. arXiv preprint 1rXiv:1906.08644 2019.
Tyler Westenbroek. Fernando Castaneda, Ayush Agrawal, Shankar Sastry, and Koushil Sreenath. Lyapumov design for robust and efficient robotic reinforcement learning. arXiv:2208.0600 06721, 2022.
arXiv preprint
14

---

Published as a conference paper at ICLR 2025
Grady Williams, Andrew Aldrich, and Evangelos A Theodorou. Model predictive path integral control: From theory to parallel computation. Journal of Guidance, Control, and Dynamics, 40 (2):344-357, 2017.
Maciej Wotczyk, Barthomiej Cupial, Mateusz Ostaszewski, Michal Bortkiewicz, Michal Zajkac, Razvan Pascanu, Lukasz Kucinski, and Piotr Milo's. Fine-tuning reinforcement learning models secretly a forgetting mitigation problem. arXiv preprint arXiv:2402.02868, 2024.
Kang Xu, Chenjia Bai, Xiaoteng Ma, Dong Wang, Bin Zhao, Zhen Wang, Xuelong Li, and Wei Li. Cross-domain policy adaptation via value-guided data filtering. Advances in Neural Information Processing Systems, 36:73395-73421, 2023.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33:14129-14142, 2020.
Wenhao Yu, Jie Tan, C. Karen Liu, and Greg Turk. universal policy with online system identification. html
Preparing for the unknown: Learning RSS, 2017. URL http://www.
Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Scan Kirmani, Kuang- Huei Lee, Montse Gonzalez Are- nas, Hao-Tien Lewis Chiang, Tom Erez, Leonard Hasenclever, Jan Humplik, et al. Language to rewards for robotic skill synthesis. arXiv preprint (eXXi:230600864: 2023.
Marvin Zhang, Sharad Vikram, Laura Smith, Pieter Abbeel, Matthew Johnson, and Sergey Levine Solar: Deep structured representations for model-based reinforcement learning. In International conference on machine learning, pp. 7444-7453. PMLR, 2019.
Yunchu Zhang, Liyiming Ke, Abhay Deshpande, Abhishek Gupta, and Siddhartha S. Srinivasa. Cherry-picking with reinforcement learning. In RSS, 2023.
Zhiyuan Zhou, Andy Peng, Qiyang Li, Sergey Levine, and Aviral Kumar. Efficient online reinforce- ment learning fine-tuning need not retain offline data. arXiv preprint arXiv:2412.07762, 2024.
Brian D. Ziebart, Andrew Maas, J. Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. In Proceedings of the 23rd National Conference on Artificial Intelligence Volume 3, AAAI088, pp. 1433-1438. AAAI Press, 2008. ISBN 9781577353683.
15

---

Published as a conference paper at ICLR 2025
A PROOFS
Notation Recap. We remind the reviewer of notation we have built up throughout the paper. We use the 'hat' notation denote generative dynamics model p. as well that the optimal values V Q7 obtained by optimizing the H-step objective under these dynamics. dir is then the policy obtained by optimizing the H-step objective under the model dynamics: RN #n((s) t max, Q7 (a, т).
We first present several Lemma's 's used in the proof of Theorem [ The first result bounds the difference between the true H-step returns for a policy тH and the H H-step returns predicted under the dynamics model p.
Lemma 1. (Bhardwaj et al. suppose år max, r(3) min, r(s) and AV max, Vs(s) min, (s( are finit.. Then, for each policy π may bound the H step returns under the model and true dynamics by:
2020 Lemma A. Suppose that [[((s, (od)-Preacd -Preat(,, a)ll S a. Further
1- 1-1 Ar
AV
IV (+)
V 1oc 5
( - 2
o.H.
(4)
2
Proof. This result follows imediatly from the proof of (Bhardwaj et al. 2020 Lemma A.I.), with changes to notation and noting that we assume access to the true reward. particular, the full result of (Bhardwaj et al. 2020 Lemma A.1.) includes extra --- term which comes from the usage of model which estimates the true reward. We do not consider such effects, and thus suppress this dependence.
The next result uses this I to bound the difference between the optimal H-step returns and the H H-step returns generated by the policy #i which is optimal under the dynamics model p:
Lemma Suppose that |lć(s a) - Preat (s,a)li a)ll I a. Further suppose Ar max, r(s)
min, rs) and AV max, Va(s) min, (s(s) are finite. Then for each state E S we have:
11 1-
(8) - a (8) 5
Ar + AV
(5)
1 1-7
where ätj maxx, V (s).
Proof. Let कH + max V (s) be the optimal H-step policy under the true dynamics. Lemma I we have both that
By
- H-1 Ar
AV
(s)() & p-n (7)
+7
+
+
aH..
(6)
1-9 2
2
1 1- Ar
AV
(s) S V (s)
+
all.
(7)
2
2
Combining these two bounds with the fact that 77 (s) < V (8) yields the desired result.
The following result establishes an important monotonicity property on the optimal H H-step value functions which is important for the main result.:
Lemma 3. Suppose that sup, Es-pocatsssn Vism (5') (s')) }]] Vesm (s) > -r()). Then we have Vi+ (s) 2
( for each s E S. Then for each s E S we have:
Vi(s) (s)
(8)
Proof. Fix an initial condition so S. Let be arbitrary, and fix the shorthand ( } for the time-narying policy n²" +- max; V (*0). Then, concatenate these poli- cies to define: म = { - J.. which is simply the result of applying the optimal policy for the (H - st-step look ahead objective Equation [ starting from sp, followed by applying π for a single step. Letting the following distributions over trajectories by generated by n². by the definition
16

---

Published as a conference paper at ICLR 2025
of Vii:
Vij (so)
H-1
2 E
Vsim +
Σ 7/((s)) Vsim(so)
= E
by (y Vsim (8H)
7 H-
Vsim (8H-1) + y r((H-1))
H-2
E
H H- (8H +
Σ rr((1) (50)
t=1
y H E
Vsim V (Si + +7 (r(( VH (50)
The first inequality follows from the fact that the return of म cannot be greater than that of n², the first equality follows from rearanging terms to isolate -i-1- and the second equality follows from the definition of - Now, since our choice of π used to define т was arbitrary, we choose и to be deterministic and such that Estxpreut(ssa (s.a) guaranteed by the assumption made for the result. This choice of policy grantees that:
(s.a)(s Veim
Veim > -r(s) at each state s E S, as
E
Vi (SH) - VVss-VVssssssaa + ( 2
The desired result follows immediately by combining the two preceding bounds, and noting that choice of initial condition was arbitrary, meaning the preceding analysis holds for all initial conditions.
Our final lemma bounds the sub-optimality of SGFT policies in terms of a) errors in the sim value function and b) additional suboptimalities cause by æ being sub-optimal for the H-step objective:
Lemma 4. Suppose that Vsim is improvable and further suppose that maxses Vvii (8) Vieat(s) (s)l < e. Then any policy a which satisfies (s x) = (T, - Vi((s) 2 - will satisfy:
5
(s) Vreat 5 7 -
(10)
Proof. Our goal is first to bound how Q7 (8, x) changes on expectation when applying the given policy for single step. We have that:
(s. + (s)
(11)
v*)+) 2 (8)
(12)
(s, ElVVa-n(s') F(s,
(13)
where the first inequality follows from the ssumption of the theorem, the second inequality follows from L and, the third inequality simply the definition of Qiz Letting s' ~ Preat(8, a) with (-1s), we can take expectations can combine the previous relations to obtain:
E Eh m) + F(s, +') 2 E (>V) + F(s, ) + 2 E [yVi- (8)) + F(s,
= (ir(s, R).
(14)
That is:
7B (2' 70 2 Q2(
(15)
Alternatively:
r($) (s. 7E())
70
(16)
Next, we use this bound to provide a lower bound for V V.) Because the previous analysis holds all states when apply n, the following holds over the distribution of trajectories generated by
17

---

Published as a conference paper at ICLR 2025
applying starting from the initial condition Su
R
Σ ((())
(s) - Va (80)
1=0
2 (p)
Σ
7'
- (0(((()+1
7d
1=0
2=0
76
(s: TT)
1-7¹
where we have repeatedly telescoped out sums to cancel out terms in the first equality, used 116 in the second equality, and canceled out terms to generate the final equality. Thus, we have the lower-bound:
Veaa (s) 2 (H, π) + (so) 78
(17)
76
(s) + Visi (sp)
Next, we may bound:
H-1
Vir (8o) + Vsim (8o) 2 E ioad (ro)) (50)
H Vsim (8H) +
Σ 7'r(st)
(18)
t=
A Vsim(811) - 7 Vx Vieaa (811) 7 H (ren Σ xr((s))
p Food (so)
[ H V Viim((1)) - H (s + Vieat (s)).
roul (50)
Invoking the assumption that max, Veim (s) (s)a e, we can combined this with the pre- ceding bound to yield:
(30) + Viim(()) 2 Vieat(s) - -
(19)
Finally, once more invoking the fact that Qi (s, π) + 2 Vi (s) for each s E S and combining this with Equation (17) and Equation 18] obtain that:
76
V (s) 2 (8, + Vosm
1 7
76
Vil (8o) + Vism
1 -
70
(8)
8 7
1 y
8
(s)
7 # c
1 -
from which the state result follows immediately.
Proof of Theorem
Proof. The result follows directly from combination of Lemma 4 and Lemma by suppressing problem-dependent constants and lower order terms in the discount factor 7.
B ENVIRONMENT DETAILS
Sim2Real Environment. We use 7-DoF Franka FR3 robot with I-DoF parallel-jaw gripper. Two calibrated Intel Realsense D455 cameras are mounted across from the robot to capture position of the object by
pointcloud readings retrieving pose estimation from aruco tags. Commands are sent to the controller at 5Hz. We restrict the end-effector workspace the robot in rectangle for safety the robot arm doesn't collide dangerously with the table and objects outside
18

---

Published as a conference paper at ICLR 2025
the workspace. We conduct extensive domain randomization and randomize the initial gripper pose during simulation training. The reward is computed from measured proprioception of the robot and estimated pose of the object. Details for each task are listed below.
Hammering, For hammering, the action is 3-dimensional and sets delta joint targets for joints of the robot using joint position control. The observation space is 12-dimensional and includes end- effector cartesian xyz, joint angles of the movable joints, joint velocites of the 3 movable joints, the z position of the nail, and the xZ position of the goal. Each trajectory is 50 timesteps. In simulation, we randomize over the position, damping, height, radius, mass, and thickness of the nail. Details are listed in Tab. [
The reward function is parameterized as r()) 100 Tnail goail-goal()) where Thail-god (Fmiit) (Tgoat): represents the distance in the z dimension of the nail head to the goal, which we set to be the height of the board the nail is on.
Pushing. For pushing, the action is 2-dimensional and sets delta cartesian xy position targets us- ing end-effector position control. The observation space is 4-dimensional and includes end-effector cartesian xy and the xy position of the puck object. Each trajectory is 40 timesteps. In simulation, randomize over the position of the puck. Details are listed listed in Tab. Bl
Let Fee be the cartesian position of the end effector and Fobj be the cartesian position of the object.
The reward function is parameterized as r(t) = - -Toe-goal (t) - -oabj- + (tdressholdt)) - -rable (t)
where ree-gual(t) (t) = (t) Toby t) + [3.5cm, 0.0cm, 0.0cm]]| represents the distance of the end effector to the back of the puck, Tobj-goal (t) = ((Taty (t)) 55cm|| represents the distance of the
puck to the goal (which is the edge of the table along the x dimension), ((threshola()) = [l V[ goal()) (t) Z
2.5cm] represents goal reaching binary signal, and (t) = I((ra)n(())= S 0.0] represents binary signal for when the object falls of the table.
Inserting. For inserting, the action is 3-dimensional and sets delta cartesian xyz position tar- gets using end effector position control. The observation space is 9-dimensional and includes end-effector cartesian xyz, the xyz of the leg, and the xyz of the table hole. Each trajectory is 40 timesteps. We enlarge the table by scale of 1.08 compared in the original table in FurnitureBench in order to make the table leg insertable without twisting. In simulation, we randomize the initial gripper position, position of the table, and friction of both the table and the leg.
Let rpust (t) and rpuuz()) represent the Cartesian positions of the leg and table hole. Let:
Edistance (t) clip (Irpsst, ze (t) Fp (t)), 0.1)
Vdistance (t) = clip [=post,. (t) - poss2. (t), 0.0, 0.1)
Adistance (t) clip (Irpost.a (t) Tpos2, (f),, 0.0, 0.1)
Let the success condition be defined as:
Tsuccess (t) = [Adissance (t) < 0.01 and Vdistance (t) < 0..01 and Edistance (t) < 0.01]
The reward function is now:
r(1) rsccess (l) 100 (Jrdistance (1) 2 + Ydistunce (1)2 Zdistance (1)²)
Sim2Sim Environment. We additionally attempt to model a sim2real dynamics gap in simu- lation by taking the hammering environment and create proxy for the real environment by fixing the domain randomization parameters, fixing the initial gripper pose, and rescaling the action mag- nitudes before rolling out in the environment.
C IMPLEMENTATION DETAILS
Algorithm Details. We use SAC as our base off-policy RL algorithm for training in simulation and fine-tuning in the real world. For our method, we additionally add in two networks: a dynamics model that predicts next state given current state and action, and state-conditioned value network which regresses towards the Q-value estimates for actions taken by the current policy. These net- works are training jointly with the actor and critic during SAC training simulation.
Network Architectures. The Q-network, value network, and dynamics model are all parame- terized by a two-layer MLP of size 512. The dynamics model is implemented as delta dynamics model where model predictions are added to the input state to generate next states. The policy net- work produces the mean Ha and state-dependent log standard deviation log aa which is jointly
19

---

Published as a conference paper at ICLR 2025
Table 1:
Domain randomization of hammer-
Table 2: Domain randomization of pushing
ing task in simulation
task in simulation
Name
Range
Name
Range
Nail position (m)
[0.3, 0.4]
Object position (m)
10.0, 0.3]
Nail z position (m)
[0.55, 0.65]
Object y position (m)
[-0.25, 0.25]
Nail damping
[250.0, 2500.0]
Nail half height (m)
[0.02, 0.06]
Table 3: Domain randomization of inserting
Nail radius (m)
[0.005, 0.015]
task in simulation
Nail head radius (m)
[0.03, 0.04]
Nail head thickness (m)
[0.001, 0.01]
Name
Range
Hammer mass (kg)
[0.015, 0.15]
Parts x/y position (m)
[-0.05, 0.05]
Parts rotation (degrees)
[0. 15]
Parts friction
[-0.01, 0.01]
learned from the action distribution. The policy network is parameterized by a two-layer MLP of size 512, with a mean head and log standard deviation head on top parameterized by FC layer.
Pretraining in Simulation. For hammering and puck pushing, we collect 25, 25,000,000 transitions of random actions and pre-compute the mean and standard deviation of each observation across this dataset. We train SAC in simulation on the desired task by sampling 50-50 from the random action dataset and the replay buffer. We normalize our observations by the pre-computed mean and standard deviation before passing them into the networks. We additionally add Gaussian noise centered at 0 with standard deviation 0.004 to our observations with 30% probability during training. For inserting, we train SAC in simulation with no normalization. We train SAC with autotuned temperature set initially to and a UTD
We use Adam optimizer with a learning rate of 3 x 10-4, batch size of 256, and discount factor 7 = .99.
Fine-tuning in Real World. We pre-collect 20 real-world trajectories with the policy learned in simulation to fill the empty replay buffer. We then reset the critic with random weights and continue training SAC with fixed temperature of a 0.01 and with a UTD of 2d with the pretrained actor and dynamics model. We freeze the value network learned from simulation and use it to relabel PBRS rewards during fine-tuning. During fine-tuning, for each state sampled from the replay buffer, we additionally hallucinate 5 branches off and add it to the training batch. As result, our batch size effectively becomes 1536. The policy, Q-network, and dynamics model are trained jointly on the real data during SAC fine-tuning. We don't train on any simulation data during real-world fine-tuning because we empirically found it didn't 't help fine-tunin fin-tunning performance in our settings.
20

---

Published as a conference paper at ICLR 2025
Visualization of Puck Pushing Trajectory
real rollout
0.6
22
hallucinated states
0.5
20
End effector y position
0.4
18
Value
0.3
16
0.2
14
0.1
12
0.0
0.15
0.20
0.25
0.30
0.35
0.40
0.45 0.50
0.55 0.60
End effector position
Figure 5: Visualization of real rollout, hallucinated states, and value function. The red dots indicate states along real rollout in simulation. The blue dots indicate hallucinated states branching off real states generated by the learned dynamics model. The green heatmap indicates the value function estimates at different states. A corresponding image of state is shown two states. Since hard directly visualize states and values due to the high-dimensee dimeeseitsii of the state space, we only show a part of the trajectory where the puck does not move. This allows us to visualize states and values along changes in only end effector xy.
D QUALITATIVE RESULTS
We analyze the characteristics of hallucinated states and value functions in Fig. 5 We visualize trajectory of executing puck pushing in simulation using the learned policy in this plot. The red dots indicate states along real rollout in simulation. The blue dots indicate hallucinated states branching off real states generated by the learned dynamics model. The green heatmap indicates the value function estimates at different states. A corresponding image of the state is shown for two states. The trajectory shown the figure shows the learned policy moving closer to the puck before pushing it. The value function heatmap shows higher values when the end effector is closer to the puck and lower values when further. Hallucinated states branching off each state show generated states for fine-tuning the learned policy.
Note that it is hard to directly visualize states and values due to the high-dimensionality of the state space. To get around this for puck pushing, we only show a part of the trajectory where the puck does not move. This allows to visualize states and values along changes only end effector xy.
21

---

Published as a conference paper at ICLR 2025
E SIM-TO-SIM EXPERIMENTS
Here we additionally test each of the proposed methods on the sim-to-sim set-up from Du et al. (202T), which is meant to mock sim-to-real gaps but for familiar RL benchmark tasks. The results are depicted Figure 7 for the Walker Walk, Cheetah Run, and Rope Peg-in-Hole environments. For all tasks, results matches our real world experiments SGFT substantially accelcrates learning and overcomes the dynamics gap between the 'simulation' and 'real' environments.
the precise settings from Du et al. (202I). Note that the general trend of these
Finally, additionally use the Walker and Peg-in-Hole environments ablate the effects of the hyper parameter H. Intuitively, the peg-in peg-in-hole environment requires much more precise actions, and thus should be more sensitive errors in the pretraining environment. Thus, we should expect that SGFT will benefit from larger values of H for this environment, this will correspond to relying more heavily on returns from the target environment. We see that this trend holds in Figure
6 where the Walker environments is barely affected by the choice of H but this hyper parameter has a large impact on the performance in the Peg-in- Peg-in-Hole Environment.
Abruteg Nale
SCT1 SAC
00
Figure 6: Normalized Rewards for Sim-to-Sim Transfer. Ablating the effects of the horizon H across two sim-to-sim expirments. The choice of H has much larger effect the peg- in-hlc task, which requires much precise actions to achieve succes.
Choech
19
200
250
Asymmetric
Domain Randomicabon
Recument Policy + Domain Randomization
PBRS SAC
ASD
SOFFETOMICC (Duns)
DROPO
SOFT SAC (Ours)
DORAEMON
ALYY
TDWPC2
x180
Figure 7: Normalized Rewards for Sim-to-Sim Transfer. plot the normalized rewards two sim-to- sim transfer tasks, where the rewards are normalized by the maximum reward achieved by any method.
22


Reviews:
[{"review_id": "1S2CPJAvBr", "reviewer": "Reviewer_ZMTP", "summary": "This paper presents “Simulation-Guided Fine-Tuning” (SGFT), a framework aimed at improving sim-to-real adaptation by leveraging simulation-guided data augmentation and model-based reinforcement learning techniques. The paper uses real-world tasks like hammering, inserting, and puck pushing to test SGFT’s ability to adapt policies from simulation to reality with limited real-world data. The framework builds on a model-based RL backbone, specifically using SAC and TDMPC, to optimize policy adaptation with efficient data usage and a reward shaping mechanism.", "questions": "1. How would SGFT handle tasks involving more complex dynamics, such as fluid or deformable object manipulation, where precise dynamics modeling is difficult? Would this require substantial engineering modifications, or does SGFT have inherent limitations in these scenarios?\n2. Can the authors provide insights into the expected engineering effort for adapting SGFT to new environments? Given the dependency on extensive domain randomization and parameter tuning, how feasible is it to scale SGFT across a diverse set of real-world tasks?\n3. Are there specific plans to expand the evaluation to include more complex tasks or different robotic systems? Given the limited range of tasks tested, a broader evaluation could provide a more comprehensive understanding of SGFT’s robustness and adaptability.", "limitations": "", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper addresses the sim-to-real gap in reinforcement learning, an essential area in robotics research. Using a model-based framework with potential-based reward shaping provides an innovative approach to sim-to-real transfer.\n- SGFT’s modular design, integrating model-based learning with reward shaping, shows the potential for efficient adaptation without extensive real-world training. The authors provide theoretical analyses \n- The evaluation on dynamic, real-world tasks such as hammering and inserting presents a realistic testing ground for examining SGFT’s efficacy.", "weaknesses": "1. Limited Task Scope and Complexity:\nThe evaluation primarily involves simple, structured tasks (hammering, inserting, and puck pushing), which limits the generalizability of the results. The tasks do not fully explore SGFT’s adaptability across a broader spectrum of real-world challenges or more complex environments. For example, the hammering task, despite involving contact dynamics, still relies on predefined, straightforward goals and does not include diverse object types or more intricate multi-step interactions. This narrow task set could mean that SGFT’s advantages may not scale effectively to more complex tasks .\n2. Baseline Comparison and Performance:\nWhile SGFT demonstrates some improvement in sample efficiency and asymptotic performance over baselines like SAC and domain randomization, the gains over TDMPC-2 and PBRS are limited. In Table 1, while SGFT performs comparably to these baselines in most cases, it does not consistently outperform them by a significant margin, raising questions about whether the added complexity of the SGFT approach is justified by these incremental improvements  .\n3. Heavy Dependence on Simulation Accuracy and Engineering Effort:\nSGFT’s approach requires a highly accurate simulation setup to guide policy learning effectively, which can be labor-intensive. The need for extensive domain randomization (e.g., Table 1 and 2 for hammering and puck pushing parameters) and environment-specific configurations could make SGFT less practical for tasks where precise dynamics modeling is challenging, such as interactions with liquids or deformable objects. This limitation suggests SGFT may struggle in scenarios where real-world dynamics differ significantly from the simulation environment  .\n4. Uncertain Performance on More Complex Dynamics:\nWhile the paper effectively adapts to straightforward tasks, its approach may not handle more nuanced dynamics. For instance, tasks involving materials with variable properties (like deformable objects or variable friction surfaces) may introduce unpredictable behaviors that SGFT, as presented, is unlikely to manage effectively. Thus, the scope of SGFT’s generalizability and flexibility across a broader range of robotics tasks remains unclear  .", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L1xK1iM6vm", "reviewer": "Reviewer_q12p", "summary": "The paper proposes a method that learns a robust transferred policy in sim-to-real settings by learning a value function, and then fixing it for H-step MPC. The deployed policy only needs to learn to optimize H step model rollouts with the assumption that even if the low level dynamics differ between simulation and the real world, the value function through temporal extension of H-steps is more reliable in deciding which are the target future states in the trajectory. The authors demonstrate both RL fast policy and MPC optimization based variants of the actor trained to optimize H step trajectories in the real environment. The method is tested on three simulation-to-real robot manipulation tasks.", "questions": "Questions:\n1. In Figure 4, why are many of the baseline methods not evaluated for the pushing task?\n2. What happens if you use smaller H for SGFT-TDMPC? What about larger H?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The method is simple and elegant, and the motivation for doing H step planning is clear.\n2. The assumption that the low level actions learnt in simulation can be incorrect, but the general value difference between states preserved is a reasonable assumption, and verified in the experiments.\n3. The value functions being frozen and just finetuning the policy is likely significantly more sample efficient than finetuning all components.\n3. The demonstration of TDMPC2 type model learning and planning in real world robot experiments is a great addition.", "weaknesses": "1. The choice of hyperparameter H seems extremely important, and no ablations have been done here. All algorithms have hyperparameters, but atleast the most important ones should have ablation experiments.\n2. Following up on the previous point, a major weakness of the paper is the lack of extensive experiments. Considering that the proposed method is mainly an engineering improvement rather than a wholly unique algorithm, more experiments in other real world and simulated domains could have been appreciated. For example, within simulated domains there could have been experiments where the model is trained on perturbed dynamics (such as different mass of limbs for locomotion agents) and then transferred to an unperturbed environment. Other experiments could have been designed to show how the optimal values of H are related to different task types. For example it would be interesting to see if tasks that require higher precision control require higher H, which in turn makes the policy transfer more difficult.\n3. There is not much novelty in the method itself, and it might even be something that people already kind of do. Especially SGFT-SAC, the authors fix H=1 for the experiments and this just means that the value function is fixed after simulation and just the policy is finetuned. The TDMPC experiments are more interesting, but this is not then evaluated in the pushing task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xCe93HLf2I", "reviewer": "Reviewer_TjP5", "summary": "The authors propose a novel framework (SGFT) for sim-to-real transfer in robot learning by finetuning RL policies pre-trained in simulation efficiently and effectively on limited real-world interactions. In particular, SGFT changes the real-world MDP problem by (1) turning it into a finite and short-horizon MDP (H-step) and (2) reshaping the new MDP's rewards with a pre-trained critic in simulation. These two changes allow the method to quickly adapt pre-trained policies with real-world interactions, and be also used in combination with model-based RL to further improve data efficiency.", "questions": "- When H=1, what is the difference between a standard actor-critic (AC) method with a frozen critic vs. your SGFT method? To me, it seems that an AC method with frozen pre-trained critic essentially describes the SGFT method. If so, this could yield further intuitions and explanations of the algorithm. In other words, when H=1, it seems to me like the finetuning problem is turned into a contextual bandit problem where no sequential decision making is involved.\n\n- I'm not convinced by the statement \"optimizing Equation (3) guides policy optimization algorithms towards policies which increase the value of V_sim over the H-step windows.\" in Sec. 4.3. To my understanding, as H increases, the algorithm actually gives more importance to real-world returns rather than increasing the value of V_sim. In other words, a final state S_H with lower V_sim(S_H) could be preferred if the real H-step return to get to that state is relatively higher than it was in sim. Conversely, the statement gets truer the lower the value of H.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper uses a clear writing style, where the critical statements are repeated multiple times and highlighted.\n- The method demonstrates good results on contact-rich real-world tasks---which are highly relevant to the community---in 100-200 real rollouts.\n- Theoretical analysis of the proposed method is included\n- The method is relatively simple to implement.", "weaknesses": "- Missing references to recent, related works: the authors discuss sim2real transfer works in the first paragraph of Sec. 2, but fail to include recent state-of-the-art methods as of 2023 and 2024 that \"adapt simulation parameters to real-world data\" (DROPO [1]), or that \"learn adaptive policies to account for changing real-world dynamics\" (DORAEMON [2]). I suggest skimming through these papers to make sure these two literature branches are well referenced and discussed. \n\n-  Limited experimental evaluation: I raise concerns over the main empirical findings in Fig. 4.\n\t- The SAC baseline seems to perform overly bad. To my understanding, this baseline essentially starts the same way as SGFT, but then diverges as finetuning goes on in that SAC also changes its own critic, whereas SGFT relies on a frozen initial critic (cf. my question below for further details). Also, in light of recent claims [3], one would expect off-policy algos to perform better if correctly finetuned.\n\t- In my opinion, the comparison to DR baselines should be done by pre-training them with omniscent critic (aka asymmetric actor-critic), as described in [4] and often done in recent works [5]. SGFT must be trained in simulation with unpriviliged critics, hence it's making a subtle further assumption that DR methods can, instead, relax and leverage. \n\t- the Recurrent policy + DR baseline seems to be missing from Fig. 4 pushing. Pushing is also a notorious benchmark task that has been used by DORAEMON [2] and Peng et al. [4] to showcase successful zero-shot transfer in their experiments, by randomizing similar properties (e.g. mass, friction coefficient). How are zero-shot DR baselines performing in this setting?\n\t- Recent state-of-the-art zero-shot transfer methods such as Doraemon [2] should be included in the experimental evaluation.\n    - Only experiments in the edge case H=1 are shown. This makes it hard to motivate the need for a more general framework.\n\n\n\n[1] Tiboni, G., Arndt K., and Kyrki V. \"DROPO: Sim-to-real transfer with offline domain randomization.\" Robotics and Autonomous Systems 166 (2023): 104432.\n\n[2] Tiboni, G. et al. \"Domain Randomization via Entropy Maximization.\" ICLR 2024.\n\n[3] Ball, Philip J., et al. \"Efficient online reinforcement learning with offline data.\" International Conference on Machine Learning. PMLR, 2023.\n\n[4] Peng, Xue Bin, et al. \"Sim-to-real transfer of robotic control with dynamics randomization.\" 2018 IEEE international conference on robotics and automation (ICRA). IEEE, 2018.\n\n[5] Handa, Ankur, et al. \"Dextreme: Transfer of agile in-hand manipulation from simulation to reality.\" 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "06Rv9ZTe7y", "reviewer": "Reviewer_HzhY", "summary": "This paper addresses the challenge of sim-to-real adaptation. It introduces a novel approach where the value function, learned in simulation, is used to reshape the reward signal, effectively guiding online reinforcement learning in real-world environments. The proposed method demonstrates superior performance compared to domain randomization and online system identification methods in three contact-rich and dynamic tasks: pushing, insertion, and hammering.", "questions": "Please refer to the weakness section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper is focusing on an important problem. \n - The proposed method is novel, built on the key observation that the value function from a well-performing policy learned in simulation can be used to effectively reshape the reward for online learning.", "weaknesses": "- The introduction is hard to follow with the logic among paragraphs not clear.\n - The methodology and figure illustrations can be further improved. \n   - Figure 2 is confusing. According to RL literature conventions, V_s in general, means the value function, but it is labeled as the world model in Figure 2. \n   - How is the dynamic model learned? There is no description of learning the dynamics model in the main script or in the main algorithm, algorithm 1.\n\nTypos:\n - Line 19 in abstract, “is to inefficient” -> ”is too inefficient”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)