Overcoming the Sim-to-Real Gap: Leveraging Simulation to Learn to Explore for Real-World RL
Authors: Andrew Wagenmaker
Kevin Huang
Liyiming Ke
Kevin Jamieson
Abhishek Gupta
Venue: NeurIPS 2024 poster (2024)
Keywords: reinforcement learning
sim2real transfer
exploration
sample complexity
naive exploration
Primary area: reinforcement_learning
TL;DR: We demonstrate that, even when direct sim-to-real transfer fails, transferring exploratory policies from a simulator can enable efficient RL in the real-world with only a least-squares regression oracle.

Abstract: In order to mitigate the sample complexity of real-world reinforcement learning, common practice is to first train a policy in a simulator where samples are cheap, and then deploy this policy in the real world, with the hope that it generalizes effectively. Such \emph{direct sim2real} transfer is not guaranteed to succeed, however, and in cases where it fails, it is unclear how to best utilize the simulator. In this work, we show that in many regimes, while direct sim2real transfer may fail, we can utilize the simulator to learn a set of \emph{exploratory} policies which enable efficient exploration in the real world. In particular, in the setting of low-rank MDPs, we show that coupling these exploratory policies with simple, practical approaches---least-squares regression oracles and naive randomized exploration---yields a polynomial sample complexity in the real world, an exponential improvement over direct sim2real transfer, or learning without access to a simulator. To the best of our knowledge, this is the first evidence that simulation transfer yields a provable gain in reinforcement learning in settings where direct sim2real transfer fails. We validate our theoretical results on several realistic robotic simulators and a real-world robotic sim2real task, demonstrating that transferring exploratory policies can yield substantial gains in practice as well.

Overcoming the Sim-to-Real Gap: Leveraging Simulation to Learn to Explore for Real-World RL
Andrew Wagenmaker" University of California, Berkeley
Kevin Huang
Liyiming Ke
University of Washington
University of Washington
Kevin Jamieson University of Washington
Abhishek Gupta
University of Washington
Abstract
order to mitigate the sample complexity of real-world reinforcement learning, common practice is to first train a policy in simulator where samples are cheap, and then deploy this policy in the real world, with the hope that it generalizes effectively. Such direct sim2real transfer is not guaranteed to succeed, however, and in cases where it fails, it is unclear how to best utilize the simulator. this work, we show that in many regimes, while direct sim2real transfer may fail, we can utilize the simulator to learn a set of exploratory policies which enable efficient exploration in the real world. In particular, in the setting of low-rank MDPs, we show that coupling these exploratory policies with simple, practical approaches- ppproaches--less-squurre least-squares regression oracles and naive randomized exploration-yields exponential improvement
polynomial sample complexity in the real world, an direct sim2real transfer, learning without access to a simulator. To the best of our knowledge, this is the first evidence that simulation transfer yields provable gain reinforcement learning settings where direct sim2real transfer fails. We validate our theoretical results on several realistic robotic simulators and real-world robotic sim2real task, demonstrating that transferring exploratory policies can yield substantial gains in practice as well.
Introduction
Over the last decade, reinforcement learning (RL) techniques have been deployed to solve variety of real-world problems, with applications robotics, the natural sciences, and beyond [27, 54, 52, 26, 46, 23]. While promising, the broad application of RL methods has been severely limited by its large sample complexity-tie number of interactions with the environment required for the algorithm to learn to solve the desired task. In applications of interest, it often the case that collecting samples is very costly, and the number of samples required by RL algorithms is prohibitively expensive.
In many domains, while collecting samples in the desired deployment environment may be very costly, we have access to a simulator where the cost of samples is virtually nonexistent. As a concrete example, in robotic applications where the goal is real-world deployment, directly training in the real world typically requires an infeasibly large number of samples. However, it is often possible covulator-derivve from first principles knowledge of the robot's actuation-wiin provides an approximate model of the real-world deployment environment. Given such a simulator, common practice is first train a policy accomplish the desired task in the simulator, and then deploy it in the real world, with the hope that the policy generalizes effectively from the simulator the goal deployment environment. Indeed, such "sim2real" transfer has become key piece in the
to obtain
Correspondance to: ajvagen@berkeley.co
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

15
Exploration Policy Transfer (Ours)
Approach
Standard Sim2Reea Transfer
10-
Reward
5-
iran expioratryy
Collect Ngt-coveegge
Use dita meam solve
polices
Standard Sim2Roo
-5
-10
0.0
0.2
0.4
0.6
0.8
1.0
policy that
ditaa in rial
salve fask
Steps
x10
Figure 1: Left: Overview of our approach compared to standard sim2real transfer puck pushing task. Standard sim2real transfer first trains a policy to solve the goal task in sim and then transfers this policy real. This policy may fail to solve the task in real due to the sim2real gap, and furthermore may not provide sufficient data coverage to successfully learn a policy that does solve the goal task in real. In contrast, our approach trains data when deployed in real, even if they are unable to solve the task O-shot. This high-coverage data can then be used to successfully learn policy that solves the goal task in real. Right: Quantitative results running our approach on the puck pushing task illustrated on left, compared to standard sim2real transfer. Over 6 real-world trials, our approach solves the task 6/6 times while standard sim2real transfer solves the task 0/6 times.
of exploratory policies in sim which achieve high-coverage
application of RL robotic settings, well many other domains of interest such the natural sciences [12, 15], and is a promising approach towards reducing the sample complexity of RL in real-world deployment [19, 4, 18).
Effective sim2real transfer can challenging, however, as there often a non ann-trivial mismatch between the simulated and real environments. The real world is difficult to model perfectly, and some discrepancy is inevitable. As such, directly transferring the policy trained in the simulator to the real world often fails, the mismatch between sim and real causing the policy-which may perfectly solve the task in sim-to never solve the task in real. While some attempts have been made to address this for example, utilizing domain randomization by simulator [60, 49], or finetuning the policy learned in sim in the real world [50, 73]- these approaches are not guaranteed succeed. In settings where such methods fail, can we still utilize a simulator to speed up real-world RL.?
extend the space of environments covered
In this work take steps towards developing principled approaches to sim2real transfer that addresses this question. Our key intuition is that is often easier to learn to explore than to learn to solve the goal task. While solving the goal task may require very precise actions, collecting high-quality exploratory data can require significantly less precision. For example, successfully solving complex robotic manipulation task requires partieular sequence of motions, but obtaining a policy that will interact with the object of interest in some way, providing useful exploratory data on its behavior, would require significantly less precision.
Formally, we show that, in the setting of low-rank MDPs where there is mismatch the dynamics between the "sim" and "real" environments, even when this mismatch is such that direct sim2real transfer fails, under certain conditions we can still effectively transfer a set of exploratory policies from sim to real. In particular, with random exploration and least-squares regression oracle-which are insufficient for efficient learning their efficient learning in real. Our results therefore demonstrate that simulators, when carefully applied, yield provable provable-exponeeee exponential gain over both naive sim2real transfer and learning without a simulator, and enable algorithms commonly used in practice to learn efficiently.
demonstrate that access to such exploratory policies, coupled
but often still favored in practice due to their simplicity-nnable enable provably
Furthermore, our results motivate simple, easy-to-implement algorithmic principle: rather than training and transferring a policy that solves the task in the simulator, utilize the simulator to train set of exploratory policies, and transfer these, coupled with random exploration, to generate high

---

quality exploratory data in real. We show experimentally- through realistic robotic simulator and real-world sim2real transfer problem on the Franka robot platform- that this principle of transferring exploratory policies from sim to real yields a significant practical gain in sample efficiency, often enabling efficient learning in settings where naive transfer fails completely (see Figure 1).
2 Related Work
Provable Transfer in RL. Perhaps the first theoretical result on transfer in RL is the "simulation lemma", which transforms bound on the on policy value [24, 25, 6, 22]-we argue that we can do significantly better with exploration transfer. More recent work has considered transfer in the setting of block MDPs [34], but requires relatively strong assumptions on the similarity between source and target MDPs, or the meta-RL. setting [69], but only consider tabular MDPs, and assume the target MDP is covered by the training distribution. Perhaps most relevant to this work is the work of [36], which presents several lower bounds showing that efficient transfer in RL not feasible in general. In relation to this work, our work can be seen as providing set of sufficient conditions that do enable efficient transfer, the lower bounds presented in [36] do not hold in the low-rank MDP setting we consider. Several other works exist, but either consider different types of transfer than what we consider (e.g., observation space mismatch), or only learn a policy that has suboptimality bounded by the sim2real mismatch [37, 56, 58]. Another somewhat tangential line of work considers representation transfer in RL, where it is assumed the source and target tasks share a common representation [35, 10, 2]. We remark as well that the formal sim2real setting we consider is special case of the MF-MDP setting of [53].
distance between the dynamics to a bound
Simulators and Low-Rank MDPs. Several existing works show that there are provable benefits to training a policy in "simulation" due to the ability to reset on command [67, 33, 5, 68, 70, 42]. These works do not consider the transfer problem, however. The setting of linear and low-rank MDPs which we consider has seen a significant amount of attention over the last several years, and many provably efficient algorithms exist [21, 1, 62, 63, 43, 41]. These works typically assume access to powerful oracles which enable efficient learning; we only consider access to simple regression oracle. Beyond the theory literature, recent work has also shown that low-rank MDPs can effectively model a variety of standard RL settings in practice [72].
Sim2Real Transfer in Practice.
The sim2real literature is vast and only highlight particularly relevant works here; see [74] for a full survey. To mitigate the inconsistency between the simulator and real world's physical parameters and modeling, domain randomization creates variety of simulated
environments with randomized properties to develop a robust policy [60, 49, 44, 8, 39j. Domain adaptation instead constructs encoding of deployment conditions (e.g., physical condition or past histories) and adapts to the deployment environment by matching the encoding [29, 9, 66, 55, 38, 40]. In contrast, our work assumes a fundamental sim2real mismatch where we do not expect the real system to match the simulator for any parameter settings. A related line of work shows that policies trained with robust exploration strategies generalize better to disturbed or unseen environments [13, 20]. Our work is complimentary to this work in that our goal is not to transfer policy that solves the task in new environment, but rather explores the environment.
3 Preliminaries
We let 4x denote the set of distributions over set X. [H] = 21,,,...., H). and || total-variation distance between distributions P and Q.
Ql/rv the
Markov Decision Processes. We consider the setting of episodic Markov Decision Processes (MDPs). An MDP is denoted by a tuple M = AS {Pr (r 81, Hf), where denotes the set of states, A the set of actions, PA S x A As the transition function, Ta : S A - (0, I] the reward (which we assume is deterministic and known), $1 the initial state, and H the horizon. We assume A is finite and denote A := |A]. Interaction with MDP starts from state $1, the agent takes some action aj transitions to state 82 ~ (- $1. a, ). and receives reward (i a1). This process continues for H steps at which points the episode terminates, and the process resets.
The goal of the learner is to find a policy म = [ nr : > A that achieves maximum reward. We can quantify the reward received by some policy in terms of the value and Q-value
3

---

functions. The Q-value function is defined as Q7 a) := E E- h The (Shr, ax:)
S. 0n
al. and value function is defined in terms of the Q-value function as V. (s) := (-1s) 12% a)]. The value of policy x, its expected reward, is denoted by V0_ == ( and the value of the optimal policy, the maximum achievable reward, by VJ= := sup, Vo-
In this work are interested in the setting where we wish to solve some task in the "real" envi- ronment, represented as an MDP, and we have access to a simulator which approximates the real environment in some sense. We denote the real MDP _M and the simulator as M We assume that Mrreal and Myim have the same state and actions spaces, reward function, and initial state, but different transition functions, preal and psim. We denote value functions in Mrrreal and Myim as Vreall (s) and y.imm.
h
hn (s), respectively. We make the following assumption.
Assumption 1. For all (s, h, 8 A x [H] and some ésim > we have:
||Pr (- s, a) P. |,, s,a)]lTTy S éxim
We do not assume that the value of ésim is known, simply that there exists some such 6sim-
Function Approximation. In order to enable efficient learning, some structure on the MDPs of interest is required. We will assume that Mrreal and .Msim are low-rank MDPs, as defined below. Definition 3.1 (Low-Rank MDP). We say an MDP is a low-rank MDP with dimension d if there exists some featurization Φ : S A + a and measure μ [H] x S TRd such that:
(- 8, a.)) = (ó(s, a), ()(())) Vs, a, h.
We assume that [|<b(,, a) < 1 for all (8, a), and for all h. [|l = ll Jees (d)n())]ll S vd.
Formally, we make the following assumption on the structure of M and M Assumption 2. Both Maim and M satisfy Definition 3.1 with feature maps and measures (Ø², 11") and (Ø', (²)), respectively. Furthermore, &² known, but all of μ², s', and μe are unknown.
In the literature, MDPs satisfying Definition 3.1 but where & is known are typically referred as "linear" MDPs, while MDPs satisfying Definition 3.1 but with ø unknown are typically referred to as "low-rank" MDPs. Given this terminology, we have that M is linear MDP", while A is low-rank MDP. We assume the following reachability condition on M
Assumption 3. There Ie > O with minA sup, Amin (EMM "[Ø"(s,. ( aI) 2 Xinin
Assumption 3 posits that each direction in the feature space in our simulator can be activated by some policy, and can be thought of measure of how easily each direction can be reached. Similar assumptions have appeared before in the literature on linear and low-rank MDPs [71, 3, 2]. Note that we only require this reachability assumption in M We also assume we are given access to function classes FA, S x A + [0, H)] and let F == Fi x F2 X X FH. Since no reward is collected in the (H + 1)th step we take su+1 O. For any ʃ S x A + R. we let n. (s) == arg max (8)
We define the Bellman operator on some function fa+1: S x A +
as:
I a) == (s, a) + Er (-1s,a) [max fn+1(s',
We make the following standard assumption on F. Assumption 4 (Bellman Completeness). For all fu+1 Fr+1 we have Treal fu+1 1,, Taii, where Treal and T denote the Bellman operators on Mreal and Msm, respectively.
Tsim Tsim fitt E Fr
PAC Reinforcement Learning. Our goal is to find a policy that achieves maximum reward in r Formally, we consider the PAC FrrobablyApproxximatlyy Correet RL setting.
Definition 3.2 (PAC Reinforcement Learning). Given some > 0 and 8 > O, with probability at least 1 identify some policy से such that: Vireal,5 Viral, e
We will be particularly interested in solving the PAC RL problem with the aid of a simulator, using the minimum number of samples from ! possible, as we will formalize in the following. As we will see, while it is straightforward to achieve this objective using Msim if O(esim), ) naive transfer methods can fail to achieve this completely if & Esim As such, our primary focus will be on developing efficient sim2real methods in this regime.
'The assumption that Φ² is known is for simplicity only- similar results could be obtained were si also unknown using more complex algorithmic tools in M

---

Theoretical Results
In this section we provide our main theoretical results. We first present two negative results: in Section 4.1 showing that "naive exploraiion""ttilinnn -utilizing only least-squares regression oracle and random exploration approaches such as (-greedy'-is provably inefficient, and in Section 4.2 showing that directly transferring the optimal policy from _M to M is unable to efficiently obtain policy with suboptimality better than (Csim) in real. Then in Section 4.3 we present our main positive result, showing that by utiliizing the same oracles as in Sections 4.1 and 4.2 least-squares regression oracle, regresssooooooooon
simulator access, and the ability to take actions randomly- we can efficiently learn an e-optimal policy for < 6sim in M by carefully utilizing the simulator to learn exploration policies.
4.1 Naive Exploration is Provably Inefficient
While variety of works have developed provably efficient methods for solving PAC RL in low-rank MDPs [1, 62, 43, 41J, these works typically either rely on complex computation oracles or carefully directed exploration strategies which are rarely utilized in practice. In contrast, RL methods utilized in practice typically rely on "simple" computation oracles and exploration strategies. Before considering the sim2real setting. we first show that such "simple" strategies are insufficient for efficient PAC RL. To instantiate such strategies, we consider least-squares regression oracle, often available in practice.
Oracle 4.1 (Least-Squares Regression Oracle). assume access to a least-squares regres- sion oracle such that, for any h and dataset D ((s", a², y²)) we can compute arg migminnees (i ,a" aaaaa="'- y²)
We couple this oracle with "naive exploration". which here we use to refer to any method that explores by randomly perturbing the action recommended by the current estimate of the optimal policy. While a variety of instantiations of naive exploration exist (see e.g. [IIJ, we consider a particularly common formulation, <-greedy exploration.
Protocol 4.1 (ç-Greedy Exploration). Given access to regression oracle, any Ç E [0, 1]. and time horizon T, consider the following protocol:
1. Interact with rrrral for T episodes. At every step of episode t + 1, play mhf (s) with probability Ç, and a ~ unif(A) otherwise, where:
fh arg min (f
of) rh max fh+ (5 (s/
2. Using collected data in any way desired, propose policy 7.
Protocol 4.1 forms the backbone of many algorithms used in practice. Despite its common application. as existing work [11] and the following result show, it is provably inefficient.
Proposition For any H > 1, C E [0, 1), and e S 1/6, there exist some M and Mreal. such that both M and M satisfy Assumptions 2 and 4, and unless T 2 (2221/2), when running Protocol 4.1 we have:
sup M ( (MMm), 1, E. E. [V
Mr c/32.
Proposition shows that, in a minimax sense, S-greedy exploration is insufficient for provably efficient reinforcement learning: on one of M and M Ç-greedy exploration will only be able to find a policy that is suboptimal by constant factor, unless we take an exponentially large number of samples. While we focus on Ç-greedy exploration in Proposition 1, this result extends to other types of naive exploration, for example, those given in [11]. See Section 5.2 for further discussion of the construction for Proposition
4.2 Understanding the Limits of Direct sim2real Transfer
Proposition I shows that in general utilizing least-squares regression oracle with <-greedy explo- ration is insufficient for provably efficient RL. Can this be made efficient with access to a simulator
Throoghout this paper, use "S-greedy" to refer to the method more commonly known as "e-greedy" in the literature, to avoid ambiguity between this and the in definition of PAC RL. Definition 3.2.
5

---

M In practice, standard sim2real methodology typically trains policy to accomplish the goal task in M and then transfers this policy to M We refer to this methodology as direct sim2real transfer. The following canonical result, usually referred to as the "simulation lemma" [24, 25, 6, 22], provides a sufficient guarantee for direct sim2real transfer to succeed under Assumption 1.
Proposition 2 (Simulation Lemma). Let π --im,* denote optimal policy in M Then under Assumption 1 we have V Vireal, 2 Vi 2
Proposition 2 shows that, long as c 2 2HH Csim- direct sim2real transfer succeeds in obtaining an e-optimal policy in M While this justifies direct sim2real transfer in settings where M and Mrreal are sufficiently close, we next show that given access only to 7/imm and least-squares regression oracle obtain a policy with suboptimality less than (Esim) on M using naive exploration. To formalize this, we consider the following interaction protocol.
when coupled with random exploration- we cannot hope to efficiently
Protocol 4.2 (Direct sim2real Transfer with Naive Exploration). Given access to sim.,, an optimal policy in Msim, any Ç E [0, 1). and time horizon T, consider the following protocol:
1. Interact with Mrrea for T episodes, and each step h and state play Th simm... (- 8) with probability 1 Ç. and a ~ (A) with probability Ç.
2. Using collected data in any way desired, propose a policy म.
Protocol 4.2 is a standard instantiation of direct sim2real transfer commonly found in the literature, and couples playing the optimal policy from Msim with naive exploration. We have the following. Proposition 3. With the same choice of M and M as in Proposition 1, there exists some Msi such that both M and M satisfy Assumption with Msim fsrm
tsim &- Assumptions 2
to hold, and unless T 2 (22H) when running Protocol 4.2, we have:
B.M [Vo
M. 0
Esim /22.
Proposition 3 shows that there exists setting where there are two possible M satisfying Assumption with M and where, using direct policy transfer, unless we interact with M for exponentially many episodes (in H), we cannot determine better than the worst-case Mrreal Together, Propositions 2 and 3 show that, while we can utilize direct sim2real transfer to learn a policy that is (o(esim--opting in M if our goal is to learn an e-optimal policy for < Ésim, direct sim2real transfer is unable to efficiently achieve this.
-optimal policy for
4.3 Efficient sim2real Transfer via Exploration Policy Transfer
Does there exist some way to utilize Mssim and a least-squares regression oracle to enable efficient learning in M even when < 6sim? Our key insight is that, rather than transferring the policy that optimally solves the task in Msimm we should instead transfer policies that explore effectively in M While learning to solve task may require very precise actions, we can often obtain sufficiently rich data with relatively imprecise actions-ip is easier to learn to explore than learn solve a task. In such settings, directly transferring policy to solve the task will likely fail due to imprecision in the simulator, but may be possible to still transfer a policy that generates exploratory data. To formalize this, we consider the following access model to Msim
Oracle 4.2 (M Access). We may interact with M by either:
1. (Trajectory Sampling) For any policy n, sampling trajectory {(sh, ah; Thi $h41) generated by playing π on Msii.
2. (Policy Optimization) For any reward F, computing a policy si maximizing 7 on Mssm
While access to such a policy optimization oracle is unrealistic in M where we want to minimize the number of samples collected, given cheap access to samples in Msim, such an oracle can often be (approximately) implemented in practice". ' Note that under Oracle 4.2 we only assume black-box
'While for simplicity we assume that the truly optimal policy can be computed, our results easily extend to settings where only have access to oracle which can compute an approximately optimal policy.

---

access to our simulator- rraher than allowing the behavior of the simulator to be queried at arbitrary states, we are simply allowed roll out policies on M and compute optimal policies. Given Ora- cle 4.2, as well as our least-squares regression oracle, Oracle 4.1, we propose the following algorithm.
Algorithm sim2real Exploration Policy Transfer
1: input: budget T, confidence 6, simulator M
// Learn policies T2xp which cover feature space in M
2: I1 +- LEERREEXPPLLCCISS
6, 14"s H h) (Algorithm 5) for all h [H]
3: Exp +- -Txxpp Texp plays Texp up to step h, then plays actions randomly, VTTexp E I
// Explore in Meaa via Mep
4: Play Texp unif [unif )== for T/2 episodes in MMreal, add data D
// Estimate optimal policy collected data
5: for h H, 1,...,,,1
6:
In f arg ge (e - max (8',(6')0))
7: Compute ximm,* via Oracle 4.2
8: Play 5im,, for T/4 episodes in real, compute average return reall
9: Play TJ for T/4 episodes in real, compute average return 1rreal, preallor
10: return मे < arg max (ar (1T, pesmiin
Algorithm first calls a subroutine LEARNEXPPOLICIES which learns of policies that provide rich data coverage on M --smm-pecischh induce covariates with lower-bounded minimum eigenvalue on M and relies only Oracle 4.2 (as well as knowledge the featurization M Φ¹) to find such policies. Algorithm then plays these exploration policies in MMrral, coupled with random exploration, and applies the regression oracle to the data they collect. Finally, it estimates the value of the policy learned by the regression oracle and simmm,,, and returns whichever is best. We have the following.
preciscly, LEARNEXPPOLICIES returns policies {II he which
Theorem 1. Assumptiinss to 4 hold and
6sim
Aean A3:
(4.1)
then as long as
T C. H
H
log
with probability at least 8, Algorithm returns a policy î such that Vereal, Vo Oracles and 4.2 are invoked most poly(d, H, log }) times.
V/real,
S and
Theorem shows that, as long as esim satisfies (4.1), utilizing a simulator and least-squares regression oracle, Oracles 4.1 and 4.2, allows for efficient learning in M achieving complexity scaling polynomially in problem parameters. This yields an exponential improvement over learning without simulator using naive exploration or direct sim2real transfer-which Propositions and show have complexity scaling exponentially in the horizon desiitt utilizing the same practical computation oracles. To the best of our knowledge, this result provides the first theoretical evidence that sim2real transfer can yield provable gains in RL beyond trivial settings where direct transfer succeeds.
Note that the condition in (4.1) is independent of c-unlike direct sim2real transfer, which requires = ((esm), we simply must assume ésim is small enough that (4.1) holds, and Theorem shows that we can efficiently learn an e-optimal policy in M for any >> In Appendix B.4, we also present extended version of Theorem 1, Theorem 3, which utilizes data from s to reduce the dependence on [Fl. particular, instead of scaling with log [F], it only scales with the log-cardinality of functions that (approximately) Bellman-consistent
Msim To illustrate the effectiveness of Theorem 1, we return to the instance of Propositions and 3, where naive exploration and direct sim2real transfer fails. We have the following.
Proposition& 4. In the setting of Propositions I and 3 and assuming that 6sim S 8192 H. running Algorithm I will require poly H, e!)- log 3 samples from M in order to identify an c-optimal policy in real with probability least 1 - for any 0.
7

---

Note that the condition required by Proposition 4 is simply that Gim S 1/H as long as our simulator satisfies this condition, we can efficiently transfer exploration policies learn an e-optimal policy, for any é > 0, while naive methods would be limited to only obtaining an (2(1 (t////oopptiall policy.
Remark 4.1 (Necessity of Random Exploration). Algorithm achieves efficient exploration in M by learning policies 12p in MMiim that span the feature space of MM (Line 2), and then playing these policies in M coupled with random exploration (Line 4). This use of random exploration is critical to obtaining Theorem 1. As show in Proposition 5, if we omit the random exploration, Assumption is not sufficient to guarantee Nexp explores effectively in M even when (4.1) 1) holds.
Remark 4.2 (Computational Efficiency). Algorithm 1, as well as its main subroutine LEARNEXP- POLICIES, relies only on calls to Oracle 4.1 and Oracle 4.2. Thus, assuming we can efficiently implement these oracles, which is often the case problem settings of interest, Algorithm can be run in computationally efficient manner.
5 Practical Algorithm and Experiments
We next validate the effectiveness of our proposal in practice: can set of diverse exploration policies obtained from simulation improve the efficiency of real-world reinforcement learning? We start by showing that this holds for simple, didactic, tabular environment in Section 5.2. From here, we consider several more realistic task domains; simulators inspired by real-world robotic manipulation tasks (sim2sim transfer, Section 5.3); and an actual real-world sim2real experiment on Franka robotic platform (sim2real transfer, Section 5.4). Further details on all experiments, including additional baselines, can be found in Appendix E. Before stating our experimental results, we first provide a practical instantiation of Algorithm I that we can apply with real robotic systems and neural network function approximators.
5.1 Practical Instantiation of Exploration Policy Transfer
The key idea behind Algorithm is quite simple: learn a set of exploratory policies in Msiim_ policies which provide rich data coverage in Mysim transfer these policies M coupled with random exploration, using the collected data to determine a near-optimal policy for M Algorithm I provides a particular instantiation of this principle, learning exploratory policies in M via the LEARNEXPPOLICIES subroutine, which aims to cover the feature space of M and utilizing a least-squares regression oracle to compute an optimal policy given the data collected in M In practice, however, other instantiations of this principle are possible by replacing LEARNEXPPOLICIEE with any procedure which generates exploratory policies in M and replacing the regression oracle with any RL algorithm able to learn from off-policy data. We consider a general meta-algorithm instantiating this in Algorithm
Algorithm 2 Practical sim2real Exploration Policy Transfer Meta-Algorithm
1: Input: Simulator Msim, real environment M simulator budget Tsim- real budget T, algorithm to generate exploratory policies in sim Neep. algorithm to solve policy optimization in real Uo // Learn exploratory policies in 'M
2: Run Nexp for Tim steps in Msim to generate set of exploratory policies Hexp
// Deploy exploratory policies in M
3: for = 1,2, ... T/2 do
4: Draw Texp unif (TTxp)) play in Mreeal for one episode, add data replay buffer of Xpo
5: Run Mp for one episode
// optional if App learns fully offline
In practice, Nexp and 2 can be instantiated with a variety of algorithms. For example, we might take Mexp to be an RND [7] or bootstrapped Q-learning-styl [45, 31] algorithm, or any unsupervised RL procedure [48, 14, 32, 47], and 2p to be an off policy policy optimization algorithm such as soft actor-critic (SAC) [16] or implicit Q-learning (IQL) [28].
For the following experiments, we instantiate Algorithm 2 by setting Neep to an algorithm inspired by recent work on inducing diverse behaviors in RL [14, 30], and Apo to SAC. In particular, Aexp simultaneously trains an ensemble of policies Iexp = (Tep))==1 and discriminator de S x
[n] + I,, where de is trained to discriminate between the behaviors of each policy Texp- and T
8

---

Actlon Action
0.31
Expention Pally Fundeerruur
Dract Fall Turake
0.23
Nolipaa fioe lopag
PR 15,40
011
Steee
Figure 2: Left: Illustration ofCombination Lock Example. Right: Results on Combination Lock.
is optimized on a weighting of the true task reward and the exploration reward induced by the discriminator, (s, == log | (xp(()( As shown in existing work [14, 30J, this simple training objective effectively induces diverse behavior with temporally correlated exploration while remaining within the vicinity of the optimal policy, using standard optimization techniques. Note that the particular choice of algorithm is less critical here than abiding by the recipes laid out in the meta-algorithm (Algorithm 2). The particular instantiation that run for experiments is detailed in Algorithm 6, along with further details in Appendix E.2.
exp(dela.d))
5.2 Didactic Combination Lock Experiment
We first consider variant of the construction used to prove Propositions 1 and 3, itself variant of the classic combination lock instance. We illustrate this instance in Figure 2. Unless noted, all transitions occur with probability 1, and rewards are 0. Here, in Msim the optimal policy, siim, plays action d2 for all steps < 1, while in Mreaa, the optimal policy plays action di at every step. Which policy is optimal is determined by the outgoing transition from 81 at the (H 1)th step and, as such, to identify the optimal policy, any algorithm must reach $1 at the (H - 1)th step. As $1 will only be reached at step H 1 by playing d1 for H - consecutive times, any algorithm relying on naive exploration will take exponentially long to identify the optimal policy. Furthermore, playing -im coupled with random exploration will similarly take exponential number of episodes, since sim. always plays 02- As such, both direct sim2real policy transfer as well as Q-learning with naive exploration (Protocol 4.1) will fail to find the optimal policy in Mrreal However, if we transfer exploratory policies from sim since M and Me behave identically up step // - 1, these policies can efficiently traverse MMreal, reach 8i at step H 1, and identify the optimal policy. We compare our approach of exploration policy transfer to these baselines methods and illustrate the performance of each in Figure As this simple tabular instance, Algorithm 1 directly here. As Figure 2 shows, the intuition described above leads to real gains in policy transfer quickly identifies the optimal policy, while more naive approach fail completely over the time horizon we considered.
implement
5.3 Realistic Robotics sim2sim Experiment
To test the ability of our proposed method to scale we next experiment on sim2sim transfer setting with a realistic robotic simulator. We consider TychoEnv, simulator of the 7DOF Tycho roboties platform introduced by [73], and shown in Figure 3. We test sim2sim transfer reaching task where the goal is to touch a small ball hanging in the air with the tip of the chopstick end effector. The agent perceives the hall and its own end effector pose and outputs a delta in its desired end effector pose as a command. We set M and rrreal to be two instances of TychoEnv with slightly different parameters to model real-world sim2real transfer. Precisely, we change the action bounds and control frequency from M to M
complex problems,
We aim to compare approach of exploration policy transfer with direct sim2real policy transfer. To this end, we first train a policy in Mssm that solves the task in Myim siim,,, and then utilize this policy in place of Texp in Algorithm 2. We instantiate our approach of exploration policy transfer as outlined above. Our aim in this experiment is to illustrate how the quality of the data provided by direct policy transfer vs. exploration policy transfer affects learning. As such, for both approaches we simply initialize our SAC agent in Meral Npo. from scratch, and set the reward in Mrreal to be sparse: the agent only receives a non-zero reward if it successfully touches the ball. For each approach, we repeat the process of training Mss four times, and for each of these run them for two trials in Meal
Figure 3: TychoEnv
Reach Task Setup

---

We illustrate our results in Figure 4. As this figure illustrates, direct policy transfer fails learn completely, while exploration policy transfer successfully solves the task. Investigating the behavior of each method, we find that the policies transferred via exploration policy transfer, while failing to solve the task with perfect accu- racy, when coupled with naive exploration are able to successfully make contact with the ball occasion. This provides sufficiently rich data for SAC to ultimately learn to solve the task. In contrast, direct policy transfer fails to collect any reward when run in M and, given the sparse reward nature of the task, SAC is unable to locate any reward and learn. We include an additional sim2sim ex- periment on the Franka Emika Panda Robot Arm in Appendix E.4,
Eig cration Pally Turalar Duni Peliy Tumnler
20
15
10
Steps
x10
Figure 4: Results on sim2sim
Transfer in TychoEnv Simulator
5.4 Real- World Robotic sim2real Experiment
Finally, we demonstrate our algorithm for actual sim2real policy transfer for manipulation task on real-world Franka Emika Panda robot arm [17] with parallel gripper. Our task to push a 75mm diameter cylindrical "puck" from the center to the edge of the surface, as shown in Figure 1, with the arm initialized at random locations. The observed state = PPe, Pobil E Rt consists of the planar Cartesian coordinate of the end effector Pee along with the center of mass of the puck Pobi- Our policy outputs planar end effector position deltas Apee E R², evaluated at 8 Hz. which are passed into a lower-level joint position PID controller. We use an Intel Realsense D435 depth camera to track the location of the puck. Our reward function is a sum of success indicator (indicating when the puck has been pushed to the edge of the surface) and terms which give negative reward if the distance from the end effector to the puck, or puck to the goal, are too large (see (E.1)); in particular, a reward greater than 0 indicates success.
We run the instantiation of Algorithm 2 outlined above. In particular, we train an ensemble of n = 15 exploration policies, training for 20 million steps in M In addition, we train policy that solves the task in A ..mm, We use custom simulator of the arm, where during training the friction of the table is randomized and noise is added to the observations.
We observe a substantial sim2real gap between our simulator and the real robot, with policies trained in simulation failing to complete the pushing task zero shot in real, even when trained with domain randomization. We compare direct sim2real policy transfer against our method of transferring exploration policies. For direct policy transfer, simply run SAC to finetune simm in the real world, using the current policy to collect data. For exploration policy transfer, we instead utilize leepp our ensemble of exploration policies, to collect data in the real world. We run this in tandem with an SAC agent, feeding the data from the exploration policies into the SAC agent's 's replay buffer. Unlike in Section 5.3, rather than initializing the SAC policy from scratch, we set the initial policy as simm, and fine-tune from this on the data collected from playing Hexp. See Appendix E.5 for additional details.
Our results are shown on the right side of Figure 1. Statistics are computed over 6 runs for each method. Direct policy transfer with finetuning is unable to solve the task real in each of the 6 runs, and converges to suboptimal solution. However, our method is able to solve the task successfully each time and achieve a substantially higher reward.
6 Discussion
In this work, we have demonstrated that simulators can make naive exploration efficient even in settings where direct sim2real transfer fails, if they are used train a set of exploration policies. We highlight several limitations of this work, which we believe are interesting future research questions:
Our focus is purely on dynamics where the dynamics of sim and real differ, but the environ- ments are otherwise the same. While dynamics shift is common in many scenarios, other types of shift can exist as well, for example perceptual shift. How can we best handle these types of shift?
How can we utilize simulator in sim2real transfer if we can reset it arbitrarily, rather than just al- lowing for black-box access? Does the ability to reset allow us to improve sample efficiency further?
Is the reachability condition, Assumption 3, necessary for successful exploration transfer?
10

---

Acknowledgements
The work of AW and KJ was partially supported by the NSF through the University of Washington Materials Research Science and Engineering Center, DMR- 2MM-9338979 and awards CCF 2007036 and CAREER 2141511. The work of LK was partially supported by Toyota Research Institute URP.
References
[1] Alekh Agarwal, Sham Kakade, Akshay Krishnamurthy, and Wen Sun. Flambe: Structural complexity and representation learning of low rank mdps. Advances in neural information processing systems, 33:20095-20107, 2020.
[2] Alekh Agarwal, Yuda Song, Wen Sun, Kaiwen Wang, Mengdi Wang, and Xuezhou Zhang. Provable benefits of representational transfer in reinforcement learning. In The Thirty Sixth Annual Conference Learning Theory, pages 2187- PMLR, 2023.
[3] Naman Agarwal, Syomantak Chaudhuri, Prateek Jain, Dheeraj Nagaraj, and Praneeth Netrapalli. Online target q-learning with reverse experience replay: Efficiently finding the optimal policy for linear mdps. arXiv preprint www.a:2ii...00033:2
2021.
[4] Ilge Akkaya, Marcin Andrychowiez, Maciek Chociej. Matcusz Litwin. Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert. Glenn Powell, Raphael Ribas, al. Solving rubik's cube with robot hand. arXiv preprint arXiv:1910.077 13, 2019.
[5] Philip Amortila, Nan Jiang, Dhruv Madeka, and Dean P Foster. A few expert queries suffices for sample-efficient rl with resets and linear value approximation. Advances in Neural Information Processing Systems, 35:29637-29648 2022.
[6] Ronen Brafman and Moshe Tennenholtz. R-max-a general polynomial time algorithm for near-optimal reinforcement learning, Journal of Machine Learning Research, 300e))2213- 2002.
[7] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv: 2018.
[8] Yevgen Chebotar, Ankur Handa, Viktor Makoviychuk, Miles Macklin, Jan Issac, Nathan Ratliff, and Dieter Fox. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. In ICRA, 2019.
[9] Tao Chen, Megha Tippur, Siyang Wu, Vikash Kumar, Edward Adelson, and Pulkit Agrawal. Visual dexterity: In-hand reorientation of novel and complex object shapes. Science Robotics, 8 (84)xeade9244, 2023.
[10] Yuan Cheng, Songtao Feng, Jing Yang, Hong Zhang, and Yingbin Liang. Provable benefit of multitask representation learning in reinforcement learning. Advances in Neural Information Processing Systems, 35:31741-31754 2022.
[11] Chris Dann, Yishay Mansour, Mehryar Mohri, Ayush Sekhari, and Karthik Sridharan. Guaran- tees for epsilon-greedy reinforcement learning with function approximation. In International conference on machine learning, pages 4666-4689. PMLR, 2022.
[12] Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, et al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897): 414-419, 2022.
[13] Benjamin Eysenbach and Sergey Levine. Maximum entropy rl (provably) solves some robust problems. arXiv preprini (tfi::2003.062..7.
[14] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv:1802.06070, 2018.
[15] Raj Ghugare, Santiago Miret, Adriana Hugessen, Mariano Phielipp, and Glen Berseth. Search- ing for high-value molecules using reinforcement learning and transformers. arXiv preprint arXiv:23 10.00900. 2023.
[16) Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference machine learning, pages 1861-1870. PMLR, 2018.
11

---

[17] Sami Haddadin, Sven Parusel, Lars Johannsmeier, Saskia Golz, Simon Gabl, Florian Walch, Mohamadreza Sabaghian, Christoph Jähne, Lukas Hausperger, and Simon Haddadin. The franka emika robot: A reference platform for robotics research and education. IEFE Robotics & Automation Magazine, 29(2):46-66) 64, 2022. doi: 10.1 (009/MRA.2002.333335.
[18] Sebastian Höfer, Kostas Bekris, Ankur Handa, Juan Camilo Gamboa, Melissa Mozifian, Florian Golemo, Chris Atkeson, Dieter Fox, Ken Goldberg, John Leonard, et al. Sim2real in robotics and automation: Applications and challenges. IEEE transactions on automation science and engineering, 18(2):398-400, 2021.
[19] Stephen James, Paul Wohlhart, Mrinal Kalakrishnan, Dmitry Kalashnikov, Alex Irpan, Julian Ibarz, Sergey Levine, Raia Hadsell, and Konstantinos Bousmalis. Sim-to-real via sim-to-sim: Data-efficient robotic grasping via -oa-iiize----cconiicaa adaptation networks. arxiv e-prints, page. arXiv preprint arXiv:1812.000 072552, 2018.
[20] Yiding Jiang, J Zico Kolter, and Roberta Raileanu. On the importance of exploration for generalization reinforcement learning. arXiv preprint arXiv:2306.055 05483, 2023.
[21] Chi Jin, Zhuoran Yang. Zhaoran Wang, and Michael Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory, pages 2137- 2143. PMLR, 2020.
[22] Sham Kakade, Michael J Kearns, and John Langford. Exploration in metrie state spaces. In Proceedings of the 20th International Conference on Machine Learning (ICML-03), pages 306-312, 2003.
[23] Elia Kaufmann, Leonard Bauersfeld, Antonio Loquercio, Matthias Müller, Vladlen Koltun, and Davide Scaramuzza. Champion-level drone racing using deep reinforcement learning. Nature, 620(7976)98229977 2023.
[24] Michael Kearns and Daphne Koller. Efficient reinforcement learning in factored mdps. In IJCAI, volume 16, pages 740-747, 1999.
[25] Michael Kearns and Satinder Singh. Near-optimal reinforcement learning in polynomial time. Machine learning. 49:209-255 232. 2002.
[26] Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey,
IEEE Transactions on Intelligen' Trunsportation Systems, 23(6):4909-49) 2021.
[27] Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey.
The International Journal of Robotics Research, (2(1 2013.
[28] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit
q q-leariig. arXiv preprint aww.v:2210.0611111111
2021.
[29] Ashish Kumar, Zipeng Fu, Deepak Pathak, and Jitendra Malik. Rma: Rapid motor adaptation for legged robots. arXiv preprini 1eXXix:200704000) 2021.
[30] Saurabh Kumar, Aviral Kumar, Sergey Levine, and Chelsea Finn. One solution is not all you need: Few-shot extrapolation via structured maxent rl. Advances in Neural Information Processing Systems, 33:8198 8210, 2020.
[31] Kimin Lee, Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning. In International Conference on Machine Learning, pages 6131-6141. PMLR, 2021.
[32] Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Rus- lan Salakhutdinov. Efficient exploration via state marginal matching. arXiv preprint arXiv: 19K6:0090600224.. 2019.
[33] Gen Li, Yuxin Chen, Yuejie Chi, Yuantao Gu, and Yuting Wei. Sample-efficient reinforcement learning is feasible for linearly realizable mdps with limited revisiting. Advances in Neural Information Processing Systems, 116616671666,,111111111111111111 2021.
[34] Yao Liu, Dipendra Misra, Miro Dudík, and Robert Schapire. Provably sample-efficient rl with side information about latent dynamics. Advances in Neural Information Processing Systems, 35:33482-33493, 2022.
[35] Rui Lu, Gao Huang, and Simon $ Du. On the power of multitask representation learning in linear mdp. arXiv preprint 2XXi:2106.08055 2021.
12

---

[36] Dhruv Malik, Yuanzhi Li, and Pradeep Ravikumar. When is generalizable reinforcement learning tractable? Advances in Neural Information Processing Systems, 34:8032-888 8045, 2021.
[37] Timothy A Mann and Yoonsuck Choe. Directed exploration in reinforcement learning with transferred knowledge. In European Workshop on Reinforcement Learning, pages 59-76. PMLR, 2013.
[38] Gabriel B Margolis, Xiang Fu, Yandong Ji, and Pulkit Agrawal. Learning physically grounded robot vision with active sensing motor policies. In CoRL, 2023.
[39]
Bhairav Mehta, Manfred Diaz, Florian Golemo, Christopher Pal, and Liam Paull. Active domain randomization. In CoRL, 2020.
[40] Marius Memmel, Andrew Wagenmaker, Chuning Zhu, Patrick Yin, Dieter Fox, and Abhishek Gupta. Asid: Active exploration for system identification in robotic manipulation. arXiv preprint arXiv:2404. 12308, 2024.
[41] Zak Mhammedi, Adam Block, Dylan J Foster, and Alexander Rakhlin. Efficient model-free exploration in low-rank mdps. Advances in Neural Information Processing Systems, 36, 2024.
[42] Zakaria Mhammedi, Dylan J Foster, and Alexander Rakhlin. The power of resets in online reinforcement learning. arXiv preprint arXiv:2404. 15417, 2024.
[43] Aditya Modi, Jinglin Chen, Akshay Krishnamurthy, Nan Jiang, and Alekh Agarwal. Model- free representation learning and exploration in low-rank mdps. Journal of Machine Learning Research, 1-76) 2024.
[44] Fabio Muratore, Michael Gienger, and Jan Peters. Assessing transferability from simulation to reality for reinforcement learning. IEEE transactions on pattern analysis and machine intelligence, 2019.
[45] Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. Advances in neural information processing systems, 29, 2016.
[46] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[47] Seohong Park, Oleh Rybkin, and Sergey Levine. Metra: Scalable unsupervised rl with metric- aware abstraction. arXiv preprint arXiv:2310.088
2023.
[48] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by supersuue prediction. In International conference on machine learning. pages 2778- 2787. PMLR, 2017.
[49] Xue Bin Peng. Marcin Andrychowicz, Wojeiech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference on robotics and automation (ICRA), pages 3803-3810. 10. IEEE, 2018.
[50] Xue Bin Peng, Erwin Coumans, Tingnan Zhang, Tsang- Wei Lee, Jie Tan, and Sergey Levine. Learning agile robotic locomotion skills by imitating animals. arXiv preprint arXiv:2004.00784, 2020.
[51] Antonin Rafiin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselinss;, Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1-8, 2021.
[52] Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprini urXiv::709 10087, 2017.
[53] FL Silva, Jiachen Yang, Mikel Landajuela, Andre Goncalves, Alexander Ladd, Daniel Faissol, and Brenden Petersen. Toward multi-fidelity reinforcement learning for symbolic optimization. Technical report, Lawrence Livermore National Laboratory (LLNL), Livermore, CA (United States), 2023.
[54] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mas- tering the game of go with deep neural networks and tree search. nature, 529((587)) 489, 2016.
13

---

[55] Rohan Sinha, James Harrison, Spencer M Richards, and Marco Pavone. Adaptive robust model predictive control with matched and unmatched uncertainty. In 2022 American Control Conference (ACC), 2022.
[56] Yuda Song, Aditi Mavalankar, Wen Sun, and Sicun Gao. Provably efficient model-based policy adaptation. arXiv preprint wrxiv:206.08005/ 2020.
[57] Yuda Song, Yifei Zhou, Ayush Sekhari, J Andrew Bagnell, Akshay Krishnamurthy, and Wen Sun. Hybrid rl: Using both offline and online data can make rl efficient. arXiv preprint
arXiv:2210006718. 2022.
[58] Yanchao Sun, Ruijie Zheng, Xiyao Wang, Andrew Cohen, and Furong Huang. Transfer rl across observation feature spaces via model-based regularization. arXiv preprint arXiv:2201.00248, 2022.
[59] Andrea Tirinzoni, Matteo Pirotta, and Alessandro Lazaric. A fully problem-dependent regret lower bound for finite-horizon mdps. arXiv preprint arXiv:2106. 13013, 2021.
[60] Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Picter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference intelligent robots and systems (IROS), pages 23-30. IEEE, 2017.
[61] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 0022-5033,20 2012. doi: 10.1 WW19//ROSS2012.6388660
[62] Masatoshi Uchara, Xuezhou Zhang, and Wen Sun. Representation learning for online and offline rl in low-rank mdps arXiv preprint arXiv:2110.04652. 2021.
[63] Andrew Wagenmaker and Kevin G Jamieson. Instance-dependent near-optimal policy identifi-
cation in linear mdps via online experiment design. Advances in Neural Information Processing Systems, 35:5968-599 11018,9912222 2022.
[64] Andrew Wagenmaker, Guanya Shi, and Kevin Jamieson. Optimal exploration for model-based rl in nonlinear systems. arXiv preprint 27Xiv:2306.092//, 2023.
[65] Andrew J Wagenmaker, Yifang Chen, Max Simchowitz, Simon Du, and Kevin Jamieson. Reward-free rl is no harder than reward-aware rl in linear markov decision processes. In International Conference on Machine Learning, pages 22430-22456. PMLR, 2022.
[66] Jane X Wang. Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. arXiv preprint arXiv: 2016.
[67] Gellert Weisz, Philip Amortila, Barnabás Janzer, Yasin Abbasi- Yadkori, Nan Jiang, and Csaba Szepesvári. On query-efficient planning in mdps under linear realizability of the optimal state-value function. In Conference on Learning Theory, pages 4355-4385 PMLR, 2021.
[68] Gellért Weisz, András György, Tadashi Kozuno, and Csaba Szepesvári. Confident approxi- mate policy iteration for efficient local planning in -rereliizable mdps. Advances in Neural Information Processing Systems, 35:25547-25559, 2022.
[69] Haotian Ye, Xiaoyu Chen, Liwei Wang, and Simon Shaolei Du On the power of pre-training for generalization in rl: provable benefits and hardness. In International Conference on Machine Learning, pages 39770-398808 39800. PMLR, 2023.
[70] Dong Yin, Botao Hao, Yasin Yadkori, Nevena Lazić, and Csaba Szepesvári. Efficient local planning with linear function approximation. In International Conference on Algorithmic Learning Theory, pages 165-1 192.. PMLR, 2022.
[71] Andrea Zanette, Alessandro Lazaric, Mykel Kochenderfer, and Emma Brunskill, Provably efficient reward-agnostic navigation with linear value iteration. Advances in Neural Information Processing Systems, 33:11756-111 1166,, 2020.
[72] Tianjun Zhang. Tongzheng Ren, Mengjiao Yang, Joseph Gonzalez, Dale Schuurmans, and Bo Dai. Making linear mdps practical via contrastive representation learning. In International Conference on Machine Learning, pages 26447-26466. PMLR, 2022.
[73] Yunchu Zhang, Liyiming Ke, Abhay Deshpande, Abhishek Gupta, and Siddhartha Srinivasa. Cherry-picking with reinforcement learning. arXiv preprint arXiv:2303.05508 055005 15, 2023.
14

---

[74] Wenshuai Zhao, Jorge Peña Queralta, and Tomi Westerlund. Sim-to-ree transfer in deep reinforcement learning for robotics: a survey. In 2020 IEEE symposium series on computational intelligence (SSCI), pages 737-744. IEFF, 2020.
15

---

A Technical Results
We denote the state-visitations for some policy as (s, == p [(shs an) (s, a)l, w (2) Pr [(5,, an) 2), for 2 C x A. For X' C R2, denote w (X) = Ip [P an) E X) for Φ the featurization of the environment.
Lemma A.I. Consider MDPs M and M with transition kernels P and P. Assume that both M and M start in the same state By and that, for each (s, a, h)::
Ph( 8.a) Pr.. s, s, I Csim-
(A.1)
Consider some reward function r such that E (n(sn, and S R for all possible sequences (sn, Then it follows thut, for any and (8, a. h.)
10 M. (s.
2n M (s,a) S HR Calm-
Proof. We prove this by induction. First, assume that for some and all 8, a, we have (. (s, a) @ (s, a)l S 61+1. By definition we have
MM (s, a) = rn(s,a) + EM (MMIQ (s ax+1) sh = s, an a)
and similarly for Q
M 1-11 (s, a). Thus:
N (s (s) - Q
M (s, a)l
(a)
|B M.T 1Q M.r (sh+1, a 88 8,0, a] M.= 1Q
M.r (8h41, am41)
8h = =, 0, =a]
MM 10 (9 - Q
M. (sh+1, ah+1)l 1)1 sh s, s, a)
M. M. ($1+1, ax+1) 8h = s, aj = a] - E T Q
M.. h+1 (sh+1, an+1) 8) = s, an a] + Eb+1
where (a) follows from the triangle inequality and (b) follows from the inductive hypothesis. Under (A.1), bound
[E (Q (SN+1, ax+1) 8h. = s, = al-EaM M (Q M.r (sh+1, ax+1) 8/ = a, = all S Csisu R.
It follows that for any (s, a), 12 MiT (s) Q कF (s, a)l en Csim R. ch+1.
The base case follows trivially with eи 0 since for any MDP have that QM (s, a) (s a) 1M (s,
Lemma A.2. Under the same setting as Lemma A.l and for any h, n. and Z S x A. we have
M.. h
(Z) M. wh (Z)) S. Hesim
Proof. This is an immediate consequence of Lemma A.1 since, setting the reward (sr a) ({(s,a) E Z, = hJ. we can set R and have V.M.* = w M.x (.))( (Z).
Lemma A.3 (Proposition 2). Under Assumption I, we have that
Vreall tim
< 2H2 Coim
and V ysin a ral.
2H² Cyim.
Proof. We prove the result for real the result for sim follows analogously. We have
reell,
y amm
reell, ral..
sim rol,
vo
toll
sim,
Sm
real
50
Vyrel rr mil.
mll
simm sn.
real
Am,
The result then follows by applying Lemma A.I to bound each of these terms by H Esim-
16

---

Lemma A.4. For any f E F,
#-1
Vo V0y¹
max TE(n/,
2 E [fa(sh; ah) T (8h+
,
h=0
Proof. We write
VoL Vot Vo
max fo(so, a) max fo(so, a) vo
(a)
(6)
and then bound each of these terms separately. By Lemma 5 of [57] have
(a)
ET
[fa(sh,
an) rh max S1+1
H
Σ
E= (shh E(ra + max fatt1 s/ anll
A=0
Similarly, by Lemma of [57] we have
(b) S
E [f (Sh, (s),an)) -Th max fh+ In+1(s14,,
E [fa(sn, 1B/r + max f a') sN, anll
a'
h=0
B Proof of Main Results
In Appendix B.I first provide a general result learning in real when collecting data via fixed set of exploration policies, given a particular coverage assumption. Then in Appendix B.2, we show that by playing set of policies which induce full-rank covariates in sim, these policies provide sufficient coverage for learning in real. Finally in Appendices B.3 and B.4, we use these results to prove Theorems 1 and Throughout the appendix we develop the supporting lemmas for our more general result, Theorem 3. which utilizes the simulator to restrict the version space (i.c. the dependence on (FF) in addition utilizing the simulator aid in exploration.
Throughout this and the following section we assume that Assumption 4 holds. We also assume that fú [0, Vmax instead of fú E [0, H), for some V O. For any f E F, we denote the Bellman residual as
(n(f)ss, T
Note that by assumption on F, we have (s)sss,,a) E I-V Vmaxl.
For any policy т, we denote A a.b (d" an (sn,, an)TT
Esi, (5" (s) [ and A
Necessity of Random Exploration. Algorithm achieves efficient exploration in M by first learning set of policies Hepp in M that span the feature space of M (Line 2). achieving
Amin
EveTTh
EMAi. T (s* an (sh, an)
A min'
(B.1)
and then playing these policies in re coupled with random exploration (Line 4). In particular, Algorithm plays policies from To& T where each Texp π is defined the policy which plays some Texp & up to step h, and then for steps 1/' + 1,..., chooses actions uniformly at random. This use of random exploration is critical to obtaining Theorem Indeed, under our
17

---

transfer model, condition (4.1) of Theorem 1 is not strong, enough to ensure that policies satisfying (B.1) collect rich enough data in M allow for learning a near-optimal policy. While (4.1) is sufficient to guarantee that playing TT eexp M collects data which spans the feature space of M that is, satisfying (B.1) but with the expectation over M replaced by expectation of M this is insufficient for learning, as the following result shows.
Proposition 5. For any fxim S 1/2, there exist some Msim, M and M such that:
1. Both M and 2 satisfy Assumption 1 with M and Assumptions 2 to 4 hold.
2. There exists some policy Texp such that Amin (B Seep (8 ah (*)) ax) = 1/2, vh E [M], and for any T 2 0, if we play Texp on M for ' steps, we haver
infa e (Mool) EM Mexep Vy
M
6sim
Proposition holds because two MDPs may be "close" in the sense of Assumption but admit very different feature representations. As result, transferring a policy that covers the feature space of Mii is not necessarily sufficient for covering the feature space of MMrea which ultimately means that data collected from Texp is unable identify the optimal policy in M Our key technical result, Lemma B.4, shows, however, that under Assumption and (4.1), policies which achieve high coverage in Msiim (i.e. satisfy (B.1)) are able to reach within logarithmic number of steps of relevant states in M While the sample complexity of random exploration typically scales exponentially in the horizon, if the horizon over which
explore only logarithmic, the total complexity is then only polynomial. Theorem critically relies on these facts by playing policies He up step and then exploring randomly, and repeating this for each [H]. show that sufficiently rich data is collected M for learning an c-optimal policy.
B.1 Learning in real with Fixed Exploration Policies
Algorithm sim2real transfer with fixed exploration policies (EXPLOREREAL)
input: exploration policies {= budget T, sim date Dsi. sim regularization 7
2: Play Texp unif & exp 1= (= for episodes real, add data to D
3: for h = H, H 1, do
4:
Fr. + arg min FEF
Σ
(ua(s,a) -r max (4t2(5',a'13" o
(..0, om
In. + arg min fEF
Σ
(fh(s, - -- max (ntzls',aa))
(s,a.ns"))D),
(B.2)
Σ
(fa(s,a) Fn(s,a))2 s
s.t.
2
( pm
5: return
Lemma B.I. Consider running Algorithm 3. Assume that D Dem was generated as in Assumption 5, via the procedure of Lemma C.3 run with some parameter 8,, and yy satisfies
43V2
82
8M
SH log
2/2
2
max
logg
+ +6V2
max 3
8
max sim
dH
8
dH
Furthermore, assume that there exists some e 0 such that, for any म. h E [H), and Z' C we have:
real.7 (2') < C w.,
real, (2') +
(B.3)
h
18

---

Then with probability at least 1 28, the policy N generated by Algorithm 3 satisfies
256V log(4H)
(4
sim exp
1/6)
V/ Vo² S V
max
+ 4HH
T
for
F(mexp = (SEF: Esim, (Sn, an. - (ani)==mmnnnnaaiin az)) 27, vh (H]).
Proof. Let & denote the good event of Lemma B.2, which holds with probability at least 1 25. By Lemma A.4 we have
H Σ
real
Vo V-eal,
Frea [F [fa(s), an) - f am)
max (x/ ,x
hm
1-11
< max
21 greal (f),, an]]] (1
h=0
Let
(eta (f), [Vmax 2 Vmax 2 1-.t 2n, == ((),a)=
Then we have, for any I,
Ereal 11 (F)
anl
(Zn,i) Vmax2 1
real.s
i=1
< C
wh
real, (Zn,s) Vmax2 2 it1
- + 2Vmaxe
S 2e Mexp (( an) 2V
where the second inequality follows from (B.3). On E, by Lemma B.2 and Jensen's inequality, have
2H| ( sim
Texp [| ( ax)l] S V Breal. Egroal (f) an )n) s V T
256V2
max log
8
As this holds for each h and n, we have therefore shown that
real 10 1/real,5
< 4e-
1
256V2 max log
2dd Fr (meim
4H V V
2/H F
sim
expp)
< 4CH
256V2 max log
+ 4HV m
8
This proves the result.
Lemma B.2. With probability at least 26, for each h E [H] simultaneously. as long as the conditions on given Lemma B.3 hold, we have
Fexp (Sn, an) - T
1st an a1)) T
256V2 max 1og(2H((F, 11/),,
T
and Fn E (m for all h E [H]. where
F (Tei)) = Ef : 3fr+1 Fr41 s.I. asimatt si [ffnss, an) Timm fn+1 an)) < 27
Proof. Let Fr denote the feasible set of (B.2) at step h. By Lemma B.3, with probability least 8, Ft. C Fr. and, furthermore, that Treal f 1 is feasible. The result then follows from Lemma of [57], since the constraint on the regression problem restricts the version space.
19

---

Lemma B.3. Assume that data in sim is generated as in Assumption 5 via the procedure of
Lemma C.3 run with some parameter and y satisfies
43V max 8
8M|
log
SH|Ful
2V
m sim
log
6V max
S 7.
dH
d.H
Then with probability at least 1 8 we have, for each h [H]
1. T Trol is feasible for (B.2).
2. The of feasible f for (B.2) is subset of
{f E F Jesim, [(fa ap.) - - (7 fn+1 an))²] 2y}.
Proof. By Lemma C.I, we have that with probability at least 8/2HH
Tin
512V2
F
2 21og
Jr 6
(T Jo+1 (5 (h) [r aT)) < 2V f
max
log
V22
max
Tsim
Tsim
Tsim
12dH By Lemma C.3, we have
S Tsmm which implies
Tsim Σ (T FA+1 (5, kh) f, ah)) < 2V max calm
Tin
m
43V2 82
max
.log
FH
V22
max
B1
log 4H1
dH
6dH
Part then follows given assumption 7,
To bound the feasible set for (B.2) we appeal to Lemma C.2 which states that with probability at least 1 6/2H we have that the feasible set of (B.2) is a subset of
log Tsim
8 SHI
fn E Fr
Essim Tees (Sfn an) 7 f -- 5
5 18V.2 max
Again using that
12dH B
Tai. have have that this is subset of
{ In Fn Rsim, exp (JJ an) -Tsim Ja+i
log 8
18V2 max B\
12dH
S f E ssim. (Sn, an) Tism Fr 1 (sh )) 27)
where the inclusion follows from our assumption on The result then follows from a union bound.
B.2 Performance of Full-Rank sim Policies in real
Lemma B.4, Consider policies (rrppp TH and assume that
Amin
AA An
Amins
Wh [H]
(B.4)
and that The plays actions uniformly at random for hi' > h. Let Texp mif(((=(pp)) any n. 7 O, h E [H], and Z' S A, we have
1 1). Then, for
AH -2
wh
real. (z')
<
real (z') + 4K,
where
A d + H
and k² = log 1 1/R.
E : 2
Hesi
Xmnin
7
log 1/E
20

---

Proof. Denote
2h+1 = {(s,a) : ('(s
(A (+p ('s,, 7
for some 7 O. We have
W/, Wh+1 (2++1) Ereal ([(S)+1,ah+))] E Zn+ill
rea
Φ (h+h)),a++1) Thisp 11t11) ( ah+1) [(sh+), En+i}
E exp
Φ ((sn-1, (A popp. 1o11) ah-1) S Ereal
7
( (Ø) (8 an+1) (00t1))1((s11111 (h+11,0+11 a tot1))
tr
d
where (a) follows since for all (s, a) E Zat1, we have 1 < <s), h+1 (41 (s, By
Lemma A.2, then have that
ammp (2++1) S d + He
W hh+1
(B.5)
Let Sh+1 = {s: 3a s.t. (s,a) E En+1} and note that
sim,
(2n+1)=
Σ
T s, + (s sh: an)
Wh hh1
Jats a
( |
2
E gismmmre Texp
(T) an)
A
1
E
8h, (x)) Temp à p (Sa+1
A
2 E
1
c [paim
(S
8h, a=)
A
where we have used the fact that Txp
(a
h + 1/A for all (s, a) by assumption, and define Sh. s, 0, a| = Jo... duf(() (s,,,)), where the
(S 8, a) == psim [sh+1
Sh+1
last equality follows from the definition of linear MDP. Letting μ1 (S = Js..: duf, (s), note that:
E gssm a (paim(i.tt
Sh, a1
A 11, (Sh+1) Bsim, (sh an ( and (5
A
A 14 (Sa+1)
(5x+1))
Amin
where the last inequality follows from (B.4). Combining this with (B.5), we have
d
Xmii
Hesim (1
Now note that, for any 2 E S x A:
d.ppim 2) (1sm dui((s) 0²(x) S (1
paim (Sa+1 22) ==)=
21

---

and we also have that Ph (S 2) 2 ( 22 Csim under Assumption 1. Putting this together we have that for all E S x A:
A Xmin
d +
Pheal
(Sp+1 (
=x)= S
Hesim
6sim-
Note that can always take esim
1, and will always have Amin 1. This implies that Esim S
H (esim) ) Thus,
Amin 7
A Amiin
d + H Heaim =:
Preal (Sn+1 12) s 22
Coverage of Texp in real. Let K* == log 1/k7 log 1/5 so that ght S.. Let 2n, = (S x A)\2,. Fix some
Z' S (S A), h E [H]. and policy
Consider some E 2n. and some S' S. Then note that
Pl: P (S' 2 = (S)) (TO)T((s ()))) = PA n (1 h-1 .n)
-1/2 111230(2>
S (1 16-2 a
1| (=)(ka)
h-1
h-1
where the last inequality follows from the definition of 2n. Note, though, that
11 (5)
F h-1
(S") (2 (SE)))))))
Ereal A
[Preal(' 2/ )3.
This implies that for all E Zn.
greal, [Preal(S') 2n) 2 Y
.rrea 2)².
For h < h, define
Swii {s wh
real, (z') 8h =s) {2 [2-1+1 (2-141,,-1))
for w
real, (2 8/. 8) = preal, [zh Z sh s). Note that then have w (2' (we,))
[2-1+] )- By what we have just shown, have that for E 2n.
E Breal My "Iprea (SW41, 2x)) ?
1
(SM+1, 1
which implies that
F -1
[ (SW+1a
2xr)] 2 7
1
preal (S 2)2
(B.6)
$If A' K-1 A. esp
is not invertible, we can repeat this argument with A n-- + Al and take + 0.
Sesp
22

---

Fix E Bw. Note that
( Wh whall (Z') 2/ 2) E (-))))))))) real, (z')
Sh1+1 s)]
Een
(1))
real.r (3'
88'41 = s) 1{s E Sw+1all
2
Par (Sh+1.112)
[1oR4/w]
Σ
2 1 pa (Sw++1, 2) +
i=1
|log 4/xl
2 Pp (Sw+4,4 2) I(Pp (Sw+1,4 12) x) + 3x
1=1
[log 4/ml
S
E EgNA wau)
Sh/+1
(Sh+11, 2))
22) 2 n} + 3xc
for any X Asw 11,5" Note also that, since wh'-1 plays randomly for all h" 2 h', have:
h real,
(Z') 1141=s)) Ah- wh "aall (z' S5/+1 = s),
realix
Wh
--11 will play the same sequence of actions since with probability 1/Ah~/ on any given episode, eex as i from steps 1 to h. follows that we can bound the above as:
Llog 1/xJ
hh
Σ
Le
ae (z'
S1/41 s)] 2) (S (2) 2 x} + 3k
(a) 2AA
May
[log [/K]//K]
Bend, [wop
real
(Z') 8h'+1 se)l ki'
(S zw [ (SM 12) 2 x3 + 3x
(SW+11,
(S
K
[log4/kc]
h-
real hl-1
real, h'-1 (2') 81/1 x]]
Eowxi (w/
wh'+11 (SW(11,) 33
R
h- 4/w]
(b)
hi-1 cap
rep
Wh
(2'
8841 s)w (8) 3
R
hh
re exp
(Z') 3k
w/
K
where (a) follows from (B.6) and since (Sw., I 2) z n, and (b) follows choosing Ac(s) =
W p11
real, o
(Sw41.) TI{s E Sw41.)) We therefore have that, for all 2 E Zny
real. (z' EN = 2) < A7 h
real hh'
wh
w
(Z') + 3k.
(B.7)
K
Controlling events.
Consider events ; {zh E Z'} and EN = {zw E Zwr We then have
w/
real.
(2') .preal.r(g" [E]
prreal [E n E + preal.t n E
h-1
h-1
preal. [8 Enr
[g] + I [E n Eh
n
W=h-k
i=h---
h
h-1
Σ
preal.st [8 (gn& -1] + preal - n
N
Eg].
i=h-k
23

---

We now analyze each of these terms. First, note that preal [E 11] [ Ew--1 |
[E - [realit
Ew--1] w/1
real.st (2') 2w E 1w-
We can then bound
27 A7An--1-1
w/
real
(Z' 21-11 11) _
reap
(2') + 31c
where the inequality follows from (B.7). For the second term, have
[g n -
{q] < preat
Ef]
h=h-k
k=h-k h-1
h-j-1
1
preal
cc Oh-
0
E/]
1=
==h=k
Note, however, that preal. n Eg] < and preal (E% therefore can bound the above as
| i= Eg] < & for all j. We
h-j-1
gh*+1 S k.
Altogether, then, have that
2
th
real,
(2))
Σ
wh
(2') + 4n.
W=h-k-+ +1
real 6'-2 (2') S Huy
Furthermore, since Texp = unif( ((TT(TTT)))) [Toxp]) )/m)) we have wh conclude that
real, (2'), so we
4/-
real,m (Z') S h
reap (Z') 4K.
w
B.3 Proof of Unconstrained Upper Bound
Theorem 2. Assume that one of the two conditions is met:
1. For each h, Texp plays actions uniformly random for h h,
Amiu (A 11) 2 Amiu-
(B.8)
and
V4 H H A2 12-(k--)) 22)
log (1/
T 2 C
max
e42 sim
for
K' log
64HV maa
2HA
1, 5= 2
Csim-
log 1/5
Amin
2. Coi s//AH² and
16H1 log
T2
Then with probability at least Algorithm returns a such that V. yreal,
V Vrreal <
Proof. We consider each of the conditions above.

---

Condition 1. First, note that by our assumption on Txx and applying Lemma B.4 with
Ymax and d for any and 3' C S x A, we have
Heam
for H max A real, (2)) + 16H Vaxx
wh
real
(3)
wh
ééaim
log
64NV
2HA
Csim-
log
1/g
Amin
By Lemma B.I we then have that, with probability at least 1 - 8
Vreal " 1o /reeal
256dHV mnax -
256V2 max
1og(2H + c/4
4H
66sim
T
S <e/2
where the last inequality follows under condition on T.
Condition 2. By Lemma A.3, we have that V,real, yreal, am.. S 2112 Thus, if Esim S <e/AH1 we have real reall, i E/2
Concluding the Proof. By what have shown, long of our conditions is met, will have that with probability at least 1 - 8/2, there exists ı { π simm+)) such that V-real, /rreal, e/2. Denote this policy म.
Note that V. greal, E rn] and that 2 th E [0, H] almost surely. Consider playing for T/ episodes in real and let R' denote the total return of the ith episode. Let
T/4
V
4
R.
T
By Hoeffding's inequality we have that, with probability at least 8/4:
4 log 1
V0 Viall,
HV
Thus, if
16H² log
T
(B.9)
have that V0. Vo
< c/2. Union bounding over this for both E frr sim.,,) }, have
that with probability at least
- 8/2:
/real,
7 0
e/4
7 - -e/4 2 re - -e/2.
It follows that
real
Vi r
<
real,
Vo rea,
+ e/2 S.
The proof follows from a union bound and our condition on (note that (B.9) satisfied in both cases).
'Note that, while Lemma B. applies to the constrained regression setting. this is equivalent to the uncon- strained regression setting considered here if choose large enough that the constraint vacuous.
25

---

Proof of Theorem 1. We first assume that Ç Algorithm by Algorithm 1, and Condition of Theorem and show that in this case A k is most polynomial in problem parameters.
S Auia for Ç the input regularization value given to 4d
First, by Lemma C.7 we have that, under the assumption that Ç < Amin
4d the policy Tipp given by
the uniform mixture policies returned by Algorithm 5 will, with probability at least 6, satisfy
Amin( (ein(A (As
Meep 1
Amtt under Assumption 3. Plugging Xmin 8d
Amia into Theorem 2, we have that 8d
8 2/
I6dHA
X
Esim
Now note that
64H log 125
64HV max
1/log 1/f
A -
S
It then suffices that show 1/ log, 1/551 S 1/A2 However, this is clearly met by condition on 6sim- Thus, as long as
V6 H (ogg((277))/00)) max
T 2
2 cim
by Theorem we have that स is e-optimal.
Now, if 6sim S e/4H2 and T 16H log 1/6 the first case, we most will require
we also have that स is c-optimal, by Theorem 2. Thus,
V6 1 (2g(2H7
T 22c c. max
produce policy that is e-optimal, since otherwise we will be in the second case.
It remains to justify the assumption that Ç Amua Note that the condition of (4.1) is only required in the first case. Furthermore, 6eim S c/4H² we will be the second case. Thus, in the first case, we will have
Xmin 64dH A3
S Esim S
4H²
Rearranging this we obtain that, to be in the first case, we have
16dA3
Amin
H
By our choice of <= 4 that Oracle 4.2 is called at most poly(d, Ht, e 1 log {) times, and we call the oracle of Oracle 4.1 only times. The result the follows from a union bound and rescaling 6.
H we then have that < Aman By Lemma C.7 and our choice of Ç, we have 4d
B.4 Reducing the Version Space
As noted, general, given that we do not assume that of is unknown, log |Fl could be significantly greater than the dimension. One might hope that, given access to M we can reduce this dependence somewhat. We next show that this is possible given access the following constrained regression oracle.
Oracle B.1 (Constrained Regression Oracle). We assume access to regression oracle such that, for any h and datasets ((s', a', T and {(st, o
can compute:
f
fn = arg min
- 222 s.t.
EUG - 5イ2 2 ST
FEF
t=1
While in general the oracle of Oracle B.1 cannot be reduced the oracle of Oracle 4.1, under certain conditions on F this is possible. Given this oracle, we have the following result.
26

---

Theorem 3. Assume that feim
X 64d
Then
if
11
H
T T2C δ
loo 8
with probability least 1 5, Algorithm returns policy it such that Vo Vreaal,
-Vorall Σ 6, where
F:= -==
F sup (E [fa(sh; an)) T futi (s);)))))))))))))))))))))))))))) an)))) a Sim
for = Ö (AAHH -log² log 1F//)) Furthermore, the computation oracles of Oracle 4.2 and Oracle B.1 saim
are called at most poly(d, A, H, e ,oog ( times.
Theorem 3 shows that, rather than paying for the full complexity of F, we can pay only for the subset of that is Bellman-consistent on M
B.4.1 Algorithm and Proof
Algorithm 4 sim-to-real transfer via simulated exploration (SIM2EXPLORE)
I: input: tolerance e, confidence ò, budget T, Q-value function class JF
2: Hxxp + LEARNEXPPOLICIES(
6, 1 H h) for all h [H]
3: & + O(log2 AdH
4: for 1
do
5:
EL 2 T + T/2,, rf + 10V2
max (Et)2
6:
Run exploration procedure Lemma C.3 with 3r 2012.x log S to obtain D
sim
7:
7
EXPLOREREAL
T Dim. 7²) (Algorithm 3)
8:
0 f- average return running 7' in real T /2 times
arg max V
9:
return
Theorem 4. Assume that one of the two conditions is met:
1. For each h, Tesp plays actions uniformly random for h' h.
Amin
A 2 Amin:
(B.
and
Vi c2c max
H A2K 2), log (1g(161111100)
ssi
for
b* log 64H
2HA
g ==2 log 1/E
feim
Amin
and
F F: [EF
sup
(8h, an) (
TTsm (1 an)))
log 32H
48d log
32H F
S log
+1
AdHV2 log max
V sim
Vmax sim
sim
2. Eiim Se/16H2 and
H², log 16c
T c
Then with probability at least 8, Algorithm returns a policy के such that V to
1.real. S
Proof. We break the proof into two cases.
27

---

Case 1: Caim 2 e/16H". Let f = llog falm and note that & in this case and that this is deterministic quantity. Further, note that 70 [10 max Note that by our assumption on Fexp and applying Lemma B.4 with UTHV and 7 Heam for any и and 2' _S A, we have
a 40V max sim and e [ssimm 26 d
max
real,x (z') S w/
H5
max A'
Feap, (2') + 16HV Max
wh
ćésim
for
log 64NV
2HA
1. 8=
log A 1/5
Amin
Csim-
By Lemma B.I, as long as oe and o satisty
43V2
F
log
811
2V2
log max
66V2
3r1 max
S
(B.11)
max sim
d.H
6
dH
we have that with probability at least 20,
256dHV mmnx A
256V2 log (
((1))) + e//4
Vrreala
4H
66
T
where
F =
Igsim [(fr an) - Tis fa- (sk an)) 2 vh [H]].
However, since V Feim S 10 2 and by our choice of 8 2012 max log sui= we see that (B.11)
met, so the conclusion holds. Note that, by Lemma C.5, we have that with probability least
Fd c (EE sup (Esim. [Sa (sh. an) T-si
on))))
1
48d Br
48d B
1 8
4 4log Bi
+ 6) A 48dH log
2 + V22
96dH log
log
B1
SfEEF sup (E [fr (sr, and -Tsim fh+ (sh, an)))
log 8H|P
48d 801
Sc log 2
Vmax
&
+1
AdHV2 max log
V Vmaa sim
cim
where the second inclusion follows from our setting of Br, and bounds on 7 Since T T/2t, it follows that if
de &H -2(
-2), (oggo (1H
T c
max
c22 sim
then we have that V y-real, val..
x I </22.
Case 2: 6sim S cc/16H2 By Lemma B.5 and our choice of T! we have that with probability
least 1 - 5,
20 max log
SHIFI 8
960dV2 max log
SHI F
0
6H
2 log
+3 192AdH log
74 4H² esim-
By our choice of (logg real, 0
Vana AdH and since 7" 10V2 2aax (2))2 10V2 max 2 24 we can bound
< </2
28

---

Completing the Proof. In either case, we have that with probability at least 1 - 6, there exists
some i E [t] such that Vi Vireal 5/2
Note that Vy Ereal SrraltTIE== k== rnJ and that 2n=0 Th E [0, HI] almost surely, Consider playing for n episodes in real and let It' denote the total return of the ith episode. Let
Vo
R
n
By Hoeffding's inequality we have that, with probability at least 8/1:
log 26
Vi real
Thus, if
16H log 2
have that V0
V/rall"0
S c/2. However, a.H² -log 3. this will be met. Union bounding over this for all #f, we 27
each x E f T./2 T/2: times, and in
either case we assume T 2 have that with probability least 1 d:
Vo V
4/42
0
e/42
E/2.
It follows that
Treal,
/real,
Vo V-real,
+ tc/2
The result then follows from union bound and rescaling d.
Proof of Theorem 3. The argument follows analogously to the proof of Theorem 1, but using Theo- rem 4 in place of Theorem 2. The bound the number of oracle calls follows from Lemma C.3 and our choice of Be.
Lemma B.5. With probability at least 5. for some E, we have
1/sim,
V/ s/iim
S 6H
2log
2012 max
log
8
+3 3
192AdH log
960dV22
8H max log 8 7
g6
7
20V2
log 8
960dV2
8H max log b
real,
I 6H
2
max
+3 3
192AdH log
78 4H² Gim
Proof. By Lemma C.4 we have, with probability at least 1- 8,
siim,
0
< 2H
2 2
Be
1
+3)
Bev
512V2 max
Al Alog
8H F
8
+
48d 96AdH log
48d
2
2AV2
96dH log
log
3ee
max
3²
2012
log
8H
960dV2
log 8HI
S 6H
2 log
max
+ +
192AdH log
max
where the second inequality holds by our setting of 3z.
We have
y real,
//raal,
reall Sm..
+ V-real, real
sim Sm.
sim Sm.-
s'
+ sim,
real
29

---

By Lemma A.3, we can bound
Vo real
/e Sin
S 2H2 Eaim
and by Lemma A.I we can bound
sim Combining this with our bound on Voim, is
0
Vreal.
S 122 Esims
Vismm
-V 1/real,
S esim-
gives the result.
C Learning in sim
In this section we provide additional supporting lemmas for our main results and in particular, we focus on linear in sim. In Appendix C.1 we provide several technical results critical to showing that sim can be utilized to restrict the version space, as done in Theorem version space using sim, sufficiently rich data must be collected from sim, and in Appendix C.2 provide results on this data collection. Finally, in Appendix C.3 we provide a procedure to compute the exploration policies in sim which we ultimately transfer to real.
order to restrict the
In Appendices C.I and C.2, we let hypothesis f anddd f be defined recursively as:
Tam
f arg min F.EFR
T (1n 7% max (1,41,1/,,,111)
and fn E Fn some hypothesis satisfying
Tan
1 Tsim
Σ (F..
- (( s
for parameter 0.
In Appendix C.1 we make the following assumption on the data generating process.
Assumption 5. Consider the dataset Dam [(3 Zb ...... - ~H 7/ 1)) We assume that episode in Deim was generated by playing an Fi---masuuublle policy Texp and denote exp unif ((Fe(( 1Tam)
We provide a specific instantiation Tesp in Appendix C.2. In Appendix C.3, we provide procedure for learning set of policies which induce full-rank covariates in sim, crucial piece in obtaining good exploration performance real.
C.1 Regularizing with Data from sim
Lemma C.1. With probability at least - 8:
512V2 Tim
4|FMl 8
2 2log Jul
T sim
(Traal Jn+ (3h. [h 2))) < 2V max sim
maxx
max
log
+ V22
max
T
5
Proof.
First, note that Treal Pr+t E Fr by Assumption
By Azoff-Hogffding and union bound, we have that, with probability at least 1 - 6, for each
T (fn f3 2z <
Tim E s Feep [[fn 2n) - -f (hh + V2
2 log F Tsim
Tyim 1m
Tgim 1m1
max
2 log F/ Tsim
ssi [V5n (Sn, an) - (s)))))a))) V2 max
30

---

In particular, this implies that
Tum
2 Fn11/0 Tsim
real fn+1 (3h, (h) (hh ah)) s pxsm [(Trea) fn+1 an) - (sh, an)) ] + V
T
We can bound
Esim wxp [(Trea) ( an) Fa (sh, an)) s 2m " (Trraal fh+ (sh: an) -T sim fn+1 (s/, an))²
2m (T fitt1 an) (sn. (n):)))))
(6)
To bound (a), we note that
T fn an) fih an) Ercal
max
(1+1(s'00f s, E [max fn+ils', a')
a
Preat (s' s, psim | s.a)) max Fn+1 (s',a')
$ V Vmaxx
Σ Prreal (s' s,a) Prim (s' s,a)]
where the last inequality follows under Assumption 1. This gives that (a) S 2V22 max 2 To bound (b), we apply Lemma [57], which gives that with probability at least 5,
512V2
4|F%
(b) S
llo
Tsim
Combining these with union bound gives the result.
Lemma C.2. Consider the set
Tam
F fn Fn
Tsim Σ (fh 2h)) F.G, {=))))ppppp sn
Then with probability 1 28 we have
log A/Fi Tsim
P. c In J Rsim
(Ja, an) - -Tsim -Tsimmnntt (8h, S + 1812
max
Proof. By Azema--o,,fiding, have that with probability least 1 ō, for each sh, Sh E Fr.
2 log F/8/8
Tam
gsim,=co) [(/n(sn an) - (sn, an)) V2
Σ
-
(n
f1, af))²
max
Tsim
Tem
which implies in particular that, for any Sn. Fr.
2 log 1/8
Tam
B sim, ooo
- Fr(Sh, an)) - V-2 y
(3).
(.) - a)))
Tsim
T
We can write
1 simm, cxp (sn, an) - Fa(sn, 2=
p ([n(sn, an) T Pr+t (sn, ann)) + -EESii ssim [[Ta (n)),,a))= - Ti Jn+ (sh, )
m [(Frs (sn, an) - si af)
T fn+ an))
B p ([nnssh an) T-si Pr.. (sh, an))
2m u [Jr((h,, an) - si ( an) sim Pn - ax))
31

---

By Lemma of [57], with probability at least 6,
256V max
2|
Esim, (ffa(sh: an) Tsim {
an))
llog 6
Tsim
We can therefore bound the final term as
gsim, T
[[Ja (sh; an) - 7 Saimmn-- fn-1 an (Sa))[(((s))) an) --si Pn- ( an))]
S Viax Exim (s,,,
(sh, ah) -Tsim fi+t 1 (sh: ax)ll
S Vmax
V sism
(Fa(sh: ahh 7 sim 1 (h))
256V2
2|
S Vmax
max
log
Tsim
5
Altogether then we have shown that, for any Sn E Fr. with probability at least 1 - 26::
Tii
log Fi Tsim
Σ (sh
F1 2 2 Esim dm
[ffn an) T ---immmn fn+ an)) 18V2 max"
Timm
Thus,
Tim
(3, - (,, sn
T sim
then
log F//81
psim, ((fa an) -Tssmm Fa+1(3h, 18V2
mux
Taim
The result follows from union bound.
C.2 Data Collection with CoverTraj
Lemma C.3. Consider running the COVERTR. AJ algorithm of [65] each E [H] with parameters < [log 1/8) and 7/2 <- 33 for some [0, 1}. and with REGMIN set the policy optimization oracle of Oracle 4.2. Then this procedure collects
Tsm =
Σ
m
24d
log
48d
2 32
2:
32
episodes, calls the policy optimization oracle at most Tsim times, and produces covariates An.s and sets Nn. such that, for each i [m].
sim.* (Kr.)) S 2-141 and & sup w
333 32,40 E X
and sup, wh sim. (B - Rh,,) S 3. Furthermore, we have
12dH
48dH
48d
< Tsim S
log
g2
g
g2
Proof. Instantiating REGMIN with the oracle of Oracle 4.2, we have that Definition 5.1 of [65] is met with C1 C2 0. Therefore, have that each stage collect exactly (using the precise form for K; given in the appendix [65])
24d
48 2'd
K, 22
log
7
77
episodes. The result then follows by Theorem 3 of [65].
32

---

Lemma C.4. Consider running the procedure of Lemma C.3 to collect data. Then with probability least 26, we have
y=
S 2H
1 2log
+3) 3
BV552 max log(4H + A6A log
48d
za YY
2AV2 max
968H log
48d
log
1
B²
Proof. By Lemma A.4:
H-1
simm* 0
//imm
2 E Esim, [fn an) - - Fr+ (S an)l
max
a (nf a
h=0
H-1
2msmm. [almm.lllfffff [1h, an) T-si fatz anlll
max
( (.mm.)).))))
h=0
Denote g(2)) = [ (8h, an) - -Tim Fh+z and Ax-1 = Em An,i + I, for An, collected as in Lemma C.3, and note that
E Esim.x [sim.*(g(2n)) = Esim.E| / g())dPp 2n-1)
= Emm
((2))(a | s)daddp (0
Esmm / (()))(())))))) s)dadlp A 1/2 A 11/2
h-1 h-1 1112000(00 (=(=))))
(C.1)
[|] / u(=)x(a ( (=^-1) ا
S Esimax
=1 -1// ((2)T(a)
Esim [| (2 (ś^-1)) 11) ا
We bound each of these terms separately. First, we have
gsim. [1[0"(=h-1)
|| ا sup Esim, (1(0"(2)1)) Xh-1,s1
Φ 1,i 1,
+
max
| sup Esim. [T(d' E Xx-1,))
DEBB)UU 1u 24 21
7 2 2-111 +3
< (2m + 1)B
where (a) follows from Lemma C.3 and since || $ always.
33

---

We turn now to bounding the first term. Note that
s(2)m(a) s)dadp (s))la
Tim
/ g(2)T(al
( --
Tim 1- (g(h)) 12
Tim
E [g(22 2/) 12
(a)
Tum
A
E Eaxep
A-1.6
19(2))2 1
(=1
Tim
Texp
h-1it
(f ah) Tsim Fitz an)) 1
Tim
Tim
2A
B Toxp
A-1if
[[fassh an)
Tsim Fu+z an)) 1 2A
E
b-1:f Eexp
[[hh ax)
Fn an))² ( 1 2
(b)
512V2 512V2 Alg 2A
Tim
E an) (sh, an)) 2
Exexp A-1i+
A-1 plays actions randomly at step and (b) holds with probability at where (a) uses the fact that Texp least 1 by Lemma C.6. By Azuma-Hoeflding, we have with probability 6:
E [(Fa ax) Pa an))² 12h11] s 2 (fr - ()))) + 2V max MaxxTsim log 1/6
1=1
< Tsim + 22 max Tsim Tii log 1/6
where the last inequality follows from the definition of Jn. Altogether then we have shown that, with probability least 28:
Visim,+
V/iim
22 + 1)8 512V2 max A Alg(44H)5)/0) + 2AT simm + 2AV2 max V 2T log 1/5.
Using that Tyim
48dH
log 48d given in Lemma C.3, we can bound this as
48d
< 22m + +1) 8 AlogennAooool4 + 96AdH log
ga
48d I6pH log
2AV2 max
log
3
B2
The result follows.
Lemma C.5. Assume that
dm
[ff on)
T fn+1 an)) (n)) S 5
34

---

Then this implies that, with probability at least 1 - Ó,
sup (ESim. [fn an)) Tis (sh, ax))2
log 6) A. 48d log
48d
48d
1
32 7
Vmax
96d log
log
Therefore.
(EEF p sit [[hn(sn an) T Sat1 (8n, an))) S
c STEE : sup (Esim, [fn (fnsss an) ) - -Tsim (7siifnn fh+1 an)))
48d
48d
1
}
4 log 6) A 48d log
B² 9
V
max
96dH log 8a 8 18
log
Proof. We follow similar argument the proof of Lemma C.4. Denoting g(th) == fsh an) Tsim 1 an), by the same calculation as (C.1) we have
Esim, [g( - / g(=)<{a| s)dadph (+))
Esim. (1h (100))1)
and as in the proof of Lemma C.4, can bound
psim. [= A| ]s (2m + 1)8
and
Tim
|| / f
( s)dadpp S
A
Texp
- T ( on)) 1
By Azuma-Hoefidingg with probability least 8 we can then bound
Tim
A-116 exp
[[fn an) sim f a2)) 2//111 S Tsim
sin [(fi(S), an) T-si fn+i an))
/ 222 max Tsim log 1/5
S Tsim? y + / /2V1 max Tsi log 1/5
where the last inequality follows by assumption, and where Texp uniuui unif(( ((mp)-1))))))) TTn) Altogether then, for all ग,, we have
Esim [fn an) Tism fa+1 (sh, an) S (2m 1)8- v ATsim AV2 max 2T sim log 1/8.
Using that Tsim <
18dH ge
log
48d zo
given in Lemma C.3, bound this as
48d + (2m + 1)
48d log
< (2m + 1)
48AdH log
AV2 max
96dH log
8.
ga
g2
The result follows from some algebra.
Lemma C.6. With probability at least 1 - 8, for each h E [H] simultaneously, we have
g a h-
[[Fn(sn, an) 7
1 af S 256V2 2ax 1og(4H)
Proof. This follows from Lemma of [57].
35

---

Algorithm 5 Learn Exploration Policies in M (LEARNEXPPOLICIES)
1: input: environment M, confidence 5, regularization ç. step
2: AR -- policy optimization oracle of Oracle 4.2
3: for 3, ( ( log 3 < log ))) do
4: N, f | 1, K2 [22:/$ T +- (NV + 1) 83
5
1ja
// DynamicOED algorithm from [64]
5: Σj _ DYNAMICOED((), N,, K,. 8j. AR) for ((An) t tr((An + I)
6: if' Aminn (2,) 2 12544dlog 4+64T; and T3 2 c 6 log
3/2 iT then
7: break
8:
return [T
C.3
Learning Full-Rank Policies
We consider running the MINEIG algorithm (Algorithm 6) of [64] in sim. For fixed h. we instantiate the setting of Appendix C of [64] with (TT) ((sa, On. ((()))((s),,) os. D 1, and Ar the policy optimization oracle of Oracle 4.2 (and so CR = 0), and set N = for MINEIG. We note that this algorithm is computationally efficient, given a policy optimization oracle.
Lemma C.7. For M +- -Miim, Algorithum will call Oracle 4.2 most ( llo 2l00-.10" 1
times, and with probability at least 1 5, under Assumption 3 and if S such that
Amin
will return policies IT 4d
min 8d
Amia
A
2
(C.2)
[I|
FEll
and each π E II plays actions randomly for h l..
Proof. We first argue that, if < Xmin, then with probability at least 1 - 6, (C.2) holds. Let 4d
denote the success event of each call to DYNAMICOED, and note that by our choice of on, we have P|E] 2 - 8/2. Let j* denote the minimal value of j such that
Ami Ti 2 log 4+ 64T, 4d
l/2 JT,
and T₁ 2. log 3/2
(C.3)
By Lemma C.4 of [64] and if 5 S Auma we then have that, on E, Amin (E,-) 2 2nt T,., which Ad
dd
implies that the termination criteria of Algorithm 5 will be met. By Lemma C.5 of [64], it follows that with probability least - 6/2, we have Amin TT Exeu, A,, 2 the desired conclusion.
Aima (since Tj- [1I D), 8d
Assume that Algorithm 5 terminates for some j < This implies that 4d T, < 12544dlog 4+64T, However, in this case, we then have that
4 64T,
XminT, 4d
Amin (E)) 2 12544d log
From Lemma C.5 [64], it then follows that with probability at least - 8/2, we have Amin' II,T ΣrEn, As (,,h) 2 Xuia.
It follows that, assuming T, is large enough that (C.3) is met, and we are in the case when Ç S Amai holds, then Algorithm will terminate and return a set of policies satisfying (C.2), with 4d
probability at least - 6. Note that T, = (22)). Given that Algorithm does not terminate until j O(log2{ log 1 + c 9. 1og 4) Ologg & log --9 log {)), will
have that T, will be large enough that (C.3) is met, if < Amin, The proof then follows since DYNAMICÕED calls Oracle 4.2 most T3 times at round j. and the total sum of T, is bounded
4d
as Õ log $ + <-9 log2/2 3/2 4) by the maximum of j, and since the actions chosen by π E Il for h' h are irrelevant for the operation of DYNAMICOED, they can be set to random.
36

---

D Lower Bound Proofs
D.1 Proof of Propositions 1, and
Construction. Consider the following variation of the combination lock. We let the action space A {1, 2), and assume there are two states, S = {si, $2), and horizon H. We start in state $1. The sim dynamics are given
vh < H 1: Phi (81 $1, 01) = 1, Ph (s2 1, 1,81, 42)
=
Pa (81 | $1, a1) = H H-1 (82 81, 01) ) = Pi (81 $1, 02)
-P (82 81, 02) 1/2
vh E [ (s2 82, = a E {ai, (2).
We define two real instances, Mi :- M and M2 == r where for both we have:
Vh < H - 1: Ph 81, a1) 1,
(s 11, =
Wh [M] ph a) 1,a E a2)
for MMi
Preal (s1 $1, a1) 1/2 + Caim Preal
1f-11 (52 $1, a1) 1/2 Csim
preal 11-1(3) (s1 $1,02) = 1/2 csim Preal (52
$1,02) 02) 1/2 + Caim
and for M2:
preal (81 81, (1; = 1/2 - faim: Preal (82 01] 1/2 + fsims
P H-1 (81 81,02) = 1/2 + feim: PH- (82 81, 81,02) = 1/2 <sim-
Note then that MI, and sim only differ at step H - in state Furthermore, it is easy to see that both Mj and M3 satisfy Assumption with misspecification égim. It is easy to see that Assumption 2 holds as well with d = 4 since this is a tabular MDP, and furthermore Assumption also holds with Amnin 1/4. We define the reward function as (note that this is deterministic, and the same for all instances):
vh E [H] (1 02) 1/22 + esim (1/mn112222 - h/4H)
TH (1181,00 a) 1,a (2),
and all other rewards are taken to be O.
In sim, we see that the optimal policy always plays ag In both Mi and M2, the optimal policy plays aj for all h < H 1, for Mi plays 01 H and for M2 plays 02 at H Note that for both Mj and M2, we have Vi 1/2 + Csim.
The most natural choice of FF would be the set of all tabular Q-value functions, however, this set is infinite, and would require a covering argument to incorporate. For simplicity, consider Fu the set of functions mapping {0, 1). and Fú, the set of functions mapping to finite set containing (0, 1/2 sim, 1/2 + <sim U {1/2 + fiim (1/2 h (/HH)
Note that such a set satisfies Assumption and can construct it such that log [F| S O(H).
Lower Bound for Direct Policy Transfer (Proposition 3). We consider direct sim2real transfer with randomized exploration. In particular, as noted, the optimal policy in sim always plays dg. so consider the 5-greedy policy that at every state plays d2 with probability - and plays unif((a, a2)) with probability Ç. Denote this policy as т. We then wish to lower bound:
inf sup g.M..il..M... M. :E(1.2)
after running our procedure for T episodes. Note that on M1, regardless of the actions â chooses in other states, we have
V. Mait
V.Mi
Gsim (1 # $1)),
>
2
37

---

since the only way π can achieve reward of 1/2 + Esim is by playing in at step H - 1, and all other sequences of actions obtain a reward of most 1/2 + Esim Similarly for M2 we have
Vo VoMai
Vi
Ma.2
Caim (1 1 s1)).
2
Using this, and replacing the max over E {1 with the average of them, we obtain
inf sup BMM,. (Vo
V.
0
2 inf
M..
fsim (1 -an (1))]] + M2.0 a Caim (1 -i })]
(1,2)
2
Esim
BM (Tn-1(a) | + Ma,a -- (a2 s1)1]
1 - 2
sup
2
Since (a1 81) = - (02 81), we have
M [4=1(a1 s1)] + M. 1 (a2 [1)] = 1+ M Ta1 |s1]] - M2. 1 1 s1))
2 TV pM22a)
S V KI.(PMM.. || PMM..))
where TV denotes the total-variation distance, KI. the KL--divergence, and the last inequality follows from Pinsker's inequality. We therefore have
inf sup E EM, [Vo BMM.iVy # EE(2.)
VoM..1
Gaim
1
KL( || [PMa..
4
Now note that, since M1 and Ma only differ at state $1 and step H 1, we have
KL(P (Ma..) = EM, [T a JKLL 51, a1) PiM (. 81, a1)) ( (LI (- 81, P (- 81, a2)): + MI. (T [T
where TH-1($11 (81, a;) denotes the total number of visits to ($1, at step H 1 after T episodes (see e.g. [59]). We have
Mi KL KL (-18. || PM (1, KL(PA
(- 81, a2)
PA (: $1,
H 1-11
1
1/4
3
3/4
3
log
+
log
<
4
3/4
4 1/4
where the last inequality holds as long as Gim < 1//6. Note that the only way for policy to reach 81 step H - is to play action a₁ H consecutive times. Since स only plays aj at any given step with probability 2/2, it follows that the probability that गे reaches at step H on any given episode is only (c/2) H -1, Thus,
3
K M2.
M1.
E
(( + M1. (T.,([Tn(((1 (
51
3
Mi [81)1
5
3
(
H-1
T.
5
2
We thus have:
inf sup E. G.M M. xE(1.2)
Ssi
1-
V
3
#-1
T
4
10
and we therefore have infş suPie(1.2)
B.M..ly.M.
Vo v.M.a.
Sim unless
#-1
>
38

---

Lower Bound for Ç-Greedy Without sim (Proposition 1). In order to quantify the performance of Ç-greedy algorithm, must specify how it chooses f when it has not yet observed any samples from a given (s, a, h).. Following the lead of Theorem 2 of [11], to avoid an overly optimistic or pessimistic initialization, we assume that the replay buffer is initialized with single sample from each (s, a, h). Note that the conclusion would hold with other initializations, however, e.g. initializing (s, a) = 0 or randomly if we have no observations from (s, a, h).
Assume that the observation from ($1, 01, .. - 1) transitions s2, which occurs with probability at least 1/4. In this case, we then have that, for each h, (s1, 02) 2 (s, a1). Thus, following the Ç-greedy policy, we have that 4% (a | (1) S 1/2. Denote this event on Eg. Furthermore, the only way we will have (31. 42) < (s1, a1) is if we visit ($1, 01 H - 1) again and observe a transition $1. For this to occur, however, we must play action aj H 1 times consecutively which,
in this case, will occur with probability at most max H I 1/2"l1
Following the argument in the direct policy transfer case, we have
inf sup E E.M. (Vo VoM
2 inf sup xE(1,2)
EM
VoMiR R [
R (1.22)
Csim
1
V 10
E. (+1) Eul
16
where E. [TH-1 (81) | Eg] is the expected number of visitations to (s1, H 1) after T episodes of running the Ç-greedy policy. We can rewrite
T
E Mi E.M (T))((S)))) Eo] = Σ
[{S-- = 81} Eg]-
t=1
Let & be the event that we have reached (s1, H 1) in the first T rounds. Then,
B. [[(H- = $1} Ed] = E.M E = 81 E, &n M [E Eg] + E.M [T(8-11 = 81) E EoIPMi Eol
}1} E En). =1 By what we have just argued, we have p.Mi[E Eg] < T
I <.Mi Eg] + M. [1[ [[[(s-11
1 2 and B.M [[(SH = [8
2--1- Thus, M. (81) Eo] S -h- It follows that,
inf sup E. (v vo 3 2
Esim
1-
3 2T
# (E(2.)
16
10 2
and we therefore have inf: (1.) B.M.. AMMiiIyMin
Vo VoM
2 Ssim/ /22 unless
5
T T2 V
2H-1
8
Upper Bound for Exploration Policy Transfer (Proposition 4). To obtain an upper bound for Algorithm 1, we can apply Theorem 1, so long as
min
Ggim
A4d
Note that in our setting we have d = 4,4 2, A 1/4, so this condition reduces to caim S 8 8192H Taking F to simply be the set of Q-functions defined above (so Vmax H), Theorem I then gives that
with probability least 8, Algorithm learns an coptimall policy as long as T 2 c. H log
D.2 Proof of Proposition 5
We define three MDPs: Mysim, and two possible real MDPs, M1 == M and M2 = r In all cases we have states S (s1, 82), actions A (a1, 02 43, a.), and H = 2, and set the starting state to 81- We define
ppim $1, .1) (1)
= 1,
ppim (s) a) = Ppii a) = 1/2, a E {az, ag, a4].
39

---

For both Mi and M2, we have:
Prea (s1 a1) = 1, P1 81, (1) = Pleal (#2 81, (a) = -1/2
for M1, we have
Prea (22 81, (2, 1 Eslm (s1 [2, = Ealm Prea |11 Ps $1 (1, 1/2
and for M2.
(rea (32 $1, (3) + Esim
Preal $1, az) = 1 Esim Preal (81 02) = Preal $1, 02) = 1/2.
We take the reward to be 0 everywhere, except (2(32, a) for all a.
Note that each these can be represented as linear MDP 2 dimensions, so Assumption holds. In particular, for M can take:
(s,,, a1) = El, (3, a) E {az, 0z s S,
r (s1) 1/2], u (52) 10,1/2]
MM can instead take:
Φ'(s, 01) = c1, ('s, [1/2.1/2], E {as, an), S, (s, [1/2 Esim 1/2 + Esim] s E S,
(11) = [1, μu (s2) = [0, 1].
M2 follows similarly with the role of a2 and 0g flipped.
It is easy to see that Assumption is met on this instance for both choices of M On M the policy Texp which in every states plays action an with probability 1/2 and action αą with probability 1/2 satisfies Amin ( Foxp (sh, on ) (s), an)) T) 2 1/2 (which shows that Assumption holds).
Note, however, that Texp does not play action 02 or a3- As Mi and M2 differ only on d2 and as. playing Txe will not allow for M1 and M2 be distinguished. As ag the optimal action on Mi and 0g the optimal action on M2, follows that playing Texp will not allow for the identification of the optimal policy on Mi and M2. This can be formalized identically to Appendix D.I, yielding the stated result.
E Experimental Details
E.1 Didactic Tabular Example
Consider the following variation of the combination lock. We let the action space A = (1, and assume there are two states, S {s1. $2), and horizon H. We start in state $1. The sim dynamics are given as:
Vh < H 11 Prim (s1 (1, = Ph (s2 1, a2) = psim (s1 (1,,a)) 1/4, Paim (s2 (1,a;) 3/4, Pit (52 (1,
Yh [H] prim(s2 s2, a) 1,a 02).
and the real dynamics given as:
vh H 1: Pieal 81, 01) 1, preat (82 1, 81 02) =1
preal (81 81, 18, a1) 3/4, preal (82 81, 01) 1/4, preal 1 (82 81, (z)
P
Yh E [ Pr (s2 52, 1,a E (2).
Note that these only differ on ($) function as (note that this is deterministic, and the same for both sim and real):
H - 1. and we have ésim = 1/2. We define the reward
Wh E [H] (n(51, = 1/8 h/8H
rH a) = 1/5, E (2),
and all other rewards are taken to be 0.
40

---

The intuition for this example is as follows. In both sim and real, the only way the agent can get reward is either end up in state 8 at step H, or take action d2 in state $1 at any point. In sim, the probability of ending up in state 81 at step H, even if the optimal sequence of actions to do this taken, is only 1/4, due to the final transition, and thus the average reward obtained by the policy which aims to end up in 81 is only 1/4. In contrast, if we take action 0g in 81, we will always collect reward of at least 3/8 (and the earlier we take action d2 the more reward we collect, up to 1/2). Thus, in sim the optimal thing to do in 81 is always to play ag. However, if we play d2 even once, we will transition out of 81 and never return, so there is no chance we will reach s at step H.
In real, the transitions at the final step are flipped, so that now the probability of finishing in if we take the optimal sequences of actions to do this, is 3/4, and the expected reward for this is then also 3/4. Since the reward for taking d2 in 81 does not change, and is bounded as 1/2, then in real the optimal policy is to seek to end up 81 at the final step.
The challenge with ending up in 81 at the end is that it requires playing action a at every step. In this sense it is then a classic combination lock instance, and randomized exploration will fail, requiring (22 episodes to reach the final state (since the probability of randomly taking at every state decreases exponentially with the horizon). Similarly, if we transfer the optimal policy from sim to real, it will never take action 01. so will never reach 81 at the end, and if we transfer the optimal policy from sim with some random exploration, it will fail for the same reason random exploration from scratch fails.
However, note that we can transfer policy from sim that is able to reach 81 the second-to-last step with probability 1, i.e. the policy that takes action dn at every step. Thus, if in sim we aim to learn exploration policies that transfer, and will allow us to easily reach at the final step, and quickly determine that it is indeed the optimal thing in real.
traverse the MDP, and we transfer these exploration policies, they will
We provide additional experimental results on this instance Appendix E.
300
Midee
100
130
(a) Varying Number of States (b) Varying Number of Actions
(c) Varying Horizon
Figure 5: Performance of Exploration Policy Transfer on instance from Section 5.2, varying number of states, actions, and horizon. We plot the number of samples required to achieve a reward of 0.35, which is approximately solving the task. All results are averaged across 20 trials. When increasing the number of states, we add additional O-reward states (i.e. states given yellow in Figure 2), and when adding additional actions we add additional low-reward actions (i.e. actions that have the same behavior as action 42 in Figure 2). We observe that increasing the number of states and horizon increases the number of samples needed, while increasing the number of actions does not substantially. We emphasize, however, that this is for a particular example, and this scaling may not be the same for all examples Theeoeem however, gives upper bound on all examples.
E.2 Practical Algorithm Details
The core of our work is to decouple the optimal policy training from exploration strategies in reinforcement learning fine-tuning. Specifically, we propose framework that uses a set of diverse exploration policies to collect samples from the environment. These exploration policies are fixed while we run off policy RI. updates on the collected samples to extract an optimal policy. Our theoretical derivation suggests that this decoupling can improve sample efficiency and overall learning performance.
41

---

Algorithm 6 sim2real transfer using OS for exploration and SAC for optimization
1: Input: Simulator M real environment M simulator training budget N, exploration reward balancing a, reward threshold e, exploration set size n.
2: Pre-train Exploration Policies in M
3: Initialize eez = (=)])] [1...n}}
4: Initialize discriminator Dg
5: for i = 1 to N do
Learn diverse exploration policies
6: Sample latent 2 ~ unif((1,)) n) and initial state 50-
7: for = 1 paa__ttes__er__epssoe do
8:
Sample action a₂ ~ (t)
9:
Step environment: St+1 ~ p($(+1 ac).
10:
Compute discriminator score d, = (($++1,2)
Compute exploration reward (5(5)+1, 2) log Σ. exp(dleee113)))
exp(d)
12:
if R 2 then
13:
Compute reward re r(stt 01) + Fo.ro(st
14:
else
15:
Compute reward r2 r(st; az)
Let 4 D U {(1 oe; Tt, T6, 81+1, 2)}. Update To to maximize Jn with SAC.
17:
18:
Update & to maximize Ju, ó + o + 1/ V. [log (s
19:
Compute Rr -,
20: Explore in MMreal and Estimate Optimal Policy:
21: Initialize SAC agent (either from scratch to weights of optimal sim policy).
22: while not converged do
23: Sample 2 u1if(1, n), play Te(- 2) in M add data replay buffer of SAC.
24: Roll out SAC policy for one step, perform standard SAC update.
Our framework is complementary (a) RL works on diversity or exploration that generate diverse policies and (b) off policy RL algorithms that optimize for policies. One can plug in (a) to extract a set of exploration policy from simulator and use them for data collect in the real world but use (b) to optimize for the final policy. The design choice to use simulator to extract set of exploration policies where each policy is not necessarily optimizing for the task at hand marks our distinction from previous works in (a) and (b).
We provide a practical instantiation of our framework using approach inspired by One Solution is Not All You Need (OS) [30] to extract exploration policies and Soft Actor Critic (SAC) [30] optimize for the optimal policy. We details the instantiation in Algorithm 6. OS trains set of policy to optimize not only the task reward but also discriminator reward where the discriminator encourages each policy to achieve different state. Unlike OS which carefully balances the task and exploration rewards to ensure all policies have a chance at solving the desired task, we emphasize only on having diverse policies. With known sim2real gap, we posit that some sub-optimal policies that are not solving the task in the simulator is actually helpful for exploration in the real world, which allows us simplify the balance between task and exploration. We uses standard off off-shelf SAC update to optimize for the policy.
E.3 TychoEnv sim2sim Experiment Details
For the TychoEnv experiment we run variant of Algorithm 6. We n 20, and set the reward to rf = (1 - a;) + arT, where we vary a; from 0 to 0.5. While we use a sparse reward in Mrreal, to speed up training in M we use dense reward that penalizes the agent for its distance the target. We train in M for 7M steps to obtain exploration policies. Rather than simply transferring the converged version of the exploration policies trained in M we found most effective to save the weights of the policies throughout training, and transfer all of these policies. As the majority of these policies do not collect any reward in Mssim we run an initial filtering stage where we identify several policies from this set that find reward (this can be seen in Figure 4 with the initial region of 0 reward). We then run SAC in Mreal, initialized from scratch, feeding the data collected by these refined exploration policies into the replay buffer. We found it most effective to only inject data from the
42

---

Hyperarameter
Value
reward balance a (OS)
[3s
= 1, 1....... 20}
38
learning rate
0.0003
Q update magnitude discount y batch size steps per episode
0.005
0.99
2048
45
replay buffer size training steps N (in M
5 x 106
7 x 10?
Table Hyperparameters used in Tycho training and finetuning
Dxploration Pe Trander (Cun)
Dinn explension Paacy Tranter
Direct Parky Thander
15
Steps
Figure Additional results on Tycho, including baselines training from scratch in M and training exploration policies in M with reward as stated above but with a; 1 (which is equivalent to simply training exploration policies with DIAYN [14]). As can be seen, while training from scratch in M is able to learn, it learns at much slower rate than exploration policy transfer, and achieves a much lower final value. Furthermore, training the exploration policies to maximize a mix of the task and diversity reward yields substantial gain over simply training them to be diverse.
exploration policies the replay buffer on episodes where they observe reward. We run vanilla SAC with UTD 3 and target entropy of -3. We rely on the implementation of SAC from stable-baselines3 [51].
For direct policy transfer, we train policy to convergence in M that solves the task (using SAC), and then transfer this single policy, otherwise following the same procedure as above.
In M our reward is chosen to have a value 50 the end effector makes contact with the ball, and otherwise 0. If the robot successfully makes contact with the ball the episode terminates. To generate a realistic transfer environment, we change the control frequency (doubling it in M and the action bounds.
For both methods, we run the Miiim training procedure times, and then with each of these run it in M twice. Error bars in our plot denote one standard error.
Nvidia V100 GPUs, and 32 Intel(R) Xeon(R) CPU E5-2620 v4 @ 2. 10GHz CPUs. Additional hyperparameters in given in Table 1.
All experiments were run
We provide results several additional baselines for the Tycho setup in Figure 6.
-20
Explcation Pally Trandee (Dust
Farmaanch
80
x107
Figure 7: Results on Franka sim2real experiment, comparing to training from scratch in real.
43

---

Hyperarameter
Value
reward balance a (OS)
0.5
reward threshold (OS)
-16
learning rate
0.0003
Q update magnitude discount batch size steps per episode
0.005
0.99
256
45
replay buffer size
1 106
training steps N Table 2: Hyperparameters used in Franka training and finetuning
2 x 107
0.6
Cenee willy Trnchi
0.4
0.0
0.0
Figure 8: Franka Hammering
Figure 9: Results on sim2sim
Task Setup
Transfer in Franka Simulator
E.4
sim2sim Transfer on Franka Emika Panda Robot Arm
We next turn the Franka Emika Panda robot [17], for which built using the MuloCo simulation engine [61]. We consider a hammering task, where the Franka arm holds hammer, and the goal is hammer nail into the board (see Figure 8). Success is obtained when the nail is fully inserted. We simulate sim2real transfer by setting M to be a version of the simulator with nail location and stiffness significantly beyond the range seen during training in M
realistic custom simulator
We compare exploration policy transfer with direct sim2real policy transfer. Unlike the Tycho experiment, where trained policies from scratch in M and simply used the policies trained in AMsim to explore, here we initialize the task policy in M to π simm,, which we then finetune on the data collected in M by running SAC. For direct sim2real transfer, by simply rolling out sim,, and feeding this data to the replay buffer of SAC. For exploration policy transfer, train ensemble of 10 exploration policies in sism and these policies in r again feeding this data to the replay buffer of SAC to finetune π sim., During training in M we utilize domain randomization for both methods, randomizing nail stiffness, location, radius, mass, board size, and damping.
collect data in Mrall
The results of this experiment are shown in Figure 9, We see that, while direct policy transfer is able to learn, it learns at a significantly slower rate than our exploration policy transfer approach, and achieves much smaller final success rate.
E.5 Franka sim2real Experiment Details
use Algorithm to train a policy on the Franka robot with n = 15.
The reward of the pushing task is given by:
r(ss, ax) = - l|eeee - Pobj |
|obd - P2 + I P 50.025 - -pasofftable
(E.1)
where Pgoal the desired position of the puck by the edge of the surface.
The network architecture of the actor and critic networks are identical, consisting a 2-layer MLP, each of size 256 and ReI.U activations.
We use stable-baselines3 [51] our SAC implementation, using all of their default hyperparameters. The implemention of OS is built on top of this SAC implementation. Values of hyperparameters
44

---

are shown in Table 2. Gaussian noise with mean 0 and standard deviation 0.005 meters is added in simulation the position of the puck. Hyperparameters are identical between exploration policy transfer and direct transfer methods.
For finetuning in real, start off by sampling exclusively from the buffer used during simulation. Then, as finetuning proceeds, we gradually start taking more samples from the real buffer, with the proportion of samples taken from sim equal After 3000 steps, all samples are taken from the real buffer.
s/ s/3000, where the current number of steps.
Experiments
using standard Nvidia RTX 4090 GPU. Training in simulation takes about hours, while finetuning was ran for about 90 minutes.
In Figure 7, we provide results on this setup running the additional baseline of training a policy from scratch real. As can seen, this significantly worse than either transfer method.
45

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We validate all our claims with theoretical results and experiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please see Discussion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
46

---

Justification: All results are precisely proved in the supplemental, and all assumptions clearly stated.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: To the extent possible, given that we are working with real-world systems, we have described our setup and implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
47

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
We have not currently released our code but hope to the future.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
snooiiiccy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
//wpps.ccppubl/// http:ccccpppcccccceccecccccccccc htppuullccuuuidssssseepppp guides/ /Coee/ubmdeabiii ssi ssisssoololooooooo oonooiiyyyyloooooooomoooomiiiilp for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have stated all parameters and algorithm details to the best of our knowledge (please see Experimental Details section in supplemental).
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: For all our experimental results, we provide error bars corresponding to standard error.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
48

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please see Experimental Details section appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //nner//neuaaae cc cc/public//eeecccccccccccclcllct caw.liie/thisccuudeeleee...e
csCuidel ines?
Answer: [Yes]
Justification:
This paper does not violate any ethical guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
This is work related to the advancement of our fundamental understanding of machine learning, As such, we do not believe there are any direct societal impacts from this work.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
49

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We are not releasing high-risk models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have cited the creators of the code used in this project.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswi wwwode.hhoe...c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
50

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This work does not involve human subjects research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This work does not involve human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
51


Reviews:
[{"review_id": "BUx7Dg6ygA", "reviewer": "Reviewer_emUx", "summary": "The paper shows that for transferring with a large sim-to-real gap, exploration policies have improved capabilities compared to optimally pre-trained policies, which tend to overfit the simulation up to a point where exploration is not sufficient to adapt to the changed circumstances.", "questions": "What are the implications of using a low-rank MDP?\n\nIf the simulation is considered to be free, wouldn't learning $\\Pi$ optimal policies improve the transfer (similar to using a pool of exploration policies)?  \n\nRegarding the transfer described in Alg. 1, are all exploration policies updated, or only the best-performing? \n\nIf we were to consider even larger shifts, is there a point at which exploration policies no longer provide benefits, and learning from scratch is the most sufficient approach? \n\nWhen considering the transfer from simulation to the real world, pre-training is often motivated by the need to ensure a certain level of safety for execution in the real world. How does this concern affect your approach?", "limitations": "Limitations have been addressed by the authors but could have been extended.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall, the paper is easy to follow and theoretically sound. It includes theoretical deliberations and empirical results with a convincing experiment design. I expect the considered perspective on sim-to-real shift to positively impact research in this area.", "weaknesses": "Overall, providing further examples for the theoretical elaborations could help comprehensibility (e.g., Proposition 3). Further minor issues include: \n\n- Definition 3.2: PAC is not defined, what is $\\delta$ used for? \n- In Proposition 1: Is the success of the transfer dependent on the horizon of M? \n- In Proposition 2: What does $\\Omega$ refer to? \n- What is the difference between $\\epsilon$ and $\\epsilon_{sim}$?\n- Why is the transition probability $P$ defined to be dependent on the horizon H? \n\nAlso, I am missing a thorough definition of the Low-Rank MDP, which seems central to the theoretical elaboration. \n\nEven though the experimental setup is convincing, I am missing comparisons directly training $\\pi$ in the real task as a baseline. Also, to better illustrate and link the aforementioned theoretical elaboration, providing the distance between the considered benchmark tasks would have been interesting. Furthermore, more detailed comparisons (both experimental and theoretical) to advanced pre-training approaches like APT (Liu et al., 2021) or DIAYN (Eysenbach et al., 2018) would further enhance empirical credibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their helpful feedback.\n\n> Providing further examples for the theoretical elaborations…\n\nThe example used to prove Prop. 1 and 2 is given in Sec. 5.2 and is a variant of the classic combination lock instance. In “real” the correct action must be taken for $H$ steps, and therefore, if we are exploring randomly, it will take exponentially long in $H$ to find this sequence of actions. In “sim”, the final transition is altered such that the optimal sequence in real is no longer optimal, and if we transfer the optimal sim policy to real, it does not reach the high reward region. However, if we transfer exploratory policies from sim, they are still able to traverse the MDP, and find the high reward, therefore quickly learning the optimal policy. Please see Figure 1 for a visual illustration of this construction. \n\n> Definition 3.2: PAC is not defined… What is the difference between $\\epsilon$ and $\\epsilon_{sim}$?\n\nPAC refers to “probably approximately correct”. $\\delta$ denotes the confidence with which you hope to find an $\\epsilon$-optimal policy. $\\epsilon_{sim}$ is the amount of mismatch between sim and real.\n\n> In Proposition 1: Is the success of the transfer…\n\nThe effectiveness of the policy learned in the simulator when deployed in real does depend on the horizon. This is illustrated in Prop. 1, which states that the optimal policy in sim can only be shown to be $2 H^2 \\epsilon_{sim}$ optimal in real—as $H$ increases, the effectiveness of the policy learned in sim could decrease.\n\n> In Proposition 2: What does $\\Omega$ refer to? \n\n$T \\ge \\Omega(2^H)$ means that $T$ must be on order $2^H$ (up to constants and lower-order terms).\n\n> Why is the transition probability $P$...\n\nThis is common in the theory literature and simply lets us also handle cases where the transitions are time-dependent. \n\n> I am missing a thorough definition of the Low-Rank MDP\n\nWe provide a standard definition for low-rank MDP in Def. 3.1. A low-rank MDP assumes that (1) there exists a featurization of the state-action pairs and (2) the transition functions can be approximated by a linear product based on the featurization. Note that the featurization does not need to be known a-priori. \n\n> I am missing comparisons directly training $\\pi$ in the real task as a baseline. \n\nWe have run an additional baseline training from scratch in “real” for both our sim2sim task and sim2real task. Please see the rebuttal pdf Figures 1 and 4. for these results. In both cases, we found that this performed significantly worse than our approach or direct policy transfer.\n\n> Providing the distance… \n\nFor the didactic example (Section 5.2), the distance between the real and sim is $\\epsilon_{sim} = 1/2$. For the other tasks we consider, there is unfortunately no straightforward way to measure the distance between tasks in terms of the closeness of their transitions, but highlight that our method does not require knowledge of $\\epsilon_{sim}$.\n\n> More detailed comparisons…\n\nWe have added a comparison to DIAYN for our sim2sim experiment (see Figure 1 in the rebuttal pdf), where we use DIAYN to train a set of exploration to transfer. While this performs worse than our approach—which instantiates our meta-algorithm, Algorithm 2, with a diversity-based approach similar to DIAYN, but also incorporating the task reward (see Algorithm 6)—we found it to still be somewhat effective. We remark that this use of DIAYN also fits directly within our meta-algorithm as an alternative method for generating exploration policies. \n\nDIAYN, APT, and all other practical approaches we are aware of lack theoretical guarantees, so a comparison between such approaches and our theoretical results is not possible. We are not aware of any other existing theoretical results that address the same problem we consider.\n\n> What are the implications of using a low-rank MDP?\n\nPlease see our comment to all reviewers for discussion of the low-rank MDP assumption. Theoretically, this assumption is necessary to show that high-coverage exploration policies in the simulator will also yield effective coverage in real. Extending our work to more general settings is an interesting direction for future work. Empirically, our experimental results show that insights derived from low-rank MDPs are also relevant in more general practical settings.\n\n> If the simulation is considered to be free…\n\nTheoretically we do not believe this would improve the result—the exploration policies are already yielding sufficient data coverage, and transferring a family of near-optimal sim policies would not yield improved coverage. In practice this may yield improved performance, however, and indeed, this is similar to the method we utilize to generate the exploratory policies for our practical experiments (see Algorithm 6). \n\n> Regarding the transfer described in Alg. 1…\n\nIn Algorithm 1, the exploration policies are not updated at all, they are simply played as is. The data collected from them is then used to train a single policy that solves the goal task in real. \n\n> If we were to consider even larger shifts…\n\nYes, equation (4.1) in Theorem 1 gives a sufficient condition $\\epsilon_{sim}$ must satisfy in order for exploration policy transfer to provably succeed. If this condition is not met, we are unable to guarantee exploration policy transfer succeeds.\n\n> When considering the transfer from simulation to the real world…\n\nWhile we did not explicitly consider it in this work, safety constraints could be incorporated by restricting the exploration policies so as not to simply explore, but to explore in a safe manner (e.g., incorporating safety-based approaches such as [Berkenkamp et al., 2017], or requiring that the exploration policies do not induce unsafe behavior in the simulator). Further study formalizing this is an interesting direction for future work.\n\nBerkenkamp, Felix, et al. \"Safe model-based reinforcement learning with stability guarantees.\" NeurIPS, 2017.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sNEfwrfhZY", "reviewer": "Reviewer_ceeA", "summary": "The authors propose a method where, instead of directly transferring a trained policy from a simulator to the real world, exploratory policies are learned in the simulator and transferred. This approach aims to enable efficient exploration in the real world, particularly in low-rank MDP settings.", "questions": "1. How robust is the method to violations of the low-rank MDP assumption?\n\n2. Can the approach be extended to more complex MDP settings?\n\n3. How does the method scale with the complexity of the task and the size of the state/action spaces?", "limitations": "As in weaknesses.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The idea of transferring exploratory policies instead of directly transferring policies trained in simulator is interesting and can potentially help sim-to-real transfer.", "weaknesses": "1. The approach relies on several assumptions (e.g., low-rank MDPs, specific access to simulators and oracles) that may not hold in all real-world scenarios.\n\n2. While the method shows promise in specific settings, its generality to a wider range of RL tasks and environments remains to be demonstrated.\n\n3. Limited comparison with other state-of-the-art methods, especially those addressing the sim-to-real gap through domain randomization or adaptation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback, and address questions and weaknesses below.\n\n## Weaknesses\n1. > The approach relies on several assumptions (e.g., low-rank MDPs, specific access to simulators and oracles) that may not hold in all real-world scenarios.\n\nPlease see our comment to all reviewers for further justification of the low-rank MDP assumption. Briefly, we consider the low-rank MDP assumption to draw theoretical conclusions, but our proposed algorithm extends beyond the setting of low-rank MDPs, and our empirical results (on a real world task that does not strictly follow the setup of a low-rank MDP) show it is effective in general settings. Furthermore, we emphasize that the low-rank MDP setting is canonical in the theory community, and we believe that results in this setting are interesting in their own right.\n\nWe note that our simulator access is quite weak—we only require black-box access to a simulator, which is the weakest type of access one could consider—and believe it is reasonable that we assume access to a simulator since we are studying sim2real transfer. Furthermore, we believe the oracles we consider are very reasonable—a regression oracle is quite standard and can be implemented even with neural networks, and many RL approaches exist that can successfully learn to solve a task in simulation (for example, standard approaches such as SAC and PPO are in general able to effectively solve tasks in simulation where samples are cheap).\n\n2. > While the method shows promise in specific settings, its generality to a wider range of RL tasks and environments remains to be demonstrated.\n\n\nPlease see our comment to all reviewers on what we believe are the key contributions of this work. Briefly, our primary contribution is theoretical: we provide the first result showing that simulators can provably help real-world RL in non-trivial settings. Our experimental results further validate this by showing that the algorithmic insights we derive are applicable in practice, which we demonstrate both through challenging sim2sim settings, as well as real-world robotic settings. We agree that further empirical work is necessary to fully validate the effectiveness of our proposal in general problem settings. Given the theoretical nature of the paper, however, we believe this is beyond the current scope, but is an interesting direction for future research.\n\n3. > Limited comparison with other state-of-the-art methods, especially those addressing the sim-to-real gap through domain randomization or adaptation.\n\nThough it was not clearly stated in the paper, the baseline in our presented results on the real robot utilized domain randomization—on the Franka experiments, we randomized the friction, the size of the puck and the observation noise. Our results, therefore, show our proposed method is more sample efficient than doing only domain randomization. We will state this more clearly in the final version. We have also added an additional baseline, DIAYN [Eysenbach et al., 2018], to our sim2sim experiment—please see the rebuttal pdf, Figure 1.\n\nRegarding adaptation, we are primarily interested in settings where there is an irreducible gap between sim and real, given a fixed simulator. Indeed, for our Franka sim2real experiment, adaptation is unlikely to help us, as the parameters used are already chosen to match the real world dynamics as effectively as possible. The source of the sim2real gap is likely due to unmodeled dynamics, perhaps latency, and thus adaptation would require redesigning and improving the fidelity of our simulation. \n\nEysenbach, Benjamin, et al. \"Diversity is all you need: Learning skills without a reward function.\" arXiv preprint arXiv:1802.06070 (2018).\n\n## Questions\n\n1. > How robust is the method to violations of the low-rank MDP assumption?\n\nWhile we did not explicitly consider this in our theory, small violations of the low-rank MDP assumption (up to tolerance $O(\\epsilon)$) would not affect our result. Furthermore, our empirical results illustrate that in practical settings where the low-rank assumption may or may not hold, our algorithmic insights are still effective. We remark as well that existing work shows that algorithms relying on the low-rank MDP assumption can be successfully applied to many standard RL benchmarks [Zhang et al., 2022].\n\nZhang, Tianjun, et al. \"Making linear mdps practical via contrastive representation learning.\" ICML, 2022.\n\n2. > Can the approach be extended to more complex MDP settings?\n\nOur experiments illustrate that our algorithmic insights are still effective in more complex MDP settings. Extending our theory to more general settings (for example, bilinear classes) is an interesting direction for future work—at present it is unclear whether efficient sim2real transfer is possible in such settings.\n\n3. > How does the method scale with the complexity of the task and the size of the state/action spaces?\n\nTheorem 1 shows that the sample complexity scales independently of the size of the action space, and only quadratically in the feature dimension (and does not directly scale with the number of states at all). We have also included additional experiments in the rebuttal pdf illustrating how our approach scales varying the number of states, actions, and horizon on the didactic example of Section 5.2.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c3vY5feOCo", "reviewer": "Reviewer_3Mfq", "summary": "The authors contribute a mathematical proof showing that transferring a \"exploration strategy\" in sim2real scenarios as opposed to transfer the learned policy result in an exponential improvement on samples needed to learn the real task.", "questions": "- Why is it assumed that the transfer HAS to be made one-shot (transferred completely in the beginning of the learning process in the \"real environment\")? Wouldn't it be better to have the exploration transfer strategy work as a \"loop\" where the exploration could be revised according to how the real world differs to the simulator?\n\n- Why weren't MF-MDPs used to describe the problem?\n\n- it is stated that the conditions for us being able to find a policy that solve the problem in the simulation is that ϵ ≥ 2H^2ϵ_sim. What is the meaning of ϵ here? Finding conditions for a direct policy transfer to work would also be an important contribution, specially if there is some way to test for those conditions with some samples from both the real world and simulator.", "limitations": "Basic research, no potential negative societal impact to explore.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- While not necessarily a fundamentally new idea (the transfer learning and sim2real areas have been long avoiding the naive transfer of policies and biasing exploration has always been one of the chosen strategies to perform transfer), the theoretical results presented confirm and fundament this idea that has been informally explored by the literature.\n\n- The authors present a practical, easy to understand, exploration method and show that it overperforms greatly a simple policy transfer method in a somewhat complex sim2real robotic evaluation.\n\n- The topic explored is not only adherent to neurips but also very timely in supporting more targeted efforts to apply RL to novel practical domains.", "weaknesses": "- I am not sure why a new basic \"sim2real\" formalization was used to in this paper when it already exists. The problem explored in this paper is completely equivalent to Multi-Fidelity MDPs (described in {1}), in fact, MF-MDPs even have something not incorporated in the description provided that is the cost of sampling in each fidelity. The MF-MDP paper was not even cited so I assume this happened because the authors have not read the paper. I suggest the paper is rewritten to describe the problem as an MF-MDP for standardization (there shouldn't be any impact to the conclusions or contribution fro the paper), but even if not rewritten at the very least this paper has to be cited and incorporated to the related works:\n\n{1} - Silva, F. L., et al. Toward multi-fidelity reinforcement learning for symbolic optimization. ALA workshop @ AAMAS, 2023.\n\n- Another thing that made me be in the fence regarding my score to this paper is the lack of baselines for the experimental evaluation. There has been many papers exploring a similar idea in the past, so I feel like they should have been incorporated to the experimental evaluation (Cuttler {2} comes to mind immediately, but you also have other newer options)..\n\n{2} Cutler, Mark, Thomas J. Walsh, and Jonathan P. How. \"Reinforcement learning with multi-fidelity simulators.\" 2014 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2014.\n\n- A limitation both in the theoretical and practical side of the paper is that it is assumed that the transfer will happen \"one shot\", neglecting the fact that either the simulator could be modified to get closer to the real world {3} and the learning process in the simulator repeated, or the approach could already account that the simulator and real world will be different and adapt for that (e.g. {2} and {3}). The present paradigm presented by the authors to \"learn from the simulation once and never get back to it\" is at least an incomplete picture of all that could be done.\n\n{3} - Hanna, Josiah, and Peter Stone. \"Grounded action transformation for robot learning in simulation.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 31. No. 1. 2017.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback, and will work to incorporate all suggested improvements.\n\n> I am not sure why a new basic \"sim2real\" formalization was used to in this paper when it already exists. The problem explored in this paper is completely equivalent to Multi-Fidelity MDPs (described in {1})...\n\nWe apologize for missing this reference. We will include it in the final version of the paper, and make clear what the connections are between our setting and MF-MDPs. In particular, our setting is a special case of MF-MDPs with two environments. However, as we wish to emphasize the role of each of these environments—a “sim” and “real” environment—and since there are only two such environments, we believe it is somewhat more clear to maintain our current terminology.\n \n> Another thing that made me be in the fence regarding my score to this paper is the lack of baselines for the experimental evaluation…\n\nPlease see our comment to all reviewers on what we believe are the key contributions of this work. Briefly, our primary contribution is theoretical: we provide the first result showing that simulators can provably help real-world RL in non-trivial settings. Our experimental results further validate this by showing that the algorithmic insights we derive are applicable in practice. We agree, however, that comparisons to additional baselines would be useful in fully validating the empirical effectiveness of our approach. We have aggregated all the reviewers' suggestions and added two additional baselines: using DIAYN [Eysenbach, 2018] to train exploration policies and transferring as our theory suggests, and training from scratch in real. We include these results in the rebuttal pdf. \n\nEysenbach, Benjamin, et al. \"Diversity is all you need: Learning skills without a reward function.\" arXiv preprint arXiv:1802.06070 (2018).\n\n> A limitation both in the theoretical and practical side of the paper is that it is assumed that the transfer will happen \"one shot\"...\n\nWe acknowledge the potential for iterative adaptation between the simulator and the real world as suggested by the reviewer. We agree that in general, simulations can be improved (i.e., through system identification). However, we are primarily interested in settings with an irreducible gap between the simulator and real world. For example, for our Franka sim2real experiment, we have chosen physical parameters to match the real world dynamics as effectively as possible. Nevertheless, a sim2real gap persists, likely coming from imperfect friction cone modeling, imperfect contact modeling, unmodeled dynamics, latency, etc. Many real-world robotics work dealing with contacts and manipulation shared our observations [OpenAI et al, 2018; Höfer et al., 2021; Zhang et al., 2023] and confirmed that, for many problems there can be an irreducible sim2 real gap.\n\nIn settings such as ours with irreducible sim2real gap, while it may be possible to use a simulator in a more iterative fashion, we see our results as proof of concept, illustrating that, even in the limited setting where you can only do 0-shot transfer, extracting exploration policies from a simulator yield a provable (exponential) gain over simply extracting the optimal policy from the simulator. Thus, while there may be other ways to utilize a simulator, this would not change the conclusion of the paper: simulators yield a provable gain in real-world RL and extracting exploration policies yield a provable gain than extracting the optimal policy with domain randomization (something not previously known). Understanding the most effective way to utilize a simulator (which may involve using a more iterative fashion) is an interesting direction for future work. \n\nAndrychowicz, OpenAI: Marcin, et al. \"Learning dexterous in-hand manipulation.\" The International Journal of Robotics Research 39.1 (2020).\n\nHöfer, Sebastian, et al. \"Sim2real in robotics and automation: Applications and challenges.\" IEEE transactions on automation science and engineering 18.2 (2021): 398-400.\n\nZhang, Yunchu, et al. \"Cherry-Picking with Reinforcement Learning: Robust Dynamic Grasping in Unstable Conditions.\" arXiv preprint arXiv:2303.05508 (2023).\n\n> It is stated that the conditions for us being able to find a policy that solve the problem in the simulation is that $\\epsilon \\ge 2 H^2 \\epsilon_{sim}$... \n\nHere, $\\epsilon$ denotes the desired optimality tolerance we wish to learn a policy up to—our goal is to find an $\\epsilon$-optimal policy—while $\\epsilon_{sim}$ denotes the mismatch between sim and real.\n\nOur results do provide necessary and sufficient conditions for direct policy transfer to succeed: as Proposition 1 shows, as long as $\\epsilon \\ge 2 H^2 \\epsilon_{sim}$, direct policy transfer succeeds in finding an $\\epsilon$-optimal policy, and as Proposition 2 shows, direct policy transfer cannot in general find a policy that is better than $\\epsilon_{sim}/32$-optimal. \n\nIt is unclear in practical settings how to measure when this condition is met—obtaining such a method is an interesting direction for future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank each of the reviewers for their feedback and suggested improvements. We will work to incorporate all suggestions. We highlight several points regarding our contributions and the low-rank MDP setting, which we believe are relevant to all reviews.


### Key Contribution of Paper

We want to reiterate what we believe to be the core contributions of this paper. Our primary contribution is our theoretical results showing that simulators can yield a provable (exponential) gain in real-world RL, as compared to training only in the real world, and that this is true even in non-trivial settings where the mismatch between sim and real is large enough that direct policy transfer fails. To the best of our knowledge, this is the first theoretical justification for sim2real transfer in non-trivial settings.

Our experimental results further illustrate that the algorithmic principles our theoretical results motivate are applicable in real-world settings of interest, and yield a significant improvement over standard methods for sim2real transfer. However, we see this primarily as a proof-of-concept complementing our theory: additional empirical evidence would certainly be useful in providing a rigorous justification that this is a practical approach in a wide range of settings and, furthermore, more work could be done optimizing the instantiation of our meta-algorithm (Algorithm 2). 

We have, however, added several additional empirical benchmarks to further support our empirical validation, as well as a new evaluation task (block hammering). The results for these experiments can be found in the rebuttal pdf.


### Low-Rank MDPs 

Several reviewers commented on the limitation of requiring the low-rank MDP assumption. We make several comments on this. First, while the low-rank MDP assumption is required for the theory, our experimental results show that the algorithmic insights obtained from studying the low-rank MDP setting yield effective algorithmic approaches in more general settings. Thus, while the theory may be limited to the low-rank setting, the principles derived from studying this setting are much more general. Second, existing work has shown that the low-rank assumption often holds true in practice, and has applied existing algorithms for low-rank MDPs to a variety of standard RL benchmarks such as the Deepmind Control Suite [Zhang et al., 2022]. Finally, the low-rank MDP setting is a canonical setting in the RL theory community, and a significant amount of work has been devoted to understanding it (see, for example, references [1, 2, 36, 38, 53] in the paper). Thus, we believe that results on low-rank MDPs are of interest in their own right.

Zhang, Tianjun, et al. "Making linear mdps practical via contrastive representation learning." International Conference on Machine Learning. PMLR, 2022.

Meta-review: A simple yet clear idea (transfer exploration policies, not exploitation policies) in an important and timely setting (sim2real), backed up with both theory and real-world robotics experiments. I concur with the reviewers' consensus that this paper should be accepted, even if there is room for further improvements (various ones pointed out in the reviews).
Decision: Accept (poster)