Title: Overcoming the Sim-to-Real Gap: Leveraging Simulation to Learn to Explore for Real-World RL
Venue: NeurIPS 2024
Accepted: yes
Keywords: reinforcement learning

sim2real transfer

exploration

sample complexity

naive exploration
Abstract: In order to mitigate the sample complexity of real-world reinforcement learning, common practice is to first train a policy in a simulator where samples are cheap, and then deploy this policy in the real world, with the hope that it generalizes effectively. Such \emph{direct sim2real} transfer is not guaranteed to succeed, however, and in cases where it fails, it is unclear how to best utilize the simulator. In this work, we show that in many regimes, while direct sim2real transfer may fail, we can utilize the simulator to learn a set of \emph{exploratory} policies which enable efficient exploration in the real world. In particular, in the setting of low-rank MDPs, we show that coupling these exploratory policies with simple, practical approaches---least-squares regression oracles and naive randomized exploration---yields a polynomial sample complexity in the real world, an exponential improvement over direct sim2real transfer, or learning without access to a simulator. To the best of our knowledge, this is the first evidence that simulation transfer yields a provable gain in reinforcement learning in settings where direct sim2real transfer fails. We validate our theoretical results on several realistic robotic simulators and a real-world robotic sim2real task, demonstrating that transferring exploratory policies can yield substantial gains in practice as well.

Overcoming the Sim-to-Real Gap: Leveraging Simulation to Learn to Explore for Real-World RL

Abstract

In order to mitigate the sample complexity of real-world reinforcement learning, common practice is to first train a policy in a simulator where samples are cheap, and then deploy this policy in the real world, with the hope that it generalizes effectively. Such direct sim2real transfer is not guaranteed to succeed, however, and in cases where it fails, it is unclear how to best utilize the simulator. In this work, we show that in many regimes, while direct sim2real transfer may fail, we can utilize the simulator to learn a set of exploratory policies which enable efficient exploration in the real world. In particular, in the setting of low-rank MDPs, we show that coupling these exploratory policies with simple, practical approaches-least-squares regression oracles and naive randomized exploration-yields a polynomial sample complexity in the real world, an exponential improvement over direct sim2real transfer, or learning without access to a simulator. To the best of our knowledge, this is the first evidence that simulation transfer yields a provable gain in reinforcement learning in settings where direct sim2real transfer fails. We validate our theoretical results on several realistic robotic simulators and a real-world robotic sim2real task, demonstrating that transferring exploratory policies can yield substantial gains in practice as well.

1 Introduction

Over the last decade, reinforcement learning (RL) techniques have been deployed to solve a variety of real-world problems, with applications in robotics, the natural sciences, and beyond [27, 54, 52, 26, 46, 23]. While promising, the broad application of RL methods has been severely limited by its large sample complexity-the number of interactions with the environment required for the algorithm to learn to solve the desired task. In applications of interest, it is often the case that collecting samples is very costly, and the number of samples required by RL algorithms is prohibitively expensive.

In many domains, while collecting samples in the desired deployment environment may be very costly, we have access to a simulator where the cost of samples is virtually nonexistent. As a concrete example, in robotic applications where the goal is real-world deployment, directly training in the real world typically requires an infeasibly large number of samples. However, it is often possible to obtain a simulator-derived from first principles or knowledge of the robot's actuation-which provides an approximate model of the real-world deployment environment. Given such a simulator, common practice is to first train a policy to accomplish the desired task in the simulator, and then deploy it in the real world, with the hope that the policy generalizes effectively from the simulator to the goal deployment environment. Indeed, such ' sim2real ' transfer has become a key piece in the

Standard Sim2Real

Train exploratory

policies in sim

Collect high-coverage

data in real

Use data to learn

to solve task

Train policy that

solves task in sim

Collect low-coverage

data in real

Unable to

solve task

application of RL to robotic settings, as well as many other domains of interest such as the natural sciences [12, 15], and is a promising approach towards reducing the sample complexity of RL in real-world deployment [19, 4, 18].

Effective sim2real transfer can be challenging, however, as there is often a non-trivial mismatch between the simulated and real environments. The real world is difficult to model perfectly, and some discrepancy is inevitable. As such, directly transferring the policy trained in the simulator to the real world often fails, the mismatch between sim and real causing the policy-which may perfectly solve the task in sim-to never solve the task in real. While some attempts have been made to address this-for example, utilizing domain randomization to extend the space of environments covered by simulator [60, 49], or finetuning the policy learned in sim in the real world [50, 73]-these approaches are not guaranteed to succeed. In settings where such methods fail, can we still utilize a simulator to speed up real-world RL?

In this work we take steps towards developing principled approaches to sim2real transfer that addresses this question. Our key intuition is that it is often easier to learn to explore than to learn to solve the goal task . While solving the goal task may require very precise actions, collecting high-quality exploratory data can require significantly less precision. For example, successfully solving a complex robotic manipulation task requires a particular sequence of motions, but obtaining a policy that will interact with the object of interest in some way, providing useful exploratory data on its behavior, would require significantly less precision.

Formally, we show that, in the setting of low-rank MDPs where there is a mismatch in the dynamics between the 'sim' and 'real' environments, even when this mismatch is such that direct sim2real transfer fails, under certain conditions we can still effectively transfer a set of exploratory policies from sim to real. In particular, we demonstrate that access to such exploratory policies, coupled with random exploration and a least-squares regression oracle-which are insufficient for efficient learning on their own, but often still favored in practice due to their simplicity-enable provably efficient learning in real. Our results therefore demonstrate that simulators, when carefully applied, can yield a provable-exponential-gain over both naive sim2real transfer and learning without a simulator, and enable algorithms commonly used in practice to learn efficiently.

Furthermore, our results motivate a simple, easy-to-implement algorithmic principle: rather than training and transferring a policy that solves the task in the simulator, utilize the simulator to train a set of exploratory policies, and transfer these, coupled with random exploration, to generate high

quality exploratory data in real. We show experimentally-through a realistic robotic simulator and real-world sim2real transfer problem on the Franka robot platform-that this principle of transferring exploratory policies from sim to real yields a significant practical gain in sample efficiency, often enabling efficient learning in settings where naive transfer fails completely (see Figure 1).

2 Related Work

Provable Transfer in RL. Perhaps the first theoretical result on transfer in RL is the 'simulation lemma', which transforms a bound on the total-variation distance between the dynamics to a bound on policy value [24, 25, 6, 22]-we argue that we can do significantly better with exploration transfer. More recent work has considered transfer in the setting of block MDPs [34], but requires relatively strong assumptions on the similarity between source and target MDPs, or the meta-RL setting [69], but only consider tabular MDPs, and assume the target MDP is covered by the training distribution. Perhaps most relevant to this work is the work of [36], which presents several lower bounds showing that efficient transfer in RL is not feasible in general. In relation to this work, our work can be seen as providing a set of sufficient conditions that do enable efficient transfer; the lower bounds presented in [36] do not hold in the low-rank MDP setting we consider. Several other works exist, but either consider different types of transfer than what we consider (e.g., observation space mismatch), or only learn a policy that has suboptimality bounded by the sim2real mismatch [37, 56, 58]. Another somewhat tangential line of work considers representation transfer in RL, where it is assumed the source and target tasks share a common representation [35, 10, 2]. We remark as well that the formal sim2real setting we consider is a special case of the MF-MDP setting of [53].

Simulators and Low-Rank MDPs. Several existing works show that there are provable benefits to training a policy in 'simulation' due to the ability to reset on command [67, 33, 5, 68, 70, 42]. These works do not consider the transfer problem, however. The setting of linear and low-rank MDPs which we consider has seen a significant amount of attention over the last several years, and many provably efficient algorithms exist [21, 1, 62, 63, 43, 41]. These works typically assume access to powerful oracles which enable efficient learning; we only consider access to a simple regression oracle. Beyond the theory literature, recent work has also shown that low-rank MDPs can effectively model a variety of standard RL settings in practice [72].

Sim2Real Transfer in Practice. The sim2real literature is vast and we only highlight particularly relevant works here; see [74] for a full survey. To mitigate the inconsistency between the simulator and real world's physical parameters and modeling, domain randomization creates a variety of simulated environments with randomized properties to develop a robust policy [60, 49, 44, 8, 39]. Domain adaptation instead constructs encoding of deployment conditions (e.g., physical condition or past histories) and adapts to the deployment environment by matching the encoding [29, 9, 66, 55, 38, 40]. In contrast, our work assumes a fundamental sim2real mismatch where we do not expect the real system to match the simulator for any parameter settings. A related line of work shows that policies trained with robust exploration strategies generalize better to disturbed or unseen environments [13, 20]. Our work is complimentary to this work in that our goal is not to transfer a policy that solves the task in new environment, but rather explores the environment.

3 Preliminaries

We let △ X denote the set of distributions over set X , [ H ] := { 1 , 2 , . . . , H } , and ∥ P -Q ∥ TV the total-variation distance between distributions P and Q .

Markov Decision Processes. We consider the setting of episodic Markov Decision Processes (MDPs). An MDP is denoted by a tuple M = ( S , A , { P h } H h =1 , { r h } H h =1 , s 1 , H ) , where S denotes the set of states, A the set of actions, P h : S × A → △ S the transition function, r h : S × A → [0 , 1] the reward (which we assume is deterministic and known), s 1 the initial state, and H the horizon. We assume A is finite and denote A := |A| . Interaction with an MDP starts from state s 1 , the agent takes some action a 1 , transitions to state s 2 ∼ P 1 ( · | s 1 , a 1 ) , and receives reward r 1 ( s 1 , a 1 ) . This process continues for H steps at which points the episode terminates, and the process resets.

The goal of the learner is to find a policy π = { π h } H h =1 , π h : S → △ A , that achieves maximum reward. We can quantify the reward received by some policy π in terms of the value and Q -value

functions. The Q -value function is defined as Q π h ( s, a ) := E π [ ∑ H h ′ = h r h ′ ( s h ′ , a h ′ ) | s h = s, a h = a ] , and value function is defined in terms of the Q -value function as V π h ( s ) := E a ∼ π h ( ·| s ) [ Q π h ( s, a )] . The value of policy π , its expected reward, is denoted by V π 0 := V π 1 ( s 1 ) , and the value of the optimal policy, the maximum achievable reward, by V ⋆ 0 := sup π V π 0 .

In this work we are interested in the setting where we wish to solve some task in the 'real' environment, represented as an MDP, and we have access to a simulator which approximates the real environment in some sense. We denote the real MDP as M real , and the simulator as M sim . We assume that M real and M sim have the same state and actions spaces, reward function, and initial state, but different transition functions, P real and P sim . We denote value functions in M real and M sim as V real ,π h ( s ) and V sim ,π h ( s ) , respectively. We make the following assumption.

Assumption 1. For all ( s, a, h ) ∈ S × A × [ H ] and some ϵ sim > 0 , we have:

We do not assume that the value of ϵ sim is known, simply that there exists some such ϵ sim .

Function Approximation. In order to enable efficient learning, some structure on the MDPs of interest is required. We will assume that M real and M sim are low-rank MDPs, as defined below.

Definition 3.1 (Low-Rank MDP) . We say an MDP is a low-rank MDP with dimension d if there exists some featurization ϕ : S × A → R d and measure µ : [ H ] ×S → R d such that:

We assume that ∥ ϕ ( s, a ) ∥ 2 ≤ 1 for all ( s, a ) , and for all h , ∥| µ h | ( S ) ∥ 2 = ∥ ∫ s ∈S | d µ h ( s ) |∥ 2 ≤ √ d .

Formally, we make the following assumption on the structure of M sim and M real .

Assumption 2. Both M sim and M real satisfy Definition 3.1 with feature maps and measures ( ϕ s , µ s ) and ( ϕ r , µ r ) , respectively. Furthermore, ϕ s is known , but all of µ s , ϕ r , and µ r are unknown.

In the literature, MDPs satisfying Definition 3.1 but where ϕ is known are typically referred to as 'linear' MDPs, while MDPs satisfying Definition 3.1 but with ϕ unknown are typically referred to as 'low-rank' MDPs. Given this terminology, we have that M sim is a linear MDP 2 , while M real is a low-rank MDP. We assume the following reachability condition on M sim .

Assumption 3. There ∃ λ ⋆ min > 0 with min h sup π λ min ( E M sim ,π [ ϕ s ( s h , a h ) ϕ s ( s h , a h ) ⊤ ]) ≥ λ ⋆ min .

Assumption 3 posits that each direction in the feature space in our simulator can be activated by some policy, and can be thought of as a measure of how easily each direction can be reached. Similar assumptions have appeared before in the literature on linear and low-rank MDPs [71, 3, 2]. Note that we only require this reachability assumption in M sim . We also assume we are given access to function classes F h : S × A → [0 , H ] and let F := F 1 ×F 2 × . . . ×F H . Since no reward is collected in the ( H +1) th step we take f H +1 = 0 . For any f : S × A → R , we let π f h ( s ) := arg max a ∈A f h ( s, a ) . We define the Bellman operator on some function f h +1 : S × A → R as:

We make the following standard assumption on F .

Assumption 4 (Bellman Completeness) . For all f h +1 ∈ F h +1 , we have T real f h +1 , T sim f h +1 ∈ F h , where T real and T sim denote the Bellman operators on M real and M sim , respectively.

PAC Reinforcement Learning. Our goal is to find a policy ̂ π that achieves maximum reward in M real . Formally, we consider the PAC (Probably-Approximately-Correct) RL setting.

Definition 3.2 (PAC Reinforcement Learning) . Given some ϵ > 0 and δ > 0 , with probability at least 1 -δ identify some policy ̂ π such that: V real , ̂ π 0 ≥ V real ,⋆ 0 -ϵ .

We will be particularly interested in solving the PAC RL problem with the aid of a simulator, using the minimum number of samples from M real possible, as we will formalize in the following. As we will see, while it is straightforward to achieve this objective using M sim if ϵ = O ( ϵ sim ) , naive transfer methods can fail to achieve this completely if ϵ ≪ ϵ sim . As such, our primary focus will be on developing efficient sim2real methods in this regime.

4 Theoretical Results

In this section we provide our main theoretical results. We first present two negative results: in Section 4.1 showing that 'naive exploration'-utilizing only a least-squares regression oracle and random exploration approaches such as ζ -greedy 3 -is provably inefficient, and in Section 4.2 showing that directly transferring the optimal policy from M sim to M real is unable to efficiently obtain a policy with suboptimality better than O ( ϵ sim ) in real. Then in Section 4.3 we present our main positive result, showing that by utilizing the same oracles as in Sections 4.1 and 4.2-a least-squares regression oracle, simulator access, and the ability to take actions randomly-we can efficiently learn an ϵ -optimal policy for ϵ ≪ ϵ sim in M real by carefully utilizing the simulator to learn exploration policies.

4.1 Naive Exploration is Provably Inefficient

While a variety of works have developed provably efficient methods for solving PAC RL in low-rank MDPs [1, 62, 43, 41], these works typically either rely on complex computation oracles or carefully directed exploration strategies which are rarely utilized in practice. In contrast, RL methods utilized in practice typically rely on 'simple' computation oracles and exploration strategies. Before considering the sim2real setting, we first show that such 'simple' strategies are insufficient for efficient PAC RL. To instantiate such strategies, we consider a least-squares regression oracle, often available in practice.

Oracle 4.1 (Least-Squares Regression Oracle) . We assume access to a least-squares regression oracle such that, for any h and dataset D = { ( s t , a t , y t ) } T t =1 , we can compute arg min f ∈F h ∑ T t =1 ( f ( s t , a t ) -y t ) 2 .

We couple this oracle with 'naive exploration', which here we use to refer to any method that explores by randomly perturbing the action recommended by the current estimate of the optimal policy. While a variety of instantiations of naive exploration exist (see e.g. [11]), we consider a particularly common formulation, ζ -greedy exploration.

Protocol 4.1 ( ζ -Greedy Exploration) . Given access to a regression oracle, any ζ ∈ [0 , 1] , and time horizon T , consider the following protocol:

Interact with M real for T episodes. At every step of episode t + 1 , play π f t h ( s ) with probability 1 -ζ , and a ∼ unif( A ) otherwise, where:

Using collected data in any way desired, propose a policy ̂ π .

Protocol 4.1 forms the backbone of many algorithms used in practice. Despite its common application, as existing work [11] and the following result show, it is provably inefficient.

Proposition 1. For any H > 1 , ζ ∈ [0 , 1] , and c ≤ 1 / 6 , there exist some M real , 1 and M real , 2 such that both M real , 1 and M real , 2 satisfy Assumptions 2 and 4, and unless T ≥ Ω(2 H/ 2 ) , when running Protocol 4.1 we have:

Proposition 1 shows that, in a minimax sense, ζ -greedy exploration is insufficient for provably efficient reinforcement learning: on one of M real , 1 and M real , 2 , ζ -greedy exploration will only be able to find a policy that is suboptimal by a constant factor, unless we take an exponentially large number of samples. While we focus on ζ -greedy exploration in Proposition 1, this result extends to other types of naive exploration, for example, those given in [11]. See Section 5.2 for further discussion of the construction for Proposition 1.

4.2 Understanding the Limits of Direct sim2real Transfer

Proposition 1 shows that in general utilizing a least-squares regression oracle with ζ -greedy exploration is insufficient for provably efficient RL. Can this be made efficient with access to a simulator

M sim ? In practice, standard sim2real methodology typically trains a policy to accomplish the goal task in M sim , and then transfers this policy to M real . We refer to this methodology as direct sim2real transfer . The following canonical result, usually referred to as the 'simulation lemma' [24, 25, 6, 22], provides a sufficient guarantee for direct sim2real transfer to succeed under Assumption 1.

Proposition 2 (Simulation Lemma) . Let π sim ,⋆ denote an optimal policy in M sim . Then under Assumption 1 we have V real ,π sim ,⋆ 0 ≥ V real ,⋆ 0 -2 H 2 ϵ sim .

Proposition 2 shows that, as long as ϵ ≥ 2 H 2 ϵ sim , direct sim2real transfer succeeds in obtaining an ϵ -optimal policy in M real . While this justifies direct sim2real transfer in settings where M sim and M real are sufficiently close, we next show that given access only to π sim ,⋆ and a least-squares regression oracle-even when coupled with random exploration-we cannot hope to efficiently obtain a policy with suboptimality less than O ( ϵ sim ) on M real using naive exploration. To formalize this, we consider the following interaction protocol.

Protocol 4.2 (Direct sim2real Transfer with Naive Exploration) . Given access to π sim ,⋆ , an optimal policy in M sim , any ζ ∈ [0 , 1] , and time horizon T , consider the following protocol:

Interact with M real for T episodes, and at each step h and state s play π sim ,⋆ h ( · | s ) with probability 1 -ζ , and a ∼ unif( A ) with probability ζ .

Using collected data in any way desired, propose a policy ̂ π .

Protocol 4.2 is a standard instantiation of direct sim2real transfer commonly found in the literature, and couples playing the optimal policy from M sim with naive exploration. We have the following.

Proposition 3. With the same choice of M real , 1 and M real , 2 as in Proposition 1, there exists some M sim such that both M real , 1 and M real , 2 satisfy Assumption 1 with M sim for ϵ sim ← c , Assumptions 2 to 4 hold, and unless T ≥ Ω(2 H ) when running Protocol 4.2, we have:

Proposition 3 shows that there exists a setting where there are two possible M real satisfying Assumption 1 with M sim , and where, using direct policy transfer, unless we interact with M real for exponentially many episodes (in H ), we cannot determine a better than Ω( ϵ sim ) -optimal policy for the worst-case M real . Together, Propositions 2 and 3 show that, while we can utilize direct sim2real transfer to learn a policy that is O ( ϵ sim ) -optimal in M real , if our goal is to learn an ϵ -optimal policy for ϵ ≪ ϵ sim , direct sim2real transfer is unable to efficiently achieve this.

4.3 Efficient sim2real Transfer via Exploration Policy Transfer

Does there exist some way to utilize M sim and a least-squares regression oracle to enable efficient learning in M real , even when ϵ ≪ ϵ sim ? Our key insight is that, rather than transferring the policy that optimally solves the task in M sim , we should instead transfer policies that explore effectively in M sim . While learning to solve a task may require very precise actions, we can often obtain sufficiently rich data with relatively imprecise actions-it is easier to learn to explore than learn to solve a task. In such settings, directly transferring a policy to solve the task will likely fail due to imprecision in the simulator, but it may be possible to still transfer a policy that generates exploratory data. To formalize this, we consider the following access model to M sim .

Oracle 4.2 ( M sim Access) . We may interact with M sim by either:

(Trajectory Sampling) For any policy π , sampling a trajectory { ( s h , a h , r h , s h +1 ) } H h =1 generated by playing π on M sim .

(Policy Optimization) For any reward ˜ r , computing a policy π sim ( ˜ r ) maximizing ˜ r on M sim .

While access to such a policy optimization oracle is unrealistic in M real , where we want to minimize the number of samples collected, given cheap access to samples in M sim , such an oracle can often be (approximately) implemented in practice 4 . Note that under Oracle 4.2 we only assume black-box

access to our simulator-rather than allowing the behavior of the simulator to be queried at arbitrary states, we are simply allowed to roll out policies on M sim , and compute optimal policies. Given Oracle 4.2, as well as our least-squares regression oracle, Oracle 4.1, we propose the following algorithm.

Algorithm 1 sim2real Exploration Policy Transfer

1: input: budget T , confidence δ , simulator M sim // Learn policies Π h exp which cover feature space in M sim

2: Π h exp ← LEARNEXPPOLICIES ( M sim , δ, 4 A 3 ϵ H , h ) (Algorithm 5) for all h ∈ [ H ]

3: ˜ Π h exp ←{ ˜ π exp : ˜ π exp plays π exp up to step h , then plays actions randomly, ∀ π exp ∈ Π h exp } // Explore in M real via ˜ Π exp

4: Play π exp ∼ unif( { unif( ˜ Π h exp ) } H h =1 ) for T/ 2 episodes in M real , add data to D // Estimate optimal policy on collected data

5: for h = H,H -1 , . . . , 1 do

6: ̂ f h ← arg min f ∈F ∑ ( s,a,r,s ′ ) ∈ D ( f h ( s, a ) -r -max a ′ ̂ f h +1 ( s ′ , a ′ )) 2

7: Compute π sim ,⋆ via Oracle 4.2

8: Play π sim ,⋆ for T/ 4 episodes in real, compute average return ̂ V real ,π sim ,⋆ 0

9: Play π ̂ f for T/ 4 episodes in real, compute average return ̂ V real ,π ̂ f 0

10: return ̂ π ← arg max π ∈{ π ̂ f ,π sim ,⋆ } ̂ V real ,π 0

Algorithm 1 first calls a subroutine LEARNEXPPOLICIES, which learns a set of policies that provide rich data coverage on M sim -precisely, LEARNEXPPOLICIES returns policies { Π h exp } h ∈ [ H ] which induce covariates with lower-bounded minimum eigenvalue on M sim and relies only on Oracle 4.2 (as well as knowledge of the featurization of M sim , ϕ s ) to find such policies. Algorithm 1 then plays these exploration policies in M real , coupled with random exploration, and applies the regression oracle to the data they collect. Finally, it estimates the value of the policy learned by the regression oracle and π sim ,⋆ , and returns whichever is best. We have the following.

Theorem 1. If Assumptions 1 to 4 hold and

with probability at least 1 -δ , Algorithm 1 returns a policy ̂ π such that V real ,⋆ 0 -V real , ̂ π 0 ≤ ϵ , and Oracles 4.1 and 4.2 are invoked at most poly( d, H, ϵ -1 , log 1 δ ) times.

Theorem 1 shows that, as long as ϵ sim satisfies (4.1), utilizing a simulator and least-squares regression oracle, Oracles 4.1 and 4.2, allows for efficient learning in M real , achieving a complexity scaling polynomially in problem parameters. This yields an exponential improvement over learning without a simulator using naive exploration or direct sim2real transfer-which Propositions 1 and 3 show have complexity scaling exponentially in the horizon-despite utilizing the same practical computation oracles. To the best of our knowledge, this result provides the first theoretical evidence that sim2real transfer can yield provable gains in RL beyond trivial settings where direct transfer succeeds.

Note that the condition in (4.1) is independent of ϵ -unlike direct sim2real transfer, which requires ϵ = O ( ϵ sim ) , we simply must assume ϵ sim is small enough that (4.1) holds, and Theorem 1 shows that we can efficiently learn an ϵ -optimal policy in M real for any ϵ > 0 . In Appendix B.4, we also present an extended version of Theorem 1, Theorem 3, which utilizes data from M sim to reduce the dependence on log |F| . In particular, instead of scaling with log |F| , it only scales with the log-cardinality of functions that are (approximately) Bellman-consistent on M sim . To illustrate the effectiveness of Theorem 1, we return to the instance of Propositions 1 and 3, where naive exploration and direct sim2real transfer fails. We have the following.

Proposition 4. In the setting of Propositions 1 and 3 and assuming that ϵ sim ≤ 1 8192 · 1 H , running Algorithm 1 will require poly( H,ϵ -1 ) · log 1 δ samples from M real in order to identify an ϵ -optimal policy in M real with probability at least 1 -δ , for any ϵ > 0 .

then as long as

Note that the condition required by Proposition 4 is simply that ϵ sim ≲ 1 /H -as long as our simulator satisfies this condition, we can efficiently transfer exploration policies to learn an ϵ -optimal policy, for any ϵ > 0 , while naive methods would be limited to only obtaining an Ω(1 /H ) -optimal policy.

Remark 4.1 (Necessity of Random Exploration) . Algorithm 1 achieves efficient exploration in M real by learning policies Π h exp in M sim that span the feature space of M sim (Line 2), and then playing these policies in M real , coupled with random exploration (Line 4). This use of random exploration is critical to obtaining Theorem 1. As we show in Proposition 5, if we omit the random exploration, Assumption 1 is not sufficient to guarantee Π h exp explores effectively in M real , even when (4.1) holds.

Remark 4.2 (Computational Efficiency) . Algorithm 1, as well as its main subroutine LEARNEXPPOLICIES, relies only on calls to Oracle 4.1 and Oracle 4.2. Thus, assuming we can efficiently implement these oracles, which is often the case in problem settings of interest, Algorithm 1 can be run in a computationally efficient manner.

5 Practical Algorithm and Experiments

We next validate the effectiveness of our proposal in practice: can a set of diverse exploration policies obtained from simulation improve the efficiency of real-world reinforcement learning? We start by showing that this holds for a simple, didactic, tabular environment in Section 5.2. From here, we consider several more realistic task domains: simulators inspired by real-world robotic manipulation tasks ( sim2sim transfer, Section 5.3); and an actual real-world sim2real experiment on a Franka robotic platform ( sim2real transfer, Section 5.4). Further details on all experiments, including additional baselines, can be found in Appendix E. Before stating our experimental results, we first provide a practical instantiation of Algorithm 1 that we can apply with real robotic systems and neural network function approximators.

5.1 Practical Instantiation of Exploration Policy Transfer

The key idea behind Algorithm 1 is quite simple: learn a set of exploratory policies in M sim -policies which provide rich data coverage in M sim -and transfer these policies to M real , coupled with random exploration, using the collected data to determine a near-optimal policy for M real . Algorithm 1 provides a particular instantiation of this principle, learning exploratory policies in M sim via the LEARNEXPPOLICIES subroutine, which aims to cover the feature space of M sim , and utilizing a least-squares regression oracle to compute an optimal policy given the data collected in M real . In practice, however, other instantiations of this principle are possible by replacing LEARNEXPPOLICIES with any procedure which generates exploratory policies in M sim , and replacing the regression oracle with any RL algorithm able to learn from off-policy data. We consider a general meta-algorithm instantiating this in Algorithm 2.

Algorithm 2 Practical sim2real Exploration Policy Transfer Meta-Algorithm

1: Input: Simulator M sim , real environment M real , simulator budget T sim , real budget T , algorithm to generate exploratory policies in sim A exp , algorithm to solve policy optimization in real A po // Learn exploratory policies in M sim 2: Run A exp for T sim steps in M sim to generate set of exploratory policies Π exp // Deploy exploratory policies in M real 3: for t = 1 , 2 , . . . , T / 2 do 4: Draw π exp ∼ unif(Π exp ) , play in M real for one episode, add data to replay buffer of A po 5: Run A po for one episode // optional if A po learns fully offline

In practice, A exp and A po can be instantiated with a variety of algorithms. For example, we might take A exp to be an RND [7] or bootstrapped Q-learning-style [45, 31] algorithm, or any unsupervised RL procedure [48, 14, 32, 47], and A po to be an off-policy policy optimization algorithm such as soft actor-critic (SAC) [16] or implicit Q -learning (IQL) [28].

For the following experiments, we instantiate Algorithm 2 by setting A exp to an algorithm inspired by recent work on inducing diverse behaviors in RL [14, 30], and A po to SAC. In particular, A exp simultaneously trains an ensemble of policies Π exp = { π i exp } n i =1 and a discriminator d θ : S × [ n ] → R , where d θ is trained to discriminate between the behaviors of each policy π i exp , and π i exp

0.0

0.5

1.0

1.5

0.10

0.15

0.20

0.25

0.30

0.35

Reward

Exploration Policy Transfer (Ours)

Direct Policy Transfer

Q-Learning with Naive Exploration

Action

a

$1

$1

$1

$1

$1

$1

Psim(sIs,a)

=

Action

a

Pi(s|s,a)=

rh(s1，a)

rH(S,a)

=

=

，a∈{a，a}

8

1

8H

h

$2

$2

$2

$2

$2

Pl(s I s,a)

=

=

4

1

4

Steps

×10 3

is optimized on a weighting of the true task reward and the exploration reward induced by the discriminator, r e ( s, i ) := log exp( d θ ( s,i )) ∑ j ∈ [ n ] exp( d θ ( s,j )) . As shown in existing work [14, 30], this simple training objective effectively induces diverse behavior with temporally correlated exploration while remaining within the vicinity of the optimal policy, using standard optimization techniques. Note that the particular choice of algorithm is less critical here than abiding by the recipes laid out in the meta-algorithm (Algorithm 2). The particular instantiation that we run for our experiments is detailed in Algorithm 6, along with further details in Appendix E.2.

5.2 Didactic Combination Lock Experiment

We first consider a variant of the construction used to prove Propositions 1 and 3, itself a variant of the classic combination lock instance. We illustrate this instance in Figure 2. Unless noted, all transitions occur with probability 1, and rewards are 0. Here, in M sim the optimal policy, π sim ,⋆ , plays action a 2 for all steps h < H -1 , while in M real , the optimal policy plays action a 1 at every step. Which policy is optimal is determined by the outgoing transition from s 1 at the ( H -1) th step and, as such, to identify the optimal policy, any algorithm must reach s 1 at the ( H -1) th step. As s 1 will only be reached at step H -1 by playing a 1 for H -1 consecutive times, any algorithm relying on naive exploration will take exponentially long to identify the optimal policy. Furthermore, playing π sim ,⋆ coupled with random exploration will similarly take an exponential number of episodes, since π sim ,⋆ always plays a 2 . As such, both direct sim2real policy transfer as well as Q -learning with naive exploration (Protocol 4.1) will fail to find the optimal policy in M real . However, if we transfer exploratory policies from M sim , since M sim and M real behave identically up to step H -1 , these policies can efficiently traverse M real , reach s 1 at step H -1 , and identify the optimal policy. We compare our approach of exploration policy transfer to these baselines methods and illustrate the performance of each in Figure 2. As this is a simple tabular instance, we implement Algorithm 1 directly here. As Figure 2 shows, the intuition described above leads to real gains in practice-exploration policy transfer quickly identifies the optimal policy, while more naive approach fail completely over the time horizon we considered.

5.3 Realistic Robotics sim2sim Experiment

To test the ability of our proposed method to scale to more complex problems, we next experiment on a sim2sim transfer setting with a realistic robotic simulator. We consider TychoEnv, a simulator of the 7DOF Tycho robotics platform introduced by [73], and shown in Figure 3. We test sim2sim transfer on a reaching task where the goal is to touch a small ball hanging in the air with the tip of the chopstick end effector. The agent perceives the ball and its own end effector pose and outputs a delta in its desired end effector pose as a command. We set M sim and M real to be two instances of TychoEnv with slightly different parameters to model real-world sim2real transfer. Precisely, we change the action bounds and control frequency from M sim to M real .

We aim to compare our approach of exploration policy transfer with direct sim2real policy transfer. To this end, we first train a policy in M sim that solves the task in M sim , π sim ,⋆ , and then utilize this policy in place of Π exp

in Algorithm 2. We instantiate our approach of exploration policy transfer as outlined above. Our aim in this experiment is to illustrate how the quality of the data provided by direct policy transfer vs. exploration policy transfer affects learning. As such, for both approaches we simply initialize our SAC agent in M real , A po , from scratch, and set the reward in M real to be sparse: the agent only receives a non-zero reward if it successfully touches the ball. For each approach, we repeat the process of training in M sim four times, and for each of these run them for two trials in M real .

Weillustrate our results in Figure 4. As this figure illustrates, direct policy transfer fails to learn completely, while exploration policy transfer successfully solves the task. Investigating the behavior of each method, we find that the policies transferred via exploration policy transfer, while failing to solve the task with perfect accuracy, when coupled with naive exploration are able to successfully make contact with the ball on occasion. This provides sufficiently rich data for SAC to ultimately learn to solve the task. In contrast, direct policy transfer fails to collect any reward when run in M real , and, given the sparse reward nature of the task, SAC is unable to locate any reward and learn.We include an additional sim2sim experiment on the Franka Emika Panda Robot Arm in Appendix E.4.

5.4 Real-World Robotic sim2real Experiment

Finally, we demonstrate our algorithm for actual sim2real policy transfer for a manipulation task on a real-world Franka Emika Panda robot arm [17] with a parallel gripper. Our task is to push a 75mm diameter cylindrical 'puck" from the center to the edge of the surface, as shown in Figure 1, with the arm initialized at random locations. The observed state s = [ p ee , p obj ] ∈ R 4 consists of the planar Cartesian coordinate of the end effector p ee along with the center of mass of the puck p obj . Our policy outputs planar end effector position deltas a = ∆ p ee ∈ R 2 , evaluated at 8 Hz, which are passed into a lower-level joint position PID controller. We use an Intel Realsense D435 depth camera to track the location of the puck. Our reward function is a sum of a success indicator (indicating when the puck has been pushed to the edge of the surface) and terms which give negative reward if the distance from the end effector to the puck, or puck to the goal, are too large (see (E.1)); in particular, a reward greater than 0 indicates success.

We run the instantiation of Algorithm 2 outlined above. In particular, we train an ensemble of n = 15 exploration policies, training for 20 million steps in M sim . In addition, we train a policy that solves the task in M sim , π sim ,⋆ . We use a custom simulator of the arm, where during training the friction of the table is randomized and noise is added to the observations.

We observe a substantial sim2real gap between our simulator and the real robot, with policies trained in simulation failing to complete the pushing task zero shot in real, even when trained with domain randomization. We compare direct sim2real policy transfer against our method of transferring exploration policies. For direct policy transfer, we simply run SAC to finetune π sim ,⋆ in the real world, using the current policy to collect data. For exploration policy transfer, we instead utilize Π exp , our ensemble of exploration policies, to collect data in the real world. We run this in tandem with an SAC agent, feeding the data from the exploration policies into the SAC agent's replay buffer. Unlike in Section 5.3, rather than initializing the SAC policy from scratch, we set the initial policy as π sim ,⋆ , and fine-tune from this on the data collected from playing Π exp . See Appendix E.5 for additional details.

Our results are shown on the right side of Figure 1. Statistics are computed over 6 runs for each method. Direct policy transfer with finetuning is unable to solve the task in real in each of the 6 runs, and converges to a suboptimal solution. However, our method is able to solve the task successfully each time and achieve a substantially higher reward.

6 Discussion

In this work, we have demonstrated that simulators can make naive exploration efficient even in settings where direct sim2real transfer fails, if they are used to train a set of exploration policies. We highlight several limitations of this work, which we believe are interesting future research questions:

Our focus is purely on dynamics shift-where the dynamics of sim and real differ, but the environments are otherwise the same. While dynamics shift is common in many scenarios, other types of shift can exist as well, for example perceptual shift. How can we best handle these types of shift?

How can we utilize a simulator in sim2real transfer if we can reset it arbitrarily, rather than just allowing for black-box access? Does the ability to reset allow us to improve sample efficiency further?

Is the reachability condition, Assumption 3, necessary for successful exploration transfer?

0

1

2

3

Steps

×10 6

0

5

10

15

20

Reward

Exploration Policy Transfer (Ours)

Direct Policy Transfer

References

Alekh Agarwal, Sham Kakade, Akshay Krishnamurthy, and Wen Sun. Flambe: Structural complexity and representation learning of low rank mdps. Advances in neural information processing systems , 33:20095-20107, 2020.

Alekh Agarwal, Yuda Song, Wen Sun, Kaiwen Wang, Mengdi Wang, and Xuezhou Zhang. Provable benefits of representational transfer in reinforcement learning. In The Thirty Sixth Annual Conference on Learning Theory , pages 2114-2187. PMLR, 2023.

Naman Agarwal, Syomantak Chaudhuri, Prateek Jain, Dheeraj Nagaraj, and Praneeth Netrapalli. Online target q-learning with reverse experience replay: Efficiently finding the optimal policy for linear mdps. arXiv preprint arXiv:2110.08440 , 2021.

Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's cube with a robot hand. arXiv preprint arXiv:1910.07113 , 2019.

Philip Amortila, Nan Jiang, Dhruv Madeka, and Dean P Foster. A few expert queries suffices for sample-efficient rl with resets and linear value approximation. Advances in Neural Information Processing Systems , 35:29637-29648, 2022.

Ronen I Brafman and Moshe Tennenholtz. R-max-a general polynomial time algorithm for near-optimal reinforcement learning. Journal of Machine Learning Research , 3(Oct):213-231, 2002.

Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv:1810.12894 , 2018.

Yevgen Chebotar, Ankur Handa, Viktor Makoviychuk, Miles Macklin, Jan Issac, Nathan Ratliff, and Dieter Fox. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. In ICRA , 2019.

Tao Chen, Megha Tippur, Siyang Wu, Vikash Kumar, Edward Adelson, and Pulkit Agrawal. Visual dexterity: In-hand reorientation of novel and complex object shapes. Science Robotics , 8 (84):eadc9244, 2023.

Yuan Cheng, Songtao Feng, Jing Yang, Hong Zhang, and Yingbin Liang. Provable benefit of multitask representation learning in reinforcement learning. Advances in Neural Information Processing Systems , 35:31741-31754, 2022.

Chris Dann, Yishay Mansour, Mehryar Mohri, Ayush Sekhari, and Karthik Sridharan. Guarantees for epsilon-greedy reinforcement learning with function approximation. In International conference on machine learning , pages 4666-4689. PMLR, 2022.

Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, et al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature , 602(7897): 414-419, 2022.

Benjamin Eysenbach and Sergey Levine. Maximum entropy rl (provably) solves some robust rl problems. arXiv preprint arXiv:2103.06257 , 2021.

Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv:1802.06070 , 2018.

Raj Ghugare, Santiago Miret, Adriana Hugessen, Mariano Phielipp, and Glen Berseth. Searching for high-value molecules using reinforcement learning and transformers. arXiv preprint arXiv:2310.02902 , 2023.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning , pages 1861-1870. PMLR, 2018.

Sami Haddadin, Sven Parusel, Lars Johannsmeier, Saskia Golz, Simon Gabl, Florian Walch, Mohamadreza Sabaghian, Christoph Jähne, Lukas Hausperger, and Simon Haddadin. The franka emika robot: A reference platform for robotics research and education. IEEE Robotics & Automation Magazine , 29(2):46-64, 2022. doi: 10.1109/MRA.2021.3138382.

Sebastian Höfer, Kostas Bekris, Ankur Handa, Juan Camilo Gamboa, Melissa Mozifian, Florian Golemo, Chris Atkeson, Dieter Fox, Ken Goldberg, John Leonard, et al. Sim2real in robotics and automation: Applications and challenges. IEEE transactions on automation science and engineering , 18(2):398-400, 2021.

Stephen James, Paul Wohlhart, Mrinal Kalakrishnan, Dmitry Kalashnikov, Alex Irpan, Julian Ibarz, Sergey Levine, Raia Hadsell, and Konstantinos Bousmalis. Sim-to-real via sim-to-sim: Data-efficient robotic grasping via randomized-to-canonical adaptation networks. arxiv e-prints, page. arXiv preprint arXiv:1812.07252 , 2018.

Yiding Jiang, J Zico Kolter, and Roberta Raileanu. On the importance of exploration for generalization in reinforcement learning. arXiv preprint arXiv:2306.05483 , 2023.

Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory , pages 21372143. PMLR, 2020.

Sham Kakade, Michael J Kearns, and John Langford. Exploration in metric state spaces. In Proceedings of the 20th International Conference on Machine Learning (ICML-03) , pages 306-312, 2003.

Elia Kaufmann, Leonard Bauersfeld, Antonio Loquercio, Matthias Müller, Vladlen Koltun, and Davide Scaramuzza. Champion-level drone racing using deep reinforcement learning. Nature , 620(7976):982-987, 2023.

Michael Kearns and Daphne Koller. Efficient reinforcement learning in factored mdps. In IJCAI , volume 16, pages 740-747, 1999.

Michael Kearns and Satinder Singh. Near-optimal reinforcement learning in polynomial time. Machine learning , 49:209-232, 2002.

B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent Transportation Systems , 23(6):4909-4926, 2021.

Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research , 32(11):1238-1274, 2013.

Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169 , 2021.

Ashish Kumar, Zipeng Fu, Deepak Pathak, and Jitendra Malik. Rma: Rapid motor adaptation for legged robots. arXiv preprint arXiv:2107.04034 , 2021.

Saurabh Kumar, Aviral Kumar, Sergey Levine, and Chelsea Finn. One solution is not all you need: Few-shot extrapolation via structured maxent rl. Advances in Neural Information Processing Systems , 33:8198-8210, 2020.

Kimin Lee, Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning. In International Conference on Machine Learning , pages 6131-6141. PMLR, 2021.

Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Ruslan Salakhutdinov. Efficient exploration via state marginal matching. arXiv preprint arXiv:1906.05274 , 2019.

Gen Li, Yuxin Chen, Yuejie Chi, Yuantao Gu, and Yuting Wei. Sample-efficient reinforcement learning is feasible for linearly realizable mdps with limited revisiting. Advances in Neural Information Processing Systems , 34:16671-16685, 2021.

Yao Liu, Dipendra Misra, Miro Dudík, and Robert E Schapire. Provably sample-efficient rl with side information about latent dynamics. Advances in Neural Information Processing Systems , 35:33482-33493, 2022.

Rui Lu, Gao Huang, and Simon S Du. On the power of multitask representation learning in linear mdp. arXiv preprint arXiv:2106.08053 , 2021.

Dhruv Malik, Yuanzhi Li, and Pradeep Ravikumar. When is generalizable reinforcement learning tractable? Advances in Neural Information Processing Systems , 34:8032-8045, 2021.

Timothy A Mann and Yoonsuck Choe. Directed exploration in reinforcement learning with transferred knowledge. In European Workshop on Reinforcement Learning , pages 59-76. PMLR, 2013.

Gabriel B Margolis, Xiang Fu, Yandong Ji, and Pulkit Agrawal. Learning physically grounded robot vision with active sensing motor policies. In CoRL , 2023.

Bhairav Mehta, Manfred Diaz, Florian Golemo, Christopher J Pal, and Liam Paull. Active domain randomization. In CoRL , 2020.

Marius Memmel, Andrew Wagenmaker, Chuning Zhu, Patrick Yin, Dieter Fox, and Abhishek Gupta. Asid: Active exploration for system identification in robotic manipulation. arXiv preprint arXiv:2404.12308 , 2024.

Zak Mhammedi, Adam Block, Dylan J Foster, and Alexander Rakhlin. Efficient model-free exploration in low-rank mdps. Advances in Neural Information Processing Systems , 36, 2024.

Zakaria Mhammedi, Dylan J Foster, and Alexander Rakhlin. The power of resets in online reinforcement learning. arXiv preprint arXiv:2404.15417 , 2024.

Aditya Modi, Jinglin Chen, Akshay Krishnamurthy, Nan Jiang, and Alekh Agarwal. Modelfree representation learning and exploration in low-rank mdps. Journal of Machine Learning Research , 25(6):1-76, 2024.

Fabio Muratore, Michael Gienger, and Jan Peters. Assessing transferability from simulation to reality for reinforcement learning. IEEE transactions on pattern analysis and machine intelligence , 2019.

Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. Advances in neural information processing systems , 29, 2016.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems , 35:27730-27744, 2022.

Seohong Park, Oleh Rybkin, and Sergey Levine. Metra: Scalable unsupervised rl with metricaware abstraction. arXiv preprint arXiv:2310.08887 , 2023.

Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning , pages 27782787. PMLR, 2017.

Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference on robotics and automation (ICRA) , pages 3803-3810. IEEE, 2018.

Xue Bin Peng, Erwin Coumans, Tingnan Zhang, Tsang-Wei Lee, Jie Tan, and Sergey Levine. Learning agile robotic locomotion skills by imitating animals. arXiv preprint arXiv:2004.00784 , 2020.

Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research , 22(268):1-8, 2021.

Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv:1709.10087 , 2017.

FL Silva, Jiachen Yang, Mikel Landajuela, Andre Goncalves, Alexander Ladd, Daniel Faissol, and Brenden Petersen. Toward multi-fidelity reinforcement learning for symbolic optimization. Technical report, Lawrence Livermore National Laboratory (LLNL), Livermore, CA (United States), 2023.

David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature , 529(7587):484-489, 2016.

Rohan Sinha, James Harrison, Spencer M Richards, and Marco Pavone. Adaptive robust model predictive control with matched and unmatched uncertainty. In 2022 American Control Conference (ACC) , 2022.

Yuda Song, Aditi Mavalankar, Wen Sun, and Sicun Gao. Provably efficient model-based policy adaptation. arXiv preprint arXiv:2006.08051 , 2020.

Yuda Song, Yifei Zhou, Ayush Sekhari, J Andrew Bagnell, Akshay Krishnamurthy, and Wen Sun. Hybrid rl: Using both offline and online data can make rl efficient. arXiv preprint arXiv:2210.06718 , 2022.

Yanchao Sun, Ruijie Zheng, Xiyao Wang, Andrew Cohen, and Furong Huang. Transfer rl across observation feature spaces via model-based regularization. arXiv preprint arXiv:2201.00248 , 2022.

Andrea Tirinzoni, Matteo Pirotta, and Alessandro Lazaric. A fully problem-dependent regret lower bound for finite-horizon mdps. arXiv preprint arXiv:2106.13013 , 2021.

Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS) , pages 23-30. IEEE, 2017.

Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 5026-5033, 2012. doi: 10.1109/IROS.2012.6386109.

Masatoshi Uehara, Xuezhou Zhang, and Wen Sun. Representation learning for online and offline rl in low-rank mdps. arXiv preprint arXiv:2110.04652 , 2021.

Andrew Wagenmaker and Kevin G Jamieson. Instance-dependent near-optimal policy identification in linear mdps via online experiment design. Advances in Neural Information Processing Systems , 35:5968-5981, 2022.

Andrew Wagenmaker, Guanya Shi, and Kevin Jamieson. Optimal exploration for model-based rl in nonlinear systems. arXiv preprint arXiv:2306.09210 , 2023.

Andrew J Wagenmaker, Yifang Chen, Max Simchowitz, Simon Du, and Kevin Jamieson. Reward-free rl is no harder than reward-aware rl in linear markov decision processes. In International Conference on Machine Learning , pages 22430-22456. PMLR, 2022.

Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. arXiv preprint arXiv:1611.05763 , 2016.

Gellert Weisz, Philip Amortila, Barnabás Janzer, Yasin Abbasi-Yadkori, Nan Jiang, and Csaba Szepesvári. On query-efficient planning in mdps under linear realizability of the optimal state-value function. In Conference on Learning Theory , pages 4355-4385. PMLR, 2021.

Gellért Weisz, András György, Tadashi Kozuno, and Csaba Szepesvári. Confident approximate policy iteration for efficient local planning in q π -realizable mdps. Advances in Neural Information Processing Systems , 35:25547-25559, 2022.

Haotian Ye, Xiaoyu Chen, Liwei Wang, and Simon Shaolei Du. On the power of pre-training for generalization in rl: provable benefits and hardness. In International Conference on Machine Learning , pages 39770-39800. PMLR, 2023.

Dong Yin, Botao Hao, Yasin Abbasi-Yadkori, Nevena Lazi´ c, and Csaba Szepesvári. Efficient local planning with linear function approximation. In International Conference on Algorithmic Learning Theory , pages 1165-1192. PMLR, 2022.

Andrea Zanette, Alessandro Lazaric, Mykel J Kochenderfer, and Emma Brunskill. Provably efficient reward-agnostic navigation with linear value iteration. Advances in Neural Information Processing Systems , 33:11756-11766, 2020.

Tianjun Zhang, Tongzheng Ren, Mengjiao Yang, Joseph Gonzalez, Dale Schuurmans, and Bo Dai. Making linear mdps practical via contrastive representation learning. In International Conference on Machine Learning , pages 26447-26466. PMLR, 2022.

Yunchu Zhang, Liyiming Ke, Abhay Deshpande, Abhishek Gupta, and Siddhartha Srinivasa. Cherry-picking with reinforcement learning. arXiv preprint arXiv:2303.05508 , 15, 2023.

Wenshuai Zhao, Jorge Peña Queralta, and Tomi Westerlund. Sim-to-real transfer in deep reinforcement learning for robotics: a survey. In 2020 IEEE symposium series on computational intelligence (SSCI) , pages 737-744. IEEE, 2020.

A Technical Results

We denote the state-visitations for some policy π as w π h ( s, a ) := P π [( s h , a h ) = ( s, a )] , w π h ( Z ) := P π [( s h , a h ) ∈ Z ] , for Z ⊆ S × A . For X ⊆ R d , we denote w π h ( X ) := P π [ ϕ ( s h , a h ) ∈ X ] , for ϕ the featurization of the environment.

Lemma A.1. Consider MDPs M and ˜ M with transition kernels P and ˜ P . Assume that both M and ˜ M start in the same state s 0 and that, for each ( s, a, h ) :

Consider some reward function r such that ∑ H h =1 r h ( s h , a h ) ≤ R for all possible sequences { ( s h , a h ) } H h =1 . Then it follows that, for any π and ( s, a, h ) ,

Proof. We prove this by induction. First, assume that for some h and all s, a , we have | Q M,π h +1 ( s, a ) -Q ˜ M,π h +1 ( s, a ) | ≤ ϵ h +1 . By definition we have

and similarly for Q ˜ M,π h +1 ( s, a ) . Thus:

where ( a ) follows from the triangle inequality and ( b ) follows from the inductive hypothesis. Under (A.1), we can bound

It follows that for any ( s, a ) , | Q M,π h ( s, a ) -Q ˜ M,π h ( s, a ) | ≤ ϵ h =: ϵ sim R + ϵ h +1 .

The base case follows trivially with ϵ H = 0 since for any MDP we have that Q M,π H ( s, a ) = r H ( s, a ) = Q ˜ M,π H ( s, a ) .

Lemma A.2. Under the same setting as Lemma A.1 and for any h , π , and Z ⊆ S × A , we have

Proof. This is an immediate consequence of Lemma A.1 since, setting the reward r h ′ ( s, a ) = I { ( s, a ) ∈ Z , h ′ = h } , we can set R = 1 and have V M,π 0 = w M,π h ( Z ) .

Lemma A.3 (Proposition 2) . Under Assumption 1, we have that

Proof. We prove the result for real -the result for sim follows analogously. We have

The result then follows by applying Lemma A.1 to bound each of these terms by H 2 ϵ sim .

Lemma A.4. For any f ∈ F ,

Proof. We write

and then bound each of these terms separately. By Lemma 5 of [57] we have

Similarly, by Lemma 4 of [57] we have

B Proof of Main Results

In Appendix B.1 we first provide a general result on learning in real when collecting data via a fixed set of exploration policies, given a particular coverage assumption. Then in Appendix B.2, we show that by playing a set of policies which induce full-rank covariates in sim , these policies provide sufficient coverage for learning in real . Finally in Appendices B.3 and B.4, we use these results to prove Theorems 1 and 3. Throughout the appendix we develop the supporting lemmas for our more general result, Theorem 3, which utilizes the simulator to restrict the version space (i.e. the dependence on |F| ) in addition to utilizing the simulator to aid in exploration.

Throughout this and the following section we assume that Assumption 4 holds. We also assume that f h ∈ [0 , V max ] instead of f h ∈ [0 , H ] , for some V max > 0 . For any f ∈ F , we denote the Bellman residual as

Note that by assumption on F , we have E h ( f )( s, a ) ∈ [ -V max , V max ] .

For any policy π , we denote Λ s π,h := E sim ,π [ ϕ s ( s h , a h ) ϕ s ( s h , a h ) ⊤ ] and Λ r π,h := E real ,π [ ϕ r ( s h , a h ) ϕ r ( s h , a h ) ⊤ ] .

Necessity of Random Exploration. Algorithm 1 achieves efficient exploration in M real by first learning a set of policies Π h exp in M sim that span the feature space of M sim (Line 2), achieving

and then playing these policies in M real , coupled with random exploration (Line 4). In particular, Algorithm 1 plays policies from ˜ Π h exp , where each ˜ π exp ∈ ˜ Π h exp is defined as the policy which plays some π exp ∈ Π h exp up to step h , and then for steps h ′ = h +1 , . . . , H chooses actions uniformly at random. This use of random exploration is critical to obtaining Theorem 1. Indeed, under our

transfer model, condition (4.1) of Theorem 1 is not strong enough to ensure that policies satisfying (B.1) collect rich enough data in M real to allow for learning a near-optimal policy. While (4.1) is sufficient to guarantee that playing Π h exp on M real collects data which spans the feature space of M sim -that is, satisfying (B.1) but with the expectation over M sim replaced by an expectation of M real -this is insufficient for learning, as the following result shows.

Proposition 5. For any ϵ sim ≤ 1 / 2 , there exist some M sim , M real , 1 , and M real , 2 such that:

Both M real , 1 and M real , 2 satisfy Assumption 1 with M sim and Assumptions 2 to 4 hold.

There exists some policy π exp such that λ min ( E M sim ,π exp [ ϕ s ( s h , a h ) ϕ s ( s h , a h ) ⊤ ]) = 1 / 2 , ∀ h ∈ [ H ] , and for any T ≥ 0 , if we play π exp on M real for T steps, we have:

Proposition 5 holds because two MDPs may be 'close' in the sense of Assumption 1 but admit very different feature representations. As a result, transferring a policy that covers the feature space of M sim is not necessarily sufficient for covering the feature space of M real , which ultimately means that data collected from π exp is unable to identify the optimal policy in M real . Our key technical result, Lemma B.4, shows, however, that under Assumption 1 and (4.1), policies which achieve high coverage in M sim (i.e. satisfy (B.1)) are able to reach within a logarithmic number of steps of relevant states in M real . While the sample complexity of random exploration typically scales exponentially in the horizon, if the horizon over which we must explore is only logarithmic, the total complexity is then only polynomial. Theorem 1 critically relies on these facts-by playing policies in Π h exp up to step h and then exploring randomly, and repeating this for each h ∈ [ H ] , we show that sufficiently rich data is collected in M real for learning an ϵ -optimal policy.

B.1 Learning in real with Fixed Exploration Policies

Algorithm 3 sim2real transfer with fixed exploration policies (EXPLOREREAL)

1: input: exploration policies { π h exp } H h =1 , budget T , sim date D sim , sim regularization γ

2: Play π exp = unif( { π h exp } H h =1 ) for T episodes in real , add data to D

3:

for

h

=

H,H

4:

5: return π ̂ f

Lemma B.1. Consider running Algorithm 3. Assume that D sim was generated as in Assumption 5, via the procedure of Lemma C.3 run with some parameter β , and γ satisfies

Furthermore, assume that there exists some C , ϵ > 0 such that, for any π , h ∈ [ H ] , and Z ′ ⊆ S × A , we have:

-

1

, . . . ,

1

do

Then with probability at least 1 -2 δ , the policy π ̂ f generated by Algorithm 3 satisfies

for

Proof. Let E denote the good event of Lemma B.2, which holds with probability at least 1 -2 δ . By Lemma A.4 we have

Let

Then we have, for any π ,

where the second inequality follows from (B.3). On E , by Lemma B.2 and Jensen's inequality, we have

As this holds for each h and π , we have therefore shown that

This proves the result.

Lemma B.2. With probability at least 1 -2 δ , for each h ∈ [ H ] simultaneously, as long as the conditions on γ given in Lemma B.3 hold, we have

and ̂ f h ∈ ˜ F h ( π sim exp ) for all h ∈ [ H ] , where

Proof. Let ̂ F h denote the feasible set of (B.2) at step h . By Lemma B.3, with probability at least 1 -δ , ̂ F t h ⊆ ˜ F h , and, furthermore, that T real ̂ f h +1 is feasible. The result then follows from Lemma 3 of [57], since the constraint on the regression problem restricts the version space.

Lemma B.3. Assume that data in D sim is generated as in Assumption 5 via the procedure of Lemma C.3 run with some parameter β , and γ satisfies

Then with probability at least 1 -δ we have, for each h ∈ [ H ] :

T real ̂ f h +1 is feasible for (B.2) .

The set of feasible f for (B.2) is a subset of

Proof. By Lemma C.1, we have that with probability at least 1 -δ/ 2 H ,

By Lemma C.3, we have 12 dH β 2 ≤ T sim , which implies

Part 1 then follows given our assumption on γ .

To bound the feasible set for (B.2) we appeal to Lemma C.2 which states that with probability at least 1 -δ/ 2 H we have that the feasible set of (B.2) is a subset of

Again using that 12 dH β 2 ≤ T sim , we have have that this is a subset of

where the inclusion follows from our assumption on γ . The result then follows from a union bound.

B.2 Performance of Full-Rank sim Policies in real

Lemma B.4. Consider policies { π h exp } H h =1 , and assume that

and that π h exp plays actions uniformly at random for h ′ > h . Let π exp = unif( { π h exp } H h =1 ) . Then, for any π , κ > 0 , γ > 0 , h ∈ [ H ] , and Z ′ ⊆ S × A , we have

where

Proof. Denote

for some γ > 0 . We have

where ( a ) follows since for all ( s, a ) ∈ ˜ Z h +1 , we have 1 < ϕ r ( s, a ) ⊤ ( Λ r π h exp ,h +1 ) -1 ϕ r ( s, a ) /γ . By Lemma A.2, we then have that

where we have used the fact that π h exp ( a | s, h +1) = 1 /A for all ( s, a ) by assumption, and define P sim h ( ˜ S h +1 | s, a ) := P sim [ s h +1 ∈ ˜ S h +1 | s h = s, a h = a ] = ∫ ˜ S h +1 d µ s h ( s ) ⊤ ϕ s ( s, a ) , where the last equality follows from the definition of a linear MDP. Letting µ s h ( ˜ S h +1 ) := ∫ ˜ S h +1 d µ s h ( s ) , note that:

where the last inequality follows from (B.4). Combining this with (B.5), we have

Now note that, for any z ∈ S × A :

and we also have that P sim h ( ˜ S h +1 | z ) ≥ P real h ( ˜ S h +1 | z ) -ϵ sim under Assumption 1. Putting this together we have that for all z ∈ S × A :

Note that we can always take ϵ sim ≤ 1 , and will always have ¯ λ min ≤ 1 . This implies that ϵ sim ≤ √ A ¯ λ min ( d γ + Hϵ sim ) . Thus,

Coverage of π exp in real . Let k ⋆ := ⌈ log 1 /κ log 1 /ξ ⌉ , so that ξ k ⋆ ≤ κ . Let ¯ Z h := ( S × A ) \ ˜ Z h . Fix some Z ′ ⊆ ( S × A ) , h ∈ [ H ] , and policy π .

Consider some z ∈ ¯ Z h , and some S ′ ⊆ S . Then note that 5

where the last inequality follows from the definition of ¯ Z h . Note, though, that

This implies that for all z ∈ ¯ Z h ,

for w real ,π h ( Z | s h ′ = s ) := P real ,π [ z h ∈ Z | s h ′ = s ] . Note that we then have w real ,π h ( Z ′ | S h ′ ,i ) ∈ [2 -i +1 , 2 -i ) . By what we have just shown, we have that for z ∈ ¯ Z h ′

which implies that

For h ′ < h , define

¯

for any λ i ∈ △ S h ′ +1 ,i . Note also that, since π h ′ -1 exp plays randomly for all h ′′ ≥ h ′ , we have:

since with probability 1 /A h -h ′ on any given episode, π h ′ -1 exp will play the same sequence of actions as π from steps h ′ to h . It follows that we can bound the above as:

where ( a ) follows from (B.6) and since P real h ′ ( S h ′ ,i | z ) ≥ κ , and ( b ) follows choosing λ i ( s ) = w real ,π h ′ -1 exp h ′ +1 ( s ) /w real ,π h ′ -1 exp h ′ +1 ( S h ′ +1 ,i ) · I { s ∈ S h ′ +1 ,i } . We therefore have that, for all z ∈ ¯ Z h ′ :

Controlling events. Consider events E := { z h ∈ Z ′ } and E h ′ := { z h ′ ∈ ¯ Z h ′ } . We then have

We now analyze each of these terms. First, note that

We can then bound

where the inequality follows from (B.7). For the second term, we have

Note, however, that P real ,π [ E | ⋂ h -1 i = h -k ⋆ E c i ] ≤ ξ and P real ,π [ E c h -j | ⋂ h -j -1 i = h -k ⋆ E c i ] ≤ ξ for all j . We therefore can bound the above as

Altogether, then, we have that

Furthermore, since π exp = unif( { π h exp } H h =1 ) , we have w real ,π h ′ -2 exp h ( Z ′ ) ≤ Hw real ,π exp h ( Z ′ ) , so we conclude that

B.3 Proof of Unconstrained Upper Bound

Theorem 2. Assume that one of the two conditions is met:

For each h , π h exp plays actions uniformly at random for h ′ > h ,

Then with probability at least 1 -δ , Algorithm 1 returns a ̂ π such that V real ,π real ,⋆ 0 -V real , ̂ π 0 ≤ ϵ .

Proof. We consider each of the conditions above.

and for

Condition 1. First, note that by our assumption on π exp and applying Lemma B.4 with κ = ϵ 64 HV max and γ = d Hϵ sim , for any π and Z ′ ⊆ S × A , we have

for

By Lemma B.1 we then have that, with probability at least 1 -δ 6 ,

where the last inequality follows under our condition on T .

Condition 2. By Lemma A.3, we have that V real ,⋆ 0 -V real ,π sim ,⋆ 0 ≤ 2 H 2 ϵ sim . Thus, if ϵ sim ≤ ϵ/ 4 H 2 , we have V real ,⋆ 0 -V real ,π sim ,⋆ 0 ≤ ϵ/ 2 .

Concluding the Proof. By what we have shown, as long as one of our conditions is met, we will have that with probability at least 1 -δ/ 2 , there exists π ∈ { π ̂ f , π sim ,⋆ } such that V real ,⋆ 0 -V real ,π 0 ≤ ϵ/ 2 . Denote this policy as ˜ π .

Note that V real ,π 0 = E real ,π [ ∑ H -1 h =0 r h ] and that ∑ H -1 h =0 r h ∈ [0 , H ] almost surely. Consider playing π for T/ 4 episodes in real and let R i denote the total return of the i th episode. Let

By Hoeffding's inequality we have that, with probability at least 1 -δ/ 4 :

we have that | ̂ V π 0 -V real ,π 0 | ≤ ϵ/ 2 . Union bounding over this for both π ∈ { π ̂ f , π sim ,⋆ } , we have that with probability at least 1 -δ/ 2 :

The proof follows from a union bound and our condition on T (note that (B.9) is satisfied in both cases).

Thus, if

It follows that

Proof of Theorem 1. We first assume that ζ ≤ λ ⋆ min 4 d , for ζ the input regularization value given to Algorithm 5 by Algorithm 1, and Condition 1 of Theorem 2, and show that in this case A k ⋆ -2 is at most polynomial in problem parameters.

First, by Lemma C.7 we have that, under the assumption that ζ ≤ λ ⋆ min 4 d , the policy π h exp given by the uniform mixture of policies returned by Algorithm 5 will, with probability at least 1 -δ , satisfy λ min ( Λ s π h exp ,h ) ≥ λ ⋆ min 8 d under Assumption 3. Plugging ¯ λ min ← λ ⋆ min 8 d into Theorem 2, we have that ξ = 2 √ 16 dHA λ ⋆ min · ϵ sim . Now note that

It then suffices that we show 1 / log A 1 /ξ ≤ 1 ⇐⇒ 1 /A ≥ ξ . However, this is clearly met by our condition on ϵ sim . Thus, as long as

by Theorem 2 we have that ̂ π is ϵ -optimal.

Now, if ϵ sim ≤ ϵ/ 4 H 2 and T ≥ 16 H 2 log 4 /δ ϵ , we also have that ̂ π is ϵ -optimal, by Theorem 2. Thus, in the first case, we at most will require

to produce a policy that is ϵ -optimal, since otherwise we will be in the second case.

It remains to justify the assumption that ζ ≤ λ ⋆ min 4 d . Note that the condition of (4.1) is only required in the first case. Furthermore, if ϵ sim ≤ ϵ/ 4 H 2 we will be in the second case. Thus, in the first case, we will have

Rearranging this we obtain that, to be in the first case, we have

By our choice of ζ = 4 A 3 ϵ H , we then have that ζ ≤ λ ⋆ min 4 d . By Lemma C.7 and our choice of ζ , we have that Oracle 4.2 is called at most poly( d, H, ϵ -1 , log 1 δ ) times, and we call the oracle of Oracle 4.1 only H times. The result the follows from a union bound and rescaling δ .

B.4 Reducing the Version Space

As we noted, in general, given that we do not assume that ϕ r is unknown, log |F| could be significantly greater than the dimension. One might hope that, given access to M sim , we can reduce this dependence somewhat. We next show that this is possible given access to the following constrained regression oracle.

Oracle B.1 (Constrained Regression Oracle) . We assume access to a regression oracle such that, for any h and datasets { ( s t , a t , y t ) } T t =1 and { ( ˜ s t , ˜ a t , ˜ y t ) } ˜ T t =1 , we can compute:

While in general the oracle of Oracle B.1 cannot be reduced to the oracle of Oracle 4.1, under certain conditions on F this is possible. Given this oracle, we have the following result.

Theorem 3. Assume that ϵ sim ≤ λ ⋆ min 64 dHA 3 . Then if

with probability at least 1 -δ , Algorithm 4 returns policy ̂ π such that V real ,π real ,⋆ 0 -V real , ̂ π 0 ≤ ϵ , where

for α = ˜ O ( AdH 3 · log 2 log |F| /δ ϵ sim ) . Furthermore, the computation oracles of Oracle 4.2 and Oracle B.1 are called at most poly( d, A, H, ϵ -1 , log |F| δ ) times.

Theorem 3 shows that, rather than paying for the full complexity of F , we can pay only for the subset of F that is Bellman-consistent on M sim .

B.4.1 Algorithm and Proof

Algorithm 4 sim -toreal transfer via simulated exploration (SIM2EXPLORE)

1: input: tolerance ϵ , confidence δ , budget T , Q -value function class F

2: Π h exp ← LEARNEXPPOLICIES ( M sim , δ, 4 A 3 ϵ H , h ) for all h ∈ [ H ]

3: ι ←O (log 2 V max AdH ϵ )

4: for ℓ = 1 , 2 , . . . , ι do

6: Run exploration procedure of Lemma C.3 with β ℓ ← γ ℓ 20 V 2 max log 8 H |F| δ to obtain D ℓ sim

9: return ̂ π ← arg max ℓ ∈ [ ι ] ̂ V ̂ π ℓ 0

8: ̂ V ̂ π ℓ 0 ← average return running ̂ π ℓ in real T ℓ / 2 times

Theorem 4. Assume that one of the two conditions is met:

For each h , π h exp plays actions uniformly at random for h ′ > h ,

and for

and

ϵ sim ≤ ϵ/ 16 H 2 and

Then with probability at least 1 -δ , Algorithm 4 returns a policy ̂ π such that V real ,π real ,⋆ 0 -V real , ̂ π 0 ≤ ϵ .

Proof. We break the proof into two cases.

Case 1: ϵ sim ≥ ϵ/ 16 H 2 . Let ¯ ℓ = ⌊ log 2 ϵ -1 sim ⌋ and note that ¯ ℓ ≤ ι in this case and that this is a deterministic quantity. Further, note that γ ¯ ℓ ∈ [10 V 2 max ϵ 2 sim , 40 V 2 max ϵ 2 sim ] and ¯ ϵ ¯ ℓ ∈ [ ϵ sim , 2 ϵ sim ] . Note that by our assumption on π exp and applying Lemma B.4 with κ = ϵ 64 HV max and γ = d Hϵ sim , for any π and Z ′ ⊆ S × A , we have

By Lemma B.1, as long as β ¯ ℓ and γ ¯ ℓ satisfy

we have that with probability at least 1 -2 δ ,

where

However, since V 2 max ϵ sim ≤ 1 10 γ ¯ ℓ , and by our choice of β ¯ ℓ = γ ¯ ℓ 20 V 2 max log 8 H |F| δ , we see that (B.11) is met, so the conclusion holds. Note that, by Lemma C.5, we have that with probability at least 1 -δ :

where the second inclusion follows from our setting of β ¯ ℓ , and bounds on γ ¯ ℓ . Since T ℓ ← T/ 2 ι , it follows that if

then we have that V real ,π real ,⋆ 0 -V real , ̂ π ¯ ℓ 0 ≤ ϵ/ 2 .

Case 2: ϵ sim ≤ ϵ/ 16 H 2 . By Lemma B.5 and our choice of T ℓ sim , we have that with probability at least 1 -δ ,

By our choice of ι = O (log 2 V max AdH ϵ ) and since γ ι = 10 V 2 max (¯ ϵ ι ) 2 = 10 V 2 max · 2 -2 ι , we can bound V real ,⋆ 0 -V real , ̂ π ι 0 ≤ ϵ/ 2 .

for

Completing the Proof. In either case, we have that with probability at least 1 -δ , there exists some ̂ i ∈ [ ι ] such that V real ,⋆ 0 -V real , ̂ π ̂ i 0 ≤ ϵ/ 2 .

Note that V real ,π 0 = E real ,π [ ∑ H -1 h =0 r h ] and that ∑ H -1 h =0 r h ∈ [0 , H ] almost surely. Consider playing π for n episodes in real and let R i denote the total return of the i th episode. Let

By Hoeffding's inequality we have that, with probability at least 1 -δ/ι :

we have that | ̂ V π 0 -V real ,π 0 | ≤ ϵ/ 2 . However, as we run each π ∈ ̂ Π ℓ T ℓ / 2 = T/ 2 ι times, and in either case we assume T ≥ cιH 2 ϵ 2 · log 4 ι δ , this will be met. Union bounding over this for all ̂ π ℓ , we have that with probability at least 1 -δ :

The result then follows from a union bound and rescaling δ .

Proof of Theorem 3. The argument follows analogously to the proof of Theorem 1, but using Theorem 4 in place of Theorem 2. The bound on the number of oracle calls follows from Lemma C.3 and our choice of β ℓ .

Lemma B.5. With probability at least 1 -δ , for some ℓ , we have

Proof. By Lemma C.4 we have, with probability at least 1 -δ ,

where the second inequality holds by our setting of β ℓ .

We have

Thus, if

It follows that

By Lemma A.3, we can bound for parameter γ > 0 .

In Appendix C.1 we make the following assumption on the data generating process.

Assumption 5. Consider the dataset D sim = { ( ˜ s t 0 , ˜ a t 0 , ˜ r t 0 , . . . , ˜ s t H -1 , ˜ a t H -1 , ˜ r t H -1 ) } T sim t =1 . We assume that episode t in D sim was generated by playing an F t -1 -measurable policy ˜ π t exp , and denote π sim exp = unif( { ˜ π t exp } T sim t =1 ) .

We provide a specific instantiation of π sim exp in Appendix C.2. In Appendix C.3, we provide a procedure for learning a set of policies which induce full-rank covariates in sim , a crucial piece in obtaining good exploration performance in real .

C.1 Regularizing with Data from sim

Lemma C.1. With probability at least 1 -δ :

Proof. First, note that T real ̂ f h +1 ∈ F h by Assumption 4.

By Azuma-Hoeffding and a union bound, we have that, with probability at least 1 -δ , for each f, f ′ ∈ F h ,

Combining this with our bound on V sim ,⋆ -V sim , ̂ π

0 ℓ 0 gives the result.

C Learning in sim

In this section we provide additional supporting lemmas for our main results and in particular, we focus on linear in sim . In Appendix C.1 we provide several technical results critical to showing that sim can be utilized to restrict the version space, as is done in Theorem 4. In order to restrict the version space using sim , sufficiently rich data must be collected from sim , and in Appendix C.2 we provide results on this data collection. Finally, in Appendix C.3 we provide a procedure to compute the exploration policies in sim which we ultimately transfer to real .

In Appendices C.1 and C.2, we let hypothesis ˜ f and ̂ f be defined recursively as:

and ̂ f h ∈ F h some hypothesis satisfying

In particular, this implies that

To bound ( a ) , we note that

where the last inequality follows under Assumption 1. This gives that ( a ) ≤ 2 V 2 max ϵ 2 sim . To bound ( b ) , we apply Lemma 3 of [57], which gives that with probability at least 1 -δ ,

Combining these with a union bound gives the result.

Then with probability 1 -2 δ we have

Proof. By Azuma-Hoeffding, we have that with probability at least 1 -δ , for each f h , f ′ h ∈ F h ,

which implies in particular that, for any f h ∈ F h ,

We can write

By Lemma 3 of [57], with probability at least 1 -δ ,

We can therefore bound the final term as

Altogether then we have shown that, for any f h ∈ F h , with probability at least 1 -2 δ :

Thus, if then

The result follows from a union bound.

C.2 Data Collection with CoverTraj

Lemma C.3. Consider running the COVERTRAJ algorithm of [65] for each h ∈ [ H ] with parameters m ←⌈ log 2 1 /β ⌉ and γ i ← 2 i · β for some β ∈ [0 , 1] , and with REGMIN set to the policy optimization oracle of Oracle 4.2. Then this procedure collects

episodes, calls the policy optimization oracle at most T sim times, and produces covariates Λ h,i and sets X h,i such that, for each i ∈ [ m ] ,

and sup π w sim ,π h ( B d \ ∪ i m =1 X h,i ) ≤ β . Furthermore, we have

Proof. Instantiating REGMIN with the oracle of Oracle 4.2, we have that Definition 5.1 of [65] is met with C 1 = C 2 = 0 . Therefore, we have that at each stage i we collect exactly (using the precise form for K i given in the appendix of [65])

episodes. The result then follows by Theorem 3 of [65].

Lemma C.4. Consider running the procedure of Lemma C.3 to collect data. Then with probability at least 1 -2 δ , we have

Proof. By Lemma A.4:

Denote g ( z h ) := | ̂ f h ( s h , a h ) -T sim ̂ f h +1 ( s h , a h ) | and Λ h -1 = ∑ m i =1 Λ h,i + I , for Λ h,i collected as in Lemma C.3, and note that

We bound each of these terms separately. First, we have

where ( a ) follows from Lemma C.3 and since ∥ ϕ ∥ Λ -1 h -1 ≤ 1 always.

We turn now to bounding the first term. Note that

where ( a ) uses the fact that π h -1 ,t exp plays actions randomly at step h and ( b ) holds with probability at least 1 -δ by Lemma C.6. By Azuma-Hoeffding, we have with probability 1 -δ :

where the last inequality follows from the definition of ̂ f h .

Altogether then we have shown that, with probability at least 1 -2 δ :

Using that T sim ≤ 48 dH β 2 log 48 d β 2 as given in Lemma C.3, we can bound this as

The result follows.

Lemma C.5. Assume that

Then this implies that, with probability at least 1 -δ ,

Therefore,

Proof. We follow a similar argument as the proof of Lemma C.4. Denoting g ( z h ) := f h ( s h , a h ) -T sim f h +1 ( s h , a h ) , by the same calculation as (C.1) we have

and as in the proof of Lemma C.4, we can bound

and

By Azuma-Hoeffding, with probability at least 1 -δ we can then bound

where the last inequality follows by assumption, and where π sim exp = unif( { π h -1 ,t exp } T sim t =1 ) . Altogether then, for all π , we have

Using that T sim ≤ 48 dH β 2 log 48 d β 2 as given in Lemma C.3, we can bound this as

The result follows from some algebra.

Lemma C.6. With probability at least 1 -δ , for each h ∈ [ H ] simultaneously, we have

Proof. This follows from Lemma 3 of [57].

Algorithm 5 Learn Exploration Policies in M sim (LEARNEXPPOLICIES)

C.3 Learning Full-Rank Policies

We consider running the MINEIG algorithm (Algorithm 6) of [64] in sim . For a fixed h , we instantiate the setting of Appendix C of [64] with ψ ( τ ) = ϕ ( s h , a h ) ϕ ( s h , a h ) ⊤ , D = 1 , and A R the policy optimization oracle of Oracle 4.2 (and so C R = 0 ), and set N = 1 for MINEIG. We note that this algorithm is computationally efficient, given a policy optimization oracle.

Lemma C.7. For M←M sim , Algorithm 5 will call Oracle 4.2 at most ˜ O ( d ζ · log 1 δ + ζ -9 · log 3 / 2 1 δ ) times, and with probability at least 1 -δ , under Assumption 3 and if ζ ≤ λ ⋆ min 4 d , will return policies Π such that

and each π ∈ Π plays actions randomly for h ′ > h .

Proof. We first argue that, if ζ ≤ λ ⋆ min 4 d , then with probability at least 1 -δ , (C.2) holds. Let E denote the success event of each call to DYNAMICOED, and note that by our choice of δ j , we have P [ E ] ≥ 1 -δ/ 2 . Let j ⋆ denote the minimal value of j such that

By Lemma C.4 of [64] and if ζ ≤ λ ⋆ min 4 d , we then have that, on E , λ min ( Σ j ⋆ ) ≥ λ ⋆ min 4 d T j ⋆ , which implies that the termination criteria of Algorithm 5 will be met. By Lemma C.5 of [64], it follows that with probability at least 1 -δ/ 2 , we have λ min ( 1 | Π j ⋆ | ∑ π ∈ Π j ⋆ Λ s π,h ) ≥ λ ⋆ min 8 d (since T j ⋆ = | Π j ⋆ | ), the desired conclusion.

Assume that Algorithm 5 terminates for some j < j ⋆ . This implies that λ ⋆ min 4 d T j < 12544 d log 4+64 T j δ . However, in this case, we then have that

From Lemma C.5 of [64], it then follows that with probability at least 1 -δ/ 2 , we have λ min ( 1 | Π j | ∑ π ∈ Π j Λ s π,h ) ≥ λ ⋆ min 8 d .

It follows that, assuming T j is large enough that (C.3) is met, and we are in the case when ζ ≤ λ ⋆ min 4 d holds, then Algorithm 5 will terminate and return a set of policies satisfying (C.2), with probability at least 1 -δ . Note that T j = O (2 j ) . Given that Algorithm 5 does not terminate until j = O (log 2 ( d ζ · log 1 δ + ζ -9 · log 3 / 2 1 δ ) ≥ O (log 2 ( d 2 λ ⋆ min · log 1 δ + ζ -9 · log 3 / 2 1 δ )) , we will have that T j will be large enough that (C.3) is met, if ζ ≤ λ ⋆ min 4 d . The proof then follows since DYNAMICOED calls Oracle 4.2 at most T j times at round j , and the total sum of T j is bounded as ˜ O ( d ζ · log 1 δ + ζ -9 · log 3 / 2 1 δ ) by the maximum of j , and since the actions chosen by π ∈ Π for h ′ > h are irrelevant for the operation of DYNAMICOED, so they can be set to random.

D Lower Bound Proofs

D.1 Proof of Propositions 1, 3 and 4

Construction. Consider the following variation of the combination lock. We let the action space A = { 1 , 2 } , and assume there are two states, S = { s 1 , s 2 } , and horizon H . We start in state s 1 . The sim dynamics are given as:

We define two real instances, M 1 := M real , 1 and M 2 := M real , 2 , where for both we have:

for M 1 :

and for M 2 :

Note then that M 1 , M 2 , and sim only differ at step H -1 in state s 1 . Furthermore, it is easy to see that both M 1 and M 2 satisfy Assumption 1 with misspecification ϵ sim . It is easy to see that Assumption 2 holds as well with d = 4 since this is a tabular MDP, and furthermore Assumption 3 also holds with λ ⋆ min = 1 / 4 . We define the reward function as (note that this is deterministic, and the same for all instances):

and all other rewards are taken to be 0.

In sim , we see that the optimal policy always plays a 2 . In both M 1 and M 2 , the optimal policy plays a 1 for all h < H -1 , for M 1 plays a 1 at H -1 , and for M 2 plays a 2 at H -1 . Note that for both M 1 and M 2 , we have V ⋆ 0 = 1 / 2 + ϵ sim .

The most natural choice of F would be the set of all tabular Q -value functions, however, this set is infinite, and would require a covering argument to incorporate. For simplicity, consider F H the set of functions mapping to { 0 , 1 } , and F h the set of functions mapping to a finite set containing { 0 , 1 / 2 -ϵ sim , 1 / 2 + ϵ sim } ∪ { 1 / 2 + ϵ sim (1 / 2 -h ′ / 4 H ) } H h ′ =0 . Note that such a set satisfies Assumption 4 and we can construct it such that log |F| ≤ O ( H ) .

Lower Bound for Direct Policy Transfer (Proposition 3). We consider direct sim2real transfer with randomized exploration. In particular, as noted, the optimal policy in sim always plays a 2 , so we consider the ζ -greedy policy that at every state plays a 2 with probability 1 -ζ , and plays unif( { a 1 , a 2 } ) with probability ζ . Denote this policy as ˜ π . We then wish to lower bound:

after running our procedure for T episodes. Note that on M 1 , regardless of the actions ̂ π chooses in other states, we have

since the only way ̂ π can achieve a reward of 1 / 2 + ϵ sim is by playing a 1 in s 1 at step H -1 , and all other sequences of actions obtain a reward of at most 1 / 2 + ϵ sim / 2 . Similarly for M 2 we have

Using this, and replacing the max over i ∈ { 1 , 2 } with the average of them, we obtain

Since ̂ π H -1 ( a 1 | s 1 ) = 1 -̂ π H -1 ( a 2 | s 1 ) , we have

where TV denotes the total-variation distance, KL the KL-divergence, and the last inequality follows from Pinsker's inequality. We therefore have

Now note that, since M 1 and M 2 only differ at state s 1 and step H -1 , we have

where T H -1 ( s 1 , a i ) denotes the total number of visits to ( s 1 , a i ) at step H -1 after T episodes (see e.g. [59]). We have

where the last inequality holds as long as ϵ sim ≤ 1 / 6 . Note that the only way for a policy to reach s 1 at step H -1 is to play action a 1 H -1 consecutive times. Since ˜ π only plays a 1 at any given step with probability ζ/ 2 , it follows that the probability that ˜ π reaches s 1 at step H -1 on any given episode is only ( ζ/ 2) H -1 . Thus,

We thus have:

and we therefore have inf ̂ π sup i ∈{ 1 , 2 } E M i , ˜ π [ V M i ,⋆ 0 -V M i , ̂ π 0 ] ≥ ϵ sim / 8 unless

Lower Bound for ζ -Greedy Without sim (Proposition 1). In order to quantify the performance of a ζ -greedy algorithm, we must specify how it chooses ̂ f when it has not yet observed any samples from a given ( s, a, h ) . Following the lead of Theorem 2 of [11], to avoid an overly optimistic or pessimistic initialization, we assume that the replay buffer is initialized with a single sample from each ( s, a, h ) . Note that the conclusion would hold with other initializations, however, e.g. initializing ̂ f h ( s, a ) = 0 or randomly if we have no observations from ( s, a, h ) .

Assume that the observation from ( s 1 , a 1 , H -1) transitions to s 2 , which occurs with probability at least 1 / 4 . In this case, we then have that, for each h , ̂ f 0 h ( s 1 , a 2 ) ≥ ̂ f 0 h ( s 1 , a 1 ) . Thus, following the ζ -greedy policy, we have that π 0 h ( a 1 | s 1 ) ≤ 1 / 2 . Denote this event on E 0 . Furthermore, the only way we will have ̂ f 0 h ( s 1 , a 2 ) < ̂ f 0 h ( s 1 , a 1 ) is if we visit ( s 1 , a 1 , H -1) again and observe a transition to s 1 . For this to occur, however, we must play action a 1 H -1 times consecutively which, in this case, will occur with probability at most max { 1 / 2 , ζ/ 2 } H -1 ≤ 1 / 2 H -1 .

Following the argument in the direct policy transfer case, we have

where E M 1 [ T H -1 ( s 1 ) | E 0 ] is the expected number of visitations to ( s 1 , H -1) after T episodes of running the ζ -greedy policy. We can rewrite

Let E be the event that we have reached ( s 1 , H -1) in the first T rounds. Then,

By what we have just argued, we have P M 1 [ E | E 0 ] ≤ T · 1 2 H -1 , and E M 1 [ I { s H -1 = s 1 } | E c , E 0 ] ≤ 1 2 H -1 . Thus, E M 1 [ T H -1 ( s 1 ) | E 0 ] ≤ 2 T 2 2 H -1 . It follows that,

and we therefore have inf ̂ π sup i ∈{ 1 , 2 } E M i , ˜ π [ V M i ,⋆ 0 -V M i , ̂ π 0 ] ≥ ϵ sim / 32 unless

Upper Bound for Exploration Policy Transfer (Proposition 4). To obtain an upper bound for Algorithm 1, we can apply Theorem 1, so long as

Note that in our setting we have d = 4 , A = 2 , λ ⋆ min = 1 / 4 , so this condition reduces to ϵ sim ≤ 1 8192 H . Taking F to simply be the set of Q -functions defined above (so V max = H ), Theorem 1 then gives that with probability at least 1 -δ , Algorithm 1 learns an ϵ -optimal policy as long as T ≥ c · H 17 ϵ 8 · log H δ .

D.2 Proof of Proposition 5

We define three MDPs: M sim , and two possible real MDPs, M 1 := M real , 1 and M 2 := M real , 2 . In all cases we have states S = { s 1 , s 2 } , actions A = { a 1 , a 2 , a 3 , a 4 } , and H = 2 , and set the starting state to s 1 . We define

For both M 1 and M 2 , we have:

for M 1 , we have

and for M 2 ,

We take the reward to be 0 everywhere, except r 2 ( s 2 , a ) = 1 for all a .

Note that each of these can be represented as a linear MDP in d = 2 dimensions, so Assumption 2 holds. In particular, for M sim we can take:

For M 1 we can instead take:

M 2 follows similarly with the role of a 2 and a 3 flipped.

It is easy to see that Assumption 1 is met on this instance for both choices of M real . On M sim , the policy π exp which in every states plays action a 1 with probability 1/2 and action a 4 with probability 1/2 satisfies λ min ( E sim ,π exp [ ϕ s ( s h , a h ) ϕ s ( s h , a h ) ⊤ ]) ≥ 1 / 2 (which shows that Assumption 3 holds).

Note, however, that π exp does not play action a 2 or a 3 . As M 1 and M 2 differ only on a 2 and a 3 , playing π exp will not allow for M 1 and M 2 to be distinguished. As a 2 is the optimal action on M 1 and a 3 the optimal action on M 2 , it follows that playing π exp will not allow for the identification of the optimal policy on M 1 and M 2 . This can be formalized identically to Appendix D.1, yielding the stated result.

E Experimental Details

E.1 Didactic Tabular Example

Consider the following variation of the combination lock. We let the action space A = { 1 , 2 } , and assume there are two states, S = { s 1 , s 2 } , and horizon H . We start in state s 1 . The sim dynamics are given as:

and the real dynamics are given as:

Note that these only differ on ( s 1 , a 1 ) at h = H -1 , and we have ϵ sim = 1 / 2 . We define the reward function as (note that this is deterministic, and the same for both sim and real ):

and all other rewards are taken to be 0.

The intuition for this example is as follows. In both sim and real , the only way the agent can get reward is to either end up in state s 1 at step H , or to take action a 2 in state s 1 at any point. In sim , the probability of ending up in state s 1 at step H , even if the optimal sequence of actions to do this is taken, is only 1 / 4 , due to the final transition, and thus the average reward obtained by the policy which aims to end up in s 1 is only 1/4. In contrast, if we take action a 2 in s 1 , we will always collect reward of at least 3/8 (and the earlier we take action a 2 the more reward we collect, up to 1/2). Thus, in sim the optimal thing to do in s 1 is always to play a 2 . However, if we play a 2 even once, we will transition out of s 1 and never return, so there is no chance we will reach s 1 at step H .

In real , the transitions at the final step are flipped, so that now the probability of finishing in s 1 , if we take the optimal sequences of actions to do this, is 3/4, and the expected reward for this is then also 3/4. Since the reward for taking a 2 in s 1 does not change, and is bounded as 1/2, then in real the optimal policy is to seek to end up in s 1 at the final step.

The challenge with ending up in s 1 at the end is that it requires playing action a 1 at every step. In this sense it is then a classic combination lock instance, and randomized exploration will fail, requiring Ω(2 H ) episodes to reach the final state (since the probability of randomly taking a 1 at every state decreases exponentially with the horizon). Similarly, if we transfer the optimal policy from sim to real , it will never take action a 1 , so will never reach s 1 at the end, and if we transfer the optimal policy from sim with some random exploration, it will fail for the same reason random exploration from scratch fails.

However, note that we can transfer a policy from sim that is able to reach s 1 at the second-to-last step with probability 1, i.e. the policy that takes action a 1 at every step. Thus, if in sim we aim to learn exploration policies that can traverse the MDP, and we transfer these exploration policies, they will transfer, and will allow us to easily reach s 1 at the final step, and quickly determine that it is indeed the optimal thing in real .

We provide additional experimental results on this instance in Appendix E.1.

(a) Varying Number of States (b) Varying Number of Actions

Samples to Achieve Total Reward of 0.35

120

Samples to Achieve Total Reward of 0.35

180

800

115

170

700

160

110

600

150

500

105

140

400

100

130

300

95

120

200

110

100

90

×

0

20

Number of States

40

09

80

100

0

20

Number of Actions

40

09

80

100

10

15

20

Horizon

30

35

40

45

50

E.2 Practical Algorithm Details

The core of our work is to decouple the optimal policy training from exploration strategies in reinforcement learning fine-tuning. Specifically, we propose a framework that uses a set of diverse exploration policies to collect samples from the environment. These exploration policies are fixed while we run off policy RL updates on the collected samples to extract an optimal policy. Our theoretical derivation suggests that this decoupling can improve sample efficiency and overall learning performance.

Algorithm 6 sim2real transfer using OS for exploration and SAC for optimization

Our framework is complementary to (a) RL works on diversity or exploration that generate diverse policies and (b) off policy RL algorithms that optimize for policies. One can plug in (a) to extract a set of exploration policy from a simulator and use them for data collect in the real world but use (b) to optimize for the final policy. The design choice to use simulator to extract a set of exploration policies where each policy is not necessarily optimizing for the task at hand marks our distinction from previous works in (a) and (b).

We provide a practical instantiation of our framework using an approach inspired by One Solution is Not All You Need (OS) [30] to extract exploration policies and Soft Actor Critic (SAC) [30] to optimize for the optimal policy. We details the instantiation in Algorithm 6. OS trains a set of policy to optimize not only the task reward but also a discriminator reward where the discriminator encourages each policy to achieve different state. Unlike OS which carefully balances the task and exploration rewards to ensure all policies have a chance at solving the desired task, we emphasize only on having diverse policies. With a known sim2real gap, we posit that some sub-optimal policies that are not solving the task in the simulator is actually helpful for exploration in the real world, which allows us to simplify the balance between task and exploration. We uses standard off-shelf SAC update to optimize for the policy.

E.3 TychoEnv sim2sim Experiment Details

For the TychoEnv experiment we run a variant of Algorithm 6. We set n = 20 , and set the reward to r i t = (1 -α i ) r + α i r e where we vary α i from 0 to 0.5. While we use a sparse reward in M real , to speed up training in M sim we use a dense reward that penalizes the agent for its distance to the target. We train in M sim for 7M steps to obtain exploration policies. Rather than simply transferring the converged version of the exploration policies trained in M sim , we found it most effective to save the weights of the policies throughout training, and transfer all of these policies. As the majority of these policies do not collect any reward in M sim , we run an initial filtering stage where we identify several policies from this set that find reward (this can be seen in Figure 4 with the initial region of 0 reward). We then run SAC in M real , initialized from scratch, feeding in the data collected by these refined exploration policies into the replay buffer. We found it most effective to only inject data from the

0

1

2

3

Steps

×10 6

0

5

10

15

20

Reward

Exploration Policy Transfer (Ours)

DIAYN Exploration Policy Transfer

Direct Policy Transfer

From Scratch in Real

Figure 6: Additional results on Tycho, including baselines training from scratch in M real , and training exploration policies in M sim with reward as stated above but with α i = 1 (which is equivalent to simply training exploration policies with DIAYN [14]). As can be seen, while training from scratch in M real is able to learn, it learns at a much slower rate than exploration policy transfer, and achieves a much lower final value. Furthermore, training the exploration policies to maximize a mix of the task and diversity reward yields a substantial gain over simply training them to be diverse.

exploration policies in the replay buffer on episodes where they observe reward. We run vanilla SAC with UTD = 3 and target entropy of -3. We rely on the implementation of SAC from stable-baselines3 [51].

For direct policy transfer, we train a policy to convergence in M sim that solves the task (using SAC), and then transfer this single policy, otherwise following the same procedure as above.

In M real , our reward is chosen to have a value of 50 if the end effector makes contact with the ball, and otherwise 0. If the robot successfully makes contact with the ball the episode terminates. To generate a realistic transfer environment, we change the control frequency (doubling it in M real ) and the action bounds.

For both methods, we run the M sim training procedure 4 times, and then with each of these run it in M real twice. Error bars in our plot denote one standard error.

All experiments were run on two Nvidia V100 GPUs, and 32 Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz CPUs. Additional hyperparameters in given in Table 1.

We provide results on several additional baselines for the Tycho setup in Figure 6.

0.0

0.2

0.4

0.6

0.8

1.0

Steps

×10 4

80

60

40

20

0

Reward

Exploration Policy Transfer (Ours)

Direct Policy Transfer

From scratch

E.4 sim2sim Transfer on Franka Emika Panda Robot Arm

We next turn to the Franka Emika Panda robot arm [17], for which we use a realistic custom simulator built using the MuJoCo simulation engine [61]. We consider a hammering task, where the Franka arm holds a hammer, and the goal is to hammer a nail into the board (see Figure 8). Success is obtained when the nail is fully inserted. We simulate sim2real transfer by setting M real to be a version of the simulator with nail location and stiffness significantly beyond the range seen during training in M sim .

We compare exploration policy transfer with direct sim2real policy transfer. Unlike the Tycho experiment, where we trained policies from scratch in M real and simply used the policies trained in M sim to explore, here we initialize the task policy in M real to π sim ,⋆ , which we then finetune on the data collected in M real by running SAC. For direct sim2real transfer, we collect data in M real by simply rolling out π sim ,⋆ and feeding this data to the replay buffer of SAC. For exploration policy transfer, we train an ensemble of n = 10 exploration policies in M sim and run these policies in M real , again feeding this data to the replay buffer of SAC to finetune π sim ,⋆ . During training in M sim , we utilize domain randomization for both methods, randomizing nail stiffness, location, radius, mass, board size, and damping.

The results of this experiment are shown in Figure 9. We see that, while direct policy transfer is able to learn, it learns at a significantly slower rate than our exploration policy transfer approach, and achieves a much smaller final success rate.

E.5 Franka sim2real Experiment Details

We use Algorithm 6 to train a policy on the Franka robot with n = 15 .

The reward of the pushing task is given by:

where p goal is the desired position of the puck by the edge of the surface.

The network architecture of the actor and critic networks are identical, consisting of a 2-layer MLP, each of size 256 and ReLU activations.

We use stable-baselines3 [51] for our SAC implementation, using all of their default hyperparameters. The implemention of OS is built on top of this SAC implementation. Values of hyperparameters

0

1

2

3

4

5

Steps

×10 3

0.0

0.2

0.4

0.6

Success Rate

Exploration Policy Transfer (Ours)

Direct Policy Transfer

are shown in Table 2. Gaussian noise with mean 0 and standard deviation 0 . 005 meters is added in simulation to the position of the puck. Hyperparameters are identical between exploration policy transfer and direct transfer methods.

For finetuning in real, we start off by sampling exclusively from the buffer used during simulation. Then, as finetuning proceeds, we gradually start taking more samples from the real buffer, with the proportion of samples taken from sim equal to 1 -s/ 3000 , where s is the current number of steps. After 3000 steps, all samples are taken from the real buffer.

Experiments were run using a standard Nvidia RTX 4090 GPU. Training in simulation takes about 3 hours, while finetuning was ran for about 90 minutes.

In Figure 7, we provide results on this setup running the additional baseline of training a policy from scratch in real. As can be seen, this is significantly worse than either transfer method.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We validate all our claims with theoretical results and experiments.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Please see Discussion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All results are precisely proved in the supplemental, and all assumptions clearly stated.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: To the extent possible, given that we are working with real-world systems, we have described our setup and implementation details.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We have not currently released our code but hope to in the future.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have stated all parameters and algorithm details to the best of our knowledge (please see Experimental Details section in supplemental).

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: For all our experimental results, we provide error bars corresponding to 1 standard error.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Please see Experimental Details section in appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This paper does not violate any ethical guidelines.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This is work is related to the advancement of our fundamental understanding of machine learning. As such, we do not believe there are any direct societal impacts from this work.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We are not releasing high-risk models.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have cited the creators of the code used in this project.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This work does not involve human subjects research.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This work does not involve human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors contribute a mathematical proof showing that transferring a \"exploration strategy\" in sim2real scenarios as opposed to transfer the learned policy result in an exponential improvement on samples needed to learn the real task.\n\nSTRENGTHS:\n- While not necessarily a fundamentally new idea (the transfer learning and sim2real areas have been long avoiding the naive transfer of policies and biasing exploration has always been one of the chosen strategies to perform transfer), the theoretical results presented confirm and fundament this idea that has been informally explored by the literature.\n\n- The authors present a practical, easy to understand, exploration method and show that it overperforms greatly a simple policy transfer method in a somewhat complex sim2real robotic evaluation.\n\n- The topic explored is not only adherent to neurips but also very timely in supporting more targeted efforts to apply RL to novel practical domains.\n\nWEAKNESSES:\n- I am not sure why a new basic \"sim2real\" formalization was used to in this paper when it already exists. The problem explored in this paper is completely equivalent to Multi-Fidelity MDPs (described in {1}), in fact, MF-MDPs even have something not incorporated in the description provided that is the cost of sampling in each fidelity. The MF-MDP paper was not even cited so I assume this happened because the authors have not read the paper. I suggest the paper is rewritten to describe the problem as an MF-MDP for standardization (there shouldn't be any impact to the conclusions or contribution fro the paper), but even if not rewritten at the very least this paper has to be cited and incorporated to the related works:\n\n{1} - Silva, F. L., et al. Toward multi-fidelity reinforcement learning for symbolic optimization. ALA workshop @ AAMAS, 2023.\n\n- Another thing that made me be in the fence regarding my score to this paper is the lack of baselines for the experimental evaluation. There has been many papers exploring a similar idea in the past, so I feel like they should have been incorporated to the experimental evaluation (Cuttler {2} comes to mind immediately, but you also have other newer options)..\n\n{2} Cutler, Mark, Thomas J. Walsh, and Jonathan P. How. \"Reinforcement learning with multi-fidelity simulators.\" 2014 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2014.\n\n- A limitation both in the theoretical and practical side of the paper is that it is assumed that the transfer will happen \"one shot\", neglecting the fact that either the simulator could be modified to get closer to the real world {3} and the learning process in the simulator repeated, or the approach could already account that the simulator and real world will be different and adapt for that (e.g. {2} and {3}). The present paradigm presented by the authors to \"learn from the simulation once and never get back to it\" is at least an incomplete picture of all that could be done.\n\n{3} - Hanna, Josiah, and Peter Stone. \"Grounded action transformation for robot learning in simulation.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 31. No. 1. 2017.\n\nQUESTIONS:\n- Why is it assumed that the transfer HAS to be made one-shot (transferred completely in the beginning of the learning process in the \"real environment\")? Wouldn't it be better to have the exploration transfer strategy work as a \"loop\" where the exploration could be revised according to how the real world differs to the simulator?\n\n- Why weren't MF-MDPs used to describe the problem?\n\n- it is stated that the conditions for us being able to find a policy that solve the problem in the simulation is that ϵ ≥ 2H^2ϵ_sim. What is the meaning of ϵ here? Finding conditions for a direct policy transfer to work would also be an important contribution, specially if there is some way to test for those conditions with some samples from both the real world and simulator.\n\nLIMITATIONS:\nBasic research, no potential negative societal impact to explore.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-19"}

comments: SUMMARY:
The authors propose a method where, instead of directly transferring a trained policy from a simulator to the real world, exploratory policies are learned in the simulator and transferred. This approach aims to enable efficient exploration in the real world, particularly in low-rank MDP settings.

STRENGTHS:
1. The idea of transferring exploratory policies instead of directly transferring policies trained in simulator is interesting and can potentially help sim-to-real transfer.

WEAKNESSES:
1. The approach relies on several assumptions (e.g., low-rank MDPs, specific access to simulators and oracles) that may not hold in all real-world scenarios.

2. While the method shows promise in specific settings, its generality to a wider range of RL tasks and environments remains to be demonstrated.

3. Limited comparison with other state-of-the-art methods, especially those addressing the sim-to-real gap through domain randomization or adaptation.

QUESTIONS:
1. How robust is the method to violations of the low-rank MDP assumption?

2. Can the approach be extended to more complex MDP settings?

3. How does the method scale with the complexity of the task and the size of the state/action spaces?

LIMITATIONS:
As in weaknesses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper shows that for transferring with a large sim-to-real gap, exploration policies have improved capabilities compared to optimally pre-trained policies, which tend to overfit the simulation up to a point where exploration is not sufficient to adapt to the changed circumstances.\n\nSTRENGTHS:\nOverall, the paper is easy to follow and theoretically sound. It includes theoretical deliberations and empirical results with a convincing experiment design. I expect the considered perspective on sim-to-real shift to positively impact research in this area.\n\nWEAKNESSES:\nOverall, providing further examples for the theoretical elaborations could help comprehensibility (e.g., Proposition 3). Further minor issues include: \n\n- Definition 3.2: PAC is not defined, what is $\\delta$ used for? \n- In Proposition 1: Is the success of the transfer dependent on the horizon of M? \n- In Proposition 2: What does $\\Omega$ refer to? \n- What is the difference between $\\epsilon$ and $\\epsilon_{sim}$?\n- Why is the transition probability $P$ defined to be dependent on the horizon H? \n\nAlso, I am missing a thorough definition of the Low-Rank MDP, which seems central to the theoretical elaboration. \n\nEven though the experimental setup is convincing, I am missing comparisons directly training $\\pi$ in the real task as a baseline. Also, to better illustrate and link the aforementioned theoretical elaboration, providing the distance between the considered benchmark tasks would have been interesting. Furthermore, more detailed comparisons (both experimental and theoretical) to advanced pre-training approaches like APT (Liu et al., 2021) or DIAYN (Eysenbach et al., 2018) would further enhance empirical credibility.\n\nQUESTIONS:\nWhat are the implications of using a low-rank MDP?\n\nIf the simulation is considered to be free, wouldn't learning $\\Pi$ optimal policies improve the transfer (similar to using a pool of exploration policies)?  \n\nRegarding the transfer described in Alg. 1, are all exploration policies updated, or only the best-performing? \n\nIf we were to consider even larger shifts, is there a point at which exploration policies no longer provide benefits, and learning from scratch is the most sufficient approach? \n\nWhen considering the transfer from simulation to the real world, pre-training is often motivated by the need to ensure a certain level of safety for execution in the real world. How does this concern affect your approach?\n\nLIMITATIONS:\nLimitations have been addressed by the authors but could have been extended.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}