Jump-Start Reinforcement Learning
Authors: Ikechukwu Uchendu
Ted Xiao
Yao Lu
Banghua Zhu
Mengyuan Yan
Joséphine Simon
Matthew Bennice
Chuyuan Fu
Cong Ma
Jiantao Jiao
Sergey Levine
Karol Hausman
Venue: Submitted to ICLR 2023 (2023)
Keywords: reinforcement learning
offline reinforcement learning
fine-tuning

Abstract: Reinforcement learning (RL) provides a theoretical framework for continuously improving an agent’s behavior via trial and error. However, efficiently learning policies from scratch can be very difficult, particularly for tasks that present exploration challenges. In such settings, it might be desirable to initialize RL with an existing policy, offline data, or demonstrations. However, naively performing such initialization in RL often works poorly, especially for value-based methods. In this paper, we present a meta algorithm that can use offline data, demonstrations, or a pre-existing policy to initialize an RL policy, and is compatible with any RL approach. In particular, we propose Jump-Start Reinforcement Learning (JSRL), an algorithm that employs two policies to solve tasks: a guide-policy, and an exploration-policy. By using the guide-policy to form a curriculum of starting states for the exploration-policy, we are able to efficiently improve performance on a set of simulated robotic tasks. We show via experiments that it is able to significantly outperform existing imitation and reinforcement learning algorithms, particularly in the small-data regime. In addition, we provide an upper bound on the sample complexity of JSRL and show that with the help of a guide-policy, one can improve the sample complexity for non-optimism exploration methods from exponential in horizon to polynomial.

Under review as a conference paper at ICLR 2023
JUMP-START REINFORCEMENT LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
Reinforcement learning (RL) provides a theoretical framework for continuously improving an agent's behavior via trial and error. However, efficiently learning policies from scratch can be very difficult, particularly for tasks that present ex- ploration challenges. In such settings, might be desirable initialize RI. with existing policy, offline data, demonstrations. However, naively performing such initialization in RL often works poorly, especially for value-based methods. offline data, demonstrations,
In this paper, we present meta algorithm that or a pre-existing policy to initialize an RL policy, and is compatible with any RL approach. In particular, we propose Jump-Start Reinforcement Learning (JSRL), an algorithm that employs two policies to solve tasks: a guide-policy, and an form a curriculum of starting we are able to efficiently improve performance
exploration-policy. By using the guide-policy states for the exploration-polic on a set of simulated robotic tasks. We show via experiments that is able significantly outperform existing imitation and reinforcement learning algorithms, particularly in the small-data regime. addition, we provide an upper bound on the sample complexity of JSRL and show that with the help of guide-policy, one can improve the sample complexity for non-optimism exploration methods from exponential in horizon to polynomial.
INTRODUCTION
promising aspect of reinforcement learning (RL) is the ability of policy to iteratively improve via trial and eтror. Often, however, the most difficult part of this process is the very beginning, where a policy that is learning without any prior data needs to randomly encounter rewards to further improve. A common way to side-step this exploration issue is to aid the policy with prior knowledge. One source of prior knowledge might come the form of prior policy, which can provide some initial guidance in collecting data with non-zero rewards, but which is not by itself fully optimal. Such policies could be obtained from demonstration data (e.g., via behavioral cloning), from sub-optimal prior data (e.g., via offline RL), or even simply via manual engineering. In the case where this prior policy is itself parameterized as a function approximator, it could serve to simply initialize a policy gradient method. However, sample-efficient algorithms based value functions difficult to bootstrap in this way. As observed in prior work (Peng et al.] 2019 Nair et al.) 2020) Kostrikov et al. [022]] Lu et al. 2021), value functions require both good and bad data to initialize successfully, and the mere availability of a starting policy does not by itself readily provide an initial value function of comparable performance. This leads the question pose in this work: how can we bootstrap value-basee based RL algorithm with prior policy that attains reasonable but sub-optimal performance?
notoriously
The main insight that we leverage to address this problem is that we can bootstrap any RL algorithm by gradually "rolling in" with the prior policy, which we refer to as the guide-policy. In particular, the guide-policy provides curriculum of starting states for the RL exploration simplifies the exploration problem and allows for fast learning. As the exploration-pole pooiicy improves, the effect of the guide-policy is diminished, leading to an RL-only policy that is capable of further autonomous improvement. Our approach is generic, as it can be applied to any RL method that explores its environment for policy improvement, though work. The only requirements of our method are that the guide-policy can select actions based on observations of the environment, and its performance is reasonable (i.e., better than random
which significantly
focus value-based methods in this

---

Under review as a conference paper at ICLR 2023
Vanilla
JSRL
Guide- Pullcy
Explonat on an-Pullcy
dagee
Cantezin Stage
E E
E
TO
F
Figure 1: We study how to efficiently bootstrap value-based RL algorithms given access to prior policy. In vanilla RL (left), the agent explores randomly from the initial state until it encounters reward (gold star). JSRL (right), leverages a guide-policy (dashed blue line) that takes the agent closer to the reward. After the guide-policy finishes, the exploration-pole poolicy (solid orange line) continues acting in the environment. As the exploration-policy improves, the influence of the policy diminishes, resulting in a learning curriculum for bootstrapping RL.
policy). Since the guide-policy significantly speeds up the early phases of RI, we call this approach Jump-Start Reinforcement Learning (JSRL). We provide an overview diagram of JSRL in Fig. [
JSRL utilize any form of prior policy to accelerate RL. It is also compatible with RL algorithms that involve rolling out policy to explore an environment. Thus, JSRL can easily be combined with existing offline and/or online RL methods. In addition, we provide theoretical justification of ISRL by deriving an upper bound on its sample complexity compared to RL alternatives. Finally, we demonstrate that JSRL outperforms previously proposed imitation and reinforcement learning approaches on a set of benchmark tasks as well as more challenging vision-based robotic problems.
2 RELATED WORK
Imitation learning combined with reinforcement learning (IL+RL).. Several previous works on leveraging prior policy to initialize RL focus on doing so by combining imitation learning and RL. Some methods treat RL sequence modelling problem and train an autoregressive model using offline data Zheng et al. (2022) Janner et al. (2021); Chen et al. (2021). One well-studied class of approaches initializes iee policy search methods with policies trained via behavioral cloning [Schaal et al.. (1997); Kober et al. (2010); Rajeswaran et al. (2017). This is effective strategy for initializing policy search methods, but is generally ineffective with actor-critic or value-based methods, where the critic also needs be initialized (Nair et al.) 2020), have been proposed to include prior data in the replay buffer for a value- based approach (Nair et al. 2018 Vecerik et al. 2018). but this requires prior data rather than just a prior policy. More recent approaches improve this strategy by using offline RL Kumar et al. (2020); Nair et al.) (2020); Lu al. 202T) to pre-train on prior data, then finetune. We compare to such methods, showing that our approach not only makes weaker assumptions (requiring only a policy rather than dataset), but also performs comparably or better.
also illustrate in Section 3 Methods
Curriculum learning and exact state resets for RL. Many prior works have investigated efficient exploration strategies RL that are based on starting exploration from specific states. Commonly, these works assume the ability to reset to arbitrary states in simulation (Salimans & Chen 2018). Some methods uniformly sample states from demonstrations as start states (Hosu & Rebedea 2016 Peng et al.] 2018 Nair et al. 2018), while others generate curriculas of start states. The latter includes methods that start at the goal state and iteratively expand the start state distribution, assuming reversible dynamics (Florensa al. 2017 McAleer al. 2019) access to approximate dynamics model (Ivanovic et al. (2019). Other approaches generate the curriculum from demonstration states (Resnick al. 2018) from online exploration (Ecoffet et al. 2021). In contrast, our method does not control the exact starting state distribution, but instead utilizes the implicit distribution
"A project wehpage is available at athstpp://jompstttttr..ttttt
2

---

Under review as a conference paper at ICLR 2023
naturally arising from rolling out the guide-policy This broadens the distribution of start states compared to exact resets along a narrow set of demonstrations, making the learning process more robust. In addition, approach could be extended to the real world, where resetting to state in the environment is impossible.
Provably efficient exploration techniques. Online exploration in RL has been well studied in theory (Osband& & Van Roy 2014] Jin et al. 20188] Zhang et al.) 2020b] Xie et al., [2021 Zanette et al.) 2020 Jin al. (020). The proposed methods either rely on the estimation of confidence intervals (e.g. UCB, Thompson sampling), which is hard to approximate and implement when combined with neural networks, or suffer from exponential sample complexity in the worst-case. In this paper, we leverage a pre- pre-trained guide-policy to design an algorithm that is more sample-efficien than these approaches while being easy to implement in practice.
"Rolling in" policies. Using pre-existing policy (or policies) to initialize RL and improve exploration has been studied in past literature. Some works use an ensemble of roll-in policies or value functions to refine exploration Jiang et al.) (2017); Agarwal et al. (2020). With policy that models the environment's 's dynamics, it is possible to look ahead to guide the training policy towards useful actions (Lin 1992p. Similar to our work, an approach from Smart& & Pack Kaelbling (2002) rolls out fixed controller to provide bootstrap data for a policy's 's value function. However, this method does not mix the prior policy and the learned policy, but only uses the prior policy for data collection. We use multi-stage curriculum to gradually reduce the contribution of the prior policy during training, which allows for on-policy experience for the learned policy. Our method is also conceptually related to DAgger (Ross & Bagnell 20100), which also bridges distributional shift by rolling in with one policy and then obtaining labels from a human expert, but DAgger intended for imitation learning and rolls in the learned policy, while our method addresses RL and rolls in with a sub-optimal guide-policy.
3 PRELIMINARIES
We define Markov decision process M = P., H), where S and A are state and action spaces, P: : S x A S + R,, is a state-transition probability function, R: S x A + I is reward function, Po: S + R+ is an initial state distribution, y is discount factor, and H is the task horizon. Our goal is to effectively utilize a prior policy of any form in value-based reinforcement learning (RL). The goal of RL is to find a policy T(a s) that maximizes the expected discounted reward over trajectories, т, induced by the policy: E [R(T)] where 8o ~ Pn, 81+1 ~ P(-Ist; 0t) and ae ~ ((-(s)). To solve this maximization problem, value-based RL methods take advantage of state or state-action value functions (Q-function) (T. a), which can be learned using approximate dynamic programming approaches. The Q-function, Q' (s, a), represents the discounted returns when starting from state s and action a, followed by the actions produced by the policy π.
In order to leverage prior data in value-based RL and continue fine-tuning, researchers commonly use
Boetstrapping antmmaze-le eetorazz-leggeddveeeee-
10L
various offline RL meth-
ods (Kostrikov et al. 1202
Kumar et al.] 2020) Nair
Tre Cuning
Missuuing
et al.] 2020 Lu al. 2021) that often rely on pre-trained, regularized Q- functions that can be fur- ther improved using online data. In the case where a pre-trained Q-function is not available and we only have access to prior policy, value-based RL methods struggle to effectively incor- porate that information depicted in Fig. 2 In this experiment, we train actor-critic method up to step 0, then we start from fresh Q-function and
-0.25
-0.50
Figure 2: Naäve policy initialization. We pre- treitrain policy to medium performance (depicted by negative steps), then use this policy to ini- tialize actor-critic fine-tuning (starting from step 0), while initializing the critic randomly. Actor performance decays, as the untrained critic provides poor learning signal, causing the good initial policy to be forgotten. In Figures 7 and 8 we repeat this experiment but allow the randomly initialized critic to "warm up" before fine-tuning.
3

---

Under review as a conference paper at ICLR 2023
continue with the pre-trained actor, simulating the case where we only have access to prior policy. This is the setting that we are concerned with in this work.
4 JUMP-STARE REINFORCEMENT LEARNING
In this section, we describe our method, Sump-Start Reinforcement Learning (ISRL), that we use to initialize value-based RL algorithms with prior policy of any form. We first describe the intuition behind our method then lay out a detailed algorithm along with theoretical analysis.
4.1 ROLLING IN WITH TWo POLICIES
We assume access to a fixed prior policy that we refer to as the "guide-policy" (a|s), which we leverage initialize an RL algorithm. It is important to note that we do not assume any particular form of n it could be learned with imitation learning, RL, or it could be manually scripted. We will refer to the RL policy that is being learned via trial and error as the since, as it is commonly done in RL literature, this is the policy that is used for exploration as well as online improvement. The only requirement for лt is that it an RL policy that can adapt with online experience. Our approach and the set of assumptions is generic in that it can handle any downstream RL method that rolls out policy for exploring an environment, though we focus on the case where x" is learned via a aalue-based RL algorithm.
pmplooationnpoliyy x² (als),
The main idea behind our method is leverage the two policies, T and H", executed sequentially to learn tasks more efficiently. During the initial phases of training, T is significantly better than the untrained policy T², so we would like to collect data using л However, this data out of distribution for π", since exploring with He will visit different states. Therefore, we would like to gradually transition data collection away from T and toward x². Intuitively, we would like to use o to get the agent into "good" states, and then let x take over and explore from those states. As gets better and better, x" should take over earlier and carlier, until all data is being collected by Tf and there is no more distributional shift. We can employ different switching strategies to switch from T to n², but the most direct curriculum simply switches from T to πe at some time step h, where h is initialized to the full task horizon and gradually decreases over the course of training. This naturally provides a curriculum for n². At each curriculum stage, x needs to master a small part of the state-space that is required reach the states covered by the previous curriculum stage.
4.2 ALGGORITHM
We provide detailed description of JSRL in Algorithm I Given an RL task with horizon H. we first choose a sequence of initial guide-steps to which we roll out our guide-policy, (H, H2. H where H, E 21, 2....... HI} denotes the number of steps that the guide-policy at the 1 iteration acts for. Let denote the iterator over such a sequence of initial guide-steps. At the beginning of each training episode, we roll out 7 for h steps. then τ" continues acting in the environment for the additional H steps until the task horizon H is reached. We can write the combination of the two policies the combined policy, where Ti.h T and 1:+ л². After we roll out r to collect online data, we use the new data to update our exploration-polee policy лt and combined policy æ by calling standard training procedure TRAINPOLICY. The TRAINPOLICY updates both the Q function and the corresponding evaluation policy. For example, the training procedure may be updating the exploration-polic, via Deep Q-Network (Mnih et al. 2013) with e-greedy as the exploration technique (i.e. (a)s) = -1- e if a = arg max,, Q(s, a) and A/A otherwise). The new combined policy is then evaluated over the course of training using a standard evaluation procedure EvALUATEEPLLICY Once the performance of the combined policy л reaches a threshold, B, we continue the for loop with the next guide step h.
While any guide-step sequence could be used with JSRL, we focus on two specific strategies for determining guide-step sequences: curriculum and random-switching With the curriculum strategy, we start with large guide-step (ie. Hi H) and use policy evaluations of the combined policy to progressively decrease H,, as T" improves. Intuitively, this means that we train our policy in a backward manner by first rolling out T to the last guide-step and then exploring with π². and then rolling out 78 to the second to last guide-step and exploring with x", and so on. With the random- switching strategy, we sample each h uniformly and independently from the set ( (H, "2, H.).
4

---

Under review as a conference paper at ICLR 2023
In the rest of the paper, refer to the curriculum variant JSRL, and the random switching variant as 1SRL--aando..
Algorithm Jump-Start Reinforcement Learning
1: Input: guide-policy x", performance threshold (9. task horizon H, a sequence of initial guide-steps H1, H2,... .H.. where H; é {1,2, 2.... H) for all i
2: Initialize exploration-poliey from scratch or with the guide-policy of +- x². Initialize Q-function Q and dataset D f- Ø9
3: for current guide step h H1, H2. H., do
4: Set the statsstttonay policy TiAh = x Th41: =
5: Roll out the policy æ to get trajectory {(s), (i, (SH, rH)): Append the trajectory to the
dataset 'D.
6: TT, Q + TRAINPOLICY(Tf,, Q.
7: if (VALUATHPOLICC(T)) 2 then
Continue
9: end if
10:
end for
4.3 THEORETICAL ANALYSIS
In this section, we provide theoretical analysis of JSRL, showing that the roll-in data collection strategy that we propose provably attains polynomial sample complexity. The sample complexity refers to the number of samples required by the algorithm to learn policy with small suboptimality, where we define the suboptimality for a policy π as Eg<po (V*(s) (s) In particular, we aim to answer two questions: Why is JSRL better than other exploration algorithms which start exploration from scratch? Under which conditions does the guide-policy provably improve exploration? To answer these questions. we study upper and lower bounds for the sample complexity of exploration algorithms. We first provide a lower bound showing that simple can--ottmiim-bassee algorithms like e-greedy suffer from sample complexity that is exponential in the horizon. Then, we show that with the help of guide-policy with good coverage of important states, the JSRL algorithm with c-greedy as the exploration strategy can achieve polynomial sample complexity.
exploration
We focus on comparing JSRL with standard non-optimism- based exploration methods, e.g. e- greedy (Langford Zhang] 2007) and FALCON: (Simchi-LLeii Xu 2020). Although the optimism- based RI. algorithms like UCB (Jin et al. 2018) and Thompson sampling (Ouyang et al.) (017) turn out to be efficient strategies for exploration from scratch, they all require uncertainty quantification, which can be hard for vision-based RL tasks with neural network parameterization. Note that the cross entropy method used in the vision- RL framework Qt-Opt (Kalashnikov et al. 2018) is also non-optimism- based method. In particular, it can be viewed as variant of e-greedy algorithm in continuous action space, with the Gaussian distribution as the exploration distribution.
We first show that without the help of guide-policy, the non-optimism--aase based method usually suffers from a sample complexity that is exponential in horizon for episodic MDP. We adapt the combination lock example in Koenig & Simmons (1993) show the hardness of exploration from scratch for non-optimism-belt based methods.
Theorem 4.1 (Koenig & Simmons (993)). For 0-initialized c-greedy, there exists an MDP instance such that one has to suffer from a sample complexity that is exponential in total horizon H in order to find policy that has suboptimality smaller than 0.5.
We include the construction of combination lock MDP and the proof in Appendix A.4.2 for com- pleteness. This lower bound also applies to any other non-optimism-bart based exploration algorithm which explores uniformly when the estimated Q for all actions are 0. As concrete example, this also shows that iteratively running FALCON+ Simchi-Levi & Xu (2020) suffers from exponential sample complexity. With the above lower bound, we are ready show the upper bound for JSRL under certain assumptions on the quide-policy. In particular, we assume that the guide-polics л is able to cover good states that are visited by the optimal policy under some feature representation:
Assumption 4.2 (Quality of guide-policy T.). Let (s) be the marginalized state occupancy distribution when we follow policy Assume that the state is parametrized by some feature mapping S + Rd such that for any policy म, Q (s. a) and x(s) depend on s only through ((s), and that in the feature space, the guide-policy T cover the states visited by the optimal policy:
5

---

Under review as a conference paper at ICLR 2023
de. (o(s)) s.h di (o(s)) SC
sup
We provide formal definition of the marginalized state occupancy distribution in Appendix A.4 In other words, the guide-policy visits only all good states in the feature space. A policy that satisfies Assumption 4.2 be far from optimal due to wrong choice of actions in each step. Assumption 4.2 is also much weaker than the single policy concentratability coefficient assumption, which requires the guide-policy visits all good state and action pairs and is a standard assumption in the literature in offfine learning Rashidinejad et al. (2021); ); Xie et al. (2021). The ratio in Assumption 4.2 is also sometimes referred to as the distribution mismatch coefficient in the literature of policy gradient methods Agarwal et al. (2021).
We show via the following theorem that given Assumption 4.2 a simplified JSRL algorithm which only explores at current guide step h + gives good performance guarantees for both tabular MDP and MDP with general function approximation. The simplified JSRL algorithm coincides with the Policy Search by Dynamic Programming (PSDP) algorithm in Bagnell al. (2003), although our method is mainly motivated by the problem of fine-tuning and efficient exploration in value based methods, while PSDP focuses on policy- methods.
Theorem 4.3 (Informal). Under Assumption 4.2 and an appropriate choice of TrainPolicy and EvaluatePolicy, JSRL Algorithm [ tabular MDP; and hear-optimal bound up to factor of C poly(H) ) MDP with general,unctionn function approximation.
guarantees suboptimality of /(C //2////// for
To achieve polynomial bound for JSRL, it suffices to take TrainPolicy as e-greedy. This is in sharp contrast Theorem 4.1] where e-greedy suffers from exponential sample complexity. As discussed in the related work section, although polynomial and even near-optimal bound can be achieved by many optimism- based methods Jin et al. (2188;; p Ouyang al. (2017), the JSRL, algorithm does not require constructing a bonus function for uncertainty quantification, and can be implemented easily based on naïve e-greedy methods. Furthermore, although we focus on analyzing the simplified JSRL which only updates policy π at current guide steps h + 1. in practice we run JSRL algorithm as in Algorithm| T. which updates all policies after step h + 1. This is the main difference between our proposed algorithm and PSDP. For a formal statement and more discussion related to Theorem 4.3 please refer to Appendix A.4.3
EXPERIMENTS
In our experimental evaluation, we study the following questions: (1) How does JSRL compare with competitive IL+RL basclines?
(2) Does JSRL scale to complex vision-based robotic manipulation tasks? (3) How sensitive is JSRL to the quality of the guide-policy? (4) How important is the curriculum component of JSRL? (5) Does JSRL. generalize? That is, can a guide-policy still be useful if it was pre-trained on related task?
5.1 COMPARISON WITH IL+RL BASELINES
To study how JSRL compares with competitive IL+RL methods, utilize the D4RL. (Fu et al. 2020> benchmark tasks, which vary in task complexity and offline dataset quality. We focus on the most challenging D4RL tasks: Ant Maze and Adroit manipulation. We consider a common setting where the agent first trains on an offline dataset (Im transitions for Ant Maze, 100k transitions for Adroit) and then runs online fine-tuning for 1m steps. We compare against algorithms designed specifically for this setting, which include AWAC (Nair et al. 2020), IQL (Kostrikov et al. [022)). CQL (Kumar et al. 2020). and behavior cloning (BC). While JSRL can be used in combination with any initial guide-policy or fine-tuning algorithm, we show the combination of JSRL. with the strongest baseline, IQI. IQI. (Implicit Q-Learning) is an actor-critic method that completely avoids estimating the values of actions that are not seen in the offline dataset. This is recent state-of-the-art method for the IL+RL setting we consider. In Table I we see that across the Ant Maze environments and Adroit environments, IQL+JSRL is able to successfully fine-tune given initial offline dataset, and is competitive with baselines. We will come back for further analysis of Table I when discussing the sensitivity the size of the dataset.
'The AWAC, BC, and CQL. performance scores for D4RL. are taken from Kostrikow et al. 2021 which only evaluated settings with full-sized datasets.

---

Under review as a conference paper at ICLR 2023
Insiance Unespeny Desoesst
Insane Craging DemrArdros
Nunden
00
2300
28391
2530
2831
00.00
A620
bresseng
Crossing Domorstictios
05 01-00e1i22
on
Figure 3: We evaluate the importance of guide-policy quality for JSRL on Instance Grasping, the most challenging task we consider. By limiting the initial demonstrations, JSRL is less sensitive to limitations of initial demonstrations compared to baselines, especially in the small-data regime. For each of these initial demonstration settings, we find that Qt-Opt+JSRL more sample efficient than SeppttJSRL-Randdde in carly stages of training, but converge the same final performances. A similar analysis for Indiscriminate Grasping is provided in Fig. 10 in the Appendix.
Indisciimnntt Demonstrations
2K Demennrrations
2.0
O6 0p--)j-4
3040
Figure 4: IL+RL methods simulated robotic grasping tasks. The baselines show improvement with fine-tuning, Qt-Opt+JSRL is more sample efficient and attains higher final performance.
5.2 VISION-BASED ROBOTIC TASKS
Utilizing offline data is challenging in complex tasks such as vision-based robotic manipulation. The high dimensionality of both the continuous control action space as well as the pixel-based state space present unique scaling challenges for IL+RL methods. To study how JSRL scales to such settings, focus on two simulated robotic manipulation tasks: Indiscriminate Grasping and Instance Grasping. In these tasks, simulated robot arm is placed in front of table with various categories of objects. When the robot lifts any object, a sparse reward is given for the Indiscriminate Grasping task; for the more challenging Instance Grasping task, the sparse reward is only given when sampled target object is grasped. An image of the task is shown in Fig. aId described in detail in Appendix 1.1 We compare JSRL against methods that have been shown scale such complex vision-based robotics settings: Qt-Opt (Kalashnikov et al. 2018), AW-Opt (Lu et al. [202I). and BC. Each method has access the same offline dataset 2,000 successful demonstrations and allowed run online fine-tuning for up to 100,000 steps. While AW-Opt and BC utilize offline successes as part of their original design motivation, we allow more fair comparison for Qt-Opt by initializing the replay buffer with the offline demonstrations, which was not the case in the original Qt-Opt paper. Since we have already shown that JSRL can work well with an offline RL algorithm in the previous experiment, to demonstrate the flexibility of our approach, in this experiment we combine JSRL with an online Q-learning method: Qt-Opt. As seen Fig. 4 the combination of Dt-Opt+JSRL (both versions of the curricula) outperforms the other methods in both sample efficiency well as final performance.
5.3 INITIAL DATASET SENSITIVITY
While most IL+RL methods are improved by more data and higher quality data, there are often practical limitations that restrict initial offline datasets. JSRL is exception this dependency, as
7

---

Under review as a conference paper at ICLR 2023
Envireament
Dataset
BC
CQU
IQL
IQL+JSRL (Ours)
Curriculum
Random
aatmazo-umaz--00
0.2 0.5
15.6 = 19.9
10.4 9.6
10k
55.5 + 12.5
71.7 + 14.5
52.3 26.7
100K
74.2 1: 25.6
93.7 1: 4.2
92.1 2.8
(standard)
Ik
0.0 = 0.0
3.1 3.118..
1.9 4.8
10%
33.1 = 10.7
72.6 - 12.2
39.4
20.1
100k
29.9
23.1
81.3 1: 23.0
82.3
14.2
Im (standard)
49.0
45.6
99.4
53.0 + 30.5
88.6
16.3
89.8
10.0
attmrze-medium-plaaxxx)
0.0
0.0
0.0
10k
0.1 0.3
16.7 16.7 17721229 12.9
3.8 5.0
100K
32.8 = 32.6
86.7 3.7
56.2 28.8
Im (standard)
0.0
0.0
92.8 2.7
91.1 3.9
87.8 4.2
unlnaaz--mciuu-ndieeeee---
IK
U.0 = 0.0
0.0 - 0.0
0.0
10k
100k
15.7 + 17.7
81.5 - 18.8
67.0 17.4
Im (standard)
0.3
0.0
32.3
92.4 4.5
93.1 3.1
86.3 5.9
uutmazz-arreeplaa-xxU
0.0:0..0
0.0
0.0
10%
0.0 0.0
0.1 0.2
0.0 0.0
Im (standard)
0.0
0.0
0.0
62.4 + 12.4
62.9 11.3
48.6 + 10.0
antmaae large diverse
1k
0.0 1 0.0
0.0
0.0
10k
100K (standard)
4.1 4.1110..4 10.4
34.4 23.0
22.4 15.4
binary-vO
100 Ik 10k 100k (standard)
18.8 11.6
24.3 1: 12.1
29.1 7.6
30.1 10.2
36.7 1: 7.9
46.3 1 6.3
38.4
44.3 6.2
52.1
70.3
0.0
9.9
65.0 1: 2.9
62.6 1: 3.6
60.6 1 2.7
door-binar--UU
100
0.8 =
0.4
0.1
Ik
1.5
0.7 1.0
0.45 1. 1.2
10k 100k (standard)
10.6 + 14.1
4.3 8.4
22.3 11.6
30.1
0.0
28.5
24.3
relocate. relocate-binary-oe vO
100
0.0 + 0.0
0.0
0.0 0.0
1k 10k
0.0 + 0.0
0.0
0.0
0.3
0.6 0.611.66 1.6
0.5 0.7
100k (standard)
2.7
0.0
0.0
8.6
7.7
0.0
4.7
Table 1:
Comparing JSRL with IL+RL baselines on D4RL tasks by using averaged normalized scores for D4RL Ant Maze and Adroit tasks. Each method pretrains runs online finetuning for Im steps. Our method IQL+JSRL competitive with IL+RL baselines in the full dataset setting, but performs significantly better in the small-data regime. For implementation details and more detailed comparisons, see Appendix A.2
offline dataset and then
Environment
# Demos
Qt-Opt
AW-Opt
Qt-Opt+JSRIL (Ours)
Indiscriminate Grasping
20
0,0 0.0
0.0 0.0
0.19 1 0.04
0.92 E 0.00
Indiscriminate Grasping
200
0.93 113::000 0.01
0.96 + 0.02
0.23 0.00
0.92 1: 0.01
Indiscriminate Grasping
0.94 0.01
0.97 + 0.01
0.44 4.44
0.94 0.03
Indiscriminate Grasping
20k
0.04
0.98
0.91 0.01
0.95
Instance Grasping
20
0.23 +.23
0.05 0.04
0.50
Instance Grasping
200
0.47 1: 0.04
0.49 0.02
0.15 1: 0.02
0.02
0.54 0.03
Instance Grasping
2k
0.15 1: 0.26
0.43
0.28 + 0.04
0.57 0.07
Instance Grasping
0.28 0.25
0.57 0.01
0.49 0.02
1 0.02
Table 2: Limiting the initial number of demonstrations is challenging for IL+RL baselines on the difficult robotic grasping tasks. Notably, only Qt-Opi+JSRL is able to learn in the smallest-data regime just 20 demonstrations, 100x less than the standard 2,000 demonstrations.
the quality of the guide- policy TH directly depends on the offline dataset when utilizing JSRL in an II+RI setting (i.e., when the guide-policy is pre-trained on an offline dataset). We study the offline dataset sensitivity of IL+RI algorithms and JSRI. on both D4RI. tasks as well as the vision-based robotic grasping tasks. The two settings presented in D4RL and Robotic Grasping quite different: IQL+JSRL in D4RL pretrains with an offline RL algorithm from mixed quality offline dataset, while Qt-Opt+JSRL pretrains with BC from high quality dataset.
For D4RL, methods typically use million transitions from mixed-quee -ualldyualyyyy policies from previous RL training runs; reduce the size of the offline datasets in Table IQL+JSRL performance perforanne degrades less than the baseline IQL performance. For the robotic grasping tasks, we provided 2,000 high- quality demonstrations. As reduce the number of demonstrations, we find that JSRL efficiently
8

---

Under review as a conference paper at ICLR 2023
learns better policies. Across both D4RL and the robotic grasping tasks, JSRL outperfonms baselines in the low- regime, as shown in Table and 2 In the high-data regime, when we increase the number of demonstrations by 10x to 20,000 demonstrations, we notice that AW-Opt and BC perform much more competitively, suggesting that the exploration challenge is no longer the bottleneck. While starting with such large numbers of demonstrations is not typically realistic setting, this results suggests that the benefits of JSRL are most prominent when the offline dataset does not densely cover good state-action pairs. This aligns with our analysis in Appendix [A.I that JSRL does not require such assumptions about the dataset, but solely requires prior policy.
5.4 JSRL-CURRICULUM Vs. JSRL-RANDOM SWITCHING
In order to disentangle these two components, we propose an augmentation of our method, JSRL- Random, that randomly selects the number of guide-steps every episode. Using the D4RL tasks and the robotic grasping tasks, we compare JSRL-Random to JSRL and previous IL+RL baselines and find that ISRL-Random performs quite competitively, as seen in Table and Table 2 However, when considering sample efficiency. Fig. 4 shows that JSRL is better than stages of training, while converged performance is comparable. These same trends hold when we limit the quality of the guide-policy by constraining the initial dataset, as seen in Fig. 3 This suggests that while a curriculum of guide-steps does help sample efficiency, the largest benefits of ISRL may stem from the presence of good visitation states induced by the guide-policy to the specific order of good visitation states, as suggested by our analysis in Appendix (A.en. For analyze hyperparameter sensitivity of JSRL-Curricllum and provide the specific implementation of hyperparameters chosen for our experiments in Appendix A.3
in early
opposed
5.5 GUIDE-POLICE GENERALIZATION
In order study how tasks, we train an indiscriminate grasping policy and use it as the guide-polic for JSRL on instance grasping (Figure 13). While the performance when using the indiscriminate guide is worse than using the instance guide, the performance for both JSRL versions outperform vanilla Qr-Opt. We also test ISRL's 's generalization capabilities in the D4RL setting. We consider two variations of Ant mazes: "play" and "diverse". In antmaze-* x-paye the agent must reach fixed set of goal locations from fixed set of starting locations. In antmaze-* -diveere,,, the agent must reach random goal locations from random starting locations. Thus, the diverse environments present greater challenge than the corresponding play environments. In Figure [1] we see that JSRL is able to better generalize to unseen goal and starting locations compared vanilla IQL.
from easier tasks can be used to efficiently explore more difficult
CONCLUSION
In this work, we propose Jump-Start Reinforcement Learning (JSRL), a method for leveraging a prior policy of any form to bolster exploration in RL to increase sample efficiency. Our algorithm creates a learning curriculum by rolling in a pre-existing guide-policy, which is then followed by the self-improving exploration policy. The job of the exploration-policy is simplified, as it starts its exploration from states closer to the goal. As the exploration policy improves, the effect of the guide- policy diminishes, leading to fully capable RL policy. Importantly, our approach is generic since it can be used with any RL method including value-based RL approaches, which have traditionally struggled in this setting. We showed the benefits of JSRL in a set of offline RI. benchmark tasks as well as more challenging vision-based robotic simulation tasks. Our experiments indicate that ISRL more sample efficient than more complex IL+RL approaches while being compatible with other approaches' benefits. In addition, we presented theoretical analysis of an upper bound on the sample complexity of JSRL, which showed noom-exoneentilltoo tameemtillto--oolyommia improvement in time horizon from non-optimism exploration methods. In the future, we plan deploying JSRL in the real world in conjunction with various types of polieceooicies to further investigate its ability to bootstrap data efficient RI.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Alekh Agarwal, Mikael Henaff, Sham Kakade, and Wen Sun. Pe-pg: Policy cover directed exploration for provable policy gradient learning. arXiv preprint arXiv:2007.08459 2020.
Alckh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research, 22(98):1-766 2021.
J Andrew Bagnell. Learning decisions: Robustness, uncertainty, and approximation. Carnegie Mellon University. 2004.
James Bagnell, Sham M Kakade, Jeff Schneider, and Andrew Ng. Policy search by dynamic programming. Advances in neural information processing systems, 16, 2003.
Jinglin Chen and Nan Jiang. information-thhorcee considerations in batch reinforcement learning.
arXiv preprint arXiv: 1rXYv:905.00366 2019.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34, 2021.
Wei Chu, Lihong Li, Lev Reyzin, and Robert Schapire. Contextual bandits with linear payoff functions. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pp. 208-214. JMLR Workshop and Conference Proceedings, 2011.
Adrien Ecoffet, Joost Huizinga. Joel Lehman, Kenneth O Stanley, and Jeff Clune. First return, then explore. Nature, 590(7847):580 1866005566600 2021.
Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel. Reverse curriculum generation for reinforcement learning. In Conference on robot learning, pp. 482-495. PMLR, 2017.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 2020.
lonel-Alexanrru Hosu and Traian Rebedea. Playing atari games with deep reinforcement learning and human checkpoint replay. arXiv preprint arXiv: 600.05077 2016.
Boris Ivanovic, James Harrison, Apoorva Sharma, Mo Chen, and Marco Pavone. Barc: Backward reachability curriculum for robotic reinforcement learning. In 2019 International Conference on Robotics and Automation (ICRA), pp. 15-21. IEEE, 2019.
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems, 34, 2021.
Nan Jiang. On value functions and the agent-environment boundary. arXiv preprint 13X14:1905.13341, 2019.
Nan Jiang, Akshay Krishnamurthy, Alekh Agarwal, John Langford, and Robert Schapire Contex- tual decision processes with low bellman rank are pac-learnable. In International Conference on Machine Learning, pp. 1704-1713, PMLR, 2017.
Chi Jin, Zeyuan Allen-Zhu, Sebastien Bubeck, and Michael Jordan. Is Q-learning provably efficient? In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 4868-49 4878, 2018
Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory, pp. 2137-2143. PMLR, 2020.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In ICML,, volume 2, pp. 267-274, 2002.
10

---

Under review as a conference paper at ICLR 2023
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. arXiv preprint arXtv:1806.10293, 2018.
Jens Kober, Betty Mohler, and Jan Peters. Imitation and reinforcement learning for motor primitives with perceptual coupling. In From motor learning to interaction learning in robots, pp. 205-225. Springer, 2010.
Sven Koenig and Reid G Simmons. Complexity analysis of real-time reinforcement learning. In AAAI, pp. 99 107, 1993.
Hya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint 10..v:2110.06166 2021.
Akshay Krishnamurthy, John Langford, Aleksandrs Slivkins, and Chicheng Zhang. Contextual bandits with continuous actions: Smoothing, zooming, and adapting. In Conference on Learning Theory, pp. 2025-2027. PMLR, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006.0477 04779, 2020.
John Langford and Tong Zhang. The epoch-greedy algorithm for contextual multi-armed bandits. Advances in neural information processing systems, 20(1):96-1, 2007.
Peng Liao, Zhengling Qi, and Susan Murphy. Batch policy learning in average reward Markov decision processes. arXiv preprint arXiv:2007.1177), 2020.
Long-Ji Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine learning, 8(3-4):293-33 29111, 1992.
Boyi Liu, Qi Cai, Zhuoran Yang, and Zhaoran Wang. Neural trust region/proximal policy optimization attains globally optimal policy. In Neural Information Processing Systems, 2019.
Yao Lu, Karol Hausman, Yevgen Chebotar, Mengyuan Yan, Eric Jang, Alexander Herzog, Ted Xiao, Alex Irpan, Mohi Khansari, Dmitry Kalashnikov, and Sergey Levine. Aw-opt: Learning robotic skills with imitation andreinforcement scale. In 202/ Conference Robot Learning (CoRL., 2021.
Stephen McAleer, Forest Agostinelli, Alexander Shmakov, and Pierre Baldi. Solving the rubik's 's cube without human knowledge. 2019.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing Atari with deep reinforcement learning. arXiv preprint 132:v:1312.5602, 2013.
Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Over- coming exploration in reinforcement learning with demonstrations. In 20/8 IEEE International Conference on Robotics and Automation (ICRA), pp. 6292-6299. IEEE, 2018
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. 2020.
Ian Osband and Benjamin Van Roy. Model-based reinforcement learning and the eluder dimension. arXiv preprini arXiv: /406. 1853, 2014.
Yi Ouyang, Mukul Gagrani, Ashutosh Nayyar, and Rahul Jain. Learning unknown markov decision processes: A thompson sampling approach. arXiv preprint arXiv:1709.045700 2017.
Xue Bin Peng, Picter Abbeel, Sergey Levine, and Michiel van de Panne. Deepmimic: Example- guided deep reinforcement learning of thysics-based character skills. ACM Trans. Graph., 37(4), July 2018.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. weiigteee-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv: 190000777 2019.
11

---

Under review as a conference paper at ICLR 2023
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforeement learning and demonstrations. arXiv preprint arXiv: 1TX.v11709.10087, 2017.
Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. Bridging offline rein- forcement learning and imitation learning; A tale of pessimism. arXiv preprint 1xXi::2103.12022
2021.
Cinjon Resnick, Roberta Raileanu, Sanyam Kapoor, Alexander Peysakhovich, Kyunghyun Cho, and Joan Bruna. Backplay:" man muss immer umkehren". arXiv preprint arXiv:1807.06919, 2018.
Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 661-668, 2010.
Tim Salimans and Richard Chen. Learning montezuma's 's revenge from single demonstration. arXiv preprint
03381, 2018
Stefan Schaal et al. Learning from demonstration. Advances neural information processing systems, pp. 1040-1046, 1997.
Bruno Scherrer. Approximate policy iteration schemes: A comparison. In International Conference on Machine Learning, pp. 1314-1322, 2014.
David Simchi-Levi and Yunzong Xu. Bypassing the monster: A faster and simpler optimal algorithm for contextual bandits under realizability. Available at SSRN 3562765, 2020.
Smart and L.. Pack Kaelbling. Effective reinforcement learning for mobile robots. In Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No.02CH37292), volume 4, pp. 3404-3410 vol.4, 2002. doi: 10.1100ROB0T22000001140
Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pictquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards, 2018.
Lingxiao Wang, Qi Cai, Zhuoran Yang, and Zhaoran Wang. Neural policy gradient methods: Global optimality and rates of convergence. In International Conference on Learning Representations, 2019.
Tengyang Xie, Nan Jiang, Huan Wang, Caiming Xiong, and Yu Bai. Policy finetuning: Bridging sample-efficient offline and online reinforcement learning. arXiv preprint arXiv:2106. 100.0995,04
04895, 2021.
Andrea Zanette, Alessandro Lazaric, Mykel Kochenderfer, and Emma Brunskill. Learning near optimal policies with low inherent bellman error. In International Conference Machine Learning, pp. 10978-10989, PMLR, 2020.
Junyu Zhang, Alec Koppel, Amrit Singh Bedi, Csaba Szepesvari, and Mengdi Wang. Varia- tional policy gradient method for reinforcement learning with general utilities. arXiv preprint arXiv:2007.02151, 2020a.
Zihan Zhang, Yuan Zhou, and Xiangyang Ji. Almost optimal model-free reinforcement learning via reference-advvantage decomposition. Advances in Neural Information Processing Systems, 33, 2020b.
Qinqing Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. arXiv preprint arXiv:2202.05607, 2022.
12

---

Under review as a conference paper at ICLR 2023
+
Figure 6: Example ant maze (left) and adroit dexterous manipulation (right) tasks.
A APPENDIX
A.1 EXPERIMENT IMPLEMENTATION DETAILS
A.1.1 D4RL: ANT MAZE AND ADROIT
We evaluate on the Ant Maze and Adroit tasks, the most challenging tasks in the D4RL benchmark Ful ct al. (020). For the baseline IL+RL method comparisons, we utilize implementations and reported results from Kostrikov et al. (202T: we the open-sourced version of IQL. and the reported results from for AWAC, BC, and CQL. While the standard initial offline datasets contain Im transitions for Ant Maze and 100k transitions for Adroit, we additionally ablate the datasets to evaluate settings with 100, 1k, 10k, and 100k transitions provided initially.
For the implementation of IQL+JSRL, we build upon the open-sourced IQL implementation Kostrikov et al. (2021) First, to obtain a guide-policy, we use IQL without mod- ification for pretraining on the offline dataset. Then, we follow Algorithm when finetuning online and use the IQI. online update the TRAINPOLICY step from AI- gorithm [ The IQL neural network architecture follows the original implementation of Kostrikov et al. (2021). For finetuning, we maintain two replay buffers for offline and online transitions. The offline buffer contains all the demonstrations, and the online buffer is FIFO with a fixed capacity of 100k transitions. For each gradient update dur- ing finetuning, we sample minibatches such that 75% of samples come from the online buffer, and 25% of samples come from the offline buffer.
O b e
Figure 5: the simulated vissen-based robotic grasping tasks, robot arm must grasp various objects placed in bins in front of it. Full implementation details are described in Appendix A.1.2 A.1.2
Our implementation of IQL+JSRL focused on two settings when switching from offline pretraining to online finetun-
ing: Warm- saarm-starting and Cold-starting. When saarm-stating,, we copy the actor, critic, target critic, and value networks from the pre-trained guide-policy to the exploration-policy. When Cold-starting, we instead start training the exploration-policy from seratch. Results for both variants are shown in Appendix A.2 We find that empirically, the performance of these two variants is highly dependent on task difficulty as well as the quality of the initial offline dataset. When initial datasets are very poor, cold-starting usually performs better; when initial datasets dense and high-quality, warm-starting seems to perform better. For the results reported in Table T we utilize Cold-start results for both CLL+JSL--Currculun and IOL+JSRL-Random
Finally, the curriculum implementation for IQL+JSRL used policy evaluation every 10,000 steps to gauge learning progress of the exploration-policy nº. When the moving average of " performance
13

---

Under review as a conference paper at ICLR 2023
increases over a few samples, move to the next curriculum stage. For the IQL+JSRL- variant, we randomly sample the number of guide-steps for every single episode.
A.1.2 SIMULATED ROBOTIC MANIPULATION
We simulate a 7 DoF arm with an over-the-shoulee camera (see Figure 5 Three bins in front of the robot are filled with various simulated objects to be picked up by the robot and sparse binary reward is assigned if any object is lifted above bin at the end of an episode. States are represented in the form of RGB images and actions continuous Cartesian displacements of the gripper's 31) positions and yaw. In addition, the policy commands discrete gripper open and close actions and may terminate an episode.
For the implementation of QI-Opt+JSRL, we build upon the Qt-Opt algorithm described in Kalash- nikov al. (2018). First, obtain a guide-policy we use BC policy trained offline on the provided demonstrations. Then, we follow Algorithm [ when finetuning online and use the Qt-Opt online update as the TRAINPOLICY step from Algorithm[]] The demonstrations are not added the Qt- Oppplayrrryplay buffer. The Qt-Opt neural network architecture follows the original implementation in Kalashnikov et (2018).
Finally, similar to Appendix A.1.1] the curriculum implementation for Qt-Opt+JSRLused policy evaluation every 1,000 steps to gauge learning progress of the exploration-pole policy π². When the moving average of r performance increases over few samples, the number of guide-steps is lowered, allowing the JSRL curriculum to continue. For the Qu-Ope-JSRL-RRadoo variant, we randomly sample the number of quide-steps. for every single episode.
A.2 ADDITIONAL EXPERIMENTS
ISRL: Random Switching
JSRL: Curriculum
Environment
Sarm-start
Cold-start
Warm-start
Cold-start
IQL
pen- cun-binary-vO
27.18 + 7.77
29.12 + 7.62
25.10 + 8.73
24.31 + 12.05
18,80 11.63
Noor-binary-vi
0.01 + 0.04
0.06 = 0.23
1.45 + 4.67
0.40 + 1.80
0.84 3.76
relocate-binary-v0
0.00 + 0.00
0.00 + 0.00
0.00 + 0.00
0.01 + 0.06
0.01 + 0.03
Table 3: Adroit 100 Offline Transitions
JSRL: Random Switching
JSRL: Curriculum
Environment pen-binary-v0
Warm-start
Cold-start
Warm-start
Cold-start
IQL
47.23 + 3.96
46.30 + 6.34
34.23 + 7.22
36.74 7.91
30.11 + 10.22
door-bine moor-binary-vO
0.15 = 0.25
0.45 1.22
0.44 + 0.89
0.68 1.02
0.53 1.46
relocate-bine 0inaa-t-
0.06 + 0.08
0.01 + 0.04
0.05 + 0.09
0.04
0.003
Table 4: Adroit Ik Offline Transitions
IQL+ISRI: Random Switching
IQL+ISRL:
Curriculum
Environment
Warm-start
Cold-start
Warm-start
Cold-start
IQL.
ven-binary-v0
51.78 = 3.00
52.11 + 3.30
38.04 + 12.71
44.31 6.22
38.41 11.18
noor-binary-v0
10.59 + 11.78
22.32 11.61
5.08 + 7.60
4.33 8.38
10.61 + 14.11
relocate-binary-v0
1.99
0.50 0.65
4.39 + 8.17
0.55 1.60
0.19 0.32
Table 5: Adroit 10k Offline Transitions
14

---

Under review as a conference paper at ICLR 2023
Naive Bootstrapping (100k Samples, 100k Warm up Steps): antmaze-umaze-vo
100
SAC
SAC+JSRL
80
Average Return
60
40
20
1.0
-0.5
0.0
0.5
1.0
Steps
1e6
Naive Bootstrapping (100k Samples, 100k Warm up Steps): antmazz-meddim-diiveesee
100
SAC
SAC+JSRL
80
Average Return
60
40
0
-1.0
-.5
0.0
0.5
1.0
Steps
1e6
Naive Bootstrapping (100k Samples, 100k Warm up Steps): antmaze-large-divereevv
100
SAC
SAC+JSRL
80
Average Return
60
20
0
-1.0
-0.5
0.0 Steps
0.5
1.0
1e6
Figure 7: policy first pre-trained on 100k offline transitions. Negative steps correspond this pre-training. We then roll out the pre-trained policy for 100k timesteps, and use these online samples to warm-up the critic network. After warming up the critic, we continue with actor-critic fine-tuning with the pre-trained policy and the warmed up critic.
15

---

Under review as a conference paper at ICLR 2023
Naive Bootstrapping (1m Samples, 100k Warm Steps): antmaze-umaze--o
100
SAC
SAC+JSRL
80
Average Return
60
40
20
0
-1.0
-0.5
0.0
0.5
1.0
Steps
1e6
Naive Bootstrapping (1m Samples, 100k Warm Steps): antmaze-medium-diierreee
100
SAC
SAC+JSRL
80
Average Return
60
40
20
-1.0
-0.5
0.0
0.5
1.0
Steps
1e6
Naive Bootstrapping (1m Samples, 100k Warm up Steps): antmaze-large-diverseww
100
SAC
SAC+JSRL
80
Average Return
60
40
20
0
-1.0
-0.5
0.0
0.5
1.0
Steps
1e6
Figure 8: policy first pre-trained on one million offline transitions. Negative steps correspond this pre-training. We then roll out the pre-trained policy for 100k timesteps, and use these online samples to warm-up the critic network. After warming up the critic, we continue with actor-critic fine-tuning with the pre-trained policy and the warmed up critic. Allowing the critic warm up provides stronger baseline to compare ISRL to, since in the case where we have a policy, but no value function, we could use that policy to train a value function.
16

---

Under review as a conference paper at ICLR 2023
Online BC Offline RL JSRL Guide Policies, 20 Demos
1.0
0.8
0.6
Success
Qt-Opt, 20 Demos Buffer
Ot-Opt JSRL, Guide Policy from Online RL
Qt-Opt JSRL, Guide Policy from BC
0.4
0.2
0.0
0
20
40
60
80
100
120
Steps (x1000)
Online BC vs. Offline RL JSRL Guide Policies, 2000 Demos
1.0
0.8
Success
0.2
Ot-Opt, 2000 Demos Buffer
Qt-Opt JSRL, Guide Palicy from Online RL
Qt- Opt JSRL. Guide Policy from BC
0.0
0
20
40
60
80
100
120
Steps (x1000)
Figure 9: QT-Opt+ISRL using guide-policis trained from-scratch online vs. guide-policies trained with BC demonstration data in the indiseriminate grasping environment. For each experiment, the guide-policy trained offline and the guide-policy trained online are of equivalent performance.
17

---

Under review as a conference paper at ICLR 2023
indiscrimingte Gresping:
1.0
0.
Graggire 5,0009
QT- OS-0pt-JSAL
ql Oot-JSRL Random 0T5Opt
JMV Opt
I0020
20000
02020
Indiscriminate Grasping: 200 Demonstrations
1.0
Glaratre S440044
02
10000
/0300
.1000
40000
50000
indiscrimingte Gresping:
Gruggin 5,0001
0.6
0.4
JSRL Random
3000
20000
23000
20000
Indiscriminate Grasping: 20c Demonstrations
1.0
0.6
GROOt+
G10gh+j52 -Radon
0.2
CT Ogt
0.0
BC
3000
30000
15000
20000
2000
2UAUU
45CU2
60000
Figure 10: Comparing IL+RL methods with JSRL on the Indiscriminate Grasping task while adjusting the initial demonstrations available. In addition, compare the sample efficiency
18

---

Under review as a conference paper at ICLR 2023
Instance Grasping: 20 Demonstrations
Orappig SaLtin
0.4
OT Jj--
OT-Qp--jont Rendem
0.1
20000
60030
60000
00000
100200
Instance Grasping: 20C Demonstretions
Graising Su0009
OSt |SRLL
01-Ops--5SL Randem
033
OT-Opt
AN Opt
022
BC
0.0
Steps
Instance Grasping: 2k Demonstrations
0.5
0.3
OT-Opt+JSRL
Pa-spp--J-.. Handom
011
OT-
AlkOpt
20000
400CU
60000
SOOBC
Instance Gresping: Demonstrations
0.6
0.5
0.3
0.
GTF CTS0gt+J55R
CT Opt 1S Random
07-0pt
0.0
Steps
Figure 11: Comparing IL.RL methods with JSRL on the Instance Grasping task while adjusting the initial demonstrations available.
19

---

Under review as a conference paper at ICLR 2023
IQL+JSRL:
Random Switching
IQL+JSRL:
Curriculum
Environment -inaain
Warm-start
Cold-start
Warm-start
Cold-start
IQL
60.06 + 2.94
60.58 = 2.73
62.81 2.79
62.59 + 3.62
64.96 2.87 2.50 8.59 7.70
door-binary-v0
27.23 + 8.90
24.27 11.47
38.70 + 17.25
28.51 + 19.54
50.21
relocate-binary-v0
5.09 + 4.39
4.69 4.16
11.18 + 11.69
0.04 + 0.14
Table 6: Adroit 100k Offline Transitions
IQL+JSRL:
Random Switching
IQL+JSRL:
Curriculum
Environment
Warm-start
Cold-start
Warm-start
Cold-start
IQL
antmaze-umaze-vv
0.10 + 0.31
10.35 9.59
0.40 + 0.94
15.60 + 19.87
0.20 = 0.52
antmarz-umaaze-diveres--
0.10 0.31
1.90 + 4.81
0.45 + 1.23
3.05 + 7.99
0.00 0.00
animaze mecilee--miimmm-p- -lay--v0
0.00 = 0.00
0.00 = 0.00
0.00 0.00
0.00 = 0.00
Hummaze-meduu-dding -iversseev
0.00 0.00
0.00 0.00
0.00
0.00
0.00 = 0.00
antmaze-large-llay-x))
0.00 + 0.00
0.00 + 0.00
0.00 + 0.00
0.00
0.00 = 0.00
antmaze-arge--ivvrse--d
0.00 0.00
0.00 0.00
0.00 0.00
0.00 + 0.00
0.00 0.00
Table 7: Ant Maze 1 Offline Transitions
IQL+JSRL:
Random Switching
IQL+JSRL: Curriculum
Environment
Warm-start
Cold-start
Warm-start
Cold-start
IQL
antmaze-umaze-vd
56.00 + 13.70
52.70 26.71
57.25 + 15.86
71.70
14.49
55.50 + 12.51
antmaze-umazz-diierss.
23.05 + 10.96
39.35 + 20.07
26.80 + 12.03
72.55 + 12.18
33.10
antmaze- mantuuz--mmciuu-llla----
0.05 + 0.22
3.75 4.97
0.00 = 0.00
16.65 12.93
0.30
0.00 + 0.00
5.10 = 8.16
0.00 = 0.00
16.60 11.71
0.00
antmaze- mutmaze-large-Pae
sarge-play-v0
0.00 + 0.00
0.00 0.00
0.00 = 0.00
0.05 + 0.22
0.00
mnmaaz-large-diveent
0.00 = 0.00
0.00 0.00
0.00 = 0.00
0.05 + 0.22
0.00
Table 8: Ant Maze 10k Offline Transitions
IQL+JSRI:
Random Switching
IQL+JSRI:
Curriculum
Environment
Warm-start
Cold-start
Warm-start
Cold-start
IQL.
antmaze-umaze-a
73.35 + 22.58
92.05 = 2.76
71.35 + 26.36
93.65 4.21
74.15
cuimaze-umaze-diveee
40.95 + 13.34
82.25 14.20
38.80 + 21.96
81.30 23.04
29.85
(limazze-mdduu--lay--000
9.55 14.42
56.15 + 28.78
22.15 + 29.82
86.85 3.67
3
antmaze-medium--divess-
14.05 + 13.30
67.00+117
15.75 + 16.48
81.50 + 18.80
antmaze-large-play
0.35 + 0.93
17.70 13.35
0.45 1.19
36.30 16.41
2.55 8.19
untmaze-large-diverse.
1.25 + 2.31
22.40 = 15.44
+ 1.16
34.35 + 22.97
4.10 + 10.37
Table 9; Ant Maze 100k Offline Transitions
IQL+JSRL:
Random Switching
IQL+ISRL: Curriculum
Environment
Warm-start
Cold-start
Warm-start
Cold-start
IQL
antmaze-umaze--e
95.35 + 2.23
94.95 = 2.95
96.70 1.69
98.05 + 1.43
97.60 3.119
antmazz-ummaze-diveeee--d
65.95 + 27.00
89.80 10.00
59.95 + 33.90
88.55 16.37
52.95
antmaze-meddinn-plaa--000
82.25 + 4.88
87.80 + 4.20
92.20 +
91.05 + 3.86
92.75
2.73
antmare-meddum-ddreeeeww
83.45 4.64
86.25 5.94
91.65 2.98
93.05 + 3.10
92.40 + 4.50
autmaze-large-play--0
50.35 9.77
48.60 = 10.01
72.15 9.66
62.85 11.31 11.31
62.35 12.42
antmaze-large-diveree.
56.80 + 9.15
58.30 - 6.54
70.55 + 17.43
68.25 - 8.76
68.25 1 8.85
Table 10: Ant Maze Im Offline Transitions
A.3 HYPERPARAMETERS OF JSRL
JSRL introduces three hyperparameters: (1) the initial number of guide-steps that the guide-policy takes the beginning of fine-tuning (H1), (2) the number of curriculum stages (n), and (3) the
20

---

Under review as a conference paper at ICLR 2023
Moving Average Horizon
10
0%
79.66
36.66
74,83
Tolerance
5%
51.12
78.8
79.78
15%
56.41
47,46
59.52
Table 11: We fix the number of curriculum stages at = 10 for antmaze-large-diveree then vary the moving average horizon and tolerance. Each number the average reward after 5 million training steps of one seed. As tolerance increases, the reward decreases since curriculum stages are not fully mastered before moving on.
performance threshold that decides whether to move on to the next curriculum stage (3). Minimal tuning was done for these hyperparameters.
IQL+JSRL:
For offline pre-training and online fine-tuning, we use the same exact hyperparameters the default implementation of IQL. [6].
Our reported results for vanilla IQL do differ from the original paper, but this due to us running more random seeds (20 vs. 5), which also consulted with the authors of IQL. For Indiscriminate and Instance Grasping experiments we utilize the same environment, task definition, and training hyperparameters as Qt-Opt and AW-Opt.
Initial Number of Suide-Steps: H1:
train the guide-policy (IQL. for D4RL and BC for grasping) then evaluate it to determine how many steps it takes to solve the task on average. For D4RL, we evaluate over one hundred episodes. For grasping, we plot training metries and observe the average episode length after convergence. This average is then used as the initial number of guide-steps. Since H1 is directly computed, no hyperparameter search required.
For all K+ISRLeeperrimmt
Curriculum Stages:
Once the number of curriculum stages was chosen, we computed the number of steps between curriculum stages Hi, Then h varies from Hi Hi HI, - H, 1 (n 1) Hi on an appropriate number of curriculum stages, we decreased (increased H4 and H, H4-1), starting from H, until the curriculum became too difficult for the agent to overcome (i.c., the agent becomes "stuck" on a curriculum stage). We then used the minimal value of n for which the agent could still solve all stages. In practice did not try every value between H and 1, but chose a very small subset of values to test in this range.
n 0. To decide
n
Performance Threshold 3: For both grasping and D4RL tasks, evaluated between fixed intervals and computed the moving average of these evaluations (5 for D4RL, 3 for grasping). If the current moving average close enough to the best previous moving average, then curriculum stage to i + 1. To define "close enough", we set a tolerance that let the agent move to the next stage if the current moving average was within some percentage of the previous best. The tolerance and moving average horizon were our "0",, generic parameter that is flexible based on how costly is evaluate the performance of In Figure 12 and Table we perform small studies to determine how varying 3 affects JSRL's performance.
from
21

---

Under review as a conference paper at ICLR 2023
1.0
0.8
0.6
Success
JSRL tolerance 0%, moving_ average
0.4
JSRL tolerance 0%, moving_ _average 3
JSRL, tolerance 0%, moving aaenrage 10
JSRL, tolerance 5%, moving_averan _averageeeeeeeee
JSRL. tolerance 5%, moving_
3
0.2
JSRL, tolerance 5%, moving average 10
JSRL, tolerance 15%, moving_average
JSRL, tolerance 15%, moving
JSRL, tolerance 15%, movingg_vveaage 10
0.0
0 10 20
30
40
50
60
Steps (x1000)
Figure 12: Ablation study for 3 in the indiscriminate grasping environment. We find that the moving average horizon does not have large impact on performance, but larger tolerance slightly hurts performance. A larger tolerance around the best moving average makes it easier for JSRL move to the next curriculum stage. This means that experiments with a larger tolerance could potentially move the next curriculum stage before JSRL masters the previous curriculum stage, leading lower performance.
Qt-Opt
Qt-Opt JSRL
Qt-Opt JSRL indiscriminate grasping guided policy
0.8
0.6
Success
0.4
0.2
0.0
0
5
10
15
20
25
30
35
40
Steps (x1000)
Figure 13: First, indiscriminate grasping policy is trained using online QT-Opt to 90% indiscrimi- nate grasping success and 5% instance grasping success (when the policy happens to randomly pick the correct object). We compare this 90% indiscriminate grasping guide policy with 8.4% success instance grasping guide policy trained with BC on 2k demonstrations. While the performance for using the indiscriminate guide is slightly worse than using the instance guide, the performance for both JSRL versions are much better than vanilla Qt-Opt.
22

---

Under review as a conference paper at ICLR 2023
Generalization: antmaze-umaze-vu to antmaze-umaze-divert
100
80
Average Return
60
40
20
IQL+JSRL
IQL
0
-1.0
-0.5
0.0
0.5
1.0
Steps
1e6
Generalization: antmaze-medium play--vv to nntmaze-medium-diveere
100
IQL+JSRL
IQL
80
Average Return
60
40
20
-1.0
-0.5
0.0
0.5
1.0
Steps
1e6
Generalization: antmaze-large-play-o0 to antmaze-larges diveeeevese- 100
IQL+JSRL
IQL
80
Average Return
60
40
20
0
-1.0
-0.5
0.0
0.5
1.0
Steps
1e6
Figure 14: First, policy is trained offline simpler mutmaze-*-pla environment for one million steps (depicted by negative steps). This policy is then used for initializing fine-tuning (depicted by positive steps) in more complex fatmaze---divers environment. We find that IQL+JSRL can better generalize to the more difficult antmazes compared to IQL even when using guide-policies trained on different tasks.
23

---

Under review as a conference paper at ICLR 2023
A.4 THEORETICAL ANALYSIS FOR JSRL
A.4.1 SETUP AND NOTATIONS
Consider finite-horizon time- iimooonnoooeeesu MDP with fixed total horizon H and bounded reward Th E [0, 1], Vh [H). The transition of state-action pair (s, a) in step h denoted as P.. s, a).
Assume that at step 0, the initial state follows distribution po-
For simplicity, we use π to denote the policy for H steps π = { We let di (s) be the marginalized state occupancy distribution in step h when we follow policy
A.4.2 PROOF SKETCHI FOR THIBOREM
h ==
h = 2
h = 3
h =
H -
h = H
g a''
a
8
8
(s aa) = 1
(s a) 0 for all other cases
Figure 15: Lower bound instance: combination lock
We construct special instance, combination lock MDP, which is depicted in Figure 15 and works as follows. The agent can only arrive at the red state 8441 in step h - 1 when it takes action oj the red state st at step h. Once it leaves state 8) the agent stays in the blue states and can never get back to red states again. At the last layer, one receives reward 1 when the agent is state s and takes action djr- For all other cases, the reward is O. In exploration from scratch, before seeing TH (s*, aa)), one only sees reward 0. Thus 0-initialized e-greedy always takes each action with probability 1/2. The probability of arriving at state sy with uniform actions is 1/2",, which means that one needs at least nd samples in expectation to see TH (s (s","".
A.4.3 UPPER BOUND OF JSRL
and and its assumption in formal way. First, we make assumption on the quality of the guide-policy, which is the key assumption that helps improve the exploration from exponential to polynomial sample complexity. One of the weakest assumption in theory of offline learning literature is the single policy concentratability coefficient RRashidinejad et al. (2021); Xie et al. 2002 Concretely, they assume that there exists a guide-policy n. such that
In this section, we restate Theorem 4.3
dh (s
sup s,a.h di. (8,a)
SC
This means that for any state action pair that the optimal policy visits, the guide-policy shall also visit with certain probability.
In the analysis, we impose a strictly weaker assumption. We only require that the guide-policy visits all good states in the feature space instead of all good state and action pairs.
Assumption A.1 (Quality of guide-policy T²). Assume that the state is parametrized by some feature mapping Ø S - Rd such that for any policy x, (QT a) and x (s) depends on only through d(s). We assume that in the feature space, the guide-policy T7 cover the states visited by the optimal policy:
dF (o(s))
sup 2 d (ø($)) SC
'The single policy concentratability assumption is already a weaker version of the traditional concentratability coefficient assumption, which takes supremum of the density ratio over all state-action pairs and all policies
(Scherrer 2014; Chen & Jiang 2019 Jiang 2019 Wang et al. 2019 Liao al. 2020 Liu al. 2019
Zhang
202
24

---

Under review as a conference paper at ICLR 2023
Note that for the tabular case when ((s) 8, one easily prove that equation implies Assump- tion A.I In real robotics, the assumption implies that the guide-policy at least sees the features the good states that the optimal policy also see. However, the guide-policy can be arbitrarily bad in terms of choosing actions.
Before we proceed to the main theorem, we need impose another assumption on the performance of the exploration step, which requires to find an exploration algorithm that performs well in the case of H = (contextual bandit).
Assumption A.2 (Performance guarantee for ExplorationOracle CB). In (online) contextual bandit with stochastic context Po and stochastic reward r(s, a) supported on [0, R), there exists some ExplorationOracle.CE which executes policy x' in each round E [T), such that the total regret is bounded:
T E B (s)) r(s x (+)) s J(T,
Σ
This assumption is usually given for free since it is implied by rich literature in contextual bandit, including tabular Langford Zhang 2007), linear Chu et al. (2011), general function approximation with finite action Simchi-Levi & Xu (2020), neural networks and continuous actions Krishnamurthy jet al. (2019), either via optimism-basee methods (UCB, Thompson sampling etc.) or non-optimism- based methods (e-greedy, inverse gap weighting etc.).
Now we are ready to present the algorithm and guarantee. The JSRL algorithm is summarized in Algorithm [ For the convenience of theoretical analysis, we make some simplification by only considering curriculum case, replacing the step of EvaluatePolicy with a fixed iteration time, and set the TrainPolicy in Algorithm [ follows: iteration h, fix the policy #h+1 unchanged, set Th ExplorationOracle CB(D), where the reward for contextual bandit is the cumulative reward
h:H Tt. For concreteness, we show the pseudocode for the algorithm below.
A Algorithm 2 Jump-Start Reinforcement Learning for Episodic MDP with CB oracle 1: Input: guide-policy x9, total time step T, horizon length
2: Initialize cxploration policy
T², online dataset D Ø.
3: for iteration h = H H 2, 2...... do
4: Execute CaplorationOOacle...m
for [T/H] rounds, with the state-aciton aate-accioo-reward tuple for
contextual bandit derived as follows: at round f, first gather a trajectory {(1 of si+1. rf) hejH-i]
by rolling out policy x, then take {sh,, a Ee ri} as the mate-action-rewan samples for contextual bandit. Let x' be the executed policy at round t. 5: Set policy n Unif Unif([=')7 ([>')T=
6: end for
Note that the Algorithm 2 is special case of [ where the policies after current step is fixed. This coincides with the idea of Policy Search by Dynamic Programming (PSDP) in Bagnell al. (2003). Notably, although PSDP mainly motivated from policy learning while JSRL is motivated from efficient online exploration and fine-tuning, the following theorem follows mostly the same line as that in BBgnelll (2004). For completeness we provide the performance guarantee of the algorithm as follows.
Theorem A.3. Under Assumption A.1 and A.2 rounds.
the JSRL Algorithm D
guarantees that after T
H-1 E Eso (so) VO" (so)) S C. Σ H(TH,HH-)) h=0
Theorem A.3 is quite general, and it depends on the choice of the exploration oracle. Below give concrete results for tabular RL and RL with function approximation.
Corollary A.4. For tabular case, when take ExplorationOracle. CB c-greedy, the rate achieved 1/3 ///3 //1///1//////// when we take ExplorationOrache CB as FALCON+, the rate becomes // //T/)///////////// Here S can be relaxed to the maximum state size that 7 visits among all steps.
25

---

Under review as a conference paper at ICLR 2023
The result above implies polynomial sample complexity when combined with non-optimism exploration techniques, including e-greedy [angford & Zhang (2007) and FALCON+ Simchi- & Xu (2020). In contrast, they both suffer from curse of horizon without such guide-policy.
Next, we move to RL with general function approximation.
Corollary A.5. For general function approximation, when take ExplorationOracle. CB as FAL- CON+, the rate becomes Õ(C E VAEz(T/H) under the following assumption.
Assumption A.6. Let be an arbitrary policy. Given training trajectories of the form (6hh "h Ph+1:7 7 semm, drawn from following policy π in given MDP, according to 5/2 ~ d of |s) ~ TA (*n), r (s1 af)) Rs (5%, a/), ( ax)) ~ Pa (t a/), there ex- ists some offline regression oracle which returns family of predictors Qn. : x A R, (H) such that for any h E [f]. we have
E - s Ex(n).
As is shown in Simchi-Levi & Xu (2020), this assumption on offline regression oracle implies our Assumption regret bound in Assumption A.2 When Ey is a polynomial function, the above rate matches the worst-case lower bound for contextual bandit in Simchi-Levi & Xu (2020), up to factor of C. poly(H).
The results above show that under Assumption A. one can achieve polynomial and sometimes near-optimal sample complexity up to polynomial factors of H without applying Bellman update, but only with contextual bandit oracle. In practice, we run Q-learning based exploration oracle, which may be more robust to the violation of assumptions. We leave the analysis for Q-learning based exploration oracle as a future work.
Remark A.7. The result generalizes to and is adaptive to the case when one has time-inhomogeneous C.i.e
di (6(8))
Yh [H], sup d (o(s)
I C(h).
The rate becomes DI C(h) JHT/H, h) in this case.
In our current analysis, heavily rely on the assumption of visitation and applied contextual bandit based exploration techniques. In our experiments, we indeed run a Q-learning based exploration algorithm which also explores the succinet states after we roll out the guide-policy. This also suggests why setting K > 1 and even random switching in Algorithm [ might achieve better performance than the case of = We conjecture that with a Q-learning based exploration algorithm, JSRL still works even when Assumption onnly holds partially. We leave the related analysis for JSRL with Q-lcarning based exploration oracle for future work.
A.4.4 PROOF OF THEOREM A.3 AND COROLLARIES
Proof. The analysis follows same line as Bagnell (2044) For completeness we include here. By the performance difference lemma Kakade Langford (2002), has
H-1
Exp-dd (so) - Vi" (so)] = Σ
E Eond;
(s
(2)
h=0
At iteration h. the algorithm adopts policy π with कं = -/, VI < h, and fixed learned m > The algorithm only updates gı during this iteration. By taking the reward as E/ Tt, this presents contextual bandit problem with initial state distribution d reward bounded in between [0, H - h), and the expected reward for taking state action (s, a) is Q (s, a). Let म be the optimal policy for this contextual bandit problem. From Assumption A.2
know that after T H rounds at iteration h.
26

---

Under review as a conference paper at ICLR 2023
one has
H-1
Ea 1Q% (s)) - (T(s
E Bendy (s, (% (x))]
0
(84)
E Band; ((1
4=0
(iis)
C
E B (0
(iv)
H-1
S
C. Σ
H
h=0
Here the inequality (i) uses the fact that # is the optimal policy for the contextual bandit problem. The equality (ii) uses the fact that Q, æ depends only through 6(s). The inequality (iii comes
from Assumption A.I The inequality (iv) comes from Assumption A.2 From Equation equation we know that the conclusion holds true.
When ExplorationOracle.cB is e-greedy, the rate in Assumption A.2 becomes f(T, R) R. ((SA/T)//3) Langford & Zhang (2007). which gives the rate for JSRL as O(CHT/331/3 A1/3 / /T1;3); ) when we take ExplorationOracle.co CB as FALCON+ tabular case, Simchi-Levi & Xu 22020), the the rate in Assumption A.2 becomes (T, R) R. ((SA2 (T) A When we take ExplorationOracle CB as FAL- final rate for JSRL becomes O(CH$/251/2 A/T4/2).
CON+ in general function approximation under Assumption A.6 the rate in Assumption A.2 becomes f(T, R) R (A) 1/2 the final rate for JSRL becomes OCC EH (TE)(
27


Reviews:
[{"review_id": "zj6hJIKq70Y", "reviewer": "Reviewer_WwEP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose Jump-Start Reinforcement Learning (JSRL) which utilizes a pre-trained guide policy to form a curriculum of starting states for a different exploration policy. Theoretical analysis shows that with a properly chosen training and evaluation algorithm, JSRL achieves a polynomial sample complexity. Empirical evaluations on D4RL and vision-based robotic tasks are provided to show the effectiveness of JSRL.", "summary_of_the_review": "The major weakness is the evaluation results, which did not convince me that JSRL is stronger than baseline IL + RL methods.", "strength_and_weaknesses": "Strength: JSRL is well-motivated to address the sample complexity requirement in training RL agents from scratch. The proposed algorithm is easy to understand and empirical evaluations show some effectiveness, especially in the low demo data regime. \n\nWeakness:\nThe empirical evaluation section could be improved. In Table 2, three baseline methods were not tested on low demo data regimes, only the 1 million standard setting. As some of them achieve better results than JSRL in the standard setting, we need to see the comparisons in the low data setting to evaluate how effective JSRL really is.\n\nMoreover, the improvement margins JSRL has over baseline methods are relatively small. In fact, in Table 1, for Instance Grasping, the result confidence intervals of 20 and 20k demos overlap with those of AW-Opt. The improvement in D4RL tasks is also small. So I am not convinced JSRL, in its current form, is really better than existing baselines.\n\nThere are some imprecise writings in the theoretical analysis, see my questions in the next section.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "sDVuXr25El", "reviewer": "Reviewer_5862", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper addresses an interesting problem for RL i.e., sample efficiency. The author proposes a Jump-Start RL (JSRL) algorithm to leverage a prior policy of any form to give a head start for exploration in RL. The proposed algorithm rolls out a pre-existing guided policy, followed by self-improving exploration policy. The proposed method is evaluated on simulated tasks and vision-based robotic tasks. Also, the author provides the upper bound on the sample complexity for proposed method.", "summary_of_the_review": "already mentioned above.", "strength_and_weaknesses": "Strengths:\n- The paper is well-written.\n- The problem addressed in the paper is very important problem for RL to be used widely as a learning method.\n- The paper evaluates the proposed method on extensive simulated and vision-based robotic tasks.\n- The upper bound on sample complexity of the proposed method is good. \n\nWeaknesses/Questions:\n- The problem of sample complexity of RL is very useful in real-world robotic tasks. I have only seen authors mentioning about real-world robotic task in the end of the paper as Future Work. Can authors discuss how the proposed method would help in real-world robotic tasks?  If authors have some small experiment on a real-world robotic task to demonstrate effectiveness of the proposed method that would make the paper more strong?\n-  The guided policy induces some bias in learning the policy to learn a task. However, RL might find a better policy from scratch if the guided policy is not there? Can author comment on the possible bias induced due to the guided policy?\n- The guided policy is chosen from similar domain task as the new task? How would JSRL would perform if a guided policy is from different domain than the new task? For example, can we use a guided policy of grasping on a door closing task?\n- Could you please mention abbreviation of IQL? I know you guys are referring to Implicit Q-Learning. However, its good for the reader to know this terms in the paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "ezzm98oPn20", "reviewer": "Reviewer_3rP7", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a general method called Just-Start Reinforcement Learning to utilize the pre-existing policy for better learning an RL policy. Two policies, a guide-policy and a exploration-policy, are used interactively to efficiently explore the spaces. The guide-policy is used at the beginning of each training episode. And an updated exploration policy will take charge of the other steps until the end of the time horizon. This paper theoretically analyzes the upper bound on the sample complexity of JSRL is polynomial. Furthermore, this paper presents simulated experiments on a set of robotics task. The proposed method significantly outperform related IL+RL works.", "summary_of_the_review": "This paper proposes a novel frame work for efficiently exploring the state by using existing knowledge. Sufficient theoretical and empirical analysis is provided to support the statement in this paper. ", "strength_and_weaknesses": "The method proposed in this paper is a very general framework for efficiently and quickly learning an RL policy by utilizing the knowledge of an existing guide-policy. Any RL algorithm and policy model can be plugged into the proposed framework to improve its performance under an reasonable assumption. Strong theoretical proofs are provided to verity the advantages of this paper. From the experimental results, the improvement is significant compared to the existing methods. Different experimental setups are considered which gives a sufficient support for the theoretical result. \n\nI understand that Assumption 4.2 is relatively weak compared to the assumptions in related works. And this assumption is also essential for the theory. But in reality, I still think it is pretty strong. It could be hard to find a guide policy which only visits all good states in feature space.  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "T6MtJBAL7PA", "reviewer": "Reviewer_hK9Y", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose JSRL to explore and learn by continuously using existing polices to reach a state before adopting a learnable policy. The authors emphasize that when using existing policy to select the arrival state, one can consider gradually decreasing the time step of their decisions, thus providing a more stable learning process.", "summary_of_the_review": "I think this paper is interesting enough that I will take into account the comments of other reviewers with the results of the rebuttal to adjust my score", "strength_and_weaknesses": "\nStrength.\n* The paper is clearly written and very well understood\n* The method is helpful when reinforcement learning is performed for application\n* Experimental results show that the method proposed by the authors has clear advantages\n\nWeaknesses.\n* The paper emphasizes the shortcomings of the value-based approach, and in fact most of the experiments are based on the AC architecture approach.\n* The paper has some overclaims, mentioning \"It is also compatible with any RL algorithm and can be easily combined with existing offline and/or online RL methods\" several times in the paper, but it is worth noting that JSRL is only applicable to some specific scenarios, such as offline to online, and I think the authors should be more rigorous in expressing the advantages of the method.\n* The results of Fig2 are intuitive, and since critic is not learned, it is natural that efficient finetuning is not possible. in conjunction with Fig7 and Fig8 in the appendix, I believe that the core problem comes from the need for reasonable policy evaluation learning at the beginning of the next phase in order to proceed well with further learning, but I find the authors' formulation in the relevant section confusing .\n* The authors should have compared transfer learning and offline2online related methods instead of a few simple baseline algorithms.\n* In Fig3, it looks like there is no significant advantage of gradually decreasing methods compared with the randomly chosen time step methods\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject