Reinforcement Learning with Sparse Rewards using Guidance from Offline Demonstration
Authors: Desik Rengarajan
Gargi Vaidya
Akshay Sarvesh
Dileep Kalathil
Srinivas Shakkottai
Venue: ICLR 2022 Spotlight (2022)
Keywords: Reinforcement Learning
Sparse Rewards
Learning from Demonstrations

Abstract: A major challenge in real-world reinforcement learning (RL) is the sparsity of reward feedback.  Often, what is available is an intuitive but sparse reward function that only indicates whether the task is completed partially or fully.  However, the lack of carefully designed, fine grain feedback implies that most existing RL algorithms fail to learn an acceptable policy in a reasonable time frame.  This is because of the large number of exploration actions that the policy has to perform before it gets any useful feedback that it can learn from.  In this work, we address this challenging problem by developing an algorithm that exploits the offline demonstration data generated by {a sub-optimal behavior policy} for faster and efficient online RL in such sparse reward settings.  The proposed algorithm, which we call the Learning Online with Guidance Offline (LOGO) algorithm, merges a policy improvement step with an additional policy guidance step by using the offline demonstration data.  The key idea is that by obtaining guidance from - not imitating - the offline {data}, LOGO orients its policy in the manner of the sub-optimal {policy}, while yet being able to learn beyond and approach optimality.  We provide a theoretical analysis of our algorithm, and provide a lower bound on the performance improvement in each learning episode.  We also extend our algorithm to the even more challenging incomplete observation setting, where the demonstration data contains only a censored version of the true state observation.  We demonstrate the superior performance of our algorithm over state-of-the-art approaches on a number of  benchmark environments with sparse rewards {and censored state}.  Further, we demonstrate the value of our approach via implementing LOGO on a mobile robot for trajectory tracking and obstacle avoidance, where it shows excellent performance.

Published as a conference paper at ICLR 2022
REINFORCEMENT LEARNING WITH SPARSE REWARDS USING GUIDANCE FROM OFFLINE DEMONSTRATION
Desik Rengarajan, Gargi Vaidya, Akshay Sarvesh, Dileep Kalathil & Srinivas Shakkottai Department of Electrical and Computer Engineering. Texas A&M University k va dya, sarves, di dileep... kala kalathil, sshakkot. @ttttetttttt @t.mm. edu
ABSTRACT
A major challenge in real-world reinforcement learning (RL) is the sparsity of reward feedback. Often, what available an intuitive but sparse reward function that only indicates whether the task is completed partially or fully. However, the lack of carefully designed, fine grain feedback implies that most existing RL algorithms fail to learn acceptable policy in a reasonable time frame. This is because the large number of exploration actions that the policy has to perform before it gets any useful feedback that it can learn from. In this work, address this challenging problem by developing an algorithm that exploits the offline demonstration data generated by sub-optimal behavior policy for faster and efficient online RL. in such sparse reward settings. The proposed algorithm, which we call the Learning Online with Guidance Offline (LOGO) algorithm, merges a policy improvement step with an additional policy guidance step by using the offline demonstration data. The key idea is that by obtaining guidance from not imitating the offline data, LOGO orients its policy in the manner of the sub-optimal policy, while yet being able to learn beyond and approach optimality. We provide a theoretical analysis of our algorithm, and provide a lower bound on the performance improvement in each learning episode. We also extend our algorithm to the even more challenging incomplete observation setting, where the demonstration data contains only censored version of the true state observation. We demonstrate the superior performance of our algorithm over state-of state-of-the-art approaches a number of benchmark environments with sparse rewards and censored state. Further, we demonstrate the value of our approach via implementing LOGO on a mobile robot for trajectory tracking and obstacle avoidance, where it shows excellent performance.
INTRODUCTION
Reinforcement Learning (RL) is being considered for application to a variety of real-world settings that have very large state-action spaces, but under the availability of accurate simulators determining how best to explore the environment is a major challenge. Oftentimes, the problem setting is such that intuitively obvious reward functions are quite sparse. Such rewards typically correspond to the attainment of a particular task, such as a robot attaining designated way points, with little fine-grain reward feedback the intermediate steps taken to complete these tasks. The sparsity of rewards implies that existing RL approaches often do not make much headway into learning viable policies unless provided with carefully hand-tuned reward feedback by a domain expert.
Many of these problems correspond to systems where data has been gathered over time using an empirically determined (sub-optimal) behavior policy, which contains information important to help bootstrap learning. An additional caveat is that this behavior data might only contain measurements of a subset of the true state. Is it possible to design an algorithm that has principled policy optimization for efficient learning, while also being able to explore by utilizing the data derived from behavior policy? ? A natural candidate to begin with is the framework of policy gradient algorithms (Schulman et al. 2015 Lillicrap al. 2016; Haarnoja et al. 2018), which performs really well in the dense reward setting. Can such a policy gradient algorithm be aligned with a behavior policy for guided exploration that is critical in the sparse reward setting?

---

Published as a conference paper at ICLR 2022
In this work, we propose principled approach for using the well known trust region policy optimiza- tion (TRPO) algorithm (Schulman et al.] 2015) with offline demonstration data for guidance in a sparse reward setting. Our choice of the TRPO approach is motivated by its analytical tractability and superior performance. Our key insight is that we can utilize the trust region approach while being guided by the behavior data by virtue of two steps. The first step is identical to traditional TRPO to generate a candidate policy. In the second step, the objective is to find a policy closest to the behavior policy, subject it being in the trust region of the candidate generated in the first step. Thus, the second step ensures that the policy chosen is always guided by the behavior policy, but the level of alignment with the behavior policy can be reduced by shrinking the trust region over time. We call our approach Learning Online with Guidance Offline (LOGO) to capture this two-step procedure. This principled approach enables us to provide analytical performance guarantees while achieving superior performance in a number of benchmark problems.
Our main results are as follows: (i) LOGO can guarantee a performance improvement as long as the behavior policy is able to offer advantage. Reducing the alignment with the behavior policy over time allows us to extract this advantage and then smoothly move beyond to find near-optimal policies, (ii) provide generalized version of the Performance Difference Lemma (Kakade & Langford (002) wherein the stage reward function can depend on the policy itself, and use this result to determine surrogate objective function for step 2 of LOGO that is straightforward evaluate. This allows us to implement LOGO in the manner of two TRPO-like steps, enabling us to leverage the TRPO code base, (iii) we show on standard MuJoCo environments that LOGO trained with sparse rewards can attain nearly the same performance as an optimal algorithm trained with dense rewards. We go further and show that this excellent performance carries over to waypoint tracking by a robot over Gazebo simulations and real-world TurtleBot experiments, (iv) finally, we show that LOGO can also be used in the case where the demonstration data only contains a censored version of the true state (incomplete state information) by simply adding projection to the available subset of the state space in step 2 of the algorithm. Again, we provide supporting evidence via excellence both MuJoCo simulations, as well as obstacle avoidance by a TurtleBot that is trained with behavior policy without a Lidar (censored state), but is tested with it (full state).
1.1 RELATED WORK
Our work is mainly related to two RL research areas:
Imitation Learning (IL): The goal of an II. algorithm is to imitate an (expert) policy using the demonstration data generated by that policy. Behavior cloning (BC) is a simple IL approach where the expert policy is estimated from the demonstration data using supervised learning. BC algorithms, however, suffer from the problem of distribution shift (Ross et al. 2011). Inverse reinforcement learning (IRL) algorithms (Ng & Russell, 2000, Ziebart et al.) 2008) estimate a reward function from the demonstration data and solve a forward RL problem using this reward function. Generative adversarial imitation learning (GAIL) (Ho & Ermon 2016) avoids the reward estimation problem by formulating the IL problem as a distribution matching problem, and provides an implicit reward for the RL algorithm using a discriminator (Goodfellow al. 2014). Most IL algorithms do not use reward feedback from the environment, and hence are restricted to the performance of the policy that generated the demonstration data. Our approach is different from pure II., and we leverage online RL. with (sparse) reward feedback for efficient learning.
Learning from Demonstration (LfD): The key idea of the LfD algorithms is to use demonstration data to aid online learning (Schaal et al., 1997). Many works propose to exploit demonstration data by adding it the replay buffer with prioritized replay mechanism to accelerate learning (Hester et al. 2018: Vecerik et al., 2017] Nair et al. 2018). Rajeswaran et al. (2018) combine a policy gradient algorithm with demonstration data by using a mix of behavior cloning and ouline RL fine tuning. Nair et al. (2020) propose AWAC algorithm to accelerate online RL by leveraging large amounts of offline data with associated rewards. Different from this, LOOGO does not need the reward observations. Moreover, we give provable guarantees on the performance of LOGO whereas AWAC does not have any such provable guarantee (further details are provided in Appendix G. Kim et al. (2013) propose a framework to integrate LfD and approximate policy iteration by formulating a coupled constraint convex optimization problem, where the expert demonstrations define a set of linear constraints. This approach is. however, limited to small problems with a discrete action space. The closest to our work is the PofD algorithm proposed by Kang et al. (2018). PofD modifies the reward function by taking a weighted combination of the sparse reward from the online interaction and an implicit
2

---

Published as a conference paper at ICLR 2022
reward obtained from the demonstration data using a discriminator. Very different from this, propose an intuitive and principled approach of using the offline demonstration data for guiding the online exploration during the initial phase of learning. Our two step approach enables us provide rigorous performance guarantee for the proposed LOGO algorithm, and to leverage trust region- approaches to solve high dimensional problems with even sparser settings. We provide an extensive comparison between our algorithm and PofD in Section S
Recently, there has been many interesting works in the area of offline RL (Kumar et al. 2019, Fujimoto et al. 2019; Siieeel et al. 2020; Wu el al. 2019a) which focus on learning a policy using only the offline data without any online learning or online fine-tuning. Different from these, LOGO is an online RL algorithm (further details are provided in Appendix G)
Offiine RL:
2
PRELIMINARIES AND PROBLEM SETTING
2.1 PRELIMINARIES
A Markov Decision Process (MDP) is denoted as tuple < S.A.R. >, where S is the state space, AA is the action space, R S x AA- -> is the reward function, P: A [0. 1] the transition probability function with P(s'|s, a) giving the probability of transitioning to state s' when action a is taken at state 8, and y E (0, is the discount factor. A policy т is a mapping from S to probability distribution over A, with ITs,, a) specifying the probability of taking action in state s. A policy π can generate state-action trajectory т, where T = (su. ag. 81, a1 a)....),50 80 ~ μ., ag ~ r ") 8t+1 P(.|st, at) and μ is the initial state distribution. The infinite horizon discounted return of policy æ is defined as (T((T) = E oo 7 R(s,. The goal of a reinforcement learning algorithm is learn the optimal policy x arg max, JR(TT)
The value function of a policy प defined as (s) = Ex-r 7 7Rss, 0z)[s0 = s]. is the expected cumulative discounted reward obtained by following the policy starting from the state 8. The action-value function of policy ा is defined similarly as Q7
E,ax 1E 2 R(s, 0e)lso s, ag = a]. The advantage function is defined as (s.a) (T)(sa) - VØ
The discounted state visitation distribution for the policy т, denoted as d', defined as d= (s) (1 - 7) Ei= o ''P(ss = s/ where the probability defined with respect to the randomness induced by и,, P and μ.
Definitions and Notations: The Kullback- (KL) divergence between two distribution p and q is defined as (p. g) = E., log p(z) lices Ti and T2 with respect to d is defined as KL π2) Ea-ara [DKL(T1(6) +), (s. )]]- The maximum KL divergence between two polices Ti and T2 is defined as DRE (T), TT2) = max, DKI(mm (8, (2(s))) .)) The total variation (TV) distance between two distribution p and q is defined as Dry (p. 7) = (1/2) Ex (p(x) - g(x))- The average TVV distance and the maximum TV dis- tance between two polices wi and IT are defined as (T T2) = Eandr [DTv(TT] (TI(s,)) (2,,)
q(z)- The average KI. divergence between two po-
and D (MI, x2) max, DTv(T1(S -), (2(5, -)), respectively.
2.2 PROBLEM SETTING
As described in the introduction, our goal is to develop an algorithm that can exploit offline demonstra- tion data generated using a sub-optimal behavior policy for faster and efficient online reinforcement learning in sparse reward setting. Formally, we assume that the algorithm has access to the demon- stration data generated by a sub-optimal behavior policy The We first consider the setting in which the demonstration data has complete state observation. In this setting, the demonstration data has the form D = {r') where 7¹ (si, af......$ s/, a. Te ~ Tb. Later, we also propose an extension to the incomplete observation setting in which only a censored version of the true state is available in demonstration data. More precisely, instead of the complete state observation sf the demonstration data will only contain š = o(st), where o(-) is projection to lower dimensional subspace. We represent the incomplete demonstration data D (7 J where 7 (31, of ii,..., St. af))
We make the following assumption about the behavior policy Tb- Assumption 1. In the initial episodes of learning, Ea-an [A (s, a)] 2 B > O, Vs, where n. is the learning policy employed by the algorithm in the kth episode of learning.
3

---

Published as a conference paper at ICLR 2022
Intuitively, the above assumption implies that taking action according to Tb will provide higher
advantage than taking actions according to Fy because Eawaz [A (s, a)] = 0. This is a reasonable assumption, since the behavior policies currently in use in many systems are likely to perform much better than an untrained policy. We also note that a similar assumption is made by Kang et al. (2018) to formalize the notion of useful behavior policy. We emphasize that T. need not be the optimal policy, and Tb could be such that (R(..) < (R(T)). Our proposed algorithm learns a policy that performs better than nh, through online learning with guided exploration.
ALGORITHM AND PERFORMANCE GUARANTEE
3.1 ALGORITHM
In this section, we describe our proposed LOGO algorithm. Each iteration of our algorithm consists of two steps, namely policy improvement step and a policy guidance step. In the following. T denotes the policy after k iterations.
Step 1:
Policy Improvement: In this step, the LOGO algorithm performs a one step policy improve- ment using the Trust Region Policy Optimization (TRPO) approach (Schulman et al. 2015). This can be expressed as
Tk+1/2 arg max Bowd [A (s, a)]
s.t. DEL (T, TT)) S 8.
(1)
The TRPO update finds the policy T/-1/2 that maximizes the objective while constraining this maximizing policy to be within the trust region around = defined as {Ι PRL (TT, m&) S 8). The TRPO approach provides a provable guarantee on the performance improvement, as stated in Proposition I
We omit the details this is a standard approach in the literature.
While the TRPO approach provides an efficient online learning strategy in the dense reward setting, it fails when the reward structure is sparse. In particular, it fails to achieve any significant performance improvement for a very large number of episodes in the initial phase of learning due to the lack of useful reward feedback that is necessary for efficient exploration. We propose to overcome this challenge by providing policy guidance using offline demonstration data after each step of the TRPO update. The policy guidance step given
Step 2: Policy Guidance:
Tk+11 = arg min (T, s.t.
DKL. (TT, T1411/) S 6z.
(2)
Intuitively, the policy guidance step aids learning by selecting the policy T in the direction of the behavior policy Tb- This is achieved by finding a policy that minimizes the KL divergence w.r.t. Tbs but at the same time lies inside the trust region around T defined as {T : DR (TT 1k+1/2) S 6x). This trust region-barc based policy guidance is the key idea that distinguishes LOGO from other approaches. In particular, this approach gives LOGO two unique advantages over other state-of-the-ar algorithms.
First, unlike imitation learning that tries to mimic the behavior policy by directly minimizing the distance (typically KL/IS divergence) between it and the current policy, LOGO only uses the behavior policy to guide initial exploration. This is achieved by starting the guidance step with a large value of the trust region oR, and gradually decaying it according Appendix [F) as learning progresses. This novel approach of trust region based policy improvement and policy guidance enables LOGO to both learn faster in the initial phase by exploiting demonstration data, and to converge to better policy than the sub-optimal behavior policy.
adaptive update rule (specified in
Second, from an implementation perspective, the trust region based approach allows approximate the objective by surrogate function that is amenable to sample based learning (see Proposition 2 This allows to implement LOGO in the manner of two TRPO-like steps, enabling us leverage the TRPO code base. Details are given Section
3.2 PERFORMANCE GUARANTEE
We derive a lower bound on the performance improvement, JI(T Jn(mk) for LOGO in each learning episode. We analyze the policy improvement step and policy guidance step separately. We begin with the performance improvement due to the TRPO update (policy improvement step). The following result and its analysis are standard in the literature and omit the details and the proof.
4

---

Published as a conference paper at ICLR 2022
Proposition 1 (Proposition (Achiam et al. 2017 Let xy and TR41/s related by I Then,
JR( - JR(TT) 2 -V287ER (1)))),
where 6n, = max 14 as)
We now give the following result which can be used obtain lower bound on the performance improvement due to the policy guidance step.
Lemma 1. Let T be a policy that satisfies Assumption L Then, for any policy n,
Jn(#) JR( 2 (1-7)-BB- (1-7) - 2D
2 (T,
(4)
Where En, = maxx, [AR =h FR (s,
Remark Lemma[]ake gives intuitive explanation for our proposed policy guidance step. The policies used in the initill phase of the learning can be far from the optimal. So, it is reasonable to assume that Ty (and hence 1/+1 satisfies Assumption D in the initial phase of learning. Then, minimizing DiL (T, Tb) can get non-negative lower bound in 4 which will imply that the performance of T is better than T/+1/2- This is indeed the idea behind the policy guidance step.
Combining the results of Proposition and Lemma I and with some more analysis, we get the following performance improvement guarantee for the LOGO algorithm.
Theorem 1.
Let Th and k/+1/2 related by ( and let T4+1/2 and Th+1 are related by 2 Let ER,
and 8/22/22 be as defined in Proposition and Lemma respectively, Let Rmax = maxe.a a R((sa))) (i) If T 1/2 satisfies Assumption 0 then
V28TER.
CR,841/2
JR(TK+1) - (R(TT)= Z (1 7}² (1 (1 (1 (1->) V (T4+1,
+
(5)
(ii) If T241/2 does satisfy Assumption 7 then
JR( JR(TK) 2 -(v -VV22 + 3lmmaxx) -7
(6)
Remark the initial phase of learning when the baseline policy is better than the current policy, the policy guidance step can add non-negative term to the standard lower bound obtained by the TRPO approach, as shown in (5). This indicates faster learning in the initial phase as compared to the naive TRPO approach. The policy improvement guarantee during the later phase of learning is given by (6). which shows that LOGO achieves similar performance guarantee as TRPO when 5y = 0(v5). We ensure this by decreasing the value of 6 as the learning progresses. Thus, Theorem Cclearly shows the key advantage of the LOGO algorithm achieved by the novel combination of the policy improvement step and the policy guidance step.
4 PRACTICAL ALGORITHM
We first develop an approximation to the policy guidance step (2) that is amenable to sample-based learning and can scale to policies paramaterized by neural networks. This step involves minimizing (TT Tb) under a trust region constraint. However, this is not easy to solve directly by a sample- based learning approach, because estimating it requires samples generated according any possible ग, which is clearly infeasible. To overcome this issue, inspired by the surrogate function idea used in the TRPO algoirthm (Schulman et al. 20155) we derive a surrogate function for D (T,, Tb) that can be estimated using only the samples from the policy T
For deriving surrogate function for DR (T, Tb) that is amenable to sample-based learning, we first define the policy dependent reward function C₁ as Cx a) = log( a)/mmss)))))) a)). Using Cr, we can also define the quantities Jc, (f), V Q and A for any policy म, exactly as defined in Section by replacing by C, Using these notations, we now present an interesting result which we call the performance difference lemma for policy dependent reward function.
Lemma 2. For any policies ा and ă.
Jc., (T) Jcs (7)
-1 B Eandd avs (s, + (1 - ->)->>>k. 7). (7)
5

---

Published as a conference paper at ICLR 2022
Note that the above result has additional term, (1 -2- -1 iD ( 7),, compared to the standard performance difference lemma (Kakade & Langford 2002). In addition to being useful for analyzing our algorithm, we believe that the above result may also be of independent interest.
We now make an interesting observation that Jc. (x) = (1 D (T, Tb) (proof is given in the Appendix), which can be used with Lemma 2 to derive the surrogate function given below.
Proposition 2.
Let T//11/2 be given in around 4/41/2 defined as {I KL (T,, Tu+ ) I 5kh, we have
Then, for any policy that lies in the trust region
DiL (T, S az Bnnd (s,) LA Me+1/2 (8, + +7(1 -1) --- Ex,, Vx. 20 + dg. 2+1/22
(8)
where ork = DKL Mx+1/2 (O (TT44/2s mb), mn,k = maxs,a |Ac, C (8,
x-1/22
Now, instead of minimizing D&L (I, Tb), we will minimize the upper bound as surrogate objective function. Note that only one term in this surrogate objective function depends on π, and that term can be estimated using only samples from 4/41/2- Thus, will approximate the policy guidance step as
Tk+1 argmin E ggdd "swd"k+'p 1/2 (na
(A 12 Cre43/ (s, a)]
s.t. DRA (m, Tk+1/2) S 8
(9)
Since DRE is difficult to implement in practice, we will further approximate it by average KI. divergence DL Daa-1/a (TT, 1/2). We note that this standard approach used in the TRPO algorithm.
We can now put steps and 9 together to obtain the full algorithm. However, as the policies are represented by large neural networks, solving them exactly is challenging. Hence. for sufficiently small 6, ōg. we can further approximate the objective functions and constraints by a Taylor series expansion to obtain readily implementable update equations. This a standard approach (Schulman) et al. 2015 Achiam et al. 2017), and we only present the final result below.
Consider the class of policies (no 8 E Q} where 0 the parameter of the policy. Let 6₃ be the parameter corresponding to policy Ty- Then, the Taylor series expansion-based approximate solution of (T) and (9) yields the final form of LOGO as follows:
26 F
25 0k+1 = -8241/0 V hi LE
0x41/2 = 0% +
F 9k:
L
(10)
T
gk
where 9k
Vo Bendd (o [A (8, a)), FR
vg
DRi (To, and h
Vo E and 1/2 (a [ACr (s, La = V3 DKL
Th+ (TT0, 7/+
1//2
While it is straightforward to compute A7, for any policy when the form of the baseline policy my is known, it is more challenging when only the demonstration data DD generated according to Th is available. We overcome this challenge by training a discriminator using the demonstration data D and the data D 1//1 generated by the policy T (Goodfellow et al. 2014 Ho & Ermon 2016 Kang et al. 2018 8) that will approximate the policy dependent reward function - Further details on
training this discriminator and a concise form of the algorithm are given in Appendix D
4.1 EXTENSION TO INCOMPLETE OBSERVATION SETTING
We now discuss how to extend LOGO to the setting where the behavior policy data contains only incomplete state observations. For instance, consider the problem of learning a policy for a mobile robot to reach a target point without colliding with any obstacles. Collision avoidance requires sensing the presence of obstacles, which is typically achieved by camera/l.idar sensors. Learning an RL policy for this problem requires high fidelity simulator that models various sensors and their interaction with the dynamics. Such high fidelity simulators are, however, typically slow and difficult to parallelize, and training an RL policy using such a simulator in a sparse reward environment can be very time consuming and computationally expensive. Often, it is much easier to train an RL policy for the trajectory tracking problem using only a simple kinematics simulator model which only has a lower dimensional state space compared to the original problem. In particular, such a kinematics simulator will only have the position and velocity of the robot as the state instead of the true state with

---

Published as a conference paper at ICLR 2022
high dimensional camera/Lidar image. Can the demonstration data or behavior policy from this low dimensional simulator to accelerate the RL training in a high dimensionalfiddlityy simulator in sparse reward environments? We answer this question affirmatively using simple idea to extend LOGO into such incomplete observation settings.
Since the behavior policy appears in LOGO only through a policy dependent reward function Cx(s,a) = log(m(s, a)/ (s, a)), we propose to replace this with a form that can handle the incomplete observation setting. For any state E S, let & o(s) be its projection to a lower dimensional state space S. As explained in the mobile robot example above, a behavior policy Tb in the incomplete observation setting can be interpreted as mapping from S the set of prob- ability distributions over A. We can then replace C= in the LOGO algorithm with C₂ (s, a) defined as Č* a) = log(n(s; a) ih (o((), a)). When only the demonstration data with incomplete observation is available instead of the representation of the behavior policy, we propose to train be the demonstration with incomplete observation, Then we train discriminator using D and D, to estimate Č,,
discriminator to estimate Cx. Let D = { i=
where 7' af...
5 EXPERIMENTS
We now evaluate LOGO from two perspectives: (i) Can LOGO learn near-optimally in a sparse reward environment when guided by demonstration data generated by a sub-optimal policy? (ii) Can LOGO retain near-optimal performance when guided by sub-optimal and incomplete demonstration data with sparse rewards? We perform an exhaustive performance analysis of LOGO, first through simulations under four standard (sparsified) environments on the widely used MuJoCo platform (Todorov et al.) 2012). Next, we conduct simulations on the Gazebo simulator (Koenig & Howard 2004) using LOGO for way-point tracking by a robot in environments with and without obstacles, with the only reward being attainment of way points. Finally, we transfer the trained models to real-world TurtleBot robot (Amsters & Slacts 2019) to demonstrate LOGO in realistic setting.
LOGO
TR-O
DAPG
Behavier
Export
Hoppar-22
1360
Sanicles
Sanslos
5amp
(a) Evaluation on MuloCo with full offline observation.
Hoppor-a +2
P0herbhh12
Walker--2
6.0
200
2.5 24
Semroca
Samples
(b) Evaluation on MuloCo with incomplete offline observation.
Figure I: Evaluation of algorithms on four sparse reward MuJoCo environments with complete (Ia) and incomplete (Tb) offline data. The solid line corresponds to the mean over five trials with different seeds and the shaded region corresponds to the standard deviation over the trials.
In what follows, we present a summary of our experiments and results. We remark that LOGO is relatively easy to implement and train, since its two TRPO-like steps imply that we can utilize much of TRPO code base.
code hase and video of the TurtleBot experiments:
com/ wtt.://ee/eeeiiiieeeeeeeeeeea
7

---

Published as a conference paper at ICLR 2022
5.1 MUJoCo SIMULATIONS
In our first set of experiments, we consider four standard environments using the MuJoCo platform. We introduce sparsity by reducing the events which reward feedback is provided. Specifically, for Hopper, HalfCheetah and Walker2d, we provide a reward of +1 at each time only after the agent moves forward over 2, 20, and 2 units from its initial position, respectively. For InvertedDoublePen- dulum, we introduce sparsity by providing a reward only at the end of the episode.
Besides LOGO, our candidate algorithms are as follows: (i) Expert: We train TRPO in the dense reward environment to provide the optimal baseline, (ii) Behavior: We use a partially trained expert that is still a sub-optimal stage of learning to provide behavior data, (iv) GAIL: We use Generative Adversarial Imitation Learning (Ho & Ermon 2016), which attempts to imitate the Behavior policy (iii) TRPO: We directly use TRPO the sparse reward setting without any guidance from behavior data (iv) POID: We use Policy Optimization from Demonstration (Kang et al. 2018) as heuristic approach to exploiting behavior data. (v) BC-TRPO: We warm start TRPO by performing behavior cloning (BC) on the sub-optimal behavior data. (vi) DAPG: We use Demo Augmented Policy Gradient (Rajeswaran et al., 2018) which warm starts Natural Policy Gradient (NPG) algorithm using BC, and fine tunes it online using behavior data in heuristic manner. Note that for all algorithms, we evaluate the final performance in the corresponding dense reward environment provided by OpenAI Gym, which provides a standardized way of comparing their relative merits.
Setting 1: Sparse Rewards, We compare the performance of our candidate algorithms in the sparse reward setting in Figure [a] which illustrates their rewards during training. As expected. TRPO fails to make much meaningful progress during training, while GAIL can at best attain the same performance as the sub-optimal behavior policy. While BC-TRPO benefits from warm starting, fails to learn later due to the absence of online guidance as in the case of LOGO. POfD and LOGO both use the behavior data to boot strap learning. POID suffers from the fact that it is influenced throughout the learning process by the behavior data, which prevents it from learning the best policy. However, LOGO's 's nuanced exploration using the demonstration data only as guidance enables it to quickly attain optimality. LOGO outperforms DAPG in all but one environment.
Sparse Rewards and Incomplete State. We next consider sparse rewards along with reducing the revealed state dimensions in the behavior data of Setting 1. These state dimensions are selected by eliminating state dimensions revealed until the expert TRPO with dense rewards starts seeing reduced performance. This ensures that we are pruning valuable information. Since GAIL, POID and LOGO all utilize the behavior data, we use the approach of projecting the appropriate reward functions that depend on the behavior data into a lower dimensional state space described in Section 4.1 We emphasize that BC-TRPO and DAPG cannot be extended to this setting as they require full state information for warm starting (in BC-TRPO and DAPG) and online fine tuning (in DAPG). We see from Figure 1b that LOGO is still capable of attaining good performance, although training duration increased. We will further explore the value of being able utilize behavior data with such incomplete state information in robot experiments in the next subsection.
Setting 2:
5.2 TURTLEBOT EXPERIMENTS
We now evaluate the performance of LOGO in a real-world using TurtleBot, a two wheeled differential drive robot (Amsters& & Slacts 2019). We train policies for two tasks, (i) Waypoint tracking and (ii) Obstacle avoidance, on Gazebo, a high fidelity 3D robotics simulator.
The goal is to train policy that takes the robot an arbitrary waypoint within I meter of its current position in an episode of 20 seconds. The episode concludes when the robot either reaches the waypont or the episode timer expires. The state space of the agent are its relative I, y. coordinates and orientation á to the waypoint. The actions are its linear and angular velocities. The agent receives a sparse reward of +1 if it reaches the waypoint, and O otherwise. We created a sub-optimal Behavior policy by training TRPO with dense rewards on our own low fidelity kinematic model Python-based simulator with the same state and action space. While it shows reasonable waypoint tracking, the trajectories that it generates in Gazebo are inefficient, and its real-world waypoint tracking is poor (Figures 2 (d)-(f).. As expected, TRPO shows poor performance in this sparse reward setting and often does not attain the desired waypoint before the episode concludes, giving the impression of aimless circling seen in Figure 2(f). LOGO is able to effectively utilize the Behavior policy and shows excellent waypoint tracking seen in Figures 2 (f)-
Task 1: Waypoint Tracking:
8

---

Published as a conference paper at ICLR 2022
(a) Gazebo setup
(b) Real-world setup
(c) Real-world 2D Lidar
Waypoint Tinkkigg
125
1990
Ovel
Balioma
30
(d) Waypoint tracking training
(e) Gazebo evaluation
(f) evaluation
0.33
635
0.24
-025
-035
-123
Samp
(g) Obstacle avoidance training
(h) Gazebo evaluation
(i) Real-world evaluation
Figure 2: Training and evaluation under Gazebo simulations and world experiments. (a)-(c) show the experiment setup, with the TurtleBot added for visual reference in (c). (d)-(() show waypoint tracking performance. (g)-(i) show waypoint tracking with obstacle avoidance performance.
Task 2: Obstacle Avoidance: The goal and rewards are the same as in Task 1, with penalty of for collision with the obstacle, shown in Figure 2 (a)-(c). The complete state space is now augmented by 2D Lidar scan in addition to coordinates and orientation described Task 1. However, the Behavior policy is still generated via the low fidelity kinematic simulator without the obstacle, i.e., it created on a lower dimensional state space. As seen in Figures 2 (g)-(i), this renders the Behavior policy impractical for Task 2, since it almost always hits the obstacle in both Gazebo and the real- world. However, it does possess information on how to track a waypoint, and when combined with the full state information, this nugget utilized very effectively by LOGO to learn a viable policy as seen in Figures (i). Further, TRPO in this sparse reward setting does poorly and often collides with the obstacle in real-world experiments as seen in Figure 2 (i).
CONCLUSION
In this paper, we studied the problem of designing RL algorithms for problems in which only sparse reward feedback provided, but offline data collected from a sub-optimal behavior policy, possibly with incomplete state information is also available. Our key insight was that by dividing the training problem into two steps (i) policy improvement and (ii) policy guidance using the offline data, each using the concept of trust region based policy optimization, we can both obtain principled policy improvement and desired alignment with the behavior policy. We designed an algorithm entitled LOGO around this insight and showed how it be instantiated in two TRPO-likke steps. We both proved analytically that LOGO can exploit the advantage of the behavior policy, as well as validated its performance through both extensive simulations and illustrative real-world experiments.

---

Published as a conference paper at ICLR 2022
7 ACKNOWLEDGEMENT
This work was supported in part by the National Science Foundation (NSF) grants NSF-CRI-CPS- 1850206,
NSF-CPS-203896 and NSF-CNS 1955696, and U.S. Army Research Office (ARO) grant W911NF-19 19-1--6711--677. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect views of the sponsoring agencies.
CARREEREECC-204555
REFERENCES
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, pp. 22-31. PMLR, 2017.
Robin Amsters and Peter Slaets. Turtlebot as a robotics education platform. In Robotics in Education Current Research and Innovations, Proceedings of the 10th RiF, Advances in Intelligent Systems and Computing, pp. 170-181, 2019.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pp. 2052-200200000 2062. PMLR, 2019.
Yang Gao, Huazhe Xu, Lin, Fisher Yu, Sergey Levine, and Trevor Darrell. Reinforcement learning from imperfect demonstrations. arXiv preprint arXiv: /802.00/0 2018.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, pp. 2672-2680, 2014.
Çaglar Gülçehre, Tom Le Paine, Bobak Shahriari, Misha Denil, Matt Hoffman, Hubert Soyer, Richard Tanburn, Steven Kapturowski, Neil C. Rabinowitz, Duncan Williams, Gabriel Barth-Maron, Ziyu Wang, Nando de Freitas, and Worlds Team. Making efficient use of demonstrations to solve hard exploration problems. International Conference on Learning Representations, ICLR, 2020.
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. Conference on Robot Learning, pp. 1025-1037, 2019.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning. JCML, pp. 1865, 2018.
Todd Hester, Matej Vecerík, Olivier Pietquin, Mare Lanetot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, Gabriel Dulac-Amold, John Agapiou, Joel Z. Leibo, and Audrunas Gruslys. Deep q-learning from demonstrations. In AAAI Conference Artificial Intelligence, pp. 3223-3230, 2018.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems, pp. 4573, 2016.
Mingxuan Jing, Xiaojian Ma, Wenbing Huang, Fuchun Sun, Chao Yang, Bin Fang, and Huaping Liu. Reinforcement learning from imperfect demonstrations under soft expert guidance. In AAAI Conference Artificial Intelligence, volume 34, pp- 5109-5116, 2020.
Sham M. Kakade and John Langford. Approximately optimal approximate reinforcement learning.
In International Conference Machine Learning, pp. 267-274, 2002.
Bingyi Kang, Zequn Jie, and Jiashi Feng. Policy optimization with demonstrations. International Conference Machine Learning, pp. 2469-2478, 2018.
Beomjoon Kim, demonstrations. In Advances in Neural Information Processing Systems, pp. 2859-28677 2013.
Farahmand, Joelle Pineau, and Doina Precup. Learning from limited
10

---

Published as a conference paper at ICLR 2022
Nathan P. Koenig and Andrew Howard. Design and use paradigms for gazebo, open-source multi-robot simulator. In IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2004.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-po q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32:11784-11799 2019.
Gabriele Libardi, Gianni De Fabritiis, and Sebastian Dittert. Guided exploration with proximal policy optimization using a single demonstration. International Conference on Machine Learning, pp- 6611-6620, 2021.
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations ICLR, 2016.
Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojeiech Zaremba, and Pieter Abbeel. Overcom- ing exploration in reinforcement learning with demonstrations. In IEEE International Conference on Robotics and Automation (ICRA), pp. 6292-6299, 2018.
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006. 09359, 2020.
Andrew Y. Ng and Stuart J. Russell. Algorithms for inverse reinforcement learning. In International Conference on Machine Learning (ICML), pp. 663-670, 2000.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito,o Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high high-performance deep learning library. In Advances in Neural Information Processing Systems, pp. 8024-8035. 2019.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. In Robotics: Science and Systems, 2018.
Stéphane Ross, Geoffrey J. Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In International Conference on Artificial Intelligence and Statistics, AISTATS, pp. 627-635, 2011.
Stefan Schaal et al. Learning from demonstration. Advances in neural information processing systems, pp. 1040-1046, 1997.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897, 2015.
Noah Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmalcki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavior modelling priors for offline reinforcement learning. In International Conference on Learning Representations (ICLR), 2020. URL pettps://openreee
Tke
Stanford Artificial Intelligence Laboratory et al. Robotic operating system. URL https://www
ros. org
Umar Syed, Michael Bowling, and Robert E Schapire. Apprenticeship learning using linear program- ming. International Conference on Machine Learning, pp. 1032-10399 1039, 2008
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In JEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033, 2012.
Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. arXiy preprint arXiv:1707.08817, 2017.
11

---

Published as a conference paper at ICLR 2022
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint ar111:191..11311, 2019a.
Yuch-Hua Wu, Nontawat Charoenphakdee, Han Bao, Voot Tangkaratt, and Masashi Sugiyama. Imita- tion learning from imperfect demonstration. In International Conference on Machine Learning, pp. 6818-6827, 2019b.
Brian D. Ziebart, Andrew L. Maas, J. Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. AAAI Conference on Artificial Intelligence, pp. 1433-1438, 2008.
12

---

Published as a conference paper at ICLR 2022
A USEFUL TECHNICAL RESULTS
We will use the well known Performance Difference Lemma in our analysis Kakade & Langford 2002
Lemma (Performance Difference Lemma). For any two policies and n.
(n( [n( (1 -7) Bgndd a~aa(s ::) (4, a)]
The following result gives a bound on the difference between discounted state visitation distributions of two policies in terms of their average total variation difference.
Lemma (Lemma 3, (Achiam et al. 2017)). For any two policies a and n²,
d d= 10
11, 2 - -) Div((T,)) x')
We will use the following well known result.
Lemma 5. For any two policies and n', Div(2,m') S VD (TT, x2)/2.
Proof. From Pinsker's inequality, (r 9) S v (PL 2)/2 for any distributions p. q- Com- bining this with Jensen's inequality, we get Di (TT, x²) I v DRR, (z, 7//2.
We will use the following result which is widely used in literature. We provide the proof for completeness.
Lemma 6. Given any policy and function fx : A + R,
E
E
=
= (1-7)-1 Bands (f
Proof.
E
Σ
Y (n a:)] = Σ>
=. PP(ss =.a = (s.
1=
8,0 1=0
"P(s = () (x00)s,))=<((ssa m(s Σ 'P((s s/
8,0 1=0
(1-7)-1
- aa (nx
-
B PROOF OF THE RESULTS IN SECTION 3
B. PROOF LEMMA
Proof. Starting from the performance difference lemma.
JR( JR( = =(1 (1-7)- Exndd aova((,)))
(A Fet1/a (s,a))
= =1 Σ (s)
(s) Σ (b, A)
Th 1/1 (s,a)
+ (1 - 7) 1 Σ (s)
(T((, a) - a)(s,a))A Fet 1/1 (2, a)
2 (1-7)-13 + (1 - - Σ (s) Σ (T( - (s, t0t (s,a)
13

---

Published as a conference paper at ICLR 2022
(6) _ (1 (1--))1B (1-7)-1 <n, 2 (TT mb)
(1-?)- " #B-
-) <R,K+1/2 2D (H,,
where (a) follows from the premise that T4+1/2 satisfies Assumption
(b) is obtained by denoting
m R
*=** a)). and (c) follows from Lemma
CR,k+1/2
B.2 PROOF THEOREM
We will first prove the following result.
Lemma 7, For any two policies и and ft,
32mxx
-n(m)) S
DTV (x, (z
7)
where Rmax = maxe, af(s
Proof. We have,
JR(T) Jn((F= =
-( -11-7)-1 Σ
(()))) - [ -
8,0
8,0
= =(1 (1-1)) (1-7)- (() Σ ())
8,0
8,0
+ (1-7)
th
Σ 8,0
(s))(ss
8,0
(1 7) -1 Ramax add d" +11 () Remax D.max (π.
(4)
(1 7) Rimax 27 - (1- Di (,,, #) + (1 -7)-- Rmax DTV (I,, ()
3R
$ (1 7)2
D4P (TT, #),
where (a) follows from Lemma 4 The lower bound also be obtained in similar way. Combining
both, we obtain the desired result.
Proof of Theorem 7 (1) Tk satisfies Assumption then 55) follows immediately by combining the results of Proposition D and Lemmal (it) If 1/=1 does not satisfy Assumption we can directly bound JR(T - ( without invoking Lemma]] [
(and hence the implicit assumption that moving towards m, improves the policy).
Use Lemma
7 we get
3Rma DIV (T&+1. Tx41/)) 2 3Rmax
Jn(T))1) Jn((T444/)) 2 (1 7)2
85.
(11)
(1 )-
where the last inequality follows from the fact D (1 m241/a S 6g according to the policy guidance step (2).
Now, combining the result of Proposition and the above inequality, we obtain (6).
C PROOF OF THE RESULTS IN SECTION
C.1 PROOF OF LEMMA
Proof. We follow the proof technique of the performance difference lemma (PDL) with some modifications to get the result. For any given initial state 8,
Vg (s) vē. (s) = Erari sp=s
Cr az) - V.3, (s)
14

---

Published as a conference paper at ICLR 2022
Byyx 80
> (Cr(s), + +V, V2, (s:)) -V, (6)
(0) E E m0=
(C+(s,,, + vV8. (9++1) - V8, (s:))
E B ===
7' (C=(st, (r) + (V2,()0411 v2.())))
t=0
E v (C (= at)]]
(12)
where (a) is obtained by rearranging the summation and telescoping. The first summation in can now be handled as in the proof of the PDL, and we omit the details. So, we get
E
(C+(s,)) + VV, ($++1) - V&, (s,))] = (1 - 7)- E a (s. (13)
The second summation can be written as
T (8,,
म (st); )]
Ey
> (Cx ar) - Ca (8t, at))] = Byas
y (log
llog
(b az)
Tb( as)
t=
1=0
Tse, de) (b) # (8t; a1)
T(2, 0z) 7(8, (t)
log
(1- Σ d= (8)
r (8, a) log
D = =1 -1 ( #):
(14)
where (b) is obtained by using Lemma
Now, by taking expectation on the both sides of (12) w.r.t the initial state distribution μu, and then substituting 113 and there, we get the desired result.
C.2 PROOF OF PROPOSITION
Lemma 2 provides an approach to evaluate the infinite horizon return (x) of policy x using the infinite horizon return Jc, (f) and advantage AC, of policy न. Unfortunately, the RHS of (7) contains two terms that requires taking expectation w.r.t. d which means that empirical estimation requires samples according to π (that are unavailable). This is well known issue in the policy gradient literature and is addressed by considering an approximation by replacing the expectation w.r.t. d= by d (Kakade & Langford [2002) Schulman et al. 2015). We follow the same approach with minor modifications in the context of our problem.
We will first prove the following lemmas. Lemma 8. For any two policies π and मt,
Jc,(m] - Jca (#) S (1 () Eandy ava (8,
+ (1 7)2
V27c
(1 (1-7) (T,+
1
DRLx 7),
where e; max,,0 |A,, a)}.
Proof. We will separately bound the two terms on the RHS of (7) in Lemma Firstly,
E Egentr (an (s, a)] = Σ d' Σ ((, ass,a) A) (s)
Σ t (8) Σ As) (s, + (dd d3 dd(s)) Σ (5 (s,
15

---

Published as a conference paper at ICLR 2022
Band amr(s, LA (s, a)] + ||d" d"
Rgnd an A (8, a)) + 2y(1 - -) <* (T,
E Ba am [4 (s, a)] + 22y(1 - -) (
(15)
where (a) follows from Lemma and (6) follows from Lemma Secondly, Di (T,, result.
5
S Dmax (T, Æ). Using this fact and the inequality (15) 7 we get the desired
We now make an interesting observation that Jc., (π) is scaled version of the average KLL divergence between π and Tb- We formally state this result below.
Lemma 9, For any arbitrary policy n,
Jc, (T) = E
y (x), =1 -1->)) " (T)
(16)
t=
Proof. We have,
x
Jc, (T) B,ax] Σ
y C (s), _0 (1-7)- Σ d" (
t=0
<<s a) (s,
-1--)) Σ d" (s) Σ T a)
(1 -1) (11-))'DR,,A1 mb),
a
where (a) follows from Lemma
6
Proof of Proposition 2 policy and म. we get
Using the result of Lemma 9 the inequality given by Lemma for any
V27e2
D (TL mb) DKL (M, mb) + Rond anx(e) (8, +
V (T + DRL (T, #).
(1 (1-
Now, replacing स by 7/41/22 we get
DEL (T, S or Band (nt((s Caparp A
Matl/2 (s)
V2 /
DKL (1,, mx43/s) DKL (IT, T441/)),
Tet
(17)
(1-
where on Tu+l/2 (TK41/21 Tb), En max.,0 A (s,a))-
Now, for any æ that lies in the trust region {Tt
DRL (It, T841/2)
dx), have
D D=+ (T,, T441/2) S DRL (T, Tepys) I 6g. Using this in 17 gives the final result.
D DETAILS OF THE PRACTICAL ALGORITHM
As explained in Section when Nb is known, obtaining C=(s a) = log(m(ss a)/ (s, a)) and estimating A2. is straightforward. Hence, LOOO can perform the policy improvement and policy guidance step according to the update equations given in (1O) by directly using тb- When the form of Th is unknown and we only have access to the demonstration data generated according to m. we have to estimate (s,,, using D and Dr, where D, is the trajectory data generated according to
a)) directly, we make use of the one-to-one correspon- dence between policy and its discounted state-action visitation distribution defined pT d" (s) a) (Syed et al., 2008). More precisely, we estimate log(p" (s, a)/p" (s,a)) using and Dz instead of estimating (s, a)/m (s, ()) directly. We can then use the powerful framework of
Instead of estimating log(T(s, (s,
(s. a)
16

---

Published as a conference paper at ICLR 2022
generative adversarial networks (GAN) Goodffllow et al. 2014) to estimate this quantity. This can be achieved by training a discriminator function
[0, 1] as
max B
((s,))=p77 (log ((s, a)] + (s.a)>=p- [og(1 B(s,a)))-
(18)
We note that the GAN-based approach to estimate a distance metric between p" and pT is popular in the imitation learning literature (Ho & Ermon 2016 Kang et 2018).
The optimal discriminator for the above problem given by (* a) (=b(e,a)+p (s,0)) Goodfellow
(s.a)
ct al. 2014 Proposition 1). Hence, given the discriminator function B obtained after training to solve (T)) C+ (s, a.) log (s,, a) in the LOGO algorithm, which provides approximation to the quantity of interest. As shown in Section 5 this approximation yields excellent results in practice.
We summarize the LOGO algorithm below.
Algorithm 1 LOGO Algorithm
1: Initialization: Initial policy To, Demonstration data D behavior policy Tb
2: for k ===0 0, 1, 2, do
3: Collect (s, a, s') ~ my and store in DTT.
4: if m. is known then
5: CTA (s,
log (og(=*(s, (s.aa)a./sss.)
6: else
7:
Train a discriminator B(s, a) according (88) using 'D and Di,,. Cr (s,a) = log B (s,
8:
9:
end if'
10:
Estimate Ob and F& using DHr.
28
11:
Perform policy improvement step: 0 Okti/2 02 +
9he Fx
9k
le
9k.
12:
Decay dy (according to the adaptive rule described in Appendix Estimate h2 and La using Cr. and 1 1/2
13:
28
14:
Perform policy guidance step: 0x+1 = O V
Li
hK
1T Li
end for
E DETAILS OF TURTLEBOT EXPERIMENTS
We evaluate the performance of LOGO in real-world setting using TurtleBot, two wheeled differential drive robot (Amsters Slacts 2019). We consider different simulators for training our policy. The first one is a simple kinematics- based low fidelity simulator. The second one is sophisticated physics-based Gazebo simulator (Koenig & Howard, 2004) with an accurate model of the TurtleBot.
We use the low fidelity simulator to get a sub-optimal Behavior policy by training TRPO with dense rewards. The details are given in Section E.I] Gazebo simulator used for implementing the LOGO algorithm sparse reward setting with complete and incomplete observations. The details are given in Section E.2
E. Low FIDELITY KINEMATICS SIMULATOR
We design the low fidelity simulator using the OpenAl gym framework where the dynamics are governed by the following equations
x++1 2, + Uy cos(oos cOs(),)), Dr+1 = Vr U, sin, oc+1 02 + Wes
where 3z, Be the coordinates of the bot, U2 is the linear velocity, Wi is the angular velocity, and d is its yaw calculated from the quaternion angles, all calculated at time t. is the time discretization factor. We define the state 8, to be the normalized relative position w.r.t. the waypoint,
17

---

Published as a conference paper at ICLR 2022
Le., St ((x, -ww)/G (ye yu Dw)/G.0, 03"), where Tw. Yw are the target coordinates of the waypoint, and d:"² is the target heading to the waypoint at time t. we define the action ae to be the linear and angular velocities, i.., a, [et, wp]. We discretize the action space into 15 actions.
We handcraft a dense reward function as follows,
+10
if Dy Twl S 0.05 and
1ye
shel I 0.05,
Tt =
M 2 G otherwise,
or
1 2 G,
0. 0.166d; - 0.3184 0g]
where de is a combination of cross track and along track emror defined as follows,
de ((xe- xx + (ye yw))) sin"(0r 02) lxz Jwol
+ l Vwl-
E.2 HIGH FIDELITY GAZEBO SIMULATOR
Gazebo is a physics-based simulator with rendering capabilities and can be used for realistic sim- ulation of the environment. is configurable and can be used model robots with multiple joint constraints, actuator physics, gravity and frictional forces and wide range of sensors in indoor as well as outdoor settings. Gazebo facilitates toose-torreal-orlke data collection from the robots and sensor models. Since it runs in real-time, it takes millions of simulation frames for RL agent to learn simple tasks. Gazebo can also speed up simulation by increasing step size. This can however lead to loss of precision. For this project, we the training scheme on Gazebo in almost real time with a simulation step-size of AT 0.001.
The Gazebo simulation setup consists of the differential drive robot (TurtleBot3 Burger) model spawned in either an empty space or custom space with obstacles at a predefined location. A default coordinate grid is setup with respect to the base-link of the robot model. A ROS (Stanford Artificial Intelligence Laboratory et al.) framework is instantiated using a custom-buit OpenAl Gym environment which acts an interface between the proposed RL algorithm and the simulation model. ROS topics are used to capture the state update information of the robot asynchronously in callback driven mechanism. For the purpose of our experiments we use the following ROS topics,
1. / /odom (for zt, Vo. 0₂): Odometry information based on the wheel encoder
2. / cmdd vel (for vr, wy): Linear and angular velocity commands
3. /scan (for obstacle avoidance): Scan values from the Lidar
The /odom and /scan topics are used determine s, while / cmd_vel is used output the required action a;. The state space and action space for waypoint tracking task is similar to state and action space section E.I] For obstacle avoidance tasks, we also include the distance values obtained form the Lidar as a part of the state.
E.3 ROBOT PLATFORM: TURTLEBOT3
We use TurtleBot Amsters & Slacts 2019), an open source differential drive robot equipped with RP Lidar as robotic platform for real-world experiments. We use ROS the middleware to setup the communication framework. The bot transmits its state (position and Lidar information) information over network to intel NUC, which transmits back the corresponding action according to the policy being executed.
E.4 TRAINING: WAYPOINT TRACKING
In navigation problems, we have a global planner that uses a high level map of the bot's surroundings for planning trajectory using waypoints unit-meter distance from each other, while the goal of the bot to achieve these waypoints. To obtain a waypoint tracking scheme, we first train behavior policy the low fidelity simulator using TRPO to reach waypoint approximately one unit distance (Im in the real world) away from its initial position. In each training episode, the bot reset to the origin and waypoint is randomly chosen Tw uniform([-1) 1]). Ya 1. The episode is terminated the robot reaches the waypoint, or if it crosses the training boundary or exceeds the maximum episode length. The behavior policy obtained after training in the low fidelity simulator
18

---

Published as a conference paper at ICLR 2022
is then used in the LOGO algorithm training Gazebo. LOGO is trained in Gazebo with sparse sparse rewards, where a reward +1 provided if the bot reaches the waypoint, and 0 otherwise. We evaluate the trained policy in Gazebo and the real world, by providing series of waypoints to track in order to reach its final goal.
E.5 TRAINING: OBSTACLE AVOIDANCE
We train our bot for obstacle avoidance Gazebo using the behavior policy described the section above. Our goal is to use the skills of waypoint navigation from the behavior policy to guide and learn the skills of obstacle avoidance. The state space includes the Lidar scan values addition to the state space described in section E.1 The / /acan provides 360 values, each of these indicate the distance to the nearest object in sector. For the purpose of our experiments, we use the minimum distance in each 602 sector. This reduces the Lidar data to 6 values. We train algorithm a fixed obstacle for random waypoints. waypoint is generated similar to the previous section. The episode is terminated if same conditions in the previous section are satisfied or if a collision with the obstacle occurs. We demonstrate the performance of our algorithms both in Gazebo well the real-world.
Gazebo with each training episode, the bot is reset to the origin and a
F IMPLEMENTATION DETAILS
We implement all the algorithms in this paper using PyTorch (Paszke al. 2019). For all experiments, we have a two layered (128 x 128) fully connected neural network with tanh activation functions to parameterize our policy and value functions. We use learning rate of 3 10 discount factor ? = 0.99, and TRPO parameter 5 = 0.01. We decay the influence of the behavior policy by decaying 5μ. We start with 6p, and we do decay 5g for the first Kg iterations. For k > Ks. we geometrically decay òū as ô₂ +- add, whenever the average return in the current iteration is greater than the average return the past 10 iterations. The rest of the hyperparameters for MuJoCo simulations, Gazebo simulation, and real-world experiments are given in table I In table 2 we provide details on the demonstration data collected using the behavior policy.
We implement LOGO based publicly available TRPO code base. We implement PofD, TRPO, BC-TRPO, and GAIL by modifying the same code base as used by LOGO. We run DAPG using code base and hyperparameters provided by the authors. For consistency, run all algorithms using the same batch size.
Environment
50 Incomplete Observation
K8
Batch Size
Complete Observation
Hopper-v2
0.01
0.02
0.95
20000
HalfCheetah-v2
0.2
0.1
0.95
50
20000
Walker2d-v2
0.03
0.05
0.95
20000
InvertedDoublePendulum
0.2
0.1
0.9
5000
Waypoint tracking
0.01
0.95
2048
Obstacle avoidance
0.008
0.9
2048
Table 1: Hyperparameters
19

---

Published as a conference paper at ICLR 2022
Average
Environment
Offline
Online S
A
Samples
Episodic Reward
Complete
Incomplete Observation
Observation
Hopper-v2
R
R'
111
R
3869
1369.81
HaliCheetah-v2
R
R44
1277
R
5000
2658.34
Walker2d-v2
R
R10
R27
R
4584
2449.21
InvertedDoubleeenduulu
R
R7
R1
R
183
340.73
Waypoint traking
R
R
15
Policy
1
Obstacle avoidance
R
R
15
Policy
-0.88
Table Demonstration data details
G
RELATED WORK
Offline RL; Recently, there have been many interesting works in the area of offline RL which use offline data to learn policy. In particular, offline RL algorithms such as BEAR (Kumar et al. 2019). BCQ (Fujimoto et al. 2019). ABM (Siegel et al.) 2020), and BRAC (Wu et al., 2019a) focus on learning a policy using only the offline data without any online learning or online fine-tuning. The key idea of these offline RL algorithms is to learn policy that is 'close' to the behavior policy that generated the data via imposing some constraints, to overcome the problem of distribution shift. This approach, however, often results in conservative policies, and hence it is difficult to guarantee significant performance improvement over the behavior policy. Moreover, standard offline RL algorithms are not immediately amenable online fine-tuning, observed in Nair et al. (2020). LOGO is different from the offline RL algorithms in the following two key aspects. First, unlike the offline RL algorithms mentioned before, LOGO an online RL algorithm which uses the offine demonstration data for guiding the online exploration during the initial phase of learning. By virtue of this clever online guidance, LOGO is able to converge to a policy that sigmificantly superior than the sub-optimal hehavior policy that generated the demonstration data. Second, offline RI. algorithms typically require large amount of state-action data with the associated rewards. LOGO only requires small amount demonstration data since it is used only for the guiding the online exploration. Moreover, LOGO requires only the state-action observation and does not need the associated reward data. In many real-world application applications. it may be possible get state-action demonstration data from human demonstration or using a baseline policy. However, is difficult to assign reward values these observation.
Advantage Weighted Actor-Critic (AWAC) algorithm (Nair et al. 2020) propose to accelerate online RL by leveraging offline data. This work is different from our approach in four crucial aspects. (i) AWAC requires offline data with associated rewards whereas LOGO requires only the state-action observations (not the reward data). In many real-world applications, may be possible to get state- action demonstration data from human demonstration or using baseline policy. However, it may be difficult to assign reward values these observations, especially in the sparse reward setting, (ii) AWAC explicitly mentions that it leverages large amounts of offline data whereas LOGO relies on small amount of sub-optimal demonstration data. (iti) LOGO gives a novel and theoretically sound approach using the double trust region structure that provides provable guarantees on its performance. AWAC algorithm does not give any such provable guarantees. (iv) LOGO can be easily extended to the setting with incomplete state information where as AWAC is not immediately amenable to such extension.
There are also many recent works on addressing multiple aspects of IL and LID, including combining IL and RL for long horizon tasks (Gupta et al. 2019). learning from imperfect (Gao et al. 2018 Jing et al., 2020; Wu et al. 2019b) and incomplete (Libardi al.) 2021 Gülçehre al. 2020 demonstrations. Their approaches, performance guarantees and experimental settings are different from that of our problem and proposed solution approach.
20

---

Published as a conference paper at ICLR 2022
Hopper-v2
Hopper-v2
Average episodic reward
Average episodic reward
3000
3000
2000
2000
1000
1000
0.0
0.5
1.0
1.5 Samples
2.0
2.5
3.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
16
Samples
d- 0.85
0.95
d- 0.99
Behavior Expert
K;=1
Kg
=
1000
Bchaviar
Expert
(a) Sensitivity of
(b) Sensitivity of Ks
H SENSITIVITY ANALYSIS OF & AND Kg
We run LOGO for different values of a and Kg by keeping the other parameters fixed on the Hopper- v2 environment. We observe from Figure 3a that the performance of LOGO is sensitive to perturbations in a. When a = 1, we do not decay the influence the behavior data, this results in performance close the behavior data which inline with our intuition. We observe from Figure 3b that LOGO is not sensitive to perturbations in K3 as well. We observe that the performance of LOGO similar for Kg 1, 50, 100. This because we only decay 6 whenever the average return in the current iteration is greater than the average return in the past 10 iterations. We further note that Kü controls the iteration from which we begin decaying the influence of the behavior data, this can be clearly seen for K3 500, 1000 where the performance rises soon as the decaying begins.
21


Reviews:
[{"review_id": "tXtCs64LyO3", "reviewer": "Reviewer_FWYZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well-written and clearly states the problem, motivation and goals, and the approach taken to solve the problem at hand. Formal explanations and methods are presented rigorously and are supplemented with helpful intuitive explanations following each major block of formal mathematical language. The exploration of learning from demonstrations to improve learning in challenging settings (sparse rewards, long horizons, etc) is, in my mind, an important concept that also has ~immediate practical applications and implications, and the method presented here adds to the toolbox of ideas on how this can be approached.\n\n\nMajor concerns / criticisms / discussion\n----------------------------------------\n\n* From a quick read of the Kang et al. 2018 is indeed appears to be very close to the idea (at least in spirit) presented in this work. Authors mention how their approach is fundamentally different in terms of methodological approach, however I think the work would benefit from a bit more detailed exploration of the differences in consequences of applying POfD vs. LOGO. If we imagine a situation where both algorithms were presented with the same data (same random seed, same everything) -- what would be crucial difference between the two policies produced by the two methods? Maybe differences in value estimation? Degree of deviation from the behavior policy? What are the arguments one could present to decide \"I will choose LOGO over POfD, because ...\"?\n\n* Conceptually there does not seem to be anything that would prevent using LOGO in a large-state-space setting, for example using RGB(D) inputs from a robot. Did the authors attempt LOGO in such an environment? Is the \"partial observation\" trick *required* to learn in such an environment or we would see the benefits of LOGO even when working with full state directly? If you did try this then were the results positive or negative, and if you did not, then what are your thoughts on feasibility and performance expectations?\n\n\nQuestions\n---------\n\n* general: How hard would it be to adapt the same idea to other algorithms, that do not have the build-in notion of a trust region? For example SAC and QR-DQN? Or another way to phrase the question: how specific is the proposed method (conceptually) to TRPO. Could including some kind of a penalty based on KL divergence on top of other algorithm also work, or this would be unlikely / highly non-trivial to achieve?\n\n* page 1 par 2: \"natural candidate to begin with is the framework of policy gradient algorithms <...> performs really well in the dense reward setting\" -- Unclear why this is a natural choice while in the previous paragraph we were discussing sparse rewards. Wouldn't it be more natural to first consider something that works well with sparse rewards?\n\n* page 2 par 1: \"Our choice of the TRPO \" -- Why not the superior methods based on similar principles like PPO?\n\n* page 3 assumption 1: What does beta stand for in this equation?\n\n* page 9 \"task 2\": What is the dimension of the state space when lidar data is included?\n\nMinor remarks\n-------------\n* page 1 par 2: \"behavior data might only contain measurements of a subset of the true state\" -- It was a bit unclear why that would be the case. If I understand correctly this isn't something that happens often. Was this included in the manuscript because the authors have observed that LOGO works well for situations like this? If that is the case perhaps the sentence can be reformulated to say that, without the implication that partial state is something that tends to happen often.\n\n* sec 5.1 par 1: \"moves forward over 2, 20, and 2 units from\" -- at typo?", "summary_of_the_paper": "In a regime where we have access to offline behavior data from a suboptimal policy (a heuristic, human demonstrations, etc) we can use that data and \"trust region\"-based methods (TRPO in this work) to nudge exploration in the right direction by keeping the learnable policy close to the behavioral one. While there are several works exploring this general idea, the particular way presented in this work is novel and achieves better results on a set of MuJoCo environment. The approach is also shown to work on a physical robot (Turtlebot).", "summary_of_the_review": "I find the paper's goals clear and interesting and the execution faithful and technically sound. This work makes a curious step in the direction of mixing together online learning and pre-existing policy data and I think would be an interesting result for others who do research in this area. I wouldn't mind seeing this work presented at the conference.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "oJFoRQN67Li", "reviewer": "Reviewer_KhX3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper provides a novel formulation of how to keep an RL policy close to a behavior policy. Instead of directly performing some sort of constrained optimization (maximizing the expected sum of rewards subject to the constraint being $ KL(\\pi||\\pi_b) < \\epsilon $), the authors formulate the problem as another RL problem with the reward $ \\frac{log(\\pi(a|s))}{log(\\pi_b(a|s))} $ while ensuring that the updated policy doesn't stray too far from the current RL policy iterate (with a KL constraint term that has a decaying coefficient). This approach is certainly novel and an intriguing way to optimize under this constraint and the empirical results certainly do appear impressive.\n\nHowever, I do have several concerns.\n1. This paper makes no mention of prior work that has also noted this problem of attempting to constrain the RL policy to remain close to the behavior policy via the same constraint mentioned in this work: $ KL(\\pi||\\pi_b) < \\epsilon $. In the offline RL literature, there has been a large body of work that has attempted to optimize this constraint in different ways: (BEAR, BCQ, ABM, BRAC, MPO). I believe these, and many other works in this area should be included with a discussion of their tradeoffs in the related works section. \n2. Empirically, this work is missing several clear comparisons. One: training a behavior policy using the offline data (do Behavior cloning) and simply initializing TRPO from this policy to bootstrap learning. This is a simple solution used as a baseline in prior work (see RPL) and should be included as a point of comparison. Two: DAPG. This method can in principle use suboptimal data and has been empirically validated to work on challenging sparse reward tasks (though with the use of expert demos). Three: AWAC. This method was designed explicitly for the scenario in which a large suboptimal dataset is provided (see results on D4RL) and a small amount of online finetuning is required. Comparison with this method is critical. Four: Comparison against behavior cloning with reward filtering. (simply train a behavior cloning policy with only 10%/20% of the optimal data as ranked by the return of the offline trajectories). This method has been demonstrated to outperform many recent offline RL algorithms as seen in Table 3 of the Decision Transformer paper. \n3. Empirical evaluation environments. The method is mostly evaluated on relatively simple Gym tasks modified to include sparse rewards. The RL community already has much more challenging sparse reward environments in which similar methods have been tested: the Adroit suite of sparse reward hand manipulation tasks. Evaluating if the proposed method's results are relevant should entail a comparison against prior work (AWAC and DAPG at the very least) on these much more challenging robotic control tasks. \n4. nitpicks: \n* upperbound should be upper bound, lowerbound should be lower bound\n* End of related work: “Their approaches, performance guarantees and experimental settings are different from that of our problem and proposed solution approach.” Please explain how this is the case\n\nCitations:\nBEAR: Aviral Kumar, Justin Fu, George Tucker, and Sergey\nLevine. Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction. In Neural Information Processing\nSystems (NeurIPS), 2019.\n\nBCQ: Scott Fujimoto, David Meger, and Doina Precup. OffPolicy Deep Reinforcement Learning without Exploration.\nIn International Conference on Machine Learning (ICML),\n2019.\n\nABM: Noah Y. Siegel, Jost Tobias Springenberg, Felix\nBerkenkamp, Abbas Abdolmaleki, Michael Neunert,\nThomas Lampe, Roland Hafner, Nicolas Heess, and\nMartin Riedmiller. Keep doing what worked: Behavioral\nmodelling priors for offline reinforcement learning, 2020\n\nBRAC: Yifan Wu, George Tucker, and Ofir Nachum. Behavior\nRegularized Offline Reinforcement Learning. 2020.\n\nMPO: Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval\nTassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a Posteriori Policy Optimisation. In\nInternational Conference on Learning Representations\n(ICLR), pp. 1–19, 2018.\n\nRPL: Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey\nLevine, and Karol Hausman. Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement\nLearning. In Conference on Robot Learning (CoRL),\n2019\n\nDecision Transformer: Chen, Lili, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. \"Decision transformer: Reinforcement learning via sequence modeling.\" \n", "summary_of_the_paper": "This paper aims to improve the performance of RL in sparse reward settings via the guidance of sub-optimal demonstrations. The idea proposed by this work is to modify TRPO to restrict its updates to remain close to the behavior policy that generated the offline dataset, while decaying this constraint over time to enable the RL policy to improve upon the behavior policy in an online fashion. The authors do so by minimizing the KL divergence between the current RL policy and the behavior policy while not moving too far away from the current RL policy. The method is evaluated on openAI gym style tasks as well as a real robot navigation task. ", "summary_of_the_review": "In general, while the work has (at first glance) impressive empirical results as well as a novel formulation of the KL constraint between the RL policy and the behavior policy, there are quite a few concerns that would need to be addressed with direct justification and experimentation in order to recommend acceptance. Hence, I currently recommend rejection, but I am open to updating my score if all of my concerns can be addressed. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "lprVe-hNky", "reviewer": "Reviewer_DFHZ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of using behavior policy to guide the reinforcement learning algorithm is promising given the fact that demonstrations are sparse to learn. While the sparsity can cause slow convergence, it is important to have a rapid algorithm for this kind of problems. The proofs provide lower bounds on reward gain between two consecutive updates. Theorem 1 shows that in the initial phase of learning the lower bound could be positive, where \\beta is larger positive number. I have a concern about this theorem. In the initial phase, the KL divergence of \\pi_k and \\pi_b is also larger given the fact that \\delta_k is decreasing. In this case, the result in Theorem 1 is not always positive which may not indicate faster learning in the initial phase. \n\nIn the experimental section, the proposed LOGO algorithm seems to have the fastest convergence compared to others. Could you give more description about convergence rates comparison from a theoretical point of view?\n", "summary_of_the_paper": "This paper considers the problem of reinforcement learning with sparse reward functions obtained from offline demonstration. The authors propose a trust region policy optimization based algorithm with offline demonstration data for guidance. The proposed LOGO algorithm is proved to be efficient by a theoretical analysis showing the lower bound on the performance improvement. On benchmark datasets, the proposed algorithm performs better than the state-of-the-art approaches. An illustration is shown by implementing LOGO on a mobile robot for trajectory tracking and obstacle avoidance.", "summary_of_the_review": "This paper proposed a novel algorithm to solve reinforcement learning problems with offline demonstration policy. The idea is interesting and nature for many real problems, especially for learning from demonstration problems. A very straightforward update rule is provided by using an estimation of KL divergence. This improves the computational efficiency of the proposed algorithm, which is very promising in real problems.  The Mujoco and TurtleBot experiments verify the performance of the algorithm with both simulation and real experiment. Overall, this paper is well written and the idea is novel and promising.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Tq7ZiCgbQ8", "reviewer": "Reviewer_DXFH", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper studies the problem of how to use demonstration data to better aid reinforcement learning with sparse reward, which is an important task setting that will have a wide impact on application of RL. Prior work in this direction has focused on directly imitating the demos or only using the demonstrated states to constrain exploration. The proposed LOGO algorithm leverages the per-state action distribution as a constraint for guiding policy search. The authors also show that an important advantage of LOGO is that the demonstration data need not cover the entire state space, and the algorithm is capable of matching distributions in down-projected state spaces.  This advantage allows the algorithm to be applicable to a wider range of demonstration types than pure imitation-based algorithms do. The authors performed a theoretical analysis on the algorithm and conducted experiments in a range of different tasks, including a navigation task on a real robot. The writing of this paper is clear and well organized for the readers to follow.\n\n\nOne weakness of the paper is that it misses one important baseline method, which is to warm-start the policy using behavior cloning with demonstration data and then run RL in the similar manner. This baseline uses the same amount of auxiliary data (i.e. the demonstrations) and potentially needs less amount of time interacting with the environment since the supervised BC step is offline. Current baseline methods do not fully leverage the data available in this problem setting. TRPO ignores the demonstrations, GAIL ignores the RL reward, and PofD only uses the state distribution instead of the state-action distribution from the demonstrations. Therefore, adding the baseline described above is important to make an apples-to-apples comparison between BC and using demos as guidance. Apparently warm-start with BC does not offer the generalizability to partially-observed demonstrations and therefore LOGO still has its unique advantages even if it is not able to outperform this baseline\n", "summary_of_the_paper": "This paper presents the learning online with guidance offline (LOGO) algorithm that leverages demonstration data to constrain policy search for reinforcement learning with sparse reward such that the initial exploration phase is guided. Experiments in locomotion tasks in simulated domains and a navigation task on a real robot demonstrates LOGO is more data-efficient compared to a range of baseline methods.\n", "summary_of_the_review": "This paper studies an important problem of how to leverage demonstrations for reinforcement learning and proposes a novel method that is demonstrated to outperform a set of baselines in various domains. It makes the case for how distribution matching should be used instead of pure supervised learning of demonstrations and shows an unique advantage of doing so, which is the capability of dealing with partially observable states. I recommend acceptance of this paper but would love to see another baseline added to the experiments as I mentioned in my review above.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "RWEIi8MYnvE", "reviewer": "Reviewer_FZbN", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper contributes a useful and apparently novel algorithm for an important problem in the application of RL methods in real-world settings. While the algorithm, theory and experimental methodologies seem sound, a lack of comparison with several relevant algorithms makes it difficult to comprehensively assess the strength of the proposed method. Claims for the merit of the proposed method would also be strengthened if the paper included a sensitivity analysis with varying learning schedules for annealing away the guidance policy data. Without these elements, this paper in its current form falls slightly below the bar for inclusion at ICLR. If the authors can furnish these additional results (or provide compelling arguments why they are not necessary), then I would be willing to increase my score for this paper.\n\n**Strengths:** The paper is very clear and well written, and the explanation of the algorithm is easy to follow. The experimental methodologies (selection of simulated experiments, measured metrics etc.) appear to be sound, and it is good to see experiments on a real-world robot which substantiates the claim that this method is useful for real-world RL. The fact that the theoretical version of the algorithm (before sampling based approximations) utilises the trust-region method means that this algorithm can be theoretically analysed, which is a nice property. The main assumption of the method (Assumption 1) is reasonable. In this reviewer’s opinion, one of the best strengths of the proposed LOGO method is that the double-TRPO structure allows this to be easily implemented using existing TRPO codebases. Implementation is often a stumbling point for RL algorithms, and this is a non-trivial strength of the method.\n\n**Weaknesses:** The discussion of related work seems to be missing a few recent papers that might be relevant. These methods should also be represented in the simulated experiments to enable a fair comparison with related approaches. Specifically, in addition to Policy Optimization from Demonstration (POfD) [1] (which is cited and included in the simulated experiments), other relevant methods seem to be Advantage Weighted Actor-Critic (AWAC) [2], (which is cited but not included in experiments - the authors dismiss this method as heuristic based, but this is not a sufficient reason to exclude it from experiments), as well as Learning with supervision from Noisy Demonstrations (LfND) [3]. RL with GAN shaping [4] and the Cycle-of-Learning (CoL) framework [5] also appear to be tackling a similar problem to LOGO.\n\nThe inclusion of GAIL in the simulated experiment and pure TRPO in the simulated and real-world experiment helps situate the results, but neither of these methods could be expected to outperform LOGO given the sparse reward structure and the sub-optimal guidance data the present problem - if anything these methods serve as lower-bounds on performance that LOGO should out-perform. It would also be helpful for the reader to include a comparison with the naive strategy of behaviour cloning the sub-optimal policy, then continuing training with vanilla TRPO for some number of iterations.\n\nAs the authors acknowledge, there is a large body of work in combining online RL with offline demonstrations in various ways, and not every paper can compare against every prior method, however I believe POfD by itself does not allow a comprehensive assessment of the performance of the proposed approach.\n\nMy other concern is that the proposed algorithm depends critically on the learning schedule used to anneal away the offline policy guidance (which is described in appendix F), however as far as I can tell, the paper does not include a sensitivity analysis over this schedule (which potentially leaves the paper open to accusations of cherry picking results). Given that the performance of LOGO critically hinges on this schedule, I believe the paper needs to include an additional experiment showing how LOGO performs under variations of the schedule (e.g. sweeping values for $K_\\delta$ as well as for $\\alpha$).\n\n**Queries and minor points:**\n\n1. Is there anything that means LOGO is specifically useful for the sparse reward setting? Would it help or hinder in a dense reward setting?\n2. Can the notion of the guidance policy ‘offer[ing] an advantage’ (Section 1) be quantified through information theoretic analyses?\n3. Please ensure your charts are fully legible in greyscale - currently Figure 1 is not, and Figure 2 is a little difficult to read in greyscale.\n\n---\n\n[1] Kang, Bingyi, Zequn Jie, and Jiashi Feng. \"Policy optimization with demonstrations.\" International Conference on Machine Learning. PMLR, 2018.\n\n[2] Nair, Ashvin, et al. \"Accelerating online reinforcement learning with offline datasets.\" arXiv preprint arXiv:2006.09359 (2020).\n\n[3] Ning, Kun-Peng, and Sheng-Jun Huang. \"Reinforcement learning with supervision from noisy demonstrations.\" arXiv preprint arXiv:2006.07808 (2020).\n\n[4] Wu, Yuchen, Melissa Mozifian, and Florian Shkurti. \"Shaping rewards for reinforcement learning with imperfect demonstrations using generative models.\" 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021.\n\n[5] Goecks, Vinicius G., et al. \"Integrating behavior cloning and reinforcement learning for improved performance in dense and sparse reward environments.\" arXiv preprint arXiv:1910.04281 (2019).\n", "summary_of_the_paper": "This paper presents ‘LOGO’, an extension of the TRPO algorithm which enables additional learning guidance from offline, sub-optimal (possibly incomplete observation) demonstration data. By annealing away the learning contribution from the sub-optimal guidance policy data during training (with a learning schedule and corresponding hyper-parameter), LOGO utilises this data for guidance, rather than directly imitating it. Furthermore, because LOGO utilises the trust-region methodology, the authors are able to provide a theoretical analysis and lower bound on performance improvement each episode. The method shows promising performance on several MuJoCo continuous control tasks, as well as in a Gazebo TurtleBot simulation, which is also able to be transferred to a real-world robot.", "summary_of_the_review": "The paper is strong but missing some key experiments, which preclude inclusion at ICLR in the present form. I would like to see one or more additional relevant comparison algorithms included in the simulated experiments, as well as a sensitivity analysis for the learning schedule, as described above. I have not checked the mathematical proofs thoroughly. I have not reviewed the code included in the attached supplementary material.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors introduce a method for improving reinforcement learning in sparse reward settings. In particular, they propose to take advantage of a suboptimal behavior policy as a guidance policy that is incorporated in a TRPO-like update. The reviewers agree that this is a novel and interesting idea and given the authors' rebuttal with additional experiments, clarifications and discussions, they agreed to accept the paper. However, they also point out several flaws (e.g. evaluation on a more challenging sparse-reward task such as Adroid) that I encourage the authors to address in the final version of the paper.
Decision: Accept (Spotlight)