Learning Off-Policy with Online Planning
Authors: Harshit Sikchi
Wenxuan Zhou
David Held
Venue: CoRL2021 Oral (2021)
Keywords: Reinforcement Learning
Trajectory Optimization
Safety

Abstract: Reinforcement learning (RL) in low-data and risk-sensitive domains requires performant and flexible deployment policies that can readily incorporate constraints during deployment. One such class of policies are the semi-parametric H-step lookahead policies, which select actions using trajectory optimization over a dynamics model for a fixed horizon with a terminal value function. In this work, we investigate a novel instantiation of H-step lookahead with a learned model and a terminal value function learned by a model-free off-policy algorithm, named Learning Off-Policy with Online Planning (LOOP). We provide a theoretical analysis of this method, suggesting a tradeoff between model errors and value function errors, and empirically demonstrate this tradeoff to be beneficial in deep reinforcement learning. Furthermore, we identify the "Actor Divergence" issue in this framework and propose Actor Regularized Control (ARC), a modified trajectory optimization procedure. We evaluate our method on a set of robotic tasks for Offline and Online RL and demonstrate improved performance. We also show the flexibility of LOOP to incorporate safety constraints during deployment with a set of navigation environments. We demonstrate that LOOP is a desirable framework for robotics applications based on its strong performance in various important RL settings.


Learning Off-Policy with Online Planning
Harshit Sikchi; Wenxuan Zhou, David Held Roboties Institute
Carnegie Mellon University
{hsikchi, wenxuanz, dheld)@cs. cmu.edu edu
Reinforcement learning (RL) in low-data and risk-sensitive domains requires performant and flexible deployment policies that can readily incorporate constraints during deployment. One such class of policies are the paramerraet H-step lookahead policies, which select actions using trajectory optimization over dynamics model for fixed horizon with a terminal value function. In this work, we investigate a novel instantiation of H-step lookahead with a learned model and a terminal value function learned by a model-free off-policy algorithm, named Learn- ing Off- Off-Policy with Online Planning (LOOP). We provide a theoretical analysis of this method, suggesting tradcoff between model errors and value function errors and empirically demonstrate this tradeoff to be beneficial in deep reinforcement learning. Furthermore, identify the "Actor Divergence" issue in this framework and propose Actor Regularized Control (ARC), a modified trajectory optimization procedure. We evaluate our method set of robotic tasks for Offline and On- line RL and demonstrate improved performance. We also show the flexibility LOOP incorporate safety constraints during deployment with of navigation environments. We demonstrate that LOOP is a desirable framework for robotics applications based on its strong performance in various important RL settings. Project video and details can be found at hari-sikchi, bari-sikchi.gitbubbooleo github.io/loop.
Abstract:
Keywords: Reinforcement Learning, Trajectory Optimization, Safety
Introduction
Online Planning
Off-Policy Learning
Off-policy reinforcement learning algorithms have been widely used in many robotic ap- plications due to their sample efficiency and their ability to incorporate data from different sources [1, 2, 3, 4J. Model-free off-policy algo- rithms sample transitions from replay buffer to learn a value function and then update the policy according to the value function [5, 6]. Thus, the performance of the policy is highly dependent on the estimation of the value function. How- ever, learning an accurate value function from off-policy data is challenging especially in deep RL due to a variety of issues, such as overes- timation bias [7, 81, delusional bias [9], rank loss [10], instability [11], and divergence [12]. Another shortfall of model-free off-policy algorithms in continuous control is that the policy is usually parametrized by feedforward neural network which lacks flexibility during deployment.
Terminal value estimates
Value Function
v(stu)
vod
Dynamics Model
100
Replay Buffer
Dynamics model rollouts
Figure 1: Overview of LOOP: A learned dynamics model is utilized for Online Planning with termi- nal value function. The value function is learned via model-free off-policy algorithm.
Previous works in model-based RL have explored different ways of using dynamics model to improve off- colicolic algorithms [13, 14, 15, 16, 17]. One way of incorporating the dynamics model to H-step lookahead policies [18]. At each timestep. H-step lookahead policies rollout the dynamics model H-step into the future from the current state to find an action sequence with the highest return. Within this trajectory optimization process, terminal value function is attached the end of the rollouts to provide an estimation of the return beyond the fixed horizon. This way of online
Currently at The University of Texas at Austin, Email: hsikchi @ @utexas.eda edu
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

planning offers us a degree of explainability missing in fully parametric methods while also allowing us take constraints into account during deployment. Previous work proves faster convergence with H-step lookahead policies in tabular setting [18] or showed improved sample complexity with a ground-truth dynamics model [19]. However, the benefit of H-step lookahead policies remains unclear under an approximate model and an approximate value function. Additionally, if H-step lookahead policies are used during the value function update [19], the required computation of value function update will be significantly increased.
In this work, we take this direction further by studying H-step lookahead both theoretically and empirically with three main contributions. First, we provide a theoretical analysis of H-step lookahead under an approximate model and approximate value function. Our analysis suggests a trade-off between model error and value function error, and we empirically show that this tradeoff can be used improve policy performance in Deep RL. Second, we introduce Learning Off- Off-Policy with Online Planning (LOOP) (Figure 1). To avoid the computational overhead of performing trajectory optimization while updating the value function as in previous work [19], the value function of LOOP is updated via a parameterized actor using a model-free off-policy algorithm ("Learning Off-Policy"). LOOP exploits the benefits of H-step lookahead policies when the agent is deployed in the environment during exploration and evaluation ("Online Planning"). This novel combination of model-based online planning and model-free off-policy learning provides sample-efficient and cumputationalll--fffccceee learning. We also identify the "Actor Divergence" issue in this combination and propose a modified trajectory optimization method called Actor Regularized Control (ARC). ARC performs implicit divergence regularization with the parameterized actor through Iterative Importance Sampling.
Third, we explore the flexibility of H-step lookahead policies for improved performance in offline RL and safe RL, which are both important settings in robotics. LOOP can be applied on top of various offline RL algorithms to improve their evaluation performance. LOOP's semiparameteric behavior policy also allows it to easily incorporate safety constraints during deployment. We evaluate LOOP on set of simulated robotic tasks including locomotion, manipulation, and controlling an RC car. We show that LOOP provides significant improvement in performance for online RL, offline RL, and safe RL, which makes it a strong choice of RL algorithm for robotic applications.
2 Related Work
Model-based RL Model-based reinforcement learning (MBRL) methods learn a dynamics model and use it to optimize the policy. State-of-the-arr model-based RL methods usually have better sample efficiency compared to model-free methods while maintaining competitive asymptotic perfor- mance [20, 13]. One approach in MBRL is to use trajectory optimization with a learned dynamics model [17, 21, 22]. These methods can reach optimal performance when large enough planning horizon is used. However, they are limited by not being able to reason about the rewards beyond the planning horizon. Increasing the planning horizon increases the number of trajectories that need to be sampled and incurs a heavy computational cost.
Various attempts have been made to combine model-free and model based RI.. GPS [23] combines tra- jectory optimization using analytical models with the on-policy policy gradient estimator. MBVE [15] and STEVE [16] use the model to improve target value estimates. Approaches such as MBPO [13]
and MAAC [24] follow Dyna-style [25] learning where imagined short-horizon trajectories are used to provide additional transitions to the replay buffer leveraging model generalization. Piché et al. [26] use Sequential Monte Carlo (SMC) to capture multimodal policies. The SMC policy relies on combining multiple 1 1-step lookahead value functions to sample a trajectory proportional to the unnormalized probability exp(E!" " (A(s, a))); this approach potentially compounds value function errors, in contrast to LOOP which uses single H-step lookahead planning for each state. POLO [19] shows advantages of trajectory optimization under ground- truth dynamics with terminal value function. The value function updates involve additional trajectory optimization routines which is one of the issues aim to address with LOOP. The computation of trajectory optimization in POLO is OT N) while LOOP is (T where T is the number of environment timesteps, H is the planning horizon, and N is the number of samples needed for training the value function.
Off-Policy RL LOOP relies on a terminal value function for long horizon reasoning which can be learned effectively via model-free off off-policy RL algorithms. Off-policy RL methods such as SAC [5]
2

---

and TD3 [6] use the replay buffer to learn a Q-function that evaluates a parameterized actor and then optimize the actor by maximizing the Q-function. Off-policy methods can be modified be used for Offline RL problems where the goal is to learn a policy from a static dataset [27, 8, 28, 29, 30, 31, 32]. MBOP [33], a recent model-based offline RI. method, leverages planning with terminal value function, but the value function is a Monte Carlo evaluation of truncated replay buffer trajectories, whereas in LOOP the value function is trained for optimality under the dataset.
3 Preliminaries
A Markov Decision Process (MDP) is defined by the tuple A,,,, with state-space S, action- space A, transition probability pl 1/ de). reward function r(s, a), and initial state distribution Pu(s). In the infinite horizon discounted MDP, the goal of reinforcement learning algorithms is to maximize the return for policy л given by J -arra((s)) EE= 7'r(st, d=)]
Value functions: V S + E represents state-value function which estimates the return from the current state 8; and following policy æ, defined as (s) = (ev (e
7'r(st. 7'ree' ar [e)] = s].
Similarly, Q S x A - IE represents action-value function, usually referred as a Q-function, defined as (8, (),,a))= = E - |E 7 as) |80 = 8, dg = a). Value functions corresponding to the optimal policy л¹ are defined to be V and Q". The value function can be updated according to the Bellman operator T:
TQ(s a.) r(s,, a.)
(1)
where To is updated to be greedy with respect to Q, the current Q-function.
Constrained MDP for safety: A constrained MDP (CMDP) is defined by the tuple (S, p, c. Po) with an additional cost function c(s, a). We define the cumulative cost of a policy be D* Ba, (ae (e0)))ppp 1E= ((c(st. oe)]- A common objective for safe reinforcement learning is to find policy π argmax, J subject D I do where dg is a safety threshold [34].
H-step Lookahead with Learned Model and Value Function
Model-based algorithms often learn an approximate dynamics model AT ($ a.) using the data collected from the environment. One way of using the model is find action sequence that maximizes the cumulative reward with the learned model using trajectory optimization [35, 36, 37]. An important limitation of this approach is that the computation grows exponentially with the planning horizon. Thus, methods like [35, 17, 21, 38, 39] plan over a fixed, short horizon and are unable to reason about long-term reward. Let лH be such a fixed horizon policy:
TH (50) = argmax max
E ( +)], where RØ (S0, 7) Σ r'r(s,,
(2)
where 7 denotes the action sequence (f....H-1) One way to enable efficient long-horizon reasoning
augment the planning trajectory with terminal value function. Given a value-function V. define policy H obtained by maximizing the H-step lookahead objective:
T TH y. (so) argmax max Egis R 61.... 1
(3)
#-1
where R H F 7) = Σ 1=0
(r ai) + + + ((n))
The quality of both the model M and the value-function V. affects the performance of the overall policy. To show the benefits of this combination model-based trajectory optimization and the falue-function, we analyze and bound the performance of the H-step look-ahead policy и # compared to its fixed-horizon counterpart without the value-function тH (Eqn. 2), well as the r(s, a) + +>(')) Following previous work, we will construct the proofs with the state-value function V, but the proofs for the action-value function Q can be derived similarly.
greedy policy obtained from the value-function के. = urgmax,E..MMrr
3

---

Lemma 1. (Singh and Yee [401) Suppose we have an approximate value function V such that - [s)]s Ev. Then the performance of the 1-step greedy policy Ty can be bounded as:
JJ
7 [26,]
(4)
Theorem 1. (H-step lookahead policy) Suppose M is an approximate dynamics model with Total Variation distance bounded by 6m- Let V be an approximate value function such that max,| max,IV (s) V(s) cv. Let the reward function r(s, a) be bounded by Rmaxl and V be bounded by [0, 10.Vmat maxl Let Cp be the suboptimality incurred in H-step lookahead optimization (Eqn. 3). Then the performance of the H-step lookahead policy "H, can be bounded as:
J N. S 1 7.
2
[C(Em, H, H27 cp +7"
(5)
where
H-1
C(Em: H, = Rmax
Σ
tm + 7 11 Vmmx
1=0
Proof. Due to the page limit, we defer the proof Appendix A.I. We also provide extension of Theorem under assumptions on model generalization and concentrability in Corollary 1 and Theorem respectively in Appendix A.
H-step Lookahead Policy vs H-step Fixed Horizon Policy: The fixed-horizon policy can be considered as special case of xI " with (s) 0 Vs E S. Following Theorem 1, cp max,|V*(s)) (s)l implies potentially large optimality gap. This suggests that learning value function that better approximates V than (s) will give us smaller optimality gap in the worst case.
H-step lookahead policy vs 1-step greedy policy: By comparing Lemma and Theorem 1, we observe that the performance of the H-step lookahead policy " #. reduces the dependency the value function error e, at least by factor of - while introducing an additional dependency on the model error <m- This implies that the H-step lookahead is beneficial when the value-function bias dominates the bias in the learned model. In the low data regime, the value function bias can result from compounded sampling errors [41] and is likely to dominate the model bias, as evidenced by the success of model-based RL methods in the low-data regime [33, 42, 13]; we observe this hypothesis be consistent with our experiments where H-step lookahead offers large gains in sample efficiency, Further, errors in value learning with function approximation can stem from number of reasons explored in previous work, some of them being Overestimation, Rank Loss, Divergence, Delusional bias, and Instability [7, 11, 6, 43, 10]. Although this result may be intuitive to many practitioners, it has not been shown theoretically; further, we demonstrate that we can use this insight to improve the performance of state-of-the -ftt----t----tttt methods for online RI, offline RI. and safe RI..
5 Learning Off-Policy with Online Planning
We propose Learning Off-Policy with Online Planning (LOOP) as a framework of using H-step lookahead policies that combines online trajectory optimization with model-free off-policy RL (Figure 1). We use the replay buffer to learn a dynamics model and a value function using an off- policy algorithm. The H-step lookahead policy (Eqn. 3) generates rollouts using the dynamics model with terminal value function and selects the best action for execution. The underlying off-policy algorithm is boosted by the H-step lookahead which improves the performance of the policy during both exploration and evaluation. From another perspective, the underlying model-based trajectory optimization improved using terminal value function for reasoning about future returns. this section, we discuss the Actor Divergence issue in the LOOP framework and introduce additional applications and instantiations of LOOP for offline RL and safe RI.
5.1 Reducing actor-divergence with Actor Regularized Control (ARC)
As discussed above, LOOP utilizes model-free off-policy algorithms to learn value function in more computationally efficient manner. It relies on actor-critie methods which use a parametrized actor Ny to facilitate the Bellman backup. However, we observe that combining trajectory optimization and policy learning naively will lead to an issue that we refer to as "actor divergence": a different policy is used for data collection (H-step lookahead policy x #H v) than the policy that is used to learn

---

the value-function (the parametrized actor πT). This leads to potential distribution shift between the state-action visitation distribution between the parametrized actor TT and the actual behavior policy TH which can lead to accumulated bootstrapping errors with the Bellman update and destabilize value learning [43]. One possible solution in this case is to use Offline RL [30]; however, in practice, we observe that offline RL in this setup leads learning instabilities. We defer discussion on this alternative to the Appendix D.7. Instead, we propose to resolve the actor-divergent issue via modified trajectory optimization method called Actor Regularized Control (ARC).
In ARC, we aim to constrain the action selection of the trajectory optimization to be close to the parametrized actor. We frame the following general constrained optimization problem for policy improvement [44]; Popt argmaxEp- 1 +) s.t DKL PXL(p"IPPprion (Prror)) S
(6)
p²
where L (sc. 7) is the expected lookahead objective (Eqn. 3) under the learned model given by L (s;. r) = Eng R R (s..((::T)) starting from state p" is distribution over action sequences of horizon H starting from 8t, and Porior is prior distribution over such action sequences. We will use the parametrized actor to derive this prior in ARC. This optimization admits closed form solution by enforeing the KKT conditions where the optimal policy is given by Poge Pirrior + 4,,,, (#677) [45, 46, 47, 48], where is the lagrangian dual variable. The above formulation generalizes number of prior work [5, 35, 45] (more details in Appendix B.3).
Approximating the optimal policy Pope as a multivariate gaussian with diagonal covariance Pops N ([Hopt: Dapt). the parameters can be estimated using importance sampling under the proposal distribution Pprior
Pope
Dope (T' Porsoor (T') (F) 12)
Popt N (Hopt, Copt), Popt EEr,,n
Perior (7') T'
Topt -Er
where 7' Perior We use iterative importance sampling estimate Pope which is parameterized as a Gaussian whose mean and variance at iteration m + are given by the empirical estimate:
m=1 E " ( where N((m ((") and N(4", 00) is set to Perior- As long we perform finite number of iterations, the final trajectory distribution is constrained in total variation to be close to the prior as a result of finite trust region updates as shown in Lemma 2 in Appendix A.4.
m+1 Ee "m.llsrrr) (T' 1
(8)
EN eb L ( E
ΣN ENeet en
To reduce actor divergence in LOOP, we constrain the be close to that of the parametrized actor wy. To do so, we Porior = 3Tg + B3) ( a). The trajectory prior is a mixture of the parametrized actor and the action sequence from the previous environment timestep with additional Gaussian noise NN0, .). Using 1-timestep shifted solution from the previous timestep allows to amortize trajectory optimization over time [33]. For online RL, we can vary a to vary the amount of exploration during training. For offline RL, set 3 1 to constrain actions to be close to those in the dataset (from which T is learned) to be more conservative.
of the trajectory optimization
5.2 Additional instantiations of LOOP: Offline-LOOP and Safe-LOOP
LOOP not only improves the performance of previous model-based and model-free RL algorithms but also shows versatility in different settings such as the offline RL setting and the safe RL setting. These potentials of H-step lookahead policies have not been explored in previous work.
LOOP for Offline RL: In offline reinforcement learning, the policy is learned from a static dataset without further data collection. We can use LOOP on top of an existing off-po algorithm as a plug-in component improve its test time performance by using the model-based rollouts as suggested by Theorem 1. Note that this is different from the online setting in the previous section which LOOP also influences exploration. In offline-LOOP. to account for the uncertainty in the model and the Q-function, ARC optimizes for the following uncertainty- pessimittite objective similar to 50]:
mean k) [R H. (St, T]] Spees R (st,T)) T)]
(9)
5

---

Figure 2: We evaluate LOOP over variety of environments ranging from locomotion, manipulation navigation including Walker2d-v2, Ant-v2, PenGoal-vl, Claw-v1, CarGoall, etc.
where [K] are the model ensembles, pess is the pessimism parameter and R R v is the H-horizon lookahead objective defined in Eqn. 3.
Safe Reinforcement Learning: Another benefit of LOOP with its semi semi-paramen mani-paammttric policy that we can easily incorporate (possibly con-staiionar) constraints with the model based rollout, while being an order of magnitude more sample efficient than existing safe model-free RL algorithms. To account for safety in the planning horizon, ARC optimizes for the following cost-pessinistic objective:
++ --1
argmax, or E
R (s,
()) s.t. max Σ
7'c(st, ad) S do
(10)
where [K] are the model ensembles, с is the constraint cost function and R H is the H-horizon lookahead objective defined in Eqn. 3 and dg is the constraint threshold. For each action rollout, the worst-case cost is considered w.r.t model uncertainty to be more conservative. The pseudocode for modified ARC to solve the above constrained optimization is given in Appendix B.3.1.
Experimental Results
In the experiments, evaluate the performance of LOOP combined with different off off-policy algorithms in the settings of online RL, offline RL and safe RL over a variety of environments (Figure 2). Implementation details of LOOP and the baselines can be found in Appendix C.
6.1 LOOP for Online RL
In this section, we evaluate the performance of LOOP for online RL. on three OpenAl Gym MuloCo [51] locomotion control tasks: Hal fCheetah-v2, Walker-v2, Ant-v2 and two manipulation tasks: PenGoal-v1, Claw-vl. In these experiments, we use Soft Actor-Critic (SAC) [5] as the underlying off-policy method with the ARC optimizer described in Section 5.1. Fur- ther experiments on
v2, Swimmer, Hopper- v2 and Humanoid-i and more details on the baselines can be found in Appendix D. and Appendix C.2 respectively.
HalfCheetah-v2
Walkor-v2
Ant-v2
PenGoal-v1
Claw-v1
Aparage Reward
12500
690
04
3000
290
-0300
2500
00
Timestops
10
Timestips
Timestops
02
5.55
0.2
Timestops
3,50
1.0
-228
Timestops
3.4
LOOP-SAC (ours)
MBPO
SAC
LOOP-SARSA
SAC-VE
PETS-restrictd
Figure 3: Comparisons of LOOP and the baselines for online RL. LOOP-SAC is significantly more sample efficient than SAC. is competitive to MBPO for locomotion tasks and outperforms MBPO for manipulation tasks (PenGoal-vi and Claw-vl). The dashed line indicates the performance of SAC at million timesteps. Additional results on more environments can be found in Appendix D.1.
Baselines: We compare the LOOP framework against the following baselines: PETS- variant of PETS [17] that uses trajectory optimization (CEM) for the same horizon as LOOP without a terminal value function. LOOP-SARSA uses a terminal value function which an evaluation of the replay buffer policy, similar to MBOP [33] in spirit. To compare with other ways of combining model-basee and model-free RL, we also compare against MBPO [13] and SAC-VE. MBPO leverages the learned model to generate additional transitions for value function learning.

---

SAC-VE utilizes the model for value expansion, similar to MBVE [15] but uses SAC as the model- free component for fair comparison with LOOP as done in [13]. We do not include comparison to STEVE [16] or SLBO [52] as they were shown to be outperformed by MBPO, and perform poorly compared to SAC in Hopper and Walker environments [13]. We were unable to reproduce the results
for SMC [26] due to missing implementation. We did not include POLO here due several reasons. An extended discussion can found Appendix D.2.
From Figure 3, we observe that LOOP-SAC is significantly more sam- ple efficient than SAC, the underlying model- free method used learn terminal value function. LOOP-SAC also scales well to high-dimensional environments like Ant-v2
Performance:
Walker-v2
Walker-v2
Acton Choomppn sopere
Average Rewarrd
600
and PenGoal-vis PETSSrestricted performs poorly due to myopic reasoning over a lim- ited horizon H. SAC-VE and MBPO repre- sent different ways of incorporating a model to improve off-policy learning. LOOP-SAC outperforms SAC-VE and performs competi- tively to MBPO, outperforming it significantly in PenGoal-vi and Claw-vl. In principle, methods like MBPO and value expansion can be combined with LOOP to potentially increase performance; we leave such combinations for future work. LOOP-SARSA has poor performance result of the poor value function that is trained for evaluating replay buffer policy rather than optimality. As an ablation study, we also run experiments using LOOP without ARC, which optimizes the unconstrained objective of Eqn. using CEM [36]. Figure 4 (left) shows that ARC reduces actor-divergence effectively and Figure 4 (right) shows that learning performance is poor in absence of ARC for Wa Walker-v2. More ablation results be found in Appendix D.5.
Timesteps
14
Timestops 165
LOOP-SAC-nOARC
LOOP-SAC
Figure 4: (Lefft ARC reduces the actor-divergence measured by the L2 distance between the mean of the parametrized actor and the output of the H- step lookahead policy. (Right) In absence of ARC, policy learning can be unstable.
6.2
LOOP for Offline RL
Env
CRR
LOOP CRR
Improve%
PLAS
LOOP
Improve%
MBOP
Dataset
PLAS
hopper
65.73
85.83
30.6
32.08
56.47
76.0
48.8
medium
halfcheetah
41.14
41.54
1.0
39.33
39.54
0.5
44.6
walker2d
69.98
79.18
13.1
46.20
52.66
14.0
41.0
hopper
27.69
29.08
5.0
29.29
31,29
6.8
12.4
med-replay
halfcheetah
42.29
42.84
1.3
43.96
44.25
0.7
42.3
walker2d
19.84
27.30
37.6
35.59
41.16
15.7
9.7
Table Normalized scores for LOOP on the D4RL datasets comparing to the underlying offline RL algorithms and baseline MBOP. LOOP improves the base algorithm across various types of datasets and environments.
For Offline RL, benchmark the performance using the D4RL datasets [53]. We combine LOOP with two value-based offline RL algorithms: Critic Regularized Regression (CRR) [54] and Policy in Latent Action Space (PLAS) [32]. We use the original offline RI. algorithms train value function from the static data and then use it as the terminal value function for LOOP. We use 3 = 1 in the trajectory prior of ARC (Section 5.1) in the offline RL setting to keep the policy conservative.
Baselines: In addition to the underlying offline RL algorithms, we also include recent work MBOP [33] baseline. MBOP
terminal value function which is evaluation of the dataset policy. In contrast, LOOP uses a terminal value function trained with offline RL algorithms which is more optimal.
Performance: Table presents the comparison of LOOP and the underlying offline RL algorithms. LOOP offers an average improvement of 15.91% over CRR and 29, 49% over PLAS on the complete D4RL MuJoCo Locomotion dataset. Full results can be found Appendix D.3. The results further highlight the benefit of the LOOP framework compared to the underlying model-free algorithms.

---

CarGoal1
PointGool1
CarGoal1
PointGoal1
Average Cost
100
Average Reward
100
20
60
-20
10"
10"
10'
10'
16'
10'
10'
Timesteps
Timesteps
Timesteps
Timesteps
OOP
LOOP
CPO
LBPO
PPO lagrangian
Figure We compare safeLOOP with other safety methods such as CPO, LBPO, and PPO-lagrangian on OpenAl Safety Gym environments. It shows significant sample efficiency while offering similar or better safety benefits as the baselines.
6.3 LOOP for Safe RL
For safe RI., we modify the H-step lookahead optimization to maximize the sum of rewards while sat- isfying the cost constraints, as described in Section 5.2. We evaluate our method on two environments from the OpenAl Safety Gym [55] and an RC-car simulation environment [56]. The objective of the Safety Gym environments to move a Point mass agent or Car agent to the goal while avoiding obstacles. The RC-car environment is rewarded for driving along circle of Im fixed radius with
desired velocity while staying within the 1.2m circle during training. Details for the environments be found in Appendix C.4.
Baselines: We compare our safety- -uggeeedd LOOP (safel.OOP) against various state-of-the- art safe learning methods such as CPO [57],
Average Reward
Averape Cos!
-00
LBPO [58], and PPO-lagrangian [59, 55). CPO uses a trust region update rule that guarantees safety. LBPO relies a barrier function formu- lated around a Lyapunov constraint for safety. PPO-lagrangian uses dual gradient descent to solve the constrained optimization. To ensure a fair comparison, all policies and dynamics mod- els are randomly initialized, as is commonly done in safe RI experiments (rather than starting from a safe initial policy). We additionally com- pare against model-based safety method that modifies PETS for safe exploration (safePETS) without the terminal value function. We mostly compare to model-free baselines due lack of safe model-based De baselines in the literature.
2000 Timestaps
Timeslaps
SafsLOOP
SETOPETS
LOOP
Figure 6: RC-car experiments show the impor- tance of the terminal value function in the LOOP framework. SafeLOOP achieves higher returns than SafePETS while being competitive in safety performance. Both safePETS and PETS fail to learn drifting policy due to limited lookahead.
Performance: For the OpenAl Safety Gym environments, observe in Figure 5 that safeLOOP can achieve performant yet safe policies in a sample efficient manner. SafeLOOP reaches a higher reward than CPO, LBPO and IPO-lagrangian while being orders of magnitude faster. SafeLOOP also achieves a policy with a lower cost faster than the baselines. From another aspect, the simulated RC-car experiments demonstrate the benefits of the terminal value function in safe RI. Figure 6 shows the performance of LOOP, safeL.OOP, PETS, and safePETS on this domain. PETS [17) and safePETS do not consider terminal value function. SafeLOOP is able achieve high performance while maintaining the fewest constraint violations during training. Qualitatively, LOOP and safeLOOP able to learn safe drifting behavior, whereas PETS and safePETS fail to do so since drifting requires longer horizon reasoning beyond the fixed planning horizon in PETS. The results suggest that safeLOOP is a desirable choice of algorithm for safe RL due to its sample efficiency and the flexibility of incorporating constraints during deployment.
7 Conclusion
In this work we analyze the H-step lookahead method under a learned model and value function and demonstrate empirically that it can lead many benefits in deep reinforcement learning. We propose a framework LOOP which removes the computational overhead of trajectory optimization for value function update. We identify the actor-divergence issue in this framework and propose a modified trajectory optimization procedure Actor Regularized Control. We show that the flexibility of H-step lookahead policy allows us to improve performance in online RL, offline RL as well as safe RL and this makes LOOP strong choice of RL algorithm for robotic applications.
8

---

Acknowledgments
We thank Tejus Gupta, Xingyu Lin and the members of R-PAD lab for insightful discussions. This material is based upon work supported by the United States Air Force and DARPA under Contract No. 18-C-0092, LG Electronics, and the National Science Foundation under Grant No. I1S-1849154.
References
[1] D. Kalashnikov, J. Varley, Y. Chebotar, B. Swanson, R. Jonschkowski, C. Finn, S. Levine, and K. Hausman. Mt-opt: Continuous multi-task robotic reinforcement learning at scale. arXiv preprint arXiv: arXiv:2104.022 104.0844. 12, 2021.
[2] T. Haarnoja, S. Ha, A. Zhou, J. Tan, G. Tucker, and S. Levine. Learning walk via deep reinforcement learning. arXiv preprint arXiv:/8/22 103, 2018.
[3] Matas, S. James, and J. Davison. Sim-to-real reinforcement learning for deformable object manipulation. In Conference on Robot Learning, pages 734-743, PMI., 2018.
[4] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakr- ishnan, V. Vanhoucke, et al. Qr-opt: Scalable deep reinforcement learning for vision-basee robotic manipulation. arXiv preprint arXiv: / 10293, 2018.
[5] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, Tan, V. Kumar, H. Zhu, A. Gupta, Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv: 1812.05905, 2018.
[6] S. Fujimoto, H. Van Hoof, and D.. Meger. Addressing function approximation error in actor-critic methods. arXiv preprint arXiv:1882.00447, 2018.
[7] S. Thrun and A. Schwartz. Issues in using function approximation for reinforcement learning. In Proceedings of the Fourth Connectionist Models Summer School, pages 253-263. Hillsdale, NJ, 1993.
[8] S. Fujimoto, D. Meger, and D. Precup. Off-policy deep reinforcement learning without explo- ration. arXiv preprint / 2018.
[9] Lu, D. Schuurmans, and C. Boutilier. Non-delusional q-learning and value-iteration. 2018.
[10] A. Kumar, R. Agarwal, D. Ghosh, and S. Levine. Implicit under- parameterizaationn inhibits data-efficient deep reinforcement learning. arXiv preprint arXiv:2010. 2020.
[11] J. Fu, A. Kumar, M. Soh, and S. Levine. Diagnosing bottlenecks in deep q-learning algorithms. In International Conference on Machine Learning, pages 2021-2030. PMLR, 2019.
[12] J. Achiam, E. Knight, and P. Abbeel. Towards characterizing divergence in deep q-learning. arXiv preprint 19X1v:903.08800 2019.
[13] M. Janner, J. Fu, M. Zhang, and S. Levine. When to trust your model: Model-based policy optimization, 2019.
[14] A. Rajeswaran, I. Mordatch, and V. Kumar. A game theoretic framework for model based reinforcement learning. In International Conference on Muchine Learning, pages 7953-7963.
PMLR, 2020.
[15] V. Feinberg, A. Wan, I. Stoica, M. Jordan, E. Gonzalez, and S. Levine. Model-based value estimation for efficient model-free reinforcement learning, arXiv preprint 1 2018.
[16] J. Buckman, D. Hafner, G. Tucker, E. Brevdo, and H. Lee. Sample-effcciet reinforcement learn- ing with stochastic ensemble value expansion. Advances in Neural Information Processing 2018.
Systems, pages

---

[17] K. Chua, R. Calandra, R. McAllister, and S. Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems, pages 4754-4765 2018.
[18] Y. Efroni, M. Ghavamzadeh, and S. Mannor. Online planning with lookahead policies. Advances in Neural Information Processing Systems, 33, 2020.
[19] K. Lowrey, A. Rajeswaran, S. Kakade, E. Todorov, and I. Mordatch. Plan online, learn offline: Efficient learning and exploration via model-based control. arXiv preprint arXiv: 1a:::11111000000 1,04 2018.
[20] T. Kurutach, I. Clavera, Y. Duan, A. Tamar, and P. Abbeel. Model-ensemble trust-region policy optimization. arXiv preprint arXiv: 10592, 2018.
[21] A. Nagabandi, K. Konoglie, S. Levine, and V. Kumar. Deep dynamics models for learning dexterous manipulation. arXiv preprint arXi:: 2019.
[22] M. Deisenroth and C. Rasmussen. Pilco: A model-based and data-efficient approach to policy search. In Proceedings of the 28th International Conference muchine learning (ICML-1I) pages 465 472. Citeseer, 2011.
[23] S. Levine and V. Koltun. Guided policy search. In International conference machine learning, pages 1-9. PMLR, 2013.
[24] I. Clavera, Fu, and P. Abbeel. Model-augmented actor-critic; Backpropagating through paths. arXiv preprint arXiv:2005.08065 2020.
[25] R. S. Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM Sigart Bulletin, 2(4):160-163: 163600--1,1111 1991.
[26] A. Piché, V. Thomas, C. Ibrahim, Y. Bengio, and C. Pal. Probabilistic planning with sequential monte carlo methods. In International Conference on Learning Representations, 2018
[27] R. Agarwal, D. Schuurmans, and M. Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference Machine Learning, pages 104- 114. PMLR, 2020.
[28] R. Zhang, B. Dai, L. Li, and D. Schuurmans. Gendice: Generalized offline estimation of stationary values. arXiv preprint arXiv:2002.09072) 2020.
[29] N. Y. Siegel, J. T. Springenberg, F. Berkenkamp, A. Abdolmaleki, M. Neunert, T. Lampe, R. Hafner, and M. Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. arXiv preprint arXiv:2002.08396, 2020.
[30] S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.
[31] A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006. 04779, 2020.
[32] W. Zhou, S. Bajracharya, and D. Held. Plas: Latent action space for offline reinforcement learning. arXiv preprint arX1v:2011.07213,, 2020.
[33] A. Argenson and G. Dulac-Arnold. Model-based offline planning. arXiv preprint arXiv:2008.05556 2020.
[34] E. Altman. Constrained Markov decision processes, volume 7. CRC Press, 1999.
[35] G. Williams, P. Drews, B. Goldfain, J. M. Rehg. and E. A. Theodorou. Aggressive driving with model predictive path integral control. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pages 1433-1440. IEFE, 2016.
[36] R. Rubinstein. The cross-entropy method for combinatorial and continuous optimization. Methodology and computing in applied probability, 1(2):127-190, 1999.
10

---

[37] E. F. Camacho and C. B. Alba. Model predictive control. Springer science & business media, 2013.
[38] Wang and J. Ba. Exploring model-based planning with policy networks. arXiv preprint arXiv: 1rXiv:1906.0000 0864,, 2019.
[39] B. Zhang, R. Rajan, I.. Pineda, N. Lambert, A. Biedenkapp. K. Chua, F. Hutter, and R. Calandra. On the importance of hyperparameter optimization for model-based reinforcement learning. In International Conference on Artificial Intelligence and Statistics, pages 4015-4023. PMLR, 2021.
[40] Singh and R. C. Yee. An upper bound on the loss from approximate optimal- value functions. Machine Learning, 16(3):227-233, 1994.
[41] A. Agarwal, N. Jiang, and S. M. Kakade. Reinforcement learning: Theory and algorithms. CS Dept. UW Seattle, Seattle, WA, USA, Tech. Rep, 2019.
[42] T. Matsushima, H. Furuta, Y. Matsuo, O. Nachum, and S. Gu. Deployment-effiieen rein- forcement learning via model- offline optimization. arXiv preprint 20X1v:2006.03647, 2020.
[43] A. Kumar, J. Fu, M. Soh, G. Tucker, and S. Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, pages 11761-11771, 2019.
[44] N. Vieillard, T. Kozuno, B. Scherrer, O. Pietquin, R. Munos, and M. Geist. Leverage the average: an analysis of regularization in rl. arXiv preprint arXiv:2003. 14089, 2020.
[45] A. Nair, M. Dalal, Gupta, and Levine. Accelerating online reinforcement learning with offline datasets. arXiv preprint ar arXiv:2006.09:3 2020.
[46] J. Peters and S. Schaal. Reinforcement learning by weightwe regression for operational space control. Proceedings of the 24th international conference on Machine learning, pages 745-750, 2007.
[47] J. Peters and S. Schaal. Natural actor-critic. Neurocomputing, 7107-9):1180-1199 2008.
[48] J. Peters, K. Mulling, and Y. Altun. Relative entropy policy search. In Twenty-Fourhh AAAI Conference on Artificial Intelligence, 2010.
[49] Yu, G. Thomas, L. Yu, S. Ermon, Zou, S. Levine, C. Finn, and T. Ma. Mopo: Model-based offline policy optimization. arXiv preprint arXiv:2005. 2020.
[50] R. Kidambi, A. Rajeswaran, P. Netrapalli, and T. Joachims. Morel: Model-based offline reinforcement learning. arXiv preprint ar arXiv:200s.0595/ 2020.
[51] E. Todorov, T. Frez, and Tassa. Mujoco: physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033, IEEE, 2012.
[52] Y. Luo, H. Xu. Li, Y. Tian, T. Darrell, and T. Ma. Algorithmic framework for model based deep reinforcement learning with theoretical guarantees. arXiv preprint arXiv:1807.03855. 2018.
[53] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint XXiv:2004.072199 2020.
[54] Z. Wang. Novikov, K. Zoina, T. Springenberg, S. Reed, B. Shahriari, N. Siegel, Merel, C. Gulcehre, N. Heess, et al. Critic regularized regression. arXiv preprint MXXi::006..1134 2020.
[55] A. Ray, J. Achiam, and D. Amodei. Benchmarking Safe Exploration in Deep Reinforcement Learning. 2019.
[56] E. Ahn. Towards safe reinforcement learning in the real world. PhD thesis, 2019.

---

[57] J. Achiam, D. Held, A. Tamar, and P. Abbeel. Constrained policy optimization. In International Conference on Machine Learning, pages 22-31. PMLR, 2017.
[58] H. Sikchi, W. Zhou, and D. Held. Lyapunov barrier policy optimization. arXiv preprint arXiv:2103.09230, 2021.
[59] E. Altman. Constrained markov decision processes with total cost criteria: Occupation measures and primal Ip. Mathematical methods of operations research, 43(1):45-72, 1996.
[60] R. Munos. Error bounds for approximate value iteration. In Proceedings of the National Conference on Artificial Intelligence, volume 20, page 1006. Menlo Park, CA; Cambridge, MA; London; AAAI Press; MIT Press; 1999, 2005.
[61] R. Munos and C. Szepesvári. Finite-time bounds for fitted value iteration. Journal of Machine Learning Research, 9(5), 2008.
[62] Z. Liu, Zhou, B. Chen, S. Zhong, M. Hebert, and D. Zhao. Safe model-based reinforcement learning with robust cross-entropy method. arXiv preprint arXiv:2010.07968, 2020.
[63] M. Wen and U. Topeu. Constrained cross-entropy method for safe reinforcement learning. IEEE Transactions on Automatic Control, 2020.
[64] B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles, arXiv preprint a12iv:661..01474 2016
12


Reviews:
[{"review_id": "czWWZiBJ2Da", "reviewer": "Reviewer_Sv1h", "summary": "This paper proposes a novel approach to reinforcement learning with planning. \nThe idea is to learn a model of the dynamics and \n\n1. Use a planning algorithm to find the optimal sequence of actions in the H-horizon.\n2. Use at the last step of the horizon an approximation of $V^*$, so to make the planning algorithm aware of the value beyond the horizon.\n3. The value function is learned with an off-policy reinforcement learning algorithm\n4. Since the policy of the trajectory planner differs from the policy induced by the off-policy RL, the authors propose to integrate in the planner a penalization term for actions that are far from the policy of the reinforcement learning counterpart.\n\nIn particular, the authors propose bounds on the return of the policy induced by the planner by both taking into consideration the error from the model and from the value function.\n\nThe authors propose both a version of their algorithm for offline RL and for constrained MDP.\n\n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "__Strenghts__\n\nThe idea of integrating the value function into the planner to extend the horizon is not new [1, 2]. The merits of the paper are, in my opinion,\n\n- the theoretical analysis \n- the introduction of the ARC (the Actor Regularized Control). \n- the usage of the algorithm in both offline RL and constrained MDP.\n\nI think that the paper is well written and follows a good argumentation of the proposed method.\n\nWhen I read the paper, I immediately thought that the RL algorithm should only evaluate the policy, rather than improve it, (e.g., should evaluate $V^\\pi$ instead of $V^*$). Doing that should avoid the problem that you mention of having a policy in the RL algorithm that differs from the policy of the planner. I saw in the experiment that you actually did that with LOOP-SARSA. I see that LOOP-SARSA does not obtain a good performance (I think that this is due to the fact that it does not \"bootstrap\" the optimal value function). It would be nice still to insert a more detailed explanation of why you choose to estimate $V^*$ instead of $V^\\pi$ (theoretically, the improvement is still guaranteed by the fact that the planner seeks optimal actions - this is the same principle driving behind SARSA).\n\nThe idea of keeping the trajectory close to the one generated by the RL agent does make sense too, otherwise, there is a distribution mismatch. \n\nI think that the empirical section is well made, there are enough comparisons with related algorithms.\n\n\n__Weakness__\n\nThe bound (eq 5) assumes that the planner is picking optimal actions. In reality, this is not the case. Therefore, the authors should insert a term that describes the situation where the planner plans a suboptimal trajectory.\n\nI would insert the discussion on whether to use policy evaluation (estimate $V^\\pi$) or policy improvement (estimate $V^*$). What are the pros and cons of the two approaches?\n\n\n\n__typo__\n\nin line 195 the inline math has an extra parenthesis.\n\nUpdate\n==\n\nI appreciated the authors including my suggestion in the paper. After reading the other reviews, I am willing to keep my score unchanged. \n\n\n__References__\n\n[1] https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=6614995\n[2] https://arxiv.org/pdf/2102.11122.pdf", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I do not think that the idea presented in the paper is very novel (planning + RL is not novel, (value-function to extend the horizon is not novel), (regularizing the actor is also not a novel concept)...\nStill, the paper is well written, and all the choices are, in my opinion, fair and logical. Also, each contribution of the paper might be small but summed all together it makes a fair level of originality.\n\nThe authors did a good job in comparing their algorithm with related work.\nTherefore, I advocate for the paper's acceptance.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "- I would like to see Theorem 1 with also an assumption on suboptimal trajectory planners.\n- I would like a discussion on the difference between providing $V^*$ instead of $V^\\pi$ to the planner.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "MEDU6-0wBms", "reviewer": "Reviewer_ztVy", "summary": "The paper presents an RL approach that combines model-based and model-free learning and improves upon previous H-step lookahead methods. In particular, H-step lookahead methods employ a model to perform trajectory optimization for H steps and then use a learned value function to estimate terminate values of the trajectories, which enables infinite-horizon optimization. The paper first performs a theoretical analysis of the induced errors of model-based and value learning parts and concludes that there is an advantage of combining them, especially in low data regimes. After that, the paper identifies the problem of “actor divergence”, which is present in previous methods. In particular, as the behavior policy used to collect the data differs from the parameterized actor used to optimize the value function, it can lead to bootstrapping errors. The paper proposes to introduce a KL-divergence constraint on the behavior policy optimization that constraints it against the actor policy and in this way reduces the actor divergence. Finally, the paper introduces ensemble-based uncertainty estimation for incorporating pessimism when learning from offline data, and points out the ability to incorporate safety constraints when performing model-based trajectory optimization. The method is evaluated in multiple simulated MuJoCo tasks, such as OpenAI Gym tasks for Online RL evaluations, D4RL for Offline RL, and OpenAI Safe Gym for Safe RL evaluations. It is shown to outperform multiple baselines and reduce the actor divergence problem.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- H-step lookahead methods are an elegant way to combine advantages of model-based and model-free learning. The paper presents an important improvement for this family of methods. \n- Theoretical error analysis is a good way to demonstrate advantages of H-step lookahead methods.\n- The paper is well-written and easy to understand and follow.\n- The method is shown to outperform a range of baselines in both online and offline problems in simulation, as well as work with simulated safety-critical tasks.\n\nWeaknesses:\n- Although mentioning POLO in the related work, experiments do not include this method as a baseline. It would be useful to see comparisons to this method as it is close to the presented approach and would help to evaluate the advantage of reducing the actor divergence.\n- It would be interesting to see an ablation of adding the pessimistic term to optimization to better understand its impact.\n- It is not completely clear how a suboptimal actor would affect the behavior policy in case the actor optimization does not work well. It would be interesting to see a discussion on this aspect of the algorithm.\n\n-- Post rebuttal --\nThanks for clarifications and additional experiments, I have increased my score.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper improves upon previous works on H-step lookahead methods and provides a range of experiments and theoretical derivations. Some improvements could be made through additional experiments and discussions.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "Please refer to \"Strengths And Weaknesses\" for a detailed list of issues.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}, {"review_id": "87WIRWJrnBn", "reviewer": "Reviewer_PXsM", "summary": "This paper presents a theoretical analysis and practical implementation of a learning framework that combines H-step lookahead policies with a learned model and a learned terminal value function using a model-free off-policy algorithm. This method, called Learning Off-Policy with Online Planning (LOOP), reduces the computational cost of the trajectory optimization by updating the value function through using a model-free off-policy algorithm. In addition, the authors propose a modified trajectory optimization method called Actor Regularized Control to address the Actor Divergence issue (different policies used for data collection and to learn the value function). Moreover, the authors show the LOOP framework can be applied to enhance performance in offline RL and safe RL tasks. Finally, the method is tested on different simulated tasks, where it is shown to provide performance improvements for online RL, offline reinforcement learning and safe RL when compared with state-of-the-art methods. ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper is well written and clear to understand. The authors do a good job introducing the state of the art methods related with their work and how their work differentiates from others. The authors also present in a clear and structured way each of the different contributions of the paper and they provide a detailed analysis of the extensive testing of their framework and comparison with other methods. \n- A theoretical analysis is presented to provide a solid foundation for the method and back up the experimental results.\n- It is interesting to see how this framework can be adapted not only for online RL, but also for offline RL and safe RL, which makes it versatile and appealing for its application in robotic tasks.\n\nWeaknesses\n- Despite the fact that a model-free off-policy algorithm is used to compute the Value function, including the H-step lookahead policy may still increase the computational cost of the algorithm as the action is selected though trajectory optimization by rolling out the dynamic model to find an action sequence with the highest return.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I would recommend the authors to include a section to discuss the application of its framework to high-dimensional complex robotics tasks e.g. humanoid and what could be the potential issues or advantages. Though they have shown their method performs as good or better than baselines available for comparison, it would be interesting to see its application in 3d complex environments and potential real robotics applications. \n", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "There are no issues to be addressed for this paper. ", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}]

Author rebuttal:

Meta-review: === comments before the discussion ===

The reviewers agree that the paper is well-written and the technical quality is good. On the other hand, reviewers provided comments to improve the paper, e.g., adding ablation study and missing discussions. I recommend the authors to carefully address the comments provided by reviewers.

=== comments after the discussion ===

The paper present a novel RL approach that combines model-based and model-free learning. As the scores provided by reviewers are very high, the area chair recommends the acceptance of the paper (oral).


Decision: Accept (Oral)