Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper presents a theoretical analysis and practical implementation of a learning framework that combines H-step lookahead policies with a learned model and a learned terminal value function using a model-free off-policy algorithm. This method, called Learning Off-Policy with Online Planning (LOOP), reduces the computational cost of the trajectory optimization by updating the value function through using a model-free off-policy algorithm. In addition, the authors propose a modified trajectory optimization method called Actor Regularized Control to address the Actor Divergence issue (different policies used for data collection and to learn the value function). Moreover, the authors show the LOOP framework can be applied to enhance performance in offline RL and safe RL tasks. Finally, the method is tested on different simulated tasks, where it is shown to provide performance improvements for online RL, offline reinforcement learning and safe RL when compared with state-of-the-art methods. 
strengths_and_weaknesses: Strengths
- The paper is well written and clear to understand. The authors do a good job introducing the state of the art methods related with their work and how their work differentiates from others. The authors also present in a clear and structured way each of the different contributions of the paper and they provide a detailed analysis of the extensive testing of their framework and comparison with other methods. 
- A theoretical analysis is presented to provide a solid foundation for the method and back up the experimental results.
- It is interesting to see how this framework can be adapted not only for online RL, but also for offline RL and safe RL, which makes it versatile and appealing for its application in robotic tasks.

Weaknesses
- Despite the fact that a model-free off-policy algorithm is used to compute the Value function, including the H-step lookahead policy may still increase the computational cost of the algorithm as the action is selected though trajectory optimization by rolling out the dynamic model to find an action sequence with the highest return.

summary_of_recommendation: I would recommend the authors to include a section to discuss the application of its framework to high-dimensional complex robotics tasks e.g. humanoid and what could be the potential issues or advantages. Though they have shown their method performs as good or better than baselines available for comparison, it would be interesting to see its application in 3d complex environments and potential real robotics applications. 

issues: There are no issues to be addressed for this paper.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper proposes a novel approach to reinforcement learning with planning. 
The idea is to learn a model of the dynamics and 

1. Use a planning algorithm to find the optimal sequence of actions in the H-horizon.
2. Use at the last step of the horizon an approximation of $V^*$, so to make the planning algorithm aware of the value beyond the horizon.
3. The value function is learned with an off-policy reinforcement learning algorithm
4. Since the policy of the trajectory planner differs from the policy induced by the off-policy RL, the authors propose to integrate in the planner a penalization term for actions that are far from the policy of the reinforcement learning counterpart.

In particular, the authors propose bounds on the return of the policy induced by the planner by both taking into consideration the error from the model and from the value function.

The authors propose both a version of their algorithm for offline RL and for constrained MDP.


strengths_and_weaknesses: __Strenghts__

The idea of integrating the value function into the planner to extend the horizon is not new [1, 2]. The merits of the paper are, in my opinion,

- the theoretical analysis 
- the introduction of the ARC (the Actor Regularized Control). 
- the usage of the algorithm in both offline RL and constrained MDP.

I think that the paper is well written and follows a good argumentation of the proposed method.

When I read the paper, I immediately thought that the RL algorithm should only evaluate the policy, rather than improve it, (e.g., should evaluate $V^\pi$ instead of $V^*$). Doing that should avoid the problem that you mention of having a policy in the RL algorithm that differs from the policy of the planner. I saw in the experiment that you actually did that with LOOP-SARSA. I see that LOOP-SARSA does not obtain a good performance (I think that this is due to the fact that it does not "bootstrap" the optimal value function). It would be nice still to insert a more detailed explanation of why you choose to estimate $V^*$ instead of $V^\pi$ (theoretically, the improvement is still guaranteed by the fact that the planner seeks optimal actions - this is the same principle driving behind SARSA).

The idea of keeping the trajectory close to the one generated by the RL agent does make sense too, otherwise, there is a distribution mismatch. 

I think that the empirical section is well made, there are enough comparisons with related algorithms.


__Weakness__

The bound (eq 5) assumes that the planner is picking optimal actions. In reality, this is not the case. Therefore, the authors should insert a term that describes the situation where the planner plans a suboptimal trajectory.

I would insert the discussion on whether to use policy evaluation (estimate $V^\pi$) or policy improvement (estimate $V^*$). What are the pros and cons of the two approaches?



__typo__

in line 195 the inline math has an extra parenthesis.

Update
==

I appreciated the authors including my suggestion in the paper. After reading the other reviews, I am willing to keep my score unchanged. 


__References__

[1] https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=6614995
[2] https://arxiv.org/pdf/2102.11122.pdf
summary_of_recommendation: I do not think that the idea presented in the paper is very novel (planning + RL is not novel, (value-function to extend the horizon is not novel), (regularizing the actor is also not a novel concept)...
Still, the paper is well written, and all the choices are, in my opinion, fair and logical. Also, each contribution of the paper might be small but summed all together it makes a fair level of originality.

The authors did a good job in comparing their algorithm with related work.
Therefore, I advocate for the paper's acceptance.
issues: - I would like to see Theorem 1 with also an assumption on suboptimal trajectory planners.
- I would like a discussion on the difference between providing $V^*$ instead of $V^\pi$ to the planner.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The paper presents an RL approach that combines model-based and model-free learning and improves upon previous H-step lookahead methods. In particular, H-step lookahead methods employ a model to perform trajectory optimization for H steps and then use a learned value function to estimate terminate values of the trajectories, which enables infinite-horizon optimization. The paper first performs a theoretical analysis of the induced errors of model-based and value learning parts and concludes that there is an advantage of combining them, especially in low data regimes. After that, the paper identifies the problem of “actor divergence”, which is present in previous methods. In particular, as the behavior policy used to collect the data differs from the parameterized actor used to optimize the value function, it can lead to bootstrapping errors. The paper proposes to introduce a KL-divergence constraint on the behavior policy optimization that constraints it against the actor policy and in this way reduces the actor divergence. Finally, the paper introduces ensemble-based uncertainty estimation for incorporating pessimism when learning from offline data, and points out the ability to incorporate safety constraints when performing model-based trajectory optimization. The method is evaluated in multiple simulated MuJoCo tasks, such as OpenAI Gym tasks for Online RL evaluations, D4RL for Offline RL, and OpenAI Safe Gym for Safe RL evaluations. It is shown to outperform multiple baselines and reduce the actor divergence problem.
strengths_and_weaknesses: Strengths:
- H-step lookahead methods are an elegant way to combine advantages of model-based and model-free learning. The paper presents an important improvement for this family of methods. 
- Theoretical error analysis is a good way to demonstrate advantages of H-step lookahead methods.
- The paper is well-written and easy to understand and follow.
- The method is shown to outperform a range of baselines in both online and offline problems in simulation, as well as work with simulated safety-critical tasks.

Weaknesses:
- Although mentioning POLO in the related work, experiments do not include this method as a baseline. It would be useful to see comparisons to this method as it is close to the presented approach and would help to evaluate the advantage of reducing the actor divergence.
- It would be interesting to see an ablation of adding the pessimistic term to optimization to better understand its impact.
- It is not completely clear how a suboptimal actor would affect the behavior policy in case the actor optimization does not work well. It would be interesting to see a discussion on this aspect of the algorithm.

-- Post rebuttal --
Thanks for clarifications and additional experiments, I have increased my score.

summary_of_recommendation: The paper improves upon previous works on H-step lookahead methods and provides a range of experiments and theoretical derivations. Some improvements could be made through additional experiments and discussions.
issues: Please refer to "Strengths And Weaknesses" for a detailed list of issues.
review_title: null

Meta-review: === comments before the discussion ===

The reviewers agree that the paper is well-written and the technical quality is good. On the other hand, reviewers provided comments to improve the paper, e.g., adding ablation study and missing discussions. I recommend the authors to carefully address the comments provided by reviewers.

=== comments after the discussion ===

The paper present a novel RL approach that combines model-based and model-free learning. As the scores provided by reviewers are very high, the area chair recommends the acceptance of the paper (oral).


Decision: Accept (Oral)