Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper considers the problem of reinforcement learning with sparse reward functions obtained from offline demonstration. The authors propose a trust region policy optimization based algorithm with offline demonstration data for guidance. The proposed LOGO algorithm is proved to be efficient by a theoretical analysis showing the lower bound on the performance improvement. On benchmark datasets, the proposed algorithm performs better than the state-of-the-art approaches. An illustration is shown by implementing LOGO on a mobile robot for trajectory tracking and obstacle avoidance.
main_review: The idea of using behavior policy to guide the reinforcement learning algorithm is promising given the fact that demonstrations are sparse to learn. While the sparsity can cause slow convergence, it is important to have a rapid algorithm for this kind of problems. The proofs provide lower bounds on reward gain between two consecutive updates. Theorem 1 shows that in the initial phase of learning the lower bound could be positive, where \beta is larger positive number. I have a concern about this theorem. In the initial phase, the KL divergence of \pi_k and \pi_b is also larger given the fact that \delta_k is decreasing. In this case, the result in Theorem 1 is not always positive which may not indicate faster learning in the initial phase. 

In the experimental section, the proposed LOGO algorithm seems to have the fastest convergence compared to others. Could you give more description about convergence rates comparison from a theoretical point of view?

summary_of_the_review: This paper proposed a novel algorithm to solve reinforcement learning problems with offline demonstration policy. The idea is interesting and nature for many real problems, especially for learning from demonstration problems. A very straightforward update rule is provided by using an estimation of KL divergence. This improves the computational efficiency of the proposed algorithm, which is very promising in real problems.  The Mujoco and TurtleBot experiments verify the performance of the algorithm with both simulation and real experiment. Overall, this paper is well written and the idea is novel and promising.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents ‘LOGO’, an extension of the TRPO algorithm which enables additional learning guidance from offline, sub-optimal (possibly incomplete observation) demonstration data. By annealing away the learning contribution from the sub-optimal guidance policy data during training (with a learning schedule and corresponding hyper-parameter), LOGO utilises this data for guidance, rather than directly imitating it. Furthermore, because LOGO utilises the trust-region methodology, the authors are able to provide a theoretical analysis and lower bound on performance improvement each episode. The method shows promising performance on several MuJoCo continuous control tasks, as well as in a Gazebo TurtleBot simulation, which is also able to be transferred to a real-world robot.
main_review: This paper contributes a useful and apparently novel algorithm for an important problem in the application of RL methods in real-world settings. While the algorithm, theory and experimental methodologies seem sound, a lack of comparison with several relevant algorithms makes it difficult to comprehensively assess the strength of the proposed method. Claims for the merit of the proposed method would also be strengthened if the paper included a sensitivity analysis with varying learning schedules for annealing away the guidance policy data. Without these elements, this paper in its current form falls slightly below the bar for inclusion at ICLR. If the authors can furnish these additional results (or provide compelling arguments why they are not necessary), then I would be willing to increase my score for this paper.

**Strengths:** The paper is very clear and well written, and the explanation of the algorithm is easy to follow. The experimental methodologies (selection of simulated experiments, measured metrics etc.) appear to be sound, and it is good to see experiments on a real-world robot which substantiates the claim that this method is useful for real-world RL. The fact that the theoretical version of the algorithm (before sampling based approximations) utilises the trust-region method means that this algorithm can be theoretically analysed, which is a nice property. The main assumption of the method (Assumption 1) is reasonable. In this reviewer’s opinion, one of the best strengths of the proposed LOGO method is that the double-TRPO structure allows this to be easily implemented using existing TRPO codebases. Implementation is often a stumbling point for RL algorithms, and this is a non-trivial strength of the method.

**Weaknesses:** The discussion of related work seems to be missing a few recent papers that might be relevant. These methods should also be represented in the simulated experiments to enable a fair comparison with related approaches. Specifically, in addition to Policy Optimization from Demonstration (POfD) [1] (which is cited and included in the simulated experiments), other relevant methods seem to be Advantage Weighted Actor-Critic (AWAC) [2], (which is cited but not included in experiments - the authors dismiss this method as heuristic based, but this is not a sufficient reason to exclude it from experiments), as well as Learning with supervision from Noisy Demonstrations (LfND) [3]. RL with GAN shaping [4] and the Cycle-of-Learning (CoL) framework [5] also appear to be tackling a similar problem to LOGO.

The inclusion of GAIL in the simulated experiment and pure TRPO in the simulated and real-world experiment helps situate the results, but neither of these methods could be expected to outperform LOGO given the sparse reward structure and the sub-optimal guidance data the present problem - if anything these methods serve as lower-bounds on performance that LOGO should out-perform. It would also be helpful for the reader to include a comparison with the naive strategy of behaviour cloning the sub-optimal policy, then continuing training with vanilla TRPO for some number of iterations.

As the authors acknowledge, there is a large body of work in combining online RL with offline demonstrations in various ways, and not every paper can compare against every prior method, however I believe POfD by itself does not allow a comprehensive assessment of the performance of the proposed approach.

My other concern is that the proposed algorithm depends critically on the learning schedule used to anneal away the offline policy guidance (which is described in appendix F), however as far as I can tell, the paper does not include a sensitivity analysis over this schedule (which potentially leaves the paper open to accusations of cherry picking results). Given that the performance of LOGO critically hinges on this schedule, I believe the paper needs to include an additional experiment showing how LOGO performs under variations of the schedule (e.g. sweeping values for $K_\delta$ as well as for $\alpha$).

**Queries and minor points:**

1. Is there anything that means LOGO is specifically useful for the sparse reward setting? Would it help or hinder in a dense reward setting?
2. Can the notion of the guidance policy ‘offer[ing] an advantage’ (Section 1) be quantified through information theoretic analyses?
3. Please ensure your charts are fully legible in greyscale - currently Figure 1 is not, and Figure 2 is a little difficult to read in greyscale.

---

[1] Kang, Bingyi, Zequn Jie, and Jiashi Feng. "Policy optimization with demonstrations." International Conference on Machine Learning. PMLR, 2018.

[2] Nair, Ashvin, et al. "Accelerating online reinforcement learning with offline datasets." arXiv preprint arXiv:2006.09359 (2020).

[3] Ning, Kun-Peng, and Sheng-Jun Huang. "Reinforcement learning with supervision from noisy demonstrations." arXiv preprint arXiv:2006.07808 (2020).

[4] Wu, Yuchen, Melissa Mozifian, and Florian Shkurti. "Shaping rewards for reinforcement learning with imperfect demonstrations using generative models." 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021.

[5] Goecks, Vinicius G., et al. "Integrating behavior cloning and reinforcement learning for improved performance in dense and sparse reward environments." arXiv preprint arXiv:1910.04281 (2019).

summary_of_the_review: The paper is strong but missing some key experiments, which preclude inclusion at ICLR in the present form. I would like to see one or more additional relevant comparison algorithms included in the simulated experiments, as well as a sensitivity analysis for the learning schedule, as described above. I have not checked the mathematical proofs thoroughly. I have not reviewed the code included in the attached supplementary material.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper aims to improve the performance of RL in sparse reward settings via the guidance of sub-optimal demonstrations. The idea proposed by this work is to modify TRPO to restrict its updates to remain close to the behavior policy that generated the offline dataset, while decaying this constraint over time to enable the RL policy to improve upon the behavior policy in an online fashion. The authors do so by minimizing the KL divergence between the current RL policy and the behavior policy while not moving too far away from the current RL policy. The method is evaluated on openAI gym style tasks as well as a real robot navigation task. 
main_review: This paper provides a novel formulation of how to keep an RL policy close to a behavior policy. Instead of directly performing some sort of constrained optimization (maximizing the expected sum of rewards subject to the constraint being $ KL(\pi||\pi_b) < \epsilon $), the authors formulate the problem as another RL problem with the reward $ \frac{log(\pi(a|s))}{log(\pi_b(a|s))} $ while ensuring that the updated policy doesn't stray too far from the current RL policy iterate (with a KL constraint term that has a decaying coefficient). This approach is certainly novel and an intriguing way to optimize under this constraint and the empirical results certainly do appear impressive.

However, I do have several concerns.
1. This paper makes no mention of prior work that has also noted this problem of attempting to constrain the RL policy to remain close to the behavior policy via the same constraint mentioned in this work: $ KL(\pi||\pi_b) < \epsilon $. In the offline RL literature, there has been a large body of work that has attempted to optimize this constraint in different ways: (BEAR, BCQ, ABM, BRAC, MPO). I believe these, and many other works in this area should be included with a discussion of their tradeoffs in the related works section. 
2. Empirically, this work is missing several clear comparisons. One: training a behavior policy using the offline data (do Behavior cloning) and simply initializing TRPO from this policy to bootstrap learning. This is a simple solution used as a baseline in prior work (see RPL) and should be included as a point of comparison. Two: DAPG. This method can in principle use suboptimal data and has been empirically validated to work on challenging sparse reward tasks (though with the use of expert demos). Three: AWAC. This method was designed explicitly for the scenario in which a large suboptimal dataset is provided (see results on D4RL) and a small amount of online finetuning is required. Comparison with this method is critical. Four: Comparison against behavior cloning with reward filtering. (simply train a behavior cloning policy with only 10%/20% of the optimal data as ranked by the return of the offline trajectories). This method has been demonstrated to outperform many recent offline RL algorithms as seen in Table 3 of the Decision Transformer paper. 
3. Empirical evaluation environments. The method is mostly evaluated on relatively simple Gym tasks modified to include sparse rewards. The RL community already has much more challenging sparse reward environments in which similar methods have been tested: the Adroit suite of sparse reward hand manipulation tasks. Evaluating if the proposed method's results are relevant should entail a comparison against prior work (AWAC and DAPG at the very least) on these much more challenging robotic control tasks. 
4. nitpicks: 
* upperbound should be upper bound, lowerbound should be lower bound
* End of related work: “Their approaches, performance guarantees and experimental settings are different from that of our problem and proposed solution approach.” Please explain how this is the case

Citations:
BEAR: Aviral Kumar, Justin Fu, George Tucker, and Sergey
Levine. Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction. In Neural Information Processing
Systems (NeurIPS), 2019.

BCQ: Scott Fujimoto, David Meger, and Doina Precup. OffPolicy Deep Reinforcement Learning without Exploration.
In International Conference on Machine Learning (ICML),
2019.

ABM: Noah Y. Siegel, Jost Tobias Springenberg, Felix
Berkenkamp, Abbas Abdolmaleki, Michael Neunert,
Thomas Lampe, Roland Hafner, Nicolas Heess, and
Martin Riedmiller. Keep doing what worked: Behavioral
modelling priors for offline reinforcement learning, 2020

BRAC: Yifan Wu, George Tucker, and Ofir Nachum. Behavior
Regularized Offline Reinforcement Learning. 2020.

MPO: Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval
Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a Posteriori Policy Optimisation. In
International Conference on Learning Representations
(ICLR), pp. 1–19, 2018.

RPL: Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey
Levine, and Karol Hausman. Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement
Learning. In Conference on Robot Learning (CoRL),
2019

Decision Transformer: Chen, Lili, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. "Decision transformer: Reinforcement learning via sequence modeling." 

summary_of_the_review: In general, while the work has (at first glance) impressive empirical results as well as a novel formulation of the KL constraint between the RL policy and the behavior policy, there are quite a few concerns that would need to be addressed with direct justification and experimentation in order to recommend acceptance. Hence, I currently recommend rejection, but I am open to updating my score if all of my concerns can be addressed.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In a regime where we have access to offline behavior data from a suboptimal policy (a heuristic, human demonstrations, etc) we can use that data and "trust region"-based methods (TRPO in this work) to nudge exploration in the right direction by keeping the learnable policy close to the behavioral one. While there are several works exploring this general idea, the particular way presented in this work is novel and achieves better results on a set of MuJoCo environment. The approach is also shown to work on a physical robot (Turtlebot).
main_review: The paper is well-written and clearly states the problem, motivation and goals, and the approach taken to solve the problem at hand. Formal explanations and methods are presented rigorously and are supplemented with helpful intuitive explanations following each major block of formal mathematical language. The exploration of learning from demonstrations to improve learning in challenging settings (sparse rewards, long horizons, etc) is, in my mind, an important concept that also has ~immediate practical applications and implications, and the method presented here adds to the toolbox of ideas on how this can be approached.


Major concerns / criticisms / discussion
----------------------------------------

* From a quick read of the Kang et al. 2018 is indeed appears to be very close to the idea (at least in spirit) presented in this work. Authors mention how their approach is fundamentally different in terms of methodological approach, however I think the work would benefit from a bit more detailed exploration of the differences in consequences of applying POfD vs. LOGO. If we imagine a situation where both algorithms were presented with the same data (same random seed, same everything) -- what would be crucial difference between the two policies produced by the two methods? Maybe differences in value estimation? Degree of deviation from the behavior policy? What are the arguments one could present to decide "I will choose LOGO over POfD, because ..."?

* Conceptually there does not seem to be anything that would prevent using LOGO in a large-state-space setting, for example using RGB(D) inputs from a robot. Did the authors attempt LOGO in such an environment? Is the "partial observation" trick *required* to learn in such an environment or we would see the benefits of LOGO even when working with full state directly? If you did try this then were the results positive or negative, and if you did not, then what are your thoughts on feasibility and performance expectations?


Questions
---------

* general: How hard would it be to adapt the same idea to other algorithms, that do not have the build-in notion of a trust region? For example SAC and QR-DQN? Or another way to phrase the question: how specific is the proposed method (conceptually) to TRPO. Could including some kind of a penalty based on KL divergence on top of other algorithm also work, or this would be unlikely / highly non-trivial to achieve?

* page 1 par 2: "natural candidate to begin with is the framework of policy gradient algorithms <...> performs really well in the dense reward setting" -- Unclear why this is a natural choice while in the previous paragraph we were discussing sparse rewards. Wouldn't it be more natural to first consider something that works well with sparse rewards?

* page 2 par 1: "Our choice of the TRPO " -- Why not the superior methods based on similar principles like PPO?

* page 3 assumption 1: What does beta stand for in this equation?

* page 9 "task 2": What is the dimension of the state space when lidar data is included?

Minor remarks
-------------
* page 1 par 2: "behavior data might only contain measurements of a subset of the true state" -- It was a bit unclear why that would be the case. If I understand correctly this isn't something that happens often. Was this included in the manuscript because the authors have observed that LOGO works well for situations like this? If that is the case perhaps the sentence can be reformulated to say that, without the implication that partial state is something that tends to happen often.

* sec 5.1 par 1: "moves forward over 2, 20, and 2 units from" -- at typo?
summary_of_the_review: I find the paper's goals clear and interesting and the execution faithful and technically sound. This work makes a curious step in the direction of mixing together online learning and pre-existing policy data and I think would be an interesting result for others who do research in this area. I wouldn't mind seeing this work presented at the conference.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents the learning online with guidance offline (LOGO) algorithm that leverages demonstration data to constrain policy search for reinforcement learning with sparse reward such that the initial exploration phase is guided. Experiments in locomotion tasks in simulated domains and a navigation task on a real robot demonstrates LOGO is more data-efficient compared to a range of baseline methods.

main_review: This paper studies the problem of how to use demonstration data to better aid reinforcement learning with sparse reward, which is an important task setting that will have a wide impact on application of RL. Prior work in this direction has focused on directly imitating the demos or only using the demonstrated states to constrain exploration. The proposed LOGO algorithm leverages the per-state action distribution as a constraint for guiding policy search. The authors also show that an important advantage of LOGO is that the demonstration data need not cover the entire state space, and the algorithm is capable of matching distributions in down-projected state spaces.  This advantage allows the algorithm to be applicable to a wider range of demonstration types than pure imitation-based algorithms do. The authors performed a theoretical analysis on the algorithm and conducted experiments in a range of different tasks, including a navigation task on a real robot. The writing of this paper is clear and well organized for the readers to follow.


One weakness of the paper is that it misses one important baseline method, which is to warm-start the policy using behavior cloning with demonstration data and then run RL in the similar manner. This baseline uses the same amount of auxiliary data (i.e. the demonstrations) and potentially needs less amount of time interacting with the environment since the supervised BC step is offline. Current baseline methods do not fully leverage the data available in this problem setting. TRPO ignores the demonstrations, GAIL ignores the RL reward, and PofD only uses the state distribution instead of the state-action distribution from the demonstrations. Therefore, adding the baseline described above is important to make an apples-to-apples comparison between BC and using demos as guidance. Apparently warm-start with BC does not offer the generalizability to partially-observed demonstrations and therefore LOGO still has its unique advantages even if it is not able to outperform this baseline

summary_of_the_review: This paper studies an important problem of how to leverage demonstrations for reinforcement learning and proposes a novel method that is demonstrated to outperform a set of baselines in various domains. It makes the case for how distribution matching should be used instead of pure supervised learning of demonstrations and shows an unique advantage of doing so, which is the capability of dealing with partially observable states. I recommend acceptance of this paper but would love to see another baseline added to the experiments as I mentioned in my review above.
review_title: null

Meta-review: The authors introduce a method for improving reinforcement learning in sparse reward settings. In particular, they propose to take advantage of a suboptimal behavior policy as a guidance policy that is incorporated in a TRPO-like update. The reviewers agree that this is a novel and interesting idea and given the authors' rebuttal with additional experiments, clarifications and discussions, they agreed to accept the paper. However, they also point out several flaws (e.g. evaluation on a more challenging sparse-reward task such as Adroid) that I encourage the authors to address in the final version of the paper.
Decision: Accept (Spotlight)