Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: 
The work is heuristically motivated by the goal of reducing the high computation of model-based learning while achieving high performance. For achieving that, the authors propose an algorithm, Adaptive Online Planning (AOP) combining a model-free policy learning method and a model-based planner. In terms of the empirical study, they test the algorithm in 3 environments, Hopper, Ant, and Maze. They compare their algorithms with several model-based methods.

From my perspective, the paper has several weaknesses for which I give a weak rejection. 

The motivation is interesting to me, but the authors do not provide enough justification. The authors claim that the proposed method is able to reduce high computation. However, seemingly they only intuitively illustrate how it saves energy without strong proofs, which weakens the claim. What’s more, the experiment is not clear to me. What are the take-aways of Figure 3 and Figure 4 while I cannot see an improvement from them? There is no comparison in Figure 6; not clear how the plots of other models look like. The last comment is about the 3 environments that are not complex enough.


Minor comments:
- Some typos and grammar mistakes, e.g., ‘planing’ and ‘(d)by’ in the third last line (p.4); the second sentence in Sec. conclusion (p.8).
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: The paper presents an adaptive online planning(AOP) strategy in a model-free policy setting, a reinforcement learning method aimed to solve catastrophic forgetting problem by combining model-based planning and model-free policy learning. AOP is able extensive plan only when necessary, leading to over all average reduced computation times. AOP can be easily integrated into other reinforcement learning frameworks such as to any offline-planning reinforcement learning algorithms.  The experiments demonstrate that AOP is computationally efficient compared to traditional baselines MPC-8 and MPC-3 while maintaining the performances.

The algorithm is developed based on heuristic solutions to address some of the fundamental problems in reinforcement learning, and although the proposed strategies definitely seem to provide some benefits in terms of computation complexity, the solution is not very elagant or noval. It is hard to justify the computational efficiency and performance in dynamically changing environments just based on the presented results. While the improvement in computation is there, what I find lacking is the experiments demonstrating clear evidence of overcoming catastrophic forgetting problem. The paper gives off a feeling that AOP as an add-on that can increase the performance of any  RL algorithm. 

review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: The authors study continual, lifelong learning. They suggest a new algorithm, named Adaptive Online Planning (AOP) that combines model-based planning with model-free learning. AOP decides how much additional planning is needed based on the uncertainty of the model-free learner and the performance of the planner. Experiments are carried out on three tasks, i.e. Hopper, Ant and a Maze.

This paper should be rejected. The main reason is that the experiments were only performed for 3 different seeds and are therefore not statistically relevant (see Henderson et al. "Deep reinforcement learning that matters." Thirty-Second AAAI Conference on Artificial Intelligence. 2018.). 

Besides the issue of significance of the results section, there are other concerns. Some of them are:
- Page 2: 'The dynamics model is updated immediately at world changes.' - Is this a reasonable assumption? Where does an accurate model come from? Given a perfect model, it is not surprising that a learner that is combined with such a model achieves a superior performance.
- Although the authors state that the 'ability to perform well in old tasks (backward transfer)' is important, they don't explicitly show their algorithm to achieve this goal. Backwards transfer might be included into the experimental section, but I could not find a statement that addresses this explicitly.
- I would like the authors to crisply define their use of the word 'online learning'. Does online learning simply mean to process each sample as it is available or does the term include real-time?
- How is \sigma_{thres} chosen? What is the influence of this parameter?
- The statement that 'AOP uses only 0 - 25% of the number of timesteps as MPC-8, but achieves generally comparable or stronger performance.' is wrong (see Fig 4, d and e). This statement is especially difficult, as results are only averaged over 3 runs.

There are furthermore a few minor concerns:
- the interval for \gamma should exclude 1 in this setting, as the return would otherwise be unbounded.
- In the background section, the authors confuse the definition of the return with reward.
- the term 'deep exploration' is used but not defined
- There are two figures between the subsection header for 4.4 and the text - this is highly confusing

review_title: Official Blind Review #1

Meta-review: A new setting for lifelong learning is analyzed and a new method, AOP, is introduced, which combines a model-free with a model-based  approach to deal with this setting.

While the idea is interesting, the main claims are insufficiently demonstrated. A theoretical justification is missing, and the experiments alone are not rigorous enough to draw strong conclusions. The three environments are rather simplistic and there are concerns about the statistical significance, for at least some of the experiments.
Decision: Reject