Adaptive Online Planning for Continual Lifelong Learning
Authors: Kevin Lu
Igor Mordatch
Pieter Abbeel
Venue:  (2020)
Keywords: reinforcement learning
model predictive control
planning
model based
model free
uncertainty
computation

Abstract: We study learning control in an online lifelong learning scenario, where mistakes can compound catastrophically into the future and the underlying dynamics of the environment may change. Traditional model-free policy learning methods have achieved successes in difficult tasks due to their broad flexibility, and capably condense broad experiences into compact networks, but struggle in this setting, as they can activate failure modes early in their lifetimes which are difficult to recover from and face performance degradation as dynamics change. On the other hand, model-based planning methods learn and adapt quickly, but require prohibitive levels of computational resources. Under constrained computation limits, the agent must allocate its resources wisely, which requires the agent to understand both its own performance and the current state of the environment: knowing that its mastery over control in the current dynamics is poor, the agent should dedicate more time to planning. We present a new algorithm, Adaptive Online Planning (AOP), that achieves strong performance in this setting by combining model-based planning with model-free learning. By measuring the performance of the planner and the uncertainty of the model-free components, AOP is able to call upon more extensive planning only when necessary, leading to reduced computation times. We show that AOP gracefully deals with novel situations, adapting behaviors and policies effectively in the face of unpredictable changes in the world -- challenges that a continual learning agent naturally faces over an extended lifetime -- even when traditional reinforcement learning methods fail.

Under review as a conference paper at ICLR 2020
ADAPTIVE ONLINE PLANNING FOR CONTINUAL LIFE- LONG LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
We study learning control in an online lifelong learning scenario, where mistakes can compound catastrophically into the future and the underlying dynamics of the environment may change. Traditional model-free policy learning methods have achieved successes in difficult tasks due to their broad flexibility, and capably condense broad experiences into compact networks, but struggle this setting. as they can activate failure modes early in their lifetimes which are difficult to recover from and face performance degradation as dynamics change. On the other hand, model-based planning methods learn and adapt quickly, but require pro- hibitive levels of computational resources. Under constrained computation limits, the agent must allocate its resources wisely, which requires the agent to understand both its own performance and the current state of the environment: knowing that its mastery over control in the current dynamics is poor, the agent should dedicate more time to planning. We present a new algorithm, Adaptive Online Planning (AOP), that achieves strong performance in this setting by combining model- planning with model-free learning. By measuring the performance of the planner and the uncertainty of the model-free components, AOP is able call upon more extensive planning only when necessary, leading to reduced computation times. We show that AOP gracefully deals with novel situations, adapting behaviors and policies effectively in the face of unpredictable changes in the world challenges that a continual learning agent naturally faces over an extended lifetime even when traditional reinforcement learning methods fail.
INTRODUCTION
We consider agents in human-like setting, where agents must simultancously act and learn in the world continuously with limited computational resources. All decisions made online; there no discrete episodes. Furthermore, the world is vast too large to feasibly explore exhaustively - and changes over the course of the agent's lifetime, like how robot's actuators might deteriorate with continued use. There are no resets to wipe away past errors. Mistakes are costly, as they compound downstream. To perform well reasonable computational costs, the agent must utilize its past experience alongside new information about the world to make careful, yet performant, decisions.
Non-stationary worlds require algorithms that are fundamentally robust changes in dynamics. Factors that would lead to a change in the environment may either be too difficult or principally undesirable to model: for example, humans might interact with the robot in unpredictable ways, or furniture in a robot's environment could be rearranged. Therefore, we assume that the world can change unpredictably in ways that cannot be learned, and focus on developing algorithms that instead handle these changes gracefully, without using extensive computation.
Model-based trajectory optimization via planning is useful for quickly learning control, but is com- putationally demanding and can lead to bias due to the finite planning horizon. Model-free rein- forcement learning is sample inefficient, but capable of cheaply accessing past experience without sacrifices asymptotic performance. Consequently, from an intelligent planner into neural networks to reduce computation for future decision making.
would like distill expensive experience
Deciding when to the planner learned policy presents a difficult challenge, as it is hard to evaluate the improvement the planner would give without actually using the planner. We tackle this a problem of uncertainty. When uncertain about course of action, humans
elongated

---

Under review as a conference paper at ICLR 2020
model-based search to evaluate long-term trajectories, but fall back habitual behaviors learned with model-free paradigms when they are certain of what to do (Dayan & Berridge, 2014 Daw
et al. 2005; Banks & Hope 2014; Kahneman 2003). By measuring this uncertainty, informed decisions about when to use model-based planning vs a model-free policy.
make
Our approach combines model- planning with model-free policy learning, along with an adap- tive computation mechanism, to tackle this setting. Like robot that is well-calibrated when first coming out of factory, we give the agent access formation about future changes to the dynamics, like different settings in which the robot may be deployed. This allows us to make progress on finite computation continual learning without worry- ing about model learning. The dynamics model is updated immediately at world changes. However, as we show empirically, knowing the dynamics alone falls far short of success at this task.
ground truth dynamics model that lacks in-
We present a new algorithm, Adaptive Online Planning (AOP), that links Model Predictive Path Integral control (MPPI) (Williams et al. 2015). model-based planner, with Twin Delayed DDPG (TD3) (Fujimoto et al.] 2018b), a model-free policy learning method. We combine the model- based planning method of iteratively updating a planned trajectory with the model-free method of updating the network weights to develop unified update rule formulation that is amenable to reducing computation when combined with a switching mechanism. We inform this mechanism with the uncertainty given by an ensemble of value functions. Access to the ground truth model is not sufficient by itself, as we show that PPO (Schulman et al.) 2017) and TD3 perform poorly, even with the ground truth model. We demonstrate empirically that AOP capable of integrating the two methodologies to reduce computation while achieving and maintaining strong performance in non- stationary worlds, outperforming other model-based planning methods and avoiding the empirical performance degradation of policy learning methods in changing world scenarios.
Our contributions include the proposal of a new algorithm combining model-based planning and model-free learning, the introduction of evaluation environments that target the challenges of life- long reinforcement learning which traditional methods struggle, and experiments showing the usefulness of utilizing both model-based and model-free methods in this setting. Code to run all algorithms and experiments is available at www. github..hh.... com/ com/((lin redacted for anonymity).
2 BACKGROUND
We consider the world an infinite-horizon Markov Decision Process (MDP), defined by the tuple M = { R, T,7), where S is the state space, AA is the action space. R S x A - IR is the reward function, T S x A - R are the transition probabilities, and [0, 1) is the discount factor. The world can change over time: the transitions and rewards (T, R) may change to some new R')... Unlike traditional reinforcement learning, the agent's state is not reset these world changes. The agent can generate rollouts using the current (T, R) starting from its current state, but not for future (T', R'). The agent's goal is execute future sequence of actions, summarized as a policy क (aa ) that maximizes the expected future return R(t) Er [E k (r(ss, at)].
2.1 CONTINUAL ONLINE LEARNING
In work, consider online learning in the same style Lowrey al. (2118), where both acting and learning must occur on a per-timestep basis, and there are no episodes that reset the state. At each timestep, the agent must execute its training procedure, and is then forced to immediately output an action. We also desire agents that are equipped, like humans, to handle different tasks in various environments. Continual learning is difficult, as agents must use their experience learn to perform well in new tasks (forward transfer) while preserving the ability to perform well in old tasks (backward transfer). In addition the these difficulties, there is also the challenge of avoiding failure sink states that prevent future learning progress. We augment this task with world where the dynamics continually change, creating difficult setting for agent learning.
2.2 MODEL-BASE PLANNING
Online model-based planning evaluates future sequences of actions using a model, develops a pro- jected future trajectory over some time horizon, and then executes the first action of that trajectory,
2

---

Under review as a conference paper at ICLR 2020
before repeating. We specifically focus Model Predictive Control (MPC), which iteratively ap- plies Gaussian noise to the prior predicted trajectory, evaluates them using the dynamics model, and combines them with an update rule. When the update rule is softmax weighting, this procedure is called Model Predictive Path Integral (MPPI) control (Williams et al.) 2015). Due to the nature of this iterative and extended update, this procedure is computationally expensive.
2.3 MODEL-FREE POLICY OPTIMIZATION
Model-free algorithms encode the agent's past experiences in a function dependent only on the cur- rent state, often in the form of value function critic and/or policy actor. As result, such algorithms can have difficulty learning long-term dependencies and struggle early on in training; temporally ex- tended exploration is difficult. In exchange, they attain high asymptotic performance. having shown successes in a variety of tasks for the traditional offline setting (Mnih et al.) 2013 Schulman et al., 2017]. As consequence of their compact nature, once learned, these algorithms tend to generate cyclic and regular behaviors, whereas model- based planners have no such guarantees.
We rum online versions of TD3 (Fujimoto et al. (018b) and PPO (Schulman al., 2017) to AOP. While there is natural way to give policy access to the ground truth model, allow the policies train on future trajectories generated
baselines
the ground truth model, in similar fashion to algorithms that learn model for this purpose (Kurutach et al. 2018 Buckman et al. 2018), in order to help facilitate fair comparisons to model-basee planners.
2.4 UPDATE RULE PERSPECTIVE ON PLANNING POLICY OPTIMIZATION
From a high-level perspective, the model-based planning and model-free policy optimization proce- dures are very similar (see Appendix for a side-by-side comparison). Where the planner generates noisy rollouts to synthesize generate data for learning. After an update step, either an action from the planned trajectory or one call of the policy is executed. These procedures only distinct in their respective update rules.
trajectory, the model-free algorithm applies noise to the policy to
The primary contribution AOP is unifying both update rules to compensate for their individ- ual weaknesses. AOP distills the learned experience from the planner into the off-policy learning method of TD3 and a value function, so that planning and acting can be done cheaper the future.
ADAPTIVE ONLINE PLANNING
Prososes uppior
Supslanirspeer
improvmnnnt&
Chocas hortron anserntie
movnible
bulfe/
arplaninvents
en to traestores
Mudel RL.
Medel Bases/R
Figure Schematic view of Adaptive Online Planning (AOP).
AOP combines model-based planning with model-free learning using an adaptive switching mecha- nism, summarized above in Fig. [ and written with more detail in Appendix B in Alg.
3.1 MODEL-BASED PLANNING WITH TERMINAL. VALUE APPROXIMATION
For the model-based component, AOP uses MPPI, as described in Section 2.2 with terminal value function V. where trajectories are evaluated in the form of Eq. This process repeated for several iterations to improve the plan, and then the first action is executed in the environment.
H-1
R(T) = Σ (r(s + +V((s)
(1)
0=0
3

---

Under review as a conference paper at ICLR 2020
V. is generated by an ensemble of value functions (see Eq. as proposed in POLO (Lowrey al. 2018) for MPC. The value ensemble improves the exploration ability of the optimization procedure (Osband al. 2016 2018). The log-sum-exp function serves softmax, enabling exploration while preserving a stable target for learning. The log n term normalizes the estimate to lie between the mean and the max of the ensemble, determined by the temperature hyperparameter K, which ensures that the approximation is still semantically meaningful for estimating the value of the state.
V(s)
log
xVi(s)-log
3.2 EARLY PLANNING TERMINATION
Past model- based planning procedures (Chua et al. 2018 Wang & Ba, 2019) run fixed number of iterations of MPC per timestep before executing an action in the environment. However, this is often wasteful. Within particular timestep, later planning iterations often improve the planned trajectory less than earlier iterations, and may not even improve the trajectory at all. We propose to decide on per-timestep basis. After generating new trajectory Tk+1
the number of planning iterations from the k-th iteration of planning, we measure the improvement A (T<++1 (7k) against the trajectory Ta of the previous iteration (Eq. 3}. When this improvement decreases below a threshold A Ahrees terminate planning for the current timestep with probability Cplan- Using a stochastic termination rule allows for robustness against local minima where more extensive planning may be required, but not evident from early planning iterations, in order to escape.
((TR+//T)) R(TK+1) R(TR)
(3)
R((Tx)]
3.3 ADAPTIVE PLANNING HORIZON
Since planning over a long time horizon is expensive, it would also be desirable to plan over shorter time horizon when the planner is confident in achieving long-term success with only short- term planming. We represent the planner's uncertainty with the value ensemble from Section 3.1 as the mean and standard deviation of the ensemble represent the epistemic uncertainty of the value of the state (Osband al., 2018). When deciding use reduced time horizon, require that the standard deviation a of the value ensemble on the current state be lower than some threshold Othres-
A problem with only considering the standard deviation is that the metric only considers uncertainty with respect to the past - it does not immediately measure uncertainty in a changing dynamics fine-tune setting, which is only observed when considering experiences in the future. Therefore, the horizon length using the Bellman error (Eq. 4 The time horizon is given by the largest H < Il full such that e (TK) > Ethres. When a Othress the full time horizon is always used, regardless of the Bellman error. This is to say that, if the value function can accurately approximate the latter part of the horizon, we can use the value function instead. While choices for these hyperparameters are somewhat arbitrary, we show in Appendix C.I.I that AOP is not particularly sensitive to them.
1
(H) TTs jull (R (T M44M 12
vi(s++)))
3.4 OFF-POLICY MODEL-FREE PRIOR
We use TD3 prior to the planning procedure, with the policy learning off of the data generated by the planner during planning, which allows the agent to recall past experience quickly. Similarly found that imitation learning can cap
to past work (Rajeswaran al. 2017a Zhu al. 2018). the asymptotic performance of the learned policy. As a baseline, we also run behavior cloning (BC) prior, and refer to the resulting algorithms AOP-TD3 and AOP-BC, respectively.
We note that MPC and policy optimization are both special cases of AOP. MPC is equivalent to AOP with constant setting for the time horizon that always full planning iterations (i.e. threshold
4

---

Under review as a conference paper at ICLR 2020
of 0). Policy optimization is equivalent AOP with planning iteration, since the first plan is noisy version of the policy, acting as the data collection procedure standard policy learning.
EMPIRICAL EVALUATIONS
We investigate several questions empirically:
1. What are the challenges in the continual lifelong learning setting? When developing further algorithms in this setting, what should we focus on?
2. How does AOP perform when acting from well-known states, novel states, and in changing worlds? How do traditional on-policy and off- off-policy methods fare in these situations?
3. Are the variance and the Bellman error of the value ensemble suitable metrics for deter- mining the planning computational budget?
4.1 LIFELONG LEARNING ENVIRONMENTS
We propose three environments to evaluate our proposed algorithm in the continual lifelong learning setting:
Hopper, Ant, and Maze, and release them for others to use. While these environments not overly complex control environments, they crisply highlight the difficulties of continual lifelong learning in MDPs. Pictures of these environments are included in Appendix[D
First, we consider the OpenAl Gym environment Hopper (Brockman et al. 2016). The agent is rewarded based on how closely it matches an unobserved target velocity. Every 4000 timesteps, this target velocity changes. The environment is tough, as it can be difficult to get up if the agent falls down strange position, and momentum from past actions affect the state greatly, which makes it easy for the agent to fall over. We consider three versions of the Hopper environ- ment: (1) a standard Hopper with an unchanging target velocity, (2) a novel states Hopper with the target velocities correspond to the agent seeing new
Hopper:
target velocity in the observation (thus state), and (3) a changing worlds Hopper, where the target velocity is not in the observation.
Ant: We also consider the Ant from Gym. The agent seeks maximize its forward velocity, but joint at random disabled every 2000 timesteps. Once flipped over, getting back up is extremely dif- ficult, which makes this environment harshly unforgiving. We consider two versions: (1) standard Ant with no disabled joints, and (2) changing worlds Ant with one changing disabled joint.
Maze: Like in POLO, we test in 2D point mass maze, where agent seeks reach goal. The observation is (Xpoint: Upoint; 3 gool, Mgoot). Every 500 timesteps, the walls of the maze change, and the goal swaps locations. The difficulty lies in adapting quickly new mazes while avoiding the negative transfer of old experience. We consider two versions: (1) a novel states Maze, where the walls of the maze remain constant, but new goals are introduced after 20 goal changes in the original positions, and (2) changing worlds Maze, which is as described above. We also test both versions dense reward and a sparse reward setting, where the reward either the negative I.2 distance or boolean value, respectively. In the sparse reward Maze, exploration can be particularly challenging.
4.2 BASELINES AND ABLATIONS
We run AOP-BC, POLO, MPC, TD3, and PPO as baselines against AOP-TD3; they can be seen as ablations/special cases of our proposed algorithm (see Section 3.4b. We consider versions of MPC, with 8 and 3 planning iterations, henceforth referred to as MPC-8 and MPC-3, respectively.
Table 1: Timesteps rolled out by planner (planning levels) as a fraction of MPC-8 for based planning algorithms. Shown are the average over environments and the range (min-max) across the environments for 5 seeds. For more detailed graphs, see Fig. A. in Appendix A
AOP-TD3
AOP-BC
POLO
MPC-8
MPC-3
11.39% % (1.40% 16.62%)
11.40% (2.86% 15.17%)
37.50%
100%
37.50%
5

---

Under review as a conference paper at ICLR 2020
Table Average lifetime rewards. NS, and CW, denote the standard, novel states, and changing worlds settings; (D) and (S) denote dense/sparse reward mazes. Shown is the average for 5 seeds with two standard deviations. Best results are bolded. See Appendix [A full learning curves.
Environment
AOP-TD3
AOP-BC
POLO
TD3
PPO
MPC-8
MPC-3
S Hopper
0.12 + 0.16
0.33 + 0.22
0.51
0.23
-14.41
0.36
0.19
NS Hopper
0.41 + 0.18
0.53 0.18
0.59
0.40
-14.22
-0.28
-0.49
CW Hopper
0.48 + 0.24
0.45
0.57
-2.42
-13.14
-0.30
-0.48
S Ant
3.02 - 0.13
3.27
3.40
2.19
n/a
3.52
3.40
CW Ant
2.76 + 0.47
3.11 0.41
2.90
2.05
n/a
3.32
3.14
NS Maze (D)
-0.21 + 0.08
-0.25 0.02
-0.25
-1.81
-2.14
-0.19
-0.25
CW Maze (D)
-0.29 + 0.07
-0.34 #
-0.30
-1.17
-2.10
-0.19
-0.30
NS Maze (S)
0.85 + 0.07
0.70 + 0.06
0.62
-0.68
-0.88
0.69
0.61
CW Maze (S)
0.69 + 0.20
0.56 0.04
0.57
-0.66
-0.74
0.58
0.52
ADP-TD3
AOP-BC
POLO
TD3
PPO
MPC-8
MP03
13
2003 3000
1300 3790
19300
170M 2004
2004
3000 1311 1930 1708
2000
122300 11000
30000
20000
(a) Changing worlds Hopper
(b) Changing worlds Ant
(c) Changing worlds Maze (D)
Figure Reward curves for changing worlds lifelong learning tasks. Rewards are for a single timestep, not over episode. Note that some worlds may be more difficult than others, and yield naturally lower reward. The results are averaged over 5 seeds; the shaded area depicts one standard deviation above and below the mean. Curves are smoothed and the rewards are clipped -3 for visual clarity. See Fig. A.2 in Appendix for corresponding plots for all environments.
4.3 CHALLENGES IN CONTINUAL LIFELONG LEARNING SETTING
Planner usage is shown in Table and rewards are in Table AOP uses only 1 - 17% of the number timesteps as MPC-8, but achieves generally comparable or stronger performance in most environments. More detailed graphs can be found in Appendix A
Even with model access, these environments are challenging the algorithms to learn. In the standard offline reinforcement learning setting, long-term action dependencies learned from past experience over time, and this experience can be utilized when the agent resets to the initial state. However, in the online setting, these dependencies must be learned the fly, and the agent falls, it must return to the prior state order to use that information. In particular, for the Ant environment, such falling is catastrophic, to standing. POLO-style optimistic exploration can thus be disadvantage, encouraging the Ant to take on new and unstable behaviors. In spite of this, AOP, with about 39% of the planning of POLO, achieves comparable performance to POLO; AOP-BC achieves very strong performance, in general.
Reset-Free Setting:
it takes complex action sequence return
The performance gain of MPC-8 over MPC-3 shows that achieving strong perfor- mance is difficult with constrained computation. In the sparse mazes, MPC is significantly outper- formed by AOP-TD3, and the model-free algorithms struggle to make any progress all, showing their lackluster exploration. Even POLO the exploration mechanism of AOP - faces weaker per- formance, indicating that AOP-TD3 has not only correctly identified when planning important, but is able to effectively leverage additional computation to increase its performance whilst still using less overall computation. The additional performance in the novel states Maze (S) over MPC-8 also before.
Vast Worlds:
shows AOP's ability to consolidate experience improve performance in mazes it has Furthermore, the changing worlds Maze (S), the performance AOP improves over time (Fig. A.2 indicating that AOP has learned value and policy functions for effective forward transfer.

---

Under review as a conference paper at ICLR 2020
Policy Degradation: TD3's performance significantly degrades in the changing worlds settings, as does PPO's s (see Fig. 2 PPO, an on-policy method, struggles in general. In the novel states Hopper, the variant where the policy is capable of directly seeing the target velocity, TD3 performs very well, even learning to outperform MPC. However, without the help of the observation, in the changing worlds, TD3's performance quickly suffers after world changes. The model-based planning methods do not suffer this degradation, and AOP is able to maintain its performance and computational savings, even through many world changes, despite its reliance on model-free components.
4.4 BEHAVIOR OF POLICIES IN CONTINUAL LIFELONG LEARNING SETTING
Egisoda
-10000
-10000
7500 10000 12500
37500
200200
400000 Timestops soan
600000
1003000
Timestop
Figure 3: Policy episode performance for changing worlds Hopper. Left: performance of policy only (no planner) throughout training. Right: performance on initial starting state and target velocity after additional TD3 training (blue: AOP-TD3, red: TD3, gray: from scratch). 5 seeds are shown.
In Fig. 3 (left), we plot the episodic reward of the policy running from the agent's current state for each timestep (for the current target velocity). Note that since the AOP policy is learned from off- policy data (the planner), it suffers from divergence issues and should be weaker than TD3 on its own (Fujimoto et al. 2018 Matching the result in Fig. (a), the TD3 policy degrades in performance over time, but the AOP policy does not. This suggests that the policy degradation effect might stem from exploration, rather than from an issue with the optimization algorithm.
We also show in Fig. 3 (right) tuning the policy learned by AOP after seeing every target velocity once (blue) vs. by TD3 (red) vs. training new policy (gray), learning from running the standard episodic TD3 algorithm on the first target velocity. The AOP policy learns much faster, showing that AOP is capable of quick backward transfer and adapting quickly different situation.
4.5 BEHAVIOR OF AOP IN
STATES/NOVEL STATES/CHANGING WORLDS
Figure 4: Graphs for Maze (D). From left to right: (I-NS): average Bellman error of the the first
time a goal is presented (blue) vs the last time it is presented (orange). (2-NS) average number of planning steps. Events denote the 4 times the set of goals switch during the agent's lifetime. (3- CW): average Bellman error by the time since the last world change. (4-CW); average number of planning steps. Both quickly decrease the agent becomes more familiar with the world.
Fig. shows AOP behavior in Maze (D). When encountering novel states, Bellman error is high, but as time progresses, when confronted with the same states again, Bellman error becomes low. The number of planning timesteps matches this: AOP correctly identifies a need to plan early on, but greatly saves computation later, when it is immediately able to know the correct action with almost no planning. The same effect occurs when measuring the time since the world changed for the changing worlds. At the beginning of a new world, the amount of planning is high, before quickly declining to nearly zero, almost running with the speed a policy: R 100x faster than MPC-8.
We plot the standard deviation and Bellman error over time of AOP for the changing worlds Hopper in Fig. 5 After each world change, the Bellman error spikes, and then decreases as time goes on. These trends are reflected in the time horizon (bottom center), which decreases as the agent trains in each world, and indicate that the standard deviation and Bellman error are suitable metrics for determining planning levels. The same effect also occurs for the number of planning iterations.
7

---

Under review as a conference paper at ICLR 2020
290
100
313
J030
2050
15300
27100
2007
2006
X0
750
1000
12M
1909
37500
2000
200
700
1908
12TM
1906
STA
2010
1900
500
750
1000
10000
27500
3000
750
2000
Figure 5: Graphs for AOP in changing worlds Hopper. Red lines denote world changes. Policy uses the percent of the time that the policy was used instead of the initial plan (see line 3 of Alg. 3
In Fig. we show qualitatively how TD3, MPC, and AOP handle world changes the changing worlds Hopper setting. An unobserved change in the target velocity from 2.5 to is encountered during the agent's lifetime. The purely model-free TD3 produced cyclic and regular behavior, but adapted slowly to the target speed, still moving at 2.5. MPC quickly adapted to the new target speed, but moved in an irregular fashion. AOP was both able to rapidly adapt and do so in a regular manner.
T
MCCA
NVV
Figure 6: Example trajectory traces of Cartesian hopper positions for different algorithms. An unobserved change in target speed is encountered at the timestep marked by red outline.
RELATED WORK AND FUTURE DIRECTIONS
Continual online learning: Much past lifelong learning work (Goodfellow al.) 2013 Parisi et al. 2018) has focused on catastrophic forgetting, which AOP is resilient to, but was not primary focus of our work. Kearns & Singh (2002) considers MDPs with various reward functions, using them to make conscious decisions about exploration exploitation, similar to our framework. Finn jtt al. (2019) uses meta-learning to perform continual online learning, but the tasks are considered in episodes. Nagabandi et al. (2018) learn multiple models to represent different tasks, which contrasts with our single unified policy/set of value functions.
Augmented planners: Algorithms that combine planning with learning have previously been stud- ied both discrete and continuous domains (Chua et al. 2018; Anthony et al. 2017). Recent work (Guez et al., 2018) generalizes the MCTS algorithm and proposes to learn the algorithm instead; fur- ther asking the algorithm to set computation levels could be effective in our setting as well. [evinee] & Koltun| (2013); MModdttch al. (2015) propose use priors that make trajectory planner stay close to policy outputs, which is problematic in changing worlds when the policy is not accurate.
Learned dynamics: Many previous works (Azizzaeenesheli al. 2018 Nagabandi et al., (017) have learned dynamics models and then performed MPC optimization on them. Kurutach et al. (2018); Clavera et al. (2018) utilized model ensembles reduce performance degradation due to model overfitting, and (Janner et al.) 2019) investigates model uncertainty for policy learning. Inte- grating AOP with learned uncertainty-ave dynamics model would be interesting future work.
CONCLUSION
We proposed AOP, which incorporates model-based planning with model-free learning, and intro- duced environments for evaluating algorithms in the continual lifelong learning setting. We em- pirically analyzed the performance of and signals from the model-free components, and showed experimentally that AOP was able to successfully reduce computation while achieving high perfor- mance in difficult tasks, often competitive with much more powerful MPC procedure.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search. CoRR, abs/1705.08439, 2017. (RRt::/://axxx http:///awxww....//asssss/. org/ 1705. 08439
Kamyar Azizzadenesheli, Brandon Yang. Weitang Liu, Emma Brunskill, Zachary C. Lipton, and Animashree Anandkumar. Sample-efficient deep RL with generative adversarial tree search. CoRR, abs/ 806.05780. 2018.
wttp:xii..rr//bb//////01 05780
Adrian Banks and Christopher Hope. Heuristic and analytic processes in reasoning: An event-related potential study of belief bias. Psychophysiology, 51, 03 2014. doi: 10. 10.1111/psyp.12161
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. CoRR, abs/1606.0155 005400 2016. Jttp://///aaaxi org/
abs/1606. 01540
Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, and Honglak Lec. Sample-efficient reinforcement learning with stochastic ensemble value expansion. CoRR,
abs/1807.01673 2018. URL http:tt:://ant /ttp///axxvvv owt./ httpg//bb/118..000 01655
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. CoRR, abs/1805. 2018.
URL httpt:tpp///ii...ct///// org/ abs 805. 111.5.21111
Ignasi Clavera, Jonas Rothfuss, John Schulman, Yasuhiro Fujita, Tamim Asfour, and Pieter Abbeel. Model-based reinforccmeet learning via meta-policy optimization. CoRR, abs/1809 180.00521,, 2018.
URL
org abs/180999 05214
Nathaniel D Daw, Yael Niv, and Peter Dayan. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nature neuroxcience, 8(12):1704, 2005.
Peter Dayan and Kent C Berridge. Model-based and model-free pavlovian reward learning: Reval- uation, revision, and revelation. 2014.
Chelsea Finn, Aravind Rajeswaran, Sham M. Kakade, and Sergey Levine. Online meta-learning. CoRR, abs/1902.084338 2019. URL Utthhtpp//aarri..oor////////
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcment learning without exploration. CoRR, abs/ abss/81// 1812.02900, 2018a. URL//ttp://aaxvvv.ttt/a//////// 02900
Scott Fujimoto, Herke van Hoof, and Dave Meger. Addressing function emor in actor- critic methods. CORR, abs/1802.099777, 2018b. URL htt.:tt:://aaxii.oggabbb///////
09477
lan J. Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investigation of catastrophic forgetting in gradient-based neural networks, 2013.
Arthur Guez, Théophane Weber, Ioannis Antonoglou, Karen Simonyan, Oriol Vinyals, Daan Wier-
stra, Rémi Munos, and David Silver. Learning to search with metsnets. CoRR, abs/1802.00497, 2018. URL hatp:: //t//aaxiii www.aaivvvoooom wwwpaawwwww./////////000 /ttp///0bbb1100046667777
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine, When to trust your model: Model- based policy optimization. CoRR, abs/1906.0822. 2019. URL http::/aaxiii..tt///// org/abg/
1906. 08253
Daniel Kahneman. Maps of bounded rationality: Psychology for behavioral economics. American Economic Review, 93:1449- 147144499475,1, 02 2003. doi: 10. (0.12577000222803325553999
Michael Kearns and Satinder Singh. Near optimal reinforcement learning in polynomial time. Ma-
chine Learning, 49(2): 232, Nov 2002. ISSN 1573-0565, doi: 10..123/A.10119844113558
URL http://doooooog/////
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. CoRR, abs/1802. 10592, 2018. URL lttt:://axxiv.... org/
abs/ 1802 11.80...00 10592

---

Under review as a conference paper at ICLR 2020
Sergey Levine and Vladlen Koltun. Guided policy search. In International Conference Machine Learning, pp. 1-9, 2013.
Kendall Lowrey, Aravind Rajeswaran, Sham M. Kakade, Emanuel Todorov, and Igor Mordatch. Plan online, learn offline: Efficient learning and exploration via model-based control. CoRR, abs/1811.01848, 2018. URLhtt::://arx:: www.//rg/oooo abs//ass.111.1e/////11/11111.111 8 01888
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. CoRR, abs/1312.5602, 2013. URL http:////azzi/./ggg/////////////
Igor Mordatch, Kendall Lowrey, Galen Andrew, Zoran Popovic, and Emanuel V Todorov. Interactive control of diverse complex characters with neural networks. In Advances in Neural Information Processing Systems, pp. 3132 3140, 2015.
Anusha Nagabandi, Gregory Kahn, Ronald S. Fearing, and Sergey Levine. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. CoRR, abs/1708.02566, 2017. URL htt.:ttt///axx/.rrrgg http://///////o///o///////////// 708. 02596
Anusha Nagabandi, Chelsea Finn, and Sergey Levine. Deep online learning via meta-learning: Con- tinual adaptation for model-based RI.. CoRR, abs/1812.07611 2018. URL http://arxiv.
org erg/abs/ 18121111112 07671
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped DQN. CoRR, abs/1602.04621,, 2016. URL
04621
lan Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep re- inforcement learning. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 8617-8629, Curran Associates, Inc., 2018. URL htttttttpppppeereeeee nips cttpsp.rcppprr///ppppppppp cwnpioor-funct.o ons for reiforrceeee learning. de
8080 randomi
pdf
German Ignacio Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter. Con- tinual lifelong learning with neural networks: review. CoRR, abs/1802.07569, 2018. URL
http://arxiv org hwtp:////11a /00.2 00222.05666 07569
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv: 170:: 10087 2017a.
Aravind Rajeswaran, Kendall Lowrey, Emanuel Todorov, and Sham M. Kakade, Towards gener- 2017b. URL http:
alization and simplicity in continuous control. CoRR,
//arxiv.org hal//ab///1/ www.a/////1.... 02650
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, 010/17770033443 2017. URL http: /atp:v/aaxvv.vvv org/ abs/
1707_ 06347
Tingwu Wang and Jimmy Ba. Exploring model-based planning with policy networks. CORR, abs/1906.08649, 2019. URL Uttpttt:://aaxi .rt. htgg/bb//ie /abbs/1996.. 08649)
Grady Williams, Andrew Aldrich, and Evangelos Theodorou. Model predictive path integral control using covariance variable importance sampling. CoRR, abs/1509.01149, 2015. URL
.0149
Yuke Zhu, Ziyu Wang, Josh Merel, Andrei Rusu, Tom Erez, Serkan Cabi, Saran Tunyasuvunakool, János Kramár, Raia Hadsell, Nando de Freitas, et al. Reinforcement and imitation learning for diverse visuomotor skills. arXiv preprint 802554180,.09544, 2018
10

---

Under review as a conference paper at ICLR 2020
A DETAILED EXPERIMENTAL GRAPHS
Additional graphs are provided for the fraction of planning timesteps vs MPC-8 and reward curves for all environments in Figures A.T and A.2 respectively.
AOPTDS
AOP-BC
POLO
MPC-8
MPC-3
(a) Standard Hopper
(b) Novel states Hopper
Changing worlds Hopper
Tirastep
Teansay
Trrstip
(d) Standard Ant
(e) Changing worlds Ant
Novel states Maze (D)
Tirestop
g Changing worlds Maze (D)
(h) Novel states Maze (S) (i) Changing worlds Maze (S)
Figure A.1: Number of timesteps rolled out by planner per timestep percentage of MPC-8.
ACP-TD3
AOP-BC
POLO
TD3
PPO
MPC-8
MPC-3
Timatig
Timamp
(a) Standard Hopper
(b) Novel states Hopper
Changing worlds Hopper
Roard
B006
2000
2000 2530
1580 1218
1750
2000
250
200
750
200
2000
2000
3550
2000
(d) Standard Ant
(c) Changing worlds Ant
Novel states Maze (D)
Timerig
(g) Changing worlds Maze (D)
(h) Novel states Maze (S)
(i) Changing worlds Maze (S)
Figure A.2: Reward curves for lifelong learning tasks.
11

---

Under review as a conference paper at ICLR 2020
B PSEUDOCODE FOR ALGORITHMS
B.I MODEL-BASED PLANNING VS POLICY OPTIMIZATION PSEUDOCODE
Algorithm 1: Model-Based Planning
Algorithm 2: Policy Optimization
Initialize action trajectory Tplan
Initialize policy *
while alive do
2 while alive do
Generate n rollouts based on Tylon
Generate rollouts based on T
4 Use rollouts update Tplan
Use rollouts update T
Execute first action of Tplan
Execute an action from T
end
end
B.2 ADAPTIVE ONLINE PLANNING PSEUDOCODE
Algorithm 3: Adaptive Online Planning
Initialize value ensemble {Vi) policy replay buffers Dv, D while alive do
Set Tplan to arg maxx,, R(T), generating from the policy To
Select time horizon H, for planning. described in Section 3.3
for 1: < to max jiterssers do
Run MPC planning generate Tplan described Section 3.1 add trajectories Dr
If A(T++IT) < Athres. stop planning with probability 1 Eylan
end
If it's time to update: update value ensemble (Vi)== and policy vo with Dv Dr. resp. Step once in the environment with first action of Tploni add (8t, at r() to Dv
10
n end
C HYPERPARAMETER DETAILS
All of our implementations and hyperparameters are available at www. ..w.giie ccmb (link redacted for anonymity).
C.1 ADAPTIVE ONLINE PLANNING HYPERPARAMETER,
For AOP, we set Othres 8, Ethres 25 and Eplan 0.2. We did not tune these hyperpa- rameters much, and similarly do not believe that the algorithm overly sensitive to the thresh- olds in dense reward environments (see Appendix C.I.I.. However, in the sparse Mazes, set Tthres = Ethres 0, in order to avoid early termination of exploration (we do not change the hy- perparameters determining the number of planning iterations). Tuning these hyperparameters (for both dense and sparse rewards) could lead to hetter performance, desired.
For the first planning iteration set Athres 0.01, and for the later planning iterations, Astres 0.05. We found that having a lower threshold for the first iteration helps the agent to avoid getting stuck in poor trajectories (i.e. avoid only using the policy), alongside the stochastic decision rule. For the Ant environment, we set Athres 0.01 and always require at least one planning iteration.
C.I.1 SENSITIVITY TO TURESHIOLDS
We run a rough grid search with wider values for Othres and Ethreas and calculate average reward in the Hopper changing worlds environment. The average reward for each setting is shown in Table C. and learning curves are shown in Fig. C.I AOP somewhat more sensitive to the setting éthres early in training, higher value corresponds to less planning, but this effect quickly dissipates. As a result, while the choice of ttres and éthres is fairly arbitrary, we do not believe that AOP is particularly sensitive them, and use the same values for all of the dense reward environments.
12

---

Under review as a conference paper at ICLR 2020
Table C.I: Effect of varying threshold hyperparameters
Standard Deviation Othrest
4
8 (Default)
Average reward
0.47 + 0.20
0.42 + 0.05
0.44 = 0.16
Bellman Error éthrest
10
25 (Default)
40
Average reward
0.47 + 0.17
0.47 + 0.09
0.43 + 0.24
Rese
10
1
2500
5000
7500
10000
17500
15030
17500
20000
2500
5000
7100
10000
12500
15000
17500
20003
Timestop
Timestop
Figure C.I: Learning curves for hyperparameter sweep. Left: standard deviation Othres-
Right:
Bellman error (thres- Legend shows value of relevant hyperparameter. 13 seeds were run in total.
C.2 MODEL PREDICTIVE CONTROL HYPERPARAMETER
Our MPPI temperature À is set 0.01. The other planning hyperparameter' (kept constant across environments) are shown below. See Section 3.4 interpretation of policy optimization as special case of AOP. Surprisingly, found TD3 perform worse with more than trajectory per iteration.
Parameter
AOP-TD3/AOP-BC
POLO
TD3
PPO
MPC
Planning horizon
1-80
80
256
128
80
Planning iterations per timestep
0-8
3, 8
Trajectories per iteration
40
40
32
40
Noise standard deviation
0.1
0.1
0.2
0.1
C.3 NETWORK ARCHITECTURES
For our value ensembles, we use an ensemble size of 6 and k = 10 The value functions are up- dated batches of size 32 for 32 gradient steps every timesteps. All networks use tanh activations and learning rate of 10-3, trained using Adam. Network sizes are shown below.
Environment
AOP-TD3/AOP-BC
POLO
TD3
PPO
Hopper
V (64, 64), Q: (400, 300), क (400, 300)
(64, 64)
(400, 300)
(64, 64)
Ant
V: (64, 64), Q (400, 300), क (400,300)
(64, 64)
(400, 300)
(64, 64)
Maze
(64, 64)
(64, 64)
(64, 64)
(64, 64)
C.4 POLICY OPTIMIZATION HYPERPARAMETERS
Our TD3 uses the same hyperparameters as the original authors (Fjjimoo et al. 2018b), where for every timestep, we run rollout of length 256 and run 256 gradient steps. In the TD33 used for the experiment in Section 4.4, we run rollouts of length 1000 and run 1000 gradient steps after each rollout, equivalent to the standard TD3 setting with no termination.
Our PPO uses 0.2, À 0.95, batch sizes of 4096, and 80 gradient steps per iteration. For behavior cloning, we run 400 gradient steps on batches of size every timesteps. For the policy in AOP-TD3, run 128 gradient steps on batches of size 100 every timesteps.
13

---

Under review as a conference paper at ICLR 2020
D ENVIRONMENT DETAILS
C
Figure D.1: Pictures of lifelong learning environments: Hopper, Ant, and Maze (from left to right). In the Maze, the agent (orange ball) must try navigate to the goal (open orange circle) while avoiding the black walls. In the picture, the red lines indicate past history of movement.
In the online setting, the agent receives signal from termination states, i.c. it becomes more difficult to know not to fall down in the cases of Hopper and Ant. To amend this, and achieve the same interpretable behavior the standard reinforcement learning setting, we set the reward functions as the following for our environments, similar to Rajeswaran et al. (2017b):
Environment
Reward Function
Hopper Ant
|__vel - x. v_velaarg + 5(2 1.8) + .1 ||a| + x x_velarg
| vel - 2| + 3(2 .92 + 001 || 2
Maze (Dense)
(2r, v) (rr, goat la (contact with wall]
Maze (Sparse)
1 iisidee goal] - 1(contact with wall]
14


Reviews:
[{"review_id": "rkgTvCJ2tB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents an adaptive online planning(AOP) strategy in a model-free policy setting, a reinforcement learning method aimed to solve catastrophic forgetting problem by combining model-based planning and model-free policy learning. AOP is able extensive plan only when necessary, leading to over all average reduced computation times. AOP can be easily integrated into other reinforcement learning frameworks such as to any offline-planning reinforcement learning algorithms.  The experiments demonstrate that AOP is computationally efficient compared to traditional baselines MPC-8 and MPC-3 while maintaining the performances.\n\nThe algorithm is developed based on heuristic solutions to address some of the fundamental problems in reinforcement learning, and although the proposed strategies definitely seem to provide some benefits in terms of computation complexity, the solution is not very elagant or noval. It is hard to justify the computational efficiency and performance in dynamically changing environments just based on the presented results. While the improvement in computation is there, what I find lacking is the experiments demonstrating clear evidence of overcoming catastrophic forgetting problem. The paper gives off a feeling that AOP as an add-on that can increase the performance of any  RL algorithm. \n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "Hygm-g2oYS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors study continual, lifelong learning. They suggest a new algorithm, named Adaptive Online Planning (AOP) that combines model-based planning with model-free learning. AOP decides how much additional planning is needed based on the uncertainty of the model-free learner and the performance of the planner. Experiments are carried out on three tasks, i.e. Hopper, Ant and a Maze.\n\nThis paper should be rejected. The main reason is that the experiments were only performed for 3 different seeds and are therefore not statistically relevant (see Henderson et al. \"Deep reinforcement learning that matters.\" Thirty-Second AAAI Conference on Artificial Intelligence. 2018.). \n\nBesides the issue of significance of the results section, there are other concerns. Some of them are:\n- Page 2: 'The dynamics model is updated immediately at world changes.' - Is this a reasonable assumption? Where does an accurate model come from? Given a perfect model, it is not surprising that a learner that is combined with such a model achieves a superior performance.\n- Although the authors state that the 'ability to perform well in old tasks (backward transfer)' is important, they don't explicitly show their algorithm to achieve this goal. Backwards transfer might be included into the experimental section, but I could not find a statement that addresses this explicitly.\n- I would like the authors to crisply define their use of the word 'online learning'. Does online learning simply mean to process each sample as it is available or does the term include real-time?\n- How is \\sigma_{thres} chosen? What is the influence of this parameter?\n- The statement that 'AOP uses only 0 - 25% of the number of timesteps as MPC-8, but achieves generally comparable or stronger performance.' is wrong (see Fig 4, d and e). This statement is especially difficult, as results are only averaged over 3 runs.\n\nThere are furthermore a few minor concerns:\n- the interval for \\gamma should exclude 1 in this setting, as the return would otherwise be unbounded.\n- In the background section, the authors confuse the definition of the return with reward.\n- the term 'deep exploration' is used but not defined\n- There are two figures between the subsection header for 4.4 and the text - this is highly confusing\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BJlIWgx6Fr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThe work is heuristically motivated by the goal of reducing the high computation of model-based learning while achieving high performance. For achieving that, the authors propose an algorithm, Adaptive Online Planning (AOP) combining a model-free policy learning method and a model-based planner. In terms of the empirical study, they test the algorithm in 3 environments, Hopper, Ant, and Maze. They compare their algorithms with several model-based methods.\n\nFrom my perspective, the paper has several weaknesses for which I give a weak rejection. \n\nThe motivation is interesting to me, but the authors do not provide enough justification. The authors claim that the proposed method is able to reduce high computation. However, seemingly they only intuitively illustrate how it saves energy without strong proofs, which weakens the claim. What’s more, the experiment is not clear to me. What are the take-aways of Figure 3 and Figure 4 while I cannot see an improvement from them? There is no comparison in Figure 6; not clear how the plots of other models look like. The last comment is about the 3 environments that are not complex enough.\n\n\nMinor comments:\n- Some typos and grammar mistakes, e.g., ‘planing’ and ‘(d)by’ in the third last line (p.4); the second sentence in Sec. conclusion (p.8).", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: A new setting for lifelong learning is analyzed and a new method, AOP, is introduced, which combines a model-free with a model-based  approach to deal with this setting.

While the idea is interesting, the main claims are insufficiently demonstrated. A theoretical justification is missing, and the experiments alone are not rigorous enough to draw strong conclusions. The three environments are rather simplistic and there are concerns about the statistical significance, for at least some of the experiments.
Decision: Reject