Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Hi, 

This was a nice read. I think overall it is a good idea. But I find the paper lacking a lot of details and to some extend confusing. 
Here are a few comments that I have:

Figure 2 is very confusing for me. Please first of all make the figures much larger. ICLR does not have a strict page limit, and the figures you have are hard to impossible to read. So you train in (a) on the steps task until 350k steps? Is (b), (d),(c) in a sequence or is testing moving from plain to different things? The plot does not explicitly account for the distillation phase. Or at least not in an intuitive way. But if the goal is transfer, then actually PLAID is slower than the MultiTasker because it has an additional cost to pay (in frames and times) for the distillation phase right? Or is this counted. 

Going then to Figure 3, I almost fill that the MultiTasker might be used to simulate two separate baselines. Indeed, because the retention of tasks is done by distilling all of them jointly, one baseline is to keep finetuning a model through the 5 stages, and then at the end after collecting the 5 policies you can do a single consolidation step that compresses all. So it will be quite important to know if the frequent integration steps of PLAID are helpful (do knowing 1,2 and 3 helps you learn 4 better? Or knowing 3 is enough). 

Where exactly is input injection used? Is it experiments from figure 3. What input is injecting? What do you do when you go back to the task that doesn't have the input, feed 0? What happens if 0 has semantics ? 

Please say in the main text that details in terms of architecture and so on are given in the appendix. And do try to copy a bit more of them in the main text where reasonable. 

What is the role of PLAID? Is it to learn a continual learning solution? So if I have 100 tasks, do I need to do 100-way distillation at the end to consolidate all skills? Will this be feasible? Wouldn't the fact of having data from all the 100 tasks at the end contradict the traditional formulation of continual learning? 
 
Or is it to obtain a multitask solution while maximizing transfer (where you always have access to all tasks, but you chose to sequentilize them to improve transfer)?  And even then maximize transfer with respect to what? Frames required from the environment? If that are you reusing the frames you used during training to distill? Can we afford to keep all of those frames around? If not we have to count the distillation frames as well. Also more baselines are needed. A simple baseline is just finetunning as going from one task to another, and just at the end distill all the policies found through out the way.  Or at least have a good argument of why this is suboptimal compared to PLAID. 

I think the idea of the paper is interesting and I'm willing to increase (and indeed decrease) my score. But I want to make sure the authors put a bit more effort into cleaning up the paper, making it more clear and easy to read. Providing at least one more baseline (if not more considering the other things cited by them). 


review_title: Interesting

final_score: rating: 7: Good paper, accept
review_content: This paper describes PLAID, a method for sequential learning and consolidation of behaviours via policy distillation; the proposed method is evaluated in the context of bipedal motor control across several terrain types, which follow a natural curriculum.

Pros:
- PLAID masters several distinct tasks in sequence, building up “skills” by learning “related” tasks of increasing difficulty.
- Although the main focus of this paper is on continual learning of “related” tasks, the authors acknowledge this limitation and convincingly argue for the chosen task domain.

Cons:
- PLAID seems designed to work with task curricula, or sequences of deeply related tasks; for this regime, classical transfer learning approaches are known to work well (e.g finetunning), and it is not clear whether the method is applicable beyond this well understood case.
- Are the experiments single runs? Due to the high amount of variance in single RL experiments it is recommended to perform several re-runs and argue about mean behaviour.

Clarifications:
- What is the zero-shot performance of policies learned on the first few tasks, when tested directly on subsequent tasks?
- How were the network architecture and network size chosen, especially for the multitasker? Would policies generalize to later tasks better with larger, or smaller networks?
- Was any kind of regularization used, how does it influence task performance vs. transfer?
- I find figure 1 (c) somewhat confusing. Is performance maintained only on the last 2 tasks, or all previously seen tasks? That’s what the figure suggests at first glance, but that’s a different goal compared to the learning strategies described in figures 1 (a) and (b).

review_title: Nice continual learning study

final_score: rating: 7: Good paper, accept
review_content: This paper aims to learn a single policy that can perform a variety of tasks that were experienced sequentially. The approach is to learn a policy for task 1, then for each task k+1: copy distilled policy that can perform tasks 1-k, finetune to task k+1, and distill again with the additional task. The results show that this PLAID algorithm outperforms a network trained on all tasks simultaneously. 

Questions:
- When distilling the policies, do you start from a randomly initialized policy, or do you start from the expert policy network?
- What data do you use for the distillation? Section 4.1 states"We use a method similar to the DAGGER algorithm", but what is your method. If you generate trajectories form the student network, and label them with the expert actions, does that mean all previous expert policies need to be kept in memory?
- I do not understand the purpose of "input injection" nor where it is used in the paper. 

Strengths:
- The method is simple but novel. The results support the method's utility.
- The testbed is nice; the tasks seem significantly different from each other. It seems that no reward shaping is used.
- Figure 3 is helpful for understanding the advantage of PLAID vs MultiTasker.

Weaknesses:
- Figure 2: the plots are too small.
- Distilling may hurt performance ( Figure 2.d)
- The method lacks details (see Questions above)
- No comparisons with prior work are provided. The paper cites many previous approaches to this but does not compare against any of them. 
- A second testbed (such as navigation or manipulation) would bring the paper up a notch. 

In conclusion, the paper's approach to multitask learning is a clever combination of prior work. The method is clear but not precisely described. The results are promising. I think that this is a good approach to the problem that could be used in real-world scenarios. With some filling out, this could be a great paper.
review_title: Good approach, needs more details.

Meta-review: The authors propose an architecture that uses a curriculum and multi-task distillation to gain higher performance without forgetting. The paper is largely a smart composition of known methods, and it requires keeping data from all tasks to do the distillation, so it is not truly a scalable continual learning approach. There were a lot of concerns about clarity in the manuscript, but many of these have been assuaged by an update to the paper. This is a borderline paper, but the author's rebuttal and update probably tip it towards acceptance. 
Decision: Accept (Poster)