Venue: ICLR 2017 workshop

Reviews:

final_score: rating: 7: Good paper, accept
review_content: I have updated my rating to a 7 after reading the authors' response.

This paper uses active sampling to select tasks to train on in a multi-task, deep RL setting. Their multi-task baseline, in comparison, chooses the next task using uniform sampling. The active sampling is done by comparing the current score on each task with its 'aspirational' high score - a score that could come from human performance or from single-task training or from published results. Tasks that are underperforming with respect to the aspirational high score are more likely to be sampled. 

Compared to the uniform sampling, active sampling yields much higher scores. In fact, uniform sampling causes some tasks to not get off the ground at all, where as all are able to learn with active sampling. This demonstrates the challenge of multi-task deep RL, where tasks may be adversarial and thus prevent others from learning.

Pros: Active sampling is a well-known approach, but it has not been tried for multi-task deep RL, which is a much harder problem than, e.g., supervised learning. This paper is simple, but the approach is quite effective. It will be of interest to those in the community that are studying continual or transfer learning in RL.  

Cons: The results are limited. The authors only used 6 Atari games, and this type of result is highly variable and might not hold for a different set. However, for a workshop paper, it is understandable. More sophisticated methods, such as a multi-armed bandit formulation, might have been tried. My one complaint is that the authors did not show how often the different games were chosen, nor show any visualisation of the game selection over time. This would have been quite interesting. 


review_title: Simple but effective - final review

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes an online method for training reinforcement learning agents to perform several tasks at once. The method samples a task to train on after each episode. The main idea is to assume that a reference score is available for each task and to use the agent's relative performance with respect to the reference score to determine the probability of sampling the task.  The result is that tasks with lower relative performance are sampled more frequently.

The experiments show that the proposed strategy outperforms uniform sampling of tasks during online training. While the idea is simple it is very ad hoc and has a number of unexplored parameters (temperature, number of recent scores, number of training steps with uniform sampling). There are more principled and well understood methods that could be applied to this problem. For example, why not apply a UCB-style non-stationary bandit algorithm that attempts to balance exploration and exploitation? Such a method would keep periodically going back to tasks with high relative performance, which could help mitigate catastrophic forgetting. The proposed method seems to ignore such potential interference between tasks.
review_title: Raising the score to 6 after considering the authors' response

Decision: Accept