Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper present online algorithms for learning multiple sequential problems. The main contribution is to introduce active learning principles for sampling the sequential tasks in an online algorithm. Experimental results are given on different multi-task instances. The contributions are interesting and experimental results seem promising. But the paper is difficult to read due to many different ideas and because some algorithms and many important explanations must be found in the Appendix (ten sections in the Appendix and 28 pages). Also, most of the paper is devoted to the study of algorithms for which the expected target scores are known. This is a very strong assumption. In my opinion, the authors should have put the focus on the DU4AC algorithm which get rids of this assumption. Therefore, I am not convinced that the paper is ready for publication at ICLR'18.
* Differences between BA3C and other algorithms are said to be a consequence of the probability distribution over tasks. The gap is so large that I am not convinced on the fairness of the comparison. For instance, BA3C (Algorithm 2 in Appendix C) does not have the knowledge of the target scores while others heavily rely on this knowledge.
* I do not see how the single output layer is defined.
* As said in the general comments, in my opinion Section 6 should be developped and more experiments should be done with the DUA4C algorithm.
* Section 7.1. It is not clear why degradation does not happen. It seems to be only an experimental fact.
review_title: New online algorithms for learning multiple sequential problems

final_score: rating: 7: Good paper, accept
review_content: In this paper active learning meets a challenging multitask domain: reinforcement learning in diverse Atari 2600 games. A state of the art deep reinforcement learning algorithm (A3C) is used together with three active learning strategies to master multitask problem sets of increasing size, far beyond previously reported works.

Although the choice of problem domain is particular to Atari and reinforcement learning, the empirical observations, especially the difficulty of learning many different policies together, go far beyond the problem instantiations in this paper. Naive multitask learning with deep neural networks fails in many practical cases, as covered in the paper. The one concern I have is perhaps the choice of distinct of Atari games to multitask learn may be almost adversarial, since naive multitask learning struggles in this case; but in practice, the observed interference can appear even with less visually diverse inputs.

Although performance is still reduced compared to single task learning in some cases, this paper delivers an important reference point for future work towards achieving generalist agents, which master diverse tasks and represent complementary behaviours compactly at scale.

I wonder how efficient the approach would be on DM lab tasks, which have much more similar visual inputs, but optimal behaviours are still distinct.

review_title: Improved multitask deep reinforcement learning with active learning

final_score: rating: 7: Good paper, accept
review_content: 
The authors show empirically that formulating multitask RL itself as an active learning and ultimately as an RL problem can be very fruitful.  They design and explore several approaches  to the active learning (or active sampling) problem, from a basic 
change to the distribution to UCB to feature-based neural-network based RL. The domain is video games.   All proposed approaches beat the uniform sampling baselines and the more sophisticated approaches do better in the scenarios with more tasks (one multitask  problem had 21 tasks).


Pros:

- very promising results with an interesting active learning approach to multitask RL

- a number of approaches developed for the basic idea

- a variety of experiments, on challenging multiple task problems (up to 21 tasks/games)

- paper is overall well written/clear

Cons:

- Comparison only to a very basic baseline (i.e. uniform sampling)
Couldn't comparisons be made, in some way, to other multitask work?



Additional  comments:

- The assumption of the availability of a target score goes against
the motivation that one need not learn individual networks ..  authors
say instead one can use 'published' scores, but that only assumes
someone else has done the work (and furthermore, published it!).

The authors do have a section on eliminating the need by doubling an
estimate for each task) which makes this work more acceptable (shown
for 6 tasks or MT1, compared to baseline uniform sampling).

Clearly there is more to be done here for a future direction (could be
mentioned in future work section).

- The averaging metrics (geometric, harmonic vs arithmetic, whether
  or not to clip max score achieved) are somewhat interesting, but in
  the main paper, I think they are only used in section 6 (seems like
  a waste of space). Consider moving some of the results, on showing
  drawbacks of arithmetic mean with no clipping (table 5 in appendix E), from the appendix to
  the main paper.


- The can be several benefits to multitask learning, in particular
  time and/or space savings in learning new tasks via learning more
  general features. Sections 7.2 and 7.3 on specificity/generality of
  features were interesting.



--> Can the authors show that a trained network (via their multitask
    approached) learns significantly faster on a brand new game
    (that's similar to games already trained on), compared to learning from
    scratch?

--> How does the performance improve/degrade (or the variance), on the
    same set of tasks, if the different multitask instances (MT_i)
    formed a supersets hierarchy, ie if MT_2 contained all the
    tasks/games in MT_1, could training on MT_2 help average
    performance on the games in MT_1 ? Could go either way since the network
   has to allocate resources to learn other games too.  But is there a pattern?



- 'Figure 7.2' in section 7.2 refers to Figure 5.


- Can you motivate/discuss better why not providing the identity of a
  game as an input is an advantage? Why not explore both
  possibilities? what are the pros/cons? (section 3)





review_title: An active learning approach to multitask RL with significant potential

Meta-review: The paper contains an interesting way to do online multi task learning, by borrowing ideas from active learning and comparing and contrasting a number of ways on the arcade learning environment.  Like the reviewers, I have some concerns about using the target scores and I think more analysis would be needed to see just how robust this method is to the choice/distribution of target scores (the authors mention that things don't break down as long as the scores are "reasonable", but that's not a particularly empirical nor precise statement).

My inclination is to accept the paper, because of the earnest efforts made by the authors in understanding how DUA4C works. However, I do agree that the paper should have a larger focus on that: basically Section 6 should be expanded, and the experiments should be rerun in such a way that the setup for DUA4C is more "favorable" (in terms of hyper-parameter optimization).  If there's any gap between any of the proposed methods and DUA4C, then this would warrant further analysis of course (since it would mean that there's an advantage to using target scores).  
Decision: Accept (Poster)