Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: In Eq. 12, wouldn’t it make each Q_b converge to the same action-value function given by Q_T, and unlearn information from source tasks? Slow update may alleviate the problem. Also, is this the only case where you re-train parameters learned in source tasks? Have you performed comparison between having source parameters fixed versus re-trainable?
review_title: details on re-training during transfer

final_score: rating: 7: Good paper, accept
review_content: This paper studies the problem of transferring solutions of existing tasks to tackle a novel task under the framework of reinforcement learning and identifies two important issues of avoiding negative transfer and being selective transfer. The proposed approach is based on a convex combination of existing solutions and the being-learned solution to the novel task. The non-negative weight of each solution implies that the solution of negative effect is ignored and more weights are allocated to more relevant solution in each state. This paper derives this so-called "A2T" learning algorithm for policy transfer and value transfer for REINFORCE and ACTOR-CRITIC algorithms and experiments with synthetic Chain World and Puddle World simulation and Atari 2600 game Pong. 
+This paper presents a novel approach for transfer reinforcement learning.
+The experiments are cleverly designed to demonstrate the ability of the proposed method.
-An important aspect of transfer learning is that the algorithm can automatically figure out if the existing solutions to known tasks are sufficient to solve the novel task so that it can save the time and energy of learning-from-scratch. This issue is not studied in this paper as most of experiments have a learning-from-scratch solution as base network. It will be interesting to see how well the algorithm performs without base network. In addition, from Figure 3, 5 and 6, the proposed algorithm seems to accelerate the learning speed, but the overall network seems not better than the solo base network. It will be more convincing to show some example that existing solutions are complementary to the base network.
-If ignoring the base network, the proposed network can be considered as ensemble reinforcement learning that take advantages of learned agents with different expertise to solve the novel task. 
review_title: null

final_score: rating: 7: Good paper, accept
review_content: The paper tackles important problems in multi-task reinforcement learning: avoid negative transfer and allow finer selective transfer. The method is based on soft attention mechanism, very general, and demonstrated to be applicable in both policy gradient and value iteration methods. The introduction of base network allows learning new policy if the prior policies aren't directly applicable. State-dependent sub policy selection allows finer control and can be thought of assigning state space to different sub policies/experts. The tasks are relatively simplistic but sufficient to demonstrate the benefits. One limitation is that the method is simple and the results/claims are mostly empirical. It would be interesting to see extensions to option-based framework, stochastic hard attention mechanism, sub-policy pruning, progressive networks. 

In figure 6, the read curve seems to perform worse than the rest in terms of final performance. Perhaps alternative information to put with figures is the attention mask activation statistics during learning, so that we may observe that it learns to turn off adversarial sub-policies and rely on newly learned base policy mostly. This is also generally good to check to see if any weird co-adaptation is happening. 
review_title: Review

final_score: rating: 7: Good paper, accept
review_content: In this paper a well known soft mixture of experts model is adapted for, and applied to, a specific type of transfer learning problem in reinforcement learning (RL), namely transfer of action policies and value functions between similar tasks. Although not treated as such, the experimental setup is reminiscent of hierarchical RL works, an aspect which the paper does not consider at length, regrettably.
One possible implication of this work is that architecture and even learning algorithm choices could simply be stated in terms of the objective of the target task, rather than being hand-engineered by the experimenter. This is clearly an interesting direction of future work which the paper illuminates.


Pros:
The paper diligently explains how the network architecture fits in with various widely used reinforcement learning setups, which does facilitate continuation of this work.
The experiments are good proofs of concept, but do not go beyond that i.m.h.o. 
Even so, this work provides convincing clues that collections of deep networks, which were trained on not entirely different tasks, generalize better to related tasks when used together rather than through conventional transfer learning (e.g. fine-tuning).

Cons:
As the paper well recounts in the related work section, libraries of fixed policies have long been formally proposed for reuse while learning similar tasks. Indeed, it is well understood in hierarchical RL literature that it can be beneficial to reuse libraries of fixed (Fernandez & Veloso 2006) or jointly learned policies which may not apply to the entire state space, e.g. options (Pricop et. al). What is not well understood is how to build such libraries, and this paper does not convincingly shed light in that direction, as far as I can tell.
The transfer tasks have been picked to effectively illustrate the potential of the proposed architecture, but the paper does not tackle negative transfer or compositional reuse in well known challenging situations outlined in previous work (e.g. Parisotto et. al 2015, Rusu el. al 2015, 2016).
Since the main contributions are of an empirical nature, I am curious how the results shown in figures 6 & 7 look plotted against wall-clock time, since relatively low data efficiency is not a limitation for achieving perfect play in Pong (see Mnih. et al, 2015). It would be more illuminating to consider tasks where final performance is plausibly limited by data availability. It would also be interesting if the presented results were achieved with reduced amounts of computation, or reduced representation sizes compared to learning from scratch, especially when one of the useful source tasks is an actual policy trained on the target task.
Finally, it is perhaps underwhelming that it takes a quarter of the data required for learning Pong from scratch just to figure out that a perfect Pong policy is already in the expert library. Simply evaluating each expert for 10 episodes and using an  average-score-weighted majority vote to mix action choices would probably achieve the same final performance for a smaller fraction of the data. 

review_title: Final Review: Learned convex combination of many fixed and a jointly learned expert is used to represent action policies in proof-of-concept, transfer/hierarchical RL, settings.

Meta-review: The authors present a mixture of experts framework to combine learnt policies to improve multi-task learning while avoiding negative transfer. The key to their approach is a soft attention mechanism, learnt with RL, which enables positive transfer. They give limited empirical validation of their approach. This is a general attention method which can be applied for policy gradient or value iteration learning methods. The authors were very responsive and added additional experiments based on the reviews.
Decision: Accept (Poster)