Title: Attend, Adapt and Transfer: Attentive Deep Architecture for Adaptive Transfer from multiple sources in the same domain
Authors: Janarthanan Rajendran
Aravind Lakshminarayanan
Mitesh M. Khapra
Prasanna P
Balaraman Ravindran
Venue: ICLR 2017
Keywords: Deep learning
Reinforcement Learning
Transfer Learning

Abstract: Transferring knowledge from prior source tasks in solving a new target task can be useful in several learning applications. The application of transfer poses two serious challenges which have not been adequately addressed. First, the agent should be able to avoid negative transfer, which happens when the transfer hampers or slows down the learning instead of helping it. Second, the agent should be able to selectively transfer, which is the ability to select and transfer from different and multiple source tasks for different parts of the state space of the target task. We propose A2T (Attend Adapt and Transfer), an attentive deep architecture which adapts and transfers from these source tasks. Our model is generic enough to effect transfer of either policies or value functions. Empirical evaluations on different learning algorithms show that A2T is an effective architecture for transfer by being able to avoid negative transfer while transferring selectively from multiple source tasks in the same domain.

TL;DR: We propose a general architecture for transfer that can avoid negative transfer and transfer selectively from multiple source tasks in the same domain.

Review discussion:
In Eq. 12, wouldn’t it make each Q_b converge to the same action-value function given by Q_T, and unlearn information from source tasks? Slow update may alleviate the problem.
Also, is this the only case where you re-train parameters learned in source tasks? Have you performed comparison between having source parameters fixed versus re-trainable?
a) In Eq. 12, wouldn’t it make each Q_b converge to the same action-value function given by Q_T, and unlearn information from source tasks? Slow update may alleviate the problem.
Yes, Eq. 12 would make each Q_b converge to the same action-value function given by Q_T, which is intended. I quote below the explanation for this design choice from the paper.
Page 4, section 3: Proposed Architecture
Even though the agent follows K_T , we update the parameters of the base network that produces K_B, as if the action taken by the agent was based only on K_B. Due to this special way of updating K_B, apart from the experience got through the unique and individual contribution of K_B to K_T in parts of the state space where the source task solutions are not relevant, K_B also uses the valuable experience got by using K_T which uses the solutions of the source tasks as well.
This also means that, if there is a source task whose solution K_j is useful for the target task in some parts of its state space, then K_B tries to replicate K_j in those parts of the state space. In practise, the source task solutions though useful, might need to be modified to suit perfectly for the target task. The base network takes care of these modifications required to make the useful source task solutions perfect for the target task. The special way of training the base network assists the architecture in achieving this faster. Note that the agent could follow/use K_j through K_T even when K_B does not attain its replication in the corresponding parts of the state space. This allows for a good performance of the agent in earlier stages training itself, when a useful source task is available and identified.
In the case of value transfer where K_T is Q_T and K_B is Q_B,  Q_B would actually converge to Q_T as you have pointed out. Since Q_T uses the knowledge of the source tasks, Q_B basically incorporates the information from source tasks into it in the long run (and not unlearn the information from source tasks).
b) Also, is this the only case where you re-train parameters learned in source tasks? Have you performed comparison between having source parameters fixed versus re-trainable?
In our model we never re-train the parameters of the source tasks. Retraining the parameters have the following issues which we verified empirically.
Page 4, section 3: Proposed Architecture
As mentioned earlier, the source task solutions, K_1, . . . , K_N remain fixed. Updating these source task’s parameters would cause a significant amount of unlearning in the source tasks solutions and result in a weaker transfer, which we observed empirically. This also enables the use of source task solutions, as long as we have the outputs alone, irrespective of how and where they come from.
Thank you for clarifying. Actually there is no confusion in the paper. I mistook some notations.
The paper tackles important problems in multi-task reinforcement learning: avoid negative transfer and allow finer selective transfer. The method is based on soft attention mechanism, very general, and demonstrated to be applicable in both policy gradient and value iteration methods. The introduction of base network allows learning new policy if the prior policies aren't directly applicable. State-dependent sub policy selection allows finer control and can be thought of assigning state space to different sub policies/experts. The tasks are relatively simplistic but sufficient to demonstrate the benefits. One limitation is that the method is simple and the results/claims are mostly empirical. It would be interesting to see extensions to option-based framework, stochastic hard attention mechanism, sub-policy pruning, progressive networks.
In figure 6, the read curve seems to perform worse than the rest in terms of final performance. Perhaps alternative information to put with figures is the attention mask activation statistics during learning, so that we may observe that it learns to turn off adversarial sub-policies and rely on newly learned base policy mostly. This is also generally good to check to see if any weird co-adaptation is happening.
Thanks for your positive review and the questions.
Regarding the final performance question for Fig 6, the difference in the number of steps across algorithms once they settle to an optimal or close to optimal navigation policy is very minimal and is attributed more to stochasticity in the world like the wind, start positions, etc.  The speed of convergence to a near optimal policy is the issue that we focus more on.
Regarding the evolution of the attention masks, we did observe, for instance, in our experiments on Pong with one positive and one negative expert, that the attention is, initially; high on the positive expert, low on negative and the base network; and slowly over time; the attention shifts to the base network from the positive network with the adversarial expert's weight remaining low throughout. We will add these masks in an updated version of the paper.
We have added the evolution of attention masks for the experiment on Avoiding negative transfer and Learning to transfer from a favorable source task (Section 4.2). We illustrate this with the case where Pong-Negated Reward agent acts as the negative expert, and Expert-Pong acts as the positive expert. The attention is initially high on the base network since both the attention and base networks haven't learned anything yet on what is a positive source task, and what is a negative source task.
However, slowly, the attention shifts to the positive expert, which provides a way to execute rewarding episodes and learn from rewarding transition tuples. The base network by then learns from the off-policy experience gathered this way and becomes an expert on the target task, making the attention network roughly ambivalent (between 0.45 and 0.55) about attending to the base network or the positive expert network. This is because the positive source task is also an expert on the target task. The attention on the negative source task expert is uniformly low (close to zero, negligible values) and this is empirical proof that no weird co-adaptation happens where there is attention on adversarial expert policies.
Please refer to Appendix H in the revised paper for the visualization. Thanks!
This paper studies the problem of transferring solutions of existing tasks to tackle a novel task under the framework of reinforcement learning and identifies two important issues of avoiding negative transfer and being selective transfer. The proposed approach is based on a convex combination of existing solutions and the being-learned solution to the novel task. The non-negative weight of each solution implies that the solution of negative effect is ignored and more weights are allocated to more relevant solution in each state. This paper derives this so-called "A2T" learning algorithm for policy transfer and value transfer for REINFORCE and ACTOR-CRITIC algorithms and experiments with synthetic Chain World and Puddle World simulation and Atari 2600 game Pong.
+This paper presents a novel approach for transfer reinforcement learning.
+The experiments are cleverly designed to demonstrate the ability of the proposed method.
-An important aspect of transfer learning is that the algorithm can automatically figure out if the existing solutions to known tasks are sufficient to solve the novel task so that it can save the time and energy of learning-from-scratch. This issue is not studied in this paper as most of experiments have a learning-from-scratch solution as base network. It will be interesting to see how well the algorithm performs without base network.
In addition, from Figure 3, 5 and 6, the proposed algorithm seems to accelerate the learning speed, but the overall network seems not better than the solo base network. It will be more convincing to show some example that existing solutions are complementary to the base network.
-If ignoring the base network, the proposed network can be considered as ensemble reinforcement learning that take advantages of learned agents with different expertise to solve the novel task.
Thanks for your thoughtful remarks.
REVIEWER POINT 1:
"An important aspect of transfer learning is that the algorithm can automatically figure out if the existing solutions to known tasks are sufficient to solve the novel task so that it can save the time and energy of learning-from-scratch. This issue is not studied in this paper as most of experiments have a learning-from-scratch solution as base network. It will be interesting to see how well the algorithm performs without base network."
RESPONSE 1:
We first explain the need for the base network in the architecture below:
a. The presence of a base network allows A2T to be complete, i.e. it can use the base network to learn solutions in the parts of the state space, where the source task solutions are not sufficient/useful.
b. Using the attention network, A2T can implicitly figure out if the source task solutions are sufficient to solve the novel task.  The base network has to learn from scratch only in the parts of the state space where the source task solutions are not sufficient to solve the new target task. In other parts of the state space, the agent learns to act (K_T) using the useful solutions of the source task(s). In parallel, the base network (K_B) learns indirectly from K_T and helps in fine tuning the source task solutions for the target task. This is explained in Page 4, section 3: Proposed Architecture.
c.
Even though the agent follows K_T , we update the parameters of the base network that produces K_B as if the action taken by the agent was based on K_B. Due to this special way of updating K_B, K_B also uses the valuable experience gathered by using K_T that focuses on the solutions of the source tasks in addition to using the experience gathered through the unique and individual contribution of K_B to K_T in parts of the state space where the source task solutions are not relevant.
d.
Thus, if there is a source task whose solution K_j is useful for the target task in some parts of its state space,  K_B tries to replicate K_j in those parts of the state space.
However, in practice, the source task solutions though useful, might need to be modified and fine-tuned to suit the needs of the target task perfectly rather than being replicated. The base network takes care of these modifications required to make the useful source task solutions perfect for the target task. The special way of training the base network assists the architecture in achieving this faster. Note that the agent could follow/use K_j through K_T even when K_B does not attain its replication in the corresponding parts of the state space. This allows for a good performance of the agent in  the early stages of training when a useful source task is available and has been identified.
Specifically, to address "It will be interesting to see how well the algorithm performs without base network",  we have one experiment each in policy transfer and value transfer which analyze the architecture without the base network. Please refer to Section 4.1 (Figure 3.a) (i) for policy transfer and figure 4 for value transfer). It is clearly able to appropriately invoke the source task solution and solve the target task reasonably well through the attention mechanism alone. (For instance, the A2T without base network scores 17.2 on Pong while blurred experts score 8 and 9.2 on the Pong task).
As for explaining the point raised in d) above where the source task policies may not just have to be replicated but fine-tuned for optimal performance in the target task, we see from the graph in Figure 5 that the final score with A2T having a base network is 19+, while just having an attention network without any base network saturated with 17.2 within a few epochs. The individual experts are just proxies for quadrant experts (like forehand-backhand in Tennis) because we had a heuristic way to create these experts (through blurred training).
Therefore, it becomes necessary to fine-tune a bit after replicating initially for solving the target task, and that's where the base network helps on top. Attention network without the base network can only help in replication of appropriate source modules.
REVIEWER POINT 2 & RESPONSE 2:
Regarding your point "It will be more convincing to show some example that existing solutions are complementary to the base network.", we have added a new experiment to specifically address this in the revision (refer to Appendix I). The new baseline introduced is a positive source task policy which is an imperfect expert on the target task.
Specifically, this is a source task DQN scoring 8 (instead of 19+ of an optimal expert). This calls for a situation where the imperfect expert should help in speeding up the learning through replication till some point, but the base network definitely has to fine-tune and refine the source task policy and acquire new skills to reach expertise on the target task.
So, complementary skills are implicitly learned, but the existing solutions are favorably used. We see from the graph in Appendix I that the imperfect expert case is better than having no positive expert but worse than having the perfect expert. This is essentially what we expect to happen. The Blurring experiment with a base network can also be considered as a case where some level of complementary skill is needed for getting to 19+ from 17+ obtained with using only the attention network as far as average performance is concerned.
REVIEWER POINT 3:
"If ignoring the base network, the proposed network can be considered as ensemble reinforcement learning that take advantages of learned agents with different expertise to solve the novel task."
RESPONSE 3:
We have clearly explained the need for the base network above, both through arguments and pointing to relevant experiments and additional experiments added.
Our objective was not to propose a model to "just" serve as a Mixture of Experts.
In fact, it should only be interpreted as an outcome of specific cases where the generic architecture can serve as an ensemble model. The framework is however general enough that it can not only serve as an ensemble mixture of experts model when no extra skills are needed from what is present as the source task policies, but also learn new skills on variety of new tasks and augment our library of policy modules / networks with a new base network.
Therefore, we believe this should not be seen as a negative (con) but rather as an inference on the ability of this architecture to serve the required purpose (mixture of experts) when appropriate (source task policies mutually sufficient when replicated to solve a target task).
We have tried to address your concerns as much as possible and look forward to clarifying any more questions on the portions revised in the paper as well as our answers.
In this paper a well known soft mixture of experts model is adapted for, and applied to, a specific type of transfer learning problem in reinforcement learning (RL), namely transfer of action policies and value functions between similar tasks.
Although not treated as such, the experimental setup is reminiscent of hierarchical RL works, an aspect which the paper does not consider at length, regrettably.
One possible implication of this work is that architecture and even learning algorithm choices could simply be stated in terms of the objective of the target task, rather than being hand-engineered by the experimenter. This is clearly an interesting direction of future work which the paper illuminates.
Pros:
The paper diligently explains how the network architecture fits in with various widely used reinforcement learning setups, which does facilitate continuation of this work.
The experiments are good proofs of concept, but do not go beyond that i.m.h.o.
Even so, this work provides convincing clues that collections of deep networks, which were trained on not entirely different tasks, generalize better to related tasks when used together rather than through conventional transfer learning (e.g. fine-tuning).
Cons:
As the paper well recounts in the related work section, libraries of fixed policies have long been formally proposed for reuse while learning similar tasks.
Indeed, it is well understood in hierarchical RL literature that it can be beneficial to reuse libraries of fixed (Fernandez & Veloso 2006) or jointly learned policies which may not apply to the entire state space, e.g. options (Pricop et. al). What is not well understood is how to build such libraries, and this paper does not convincingly shed light in that direction, as far as I can tell.
The transfer tasks have been picked to effectively illustrate the potential of the proposed architecture, but the paper does not tackle negative transfer or compositional reuse in well known challenging situations outlined in previous work (e.g. Parisotto et. al 2015, Rusu el. al 2015, 2016).
Since the main contributions are of an empirical nature, I am curious how the results shown in figures 6 & 7 look plotted against wall-clock time, since relatively low data efficiency is not a limitation for achieving perfect play in Pong (see Mnih. et al, 2015). It would be more illuminating to consider tasks where final performance is plausibly limited by data availability. It would also be interesting if the presented results were achieved with reduced amounts of computation, or reduced representation sizes compared to learning from scratch, especially when one of the useful source tasks is an actual policy trained on the target task.
Finally, it is perhaps underwhelming that it takes a quarter of the data required for learning Pong from scratch just to figure out that a perfect Pong policy is already in the expert library. Simply evaluating each expert for 10 episodes and using an  average-score-weighted majority vote to mix action choices would probably achieve the same final performance for a smaller fraction of the data.
Thanks for a thoughtful review.
REVIEWER POINT 1:
"Although not treated as such, the experimental setup is reminiscent of hierarchical RL works, an aspect which the paper does not consider at length, regrettably."
RESPONSE 1:
We agree that the architecture and experimental setup is natural for Hierarchical RL.
For instance, we discussed the specific Tennis example (which motivates the Pong experiments) in detail in the Introduction Section, where we talk about skills like Forehand, Backhand and Drop-shots. One could interpret playing good Forehands, Backhands and Drop-shots as three important subgoals for a much broader and complex game like Tennis.
However, we would like to point out that this paper is intended to show the efficacy of the A2T framework for transfer from multiple source tasks in the same domain and specifically delve on the issues of Negative and Selective Transfer (which have not been adequately addressed yet in Deep RL research). Automatically constructing a set of tasks for Lifelong Hierarchical RL and demonstrating the applicability of this framework is beyond the scope of the current paper. Exploring the A2T framework in the context of Hierarchical and Lifelong RL is an exciting future research direction and we have mentioned this in the Conclusion Section in our revised version.
REVIEWER POINT 2:
"As the paper well recounts in the related work section, libraries of fixed policies have long been formally proposed for reuse while learning similar tasks.
Indeed, it is well understood in hierarchical RL literature that it can be beneficial to reuse libraries of fixed (Fernandez & Veloso 2006) or jointly learned policies which may not apply to the entire state space, e.g. options (Pricop et. al). What is not well understood is how to build such libraries, and this paper does not convincingly shed light in that direction, as far as I can tell."
RESPONSE 2:
As pointed out in the response to the comment above, our paper specifically focuses on the issue of Transfer given multiple previous libraries.
Though the nature of tasks naturally calls upon the Hierarchical RL connection, it is not the focus of this Transfer Learning paper.
Further, we would like to point out that even though it is known in the Options literature that reusing libraries is beneficial, we do not just delve on "re-using beneficial libraries", but also on "learning to avoid the reuse of irrelevant or harmful previous libraries selectively across the state-space"; "learning to specifically pick the apt library dynamically during the task execution".
That is, we "learn what is the beneficial library at a given state for a new task". We deal with these specific issues on a reasonably complex task with a large perceptual state-space such as Atari Pong, as well as a Puddle World Task.
Though Hierarchical Lifelong RL is a natural future extension, connecting HRL with this work based on the nature of the experiments is not appropriate.
REVIEWER POINT 3:
"The transfer tasks have been picked to effectively illustrate the potential of the proposed architecture, but the paper does not tackle negative transfer or compositional reuse in well known challenging situations outlined in previous work (e.g. Parisotto et. al 2015, Rusu el. al 2015, 2016)."
RESPONSE 3:
We have introduced new experiments for negative transfer:
i) Pong trained with negated reward function
ii) Network trained on a different game - Freeway
iii) Network with final layer weights negated - Puddle World
WRT ii), we used the network from another game purely as a proxy for having a source network whose weights are adversarial or hard to fine-tune when compared to learning from scratch. This is a generic scenario for Negative Transfer.
In fact, ii) is similar to the Seaquest-Gopher experiment shown in Rusu et al 2016, where a network learned on one game (Seaquest) performs worse than learning from scratch for the other game (Gopher).
As additional experiments, we also tried some of the Pong-Soup source-target transfer pairs described in Rusu et al 2016 (Progressive Neural Networks - PNNs). As far as we saw, we did not observe negative transfer in the experiments we tried (transfer from white-> black background, transfer from normal -> horizontal flipping, transfer from horizontal -> vertical flipping, etc). Some of this is shown in Rusu et al 2016 as well in the Page 4 of the Supplementary Material (https://arxiv.org/pdf/1606.04671v3.pdf). Using their notations, Baseline1 is learning-from-scratch, while Baseline 3 is fine-tuning the source task network. In all the graphs shown, Baseline 3 is better performing than Baseline 1 (which is essentially positive transfer). This clearly shows that the Pong Soup experiments in Rusu et al 2016 do not really have negative transfer.
As for the experiments on Page 3 of PNNs Supplementary Material dealing with transfer across different games, it is only in the experiments where the target task is Gopher and Star Gunner that there is negative transfer (Baseline 1 better than Baseline 3). The performance of PNNs on source-target pairs suffering from negative transfer is worse than learning from scratch (Baseline 1) when using only one source task expert. Our Pong-Freeway experiment is similar to the Seaquest-Gopher or Seaquest-Star Gunner experiment, and our graphs clearly show that A2T with one negative expert is as good as Learning from Scratch.
This is an important result and is possible mainly because of the clever way to focus attention on the outputs of the networks alone and allowing the base network (K_B) to dictate behavioral policy (K_T) through an independently learned attention mechanism on the target task. PNNs try to learn the task dependency through complicated gating mechanisms and while they may be able to re-use features at different levels, the ability to avoid using a negative expert is not very natural through multiple adaptation filters at different levels.
Rather, ignoring existing solutions (outputs) and using the solution of a newly learned network through an attention network (as in A2T) is simpler and works in practice.
As pointed out by AnonReviewer4, we believe the experiments, though simplistic, are sufficient to demonstrate the benefits of this framework. Having a source network with a set of weights that leads to slower learning when finetuned-on-top-of, as compared to learning from random weights, is a good way to demonstrate negative transfer.
Finally, it is important to understand that our A2T framework is not a competing model with PNNs (Rusu et al 2016) or Actor Mimic / Policy Distillation (Rusu et al 2015, Parisotto et al 2015), but in a sense, tangential to the above mentioned models and the benefits from A2T can help reduce Negative Transfer observed in PNN and Actor-Mimic respectively as follows:
i) PNN+A2T: We could take advantage of reusing features at multiple levels through gating as in PNNs as well as attentively combining the output policies or value functions using A2T. This way, we ensure reuse of lower level features from the source tasks for the base network, but at the same time, ignore the policies from the source tasks if they are orthogonal or adversarial strategies with respect to the reward functions in the target task.
ii) PNN+ Actor Mimic : We could learn a multi-task network on the source tasks. This could be treated as a new source task expert network. The A2T framework can then learn to give appropriate attention weights between a random base network learning from scratch and a multi-task network on the source tasks.
REVIEWER POINT 4:
"Since the main contributions are of an empirical nature, I am curious how the results shown in figures 6 & 7 look plotted against wall-clock time, since relatively low data efficiency is not a limitation for achieving perfect play in Pong (see Mnih. et al, 2015). "
RESPONSE 4:
There are more feed-forward computations when compared to having only a single network while training.
However, the feedforward computations across the different source networks, the base network and the attention network can be "parallelized".
Secondly, the backward computations for the base and attention networks are independent due to the nature of the updates.
Hence, the backward computations can be parallelized as well. This way, the wall clock time is comparable (barring minor overheads) to having only a single network while learning. We did not focus on the parallelism in implementation while running our experiments.
But in practice, we believe wall clock time is not an issue once we have a parallelized implementation. We agree with the need for this parallelism for practical implementations. We would also like to point out that previous works (Parisotto, Rusu et al) did not focus on wall-clock time in their experiments either, simply because there is a natural way to integrate parallelism into the feed-forward and backward operations.
REVIEWER POINT 5:
"It would be more illuminating to consider tasks where final performance is plausibly limited by data availability."
RESPONSE 5:
Thanks for the suggestion. We have added an experiment to specifically address this issue. (Refer to APPENDIX J, Pg 18 in the revised version).  We sparsify the Pong environment with fewer transitions of positive rewards sampled from the replay memory, so that learning a good control policy would demand more sample complexity.
Specifically, we discard 90% of the non-zero rewarding transitions thereby synthetically making Pong a harder task to solve due to reward sparsity in the replay memory. Our results clearly show that this has a significant impact in slowing down the learning when compared to learning on a normal version of Pong, as seen from Fig 13 (a) in the paper. We also show that having a positive source task expert significantly helps improve the learning when compared to learning from scratch. This is a clear illustration of a target task where performance is limited by data availability, and where the transfer learning approach is very helpful.
We think this is an interesting direction to explore in future for work in the intersection of learning from demonstrations and sparse-reward tasks, where the agent will have to bootstrap knowledge from experts to compensate for lack of observing rewards (could be due to difficulty in reward shaping or budgets on obtaining rewards, etc). We also see that the A2T framework can learn to avoid negative transfer and benefit from a source task expert even in sparse reward scenarios.
REVIEWER POINT 6:
" It would also be interesting if the presented results were achieved with reduced amounts of computation, or reduced representation sizes compared to learning from scratch, especially when one of the useful source tasks is an actual policy trained on the target task."
RESPONSE 6:
Thanks for asking about this. Apologies for not being correct in our description earlier. We have described it now in Appendix A. In all our experiments with A2T for Value Transfer except for Blurring (Section 4.1), we use reduced representation sizes for the attention and base networks. The Nature Network (Mnih et al 2015) has (32,8,8,4,4), (64,4,4,2,2), (64,3,3,1,1), (512), (output). [(32,8,8,4,4) means 32 convolution filters of dimensions 8 x8 and stride 4 x 4, while (512) means a fully connected hidden layer with 512 units.] The NIPS network (Mnih et al 2013) has (16,8,8,4,4), (32,4,4,2,2),(256), (output). Ignoring bias terms, the former has 1,684,992 parameters, while the latter has 676,608 parameters (difference of roughly 1 million parameters).  Since we have both the attention and the base networks, our total # of parameters for value transfer A2T in all experiments (other than 4.2) would be 676,608*2 = 1,353,216. This still has 300K (approximately) parameters less than a network learning from scratch (which has the Nature architecture).
However, we believe tying the weights of the attention and base network could help reduce parameters even further. This could depend on the specific source-target pair of tasks.
Regarding the experiments in Sec 4.1 (Blurring), we used the Nature Network because it gave a better end-score on the target task.
However, in this case, the source tasks are NOT expert policies on the target task. Even by using the NIPS architecture for the base and attention networks for the Blurring domain, we were able to get near-optimal scores but the final score was on average between 18 and 19, while with the Nature network, we could report scores between 19 and 20. The improvement is incremental.
REVIEWER POINT 7:
"Finally, it is perhaps underwhelming that it takes a quarter of the data required for learning Pong from scratch just to figure out that a perfect Pong policy is already in the expert library. Simply evaluating each expert for 10 episodes and using an  average-score-weighted majority vote to mix action choices would probably achieve the same final performance for a smaller fraction of the data. "
RESPONSE 7:
There is a catch in this argument. Our intention is to not just to reuse appropriate the available source task libraries, but also to supplement the existing library with a new library for the target task.
Therefore, just using the source libraries will not result in the learning of a new stand-alone library.
Secondly, it is only for the sake of experiments that can illustrate negative transfer and positive transfer together, that we constructed source task libraries containing a fully trained expert on the target task to serve as an entity for positive transfer.
However, the framework is generic and applicable for situations where new skills have to be learned for the target task, and just relying on the source expert libraries will not be sufficient. To specifically address this, we added an experiment where the positive source task is not a perfect expert policy on the target task, but rather an imperfect expert (scores 8 on an average, while a perfect expert scores 19 or more). Please refer to Appendix I in the revision uploaded. We clearly see from that result that A2T is still able to learn a perfect policy on the target task, faster than not having any positive expert, but slower than having a perfect target task policy as source expert. Just relying on voting would limit your performance to the best available source module.
Thirdly, your argument applies only to cases where the source task selection is uniform over the entire state-space. In cases where state-specific source modules have to be invoked (such as Blurring in Pong example for Value Transfer, and the Chain world example for Policy Transfer), the advantage of A2T over voting based methods is clear.
Finally, it is important to appreciate that this is a generic framework that dynamically combines multiple advantages such as Mixture of Experts when source task modules can be replicated appropriately to solve the target task, Prevention of Negative Transfer, Learning new skills when source task solutions are not available, Fine-tuning the source task modules' usage for the target task when replication maybe exhaustive but non-optimal.
We have tried to address your concerns as much as possible and look forward to clarifying any more questions on the portions revised in the paper as well as our answers. We believe the rating of the paper can be improved after the revisions and clarifications.
References:
Mnih et al 2015 - Human-level control through deep reinforcement learning Nature
Mnih et al 2013 - Playing Atari with Deep Reinforcement Learning NIPS Workshop
Parisotto et al 2015 - Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning, arxiv
Rusu et al 2016 - Progressive Neural Networks, arxiv
Rusu et al 2015 - Policy Distillation, arxiv
1) Appendix H - Visualization of Attention Weight Evolution - (Asked by AnonReviewer4)
2) Appendix I - Partial Expert Positive Source Task A2T experiment - (Asked by AnonReviewer3 WRT base network learning complementary skills and AnonReviewer5 pointing out a con that experiments relied on source tasks containing the optimal policy for the target task.)
3) Appendix J - Sparse Pong Target Task Added - (Asked by AnonReviewer5 for illustrating transfer learning in a case where the target task performance is limited by data availability)
4) Appendix A - Revised to give precise details of network architecture (Asked by AnonReviewer5 specifically WRT comparing # of parameters in A2T vs Learning from scratch when source tasks contain an optimal policy for the target task, where it is better if there are fewer parameters for A2T).
5) Mentioned in the Conclusion Section about extensions of A2T to HRL and Lifelong Learning. (Pointed out by AnonReviewer5)

Meta-review:
role: PC
data: comment: The authors present a mixture of experts framework to combine learnt policies to improve multi-task learning while avoiding negative transfer. The key to their approach is a soft attention mechanism, learnt with RL, which enables positive transfer. They give limited empirical validation of their approach. This is a general attention method which can be applied for policy gradient or value iteration learning methods. The authors were very responsive and added additional experiments based on the reviews.

Decision: Accept (Poster)