Task: Language Identification
Input (Spanish, English): In this paper a well known soft mixture of experts model is adapted for, and applied to, a specific type of transfer learning problem in reinforcement learning (RL), namely transfer of action policies and value functions between similar tasks. Although not treated as such, the experimental setup is reminiscent of hierarchical RL works, an aspect which the paper does not consider at length, regrettably. One possible implication of this work is that architecture and even learning algorithm choices could simply be stated in terms of the objective of the target task, rather than being hand-engineered by the experimenter. This is clearly an interesting direction of future work which the paper illuminates.   Pros: The paper diligently explains how the network architecture fits in with various widely used reinforcement learning setups, which does facilitate continuation of this work. The experiments are good proofs of concept, but do not go beyond that i.m.h.o.  Even so, this work provides convincing clues that collections of deep networks, which were trained on not entirely different tasks, generalize better to related tasks when used together rather than through conventional transfer learning (e.g. fine-tuning).  Cons: As the paper well recounts in the related work section, libraries of fixed policies have long been formally proposed for reuse while learning similar tasks. Indeed, it is well understood in hierarchical RL literature that it can be beneficial to reuse libraries of fixed (Fernandez & Veloso 2006) or jointly learned policies which may not apply to the entire state space, e.g. options (Pricop et. al). What is not well understood is how to build such libraries, and this paper does not convincingly shed light in that direction, as far as I can tell. The transfer tasks have been picked to effectively illustrate the potential of the proposed architecture, but the paper does not tackle negative transfer or compositional reuse in well known challenging situations outlined in previous work (e.g. Parisotto et. al 2015, Rusu el. al 2015, 2016). Since the main contributions are of an empirical nature, I am curious how the results shown in figures 6 & 7 look plotted against wall-clock time, since relatively low data efficiency is not a limitation for achieving perfect play in Pong (see Mnih. et al, 2015). It would be more illuminating to consider tasks where final performance is plausibly limited by data availability. It would also be interesting if the presented results were achieved with reduced amounts of computation, or reduced representation sizes compared to learning from scratch, especially when one of the useful source tasks is an actual policy trained on the target task. Finally, it is perhaps underwhelming that it takes a quarter of the data required for learning Pong from scratch just to figure out that a perfect Pong policy is already in the expert library. Simply evaluating each expert for 10 episodes and using an  average-score-weighted majority vote to mix action choices would probably achieve the same final performance for a smaller fraction of the data.
Output (English): en