Venue: ICLR 2017 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: 
This paper explores transfer in reinforcement learning between agents that may be morphologically distinct. The key idea is for the source and target agent to have learned a shared skill, and then to use this to construct abstract feature spaces to enable the transfer of a new unshared skill in the source agent to the target agent. The paper is related to much other work on transfer that uses shared latent spaces, such as CCA and its variants, including manifold alignment and kernel CCA. 


The paper reports on experiments using a simple physics simulator between robot arms consisting of three vs. four links. For comparison, a simple CCA based approach is shown, although it would have been preferable to see comparisons for something more current and up to date, such as manifold alignment or kernel CCA. A three layer neural net is used to construct the latent feature spaces. 

The problem of transfer in RL is extremely important, and receives less attention than it should. This work uses an interesting hypothesis of trying to construct transfer based on shared skills between source and target agent. This is a promising approach. However, the comparisons to related approaches is not very up to date, and the domains are fairly simplistic. There is little by way of theoretical development of the ideas using MDP theory. 

review_title: Transfer learning in RL using a nonlinear CCA like approach 

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper considers the problem of transferring skills between robots with different morphologies, in the context of agents that have to perform several tasks.  A core component of the proposed approach is to use a task-invariant future space, which can be shared between tasks & between agents.

Compared to previous work (Ammar et al. 2015), it seems the main contribution here is to “assume that good correspondences in episodic tasks can be extracted through time alignment” (Sec. 2).  This is an interesting hypothesis. There is also similarity to work by Raimalwala et al (2016), but the authors argue their method is better equipped to handle non-linear dynamics. These are two interesting hypotheses, however I don’t see that they have been verified in the presented empirical results.  In particular, the question of the pairing correspondence seems crucial. What happens when the time alignment is not suitable. Is it possible to use dynamic time warping (or similar method) to achieve reasonable results?  Robustness to misspecification of the pairing correspondence P seems a major concern.

In general, more comparison to other transfer methods, including those listed in Sec.2, would be very valuable.  The addition of Sec.5.1 is definitely a right step in this direction, but represents a small portion of the recent work on transfer learning.  I appreciate that other methods transfer other pieces of information (e.g. the policy), but still if the end goal is better performance, what is worth transferring (in addition to how to do the transfer) should be a reasonable question to explore.

Overall, the paper tackles an important problem, but this is a very active area of research, and further comparison to other methods would be worthwhile.  The method proposed of transferring the representation is well motivated, cleanly described, and conceptually sound.  The assumption that time alignment can be used for the state pairing seems problematic, and should be further validated.
review_title: Review

final_score: rating: 7: Good paper, accept
review_content: This paper presents an approach for skills transfer from one task to another in a control setting (trained by RL) by forcing the embeddings learned on two different tasks to be close (L2 penalty). The experiments are conducted in MuJoCo, with a set of experiments being from the state of the joints/links (5.2/5.3) and a set of experiments on the pixels (5.4). They exhibit transfer from arms with different number of links, and from a torque-driven arm to a tendon-driven arm.

One limitation of the paper is that the authors suppose that time alignment is trivial, because the tasks are all episodic and in the same domain. Time alignment is one form of domain adaptation / transfer that is not dealt with in the paper, that could be dealt with through subsampling, dynamic time warping, or learning a matching function (e.g. neural network).

General remarks: The approach is compared to CCA, which is a relevant baseline. However, as the paper is purely experimental, another baseline (worse than CCA) would be to just have the random projections for "f" and "g" (the embedding functions on the two domains), to check that the bad performance of the "no transfer" version of the model is due to over-specialisation of these embeddings. I would also add (for information) that the problem of learning invariant feature spaces is also linked to metric learning (e.g. [Xing et al. 2002]). More generally, no parallel is drawn with multi-task learning in ML. In the case of knowledge transfer (4.1.1), it may make sense to anneal \alpha.

The experiments feel a bit rushed. In particular, the performance of the baseline being always 0 (no transfer at all) is uninformative, at least a much bigger sample budget should be tested. Also, why does Figure 7.b contain no "CCA" nor "direct mapping" results? Another concern that I have with the experiments: (if/how) did the author control for the fact that the embeddings were trained with more iterations in the case of doing transfer?

Overall, the study of transfer is most welcomed in RL. The experiments in this paper are interesting enough for publication, but the paper could have been more thorough.
review_title: Review

final_score: rating: null
review_content: The most widely used method for constructing latent feature spaces across source and domain tasks is canonical correlational analysis (CCA), invented more than 80 years ago. CCA requires explicit correspondences, but recent work by Wang et al. (ICML 2008, IJCAI 2009, IJCAI 2011, AAAI 2014) on manifold alignment generalizes CCA to allow for weak correspondences, as well as enables other objective functions, such as preserving global distances or local geometry. In recent years, a number of researchers in RL and robotics have used manifold alignment methods to show successful transfer in RL. It would be useful to compare against these methods, both experimentally as well as theoretically. 

Examples of such work include: 

Amar et al., ("Unsupervised Cross-Domain Transfer in Policy Gradient Reinforcement Learning via Manifold Alignment", AAAI 2015) apply manifold alignment to show transfer in RL, such as from inverted pendulum to quad rotor control. Raimalwala et al. ("A Preliminary Study of Transfer Learning between Unicycle Robots", AAAI SS 2016) apply a similar approach to show transfer in robotics as well 
review_title: Comparison to other work on feature space construction 

final_score: rating: null
review_content: I was confused by P.  Can you define this more precisely: is it a distribution?  or a mapping?  or a function?

The loss equation at the beginning of Sec.4.1 should probably have super-script “j” in the last term.

Did you compare to any other transfer method?  What would be the most appropriate?

How does the Baseline method do if it gets as much data as the transfer method (i.e. 3 times more for Fig.5 to compare with “All proxies”)

review_title: Questions

Meta-review: pros:
 - tackles a fundamental problem of interest to many
 - novel approach
 
 cons:
 - originally not evaluated against some reasonable benchmarks. Note: now added or addressed
 - little theoretical development cf MDP theory
 - some remaining questions about the necessity (and ability) to find good time alignments
 
 I personally found the ideas to be quite compelling, and believe that this is likely to inspire future work.
 The experiments represent interesting scenarios for transfer, with the caveat that they are just in simulation.
Decision: Accept (Poster)