Title: Learning Invariant Feature Spaces to Transfer Skills with Reinforcement Learning
Authors: Abhishek Gupta
Coline Devin
YuXuan Liu
Pieter Abbeel
Sergey Levine
Venue: ICLR 2017
Keywords: Deep learning
Reinforcement Learning
Transfer Learning

Abstract: People can learn a wide range of tasks from their own experience, but can also learn from observing other creatures. This can accelerate acquisition of new skills even when the observed agent differs substantially from the learning agent in terms of morphology. In this paper, we examine how reinforcement learning algorithms can transfer knowledge between morphologically different agents (e.g., different robots). We introduce a problem formulation where twp agents are tasked with learning multiple skills by sharing information. Our method uses the skills that were learned by both agents to train invariant feature spaces that can then be used to transfer other skills from one agent to another. The process of learning these invariant feature spaces can be viewed as a kind of ``analogy making,'' or implicit learning of partial correspondences between two distinct domains. We evaluate our transfer learning algorithm in two simulated robotic manipulation skills, and illustrate that we can transfer knowledge between simulated robotic arms with different numbers of links, as well as simulated arms with different actuation mechanisms, where one robot is torque-driven while the other is tendon-driven.

TL;DR: Learning a common feature space between robots with different morphology or actuation to transfer skills.

Review discussion:
The most widely used method for constructing latent feature spaces across source and domain tasks is canonical correlational analysis (CCA), invented more than 80 years ago. CCA requires explicit correspondences, but recent work by Wang et al. (ICML 2008, IJCAI 2009, IJCAI 2011, AAAI 2014) on manifold alignment generalizes CCA to allow for weak correspondences, as well as enables other objective functions, such as preserving global distances or local geometry. In recent years, a number of researchers in RL and robotics have used manifold alignment methods to show successful transfer in RL. It would be useful to compare against these methods, both experimentally as well as theoretically.
Examples of such work include:
Amar et al., ("Unsupervised Cross-Domain Transfer in Policy Gradient Reinforcement Learning via Manifold Alignment", AAAI 2015) apply manifold alignment to show transfer in RL, such as from inverted pendulum to quad rotor control. Raimalwala et al. ("A Preliminary Study of Transfer Learning between Unicycle Robots", AAAI SS 2016) apply a similar approach to show transfer in robotics as well
Thank you for the pointers to CCA and the papers using unsupervised manifold alignment. We have added experiments in Figures 5 and 7 using CCA to determine the functions F and G. We find that this method performs poorly because a linear mapping into a common feature space is not sufficiently expressive to capture the complex nonlinear interactions present.
We have also included included an additional comparison between our method (learning a common embedding between robots) and directly mapping from source states to target states. These comparisons show that using invariant nonlinear embedding functions allows for better transfer than a simple linear mapping or a prediction of the whole source state.
We find that the Bou Ammar et al. reference addresses a complementary problem to the one we aim to solve.
For example, it assumes the presence of a feature mapping phi that provides distances between states, and use these (hand designed) features to assign correspondences between states in the different domains.
In contrast, we assume that good correspondences in episodic tasks can be extracted through time alignment, and focus on learning the feature mapping itself. The focus in our work is on learning complex nonlinear mappings from states to a shared feature space.
The work from Raimalwala et al was very effective in enabling transfer in the LTI single input single output systems considered, but we believe it is not suitable for the nonlinear, more complex systems being considered in our experiments without restrictive linearizations.
Additionally these methods also learn a direct mapping between state spaces, rather than mappings to a shared feature space, which as seen from our additional comparisons, do not perform as well.  We have added information about these works to Section 2.
I was confused by P.  Can you define this more precisely: is it a distribution?  or a mapping?  or a function?
The loss equation at the beginning of Sec.4.1 should probably have super-script “j” in the last term.
Did you compare to any other transfer method?  What would be the most appropriate?
How does the Baseline method do if it gets as much data as the transfer method (i.e. 3 times more for Fig.5 to compare with “All proxies”)
Thank you for your questions.
> I was confused by P.  Can you define this more precisely: is it a distribution?  or a mapping?  or a function?:
P is a set of tuples (s_S, s_T) that put the source and target states into correspondence. The correspondences are used to specify which states should be placed close together in the feature space via the contrastive loss.
> The loss equation at the beginning of Sec.4.1 should probably have super-script “j” in the last term:
Yes, that is indeed a typo. We have updated the notation accordingly.
> Did you compare to any other transfer method?  What would be the most appropriate?
We have added two comparisons: finding the feature space using canonical correlation analysis, and learning a direct mapping between state spaces of domains. Please see Section 5.1 for further descriptions.
> How does the Baseline method do if it gets as much data as the transfer method (i.e. 3 times more for Fig.5 to compare with “All proxies”):
We ran the baseline for a larger number of iterations, and the results are shown in Table 1. These results suggest that the baseline method does not substantially benefit from the additional samples.
To address the reviewers’ questions, we have made the following changes:
-We added comparisons of our method to using canonical correlation analysis to find the embedding functions F and G (see Figures 5 and 8a)
-We added comparisons of our method to directly predicting the target states from the source states (see Figures 5, 8a)
-We have clarified the description of P in Sec 4.1
-We have added two previous approaches to the related work section.
The paper considers the problem of transferring skills between robots with different morphologies, in the context of agents that have to perform several tasks.  A core component of the proposed approach is to use a task-invariant future space, which can be shared between tasks & between agents.
Compared to previous work (Ammar et al. 2015), it seems the main contribution here is to “assume that good correspondences in episodic tasks can be extracted through time alignment” (Sec. 2).  This is an interesting hypothesis. There is also similarity to work by Raimalwala et al (2016), but the authors argue their method is better equipped to handle non-linear dynamics. These are two interesting hypotheses, however I don’t see that they have been verified in the presented empirical results.
In particular, the question of the pairing correspondence seems crucial. What happens when the time alignment is not suitable. Is it possible to use dynamic time warping (or similar method) to achieve reasonable results?  Robustness to misspecification of the pairing correspondence P seems a major concern.
In general, more comparison to other transfer methods, including those listed in Sec.2, would be very valuable.  The addition of Sec.5.1 is definitely a right step in this direction, but represents a small portion of the recent work on transfer learning.  I appreciate that other methods transfer other pieces of information (e.g. the policy), but still if the end goal is better performance, what is worth transferring (in addition to how to do the transfer) should be a reasonable question to explore.
Overall, the paper tackles an important problem, but this is a very active area of research, and further comparison to other methods would be worthwhile.  The method proposed of transferring the representation is well motivated, cleanly described, and conceptually sound.  The assumption that time alignment can be used for the state pairing seems problematic, and should be further validated.
Thank you for pointing out the highly related work by (Ammar et al. 2015) . In order to ensure a fair comparison we use the Matlab code provided by Chang Wang (https://sites.google.com/site/changwangnk/home/ma-html). We find that the unsupervised manifold learning method described in (Wang and Mahadevan 2009) and (Ammar et al. 2015) performs poorly. We suspect that this is because the method relies on pairwise distance and the raw state spaces do not provide a meaningful metric space. Ammar et al use a hand-specified feature mapping phi that provides a good metric space for the method. As we assume no prior knowledge about the state space, we do not have a hand-specific feature mapping in our experiments.
We would like to clarify that our primary contribution is to learn a feature space in which the source and target states can be compared. This differs from prior work because we do not reconstruct the source states from target states, allowing the embeddings to discard information that is not common. This is more suitable for transferring between robots with varying morphologies.
To address your comment about assuming correspondences, we have formulated an alternating optimization, alternating between establishing common feature space using current correspondences, and reestablishing correspondences using current feature space and dynamic time warping. This is described in detail in Section 3.3.2, and results are seen in Fig 5, Fig 7a.
As suggested, we have also run comparisons with several prior methods such as (Ammar et al. 2015), kernel CCA, random projections, CCA and using state mappings instead of embeddings. These results are shown in Figures 5 and 7a, and show the advantage of using our proposed method over several prior methods.
Although kernel CCA is quite competitive with our method on the tendon task, it performs very poorly on the button task while our method is able to achieve excellent performance on both.
Unfortunately we could not compare to the work in Raimalwala et al (2016) because it is quite specific to systems which have a single input, a single output, and can be modeled as linear time invariant. Our tasks are nonlinear and have multidimensional states.
This paper explores transfer in reinforcement learning between agents that may be morphologically distinct. The key idea is for the source and target agent to have learned a shared skill, and then to use this to construct abstract feature spaces to enable the transfer of a new unshared skill in the source agent to the target agent. The paper is related to much other work on transfer that uses shared latent spaces, such as CCA and its variants, including manifold alignment and kernel CCA.
The paper reports on experiments using a simple physics simulator between robot arms consisting of three vs. four links. For comparison, a simple CCA based approach is shown, although it would have been preferable to see comparisons for something more current and up to date, such as manifold alignment or kernel CCA. A three layer neural net is used to construct the latent feature spaces.
The problem of transfer in RL is extremely important, and receives less attention than it should. This work uses an interesting hypothesis of trying to construct transfer based on shared skills between source and target agent. This is a promising approach.
However, the comparisons to related approaches is not very up to date, and the domains are fairly simplistic. There is little by way of theoretical development of the ideas using MDP theory.
Thank you for bringing up several important points in the review. We have added several more comparisons to our results in Figures 5 and 7a. The first is using random projections, which performs slightly better than no transfer. The second is kernel CCA, and the last is unsupervised manifold learning, using the method presented in (Wang and Mahadevan 2009) and (Ammar et al 2015).  Please see the response to Reviewer 3 for more details on why we believe this last method performs poorly in our settings.
Additionally, we have formulated a method to find the state alignment jointly with the feature embedding as described in Section 3.3.2, allowing us to remove the assumption that the states are aligned by timestep. The method alternates between common feature space learning using current correspondences and dynamic time warping to reestimate correspondences using the current learned feature space. Figure 5 shows that this method actually provides better results than just assuming correspondences by time-step.
This paper presents an approach for skills transfer from one task to another in a control setting (trained by RL) by forcing the embeddings learned on two different tasks to be close (L2 penalty). The experiments are conducted in MuJoCo, with a set of experiments being from the state of the joints/links (5.2/5.3) and a set of experiments on the pixels (5.4). They exhibit transfer from arms with different number of links, and from a torque-driven arm to a tendon-driven arm.
One limitation of the paper is that the authors suppose that time alignment is trivial, because the tasks are all episodic and in the same domain. Time alignment is one form of domain adaptation / transfer that is not dealt with in the paper, that could be dealt with through subsampling, dynamic time warping, or learning a matching function (e.g. neural network).
General remarks: The approach is compared to CCA, which is a relevant baseline.
However, as the paper is purely experimental, another baseline (worse than CCA) would be to just have the random projections for "f" and "g" (the embedding functions on the two domains), to check that the bad performance of the "no transfer" version of the model is due to over-specialisation of these embeddings. I would also add (for information) that the problem of learning invariant feature spaces is also linked to metric learning (e.g. [Xing et al. 2002]). More generally, no parallel is drawn with multi-task learning in ML. In the case of knowledge transfer (4.1.1), it may make sense to anneal \alpha.
The experiments feel a bit rushed.
In particular, the performance of the baseline being always 0 (no transfer at all) is uninformative, at least a much bigger sample budget should be tested.
Also, why does Figure 7.b contain no "CCA" nor "direct mapping" results?
Another concern that I have with the experiments: (if/how) did the author control for the fact that the embeddings were trained with more iterations in the case of doing transfer?
Overall, the study of transfer is most welcomed in RL. The experiments in this paper are interesting enough for publication, but the paper could have been more thorough.
Thank you for your review. We agree that assuming a trivial time alignment is somewhat unsatisfying, and to address this we have formulated an alternating optimization to learn the alignment jointly with the embeddings, described in Section 3.3.2. The method alternates between common feature space learning using current correspondences and dynamic time warping to reestimate correspondences using the current learned feature space, as described in Section 3.3.2. Results using this method can be found in Figure 5, Figure 7a. Using this approach usually works better than just assuming timestep correspondences, or as well in the case of well aligned data.
As per the suggestion of comparing with random projections, we have added a comparison using random projections as feature embeddings. As shown in Figures 5 and 7a, this method achieves a low level of performance in between the CCA method and no transfer.
Also, as pointed out in the review we do indeed anneal \alpha over the course of learning in all our experiments.
As shown in Table 1, we have run the “No Transfer” method for 75 iterations (seeing 3 times more samples). A likely reason the no transfer baseline doesn’t work even with this greater sample budget is that the exploration is insufficiently coherent for any reward to be obtained at all, leading to the behavior seen.
Finally, we have added a connection to metric learning in the last paragraph of Section 2, as well as a paragraph on multitask learning (the third paragraph in Section 2).
Figure 7b does not yet contain the comparisons with CCA and direct mapping because we have been adding these comparisons over the course of the response period. This figure will contain all comparisons for the final version. Thanks!
I think that the additional work and additional details that the authors include make the paper better.
In particular all the baselines are welcomed, and the EM with DTW can be considered a baseline for subsequent work that would be based on this paper. I updated my rating from 6 to 7 (accept).
We thank the reviewers for their comments. Most comments asked for additional comparisons and learned state correspondences.
In response to reviewer comments, we have added the following to our paper:
1. Learning state correspondences by using an alternating optimization to jointly assign the state pairs in P and learn the embedding functions f and g. This significantly relaxes the assumption that states can be paired by time-step and provides better performance in experiment 1. See section 3.3.2 for a description of this method and figures 5 and 7a for results.
2. All of the comparisons suggested by the reviewers: kernel-CCA, unsupervised manifold alignment (Ammar et al. 2015), and random projections. Results are in Figures 5 and 7a.
3. Connections to metric learning and multitask learning in Section 2.
The comparisons have been run on experiments 1 and 2 and will be added to experiment 3 for the final version.
Dear reviewers,
We have addressed the concerns presented in the reviews, including comparisons to several prior methods and a method for automatically determining alignment, as detailed in the individual reviewer responses. If you have suggestions for additional comparisons, we would be happy to add them in the final version.
As such, we hope you will consider adjusting your reviews accordingly.
Regards
Abhishek, Coline, Yuxuan, Pieter, Sergey
Dear Reviewers and Area Chair,
The reviewers for the paper provided a positive evaluation, with suggestions for additional experiments providing comparisons with several prior methods, some clarifications, and brought up the important problem of addressing time-based correspondence alignment. We have edited the paper to include additional experiments that address these issues, and therefore believe that the reviewer concerns about the work have been addressed. We discuss the specifics below.
AnonReviewer 1 Comments:
“Compared to previous work (Ammar et al. 2015)”
- Performed this comparison and added the results into the Figures 5 and 8. We found that our method performed significantly better than this prior work.
“Is it possible to use dynamic time warping (or similar method) to achieve reasonable results?  Robustness to misspecification of the pairing correspondence P seems a major concern.”
- We introduced an EM style algorithm which removes this assumption. Description of this method are in Section 3.3.2, and results in Figures 5 and 8.
“more comparison to other transfer methods, including those listed in Sec.2, would be very valuable”
- We have implemented several more baselines: KCCA, Unsupervised Manifold Alignment as suggested by Bou Ammar et al, Direct mappings, random projections, CCA in Figures 5 and 8.
AnonReviewer3 Comments:
“preferable to see comparisons for something more current and up to date, such as manifold alignment or kernel CCA”, “comparisons to related approaches is not very up to date”
-Performed comparisons to Kernel CCA and Manifold Alignment using Unsupervised Manifold Alignment (Wang and Mahadevan, Bou Ammar et al) and added to our experiments in Figures 5 and 8.
AnonReviewer4 Comments:
“A limitation of the paper is that the authors suppose that time alignment is trivial [...] could be dealt with through subsampling, dynamic time warping, or learning a matching function”
-We introduce a new EM style algorithm alternating between feature learning and dynamic time warping. Description is in Section 3.3.2 and results in Figures 5 and 8.
“Another baseline (worse than CCA) would be to just have the random projections for "f" and "g"”
-We added this comparison in Figures 5 and 8.
“at least a much bigger sample budget should be tested [...] control for the fact that the embeddings were trained with more iterations in the case of doing transfer”
- We ran the baseline with a significantly higher sample budget as shown in Table 1. The poor performance is likely due to not enough guided exploration happening without good reward shaping for the baseline.
“problem of learning invariant feature spaces is also linked to metric learning [...] no parallel is drawn with Multi-Task learning in ML ”
-The additional references suggested have been added in the Related Work (Section 2).

Meta-review:
role: PC
data: comment: pros:
 - tackles a fundamental problem of interest to many
 - novel approach
 
 cons:
 - originally not evaluated against some reasonable benchmarks. Note: now added or addressed
 - little theoretical development cf MDP theory
 - some remaining questions about the necessity (and ability) to find good time alignments
 
 I personally found the ideas to be quite compelling, and believe that this is likely to inspire future work.
 The experiments represent interesting scenarios for transfer, with the caveat that they are just in simulation.

Decision: Accept (Poster)