Policy Stitching: Learning Transferable Robot Policies
Authors: Pingcheng Jian
Easop Lee
Zachary Bell
Michael M. Zavlanos
Boyuan Chen
Venue: CoRL 2023 Poster (2023)
Keywords: robot transfer learning
policy stitching

Abstract: Training robots with reinforcement learning (RL) typically involves heavy interactions with the environment, and the acquired skills are often sensitive to changes in task environments and robot kinematics. Transfer RL aims to leverage previous knowledge to accelerate learning of new tasks or new body configurations. However, existing methods struggle to generalize to novel robot-task combinations and scale to realistic tasks due to complex architecture design or strong regularization that limits the capacity of the learned policy. We propose Policy Stitching, a novel framework that facilitates robot transfer learning for novel combinations of robots and tasks. Our key idea is to apply modular policy design and align the latent representations between the modular interfaces. Our method allows direct stitching of the robot and task modules trained separately to form a new policy for fast adaptation. Our simulated and real-world experiments on various 3D manipulation tasks demonstrate the superior zero-shot and few-shot transfer learning performances of our method.

Policy Stitching: Learning Transferable Robot Policies
Pingcheng Jian'
Easop Lee"
Zachary Bell?
Michael M. Zavlanos'
Boyuan Chen'
'uuee University Air Force Research Laboratory
Training robots with reinforcement learning (RL) typically involves heavy interactions with the environment, and the acquired skills are often sensi- tive to changes in task environments and robot kinematics. Transfer RL aims leverage previous knowledge accelerate learning of new tasks new body figurations. However, existing methods struggle to generalize to novel robot-task combinations and scale to realistic tasks due to complex architecture design or strong regularization that limits the capacity of the learned policy. We propose Policy Stitching. novel framework that facilitates robot transfer learning for novel combinations robots and tasks. Our key idea is to apply modular policy design and align the latent representations between the modular interfaces. Our method allows direct stitching of the robot and task modules trained separately to form new policy for fast adaptation. Our simulated and real-world experiments on various 3D manipulation tasks demonstrate the superior zero-shot and few-shot transfer learning performances of our method.
Abstract:
Keywords: robot transfer learning, policy stitching
(A)
Original Policies
Policy Stitching
(B)
Seedi
Sead 101
Morile
Seedi 101
Seedi 104
Tars Modade
Bober Modie
Tak Nodulk
Fabeet Wottle
Fig. I: Policy Stitching, (A) Our framework facilitates robot transfer learning among novel combinations of robots and tasks by decoupling and stitching robot and task modules. (B) Motivation example: robot arm trained reach goals four different target regions using the modular policy, Results from separate training runs with different random seeds (101-104) show misaligned latent representations.
Introduction
Robots are typically trained to excel at specific tasks such as relocating objects or navigating to predetermined locations. However, such robots need to retrained from scratch when faced with new tasks or body changes. In contrast, humans demonstrate remarkable capabilities [1] continuously acquire skills by drawing on past experiences. Even under physical constraints imposed by injuries, we can still rapidly adapt to perform new tasks. Despite significant advancements in robotics and machine learning, robots still cannot generalize their experience across a wide range of tasks and body configurations.
Model-based robot learning, in particular self-modeling or self-identification learning [2, 3, 4, 6, 7, 9, 10], aims learn predictive model of the robot's 's kinematics and dynamics and then employ this model for various downstream tasks through model predictive control. However, the
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

learning process needs to be separated into two stages of robot-specific and task-specini learning. On the other hand, model-free reinforcement learning trains policies end to end, but often has limited transfer learning performance [11]. Existing efforts regularize one large policy network for multi-task learning by learning routing connections to reuse part of the network weights [12] or assigning task-specific sub-networks [13]. However, the capacity of the large policy network grows exponentially as the number of tasks increases.
We introduce Policy Stitching (PS), model-free learning framework for knowledge transfer among novel robot and task combinations through modular policy design and transferable representation learning (Fig.1 (A)). We explicitly decouple robot-specific representation (e.g., robot kinematics and dynamics) and task-specific representation (e.g., object states) in our policy design enable the reassembly of both modules for new robot-task combinations. For instance, given one policy trained for 3-DoF manipulator < pick up cube and another policy trained for 5-DoF I manipulator to pick up a stick i. if we would like to have the 3-DoF manipulator pick up stick now (i.e.. < + 1), our method can directly take the robot module in the first policy and stitch it with the task module in the second policy.
While the modular design allows direct stitching, the reassembled policy may not work at all. We find that this is because the output representation from one neural network module does not align with the desired input representation of another module, particularly when modules trained on different tasks and robot body configurations. Past work in supervised learning [14, 15] has made similar observations. As motivating example, in Fig. (BB,, we show that, even under the same task and robot setup, latent representations from RL policies trained with different random seeds do not align with each other. More interestingly, they exhibit similar isometric transformations as shown in recent work on supervised learning [14, 15]. See Appendix C.I for additional details of the isometric transformation phenomenon.
To this end, we further propose to generalize the latent representation alignment techniques from supervised learning to reinforcement learning for robot transfer learning. The key idea is to enforce transformation invariances by projecting the intermediate representations into the same latent coordi- nate system. Unlike supervised learning, RL does not come with human labels to help select anchor coordinates. We propose to use unsupervised clustering of target states to help resolve this gap. Our method produces aligned representations for effective policy stitching. In summary, our contributions are three-fold:
Policy Stitching, model-free reinforcement learning framework for robot transfer learning among novel robot and task combinations.
Modular Policy Design for robot-specific and task-specific module reassembly. Represen- tation Alignment to learn transferable latent space for direct module stitching.
Demonstration of the clear advantages our method in both zero-shot and few-shot transfer learning through simulated and physical 3D manipulation tasks.
2
Related Work
Robot Transfer Learning. Transferring robot policies across various environment dynamics or novel tasks is still open challenge [16, 17, 18, 19]. Past efforts have proposed to transfer different components in reinforcement learning framework, such as value functions [20, 21, 22], rewards [23], experience samples [24], policies [25, 26, 27], parameters [28, 29, 30], and features [31]. In contrast, our method transfers structured network modules of a policy network across novel tasks or novel robot body configurations. Our modular design is similar to previous work [27], but we do not limit the capacity of the latent space to avoid overfitting. Instead, we enforce invariances among the learned latent representations of different modules through feature alignment. Moreover, our work surpasses previous accomplishments in 2D tasks by demonstrating 3D manipulation skills, both in simulation and the real world.

---

2
module A
Benl
Itan)
Peletive Representadion
Rebstiie Represorcation
Juiee (Sh
Retatton Resonsemaaionn
robor module
roboc modulc Itf,
Action
Policy
Pollcy
Policy Stitching
Fig. 2: Method Overview. Our modular policy design enables the task module to process task-specific states such object and environmental states and the robot module process robot-specific states such as kinematics and dynamics. We generalize relative representations model-free RI setup align latent representations from separately trained policies. With both modular design and latent representation alignment, our method allows direct stitching of unseen combination of task and robot module for effective and efficient transfer.
Meta Learning. Meta-learning [32, 33, 34, 35, 36, 37] aims to achieve fast adaptation on new task based on past learning experiences. Recent research has proposed generating better parameter initialization for learning new tasks [30, 38, 39, 40] or using memory-augmenteed neural networks [41, 42, 43] to assimilate new data swiftly without forgetting the previous knowledge. Another category of methods [44, 45, 46, 47, 48, 49] proposes to use a hypernetwork to generate the parameters of policy networks for different tasks. Our work also aims fast adaptation novel robot and task combinations, but method reuses structured components in policy networks.
Compositional Reinforcement Learning. Functional compositional RL has been used in zero-shot transfer learning [27], multi-task learning [12], and lifelong learning [13]. Some approaches learn the structure of the modules [50, 51, 12, 13], but these trained modules can only be functional in large modular system and cannot work when stitched with new modules. Another method [27] tries make the network module reusable, but the module alignment problem prevents it from working on 3D tasks. Our work directly stitch policy modules to improve zero-shot and few-shot learning performances in simulated and physical 3D manipulation tasks.
3 Method: Policy Stitching
Policy Stitching focuses on enabling effective knowledge transfer between different tasks or robot bodies. Our method consists of two main components: the modular policy design and transferable representations achieved through latent space alignment. Our framework is designed to be compatible with various model-free RI. algorithms.
3.1 Modular Policy Design
We propose modular policy design to decompose the policy into two distinct modules focusing robot specific and task specific information. Our design provides straightforward ingredient for module reuse and stitching. Consider an environment, E, with a robot and presented task k, we formulate the problem as Markov Decision Process with an observed state sg and action ag. We denote the learning policy as Te.. (ag $g) parameterized by function (8g). We utilize the model-free RL formulation, specifically the Soft Actor- (SAC) [52] algorithm.
We decompose the state SF, the policy function &F,,, (sæ), and the Q-function into robot-specific module and a task-specific module (Fig. 2). The state s g is decomposed into a robot-specific state SE, and a task-specific state SE,T. The robot-specific state SE,R. only consists of the joint angles of the robot, while the task-specific state $E.T includes task information at hand such as the current position and orientation, linear and angular velocity, and goal position of the object. Similarly, the policy function &F,, (Sæ) is also decomposed into a task-specific module gs encode the input task states, and robot-specifee module to implicitly capture robot kinematics and dynamics from input robot states. We follow the same design to decompose the Q-function into task-specific module qk

---

and robot-specific module h,. Note that this decomposition is similar to previous work [27] with generalization on both actor and critic network. Formally, our policy function and Q-function can be re-written as follows:
EE (SE) = ΦE,, (SE,T, = fr (9k (SE,T) ,E,R)
(I)
QE (g) QE (SE,T, SE,R, ag) = hr (gk (SE,T) ,..,,
(2)
For two policy networks fri (9k1 , and J+2 (9k2 (#E,T) .E,R), ) we define Policy Stitch- ing as constructing another policy network fr2 (gta ($E,7), , SE,R) by initializing the task module with parameters from 9&1 and initializing the robot module with parameters from fr2. The Q-function stitched in similar way.
We name the modules after their main functionalities. Such modular design does not completely separate the information processing from each module. Since we train the entire policy end to end, the gradients flow in between the two modules during backpropagation. However, by explicitly enforcing their main functionalities through our modular design, we can still obtain effective transfer learning by stitching modules to leverage previously acquired knowledge. After the stitching, we can either use the new policy for zero-shot execution or perform few-shot learning for fast adaptation.
3.2 Transferable Representations through Latent Space Alignment
Our modular policy design allows for direct stitching, but simple stitching approach may not yield optimal performance. Previous research [27] attributes the issue as overfitting to a particular robot and task, and has proposed to use dropout and very small bottleneck dimensions. However, the capacity of the policy is largely limited to simple 2D tasks and dimenniiional task states. Instead, we identify the fundamental issue the lack of alignment enforcement of the latent embedding space the output/input interface of two modules. Even under the same robot-task combination, the latent embedding vectors between the robot and task modules from different training seeds exhibit an almost isometric transformation relationship (Fig. (B)). While similar observation has been made in supervised learning [15, 53], we have identified and addressed this issue in the context of RL and demonstrate the effectiveness in knowledge transfer across novel robot-task combinations.
We propose generalize Relative Representations [15] in supervised learning model-free RL. Un- like other techniques in supervised learning [54, 14, 55), Relative Representations does not introduce additional learnable parameters. The key idea is to project all latent representations at the interface between two modules to a shared coordinate system among multiple policy networks. Through this invariance enforcement of transformation, we can establish consistent latent representation of the task-state that aligns with the desired latent input of the robot module.
However, the original Relative Representations relies on the ground-truth labels in the supervised dataset to provide the anchor points for building the latent coordinate system. Our setup in RL does not come with such labels. The anchor points are analogous to the concept of basis in a linear system, hence should be as dissimilar from each other as possible. To overcome this challenge, we first collect a task state set S by rolling out two trained naive policies in their own training environments. We then perform unsupervised clustering with k-means [56] on these task states to select an anchor set A. Specifically, we select anchor states that are closest the centroids of the clusters. The number of clusters i depends the dimension of the latent representation.
We want to represent every embedded task state e = 9k (s) with respect to the embedded anchor states Cat)) 9k )), where a A and 9& is the task-specific module that embeds the task states. To this end, we capture the relationship between an embedded task state egto and an embedded anchor state eash using a similarity function sim IRd x RRd + R. which calculates a similarity score r = sim (e,m) e,()) Given the anchor set all) .((.),, the transferable representation of an input task state ((1) E is calculated by:
T sim (Eg()) e aa.))) sim (eal), e ( e
...., sim (a00,ea(44))) (
(3)
which is vector of length |A], and each element is the similarity score between the embedded task state and an embedded anchor state. We intentionally choose the cosine similarity our similarity

---

Pushi
Puarz
PuBi3
Pik1
Ph32
Plaest
Fubo2
Rubus
Fig. 3: Simulation Experimental Setup: From left to right, show different task and robot configurations: standard table, slippery table, rough terrain with small rocks, cube object, stick object, 7-DoF Franka Emika Panda arm, third joint locked, and fifth joint locked.
measure because it is invariant reflection, rotation ,and rescaling. Although it is not invariant to vector translation, we add normalization layer before calculating cosine similarity mitigate this issue. In this way, our proposed Relative Representation for RL projects latent representations from different policies into a common coordinate system. Therefore, it overcomes the isometric transformation issue of the original latent representations and makes them better aligned and invariant.
3.3 Implementation Details
We challenge our methods in sparse reward setup and use hindsight experience replay [57] to encourage exploration. Both the task and robot modules are represented as MLPs [58] with four layers and 256 hidden dimensions. The last layer of the task module and the first layer of the robot module have a dimension 128, thus the dimension the latent representation and the selected anchor set are also 128 = 128). Detailed network structures of our method and other baselines can be found in Appendix Each of our policy is trained 7 threads of the AMD EPYC 7513 CPU and NVIDIA GeForce RTX 3090 GPU. Each training epoch consists of 35, 000 steps.
4 Experiments
We aim to evaluate the performance of Policy Stitching (PS) in both zero-shot and few-shot transfer setups. Furthermore, we generalize our experiments to physical robot setup to demonstrate the practical applicability and real-world performance. Finally, we provide quantitative and qualitative analysis to understand the role of the learned transferable representations in effective policy stitching.
4.1 Simulation Experiment Setup and Baselines
We modify the panda- gym environment [59] to simulate manipulation task environments and 3 distinct robots with varying kinematics (Fig.3). The tasks consist of 3 push scenarios, where the robot must push cube to the goal position on surfaces with very different friction properties, and 2 pick tasks that involve pick and place task of objects with different shapes. zero-shot evaluation, the stitched policy is directly tested without fine-tuning and each experiment consists of 200 test trajectories, which are repeated times report the mean and standard deviation. few-shot evaluation, the stitched policy first interacts with the environment for additional 10 epochs to store data in the replay buffer and is then fine-tune funet for few epochs with SAC. We train with 3 random seeds report the mean and standard deviation.
Robot-Task Combination. We evaluate PS on 8 robot-task combinations to cover policy stitching in both similar and dissimilar setup. We list all combinations as the titles of sub-figures in Fig.4. For instance, "t: Pi1-R1" represents task module trained in the Pickl task with Robotl, while "r: Pul-R2" represents robot module trained in the Pushl task with Robot2. These modules are then stitched together to form new combination that has not been jointly trained together.
Baselines. We compare our method to the approach of Devin et al. [27], which uses similar modular policy but with small bottleneck and dropout as regularization to alleviate the modules misalignment issue. In the few-shot test, we also include an additional comparison to increase the bottleneck dimension the Devin et al. [27]'s method to the same dimension as our method, since the original bottleneck dimension is too small to capture necessary information for complex 3D tasks. Furthermore, we provide ablation study with our modular policy design but no latent
5

---

Touching Rate (%)
Success Rate (%)
PS
PS(Ablation)
Devin et al.
Plain
PS
PS(Ablation)
Devin et al.
Plain
E1
73.0 + 1.4
75.8 4.4
44.8
0.0 0.0
26.9 + 3.6
13.9 3.5
11.7 1 2.6
6.5 2.5
E2
99.9
= 0.2
93.5 2.0
78.3 5.4
27.9
24.4
1.5
6.8
10.0 0.9
9.3
E3
90.8 2.3
9
82.6 1.6
0.0 0.0
16.5 1.9
7.2
9.5
3.7
E4 9.94
88.7 + 1.1
1.1
1.6 1.0
1.12
14.9
11.8
9.0 1.4
E5
14.7 2.7
9.6 164:1.6 1.6
8.6
0.9
2.1
1.9
3.8 1.6 2.9
E6
55.6
80.8 + 3.4
3..5
0.1
8.8
11.6
1..5
9.0
E7
41.8 + 1.7
13.1 + 2.9
9.8 1.3
0.0 0.0
4.7
3.6
3.0
8.2 1.4
E8
18.4
18.8 + 2.6
13.6 1.1
0.0 0.0
1.7
3.7
3.1
3.0 0.7
Tab. 1: Success rates and touching rates of zero-shot transfer
representation alignment to study its importance. We also build a Plain baseline which is an MLP that takes in all the states the first layer and has modular structure. When transferring novel task-robot combination, as in Appendix Fig. 8(c)) and perform the same reassembling operation as the modular networks. Specifically, when performing the stitching operation of the Plain baseline, the top half of a Plain network is stitched to the bottom half of another Plain network. This operation that preserves a sub-network from the old task and adds new sub-network for the new task has been widely used in other works [60, 61, 62, 63, 64]. See Appendix A for architecture details.
split the Plain MLP network into two parts (i.e..to-hhalf and bottom-hat half
Metrics. We use success rate (task completion within fixed number of steps) and touching rate (contact with the object during the task) as our evaluation metrics, In challenging transfer scenarios (e.g., a robot trained for pushing task is required to perform a picking task), all methods may exhibit low success rates. Therefore, we use touching rate as a more informative metric to indicate whether the robot is engaging in meaningful interactions rather than arbitrary movements. Effective touching behavior can result improved exploration for few-shot transfer learning, since touching serves as preliminary behavior for picking pushing.
4.1.1 Results: Zero-Shot Transfer in Simulation
E1-E3 are easier robot-task combinations than E4-ES since the unseen setup is closer to their original combinations. As shown in Tab. 1. for E1-E3, PS achieves significantly better or comparable results than Devin et al. [27] and the PS(Ablation) method, suggesting the latent representation alignment serves as the fundamental step to enable effective module stitching. Our method also outperforms Plain MLP policy, which indicates both modular policy design and latent representation alignment beneficial.
For more challenging cases (E4-E8) where both task and robot configurations are drastically different, though all methods do not exhibit strong success rates, PS demonstrates notably higher touching rates. This implies more meaningful interactions that can potentially lead to a higher success rate
PS(Ablation)
al.(Large)
S1.. PutaI & PuZ-12
PuZ-I1 & Fut-12
Pa1-55 & Jusel!
Ee: Putan Pa1-2
Fig. 4: Success rates of the few-shot transfer learning in simulation.

---

and faster adaption in our few-shot experiments. Qualitatively, Plain baseline shows limited attempts to complete the task with frequent aimless arm swinging. Though PS(Ablation) and Devin et al. [27] methods show higher touching rates than the Plain baseline, they lack consistent attempts to move the target object to the goal after touching. In contrast, our PS method makes the robot attempt to push the object to the goal in most experiments. We show such qualitative comparisons in our supplementary video.
Sh locked
4.1.2 Results: Few-shot Transfer in Simulation
In Fig.4, PS achieves the highest or comparable success rates. No- tably, in E2 and E4, while other methods tend get stuck local minima and reach a plateau at sub-optimal success rates, PS con- verges to significantly higher rates. In some experiments such as E1, E4, E5 and E7, PS transfers much faster and achieve higher success rates during the early phase of fine-tuning. This high transfer effi- ciency supports our hypothesis in the zero-shot experiment that the high touching rates achieved by PS can facilitate transfer learning through meaningful interactions. Even in challenging scenarios where all methods struggle to achieve a high success rate, PS quickly adapt based limited interactions. Furthermore, while some methods may achieve comparable results in a few instances, PS is the only method that demonstrates consistent satisfactory performance across all scenarios, suggesting that PS serves as a stable transfer learning framework.
Fig. 5: Real World Setup.
4.2 Real World Experiment
First, we aim to evaluate the zero-shot transfer performance of PS in the real world by directly transferring the simulated policies to novel robot-task combinations. Moreover, we are interested in accessing the feasibility of continuously improving such policies given limited real-worlh interactions.
Setup. In our real-world experiment (Fig.5), we include Robot1 as 6-DoF URS arm, a Robot2 the same UR5 arm but with the fifth joint being locked, Task1 as pushing cube to goal position, and a Task2 as pushing a cylinder. The friction coefficient between the object and the table is 0.16 for Task and 0.72 for Task 2. We first train Robotl Task2 and Robot2-Task1 1 pairs in simulation. We then create I through policy stitching. For the few-shot experiments, we first fine-tune the policy for 50 epochs in simulation and test it in the real world. Following this, we then allow the policy to interact with the physical world for an additional 6 epochs (5 hours). We use an Intel RealSense D435i to detect the ArUco markers [65] attached to the object and the goal position on the table. Both objects are 3D printed. The cylinder has a sandpaper its bottom increase the friction coefficient. All evaluation results are based on 20 testing instances.
MOM
Pijailatint ei Simu lolion
in srrulatiion
Epectic
Fig. 6: Few-Shot Results. Solid curves show the fine-tuning in simulation. Dotted curves show the further fine-tuning in the real world.
Zero-Shot Results. PS achieves a success rate of 40% and touching rate of 100% in zero-shot transfer. In comparison, both the PS(Ablation) and the Plain baseline have 0% success rate. The PS(Ablation) achieves a 75% touching rate, while the Plain baseline cannot even touch the object. The superior performances PS indicate the great potential of policy stitching to generalize to novel robot-task combinations in real-world settings.
Few-Shot Results. As shown by the solid curves in Fig.6, after the first stage of fine-tuning in simulation, PS achieves the highest success rate of about 90% in simulation, surpassing both the PS(Ablation) and Plain methods (~70%). When this policy is directly transferred to the real URS arm, PS still achieves the highest success rate at 75%, followed by the PS(Ablation) at 50% and the Plain method at 45%.
These results indicate that the modular design with relative representation achieves promising success rate new task in both simulation and the real world after a few epochs of fine-tuning, Due
7

---

Interface of PS
Interface of PS(Ablation)
Fig. 7: Latent Space Anal-
(A)
Taskc Reach
Task Reath
Tosk: Reathh
Teak: Beach
Trek: Beach
Tesk: Reach
ysis. We train robots with 3 different kinematics for the reaching task and visualize their latent space colorized by 4 groups of task states, similar to Fig. I. (A) Visu- alization of the 2D princi- pal components of the la- tent representations. (B) The average pairwise cosine distances and L2 distances between each pair of the stitched latent states.
smaefhocee mockise reckerkk with rebete reprrsentaiion
Interaacss of modela network rdalive ropresoriiaion
with relatie spressrttate sadee
with valiv rep vsnttaaiion
Mart cacie vist auuu mornssiitee
1' kittec retatia appmartatton
63
KONO
10000 10b1z Anlec
100000 10E22 Mdd
IREAL Kaee
to the sim2real gap, the success rates of all methods drop to some extent. However, after 6 epochs of fine-tuning on the real robot platform shown as the dotted curves, PS results in a final success rate of 90%. Although the success rates of both baselines (i.e., 60% and 55%) are improved after the fine-tuning, there is still a significant gap with PS. Through both zero-shot and few-shot experiments, we ensure that our method performs effectively, efficiently, and reliably in physical scenarios.
4.3 Analysis: Latent Space at Module Interface
To understand how our latent space alignment benefits effective policy stitching by learning transfer- able representations, we provide further quantitative and qualitative analysis. Since the dimension of the latent embedding at the task-robot module interface is 128, we cannot directly visualize the latent representations. We apply the Principal Component Analysis (PCA) [66] to reduce the dimension of the representation to 2. We train robots with 3 different kinematics on reaching task with and without our latent space alignment. We then colorize the principal components based on their task states and group them into four directions as in motivation example (C)). As shown in F(g.7(A), the latent spaces of all policies trained with PS remain consistent across various training environments, while others without latent space alignment still show approximate isometric transformations. The consistent latent space explains the effectiveness of our policy stitching method.
To account for potential information loss during PCA visualization, we also provide quantitative analysis. In Fig.7(B), we calculate the pairwise cosine distances and 1.2 distances (Appendix C.2) between the raw tigh-dimensional latent states of different stitched policies. The pairwise distances are significantly smaller when the latent representations are aligned. PS achieves an average cosine distance of 0.0055 and L2 distance of 0.240, while ablation has much higher cosine distance of 0.865 and L.2 distance of 1.275. This analysis indicates that the latent representations of normal modular networks differ greatly from each other, and the latent representation alignment can successfully reduce such differences and facilitate the learning of transferable representations for policy stitching-
Conclusion, Limitation and Future Work
In this work, we propose Policy Stitching, novel framework to transfer robot learning policy to novel robot-task combinations. Through both simulated and real-world 3D robot manipulation experiments, we demonstrate the significant benefits of our modular policy design and latent space alignment in PS. PS paves promising direction for effective and efficient robot policy transfer under the end-to-end model-free RL framework.
One limitation that our current method of selecting anchor states based on clusters of test states may not generalize scenarios with high- high-dimensional state representations, such as images. An exciting future direction is to study self supervueee methods disentangle latent features for anchor selections. This can further help generalize PS to more complex tasks and reward settings. Another limitation is that our adapted relative representation method still cannot enforce strict invariance of the latent features, hence requires some level of fine-tuning. It will be interesting to explore alternative methods for aligning modules of networks without the need for anchor states. We also plan to explore various robot platforms with different morphology to generalize PS to more diverse platforms.
8

---

Acknowledgments
This work supported in part by ARL under awards W91 and INN122220113,, by AFOSR under award 19--550019-1-0169, and by NSF under award CNS-1932011.
References
[1] S. J. Pan and Q. Yang. A survey transfer learning. IEEE Transactions on knowledge and data engineering, 1315019335,,335, 2010.
[2] Bongard, V. Zykov, and H. Lipson. Resilient machines through continuous self-modeling. Science, 314(5802)11118-11211 2006.
[3] A. S. Polydoros and L. Nalpantidis. Survey of model-based reinforcement learning: Applications robotics. Journal of Intelligent & Robotic Systems, 86(2):153-173, 2017.
[4] Ebert, C. Finn, S. Dasari, A. Xie, A. Lee, and S. Levine. Visual foresight: Model-based deep reinforcement learning for rision-based robotic control. arXiv preprint arXiv:1812.00568. 2018.
[5] R. Kwiatkowski and Lipson. aank- self-modeling machines. Science Robotics, (2)) 2019.
[6] B. Chen, Y. Hu, L. Li, Cummings, and H. Lipson. Smile like you mean it: Driving animatronic robotic face with learned models. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 2739-2746. IEFE, 2021.
[7] K. Hang. W. G. Bircher, A. Morgan, and M. Dollar. Manipulation for self-identiffiattin,, and self-identification for better manipulation. Science robotics, 6(54)xebbe132) 2021.
[8] B. Chen, R. Kwiatkowski, Vondrick, and H. Lipson. Fully body visual self-modeling of robot morphologies. Science Robotics, 7(68) 1944, 2022.
[9] R. Kwiatkowski, Hu, B. Chen, and Lipson. On the origins of self seff-modeling. arXiv preprint arXiv:2209.088 02010, 2022.
[10] Y. Hu, B. Chen, and H. Lipson. Egocentric visual self-modeling for legged robot locomotion. arXiv preprint 2xXi:22077033866 2022.
[11] J. Hua, L. Zeng, G. Li, and Z. Ju. Learning for robot: Deep reinforcement learning, imitation learning, transfer learning. Sensors, (4):11277, 2021.
[12] R. Yang, H. Xu, Y. Wu, and X. Wang. Multi-task reinforcement learning with soft modulariza-
tion. Advances in Neural Information Processing Systems, 33:4767-4477, 2020.
[13] J. A. Mendez, H. van Seijen, and E. Eaton. Modular lifelong reinforcement learning via neural composition. arXiv preprint arXiv:2207.0044 2022.
[14] M. Gygli, J. Uijlings, and V. Ferrari. Towards reusable network components by learning compatible representations. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pages 7620-7629, 2021.
[15] L. Moschella, V. Maiorca, M. Fumero, A. Norelli, F. Locatello, and E. Rodolà. Relative representations enable zero-shot latent space communication. arXiv preprint arXiv:2209.15488 15430, 2022.
[16] M. E. Taylor and P. Stone. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research, 10(7), 2009.
[17] Z. Zhu, K. Lin, and J. Zhou. Transfer learning in deep reinforcement learning: A survey. arXiv preprint arXiv:2009. 07888, 2020.

---

[18] S. Mohanty, J. Poonganam, A. Gaidon, A. Kolobov, B. Wulfe, D. Chakraborty, G. Semetulskis, J. Schapke, J. Kubilius, J. Pašukonis, et al. Measuring sample efficiency and generalization in reinforcement learning benchmarks: Neurips 2020 procgen benchmark. arXiv preprint arXiv:2103. 15332, 2021.
[19] P. Jian, C. Yang, D. Guo, H. Liu, and F. Sun. Adversarial skill learning for robust manipulation. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 2555-2561. IEEE, 2021.
[20] A. Tirinzoni, R. Rodriguez. Sanchez, and M. Restelli. Transfer of value functions via variational methods. Advances in Neural Information Processing Systems, 31, 2018.
[21] Y. Zhang and M. M. Zavlanos. Transfer reinforcement learning under unobserved contextual information. In 2020 ACM/IEEE 11th International Conference Cyber-Physical Systems (ICCPS), pages 75-86. IEEE, 2020.
[22] C. Liu, Y. Zhang, Y. Shen, and M. M. Zavlanos. Learning without knowing: Unobserved context in continuous transfer reinforcement learning. In Learning for Dynamics and Control, pages 791-802. PMLR, 2021.
[23] G. Konidaris and A. Barto. Autonomous shaping: Knowledge transfer in reinforcement learning. In Proceedings the 23rd international conference on Machine learning, pages 489-496 2006.
[24] A. Lazaric, M. Restelli, and A. Bonarini. Transfer of samples in batch reinforcement learning. In Proceedings of the 25th international conference on Machine learning, pages 544 551, 2008.
[25] F. Fernández and M. Veloso. Probabilistic policy reuse in a reinforcement learning agent. In Proceedings of the fifth international joint conference on Autonomous agents and multiagens systems, pages 720-727, 2006.
[26] G. D. Konidaris and A. G. Barto. Building portable options: Skill transfer in reinforcement learning. In Ijcai, volume 7, pages 895-900, 2007.
[27] Devin, A. Gupta, T. Darrell, P. Abbeel, and S. Levine. Learning modular neural network policies for multi-task and multi-robot transfer. In 2017 IEFE international conference on robotics and automation (ICRA), pages 2169-2176. IEEE, 2017.
(28) F. Dosei-Velez and G. Konidaris. Hidden parameter markov decision processes: A semipara- metric regression approach for discovering latent task parametrizations. In IJCAI: proceedings of the conference, volume 2016, page 1432. NIH Public Access, 2016.
[29] T. W. Killian, S. Daulton, G. Konidaris, and F. Doshi-Velez. Robust and efficient transfer learning with hidden parameter markov decision processes. Advances in neural information processing systems, 30, 2017.
[30] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pages 1126-1135. PMLR, 2017.
[31] A. Barreto, W. Dabuey, R. Munos, J. Hunt, T. Schaul, H. P. van Hasselt, and D. Silver. Successor features for transfer in reinforcement learning. Advances in neural information processing systems, 30, 2017.
[32] J. Schmidhuber. Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta-.. hook. PhD thesis, Technische Universität München, 1987.
[33] J. Schmidhuber. Learning to control fast-weight memories: An alternative to dynamic recurrent networks. Neural Computation, 4(1):131-139, 1992.
[34] J. Schmidhuber. A neural network that embeds its own meta-levels. In IEEE International Conference Neural Networks, pages 407-412. IEBE, 1993.
10

---

[35] S. Thrun and L. Pratt. Learning to learn. Springer Science & Business Media, 2012.
[36] B. M. Lake, T. D. Ullman, J. B. Tenenbaum, and S. J. Gershman, Building machines that learn and think like people. Behavioral and brain sciences, 40:e253, 2017.
[37] J. Vanschoren. Meta-learning: survey. arXiv preprint arXiv:/810.03545 2018.
[38] C. Finn, K. Xu, and Levine. Probabilistic model-agnostic meta-learning. Advances in neural information processing systems, 31, 2018.
[39] C. Finn, A. Rajeswaran, S. Kakade, and S. Levine. Online meta-learning In International Conference on Machine Learning, pages 1920-1930. PMLR, 2019.
[40] A. Nagabandi, I. Clavera, S. Liu, R. S. Fearing, P. Abbeel, S. Levine, and C. Finn. Learning to adapt in dynamic, real-world environments through reitforeiifo learning. arXiv preprint arXiv: 1803. 1347, 2018.
[41] A. Graves, G. Wayne, and I. Danihelka. Neural turing machines. arXiv preprint arXiv: 1410.5401, 2014.
[42] A. Santoro, S. Bartunov, M. Botvinick, D. Wierstra, and T. Lillicrap. Meta-learning with memory-augmentedd neural networks. In International conference on machine learning, pages 1842-1850. PMLR, 2016.
[43] Munkhdalai and Yu. Meta networks. International conference on machine learning. pages 2554-2563. PMLR, 2017.
[44] Y. Duan, Schulman, X. Chen, P. L. Bartlett, I. Sutskever, and P. Abbeel. RI 2: Fast reinforce- ment learning via slow reinforcement learning, arXiv preprint arXi::1611.02799 2016.
[45] D. Zhao, J. von Oswald, S. Kobayashi, J. Sacramento, and B. F. Grewe. Meta-learning via hypernetworss. 4th Workshop on Meta-Learning at NeurlPS 2020, Vancouver Canada, 2020.
[46] D. Ha, A. Dai, and Q Le. Hypernetworks. arXiv preprint arXiv: 16Xi::1609.09166 2016.
[47] J. Von Oswald, C. Henning, J. Sacramento, and B. F. Grewe, Continual learning with hypernet- works. arXiv preprint 1rxiv:1906.00695, 2019.
[48] Beck, M. T. Jackson, R. Vuorio, and S. Whiteson. Hypernetworks in meta-reinforcement learning. arXiv preprint arXiv:2210. 2022.
[49] Z. Xian, Lal, H.-Y. Tung. E. A. Platanios, and K. Fragkiadaki. Hyperdynamics: Meta-learning object and agent dynamics with hypernetworks. arXiv preprint arXiv:2103.094339 2021.
[50] A. Goyal, A. Lamb, J. Hoffmann, S. Sodhani, S. Levine, Y. Bengio, and B. Schölkopf. Recurrent independent mechanisms. arXiv preprint arXiv:1909, 10893, 2019.
[51] S. Mittal, A. Lamb, A. Goyal, V. Voleti, M. Shanahan, G. Lajoie, M. Mozer, and Y. Bengio. Learning to combine top-down and bottom-up signals in recurrent neural networks with attention over modules. In International Conference on Machine Learning, pages 6972-6986. PMLR, 2020.
[52] Haarnoja, Zhou, P. Abbeel, and Levine. Soft actor-critie: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[53] C. Olah. 2015-01-Visualin Rerpraiiiiinggpppereeesstiiossss 2015. Accessed: Month Day, Year.
Visualizing representations.
http://colah. woolla..gtthiie rithub. io/posts/

---

[54] K. Lenc and A. Vedaldi. Understanding image representations by measuring their equivariance and equivalence. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 991-999, 2015.
[55] T. Hofmann, B. Schölkopf, and A. J. Smola. Kernel methods in machine learning. The annals of statistics,
2008.
[56] A. Hartigan and M. A. Wong. Algorithm 136: A k-means clustering algorithm. Journal of the royal statistical society. series (applied statistics), 28(1):100-108, 1979.
[57] M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin,
O. Pieter Abbeel, and Zaremba. Hindsight experience replay. Advances neural information processing systems, 30, 2017.
[58] S. Haykin. Neural networks: comprehensive foundaattoiiveeutioiion Prentice Hall PTR, 1998.
[59] Q Gallouédec, N. Cazin, E. Dellandréa, and L.. Chen. panda-gym: Open-Source Goal- Conditioned Environments for Robotic Learning. 4th Robot Learning Workshop: Self- Supervised and Lifelong Learning at NeurlPS, 2021.
[60] J.-T. Huang, Li, D. Yu, L. Deng, and Y. Gong. Cross-language knowledge transfer using multilingual deep neural network with shared hidden layers. In 2013 IEEE international conference acoustics, speech and signal processing, pages 7304-7308. IEEE, 2013.
[61] M. Oquab, Bottou, I. Laptev, and J. Sivic. Learning and transferring mid-level image representations using convolutional neural networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1717-1724, 2014.
[62] M. Long, H. Zhu, Wang, and M. Jordan. Unsupervised domain adaptation with residual transfer networks. Advances in neural information processing systems, 29, 2016.
[63] F. Yang, C. Yang, H. Liu, and F. Sun. Evaluations of the gap between supervised and reinforce- ment lifelong learning on robotic manipulation tasks. In Conference on Robot Learning, pages 547-556. PMLR, 2022.
[64] Yoon, S. Kim, E. Yang, and S. J. Hwang. Scalable and order-robust continual learning with additive parameter decomposition. arXiv preprint arXiv:1902. 2019.
[65] S. Gurrido-Jurado, Muñoz-Salinas, F. Madrid-Cuevas, -uevas, and M. Marín-Jiménez. Aruco: a minimal library for augmented reality applications based on openev. Journal of Real-Time Image Processing, 9(2):399-406, 2014.
URL https://doi.
[66) K. Pearson. lines and planes of closest fit to systems points in space. Philosophical Magazine, 2(11):559-577) 1901, doi:10. 10801.0010111111 47864401094627200.
12

---

Appendix
A Network Structure
[joint task stale]
[lask state]
state dimemioon
1esh modul
dimension
256 units
taok module
units
256 units
120 units
repemestatto
normalizzionn
bottom
[oint statel
5im (attnt losk pate lotent anthor stal)
128 units
[nint state)
128 units
256 units
256 units
robot
anits
robut modile
256 units
26 unils
action mean, std
aclion muan, std
Action
Action
(a)
(b) PS(Ablation)
Plain Baveline
baak state]
(lask state]
state dimension
state dimension
task module
module
units
256 units
units
16 units
128 units
dirppout
drapout
[joint statn)
state)
units
[jattt state]
136 units
robot module
256 units
robot
256 units
module
256 units
256 units
action mean, std
action mean, std
Action
Action
(d) Devin et al. 1271
(e) Devin et [27] (Large)
Fig. 8: Detailed network structure of PS, PS(Ablation), Devin al., Devin et al.(aarge) and Plain Baseline methods.
13

---

B Policy Stitching Algorithm
Algorithm 1 Zero-shot Transfer with Policy Stitching
Train Plain Policies in Source Environments:
Train Plain policy TT1k1 with SAC in the environment 1,111 Train Plain policy 7,222 with SAC in the environment 2,
Collect Anchor States:
Roll out Teiki in EIIRI to gather a set of task states & Roll out Ty222 in Eyg22 to gather set of task states $2
task state set < {S), S2
K-means center C < (Sa
Select anchor set {0) which task states closest to the K-means center C.
Train Modular Policies in Source Environments:
Initialize modular policy 1 Óg.. with Ski for its task module and fri for its robot module. Initialize modular policy 2 @E,212 with gk2 for its task module and fre for its robot module. for each iteration do
for each environment step
Embed task state es; < 9hh1 (8x)
Embed anchor state Cata + 9k1 (a
Calculate the relative representation ra. according to equation (3) action, ff1 (rr.)
8141 ~ -Prkk (81+1 action:) + DU ((s,, action, (St, actiona), ,8++1)
end for for each gradient step do update Jk1 and fri with SAC algorithm end for end for for each iteration do
for each environment step do
Embed task state es, f (8:2(8:)
Embed anchor state ea(i) (0)
Calculate the relative representation rx, according to equation (3)
action, (T.(
8t+11 PPr222 ( St, action;)
+ [(St, action, r (st, action:) St41)3
end for for each gradient step do
update 9k2 and fr2 with SAC algorithm
end for end for
Policy Stitching:
Initialize the task module with parameters from gei-
Initialize the robot module with parameters from fr2.
Construct a stitched policy & (sg) fr2 (ga1 (SE,T) , SE.R)
Test the Stitched Policy the Target Environment:
Roll out the stitched policy ΦB (sg). Calculate the success rate and touching rate.
14

---

Algorithm 2 Few-shot Transfer Learning with Policy Stitching
Train Plain Policies in Source Environments:
Train Plain policy Trik with SAC in the environment 111
Train Plain policy TT222 with SAC in the environment 2,2x2.
Collect Anchor States:
Roll out two trained policies in their own training environments gather set of task states during these interaction processes.
K-means center set <- means(()
Select anchor set A which are task states closest to the K-means center set C.
Train Modular Policies in Source Environments:
Initialize modular policy I PELI with J&i for its task module and frr for its robot module. Initialize modular policy 2 6,,2x2 with 9&2 for its task module and fr2 for its robot module.
for each iteration do for each environment step do Embed task state cs, f 9h1 Embed anchor state Ea()) + 9k1 (a
Calculate the relative representation Ts, according to equation (3) action; ~ (fr)
81+1 -PAIRI (8 | action;)
D DD ((8,, action,, r (St, action)) 1
end for for each gradient step do
update gk1 and fri with SAC algorithm end for end for for each iteration do
for each environment step do
Embed task state es. + 9k2(S:)
Embed anchor state ea()) + (0)
Calculate the relative representation Fs, according to equation (3)
action; fr2((r,))
8141 ~ (1+212 action:)
DU action,, r (St, actions) (staaction) end for for each gradient step do
;
update Ok2 and fr2 with SAC algorithm end for end for
Policy Stitching and Q-function Stitching:
Initialize the policy task module with parameters from Ik1-
Initialize the policy robot module with parameters from fr2.
Construct stitched policy & (sg) fr2 (gei (SE,T) S.,R).
Initialize the Q-function task module with parameters from Gki-
Initialize the Q-function robot module with parameters from 1r
Construct stitched O-function Qerzer (sg, = hr2 (4k1 (SE.T), SE.R
Few-Shot Transfer Learning: Fine-tune the stitched policy Φ3,20 (sg) and the stitched Q-function Q Evaa (Sg, with SAC in the target environment E,2t1.
15

---

stale)
state
state dimersion
state dimension
state dimensior
task
task
module
64 un ts
128 units
256 units
units
units
units
units
128 units
state state]
(jint stale]
state]
units
robot module
units
rabot module
units
rabot medule
units
units
units
256 units
oction moan, std
action moan, std
action mean, std
Action
Action
Action
(a) Small modular network
(b) Medium modular network
(c) Large modular network
Fig. 9: The detailed architectures of the modular networks with three different interface dimensions.
C Analysis of the Module Interface
We carry out additional analysis of the latent representations across different sizes of modular networks. We build three different sizes of modular networks, each interface dimension being 3D, 16D, and 128D shown in Fig.9. The transferable representation is added to these networks as shown in Fig We construct six networks (3 different sizes, with and without relative representation) to perform the reaching task as described in 1ig.1.
C.1 Visualization of the latent representations at the modules interface
For the small networks with 3D interfaces, we plot the 3D latent representations directly. For the medium and large networks with 16D and 128D interfaces, we use PCA [66] to reduce the dimension to 2D. Fig. 10 shows the visualization of the interface of the six networks trained with different random seeds. The isometric transformation relationship shown for the PS(Ablation) method across all sizes of modular networks, and with the help of transferable presentation, PS achieves near invariance. Similarly, Fig. 11 shows the interface of the networks trained with different robot types. The transferable representation achieves near invariance isometric transformations across all types of robots.
PCA is an information lossy compression process. The PCA method only guarantees identical output results when the input data sets are identical. When the input data sets are similar but not identical, the output results may vary considerably, In our experiments using PCA for visualization. we have observed that the PCA results of most interfaces with transferable representations are similar. However, in rare cases, have noticed significant differences in the PCA results. As shown in Fig. 12, the original 3D latent states have very similar distributions across the four different runs, but after the dimension reduction to 2D with PCA, the visualization results show isometric transformations. Moreover, in the case of small modular networks with 3D interfaces, achieving a high success rate of approximately 100% often requires considerable amount of training time. Occasionally, the network may converge local minimum with a success rate of around 90%. When it converges to local minimum. the latent representation its interface typically differs from those that converge to the global minimum. PCA only provides intuitive idea of the behavior at module interface, thus, accompany these visualizations with quantitative analysis.
16

---

Random Seeds (101-104)
Small network- PS(Ablation)
Medium network PS(Ablation)
Large network PS[Ablation]
Large network- PS
Fig. 10: Latent Space Visualization Train each policy network four times with four different random seeds (101-104). Without transfcrable representation, the latent representations at the interfaces have an approximate isometric transformation relationship. With relative representation, these latent representations are isometrically similar.
Different Robot Kinemattcss
Small network PS(Ablation)
Medium network PS(Ablation)
Large network PS(Ablation)
Small network- P5
Medium network
Large network PS
Fig. 11: Latent Space Visualization Train each policy network for the reaching task with three different types of robots shown Figure 3. With relative representation, the latent representations the module interfaces nearly the same across diffcrent environments. Without it, they have an approximately isometric transformation relationship.
C.2 Quantitative analysis of the latent representations at the modules interface
To measure the similarity between two different latent representations, we use cosine and L2 pairwise distances. We compute the pairwise distance between two latent task states derived from the same input state. By considering dataset of input states, we calculate the mean of the pairwise distances across all input states, obtaining the average pairwise distance between two modular networks.
Given input task state set SE.T, the average pairwise cosine distance and 1.2 distance are defined as
dons
Σ
(1 So (gt (s'e.)) o6 (s'2.))) (E.7|.
Bg.T
dr =
Σ
dra (g) (B'E,) 2 (KE,T) [S,.T
(5)
1=1
17

---

Small network PS
Small network PS
PCA: to
Fig. 12: Limitation of visualization with PCA. Raw latent distributions very similar each other in the 3D space, but after compression to 2D with PCA, the visualization results are quite different.
Cocine
Cosine small actworks rostive opreserration
with rolttve repredoraztion
with reative ispresotttion
without reaatie log ceprooott on
Iran3 Danz train
0,078 8.55
Can bann bonz Crams
2.075
u.coea 0J0004
D.45
0.38
0.38
1.48
D443
3.3
CORAL
1.9
2.89
2.4
0.063 0.0054 0.08L
0.48
018
0.05
1.1
0.99
13
DSin tnin vain mnh
bann mih 2 tran mah
Dain 7ah2 mn
13h
mAh1 25n tain triin
Small modular network
setworks
With retatve represtrtation
with restive impresentation
withuu enirtee incecectation
Alhoou plarive repioemnnt
bon' Crann bran
0.0035
0.0006 0.002
2.2
0.37
0.15
13
0
0.25
0.55
12
1.4
0.2011 0.0026
0.15
0.75
0,0027 0.002
8002
0.86
0:36
1 tran vanh mih
trant Irhh 2 trae 1 trann
Inaen man2
m
crain
Imh train tran train
(b) Mediumn modular network
with celatton large relnnets
siutance large rehurrio
Codlee stancce networbs without Ieprrsattation
citttrre the large retsentoo
withous metati repratemeen
baany trann Da91
0L506
0.C081 0.008
Done bon' tranz bant
0.50
1.4
0.67
5.006
0,0009
0.021
1.3
0:57
0:79
04
9.3
0.0051 10099
0.023
0.50
0.77
1.5
0:04
0.79
0.13
2:4
8.3
1.8
I Irin
bet tran bain
Inan train
tran Dern train4
(c) Large modular network
Fig. 13: Cosine and L2 distances of different networks with and without transferable representation for different random seeds.
where Sc b)
the cosine similarity and dr.(p..) q) = 1|p - g/l the L.2 distance.
|
Fig. shows the average pairwise distances of modular networks trained with four different random seeds. We calculate the distances for different sizes of networks shown Figure 9. We also calculate the mean and standard deviations of the data in Fig.9 and present them in Tab. 2. The results show that the transferable representation largely reduces the average pairwise distances of the latent spaces between different training runs.
18

---

We also train the modular networks in different environments and calculate the pairwise distances at the interfaces. Specifically, we train the policy networks on the reaching task with different robots shown in Figure 3. The average pairwise distances are shown in Figure 14 and we calculate the mean values and standard deviations in Tab.3. These quantitative results show that the relative representation makes the module interfaces much more similar each other when trained different environments.
cosine distance
1.2 distance
0.0363 0. 0.0319
0.289 + 0.141
PS(Ablation)
1.106 + 0.434
1.426 + 0.322
(a) Small modular network
cosine distance
L2 distance
PS
0.00231 + 0.00073
0.1633 + 0.0294.
PS(Ablation)
1.054 0.218
1.442 + 0.151
(b) Meidum modular network
cosine distance
L2 distance
PS
0.013 1 0.007
1.051 +: 0.368
PS(Ablation)
0.753 0.081
8.386 + 0.687
(c) Large modular network
Tab. 2: Mean and standard deviation values of the average pairwise distances between trainings with four different random seeds (101-104)
19

---

with nelative ruprosontation
with velativv suprosentttin
arthout relatov represartation
authout relaziv reprosenat on
0039
0:022
0.53
0:99
6.9)
RubotZ
Tobor
0.52
0.E5
0.99
DV232
0.15
0.53
D.91
INDOLL
P00002
100013
Roboc
Robo22
Po0oc3
NoboLL
Pobox
Robor3
Robok
Robott
Small modular network
networks
aetworks
with rslabie represtrtation
restive rsprosoitatio
withoutrrbbtoo oppecontaaion
without relgive ropregorzzt
D. COOS
1000
tab-0
0.28
0.14
0.26
D.04
1.A
box03
Robott
Robot2
R00o13
Roboot
Robor2
Robor3
Robott
Fobot2
Robot3
Roboot
Robott
Robol
(b) Medium modular network
distance
Costne distance the large networks
1a dleettncce of the large networs
relarivee represset bariarr
with representar
20
DOGS
Nutor2
0.5
Robod
36
hozer2
0.00
0.18
0.2
Bober3
022
Rebstt
Rebst2
Rebot3
Rebett
Rebor2
Rebar3
Robett
Foberk
Robet3
Robett
Robett
Rabeet
(c) Large modular network
Fig. 14: Cosine and L2 distances of different networks with and without transferable representation for different robot sctup.
cosine distance
L.2 distance
PS PS(Ablation)
0.0082 + 0. 0.0035
0.149 + 0.032
0.633 0.3
1.066 + 0.165
(a) Small modular network
cosine distance
L.2 distance
0. 0.0055 0. 0.0029
0.240 + 0.075
PS(Ablation)
0.865 1 0.367
1.275 1 0.311
(b) Medium modular network
cosine distance
distance
PS PS(Ablation)
0.0071 1 0. 0.0006
0.743 1 0.152
0.760 + 0.094
8.822 + 0.448
(c) Large modular
Tab. Mean and standard deviation values of the average pairwise distances between trainings with three different types of robot kinematics
20


Reviews:
[{"review_id": "nLMd5dEx-hc", "reviewer": "Reviewer_ZQPk", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a method to learn task transferable policies. The paper considers training policies for different robot arms on different tasks. Policies take task representation and robot representation as input and output actions to solve the task. A relative representation is used; after building a K-means best anchor set, it projects task information into a certain transferable representation, enabling the transferability of the robot policy. Experiments demonstrate good few-shot transfer in the simulation and also can learn transferable policy in the real world.", "summary_of_recommendation": "The paper studies an interesting problem. However, the writing is hard to follow. It requires a major revision to be accepted.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Poor: Difficult to follow and understand.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "I think the authors should better present and motivate their approaches. Providng clear definition and pseudocode will be of great help.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Advantage: The paper studies an interesting problem of learning transferable policy, especially for novel combinations of robots and tasks, through modularized design. The problem is valuable, and the modularized idea is promising. It is impressive that the learned policy can be transferred to real-world scenarios. \n\nHowever, there is still significant room for improvement. The paper’s presentation is rather vague and unclear. For example, I found the terminology “stitching” is not strictly defined in Section 3. The training set in line 141 is not defined. Where does it come from? The method section also does not mention how the policy uses the similarity scores. I feel the description of the method is incomplete.\n\nThe lack of clarity also makes me doubt the motivation for the use of relative representation. Actually, I do not get why changing the task representation only will enable transferring policy as it does not really affect the training of the policy. Moreover, I don’t see why relative representation of tasks will be better than PCA or domain adaptation methods like information bottleneck or MMD, let alone other representation learning methods based on generative models. The authors may need to motivate the relative representation better.\n\nLast but not least, the zero-shot success rate is rather low. The policy seems to mostly learn to touch the object instead of solving tasks. This also makes me question the effectiveness of the approach.", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}, {"review_id": "e2QzCbd2vI", "reviewer": "Reviewer_BozE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes an approach for policy stitching, by combining different parts of a policy representing robot state information, and task information for a novel test-time combination. The main insight in the paper is to have a modular policy design and align the latent representations between the modular components. Experiments on different simulated and real robot manipulation tasks show that the proposed policy stitching approach can solve novel robot-task combinations zero-shot, and achieve better performance with few-shot fine-tuning. ", "summary_of_recommendation": "The paper proposes an interesting formulation of policy stitching, building upon prior works in this domain. Although I have concerns about scaling the approach to high dimensional observations and more complex tasks (since object state information is required), but the current instantiation is still useful for several tasks like pushing, pick and place etc. and so I am recommending weak object.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- Is it possible to scale the current appraoch to tasks beyond pushing in the real world (e.g. articulated object manipulation) where it is difficult to quantify object state information?\n\n- Another weakness is that the representations for robot state and task are not \"really\" separated as it is the same neural network encoding both the information - this is true for the policy network as well as for the q function network. With high dimensional observations, and complex tasks, the current approach might fail because of this, and more careful modular design might be necessary. Is it possible to better ensure this modularity or it is actually not necessary to do this even for complex tasks and high-dim observations (e.g. images)?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths\n\n- The key insights of the paper regarding modular policy design although is not particularly novel, the instantiation with aligning latent representations is simple, insightful, and practical.\n\n- The intuition regarding latent space alignment is very clearly described in the text and in figure 2, and the paper overall is easy to follow.\n\n- The experiments with different robot morphologies, and different task settings are carefully chosen to answer the research questions. In addition, sim2real results demonstrate feasibility of the approach on real robot pushing tasks. \n\nWeaknesses\n\n- The main weakness of the paper is that low-dimensional state information about objects (like position, orientation etc.) are necessary for the current instantiation of the framework, so application to tasks beyond pushing in the real world (e.g. articulated object manipulation) are limited, as it is difficult to quantify object state information.\n\n- Another weakness is that the representations for robot state and task are not \"really\" separated as it is the same neural network encoding both the information - this is true for the policy network as well as for the q function network. With high dimensional observations, and complex tasks, the current approach might fail because of this, and more careful modular design might be necessary.\n\n  ", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "_d2eLrKx4RX", "reviewer": "Reviewer_DSoM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents Policy Stitching, a method that aims to facilitate transfer to new robots and tasks by learning modular policies. This work builds on the prior work of learning modular policies proposed Devin et al. The paper highlights a major limitation of existing work: the internal task representation is not guaranteed to be invariant to factors such as training seeds. Consequently the paper stresses the importance of learning invariant task representations for facilitating transfer to new settings. To enable this invariance, this paper utilizes the recent work of Relative Representations, by representing the task state as a sequence of similarity scores to anchor states. In experimental evaluations the authors show that the proposed enables better zero-shot and few-shot transfer to a set of tasks with different task and robot designs. Additional quantitative visualizations confirm that the method learns invariant task representations.", "summary_of_recommendation": "The paper is well motivated with a good set of experiments. Still experiments have room for growth and writing has room for improvement (see list of weaknesses).\n\n**update after rebuttal** I have carefully reviewed the reviews and the author responses to all reviews. I am still recommending weak accept. For a stronger paper I recommend that the authors study transfer to different morphologies of robots.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "See the list of weaknesses in the section above", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is overall well-motivated with a broad set of experiments supporting the claims. There are areas for improvement regarding clarity and presentation quality. Please see the specific breakdown of points as follows:\n\nStrengths:\n* Studies an important problem (transfer) and introduction motivates the problem well\n* Experiment results show meaningful improvements in performance compared to baselines and self-ablations\n* Great to see real robot results supporting the simulation results!\n* Visualizations of the task representations support the claim that this method learns invariant representations\n\nWeaknesses:\n* When presenting the modular policy design (102-119) the authors should acknowledge the prior work of Devin et al., as this design was proposed before in their work. While Devin et al. is acknowledged in the related works and experiments section, their work should be directly acknowledged in this section as well.\n* The work of Relative Representations is a significant component in this work, however the authors dedicate relative little writing to explain this work. It would be great to expand on why Relative Representations induces invariant representations, as this was not clear to me immediately.\n* Experiments can be more impressive if demonstrating different robots with different morphologies rather than a Panda arm with locked joints\n* This is already mentioned in limitations section, but experiments in image domains can help to further strengthen the experiments.\n* Further clarification needed on the Plain baseline. What do you mean on line 192 by “perform the same reassembling operation as the modular networks”?", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "6IEE_5FSH_1", "reviewer": "Reviewer_2mFg", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work  introduces a model-free learning framework called `Policy Stitching (PS)`\nthat facilitates the transfer of knowledge among different combinations of robots and \ntasks through a modular policy design and transferable representation learning. \nPS distinctly separates robot-specific aspects (like kinematics and dynamics) and \ntask-specific aspects (like object states) within the policy design. \nThis separation enables the recombination of these modules for new robot-task pairings. \nFor example a robot arm's module trained to pick up a cube can be combined with a task module trained for a \ndifferent robot to pick up a stick, thus enabling the original robot to pick up a stick by \"stitching\" these policies together.\n", "summary_of_recommendation": "This paper proposes to learn decomposable dynamics vs tasks representations to solve robotics manipulation tasks. The methodology presented here is novel and practical. Overall the presentation is decent. However I would like to see more challenging robotics tasks, because the capability of this method is only demonstrated on simple pushing tasks.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- I understand that for the experiments, the tasks are ran under sparse reward setup and with hindsight experience replay. But of course this might limit the choice of tasks that can be ran (object picking are still fairly simplistic experiments). I was wondering what prevents this method to work for more challenging manipulation tasks, perhaps under dense reward setting? I'm wondering in terms of the tasks to solve, how far it can be pushed and whether the authors tried other tasks that failed.\n- In terms of terminology, usually in the field, we refer to dynamics vs task related representations. There are many frameworks trying to attempt to learn a decomposed representation for the reward function (assuming a dense reward that already captures environment dynamics and the goals of the task itself) . So I find the terminology used throughout this paper as `robot-specific information` slightly divergent. Even though its accurate, you are explicitly using robot related information such as joints, velocity etc but all these capture the robot dynamics. So I still prefer to stick with the terminology of decomposing dynamics and task (i.e. reward related component).\n- How tied is this decomposition to the reward itself? Because as far as I understood, this method tries to capture robot-related and task/goal related observations very separately without considering the reward itself (as its sparse reward in this case, the reward provides no information until task completion). ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a modular policy design in Policy Stitching, where the policy is divided into two separate modules - \none concentrating on robot-specific information and the other on task-specific information. \nThis design facilitates easy reuse and combination of modules. The environment is defined with a robot `r` and task `k` \nand is formulated as a Markov Decision Process with state `s_E` and action `a_E`. The learning policy `π` is \nparameterized by a function $\\phi E_{r,k}$. The Soft Actor-Critic (SAC) algorithm is used as model-free RL. \nThe state `s_E`, policy function $\\phi E_{r,k}$., and Q-function are decomposed into robot-specific and task-specific modules. \nThe robot-specific state `(s_E,R)` includes information like joint angles (i.e. observations that relate to the robotics dynamics), whereas the task-specific state `(s_E,T)` includes information related to the task, such as position, orientation, linear and angular velocity, and goal position of the object. \n\nEquations (1) and 2  show how the `policy function` and `Q-function` can be broken down into task-specific and robot-specific components, \nfacilitating modular design.  The authors acknowledge that this approach may face issues when combining modules, as the output representation of one module may not align with the input representation of another, particularly when they are trained on different tasks or robot configurations. This issue is exemplified through latent representations from reinforcement learning policies that, even under identical conditions,  do not align due to differences in random seeds during training, and interestingly, exhibit a phenomenon called isometric transformations.\n\nThis work further proposes adapting latent representation alignment techniques from supervised learning to reinforcement \nlearning to address the alignment issue in robot transfer learning through Policy Stitching. \nThe core concept is to enforce transformation invariances by projecting the intermediate representations \ninto a common latent coordinate system.  Since the method used here, `Relative Representations` relies on ground-truth labels in supervised datasets to establish anchor points, the authors suggest utilizing the k-means clustering algorithm  to select anchor points from the training set, which serves as the basis for the latent coordinate system. \nThese anchor points should be as dissimilar as possible and are selected based on their proximity to the centroids of the clusters formed by k-means.\n\n**Experiments**\n\nPS is evaluated in both zero-shot and few-shot transfer setups. The robotics simulation tasks include object picking tasks such as `pick up a cube` and `pick up a stick`. These tasks are ran under sparse reward setup using hindsight experience replay.  Real world experiment are `Task1` : pushing a cube to a goal position, and a `Task2` as pushing a cylinder.\n\nFigure 7 is also interesting where the authors present a latent space analysis where they train robots with three different kinematics for a \nreaching task and visualize their latent space, which is color-coded based on four groups of task states. These representations are 2D principal components of the latent representations, providing insights into the structure of the  latent space and `(B)` displays the average pairwise cosine distances and L2 distances between each pair of the stitched latent states. This visualization and analysis characterize effectiveness of the latent representations.\n\n**Baselines** \n\n`PS` is compared with `Devin et al` approach which also uses a modular policy but  employs small bottlenecks and dropout for regularization to tackle module misalignment. Additionally, the authors conduct an ablation study using their modular policy design without \nlatent representation. They also introduce a baseline model, `Plain`, which is a simple Multi-Layer Perceptron (MLP) without a modular structure.  \n\nAdditionally,  the authors conduct an ablation study using their modular policy design without latent representation alignment.\n\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a novel way to learn modular policies. The idea is to put together information from different sources such as kinematics and dynamics, and task-specific things such as object states. In experiments, the approach allows transfer to different tasks and robot designs. The approach is evaluated both in simulation and on a real robot. The authors should consider expanding the experimental evaluation on a wider variety of robotic tasks.
Decision: Accept (Poster)