Perception Stitching: Zero-Shot Perception Encoder Transfer for Visuomotor Robot Policies
Authors: Pingcheng Jian
Easop Lee
Zachary I. Bell
Michael M. Zavlanos
Boyuan Chen
Venue: Accepted by TMLR ()

Abstract: Vision-based imitation learning has shown promising capabilities of endowing robots with various motion skills given visual observation. However, current visuomotor policies fail to adapt to drastic changes in their visual observations. We present Perception Stitching that enables strong zero-shot adaptation to large visual changes by directly stitching novel combinations of visual encoders. Our key idea is to enforce modularity of visual encoders by aligning the latent visual features among different visuomotor policies. Our method disentangles the perceptual knowledge with the downstream motion skills and allows the reuse of the visual encoders by directly stitching them to a policy network trained with partially different visual conditions. We evaluate our method in various simulated and real-world manipulation tasks. While baseline methods failed at all attempts, our method could achieve zero-shot success in real-world visuomotor tasks. Our quantitative and qualitative analysis of the learned features of the policy network provides more insights into the high performance of our proposed method.

Published in Transactions on Machine Learning Research (11/2024)
Perception Stitching: Zero-Shot Perception Encoder Transfer for Visuomotor Robot Policies
Pingcheng Jian'
Easop Lee'
Zachary Bell? Michael M. Zavlanos'
Boyuan Chen
Duke University Air Force Rescarch Laboratory
Reviewed OpenReview: httpp://opaaaaeeiee..er////ouat/
Abstract
Vision-based imitation learning has shown promising capabilities of endowing robots with various motion skills given visual observation. However, current visuomotor policies fail lo adapt to drastic changes in their visual observations. We present Perception Stitching that enables strong zero-shot adaptation to large visual changes by directly stitching novel combinations visual encoders. Our key idea is to enforce modularity visual encoders by aligning the latent visual features among different visuomotor policies. Our method disentangles the perceptual knowledge with the downstream motion skills and allows the reuse of the visual encoders by directly stitching them a policy network trained with partially different visual conditions. We evaluate our method in various simulated and real- world manipulation tasks. While baseline methods failed at all attempis, our method could achieve zero-shot success in real-world visuomotor tasks. Our quantitative and qualitative analysis of the learned features of the policy network provides more insights into the high performance of our proposed method.
Policy A:
Policy B:
Policy C:
Stitch
Action Decoder
Action Decoder
Action Decoder
Figure 1: Perception Stitching: "Policy A was trained with an in-hand camera and front-view camera. "Policy B" was trained with a close-up camera and a side-view camera. Perception Stitching enables zero-shot stitching of the original Policy A and B by reusing their relevant components for each sensing configuration to form a "Policy C "Policy C" can maintain strong wero-shot transfer performance with an in-hand camera and side-view camera.
1 Introduction
Despite recent advances in vision-based imitation learning for acquiring diverse motor skills (Chi et al. 2023; Fu et al.) 2024), significant challenge in deploying visuomotor policies in real-world settings is ensuring the perceptual configurations are identical during training and policy execution. With the growth of hybrid robol datasets where robots are trained lo perform similar lasks across the world, remains difficult. to share their learned experiences, even under the same task but with different visual observations. Such a challenge often stems from the unique configurations and perspectives cach camera sctup brings, which, while enriching the dataset, complicates the sharing of learned experiences across different systems. Different institutions worldwide usually place different sensors at different perspectives when collecting their datasets. In these

---

Published in Transactions on Machine Learning Research (11/2024)
cases, previously collected datasets trained policies are not interchangeable, and new training data must be collected in each enviromment.
Instead, what if we could directly stitch the perception encoder trained in one visuomotor policy to the rest of the components of another visuomotor policy? This approach would enable zero-shot transfer of the trained visuomotor policies to a novel combination of perceptual configurations. To address the challenges associated with zero-shot transfer across different settings, previous efforts have aimed to learn an invariant feature space universal representation for quick adaptation to new environments, with approaches ranging from extensive pre-training with video data (Nair et al.,
2022,, employing contrastive learning between two 2017], and concentrating on low-dimensional data over vision-based observations (Jian et al. 2023). In particular, recent studies have proposed to achieve fast policy transfer (Jian et al. 2023) by aligning the latent representations of different perception encoders with relative representation (Moschella et al. ) However, it remains unclear how to scale similar approaches from few-shot transfer to zero-shot transfer and high-dimensional observations.
policies to find a common feature space (Gupta et al.
We present Perception Stitching (PeS) (Fig. I to enable zero-shot perception encoder transfer for visuomotor robot policies. PeS advances the previous studies through a novel training scheme under various camera configurations and effectively processing nigh-dimensional image data. Our approach can train modular perception encoders for specific visual configurations (e.g. camera parameters and positions) and reuse the trained perception encoders in a novel environment in a plug-and-go manner. In the simulation, we evaluate PeS in five different robotic manipulation tasks, each with seven unique visual configurations. It constantly shows significant performance improvement compared to four baseline methods and two ablation studies. We also evaluate PeS in four real-world manipulation tasks. While the baseline struggles to get any successful attempts, PeS achieves pronounced success rates, indicating that directly stitching modular perception encoders in the real world has been turned from impossible to possible by this work. Additionally, we contribute quantitative and qualitative analysis to provide more insights on the high performance of our proposed method.
2 Related Work
Learning Visuomotor Policy for Robotic Manipulation A wide range of previous work has focused on
learning visuomotor policy for robotic manipulation (Levine et al.
2016
Finn et al. 2016 2017b Kalashnikov
et al.
2018 Srinivas et al.) 2018 Ebert et al. 2018 Zhu et al.
2018
Rafailov et al. 2021 Jain et al.) 2019:
"lämäläinen et al. 2019) F Florence et al. 2022
Brohan et al.
2022
2023 Padalkar et al. 2023 Sermanet
ct al.
2018). Certain works have investigated the impact of camera placements Zakky ct al. 2020 Hsu
et 2022) design of the hardware and software (Zhao el al. 2023 Fu et al. 2024 Kim el al.
al.
2023)
novel
network architectures and optimization techniques DDsaar & Gupta, 2021 Kim et al.) 2021 Zhu et al.
2023
Abolghasemi et al. 2019; Ramachandruni et al., 2020 Brohan et al.,
2023
2022 Padalkar ct al. 2023
et al. 2023 Li et al.] 2023). In this work, we show that perception encoders trained with Behavior Cloning
(BC)
Pomerleau 1988 often lack the flexibility for module reuse. Our Perception Stitching (PeS) method
enables perceptual knowledge reuse and facilitates zero-shot transfer between diverse visual configurations, advancing existing rescarch on fast adaptable visuomotor policy design.
Robot Transfer Learning Transfer learning has long been considered primary challenge in robotics
(Tan et al.] 2018 TTylor & Stone 2009). In the context of reinforcement learning, many previous work
transfer different components such as policies (Devin et al.
2017: Konidaris & Barto 2007 Fernández &
Veloso 2006), parameters (Finn et al. 2017a Killian et al.
2017: Doshi- & Konidaris 2016), features
[Barrtto et al. 2017 Gupta et al. 2017) experience samples Lazaric et al. 2008), value functions (Liu al.,
2021 Zhang & Zavlanos 2020 Tirizooi et al. 2018), and reward functions (Konidaris & Barto
2006). In
imitation learning, additional studies have made progress via domain adaptation (Kim et al. 2020 Yu et al,
2018), querying unlabeled datasets (Du et al. 2023). abstracting and transferring concepts
EizzaroGGredila,
et al.
2019
Shao et al., 2021],, or conditioning on other information such as language instructions (Seppuutis
et al.
2020 Lynch & Sermanet] 2020) Jang et. al. 2022) and goal images (Pathak et al. 2018 Our work
focuses on neural network policy sub-module reuse through direct stitching to achieve zero-shot transfer.
Sim-to-real transfer is another important topic within transfer learning (Sadeghi al.
2018
James et al.
2019 Zhang et al. 2019 Tobin et al. 2018 Mehta et al. 2020 James et. al. 2017
Nguyen et al.
2018 Rusu
2

---

Published in Transactions on Machine Learning Research (11/2024)
V.E. A
V.E. B
V.E. C
V.E. D
Obsorvation
Anchors
Obsonation
Anchors
prioception
jorioception
R.R.
R.R.
R.R.
R.R.
V.E.
V.E. D
A.D. 1
A.D. 2
Stitch
Isametric
Action
Action
Transformetion
Embedded Anchors
Embeddeed Anchors
V.E. A
V.E.
R.R.
R.R.
prioception
R.R.
R.R.
Approkimate
Invariance
V.E.: Visull Encoder D. Action Decoder
A.D.
R.R.; Relative Represeniation
Action
Figure 2: Method Overview. Two visual encoders process the RGB images from two cameras separately, and the latent representations are concatenated with the proprioception of the robot end effector state. The original latent representations of the images are observed to have an approximate isometric transformation relationship. Relative representations with disentan- glement regularization can maintain an approximate invariance and, therefore, help achieve high zero-shot transfer performance.
ct al.
2017). To merge the sim2real gap, previous methods include domain adaptation (Zhang et al
2019
Tobin et al. 2018 Mehta et al. 2020 James et al. 2017], adopting progressive network (Rusu el al. 2017
2016), fine-tuning the visual layers (Sadeghi et al. 2018 , training a generator that translates
images to canonical simulation images (James et al. 2019). or training a CycleGAN (Zhu et al. 2017 to
synthesize real images from simulation images(Nguyen el al. 2018). Most recent research has proposed Lo learn invariant or universal feature representations for transfer learning (Gupta et al. 2017 Jian et al. 2023
Nair al.) 2022). Our work fits in this category. Unlike previous studies, our method does not require a large amount of online data for pre-training (Nair et al. 2022 training two policies SaauultanooullyGuuaaa et al. 2017). Compared with similar previous work by Jain et al. (2019), our method is not limited to low dimensional observations, does not require few-shot fine-tuning, and can solve much more difficult tasks.
Compositional Robot Learning Compositional robot learning reuses the learned knowledge saved in
some portion of the policy network new tasks instead of training from scratch (Pleiffer et al.) 2023 Devin has achieved promising results in multi-task learning Yang
2020 Alet et al., 2018: Chen al.
2020).
ee al. 2020; Hussing et al.
2023 Chen al. 2022 Kwiatkowski et. al., 2022: Hu al.) 2022), transfer
learning (Devin et al. 2017 Jian et al.
2023
Gupta et al. 2017), and lifelong learning (Mendez et al. 2022:
Mendez & Eaton 2022: Méndez 2022
Some previous works train graph of network modules and generate
different paths to connect specilic modules for different tasks (Yang et al. 2020 Mendez et al.) 2022) but
the network modules cannot: be reused out of the graph it embeds in. Devin et al. (017)) reuses the network module without aligning latent space representations and thus fails to achieve satisfying performance in more complex tasks. JJin et al. (2023) aligns the latent spaces by selecting anchor states and calculating relative representations (Moschella et al., 2022) al the module interface, which no longer requires simultaneously training multiple policies. However, the results fine-tuning in simple tasks such as pushing, In this work, extend its success to high-dimennional image observations and more difficult tasks such as cube stacking and door opening with zero-shot transfer.
limited to low-dimensional observations and few-shot
3 Perception Stitching: Learning Reusable Perception Network Module
Perception Stitching (PeS) (Fig. 2 is a compositional vision-based robot learning framework that allows zero-shot transfer of perceptual knowledge between different camera configurations. It designed to be compatible with conventional behavior cloning algorithms (Bain & Sammut 1995; Ross al. 2011 Torabi
et al. 2018 Chi et al. 2023), various visual encoder structures such CNN (LeCun et al. 1989 and ResNet
3

---

Published in Transactions on Machine Learning Research (11/2024)
(ee et al, 2016), and action decoder structures such as MLP (Hinton et al. 2012 Mandlekar et al. 2021 and LSTM (Hochreiter& & Schmidhuber 1997). In this section, we will demonstrate the two main components of PeS: the modular visuomotor policy design and the latent space alignment for transferable representation.
3.1 Modular Visuomotor Policy Design
Consider an enviromment. E with observation oF. We denole the observations from two camera views as of and of, and the robot proprioception of the end effector position, orientation, and gripper open width as of For an MLP-based policy, we denote it as Tg(a | If parameterized by a function $g(oF). For an RNN- based policy, we denote it as Te(a 11f11 <f | oF, .hffcf) parameterized by a function @ N.. cF., where fff is the hidden state of the RNN network at time step and of is the cell state. The visuomotor policy can be decomposed into the visual encoder of for camera 1, the visual encoder of for camera 2, and the action decoder J We can represent the MI.P-based policy as Eq. 1 and the RNN-based policy as Eq. 2
= o of 0)) = 58 (of((o 3 (005).
(1)
0n(oR,np F. Mr ,,, - - 02 of p , c (o (o MF,
(2)
Without loss of generality, we demonstrate the perception stitching process with the MLP-based policy. With two visuomotor policies (g (oft). (o²"), ), and f {Ba (of*). ). L (o (, in two environments Ej and Ez with different cameras, we define perception stitching as constructing another visuomotor policy network f (of" (of²), Ea (oza), " ofo)) by initializing the visual encoder 1 with parameters from 9f",, visual encoder 2 with parameters from 92 E2 y and action decoder with parameters from f Then this stitched policy is zero-shot transferred to the new environment Es with the same perception configuration 1 as that of Fi and perception configuration as that of Eg. For example, as shown Fig. 2 (left. half), we train policy Φg, in En with an in-hand camera and a side-view camera, and policy Φ in Es with a side-view camera and a front view camera. Now, if we need a policy for Ez with an in-hand camera and a front-view Ea to the of and f to form stitched policy that directly works in camera, stitch the visual encoder 92
Eg. Note that though we formalize PeS set up with dual-camera settings, PeS is not constrained with only two cameras, as demonstrated in our experiments in the Section 4.2 In addition, although we choose the action decoder 1 for the experiments in section [ the choice of which action decoder to be used doesn't 't affect the performance of PeS, and the experiment results of comparing the influence of the two action decoders are presented in Appendix
3.2 Latent Space Alignment for Transferable Representation
Relative Representation Simply stitching one portion of a neural network to another neural network usually cannot yield optimal performance in the target environment due to the misalignment of latent space (Jian et al. 2023: Devin et al. 2017). Previous works have observed an approximate isometric transformation relationship between the latent representations trained with different random seeds (Moschella et al., 2022) and different robot kinematics (Jian et al., 2023). These isometric transformations include rotation, reflecting, rescaling, and translation. In this work, we obscrve a similar phenomenon in the latent spaces of the visual encoders. Hence, we calculate a relative representation at the latent. space (Jian el al. 2023 Moschella et al. 2022) to align the latent features from different policy modules.
As shown in Fig. 2 we first collect set A of anchor images a from the dataset ID for the behavior cloming: A == (a C DD. By applying the visual encoder to both the input image s(() (s(()) c.) and the anchor image a we obtain their embedded forms eg( = g()(f) and e.o: = g(a
We want to project the embedded input images to coordinate system consisting of the embedded anchor images, and if this coordinate system is invariant to isometric transformations, we can alleviate the latent space misalignment issue. Therefore, we calculate similarity score r = sim (e,()) eqi)) between an embed- ded task state and an embedded anchor state where sim : Rd x Rd + R. Then the relative representation (Jian et al. 2023 Moschella et al., 2022) of the input image s(() with respect to the anchor set A is given by:
r. (sim(ex(a, Caw)) sim((, eate), ... sim(e simle)c)ea
(3)
4

---

Published in Transactions on Machine Learning Research (11/2024)
We choose the cosine similarity due to its invariance to reflection, rotation, and ro-scaling. Additionally, we add normalization layer before calculating cosine similarity to mitigate the translation transformation. Therefore, this relative representation is invariant to the isometric transformation, leading to better latent space alignment.
Compared with previous works (Jian et al. 2023 Moschella et al. 2022). we develop two novel techniques for the latent space alignment of visuomotor robot policies: (I) a novel anchors selection method designed for imitation learning and (2) the use of a disentanglement regularization for better latent space alignment.
Anchors Selection. When PeS is applied to visuomotor policies with two visual encoders, these two visual encoders encode images observed in two different cameras, and our proposed anchor selection method utilizes this correspondence between these two visual configurations. As shown in Fig. 3 after collecting a dalaset D1 in an environment E1. we perform k-means (Hartigan & Wong 1979) on Di and select the images closest to the cluster centers as the anchors in the anchor set A1. We then replay the trajectorics of the datasct DD in another environment E2, which requires the robot to accomplish the same task in Ei, but the cameras are different. There is no additional policy training required due to the replay. The dataset DD2 collected by replaying these trajectories in Ea has states corresponding to the states in D3. We then use the indices of anchors in A1 to select the anchor set A2.
Correspondngg
Disentanglement Regularization. In addition to the negative log-likelihood loss (LCc) used for the standard behavior cloning algorithm, we also apply disentanglement regularization [Wang et al. 2022 Ldisent to further refine the latent space alignment. We first calculate the covariance of the hthh and 1th dimension of the batch of embedded representations with
Indices
Side View Anchars
Front View Anchors
K- Meens
Side View Dataset:
Front View
Dataset:
Replay
1 Cov (zu, 21) N 1=1 (Zsk
Σ
žx) (zit Z),
Trajectories
Figure 3: Anchors Selection. Select the anchor images in one dataset with the k-means algorithm (Hartigan & Wong (1779) Replay the trajectories of the first dataset to collect another dataset with a different camera. We select the images with the corrosponding indices of the anchors in the first datasct as the anchors in the second dataset.
(4)
where z is a batch of latent representations embed- ded by the ResNet before going through the relative representation calculation process. 2ik and zu are the values of the 4th and 1th dimension of the zth embedded data point. Eg is the mean of the 4tth dimension across all N data points in the batch, calculated as Σ ===0 Σi= 2sk- ž is the mean of the 7th dimension, similarly calculated. Then the disentanglement loss is calculated by:
I.diaent Z(Z Σ Z
E |cov (Ek: 21)
where Z is the dimensionality of the latent space. Overall, this disentanglement loss calculates the covariance of a latent representation feature 2 with all other features 2 (1 + k), sums up these absolute values, normalizes it with Z 1, and calculates the mean over all the features 2 (k = 1,2...... Z). By encouraging different features at the latent space to be independent with each other, we encourage them to capture different underlying factors hidden in the observation (e.g. object color, position etc.). The disentangled representation has been used in many applications in supervised learning (Tran et al. 2017 Kim & Mnih
2018 Chen et al. 2018 Higgins et al. 2017 Quessard et al.) 2020 Higgins et al. 2018 Zbontar et al.] 2021 Ermolov et al. 2021 Bardes et al. 2021 and we empirically find it significantly improves the performance of PeS in difficult tasks.
The final PeS loss function is
L Lnc + XLdisent:
(6)
where we choose the weight À = 0.002. For an ablation study, we also experiment without the disentanglement loss
T.peS(w/o dieent. loss) L.BC-
(7)
5

---

Published in Transactions on Machine Learning Research (11/2024)
We also experiment with replacing the disentanglement loss with L1 and L2 normalizations of the latent representations
L PeS(( 11822 loss) LBC A1 Σ ||
||2/||2,
where the weights A = 0.001, A2 = 0.001, 2; is the 7th latent representation, and is the batch size. These L1 and L2 regularizations have been used in previous work (Nair al. 2022) avoid the state-distribution shift failure in imitation learning (Ross al. 2011) by limiting the latent space dimension. We find it improves the zero-shot transfer performance in some cases but generally doesn't perform well the disenlanglement loss.
3.3 Implementation Details
Each input image is (84, 84,3) RGB image. The visual encoder consists of a ResNet-18 net (He ct al. 2016), followed by a spatial-softmax layer (Finn et al. 2016; Mandlekar et al., 2021], and then 256-dimensional last layer at the module interface. We use 256 anchors to match the dimension of the latent representation. The dight-dimensional proprioception consists of the end effector position (3D), end effector quaternion (4D), and the gripper open width (1D). It embedded by 64-dimensiioall linear layer and then concatenated with two latent representations of the two images. We the RNN-based action decoder for the Can and Door Open lask and the MLP-based action decoder for other tasks. We list all the parameters of the neural network in the Appendix A with our code base. The actions output by Gaussian Mixture Model in the last layer of the policy network where the cight-dimensional action vector is the delta value of the 8D proprioception. Before the perception stitching and zero-shol transfer, all policies are trained lo 100% success rates with BC (Pomerleau] 1988). The dataset of each task contains 200 trajectories of the expert demonstrations. The pseudo code of PeS presented Appendix B B
4 Experiments
Our experiments aim to (1) evaluate the effectiveness of PeS enhancing zero-shot policy transfer the stitched policies and (2) understand the mechanisms behind the performance advantage of PeS. Our experiments consist of five parts. First, we evaluate the zero-shot transfer performance of PeS for train- ing double-camera-vieww visuomoloor policies in five simulated manipulation lasks with seven variations of camera configurations. Next, we apply PeS to single-camera-view and tripl-camera-viiew policies to test generalizability to arbitrary camera number settings. We then assess the performance of PeS four real-world manipulation tasks with three different camera configurations. In addition, we analyze the latent representations of the visual encoders both quantitatively and qualitatively to understand the effectiveness of PeS in latent space alignment. Lastly, we adopt Gradient- Class Activation Mapping (Grad-CAM) (Selvaraju et al. 2017 to visualize the regions that the visuomotor policies focus on, offering an inluitive insight into the success of PeS.
(b)
Push
Lift
Masied
Zeom
Buued
Indtand View
Clseedio View
Door Open
Fisheye
Lightia
Position
Side
Figure 4: Simulation Experiment Setup, (a) Five simulation tasks from Robomimic (Mandlekar et al. 2021 benchmark.
(b) Seven camera configuration variations include. (c) Four camera mounting positions.
4.1 Simulation Experiments with Double Camera Views
Evaluation Tasks We evaluate PeS in five manipulation tasks from the Robomimic (Mandlekar et al.
2021) benchmark (Fig. [a)): (1) Push: robot starts from a random initial position in the air and pushes

---

Published in Transactions on Machine Learning Research (11/2024)
the cube forward for certain distance (30 cm in simulation, 20 cm in real world). (2) Lift: robot lifts up cube randomly placed on the table. (3) Can task: robot picks up a can on its right and places it into the bins on its left. (4) Stack: a robot picks up the red cube on the table and stacks it on the green cube. (5) Door Open: a robot grabs the handle and pulls the door open.
Camera Configurations For each task, we test PeS with seven camera configurations in Fig. Tb). (1)
Masked: A black square mask is put on the upper-left corner of the image to mimic partially occluded camera lens. Although the mask makes little difference for humans, our experiments suggest strong negative effect on current visuomotor policies. (2) Zoom in: zoom in the camera to enlarge the object in a smaller field of view. 3) Blurred: Gaussian blur effect is added to the image. (4) Gaussian Noisc: Gaussian
noise is added to the image to simulate low-light condition image capture. (5) Fisheye: a fisheye effect is
simulated. (6) Lighting: the image is darker or brighter than the normal images, mimicking the overexposed or underexposod situation. (7) Position: the camera. is put in one out of four possible positions as in Fig. c).
The first six configurations used in-hand and close-up cameras. For example, in the we first train policy with a normal in- camera and blurry close-up camera, and policy 2 with blurry in-hand camera and normal close-up camera. Then stitch the close-up view visual encoder of the policy 2 to the in- hand view visual encoder and action decoder of the policy 1 and test this stitched policy in the Push task with a normal in-hand camera and a normal close-up camera without any fine-tuning. The last configuration is designed to test zero-shot transfer ability across different camera positions. Policy 1 is trained with an in-hand camera and a front-view camera. Policy 2 is trained with a close-up view camera and a side-view camera. Then the side-view encoder of policy 2 is stitched to the in hand view encoder and action decoder of policy 1, and tested with the in-hand and side-view cameras.
experiment,
Baselines We adopt four baseline methods from previous works and two ablation studies: (1) RT-1 (Brohan et al.] 2022) is a multi-task large model for robotic control built on the Transformer architecture. It is trained on a large dataset of varions robotic tasks and can be generalized in zero-shot to new tasks. (2) Devin et al. (2017) uses a task network module to embed the observation (task state) agnostic to the robot itself, concatenates it with the proprioceptive robot state, and then inputs it into the downstream robot network module. For a fair comparison, the task module and the robot module use the same structure as our PeS method. The key difference is that Devin et al. (2017) docs not adopt the relative represcntation and does not apply our disentanglement regularization. (3) Cannistraci et al. (2023) (linear) uses four similarily functions, L1, L2, L cos to calculate four relative representations and then calculates the element- wise linear sum of the four relative representations to be the final latent state passed over to the action decoder. The weights of the L1, L2, Loc, cos similarities are 0.05, 0.1, 0.05, and 0.8. In contrast, our PeS method only uses cosine similarity. (4) Cannistraci et al. (2023) (nonlinear) also adopts these four similarity functions but passes them through non-linear layer (consists of a LayerNorm, a linear layer, and Tanh) followed by an element-wiee wise summation. (5) The PeS (w/o disent. loss) ablation method does not have the disentanglement loss Eq. 7 (6) The PeS (w. 11 & 12 loss) replaces the disentanglement loss in PeS with L1 and L2 penalty, as shown in Eq. S In addition, we also report the performance of policies 1 and 2 to be stitched if they are transferred to the target environment in zero-shot. These two baselines help us understand how much this transfer preserves the performance of the policy in the initial setting, and how much extra performance increase PeS can bring.
Results Tab. D shows that PeS and its two ablation methods can all achieve satisfying performance with around 90% of success rates on Push and Lift. Compared with directly transferring policy 1 or policy 2 to the new environment, PeS results in 25% to 45% success rate increase. RT-1 demonstrates satisfying robustness on moderate visual variations such as "Mask" and "Noise". However, RT-1 doesn't perform well when the task becomes harder and the visual configuration changes more drastically. For example, it only achieves a 14% of success rate in the List-Fisheye task. Cannistraci et al. 2024 (linear) has lower success rates around 70% 80%, suggesting that introducing a linear combination of multiple similarity measurements for the relative representation hurts the model's performance. Since isometric transformation is the major transformation observed at the latent space, the single cosine similarity of PeS offers simple but effective features. Cannistraci et al. 2024 (uon-linear) performs worse than the linear baseline and reaches about 20% to 50% success rates. We hypothesize that the nonlinear combination introduces learnable parameters for the non-lineat summation process. However, these parameters are not optimized to align the latent spaces
7

---

Published in Transactions on Machine Learning Research (11/2024)
Mask
Zoom In
Blurred
Noise
Fishcye
Position
Lighting
Average
Policy Policy? 2
90.0+2 83
88.7=0.9
71.7=5.73
69. 69.340.91
11.32:094
38.0=1.63
70.6
96.740.94
86 16.0=1663
64.743.00 40
23.3222.9
68.741.88
0.320.97
36,0222.83
6..
Push
RT-1
95.342.40
7.36
66.0444.32
77.94-8.22
76.04..49
45.31:5.73
40.0.-6.53
72.7
Desin ot al. 2017
00.7+10.6
8.7+4.90
16.743.77
59.3+6.80
29.3+7.36
19.345.73
36.7+3.40
33.0
Cannistraci et al. 2024 (liner)
89.3:t4.11
94.0=2883
64.7:11.80
74.7=6.18
74.04:2.88 83
78.742249
87.3+0.90
80.4
Counistraci et al. 2021 (non-lineas
12.7+1.89
18.7=1.98
12.8433.7
29.3::0.91
6.02-1.92
5.3:22.99
32.
20,2
PeS (w/o disent. loes)
100.0:10.0
2..0
80.7.1..84
100.01.0.0
100.010.0
100.0.10.0
86.74.0.94
93.1
PoB (w. & loes)
88.7+44.99
95.3-1.80 1.89
90.0++5.66
100.0+0.0
90.3++0.44
80.7++4.99
80-3+0.94
91.0
Pv8
100.0=0.00
100.0=0.00
95.3:+0.94
100.0+0.0
92.74:2.00 50
100.0=0.00
94.0=24322
97.4
Policy
86.0+1.63
24.7=3.77
68.0:444.90
88.0+1.63
38.74:5.25
0.0++0.00
15.3+6.18
45.8
Pollcyy 2
82.0+44322 32
12.0=7.12
91.3++3.40
83.3+00.44
13.3+00.44
0.0+00000
44.0+22.33
46.6
LiB
RT-1
86.022.88 82
0.0+0.00
66.7:6..1
85.3+:1.99
11.04:3.22 27
35. 35.0+7.07
16.0+:1.22
50.1
Devin al. 2017
0.0:10.00
5.31:244
18.015.89
9.31:4111
14.71:4.99
36.01:1.63
18.9
18.9
Camistraci et al. 2024 (liear)
72.7::3.77
0..0=2288
66.0:14.322
68. 08.7
88.7+:1.85 188
57.54:2.49
56.0++5.89
70.5
Cannistracl et al. 20/4 (mon-linear)
80.3++2.44
36.0-3:27
02.7++3.4
93-3+2.49
16.7422.49
21.3++094
37.3+2.49
49.5
PeS (w/o disent. lose)
83.3:2 GD
50.7=0.73
93.3:10.94
91. 3:5573
79.3+:2. 19
93.3: 49
76.7 T-. 40
85.4
(w. & loss)
97.3:2.49
05.320.94
90.7:10.91
86.01:4.32
81.713377
61:71:3.10
85.2
P68
92.7+2.50
91.7=1189
89.3:::.111
96.0=1.63
88.7+0094
98.0+0.03
64.7=6660
91.3
Table Zero-Shot Transfer Success Rates in basic Simulation tasks. We test all methods across six different visual configurations and also calculate the average performance. In the two basic tasks, Push and Lift, Pc8 and its two ablation methods can all get about a 90% success rate on average. 'he Cannistraci et al. 2024 (lincar) bascline has 70% 80% of success rate, while the Cannistraci et al. 2024 (non-linear) and Devin et al. 2017 baselines perform poorly with about 20% 50% of success rate.
Mask
Zoom In
Blurred
Noise
Fisheye
Position
Lighting
Average
Policy 1
76.7+33.40 3.40
12. 72..4
80.0::2.83
23.3#2.49
16.0+1.63
0.0++0.00
0.0+00000
38.4
Policy RT-1
80.0=0.00
4.0+1.63
78.0+0000
86.7+0.54
13.3+4.11
0.0++0.00
2.0+0.00
37.7
Can
88.3:6624
0:0:0.000
86.728.50
28.3:2.56
0.020.00
8. 8.3:26.2 21
1.020.000
39.1
Devin 2017
19.3+3.25
24.7+1.89
1.7+1.89
60++4.32
20.++3.0
1.3+1.89
3.3+1.80
12..4
Camnistrac el al. 2021 (liusar)
13.3:20.91
1..0
18.7222.99
05.32:0.4
26.7233777
34.722.777
12.7=0.91
42.8
Cancistraci al 2024 (asa (aea-linear)
72.720.94
24.712.49
37.314199
12.74:3.00
1.71
39 39.31:1.89
38. 38.7211.89
37.7
PeS (w/o disent. loss)
44.7+8.06
4.13
34.7
30.7=-6.00
92.7
44.7+:3.40
42.7=
54.2
Pe8 [w. PeS
loss)
17.320.94
18.71:168
54.018644
36.017.12
58.71:1.88
61.7166.00
L:322
51.1
83.3-5.24
80.0+:2.4
74.04-2.83
78.7+4.11
56.0+22.83
78.7 2.49
6..3=6.80
76.2
Policy
86-0+283
16:0+3.00
1.3-094
65-3+0.94
20.7+2.0
0:0+0.00
15-3-6.18
27.2
Policy 2
83.322.19 49
0.0+0.00
1.021.63
64.0=00.00
20011
2.0:1..6
25.6
RT-1
55.02:4.6 0.08
0,010.00
0.0:0.00
76.7
13.315.73
0,0100.00
31.3
29.5
Stack
Devin al. 2017
0.7:0.84
8.0:1.63
0.720.94
24.022288
0.0::0000
14.0=3227
1.74:2.19
7.4
Cammistraci et al. 2021 (linear)
17.320.91
62.02:4.32
32. 32.71:3.77
30.7100.4
54.011.64
14.71:6118
0.71:0.94
34.6
Cannistraci al. 2024 (oon-linear)
1..0:1163
12.0:0.00
0.0:00.00
3.310.94
0.01:0.00
0.71:0944
8.71:3.77
5.0
PeS (w/o disent. lose)
31.0::1.433
10.72:4.11
62. 02.02:0.71
31.0
22.7 22.71:3.77
26.02:4.32
36.728.06
32.3
PeS (w. 11 & loes)
92.710.94
98.0:0.00
62.7166.0
24.0:44.0
59.117.36
1887
48.024.32
63.3
PcB
94.7+0.94
96.7+0.94
90.0+1.63
96.7+1.89
97.3+2.49
80.0+4.90
82.0-1.63
91.1
Policy I
15.3::189
0:000:00
19.32:.71
06:0::4332
10211:
0:0:::000
0.740.94
26.5
Policy 2
89 $9.3220.11
1.8
21.355.25
52.710091
1.01:.6
0.02:000
14.7=3.00
28.8
Door Open
RT-1
96.0#1163
0.0::0.00
26.71-3.40
94.7=33.40
0.710.94
0.0::0.00
0:010.000
31.2
Devia et al. 2017
9.31:1.11
5.310.91
0.0:20.00
1.04:1663
0.720.91
0.0::0.00
6.0111633
3.6
Cannistraci
(linear)
0.0::0.00
L.3:0.94
10.71:2.49
10.79:4.99
2.03:1.63
47.31:9.29
12.027.12
12.0
Cannistrael et al. 2024 (non-linear)
26.0+2.83
31.3+4.99
49.3+-8.22
48.0+55.0
02.7+3340
44.7+33.40
33:3=4.99
42.2
PaS (w/o disent. loss)
24.757.77
44.01:2.83
34.71-9.77
0.74:0.44
36.74.0.44
23.01:3.40
24.0:22.3
26.9
PeS (w. It & 12 loos) Pd5
4.0+-1.63
78.0+-5.66
3.3+0.94
2.0+-1633
42.7+-4.00
6.0+-3.26
14.7=4.99
21.5
58.7:44.11
68.7:0094
70.7:00.4
62.75:3.40
64.7::4.99
48.7:3.40
56.7=5.78
60.1
Table 2: Zero-Shot Transfer Success Rates in difficult Simulation tasks. The three difficult tasks include Can, Stack, and Door Open. We test all methods across six different visual configurations and also calculate the average performance. The three baselines achieve about 0% 40% of success rates in these tasks. In the Can and Stack tasks, PeS achieves 75% -95% of success rates, while the ablation methods achioves about 30% 65% of success rates. In the Door task, PcS achieves 60.7% of success rates, while the two ablation methods onno only have 20% 30% of success rates.
but are optimized for the accuracy of behavior cloning, leading to unaligned latent space and hurting the performance. Devin et al. 2017 bascline achioves very low success rates between 18% to 33%, indicating that the unaligned latent representations cause the failure of the stitched policy for zero-shot transfer.
Tab. 2 reports the performance in more difficult tasks: Can, Stack, and Door Open. PeS achieves 60% to 93% of success rates, which has about 30% to 60% of advantage to the basc policy 1 and 2. The success rates of all four baselines are lower Ihan 43%. This result suggests that previous methods cannot solve the zero- shot transfer problem for difficult tasks with satisfactory performance, which usually involve complicated visual background (e.g., Can), long-horizon multi-stage motions (c.g., Stack), and articulated objects (e.g., Door Open). Among these baselines, we find that RT-1 sometimes can achieve very good performance when the visual variation is moderate. For example, in the Can-Mask task and the Door Open-Noise task,
8

---

Published in Transactions on Machine Learning Research (11/2024)
RT-1 achieves better performance than all other methods including PeS. This is somewhat not surprising since RT-1 was trained on a much larger dataset with a much larger model that could cover some of these variations in training. However, for the harder tasks with more drastic visual configuration changes, such as "Door-Fisheye" and Canack Position", we found that the RT-1 method could not perform well. This additional result suggests that the trade-off between modular and end-lo-end architectures is that an end-to-end network is simpler in the end-to-end training procedure but usually cannot generalize to drastic changes in visual configuration that have not been encountered during training. In contrast, although the modular policies usually involve more careful structure designs in the training procedure, our results show that they performed better in drastic visual changes.
We also notice that the two ablation methods have siguificant performance drops (20% to 60%) in these difficult tasks, showing that the disentanglement regularization in PeS largely improves the zero-shot transfer performance. Although using L1 and 1,2 regularization also leads to good success rates in some cases, the disentanglement regularization has more consistent performance and higher success rates on average.
4.2 Simulation Experiments with Single Camera View and Triple Camera Views
While most visuomotor policies adopt two cameras in their applications, also conduct experiments for poli- cies with only one camera and policies with three cameras to verify the generalizability of PeS to arbitrary camera nunber set sings.
inilar van Aschondd
Cloe Yp Aetessal
Single Camera View We train two policies with two different cameras separately. Each policy still has two visual encoders, but one visual encoder takes in black image and the other encoder takes in the images from the single camera. For both encoders of the policy, they use the same anchor images set collected by the only camera, because we assume that each policy is trained separately and only has access to the dataset of one camera.
Vied Escaderk
ExvodarD
Excoder
Excaderp
den
di
Releive Mepince Litee
Ro Meproemaaton
Reler Seponer lator
In the perception stitching process, we stitch the visual encoder of policy 2 to the visual encoder and action de- coder of policy 1, shown in Figure and assemble stitched policy for double camera inputs. We suggest that this is the most reasonable setting for the perception stitching of singlo-camera policies. In contrast, if we re- move the black image encoder and stitch the only vision encoder of the policy to the action decoder of the policy 2, applying this reassembled policy in either environment or 2 won't lead to better performance compared with the original policy trained to maximum success rate in that environment.
Adae Drader
Arten Desntor3
slitch
Actis
Actin
Vical Ewiider
Vita ExcideD
Lce Dan
Ielavve
Peprnemiabbe
Eppecsntatin
Actus
Figure 5: Perception Stitching with Single Cam- era. The original two policies are trained with only one camera. The other visual encoder takes in a black image. The corresponding anchors are selected with our proposed method 3 The visual encoder of the black image uses the same anchor images as the other encoder of this policy.
As shown in table g PeS outperforms the other methods in the three difficult tasks and achieves close-to-optimal performance in the two basic tasks. It has the highest average success rale over the five tasks.
Ideally, the decoder should learn to completely ignore the latent representations from the encoder with empty input if we train the policy with a large amount of expert data and many epochs. In practice, however, the policy is trained with only 200 expert demonstrations of data and limited epochs. The policy will learn to assign more importance to the encoder with RGB image input and less importance to the encoder with empty input, but will not fully ignore that encoder with empty input. Therefore, stitching another encoder to replace the encoder with empty input will have less influence on the policy performance compared with the double camera experiments, but the disturbance caused by this new encoder will still exist due to the latent
9

---

Published in Transactions on Machine Learning Research (11/2024)
space misalignment, although in some easier experiments, the disturbance is marginal. As supported by our single-camera experiment results, the performance decreases of the baseline methods after the zero-shot transfer are smaller, compared with that in the double-camera experiments, but the performance decreases are not zero.
Push
Lift
Can
Stack
Door Open
Average
Devin et al. 2017
73.3+11.47
72.7+0.94
66.7+6.18
14.0+33.77
22.7+2.50
49.8
Cannistraci et al. 2024 (linear)
89.3+4.11
86.0++1.33
72.8+3.27
14.7+6.18
39.3+3.40
60.4
Cannistraci et al. 2021 (non-linear)
60.7+11.88
80.7+33.10
21.0+2.83
0.7+0.99
41.3+41.9
41.5
PeS (-w/o disent, loss)
88.744.11
94.0#11.63
91.0=3.40
26.0::432
32.7+3.77
66.5
PeS (w. 11 & 12 loss) PeS
94.7+2.49
90.0:22.83
86.7=1899
58.7=1.88
40.7:0.94
74.2
91.34:2.49
92.6
94.6=0094
80.014.90
72.7+3.77
86.2
Table 3: Zero-Shot Transfer Success Rates of single camera policies. PeS achieves optimal performance in the three difficult tasks (Can, Stack, Door Open), and close-to optimal performance in the two basic tasks (Push, Lift). Its average success rate outperforms all the baselines and the ablation methods.
Mask
Zoom in
Blurred
Noise
Fisheye
Camera Position
Average
Devin al. 2017
10011.63
5.312249
1.310.944
13.3+180 89
1.311.89
8.0::1.63
5.5
Cannistraci et al. 2024 (linear)
46.743.40
20.7+188
13.
56.0=2.83
16.02:1.2
30.0=4.32
30.5
Canmistraci et al. 2024 (non-lineur)
30.7:10.94
22.740.94
19.3:44.1
42.741.89
12.7:0.94
4.7::2.49
22.1
PoS (-w/o disent. loss)
21.3:12.49
20.7:11.0
72.743.77
56.7:44.11
37.314.99
9.31:1899
36.3
PeS (w. II & 12 loss) PoS
46.7+33.40
56.7+44.11
84.0+2.83
82.7+2249
75.3+22.9
61.3=3.40
67.8
97.3+2.49
92.7+0094
93.3+0.94
90.0+1.63
94.6+0.94
86.7+2.49
92.4
Table 4: Zero-Shot Transfer Success Rates of triple camera policies, All the experiments carried out with the Stack task. PeS achieves optimal performance for all the six different visual configurations. Its average success rate has an around 25% advantage to the second best method.
Triple Camera Views We pick the Stack task and train policies with three visual encoders that embed three different visual configurations. For the first five visual configurations that add different effects to the observations, policies are all trained with the in-hand view, close-up view, and side view. Policy has the effect (e.g. Gaussian noise) only in vision 3, and policy 2 has the same effect in vision and vision 2. Then stitch the encoder 3 of policy 2 to the rest parts of policy and test the stitched policy with three normal cameras. For the last experiment that involves different. camera positions, policy 1 is trained with the in-hand view, close-up view, and front view, and policy 2 is trained with close-up view, front view, and side view. Then we stitch the side view encoder of policy 2 to the in-hand view, close-up view encoders, and decoder of policy 1. The stitched policy is tested with in-hand view, close-up view, and side view. This triple camera. perception stitching process is presented Figure 0
Table 4 reports the zero-shot transfer success rates for the Stack task with six different vi- sual configurations. The PeS method consis- tently outperforms the other baseline and ab- lation methods for all six visual configurations. On average, reaches a 92.4% success rate on the stacking task, which is about 25% higher than the second-best. method.
Policy
Policy
In- Hand View
Close-Up View
Pront View
Close-Up View
Front View
Side View
Stitch
Stitched Policy:
4.3 Real World Experiment
Camera Configurations We conducted Reach, Push, Lift, and Stack in our real-world experiments (Fig. 7). In Reach, the robot starts from some random position in the air and reaches the red cube. In this experiment, the policy is trained with front-view cam- era with broken lens (Fig. 7 and a normal
In-Hand View
View
Side View
Figure Perception Stitching with Three Cameras. The orig- inal two policies are trained with three cameras. In the Perception Stitching process, the visual encoder the policy is stitched to the visual encoder and 2 and the action decoder of the policy
in-hand camera. The policy 2 is trained with normal front-view camera and in-hand camera with
10

---

Published in Transactions on Machine Learning Research (11/2024)
broken lens. Then we stitch the front view encoder of the policy 2 to the in-hand view encoder and action decoder of the policy 1 and test. the stitched policy with two normal cameras. Tn Push, the robot starts from some random position to reach the cube and then pushes the cube across the green line on the table. The training and testing process of Push is similar to that of Reach, and the only difference is replacing the camera with broken lens with camera with a masked lens, as shown in Fig. T In Lift, the red cube is placed at a random position on the table with a random orientation. The robot needs to grab the cube and lift it up. In this experiment, the policy 1 is trained with in-hand and side-view cameras. The policy 2 is trained with side-view and front-view cameras. Then stitch the front view encoder of the policy 2 to the in-hand view encoder and action decoder of the policy 1, and test the stitched policy with an in-hand camera and front-view camera, as shown in the left half part of the Fig. 2 In Stack, the red cube is placed at random position on the left side of the table, and the robol. should lift. it up and stack it on the blue cube on the right. Similar to Lift, it also involves the stitching between cameras at different positions (left, front, and right). Stack the most difficult one in all the real-world experiments. It requires the most sophisticated manipulation skills and long- horizon multi-stage motions.
RReach
Push
Lift
Stack
In-kand
Camera
Front Vee
Camera
Side View
Camera
reaching, pushing, lifting, and stacking cube. Three cameras mounted the front view, side view, and in-hand positions separately. (B) Broken lens. (C) Masked lens.
Figure (A) Real- World Tasks:
Results As shown in Tab. 5 PeS achieves 100% of success rate in Reach, 85% in Push, 80% in Lift, and 45% in Stack, while Devin et al. 2017 base- line gets 0% of success rates in all the experiments. In the experiments, we observe that PeS policies show more accu- rate motions compared with the baseline policies. In comparison, Devin et al. 2017 baseline policies cannot output actions accurately enough to accomplish the tasks, although they have some attempts to complete the tasks in some cases. Take the Lift task as an example, the robot trained with the baseline method reaches some positions close the cube but always fails to grasp it. In the most difficult Stack task, PoS robot can still achieve many success cases, while the baseline robol cannot output meaningful motions. Please refer to the supplementary material for the experiment videos.
Reach broken lens masked lens different positions
Push
Lift
Stack different positions
PeS Devin et al. 2017
100.0
85.0
80.0
45.0
0.0
0.0
Table 5: Zero-Shot Transfer Success Rates in Real World. We report the success rates of three manipulation tasks real world. Each success rate is calculated with 20 games, PeS achieves a 100% success rate the casiest Reach task, 85% in the Push task, 80% the Lift task, and 45% in the hardest Stack task. In comparison, the baseline method doesn't have any success case in all these tasks.
In summary, we find that the performance advantage of PeS is more pronounced in the real world than in the simulation. We assume that this is because there are more noises and disturbances in the real world. To the best of our knowledge, PeS is the first method that enables vision-based zero-shot transfer in the real world via reassembling neural network components.
4.4 Analysis: Latent Space at Module Interface
To understand the mechanism behind the high success rate of PeS, we perform visualization and quantitative analysis of the latent representations of the visual encoders.
We choose the Push-Camera Position experiment. in the simulation and visualize the corresponding latent representations to have an intuitive understanding of the mechanism of PeS. We first reduced the 256D ropresentations to 3D with PCA PCA(Hotelling 1933) for visualization. Since the side view encoder of the policy 2 is stitched to the policy at the position of its original front view encoder, we compare the latent representations of these two encoders. As shown in Fig. 8 a),, the latent representations trained with PeS have
11

---

Published in Transactions on Machine Learning Research (11/2024)
similar shapes to each other. In comparison, the latent representations with the Devin al. 2017 bascline are not similar but have an approximately isometric transformation (rotation in this case) relationship with each other. The latent representations visualization of other tasks can be found in Appendix
(a)
Pe5 Push Task
(b)
(c)
Cosine Distance Comparison
L2 Distance Comparison
1.2
PeS
Devin 2017
PeS
Devin 2017
7-
1.0
Distance
0.6
front view!
side view
0.4
0.2
1-
0.0
Risheye Camera
Riseyye
Figure 8: Latent Space Analysis. (a) We visualize the latent representations of the front-view encoder of policy and the side-view encoder of policy of the Push-Camera Position experiment. The 256D) representations reduced to 3D) with PCA (Hotelling (933). The red dots represent samples where the robot's 's end effector is at higher positions, blue dots indicate medium heights and green dots correspond (Devin et al. 2017) method. (b) & (c) We compare the distances of the latent representations in all the experiments in the Push task. One representation from the second view encoder of policy and the other is from the second view encoder of policy 2.)
lower positions
the cube. compare
(PeS) method with the baseline
For further quantitative analysis, we select the Push task in the simulation and calculate the pairwise distances of the latent. representations all Push experiments with different visual configurations. Fig. (b) shows that PeS significantly reduces the cosine distances of the latent representations in all these experiments. Fig. (c) shows that the L2 distances with PeS generally smaller than that with the Devin et al. 2017 baseline, but the differences not distinguishable in some cases (e.g. Tush-Zoom in). More mathematical details of calculating the cosine and T.2 distances can be found in Appendix C
4.5 Analysis: Highlight Attention Regions with Grad-CAN CAd
We visualize the attention map from the policy modules further explain why certain policies work well while others fail. To this end, modify the Gradient-weighted Class Activation Mapping (Grad-CAM) (Selvaraju et al. 2017) approach to highlight the regions that the policies pay attention to. Grad- CAM is widely applied to neural classi- fication models with convolutional lay- ers. To adapt the Grad-CAM from im- age classification to robot learning, we replace the beforo-softmax score of for
PeS
Devin et al. 2017
PeS
Devin et al. 2017
(a) In-hand View
(b) Side View
Figure 9: Attention Heatmap. (a) In the in-hand view, our policy pays attention to the cube on the table, while the baseline policy pays meaningless attention to the upper region of the image. (b) In the front view image, our policy pays attention to the robot end effector, while the bascline method has slightly more attention to the two sides of the image.
class e of the image classification net- works with the log-likelihood I(a) of the robot action a in the training dataset. We denote the 4th feature map activation output from the last convolutional layer as A Then the backpropagated gradient of I(a) d((a)
with respect to _ is computed as 045 We do global average pooling of these gradients over the width (indexed by i) and height (indexed by j) dimensions of the feature map to get the neuron importance weight
global average ponling
Øl(a)
04
ΣΣ
(9)
7
0A
gradients backprop
12

---

Published in Transactions on Machine Learning Research (11/2024)
This weight aj captures the "importance" of feature map & for robot action a. Then, the attention map I CAM is calculated as the weighted combination of forward activation maps followed by ReLU Sel- varaju et al. 2017):
Σ o
I Grad-CAM = ReLU
(10)
linear combination
We apply ReLU because we are only interested in the features that have positive influence on the actions. The inlensity of these pixels should be increased in order to increase the log-likelihood I(a) (Selvaraju el al. 2017j. This GIad-CAAM is a heatmap of the same size as the convolutional feature maps A We upsample it to the input image size with bilincar interpolation to get the final attention heatmap of the input image (Selvaraju et al. 2017). larger value on this heatmap means this pixel contributes to a larger gradient of the log-likelihood of the robot action.
We choose the Lift-Camera Position experiment and visualize the attention heatmap of the stitched policies with PeS and the baseline separately. As shown in Fig. [(a] from the in-hand camera, our policy is focusing on the cube between the two gripper fingers, which is intuitively what humans pay attention to in Lift task. In comparison, the Devin et al. 2017 baseline policy has more attention to the upper middle part of the image, which is the desk surface, and is not informative for this task. In Fig. [(b] our policy is paying attention to the robot end effector from the side-view camera, while the baseline policy has more attention to the two sides of the image, while these regions have no crucial object. To sum up, the attention maps from the Grad-CAM suggest that. the stitched policy with PeS performs better than the baseline because it can pay attention to the crucial regions for accomplishing the task, while the baseline policy cannot. This result provides an intuitive explanation of the good performance of PeS. Videos of the attention heatmaps during the manipulation process can be found in the supplementary material.
5 Discussion
Conclusion We present Perception Stitching (PeS), a method for zero-shot visuomotor policies transfer via latent spaces aligmment. PeS aligns the latent spaces of different visual encoders by enforcing the relative representations invariant to isometric transformations and, therefore, allows the trained visual encoders to be reused in a alug-and-go manner. Our evaluation covers 35 simulation experiments and real-world exper- iments with variety of manipulation tasks and camera configurations. The results demonstrate significant performance improvement with PoS for zero-shot transfer, and its advantage is especially pronounced in real- world lasks. Moreover, we conduct quantitative and qualitative analyses to understand the mechanism of the superior performance of PeS. We hope that this work can inspire further exploration of the compositionality and modularity in robot learning.
Limitations and Future Work There no additional training required during transfer, but to obtain the anchor states our framework still requires the robot to replay the trajectories in the previous dataset to collect a new datasct and then select anchors with the corresponding indices. Although this is feasible, we have demonstrated in our real-world experiments, the trajectories replaying process in the real world usually takes longer time than collecting new dataset by random sampling. Future work can develop an alternative algorithm for more efficient anchor selection. We also acknowledge that PeS cannot yet achieve perfect performance on some tasks that require very high precision or have very long horizons. One major cause of the potential failure that the PeS can only enforce an approximate invariance of the latent ropresentations but not a strict invariance, which leads the loss of some accuracy of the stitched policy. It is an interesting topic for future work to further refine the latent space alignment.
In addition, the policy trained with the Behavior Cloning algorithm usually cannot recover from the error accumulation very well in long-horizon tasks. Although our paper focuses on imitation learning, one exciting future direction is to explore our techniques in a reinforcement learning setup (Ricciardi et al.) 2024) or explore more advanced imitation learning that handles error accumulation, since these approaches have shown some capabilities of recovery from failures in recent works. Another promising direction is explore the combination of PeS with other robot transfer learning techniques that focus beyond vision encoder transfer, such embodiment transfer and scale to large robot. learning datasets.
13

---

Published in Transactions on Machine Learning Research (11/2024)
References
Pooya Abolghasemi, Amir Mazaheri, Mubarak Shah, and Ladislau Boloni. Pay attention!-robuustiyingg a deep visuomotor policy through task-focused visual attention. In Proceedings of the IEEE/C /CVF Conference on Computer Vision and Pattern Recognition, pp. 1254-4262, 2019.
Ferran Alet, Tomás Lozano-Pérez, and Leslic P Kaclbling. Modular meta-lcarning. In Conference on robot learning, pp. 856-868. PMLR, 2018.
Michael Bain and Claude Sammut. A framework for behavioural cloning. In Machine Intelligence 15, pp. 103-129, 1995.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Vatann-innarranc--cvaaiin regularization for self-supervised learning. arXiv preprint arx: 05. 040004 2021.
André Barrelo, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado P van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. Advonces in neural information processing systems, 30, 2017.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real- world control at scale. arXiv preprint arXiv:2212.0688 2022.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. R1-2: Vision--anguagg-scciinn models transfer web knowledge to robotic control. arxiv preprint 0rXiv:2307.15818 2023.
Irene Cannistraci, Luca Moschella, Marco Fumero, Valentino Maiorca, and Emanuele Rodolå. From bricks to bridges: Product of invariances to enhance latent space communication. ar preprint orXin:2310.01211, 2023.
Boyuan Chen, Robert Kwiatkowski, Carl Vondrick, and Hod Lipson. Fully body visual self- self-modeling of robot morphologics. Science Robotics, 7(68):cabn1944, 2022.
Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentanglement in variational autoencoders. Advances in neural information processing systems, 31, 2018.
Yutian Chen, Abram L Friesen, Feryal Behbahani, Arnaud Doucet, David Budden, Matthew Hoffman, and Nando de Freitas. Modular meta-learning with shrinkage. Advances in Newral Information Processing Systems, 33:2858 2869, 2020.
Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song, Diffusion policy: Visuomotor policy learning via action diffusion. orXiv preprint arXiv:2308. 04137, 2023.
Sudeep Dasari and Abhinav Gupta. Transformers for ono-shot visual imitation. In Conference on Robot Learning, pp. 2071-200 1771-004.. PMLR, 2021.
Coline Devin. Compositionality and Modularity for Robot Learning, PhD thesis, EECS Department, Uni- versity of California, Berkeley, Dec 2020. URL htt.:t/pwwwww....ooooorr/r////// 2TCS-2020-207 html
Coline Devin, A Abhishek Gupta, Trevor Darrell, Pieter Abbeel, and Sergey Levine. Learning modular neural network policies for multi-task and multi-robot transfer. In 2017 IEEE international conference on robotics and automation (ICRA), pp. 2169-2176. IEEE, 2017.
Finale Doshi- Velez and George Konidaris. Hidden parameter markov docision processes: A semiparametric regression approach for discovering latent task parametrizations. In IJCAI: proceedings of the conference, volume 2016, pp. 1432. NIH Public Access, 2016.
Doshi-Velez
Maximilian Du, Suraj Nair, Dorsa Sadigh, and Chelsea Finn. Behavior retrieval: Few-shot imitation learning by querying unlabeled datasets. arXiv preprint or arXiv:2304.088 08742, 2023.
14

---

Published in Transactions on Machine Learning Research (11/2024)
Frederik Ebert, Chelseaa Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine. Visual fore- sight: Model-based deep reinforcement learning for vision-based robotic control. ar preprinl 0at:11822000633 2018.
Alcksandr Ermolov, Aliaksandr Siarohin, Enver Sangincto, and Nicu Sobe. Whitening for self-supervised representation learning. In International conference on machine learning, pp. 3015 3024. PMLR, 2021.
Fernando Fernández and Manuela Veloso. Probabilistic policy reuse in reinforcement learning agent. In Proceedings of the fifth international joint conference Autonomous agents and malliagent systems, pp. 720 727, 2006.
Chelsca Finn, Xin Yu Tan, Yan Duan, Trevor Darrell, Sergey Levine, and Picter Abbeel. Deep spatial auloencoders for visuomotor learning. In 2016 JEEE International Conference on Robolics and Aulomation (ICRA), pp. 512-519. IEEE, 2016.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pp. 1126-1135. PMLR, 2017a.
Chelsea Finn, Tianhe Yu, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot visual imitation learning via meta-learning. In Conference robot learning, pp. 357-368. PMLR, 2017b.
Pete Florence, Corey Lynch, Andy Zeng, Oscar A Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong, Johnny Lee, Igor Mordatch, and Jonathan Tompson. Implicit behavioral cloning. In Conference on Robot Learning, pp. 158-168. PMLR, 2022.
Zipeng Fu, Tony Z Zhao, and Chelsea Finn. Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation. arXie preprint arXiv:2401.00111, 2024.
Abhishek Gupta, Coline Devin, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Learning invariant feature spaces to transfer skills with reinforcement learning, arXiv preprint arvv 703. 02949, 2017.
Alcksi Hämäläinen, Karol Arndt, Ali Ghadirzadch, and Ville Kyrki. Affordance lcarning for end-to-end visuomotor robot control. In 2019 IEEE/RS] International Conference on Intelligent Robols and Systems (IROS), pp. 1781-1788. IEFE, 2019.
John A Hartigan and Manchek A Wong. Algorithm as 136: A k-means clustering algorithm. Journal of the royal statistical society. series e (applied statistics), 28(1):100-108, 1979.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Trina Higgins, Loic Matthey, Arka Pal, Christopher P Burgess, Xavier Glorot, Matthew M Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. ICLR (Poster), 3, 2017.
Trina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Malthey, Danilo Rezende, and Alexander Lerchner. Towards definition of disentangled representations. arXiv preprint XXiv:1812.02230 2018.
Geoffrey Hinton, Li Deng, Dong Yu, George E Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N Sainath, et al. Deep neural networks for acous- tic modeling in speech recognition: The shared views of four research groups. IEEE Signal processing magazine, 29(6):82-9) 97, 2012.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 98): 1997.
Harold Hotelling. Analysis of a complex of statistical variables into principal components. Journal of educational psychology, 24(6):417, 1933.
15

---

Published in Transactions on Machine Learning Research (11/2024)
Kyle Hsu, Moo Jin Kim, Rafacl Rafailov, Jiajun Wu, and Chelsea Finn. Vision-based manipulators need to also see from their hands. arXiv preprint ar arXin:2203. 00022203..6777 12677, 2022.
Yuhang Hu, Boyuan Chen, and Hod Lipson. Egocentric visual self-modeling for legged robot locomotion. arXiv preprint orXm:2207.0880 03386, 2022.
Marcel Hussing, Jorge A Mendez, Anisha Singrodia, Cassandra Kent, and Eric Eaton. Robotic manipulation datasots for offline compositional reinforcement lcarning. arXiv preprint ar arXin:2307.07091, 2023.
Divye Jain, Andrew Li, Shivam Singhal, Aravind Rajeswaran, Vikash Kumar, and Emanuel Todorov. Learn- ing deep visuomotor policies for dexterous hand manipulation. In 2019 international conference on robotics and automation (ICRA), pp. 3636-3643, IEFE, 2019.
Stephen James, Andrew J Davison, and Edward Johns. Transferring end-to-end visuomotor control from simulation to real world for a multi-stage task. Conference on Robot Learning, pp. 334-343. PMLR, 2017.
Stephen James, Paul Wohlhart, Mrinal Kalakrishnan, Dmitry Kalashnikov, Alex Irpan, Julian Ibarz, Sergey Levine, Raia Hadsell, and Konstantinos Bousmalis. Sim-to-real via sim-to-sim: Data-ellicient robolic grasping via cooomizeddto-canonioe adaptation networks. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 12627 12637, 2019.
Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. Be-z: Zero-shot lask generalization with robotic imitation learning. In Conference Robol Learning, pp. 991-1002. PMLR, 2022.
Pingcheng Jian, Easop Lee, Zachary Bell, Michael M Zavlanos, and Boyuan Chen. Policy stitching: Learning transferable robot policies. Conference on Robot Learning, pp. 3789 3808. PMLR, 2023.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for basedn-based robotic manipulation. arXiv preprint arXiv:1806. 10293, 2018.
Taylor W Killian, Samuel Daulton, George Konidaris, and Finale Doshi- Robust and efficient. transfer learning with hidden parameter markov decision processes. Advances in neural information processing systems, 30, 2017.
Heecheol Kim, Yoshiyuki Ohmura, and Yasuo Kuniyoshi. Transformer-based deep imitation learning for dual-arm robot manipulation. In 2021 JEEE/RS) International Conference on Intelligent Robots and Systems (IROS), pp. 8963 8972. IEEE, 2021.
Heecheol Kim, Yoshiyuki Ohmura, Akihiko Nagakubo, and Yasuo Kuniyoshi. Training robots without robots: deep imitation learning for master-to-robot policy transfer. IEEE Robotics and Automation Letters, 8(5): 2906-2913, 2023.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising, In International conference on machine learning, pp.
PMLR, 2018
Kuno Kim, Yihong Gu, Jiaming Song, Shengjia Zhao, and Stefano Ermon. Domain adaptive imitation learning. In International Conference on Machine Learning, pp. 5286-500 5295. PMLR, 2020.
George Konidaris and Andrew Barto. Autonomous shaping: Knowledge transfer in reinforcement learning. In Proceedings of the 23rd international conference on Machine learning, pp. 489 496, 2006.
George Dimitri Konidaris and Andrew G Barto. Building portable options: Skill transfer in reinforcement learning. In Ijeui, volume 7, pp. 895 900, 2007.
Robert Kwiatkowski, Yuhang Hu, Boyuan Chen, and Hod Lipson. On the origins of self melf-modeling. arXiv preprint ar Viv:2209.02014 2022.
16

---

Published in Transactions on Machine Learning Research (11/2024)
Alessandro Lazaric, Marcello Restelli, and Andrea Bonarini. Transfer of samples in batch reinforcement learning. In Proceedings of the 25th international conference on Machine learning, pp- 544-551, 2008.
Miguel Aizaro-Gredille Dianhuan Lin, J Swaroop Guntupalli, and Dileep George. Beyond imitation: Zero- shot task transfer on robots by learning concepts as cognitive programs. Science Robotics, 4(26):coav31100 2019.
Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1 (4):541 551, 1989.
Sergey Levine, Chelsoa Finn, Trevor Darrell, and Picter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334-1333 2016.
Xiang Li, Varun Belagali, Jinghuan Shang, and Michael Ryoo. Crossway diffusion: Improving diffusion-
based visuomotor policy via self-supervised learning. arXiv preprint arXtt:2307.018/3 2023.
Chenyu Liu, Yan Zhang, Yi Shen, and Michael M Zavlanos. Learning without knowing: Unobserved context in continuous transfer reinforcement learning. In Learning for Dynamics and Control, pp. 791 802. PMLR, 2021.
Corey Lynch and Pierre Sermanet. Grounding language in play. arXiv preprint arXim:2005.07648 3, 2020.
Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, and Roberto Martin-Martin. What matters in learning from offline human demonstrations for robot manipulation. arXiv preprint arXiv:2108.0222 2021.
Bhairav Mehta, Manfred Diaz, Florian Golemo, Christopher J Pal, and Liam Paull. Active domain random- ization. In Conference Robot Learning, pp. 1162-1176. PMTR, 2020.
Jorge A Mendez and Eric Eaton. How to reuse and compose knowledge for lifetime of tasks: A survey continual lcarning and functional composition. arXiv preprint arXiv:2207.07790, 2022.
Jorge A Mendez, Harm van Seijen, and Eric Eaton. Modular lifelong reinforcement learning via neural composition. arXiv preprint orXic:2207.00429 2022.
Jorge Armando Méndez. Lifelong Machine Learning of Punctionally Compositional Structures. PhD thesis, University of Pennsylvania, 2022.
Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norelli, Franccsco Locatello, and Emanuele Rodola. Relative representations enable zero-shot latent space communication. arXie preprinl arXiv:2209.15430 15490 2022.
Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. arxiv preprint arXiv:2203. 12601, 2022.
Phuong DH Nguyen, Tobias Fischer, Hyung Jin Chang, Ugo Pattacini, Giorgio Metta, and Yiannis Demiris. Transferring visuomotor learning from simulation to the real world for robotics manipulation tasks. In 2018 TEEE/RSJ International Conference on Intelligent Robots and Systems (TROS), pp. 6667-6674. IEEE, 2018
Abhishek Padalkar, Acorn Pooley, Ajinkya Jain, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khaz- atsky, Anant Rai, Anikait. Singh, Anthony Brohan, et al. Open x-embodiment: Robotic learning datasets and rt-x models. arXiv preprint arXiv:2310.08864, 2023.
Decpak Pathak, Parsa Mahmoudich, Guanghao Luo, Pulkit Agrawal, Dian Chen, Yide Shentu, Evan Shel- hamer, Jitendra Malik, Alexci A Efros, and Trevor Darrell. Zero-shot visual imitation. In Proceedings of the IEFE conference on computer vision and poltern recognition workshops, pp. 2050-2053, 2053, 2018.
17

---

Published in Transactions on Machine Learning Research (11/2024)
Jonas Pfeiffer, Sebastian Ruder, Ivan Vulié, and Edoardo Ponti. Modular deep learning. Transactions on Ma- chine Learning Research, 2023. ISSN 2835-8856. URL Survey Certification.
Dean A Pomerleau. Alvinn: An autonomous land vehicle in neural network. Advances in neural information processing systems, 1, 1988.
Robin Quessard, Thomas Barrett, and William Clements. Learning disentangled representations and group structure of dynamical environments. Advances in Neural Information Processing Systems, 33:19727- 19737, 2020.
Rafael Rafailov, Tianhe Yu, Aravind Rajeswaran, and Chelsea Finn. Offline reinforcement learning from images with latent space models. In Learning for Dynamics and Control, pp. 1154 1168. PMLR, 2021.
Kartik Ramachandruni, Madhu Babu, Anima Majumder, Samrat Dutta, and Swagat Kumar. Attentive task-net: Self supervised task-attention network for imitation learning using video demonstration. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 4760-4766 IEEE, 2020.
Antonio Pio Ricciardi, Valentino Maiorca, Luca Moschella, Riccardo Marin, and Emanuele Rodolà. Zero- shot stitching reinforcement learning using relative representations. arXiv preprint or urxii:2444. 12917, 2024.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured pre- diction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 627-635. JMLR Workshop and Conference Proceedings, 2011.
Andrei A RUsu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXio preprint arXiv:1606.00 04661, 2016.
Andrei A Rusu, Matej Večerík, Thomas Rothörl, Nicolas Heess, Razvan Pascanu, and Raia Hadsell. Sim- to-real robot learning from pixels with progressive nets. In Conference on robot learning, pp. 262 270. PMLR, 2017.
Fereshteh Sadeghi, Alexander Toshev, Eric Jang, and Sergey Levine. Sim2real viewpoint invariant visual servoing by recurrent control. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4691-4699, 2018.
Stefan Schaal. Learning from demonstration. Advances in neural information processing systems, 9, 1996.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEFE international conference on compuler vision, pp. 618 22-6666 2017.
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. Time-contrastive networks: Self-supervised learning from video. In 2018 JEEE international conference on robotics and automation (ICRA), pp. 1134-1141. IEEE, 2018.
Lin Shao, Toki Migimatsu, Qiang Zhang, Karen Yang, and Jeannette Bohg. Concept2robot: Learning ma- nipulation concepts from instructions and human demonstrations. The International Journal of Robotics Research, 40(12-14):1419 14344144444444444 2021.
Aravind Srinivas, Allan Jabri, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Universal planning networks: Learning generalizable representations for visuomotor control. In International Conference on Machine Learning, pp. 4732-4741. PMLR, 2018.
Simon Stepputtis, Joseph Campbell, Mariano Phiclipp, Stefan Lee, Chitta Baral, and Heni Ben Amor. Language-conditioned imitation learning for robot manipulation tasks. Advances in Neural Information Processing Systems, 33:13139-13155 2020.
18

---

Published in Transactions on Machine Learning Research (11/2024)
Chuanqi Tan, Fuchun Sun, Tao Kong, Wenchang Zhang, Chao Yang. and Chunfang Liu. A survey on deep transfer learning. In Artificial Neural Networks and Machine Learning Learning-ICANN ICANN 2018: 27th International Conference on Artificial Neural Networks, Rhodes, Greece, October 4-7, 2018, Proceedings, Part 111 27, pp. 270-279. Springer, 2018
Matthew E Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research, 10(7), 2009.
Andrea Tirinzoni, Rafacl Rodriguez Sanchez, and Marcello Restelli, Transfer of value functions via variational methods. Advances in Neural Information Processing Systems, 31, 2018.
Josh Tobin, Lukas Biewald, Rocky Duan, Marcin Andrychowiez, Ankur Handa, Vikash Kumar, Bob Me- Grew, Alex Ray, Jonas Schneider, Peter Welinder, et al. Domain randomization and generative models for robotic grasping. In 2018 RRE/RSJ International Conference on Intelligent. Robots and Systems (TROS), pp. 1482-34899 IEEE, 2018.
Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. arXiv preprint arXiv:1805. 01954,, 2018.
Luan Tran, Xi Yin, and Xiaoming Liu. Disentangled representation learning gan for pose-invariant face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1415 1424, 2017.
Xin Wang, Hong Chen, Si'ao Tang, Zihao Wu, and Wenwu Zhu. Discntangled representation learning, arXiv preprinl ar Xrx:v:2211. 1. 1695,, 2022.
Ruihan Yang, Huazhe Xu, Yi Wu, and Xiaolong Wang. Multi-task reinforcement lcarning with soft modu- larization. Advunces in Newral Information Processing Systems, 33:4767 13:4767-47777, 2020.
Tianhe Yu, Chelsea Finn, Annie Xie, Sudeep Dasari, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot imitation from observing humans via domain-adaptive leta-learning, arXiv preprint 5xXiv:1802.0155% 2018
Youssel Zaky, Gaurav Paruthi, Bryan Tripp, and James Bergstra. Active perception and representation for robotic manipulation. arXiv preprint orXiv:2003.0673/ 2020.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International conference on machine learning, pp. 12310-12320. PMLR, 2021. 2021.
Jingwei Zhang, Lei Tai, Peng Yun, Yufeng Xiong, Ming Liu, Joschka Boedecker, and Wolfram Burgard. Vr-goggles for robots: Real-to-sim domain adaptation for visual control. IEEE Robotics and Automation Letters, 4(2):1148 1155, 2019.
Yan Zhang and Michael M Zavlanos. Transfer reinforcement learning under unobserved contextual informa- tion. In 2020 A ACM/IEEE 11th International Conference on Cyber-Physiene Systems (ICCPS), pp. 75-86. IEEE, 2020.
Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine- fine-grained bimanual manipulation with low-cost hardware. arXio preprint arXiv:2304. 13705, 2023.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on compuler vision, pp. 2223-2222222 2202,222222222 2017.
Yifeng Zhu, Abhishek Joshi, Peter Stone, and Yuke Zhu. Viola: Object-centric imitation learning for vision- based robot manipulation. In Conference on Robot Learning, pp.
PMLR, 2023.
Yuke Zhu, Ziyu Wang, Josh Merel, Andrei Rusu, Tom Erez, Serkan Cabi, Saran Tunyasuvunakoo,, János Kramár, Raia Hadsell, Nando de Freitas, et al. Reinforcement and imitation learning for diverse visuomotor skills. or Xiu preprint ar Xrv: 00..1102.000 09564, 2018
19

---

Published in Transactions on Machine Learning Research (11/2024)
A
Network Structure
ReaNet18
ReaNet18
ReaNet18
ResNet18
Spatial Softmax
Spatial Softmax
Spatial Softmax
Spatial Softmax
low dim obs
low dim obs
Linear Layer units)
Linear Layer (256 units)
Linear Layer (256 units)
Linear Layer units)
Linear Layer (256 units)
Linear Layer (256 units)
GMM-MLP
GMM-LSTM
action
action
(a) MI.--based policy
(b) RNN-based policy
Figure 10: Policy Network Structure. The detailed architectures of the visuomotor policy networks, We adopt the RNN-
based network for the Pick-And-Plaacee Can task, and the MLP-based network for other tasks.
Hyperparameter
Default
Learning Rate
1 10-4
Action Decoder MLP Dims
[1024, 1024]
GMM Num Modes
5
Image Encoder
ResNet-18
SpatialSoftmax (mum-KP)
64
Image Embedding Layer
256 units
Low Dim Obs Embedding Layer
64 units
Table 6: MLP-based policy Hyperparameters.
For the baseline methods, we demonstrate the detailed structures of the ML.P-base policy network in Fig. 10(a] and the RNN-based policy network in Fig. 10(b] The hyperparameters of the ML.P-base network and the RNN-based network are listed in TABLE and TABLE separately.
The policy network of the Perception Stitching method shares all the network structures and hyperparameters as that of the bascline mothod. The only difference that it caleulates the relative reprosentations of the images after the 256-unit linear layer, as shown in Fig, 2 and equation 3
B Perception Stitching Algorithm
The algorithm of the Perception Stitching (PeS) is shown in Algorithm 1 The first step of this algorithm is collect expert dataset DD in the environment E, by teleoperating the robot by proficient human expert.
20

---

Published in Transactions on Machine Learning Research (11/2024)
Algorithm Zero-shot Transfer with Perception Stitching
Collect Dataset 1 with random sampling:
Task T in the environment Ej with two visual configurations of and 02 Bi D1 4 0.
Initialize empty dataset
for each game i of the task do Random sample the initial state of the task.
Execute the Expert policy to collect the expert trajectory 7|² of this game.
Push T into the dataset D1. end for
Collect Dataset 2 with trajectories replay:
Task 7 in the environment Eg with two visual configurations of and of Initialize an empty dataset D22 + 0.
for each game i of the task do
Replay the trajectory T! in Eg to collect the trajectory 7, of this game.
Push T? into the datasct D2.
end for
Collect Anchor States:
Task T in the environment Es with two visual configurations Fr and 02 Fa K-means center set C₁ + k means(ID1).
Select anchor set A {a ( from Di which are images closest to the K-means center Ci.
Select anchor set A2 = {az (1) from B2 which have the same indices as & in Dj.
Anchor set A3 ( consists of images of of from A and images of 02 Ea from Az.
Train Modular Policies in Source Environments:
Enviromment Ej has two visual configurations 0 Fi Fi and low dimensional observation Fi
Environment Eg has two visual configurations 01 02
F2. E2 and low dimensional observation 0
Initialize policy FE (of) (of1), ) (oF1).
).
of A1) Ej.
Ei.
Initialize policy I (2 (o ).92 (o
), F2 A2) in Ez. Fa
Optimize f (
Es (o (of)),
) A1) with dataset DD with the Behavior Cloning algorithm Ei (o
Schaal
1996).
Optimize B (01
F2 Fi ), 92 (02
Fa
Fa A2) with dataset D2 with the Behavior Cloning algorithm 22) )
Schaal
1996).
Perception Stitching:
Enviromment E& has visual configurations of Et and 02 Eg
Initialize the visual encoder 1 with parameters from 91 Initialize the visual encoder 2 with parameters from 92 Initialize the action decoder with parameters from fEi Construct the stitched policy f (g (ofi), ),9
Ez (oa
(o of As).
Test the Stitched Policy in the Target Environment:
Rollout the stitched policy f (gf (i (oF'),
Fi (1,,a)) in Eg. " Ea )) of
Calculate the success rate of task T with the policy f Fs (g) (o ),9g 2 (o
Fl
R3), Fo
0
Az) Es.
21

---

Published in Transactions on Machine Learning Research (11/2024)
Hyperparameter
Default
Learning Rate
1 10-4
Action Decoder MLP Dims
[] 1000
RNN Hidden Dim
RNN Seq Len
10
GMM Num Modes
5
Image Encoder
ResNet-18
SpatialSoftmax (num-KP)
64
Image Embedding Layer
256 units
Low Dim Obs Embedding Layer
64 units
Tahle 7: RNN-based policy Hyperparameters.
Then we replay the expert trajectories in Dj collect another expert dalaset D2 in the environment Ez- We collect the anchor set A1 in the dataset D1 via the K-means algorithm Hartigan & Wong (197)). Then we collect the anchor set A2 which has the same indicos in D2 as A in 1 For the environment Eg with two
visual configurations E. and Es we assemble an anchor set A3 = {aa (i) consists of images of from A and images of og from Az. In the next step, we train the two polices ( (o (. (0, [0
) 0, ,,A)) and
f E (of (of*), ). 92 E2 (o ofa A2) in E₁ and Eg with the Behavior Cloning algorithm Schaal (1996) separately.
Then initialize stitched policy f (of (o (of (of)),)))))))))) ( (o²"), "f Ag)- This stitched policy tested in the task T in Ez, Its performance is measured by its success rate.
C Quantitative Analysis of the Module Interface
The cosine and L2 pairwise distance shown in Fig. g measures the similarity between two latent represen- tations. For a group of states SE. = (s in the environment of task T, they are observed from two cameras and get two groups of observed images O1,E = {o),,T) and O2, = F We obtain the average pairwise distance between the latent representations of two visual encoders by calculating the mean of the pairwise distances across all input states:
[gg.l
doos
Σ
(1
So (of (oi,E.)) .h (db.)))) / [E,T),
(11)
[Sn.21
d12 = Σ diz (of (oi, 92 (o2,n,T) /Sg..l
(12)
ab is the cosine where of and of are the visual encoders for 01,E and 02 separately, Sc b) || similarity and dr2(p.q) = |p - 4/| is the L2 distance. Fig. g shows the cosine and L2 distances in all the experiments in the Push task. We record the distances data and the mean cosine and L,2 distances across all these experiments in Table S
D Influence of Using Decoder 1 and Decoder 2
This section aims to answer the question: Is there any difference between choosing which action decoder (action decoder v.s. action decoder 2)?
We carry out the zero-shot transfer in the Stack task with six different visual configurations. For each experiment, we try action decoder 1 and action decoder 2, and report their success rates side-by-side in Table We notice that the average success rates of docoder 1 and decoder 2 across the six visual configurations are close to each other for all the methods, and the maximum difference within 15%. We believe that this difference is generated by the randomness of the testing process, but not the systematic advantage of
22

---

Published in Transactions on Machine Learning Research (11/2024)
Cosine Distance
L,2 Distance
bascline
bascline
Masked
0.065
0.422
4.943
6.323
Zoom in
0.083
0.687
4.068
4.469
Blurred
0.043
0.982
4.937
5.828
Gaussian Noise
0.062
1.044
3.196
6.865
Camera Type
0.045
0.759
3.588
5.764
Camera Position
0.003
1.072
1.936
5.314
Mean
0.050
0.828
3.778
5.761
Table & Latent Representations Distances Data. The distances of the latent representations in all the experiments in the Push task are recorded. We also calculate the mean distances across all these experiments.
Mask
Zoom in
Blurred
Noise
Fisheye
Camera Position
Average
Deviu el al. 2017
Decoder 1
0.7:0.94
8.0#1.63
0.7:00.44
24.0=2.83
0:0:0:00
14.0+3.27
7.9
Decoder
5.312.19
0.0=0.00
6.722.10
1.321.89
Commistraci 2024 (linear)
Dooder
47.3-0.94
62.01-4.32
32.7+3.77
30.750.94
54.0=8.64
14.7+6.18
40.22
Decoder 2
39.3=1.11
56.74189
26.726.60
51.3:4.11
16.0=2.8 83
23.3:1.89
10.6
Concistruci (non-linear)
Decoder
10.0=1.63
12.0110.0
0.0:10:00
3.310.94
0.010.00
0.7=0.94
4.3
Decoder 2
5.3+0034
6.7=0.94
8.0 2.83
14.7+1.89
2.0::163
0.04:0000
6.1
PeS (ow/o diseat. loss)
Decoder
36.0111-45
10.71:111
22.773377
Decoder 2
31.3=2.49
52.716.60
28 28.01:3.7
26.71:1.0
32.7:3.40
19.314.11
31.8
Pol (w. 11 & 12 loes)
Decoder 1
92.7=0.94
98.0110.0
62. 72..6666
21.01:900
59.317.38
58.721.88
65.9
Decoder 2
72.7=3.77
36.041.61
88.7114.99
18.7
23.31:3.0
33.310.94
52.1
PeS
Decoder 1
91.7=0.91
96.720.91
91.011163
98.711.89
97.312.49
80.01:1.90
92.6
94.012.83
92.74:0.4
95.320094
Table 9: Action Decoder 1 V.S. Action Decoder 2. All the experiments are carried out with the Stack task. For the PeS method and other baseline and ablation methods, the difference in the average success rates of using action decoder 1 or action decoder 2 is within 15%. The choice of the action decoder does not have distinct influence on the zero-shot transfer success rates.
decoder to another decoder. This result suggests that the choice of the action decoder during the zero-shot transfer process does not make distinct difference for all the methods average.
If we look at the success rates of each visual configuration, we can see that the success rates difference of the PeS method is within 11%. The choice of decoder does not affect the reassembled policy's 's performance with PeS, and both docoders can load to very satisfying success rates over 85%. However, in some experiments with the baselines and the ablation methods (e.g. PeS (w. & loss)-Fisheyye, we can see a huge success rate difference. It indicates that the choice of different decoders has a random influence on the performance of the basclines and ablation methods for different visual configurations, but it doesn't 't affect the average success rates drastically.
E Anchors from failure trajectories
Mask
Zoom
Blurred
Noise
Fisheye
Camera Position
Average
100% success rate
94.7=0.94
96.7100.4
90.0:11.63
96.711189
97.3:12.99
40.0
92.6
51% success rate
98.7+0.94
95.34-3.77
46.0+-1.63
85.3+0.94
52.7+2249
84.0+3.27
77.0
6% success rate
49.3:16.80
48.714.11
76.748.06
98.710.94
44.7:13.40
98.0+11.63
69.4
Table 10: Anchors from Failure Trajectories. All the experiments carried out with the Stack task. We chose anchor datascts from three different datasets with K-means algorithm: (1) The dataset collected by expert agent, and contains 200 success trajectories. (2) The dataset collected used collect 200 trajectories, among which 34% successful trajectories and 46% failure trajectories. (3) The dataset collected poorly behaved policy with close to only 6% are successful trajectories and 94% are failure trajectories. average success rate decreases when the percentage of successful trajectories the dataset for anchor selection decreases.
semi-trained policy which has around 50% of success rate. It is
It is used collect 200 trajectories, among which
23

---

Published in Transactions on Machine Learning Research (11/2024)
This section aims to answer the question: Does the data collection process require some success trajectory? Or even failure/exploratory trajectories are still useful?
We use the Stack task to carry out the experiments. We collect a dataset with 200 successful trajectories and collect an anchor set 1 from this fully successful dataset. Then we train a policy to about 50% of training success rates and execute it to collect 200 trajectories during testing. 54% of these trajectories are successful and the rest 46% are failed. We collect an anchor set 2 from this semi-successful dataset. We also train a policy for only 1 minute that it has very low success rate. We use this poorly trained policy to collect dataset of 200 trajectories with only 6% of them being successful. We collect anchor set 3 from this dataset in which most trajectories fail. All the anchors are collected with the k-means algorithm as shown in Figure
We use these three different anchor sets for training the policies across the six visual configurations. The data set used for training is still the same datasot collected by an expert agent with 200 successful trajectorios, and the only difference is that the anchors are selected from data sets with different success rates. Table I0 reports the success rates. On average, the success rate decreases when the proportion of successful trajectories decreases for the anchor selection. In addition, unstable when the failure trajectory number during anchor selection increases. When there are no failure trajectories, the policy success rates are stably above 80%. In contrast, with the failure trajectories number increases, although there are still some cases where the trained policy can get over 90% of success rates, there appear more cases where the policy gets around 40% to 50% of success rates. These low-performance experiments make the average success rate decrease.
notice that the performance of the trained policy becomes
Since need to an expert dataset for training the policics, and it is small data set with only 200 trajectories that are easy to collect, we encourage the users of the PeS method to directly collect the anchor set from the training data set with K-means algorithm. Our experiment result shows that this anchor selection method can lead to more stable zero-shot transfer performance and a higher average success rate.
F Latent Representations Visualization
We visualize the latent representations of the Lift, Can, Stack and Door tasks. Among all the visual configuration changing experiments, we choose the camera position variation experiments. We first reduced the 256D representations to 3D with PCA (Hotelling) 1933) for visualization. Since the side view encoder of the policy is stitched to the policy at the position of its original front view encoder, we compare the latent representations of these two encoders.
In these visualizations, the red data points are the first 5 steps of images in each of the 200 games in the dataset of certain task. The blue data points are the last 5 steps of images, and the green data points are the 5 steps of images in the middle of each trajectory. Therefore, we have 1000 data points for each color which represent the starting, middle, and ending stages of a task.
The visualization results show that PeS can better align the latent space and force an approximate invariance of the latent representations. In contrast, the Devin al. (2017) baseline without adopting the relative representation and the disentanglement regularization leads to different latent representations between the two encoders, and they have an approximately isometric transformation relationship. These visualizations support our conclusions in the paper.
G Impact of Anchor Number
To study the impact of number of anchors on transfer performance, we picked challenging task, Stack, and reported the success rates with standard errors over 3 random seeds on all the different visual configurations in Table I We found that when the anchor nunber is loo small, the latent space of the visual encoder does not have enough capacity to capture effective visual representations while maintaining an approximate invariance at the same time. Therefore, the zero-shot transfer performance drops drastically. On the other hand, when the number of anchors becomes too large, there are more redundant anchors which are similar to each other. This will make disentangling the features at the latent space with the disentanglement loss
24

---

Published in Transactions on Machine Learning Research (11/2024)
Lift Task
Can Task
Pe5
Pe5 frre
Pe5
forts
Jide
Devin 2017 frrtwen
Devin et al. 2017 vew
Devin 2017 view
Devin et al, 2017 sice view
(a)
(b)
Stack Task
Door Task
Pu5
Dede et 2017 frotwen:
Devin et al. 2017 ude vew
Devin et 2017 froet vew
Devin at 2017 sick view
(c)
(d)
Figure 11: Latent Representations Visualization. We visualize the latent representations of the front-viow encoder of policy and the side-view encoder of policy 2 of the Lift, Can, Stack and Door tasks with the Camera Position changes in the visual configuration.
25

---

Published in Transactions on Machine Learning Research (11/2024)
harder, which can also lead to some drop of the performance. We empirically found that an anchor number between 256 to 512 can achieve optimal performance in our tasks.
Anchor Number
Masked
Zoom in
Blurred
Noise
Fisheye
Position
Average
1024
72.7100.44
56.0144.32
88.713.40
86.713.77
65.3:12.49
69.3112.49
73.1
512
90.713.40
94.710994
91.3110.4
92.745.73
95.313.40
88.719.84
92.2
256
94.7+0.94
96.7+0.94
90.0++1633
96.7+1.89
97.3+2.49
80.0+4.90
92.6
128
72.7+10.87
43.3+5.73
42.0+1.63
24.0+2.83
73.3+1.89
30.0+7.12
47.6
1.3+0.00
6.7+0.94
6.7+4.11
0.0++0.00
5.342.49
0.0++0.00
3.3
Table 11: Success rates of the Stack task with different anchor numbers
We also investigated the impact the anchor number on computational efficiency. We picked the Stack task and trained the policies a NVIDIA A6000 GPU with the batch size of 32. As shown in Table 12 larger anchor number will lead to larger model size, longer anchor searching time, and longer training time for each epoch.
Anchor Number
Model Size
Anchors Searching Time (s) Training Time Per Epoch (s)
1024
26,835,803
1455
182
512
25,720,667
922
107
256
19,1660099
570
66
128
24,884,315
288
64
24, 24,744,923
194
Table 12: network model sizes, anchors searching time, and training time per epoch of the Stack task with different anchor numbers
To sum up, either a too large too small anchor number can hurt the transfer performance, and larger anchor numbers can reduce the computational efficiency. It is important select an appropriate anchor number for each task.
H Impact of Disentanglement Loss Weight
Weights
Masked
Zoom in
Blurred
Noise
Fisheye
Camera Position
Average
0.0
34.0411.43
10.7:4.11
62.0=10.71
34.0=7.12
22.7=3.77
26.0:t:.32
31.6
0.0002
34.7+1.89
8.0+1.63
88.0+2.83
30.0#1.63
16.0=2.83
24.7#22.99
33.6
0.002
94.7#0.94
96.7::0.94
90.0+1.63
96.7=1.89
97.3=2.49
80.0+:.90
92.6
0.02
00.712.49
80.7110.87
92.0.14.90
88.0.3.27
5.23_5.25
70.0.14.32
83.8
0.2
35.3113.4
56.0.11.63
88.712.49
31.3100.5
48.0=1.63
16.0.14..22
45.9
Table 13: Success rates of the Stack task with diffferent disentanglement loss weight
We have also studied the effect of different weights of disentanglement loss. We picked the Stack task and tested different weights all the various visual configurations. When the weight is close to zero, the transfer performance gets close to that of the PeS (w/o disent. loss) ablation method and is significantly lower than the PeS full method. When the weight becomes too large, the optimization process leans too much to disentangling the latent features than imitating the expert behaviors. Therefore, the weaker imitation of the expert agent will also cause the performance drop. We empirically found that the optimal weight of the disentanglement loss in our tasks should be around the range of 0.002 to 0.02. The success rates with standard errors over 3 random seeds shown in Table 13
26

---

Published in Transactions on Machine Learning Research (11/2024)
Impact of Replay Trajectory Deviations
One limitation of PeS is that it requires a trajectory replay to collect the anchor images. However, in real- world applications, it is usually hard to accurately replay the exact trajectories. In order to understand how the trajectory deviation errors in replay could influence the performance of PeS, we conduct additional experiments to introduce trajectory deviations with different amplitudes.
We use the Stack task in the simulation test the effect of trajectory deviations. During the trajectory replaying, add random horizontal vector to every position point on the trajectory except for the gripper close or open action point. The length of the random vector to cause deviation is sampled from a uniform distribution within a certain range, and we test out the range options of tol em, O to 3 cm, and 0 to 5 cm. The direction of the deviation vector is randomly sampled within the x-y plane.
Amplitude
Masked
Zoom in
Blurred
Noise
Fisheye
Position
Lighting
Average
0 cm
94.7+0.94
96.7+0.94
90.0+1163
96.7+1.89
97.3+2.49
80.0+4.90
82.0+1.63
91.1
1
72.0+5.89
86.7+4.71
77.3+6.18
88.0+1.63
93.3+0094
74.7+7.36
78.0+4.32
81.4
3 cm
34.7+3.77
35.3+1.89
44.7+8.22
26.7+1.89
37.3+2249
19.3+7.36
23.0+2..3
31.6
5 em
12.042.83
14.7:4499
16.7:0.94
23.323.40
16.713.40
8.341.89
3.310.94
13.6
Table 14: Success rates of the Stack task with different replay trajectory deviation amplitudes
The experiment results are shown in the table below. When the trajectory deviation within the range of cm, The average performance of PeS drops by about 10%, but can still achieve 81.4% of average success rate, which is much higher than all the baselines that don't require trajectory replay and all the ablation methods that require trajectory replay without deviation. When the trajectory deviation goes up to the range of 3cm, PeS can achieve an average success rate of 31.6%, which is on par with the best performing baselines RT1 (29.5%) and Cannistraci el al. 2024 (linear) (34.6%). When the trajectory deviation goes up to very large range 5cm, the average success rate PeS drops to 13.6%.
In summary, although PoS currently requires trajectory replays for anchor selection, docsn't require very precise replay and can perform well within cm of replay error range.
27


Reviews:
[{"review_id": "DgvWWRD2wt", "reviewer": "Reviewer_p1mS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n1. The paper presents well and is easy to understand.\n2. The wide variety of experimental configurations in simulation and real world setup are well appreciated. \n3. The visualization of latent correspondence in Figure 8 and cosine distance of embeddings is reasonable evidence as to the claims for why the proposed approach outperforms baselines. Please also see related requested change 3.\n4. Covariance regularization is a simple to implement but very useful contribution here compared to previous approaches for this task.\n\n## Weaknesses\n1. The motivation is pitched as solving the problem of different institutions having varying robotics setups, and transferring policies can be difficult for example when camera configurations change. However, as noted by the authors in the limitations, replaying trajectories exactly to create duplicate anchors dataset in the new environment E_2 would be a severe bottleneck in many real world domains where controlling the environment variables to behave identically is difficult. Collecting a new dataset here and just training with BC might be preferable to doing stitching, which diminishes the value of this method.\n2. Can this stitching actually improve over the policy trained with little data that can be collected in the new setting directly? If there is a strong pretrained policy trained in one setting, and is being transferred to a new setting with a different camera configuration, how much does this transfer preserve the performance of the policy in the initial setting? Please see the requested change 1. below where I follow up on this.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a simple zero-shot approach to stitch the visual encoders of a robot policy with the visual encoders and decoder of another one. The method requires collecting a correspondent dataset of anchor images, and using a relative representation which projects the latents from the encoders to the anchor bases. The other important proposed improvement is the addition of batch covariance regularization of the encoder latents to the behavior cloning loss of policies. The stitching happens in a straightforward way, by projecting the new encoder to the bases of the correspondent anchor vectors. The paper evaluates the method compared to baselines across several simulated and real world robotics tasks, with many different camera configurations.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. An important comparison that seems to be missing is what the performance of policies 1 and 2 to be stitched are in the first place. Only the performance of stitched policies are shown for the experiments. I would assume policy stitching even when done well would generally result in some degradation of performance from the base policies unless one of the base policy decoders isn’t good for some reason, or the other camera configuration for that policy is faulty. If policy 2 is worse than policy 1 (suppose it is trained with less data), then would stitching a policy 2 observation to policy 1 decoder result in improved performance? This would be of interest for example if policy 1 is some pretrained open model available, and policy 2 can be cheaply trained in a new setting with a different camera configuration. It would be very useful if the authors can provide performance of the base policy 1 and policy 2 in all their experiments.\n2. I dont fully understand the single camera view experiment. If you train the two policies such that one of the encoders contains no information because of the completely dark input, won’t the decoder learn to completely just ignore the latents from that encoder? So for zero shot stitching where you use the policy 1 decoder, wouldn't you expect to get essentially the same performance as policy 1 before stitching? What was the performance of the policies before and after stitching? I would appreciate this section having a bit more clarification.\n3. Given that the main contribution of this work is dependent on this alignment of latents from different views after the relative representation step, it would be nice if latent visualization from other tasks could also be added to the main paper or atleast the appendix.", "broader_impact_concerns": "No broader impact concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "f9JzaoayUD", "reviewer": "Reviewer_rnZY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. *Method requires no training constraints*: Unlike many other methods that demand extensive pre-training on large datasets, fine-tuning on new tasks, or are limited to low-dimensional observations, PeS circumvents these constraints. By directly focusing on high-dimensional image observations, PeS can handle complex real-world robotic manipulation tasks. Additionally, PeS enables zero-shot transfer, meaning the system can generalize to novel visual configurations without the need for retraining or fine-tuning, which broadens its applicability to diverse environments and tasks without the overhead of additional training.\nMoreover, the use of disentanglement regularization, ensures that the latent features are independent of each other, promoting better alignment between visual encoders.\n\n2. *Strong results:* The experimental results clearly demonstrate that PeS consistently outperforms baseline methods, particularly in challenging tasks. Its ability to handle zero-shot transfer is especially notable in real-world tasks, where baseline methods struggle. In tasks like object manipulation or stacking, PeS achieves significantly higher success rates than competing approaches, showcasing its robustness and effectiveness in translating learned visuomotor policies from one context to another with minimal adaptation efforts.\n\n3. *Comprehensive evaluation:* PeS was subjected to a wide array of evaluations, encompassing numerous simulated and real-world tasks. Tests include varying camera configurations, such as occlusions, zoom effects, and blurred or noisy images, number of cameras, ensuring that PeS was rigorously tested under challenging visual conditions. Additionally, the inclusion of real-world experiments provides a clear demonstration of how the method performs outside controlled simulations, which showcases its adaptability to various visuomotor tasks.\n\n4. *Visual analysis with Grad-CAM:* The authors provide visual analysis of PeS’s performance through the use of Grad-CAM, highlighting which areas of the visual input the model focuses on when making decisions. The use of Grad-CAM allows for an intuitive understanding of why PeS succeeds where other methods fail, showing that PeS’s attention is directed to crucial task-relevant objects in the image.\n\n**Weaknesses**\n\n1. *Anchors collection:* The current requirement to replay trajectories for anchor selection imposes limitations on the method’s applicability, particularly in real-world settings where re-enacting the exact same trajectory can be time-consuming or impractical. A more flexible approach that enables automatic pairing of data across different visual encoders without needing identical trajectory replays could significantly broaden the method’s impact. \n\n2. *Baseline Comparisons:* While PeS outperforms existing methods, the baselines used for comparison could be more diverse.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents “Perception Stitching” (PeS), a novel approach that enables zero-shot transfer of visuomotor policies between environments with different visual configurations. The key contribution is the modularization and alignment of visual encoders via relative representations across policies to facilitate their reuse in novel perceptual configurations without retraining. The authors show that PeS disentangles perceptual knowledge from motion skills and demonstrates robust zero-shot transfer capabilities across different robotic tasks in both simulated and real-world environments. The method achieves higher success rates compared to baselines, especially in challenging tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "*Sim2Real Data Collection:* It would be interesting to leverage on a sim2real approach where simulation environments are used to collect additional data for training or enhancing the real-world controller. This would allow the method to gather diverse data in a controlled simulation environment, which could then be transferred to the real controller, potentially reducing the reliance on real-world data collection and improving zero-shot transfer performance in real-world tasks.\n\n*Explaining performance drops in real-world tests:* Some tasks in the real-world tests, especially Stack, exhibit a noticeable drop in performance compared to others like Reach or Push. Given that Stack involves multiple stages of precise manipulation—such as locating, grasping, lifting, and accurately placing an object—this complexity could naturally lead to more errors. However, the performance gap is significant, raising the question: Have you investigated the root causes behind this drop in performance?\n\n*A reinforcement learning approach that could be relevant to your research:* While your work focuses on behavior cloning, have you considered looking into the following paper [1]? It explores the use of relative representations to achieve zero-shot stitching in simpler environments like CarRacing and the Atari suite. It may offer insights applicable to your approach, despite the difference in learning paradigms.\n\n[1]: Zero-Shot Stitching in Reinforcement Learning using Relative Representations. https://arxiv.org/abs/2404.12917", "broader_impact_concerns": "The broader impact section is not present and not needed, according to my opinion.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "RVnghDV4Cg", "reviewer": "Reviewer_FoMS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. Novel approach to zero-shot transfer in visuomotor policies.\n2. Comprehensive evaluation in both simulated and real-world environments.\n3. Convincing ablation studies and comparisons with relevant baselines.\n4. In-depth analysis of latent representations and attention mechanisms using Grad-CAM.\n5. Demonstrated effectiveness in complex manipulation tasks.\n6. Potential for significant impact in robotics and transfer learning.\n\nWeaknesses:\n\n1. The method requires replaying trajectories to collect new datasets for anchor selection, which can be time-consuming in real-world settings.\n2. Limited exploration of scalability to larger datasets or more diverse tasks.\n3. Insufficient discussion of potential limitations or failure cases.\n4. While the real-world experiments cover several tasks (Reach, Push, Lift, Stack), they are still limited in scope, particularly lacking long-horizon planning or multi-step manipulation sequences.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Perception Stitching (PeS), a novel approach for zero-shot transfer of visuomotor robot policies across different visual configurations. The key contributions include:\n\n1. A modular visuomotor policy design that separates visual encoders and action decoders.\n2. A method for latent space alignment using relative representations, enabling the reuse of visual encoders.\n3. A novel anchor selection technique for imitation learning scenarios.\n4. The application of disentanglement regularisation to improve latent space alignment.\n\nThe authors demonstrate the effectiveness of PeS through comprehensive experiments in both simulated and real-world environments, covering various manipulation tasks and visual configurations. The method consistently outperforms existing baselines, particularly in complex tasks and real-world scenarios.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Critical changes:\n\n1. Provide a more thorough discussion of potential limitations and failure cases of the proposed method.\n2. While the authors compare with modular approaches (Devin et al. 2017 and Cannistraci et al. 2023), the paper could benefit from comparison or discussion with a recent end-to-end approach for zero-shot or few-shot transfer in visuomotor policies. Consider comparing with or discussing differences from an end-to-end method such as RT-1 (Brohan et al., 2022) or a similar recently proposed approach. This could highlight the trade-offs between modular and end-to-end architectures in the context of zero-shot transfer for robotic tasks.\n3. Clarify the novelty of each component of PeS in relation to existing techniques, particularly the use of relative representations and disentanglement regularisation.\n4. Provide an analysis and discussion of the method's sensitivity to hyperparameters, particularly the number of anchors and its impact on transfer performance and computational efficiency, and the weight of the disentanglement loss and its effect on the trade-off between transfer performance and representation quality. \n\nSuggested improvements:\n\n1. Expand real-world experiments to include a broader range of tasks and visual configurations, particularly those requiring long-horizon planning or multi-step manipulation sequences\n2. While the paper does explore different camera configurations, consider investigating the method's robustness to more extreme domain shifts, such as significant lighting changes or novel object appearances.\n3. Explore alternatives to k-means for anchor selection. How would density-based clustering methods or learnable anchor points affect performance and efficiency, especially in the presence of outliers. \n4. Clarify why certain tasks may benefit from the temporal dependencies that RNNs can capture, while the other tasks may not require this temporal modeling.\n5. Consider discussing whether data resampling methods that enhance the quality of behavioral cloning, such as Model-Based Trajectory Stitching (Hepburn & Montana, 2024), could potentially further improve the performance of your approach.", "broader_impact_concerns": "Given that the authors have already addressed these key points, there are no major omissions in their broader impact statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Overall, reviewers are positive about this work and its extensive empirical work. They all agree that the method is novel, interesting, well-supported, and motivates further research. While the reviewers are concerned that the work is fairly narrow, requires possibly specific kinds of data, and is more of a robotics contribution than a fundamental ML contribution, they agree that the method should be of interest to many in the field.
Decision: Accept (poster)