One-Shot Imitation Learning: A Pose Estimation Perspective
Authors: Pietro Vitiello
Kamil Dreczkowski
Edward Johns
Venue: CoRL 2023 Poster (2023)
Keywords: One-Shot Imitation Learning
Unseen Object Pose Estimation
Robot Manipulation

Abstract: In this paper, we study imitation learning under the challenging setting of: (1) only a single demonstration, (2) no further data collection, and (3) no prior task or object knowledge. We show how, with these constraints, imitation learning can be formulated as a combination of trajectory transfer and unseen object pose estimation. To explore this idea, we provide an in-depth study on how state-of-the-art unseen object pose estimators perform for one-shot imitation learning on ten real-world tasks, and we take a deep dive into the effects that camera calibration, pose estimation error, and spatial generalisation have on task success rates. For videos, please visit www.robot-learning.uk/pose-estimation-perspective.

One-Shot Imitation Learning: A Pose Estimation Perspective
Pietro Vitiello" The Robot Learning Lab Imperial College London pv2017@ic aca. uk
Kamil Dreczkowski" The Robot Learning Lab Imperial College London krd1 15@ic.0i... ac. uk
Edward Johns The Robot Learning Lab Imperial College London
e joons@imperiall ac. uk
Section 4.1:
Section Benchmarking on Real-Worid Tasks
Effect of Calibration Errors Task Success
Section 4.1 and 422
Section
Effect of Pose Estimation Errors on Task Success Rates
Spatial Generalisation
We model one-shot imitation learning as trajectory transfer, where we use unseen object pose estimation adapt end-effector trajectory from single demonstration, new scene where the object is in novel pose. In this paper, we are going to study this formulation through a series of four investigations shown in the above boxes.
Figure 1:
Abstract: this paper, we study imitation learning under the challenging setting of: (1) only single demonstration, (2) no further data collection, and (3) no prior task or object knowledge. We show how, with these constraints, imitation learning can be formulated
combination of trajectory transfer and unseen object pose estimation. To explore this idea, we provide an in-depth study on how state-of-the- art unseen object pose estimators perform for one-shot imitation learning ten real-world tasks, and we take deep dive into the effects that camera calibration, pose estimation error, and spatial generalisation have task success rates. For videos, please visit www.robot-learnnng.. wwwp://///oeepeaaaatiiee ceiiinatioonprrspeetiiie
Keywords: One-Shot Imitation Learning, Unseen Object Pose Estimation, Robot Manipulation
Introduction
Imitation Learning (IL) can be convenient and intuitive approach for teaching a robot how to perform task. However, many of today's methods for learning vision-based policies require tens to hundreds demonstrations per task [1, 2, 3, 4, 5, 6]. Whilst combining with reinforcement learning [7, 8, 9] pre-training similar tasks [10, 11, 12] can help, in this paper take a look at one-shot imitation learning. where we assume: (1) only a single demonstration, (2) no further
data collection following the demonstration, and (3) prior task object knowledge.
"Joint First Author Contribution
7th Conference on Robot Learning (CoRI. 2023), Atlanta, USA.

---

With only a single demonstration and no prior knowledge about the task or the object(s) the robot is interacting with, the optimal imitation is one where the robot and object(s) are aligned in the same way as during the demonstration. For example, imitating "scoop the egg" task (see Figure 1) could be achieved by aligning the spatula and the egg with the same sequence of relative poses as was provided during the demonstration.
But without any prior knowledge about the object(s), such as 3D object models, the reasoning re- quired by the robot now distils down to an unseen object pose estimation problem: the robot must infer the relative pose between its current observation of the object(s) and its observation during the demonstration, in order to perform this trajectory transfer [13] (see Figure 2). Unseen object pose estimation is already a challenging field within the computer vision community [14, 15, 16, 17], and these challenges are further compounded in robotics setting.
From this standpoint, are the first study the utility of unseen object pose estimation for tra- jectory transfer in the context of one-shot II., and we reveal new insights into the characteristics of such a formulation and how to mitigate its challenges. We begin our study by analysing how camera calibration and pose estimation errors affect the success rates of ten diverse real-world manipulation tasks, such inserting plug into a socket or placing a plate in a dishwasher rack.
Following this, we estimate the pose estimation errors of eight different unseen object pose estima- tors in simulation, including one based on NOPE [18], ofate-of-the-ar unseen object orientation estimation method, and one based on ASpanFormer [19], a state-of-the-art art correspondence estima- tion method. We then benchmark trajectory transfer using these eight unseen object pose estimators against DOME [20], a state-of-the-ar one-shot II. method, on the same ten real-world tasks as mentioned above. Our results not only show that the unseen object pose estimation formulation of one-shot IL is capable of outperforming DOME by 22% on average, but it is also applicable to a much wider range of tasks, including those for which third-person perspective is necessary [21].
Finally, we evaluate the robustness of this formulation to changes in lighting conditions, and con- clude our study by investigating how well it generalises spatially, an object's 's pose differs to its pose during the demonstration.
2 Related Work
Whilst there are many methods that study imitation learning with multiple demonstrations per task [3, 22, 23], in this section, we set our paper within the context of existing one-shot IL methods.
Trajectory Transfer. Trajectory transfer refers to adapting demonstrated trajectory to a new test scene. Previous work has considered how to warp a trajectory from the geometry during the demonstration to the geometry at test time [13], focusing on non-rigid registration for manipulating deformable objects. However, when relying on only a single demonstration, they displayed very local generalisation to changes in object poses, suggesting the need for multiple demonstrations in order to achieve greater spatial generalisation [13, 24, 25]. In contrast, we are the first to study unseen object pose estimation for trajectory transfer, which enables spatial generalisation from only a single demonstration.
Methods that require further data collection. Since one demonstration often does not provide sufficient information to satisfactorily learn a task, some methods rely on further data collection. approaches [26, 27] train visual servoing policy by collecting data
For instance, Coarse-to- around the object in a self-supervised manner. On the other hand, FISH [9] fine-tunes base policy learned with IL using reinforcement learning and interactions with the environment. While these approaches have their strengths, the additional environment interactions require time and sometimes human supervision. In contrast, modelling one-shot II. as unseen object pose estimation avoids the need for real-world data collection, hence enabling scalable learning.
Methods that require prior knowledge. Another way of compensating for the lack of demonstra- tion data is leverage prior task knowledge. For instance, many IL methods require access to object poses [28, 29, 30] or knowledge of the manipulated object categories [31], which is often impracti- cal in everyday scenarios. Another approach that assumes prior knowledge for learning tasks from a single demonstration is meta- meta-leariing [10, 11, 12, 32, 33, 34, 35]. In this paradigm, policy is pre-trained on a set of related tasks in order to infer actions for similar tasks from a single demon- stration. However, the applicability of the learned policy is limited to tasks closely related to the veta-trainnng dataset. Contrary to the meta-learning formulation of one-shot II., we approach it as unseen object pose estimation, which assumes prior knowledge and thus increases its generality.

---

RGB-D Image
Unseen Object Pose Estimation
Trajectory
Task Execution
What should offector
Trejectory
T6
Observation
?
T6
Demonstration
Deployment
Trajectory Transfer
Figure 2: Overview of our formulation for one-shot IL: (1) The robot receives a demonstration as an RGB-D image and an end-effector trajectory. (2) At deployment, the robot sees the object in new pose and must adapt the demonstrated trajectory accordingly. (3) To do so, the robot uses unseen object pose estimation to estimate the object transformation between demonstration and deployment. (4) It then applies this transformation to the demonstrated trajectory. (5) Ultimately, this aligns the end-effector with the same object-centric poses as experienced during the demonstration.
Methods which do not require further training or prior knowledge. DOME [20] and FlowCon- trol [36] are one-shot II. algorithms that assume no prior knowledge. The effectiveness of these methods hinges on their reliance on wrist-mounted camera, limiting their applicability where hand-centric observability is sufficient [21] and in which hand-held objects do not occlude wrist-mounted camera. In contrast, the one-shot IL formulation explored this paper applies to much wider spectrum of tasks, including those for which third-person perspective is necessary [21], such as the dishwasher task considered our experiments (Section 4).
tasks
3 One-shot Imitation Learning for Robotic Manipulation
In this work, we study one-shot IL under the challenging setting when there is: (1) only a single demonstration, (2) no further data collection, and (3) prior task object knowledge. This is an appealing setting to aim for, since encourages the design section, explore this from the perspective of object pose estimation. First, provide formu- lation of IL. Then, we model one-shot IL for manipulation as a trajectory transfer problem. Finally, introduce unseen object pose estimation, which underpins the trajectory transfer problem.
general and efficient method. this
Trajectory transfer, illustrated in Figure 2, involves the robot adapting a demonstrated end-effector (EEF) trajectory for deployment. This is done by estimating the relative object pose using unseen object pose estimation from pair of RGB-D images captured at the beginning of the demonstration and deployment phases, allowing for spatial generalisation of the demonstrated trajectory.
3.1 Imitation Learning Formulation
We observe demonstrations as trajectories T (T s), where represents the state of the system, T finite horizon, and the context vector. The state encompasses various measurements relevant to the task and should include sufficient information for inferring optimal actions. In the context robotic manipulation,
state could the context vector captures various information regarding the conditions the demonstration was recorded under, and can assume different forms, ranging from an image of the task space captured before the demonstration to task identifier in multi-task setting.
EEF and/or object(s) pose(s). Similarly,
Given dataset of demonstrated trajectories D {T. J , IL aims learn a policy n² that satisfies the objective:
arg min (((()). p(x)),
where (xx) and p(z) are the distributions of states induced by the demonstrator and policy respec- tively, and D(g, p) is a distance measure between and

---

3.2 Modelling One-Shot Imitation Learning as Trajectory Transfer
In this section, we model one-shot IL as trajectory transfer (see Figure 2), which we define as the process at test time of moving the EEF, or a grasped object, to the same set of relative poses, with respect to a target object, that it had during the demonstration. Let R define the frame of the robot and F. that of the EEF at time step I. A homogeneous transformation matrix TAB represents frame B expressed in frame A. During demonstrations, the robot receives instructions through teleoperation or kinesthetic teaching, which are defined as:
T -Demo = (XR (x Demo Demo),
and comprise of an RGB-D image DDemo of the task space captured before the demonstration using an external camera, and sequence of EEF poses from the demonstration, X Demo = Dememm \
=t=11
expressed in the robot frame.
With only single demonstration and no prior knowledge about the object the robot is interacting with, the optimal imitation can be considered to be where the robot and object(s) are aligned in the same way during the demonstration, throughout the task. For example, imitating grasping mug (see Figure 2). could be achieved by aligning the EEF in such a way that the relative pose between the EEF and mug are the same during the demonstration. Note that this also holds for more complex trajectories beyond grasping, such as insertion or twisting manoeuvres. Moreover, the task involves a grasped object, assuming that the latter is fixed and rigidly attached to the gripper, aligning the EEF will also align the grasped object.
Now, consider Equation I in the context of manipulation, where the optimal policy n² should result in replicating the demonstrated task state, x = TOg, where O is the object frame, at every timestep during deployment. This demonstrated sequence of BEF poses, expressed in the object frame, is defined as XB Deem = (ToE, T It=1, where
DDDemTTDem Demo (TRomo)
TOF, TOR
TRF.
and Tomo is the unknown object pose during the demonstration. During deployment, for op- timal imitation, n' should replicate XDemo given a novel unknown object pose T i.e. we would like To = ToE during every timestep of the interaction. Hence, the sequence of EEF poses (expressed in the robot frame) that aligns with the demonstration can be defined as T JT X (TTE 1i=1, where
R.F TTest TreesTDeee TOE,
(4)
Substituting Equation 3 into Equation 4 yields
The
RO
TT RE.
DemoTDerr Demo
RT,T Demo "TTTRem
(5)
where we define
"Ta = T'Iest Teeoo
(6)
RD
which represents the transformation of the object between the demonstration and deployment scenes, where we use the superscript A to indicate that "T, is expressed in the robot frame R.
This then leads to the crux of our investigation: the trajectory transfer problem, i.e. computing X from X R Demo, distils down to the problem of estimating the relative object pose between the demonstration and deployment scenes, given single image from each. Once this pose is estimated, controlling the EEF to follow this trajectory can simply make use of inverse kinematics. And given that we assume no prior object knowledge, the challenge becomes one of one-shot unseen object pose estimation, an active field in the computer vision community [14, 15, 16, 17, 18).
3.3 One-shot Unseen Object Pose Estimation for Trajectory Transfer
One-shot unseen object pose estimation is concerned with estimating the relative pose of novel object visible in two images. Formally, let C denote the frame of reference of a camera, and consider one image / Demo taken when novel object was pose TDemo, and second image /Test taken when that same object was at pose TCo One-shot unseen object pose estimation aims to estimate Demo
the relative transformation between the two object poses, CT2, that satisfies TTest CT₁ CO where we use the superscript to indicate that CT; is expressed in the camera frame C. Rearranging this equation yields
CT -TTest (Toomo)
Too TTTsst

---

Comparing Equations 6 and 7 reveals that TT and RT; both represent the relative object pose, but are expressed in different frames of reference. In fact, after estimating "Ta using one-shot unseen object pose estimation, we can find RTg from the following relationship derived in Appendix A:
TT TRC TT (TRC) TRC
TRO-1E TRC ToTCR:
where TnC is the pose of the camera in the robot frame. Hence, the trajectory transfer problem can now be solved by using one-shot unseen object pose estimation to calculate the value of TTg
Examining Equation 8 reveals that there two potential sources of error that could degrade the accuracy of "Ty and compromise performance during deployment. The first source of error is the error in extrinsic camera calibration TRC, and the second is the error in unseen object pose estimation itself T, both of which we discuss and study in the following sections.
Experiments
We now introduce ten represen- tative everyday robotics tasks that span broad range of complexities. As depicted in Figure these tasks placing one bowl into another (Bowls), inserting a plug into socket (Plug), grasping a mug by the handle (Mug), scoop- ing egg from pan (Egg), inserting bread into toaster (Toaster), inserting a plate into a specific slot in a dish rack (Dishwasher), inserting a cap into bottle (Cap), pouring a marble from kettle into mug (Tea), grasping a can (Can), and placing lid onto pot (Lid).
Stack Bowls
Insert Plug into
Grasp Mug
Scoop Egg
Insert bread Toaster
Put plate Dishwasher
Insert Cap
Pour Tea from kettle
Place Lid on pot
Into bottle
Grasp Can
Figure 3: The 10 real-world tasks we use for evaluation.
We begin this section by studying the effect of calibration and pose estimation errors on the suc- cess rate for each of these tasks (Section 4.1). We then consider eight unseen object pose estimation methods and estimate their pose estimation errors in simulation (Section 4.2). And finally, we bench- mark these pose estimation methods when used for trajectory transfer on the discussed real-world tasks (Section 4.3), we study the robustness to changes lighting (Section 4.3) and distractors (Ap- pendix G.2), and we examine the spatial generalisation capabilities of trajectory transfer (Section 4.4). For videos of our experiments, please visit our website.
4.1 Sensitivity Analysis of Task Success Rates to Calibration and Pose Estimation Errors
Correlating task success rates with calibration and pose estimation errors in the real world is chal- lenging. To establish a relationship between these errors and task success rates, we begin by pro- viding single demonstration via kinesthetic teaching from a last-inch setting. We then measure the correlation between the task success rate and the starting EEF position error prior to imitating the demonstration (see Appendix F.2). Finally, map starting EEF position errors either calibration or pose estimation errors using an empirically defined mapping (see Appendix B).
Specifically, for each considered task, we reset the object position, add position error to the starting EEF pose, replay the demonstration, and note if the task execution successful. This repeated 10 times for each position noise magnitude, with noise magnitudes starting from 0 mm and increasing in 2 mm increments until the success rate 0% over three consecutive noise magnitudes. This resulted in total of approximately 1,500 real-world trajectories in order to establish the relationship between EEF position errors and task success rates for the considered tasks.
Then, to empirically map calibration errors pose estimation errors to these starting EEF position errors, only one potential source of error was considered translation errors in calibration to starting EEF position errors, we assumed that rotation errors in calibration as well as rotation and translation errors in pose estimation are all zero, which isolates the effect of translation errors in calibration on task success rates.
time. For example, when mapping
5

---

The results for this experiment are shown in Figure 4, where each of the x-axes corresponds to a different type of error in calibration or pose estimation. The shape of the graph is iden- tical for each error type, be- cause there is a linear relation- ship between each of these rors and the error in starting EEF position. From these results, we can draw two main conclu- sions. Firstly, the task success rate for all tasks is more sen- sitive to errors in pose estima- tion than errors in camera cali- bration (for both translation and rotation errors), highlighting the importance of good pose esti- mation methods for this frame- work. Secondly, given typical performance of pose estimation and calibration methods, rotation errors are more problematic than position errors. For example, considering that rotation errors < 4 are more probable than position errors > 2 cm with today's s methods, we can see that a mere ~49 error in calibration or ~2 in pose estimation leads to an average success rate of ~50% whereas this same success rate would require a ~ em error in calibration or ~2 em in pose estimation.
1.0
Meen + Std.
Bowls
0.8
Can
Surress Iate
0.6
Dishwasher
Egu
Lid
Plug
Toaster
10
20
30
40
50
60
Transiation Froo Poss Fatimation [mm]
50
100
150
200
Transiation Error in Camera Callbration [mm]
Rotation Error Estimation [dog]
Rocacion Error Camera Calibration
Correlation between error magnitudes in either cali- bration or pose estimation, and task success rates, assuming a distance of 80 cm between the camera and the task space.
Figure 4:
4.2 Pose Estimation Errors of One-Shot Unseen Object Pose Estimation Methods
We now consider eight different unseen ob- ject pose estimation methods and evaluate their pose estimation errors in simulation. To this end, we generate simulated dataset consisting of I 100 image pairs of 55 differ- ent objects from Google Scan Object [37], using Blender [38] (see Appendix D.2).
Translation Error [cm]
Rotation Error [deg]
Class.
5.9 + 11.2
4.3 + 10.1
ASpan. (FT)
6.0 6.0 + 13.0
6.1 + 13.4
Reg.
+ 15.3
9.4 + 15.3
DINO
11.3
11.6 +
ASpan.
17.4
11.7 18.8
NOPE
18.4
18.4 17.2
The considered methods are: 1) ICP: We use the Open3D [39] implementation of point- to-point ICP [40]. ICP is given total of 5 seconds to make each estimate, giving it
ICP
14.3
32.4
GMFlow
28.4 24.6
27.6 24.5
Table 1: Simulation pose estimation errors and stan- dard deviations for eight different methods.
enough time try 50- 150 different initial-
isations. 2) GMFlow: We use GMFlow [41]
to estimate correspondences between two RGB images and solve for the relative object pose CTa with Singular Value Decomposition (SVD) [42] using the depth image. 3) DINO: We use DINO [43] to extract descriptors for pixels in the two RGB images, and use the SciPy [44] implementation of the Hungarian algorithm to establish correspondences. Again, the relative pose TT is obtained using SVD. 4) ASpan.: We use the pre-trained ASpanFormer [19] to establish correspondences be- tween two RGB images and estimate "T₁ using SVD. 5) ASpan. (FT): The ASpan. baseline, with model weights fine-tuned on a custom object-centric dataset generated Blender using ShapeNet and Google Scan Objects [37]. 6) NOPE: We use the pre-trained NOPE [18] model to estimate the relative object rotation from two images, and a heuristic that centres two partial point clouds to predict the relative translation. 7) Reg.: We train an object-agnostic PointNet++ regress rela- tive object orientations around the world's vertical axis from two coloured point clouds, using data generated in simulation and domain randomisation. We solve for the relative translation using heuristic that centres two partial point clouds. 8) Class.: This is equivalent to the Reg. baseline with the exception that PointNet++ is trained to classify the relative object orientation.
We also experimented with predicting the relative object translation from pairs of RGB-D images for the NOPE, Reg. and the Class. baselines. However, we found that a simple heuristic that centres partial point clouds for translation prediction had a similar performance, and thus used this during inference. We refer the reader to Appendix C for a more detailed description of all of these methods.

---

Plug
Pot
Toaster
Dishwasher
Mug
Egg
Bottle
Tea
Bowls
Can
Mean
TT (GMFlow)
10
40
20
40
20
20
20
60
60
29
TT (ASpanFormer (FT)
0
10
10
50
60
50
30
34
TT Former)ormer)
50
50
60
35
TT (DINO)
0
20
10
30
50
60
40
40
80
70
40
TT (NOPE)
0
10
0
50
0
70
90
100
70
100
49
DOME
0
10
80
0
100
70
40
90
70
100
56
TT (ICP)
10
70
40
60
80
100
100
100
100
74
TT (Class.)
20
100
100
100
80
76
TT (Reg.)
20
70
100
100
100
80
100
78
Mean
6.7
23.3
40
46.7
62.2
64.4
65.6
68.9
70
83.3
Table 2: Real-world success rates (%), from ten trials for each combination of method and task. TT (Trajectory Transfer) is used to distinguish all the previously discussed baselines from DOME [20].
The results for this experiment are shown in Table 1 (see Appendix E for an error definition and a discussion of these results). Although directly comparing these results to Figure 4 suggests that none of these baselines would be suitable for learning the considered tasks, in practice we found that translation and rotation errors in pose estimates are often coupled and partially cancel each other out, while Figure 4 only considers isolated errors. These observations are further reinforced by the strong performance we found with these baselines in our real-world experiments.
4.3 Real- World Evaluation
We now investigate if the trajectory transfer formulation of one-shot IL can learn real-world, every- day tasks, of varying tolerances.
Implementation Details For trajectory transfer, we use given unseen object pose estimator to estimate TT3 and Equations 8 and 5 alongside inverse kinematics align the robot with the first state of the demonstration. From this state, we align the full robot trajectory with the demonstrated trajectory, following Appendix F.2. In order isolate the object of interest from the background, we segment it from the RGB-D image captured before the demonstration and deployment using combination of OWL-ViT [45] and SAM [46]. Both segmented RGB-D images are subsequently downsampled to ensure compatibility with a given method. See Appendix F for further details.
Experimental Procedure We conduct experiments using 7-DoF Sawyer robot operating at 30 Hz. The robot is equipped with head-mounted camera capturing 640-by-480 RGB-D images. The task space is defined as a 30 x 75 em region on table in front of the robot, which is further divided into 10 quadrants measuring 15 x 15 cm each. During the demonstration phase, all objects are positioned in approximately the same location near the middle of the task space (see Figure 5), and a single demonstration is provided for each task via kinesthetic teaching from a last-inch setting. In the testing phase, the object is randomly placed within each quadrant with a random orientation difference of up to +:45 relative to the demonstration orientation. We test each method on a single object pose within each of the quadrants, resulting in 10 evaluations per method.
Results. The results for this experiment are shown in Table 2, with tasks ordered by mean success rate across methods and methods ordered by mean success rate across tasks. These results also include a comparison against DOME [20], a state-of-the-ar one-shot IL method. We observe that for DOME, the majority of failure cases are caused by its inaccurate segmentation of target objects. Its poor performance on the dishwasher task is attributed to the fact that the demonstration had to be started from further away, as DOME requires the object to be fully visible from a wrist-mounted camera. As result, DOME was beaten on average by three of the baselines.
The Reg. and Class. baselines had the best performance on average, likely due to the fact that their training data was tailored to object manipulation (see Appendix D. ICP's performance was affected by the partial nature of the point clouds, which sometimes caused to converge to local optimums. NOPE found itself out of its training distribution. Being trained on images with the object in the centre, NOPE can confuse relative translation for rotation when an object is displaced from the image centre. DINO uses semantic descriptors, which cause keypoints to be locally similar, translating into matches that are coarse and not precise. ASpanFormer was trained on images of entire scenes with many objects, hence expecting scenes rich with features. Therefore, predicting correspondences for a single segmented object causes this method to perform poorly. Meanwhile, we note that the fine-tuned ASpanFormer's performance drops significantly more with the sim-to-real gap than that of the Reg. and Class. methods. Lastly, GMFlow was found to poorly estimate rotations as the predicted flow tended to be smooth and consistent across pixels.
7

---

Robustness to Changes in Lighting Conditions. We now focus on trajectory transfer using re- gression, the best-perf method our real-world experiments, and analyse its robustness to changes in lighting conditions. To this end, we rerun the real-world experiment for this method while additionally randomising the position, luminosity, and colour temperature of an external LED light source before each rollout. The results from this experiment indicate that trajectory transfer using regression remains strong, with an average decrease in performance of only 8% when the lighting conditions are randomised significantly between the demonstration and test scene. We attribute this strong performance to the fact that the dataset used to train this baseline randomises lighting condi- tions between the two imput images as part of domain randomisation. For full details regarding this experiment and its results, we refer the reader to Appendix G.I.
4.4 Spatial Generalisation
Another insight that emerged from the real-world experi- ments is the impact of the relative object pose between the demonstration and deploy- ment on the average perfor- mance of trajectory transfer. When we aggregate the suc- cess rates across all baselines, tasks and poses within each of the quadrants, we notice decline in the success rate of trajectory transfer as the object pose deviates from the demonstration pose. In Figure 5, we display a mug at the approximate location where all objects were placed during demonstrations (labelled as DEMO), as well as mug at the centre of each of the quadrants. The opacity of the mugs located in the different quadrants is proportional to the average success rate for those quadrants, which is also displayed in white text. Note that whilst in this figure the orientation of the mug is fixed, experiments did randomise the orientations.
50%
60
58
44 %
44
DEMO
44% %
64
65% %
50%
47%
Figure 5:
The correlation between success rate and displacement from the place where the demonstration was given.
The cause of this behaviour lies in the camera perspective. Specifically, even when kept at fixed orientation, simply changing the position of an object will result changes to its visual appearance. Moreover, contrary to the effect of errors in camera calibration (see Appendix B. B.1), the changes in the visual appearance lessen as the camera is placed further away from the task space. These insights might seem intuitive, but for this same reason, they could be easily overlooked by researchers in the field. As a result, for optimal spatial generalisation, we recommend providing demonstrations at the centre of the task space, this minimises the variations in the object appearance when the object's pose deviates from the demonstration pose.
5 Discussion and Limitations
By formulating one-shot IL using unseen object pose estimation, we are able to learn new tasks with- out prior knowledge, from single demonstration and further data collection. We demonstrate this from a theoretical perspective and show its potential when applied to real-world tasks.
One limitation of this method is that we do not address generalisation to intra-class instances. Us- ing semantic visual correspondences [47] promising future direction here. Another limitation is the reliance on camera calibration. However, our analysis of calibration errors and real-world experiments do indicate good performance given typical calibration errors.
Although the proposed method has demonstrated to be very versatile in the types of tasks it can learn, in our setup it required static scene. This is because the robot arm often occludes the task space given the head-mounted camera on our Sawyer robot, making it not possible to continuously estimate the object pose during deployment. However, this is a limitation of the hardware setup and not fundamental limitation of the method. By optimising the camera placement for minimum occlusions, trajectory transfer could be deployed in a closed-loop and in dynamic scenes.
Finally, the current formulation is unsuitable for tasks that depend on the relative pose between two objects, where neither of them is rigidly attached to the EEF. For instance, pushing object close to another cannot rely on the rigid transfer of the trajectory, because the latter needs to be adapted according to the relative pose of the two objects. However, such tasks are fundamentally ambiguous with only a single demonstration, and multiple demonstrations would be required.
8

---

Acknowledgments
We would like to thank all reviewers for their thorough and insightful feedback, which had signif- icant impact on our paper.
References
[1] S. Young, D. Gandhi, S. Tulsiani, A. Gupta, Abbeel, and L. Pinto. Visual imitation made easy. In J. Kober, F. Ramos, and C. Tomlin, editors, Proceedings of the 2020 Confer- ence on Robot Learning, volume 155 of Proceedings of Machine Learning Research, pages 1992-2005. PMLR, 16-18 Nov 2021. URL. https::/proceeeeee http://ppoccedng..///eem mlr press/v155/ young21a. html.
[2] E. Jang, A. Irpan, M. Khansari, D. Kappler, Ebert, C. Lynch, S. Levine, and C. Finn. BC-z: Zero-shot task generalization with robotic imitation learning. In 5th Annual Conference Robot Learning, 2021. URL https://openreviiee wtt.://oorommmddddatt222tttt
[3] A. Brohan et al. RT-1: Robotics Transformer for Real-World Control at Scale. In Proceedings of Robotics: Science and Systems, Daegu, Republic of Korea, July 2023. doi: 16.100//SSS X02..XIX.025.
[4] T. Yu, T. Xiao, J. Tompson, A. Stone, S. Wang, A. Brohan, J. Singh, C. Tan. D. M, J. Peralta, K. Hausman, B. Ichter, and F. Xia. Scaling Robot Learning with Semantically Imagined Ex- perience. In Proceedings of Robotics: Science and Systems, Daegu, Republic of Korea, July 2023. doi:10. 15611/55S.SSS.. XIX.027..
[5] A. Stone, T. Xiao, Y. Lu, K. Gopalakrishnan, K.-H. Lee, Q. Vuong, P. Wohlhart, S. Kirmani, B. Zitkovich, F. Xia, C. Finn, and K. Hausman. Open-world object manipulation using pre- trained vision-language models. In 7th Annual Conference on Robol Learning, 2023. URL httts:///penreviess wwt./foruumm ?id=9al 1d
[6] Z. Zhao, V. Kumar, Levine, and Finn. Learning fine tine-grained bimanual manipulation with low-cost hardware. In K. E. Bekris, K. Hauser, S. L. Herbert, and J. Yu, editors, Robotics: Science and Systems XIX, Daegu, Republic of Korea, July 10-14, 2023, 2023. doi:10.15600 RSS.2023.XIX. X.X.2XXXXXX166...... URL https: / //doi org/18 10. 15607 2023. XIX. 016.
[7] Y. Zhu, Z. Wang. J. Merel, A. Rusu, T. Erez, S. Cabi, S. Tunyasuvunakool. J. KramAjr, R. Had- sell, N. de Freitas, and N. Heess. Reinforcement and imitation learning for diverse visuomotor skills. In Proceedings of Robotics: Science and Systems, Pittsburgh, Pennsylvania, June 2018. doi: 10.1005557/RSSS
[8] N. Das, S. Bechtle, T. Davchev, D. Jayaraman, A. Rai, and F. Meier. Model-based inverse reinforcement learning from visual demonstrations. In J. Kober, F. Ramos, and C. Tomlin, editors, Proceedings of the 2020 Conference on Robot Learning, volume 155 of Proceedings of Machine Learning Research, pages 1930-1942. PMLR, 16-18 Nov 2021. URL https: // //proceedings... mlr grres/vv15/dast
html.
[9] S. Haldar, J. Pari, A. Rai, and L. Pinto. Teach Robot to FISH: Versatile Imitation from One Minute of Demonstrations. In Proceedings of Robotics: Science and Systems, Daegu, Republic of Korea, July 2023. Nod:10..1607/RR. 200.XXX00002...0.0
[10] Y. Duan, M. Andrychowicz, B. Stadie, O. Jonathan Ho, J. Schneider, I. Sutskever, Abbeel, and W. Zaremba. One-shot imitation learning. In Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: / htttp:://proceeeinggs. httpssssssssspppppppaap.iiiiiiie faaaper_flee/ppppeeee braaper_files/papeer//aee 0.366600:55566777155776662601660
Paper .df.
[11] C. Finn, T. Yu, T. Zhang, P. Abbeel, and S. Levine. One-shot visual imitation learning via meta- learning. In Levine, Vanhoucke, and Goldberg, editors, Proceedings of the 1st Annual Conference on Robot Learning, volume 78 of Proceedings of Machine Learning Research, pages 357-368. PMLR, 13-15 Nov 2017. URL https: ptttts://proceeeee http///preeenn.../// wwweeiinnnsmm.......mmmmmee htt.:gg..........//eeemm.m presss/788/ finn17a 1a. html.

---

[12] т. Yu, C. Finn, S. Dasari, A. Xie, T. Zhang, P. Abbeel, and S. Levine. One-shot imitation from observing humans domain domain-adaptive meta-learning In H. Kress-Gaue Gazit, S. Srinivasa, T. Howard, and N. Atanasov, editors, Robotics: Science and Systems XIV, Carnegie Mellon University, Pittsburgh, Pennsylvania, USA, June 26-30, 2018, 2018. doi: 15007/5555 org/rss14//
XIV.002. URL http: //www//wwwwww/ hwwp::.boooiiii.. www.oooccppooceedinnng..ooo.
html.
[13] Schulman, Ho. Lee, and P. Abbeel. Learning from Demonstrations Through the Use of Non-rigid Registration, pages 339-354. Springer International Publishing, Robotics Research:
The 16th International Symposium ISRR, Cham, 2016. ISBN 978- -78-3-319-28872-7. doi:
URL /
1007 2110:/9778---119--87-------
[14] G. Pitteri, A. Bugeau, S. Ili,, and V. Lepetit. 3D Object Detection and Pose Estimation of Unseen Objects Color Images with Local Surface Embeddings. In Asian Conference on Computer Vision, 2020.
[15] M. Gou, H. Pan, H. Fang, Z. Liu, C. Lu, and P. Tan. Unseen Object 6D Pose Estimation: A Benchmark and Baselines. ArXiv, abs/2206. 1808, 2022.
[16] J. Wu, Y. Wang, and R. Xiong. Unseen object pose estimation via registration. In 2021 IEEE International Conference on Real-time Computing and Robotics (RCAR), pages 974 979, 2021. doi:10.1 109/RCAR52367. 99119999
[17] K. Park, A. Mousavian, Y. Xiang, and D. Fox. LatentFusion: End-to-End Differentiable Re- construction and Rendering for Unscen Object Pose Estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020.
[18] V. N. Nguyen, T. Groueix, Y. Hu, M. Salzmann, and V. Lepetit. NOPE: Novel Object Pose Estimation from a Single Image. arXiv preprint arXiv:2303. 13612, 2023.
[19] H. Chen, Z. Luo, L. Zhou, Y. Tian, M. Zhen, T. Fang. D. McKinnon, Y. Tsin, and L. Quan. Aspanformer: Detector-free image matching with adaptive span transformer. European Con- ference on Computer Vision (ECCV), 2022.
[20] E. Valassakis, G. Papagiannis, Di Palo, and E. Johns. Demonstrate Once, Imitate Imme- diately (DOME): Learning Visual Servoing for One-Shot Imitation Learning. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2022.
[21] K. Hsu, M. J. Kim, R. Rafailov, J. Wu, and C. Finn. Vision-based manipulators need to also from their hands. In International Conference Learning Representations, 2022. URL /ttpp::///ppeeeiiiiee www.:eee/ooooooeeee woroum/dddJJAAKp///P..
[22] S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-maron, M. Giménez, Y. Sulsky, J. Kay, J. T. Springenberg. T. Eccles, J. Bruce, A. Razavi, A. Edwards, N. Heess, Y. Chen, R. Hadsell, O. Vinyals, M. Bordbar, and N. de Freitas. A generalist agent. Transac- tions on Machine Learning Research, 2022. ISSN 2835-8856. URL https: /ppnreevee.. net/forum? d=d iikiikkkHjjjj Featured Certification, Outstanding Certification.
[23] V. Vosylius and E. Johns. Few-shot in-context imitation learning via implicit graph align- ment. In 7th Annual Conference Robot Learning. 2023. URL https://openrevierr net/forumiid dnn.rmmttiddddtttytttt.....
[24] A. X. Lee, A. Gupta, H. Lu, S. Levine, and P. Abbeel. Learning from multiple demonstrations using trajectory-aware non-rigid registration with applications to deformable object manip- ulation. 2015 IEEE/SSJ International Conference on Intelligent Robots and Systems (IROS), pages
2015. URL //api. semanticscholar arg/CorpusID:
[25] A. X. Lee, S. H. Huang, D. Hadfield Menell, E. Tzeng, and P. Abbeel. Unifying scene registra- tion and trajectory optimization for learning from demonstrations with application manipula- tion of deformable objects. 2014 IEEF/RSJ International Conference on Intelligent Robots and Systems, pages 4402-4407, 2014. URL. https: / / /api. wwwanti cacholar.. worgeoogg//ooo www./oorgpiiu.. IIr 161 1612092..
10

---

[26] E. Johns. Coarse-to-Fing Imitation Learning: Robot Manipulation from Single Demonstra- tion. In IEEE International Conference on Robotics and Automation (ICRA), 2021.
[27] N. Di Palo and E. Johns. Learning Multi-Stage Tasks with One Demonstration via Self-Replay In Conference on Robot Learning (CoRL), 2021.
[28] Y. Huang, J. Silvério, L. Rozo, and D. G. Caldwell. Generalized Task-Paramnn Parametramtte Skill Learning. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 5667-5474, 2018.
100...10//RRAA..1....... 88866079.
[29] Li, M. Cong, D. Liu, and Y. Du. Enhanced task parameterized dynamic movement primitives by gmm to solve manipulation tasks. Robotic Intelligence and Automation, 43(2):85- 95, 2023.
[30] Y. Hu, M. Cui, Duan, W. Liu, D. Huang, A. Knoll, and G. Chen. Model predictive opti- mization for imitation learning from demonstrations. Robotics and Autonomous Systems, 163: 104381, 2023.
[31] B. Wen, W. Lian, K. Bekris, and S. Schaal. You Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstration. In Proceedings of Roboties: Science and Systems, New York City, NY, USA, June 2022. doi: 15507//55..02222 V
[32] A. Bonardi, S. James, and A. J. Davison. Learning one-shot imitation from humans without humans. IEEE Robotics and Automation Letters, 5(2):3533-3533) 2020. doi: 109//RA.. 2020.2977835.
[33] X. Yang, Y. Peng. W. Li, J. Z. Wen, and D. Zhou, Vision-based one-shot imitation learning supplemented with target recognition via meta learning. In 202/ IEEE Interna- tional Conference on Mechatronics and Automation (ICMA), pages 10. 101109//CMA//203622002 9512607.
2021. doi:
[34] Z. Mandi, F. Liu, K. Lee, and P. Abbeel. Towards more generalizable one-shot visual imitation learning. In 2022 International Conference on Robotics and Automation (ICRA), pages 2434- 2444, 2022. doi: 10. 1191/1////66399 200..99922229122229
[35] M. Sieb, Z. Xian, A. Huang, O. Kroemer, and K. Fragkiadaki. Graph-structured visual imita- tion. In L. P. Kaelbling, D. Kragic, and K. Sugiura, editors, Proceedings of the Conference on Robot Learning, volume 100 of Proceedings of Machine Learning Research, pages 979-989, PMLR, 30 Oct-01 Nov 2020. URL https: htttpp://prrceeeinggg mlr. htt.eeeddnnn.......eeeeesssseeee ppress/v100/siee000ae
html.
[36] M. Argus, L. Hermann, J. Long, and T. Brox. Flowcontrol: Optical flow based visual servoing. In 2020 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7534-7541, 2020. 1oi: Www.////RSS574.20000000....
[37] L. Downs, A. Francis, N. Koenig, B. Kinman, R. Hickman, K. Reymann, B. McHugh, and V. Vanhoucke. Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Items. 2022 International Conference on Robotics and Automation (ICRA), page 2553-2560 IEEE Press, 2022. 10.: //doi.org//l 1 1101 010.099/IIAA4669.9000 2022. 2022.98118099 9811809.
URL https:
[38] B. O. Community. Blender 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018 URL http://wwwet blender org.
[39] Q.-Y. Zhou, J. Park, and V. Koltun. Open3D: A modern library for 3D data processing. 2018.
[40] P. Besl et al. A method for registration of 3-D shapes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 14(2):239-22) 256,, 1992. 11111.119993..11199999
[41] H. Xu, J. Zhang, J. Cai, H. Rezatofighi, and D. Tao. Gmflow: Learning optical flow via global matching. In Proceedings the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages
2022.

---

[42] K. Arun, T. S. Huang, and S. D. Blostein. Least- squares fitting of two 3-d point sets. 1EEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-9(5):998-00)) 1987. doi: 10.1 10.1109/TPAMI, 198 1967,4767965.
[43] S. Amir, Y. Gandelsman, S. Bagon, and T. Dekel. Deep VIT Features as Dense Visual Descrip- tors. ECCVW What Motion For?, 2022.
[44] Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski,
P. Peterson, W. Weckesser, J. Bright, S. J. van der Walt, M. Brett, J. Wilson, K. J. Millman, N. Mayorov, A. R. J. Nelson, E. Jones, R. Kern, E. Larson, C. J. Carey, i. Polat, Y. Feng, E. W. Moore. J. VanderPlas, D. Laxalde, J. Perktold, R. Cimrman, I. Henriksen, E. A. Quintero, C. R. Harris, A. M. Archibald, H. Ribeiro. F. Pedregosa, van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-272 2020. 10i:10. 1038/s4 1008///1192--00
[45] M. Minderer, A. Gritsenko, A. Stone, M. Neumann, D. Weissenborn, Dosovitskiy, Ma- hendran, A. Arnab, M. Dehghani, Z. Shen, X. Wang. X. Zhai, T. Kipf, and N. Houlsby. Simple open-vocabulary object detection with vision transformers. ECCV, 2022.
[46] A. Kirillov, E. Mintun, N. Ravi, H. Mao, Rolland, Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollar, and R. Girshick. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4015-4026, October 2023.
[47] D. Hadjivelichkov, S. Zwane, M. Deisenroth, I.. Agapito, and D. Kanoulas. One-Shot Transfer of Affordance Regions? AffCorrs! In K. Liu, D. Kulic, and J. Ichnowski, editors, Proceedings of The 6th Conference on Robot Learning (CoRL). volume 205 of Proceedings of Machine Learning Research, pages 550-560, 14-18 Dec 2023.
[48] R. Raguram, O. Chum, M. Pollefeys, J. Matas, and J.-M. Frahm. Usac: A universal frame- work for random sample consensus. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35()):2022-2038, 2013.
[49] M. A. Fischler and R. C. Bolles. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. In M. A. Fis- chler and O. Firschein, editors, Readings in Computer Vision, pages 726-740. Morgan Kaufmann, San Francisco (CA), 1987. ISBN 978-0-08-05158116. 0-0088778---08--555888---------1 arricle/p11/B8770000188888
doi:http::/doi..ogg http://www.cceccceeeeeeec//c///e
URL
[50] G. Bradski. The OpenCV Library. Dr. Dobb's Journal of Software Tools, 2000.
[51] H. W. Kuhn. The Hungarian Method for the Assignment Problem. Naval Research Logistics Quarterly, 2(1-2):83-99) 97, March 1955. 4b0110.1002ha.33800220000
[52] C. R. Qi. L. Yi, H. Su, and L. J. Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In Guyon, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Sys- tems, volume 30. Curran Associates, Inc., 2017. URL http:::: wwwp:eeeeeediieeeeeneneeeeeeeeee neurips. cc/ paper_ files
Luxburg. S. Bengio, H. Wallach, R. Fergus,
Paper pdff.
[53] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. 2017 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 23-30, 2017. doi: 10. 10.1109/ROS.22 8202133..
[54] A. X. Chang, Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, J. Xiao, L. Yi, and F. Yu. ShapeNet: An Information-Rich 3D Model Reposi- tory. Technical Report arXiv: 1513:00 [es. GRJ, Stanford University Princeton University -ooooa Technological Institute at Chicago, 2015.
12

---

[55] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In Y. Bengio and Y. LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http: //arxiv.org/ aaaaab///tt htss//12.... 6980.
[56] Heave, 2023. URL https: //oplyav/polyhaven...o com.
[57] E. Olson. Apriltag: A robust and flexible visual fiducial system. In 20// JEEE International Conference Robotics and Automation, pages 3400-3407, 2011. doi:10.1 109/CRA..011.. 5979561,
13

---

Appendix A Changing the Coordinate Frame of To
In the main paper, we state that the equation below changes the frame of reference of from the camera's frame C' to the robot's frame R:
TTg TRC TTTCR = TRC TT (TRC)"*,
where TRC is the pose of the camera in the robot frame. To derive this relationship. begin by referring to the definition of TT presented in Equation 7 of the main paper:
T3 co TTeaTee Toos (Teo
Rearranging this equation yields:
T Test TT
Demo
(10)
Additionally, we know that
Test T Too
co
(11)
By substituting Equation 10 into Equation 11, we obtain:
Th'st = TRC Tifest CO
-RC TTTTComo
(12)
However,
TCRTRRO Demo (Tnc) RO
CO
RO
By substitute this relationship into Equation 12 we obtain
T
Test RO
TRC TcTCRTRO
Demo
which can be rearranged to obtain
TRC C TTTTR Testt
T (TRo
OR
Since the right-hand side of this equation is consistent with the definition of RTs presented in Equa- tion 6 of the main paper, we conclude that
"T = Tne TTATER
Appendix B Mapping End-effector Errors to Calibration and Pose Estimation Errors
In Section 4.1 of the main paper, detail our experimental procedure to derive the correlation between task success rates and starting end-effector position errors prior to replaying a last-inch demonstration. We then mention that map these end-effector position errors to the correspond- ing calibration or pose estimation errors. this section, we describe our procedure for deriving the empirical mapping between end-effector position errors and the corresponding calibration (Ap- pendix B. 1) and pose estimation errors (Appendix B.2).
B.1 Errors Camera Calibration
We derive the empirical relationship between end-effector position errors and camera calibration errors using an experiment in which we repeatedly (1) sample starting end-effector pose for demonstration, Demo (2) sample a relative object pose between the demonstration and deploy- ment, CT, and (3) compute the accuracy with which we can calculate the corresponding starting end-effector pose for deployment, Tiest using trajectory transfer, after injecting controlled amounts of noise to the camera calibration matrix TRc-
Experimental Procedure We begin by calibrating a thad-mounted camera to Sawyer robot in the real world, obtaining an estimate of the camera pose:
Tec Rac))tc).
where Rac E 50(3) is a rotation matrix and n. E R& is translation vector. We then sample random end-effector pose RE which can be interpreted the end-effector pose at the beginning
14

---

of demonstration. Further, we sample random relative object movement expressed in the camera frame,
TTo = [ [o
where C Rs E (3) is a rotation matrix obtained from a randomly sampled rotation vector with a rotation magnitude sampled from the interval [0, 45]", and Cts E R is randomly sampled trans- lation vector with magnitude sampled from the interval [0, 0.4]m. The interval [0, 45) was chosen for consistency with our real-world experiments, while the maximum magnitude of the translation equal to half of the longest side of our workspace, and corresponds to the maximum translation that is allowable between a demonstration and deployment when the demonstration is given with the object at the centre of the workspace.
After sampling the hypothetical end-effector pose TRE and the relative object pose T, we use them together with the camera extrinsic matrix TRC to calculate the desired end-effector pose at test time,
Th
Test Test R
RR
using trajectory transfer (see Equations and in the main paper).
Once we compute the desired end-effector pose via trajectory transfer, we either perturb the rotation matrix of the camera calibration matrix, resulting in
TnC = RR RRC|tRR)
where R. E SO(3) is a rotation matrix obtained from a randomly sampled rotation vector with a predetermined rotation magnitude, or perturb the translation vector of the camera calibration matrix, resulting in
TR = |Ra + tz],
where t R is randomly sampled vector with predetermined magnitude.
After perturbing the camera pose, we estimate the desired end-effector pose,
Test RE
Test Test RE
CRE
using trajectory transfer and the noisy camera calibration matrix TRO. Finally, we calculate the error between the ground truth and estimated end-effector pose, Test and Test using the following equations:
temrr = Tist
-t
#Test| FRE
(Fl'sst Rerror = log R
We repeat this procedure for 1000 different randomly sampled relative object poses CT3 for hy- pothetical end-effector poses Demo with end-effector-to-camerr distances ranging from 0.2m to 1.2m, increasing in increments of 0.
We present the results from this experiment in Figure 6. The first two graphs focus translation errors in camera calibration and their impact on trajectory transfer. The bottom two graphs examine rotation errors in camera calibration and their influence trajectory transfer.
Interesting Findings The top graph of Figure reveals that translation errors in camera calibration lead to proportional errors in trajectory transfer. However, it is important to note that the magnitude of the errors the starting end-effector positions are smaller compared to the magnitude of the calibration errors. Additionally, from the second graph we find that translation errors in camera calibration do not affect rotation errors in trajectory transfer, which aligns with our expectations.
Moving on to rotation eπrоrs in camera calibration (third graph in Figure 6), we notice that the translation error in trajectory transfer depends not only on the rotation error but also on the distance between the end-effector and the camera. This relationship is logical since rotations occur around the camera frame, and the resulting translation induced by an error in rotation proportional to the distance from the frame of rotation. Hence, a camera should be placed as close as possible to the robot's workspace to attenuate the effects of rotation errors in camera calibration on trajectory transfer. Furthermore, we observe that rotation emors in camera calibration (last graph in Figure 6) correspond to proportional rotation errors trajectory transfer, although the latter are consistently smaller in magnitude.
15

---

End Effector to camera Distance [m]
0.0
0.3
06
0.9
12
1.5
1.8
21
24
2.6
30
31
35
39
4.1
4.4
51
5.2
5.6
58
6.1
65
0.0
0.3
96
0.9
12
14
1.7
2.1
2.3
2.6
2.9
3.2
3.5
3.8
4.1
4.4
4.7
5.0
5.2
5.6
5.8
6.2
6.8
0.0
96
9.9
12
1.5
1.8
2.1
2.3
2.6
3.0
3.6
3.7
4.1
5.2
5.6
5.8
6.2
6.5
End Effector Postion
6.1
6.5
633
0.0
0.9
12
1.5
2.4
2.7
3.6
3.9
5.0
5.3
5.6
6.1
6.5
0.6
0.0
0.3
0.6
0.9
1.2
1.5
1.7
2.1
2.4
2.6
2.9
3.3
3.5
3.9
4.2
4.4
4.7
4.9
5.2
5.5
5.8
6.2
0.3
05
0.0
0.3
0.6
0.9
1.2
1.3
1.8
2.1
2.4
2.6
2.9
3.2
3.5
3.5
4.1
4.3
4.8
5.0
5.2
5.7
5.9
6.2
0.0
90
120
130
340
273
180
100
210
220
Ent-Efictorr cameraa Distance [m]
0.0
3.3
0.0
0.0
0.0
0.0
0.0
0.0
0.0
013
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
End Lffeccor [cog]
0.0
3.3
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.3
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.3
0.0
0.0
0.0
8.3
0.0
0.0
3.0
0.0
6.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
3.3
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.3
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
6.n
0.0
0.0
R.0
0.0
10
J03
AED
1b0
180
LSU
Camera Callbration Thanslation [mm]
Distarce [m)
30
5.4
7.5
End ERecttr Posmin Emro [cm)
100
0.0
0.5
11
16
2.2
27
3.4
38
4.5
5.0
5.4
6.0
6.6
7.1
7.6
8.2
8.9
9.2
9.9
10.5
10.9
11.5
12.0
J0
03
0.0
0.5
1.0
1.5
2.0
23
3.1
3.6
4.2
4.6
5.1
3.6
6.2
6.7
7.2
7.5
8.2
8.8
9.4
9.7
30.2
10.5
11.3
05
7.4
6.7
8.7
10.5
4.0
Elfector
00
0.4
1.7
20
3.7
4.1
5.4
5.7
2.3
9.3
0.0
0.4
0.8
3.1
1.5
1.9
2.3
2.6
3.1
3.5
3.8
4.1
4.6
4.9
5.4
5.7
6.2
6.9
7.2
7.5
7.9
8.5
0.0
0.3
0.6
0.9
1.2
1.6
1.8
2.2
2.5
2.8
3.2
3.4
3.6
3.9
4.4
4.6
4.9
5.4
5.6
5.8
6.2
6.5
6.9
Effecctoo Rocation EnT [d0g]
End for Disisike
0.0
0.3
0.6
0.9
1.2
1.5
1.9
2.1
2.5
2.8
3.1
3.4
3.7
4.1
4.4
4.6
4.9
5.3
5.7
5.9
6.2
5.5
6.8
0.
0.3
0.8
0.9
1.2
1.0
19
222
23
2.8
3.1
3.7
4.0
4.3
4.7
4.9
5.6
5.9
0.3
8.4
6.85
1.5
2.8
0/6
0.0
0.3
0.6
0.9
1.2
1.6
199
2.2
2.5
2.8
3.1
3.4
3.7
4.1
4.3
4.7
4.9
5.3
5.5
5.9
6.2
0.
6.9
5.5
6.7
10
13
14
15
10
19
21
22
Camera Calibration Rotation Error (clgq]
Figure 6: The relationship between the end-effector-to-camee distance, the magnitude of the error in camera calibration, and the error in trajectory transfer (i.e. the emror in the calculated starting pose of the end-effector prior replaying demonstration during deployment).
16

---

Mapping starting end-effector errors to calibration errors To map starting end-effector position errors to translation errors extrinsic calibration, we fit first-order bivariate spline" to map end- effector-to-camdee to-mmmera distances and end-effector position errors to corresponding camera calibration translation errors (topmost graph of Figure 6). Similarly, to map starting end-effector position errors to rotation errors in camera calibration, we fit first-order bivariate spline to map end-effector-te camera distances and end-effector position errors to corresponding camera calibration rotation errors (third graph of Figure 6).
B.2 Errors in Pose Estimation
The empirical relationship between end-effector position errors and pose estimation errors is derived using a very similar experimental procedure to that outlined above. However, instead of injecting controlled amounts of noise to the camera calibration matrix TRC, we now instead inject noise to the relative object pose TTg
Experimental Procedure
Just like in the previous experiment, we begin by calibrating head- mounted camera to a Sawyer robot in the real world, obtaining an estimate of the camera pose TRC- We then sample random starting end-effector pose for demonstration, TRE and a relative object movement expressed in the camera frame, CTy, using the same procedure as in the previous experiment. After sampling the starting end-effector pose Demoo and relative object pose CTo, use them and the camera extrinsic matrix TRC to calculate the desired end-effector pose during deployment, TTest, using trajectory transfer (see Equation and in the main paper).
Once we compute the desired end-effector pose via trajectory transfer, we either perturb the rotation matrix of the relative object pose TT,, resulting in
'T, = [R [ R, 1t)
where R. E S0(3) is rotation matrix obtained from randomly sampled rotation vector with a predetermined rotation magnitude, or perturb the translation vector the relative object pose, resulting in
TT [ (R/
Ct + tt.
where t. E R a randomly sampled vector with a fixed magnitude.
We then estimate the desired end-effector pose,
TRE [ [Test]
using trajectory transfer and the noisy relative object pose CT and calculate the eπто between the ground truth and estimated end-effector poses, 2e and TTast in the previous experiment. We repeat this for 1000 different randomly sampled relative object poses TT for hypothetical demonstrated end-effector poses Demoo with wtl-eeffecootto--caaeee distances ranging from 0.2m to 1.2m in increments of 1.
RE using the same procedure as
We present the results from this experiment in Figure Just like in Figure 6, the first two graphs focus on translation errors in pose estimation and their impact on trajectory transfer, and the bottom two graphs focus on rotation errors pose estimation and their influence on trajectory transfer.
Interesting Findings The top graph of Figure 7 reveals that translation errors in pose estimates lead to equal errors in trajectory transfer. Additionally, from the second graph of Figure 7 we find that translation errors in pose estimates do not affect rotation errors in trajectory transfer, which aligns with our expectations.
Moving on to rotation errors in pose estimation (third graph of Figure 7), we notice that the trans- lation error in trajectory transfer depends not only on the rotation error but also on the distance between the end-effector and the camera. This relationship is expected since rotations occur around the camera frame, and the resulting translation induced by error in rotation is proportional to the distance from the frame of rotation. Furthermore, we observe that rotation errors in pose estimates equal rotation errors in trajectory transfer (bottom graph of Figure 7). Finally, note that the errors in trajectory transfer induced by errors in pose estimation are far greater in magnitude than those induced by errors in calibration.
We have experimented with fitting higher order bivariate splines. However, we have found the first-order spline to result in the lowest root mean squared error validation set.
17

---

0.0
35
10
15
20
23
3.3
3.5
4.0
43
5.0
5.5
6.0
6.3
1.0
End Etocoar camera Diztnnc [n]
10
200
45
6.3
Endiiifucto Position Error [cm]
2.0
8.5
6.
0.0
35
10
1.5
20
2,5
3.3
3.5
4.0
4.5
5.0
5.5
6.0
6.5
7.0
0.7
0.0
2.0
45
6.5
3.0
08
2.0
R3
0.3
65
0.0
3.5
1.0
1.5
2.0
2.5
3.3
3.5
4.0
4.5
5.0
5.5
6.0
6.5
7.0
0.0
0.5
10
20
25
30
4.0
45
5.0
555
6.0
63
12
0.0
3.0
0.0
0.0
0.0
0.0
0.9
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
End Efoccor mort Dizzance [m]]
0.0
0.0
0.0
Ené-Effector Rotatton Enoo [deg]
33
0.0
0.0
0.0
08
0.0
3.0
0.0
0.0
0.0
0.0
0.5
0.0
3.0
0.0
0.0
0.0
0.0
0.0
0.0
0.7
0.0
3.0
0.0
0.6
0.0
0.3
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.5
0.
0.5
0.0
30
00
10.0
0.3
0.0
0.01
010
11U
010
100
0.0
0.0
0.0
0.5
10
20
2.5
3.
4.0
45
5.0
6.0
65
7.0
Pose Est mation Transletion Eroor [cm]
0.0
0.8
1.5
3,7
45
5.3
6.9
7.5
3.3
9.0
10.5
10
100
0.0
0.7
1.4
2.7
3,4
41
4.8
5.5
6.2
6.9
7.5
3.2
8.9
9.8
-6
ErelSffocte Prilio Eroi Fomd
0.0
0.6
1.2
1.9
2.5
311
3.7
4.3
4.9
5.6
6.2
6.7
7,4
8.0
6.7
0.0
0.5
1.1
1.6
2.2
2.9
33
3.
4.4
4.9
53
3.5
5.5
Y.3
7.0
0.0
0.5
1.0
1.4
2,4
2.9
3.4
4.3
4.7
3.2
5,7
6.7
0.4
2.5
4.1
4.6
2.1
0.5
3.0
1.5
2.3
3.0
5.0
3.5
0.5
1.0
2.5
3.0
5.0
0.0
0.5
1.0
1.5
2,0
255
3.0
35
4.0
4.5
5.
5.2
5.0
8.5
7.0
Lnd Lifector Rortation bror deg)
0.0
0.5
1.0
1.5
2,0
2,5
3.0
355
4.0
4.5
5.
3.5
5.0
6.5
7.0
0.6
8.5
1.0
1.5
2.0
255
300
1.5
4.0
4.5
55
5.0
6%
0.0
0.5
3.0
1.5
2,0
25
3.0
355
4.5
4.5
5.0
3.5
5.5
5.5
7.0
00
85
1.0
1.5
7.0
2.5
3.0
355
4
4.5
5.
5.5
5.0
6.5
7.0
0.3
1.2
2,5
3.0
3.0
5.5
Estimation
Figure 7: The relationship between the mud-effecttorto-caaee distance, the magnitude of the error in pose estimation, and the error in trajectory transfer (i.c. the error in the calculated starting pose of the end-effector prior to replaying a demonstration during deployment).
18

---

Mapping starting end-effector errors to pose estimation errors To map starting end-effector position errors translation errors in pose estimation, we fit first-order bivariate spline' map und-effector-tt-canne distances and end-effector position errors to corresponding pose estimation translation errors (topmost graph of Figure 7). Similarly, to map starting end-effector position errors to rotation errors in pose estimation, we fit first-order bivariate spline to map end-effector-to camera distances and end-effector position errors to corresponding pose estimation rotation errors (third graph of Figure 7).
Appendix C
Unseen Object Pose Estimation Baselines
C.1 Iterative Closest Point
We use the Open3D [39] implementation of point-to-point ICP [40] directly estimate TTg We set the maximum distance between correspondences to 10cm, the maximum number of ICP iterations to 10, and allow ICP to try as many random initialisations as possible within seconds. This typi- cally resulted in ICP trying approximately 100 random initialisations. We have tried increasing the maximum number of ICP iterations, but observed that it is better to try more random initialisations than to have more ICP iterations per initialisation.
To initialise ICP, we first sample random rotation around the z-axis in the robot frame, and map this rotation to the camera frame using the camera extrinsic matrix. Sampling rotations in this way exploits the prior knowledge that objects are translated in 3DoF while being rotated only around the robot's s z-axis between the demonstration and deployment, which is the case for the majority of manipulation tasks. To obtain the initialisation for translation, we first rotate the first partial point cloud using the sampled rotation, and then centre the two partial point clouds. Finally, we add Gaussian noise to the translation component with standard deviation equal lcm.
C.2 Carrespondence-Esstiatttoo
Methods
We explore four different methods for estimating correspondences:
DINO:
We use Deep ViT features [43] to establish correspondences between two RGB images.
GMFlow: We use the pre-trained GMFlow [41] model predict optical flow, which is used to establish correspondences between two RGB images.
ASpanFormer: We use the pre-trained ASpanFormer [19] model to directly predict correspon- dences between two RGB images.
ASpanFormer (FT): We fine-tune the pre-trained ASpanFormer model using an object-centric dataset generated in simulation (see Appendix D.1), and use directly predict correspondences.
After establishing correspondences using any of these methods, we apply a filtering step to re- move outliers. To accomplish this, leverage the Universal Sample Consensus (USAC) algo- rithm [48], which is an extension and generalisation of the Random Sample Consensus (RANSAC) algorithm [49]. Specifically, we utilise the OpenCV [50] implementation of USAC and set the RANSAC reprojection threshold to pixels.
Once correspondences are established and outliers have been removed, all methods rely on Singular Value Decomposition (SVD) [42] to predict the relative object pose CT₃ using correspondences and their depth values.
C.2.1 DINO
Our implementation of the DINO correspondence estimator begins by cropping segmented RGB images around their segmentation masks, and resizing them so that their longest side measures 224 pixels, while maintaining their original aspect ratio. Next, extract DINO features from both seg- mented image crops using the pre-traines dino_vit8 model [43] with a stride of 4. To establish corre- spondences, we compute the cosine similarity between the descriptors of all patches and employ the Hungarian Algorithm [51]. Once correspondences are established, we discard all correspondences with cosine similarity lower than 0.1.
*We attempted fitting higher order splines but found the first-order spline to result in the lowest root mean squared error on validation set.
19

---

C.2.2 GMFlow
Our implementation of the GMFlow correspondence estimator begins by cropping the two (non- segmented) RGB images around their segmentation masks, and resizing them so that the width of the wider image measures 128 pixels, while preserving the original aspect ratio of both images. Then, the pre-trained GMFlow [41] model is used predict the optical flow between the two image crops, which we segment using the segmentation mask and map to correspondences.
C.2.3 ASpanFormer
AspanFormer [19] is Transformer- detector-free model for correspondence estimation. It uses estimated flow maps to adaptively determine the size of the regions within which to perform attention. The latter is done via their proposed Global-Local Attention (GLA) block, which allows them to achieve state-of-the-ar performance on a variety of matching benchmarks.
In this project, we use the pre-trained indoor model [19] that has been open-sourced by the authors of the paper. The correspondence estimation pipeline begins by cropping segmented RGB images around the segmentation masks. Both cropped images then resized so that their longer side measures 320 pixels, while preserving the original aspect ratio. For compatibility with the pre- trained model the shorter size is then padded with zeros, resulting in 320 320 images. Both resized RGB crops are then converted to grayscale images, which are then passed directly as input to the pre-trained model.
C.2.4 ASpanFormer (FT)
We further fine-tune the pre-trained weights of the ASpanFormer model on object-centric dataset which we have generated in simulation (see Appendix D.I). We have chosen to do this, as fine- tuning on an object-centric dataset should allow the model be more in-distribution when dealing with a robot manipulation setting compared to the original ASpanFormer model that was trained on feature-rich scenes with multiple objects.
C.3 Relative Orientation Estimation Methods
We consider three different methods for predicting the relative object orientation, around the robot's z-axis, from pair of RGB-D images. These methods include NOPE [18], recently proposed un- seen object relative orientation estimator, and PointNet++ [52]-based regression and classification models, that we have trained on an object-centric dataset generated in simulation (see Appendix D.I), relying on domain randomisation and data augmentation techniques for sim-to-real transfer [53].
We note that have tried using NOPE, and training both the regression and classification models to predict full 3DoF relative orientations, but found this to be not very accurate, leading to a poor performance of the final implementation in the real world. Hence, we leverage the prior knowledge that objects are going to be rotated only around the robot's z-axis between the demonstration and deployment make the problem tractable.
Once a model predicts the relative object orientation, we use a heuristic that applics this rotation to the first partial point cloud and then centres the two partial point clouds predict the relative translation. We have also experimented with training another PointNet++ [52] for learning a residual correction this heuristic but did not find this to bring significant improvements.
C.3.1 NOPE
NOPE is a recently proposed unseen object relative orientation estimator. We use the pre-trained weights provided by the author, trained on 1000 random object instances from each of the following 13 categories from the ShapeNet dataset [54]: airplane, bench, cabinet, car, chair, display, lamp, loudspeaker, rifle, sofa, table, telephone, vessel. During training, NOPE trains a U-Net to predict the embedding of a novel view of an object, given a reference image and a relative pose. Then at inference, it first takes as input support image of an object and predicts the embedding of that object under many relative orientations, effectively creating templates for template matching. Then given a query image of that same object, NOPE first computes its embedding and then finds the embedding 's distance to all the templates, giving a distribution over the possible relative orientations between the query and the support image. The predicted orientation will correspond to the most similar template.
20

---

To use NOPE to only predict the relative orientation around the robot's z-axis, we only sample rotation matrices that correspond to relative orientations around the robot's z-axis (which has the same direction as the object's z-axis). To he specific, we create 90 templates corresponding to rotations ranging from -44.5 to 44.5 spaced 1 apart. NOPE then encodes all of the templates as well as the query object orientation and selects the rotation whose encoding is most similar to that of the query according to the root mean squared error. Once we have the orientation predicted by NOPE, we use the heuristic described at the beginning of this Subsection to estimate the relative translation, completing the process pose estimation using NOPE.
C.3.2 Regression
We implement both the regression and the classification baselines to compare simpler approaches trained on an object-centric dataset (see Appendix D.I) to more sophisticated baselines trained on more general datasets, such DINO, GMFlow and the ASpanFormer. To this end, we implement a Siamese PointNet++ [52] encoder made of three set-abstraction layers and three linear layers, for total of ~18 M parameters. The encoder independently encodes the point clouds of target object obtained from the demonstration and the test scene. The two output embeddings are then concatenated into single 512 2--dimensional vector that we feed as input into 3-layer perceptron to fuse the information together and regress the object's rotation magnitude around the robot's 's z-axis. More specifically, the network returns normalised rotation, where -1 and correspond to -45- and 45 respectively. This multilayer perceptron is composed of layers with 256, 128 and I hidden node respectively, summing up to ~0.2 M parameters, for total model size of ~2 M parameters. The model was trained with the ADAM [55] optimiser and the mean squared error loss.
The input point clouds to the Siamese PointNet+++ are expressed in the robot's frame, have zero mean, and are downsampled to 2048 points. The features of each of the points include the point po- sition and colour. Since point positions used as point coordinates in the PoinNet++ architecture, including them as additional features may seem like including redundant information. However, we have found that doing improves performance in practice. The most likely reason for this is that PoinNet++ uses point coordinates to cluster points and aggregates features for the different clusters. Without including point positions features, this information would not be explicitly used derive the global point cloud feature vector.
C.3.3 Classification
This baseline is equivalent to the regression method described above, with the exception that the last layer of the 3-layer perceptron does not regress the rotation but instead outputs a probability distribution over 90 possible classes, where each class represents an angle between -44.5 and 44.5%, equally spaced 1 apart. This model has ~2 M parameters and was trained with the ADAM [55] optimiser using the binary cross entropy loss.
Appendix D Datasets
We begin this section by describing the object-centric dataset we have generated to train the regres- sion and classification models, and to fine-tune the ASpanFormer model (Appendix D.1). We then describe the dataset we have generated to benchmark all considered unseen object pose estimation methods in simulation (Appendix D.2). Finally, we describe the real-world dataset we used to de- termine when to stop training the regression and classification models to bridge the sim-to-real gap (Appendix D.3).
D.1 Training Dataset
The object-centric dataset used to train the regression and classification models, and to fine-tune the ASpanFormer model, was generated using Blender [38] and consists of ~ 226K image pairs. For each image pair, we first create scene and import a random object from either the ShapeNet dataset [54] or the Google Scan Objects dataset [37] The object is imported at a random pose and there is a 90% probability that its texture will also be randomised, by changing its colour and material properties, such as reflectivity. We then randomise the number, position, energy and strength of external light sources and render an RGB-D image and segmentation mask of the object.
After that, we perturb the object by rotating around the world's z-axis (perpendicular to the floor) by maximum of 45 either clockwise or counterclockwise and randomly displacing it somewhere
21

---

within the visible scene. Finally, we again randomise the position, energy and strength of the external light sources, and render another RGB-D image and segmentation mask. We conclude by recording the relative object pose between the two scenes. Additional data generation details are summarised Table and some examples of generated image pairs can be seen Figure 8.
Characteristic
Randomisation
Camera Extrinsic
In simulation, we set the camera extrinsic matrix equal to that for our real-world setup. However, to account for possible calibra- tion errors, for each scene simulation, we randomly perturb the extrinsic matrix by maximum of lcm in translation and 2 in orientation.
Object Type
Objects are chosen from ShapeNet with probability of 85% and from Google Scan with 15% probability. Within these two fam- ilies, the objects are sampled uniformly, but avoiding categories that were excessively out of distribution for object manipulation, such as airplanes, pistols or watercrafts.
Object Position
The object position is randomised by an arbitrary magnitude as long as the object remains fully visible to the camera. We use rejection sampling to ensure this.
Object Orientation
Once a random object pose generated to capture the first im- age, the object's ss orientation is changed by random rotation be- tween 45 and 45 around the world's z-axis.
Object Texture
The appearance of each different component of the object's model gets randomised with a 10% probability. In particular, if randomised, in 20% of cases, the component's colour is set to be monochromatic and the colour could be any of the follow- ing, according to these probabilities: normal (35%), dark (15%), very dark (5%), bright (15%), very bright (5%), pale (15%), dark pale (5%), bright pale (5%). If not monochromatic, with 20% probability the randomisation is applied as a texture, where the latter sampled uniformly from either the Haven [56] or MIL [11] texture datasets. Additionally. the material properties get randomised as well. More specifically with 50% probability the roughness of the material gets changed, with 30% probability the metallic properties get altered, with 30% probability the specu- larity properties, with 30% probability the material's anisotropy, with 15% probability its sheen, and finally with 5% probability the clearcoat property of the material is varied.
Lighting
When generating scene, the lighting conditions get ran- domised. There are three main modes the light can be in, whose parameters get further randomised. More specifically, the five modalities with their probabilities are: mostly ambient (5%), strong top shadow (30%), generic shadow (30%), very bright (5%), very dim (30%). Once the modality has been sampled, aside from the "strong top shadow" mode, either one or two light sources get placed in the scene. The light location, energy and ambient strength get then randomised for cach of the light sources independently, creating very diverse lighting conditions.
Table 3: Detailed explanation of the various randomisation strategies applied for the process of data generation.
22

---

Figure 8: Eight examples of image pairs generated in simulation. Each row contains two image pairs, showing the same object with random texture and colour. The object position and orientation, as well as lighting conditions are randomised between the two images. The background is not randomised as the objects will later be segmented out, making the choice of background irrelevant.
D.2 Evaluation Dataset
In Section 4.2 of the main paper, we evaluate the performance of eight different one-shot unseen object pose estimation methods in simulation. To evaluate these methods, we generated separate dataset that used for training, using objects that the ASpanFormer, regression and classification models were not trained on. We divided these objects into five categories, briefly explained hereafter. Examples for each of the categories can be found in Figure 9
1. Non-Symmetrical (Non-Sym.) are objects that are not symmetric around any of their axes. This category has 25 objects.
2. o--Symmetrical (co-Sym.) are objects that have an infinite degree of symmetry around their z-axis. This category has 10 objects.
3. xc-Symmetrical Geometry (so-Sym. Geo.) are objects whose geometry has an infinite degree of symmetry around the object's z-axis, but have non-symmetric texture.
4. W-Symmetrical (V-Sym.) are objects which have a finite degree of symmetry around their z-axis. This category has 10 objects. For instance, a cube has a rotation symmetry of order 4 around its 7-axis.
5. N W-Symmetrical Geometry (N-Sym. Geo.) are objects which have a non-symmetrical texture but whose geometry has finite degree of symmetry around the object's z-axis. This category has 5 objects.
Examples of the objects that could be found in each category are the following. 1) Non- Symmetrical: kettles, mugs, shoes, caps, etc. 2) oc-Symmetrical: ramekins, bowls, vases, cups, etc. 3) oc-Symmetrical Geometry: cans, tape, cylindrical medicine packages, etc. 4) N- Symmetrical: square plates, square bowls, boxes, chests, etc. 5) Lastly N-Symmetrical Geome- try: non-uniform boxes, sponges, cylindrical speakers, etc.
Overall, for each different object, we generated 20 image pairs, resulting in total dataset size of 1, 100 image pairs.
23

---

Figure 9: Examples of image pairs generated for the evaluation dataset. The top row shows two image pairs for non-symmetric objects. Second row shows image pairs for (left) an oo-Symmetrical object and (right) oc-Symmetrical Geometry object. Bottom shows image pairs for (left) an N N-Symmetrical object and (right) an N-Symmetrical Geometry object.
D.3
Real-Woo World Validation Dataset
We collect a real-world validation dataset to obtain a criteria for early stopping when training the regression and classification models. To this end, collect 71 image pairs of 7 different everyday objects, including two different toasters, one pan, one pot, a wooden box, a small black box and rigid plastic container. We collect the images mimicking the data generation process in simulation. We firstly place an object on the table along with removable AprilTag [57]. which allows us to initially record the ground truth pose and then remove the tag to collect the desired RGB-D image without the tag being visible. Subsequently. we perturb the object's pose following the same strategy as with the simulated scenes, and we record another pose and RGB-D image. Examples of the collected data can be found in Figure 10. When training model, we monitor its pose estimation error on this dataset to determine when stop the training.
Figure 10: The left and right double columns show pairs of images for a wooden box and red toaster respectively. Within one image pair, to go from one picture to the other the object has been randomly translated and then randomly rotated. The first row illustrates how the ground truth relative pose has been determined, that is through the use of AprilTags. These they could be hidden without affecting the pose the object, allowing for the capture the images shown in the second row.
carefully placed that
24

---

Non Sym.
Sy-Sym.
00-Sym. Geo.
N-Sym.
N-Sym. Geo.
Mean
Class.
8.4 14.5
0.8
1.8
6..3
3.9 4.8
5.9 11.2
ASpan. (FT)
8.1 + 16.3
3.9
3.2+3.9
6.9+13.8
2.0 4.7
6.0 13.0
Reg.
13.7 17.8
2.
1.2
1
9.7
9.8 + 15.3
DINO
21.5
6.9
3.7
20.2
5.9 13.3
11.3 18.2
Aspan.
15.3 1.332110
7.2
5.4
12.6
6.5
11.5 17.4
NOPE
25.3 16.9
3.9
1.7
24.5 1.5-155 15.4
23.1
18.8 17.4
ICP
40.1
3.7
14.2
9.4 17.3
9.4
14.3
29.3
30.8
GMFlow
30.9 = 23.7
21.7 26.1
20.9 15.5
30 + 26.2
31.5 25.2
28.4 24.6
Mcan
16.6
4.4
3.9
13.5
10.3
Table 4: Full results for the simulation benchmarking experiments regarding translation errors in pose estimation, expressed in centimetres.
Non-Sym.
00-Sym.
O0-Sym. Geo.
N-Sym.
N-Sym. Geo.
Mean
Class.
6.7
13.1
0.2 0.1 0.1
0.4 1.4
5.4
2.1 6,3
4.3 1 10.1
ASpan. (FT)
8.6 17.1
3.8
2.6 3.1
6.7
2.2
6.1 + 13.4
Reg.
13.7 18.0
0.7
0.3 0.1
11.4
9.3
9.4 15.3
DINO
16.4 22.9
5.7
3.1
19.6
12.4
11.6 18.9
ASpan
16.8 24.2
5.7
4.6
15.5
6.5111.
11.7 18.8
NOPE
25.4 16.1
0.9
0.4+14
23.8 14.4
23.2 - 15.7
18.4 17.2
ICP
24.1 + 43.4
2.6
+ 3.9
3.2 11.8
8.4 16.0
13.1 + 25.9
14.4
32.4
GMFlow
33.3 = 27.7
17.0 + 18.9
17.9 13.2
28.1 = 22.3
27.8 21
27.6 + 24.5
Mean
16.4
4.2
3.7
13.3
10.05
Table 5: Full results for simulation benchmarking experiments regarding rotation errors in pose estimation, expressed in degrees.
Appendix E
Benchmarking One-Shot Unseen Object Pose Estimators
In Section 4.2 of our main paper, we benchmark the eight unseen object pose estimation baselines introduced in Appendix C, on the simulated dataset described in Appendix D).2
Error Definition Consider a ground truth relative object pose between a pair of images, "To, and pose estimate "T,, We begin by calculating the transformation T. [R. [te] that maps the pose estimate to the ground truth pose, i.c.
RTs T RT,
T. RTs RT:;)
We then define the translation and rotation error as
Letoor
||tell2
Renor |log (Re)ll2
where log is the logarithmic map for the SO(3) group.
In practice, for object categories with an order symmetry (see Appendix D.2), there is always more than single ground truth pose Ts for each image pair. In those cases, we independently compute the translation and rotation error between the pose estimate and each the valid relative object poses, and consider the pair of errors corresponding to the ground truth pose that gives rise to the smallest rotation error.
For objects with an infinite degree of symmetry around the z-axis, we create 360 ground truth relative object poses for each image pair. That is, discretise the possible rotations around the z-axis into 360 bins, and find a relative object pose corresponding to each possible relative rotation. For objects with a degree of symmetry of around the z-axis, we create 4 possible relative object poses per image pair. Finally, for objects with degree of symmetry of 2 around the z-axis, we create 2 possible relative object poses per image pair.
Results In Table 4 we show the full results concerning the translation error in pose estimation for the individual object categories discussed in Appendix D.2. Similarly, do the same for rotation errors in Table 5.
Discussion From Tables 4 and 5, we can clearly see that on average, the higher the degree of sym- metry an object, the lower the error in the predictions of all methods. This is intuitive, as the larger the order of symmetry of an object, the larger the possible set of correct relative pose labels.
25

---

Surprisingly, predicting the rotation for the symmetrical geometry categories has turned out to be easier than for the corresponding categories where the visual textures are symmetric as well. How- ever, as we have only considered 5 and 10 objects each for these categories, these results may not be statistically significant.
Appendix F Trajectory Transfer Implementation Details
E1 Incorporating Inductive Bias
As mentioned in Appendix C.3, when training the regression and classification models, and when using NOPE [18] to predict 3DoF relative orientations, we found high pose estimation errors that compromised real-world performance. Hence, have chosen to train the regression and classifica- tion networks, and to use the pre-trained NOPE model, only to predict the relative object orientation around the robot's z-axis. This design choice is motivated by the fact that for most manipulation tasks, a test object translates in 3DoF while being rotated only around the world's 's z-axis between the demonstration and deployment, and the world's and robot's z-axes aligned.
To ensure a fair comparison between regression, classification, NOPE, and the remaining considered baselines, we also incorporate this predictive bias into their predictions. Specifically, consider relative pose estimate expressed in the robot frame (see Appendix A):
T L tss)
where RR3= E SO(3) is the relative orientation prediction and Rtg E R the relative translation prediction. To incorporate the inductive bias that the object only rotates around the robot's z-axis into such an estimate, we first convert R R& to Euler angles, set rotations around the non-z axes to zero, and convert back to a rotation matrix f Rss. Now, given " "Rs and an end-effector pose T Demoo RE that we would like align with an object at test time via trajectory transfer (see Equation 5 of the main paper), we adjust the translation component of TTy to account for the modification of R Ras using the equation:
Demo R RRt "RsRRE" + tā. RE R R&tRE
where Ri is the adjusted relative translation, and &o is the translation component of T RE The intuition behind this equation is that it ensures that the position of the end-effector after trajec- tory transfer with a modified pose estimate "T, = [ [ss] is the same as the position that would have been obtained when using the non-modified pose estimate RTs.
F22 Aligning the Full Trajectory
In theory, we could use trajectory transfer (see Equation 5 of the main paper) to solve for the full end-effector trajectory aligned with the object at the deployment pose and could track this trajectory using trajectory tracking. However, in practice, we have found that this resulted in very jerky tra- jectories. Hence, instead, we use trajectory transfer to only align the first end-effector pose of the demonstration with the deployment scene, and send the robot to that pose using inverse kinematics. From there, we align the full end-effector trajectory with the demonstrated trajectory by replaying the demonstrated end-effector velocities expressed in the end-effector frame. Although the robot does not realise these velocities instantaneously. in practice, we have found this to work sufficiently well and to perform better than using trajectory tracking system.
Appendix G Sensitivity to Non-Geometric Noise
In this section, we focus on trajectory transfer using regression for unseen object pose estimation, which was the best-performing method in our real-world experiments (see Section 4.3 of our main paper), and analyse its sensitivity to distractors and changes in lighting conditions and backgrounds.
G.1 Sensitivity to Changes in Lighting Conditions
To investigate the robustness of trajectory transfer changes in lighting conditions, follow the same experimental procedure as described in Section 4.3 of our main paper. That is, we divide a 30 75em region table in front of the robot into ten quadrants measuring 15 15cm and use the demonstrations collected when carrying out the main experiment to facilitate a direct comparison to the remaining results presented in the main paper.
At test time, for each quadrant, we randomly perturb the position, luminosity and colour temperature of an external LED light source (see Figure 11 for examples), and randomly place the test object
26

---

Figure 11: Examples of how the position, luminosity and colour temperature of an external LED light source have been randomly perturbed between different evaluations study the robustness of trajectory transfer using regression for unseen object pose estimation to changes in lighting condi- tions.
within that quadrant with random orientation between +450 of the demonstrated orientation. This results in ten evaluations per task.
Plug
Pot
Toaster
Dishwasher
Mug
Egg
Bottle
Bowls
Can
Mcan
Fixed Lighting Changes in Lighting
20
70
100
90
100
100
80
100
10
70
100
70
80
100
90
100
Table 6: Real-world success rates (%) of TT (Reg.) averaged over ten trials under fixed lighting
conditions and changes lighting conditions.
The results from this experiment are shown in Table 6. For reference, this table also includes the results for trajectory transfer using regression under fixed lighting conditions from our main exper- iment which is described in Section 4.3 of our main paper. As these results illustrate, trajectory transfer using regression displays a strong performance as lighting conditions are randomised be- tween the demonstration and test scene, with an average decrease in performance of only 8%. We attribute the strong performance of this baseline under changes in lighting conditions to the fact that the dataset used to train this baseline randomises lighting conditions between the two input images, alongside the hue, value and saturation the two images, as part domain randomisation (see Appendix C.3 and D.I).
G.2 Sensitivity to Distractors and Changes in Background
Figure 12: Example of the combined performance of Owl-ViT and SAM when segmenting toaster in cluttered scenes with different backgrounds. By isolating only the object of interest, the chosen unseen object pose estimator is unaffected by the mentioned changes.
27

---

Our formulation of trajectory transfer using unseen object pose estimation assumes segmented input RGB-D images. Hence, the robustness of the framework distractors and changes in the back- ground is only dependent on the used segmentation pipeline and not on the backbone pose estimator itself. our implementation, we use combination of OWI-VVT [45] and SAM [46] for the seg- mentation pipeline. That is, we first query OWL-ViT for a bounding box of the test object using language prompt "An image of a X", where X the category of the considered test object (e.g. can, mug. toaster, plug etc.). We then crop the RGB-D image using the output bounding box and pass the cropped RGB image to SAM obtain a segmentation mask of the target object. With this pipeline, have observed consistent good performance even under clutter. Examples of segmen- tations of the toaster in three different cluttered scenes with three different backgrounds can be seen in Figure 12.
28


Reviews:
[{"review_id": "rFs1LDZx7gl", "reviewer": "Reviewer_DHE8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "In this paper the authors aim to solve the problem of relative pose estimation given RGB-D images. The approach is imitation-based and it's constrained to learn only from one single demonstration per task.  A demonstration is defined as an RGB-D image and an end-effector trajectory. The problem is modeled as trajectory alignment problem and boiled down to finding a transformation matrix between the demonstration and the deployment. Intuitively, once the pose is estimated, the robot can \"repeat\" the demonstration once the estimated relative pose is applied. The paper is accompanied by impressive real world videos.", "summary_of_recommendation": "The paper presents impressive real-world results for the task of teaching a manipulation robot how to do a task with only one demonstration per task. The problem of learning a task is posed as a trajectory alignment problem, and many baselines are compared for solving it. I believe this paper can be useful for the community once the authors address the structural flaws I highlight in my review.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "The paper can be improved significantly, specially in terms of clarity and structurally (as written in the weakness part of my review). I would rather write this paper as a benchmark paper than as a contributing paper, since the technical contribution is very limited. ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strenghts:\n- The paper presents a simple approach that is able to render impressive results in real-world experiments. I think the paper can be useful for the community.\n- The paper presents extensive comparisons in simulation with 8 different backbones for pose estimation, which are carefully chosen, implemented and tuned.\n\nWeaknesses:\n- There seems to be little technical novelty in this work.\n-  The clarity of the paper can be significantly improved. The only part where an overview the method is clearly explained is in the caption of Fig. 2. In my opinion, the abstract and introduction don't address the contributions or the main methodology of the paper, but rather focus on why the problem is hard. The paper presents more prominently the four conducted investigations than the method itself, and presents the method as a byproduct of that. In my opinion, this structure could be clearer.\n- Even if the method presented seems to be quite simple, it shows that is effective. Is this the first method that poses the one shot imitation learning problem as a trajectory alignment problem? If so, mention it in the introduction.\n- It's not clear if the robot gets only one picture at the beginning of the deployment phase (open-loop) or if it constantly gets pictures until the end effector is aligned with the demonstration picture (closed-loop). Would be beneficial to explain this in the manuscript.\n- Section 4 lacks a clear motivation. These ablation studies seem to be one important contribution of the paper, but however are not motivated well enough in the section.\n- In Section 4, the order in which Fig. 3 and 4 appear compared to the order in which these figures are referenced seems to be swapped. Fig. 4 seems to be connected with the first part of Section 4, and this can be confusing for the reader.\n- Reading Table 1 and Table 2 can give the impression that this paper is a benchmark paper. Would maybe be easier to understand the contributions if the paper is presented as so.\n- I wonder why in Section 5.2 an ablation with respect to the rotation of the object (and not only the translation) is not presented.\n\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "lzpRjxdVji", "reviewer": "Reviewer_NRUS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a new framework for one-shot imitation learning. Specifically, the demonstration is composed of one third-person image of the scene at the beginning of the episode, and also the end-effector trajectory. At test time, the object may be placed at a different position and the robot should still perform the task.\n\nThe approach authors proposed is to perform relative object pose estimation between the test time image and the demo image, i.e. predict a homogeneous transformation from two RGB-D images. The transformation is then used to move the robot to a similar relative starting pose w.r.t the object of interest, after which replaying the demo should accomplish the task.", "summary_of_recommendation": "I recommend weak reject as this work lacks comparisons to related works and is not well motivated, despite relatively thorough ablations.\n\nUpdated to weak accept after rebuttal.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "*For the first point under Quality -> Cons:*\nProvide a more thorough empirical comparison or discussion compared to previous methods such as DOME. As noted in the previous section, the proposed method requires better motivation and justification.\n\n*For the second point under Quality -> Cons:*\nProvide a more detailed discussion of this limitation.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*Quality*\n\nPros\n- The paper performed a comprehensive analysis of 8 relative pose estimation methods. For each method, the paper reports both the pose estimation error and the success rate when deployed on a real robot.\n- The experiments covers 10 different tasks that covers diverse object interactions. 10 evaluation trials are performed for each task times each pose estimation method. However, the tasks all share similar attributes and may not reflect real-world challenges. See next paragraph for more detail.\n\nCons\n- The proposed framework is incremental to previous works e.g. DOME cited in the paper. The authors did a fair comparison on page 2 line 67, and acknowledges that:\n\n    \"The main difference [...] is that they focus on top-down settings and rely on a wrist-mounted camera. On the contrary, we [...] rely on an external camera, allowing the robot to constantly have a complete view of the workspace that cannot get occluded by held objects.\" \n\n    **Reducing the need of a RGB wrist camera while requiring a third-person RGB-D camera is not sufficient to motivate the new method.** The author also does not provide empirical comparison, or any detailed discussion, to justify why the proposed relative pose estimation approach is preferred compared to the visual-servoing approach of previous work.\n- **The proposed framework is limited to trajectories that involve a single (moving) object**, which hinders its usefulness in real-world scenarios. If there are two objects of interest in a single trajectory, it is unclear how the two transformations can be used to align the trajectory. The paper also does not include a explicit discussion of this limitation.\n\n*Clarity*\n\nThe presentation is clear.\n\n*Originality*\n\nThe work is original, though limited novelty.\n\n*Significance*\n\nThe empirical analysis is substantial in evaluating ablations of the proposed method, while it leaves out comparisons to previous works that tackle similar problems.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "h6fQo7a4r4h", "reviewer": "Reviewer_Sjn1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper studies the one-shot imitation learning problem as described as a zero-shot pose estimation and trajectory alignment problem. The authors study the effects of camera extrinsics calibration as well as pose estimation error on end effector error and evaluate several methods of zero-shot pose estimation over ten manipulation tasks. In this study, they train a regression and classification neural network to estimate the relative pose between two objects. In the study, these classification and regression models performed best over most tasks. Additionally, a spatial error analysis was performed over all methods to determine the correlation between object location on the table and task success. This showed that task success was correlated with proximity to the demonstration location, in the center of the table.", "summary_of_recommendation": "While I do think that evaluating one-shot pose methods on actual tasks gives a better understanding of their utility, I think the analysis beyond which worked best was lacking. A better analysis of what types of pose errors caused tasks failures and a more thorough look into each tasks error tolerance would have given the reader a better understanding of how to correlate estimator error and task success. With the limited scope of analysis and no added novelty, I can not recommend acceptance.\n\n(Post Rebuttal) With the new analysis, this work could be used to evaluate the accuracies needed for future pose estimation to complete these tasks. I raise my recommendation to weak accept. ", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "See review. ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Strengths\n## Task-centric evaluation of one-shot pose estimation methods\nThis paper evaluates several one-shot pose estimation methods over 10 different tasks. Many of these methods are only evaluated on pose or pixel level metrics. These metrics rarely directly map to end performance for robotic applications. \n\n# Weaknesses\n## Limited analysis\nThe analysis done on the effects of different types of errors on trajectory alignment tasks are limited. The effects of the noise analyzed in this paper are transforming errors though a chain of transformations. This does not give us a better understanding of what types of errors actually trigger failures in each type of problem. The spatial generalization analysis in section 5.2 is more along the lines of the analysis that would be useful, but it is relatively light in scope. Additionally, robustness to non-geometric noise, such as lighting was not analyzed. \n\n(Post Rebuttal) Authors greatly improved the analysis of the effects of pose errors and lighting changes on a variety of tasks. \n\n## No novelty\nThe described “new framework” is a PointNet++ used to regress or classify relative pose. This is basically a simplified version of DCP [1]. Without further details I can’t be sure how novel this approach is. I do understand that this is meant to be an analysis paper, novelty is less crucial, but it should be noted.\n\n[1] Wang, Yue, and Justin M. Solomon. \"Deep closest point: Learning representations for point cloud registration.\" Proceedings of the IEEE/CVF international conference on computer vision. 2019.", "extra_scores": {"impact": "1: Not very relevant to the robot learning community."}, "extra_text": {}}, {"review_id": "QuxVFAQdp7U", "reviewer": "Reviewer_cUKd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper phrases one-shot imitation learning as an unseen object pose estimation problem which allows use of a single demonstration to learn a new task. With only a single demonstration and no prior knowledge about the object the robot is interacting with, the optimal imitation can be considered to be where the robot and object are aligned in the same way as during the demonstration, throughout the task. Although the observation is appreciable, I guess this is already encoded in many policy generalization schemes via modulation of demonstrated policy based on observations. The interesting sensitivity analysis affecting the performance of the generalization is worth noting.", "summary_of_recommendation": "The paper has interesting insights about the pose perspective generalization of policy.\nHowever, I would like to see some amendments as mentioned in my comments above.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "The paper should compare and establish some quantitative metric for comparison against established pose based skill learning in robotics e.g. \n B. Wen, W. Lian, K. Bekris, and S. Schaal, “You Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstration,” in Proceedings of Robotics: Science and Systems, New York City, NY, USA, June 2022.\nE. Valassakis, G. Papagiannis, N. Di Palo, and E. Johns, “Demonstrate once, imitate immediately (dome): Learning visual servoing for one-shot imitation learning,” in 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2022, pp. 8614–8621.\nAnd additionally also distinguish it from the other skill generalization like DMPs and TP-GMM which also uses pose based generalization of policies. \n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths: \nThe paper addresses a very important problem in robot learning from demonstration. The generalization of learnt policy in new situations is formulated as A Pose Estimation Perspective with interesting insights on sensitivity analysis of related factors.\n\nThe experiments on real robot demonstrate the validity of the approach\n\nWeakness:\nThe significance of the proposed is not so well placed with respect to the state-of-the-art. The experimental test cases are relatively simple (linear top down motion)\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a new method to tackle the challenging problem of one-shot imitation learning, which is formulated as a trajectory alignment problem. After the authors' rebuttal, most concerns of the reviewers have been addressed, and majority reviewers agree on accepting the paper.

Due to concerns of the reviewers on the lack of related work and comparison of previous works, I recommend the paper as a poster presentation.
Decision: Accept (Poster)