6D Object Pose Tracking in Internet Videos for Robotic Manipulation
Authors: Georgy Ponimatkin
Martin Cífka
Tomas Soucek
Médéric Fourmy
Yann Labbé
Vladimir Petrik
Josef Sivic
Venue: ICLR 2025 Poster (2025)
Keywords: 6DoF pose estimation
robotic manipulation from video
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: A method to estimate 6D pose and trajectory of an object in the Internet video

Abstract: We seek to extract a temporally consistent 6D pose trajectory of a manipulated  object from an Internet instructional video. This is a challenging set-up for current 6D pose estimation methods due to uncontrolled capturing conditions, subtle but dynamic object motions, and the fact that the exact mesh of the manipulated object is not known. To address these challenges, we present the following contributions. First, we develop a new method that estimates the 6D pose of any object in the input image without prior knowledge of the object itself. The method proceeds by (i) retrieving a CAD model similar to the depicted object from a large-scale model database, (ii) 6D aligning the retrieved CAD model with the input image, and (iii) grounding the absolute scale of the object with respect to the scene. Second, we extract smooth 6D object trajectories from Internet videos by carefully tracking the detected objects across video frames. The extracted object trajectories are then retargeted via trajectory optimization into the configuration space of a robotic manipulator. Third, we thoroughly evaluate and ablate our 6D pose estimation method on YCB-V and HOPE-Video datasets as well as a new dataset of instructional videos manually annotated with approximate 6D object trajectories. We demonstrate significant improvements over existing state-of-the-art RGB 6D pose estimation methods. Finally,  we show that the 6D object motion estimated from Internet videos can be transferred to a 7-axis robotic manipulator both in a virtual simulator as well as in a real world set-up. We also successfully apply our method to egocentric videos taken from the EPIC-KITCHENS dataset, demonstrating potential for Embodied AI applications.

Published as a conference paper at ICLR 2025
6D OBJECT POSE TRACKING IN INTERNET VIDEOS FOR ROBOTIC MANIPULATION
Georgy Ponimatkin 1.2
Martin Cifka 122
Tomás Souček'
Médéric Fourmy'
Yann Labbe
Vladimír Petrík'
Josef Sivic!
Czzec Institute of Informatics, Roboties and Cyberneties, Czech Technical University in Prague Faculty of Electrical Engineering. Czech Technical University in Prague
3 H Company
ABSTRACT
We seek extract temporally consistent 6D pose trajectory of manipulated ob- ject from an Internet instructional video. This is a challenging set-up for current 6D pose estimation methods due to uncontrolled capturing conditions, subtle but dynamic object motions, and the fact that the exact mesh of the manipulated object is not known. To address these challenges, we present the following contributions. First, we develop new method that estimates the 6D pose of any object the input image without prior knowledge of the object itself. The method proceeds by (i) retrieving a CAD model similar to the depicted object from a large-scale model database, (ii) 6D aligning the retrieved CAD model with the input image, and (iii) grounding the absolute scale of the object with respect to the scene. Sec- ond, we extract smooth 6D object trajectories from Internet videos by carefully tracking the detected objects across video frames. The extracted object trajecto- ries are then retargeted via trajectory optimization into the configuration space of a robotic manipulator: Third, we thoroughly evaluate and ablate our 6D pose esti- mation method on YCB-V and HOPE-Videe datasets as well as a new dataset instructional videos manually annotated with approximate 6D object trajectories. We demonstrate significant improvements over existing state-of-tth--ar RGB 6D pose estimation methods. Finally, show that the 6D object motion estimated from Internet videos can be transferred to a 7-axis robotic manipulator both in a virtual simulator well in real world set-up. We also successfully apply our method to egocentric videos taken from the EPIC-KITCHENS dataset, demon- strating potential for Embodied Al applications.
DRIT
(a) input video
(b) retrieved mesh
(c) 6D pose trajectory
(d) robot trajectory
Figure 1: Robotic manipulation guided by instructional video. (a) Given instructional video from the Internet, our approach: (b) retrieves a visually similar mesh for the manipulated object from large mesh database, (c) estimates the approximate 6D pose trajectory of the object across vidco frames, and (d) transfers the object trajectory onto a 7-axis robotic manipulator.
INTRODUCTION
In this work, we make step towards learning robotic object manipulation skills from Internet in- structional videos, as illustrated Figure!] This is a challenging problem as Internet instructional
"Equal contribution.

---

Published as a conference paper at ICLR 2025
videos present uncontrolled capturing conditions where people manipulate large variety of objects often in a dynamic fashion. While humans can easily learn even complex manipulation tasks from one more instructional videos, building machines with similar cognitive and sensory-man motoor capa- bilities remains a major open challenge. Yet, such capabilities would enable tapping into millions of instructional videos available online (Miech et al. (019)).
Current methods for learning object manipulation skills from instructional videos have focused on learning manipulation for stick-like objects (e.g., hammer, scythe, or spade) (Li et al. 2022 Zo- rina et al. 202T), require tens or even hundreds of training examples captured from multiple view- points (Sermanet al. 2018) or focus on recovering a static pose of the manipulated object (Patel et al. [222 Wu et al. 2024). In contrast, we leverage the recent progress in 6D object pose esti- mation and object tracking, and develop novel method for 6D object pose estimation that allows for extracting temporally consistent 6D) object trajectories from uncontrolled Internet videos without any prior knowledge about the object itself. Despite being simple and training-free, the method can extract object trajectories that can be transferred to robotic manipulator, making significant step towards large-scale learning of robotic manipulation skills from Internet's ' instructional videos.
In detail, we make the following contributions:
First, we develop a method that estimates the 6D pose of any object in an image without prior knowledge of the object itself. The method works by (i) retrieving a CAD model similar to the object in the image from a large-scale database, (ii) aligning the retrieved CAD model with the object, and (iii) grounding the object scale with respect to the scene.
Second, we extract smooth 6D object trajectories from Internet videos by carefully tracking the detected objects across video frames. The extracted 6D object trajectories are then retargeted via trajectory optimization into the configuration space of a robotic manipulator.
Third, we thoroughly evaluate and ablate our 6D pose estimation method on YCB-V and HOPE- Video datasets as well as new dataset of instructional videos manually annotated with approx- imate 6D object trajectories. We demonstrate significant improvements over existing state-of- the-art RGB 6D pose estimation methods.
Finally, we show that the 6D object motion estimated from Internet vidcos can be transferred to a 7-axis robotic manipulator both in virtual simulator as well as in a real world robotic set- up. Additionally, we apply our method to egocentric videos taken from the EPIC-KITCHENS dataset, showing our method can be useful in various Embodied Al applications.
2 RELATED WORK
Object instance detection and mesh retrieval. Image based pose estimation methods are usually decomposed into successive steps: object instance detection, often paired with mesh identification. followed by pose estimation. When the depicted objects all belong to a relatively small dataset (Xi-
ang al. 2018 Hodan et al. 2017: Tyree et al. 2022 most methods train off-the- 2D object detector, such as Faster
Ren et al. (2016); Li et al. (2020). MaskRCNN (He et al.]
2017 Labbé al. (020) YOLOVS (Jocher al. 2023 Wang al. 2021). using synthetic train-
ing data (Denninger et al. 2023). These supervised detectors achieve very high performance on standard benchmarks (Hodan et al. 2024) but are limited by the need of retraining for each new object instance. More recently. open-ended object detectors have appeared thanks to the advent of powerful foundation models (Ke et a 2023 Oquab al. 2024). For given set of textured meshes, CNOS (gguyen et al. 2023) proposes an onboarding phase where the known meshes are rendered across a set of viewpoints that are encoded with powerful image features such as (Oquab et al., 2024). At run time, candidate objects are segmented in the input RGB image using an object segmenter, such as SAM (Kirillov et al. 2023) Zhao al. 2023). and encoded using the same image features. The depicted object instance is then matched to one of the known meshes whose image encoding is the most similar with the instance features across views. Recent advancements include reducing the number of false detections (Lu et al. 2024) by improving the instance segmentation (Nguyen al., 2023) using (Liu et al.; 2024b), or developing better view-level feature aggregation techniques such as Foreground Feature Averaging (Kotar al. 2023).
Model-based 6D pose estimation. Recent progress in deep-learning based 6D pose estimation has been focused on increasing precision of alignment-based methods (Wang al. 2021 Labbé al.,
2

---

Published as a conference paper at ICLR 2025
2020).
This progress was made possible by the availability of large scale object databases (Downs)
et al.
2022 Deitke et al.] 2023), procedural rendering software to produce synthetic training
data
(Denninger et al. 2023). and well-maintained public test datasets and benchmarks (Hodan
eet al.
2020). The recent results of the BOP challenge indicate that the metrics on 6D localization
of objects known during training are starting saturate (Hodan et al.] 2024). The community effort has now shifted toward the development of models able to generalize to objects unseen during train-
ing (Zhao et al., 2022 Labbé al. 2022 Nguyen et al. 2024; Örnek et al. 2024) which are often decomposed in coarse template matching (Nguyen et al. 2022) followed by iterative refinement (Hai ct al. 2023). However, these model-based based methods tend to lack robustness when the retrieved mesh differs too much from the object depicted in the input image, which limits their application in less controlled, "in the wild" settings. this work, we aim to overcome this key limitation.
6D pose estimation in the wild. For uncontrolled setups, such as in Internet videos, one typically does not have access to exact models of the depicted objects. Category-level pose estimation meth- ods drop the requirement for exact models by assuming the existence of canonical reference frame among a class of objects (e.g., mugs or cars). Some of the works make simplifying assumptions by constraining the object rotation along the gravity direction, which is reasonable for large items such as furniture or vehicles (Papon & Schoeler [2015) Braun et al.] 2016). For 6D category pose estimation, the normalized object coordinate space representation is popular choice (Brachmann) let al. 2014; Wang et al.] 2019 Xiao et al. 2022). In contrast, model-free pose estimation methods make no assumption on the type of objects present in the scene but rely either on a onboarding procedure where several reference images of the target object are captured together with the cor- responding metric camera poses (Sun al. 2022 He al.) 2022a) (e.g.. using ARKit) or require the use of RGBD images as input at run-time (Wen & Bekris 2021 He et al. 2022b). While the metric mesh or volumetric model limits their use for applications in simulation or robotics. Finally, another approach is to jointly reconstruct the object shape and solve for the camera pose(s) (Irshad et al. 2022). Object reconstruction may be done during an onboarding phase using one image (Long al. 2024 [ Nguyen et al. 2024) multi- view reconstruction (Yariv et al.] 2020 Wen et 2004). However, the onboarding images are often required to display the object in a static pose and from several different views (> 20), which makes these methods unsuitable for directly estimating object poses from in-the-wild Internet videos. An
model-free free methods are versatile, the absence
alternative to onboarding can be test-time adaptation (Shi et al. 2024) to address the novel shapes
test time. Some works (Hasson et al., 2019 Zhang et al. 2020 Cao et al. 2021 Wen et al. 2023) have considered jointly reconstructing and tracking of objects interacting with hands, but none of these have demonstrated transferring object motion from in-the-wild Internet videos to a robotic ma- nipulator. In contrast, we propose to tackle uncontrolled Internet videos via an approach combining mesh retrieval, 6D) pose alignment, object tracking and trajectory optimization to go from the input unconstrained Internet video to a motion imitating the fine-grained human manipulation on a robot.
Learning robotic manipulation from video. One strategy for learning robot manipulation skills from human demonstrations consists of extracting reward signals from the videos and using the reward to train robotic policy using reinforcement learning (a good survey of different meth- ods is in (McCarthy et al., 2024)). Examples include (Sermanet et al. 2018), which defines the reward using image features extracted with an embedding model trained with time-contrastive self- supervision. The model requires a relatively large number of real demonstrations (around 100). Other works operate from single demonstration and leverage explicit 3D object representations and estimated object poses to define the rewards. For example, (Petrík et al.] (202)) uses a differ- entiable renderer to estimate the pose of geometric primitives approximating the objects shapes, and (Zorina et al. 2021) estimates the pose of stick-like objects (hammer, scythe) using the method of of al. (2022) and physics- based temporal smoothing. Closest to ours is(Patel al. (2022) which considers any type of object with a known 3D model. They demonstrate results for positioning ob- jects into a static pose given by video. In contrast, we do not assume the exact 3D model to be known and we show results for a complete task (e.g., pouring).
3 APPROACH
We first in Section 3.1 present method for computing 6D poses for objects without knowing their exact 3D models. Then, in Section (3.2, we describe how we extract smooth 6D object trajectories from Internet videos and transfer them to real robotic manipulator.
3

---

Published as a conference paper at ICLR 2025
(o) Object detection
(b) CAD model retrieval
(d) pose estimation
Input image
Encoder
Patch basad
Enc:
R
Mesh DB
(c) Object scale estimation
odjjects
Detector
CLIP LLM
*A bottlle" te
20 cv
R.
"A gloap"
10
Glocal reacaling
S
Depth
est.
Relat oca Lee
Overview of our 6D object pose estimation without a known 3D mesh (Sec. 3.1 Given input RGB image, our method: (a) detects and segments objects present in the image, (b) retrieves similar meshes from a large-scale object database via patch-based retrieval, (c) estimates the absolute scale of depicted objects in the scene via LLM-based re-scaling, and (d) estimates the camera-to-object rotation R and translation via alignment of the retrieved (approximate) mesh.
Figure 2:
3.1 6D OBJECT POSE ESTIMATION WITHOUT A KNOWN 3D MESH
Given image video frame, our goal is to detect and estimate the 6D pose of objects in the image. We wish to predict the object poses for in-the-wild images and videos without any prior knowledge of the objects themselves. Therefore, in contrast to the standard 6D pose estimation methods, our main challenge is the lack of a known CAD model of the object. Further, the CAD model is unknown, the size of the object in the image is also unknown. The object can appear small in the image because it is small, or it is large but far from the camera.
To tackle these challenges, we present a novel method that (a) after detection and segmentation of the object in the image using an open-set detector, (b) retrieves CAD model semantically and geometrically similar to the object the image from large database, (c) computes the object scale by estimating the monocular depth and grounding it to the real world using large language model, and (d) estimates the 6D pose of the object the image by aligning with the retrieved CAD model. The overview of approach is shown in Figure and each of the steps is deseribed below.
Object detection and segmentation. We use an open-set object detector with an image segmenter to generate high-quality object proposals and masks for objects present in the input image. Additional details can be found in Appendix A.I]
CAD model retrieval. Given image video frame, goal is obtain CAD models of salient objects present the image. While single-view 3D reconstruction methods (Wang et al., [024] Szymanowicz al., 2024 Liu et al. (2023) offer possible solution, found them unsatisfiable for our task of 6D pose detection and tracking in the wild. Instead, for each object in the image, we retrieve the most similar CAD model from large-scale reference database composed of Objaverse- LVIS (Deitke et al. 2023) and Google Scanned Objects (Downs et al. 2022).
The key question is how to perform retrieval of the most similar CAD model for an object in the input RGB image. This is hard as the database typically does not contain the exact CAD model for the depicted object. Hence, the goal is retrieve another model instance from the same category (e.g., another similarly-looking mug). To perform such "category-level retrieval, retrivia,, we use pre-trained foundation image features, as they are invariant to appearance differennees between real images and CAD model renders. In detail, we represent each CAD model in the database as a single feature descriptor obtained by foreground feature averaging (FFA) (Kotar et al. 2023) patch-level feature maps (Oquab et al. 2024) obtained from multiple rendered views of the given CAD model. For each proposal generated by the object detector and segmenter, we then extract the same F FFA feature descriptor and retrieve the most similar CAD model from the database.
4

---

Published as a conference paper at ICLR 2025
6D object pose estimation via category-level view alignment. Given the retrieved CAD model with its associated object mask in the input image, the goal is to estimate rotation R E SO(3) and translation t R2 of the object in camera coordinate space. We assume pinhole camera model with focal length f. principal point e, and an approximate scale of the model. In Appendix A.2 we discuss how these assumptions can be relaxed.
First, we estimate the object's rotation via view alignment. In particular, we proceed as follows: we crop, mask. and rescale the image according to the object's mask and compute its patch fea- tures (Oquab et al. 2024), here denoted as {p% query }& where k is the patch index. Then, we compute the dot product with the patch features {p% CAD {R) of view of the CAD model rendered in ro- tation R and select the best rotation R maximizing the dot product averaged across all the N patches:
R
arg max R
Pk Pk
query CAD (R)).
We use views rendered with rotations sampled using the viewpoint sampler of Alexa 2022
The translation is computed in two steps. First, the = (depth) coordinate is computed by rescaling the CAD model's width, height, and depth [ow,, oh, oa] in the rendered coordinate space [R 0] by the object's bounding box width by and height by as:
t; 2 (5 +
2
Oh by.
(2)
Then, the x and y coordinates are computed using the bounding box center [bx, by] and the camera intrinsics as: tz (b, Cv / and ty (by cy)ts/f. If the intrinsics of the camera are unknown, we observed that using common intrinsic values (cqual to the field of view of 53 degrees) yields good results up to the absolute scale of the scene. The detailed discussion of this topic is in Appendix A.2 Determining the absolute scale is discussed next.
Object scale estimation via monocular depth prediction and LLM. Obtaining correct 6D pose of an object requires knowledge of its absolute scale. However, we cannot rely on the arbitrary scales of the CAD models found in our database. For example, the database models can have significantly different scales (metric or imperial, millimeters or meters), which makes it impossible to automati- cally use them for pose estimation tasks. Although one can roughly estimate the object scales from point clouds obtained by back- pooj-projettion using a depth map, for in-the-wild scenarios, such as in Internet videos, the depth map often not available. Instead, we develop a scale estimation method requiring only single-view RGB input, that takes advantage of depth map estimated using a neural network and information about common object scales obtained from a large language model (LLM).
While it is impossible to predict a metrically accurate depth map from single image, especially the camera intrinsic parameters are not known, we observe that monocular depth predictors can be used to obtain relative scales of the objects even with unknown intrinsic parameters. We assign relative scale r, to every object i as the distance between the furthest object point and object center along object's ss principal axis as computed from the point cloud obtained from the monocular depth predictor (Bhat et al. 2023), while using the common intrinsic priors (see the second paragraph in Appendix A.2). A.22). To obtain the metric scales m,, we first ask GPT-4 (OpenAl 2024) to generate a list of common object text descriptions with their typical size in meters. Then, we classify the depicted object using the dot product between CLIP (Cherti et al., 2023) features of the object's image and the text descriptions generated by GPT-4. We retrieve the corresponding scale as the scale of the best matching text descriptions as measured by the dot product. Note, that our set of generated text descriptions is not tailored to any specific scene type. Instead, it is designed to encompass wide range of everyday objects, therefore it can be generated offline. Moreover, the set can be easily adapted to specific use cases by generating new descriptions using a modified prompt, if needed. Finally, we use the metric scales m, to globally rescale the relative scales r; obtained from the depth map. In this way, we keep the estimated ratios between the object scales, but employ the metric information obtained using the LLM. More details about the scale estimation method, including the outlier rejection procedure, can be found in Appendix A.3 Note that our approach aggregates scale estimates from multiple objects in the scene in a robust manner, which makes our method tolerant to incorrect scale estimates of some of the objects within the scene.
5

---

Published as a conference paper at ICLR 2025
3.2 ROBOTIC OBJECT MANIPULATION FROM INTERNET VIDEOS
In this section describe how we obtain temporally consistent 6D pose trajectories from Internet videos and how we use those trajectories to drive a robotic manipulator. This problem is challenging as the input videos often depict lineggrained dynamic object motions (think person trying to empty ketchup bottle). To address this issue we design an approach that outputs a smooth 6D motion in the configuration space of the robot by leveraging (i) approximate 6D object pose estimates obtained by the approach described in Section 3.1 with (ii) recent advances in accurate 2D feature point tracking in video (Karaev et al. 2024) and (1I) trajectory optimization techniques in roboties.
Pose tracking for recovering object trajectories from Internet videos. The category-level align- ment approach described in Section 3.1 predicts discrete set of rotations. Also, the alignment can- not recover the relative inter-frame rotation of cylindrically symmetric objects such as cups, bottles, or bowls due to the different textures of the real and the retrieved 3D objects. To produce smooth trajectories with correct inter-frame rotations that can be transferred to robotic manipulator, we track the object across video frames using a dense point video tracker (Karaev et al. 2024). The tracking is initialized by sampling 3D points Pc E R on the retrieved object and projecting them into a video frame using the object pose estimate obtained using our 6D pose estimation approach, resulting in a set of 2D image points Lı RR. The points Pı tracked across video frames to produce set of correspondences {(pc. p() where is the index of video frame. Assuming static camera, the temporally refined 6D pose of the object is computed for each frame j using the PnP algorithm (Lepetit et al. 2009) on the set of 3D-2D correspondences {(pc.') p/)]r- The tempo- rally refined poses obtained from this procedure result in smoother trajectories with fewer errors for video frames with partial occlusions. Note that for moving cameras, camera movement correction can be made by estimating the relative camera poses from the static video background using readily available software such as COLMAP (Schönberger & Frahm 2016). In this paper, we focus on object motion relative to the camera and thus do not explicitly estimate the camera's 's relative poses in the video.
Retargeting trajectories to a robotic manipulator. The temporally refined sequence of poses of the manipulated object is then used target to compute trajectory in the configuration space of the robot imitating the object 6D trajectory extracted from the video. We address this problem using trajectory optimization as described next. To express the sequence of estimated object poses in the robot frame of reference, we manually locate the camera in the robot space and recompute the object poses estimated in the camera frame into the robot frame. We express the pose of the object with respect to the robot at time using transformation T, that captures both the rotation and translation of the object in a homogeneous representation, as explained in (Sola et al.) 2018). To obtain the robot trajectory. we use trajectory optimization (Jallet et al. 2023) to solve the following optimization problem:
N-1 Wa log TT:) + my || w. ||T
TO
TN arg min
where T is the torque that controls the robot joints; the first cost term weighted by penalizes SE(3) distance via SE(3) log function (Sola et al., 2018) between T., pose of the object at time t held by the robot, and T., the pose of the object estimated from the video; the second and third terms of the cost regularize the velocity i and torque of the robot to prevent redundant motion of the robot. Optimization is solved using torque control actions to achieve a smooth trajectory in the robot configuration space. See Appendix
Appendix/A.6ffor for more details.
4 EXPERIMENTS
In this section, we first describe the datasets and metrics used. Then we present comparison of our method with state-of-the-art 6D pose estimation methods the YCB-V and HOPE-Video datasets and we ablate the key components of our approach. Finally, we evaluate the proposed method on challenging Internet videos and demonstrate that the estimated 6D object trajectories can be imitated by a robotic manipulator. Additionally, in Appendix C,5 we provide quantitative results on DTTD2 (Huang et al. 2023) dataset.

---

Published as a conference paper at ICLR 2025
Table 1: Comparison with the state-of-the-art on the YCB-V and HOPE- datasets. The
6D pose estimation results of our approach compared to MegaPose (Labbé et al., 2022), Giga- Pose (Nguyen al. 2024). and FoundPose (Ornek al., 2024 Our method outperforms all other methods in all the metrics.
YCB-V
HOPE-Vides
Method
ART
ARcout
ARcht
ARpart
ARt
ARcout
ARcut
ARpcu
MegaPose (coarse)
23.75
10.08
10.65
50.53
31.77
9.96
6.87
78.50
MegaPose
25.76
14.01
11.91
51.37
33.03
13.07
6.38
79.64
GigaPose
29.18
11.90
9.20
66.45
23.12
4.15
4.90
60.30
FoundPose
42.95
35.40
15.69
77.75
42.30
31.18
9.58
86.13
Ours
49.86
45.20
18.53
85.83
45.98
39.21
10.72
88.01
Table Influence of different model retrieval methods 6D pose estimation.
Table 3: Effects of scale estimation method the 6D pose estimation.
Retrieval
ARt
ARcout
ARcht
ARpcht
Scale Estimator
ARch 1
(a) OpenShape
34.51
20.25
10.59
72.69
Constant
11.89
(b) Ours (CLS)
45.05
40.93
16.16
78.06
Ours
18.53
(c) Ours
49.86
45.20
18.53
85.83
Oracle
45.95
(d) Oracle
62.93
51.99
45.95
90.85
Other metrics are scale-indeeendenn
4.1 EVALUATION OF 6D POSE ESTIMATION ON STANDARD DATASETS
Datasets and metries. We evaluate method YCB-V (Xiang et al. 2018) and HOPE- Video (Lin et al. 2021). The datasets feature cluttered scenes with partially occluded objects, re- sembling our target Internet videos. Both datasets contain full 6D pose annotation with the known object meshes available, yet, in our experiments, we utilize the meshes retrieved from Objaverse- LVIS (Deitke et al. 2023) and Google Scanned Objects (Downs al. 2022) datasets instead, This simulates the wild scenario where no ground-truth meshes are available.
For evaluation, we follow the BOP-inspired protocol, where average recall (AR) is estimated for every metric. We report the standard Chamfer distance (CH) metric. However, because this metric is highly dependent on the correct CAD model scale which is unavailable in our scenario, we also report projection-base based metrics, i... projected Chamfer distance (pCH) and complement over union (CoU). We also report mean AR across all metrics. For comparison proposal generation methods, also report average precision. See details of the metrics in Appendix
Comparison with the state of the art We compared our RGB-only approach against state-of-the- art RGB-only methods from the BOP challenge's 's "unseen-objeets" category, videos lack depth information. The top open-source methods compared were GigaPose (Nguyen al. 2024), FoundPose (Orek al. 2024),, and MegaPose (Labbé et al. 2022 We always retrieved meshes and scales estimates instead of the ground truth meshes for a fair comparison. The results of the methods on the YCB-V and HOPE-Vider datasets are shown Table I The table shows our method significantly outperforms all three methods, and impressively, our method achieves that without any in-domain training or fine-tuning. This indicates that MegaPose, GigaPose and FoundPose, designed for 6D pose estimation with precise and known meshes, do not generalize well to scenarios with unknown imprecise meshes and are unsuitable for the in-the-wild task.
target YouTube
4.1.1 1 ABLATIONS OF KEY COMPONENTS
CAD model retrieval. We compare our method for CAD model retrieval to the OpenShape (Liu et al. 2024a) baseline on the YCB-V dataset in Table OpenShape is multi-modal model that aligns point clouds into CLIP space using PointBERT (Yu et al. 2022). While OpenShape works well when the ground truth models are present in the database, in our scenario, when the exact models are unavailable, our method (c) outperforms OpenShape (a) by roughly 44% as measured by final pose estimation quality. As the upper bound for CAD model retrieval, we also show method in the scenario when the exact meshes of the objects depicted in the image and their scales known (d). Additionally, compare our choice of using the FFA descriptor with the per-view

---

Published as a conference paper at ICLR 2025
Input
Predictions
Overlay
Input
Predictions
Overlay
D
Figure 3: Qualitative results of our method on the YCB-V dataset. On the YCB-V dataset our method is able to detect objects that not part of the dataset (such as keyboard in image F and chair in image C), which highlights the benefit of our method of not being restricted to the ground truth meshes.
Input
Predictions
Overlay
Input
Predictions
Overlay
B
D
F
Mile
H
Figure 4: Qualitative results of our method on the HOPE-Video dataset. On the HOPE- Video dataset method reconstructs complex scenes with multiple objects while respecting partial oc- clusions.
averaged CLS token descriptor. We show that foreground feature averaging (c) outperforms the CLS token (b) substantially. We provide further ablations on the descriptor choice Appendix C.I
Scale estimation. We compare our object scale estimation method to the constant scale baseline of 10cm on the YCB-V dataset in Table Even though estimating scale from a single image is an ill- posed problem, our method that uses a scale prior obtained from a large language model improves the average Chamfer distance recall by 55% over the constant model scaling. This shows the LLM brings insightful information about the object sizes.
4.1.2 QUALITATIVE RESULTS OF 6D POSE ESTIMATION
The qualitative results on YCB-V images are shown in Figure and qualitative results on HOPE- Video are shown in Figure The successful examples in the figure demonstrate the capability of the proposed method, that is, being able to retrieve an object in the same category and to align its pose despite visual differences. For example, consider the drill in the Figure A our method retrieves another type of drill whose texture differs from the texture in the query image, but the retrieved CAD model is still aligned with the input query image. A similar example is shown in figure/4.A where our method aligns meshes with textures different from the objects depicted in the input image. An added benefit of our method is it's capability to align objects that outside of the training dataset.
8

---

Published as a conference paper at ICLR 2025
Figure 5: Qualitative results of our method on the EPIC-Kitchens dataset. Our method is able detect and align objects from an egocentric point of view. See Figures X and for additional examples.
In Figure C our method retrieves a chair and in Figure 3 computer keyboard, both of which are not set of ground truth objects of the YCB-V dataset.
Failure modes. Our method has two main failure modes: (i) failure to retrieve similar mesh and (ii) failure estimate the correct scale. The example of the first failure mode is shown in Figure f
C. where instead of the box (top left) red and white checkerboard object is retrieved and consequence the pose estimation fails. Another failure mode shown Figure TE where improper scale estimation leads incorrect scene occlusions.
4.2 VIDEO RESULTS AND APPLICATIONS
We show quantitative and qualitative results of our full 6D pose detection and tracking method on Internet videos as well as on egocentric videos from the APIC-KITCHENG dataset (Damen al., 2022). We also show an application of the extracted object manipulation trajectories on real robot. More results are available the appendix and the supplementary video.
Quantitative evaluation. For the quantitative evaluation of object tracking in-the-wild, we man- ually annotated an approximate 6D pose of human interacted objects 32 "in-the-wild" videos containing total of 6947 frames. To annotate the ground truth poses, the annotator selected an object from the top Objaverse retrievals and then mamually aligned it to visually match each frame of the video. The results of our approach are compared to several state-of-the-an methods shown in Table 4 Our method consistently improves upon the previous state-of-the-an More details on the evaluation metries in Appendix B.2
Qualitative results on Internet and egocentric videos. For a qualitative evaluation the pro- posed method in the wild, consider instructional videos, e.g., cooking videos from the Internet and EPIC-KITCHENS (Damen et al. 2022) dataset. The resulting 6D object trajectories obtained from videos from the EPIC-KITCHENS dataset are shown in Figure 5 and Figures & and 9 in the
Table 4: Comparison with state-of-the-art 6D object estimation methods on 'in the wild" videos. The 6D object tracking results of our approach are compared to MegaPose (Labbé et al.) 2022), GigaPose (Nguyen et al. 2024). and FoundPose (Örek et al. 2024 Our method outper- forms these baselines in all metrics.
MegaPose (coarse)
MegaPose
FoundPose
Ours
Ours (refined)
Relative rotation +
6.866
6.822
7.736 0.405
1.719
1.205
Relative projected translation |
0.107
0.161
0.091
0.086

---

Published as a conference paper at ICLR 2025
Input video
Prediction
Simulation
Real robot
Figure 6: The proposed approach applied on an Internet video. Top row: several input frames. Second row: alignment obtained by our method together with tracked 2D-3D correspondences shown as colored tracks. Third row: Temporally smooth poses are used to compute robot mo- tion via trajectory optimization. Bottom row: vidco of real robot execution. Please additional results the Appendix and the supplementary video.
appendix. Please note, our method can detect and track multiple objects. We use the same approach for the Internet instructional videos and then perform experiments real robot, described in the following paragraph.
Application: Imitating object motion from Internet videos. The temporally smooth 6D poses used to optimize the trajectory of the Franka Emika Panda robot. The optimized trajectory can then be executed in simulation on real robot. The proposed approach successfully replicates actions shown in instructional videos, as shown in Figure 6 Additional results are shown in Figure [0O and Figure 11 in Appendix [ For full videos of the real-world robot manipulation, please see supplementary video. Please note how in the video the robot imitates the extracted subtle human motions that would otherwise hard to program, e.g., shaking the jug before pouring.
Limitations. Two main limitations persist: (i) if the quality of object segmentation is low or the object is significantly occluded, the retrieval and scale estimation may not be successful; and (ii) the mesh retrieval sometimes selects the most semantically similar object, which however, is not always geometrically similar to the object in the video, which can lead to inaccuracies pose estimation.
CONCLUSION
We present an approach for category-level 6D object alignment for in-the-wild images and videos. We show that similar mesh can be retrieved from large-scale database of CAD meshes given the input query image, and can then aligned to the query image despite the visual dissimilarities. We demonstrate quantitatively two BOP datasets and new dataset of "in-the-wild" instructional videos that our method outperforms the state-of sitee-ofthee. offthe-art art 6D pose estimation baselines. Qualitatively, show application of the proposed method guiding robotic manipulation using an instruc- tional video, making a step towards large-scale learning object manipulation skills from Internet's instructional videos.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was partly supported by the Ministry of Education, Youth and Sports of the Czech Re- public through the e-INFRA CZ (ID:90254), and by the European Union's Horizon Europe projects AGIMUS (No. 101070165), euROBIN (No. 101070596), and ERC FRONTIER (No. 101097822). Views and opinions expressed are however those of the author(s) only and do not necessarily re- flect those of the European Union or the European Commission. Neither the European Union nor the European Commission can held responsible for them. Georgy Ponimatkin and Martin Cífka were also partly supported by Grant Agency of the Czech Technical University in Prague under allocations 56SS5/15500HKK//T/1 (GP) and SGS25/152/0HKK33TTV3 (MC).
REFERENCES
Mare Alexa. Super-Fibonacci Spirals: Fast, Low-Discrepancy Sampling of SO(3). CVPR, 2022.
Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep ViT Features as Dense Visual Descriptors. ECCVW, 2022.
Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller. ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth. arXiv:2302.12280 2023.
Eric Brachmann, Alexander Krull, Frank Michel, Stefan Gumhold, Jamie Shotton, and Carsten Rother. Learning 6D Object Pose Estimation using 3D Object Coordinates. ECCV, 2014.
Markus Braun, Qing Rao, Yikang Wang, and Fabian Flohr. Pose-RCNN: Joint Object Detection and Pose Estimation Using 3D Object Proposals. ITSC, 2016.
Zhe Cao, Ilija Radosavovie, Angjoo Kanazawa, and Jitendra Malik. Reconstructing Hand-Object Interactions in the Wild. ICCV, 2021.
Justin Carpentier, Guilhem Saurel, Gabriele Buondonno, Joseph Mirabel, Florent Lamiraux, Olivier Stasse, and Nicolas Mansard. The Pinocchio C++ library fast and flexible implementation of rigid body dynamics algorithms and their analytical derivatives. IEEE SII, 2019.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gor- don, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-iman image learning. CVPR, 2023.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Jian Ma, Evange- los Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS--O0) IJCV, 2022.
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: Universe of Anno- tated 3D Objects. CVPR, 2023.
Maximilian Denninger, Dominik Winkelbauer, Martin Sundermeyer, Wout Boerdijk, Markus Knauer, Klaus H. Strobl, Matthias Humt, and Rudolph Triebel. BlenderProc2: A Procedural Pipeline for Photorealistic Rendering. Journal of Open Source Software, 2023.
Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent Vanhoucke. Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Items. ICRA, 2022.
Hao-Shu Fang, Chenxi Wang. Minghao Gou, and Cewu Lu. GraspNet-1Biinn BBillion: Large-Scale Benchmark for General Object Grasping. CVPR, 2020.
Gemma Team. Gemma 2: Improving Open Language Models at Practical Size. arXiv:2408.00116 2024.
Yang Hai, Rui Song, Jiaojiao Li, and Yinlin Hu. Shape-Constraint Recurrent Flow for 6D Object Pose Estimation. CVPR, 2023.
11

---

Published as a conference paper at ICLR 2025
Yana Hasson, Gul Varol, Dimitrios Tzionas, Igor Kalevatykh, Michael J Black, Ivan Laptev, and Cordelia Schmid. Learning joint reconstruction of hands and manipulated objects. CVPR, 2019.
Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Girshick. Mask R-CNN. ICCV, 2017.
Xingyi He, Jiaming Sun, Yuang Wang, Di Huang, Hujun Bao, and Xiaowei Zhou. OnePose+4: Keypoint-Free One-Shot Object Pose Estimation without CAD Models. NeurlPS, 2022a.
Yisheng He, Yao Wang. Haoqiang Fan, Jian Sun, and Qifeng Chen. FS6D: Few-Shot 6D Pose Estimation of Novel Objects. CVPR, 2022b.
Tomáš Hodan, Pavel Haluza, Stepán Obdržálek, Jiri Matas, Manolis Lourakis, and Xenophon Zab- ulis. T-LESS: An RGB-D Dataset for 6D Pose Estimation of Texture-less Objects. WACV, 2017.
Tomas Hodan, Martin Sundermeyer, Bertram Drost, Yann Labbé, Eric Brachmann, Frank Michel, Carsten Rother, and Jiri Matas. BOP Challenge 2020 on 6D Object Localization. ECCVW, 2020.
Tomas Hodan, Martin Sundermeyer, Yann Labbe, Van Nguyen Nguyen, Gu Wang, Eric Brachmann, Bertram Drost, Vincent Lepetit, Carsten Rother, and Jiri Matas. BOP Challenge 2023 on Detec- tion, Segmentation and Pose Estimation of Seen and Unseen Rigid Objects. CVPRW, 2024.
Zixun Huang, Keling Yao, Seth Z. Zhao, Chuanyu Pan, Tianjian Xu, Weiyu Feng, and Allen Y. Yang. Robust Digital-Twin Localization via An RGBD-based Transformer Network and A Com- prehensive Evaluation on a Mobile Dataset. ICMLW, 2023.
Muhammad Zubair Irshad, Sergey Zakharov, Rares Ambrus, Thomas Kollar, Zsolt Kira, and Adrien Gaidon. ShAPO: Implicit Representations for Multi-Object Shape, Appearance, and Pose Opti- mization. ECCV, 2022.
Wilson Jallet, Antoine Bambade, Etienne Arlaud, Sarah El-Kazdadi, Nicolas Mansard, and Justin Carpentier. PROXDDP: Proximal Constrained Trajectory Optimization. hal-04332348, 2023.
Glenn Jocher, Jing Qiu, and Ayush Chaurasia. Ultralytics YOLO, 2023. URthttps::///ithha Com/ http://twraalttcc/llaaaaatteeeea
Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. CoTracker: It is Better to Track Together. ECCV, 2024.
Lei Ke, Mingqiao Ye, Martin Danelljan, Yifan Liu, Yu-Wing Tai, Chi-Keung Tang, and Fisher Yu. Segment Anything in High Quality. NeurIPS, 2023.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick. Segment Anything. ICCV, 2023.
Klemen Kotar, Stephen Tian, Hong-Xing Yu, Daniel L.K. Yamins, and Jiajun Wu. Are These the Same Apple? Comparing Images Based Object Intrinsics. NeurIPS, 2023.
Yann Labbé, Justin Carpentier, Mathicu Aubry, and Josef Sivic. CosyPose: Consistent multi-view multi-object 6d pose estimation. ECCV, 2020.
Yann Labbé, Lucas Manuelli, Arsalan Mousavian, Stephen Tyree, Stan Birchfield, Jonathan Trem- blay, Justin Carpentier, Mathieu Aubry, Dieter Fox, and Josef Sivic. MegaPose: 6D Pose Estima- tion of Novel Objects via Render & Compare. CoRL, 2022.
Vincent Lepetit, Francese Moreno-Noguer, and Pascal Fua. EPnP: An Accurate O(n) Solution to the PnP Problem. IJCV, 2009.
Yi Li, Gu Wang, Xiangyang Ji, Yu Xiang, and Dieter Fox. DeepIM: Deep Iterative Matching for 6D) Pose Estimation. IJCV, 2020.
Zongmian Li, Jiri Sedlar, Justin Carpentier, Ivan Laptev, Nicolas Mansard, and Josef Sivie. Estimat- ing 3D Motion and Forces of Human-Object Interactions from Internet Videos. IJCV, 2022.
12

---

Published as a conference paper at ICLR 2025
Yunzhi Lin, Jonathan Tremblay, Stephen Tyree, Patricio A. Vela, and Stan Birchfield. Multi-view Fusion for Multi-level Robotic Scene Understanding. JROS, 2021.
Minghua Liu, Ruoxi Shi, Kaiming Kuang, Yinhao Zhu, Xuanlin Li, Shizhong Han, Hong Cai, Fatih Porikli, and Hao Su. OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understanding. NeurIPS, 2024a.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-I-to-3; Zero-shot One Image to 3D Object. ICCV, 2023.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang. Hang Su, Jun Zhu, al. Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection. ECCV, 2024b.
Llama Team. The Llama 3 Herd of Models. arXiv:2407.21783, 2024.
Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Mare Habermann, Christian Theobalt, and Wenping Wang. Wonder3D: Single Image to 3D using Cross-Domaie Domain Diffusion. In CVPR, 2024.
Yangxiao Lu, Yunhui Guo, Nicholas Ruozzi, Yu Xiang, al. Adapting Pre-Trained Vision Models for Novel Instance Detection and Segmentation. arXiv:2405.17833 2024.
Robert McCarthy, Daniel C.H. Tan, Dominik Schmidt, Fernando Acero, Nathan Herr, Yilun Du, Thomas G Thuruthel, and Zhibin Li. Towards Generalist Robot Learning from Internet Video: A Survey. arXiv:2404. 19664, 2024.
Antoine Miech, Dimitri Zhukov, Jean- Sann-Batiiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. How 1o100 Learning a Text- Video Embedding by Watching Hundred Million Narrated Video Clips. ICCV, 2019.
Andrew Miller and Peter K Allen. Graspit!: A Versatile Simulator for Robotic Grasping. RAM, 2004.
Van Nguyen Nguyen, Yinlin Hu, Yang Xiao, Mathieu Salzmann, and Vincent Lepetit. Templates for 3D Object Pose Estimation Revisited: Generalization New Objects and Robustness to Oc- clusions. CVPR, 2022.
Van Nguyen Nguyen, Thibault Groueix, Georgy Ponimatkin, Vincent Lepetit, and Tomas Hodan. CNOS: A Strong Baseline for CAD-based Novel Object Segmentation. ICCVW, 2023.
Van Nguyen Nguyen, Thibault Groueix, Mathieu Salzmann, and Vincent Lepetit. GigaPose: Fast and Robust Novel Object Pose Estimation via One Correspondence. CVPR, 2024.
OpenAl. GPT-4 Technical Report, 2024.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Yao Huang, Shang- Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without super- vision. TMLR, 2024.
Evin Pınar Örnk,, Yann Labbé, Bugra Tekin, Lingni Ma, Cem Keskin, Christian Forster, and Tomas Hodan. FoundPose: Unseen Object Pose Estimation with Foundation Features. ECCV, 2024.
Jeremie Papon and Markus Schoeler. Semantic Pose using Deep Networks Trained on Synthetic RGB-D. ICCV, 2015.
Austin Patel, Andrew Wang, Ilija Radosavovic, and Jitendra Malik. Learning to Imitate Object Interactions from Internet Videos. arXiv:2211.13225 2022.
Vladimír Petrik, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Learning Object Manipulation Skills via Approximate State Estimation from Real Videos. CoRI., 2021.
13

---

Published as a conference paper at ICLR 2025
Nikhila Ravi, Valentin Gabeur, Yuan- Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Va- sudev Alwala, Nicolas Carion, Chao- Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Fe- ichtenhofer. SAM 2: Segment Anything Images and Videos. mrXi::248.000744 2024.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. TPAMI, 2016.
Johannes Lutz Schönberger and Jan-Michael Frahm. Meuuture-from-Monn Revisited. CVPR, 2016.
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. Jime-Contrastiv. Networks: SuperSuueee Learning from Video. In ICRA, 2018.
Jingnan Shi, Rajat Talak, Harry Zhang. David Jin, and Luca Carlone. CRISP: Object Pose and Shape Estimation with Test-Time Adaptation. arXiv:2412.01052, 2024.
Joan Sola, Jeremie Deray, and Dinesh Atchuthan. A micro Lie theory for state estimation in robotics. arXiv:18/2.01137, 2018.
Jiaming Sun, Zihao Wang, Siyu Zhang, Xingyi He, Hongcheng Zhao, Guofeng Zhang, and Xiaowei Zhou, OnePose: One-Shot Object Pose Estimation without CAD Models. CVPR, 2022.
Stanislaw Szymanowicz. Christian Rupprecht, and Andrea Vedaldi. Single-Viev View 3D Reconstruction. CVPR, 2024.
Splatter Image: Ultra-Fast
Stephen Tyree Jonathan Tremblay, Thang To, Jia Cheng, Terry Mosier, Jeffrey Smith, and Stan Birchfield. 6-DoF Pose Estimation of Houschold Objects for Robotic Manipulation: An Accessi- ble Dataset and Benchmark. IROS, 2022.
Gu Wang, Fabian Manhardt, Federico Tombari, and Xiangyang Ji. GDR-Net: Geodetry-Guided Direct Regression Network for Monocular 6D Object Pose Estimation. CVPR, 2021.
He Wang, Srinath Sridhar, Jingwei Huang, Julien Valentin, Shuran Song, and Leonidas J Guibas. Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation. CVPR, 2019.
Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. DUSt3:: Geometric 3D Vision Made Easy. CVPR, 2024.
Bowen Wen and Kostas Bekris. BuncleTrack: 6D Pose Tracking for Novel Objects without Instance or Category-Level 3D Models. IROS, 2021.
Bowen Wen, Jonathan Tremblay, Valts Blukis, Stephen Tyree, Thomas Müller, Alex Evans, Dieter Fox, Jan Kautz, and Stan Birchfield. BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objects. CVPR, 2023.
Bowen Wen, Wei Yang, Jan Kautz, and Stan Birchfield. FoundationPose: Unified 6D Pose Estima- tion and Tracking of Novel Objects. CVPR, 2024.
Jane Wu, Georgios Pavlakos, Georgia Gkioxari, and Jitendra Malik. Reconstructing Hand-Held Objects in 3D. arXiv:2404.06507, 2024.
Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, and Dieter Fox. PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes. RSS, 2018.
Yang Xiao, Vincent Lepetit, and Renaud Marlet. Few-Shot Object Detection and Viewpoint Esti- mation for Objects in the Wild. TPAMI, 2022.
Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lip- man. Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance. NeurlPS, 2020.
14

---

Published as a conference paper at ICLR 2025
Xumin Yu. Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point-BERT: Pre- training 3D Point Cloud Transformers with Masked Point Modeling. CVPR, 2022.
Jason Y Zhang, Sam Pepose, Hanbyul Joo, Deva Ramanan, Jitendra Malik, and Angjoo Kanazawa. Perceiving 3D Human-Object Spatial Arrangements from Single Image in the Wild. ECCV, 2020.
Chen Zhao, Yinlin Hu, and Mathieu Salzmann. Fusing Local Similarities for Retrieval-based 3D Orientation Estimation of Unseen Objects. ECCV, 2022.
Xu Zhao, Wenchao Ding, Yongqi An, Yinglong Du, Tao Yu, Min Li, Ming Tang, and Jinqiao Wang. Fast Segment Anything. 0rXiv:2306./2/56, 2023.
Kateryna Zorina, Justin Carpentier, Josef Sivie, and Vladimír Petrík. Learning Manipulate Tools by Aligning Simulation to Video Demonstration. RA-L, 2021.
15

---

Published as a conference paper at ICLR 2025
APPENDIX
In this Appendix, we provide additional technical information related to our method. Section A describe implementation details of our method. The details CAD model retrieval are de- scribed in Section A.I In Section A.2 we provide details on the rotation estimation part of our pipeline together with a discussion
how camera focal length priors can be used in case of un- known focal length. Then, in Section A.3] we describe the technical details of the scale estimation part of method. In Section [A.4] provide more technical details object tracking. Sec- tion A.55then provides details on the runtime of our method. Section [.6 provides details on motion retargeting robotic manipulator. Section B provides details on metries used in the evaluation of our method. Section C describes additional abblation studies of our method. Finally, in Section C.5 and Section D we provide additional quantitative and qualitative results obtained by our method.
A IMPLEMENTATION DETAILS
A.1 CAD MODEL. RETRIEVAL DETAILS
Our model database contains approximately 500,00 objects from Objaverse-LVIS (Deitke et al. )023) and Google Scanned Objects (Downs et al. 2022) datasets. Each CAD model is rendered 600 views at 420x420 pixels. with rotations sampled from SO(3) using viewpoint sam- pler (Alexa 2022) from which 1024D) FFA representation descriptor is computed. When construct- ing FFA mesh descriptor the averaged foreground patch token is computed from layer 22 of the DINOv2 model (Oquab al., 2024) for each of the M 600 rendered views of the CAD model. The resulting per-view foreground patch tokens are then aggregated to create per-object patch token. While such representation may seem expensive to compute, it needs to be computed only once for each CAD model in the database.
in M
To retrieve CAD models for a given image, we first generate proposals using GIONndingDINO (Liu et al. 2024b) with prompt "objects" combined with SAM 2 (Ravi et al. 2024). Then, the retrieval process done by matching a single 1024D FFA descriptor extracted from a query image to database of ~50,000 1024D descriptors (one descriptor for one object) by means of the dot product.
A.2 6D OBJECT POSE ESTIMATION DETAILS
Rotation estimation. The rotation estimation part is built the pre-trained DINOv2 VIT-L / 14 (Oquab et al. 2024) model. The rotation is estimated using spatial patch features from layer 22 of DINOv2 model by directly matching proposal image to M = 600 pre-rendered templates of the retrieved CAD model. The proposal image is cropped and resized to 420x420 pixels to match the resolution of pre-rendered templates, which gives feature maps of 30x30 tokens for matching.
Camera focal length priors. In general, for the complete 6D pose estimation, both the camera intrinsics and the exact CAD model with the correct absolute scale must be known. Knowing only one of these is insufficient. Therefore, when ground truth camera focal length is not available, we use the commonly used prior of the camera focal length f = V + 12,, where w and h are image width and height in pixels, obtain an approximate point cloud. Our method tackles the problem when the exact CAD model with the ground truth scale is unavailable. Hence, by definition, there are infinitely many solutions (up to the unknown scale of the object) and defining absolute translation emor is not meaningful both with and without known camera intrinsics. To get approximate translation, our method estimates the absolute scales of objects the scene by leveraging prior knowledge available in LLM as is described in Section 3.1 and, similarly, we use the commonly used prior of the camera focal length as stated above when the intrinsics are not available.
For robotic experiments, we rescale the predicted object trajectory so that the robotic arm can exe- cute the motion. This rescaling would be necessary, even both the camera intrinsics and the ground truth CAD model were known, the robots and manipulated objects vary in size and the motion executed by a human in the video needs to be translated into the space the robot. However, this task-specific scale refinement be done automatically using, for example, task specific reward,
16

---

Published as a conference paper at ICLR 2025
e.g. to maximize the amount of poured liquid, in the aligned simulated environment as used, e.g., in
Zorina et al. (2021).
A.3
OBJECT SCALE ESTIMATION DETAILS
LLM scale database generation. When obtaining metric scales using an LLM, we precompute a database object text descriptions and their typical size in meters. To create the database, the keywords were generated by the GPT-4 (OpenAl 2024) model by repeatedly querying the following prompt:
You help researchers to understand the 3D world around them. You need to generate 250 keywords for common real-world objects and you also need to estimate the average object scale in that categery by considering the average largest dimension in meters. The keyword should be generated in the form of an article, followed by adjective (if needed), and then by noun.
Examples: a coffee mug, computer mouse, a pan, a laptop.
In output, please ignore natural entities such as mountains, planets, do not include imaginary creatures, and buildings. Also, do not consider niche objects such as diving equipment or rare scientific equipment.
The output should be following this structure:
object.1. 1: scale.1
object_2: scale_2
The input should be made only of this list. The scales should represent numbers in meters, but not include units.
The scales obtained were accumulated and, in case of repeated occurrence, the median scale was taken, and saved in the database with CLIP feature vectors of the text descriptions. In total, we collected a list of 2200 objects, with scale values ranging from ccm to 300m ("a cruise ship"), with 90% of the objects being under 2m. This makes it possible to use the proposed approach for the most common indoor scenarios, but possibly some outdoor scenarios as well. However, when a different environment is expected, the list of objects can be easily adapted by adjusting the LLM prompts to generate objects from this environment.
LLM scale retrieval. During inference, extract CLIP features from the images of masked objects depicted the input scene. Then, we retrieve the most similar text descriptions with the corre- sponding scales for each object using the extracted CLIP features, and aggregate the K retrieved scales using the median into metric scale m, per object i.
Relative scale estimation. To obtain relative scales of objects in scene, use the depth estimation approach (Bhat et al. (2033) to obtain a depth map and compute pointcloud for each object i in the scene using its segmentation mask. We then identify the direction of the centered point cloud with the largest variance using the SVD decomposition and estimate the relative object scale r; for object i as the size of the largest dimension of the point cloud.
Global rescaling, To combine the metric scale m, from the LLM with the relative scale from the depth map for object i, we first compute the ratio p; m,/r; between the scales obtained using these two methods. Then, we take the median value of these ratios over all objects in the scene obtaining a single correction factor for the entire scene. Finally, we obtain the metric scale 8; for each object by correcting the relative scale obtained from the automatic depth estimation using the scene correction factor p as 8; = Ti p. The outcome is an estimate of a real-worke seale s; (in meters) for each extracted object i.
Please note that for some objects with high size variation, the size can be difficult to estimate, e.g. bicycle for kid adult, real-sized car compared toy car. To cope with these issues, we use median aggregation during the global rescaling, making our method robust to outliers, ie. tolerating large portions of incorrect scale estimates within the same scene.
17

---

Published as a conference paper at ICLR 2025
A.4 POSE TRACKING DETAILS
To track the object, we detect objects in the first frame of the input video using a strong open set object detector GroundingDINO (Liu al., 2024b) to localize salient object candidates, and then track them in the video frames using SAM2 (Ravi et al. 2024). We then use our 6D pose estimation method on each frame to retrieve similar CAD model from the database and align them in each frame of the video. The highest confidence alignment is used to initialize pose tracking based on 3D-2D correspondences obtained using the 6D pose from the selected frame and 2D tracks from CoTracker (Karaev et al. 2024).
A.5 COMPUTATIONAL RESOURCES AND RUNTIMES
All experiments were carried out on a cluster featuring nodes with 8x NVIDIA A100 (40 GB of VRAM per GPU), 2x 64-core AMD EPYC 7763 CPU, 1024 GB RAM. The total storage needed for this project was around 4TB. Each experiment on the YCB-V dataset takes around hours on a single GPU.
single GPU and on the HOPE- Video dataset around 14 hours
The method onboarding stage consists of pre-rendering the meshes and extracting the features. The Objaverse-LVIS database with roughly ~50,000 objects can be rendered in 3 days on one 8-GPU node. The extraction of visual features for retrieval then takes around 3 days on one 8-GPU node as well, and takes around 200MB of disk space. In practice we used the before mentioned cluster to parallelize and speed up the rendering and extraction processes significantly.
single image, the resulting runtime for detection, retrieval, and scale estimation is ~- per image. For the pose estimation part, the runtime is on average ~0.28 per object with caching. However, when running on an instructional video, we run the detector only on the first frame and track the detected objects through the video using SAM 2 (Ravi et al. 20224) which can run in real-time. Then we run the retrieval and scale estimation, which can benefit from multi-frame prediction aggregation, but does not necessarily have to use all video frames, especially for longer videos. In practice, we used 30 frames throughout the video for the retrieval and scale estimation. In contrast to single-image estimation, we use CoTracker (Karaev et al. (024) combined with PnP to extract smooth poses from the video, with run-time of ~ per frame.
When running
A.6 RETARGETING TRAJECTORIES TO ROBOTIC MANIPULATOR
Initial object pose replication. To initiate the trajectory following process, the object needs to be placed in an initial configuration that corresponds to the starting pose of the video demonstration. We manually placed the object into the gripper (effectively fixing the transformation from the gripper to the object). The robot was then commanded to move the object to the initial pose, calculated as TRo where Teo is the object pose relative to the camera frame at the first frame of the video, as estimated by our method, and TRC the fixed transformation from the robot base to the virtual camera frame, manually chosen to simulate camera viewing the robot from the front with a 30 clevation relative to the gravity vector. Transformation TRC was manually defined and held constant across all demonstrations. This allows to move the object to a pose visually similar the initial state in the videos. However, in a practical robotic system, this manual initialization would be superseded by pre-existing grasp configurations or an automated grasping procedure (e.g., using a combination of motion planning and GraspIT (Miller & Allen 2004) or GraspNet (Fang et al. 2020
Trajectory following via optimization. To achieve trajectory transfer, we represent the ob- ject's motion relative its initial pose, Teo. This yields sequence of relative transformations, (Too)), that define the object's movement. A reference trajectory is then generated by ap- plying these relative transformations to a chosen initial pose in the robot's task space in simulation or real life. Finally, trajectory optimization (Eq. 4) determines the joint torques necessary for the robot to imitate this reference trajectory. We employed the publicly available dynamic model for the Franka Emika Panda robot from the example eooeeerr-ooottoooe bobot-data data package (available via Conda and PyPI). which provides kinematic, geometric, and inertial parameters for each robot link. For utilized the P Pinocchio library (Carpentier al. 2019), also employed internally by the A1 Aligator trajectory optimization package (Jallet et al. (023). Given a sequence of joint torques, the dynamic model computes corresponding joint positions and velocities.
forward dynamics computations,
18

---

Published as a conference paper at ICLR 2025
Forward kinematics, based on the kinematic model, then maps these joint states to the object's pose. Consequently, the optimized joint torques (Eq. 4) directly influence both the end-effector pose and the joint velocities.
B METRICS
B.I SINGLE FRAME METRICS
Our method is evaluated on three different metries, Complement over Union, Chamfer Distance and Projected Chamfer Distance. Each of the metrics is described in more details below.
Complement over Union (CoU) is defined as IoU between the rendered mask of the ground truth mesh under the ground truth pose and the predicted mesh under the predicted pose. This metric measures the visual alignment of the masks well the silhouette appearance of the two objects.
Chamfer Distance (CH) standard measure used compare similarity of two pointclouds. This metric was chosen since the only way to compare quality of the pose estimate in 3D is via geometric aligmment of the ground truth (GT) and predicted meshes under GT and predicted poses accordingly. This metric takes into account both pose and scale predictions and hence allows to see full performance of our model 3D.
Projected Chamfer Distance (pCH) is complementary to the full Chamfer distance since it allows to evaluate how well estimated poses reproject into the image, essentially marginalizing over the scale in the chamfer distance. Inspired by the MSPD (Maximum Symmetry-Awan Projection Distance) metric used in the BOP challenge, PCH is given by
PCH
1 [M] REM
Σ
minl/ (z, K, T yEM
o (v, K, T
1 Σ yEAT
min K, T) ग (v. K, T)
M
aEM
where K is camera intrinsics, a predicted pose, f is ground truth pose, M is a set of points sampled from the predicted mesh, M is set of points sampled from the ground truth mesh, x and y are vertices of the meshes, and function projects 3D vertex into 2D pixel. The of this metric is to evaluate how well the estimated pose visually aligns when projected onto the image plane. As the closest vertex found for each vertex of other mesh, this metric can be used for non-identical meshes or for symmetric meshes. This approach allows us to assess alignment quality, even when the predicted scale is not exact, since the scale is factored out during the projection.
idea
B.2 TRACKING METRICS
To allow comparison of different methods, we collect 32 videos consisting of 15 instructional YouTube videos and 17 egocentric videos from the EPIC-KITCHENS (Damen et al. (022) dataset. These videos contain altogether 6947 frames. We manually create the ground-truth annotations by visually aligning the semi-automatically retrieved mesh from the Objaverse dataset with the manip- ulated object in the video. When comparing the estimated object trajectories of different methods, focus more the relative object motion rather than precise 6D object trajectory. To this end, we use three metrics that measure spatial velocity error in rotation, projected translation, and depth. Specifically, compute errors the average by a temporal distance 6, where 5 E and I is a set containing 10 integer values linearly spaced from [N/2]. Here, N represents the total number of frames in the video, and [-] denotes the floor function, rounding down to the nearest integer. The overall errors are computed as:
over all pairs of poses in video frames separated
6
rot 2,445
(5)
N
8
8
dET
1=
N
1
pnyj
proj
(6)
N
1,446
19

---

Published as a conference paper at ICLR 2025
N-6 8 N
e
deppt
depth 2,646-
8
8
SET
where e 6. 1,5+5 and depth 2,3-5 are errors computed between frames i and i--ô for rotation, projected translation and depth. These emrors are divided by the time difference ô to obtain relative error per frame. The inner sum computes the error across the N - 5 frames of the whole video, which is then normalized by N- 8 to obtain the average error. With the averaging over multiple time differences 5 E I (the outer sum), we penalize inconsistencies in velocities both locally (6 1) and globally 8 [N/2].
Rotation error. Given the estimated rotations R. R, and the ground-truth rotations R, Rí for frames i and j. we compute the rotation error e.j as the error of the ground-truth and the estimated rotation spatial velocities in the camera frame, expressed in degrees. In addition, we minimize the rotation error over the set of discretized ground-truth object symmetries S, when the object symmetry axis is defined by the human annotator (if available). The error is computed as:
Jor
min SES
log R R.) - log (R
(8)
where log SO(3) logarithm (Sola et al. 2018). Note that because we compare spatial velocities when computing rotation error, having different meshes for ground-truth annotation and the estima- tion does not affect the rotation error. However, this not necessarily true for projected translation and depth errors, as explained in the following paragraph.
Projected translation error. When computing projected translation error, we need to consider that we can have two different meshes for the ground-truth and the estimation, as their coordinate system origins do not necessarily have to project to the same 2D point in the image. As illustrated in Figure [ the velocity of the projected object center can change differently, depending on the 3D position and the rotation of the object. We therefore recompute the translational part of the estimated object trajectory to new trajectory t" that corresponds to the same object motion but with a changed origin of the object coordinate system. In the best-case scenario, where the ground- truth and estimated trajectory would be exactly the same up to the choice of the object origin, would shift the estimated object origin so that is the same 3D point as the origin of the ground- truth object when expressed in the camera coordinate frame. This is, however, impossible because the objects can have different velocities in different video frames, and also because of the ambiguity between the object's depth and scale. We, therefore, use the following approach instead: In each video frame i, first compute a point that lies the ray to the ground-truth object center, i.e. C, but having the same distance from the camera as the estimated object. Then, we express this point in the coordinate frame of the estimated object and compute the mean over all video frames, resulting in the new object origin o":
T
/
||t/]l-
N
Finally, by changing the object coordinate system, we compute the new translations
(10) Note that by using this approach, the positions of the mesh vertices do not change, but we can compensate for undesirable errors caused by having trajectories of two different meshes. In addition, limit the change of object origin by half of the estimated object scale to avoid significant unintentional error decrease in special cases of degenerate trajectories.
to = + R,o".
Given the ground-truth truth translations é and the corrected translations t", we now compute error of the projected translation e as the error between the spatial velocities of the projected object origins:
100
(m(() K) x (t,,,K) (n (É,, K) - (Lg, K K|)l
(11)
V + h2
projects 3D point to 2D image coordinates using camera matrix The first difference in Equation [] measures velocity for object with corrected translation, while the second difference measures the velocity for object with ground-truth
where the function πxx K
K described in Section A.2
20

---

Published as a conference paper at ICLR 2025
Figure 7: Visualization of object center projection. When evaluating translation and depth error, the ground-truth trrth (green) and estimated (red) meshes can have different object centers, causing the spatial velocity between two frames differ. We illustrate this in the image by showing a manipula- tion of an object between two different frames (left) and j (right) with visualization of trajectories of two different meshes representing the motion of the same manipulated object. We overlay video frames with mesh contours and projected object centers, and corresponding velocities (dashed lines). Even though both red and green objects reasonably describe the motion of the manipulated object, the spatial velocities of the projected objects are different, requiring change of the object coordinate frame to prevent an unintended increase the measured errors.
Table 5: Comparison with the state-of-the-aat in 6D object tracking in the wild on the depth metric. The 6D object tracking results of our approach are compared to MegaPose (Labbé et al. 2022) and FoundPose (Örnek et al. 2024). Even though our method outperforms baselines on the depth metric, note that in our experiments, we did not find the depth error to informative. In be outperformed by simple baseline that has zero velocity in the depth direction, resulting in depth error of 0.02899. We hypothesize this could be due to, e.g. the manipulated objects in both instructional and egocentric videos tend to move less in the depth direction, and due to difficulty of the manual ground-truth object annotation performed by visually aligning an approximate object mesh in the video.
particular, all compared methods
MegaPose (coarse)
MegaPose
FoundPose
Ours
Ours (refined)
Relative depth +
0.1198
0.1192
0.2592
0.0801
0.0800
translation. To consistently compare methods on videos of different resolutions, we normalize the етror and express it as percentage of the video's diagonal size.
Depth error. In a similar manner as the other two metrics, we measure spatial velocity error of the estimated and ground-truth depths. However, normalized by the object scales and a:
6,,6 - -1(c d,)/s (d/-s (d: d,)/8
depth
(12)
where d; || the estimated depth and d, || is the ground-truth depth. our experi- ments, we have found this metric to be insufficiently informative, as all compared methods can be easily outperformed by a trivial assumption that the manipulated object does not move in the depth direction at all, achieving depth error of 0.0289 compared 0.0801 of our method. We hypoth- esize that this is due to two factors. First, objects both instructional and egocentric videos tend to move less in the depth direction. Second, it is hard for human annotator to annotate depth by visually aligning an object in the video, especially when the aligned object does not perfectly match the shape of the depicted object. We show the complementary depth metric results in the Table 5
21

---

Published as a conference paper at ICLR 2025
Table 6: Influence of different 3D model retrieval methods on 6D pose estimation. Oracle (gray) shows an upper bound on the performance when the exact meshes of the objects depicted the image and their scales are known. Our approach outperforms the multimodal OpenShape representation. Additionally, we observe that the model achieves the strongest performance using
features from layer 22 of the DINOv2 model.
Retrieval Method
ARt
ARcaut
ARait
ARpcHt
OpenShape (Liu al.
2024a)
34.51
20.25
10.59
72.69
Ours (layer 18 CLS)
46.42
39.78
12.49
86.99
Ours (layer 20 - CLS)
48.36
43.46
13.97
87.66
Ours (layer 22 CLS)
45.05
40.93
16.16
78.06
Ours (layer 24 CLS)
41.14
36.10
15.62
71.68
Ours (layer 18 FFA)
47.40
43.68
15.35
83.16
Ours (layer 20 FFA)
48.46
44.67
17.37
83.33
Ours (layer 22 FFA)
49.86
45.20
18.53
85.83
Ours (layer 24 FFA)
46.73
42.04
17.32
80.84
Oracle CLS
63.31
52.42
46,43
91.07
Oracle FFA
62.93
51.99
45.95
90.85
C ADDITIONAL ABLATIONS AND QUANTITATIVE RESULTS
C.I ABLATION OF CAD MODEL RETRIEVAL AND ALIGNMENT
In Table we explore how the overall performance of the pipeline changes by choosing tokens from different layers the DINOv2 model. We also provide comparison to using CLS token baseline inspired by CNOS (Nguyen et al. 2023). In experiments, pose estimation using patch features from the layer 22 of the DINOv2 model yields the best per- formance. Compared the related work of FoundPose Örnek al. 2024), which used layer 18 in their pipeline, our method achieves optimal performance using layer 22 features, indicating that our task needs more semantic information, which is predominately in the last layers of ViT models Amir et al. 2022
observe that doing retrieval and
C.2 EFFECT OF PROPOSAL GENERATION ON SCALE ESTIMATION
Our scale estimation approach uses all detected objects in the image compute the absolute object sizes. If the detections are of poor quality, the estimated object sizes will be incorrect. Therefore, validate object proposal methods on the YCB-V dataset using the standard object detection met- rics (average precision and recall) as well as the median scale eттоr relative to the ground-true -tuth. In Table we show that our approach based on an off the-tff object detector and mask segmenter significantly outperforms the widely used proposal generator method CNOS (Nguyen et al.) (023). Crucially, we observe the CNOS method oversegments the scene and produces a large amount of spurious proposals. The number of spurious proposals of CNOS can be over 50%, negatively af- fecting object scale estimation even as our approach uses an outlier rejection method to cope with incorrect proposals and scale estimates. Finally, also show the scale estimation error when using the ground truth object masks. Interestingly. the scale estimation error in this case worse than using our method to obtain the masks. This be explained by the fact method detects additional objects in the background, such as chairs, that are not in the ground truth masks but help with the scale estimation.
C.3 ABLATION OF NUMBER OF VIEWPOINTS
To examine the effect of the rotation sampling on the overall pipeline performance, we perform an ablation study YCB-V dataset shown in Table 8 in which increase the number of sampled views in the utilized sampling strategy (Alexal 2022). In our method, we use N = 600 samples, which, average, leads to 250 geodesic error between the closest rotations. When increasing
22

---

Published as a conference paper at ICLR 2025
Table 7: Effect of proposal generation methods on scale estimation, precision, and recall. We compare our proposal generation approach to the CNOS method (Nguyen et al. (023) with Fast- SAM segmenter (Zhao et al. 2023
Method
Scale error [%]] +
AP1
AR@ 10
Proposals per image
CNOS (FastSAM)
118.55
61.60
71.80
34
Ours
14.42
69.70
76.70
6
GT Masks
16.65
100
100
N to N = 1200 N 1800, we observe that the overall pipeline performance remains similar, while the computational requirements for storage and runtime increase linearly. Our prerendered mesh database takes ~ JTB of disk space for N 600 views, ~2 TB for N 1200 views and ~3 TB for N = 1800 views. The runtime also scales linearly from ~0.2s per object to ~00.4s per object for 1200 views and ~0.6s per object for 1800 views.
Table 8: Ablation of Number of Viewpoints. The overall pose estimation performance compared to our choice of N = 600 views is not substantially affected by increasing the number of rendered viewpoints of each object, while the overall runtime and storage requirements increase linearly.
N samples
ARt
ARcaut
ARcht
ARpcht
Avg Err
600
49.86
45.20
18.53
85.83
250
1200
49.66
44.64
19.10
85.24
20
1800
49.10
43.95
18.25
85.11
16
C.4 LLM MODEL ABLATION
To evaluate the dependence of our scale estimation method on the choice of LLM, we conducted additional ablation study using three other models: one proprietary (GPT-3.5-Turbo) and two open-source models (1lama-3.1188) I.lam Team (024)) and Gemma2-9B Gemma Team (2024)) We adapted prompts for each model ensure the generation of a scale database in the expected format and the best possible coverage of real-world objects.
The results of our experiments are shown in the Table We observe that GPT-4 outperforms all other models, with Llama-3.1 -.1-BBB achieving similar score. Surprisingly, GPT-3.5-Ture performs the worst, while the Gemma2-99 model is slightly better than Gurro.5-Turbo but still lags behind GPT44 and 1amma-3.1-8B Please note that scale estimation does not affect the projected metrics and therefore ARcsu and ARpc remain constant.
Table 9: Ablation of LLM Model. While GPT-4 performs the best, the open-source Llama-3.1- 1- 8B model achieves a similar performance. The GPT-3.5-Turbo and Gemma2-9B models perform significantly worse in our study.
LLM model
ARt
ARcaut
ARcut
ARşCrt
GPT-4
49.86
45,20
18.53
85.83
GPT-3.5-Turbo
45.29
45.20
4.85
85,83
Llama-3. 3.1na-3.1--BB
49.32
45.20
16,92
85,83
Gemma2-9B
46.00
45.20
6.98
85.83
C.5 RESULTS ON DTTD2 DATASET
In Table 10 we provide additional quantitative results on the DTTD2 (Huang et al., 2023) dataset, which is dataset featuring 100 scenes and 18 objects, which partially overlap with the widely used YCB-V V dataset. On this dataset, our method outperforms all previous state-of-the-ar methods by large margin.
23

---

Published as a conference paper at ICLR 2025
Table 10: Comparison with the state-of-the-art on the DTTD2 dataset. The 6D pose estimation
results of our approach are compared to MegaPose (Labbé et al. 2022). GigaPose (Nguyen et al., 2024). and FoundPose (Omek et al. 2024). Our method outperforms all other methods in all the metrics.
Method
ARt
ARcout
ARcht
ARacHt
MegaPose (coarse)
26.15
13.72
6.67
58.05
MegaPose
27.84
17.73
8.77
57.00
GigaPose
41.84
39.32
12.16
74.00
FoundPose
37.39
31.32
11.41
69.44
Ours
47.11
52.10
12.68
76.54
D ADDITIONAL QUALITATIVE RESULTS
In Figures and we provide more qualitative results on the EPIC-Kitchens dataset, showing that our approach can work from egocentric point of view. In Figures 10 and [1] we provide additional qualitative results featuring extraction of object poses from the Internet videos and their transfer to the real robot. The robot trajectory is always computed closely imitate the relative 6D pose transformations of the detected object from the video. We only transform the trajectory from the camera frame into the robot frame to account for differences in scale and robot position. The perceived differences between the estimated object trajectory from the video, the object trajectory executed by the robot in the simulation and the object trajectory executed by the real robot are due to the differences of camera viewpoints.
24

---

Published as a conference paper at ICLR 2025
Figure 8: Additional results of our method on the EPIC-Kitchens dataset. Our method is able to detect and align objects from egocentric point of view. We show frames from the input vidco in the top row of each example, and the tracked 6D poses of the retrieved object together with 2D point tracks in the second row of each example. Please note the differences in object geometry and texture between the objects depicted in the video and the retrieved objects from the database.
25

---

Published as a conference paper at ICLR 2025
Figure 9: Additional results of our method on the EPIC-Kitchens dataset. Our method is able detect and align objects from egocentric point of view. We show frames from the input video in the top row of each example, and the tracked 6D poses of the retrieved object together with 2D point tracks in the second row of each example. Please note the differences in object geometry and texture between the objects depicted in the video and the retrieved objects from the database.
26

---

Published as a conference paper at ICLR 2025
Input video
Prediction
Simulation
Real robot
Input video
Prediction
Simulation
Real robot
Additional qualitative results on Internet videos. For each video, we first show several input frames, followed by the alignment obtained by method together with tracked 2D- 3D correspondences shown as colored tracks. Please note the differences in object geometry and texture between the objects depicted in the video and the retrieved (and aligned) objects from the database. Temporally smooth poses are used to compute robot motion via trajectory optimization as shown in the third row, followed by the video of real robot execution.
Figure 10:
27

---

Published as a conference paper at ICLR 2025
Input video
Prediction
Simulation
Real robot
Input video
Prediction
Simulation
Real robot
Additional qualitative results on Internet videos. For each video, we first show several input frames, followed by the alignment obtained by method together with tracked 2D- 3D correspondences shown as colored tracks. Please note the differences in object geometry and texture between the objects depicted in the video and the retrieved (and aligned) objects from the database. Temporally smooth poses are used to compute robot motion via trajectory optimization as shown in the third row, followed the video of real robot execution.
Figure 11:
28


Reviews:
[{"review_id": "3MTr8u7xBW", "reviewer": "Reviewer_x6Vw", "summary": "The paper introduces a pipeline for extracting the 6D pose trajectory from an internet video without the need of the CAD for the specific object. The authors leverage vision features to retrieve the most similar CAD model of the object, then do per-frame alignment leveraging the same vision features of the original image and rendered from the CAD. They further estimate the rough object size using LLM and leverage 2D tracking models to get inter-frame rotation consistency. The authors conduct experiments and demonstrate their superior performance. They also show demos that their trajectory can be retargeted to guide the movement of the robot.", "questions": "1. Why in Figure 1 and Figure 2, the same image has two different retrieved CAD models?\n2. Can you provide the results of the error based on the quality of the retrieved CAD model?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The task of predicting the 6D pose of internet videos without additional prior is important for a lot of downstream tasks.\n2. The whole pipeline is reasonable, fetch the similar CAD model and do rough alignment. Then further leverage the 2D tracking results to get the smoothed trajectories, that are more motion-consistent across time.\n3. The experiments on the retargeted motion on robotics further show the usefulness of the extracted smoothed trajectories.", "weaknesses": "1. The authors demonstrate that compared to model-based methods, whose performances suffer from the inaccurate CAD mode, their method addresses the challenge. However, there is lack of experiments compared to SOTA model-based methods with their fetched CAD models (e.g. FoundationPose with their retrieved CAD model).\n2. In the 6D pose alignment part, the method applies a sapling-based trajectory to get the rotation, which potentially limits the accuracy of the rotation. In the results figure, there are some rotation errors, not sure if due to the sampling-based strategy or the DINO feature extractor.\n3. For the robotics demo, the end-effector position control is on 6D pose or only on the rotation? From the Figure 9, the translation of the end-effector seems not consistent with the original video and in the simulator", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OEAqMHok44", "reviewer": "Reviewer_JJFi", "summary": "This paper proposes a new approach to detect and track the 6-DoF pose of unknown objects from RGB video. The approach is motivated by robot imitation learning from internet video. The approach uses off-the-shelf open-set object detectors, foundation models for segmentation, vision-language (CLIP), and visual features (DINOv2) to detect objects, retrieve similar shapes from a database of CAD models, and matching the object image with a set of rendered views of the object CAD model to estimate 3D orientation. Experimental evaluation is performed quantititvely on YCB-Video and HOPE-Video datasets and a comparison is made with state of the art object detectors for unseen objects for which the CAD model is assumed known (MegaPose, GigaPose). Also, qualitative results on EPIC-Kitchen, and an example of executing the estimated object trajectories on a real robot are shown.", "questions": "- l. 323, are the ground-truth meshes contained in the object datasets? \n- Table 1, was the same scale estimate for the meshes used for MegaPose and GigaPose like for the proposed method? \n- Which dynamics model is used for the optimization problem in eq 4? How is tracking of the optimized trajectory implemented?\n- See additional questions in sec. \"Weaknesses\".", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed approach for detecting and estimating 6D motion of unknown objects from RGB images is novel and interesting.\n- The paper is well written and easy to follow.\n- The set of experiments demonstrate the shape retrieval and pose estimation well and also compare with state of the art methods.\n- A qualitative example is provided with a real robot which show the robot pouring from one object to another.", "weaknesses": "- l. 197ff, CAD model retrieval by rendering views and calculating visual features seems expensive in both, the database generation and the retrieval stage for large datasets such as Objaverse-LVIS. What is the retrieval time for these datasets and how is it implemented to make retrieval efficient?\n- l. 220ff proposes to retrieve rotation by matching to a set of rendered views. What is the choice of N in the experiments? What is the avg/std angular distance between sampled rotations?\n- l. 243ff, the way to prompt the LLM in the supplementary is an offline procedure to collect size estimates for approximately 2200 objects. In the main paper, the description reads as if the LLM is prompted for each detected object using the CLIP text classification. Please describe this more clearly. What if the detected object is not included in the offline calculated set ? \n- l. 286, was estimating the motion of the camera relative to the static background evaluated in this work ? Please clarify.\n- The optimization problem in eq 4 does not provide a description of the used system dynamics model. \n- l. 361, please write more clearly, that while a similar mesh is known, the retrieved mesh does not exactly correspond to the ground truth mesh which is an assumption used for MegaPose and GigaPose.  \n- Please introduce the pCH metric formally, at least in the supplemental material. The current description is insufficient.\n- l. 519ff, the real robot experiment is rather anecdotal and lacks important details in its descriptions and quantitative evaluation (e.g., success rate). How are the observed object trajectories transfered to the real robot experiment incl. considering the change of view point and embodiment? How does the robot know where the manipulated objects are and how is this matched to the observed object motion? \n- Fig. 8, in the upper additional qualitative result, the bowl object pose is not correctly tracked. Why does the robot still turn the object in a quite different angle ?\n\nAdditional minor comments:\n- Fig. 6, rightmost real robot image seems to be a repetition of the image next to it. Was the wrong image included?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eTQdoTkw10", "reviewer": "Reviewer_4DS1", "summary": "The authors present a novel approach to extract temporally consistent 6D pose trajectories of manipulated objects from Internet videos to be applied with robotic manipulation task. It tackles the challenges posed by uncontrolled capture conditions, unknown object meshes, and complex object motions. Their evaluation on YCB-V and HOPE-Video datasets shows state-of-the-art performance, with successful motion transfer to a robotic manipulator in both simulated and real-world settings.\n\n----------------------------------------------------------------------------------------------------\nThe authors addressed most of my concerns in the rebuttal phase, and thus I would like to raise my score to 6.", "questions": "1. It might be great if the authors could ablate on the performance variation under different LLMs. Currently it only applies GPT-4, but it is important to know how different LLMs might influence the performance (i.e. one GPT-3.5 & one open-source LLM).\n2. What's the efficiency & cost of such pipeline when performing inference on a 1-minute Instructional videos? \n3. Using a CAD model can be costly since it requires a large database to store predefined meshes, and in open-world scenarios, finding an exact match is often unlikely. However, numerous approaches avoid relying on CAD models. For instance, \"6DGS: 6D Pose Estimation from a Single Image and a 3D Gaussian Splatting Model\" [ECCV 2024]. Have you tried experimenting with such methods? Or say, how do you envision those methods' strengths and weaknesses compared to your method.\n4. For the standard evaluation, it might be beneficial to add another dataset evaluation using different cameras, say iPhone sensor as proposed in \"Robust 6DoF Pose Estimation Against Depth Noise and a Comprehensive Evaluation on a Mobile Dataset\" to further validate the approach's generalizability.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The impact of the paper is dominant in the way that it provides an envision of enriched data for robotic manipulation without human labor force to construct the specific datasets. The methodology is intuitive and the performance enhancement is non-trivial. The paper is overall well-written.", "weaknesses": "My primary concern lies with the methodological novelty, as the approach largely involves applying an existing pipeline to internet videos. Specifically, the use of an LLM for estimating object scale may be questionable, given potential uncertainties around its accuracy in providing a realistic scale for each object. Aside from this, the methodology essentially adapts previous methods to fit the proposed pipeline. Given these factors, I feel this work might not align with ICLR's focus but could be more suited to a robotics conference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0m58ucuesg", "reviewer": "Reviewer_TaH5", "summary": "The paper pays attention on 6D pose trajectory estimation of a manipulated object from an Internet instructional video with a novel framework. The framework first predicts the 6D pose of any object by CAD model retrieval. Then the smooth 6D object trajectories are extracted and retargeted via trajectory optimization into a robotic manipulator. Experiments on YCB-V and HOPE-Video datasets demonstrate the improvements over RGB 6D pose methods. Moreover, the 6D object motion can be transferred to a 7-axis robotic manipulator.", "questions": "1 With the similar CAD model retrieval, the classification can also be obtained. I wonder if it is possible to use the CAD model to perform classification directly?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1 The pose estimation method by retrieving a CAD model, aligning the retrieved CAD model with the object, and grounding the object scale with respect to the scene.\n\n2 Consistent 6D pose trajectory estimation from Internet videos and retargeting trajectories to a robotic manipulator.\n\n3 The pose estimation improvement on YCB-V and HOPEVideo datasets, and transfer from 6D object motion to a 7-axis robotic manipulator.", "weaknesses": "1 The original contributions should be expressed more clearly. In the proposed method, various existing methods are employed. It is suggested to clearly distinguish the original contributions in this paper and usage of other methods. Specifically, the first contribution locates in the pose estimation method by retrieving a CAD model, aligning the retrieved CAD model, and grounding the object scale with respect to the scene. The subsequent question is that what is the original contribution, the whole pipeline or the detailed design of a particular module? The authors are suggested to express this more clearly in the revised version. For the second and third contributions, it is also recommended to present more clear expressions. \n\n2 For robotic manipulation, the running time of the pose estimation method is a key factor. The proposed method in the paper is somewhat time-consuming with 2s for detector, retrieval and scale estimation per scene and 0.2s for pose estimation per object. To further improve the paper, two suggestions are given. For one thing, the comparaions with other methods on running time are suggested to add.  For another, more analysis about the running time is also preferred, such as the recommendations for accelerate the whole method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)