You Only Look at One: Category-Level Object Representations for Pose Estimation From a Single Example
Authors: Walter Goodwin
Ioannis Havoutis
Ingmar Posner
Venue: CoRL 2022 Poster (2022)
Keywords: pose estimation
object representations
one-shot

Abstract: In order to meaningfully interact with the world, robot manipulators must be able to interpret objects they encounter. A critical aspect of this interpretation is pose estimation: inferring quantities that describe the position and orientation of an object in 3D space. Most existing approaches to pose estimation make limiting assumptions, often working only for specific, known object instances, or at best generalising to an object category using large pose-labelled datasets. In this work, we present a method for achieving category-level pose estimation by inspection of just a single object from a desired category. We show that we can subsequently perform accurate pose estimation for unseen objects from an inspected category, and considerably outperform prior work by exploiting multi-view correspondences. We demonstrate that our method runs in real-time, enabling a robot manipulator to rearrange previously unseen objects faithfully in terms of placement and orientation. Finally, we showcase our method in a continual learning setting, with a robot able to determine whether objects belong to known categories, and if not, use active perception to produce a one-shot category representation for subsequent pose estimation

You Only Look at One:
Category-Level Object Representations for Pose Estimation From a Single Example
Walter Goodwin, Ioannis Havoutis, Ingmar Posner Oxford Robotics Institute University of Oxford f firstname@robots. ox ac uk
Abstract: In order meaningfully interact with the world, robot manipulators must be able interpret objects they encounter. A critical aspect of this inter- pretation is pose estimation: inferring quantities that deseribe the position and orientation of an object in 3D space. Most existing approaches to pose estima- tion make limiting assumptions, often working only for specific, known object instances, or at best generalising to an object category using large pose-labelled datasets. this work, we present a method for achieving category-level pose es- timation by inspection of just single object from desired category. We show that we subsequently perform accurate pose estimation for unseen objects from inspected category, and considerably outperform prior work by exploiting multi -iel correspondences. We demonstrate that our method runs in real-time, enabling a robot manipulator equipped with an RGBD sensor to perform online 6D pose estimation for novel objects. Finally, we showcase our method in con- tinual learning setting, with a robot able to determine whether objects belong to known categories, and not, use active perception produce one-shot category representation for subsequent pose estimation.
Introduction
Novel category detected
Add representation
New object
One-shat pase estimation
i
Exattig calepory representatoons
Category before?
No
category
Figure An overview our method. Multiple views an object from new categories are taken and turned into category-level representation in 10s process. When novel object is encountered, it is compared to all current representations. If it is deemed to belong to previously captured category, method performs 6D pose estimation in real-time. If it is not, it scanned and The approach runs in real-time and representations allow accurate pose estimation unseen objects from captured categories.
category-level representation acquired.
Many practical applications of robotics require that a robot is able to estimate the poses of objects that are encountered, that is, the parameters that describe their position and orientation in 3DD space relative to some coordinate frame. This capability is central to object manipulation, where to suc- cessfully rearrange objects, robot needs to estimate object pose to infer how it differs from some
6th Conference Robot Learning (CoRL. 2022), Auckland, New Zealand.

---

goal state [1]. Object pose estimation can also be helpful for mobile robots seeking to localise themselves in the world, by providing semantically grounded anchors [2, 3].
Despite its central importance robotics, approaches for object pose estimation remain limited in their abilities generalise novel objects. State of the art approaches often demand large labelled datasets or accurate CAD models just to produce pose estimators for single object instances [4, 5]. Over the last few years, attempts have been made to alleviate these issues by learning category level pose estimators that train over multiple object instances from a category and can generalise to unseen instances from that category [6, 7, 8]. Such generalisation to unseen objects a much sought-after property in robotics, with some key examples being generalised manipulation [9, 10, 11] and object- based SLAM [2]. However, these category-leve approaches to pose estimation still have substantial data requirements in order generalise- for instance, the Objectron dataset [12, 8] has over 17,000 object instances and 4 million frames cover 9 object categories.
Separately, some recent works have achieved very promising results on pose estimation for single object instances by avoiding object-specific deep models, instead collecting a set of reference views of an object before performing robust feature matching for view retrieval and pose estimation [13, 14, 15]. These approaches remove the need for large curated datasets required by deep learning approaches, but designed for single object instances and thus fail to exhibit desirable category- level generalisation.
In this work, we present an approach to object pose estimation that has the desirable properties of both of these paradigms. By leveraging the pre- pre-trained features of a deep vision transformer (ViT) network [16], we show that we are able to leverage multiple views of a reference object to achieve pose estimation on any novel object from the same category. Extending recent work on leverag- ing such features for pose estimation [17], we demonstrate much improved performance by also making use of multiple views of the novel target object. Further, in contrast to prior work, both the processing of object reference views into reference object 'model' suitable for downstream pose estimation, and the pose estimation process itself, can run in real-time and are thus suitable for in-the-wild robotic deployment. We demonstrate that our method drastically outperforms alter- native methods in this one-shot category-level pose estimation setting, and show that this approach enables a robot manipulator to perform generalised manipulation tasks, in which a single 'goal' pose for an object category is inspected by robot manipulator, which then performs pick-and-place rearrangement to satisfy this goal pose when confronted with novel items from the same category.
In summary, our main contributions are:
A training-free, multi-view approach to object pose estimation that only requires views of a single reference object, yet generalises to novel objects from the same category.
A fast implementation of our pose estimation method that runs orders of magnitude faster than the closest baseline, enabling both real-time capturing objects from novel cate- gories, and real-time pose estimation for objects from known categories.
Validation of the method's utility in a continual learning setting, in which the method deployed tabula rasa and distinguishes novel categories from known categories, learning representations where appropriate.
2 Related Work
2.1 Category-level object pose estimation
Approaches to category-level object pose estimation can generally be divided into those which learn a category-level representation against which novel objects and images can be compared [6, 18, 19, 20, 21, 22], those which leverage (a set of) CAD models for a category [23, 24, 25, 26), and those which train deep models to directly regress to parameters describing object pose [27, 28, 29]. These methods require large pose-labelled datasets containing multiple instances of objects from each category of interest in order to generalise successfully.

---

In contrast, in this work we build on a recent paper that proposes novel zero-shot category-level pose estimation setting [17]. in which, given a single image of some reference object and no pose- labelled data, the task is to estimate the pose of novel objects from the same category, using a handful RGB-D views. Building on [17], our work enables the use of multiple reference object views, and we show that this leads over 5 seconds to estimate pose for single frame, our method can be run at over 15 Hz, enabling real-time pose estimation robot manipulator.
considerable improvement in performance. Further, while [17] takes
2.2 Template-based object pose estimation
In making use of collection of reference views of an object for pose estimation, our work is related to the literature on template matching- Template matching for pose estimation entails collecting many views of a reference object, and retrieving the closest view at run-time through visual simi- larity measure. Recent work achieves impressive performance for single instances by constructing point cloud of an object, and leveraging learnt 2D>-to-3D correspondence matching and Perspective- n-Point (PnP) to recover pose when the object is next encountered [30]. This method just requires an object point cloud to perform pose estimation for novel objects. Similarly, [15] uses a two-step matching process with dense CNN features, before a final PnP step is used to estimate object pose. Similar viewpoint-matching approaches have been used for pose tracking during manipulation [14]. [31] solves a continual learning case, in which 3D object representations are captured when an object first encountered by a robot, and subsequently matched against in render-and-compare manner to provide 6D pose estimation. Template matching approaches have recently been employed in few-shot setting, where following pre- pre-training stage on diverse dataset, just a few views of novel object are sufficient to estimate accurate pose [32]. All of the aforementioned methods enable pose estimation of novel objects provided that a collection of reference views, or a 3D model, can be acquired. However, the matching processes they employ are instance-specific, meaning that object templates cannot be used for pose estimation of previously unseen objects from related categories.
2.3 Category-level object representations for manipulation
Several prior works seek to learn category-level object descriptors to enable generalising robot ma- nipulation skills from single object instances to broader object categories. [9] learn dense object descriptors that can be trained in a mostly self-supervised manner from multiple views, and show that for several categories, after training on 15 object instances nearest-neighhours in descriptor space between different objects tend correlate with meaningful correspondences. Other work has sought to learn category level keypoints [10]. Manipulation goals described as costs keypoint positions can then be achieved on novel objects from known categories. More recently, category- level descriptors have been learnt on 3D object representations I]. by self-supervised training over a large set of CAD models from a category. These descriptors enable finding oriented correspon- dences between semantically equivalent points on objects from a training category, and have been used to place novel objects in target poses with a robot manipulator. These prior works require time-consuming training, over datasets collected from a not inconsiderable diversity of objects from category of interest, and thus do not scale well. In this work, we show that dense object descriptors that are similarly able to produce meaningful part-based correspondences between objects from given category can be achieved through inspection of just single object instance in real-time. As in prior work, these descriptors can be used to match object poses, but in a considerably more scalable manner, with minimal data requirements.
3 Methods
3.1 One-shot pose estimation setting
To formalise the setting for one-shot pose estimation addressed in this work, adopt and extend the notation from [17]. We consider an object from category to be represented by M views, to
3

---

produce reference image set, IR Except in certain ablations, we assume that depth images are also available from these views, DREM- We assume that camera extrinsics are known, a realistic assumption in most robotics contexts (e.g. a manipulator with a wrist-mounted camera), and denote these as TREM for the reference object views. At pose estimation time, a target object is encoun- tered, and we assume that N 1 views of this object are also captured, with RGB images I depth images D and camera extrinsics TRI.x. We seek to recover a rotation R and translation t that describes the pose of the target object with respect to the reference object.
The reference object in the one-shot pose setting serves both model against which target object correspondences for pose estimation are made, but also to establish a canonical frame against which target object pose is measured. Previous work describes similar setting in which just a single ref- erence view is used as entailing 'zero-shot' pose estimation, because this reference view is a bare minimum in order to render the pose-estimation problem well-posed. In our setting, because multi- ple reference views of an object are aggregated to form a representation for category. we choose to describe this as one-shot pose estimation. Estimating the pose of novel object relative to the reference object is sufficient to generalise manipulation behaviours demonstrated on the reference object.
3.2 Descriptors and correspondences
[17] uses features extracted from a vision transformer network (ViT) for category-level pose estima- tion, showing that they generalise well across instances within a category. In this work we use the same ViT, trained with DINO [16], an unsupervised contrastive method that uses a multi-scale crop- ping process during training which, intuitively, encourages the network to discover correspondences between small local crops of an image, and larger global crop. Empirically, DINO's features can identify objects but also object parts. Further, ViTs take a patch-based approach to image processing which retains relatively high spatial resolution throughout the network, in contrast to CNNs, which spatially downsample such that later layers with expressive features lack spatial resolution. By using a ViT with a patch size of 8x8 pixels on RGB images resized to 224x224, the resulting feature maps at all layers are 28x28. This enables patch- based correspondences to be localised accurately in the image, and thus in 3D space through backprojection. We denote normalised feature maps extracted from image as & (It) E R 282828x where D is the descriptor dimensionality (see Section 3.3).
We construct correspondences between the reference and target objects by finding P patch corre- spondences between every reference-target feature map pair (1(,), (1T,, ) pair /R., IT we denote these {ui,, Uix J.IPP. where u, are indexes into the respective feature maps. The problem of finding strong correspondences between two feature maps Φ1, 42 is much studied, with many proposed solutions. Naive nearest neighbours can result in many-to-one matches that are physically implausible and thus not appropriate for downstream pose estimation. Mutual nearest neighbours [33] is a stronger condition, in which points u, v considered correspondences only if the closest descriptor to a point in Φ is (the standard nearest neighbours condition) and the closest descriptor to in 4 is u. Formally, for e = argmax,,, d( T2..)) ). we have u' = argmax, d( & )1_)) = u, where d(-,-) is the cosine similarity. Requiring this perfect 'cy- cle' between the descriptors reduces spurious correspondences, but cannot guarantee that a certain number- or indeed any correspondences will be found, which affects downstream pose estimation in our method.
/E1:N I' I.M For an image
To ensure that P correspondences are found for each reference-target pair, we use the cyclical dis- tance metric proposed in [17], which can be thought of as a relaxation of mutual nearest neighbours matching: there are < P mutual nearest neighbours, additional correspondences are given points for which - u as defined above is minimal. This acts as a spatial prior: correspondences that are almost spatially consistent under the cycle are preferred: u' and are likely to still belong to the same semantic part of an object. We also experiment with novel variant of this approach, in which the 'cyclical distance' is measured in 3D world coordinates rather than pixel space, using the back projeeroeected pixel coordinates (details in supplementary).

---

3.3 Dimensionality reduction
In well-established approaches to learning category-level dense descriptors for robotic manipulation (e.g. (9]), descriptor dimensionality tends to be low. The authors of [9] note that for single instances, a 3-dimensional descriptor space tends to be sufficient to discriminate between object parts, though this requirement scales somewhat for multi-object descriptor networks. The raw ViT features we use as descriptors in this work are 384-dimensional. Such dimensionality is critical to the network's ability satisfy the contrastive objective over the diverse dataset it is trained on. At the pose estima- tion stage, would like to retain those factors of descriptor variation that enable to distinguish different object parts and viewpoints, but it is natural to wonder whether there may be considerable redundancy in this descriptor space. To explore this, we perform PCA on the descriptors taken over all reference object views. The majority of descriptor variance is captured in a relatively small num- ber of descriptor dimensions (supplementary). Further, we find that projecting features onto a small subset of principal components actually results in improved pose estimation results. Certain results show in Table 2 and discussed in Section 4.1.1. The category representations in Fig. 1 the projections of descriptors onto the first three principal components.
In our experiments, we calculate principal components based on the reference feature maps only, and use these to project both reference and target descriptors prior to finding correspondences.
3.4 Viewpoint estimation
3.4.1
Aggregating multi-view correspondences
WI
09
Figure 2: For visualisation, point clouds are created by a threshold ViT attention masks mask each view, aggregating, and removing outliers. (a) The matrices show (lefi) the view-to-view similarities (sum top-K correspondences) for each reference-target view pair ( =high. =low). (right) the true geodesic error between the object poses in these views, which can be seen to be correlate with the similarities. Below, random subset of correspondences across all view pairs is visualised. (b) For each target view, the corresponding reference view with the highest similarity chosen, with these pairs highlighted in yellow on the matrix. A subset of the resulting filtered correspondences is shown. (c) The resulting inlier frames by consensus filtering with a threshold of 30" (Section 3.4.1) and correspondences. (d) The same with tighter inlier threshold of 15", (c) The inlier correspondence set following RANSAC. (f) Objects aligned by estimated pose, offset by translation for clarity. (g) Taking top-K correspondences globally from the whole matching process (rather than filtering good views) leads to conflicting and erroneous correspondences.
Following [17], we assign similarities S,j reference-target view pairs e.g. LR., IT, the sum of the similarities between the K correspondences for this pair, {ug. Dj. heEIPP, found as described previously. That is, Sūj- D/ d(ui. U/2). Considering all reference images IR. and target images I arrive at view similarity matrix SMxN, with elements SMxN [i. Sij- An example of such similarity matrix, and subsequent processing steps and their effect on the correspondence set, is shown in Fig. 2. We leverage this matrix to filter to a subset of best-fit view pairs. First, for each target frame we retain just the reference frame with maximum similarity. Subsequently, we further filter based on the estimated relative orientation estimates we would arrive if we took each pair to represent perfect alignment. With this set of relative poses, we find the largest consensus set by sampling many random poses in SO(3) and finding the viewpoint pairs with relative pose within some threshold 6 distannce from this pose. These largest such set is chosen. Correspondences from these remaining views are used in the final rigid body transform solution.
5

---

3.4.2 Final pose estimation
Given a filtered subset of reference-target view pairs which have a consensus on an approximate SO(3) pose prediction, we seek estimate refined SO(3) pose, along with translation, to produce full 6D object pose estimate, and a scaling, handle intra-category size differences. For this, assume that the reference and target objects are related by such a 7-D rigid-body transform, and solve for this using RANSAC and Umeyama's 's method [34]. With K correspondences per view pair and subset of Q view pairs, we have correspondences (uk: Ue) k=1:QK- Using reference and target depth maps DR.-I:M and ITy we back project these pixel coordinates to 3D points {uR, V=IR-I:QKK For each RANSAC trial we sample four pairs of corresponding points as required by Umeyama's method, and the estimated rotation R. translation i and scaling À satisfy
(2, R. 1)
arggmin (A.R.t)
Σ Vk - (ARus +
We run RANSAC for 1000 trials and make a final estimate using the largest resulting inlier set.
3.5 Continual category learning
An often overlooked upstream requirement of most object pose estimation methods used in robotics object detection and classification. Many approaches to category-level and instance-level pose estimation require knowledge of the target object's category or identity choose the appropriate network [8], network head [35], or template [30]. We would like to be able to leverage the one- shot setting presented in this work in fully autonomous context, where the category identities of objects encountered by a robot are estimated for subsequent pose estimation. Specifically, compelling setting in robotics is that of continual learning, where a robot encountering novel objects can either assign them to categories seen before, or determine that they are a novel category. In our case, the former would lead to selection of a suitable reference object for pose estimation, while the latter, the robot would employ active perception produce multiple views of this novel object, forming a representation for novel category. We denote the set of previously 'discovered' categories Ci.s. and given one more views of novel object TT... wish to determine the object's s identity Ć E [C1...Cs+1h: where Cs+1 would imply this object belongs to an unseen category. We seek a similarity metric that enables fast checking against a potentially large number of reference objects. For this, we use the [CLS] tokens from the last layer of the DINO ViT [16]. For an existing category, we represent it as the set of [CLS] tokens over all sequences and views that have been assigned it, such that the whole feature set is (V/ "h.,..... thp >==1: For new category with (CLS) tokens over its views 22y --y VrrG assign the category as:
F
G
argmax f(v.. Mx) if f(V, M2) > 0, where f(V.. M2)
d(os, 2z.))
(2)
Where 0 is threshold on similarity (details in supplementary). If no category meets this threshold, the object is assumed to belong to novel category.
4 Experimental Results
4.1 Multi-view one-shot pose estimation
To evaluate the performance of the proposed pose estimation method on a diverse range of categories and instances, we use a pose-labelled subset [17] of the Common Objects 3D (CO3D) dataset [36], for which the ground truth relative pose between video sequences of 10 distinct objects from each of 20 categories is labelled. Each sequence contains approximately 100 frames, and taken by smartphone in hand-held turntable-style object scan. Camera extrinsics and depth images in this dataset approximate, being recovered from RGB video sequences by Structure-free Motion approach [37]. While we find that (with appropriate real-time depth completion, described

---

Categories
Category (Acc
Method
Med. Er (P)
Acc30 (*)
Acc15
Acc7,5
B' pack
Car
Chair
Keyboard
Laptop
M M'cycle
TEASER++ [13]
1264
3.75
1.1
0.3
5
Goodwin2022 [17]
47.7
49.4
28.35
9.6
44
66
47
69
85
85
Ouns-R
13.4
Ours RC
32.7
56.6
35.0
13.1
64
74
58
57
91
82
Ours-If
36.9
45.7
26.6
92
80
97
Ours-UC
31.9
61.5
47.5
27.4
78
67
98
Ours-UCC
26.2
64.5
49.4
29.2
87
70
70
99
99
Ours-UCD-
54.6
100
Table 1: Pose estimation accuracy (orientation). We report Acc@0" with 0 E {30", ..5%,.5" }. giving the percentage of estimates that fall certain maximum threshold geodesic enror, and the median error 'Med. Err' (per category, then averaged). We compare to TEASER+ [13], Goodwin2022 [17]. Suffixes on our method ablations: R: retrieval; pose estimated based on mean of most similar views. C: consensus, iterative removing far-from-mcan views. U: Umeyama's 's method; rigid body solution using best-view correspondences. D: descriptor dimensionality reduction (to 32 components). Methods use 10 reference and target views (Good- win2022 uses just reference). indicates 30 views used.
supplementary) the extrinsics and depth are of sufficiently high quality, we consider this dataset be lower bound on data quality from wrist-mounted RGB-D camera.
The results of method and certain ablations are shown in Table We compare the method of [17], which uses a single reference image, and to TEASER++, a fast point cloud registration algorithm that works the presence of a large number of outliers, unknown correspondences, and unlike ICP does not require a strong initial guess. Further, it leverages an explicit bound on the noise it assumes present the non-outlerr points, which is analogous to the inlier threshold we use in our RANSAC stage (see supplementary). and thus forms an interesting baseline for leveraging purely geometric correspondences for category-level pose estimation.
The improvement over [17] can be seen especially strongly in at the more precise Acc@ 15 and Acc@7 @7.5,, where our full method demonstrates almost double and triple the accuracy respectively, and scores 20.4% higher on Acc@ 30%. This underlines the importance of aggregating many-to-ma to-many view comparisons to arrive at pose estimates. While [13] shows better-than-randee for most categories, indicating that stronger priors than just geometric distances necessary in this setting. Further results, including those for translation errors, are in the supplementary.
performance not in general able to solve the category-level pose estimation problem,
4.1.1 Effect of descriptor dimensionality reduction
In Section 3.3, we discuss our approach to reducing descriptor dimensionality. Table 2 shows the impact of using differing numbers of principal components on the performance of our pose estima- tion method. We find that projecting descriptors onto a small subset of principal components not only serves to reduce both computation time, and memory usage per reference object, but also have a beneficial effect on performance. For 10. 20 and 30 view regimes, we see 8%, 2% and 3% improve- ments respectively when reducing from 384D to 32D descriptors. An intuitive explanation for the power of retaining just the top principal components from multi-view features is that doing dis- cards directions in feature space which are not variant under viewpoint. For instance, a ViT feature for a patch containing a wing-mirror of red motorbike might capture the semantic nature of this part which is important spatially for predicting pose - but might also capture the redness: the latter invariant over the views, so would not feature in the first principal components. This would be an example of an instance-specifie property that might encoded in the full feature dimensionality, yet would detrimental finding strong pross-instance correspondences.
4.2 Novel category discovery
Having verified that our method can produce accurate pose estimation for a range of objects using just single reference object for each category, note that in the setting above, assume that the object identities are known, That is, we never attempt to predict a bicycle's pose from teddy-
7

---

Views
# Dim
Med. Err
Acc@30
Acc@15 15"
Acc@7.5
Time (ms)
Memory (MB)
10
384
25.8
61.4
39.7
17.2
2.96
12.0
10
32
23.7
66.2
43.0
18.3
2.60
1.0
10
6
32.0
51.6
29.5
11.2
2.60
0.2
10
3
59.2
35.8
17.1
5.9
2.60
0.1
20
384
22.1
68.4
49.6
25.9
4.06
24.1
20
32
22.0
69.8
50.1
24.0
3.36
2.0
30
384
20.4
69.9
51.4
25.4
5.18
36.1
32
20.3
72.1
52.4
26.2
4.23
3.0
Table 2: Pose estimation (orientation) performance over 20 object categories, with varying descriptor dimen- sionality reduction and number of reference and target views. Dimensionality reduction by an order of magni- tude (from 384D 32D 24D) both reduces computational burden and improves results.
bear reference. Our approach for continual learning of categories (Section 3.5) should enable such identity matching (classification and novel category discovery) be fully autonomous. To assess this, simulate 1,000 continual learning episodes using 10 sequences from each of 20 CO3D categories. In each episode, 250 sequences are drawn sequentially and at random. For each sequence we either assign it to an existing erroneously believed to be novel categories are False Negatives (FN), and those assigned to a set where they are not the majority true category are False Positives (FP). Through tuning the similarity threshold, a 0% FP rate can be reached with 26% FN rate.
novel category as described in Section 3.5. Objects that are
4.3 Robotic novel object pose rearrangement
We deploy our method RealSense D435i camera wrist-mounted Panda robot. To scan objects from novel categories, sample 10 or 20 views in approximate hemisphere above the object, with the camera oriented towards the object. When an object is detected belonging to a known class (Section 3.5), the robot traverses between several random waypoints that look at the object, and we inspect pose estimates through a visualised oriented 3D bounding box. Further details are in the supplementary.
Limitations
Our work assumes that camera extrinsics are available between frames at runtime, and for best performance also assumes that a depth camera is available. While these are both readily satisfied in the motivating example of robot manipulator with a wrist-mounter camera, these conditions may not be so readily available in, for example, an AR context in which moving mobile phone is the candidate device for running pose estimation. However, the correspondence component of our method can still be used in depth-free setting (see Section 4.1). with either Perspective-n-Poin if depth is still available for the reference object, essential matrix estimation if depth is available for neither. This work assumes that the relationship between different objects within category can be captured reasonably well by 7-parameter rigid-body transform between the spatial locations of corresponding salient points on the objects. This would not hold for deformable or articulated objects, objects categories exhibiting drastic shape diversity. Relaxing this assumption would be a promising direction for future work. An extension the proposed method which models inter- instance relationships with 9-I affine transform is described in the supplementary.
Conclusion
In this work we propose an approach to object pose estimation that runs in real-time on RGB-D data, and can provide accurate pose estimates at the category level by leveraging multiple views of a single reference object from the category. We show that high dimensional features from a pre- trained ViT can become high quality and efficient descriptors through dimensionality reduction. We demonstrate that our method can function of category representations for pose estimation, with scalable approach to classification of novel objects. Our method runs at over 15 Hz on camera wrist-mounted on robot arm.
fully autonomous approach continual learning

---

Acknowledgments
The authors gratefully acknowledge the use of the University of Oxford Advanced Research Com- puting (ARC) facility http://dx doi. doi.org/11 10t.052212eo0. 22558. We thank Jack Collins, Oiwi Parker Jones, and Sagar Vaze for looking at early drafts.
References
[1] K. Wada, S. James, and A. J. Davison. ReorientBot: Learning Object Reorientation for Specific-Posed Placement. In ICRA, 2022.
[2] E. Sucar, K. Wada, and A. Davison. NodeSLAM: Neural Object Descriptors for Multi-View Shape Reconstruction. Proceedings 2020 International Conference on 3D Vision, 3DV 2020, pages 949-958, 2020.
[3] W.-C. Ma, J. Yang, S. Wang, R. Urtasun, and A. Torralba. Virtual Correspondence: Humans Cue for Extreme-Vicc Geometry. In CVPR, 2022.
[4] C. Wang, D. Xu, Zhu, R. Martin-Martin, C. Lu, L. Fei-Fei, and S. Savarese. DenseFusion:
6DD object pose estimation by iterative dense fusion. In CVPR, 2019.
[5] Y. He, W. Sun, Huang, J. Liu, Fan, and Sun. PVN3D: A deep point-wise 3D keypoints voting network for 6DoF pose estimation. In CVPR, 2020.
[6] H. Wang, S. Sridhar, J. Huang, J. Valentin, S. Song, and L.. Guibas. Normalized Object Co- ordinate Space for Category- 6D Object Pose and Size Estimation. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2019-June, 2019.
[7] C. Wang, R. MartinnMartin, D. Xu, J. Lv, C. Lu, L. Fei-Fei, S. Savarese, and Zhu. 6-PACK: Category-level 6D Pose Tracker with Anchor-Basse Keypoints. ICRA, 2020.
[8] Y. Lin, J. Tremblay, S. Tyree, P. A. Vela, and S. Birchfield. Single-stage Keypoint-based Category-level Object Pose Estimation from an RGB Image. 2021. URL http://arxiv. org/abs/2109.com 06161.
[9] R. Florence, L.. Manuelli, and R. Tedrake. Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation. In Conference on Robotic Learning, 2018.
[10] L. Manuelli, W. Gao, P. Florence, and R. Tedrake. kPAM: KeyPoint Affordances for Category- Level Robotic Manipulation. 2019.
[11] A. Simeonov, Y. Du, A. Tagliasacchi, B. Tenenbaum, A. Rodriguez, P. Agrawal, and V. Sitz- mann. Neural Descriptor Fields: 3E(3)-Equivarian Object Representations for Manipulation. 2021.
[12] A. Ahmadyan, L. Zhang, A. Ablavatski, J. Wei, and M. Grundmann. Objectron: A Large Scale Dataset of Object-Centric Videos in theWild with Pose Annotations. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE Computer Society, 2021.
[13] H. Yang, J. Shi, and L. Carlone. Teaser: Fast and certifiable point cloud registration. IEEE Transactions on Robotics, 37, 2021.
[14] B. Wen and K. Bekris. BundleTrack: 6D Pose Tracking for Novel Objects without Instance or Category-Level 3D Models. In IEEE International Conference Intelligent Robots and Systems, 2021.
[15] I. Shugurov, F. Li, B. Busam, and S. Ilc.. OSOP: A Multi-Stage One Shot Object Pose Estima- tion Framework. arXiv preprint, 2022.

---

[16] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging Properties Self-Supervised Vision Transformers. In ICCV, apr 2021.
[17] W. Goodwin, S. Vaze, Havoutis, and Posner. Zero-Shot Category-Level Object Pose Estimation. arXiv preprint, 2022.
[18] X. Chen, Z. Dong, J. Song, A. Geiger, and O. Hilliges. Category Level Object Pose Estimation via Neural Seailysii-b--Syythheee
In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), volume 12371 LNCS. Springer Science and Business Media Deutschland GmbH, 2020.
[19] K. Chen and Q. Dou. SGPA: Atructure-Guided Prior Adaptation for Category-Level 6D Object Pose Estimation. In /CCV, 2021.
[20] A. Wang, A. Kortylewski, and A. Yuille. NeMo: Neural Mesh Models of Contrastive Features for Robust 3D Pose Estimation. In ICLR, 2021.
[21] M. Tian, M. H. Ang, and G. H. Lee. Shape Prior Deformation for Categorical 6D Object Pose and Size Estimation. In ECCV, 2020.
[22] D. Chen, J. Li, Wang, and Xu. Learning canonical shape space for category-level 6D object pose and size estimation. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE Computer Society, 2020.
[23] J. Shi, H. Yang, and L. Carlone. Optimal Pose and Shape Estimation for Category-level 3D Object Perception. In Robotics: Science and Systems XVII, 2021.
[24] Y. Xiao, X. Qiu, P. A. Langlois, M. Aubry, and Marlet. Pose from Shape: Deep pose estimation for arbitrary 3D objects. In 30th British Machine Vision Conference 2019, BMVC 2019, 2019.
[25] C. Sahin and T. K. Kim. Category-level 6D object pose recovery in depth images. Lecture Notes in Computer Science (including subseries Lecture Notes Artificial Intelligence and Lecture Notes in Bioinformatics). 2019.
[26] A. Grabner, P. M. Roth, and V. Lepetit. 3D Pose Estimation and 3D Model Retrieval for Objects in the Wild. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2018
[27] S. Gupta, P. Arbeláez, R. Girshick, and Malik. Inferring 3D Object Pose in RGB-D Images. 2015.
[28] Y. Xiao, Y. Du, and R. Marlet. PoseContrast: Class-Agnostic Object Viewpoint Estimation in the Wild with Pose-Aware Contrastive Learning. In 2021 International Conference on 3D Vision (3DV). IEEE, 2021.
[29] W. Chen, X. Jia, H. J. Chang, J. Duan, L. Shen, and A. Leonardis. FS-Net: Fast Shape-based Network for Category- Level 6D Object Pose Estimation with Decoupled Rotation Mechanism. In Proceedings of the IEFE Computer Society Conference on Computer Vision and Pattern Recognition, 2021.
[30] J. Sun, Z. Wang, S. Zhang, X. He, H. Zhao, G. Zhang, and X. Zhou. OnePose: One-Shot Object Pose Estimation without CAD Models. arXiv preprint, 2022.
[31] S. Lu, R. Wang, Y. Miao, C. Mitash, and K. Bekris. Online Object Model Reconstruction and Reuse for Lifelong Improvement of Robot Manipulation. ICRA, 2022.
[32] Y. He, Y. Wang. H. Fan, J. Sun, and Q. Chen. FS6D: Few-Shot 6D Pose Estimation of Novel Objects. In CVPR, 2022.
10

---

[33] K. Aberman, J. Liao, M. Shi, D. Lischinski, B. Chen, and D. Cohen-Or. Neural best-buddies: Sparse cross-domain correspondence. ACM Transactions on Graphies, 2018.
[34] S. Umeyama. Least-Squares Estimation of Transformation Parameters Between Two Point Patterns. IEEE Transactions Pattern Analysis and Machine Intelligence, 1991.
[35] Y. Xiang, Schmidt, V. Narayanan, and D. Fox. PoseCNN: Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes. In Robotics: Science and Systems XIV, 2018.
[36] Reizenstein, R. Shapovalov, P. Henzler, L. Sbordone, Labatut, and D. Novotny. Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction. In ICCV, 2021.
[37] J. L. Schonberger and J.-M. Frahm. StructurefrromMMotie Revisited. In 2016 IEEE Confer- ence on Computer Vision and Puttern Recognition (CVPR), 2016.


Reviews:
[{"review_id": "siFAzLWJtMa", "reviewer": "Reviewer_kBkt", "summary": "This work tackles the task of 6D pose estimation for novel objects given prior observations of a single object of the same category (where multiple RGB-D images from known relative poses are available for both, the template and the query object, and the goal is to find a rotation and translation to align the template and the query object).\n\nThe proposed approach is to: a) extract spatial features from all images using a pre-trained ViT (with dimensionality reduction using PCA) , b) find a rotation that aligns the template and query object using a set of (query image, template image) pairs that match well and are consistent with other pairs, and c) estimates a 7D alignment (R, t, scale) using patchwise matches between the selected pairs and leveraging RANSAC.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper tackles a relevant problem and considers a natural setup for it. Given multiple views of a query object and a template object, computing alignment between the two is a great formulation for the task of one-shot pose estimation..\n\n- The experiments are reported on the Co3D dataset which represents a challenging setup, and shows that the proposed method does indeed generalize across objects.\n\n- A key implementation detail that this work highlights is the benefit of feature dimensionality reduction. This is well-ablated and leads to consistent improvements.\n\nWeaknesses:\n\n- One concern is that the paper relies rather heavily on the framework presented in [17], and the primary contribution here is to extend that framework to leverage multiple query views (in a slightly straightforward manner). While this obviously leads to significant empirical gains (as using multiple views should!), this does limit the technical contribution. Moreover, this should also necessitate an alternate baseline where one simply uses the approach from [17] to get multiple estimates and chooses the ‘mode’.\n\n- While the work does use multiple query views, the precise methodology feels somewhat constrained. In particular, the only patch ‘correspondences’ that are allowed (and ultimately drive the optimization) are between batches at the same grid locations across images. This is suboptimal for two reasons as: a) it implicitly assumes that images from very similar viewpoints are available for matching and may not be a valid assumption, and b) cannot leverage observations of matching content from dissimilar viewpoints (the handle of a cup might match between a left-view and a right-view, and should drive the pose estimation).\n\n- On a note slightly related to the above, I am curious why a more ‘traditional’ SfM like approach is not considered (as the method, or atleast as a baseline). In particular, given depth images, the patches can be considered as 3D points with associated (deep) features. One could then construct matches between the 3D points observed for the source and target objects, and using RANSAC, optimize the 7D pose to best align them. I would be interested to hear why this would not suffice (particularly as it is the ‘typical’ approach that drives large-scale structure from motion — and in this case the depth + known extrinsics within each object would only simplify the task further)?\n\n- (comparatively minor concern) The continual learning aspect seems a bit weak and is more of a post-hoc application rather than something integral to the method. In particular:\na) the notion of a ‘category’ does not really keep evolving: if one sees multiple chairs, the category is still represented by a single template (or perhaps a collection of independent template — the text is unclear on this). In any case, multiple examples of a category do not lead to an evolving notion of the category. \nb) The determination of a ‘new’ vs seen category is simply done via average cosine similarity of the [CLS] token of the ViT model and does not have much in common with the approach for 6D pose alignment.\nConsidering both these, the continual learning aspect of the work, while tackling a good goal, seems a bit adhoc and technically disconnected.  \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, I am slightly torn about this work. On the one hand, it considers a relevant setup and shows promising results. However, this is currently outweighed by the concerns (similarity to [17], and possibilities of a simpler SfM-like solution).\n\nUpdate:\nThanks for the response, and in particular for addressing my misunderstanding regarding the matching, which also addressed the concerns regarding a more 'regular' SfM-like method. I am happy to update the rating towards an accept and would recommend acceptance as a poster.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "- Please see the Weakness section. In particular, I would be interested in the author response to why directly reasoning in 3D space and simply searching for (robustly via RANSAC) a 7D pose that maximizes alignment between matching 3D points is not a good solution.\n- As a minor concern, the only evaluation in the paper focuses on rotation estimation, and it would be good to see some results on the accuracy of the full pose estimated."}}, {"review_id": "lkDllc7UBN3", "reviewer": "Reviewer_kWPG", "summary": "This paper presents an approach to object pose estimation at the category level by leveraging multiple views of a single reference object from the category. The approach combines advantages of different paradigms : deep models and feature matching.  High dimensional features from a pre-trained ViT is used for efficient descriptors through dimensionality reduction. These set of image descriptors from multiple views of an object are used to find correspondences between the target and the reference object of the same category. An average pose estimate is obtained by filtering best reference-target view pairs. Finally the filtered subset are aligned using a robust least square estimation to give a 6D object pose.\nThe paper builds upon [17] by using multiple reference object views which leads to considerable improvement in performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe paper is well written and supported with relevant media content.\nThe technical contribution is incremental yet considerable.\nThe proposed approach results in better performance in terms of speed and accuracy than existing category level technique.\n\nWeakness:\nAlthough the performnace improvement in Table 1 has promising results, the extensive evaluations with other techniques would be desirable.\nFor e.g., [14], You Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstration- RSS2022\nAlso, in line 38, I do not think its well argued.  Do you mean method in [14] is designed for single object instances? Please check", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I believe the approach is interesting and results in improved performance than recent approach.\nHowever, I do want to see more comparative evaluations to be convinced.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Comparative evaluations mentioned earlier.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "Bt68Ih_Upg", "reviewer": "Reviewer_DGYB", "summary": "\nThis paper proposes an approach for one-shot category-level pose estimation. Given M reference views of an object belonging to a specific category (which defines a category-level pose) and N views of a novel instance of that category the task is to define the category-level pose estimate. The authors propose to use a pre-trained DINO ViT to produce 2D-2D semantic correspondences between the reference and target images and then solve for the relative pose by (1)  lifting the 2D correspondences to 3D using provided depth images and (2) aligning the correspondence sets using RANSAC.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is an extension of prior work [17] (note that this work is only on arxiv and I was unable to find any conference/journal publication of [17]). In particular the main innovation of this work seems to be in using multiple views $$I_{R_{1:M}}$$ of the reference object instead of just one view as in [17]. \n\n\n**Strengths**\n\n\nThe main strength of the paper is that it tackles an interesting and challenging problem of one-shot category-level pose estimation. They demonstrate that using multiple reference views leads to significantly improved performance over prior work [17].\n\n\n**Weaknesses**\n\n\n\n* The main experiments (Table 1, Table 2) assume 10+ views of the target object at inference time. This is typically not a realistic assumption in most robotics settings and thus requires more justification if this is to be the main test condition of the proposed approach. How would this affect the method in terms of the consensus and retrieval steps?\n* The robot experiments are quite weak in the current paper. More qualitative examples of the pose estimates would be helpful. From the video it doesn’t seem that the pose estimates would be accurate enough to actually enable any meaningful robot manipulation tasks (e.g. category-level tasks such as those performed in [10, 11], category-level grasping, category-level pick and place, etc.)\n\n* The authors should include at least some comparison the performance of this method vs. what can be achieved with a supervised category-level pose estimator such as [6, 8]. This would allow the reader to understand the tradeoffs between using a supervised approach (that has more requirements/restrictions) vs. a more general approach such as proposed one. A comparison on a standard category-level pose estimation benchmark such as the NOCS dataset from [6] would make sense.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall this paper tackles the interesting and challenging problem of category-level pose estimation without category-level training, only relying on multiple reference views of an example from the category. The current approach leverages a pre-trained DINO ViT model to extract correspondences which are then used in a fairly standard RANSAC + Umeyama algorithm for producing pose estimates. The current paper lacks comparisons to some supervised category-level pose estimation baselines (such as [6,8]). Additionally it is not at all clear that these pose estimates are sufficiently accurate to actually be used to perform category-level tasks. The method also has several limitations. In particular it only considers a 7D pose transform (rigid + scale) for characterizing the category, which may fail to capture the intra-category shape variation in certain categories (e.g. comparing a sneaker to a tall boot).", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "List issues to be addressed during the period of author response and revision. The authors have the opportunity to submit a revised manuscript during the period. (max 5000 characters)\n\n* Can the authors expand on how category-level pose is actually defined in the extension of the CO3D dataset. It would be useful to have this in the appendix at least so the reader doesn’t need to refer to [17]. \n* The authors should include metrics on the translation errors of the pose estimates after orientation. Translation is just as important as orientation in terms of usefulness in a downstream robotic application.\n* How would the method perform in a more cluttered setting that is typical of robotics scenarios? Would be helpful to have some quantitative and/or qualitative results here.\n* Can you expand on why you get such a speedup compared to [17], the underlying algorithms seem to be the same with exception of the current work using multiple views.\n* The authors should discuss how the method would handle more cluttered scenes (as is typically encountered in robotics scenarios). Would any significant changes be needed to the approach."}}, {"review_id": "3Upnzqh6VK4", "reviewer": "Reviewer_5Bj6", "summary": "The paper presents a method for category-level object pose estimation based on RGB-D input. The approach does not require pose-labeled data, 3-D models, or category-specific networks, but only relies on few views of a target object and of a reference one of the same category. Similarly to [17], the paper shows that features extracted from a pre-trained ViT can be used as descriptors to find category-level correspondences for pose estimation. Compared to [17], the paper proposes to use multiple views from the reference object and shows that this has a significant impact on the estimated poses. Further, the method is applied in a continual learning setting for novel category discovery. The evaluations show that proposed approach significantly outperforms [17] and the baseline of [13], which however uses purely geometric correspondences. The method is demonstrated in a robotic setup to perform real-time pose estimation of a novel object based on a reference object from the same category.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The motivations and the research gaps addressed by the proposed paper are stated clearly. A detailed review of the most relevant related work is provided, with a sensible and targeted classification of the previous approaches.\n\nThe paper is generally very well written, with virtually no typos or notation inconsistencies. With very limited exceptions (mentioned in \"Issues\"), the language is clear and the explanations are thorough, making the paper easy to read.\n\nThe quality of the presentation is very high. Despite minor possible improvements, I find the visualizations professional and very informative, particularly Figure 1, which provides an effective overview of the method, and Figure 2, which illustrates clearly the process of finding correspondences and highlights interesting, not obvious points, such as the correlation between view-to-view similarities and true geodesic errors. The attention to the details is also evident in the video (e.g., the visualization of the actual reference-target views in the correspondence matrix, or the method-overview figure edited to visualize the moving robot arm).\n\nA number of minor technical details, particularly in the choice of the hyperparameters, could be provided for completeness. Furthermore, for multiple explanations and additional results, though all supporting non-major points, the reader is referred to the supplementary material, but the latter does not include the aforementioned additions.\n\nThe main point that could be seen as a weakness is that the viewpoint estimation process is heavily based on the previous work of [17] (as repeatedly acknowledged), and that particularly one of the main differences, i.e, the use of multiple reference views as opposed of a single reference view (as in [17]) could render the setup more complicated. However, as shown in the experiments, this choice results in a large improvement of the pose estimation accuracy. Furthermore:\n\n- A number of other relevant insights (e.g., the impact of feature dimensionality) or modifications (consensus filtering vs. RANSAC, cosine similarity vs L2 distance) are presented\n- The method runs in real time and was demonstrated on a real robotic system\n- The whole application in a continual category learning scenario for new category discovery is novel and of high relevance\n- Several interesting ideas are introduced or investigated, for instance the use of the [CLS] token for category comparison, the evaluation of feature similarity in 3-D rather than in pixel space, or the use of the attention maps to produce a segmentation mask", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper presents a number of novel insights on the use of vision transformers for one-shot category level object pose estimation and novel category discovery, which could significantly question the need for training object- or category-specific feature networks. The approach runs in real time and is demonstrated on a robotic platform. The explored setup, i.e., category-level, without the assumption of CAD models, instance-level networks, or pose-labeled data, and only relying on a small number of reference and target views, has a high potential of concrete use in robotics. \n\nThe experiments are solid and the presentation is professional and effective for illustrating the method and the results.\n\nI believe this paper could be a strong contribution to this Conference, after addressing a few minor points.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "### Technical details\n- L144: In what sense \"normalised\"?\n- Sec. 3: What values were used for $M$ and $N$? Was an ablation study performed to select them?\n- Fig. 2: \"Three consensus filters\"/\"iterations of filtering\": Where does the \"three\" come from? Does this value always guarantee that the criteria mentioned in the text (L197) are met?\n- Fig. 2: \"good\" -> \"closest\", as in the subfigure title of (b)? That is, what does \"good\" mean here?\n- Sec. 3.4: What value of $K$ was used? What impact does this have on the resulting pose estimate?\n- Sec. 3.4: The correspondence indices $u_k, v_k$ come from a $28\\times 28$ feature map, whereas the RGB-D image has larger resolution. How do you select the depth of the 3-D point associated to $u_k, v_k$, given the difference in resolution? Have you investigated matching the RGB-D resolution, e.g., using interpolation? Intuitively, this could produce more accurate estimates of the 3-D point positions.\n- L193-197: I find this explanation a bit articulated and hard to parse. If I interpret it correctly, for each pair $(i, j)\\in \\mathcal{Sel}\\subset M\\times N$ , where $\\mathcal{Sel}$ are the retained pairs (Line 192), a mean pose $\\mathbf{T}\\overline{ij}$ is computed from all the other pairs $\\{(m, n)\\in \\mathcal{Sel}\\ \\backslash(i, j)\\}$, and then the \"views\" that are removed at each iteration are the _pairs_ $(i^\\prime, j^\\prime) := \\arg\\max_{(m, n)\\in\\mathcal{Sel}} \\textrm{dist}(\\mathbf{T}{mn}, \\mathbf{T}\\overline{mn})$, where $\\textrm{dist}$ is the pose distance and $\\mathbf{T}{mn}$ is the relative pose obtained from pair $(m, n)$. I find confusing the concurrent use of \"view\" and \"view pair\", and also the computation of the \"mean\" could be rephrased to be more clear (e.g., is the mean pair-specific, as I interpreted above?)\n- L222: The idea of using [CLS] tokens for category comparison is interesting. Were other options considered, or any ablations performed?\n- L297: It should be explained in the text what exactly is meant by \"$15^\\circ$\". That is, how can a single angle parametrize a distance between poses\n- L333 (Supp.): The concept of \"(ViT) saliency\" should be introduced\n- From the real-world experiments shown in the video it seems that pose estimate is not very stable across frames. Did you evaluate quantitatively the extent of the pose robustness and its impact on manipulation (e.g., would inconsistencies lead to failures in grasping the object)?\n\n### Missing details that should be in the Supplementary\n- L226: Threshold on $\\theta$ \n- L242-243: How was the threshold for TEASER++ chosen? This intuitively might have an impact on the results\n- L252-253: \"effect of reducing viewpoint overlap between the image sets of reference and target objects\"\n- L279-280: \"a subsequent K-means cluster refinement approach\" (no details on the category discovery seem to be provided in the Supp.)\n\n### Suggestions/general questions\n- What are the most important factors that produce the speed-up w.r.t. to [17] mentioned, e.g., on L75? Intuitively, by having to compute correspondences w.r.t. multiple instead of a single reference view, one would think that your method should if anything be slower than [17]\n- The resizing results in the object being \"stretched\" in some reference views. Did you evaluate if this has an effect? Did you consider using, e.g., padding, to keep the object aspect ratio fixed, and see if this has an influence on the results?\n\n### Typos, language and bibliography\n- Figure 1, caption: Extra space before \"10 s\"\n- L41-43: Repetition of \"leverage\"\n- L99: Extra space before \"15 object\"\n- L122: $\\mathbb{T}_{\\mathcal{R}_{1:N}}$ -> $\\mathbb{T}_{\\mathcal{T}_{1:N}}$\n- L157: Dashes should be em dashes\n- L179 and 239: \"Certain\" -> \"A selection of the\"?\n- L180: \"show\" -> \"shown\"\n- L188: $d$ is previously (Sec. 3.1) and later (Sec. 3.5) a function of feature maps $\\Phi$, but here is defined as a function of descriptor indices\n- Eq. 1: $\\mathbf{u}_k$ and $\\mathbf{v}_k$ are here written in bold letters, whereas the correspondences are indicated in the previous lines in the text as $u_k$ as $v_k$. There is a missing step to describe how to obtain $\\mathbf{u}_k, \\mathbf{v}_k$ from $u_k, v_k$\n- L224-225 and Eq. 2: The meaning of the symbols $F$, $G$, and $x$ should be explicitly introduced in the text\n- Eq. 2: For consistency with the previous notation (see Sec. 3.2), $\\phi$ should be replaced with $\\Phi$ or otherwise explained in the text\n- Eq. 2: The brackets around the summation could be removed. Otherwise, I suggest adapting them so as to better fit their content (\"\\left(\" and \"\\right)\")\n- L257-258: \"not only serves [...], but also have\" -> \"not only serves [...], but also has\" or \"serves not only [...], but also _to_ have\"\n- L258: Unnecessary comma after \"time\"\n- L259: \"30 view\" -> \"30-view\"\n- L301: \"work;\" -> \"work.\"\n- L334: References [10], [27] and [39] are lacking their venue\n- L339: \"theWild\" -> \"the Wild\"\n- L341 (Supplementary): Missing brackets around \"Section 4.3\""}}]

Author rebuttal:

Decision: Accept (poster)