SA6D: Self-Adaptive Few-Shot 6D Pose Estimator for Novel and Occluded Objects
Authors: Ning Gao
Vien Anh Ngo
Hanna Ziesche
Gerhard Neumann
Venue: CoRL 2023 Poster (2023)

Abstract: To enable meaningful robotic manipulation of objects in the real-world, 6D pose estimation is one of the critical aspects. Most existing approaches have difficulties to extend predictions to scenarios where novel object instances are continuously introduced, especially with heavy occlusions. In this work, we propose a few-shot pose estimation (FSPE) approach called SA6D, which uses a self-adaptive segmentation module to identify the novel target object and construct a point cloud model of the target object using only a small number of cluttered reference images. Unlike existing methods, SA6D does not require object-centric reference images or any additional object information, making it a more generalizable and scalable solution across categories. We evaluate SA6D on real-world tabletop object datasets and demonstrate that SA6D outperforms existing FSPE methods, particularly in cluttered scenes with occlusions, while requiring fewer reference images.

SA6D: Self-Adaptive Few-Shot 6D Pose Estimator for Novel and Occluded Objects
Ning Gao Ngo Anh Vien' Hanna Ziesche' Gerhard Neumann
Bosch Center for Artificial Intelligence 2 AAutonomous Learning Robots, KIT Ening.gao, anhvien. ngo, hanna aeschheedde. wosch.cchhoooo..........cccc.cc. com gerhard. nemmann .ataannnkkiiei..
Abstract: 6D pose estimation is one of the critical aspects in robotic manip- ulation. Most existing approaches have difficulties in extending predictions scenarios where novel object instances are introduced, especially with heavy oc- clusions. In this work, we propose a few-shot pose estimation (FSPE) approach called SA6D on novel objects, which uses self-adaptive segmentation module identify the novel target object and construct a point cloud model of the target object using only small number of cluttered reference images. Unlike existing methods, SA6D does not require object-centric reference images any additional object information, making it more generalizable and scalable solution across categories. We evaluate SA6D on real-world tabletop object datasets and demon- strate that SA6D outperforms existing FSPE methods, particularly in cluttered scenes with occlusions, while requiring fewer reference images.
Keywords: Contrastive Learning, 6D Pose Estimation, Self-Adaptation
Introduction
Accurately estimating the 6D pose of novel objects is critical for robotic grasping, especially for the tabletop setup. Prior work has investigated instance-level 6D pose estimation [1, 2, 3, 4], where the objects are predefined. Although achieving satisfying performance, these methods are prone to overfit specific objects and suffer from poor generalization. Recently, several approaches [5, 6, 7, 8, 9. 10, 11, 12] have been proposed for category-leve 6D pose estimation instead of specific objects. However, conditioning on specific object categories limits the generalization to objects from novel categories with strong object variations. Meanwhile, some approaches [13, 14, 15, 16, 17, 18] investigate generalizable 6D pose estimation as a few-shot learning problem, i.e., predicting the 6D pose of novel and category-agnostic objects given few labeled reference images with the known pose of the novel object to define the object canonical coordinates. Although achieving promising results, these methods far only work well non-occluded and object-centric images, i.e., without the interference of other objects. This limits the generalization real-world scenarios with multiple objects in cluttered and occluded scenes. Furthermore, additional object information is required such as object diameter [14], mesh model [18, 13]. object 2D bounding box [17] or ground-truth mask [16, 19], which is not always available for novel object categories. Our method aims to enable a fully generalizable few-shot 6D object pose estimation (FSPE) model.
In summary, identify three primary challenges that are not adequately addressed by the current state-of- afetthe-art methods [14, 16, 17, 18]: i) The category-agnostic 6D pose estimation in cluttered scenes with heavy occlusions is performing poorly. ii) The object-centric reference images from cluttered scenes are cropped by ground-truth segmentation or bounding box of the target object, which limits the generalization real-world scenarios. iii) The requirement of extensive reference images covering all different view-points is not practical.
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

Online Adaptation Nodula
Object Modal
Region Proposal
Refirement
[R. Tjink
Prediction
Reference Images
Test Image
Segmentation
Local Reconstraction
We present generalizable and category-agnostic few-shot 6D object pose estimator using small number of posed RGB-D images reference. Compared to existing methods, our approach provides robust and accurate predictions on novel objects against occlusions without requiring re- training any object information.
Figure 1:
To address the aforementioned challenges, propose robust self-adaptive 6D pose estimation ap- proach called SA6D. As shown in Fig. 1, SA6D uses RGB-D images input since depth images are normally easy to obtain along with RGB images in robotic setup, and ii) depth images reveal additional geometric features and can improve the robustness of prediction against occlusion. SA6D employs an online self-adaptative segmentation module to contrastively learn a distinguishable rep- resentation of the novel target object from the reference images of cluttered scenes. Meanwhile, canonical point cloud model of the object constructed from the depth images. After the online adaptation, the segmentation module is capable segment the target object from new images and construct the local point cloud from depth. Incorporating geometric features from the extracted point cloud, region proposal module crops the test image by localizing the target object. With the cropped test image and the reference images, we employ Gen6D [14] to first predict an initial pose using visual input, followed by a refinement module using the induced geometric features.
Our work focuses on the scenario with tabletop objects used for robotic manipulation. Our primary contributions are summarized follows:
SA6D is fully generalizable mation such as ground-truth segmentation, mesh model, object-centric image. Instead, only limited number of RGB-D reference images with the ground-truth 6D pose of the predicted object are needed.
datasets without requiring any object category infor-
A self-adan aelf-adaptive segmentation module is proposed to learn distinguishable representation of novel objects during inference.
SA6D significantly outperforms current sate-of-the-ar methods against occlusion in real- world scenarios while trained entirely on synthetic data.
Related Work
Letegory-Level 6D Object Pose Estimation. Methods for generalizable 6D object pose estima- tion can be divided into category-specific and category-agnoste models. For the category-specific estimation, Wang et al. [5] first propose canonical representation for all possible object instances within category using Normalized Object Coordinate Space (NOCS). However, inferring the object pose by predicting only the NOCS representation is non-trivial given large object variations [21]. To tackle this problem, Tian et al. [22] account for intra-categorical shape variations by explicitly modelling the deformation from shape prior to the object model, while CASS [6] generates 3D point clouds in the canonical space using a variational autoencoder (VAE) [23]. FS-Net [8] proposes a shape-based model using 3D graph convolutions and decoupled rotation mechanism further reduce the feature sensitivity the color variations. Wang al. [7] predict the relative 6D pose between two consecutive images using category- keypoint matching model. Chen al. [9] employ VAE-based generator to learn categorical prior and update the prior with online rendering

---

Online Self-Adaptation Module
Region Proposal Module
Refinement Module
[R, Tlref
Tref
Segmentor
Gen6D
Local Object Reconstruction
[R. Tjinit
Object Model
Segmentor
4r
Global Reglstration
Reference Images
Referance RDI
Test
Target
Local Reconstruction
Image
Segmentation
Proposed Region
[R. Tifinal
Figure 2: Overview. SA6D includes three modules: i) The online self-adaptation module discovers and segments the target object (milk cow) from a cluttered scene giving few posed RGB-D images as reference. Subsequently, the canonical object point cloud model from the reference images and local model from the test image are constructed based on the segments. ii) The region proposal module outputs robust region of interest (ROI) of the target object against occlusion by incorpo- rating visual and geometric features. coarse 6D pose is then estimated by comparing the cropped test and reference images using Gen6D [14] and iii) further fine-tuned using ICP [20].
W.r.t. the test image. Recently, al. [11] facilitate the generalization by collecting a large-scale dataset with object-centric RGB-D videos called Wild6D, Based on Wild6D, Zhang et al. [12] pro- pose to learn the dense 2D-3D correspondences between the 2D image pixels and the categorical shape prior while the final 6D pose is computed by the least-square- fitting algorithm [24]. Similar to our work, UDA-COPE [25] employs self-supervised training while TTA-COPE [26] addresses the source-to-target domain gap using test time adaptation. Nevertheless, these methods require a man- ually defined categorical prior for training and therefore are limited to generalize across categories. In contrast, our method learns 6D pose estimation
category-agnoster manner.
Category-Agnosse 6D Object Pose Estimation. -ggosocy-agmostic pose estimation be formu- lated as a few-shot learning problem [15]: During inference, the model can generalize and predict the pose of novel objects given few images with known poses as reference. LatentFusion [16] and iNeRF [19] employ the neural rendering technique [27] to refine the predicted pose based on a latent representation obtained from the reference images while a segmentation of the object is required as input. FS6D [17] extracts features from both the reference images and test images fol- lowed by a prototype matching algorithm obtain the point-wise correspondences. OnePose [28] and OnePose++ [29] build object model from single RGB video and employ feature mapping between the test image and the object model, which not end-to-end and deviate from the few-shot domain. Furthermore, all the aforementioned methods require object-centric images for either refer- ence or test images. In contrast, Gen6D [14] is applicable for cluttered scenes where both reference and test images contain multiple objects, although it struggles with occlusion. Our work is inspired by Gen6D and exploits the geometric information of the target object enable robust prediction against occlusion.
Unseen Object Segmentation. Recently, several approaches have been proposed to close the gap between learning unseen object segmentation from synthetic datasets and real world datasets [30, 32, 33]. Xie et al. [34] propose to learn a two-stage segmentation model by separately leveraging RGB and depth information
hierarchical way, where the model is fully trained on the synthetic data. UCN [35] proposes to learn from RGB and depth images jointly and generate pixel-wise feature embeddings. To enable a generalizable 6D pose estimation in cluttered scenes, we design self-adaptiv module to generate turget-object-orieneed segmentation model using UCN as a base segmentor.

---

Transiation Tref
6D Pose [R, Tlref
Segmentor
Negative Segments
Positive Segments
s*
Copy
Object Model Iters
Segmentor
Adaptive
Dense Feature
Dense Feature
Contrastive
Reference
Images
Object-Level Representations
Model Updating
Averaging Dense Features by the Segmentation from
Target Object Representation +*
Segmentor
Cosine similarity
Adaptive Segmentor
Test Image
4
Representation
Sogmentation
Reconstruction
Figure 3: Online self-adaptation module. A pretrained segmentor yp is first applied on reference images to predict segmentations. Meanwhile, adaptive segmentor fo* is initialized from y. With the ground-trute -rutt translation of the target object in the reference images Teos the object center can be reprojected to the image. For each reference image, segment chosen it includes the reprojected object center while the remaining segments are considered as negative samples. Subsequently, object-level representation of each segment is computed by averaging the pixel-wise dense features from yo*. contrastive loss then applied over the positive and negative object representations and updates fou iteratively. After adaptation, fo* generates the target object representation r* by averaging over all positive representations from reference images. Given a test image, we obtain the representation of each candidate segment in the same way and compute the cosine similarity between each candidate and r*, where the most similar candidate is chosen as the segment of the target object. Meanwhile, the canonical and local object models are computed based on the segments and depth images.
positive sample
Method
SA6D comprised of three parts, i.e., an online self-adaptation module (OSM) for target object segmentation from cluttered scenes, region proposal module (RPM) infer the region of interest (ROI) for the target object against occlusion, and refinement module (RFM) refine the predicted 6D) pose of the target object using both visual and the inferred geometric features. The proposed pipeline is shown in Fig. 2.
3.1 Online Self-Adaptation Module
To alleviate the dependence on prior object information and object-centric reference images, and improve the prediction against occlusions, it is essential to build a model which can discover the objects from the cluttered scene and identify the occluded target object from other objects. To achieve this, we design adlf-adaptive segmentation module which is updated only during inference in self-supervised manner given posed reference images, where retraining needed.
In particular, we employ the segmentation model from Xiang et al. [35] as our base segmentor sp which segments all instances of each image by clustering the pixel- wise features using the mean-shifi algorithm [36]. Examples of predicted segmentation are shown in Fig. Given the ground-trui translation TV E RB3 of the target object in the i-th reference image, we can reproject the object center on the image plane and select the segment, which includes the reprojected object center, as positive target segment sf" while the remaining segments considered negative segments S:N { ...., x K denotes the number of negative segments in each reference image. Given

---

S
AAOOO
CURAMOO
1
LM Duck
LM Benchuise
LMO Eighbx
LMO Driller
HB
rabbit
FeWSOL
Figure 4: Qualitative results. The green bounding box denotes the ground-truth pose and blue denotes the prediction. In SA6D, blue denotes prediction before refinement while red is the final prediction.
M reference images, obtain positive set of target object segments S {&f ... h and negative set of segments S =UM S.N.
The adaptive segmentor sp" is initialized by copying the parameters yp and used generate dis- tinguishable representations between the target object and the remaining objects, not for generating the segmentation. The positive and negative object-level representations, Rº and R& are computed by averaging the pixel-wise dense features of while grouping by each segment from SP and S.N Based on the positive and negative representation sets R. and R. yp is updated iteratively using contrastive loss [37]. Specifically, for each positive pair rf r E RP, the loss is computed as
4i = llo exp(sim(rf frfff(r))//) )/T)
EvERMUGS) exp(sim(r( ,r)/T)²
where is hyper-parameter and 0.07, sim denotes the cosine similarity between rep- resentations. The loss is summed all combinations of the positive pairs from R and back- propagated through yp" After adaptation, s* generates the target object representation r* by aver- aging over all positive representations r* = mean(rf, r...." Mg Note that Rº and RN are updated along with y0 simultaneously.
Given test image, the candidate segments are obtained in the same way from &p followed by com- puting the representations from yps. By comparing the cosine similarity between the candidates and the target object representation r", the candidate with the highest similarity score is selected as the segment of the target novel object.
Object Model Reconstruction. We reconstruct the object model from the reference images by computing the partial point clouds for each positive segment and transfer them to the canonical co- ordinates given the ground-truth 6D pose [Rt Jef The combination of partial point clouds obtained from the reference images assembles coarse geometrie model of the object. For inference, obtain partial point cloud model (local reconstruction) using the predicted target segment from the test image.
3.2 Region Proposal Module
The region proposal module combines 2D image features and the geometric features from induced point cloud model. The idea is to improve the robustness against clutter and occlusion comparison Gen6D [14]. The region interest (ROI) denotes a squared area where the target object located. While Gen6D can predict the ROI of novel objects, it suffers from occlusion since the prediction depends purely the visual similarity between the reference and test images.

---

Medbed
Mikk
Ref. Non
0gghes
rock
Teachsiion
UiuMOD [19]
con
m0g
LiwMED-OCCE [60]
diack
201
14g
BorwllewndDR (41)
Baage
car
GooDD
0..30
0.45
0.25
0.25
0.02
0.12
1.36
0.30
SAID OCP odly
20
0.50
0.31
0.37
0.25
0.21
0.17
0.31
0.17
0.35
0.30
0.88
0.14
0.23
0.14
0.23
0.17
0.20
0.46
0.35
SAGD (wx RPSO
20
0.63
0.47
0.50
0.31
0.36
0.58
0.45
0.19
0.15
0.13
0.10
0.17
0.38
0.17
0.37
0.18
0.21
8.1
0.35
SA6D
20
0.73
0.73
0.55
0.50
9,47
0682
0.26
0.21
0.32
0.53
0.38
8A2
0.38
0.78
0.32
GeeD
64
0.78
0.91
0.1
0.1
0.60
0:11
0.31
0:38
016
0.78
0:11
[1 SAOD Owv RFM)
20
0.6
0.61
0:66
0.6
0.72
0.78
0.67
0.58
061
0.00
0.18
0.45
0.11
0.18
0.31
0.00
0.00
0.16
0.12
70
0.56
0.30
0.51
0.30
0.26
0.20
0.78
0110
866
0.0
0.16
0.2
0.11
0.41
0.13
0.51
0:31
Table I: Evaluation of ADD-0.1d on LineMOD, LineMOD-OCC, and HomeBrewedDB datasets
against sategory-agnoste baselines.
Method
10Ug.
5"2cm
5-5cm
10m5cm
CASS [6]
0.01
0.0
0.0
0.0
Methed
ADD-0.1d
Shape-Prior [22]
0.33
0.03
0.04
0.14
DualPoseNet
0.70
0.23
[14]
RePoNet [1]]
0.71
0.30
0.34
0.43
SAND
0,5595
SA6D
0.65
0.37
0.40
0.42
Table 2: Evaluation on FewSOL [42] dataset
Table 3:
Evaluation on Wild6D [11] dataset
over 336 objects using reference images.
against category-leve baselines.
Instead of requiring the object diameter in Gen6D, we estimate the object diameter d from the reconstructed object model. With the reconstructed object point cloud model from inference images and the partial point cloud model from the test image, we estimate an initial translation Timit using global registration, i.e., using RANSAC to first match the points between the two models followed by the fast point registration proposed by Zhou el al. [38]. With d and the estimated depth Tina- where f is the camera focal length. Meanwhile, the ROI position [u, t] is calculated by u = T f/TTi and v T f/TT As shown in Fig. 2, we can accurately predict the ROI against occlusion using the geometric information without interference from the environment. Similar to the test image, the cropped reference images are obtained given the reconstructed object model and ground-truth pose [R, Jee- Subsequently, we employ the pretrained Gen6D detector to predict an initial pose based on the visual input.
the ROI scale is computed by d * f //T
3.3 Refinement Module
Based on the reconstructed geometric features of the object, we employ Iterative Closest Point (ICP) [20] and use the output of Gen6D initial pose, which helps alleviate the problem of local optimal in ICP. In our experiments, we found this is particularly useful for rotation estimation where the global point registration struggles.
4 Experiments
We employ two baselines that are most relevant our work on category-agnostic unseen objects, namely LatentFusion (LF) [16] and Gen6D [14]. Besides the input image, LatentFusion requires ground-truth segmentation of the target object while Gen6l requires the object diameter input. In contrast, our method does not require any additional information. We also compare SA6D against category-level SOTA methods which use RGB-D as input. is good to note that SA6D is not trained on any specific category while all category- levell baselines are trained and tested on the objects within the same category.
4.1 Datasets and Metrics
Evaluation datasets. We use four datasets for evaluation against category-agnostic methods, namely LineMOD [39], LineMOD-OCC [40], HomebrewedDB [41] and FewSOL [42]. None of these datasets is used during the training phase. LineMOD (LM) includes annotations of 15 test objects in cluttered scenes without occlusion while LineMOD-OCC (LMO) and HomebrewedDB

---

data. In (HB) have heavy occlusion. FewSOL includes 336 real-world objects and 9 RGB-D images for each object from different viewpoints, where we randomly select images as references and test on the remaining image. FewSOL includes large-scale object variations but without occlusion or clut- ter. Furthermore, we compare against category-level methods on Wild6D [11] which is a RGB-D video dataset including 5 object categories.
Training datasets. The base segmentor is trained fully on the synthetic Tabletop Object Dataset (TOD) generated by Xie er al. [44] and the pretrained Gen6D model uses the rendered images from 2000 ShapeNet [45] models and 1023 Google Scanned Object by Wang et al. [46]. Note that only synthetic datasets are used for training.
Evaluation metrics We use the average distance (ADD) [39] to evaluate the object points after being transformed by the ground-truth and predicted pose. ADD-0. o.Id (ADD-0.3d) denotes the accuracy of the predictions with an average distance below 10% (30%) of the object diameter. ADD-S is used in FewSOL dataset due to the large amount of symmetric objects, where the average distance is computed based on the closest point. For comparison against category-level methods, we employ the same BOP [47] metrics as used in RePoNet [11]. n², m cm denotes the accuracy when both rotation error is less than n and translation error less than m cm.
4.2 Results and Discussion
Comparison against category-agnostic methods. As shown in Tab. 1, although baselines show promising results on LineMOD dataset, they perform poorly and cannot generalize on occluded datasets (LineMOD-OCC and HomeBrewedDB). In contrast, without requiring ground-trume -round-truth seg- mentation or object diameter, SA6D significantly increases the performance over all datasets, espe- cially under the circumstances where fewer reference images are given or the objects are occluded. Furthermore, without ground-truth segmentation, SA6D still outperforms LatentFusion occluded datasets by a large margin. Tab. shows SA6D is able to generalize large object variations while LatentFusion cannot generalize even without occlusion. We find that LatentFusion requires high- quality depth images and more reference images to reconstruct the latent representation, and works poorly on flat objects. Examples are shown in Fig. 4. Furthermore, SA6D demonstrates superior performance against Gen6l) even without using the geometric features in the refinement module (RFM). The reason is, Gen6D struggles with localizing the target object in FewSOL dataset since the evaluated objects in FewSOL dataset are close to the camera and occupy larger area than the dataset used for training, indicating a poor generalization of Gen6D on contrast, the region proposal module (RPM) used in SA6D alleviates the problem.
Ablation study on components of SA6D. To evaluate the importance of different components in SA6D, we conduct an ablation study by removing the region proposal module (wo/ RPM), the refine- ment module (wo/ RFM), and remove both by only using the global point cloud registration between the reconstructed global and local object model (ICP only). The results are shown in Tab. 1. The performance decrease of ICP only indicates that classic point cloud registration between partial and global point clouds is often stuck at suboptimal position. The performance drop on wo/ RFM demonstrates the importance of the induced geometric features and the notable performance drop on LinMOD-OCE and BreweDBB without using RPM shows its crucial role against occlusion. In Fig. 5c, we show an example of test image and visualize the pixel-wise visual similarity between reference and test images on top, where higher brightness indicates higher visual similarity. RPM capable localize the target object (cow) while Gen6D depends purely visual similarity and selects wrong object.
Accuracy vs Reference Number. We report the ADD-0.14 w.r.t. the number of reference in Fig. 5a on HomebrewedDB/cow. Increasing the number of reference images overall benefits all methods except LatentFusion, which sometimes shows degradation in performance because a heavily oc- cluded reference image can drastically alter the implicit representation in the latent space due the employed online rendering. Notably, SA6D performs consistently better than baselines and shows reasonable prediction with one-shot reference image.
7

---

0.5
SAGO
0.3
4000
0.3
0.1
Outs
10
1.0
Reference Num
Onilng-ddaptyve itoration
(a) Acc. VS reference number
(b) Acc. vs online iterations
(c) Region proposal
Figure 5: Analysis of the number of (a) reference and (b) online iterations. An example of
proposed ROI from SA6D (red) and GenóD (blue), the red cross denotes the target object.
Analysis of Online Self-Adaptation. The performance of SA6D w.r.t the number of iterations in the online adaptattn module shown in Fig. 5b on LineMOD-OCC/drilles SA6D performs poorly since the segmentor yo* cannot learn and differentiate the representation of the target object from others, which also leads performance decrease. After 12 iterations, with a learned distinguishable target object representation, SA6D gains significant improvement. With more iterations, the performance decreases again since the updated segmentor fo* starts overfitting to the reference images. We prevent overfitting by automatically stopping updating fp* with a defined threshold to the contrastive loss in Eq. (1). In our experiments, we set the threshold to 0.01 over all datasets.
the beginning,
Comparison against category-level methods. Tab. 3 demonstrates the comparison against category-level SOTA methods Wild6D dataset. Although SA6D is not trained specifically on each category, it overall achieves competitive performance and even outperforms baselines using the strict criteria (5*2cm), which indicates SA6D can predict more accurate poses than all baselines. In the appendix, we also visualize the predictions of SA6D and RePoNet [11] for comparison.
Discussions. We find that online self-adaptation module is robust against false positive sam- ples and is able to learn correct larget-oriented representation and shows remarkable performance against severe occlusion and truncation. Moreover, SA6D provides explainable confidence scores by computing the cosine similarity among the candidate segments. We also tried replacing ICP with a lcarning-basee based method, namely RPM-Net [48]. However, the prediction is always stuck at the sub-optimum. Nevertheless, we believe SA6D can be further improved with future development in the area of segmentation and learning- based point could registration, which is not the main focus of this work. The inference running time single image costs ~0.93s in total Nvidia Tesla V100 for SA6D. More discussion on dynamic scenes are shown in the appendix A.3.
4.3 Limitations
Our work does not consider deformable or articulated objects, especially for cases where reference and test images have drastic shape diversity. Another notorious concern predicting transparent ob- jects where sensors are often failed to capture depth information. Recent work on depth completion for transparent such as Zhu et al. [49] can alleviate the problem. Furthermore, our method requires accurate and generalizable base segmentor yp Although SA6D achieves promising results in most cases for tabletop objects, the under- and serreeegmetttiion behaviors still limit the performance. Moreover, a more generalizable learning- based registration method between partial and global point clouds would be an interesting direction to replace ICP.
Conclusion
We propose an approach that can efficiently and robustly predict the 6D pose of novel objects with heavy occlusions while not requiring any object information or object-centric images. We hope our approach can facilitate generalizable 6D object pose estimation in robotic applications.

---

References
[1] C. Wang, D. Xu, Y. Zhu, R. Martin-Martin, C. Lu, L. Fei-Fei, and S. Savarese. Densefusion: 6d object pose estimation by iterative dense fusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[2] S. Peng, Y. Liu, Q. Huang, X. Zhou, and H. Bao. Pynet: Pixel-wise voting network for 6dof pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[3] Y. He, W. Sun, H. Huang, J. Liu, H. Fan, and J. Sun. Pvn3d: A deep point-wise 3d keypoints voting network for 6dof pose estimation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[4] Y. He, H. Huang, H. Fan, Q. Chen, and J. Sun. Ffb6d: A full flow bidirectional fusion network for 6d pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3003-3013, June 2021.
[5] H. Wang, S. Sridhar, J. Huang, J. Valentin, S. Song, and L. J. Guibas. Normalized object coordinate space for category-level 6d object pose and size estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[6] D. Chen, J. Li, Z. Wang, and K. Xu. Learning canonical shape space for category-level 6d object pose and size estimation. In /EEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[7] C. Wang, R. Martfn-Martín, D. Xu, J. Lv, C. Lu, L. Fei-Fei, S. Savarese, and Y. Zhu. 6- pack: Category-level 6d pose tracker with anchor- keypoints. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 10059-10066, 2020. doi:10. 1.0/ SBRA40945.0020.91966
[8] W. Chen, X. Jia, J. Chang, J. Duan, L. Shen, and A. Leonardis. Fs-net: Fast chape-based net- work for category-level 6d object pose estimation with decoupled rotation mechanism. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1581-1590, June 2021.
[9] X. Chen, Z. Dong, J. Song, A. Geiger, and O. Hilliges. Category level object pose estimation via neural mmalysis-by-syntheee In European Conference on Computer Vision (ECCV), Cham, Aug. 2020. Springer International Publishing.
[10] K. Chen and Q. Dou. Sgpa: Structure-guided prior adaptation for category-level 6d object pose estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2773-2782, October 2021.
[11] Y. Fu and X. Wang. Category-level 6d object pose estimation in the wild: semi-supervised learning approach and new dataset. abv/arXiv22266, 15436, 2022.
[12] K. Zhang, Y. Fu, S. Borse, H. Cai, F. Porikli, and X. Wang. Self-supervised geometric corre- spondence for category-level 6d object pose estimation in the wild. ArXiv, 2210.07199, 2022.
[13] Y. Li, N. Gao, H. Ziesche, and G. Neumann. Catesory-agnostic 6d pose estimation with conditional neural processes. Women in Computer Vision (WiCV) workshop, CVPR, 2022.
[14] Y. Liu, Y. Wen, S. Peng, C. Lin, X. Long, T. Komura, and W. Wang. Genód: Generalizable model-free 6-dof object pose estimation from rgb images. In The European Conference Computer Vision (ECCV), 2022.
[15] N. Gao, H. Ziesche, N. Vien, M. Volpp. and G. Neumann. What matters for meta-learning vision regression tasks? In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 14776-14786. June 2022.

---

[16] K. Park, A. Mousavian, Y. Xiang, and D. Fox. Latentfusion: End-to-end differentiable re- construction and rendering for unseen object pose estimation. 2020 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10707-10716, 2020.
[17] Y. He, Y. Wang, Fan, J. Sun, and Q. Chen. Fs6d: Few-shot pose estimation of novel objects. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recog- nition (CVPR), pages 6814-6824 June 2022.
[18] I. Shugurov, F. Li, B. Busam, and Ilic. Osop: A multi-stage one shot object pose estimation framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pages 6835 0835-6844, June 2022.
[19] Y.-.. Lin, P. R. Florence, J. T. Barron, A. Rodriguez. P. Isola, and T.-Y. Lin. inerf: Invert- ing neural radiance fields for pose estimation. 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1323-1330, 2021.
[20] S. M. Rusinkiewicz and M. Levoy. Efficient variants of the icp algorithm. Proceedings Third International Conference on 3-D Digital Imaging and Modeling, pages 145-152, 2001.
[21] Z. Fan, Y. Zhu, Y. He, Q. Sun, H. Liu, and He. Deep learning on monocular object pose detection and tracking: A comprehensive overview. ArXiv, abs/2105. 14291, 2021.
[22] M. Tian, M. H. Ang, and G. H. Lee. Shape prior deformation categorical 6d object pose and size estimation. A. Vedaldi, H. Bischof, Brox, and J.-M. Frahm, editors, The European Conference on Computer Vision (ECCV), pages 530-546, Cham, 2020. Springer International Publishing. ISBN
[23] D. P. Kingma and M. Welling, Auto-encoding variational bayes. International Conference Learning Representations (ICLR), 2014.
[24] S. Umeyama. Least-squares estimation of transformation parameters between two point pat- terns. IEEE Transactions on Pattern Analysis and Machine Intelligence, 13(4):376-380, 1991. doi: 10i:10.1109/// 11t.0..199/34888773
[25] T. Lee, B. uk Lee, I. Shin, J. Choe, U. Shin, I.-S. Kweon, and K.-J Yoon. Uda-cope: Unsu- pervised domain adaptation for category-level object pose estimation. 2022 IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), pages 14871-14880, 2021.
[26] Lee, J. Tremblay, V. Blukis, Wen, B.-U. Lee, I. Shin, Birchfield, S. Kweon, and K.-J. Yoon. Tta-cope: Test-time adaptation for category-level object pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21285-21299 June 2023.
[27] B. Mildenhall, P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In The European Conference on Computer Vision (ECCV), 2020.
[28] Sun, Z. Wang, S. Zhang, X. He, H. Zhao, G. Zhang, and X. Zhou. Onepose: One-shot object pose estimation without cad models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6825-6834, June 2022.
[29] X. He, Sun, Wang, D. Huang, Bao, and Zhou. Onepose++: Keypoint- free one-shot object pose estimation without CAD models. In Advances in Neural Information Processing Systems, 2022.
[30] A. Gouda, A. Ghanem, and C. Reining. Category-agnostee segmentation for robotic grasping. ArXiv, abs/2204, 13613, 2022.
10

---

[31] M. Danielczuk, M. Matl, S. Gupta, A. Li, A. Lee. J. Mahler, and K. Goldberg. Segmenting unknown 3d objects from real depth images using mask r-enn trained on synthetic data. 2019 International Conference on Robotics and Automation (ICRA), pages 7283-7290, 2019.
[32] C. Xie, Y. Xiang, A. Mousavian, and D. Fox. The best of both modes: Separately leveraging rgb and depth for unseen object instance segmentation. In CoRL, 2019.
[33] N. Gao, B. Hohmann, and G. Neumann. Enhancing interpretable object abstraction via clustering- based slot initialization. 34rd British Machine Vision Conference (BMVC), 2023.
[34] C. Xie, Y. Xiang, A. Mousavian, and D. Fox. Unseen object instance segmentation for robotic environments. IEEE Transactions on Robotics (T-RO), 2021.
[35] Y. Xiang, C. Xie, Mousavian, and D. Fox. Learning rgb-d feature embeddings for unseen object instance segmentation. CoRL, 2020.
[36] T. Kobayashi and N. Otsu. Von mises-fisher mean shift for clustering on a hypersphere. 2010 20th International Conference on Pattern Recognition, pages 2130-2133, 2010.
[37] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In H. D. III and A. Singh, editors, Proceedings of the 37th Interna- tional Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1597-1607. PMLR, 13-18 Jul 2020.
[38] Q.-Y. Zhou, J. Park, and V. Koltun. Fast global registration. In European Conference Computer Vision, 2016.
[39] S. Hinterstoisser, Lepetit, S. Ilie, S. Holzer, G. Bradski, K. Konolige, and N. Navab. Model based training, detection and pose estimation of texture-less 3d objects in heavily cluttered scenes. In K. M. Lee, Y. Matsushita. J. M. Rehg, and Z. Hu, editors, Computer Vision ACCV 2012, pages 548-562, Berlin, Heidelberg, 2013. Springer Berlin Heidelberg. ISBN 978-3-642- 37331-2.
[40] E. Brachmann, A. Krull, F. Michel, S. Gumhold, J. Shotton, and C. Rother. Learning 6d object pose estimation using object coordinates. The European Conference on Computer Vision (ECCV), 2014.
[41] R. Kaskman, pose estimation of 3d objects. 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pages 2767-2776, 2019.
S. Zakharov, I. S. Shugurov, and S. Ili.. Homebreweddb:: Rgb-d dataset for 6d
[42] P. JishnuJaykumar, Y.-W. Chao, and Y. Xiang. Fewsol: A dataset for few-shot object learning in robotic environments. ArXiv, abs/2207 .00333, 2022.
[43] Lin, Z. Wei, Z. Li, Xu, Jia, and Y. Li. Dualposenet: Category-level 6d object pose and size estimation using dual pose network with refined learning of pose consistency. 202/ IEEE/CVF International Conference on Computer Vision (ICCV) pages 3540-3549, 2021.
[44] C. Xie, Y. Xiang, A. Mousavian, and D. Fox. The best of both modes: Separately leveraging rgb and depth for unseen object instance segmentation. CoRL, 2019.
[45] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li. S. Savarese, M. Savva, S. Song. H. Su, J. Xiao, L. Yi. and F. Yu. Shapenet: An information-rich 3d model repository. ArXiv, abs/1512.03012, 2015.
[46] Q. Wang, Wang, K. Genova, P. Srinivasan, H. Zhou, J. Barron, Martin-Brualla, N. Snavely, and T. A. Funkhouser. Ibrnet: Learning multi-view image-based rendering. 2021 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 4688 4697, 2021.

---

[47] M. Sundermeyer, T. Hodan, Y. Labbe, G. Wang. E. Brachmann, B. Drost, C. Rother, and J. Matas. Bop challenge 2022 on detection, segmentation and pose estimation of specific rigid objects, 2023.
[48] Z. J. Yew and G. H. Lee. Rpm-net: Robust point matching using learned features. 2020 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11821- 11830, 2020.
[49] L. Zhu, A. Mousavian, Y. Xiang, H. Mazhar, J. van Benbergen, Debnath, and D. Fox. Rgb-d local implicit function for depth completion of transparent objects. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4647-4656, 2021.
12

---

In appendix, we show additional qualitative results on the predicted target object segmentation in- cluding severe occlusion and truncation, qualitative results on the final 6D pose estimation against Gen6D, comparison between ICP and learning-bant based point cloud registration method, additional ablation studies, and further explanation and analysis on existing methods compared to our method, e.g., the selection of reference images, the effort of annotation, and the practical use case. We also submit video introducing SA6D in the supplementary material.
A
Additional Results
A.1
GenóD without ground-truth object diameter.
In Tab. 4, we demonstrate that using the object diameter as input a strong prior knowledge which limits the generalization Gen6D, by fixing the diameter over objects with two different values, namely 10 cm and 50 em. Without ground-truth diameter, Gen6D cannot generalize well on any of the datasets.
Dataset
Diam. (m)
LM
LMO
FewSOL
HB
0.1
0.06
0.06
0.04
0.10
0.5
0.16
0.05
0.00
0.19
GT
0.35
0.08
0.36
0.30
Table 4: Evaluation on Gen6D with different object diameters as prior knowledge. Results are averaged over objects for each dataset.
A.2 Compare ICP with learning-based point cloud registration algorithm
We show a few predicted examples of state-of-the-art learning-based point cloud registration model, namely RPM-Net, on the DHeMMDD-CC//rill in Fig. 6. RPM-Net is prone to the lo- cal optimal position for 6D pose estimation, especially for rotation. In our experiments, ICP is more robust to unseen objects.
Figure 6: Examples of using RPM-Net for point cloud registration instead of ICP. The yellow point cloud denotes the reconstructed object point cloud model and the blue one denotes the prediction after transformation using the predicted pose from RPM-Net. Better overlapping between two point clouds indicates better performance. RPM-Net cannot generalize on unseen objects and is prone to get stuck in local optima.
A.3 Results on dynamic scenes.
We have conducted the evaluation on dynamic scenes where we sample 20 images from LineMOD- OCC (LMO) as reference images and evaluate query images from LineMOD (LM) which includes different background and objects. Thus, the reference and query images are from different scenes with changing light conditions and configuration of surrounding objects. The results shown in Tab. 5. SA6D achieves competitive results compared to the original setup (first line in the table).
13

---

Note that this is still not fair comparison for our method since we use reference images with occluded target objects, which makes it more difficult to reconstruct the object model. However, this experiment demonstrates the capability of our method on query images from novel scenes.
Ref. image
Query image
eggbox
duck
cat
glue
avg.
LM
LM
0.73
0.73
0.47
0.72
0.66
LMO
LM
0.70
0.71
0.46
0.72
0.64
Table 5:
Evaluation on LineMOD using LineMOD-OCC reference.
A.4 SA6D is robust to false positive samples in reference
Using reprojected object center to select positive segments sometimes leads to false positive sam- ple given the target object center is occluded. An example is shown in Fig. 7a, in which wrong segment (yellow rabbit) is selected as a positive sample for the target object (milk cow). However, we find that our online self-adaptation module is robust against false positive samples and is able to learn a correct target-oriented representation. Moreover, SA6D provides explainable confidence scores by computing the cosine similarity between each segment representation and the target ob- ject representation. Fig. 7b shows an example of the predicted target (milk cow) segments with reasonable induced confidence score though wrong positive samples are given in the reference set.
PR... #conf: O 933040
PR 1. #cont 2oone.0.22000 0,226777
t E EEE SSBSS
(a) False positive sample.
(b) Robust prediction.
Figure 7: Discussion. (a) A false positive sample is selected given the reprojected center the target object (milk cow) is occluded by another object (yellow rabbit). Nevertheless, (b) SA6D provides robust prediction with explainable confidence scores.
A.5 SA6D demonstrates remarkable performance against severe occlusion and truncation
We show superior performance of SA6D on challenging scenes with severe occlusion and truncation in Fig. 8, where the input images, predicted segmentations from the base segmentor sp, ground-truth segmentation of target object based the reprojected object center, and three predicted candidates with the highest predicted confidence scores given each column from left to right. The lected segments are marked in white color. The confidence score conf denotes the cosine similarity between the candidate segment representation and the target object representation r*. The conf_seg computed by dividing the confidence scores between the first and second most similar segment candidates w.r.t. the target object representation. Thus, it can be used in crucial scenarios the prediction uncertain among different segments. Note that in Fig. 8a, our method is able dif- ferentiate the target object segment while the provided ground-truth segmentation points to wrong segment due to the center of target object is occluded.
A.6 Robust and explainable confidence score of the online self-adaptation module
We show more results on the predicted segmentation of the online self-adaptation module in Fig. on LineMOD dataset, Fig. 10 on LineMOD-OCC, and Fig. 11 on HomebrewedDB. Some candidates in Fig. 11 with white background indicate the background segments selected.
14

---

(a) cat
(b) driller
Sa Mar
Seats LO LAISH Seeteairc PP1 RIM SASIETY
DODOS
DOODA
DODDRR
duck
20-
flange
maay
Toyr Meso 20133 Nayl Buro W. ane Anidee oer
Mog Noo LeIT CCIMT
(e) yellow rabbit
Figure 8: Online- Adaptttton results challenging scenes against severe occlusion and truncation. Three candidates with the highest confidence scores are visualized in order.
A.7 More Qualitative Results
We show more qualitative results of the 6D pose prediction and compare our method with Gen6D on LineMOD dataset in Fig. 13, LineMOD-OCC in Fig. 14, HomebrewedDB in Fig. 15 and FewSOL. Fig. 16. The comparison on Wild6D dataset between SA6D and category-level SOTA method RePoNet is shown in Fig. 17.
A.8 Failure Cases
We show the examples in Fig. where using ICP leads to worse prediction than without using ICP in the refinement module. Results evaluated the FewSOL dataset, indicating future work generalizable and learnable point cloud registration is essential to further improve the performance.
Additional Explanation
B.1 Selection of Reference Images
Regarding the selection of the reference images on the LM, LM-O and HB datasets, the original Gen6D selects 64 reference images from predefined of images with farthest point sampling (FPS) to make sure that the view distributes evenly among the reference images. We follow the
15

---

Dopel and
(a) benchvise
cam
cat
meet AAIDN MANCE Eryet W Aeni Linee
Sogo Molee A3300l
driller
Sew Nea soams Sopdues at mo SOOW
Tooceeen and JTHSM
duck
(l) eggbox
(g) glue
Figure 9: Robust prediction of target segmentation on LineMOD. Three candidates with the highest scores visualized in order.
16

---

B0006
DOODE
(a) ape
b can
Taya buth
Teard Pato N3 avet 110011
(c) cat
ROOO
JOOO
CADDE BOOD
(d) driller
duck
(f) eggbox
DOODO
glue
Figure 10: Robust prediction of target segmentation on Lin-MOD-OCC. Three candidates with the
highest scores visualized in order.
17

---

(a) COW
Tarpt Newn ROler Tuper Uade Roreh EATTA LLSAN baon Rorf 411MTM
P1 Aook JANAAS
Tean Nepo eboor Tran Sath men LOANN A0 Modeeee Io aoer ACMONO
(b) flange
Syp Medi edpeto
MITTO CUEIICA Tage EI0811
car
(d) yellow rabbit
Figure 11: Robust prediction of target segmentation on HomebrewedDB. Three candidates with the highest scores visualized in order.
18

---

Gen6D
Ours
Figure 12: Failure cases. Using ICP in the refinement module leads to worse prediction than the initial prediction. The green bounding box is the ground-truth pose. The blue bounding box denotes the prediction in Gen6D and the prediction before using ICP in the refinement module our method while the red one denotes the prediction after ICP.
same setup when all models evaluated with 64 reference images. However, it is not efficient to sample 64 images and it is often the case that the reference images are not distributed evenly in the realdworld. Therefore, we also evaluate all methods by randomly selecting 20 reference images from the dataset, which significantly increases the task difficulty but is more realistic and plausible because is not always obtainable to collect reference images that could cover all viewpoints.
B.2 Comparison with FS6D and Model-Based Models
Similar to LatentFusion, FS6D [17] also requires object-centric reference images with ground-truth segmentations for cluttered scenes. Considering that its code is not published and could not reproduce its results, we hence excluded FS6D in our comparisons. Meanwhile, we cannot add the model-based methods [4, 3, 2, 1] into comparison due to their limitation, i.e., the model-based methods can only be applied on the specifically trained object and cannot work in our setup where the results are evaluated on new objects. Also, unfair to compare them with our work we train the model-based methods on the new objects. Moreover, FewSOI. dataset contains only images for each object, which is insufficient train the model-based methods. Considering all these limitations of the model-based methods, it is also one of motivations work this paper.
B.3 Effort of Annotation Compared with Prior Work
The annotation of a limited number of reference images requires human effort. However, the effort of annotation is also essential in prior work [8, 4, 3, 2, 22, 1, 5] where thousands of annotated images are required for every single object or category. Category-agnostee tremendously reduce human effort by requiring only a small number annotations. Still, similar to Gen6D and LatentFusion, it is necessary have small number of posed reference images for an unseen object the canonical object coordinates further determine the object rotation w.r.t. the camera. Importantly, method does not require any additional effort compared to existing methods.
methods such as our method
B.4 Practical Use Case
Our method can be used in the lifelong robot item picking/sorting in industry. Each time when a new product comes in, the robot only needs to sample a small number of images with ground- truth 6D) pose between the new product and the camera by moving the robot arm around the new product where the camera is mounted the robot arm and the other objects together with the new product are placed calibrated picking plate. The pose between the camera and the new product is easily obtainable since the pose of the camera and new product w.r.t. the robot base coordinates
19

---

SEDOO
Gen6D
1000
MOOO
Ours
LM / Duck
So Boodaa
DOOOR
Gen6D
OOOD
Ours
200008
00008
LM / Driller
PORER
BOOODE Coo
Gen6D
DAOOPE
DODO
Eggbox
Gen6D
Ours
COOR
GOOO
LM Benchvise
Gen6D
OOPOCR
CORB
BOOOGO
/
Figure 13: Prediction on LineMOD dataset with 20 reference images. The green bounding box is
the ground-truth pose. The blue bounding box denotes the prediction Gen6D and the prediction before using ICP in the refinement module in method while the red one denotes the prediction after ICP.
known. Thus, the whole system be fully automatic and does require further training for new products.
20

---

Gen6D
Sandus
RAODE
Ours
1000
MAOP
LMO Driller
CICOR
100000
Gen6D
ODDGS
DEG
GOOON
PARR
DOOD
Ours
00005
LMO Duck
Gen6D
Ours
LMO Eggbox
Gen6D
Ours
BOOOR
LMO Glue
Figure 14: Prediction on LineMOD-OCC dataset with 20 reference images. The green bounding box the ground-truth pose. The blue bounding box denotes the prediction in Gen6D and the prediction before using ICP in the refinement module in our method while the red one denotes the prediction after ICP.
21

---

Gen6D
Ours
HB Milk Cow
Gen6D
Gengnsses
Ours
HB Flange
Gen6D
Ours
White car
Gen6D
Ours
Yellow rabbit
Figure 15: Prediction HomebrewedDB dataset with 20 reference images. The green bounding box is the ground-truth pose. The blue bounding box denotes the prediction Gen6l) and the prediction before using ICP in the refinement module in method while the red denotes the prediction after ICP.
22

---

P B
B
Gen6D
B
Ours
B
E
Gen6D
8 B
m G
Ours
G m
Gen6D
a
Ours
48
B
1
Gen6D
0
Ours
Figure 16:
Prediction FewSOL dataset with 20 reference images. The green bounding box is the ground-truth pose. The blue bounding box denotes the prediction Gen6D and the prediction before using ICP in the refinement module in method while the red denotes the prediction after ICP.
23

---

RePoNet
Ours
Mug
RePoNet
Ours
Camera
RePoNet
Ours
Bowl
Figure 17: Prediction on Wild6D dataset with 20 reference images. The red bounding box is the ground-truth pose and the green bounding box denotes the prediction.
24


Reviews:
[{"review_id": "yciA0CaQL6I", "reviewer": "Reviewer_pbhk", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces an approach for 6D pose estimation of novel objects with heavy occlusions while not requiring any object information or object-centric images. \n\nMore specifically, it fine-tunes the weights of a base image segmentation model using reference images with pose labels, which allows to better localize the target object. Moreover, it leverages the geometric features to improve the robustness against occlusion in comparison to Gen6D.\n\nExperimental results on various benchmarks demonstrated the effectiveness of the proposed method, which could potentially facilitate generalizable 6D object pose estimation in robotic applications.", "summary_of_recommendation": "This paper proposes a simple yet effective object pose estimation approach with good representation and convincing results. In particular, it achieves generalizable object pose estimation in heavy occlusions, with models trained on synthetic data only.\nThis leads to promising robotic applications where the robot needs to interact with novel objects in cluttered scenes.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Please refer to the weaknesses.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Strengthes\n- The paper is well-written with nice figures and clear equations. \n\n- The proposed method achieves superior object pose estimation performance in various settings while being trained on synthetic data only.\n\n- The proposed method builds upon a simple image segmentation model and basic point cloud registration methods, which is simple yet effective.\n\n### Weaknesses\n\n- The method assumes that the target object is always contained by the test image, what if the object is completely occluded in the test image? Can we tell the absence of the target object from the feature similarity, thus avoiding invalid pose estimation?\n\n- How long does it take to complete the online adaptation? How will the adaptation time scale in the case of large-scene containing hundreds of objects?", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}, {"review_id": "wzeJfAEltR", "reviewer": "Reviewer_L1f1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes an RGB-D approach for few-shot 6-D pose estimation of novel objects, requiring as input a small set of posed RGB-D reference images and also handling severe occlusions. The method effectively combines pre-existing tools, including: 1. the object-agnostic segmentation module UCN, and an InfoNCE contrastive loss, used to extract object masks in both the reference images and in the test one and train an global object feature representation; 2. The Gen6D pose estimator, used to provide an initial 6-D pose estimate from an object crop; 3. ICP for point-cloud based pose refinement. This combination results in improved accuracy over state of the art and increased robustness to occlusions. Additional contributions include:\n- The removal of the need for object-centric reference images or object information (e.g., 3-D model or object diameter).\n- A method to extract the object crop in a test image which is more robust to occlusion and clutter, by using depth information.", "summary_of_recommendation": "While mostly based on a combination of pre-existing tools, the proposed approach advances the performance on the problem of few-shot 6-D pose estimation of novel objects. Furthermore, despite the additional requirement of depth information, it relaxes the assumptions of previous methods and further increases the robustness of these methods against occlusions and clutter, which are practical problems relevant for robotic applications. I therefore believe the paper would be a valuable contribution for the community and recommend acceptance.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- In eq. (1), how are the negatives sampled? Does each term $l_{ij}$ actually include all the negatives in the denominator?\n- How long is the training time for each new object?\n- Could the authors comment also on the parameter size of the used UCN models? The algorithm requires instantiating an \"adaptive\" UCN for each new object, which potentially does not scale up well as many new objects are added.\n- How exactly is the object diameter $\\hat{d}$ estimated?\n- In multiple points of the paper, and in particular for the Refinement Module, \"geometric features\" are mentioned. What exact geometric features were used? In particular, was standard ICP used or somehow additional geometric features/descriptors are used, both for the region proposal module and for ICP?\n- L233, Fig. 5b: Could the authors comment on why the increase in accuracy is so sharp and sudden?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\n### Strengths\n- Improved performance and robustness to occlusion and clutter w.r.t. state-of-the-art methods.\n- Thorough experiments and ablations.\n- Very good quality of the manuscript.\n- The supplementary material (particularly the video) is useful and facilitates the understanding of the main manuscript.\n\n### Weaknesses\n- The method is not extremely novel, in the sense that it combines pre-existing tools (although this combination is not trivial and results in improved performance).\n- Compared to previous few-shot approaches, the method assumes the availability of depth information.\n\n## Detailed review\n### Quality and clarity\n- The problem that the paper addresses is clearly stated and motivated, and an extensive list of related work is provided.\n- The experiments are thorough, with evaluations on several standard datasets and with multiple types of baselines (both few-shot methods and RGB-D category-level methods) and ablations on the model components. All my main questions regarding the influence of the difference components were answered through these experiments.\n- The quality of the manuscript, of the visualization and the supporting material is in general high, with a very limited amount of typos and mostly clear and linear language (cf. \"Suggestions and minor questions\" below).\n\n\n### Originality and significance\n- Although the method does not present groundbreaking new ideas, it combines pre-existing tools in a effective way, resulting in a clear improvement over previous methods.\n- No experiments from a real-world setup are presented, but the method is evaluated in several real-world benchmarks.\n- I find particularly interesting the simple, but smart idea, behind the selection of the mask for the object of interest in the reference images, by using the known extrinsics. I also appreciate the discussion in Sec. A.3 about how such a strategy might fail, but the confidence score can still compensate for it.\n\n### Suggestions and minor questions\n**Language**\n\n- The abstract is probably the main part for which I would suggest rephrasing. In particular, the sentence \"[...] meaningful robotic manipulation [...] one of the critical aspects\" sounds vague and a bit clunky (e.g., what does \"meaningful\" mean exactly? \"Critical\" for what?). Similarly, the term \"continuously [introduced]\" (L4) can have a fairly specific (and diverse) meaning in the literature, so when reading it I would expect the paper to put emphasis for instance on the \"open-set\" problem, or potentially on some continual learning problems, which is not done in the rest of the paper.\n- Minor: Throughout the paper, \"i.e.\" could be replaced with the macro \"\\ie\".\n- L124: \"copying $\\phi$\" -> \"copying **the parameters of** $\\phi$\".\n- L128: $\\phi*$ -> $\\phi^*$ or $\\phi^\\star$.\n- Caption of Table 1: \"HomwBrewedDB\" -> \"HomeBrewedDB\".\n- In all its occurrences (header of Table 1, L223, caption of Fig. 5, etc.), I would recommended replacing \"reference number\"/\"reference\" with \"number of reference images\".\n- L203: Fix \"cannot _even though without_ [...]\". Also, on the same line, the tense of \"found\" is not coherent with the rest of the paragraph.\n- L225: \"[...] except that LatentFusion sometimes [...]\" -> \"[...] except LatentFusion, which sometimes [...]\".\n- L246: \"replace\" -> \"tried replacing\" or \"experimented replacing\" (tense coherency).\n- L250: \"costs\" -> \"is\".\n- Fig. 7 in the Suppl.: \"Even though\" -> \"Nevertheless\".\n- L58 Suppl.: Missing \"the case\" after \"It is often\".\n- L65 Suppl.: \"exclude\" -> \"excluded\", \"We\" -> \"we\"\n\n**Notation and formatting**\n- L123: Technically, $S^N$ should be defined as $\\bigcup_{i=1}^M S^N_i$, since the $S^N_i$s are sets. A space is missing after the word \"segments\".\n- L155-157: It could be clarified that $T^x_{init}$, $T^y_{init}$, $T^z_{init}$ are part of $T_{init}$.\n\n**Additional details**\n- The $n^\\circ, m~\\mathrm{cm}$ metric, used in Tab. 3, was not mentioned in the \"Evaluation metrics\" paragraph (L189-194).", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "byQKKuTFWkY", "reviewer": "Reviewer_Zris", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work tackles the task of 6D pose estimation of a generic object in occluded scene, given a few RGB-D images (under clutter) with known 6D poses for the target object as reference. The approach is to: a) adapt an off-the shelf instance segmentation system to better identify the target object, and b) build a object-specific 3D model using the 6D pose annotations and predicted segmentations. Given a test image, the adapted segmentation system can detect the object and an off-the-shelf 6D pose estimation system can yield initial 6D pose estimate (given object-centric crops from test and reference images), which is then refined via ICP. Experiments show that this approach performs better than prior methods across several examined datasets.", "summary_of_recommendation": "First and foremost, I have significant concerns regarding the setup (specifically, whether the train and test images come from common scenes). That said, even assuming these are addressed by the author response, I am still not overly positive about the work and would probably rate this as borderline because the primary contribution seems to be in few-shot adaptation of a segmentation system as opposed to any innovation in 6D pose estimation, which is done via Gen6D + ICP. On average, I am currently indicating my rating as a Weak Reject. \n---\n\nPost-discussion update: The author responses (specifically the new experiment showing ability to test in different scenes) did address my concern, and as indicated in my comments in the discussion below, I'd be happy to lean towards acceptance.\n\nHopefully the authors will include this experiment in the final version.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "a) Please clarify the concern above regarding how the reference images and test images are selected, and what is the variation in the scene (if any) across them.\n\nb) In the ablations, is “SA6D w/o RPM” using the pre-trained segmentor, or is there no segmentor used? If the latter, how is the local region for ICP selected as one should only use points from the forground?\n\nc) On a related note to b), while Gen6D uses only visual cues for prediction, can it not be combined with a refinement model to perform ICP (using an off-the-shelf segmentation)? This would better help disentangle the need for the adaptive segmentation that the paper proposes to learn.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n+ The paper tackles an interesting setting. Specifically, unlike some prior works which require either object meshes or reference images without clutter, this paper can leverage reference images in a cluttered setting. \n\n+ The empirical results are shown across datasets, and the importance of the contributions is  evaluated well.\n\nConcerns:\n- Please correct me if this is wrong, but it seems that reference and test images are from overlapping scenes (i.e. common background and object configurations across reference images and test image). In my view, this is a significant drawback of the setup as it’s not clear if the 6D pose estimation systems or segmentation adaptation actually generalizes (and pays attention to the object vs scene). Moreover, if the reference and test images are indeed the same scene, why do we even need to estimate 6D poses of objects? Instead, we can simply try to obtain relative camera poses between reference and test images (e.g. via COLMAP), and given 6D object pose in any reference image, can trivially obtain 6D pose of object is test image.\n\n- While the paper presents itself as tackling the task of 6D pose estimation, the core technical contributions infact lie in adapting a segmentation model. The 6D pose estimation is actually done via a combination of a previous approach (Gen6D) followed by ICP. The key contribution here lies in improving the detection/segmentation so that clean inputs can be fed to Gen6D and ICP. While this is still a reasonable thing to do, there are no 2D detection/segmentation evaluations. I feel the paper would be better presented (and evaluated) as one for improving few-shot segmentation/detection for a generic object, with one possible application to 6D pose estimation.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "VdgF-dkm0g", "reviewer": "Reviewer_8PhQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "- Online self-adaptation module\n  - Extract standard object model: Given reference images and pose of target, a segmentor is employed for segmenting the target objects. Then, by combining the point cloud of the target object, the \"standard\" object point cloud model is generated.\n  - Generate a target-specific segmentor: Given the masks for target/non-target objects, contrastive learning is employed for finetuning the segmentor.\n  - Segment test image: Test images are fed into both pretrained and finetuned segmentor for per-pixel label and dense feature, respectively. Target object is selected by comparing the feature with reference.\n- Region Proposal Module: Estimate the size of target object based on the standard object model and crop the region of the same size on the target image. Then images are fed into Gen6D to generate a initial guess for the pose.\n- Refinement module: Performs ICP between standard object model and segmented object from test image to refine the pose prediction. ", "summary_of_recommendation": "For now, a weak acceptance is given for the online adaptation. However, more results are required for supporting the effectiveness of the method, especially the effectiveness of the proposed online adaptation module.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "- In Table 1, why SA6D with ICP only performs worse than Gen6D?\n- What's the configuration (like the number of pairs and training time) for contrastive learning?\n- What's the sensitivity to the performance of the segmentor?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "- Strength\n  - The paper is well-written and easy to follow.\n  - The topic of self-adaptative pose estimation is important to many practical applications\n- Weakness\n  - Need ablation study without online adaptation. Say both pixel label and dense feature are generated with the pretrained UCN.\n  - Need baseline results for RePoNet, DualPoseNet on LineMod/LineMOD-OCC, etc.\n  - Performance in uncontrolled environment is questionable. As shown in Table 3, on Wild6D, SA6D can hardly out-performs RePoNet, which has a much simpler inference pipeline.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a novel method on few-shot object pose estimation that can handle novel objects.

After the rebuttal, most concerns of the reviewers have been addressed, and all the reviewers agree on accepting the paper.

I recommend the paper as a poster presentation due to the lack of applying the pose estimation method to real robotic applications such as grasping or manipulation of objects.
Decision: Accept (Poster)