Learning Shape-Independent Transformation via Spherical Representations for Category-Level Object Pose Estimation

Authors: Huan Ren, Wenfei Yang, Xiang Liu, Shifeng Zhang, Tianzhu Zhang
Primary area: applications to robotics, autonomy, planning
Keywords: category-level object pose estimation, spherical representations, shape-independence, correspondence prediction

Abstract:
Category-level object pose estimation aims to determine the pose and size of novel objects in specific categories. Existing correspondence-based approaches typically adopt point-based representations to establish the correspondences between primitive observed points and normalized object coordinates. However, due to the inherent shape-dependence of canonical coordinates, these methods suffer from semantic incoherence across diverse object shapes. To resolve this issue, we innovatively leverage the sphere as a shared proxy shape of objects to learn shape-independent transformation via spherical representations. Based on this insight, we introduce a novel architecture called SpherePose, which yields precise correspondence prediction through three core designs. Firstly, We endow the point-wise feature extraction with SO(3)-invariance, which facilitates robust mapping between camera coordinate space and object coordinate space regardless of rotation transformation. Secondly, the spherical attention mechanism is designed to propagate and integrate features among spherical anchors from a comprehensive perspective, thus mitigating the interference of noise and incomplete point cloud. Lastly, a hyperbolic correspondence loss function is designed to distinguish subtle distinctions, which can promote the precision of correspondence prediction. Experimental results on CAMERA25, REAL275 and HouseCat6D benchmarks demonstrate the superior performance of our method, verifying the effectiveness of spherical representations and architectural innovations.

Decision: accept (poster)
Reviewer scores: 8, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730305829250

thread_type:
official_review

<thread_object>
creator: reviewer_p4nq

<content>
summary:
This paper introduces a new method for category-level object pose estimation. The authors argue that the shape-dependent representations in previous approaches lead to semantic inconsistencies across different objects within the same category. To handle this issue,  a shape-independent transformation is presented, transforming features to a sphere. The authors also introduce the importance of SO(3)-invariant features for category-level object pose estimation. Multiple SO(3)-invariant features extracted from RGB images and point clouds are combined to learn the NOCS coordinates on a sphere. Experimental results show SpherePose outperforms state-of-the-art methods on benchmarks like CAMERA25, REAL275, and HouseCat6D, showing its effectiveness in handling category-level pose estimation challenges.

strengths:
The idea of leveraging an object-agnostic representation to handle the problem of semantic inconsistencies is technically sound and interesting. The authors propose to combine multiple deep feature extractors, and its effectiveness is demonstrated in the ablation studies. The paper conducts extensive experiments on multiple benchmarks, showing better performance compared with existing methods. The paper is well-written and easy to understand.

weaknesses:
-	The presented ColorPointNet++ is not impressive. There are many alternatives [1,2,3] in the literature which can be used to extract SO(3)-invariant features.

-	The effectiveness of DINOv2 towards rotation invariance is questionable. The authors mentioned that it is “approximately” SO(3)-invariant, which is quite vague. It would be more convincing if the authors could provide some experimental results about the invariance of DINOv2 towards 3D rotations.

-	The difference compared with some previous methods that also use sphere representations, such as SpherePose, is unclear.  What is the major superiority compared with these methods?

-	Some important ablation studies are missing. For instance, the authors present a spherical feature interaction module to handle the challenges of self-occlusion. However, an ablation study on the effectiveness of this module is missing. Moreover, an experiment regarding the importance of SO(3)-invariant features for category-level object pose estimation is missing.

[1] Deng, Congyue, et al. "Vector neurons: A general framework for so (3)-equivariant networks." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

[2] Zhao, Chen, et al. "Rotation invariant point cloud analysis: Where local geometry meets global topology." Pattern Recognition 127 (2022): 108626.

[3] Fei, Jiajun, and Zhidong Deng. "Rotation invariance and equivariance in 3D deep learning: a survey." Artificial Intelligence Review 57.7 (2024): 168.

questions:
-	I am a bit confused about the SO(3)-invariant features in the field of object rotation estimation. We could formulate the rotation estimation problem as R = f(g(x)|w). g(x) means the feature extractor. If the extracted features are rotation-invariant, it would be g(R(x))=g(x). If this is the case, it means f(.) would be unaware of the rotation information. Why is the network able to make the prediction related to rotations? To me, SO(3)-equivariant features seem more reasonable. 
-	The proposed spherical representation is designed to be object-agnostic across different categories. What factors limit the method’s ability to generalize to objects from novel categories?
-	Why is the IoU75 worse than that of a point-based method AG-Pose on REAL275?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730562452159

thread_type:
official_review

<thread_object>
creator: reviewer_v4xt

<content>
summary:
This paper addresses the task of category-level object pose and size estimation, introducing a novel method called SpherePose. This method uses a sphere as a shared proxy shape for objects, enabling the learning of shape-independent transformations from spherical representations. To enhance the precision of correspondences on the sphere, SpherePose incorporates three core components, including  SO(3)-invariant point-wise feature extraction, spherical feature interaction, and a hyperbolic correspondence loss function. Experiments conducted on the CAMERA25, REAL275, and HouseCat6D datasets validate the effectiveness of SpherePose.

strengths:
- Unlike point-based representations, SpherePose uses a sphere as a shared proxy shape for objects and employs spherical representations to learn shape-independent transformations.

- Three core components are introduced based on spherical representations to enhance the precision of correspondences.

- SpherePose achieves state-of-the-art results on the CAMERA25, REAL275, and HouseCat6D datasets.

weaknesses:
- Are the spherical NOCS coordinates derived by normalizing the original NOCS coordinates to unit vectors? If so, it would be beneficial to provide results for regressing the original NOCS coordinates.

- Are the resulting poses obtained from the observed anchors or all sampled anchors on the sphere? If it’s the former, how can we verify that the spherical feature interaction aids in reasoning about the features of occluded anchors?

- Unlike most existing methods that use the Umeyama algorithm, SpherePose incorporates RANSAC for solving rotations. For a fair comparison, results without RANSAC for SpherePose should be included in Table 1.

- It is recommended to include results that do not use RGB or radius values as inputs in Table 4.

questions:
See Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730710007309

thread_type:
official_review

<thread_object>
creator: reviewer_rb6i

<content>
summary:
This paper addresses the challenge of category-level object pose estimation by proposing a shape-independent representation using spherical features. Starting with an RGB-D image and instance masks, the method processes "N" partial point clouds to estimate object pose parameters: rotation R \in SO(3), translation t \in R^3, and size s \in R^3 of the observed instance. The spherical feature construction on the SO(3) HEALPix grid, combined with a hyperbolic correspondence loss, significantly improves pose estimation performance. The paper categorizes pose estimation into correspondence-based vs. regression-based and point-based vs. spherical-based approaches. The proposed method, correspondence-based/spherical-based SpherePose, demonstrates state-of-the-art results on REAL275 and HouseCat6D benchmarks.

strengths:
1. Strong motivation: The proposed shape-independent proxy representation addresses a crucial need in category-level pose estimation, enhancing generalizability across different object instances.

2. Good analysis of correspondence errors: The method effectively identifies correspondences, as shown in Table 3, where accurate correspondence significantly enhances pose estimation accuracy. 
However, consistency of correspondence errors should be validated on another dataset, such as HouseCat6D.

3. Comprehensive Ablation Studies: 	The authors provide a detailed analysis of different feature extractors (DINOv2, ColorPointNet++) and loss functions (L2 vs. hyperbolic L2). While not the primary contribution, Table 4 highlights the role of backbone networks in performance improvement. Table 6 effectively isolates the impact of the hyperbolic L2 loss, validating its importance. 
However, the authors should clearly differentiate the contribution of the backbone networks from the main innovations of the paper (spherical-based proxy and correspondence-based loss), by comparing with existing methods under the same configurations.

weaknesses:
1. Justification for Spherical Proxy: 
Why is the 2-sphere an optimal proxy shape for this task? While the paper adopts the spherical representation, a more thorough explanation of the choice of spherical shapes would be helpful.

2. Size Estimation Despite Normalization: Section 3.1 mentions that the point cloud utilizes normalized coordinates, yet the method predicts the size parameter “s”.  How is accurate size estimation achieved despite normalization? Clarification on this aspect is necessary.

3. Focus on SO(3) Over SE(3): The method predicts rotation “R” and translation “t”, focusing on SO(3) invariance. Why is roto-translation-invariance (SE(3)) not considered? (and scale-invariance) This oversight could limit the method’s robustness in varying spatial contexts.

4. SO(3)-Invariant Features and Rotation Discrimination:
In Section 3.1, if point-wise features are SO(3)-invariant, how does the method differentiate between different rotations? Wouldn't rotation-“invariant” features cause the model to lose rotational information? An explanation on how SO(3)-equivariant features are maintained would be insightful.

5. Shape-Independence and HEALPix Projection:
In section 3.1. HEALPix Spherical projection, does that anchor-based projection guarantee to preserve shape-independence? Even if the spherical projection is already proposed in VI-Net (Lin et al., 2023), ensuring its effectiveness in preserving shape-independence is critical in this framework.

6. Correspondence-Based Loss in Linear Space:
In lines 357-358, the paper claims that learning correspondence-based loss is easier in linear space. Why is this the easier case? Would direct regression provide a better supervisory signal? Providing supporting evidence would strengthen this claim.

7. HEALPix vs. Other Grids: It is convinced that the choice of the HEALPix grid over random rectangular SO(3) grids is explained by the drawbacks of equirectangular grids near the poles. 
However, could a random SO(3) grid serve as a viable alternative? 
Additionally, in Table 7, evaluating other SO(3) spherical grids, such as SuperFibonacci spirals [A], would provide a comprehensive comparison, given its fast construction properties.

[A] Super-Fibonacci Spirals: Fast, Low-Discrepancy Sampling of SO(3) (Marc Alexa, CVPR 2022)

8. Clarification of this paper’s contribution: Please organize Sections 3 and 4 to focus on the main contribution of this paper.

questions:
1. Rotational Invariance/Equivariance in HEALPix Projection:
How does the HEALPix spherical projection in Section 3.1 maintain rotational invariance or equivariance between input and output?

2. Related Work and Citations: There are two concurrent works leveraging spherical grids for pose estimation—correspondence-based [B] and regression-based [C]. Please cite and discuss these works to position your method within the broader context.

[B] Improving Semantic Correspondence with Viewpoint-Guided Spherical Maps (Mariotti et al., CVPR 2024 ): https://arxiv.org/abs/2312.13216

[C] 3D Equivariant Pose Regression via Direct Wigner-D Harmonics Prediction (Lee et al., NeurIPS 2024 ): https://arxiv.org/abs/2411.00543
</content>
</thread_object>

</thread>
<thread>
creation date:
1730737584252

thread_type:
official_review

<thread_object>
creator: reviewer_lylo

<content>
summary:
The paper addresses the challenge of object pose estimation in 3D space, specifically overcoming the limitations of existing methods that rely heavily on 3D model shapes. It proposes an approach called SpherePose, which utilizes spherical representations to create shape-independent transformations, thereby improving correspondence prediction accuracy. The method incorporates three core innovations: SO(3)-invariant feature extraction, spherical feature interaction using attention mechanisms, and a hyperbolic correspondence loss function for precise supervision. This paper mainly introduces a new proxy shape for objects and a robust architecture for category-level pose estimation. Empirical validation shows superior performance against state-of-the-art methods.

strengths:
- This paper uses a sphere as a proxy to implement category-level object pose estimation. It transforms the 3D shape to a uniform sphere via HEALPix spherical representations, which leads the network to focus on the semantic consistency between different objects instead of shape deviation.
- The architecture for category-level object pose estimation that achieves precise different objects in one category correspondence prediction.
- The experiments show that the proposed methods achieve the SOTA results on 6D pose estimation tasks in several datasets.

weaknesses:
- The point cloud patches are occluded. The center of a point cloud patch may not be the corresponding object's center. The projection results changed a lot when the center moved. Besides, different object point cloud patches have various occlusions. The correspondence by the spherical proxy, whether a real semantic correspondence of objects, could have more evidence or qualitative results.
- The positional encoding of the anchors needs more explanation. The way to describe the anchor position is vague, spherical coordinates or something else. The position of an anchor could influence the rotation invariance, making the results sensitive to rotation.

questions:
- Given that the method relies on SO(3)-invariant feature extraction and RGB-based features, how robust is the approach to diverse lighting conditions and texture variations across different objects?
- Can the spherical feature interaction using attention mechanisms generalize well to objects with high self-occlusion or cluttered environments, and how is this tested?
- The hyperbolic correspondence loss function is designed to improve gradient behavior near zero. How does this compare quantitatively with traditional loss functions in terms of convergence speed and accuracy?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732467352122

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common Response to All Reviewers

comment:
We thank all reviewers for their thoughtful comments and valuable feedback. Here, we provide a common response to some of the recurring issues, and summarize the revisions made to the manuscript according to the reviewers' suggestions, along with the references cited during the response process. Reviewer-specific responses are left as direct comments.

------

**Summary of Changes**

* As suggested by **Reviewer p4nq**, we have refined the description about ColorPointNet++ and DINOv2 features in **Section 1** and **Section 3**.
* As suggested by **Reviewer rB6i**, we have reorganized **Table 4** in Section 4 to emphasize the importance of SO(3)-invariant point-wise features.
* As suggested by **Reviewer rB6i**, we have included comparison with Super-Fibonacci Spirals [6] grids in **Table 7** in Section 4.
* As suggested by **Reviewer rB6i** and **Reviewer p4nq**, we have included a detailed explanation of SO(3)-invariance/equivariance in the **Appendix A.1**.
* As suggested by **Reviewer p4nq**, we have included an analysis of the SO(3)-invariance of DINOv2 [7] features in **Figure 5** in the Appendix.
* As suggested by **Reviewer v4xt**, we have included results that exclude RGB or radius values in **Table 10** in the Appendix.
* As suggested by **Reviewer LyLo**, we have included comparison of convergence speed and accuracy for the hyperbolic correspondence loss function in **Figure 7** in the Appendix.
* As suggested by **Reviewer rB6i**, we have discussed the distinctions between our method and two concurrent works [13, 14] in the **Appendix A.4**.
* We have made minor adjustments to the structure of the Appendix for better readability.
* We have refined the alignment of tables to enhance clarity and visual appeal.

------

**References**

Here are references cited in all responses:

[1] Umeyama S. Least-squares estimation of transformation parameters between two point patterns[J]. IEEE Transactions on Pattern Analysis & Machine Intelligence, 1991, 13(04): 376-380.

[2] Lin J, Wei Z, Zhang Y, et al. Vi-net: Boosting category-level 6d object pose estimation via learning decoupled rotations on the spherical representations[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 14001-14011.

[3] Qi C R, Yi L, Su H, et al. Pointnet++: Deep hierarchical feature learning on point sets in a metric space[J]. Advances in neural information processing systems, 2017, 30.

[4] Gorski K M, Hivon E, Banday A J, et al. HEALPix: A framework for high-resolution discretization and fast analysis of data distributed on the sphere[J]. The Astrophysical Journal, 2005, 622(2): 759.

[5] Wang H, Sridhar S, Huang J, et al. Normalized object coordinate space for category-level 6d object pose and size estimation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019: 2642-2651.

[6] Alexa M. Super-fibonacci spirals: Fast, low-discrepancy sampling of so (3)[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 8291-8300.

[7] Oquab M, Darcet T, Moutakanni T, et al. DINOv2: Learning Robust Visual Features without Supervision[J]. Transactions on Machine Learning Research.

[8] Lin J, Wei Z, Ding C, et al. Category-level 6d object pose and size estimation using self-supervised deep prior deformation networks[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 19-34.

[9] Lin X, Yang W, Gao Y, et al. Instance-adaptive and geometric-aware keypoint learning for category-level 6d object pose estimation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 21040-21049.

[10] Lin J, Wei Z, Li Z, et al. Dualposenet: Category-level 6d object pose and size estimation using dual pose network with refined learning of pose consistency[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021: 3560-3569.

[11] Chen Y, Di Y, Zhai G, et al. Secondpose: Se (3)-consistent dual-stream feature fusion for category-level pose estimation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 9959-9969.

[12] Lin H, Liu Z, Cheang C, et al. Sar-net: Shape alignment and recovery network for category-level 6d object pose and size estimation[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 6707-6717.

[13] Mariotti O, Mac Aodha O, Bilen H. Improving semantic correspondence with viewpoint-guided spherical maps[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 19521-19530.

[14] Lee J, Cho M. 3D Equivariant Pose Regression via Direct Wigner-D Harmonics Prediction[C]//The Thirty-eighth Annual Conference on Neural Information Processing Systems.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper addresses the task of object pose estimation in 3D space, overcoming the limitations of existing methods that rely heavily on 3D model shapes. The proposed method uses a sphere as a shared proxy shape for objects, enabling the learning of shape-independent transformations from spherical representations. To enhance the precision of correspondences on the sphere, it incorporates three core components, including SO(3)-invariant point-wise feature extraction, spherical feature interaction, and a hyperbolic correspondence loss function. Empirical validation shows superior performance against state-of-the-art methods.
All reviewers appreciated the novelty of the proposed approach and its good empirical performance. The main concerns were unclear writings/details and missing experiments/analyses. The authors’ detailed rebuttal addressed most of them, resulting in unanimous acceptance at the end of the discussion. AC thus recommends acceptance.