Generative Category-Level Shape and Pose Estimation with Semantic Primitives
Authors: Guanglin Li
Yifeng Li
Zhichao Ye
Qihang Zhang
Tao Kong
Zhaopeng Cui
Guofeng Zhang
Venue: CoRL 2022 Poster (2022)
Keywords: Category-level Pose Estimation
Shape Estimation

Abstract: Empowering autonomous agents with 3D understanding for daily objects is a grand challenge in robotics applications. When exploring in an unknown environment, existing methods for object pose estimation are still not satisfactory due to the diversity of object shapes. In this paper, we propose a novel framework for category-level object shape and pose estimation from a single RGB-D image. To handle the intra-category variation, we adopt a semantic primitive representation that encodes diverse shapes into a unified latent space, which is the key to establish reliable correspondences between observed point clouds and estimated shapes. Then, by using a SIM(3)-invariant shape descriptor, we gracefully decouple the shape and pose of an object, thus supporting latent shape optimization of target objects in arbitrary poses. Extensive experiments show that the proposed method achieves SOTA pose estimation performance and better generalization in the real-world dataset. Code and video are available at \url{https://zju3dv.github.io/gCasp}.

Generative Category-Level Shape and Pose Estimation with Semantic Primitives
Guanglin Li Yifeng Li Zhichao Ye Qihang Zhang'
Tao Kong² Zhaopeng Cui Guofeng Zhang' State Key Lab of CAD&CG, Zhejiang University *ByteDance AI Lab
Multimedia Laboratory, The Chinese University of Hong Kong
Abstract: Empowering autonomous agents with 3D understanding for daily ob- jects is grand challenge in robotics applications. When exploring in an un- known environment, existing methods for object pose estimation are still not satisfactory due to the diversity of object shapes. In this paper, we propose a novel framework for category-level object shape and pose estimation from sin- gle RGB-D image. To handle the intra-category variation, we adopt semantic primitive representation that encodes diverse shapes into unified latent space, which is the key to establish reliable correspondences between observed point clouds and estimated shapes. Then, by using a SIM(3)-invariant shape descriptor, we gracefully decouple the shape and pose of an object, thus supporting latent target objects in arbitrary poses. Extensive experiments
shape optimization show that the proposed method achieves SOTA pose estimation performance and better generalization in the real-world dataset. Code and video are available at https://zjuddy .ittt.g............. io/gCasp.
Keywords: Category-level Pose Estimation, Shape Estimation
Introduction
Estimating the shape and pose for daily objects a fundamental function which has various applications, including 3D scene understanding, robot manipulation and autonomous warehous- ing [1, 2, 3, 4, 5, 6]. Early works for this task are focused on instance-level pose estima- tion [7, 8, 9, 10, 11], which aligns the observed object with the given CAD model. However, such setting is limited in real- cal-w scenarios since is hard to obtain the exact model of a casual object in advance. To generalize to these unseen but semantically familiar objects, category-lvvel pose esti- mation is raising more and more research attention [12, 13, 14, 15, 16, 17] since it could potentially handle various instances of the same category in real scenes.
Existing works on category- pose estimation usually try to predict pixel-wise normalized coor- dinates for instances within one class [12] or adopt a canonical prior model with shape deformations to estimate object poses [14, 15]. Although great advances have been made, these one-pass pre- diction methods are still faced with difficulties when large shape differences exist within the same category. To handle the variety of intra-class objects, some works [18, 16) leverage neural implicit representation [19] to fit the shape of the target object by iteratively optimizing the pose and shape in a latent space, and achieve better performance. However, in such methods, the pose and shape esti- mation are coupled together and rely each other to get reliable results. Thus, their performance is unsatisfactory real scene (see Sec. 4.2). So we can see that the huge intra-class shape differences and coupled estimation of shapes and poses two main challenges for the existing category- level pose estimation methods.
To tackle these challenges, we propose to estimate the object poses and shapes with semantic primi- rives from a generative perspective. The insight behind this is that the objects of a category are often composed of components with the same semantics although their shapes are various, e.g., a cup is usually composed of semicircular handle and a cylindrical body (see Fig. (a)).. This property
Work done during Guanglin Li's internship at ByteDance.
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Shared Latent Space
gf
Original Shape
g
Original Shape
23
Intra-Category Semantically Consistent Parts
Semantic Primitives
Semantic Primidives
(a)
(b)
Figure I: (a) An example of different instances in the same categories, we highlight the same seman- tic parts (e-g., lens of cameras) between different instances. (b) The detailed shapes (top row) decomposed into semantic primitives (bottom row). Different colors on primitives indicate different semantic labels, which model consisting of two auto-decoders (i.e., 9y and 9r) sharing the same latent space. 9j captures fine details, and Gc represents the shapes by simple and semantically consistent shape primitives.
consistent among different shapes. (c) An overview of the generative
inspired handle huge differences in intra-class shapes by fusing this category-level seman- tie information. Specifically, we propose to encode object shapes into semantic primitives, which construct a unified semantic representation among objects in the same category (see Fig. 1(b)). Fol- lowing Hao et al. [20], we map diverse shapes and the corresponding semantic primitives into the same shared latent space of generative model. In this way, can obtain the correspondence be- tween the observed point cloud and the latent space by dividing the observed point cloud into several semantic primitives with part segmentation network.
In order to disentangle the shape and pose estimation, we further extract a novel SIM(33-invariann descriptor from the geometric distribution of the point cloud's s semantic primitives. By leveraging such (ii())-nnaaraat descriptors, we can perform latent optimization poses and obtain the resulting shape consistent with the observation. After obtaining the estimated shape, we use a correspondence-basee algorithm to recover the size and pose by aligning semantic primitives of the target object with that of the estimated shape.
object shapes in arbitrary
In summary, our contributions are as follows: 1) We propose a novel category-level pose estima- tion method, which point clouds with implicitly generated shapes. 2) We propose an online shape estimation method, which enables object shape optimization in arbitrary poses with novel
semantic primitives to model diverse shapes and bridges the observed
descrip- tor. 3) Even when training on the synthetic dataset only, our method shows good generalization and achieves rate-of-the-arr category-leve pose estimation performance the real dataset.
2 Related Works
Instance-Level Object Pose Estimation. Instance-level pose estimation assumes the CAD model of test objects is known in the training phase, and previous methods are mainly divided into three categories. The first category of methods [21, 8, 7, 22] directly regress the 6D object pose from RGB or RGB-D images. PoseCNN [7] extends 2D detection architecture to regress 6D object pose from RGB images, and DenseFusion [8] combines color and depth information from the RGB-D input and makes the regressed pose more accurate. The second are correspondence- based methods [9, 10, 11, 23]. They predict the correspondences between 2D images and 3D models, and then recover the pose by Perspective-n-
algorithm [24]. For example, PVNet [9] predicts 3D keypoints the RGB image by voting scheme. CDPN [11] predicts the dense correspondences between image pixels and 3D object surface. The third category is rendering-baas which recover the pose by minimizing the re-projection error between the posed 3D model and 2D image through a differentiable render. Compared with these instance-level methods, our category- level method does not need the known CAD model prior.
methods [25, 26),
Levegory-Level Object Pose Estimation. Category-level methods [12, 27, 13] aim at estimating the pose for the arbitrary shape of the same category. To overcome the intra-category variations, NOCS [12] introduces normalized object canonical space, which establishes a unified coordinate space among instances of the same category. CASS [27] learns variational auto-encoder to recover object shape and predicts the pose an end-to-end neural network. These methods directly regress the pose coordinates, and thus struggle to form intra-category representation. Other methods

---

Semantic Primitives Extraction
(b) Generative Shape Optimization
Pa
KC
R,C3
f
g
badepmiction
x
8,(2)
Observed Point Cloud
Semantic Primsitives
Shape Deseriptor
Lafent Space
Generative Decoder
R.CNN
8AEE)
RAA0
Scale Rotatiso Transhation
Retativn Transtation
Pose Size Fatimation
(d) Object Reconstruction
Figure 2: Overview the proposed method. The input of our method is the point cloud observed from single-view depth image. (a) Our method first extracts semantic primitives C from object point cloud Po by part segmentation network ø. (b) We calculate a SIM(3)-invariant shape descrip- for f(C) from and optimize a shape embedding å in the latent space by the difference between f(C) and (=)(=)), where Je and 9f are the coarse and fine branches of the generative model g, detailed in Sec. 3.1. (c) The similarity transformation {5, R, Ł} is recovered through the semantic correspondences between Ć and optimized shape ge(2). (d) Further, we simply apply the trans- formation {3, R, E} on the fine geometry generated through gr (2), and reconstruct the object-level scene as by-product besides our main results, which is detailed in the Sec. B in the supp. material.
utilize a prior model of each category for the category-level representation. For example, SPD [14] and SGPA [15] predict deformation canonical prior models and recover the 6D pose through correspondence between the deformed model and the observed point cloud. However, when facing significant difference between the prior model and the target object, the deformation prediction tend to fail. DualPose [28] takes the consistency between an implicit pose decoder and an explicit one into account and refines the predicted pose during testing. Besides, FS-Net [13] proposes novel data augmentation mechanism to overcome intra-category variations. SAR-Net [17] recovers the shape by modeling symmetric correspondence of objects, which makes pose and size prediction easier. To utilize the diverse and continuous shapes in generative models, iCaps[16] and ELLIPSDF(IS) introduce the generative models [19, 20] to their shape and pose estimation. Shapes and poses are jointly optimized these works. Unlike these previous works, we establish a semantically consistent representation and decouple the shape and pose estimation, which makes the optimization more robust.
3 Method
Problem Formulation. Given observed point cloud Po (pili 1....,,No No) of an object with known category, our goal is to estimate the complete shape, scale and 6D pose for this object. The recovered shape is represented by a shape embedding & in a latent space. The estimated scale and 6D pose are represented as similarity transformation {s, R, t SIM(3), where scale R, 3D rotation R E SO(3) and 3D translation & E R3. SIM(3) and SO(3) indicate the Lie group of 3D similarity transformations and 3D rotation, respectively.
Overview. As illustrated in Fig 2, our method takes the RGB-D image as input. A Mask-RCNN[29] network is used to obtain the mask and category of each object instance. In the following stages, our method only processes the observed instance point cloud back-projected from the masked depth image. To overcome the huge intra- entra-category variations, we utilize a unified intra-category rep- resentation of semantic primitives (Sec 3.1). Then, we process the observed point cloud with part segmentation network to extract semantic primitives which establish a connection between the observed object and the latent space (Sec 3.2). To estimate the shape the target with its pose unknown, we design SIM(3)-invariant descriptor to optimize the generative shape in the latent space (Sec 3.3). Finally, we estimate the 6D pose and seale of object through the semantic correspondence between the observed point cloud and our recovered generative shape (Sec 3.4).

---

3.1 Semantic Primitive Representation
The huge intra-category differences make it challenging to represent objects with a standard model. To overcome this problem, we utilize semantic primitive representation. The insight behind the semantic primitive representation that although different instances the same category have var- ious shapes, they tend to have similar semantic parts, e.g., each camera instance has a lens, and each laptop has four corners (see Fig ((a)). In semantic primitive representation, each instance is decom- posed into several parts, and each part always corresponds to the same semantic part in different instances (see Fig ((b).. To generate this representation, we learn a generative model g following the network structure in [20] and map instances of the same category into latent space.
Specifically, Fig I(c) gives an overview of the generative model. The generative model g ex- presses shapes at two granular levels, one capturing detailed shapes and the other representing an abstracted shape using semantically consistent shape primitives. We note the fine-scale model as 9f and the coarse-scale one as Ic- 9f and Ic are two auto-decoders sharing the same latent code z. gr (z, x) = SDF(w), where x is a 3D position in canonical space and SDF denotes the signed distance function. ge(z)= = {au,/i
1......V. where Ne is the number of primitives and a is the parameters of primitive. We use sphere primitives here and a = (c, r). where c is the 3D center of sphere and is the radius. Please refer to [20] and Sec. A.3 in the supp. material for more details.
3.2 Extract Semantic Primitives from Point Cloud
We use part segmentation network to obtain the semantic primitives of the observation point cloud in two stages. First, we predict a semantic label 1, for each point p; in the observed point cloud Po- and then a centralization operation is performed on the points to extract the center of each primitive.
Semantic Label Prediction. For each point p; E Po. its semantic label 4 defined in terms of closest primitive center cy:
4 argmin| p: ccyl2.
We treat point label prediction as a classification problem and follow the part segmentation version of 3D-GCN [30] to perform point-wise classification. Standard cross entropy is used as the loss function. In addition, because of imperfect segmentation masks produced by Mask R-CNN, the point cloud vack-projected from the masked depth image may contain extra background points. For these outlier points, we add another "dust" label in the point-wise classification to filter out them.
Symmetric Loss. Many common objects in robot manipulation have symmetrical structures, and most objects (e.g., bottles) are symmetrical about an axis. The unique pre-labeled ground truth cause ambiguity when the object rotates around the axis. In order to eliminate the influence of symmetric structure on our classification task, we introduce the symmetric loss as the same as [12]. For each symmetric object in the dataset, we define symmetric axis and make the loss identical when the object rotates around the axis. Specifically, we generate a separate ground truth for each possible angle of rotational symmetry. Given the rotation angle 0, the rotated semantic label 4e is defined as:
lo. argmin P. co, j=
where Co.j is the position of ej after degrees of rotation about the axis. Then we define the symmetric loss Layma
Laym
min L(T,, dee
(3)
where is the standard cross entropy, 1, is the predicted label by the network, and we set = { for non-symmetric objects and θ = (i- 60% 5) for symmetric objects in our experiments.
Primitive centralization. After predicting the semantic label of each 3D point, we count the se- mantic labels č = {4]i I,... Noj which appear in the partially observed point cloud. Then, we calculate the primitive centers C {a|l = 41...., xx. } from the labeled points by averaging the points with the same label. Specifically,
No
C = Nv
Σ
P.I( = 1).
(4)
4

---

02
SIME3)
Transkrmaiinn
Figure 3: The proposed atomic SIM(3) deseriptor. The angle 01 and 02. which we called atomic descriptors are invariant with different object poses. And our shape descriptor concatenated by multiple atomic descriptors also invariant.
where N₁ is the total number of points with predicted label 1, is an indication function indicating whether the predicted label I, equals l.
3.3 Shape Optimization with SIM(3)-Invarian Descriptor
We decouple the shape estimation from the pose estimation to perform shape estimation for the ob- served object when the pose is unknown. In order to achieve this pose-invarian invariant shape optimization, propose novel
invariant descriptor extracted from semantic primitive representation.
SIM(3)-invariant Descriptor. Given a set of 3D points C {cgli 1,... N. and a SIM(3) transformation {s, R, t). SIM(3) -invaaann descriptor fo can be expressed as:
fo(C) {R + t).
(5)
To make our descriptor -invaainnaari first, we obtain a caanllatonninvariinn descriptor by calcu- lating the vectors v composed of every two points:
Vij (c, + t) - (c, + t) = Cj. After that, normalize the vector eliminate the scale factor
(6)
Bij
Uij s l ||:s
Finally, we eliminate the effect of rotation transformation by calculating the dot product between two vectors (see Fig. 3):
((c,, Ck, c.) Roy Ror - = R" Ru = .. Uxo:
where denotes the dot product of two vectors, and denotes a function which takes 4 points as input and a SIM(3)-nnvarian scalar output (as the angles shown in Fig. 3)
We call 0 the atomic descriptor. However, only a single atomic descriptor calculated from points (primitives centers) is not enough to describe an object's shape. Thus, we design the SIM(3)- invariant shape descriptor through function f, which concatenates multiple atomic descriptors to form a SIM(3)-invarian shape descriptor. With our estimated primitives Ć and their semantic labels Ê. the set E is denoted as:
[ o) i,j,k, o E ł j,k # o}.
and the (IM(3)-invarian) shape descriptor f is denoted as: f(C, [) (o).
((c, Cj. Ck co),
(10)
where @ denotes the concatenate operation which concatenates multiple scalars into vector in order, and Lu is subset randomly chosen from _ for each optimization. We choose the subset of _4 because |E$1 would be large, which is computational expensive. In Sec. 4.3, an ablation is conducted to analyze the effect of [ which indicates the number of atomic descriptors shape descriptor.
Online Shape Optimization. Given the shape embedding in the latent space and estimated prim- itives Č with semantic labels É, we can measure the shape difference between the generative shape (2) and the observation Po through the SIM(3)-invariant descriptor. We denote this difference as
== ==I1(CČ, ((C(((((())) D)12 + +
(11)
where (() = (é/li 1,... N.) is the primitive centers of g2). Obviously, f is differen- tiable function and ga is a differentiable neural network. We optimize the shape embedding via gradient descent in an online iterative way and set 77 0.0001 to weight the regularization term.
5

---

Table Comparison our method with other five SOTA methods [12, 14, 15, 17, 16] on REAL275 and CAMERA25. 'S' is synthetic data and 'R' is real data.
Training Data
Canonical
Learnable
Dataset
Methods
IoU50 IoU75 5'2mm 5*5cm 10m 10" 5om Parameters(M)
Model
NOCS [12)
RGB(S+R)
Regress
78.0
30.1
7.2
10.0
13.8
25.2
SPD [14]
RGBD(S+R)
Deform
77.3
53.2
19.3
21.4
43.2
54.1
18.3
REAL275
SGPA [15]
RGBD(S+R)
Deform
80.1
61.9
35.9
39.6
61.3
70.7
23.3
(real)
SAR-Net
D(S)
Deform
79.3
62.4
31.6
42.3
50.3
68.3
6.3
iCaps [16]
D(S)
Generate
22.3
80.9
Ours
D(S)
Generate 79.0
65.3
46.9
54.7
64.2
76.3
2.3
NOCS [12] RGB(S+R)
Regress
83.9
69.5
32.3
40.9
48.2
64.6
CAMERA25
SGPA [15] RGBD(S+R) Deform
93.2
74.5
82.7
88.4
23.3
(syathetic)
SAR-Net [17]
D(S)
Deform
86.8
79.0
70.9
75.3
80.3
Ours
D(S)
Generate
95.7
77.0
2.3
SGPA
ASEN
V
Ours
Figure 4: Qualitative comparisons between our method and SGPA [15] on REAL275 dataset [12]. The red arrows and dotted box point out the inaccurate results of SGPA compared with our method.
3.4 6D Pose and Size Estimation
With the primitive centers (()((2) the optimized shape and the observed point cloud, we use Umeyama algorithm [31] to compute the SIM(3) transformation by minimizing following error function:
No
s, R. é argmin s.R.t
lp,, sR(,, + +
(12)
No
1=1
4 Experiments
4.1 Datasets and Evaluation Metrics
We evaluate our method on NOCS [12] dataset which consists of six objects categories including bottle, bowl, camera,
laptop and mug. The NOCS dataset contains virtual dataset called CAMERA25 and a real-world dataset called REAL275. The CAMERA25 dataset contains 300K synthetic images and there 25K images for evaluation among them. The REAL275 contains 4.3K real-scene images for training and 2.75K real-scene images for evaluation.
Two aspects of metrics are adopted for evaluation: 3D IoU and pose error. 3D IoU measures the overlap of two posed 3D bounding boxes, and we compute the average precision threshold values of 50% and 75%. The pose error measures the rotation and translation errors between the predicted pose and ground truth pose, and we compute the average precision at threshold values (5", 2cm), (5%, 5cm), (10", 2cm) and (10" 5em).
4.2 Comparison with State-of-the-Art Methods
We compared our proposed method with several SOTA methods [12, 14, 15, 17. 16). NOCS [12] directly regress pixel-level canonical coordinate. SPD [14] and SGPA [15] are the methods which recover canonical shape by deforming a prior shape. iCaps [16] and our methods utilize gener- ative model to refine and recover the canonical model respectively. Table 1 shows the quantitative results. Overall, our method needs the minimum parameters learned for the shape and pose esti- mation task and shows the results of the smallest domain gap between the synthetic dataset and

---

Pose estimation results with recovered shape by mean, random and optimized shape em- bedding on REAL2275.
Table 2:
Table 3: Results of our method with different
numbers of primitives REAL2275
Number IoU50 IoU75 2 5 555m 2cm 10 5em
Embedding IoU50 IoU75 2cm 59 5cm 2cm 10 5cm
64
79.2
63.7
41.2
51.4
58.9
74.9
Mean
77.7
54.5
31.7
36.0
51.4
61.0
128
78.9
64.5
43.5
51.2
61.2
74.6
Random
76.8
51.6
28.7
34.4
47.6
60.3
256
79.0
65.3
46.9
54.7
64.2
76.3
Optimized
79.0
65.3
46.9
54.7
64.2
76.3
512
77.2
63.3
45.7
53.6
61.7
74.4
100
Rotation
100
Rotation
Translation
Awersge Precision
80
80
60
fveeaag
40
40
20
20
25
20
25
50
75
100
20
60
10
Degres
Centimeter
Percent
Centimeter
Figure 5: Curves of each category under different thresholds on IoU, rotation and translation on REAL275 (left) and CAMERA (right) dataset.
the real-world dataset. In detail, our method outperforms other existing methods on all metrics but loU50 on the weal-world REAL275 dataset. Fig. shows qualitative comparisons with SGPA [15]. On the synthetic CAMERA25 dataset, our method also shows improvement on most metrics com- pared with other methods, except for a small gap with SGPA [15] on two metrics, (10", 2cm) and 5cm)). Besides, Fig. 5 shows the average precision curves of each category under different thresholds on IoU, rotation and translation on REAL275 and CAMERA25 dataset. Compared the other generative method iCaps [16), our method shows far better results, indicating the advantage of our decoupled shape and pose estimation. It should be noted here that our trained generative model (See. 3.1) consists of about 4.3M parameters for each category. The generative models are only pre-trained on ShapeNetCore [32] and need no more fine-tuning for the specific shape and pose estimation task. Thus, we compare the learnable parameters with other methods. Specifically, method needs only additionally train a part segmentation network (2.3M) for all categories on the synthetic training data provided by CAMERA25 [12], which leads our better generalization in real scenes.
4.3 Ablation Study
To justify the design choices of our method, we explore the following issues through experiments:
Effectiveness of the Sin(3a-invariant descriptor. Does the descriptor- based shape opti- mization make the optimized shape more accurate and benefit the pose estimation?
Effect of different numbers of primitives. Semantic primitives the basis of our method. How many primitives are better for pose estimation?
Effect of different numbers of atomic descriptors [/tmp// Our shape descriptor concate- nated by multiple atomic descriptors. How many atomic descriptors are more effective for shape optimization?
Effectiveness of -invaa-invaraant descriptors. We verify the effectiveness of our SIM(3)- invariant descriptors in two aspects. First, we verify that descriptor- shape optimization im- proves the accuracy of shape reconstruction. We compare the reconstruction error of our optimized shape with the shape when the shape embedding z = O (denoted as "Mean" shape) and a random shape when the shape embedding z ~ NO 1) (denoted our optimized shapes are more accurate. In Table 4, we also compare our method with SGPA [15], even without shape loss to supervise our method, our coarse-level shape reconstruction shows com- parable results with SGPA. Second, we verify that descriptor-basee shape optimization improves the pose estimation results. We compare the results the pose estimation under the three canon- ical shapes (mean, random and optimized) in the Table 2. The results show that we get far better pose estimation results based on the optimized shape. Besides, Fig. visualizes cases where shapes become more consistent with observations optimization progresses.
"Random" shape). Table 4 shows that
7

---

Table 4: Comparison of shape reconstruction accuracy of methods and SGPA in CD metric (x10-3) on REAL275. Mean and Random refer to the shape when shape embedding
Table 5: Comparison of shape reconstruc- tion accuracy when different number of atomic primitives are randomly chosen to optimize the shape in CD (x (x10-) metric.
and (O, 1).
Methods
bottle bowl camera
can
laptop
mug
all
1
w/o. RANSAC
w. RANSAC
SGPA [15]
2.93
0.89
5.51
1.75
1.62
1.12
2.44
10'
6.79 + 6.50
4,38 + 5.41
Mean
9.55
4.39
12.4
3.03
6.59
2.26
6.05
102
4,86 + 5.38
3,97 + 5.15
Random
11.6
5.49
16.1
3.74
7.12
3.63
7.56
10
3.68
+ 4.34
3,59 + 4.54
Optimized
2.05
1.55
10.1
1.63
2.12
2.93
3,46
10c
3.46 1 4.05
3,44 + 4.13
Iterative Optimization Figure 6: Visualization of the shape optimization process. As optimization progresses, our opti- mized shapes more consist with observations.
Real Scene
Target Objects
Effect of different numbers of primitives. We conduct ablation study the number of mantic primitives. We experiment the number with 64, 128, 256, and 512. As shown Table 3, the results show that the accuracy of pose estimation is improved with the increase of the number of primitives, which indicates that more primitives can represent more complex geometry. The results show a decline when the number is set 512 because 512 primitives are relatively redundant for our 1024 input points, and more primitive classes also decline the accuracy of part segmentation network (Sec. 3.2). Based on these observations, we use 256 primitives the default setting in other experiments.
Effect of different numbers of atomic descriptors. As illustrated in Sec. 3.3, single atomic descriptor is insufficient to describe an object's shape, and concatenating all atomic descriptors to form shape descriptor would be computationally expensive and unnecessary. Thus, conduct experiments and analyze the influence of the number atomic descriptors on the shape optimization results. Specifically, we randomly choose {10*, 102, 10 10 atomic descriptors form the shape descriptor and use the chamfer distance to evaluate the quality of the optimized shape. As shown in Table. 5, the reconstruction eгror of the optimized shape decreases the number of atomic descrip- tors increases, and applying RANSAC in the random selection can further reduce this error. In the main experiments, we choose 10 atomic descriptors depending on the computational resources and do not use RANSAC for efficiency.
5 Conclusion and Limitations
We propose novel category-level shape and pose estimation framework by utilizing the semantic primitive representation. Leveraging novel SIM(3)-invariant descriptor, decouple shape and pose estimation and optimize the implicitly generated shape in an online iterative way. Experiments results demonstrate the advantages of our method on the real-scene data compared with other SOTA methods.
Limitations. As other methods [14, 15, 17], our method also relies on the pre-processed masks of object instances by Mask R-CNN and extra annotation of symmetry axes when training. Further- more, although we can reconstruct the object mesh (Sec. B in the supp. material), this by-product has no influence or contribution to our work. Future work could consider the mesh reconstruction to get better shape and pose estimation results. Besides, our method does not consider the occlusion and collision between objects, and may cause ambiguity from single-view observation. Feasible fu- ture work could fuse color information into our input, expand the proposed method to multi-view observation to address these problems.

---

Acknowledgments
We would like to thank Bangbang Yang for the useful discussion on this paper.
References
[1] T. Wang, Z. Xinge, J. Pang. and D. Lin. Probabilistic and geometric depth: Detecting objects in perspective. Conference on Robot Learning, pages 1475-1485. PMLR, 2022.
[2] W. Yuan, C. Paxton, K. Desingh, and D. Fox. Sornet: Spatial object-centric representations for
sequential manipulation. In Conference Robot Learning, pages 148-157. PMLR, 2022.
[3] Chen, J. Xu, and P. Agrawal. A system for general in-hand object re-orientation. In Confer- ence on Robot Learning, pages 297-307. PMLR, 2022.
[4] S. Cheng, K. Mo, and L.. Shao. Learning regrasp by learning to place. In Conference on Robot Learning, pages 277-286. PMLR, 2022.
[5] T. Chen, J. Xu, and P. Agrawal. A system for general in-hand object re-orientation. In Confer- ence on Robot Learning, pages 297-307. PMLR, 2022.
[6] E. Sucar, K. Wada, and A. Davison. NodeSLAM: Neural object descriptors for multi-view shape reconstruction. In 2020 International Conference on 3D Vision (3DV), pages 949-958. IEEE, 2020.
[7] Y. Xiang, T. Schmidt, V. Narayanan, and D. Fox. Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes. arXiv preprint urXiv:1711.00190 2017.
[8] C. Wang, D. Xu, Y. Zhu, R. Martín-Martin, C. Lu, L. Fei-Fei, and S. Savarese. Densefusion: 6d object pose estimation by iterative dense fusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3343-3352, 2019.
[9] S. Peng. Y. Liu, Q. Huang, X. Zhou, and H. Bao. Pynet: Pixel-wise voting network for 6dof pose estimation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4561-4570, 2019.
[10] Y. He, W. Sun, H. Huang, Liu, H. Fan, and J. Sun. Pvn3d: deep point-wise 3d keypoints voting network for 6dof pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11632-11641, 2020.
[11] Z. Li, G. Wang, and X. Ji. Cdpn: Coordinates based disentangled pose network for real- time rgb-based 6-dof object pose estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7678-7687, 2019.
[12] H. Wang, S. Sridhar, J. Huang, J. Valentin, S. Song, and L. J. Guibas. Normalized object coordinate space for category-level 6d object pose and size estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2642-26 222-2651,55 2019.
[13] W. Chen, X. Jia, H. J. Chang. J. Duan, L. Shen, and A. Leonardis. Fs-net: Fast shape- based network for category-level 6d object pose estimation with decoupled rotation mechanism. In Proceedings the LEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1581-1590, 2021.
[14] M. Tian, M. H. Ang, and G. H. Lee. Shape prior deformation for categorical 6d object pose and size estimation. In European Conference on Computer Vision, pages 530-546. Springer, 2020.
[15] K. Chen and Q. Dou. Sgpa: Structure-guided prior adaptation for category-level 6d object pose estimation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 2773-2782, 2021.
[16] X. Deng, J. Geng, T. Bretl, Y. Xiang, and D. Fox. iCaps: Iterative category-level object pose and shape estimation. IEEE Roboties and Automation Letters, 7(2):1884-1791 2022.

---

[17] H. Lin, Z. Liu, C. Cheang, Y. Fu, G. Guo, and X. Xue. SAR-Net: Shape alignment and recovery network for category-level 6d object pose and size estimation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pottern Recognition, pages 6707-6717, 2022.
[18] M. Shan, Q. Feng, Y.-Y. Jau, and N. Atanasov. ELLIPSDF:
Joint object pose and shape opti- mization with a bi-level ellipsoid and signed distance function description. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9946-5955, 2021.
[19] J. J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove. DeepSDF: Learning con- tinuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 165-174, 2019.
[20] Z. Hao, H. Averbuch-Elor, N. Snavely, and Belongie. Dualsdf: Semantic shape manipulation using two-level representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7631-7641, 2020.
[21] M. Schwarz, H. Schulz, and S. Behnke. RGB-D object recognition and pose estimation based on pre-trained convolutional neural network features. In 2015 IEEE International Conference on Robotics and Automation (ICRA), pages 1329-1335. IEEE, 2015.
[22] W. Kehl, F. Manhardt, F. Tombari, S. Ilie, and N. Navab. Ssd-6d: Making rgb-based 3d detec- tion and 6d pose estimation great again. In Proceedings of the IEEE International Conference on Computer Vision, pages 1521-1529, 2017.
[23] M. Oberweger, M. Rad, and V. Lepetit. Making deep heatmaps robust partial occlusions for 3d object pose estimation. In Proceedings of the European Conference on Computer Vision (ECCV), pages 119-134, 2018.
[24] V. Lepetit, F. MMoreno-Noguer and P. Fua. Epnp: An accurate (n) solution to the pnp problem. International Journal of Computer Vision, 81(2):155- 166, 2009.
[25] K. Park, A. Mousavian, Y. Xiang, and D. Fox. Latentfusion: End-to-end differentiable recon- struction and rendering for unseen object pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 10710-10719, 2020.
[26] L. Yen-Chen, P. Florence, J. T. Barron, A. Rodriguez, P. Isola, and T.-Y. Lin. inerf: Inverting neural radiance fields for pose estimation. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1323-1330. IEEE, 2021.
[27] D. Chen, J. Li, Z. Wang, and K. Xu. Learning canonical shape space for category-level 6d object pose and size estimation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 1 11973-11982, 2020.
[28] J. Lin, Z. Wei, Z. Li, S. Xu, K. Jia, and Y. Li. Dualposenet: Category-level 6d object pose and size estimation using dual pose network with refined learning of pose consistency. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 3560- 3569, 2021.
[29] K. He, G. Gkioxari, Dollár, and R. Girshick. Mask r-enn. In Proceedings of the IEEE International Conference Computer Vision, pages 2961-2969 2017.
[30] Z.-H. Lin, S.-Y. Huang, and Y.-C. F. Wang. Convolution in the cloud: Learning deformable kernels in 3d graph convolution networks for point cloud analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1800-1809, 2020.
[31] S. Umeyama. Least-squares estimation of transformation parameters between two point pat- terns. IEEE Transactions on Pattern Analysis & Machine Intelligence, 13((4))376-880, 1991.
[32] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang. Z. Li. S. Savarese. M. Savva, S. Song, H. Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012 2015.
[33] C. Wang, R. Martín-Martin, D. Xu, J. Lv, C. Lu, L. Fei-Fei, S. Savarese, and Y. Zhu. 6- pack: Category-level 6d pose tracker with anchor-based keypoints. 2020 IEEE International Conference Robotics and Automation (ICRA), pages 10059-10066. IEEE, 2020.
10

---

[34] W. E. Lorensen and H. E. Cline. Marching cubes: A high resolution 3d surface construction algorithm. ACM Siggraph Computer Graphics, 21(4):163-163 1987.
[35] X. Zhou, R. Girdhar, A. Joulin, P. Krähenbühl, and I. Misra. Detecting twenty-thousand classes using image-level supervision. arXiv preprint urXiv:2201.02600 2022.
[36] S. Chitta, I. Sucan, and S. Cousins. Moveit![ros topies). IEEE Robotics & Automation Maga- zine, 19(1):18-11) 19, 2012.


Reviews:
[{"review_id": "yHXcJIhZazH", "reviewer": "Reviewer_jxdo", "summary": "This work proposes an approach to monocular (RGB-D) category-level shape and pose estimation that decouples the two predictions through the use of a similarity-transformation-invariant descriptor. The approach uses the output of a pretrained segmentation network and operates on the resulting segmented point cloud.\n\nThe approach has two parts, first, the object shape is estimated, generatively via pretraining on ShapeNet, as both a fine-grained SDF and a course set of primitives (spheres). Then, correspondence between the generated spheres and the observed pointcloud are used to estimate scale, rotation, and translation between the predicted shape and the the robot/world/sensor.\n\nThe main novelty in this paper is the use of a descriptor that is invariant to similarity transformations as part of their shape-estimation process and the use of a course-grained shape approximation for correspondence prediction.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The SIM(3)-invariant descriptor is clever and seems well suited to this application.\n- The paper provides a reasonably thorough summary of related work and the state of the field\n- The authors show good performance in their experimental evaluation and well-motivate most of their design choices\n\nWeaknesses:\n- Limited ablations are performed making it difficult to determine what the takeaway from this work should be. Is the critical insight\nthat correspondence matching between coarse shapes proves more effective? Is the key contribution their descriptor?\n- The experimental results section is quite limited\n    - The qualitative results are particularly lacking and the paper would be much stronger if this were improved and more examples were added.\n    - The quantitative metrics used for shape are limited (IoU). Chamfer distance seems quite appropriate here (and the authors do use it when ablating number of primitives).\n    - Pose error is a bit hard to interpret as is. This would benefit significantly from also providing histograms of pose errors for each approach as well as mean and percentile metrics (e.g. 50 percentile - median error, 75th percentile error, 90th percentile error, ...)\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I advocate for acceptance of this work due to the interesting and novel shape descriptor and I think publication would be a benefit to the community. That being said, this work would be dramatically strengthened with an improved experimental section, particularly more qualitative results.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "- The authors note that too many primitives reduced performance due to 'difficulty with the classification task'. Why is this the case? Naively, it seems that because each primitive constitutes its own classification decision that more primitives would not necessarily be expected to hurt performance.\n\n- Some of the prose in the document isn't quite grammatically correct. I recommend doing a further editing pass for such things."}}, {"review_id": "a4wsNEBqn7a", "reviewer": "Reviewer_imVj", "summary": "This paper proposes a method for joint shape and pose estimation for categories of objects. The method disentangles the two estimation tasks by a proposed SIM(3)-invariant shape descriptor, combined with using DualSDF (Hao et al., 2020), for shape estimation. On the NOCS dataset, the paper demonstrates the usefulness of the disentangled estimation approach, showing improved performance compared to state-of-the-art category-level pose estimation methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\nThe main innovation of the paper is the SIM(3)-invariant shape descriptor used in conjunction with combined shape estimation and pose estimation. The methods are clearly described. The writing in the paper is clear, and it is overall a pleasurable read. The results on the REAL275 subset of the NOCS dataset are encouraging, especially as much fewer parameters are needed than competing methods to reach competitive results.\n\n\n## Weaknesses\n\n- The evaluation scores chosen ignore object symmetry. Reported scores may look arbitrarily better or worse depending on which symmetry mode the methods predict. The paper should provide data for other metrics such as ADD-S (see, e.g., PoseCNN) that handle symmetry. This would also be a more realistic evaluation for applications where rotational symmetry can be ignored. *Updated after author response*: the authors have clarified the evaluation does consider symmetry.\n\n- The pose and shape estimation aspects are coupled in the evaluation. It is not clear, how much the non-perfect models influence pose estimation performance. Also open is the question of how well the proposed decoupling of pose and shape estimation really works, or whether the networks simply co-adapted to work around each others' deficiencies. Perhaps an ablation by evaluating on instance-level 6D pose estimation, where the ground truth models are available, could be helpful. *Updated after author response*: the new ablation study addresses this concern.\n\n- The training process in the proposed method relies on the ShapeNet CAD models for training the generative models. How could these findings be translated to the case where no CAD models are available? In which domains could one expect this to be critical?  To be clear, I think this limitation is shared by many of the category-level pose estimation works, and not specific to this work. However, some discussion would be welcome as this is also not mentioned in the limitations. *Updated after author response*: in their response, the authors provided comments on this.\n\n\n## Questions\n- How does the representation in Section 3.1 differ from DualSDF? If DualSDF representation is directly adopted, as is the case as far as I can tell, it would be better to remove language such as \"we propose\" that imply novelty. Otherwise, a clear statement indicating the difference to Hao et al. is needed.\n\n- More details and clarification on the SIM(3)-invariant descriptor and in particular Eq. (9) are needed.\n* Is the \"concatenate\" operator taken over all unique 4-tuples of semantic labels?\n* It seems that the descriptor \\theta is invariant to the labels \\tilde{\\mathcal{L}} - why is this the case? Why do we need the label set as an argument of f?\n\n- Sect 3.5: \"We sample 1024 points from...\" - do you sample the points uniformly at random, or by another method? I suggest to try in the future farthest point sampling to guarantee uniform coverage of the input.\n\n\n## Minor\n- When referencing the supplementary material in the main paper, it would be helpful to always indicate a specific section in the supplement.\n- \\tilde{l}_i in Eq. (3) is not defined. Is this the label predicted by the network?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper combines implicit shape representations with a novel SIM(3)-invariant descriptor in an interesting way for category-level pose estimation, and shows some promising results. In their response, the authors provided an ablation study and addressed my other comments. I recommend accepting the paper.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Please address the points mentioned in \"weaknesses\" and \"questions\" above, in that order of priority.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "ECKJvgU5iVn", "reviewer": "Reviewer_4hYs", "summary": "This paper proposes a category-level shape and pose estimation with semantic primitives. The semantic primitives are learned to represent general shape information of objects within the same category. Then SIM(3) invariant descriptors are learned to optimize the shape of each object. The main contributions are as follow:\n\n1. A category-level pose and shape estimation framework that uses semantic primitives to encode object shapes.\n2. A SIM(3) invariant descriptors to optimize the 3D shape of the object. \n\nThe proposed method is evaluated on the NOCS dataset and shown to perform better on real data than the state-of-the-art methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n* The proposed method is able to reconstruct the shape and pose of the objects with only a RGB-D image. \n* Semantic primitives are adopted to allow the network to learn a general representation of objects within one category.\n* A SIM(3) invariant descriptor is designed to help the shape estimation process by decoupling the poes estimation task.\n* The experimental results show the proposed method performs better in the real data set than the state-of-the-art method.\n* In the supplementary materials, the proposed method is evaluated on unseen objects. This demonstrates the generalizability of the proposed methods.\n\n**Weaknesses** (Also see issues)\n* The symmetric loss appears to require extra annotating on the symmetric axis of each object. \n* I appreciate the ablation study section. However, no experiments are designed to show the effectiveness of the Semantic primitives and the SIM(3) invariant descriptors.\n* Some questions need to be clarified. (See issues)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper proposes a category-level shape and pose estimation framework. The semantic primitives and SIM(3) invariant descriptors are not new in robotics. However, it is interesting to see the combination of the two ideas for pose and shape estimation. The experimental results show the proposed method performs better on real datasets compared to the state-of-the-art methods. In addition, the generalizability of the method is demonstrated by evaluating a different dataset. However, there exist some issues that need to be addressed. Thus, I recommend a weak accept.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "* The IOU50 in Table 2 seems to decrease when the number of primitives increases. Any idea why this is the case?\n* It looks like the symmetric loss assumes the objects to have at most one symmetric axis. However, many objects have more than 1 axis of symmetry. (For example spheres.) I wonder how the proposed method deals with this issue?\n* How is RANSAC implemented to reduce the shape reconstruction error as mentioned in Section 4.2?\n* From my perspective, it is a little unfair to claim the learnable parameters to be only 2.3 M from the part segmentation network. Why is this the case? I would appreciate it if the authors could elaborate on this more. \n* Section 4.3, Shape Optimization for Pose Estimation is a little confusing. What is the difference between mean, random, and optimized methods?\n* The objects seem to have similar sizes in the dataset. Does the proposed method work well if the sizes of the objects differ drastically? (For example, a wine bottle v.s. a 12oz coke bottle.)\n* This might be nit-picking but I'm curious about how the proposed method works with category ambiguity. For example, a laptop can have two forms (open and close, and potentially any pose between the two.), but they all belong to the laptop category. Will the proposed method work well in this case?\n"}}]

Author rebuttal:

Decision: Accept (poster)