Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes a category-level shape and pose estimation with semantic primitives. The semantic primitives are learned to represent general shape information of objects within the same category. Then SIM(3) invariant descriptors are learned to optimize the shape of each object. The main contributions are as follow:

1. A category-level pose and shape estimation framework that uses semantic primitives to encode object shapes.
2. A SIM(3) invariant descriptors to optimize the 3D shape of the object. 

The proposed method is evaluated on the NOCS dataset and shown to perform better on real data than the state-of-the-art methods.
strengths_and_weaknesses: **Strengths**
* The proposed method is able to reconstruct the shape and pose of the objects with only a RGB-D image. 
* Semantic primitives are adopted to allow the network to learn a general representation of objects within one category.
* A SIM(3) invariant descriptor is designed to help the shape estimation process by decoupling the poes estimation task.
* The experimental results show the proposed method performs better in the real data set than the state-of-the-art method.
* In the supplementary materials, the proposed method is evaluated on unseen objects. This demonstrates the generalizability of the proposed methods.

**Weaknesses** (Also see issues)
* The symmetric loss appears to require extra annotating on the symmetric axis of each object. 
* I appreciate the ablation study section. However, no experiments are designed to show the effectiveness of the Semantic primitives and the SIM(3) invariant descriptors.
* Some questions need to be clarified. (See issues)
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: This paper proposes a category-level shape and pose estimation framework. The semantic primitives and SIM(3) invariant descriptors are not new in robotics. However, it is interesting to see the combination of the two ideas for pose and shape estimation. The experimental results show the proposed method performs better on real datasets compared to the state-of-the-art methods. In addition, the generalizability of the method is demonstrated by evaluating a different dataset. However, there exist some issues that need to be addressed. Thus, I recommend a weak accept.
issues: * The IOU50 in Table 2 seems to decrease when the number of primitives increases. Any idea why this is the case?
* It looks like the symmetric loss assumes the objects to have at most one symmetric axis. However, many objects have more than 1 axis of symmetry. (For example spheres.) I wonder how the proposed method deals with this issue?
* How is RANSAC implemented to reduce the shape reconstruction error as mentioned in Section 4.2?
* From my perspective, it is a little unfair to claim the learnable parameters to be only 2.3 M from the part segmentation network. Why is this the case? I would appreciate it if the authors could elaborate on this more. 
* Section 4.3, Shape Optimization for Pose Estimation is a little confusing. What is the difference between mean, random, and optimized methods?
* The objects seem to have similar sizes in the dataset. Does the proposed method work well if the sizes of the objects differ drastically? (For example, a wine bottle v.s. a 12oz coke bottle.)
* This might be nit-picking but I'm curious about how the proposed method works with category ambiguity. For example, a laptop can have two forms (open and close, and potentially any pose between the two.), but they all belong to the laptop category. Will the proposed method work well in this case?
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This work proposes an approach to monocular (RGB-D) category-level shape and pose estimation that decouples the two predictions through the use of a similarity-transformation-invariant descriptor. The approach uses the output of a pretrained segmentation network and operates on the resulting segmented point cloud.

The approach has two parts, first, the object shape is estimated, generatively via pretraining on ShapeNet, as both a fine-grained SDF and a course set of primitives (spheres). Then, correspondence between the generated spheres and the observed pointcloud are used to estimate scale, rotation, and translation between the predicted shape and the the robot/world/sensor.

The main novelty in this paper is the use of a descriptor that is invariant to similarity transformations as part of their shape-estimation process and the use of a course-grained shape approximation for correspondence prediction.
strengths_and_weaknesses: Strengths:
- The SIM(3)-invariant descriptor is clever and seems well suited to this application.
- The paper provides a reasonably thorough summary of related work and the state of the field
- The authors show good performance in their experimental evaluation and well-motivate most of their design choices

Weaknesses:
- Limited ablations are performed making it difficult to determine what the takeaway from this work should be. Is the critical insight
that correspondence matching between coarse shapes proves more effective? Is the key contribution their descriptor?
- The experimental results section is quite limited
    - The qualitative results are particularly lacking and the paper would be much stronger if this were improved and more examples were added.
    - The quantitative metrics used for shape are limited (IoU). Chamfer distance seems quite appropriate here (and the authors do use it when ablating number of primitives).
    - Pose error is a bit hard to interpret as is. This would benefit significantly from also providing histograms of pose errors for each approach as well as mean and percentile metrics (e.g. 50 percentile - median error, 75th percentile error, 90th percentile error, ...)

robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: I advocate for acceptance of this work due to the interesting and novel shape descriptor and I think publication would be a benefit to the community. That being said, this work would be dramatically strengthened with an improved experimental section, particularly more qualitative results.
issues: - The authors note that too many primitives reduced performance due to 'difficulty with the classification task'. Why is this the case? Naively, it seems that because each primitive constitutes its own classification decision that more primitives would not necessarily be expected to hurt performance.

- Some of the prose in the document isn't quite grammatically correct. I recommend doing a further editing pass for such things.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes a method for joint shape and pose estimation for categories of objects. The method disentangles the two estimation tasks by a proposed SIM(3)-invariant shape descriptor, combined with using DualSDF (Hao et al., 2020), for shape estimation. On the NOCS dataset, the paper demonstrates the usefulness of the disentangled estimation approach, showing improved performance compared to state-of-the-art category-level pose estimation methods.
strengths_and_weaknesses: ## Strengths
The main innovation of the paper is the SIM(3)-invariant shape descriptor used in conjunction with combined shape estimation and pose estimation. The methods are clearly described. The writing in the paper is clear, and it is overall a pleasurable read. The results on the REAL275 subset of the NOCS dataset are encouraging, especially as much fewer parameters are needed than competing methods to reach competitive results.


## Weaknesses

- The evaluation scores chosen ignore object symmetry. Reported scores may look arbitrarily better or worse depending on which symmetry mode the methods predict. The paper should provide data for other metrics such as ADD-S (see, e.g., PoseCNN) that handle symmetry. This would also be a more realistic evaluation for applications where rotational symmetry can be ignored. *Updated after author response*: the authors have clarified the evaluation does consider symmetry.

- The pose and shape estimation aspects are coupled in the evaluation. It is not clear, how much the non-perfect models influence pose estimation performance. Also open is the question of how well the proposed decoupling of pose and shape estimation really works, or whether the networks simply co-adapted to work around each others' deficiencies. Perhaps an ablation by evaluating on instance-level 6D pose estimation, where the ground truth models are available, could be helpful. *Updated after author response*: the new ablation study addresses this concern.

- The training process in the proposed method relies on the ShapeNet CAD models for training the generative models. How could these findings be translated to the case where no CAD models are available? In which domains could one expect this to be critical?  To be clear, I think this limitation is shared by many of the category-level pose estimation works, and not specific to this work. However, some discussion would be welcome as this is also not mentioned in the limitations. *Updated after author response*: in their response, the authors provided comments on this.


## Questions
- How does the representation in Section 3.1 differ from DualSDF? If DualSDF representation is directly adopted, as is the case as far as I can tell, it would be better to remove language such as "we propose" that imply novelty. Otherwise, a clear statement indicating the difference to Hao et al. is needed.

- More details and clarification on the SIM(3)-invariant descriptor and in particular Eq. (9) are needed.
* Is the "concatenate" operator taken over all unique 4-tuples of semantic labels?
* It seems that the descriptor \theta is invariant to the labels \tilde{\mathcal{L}} - why is this the case? Why do we need the label set as an argument of f?

- Sect 3.5: "We sample 1024 points from..." - do you sample the points uniformly at random, or by another method? I suggest to try in the future farthest point sampling to guarantee uniform coverage of the input.


## Minor
- When referencing the supplementary material in the main paper, it would be helpful to always indicate a specific section in the supplement.
- \tilde{l}_i in Eq. (3) is not defined. Is this the label predicted by the network?
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: This paper combines implicit shape representations with a novel SIM(3)-invariant descriptor in an interesting way for category-level pose estimation, and shows some promising results. In their response, the authors provided an ablation study and addressed my other comments. I recommend accepting the paper.
issues: Please address the points mentioned in "weaknesses" and "questions" above, in that order of priority.
review_title: null

Meta-review: metareview: The paper introduces a new method for category-level object shape and pose estimation. The novelty of the work is on introducing a semantic primitive representation with SIM(3)-invariant descriptors that can be used for object category shape and pose estimation. Experiments conducted on the NOCS dataset and the real world verify the efficiency of the proposed method.

There are several concerns about the experimental evaluation from the reviewers including more ablation studies. The shape reconstruction is usually evaluated by Chamfer distance. The method did not evaluate shape reconstruction separately from pose estimation.

The authors have successfully addressed the concerns from the reviewers during the rebuttal. The authors are encouraged to revise the final paper accordingly.
confidence: 5: The area chair is absolutely certain
Decision: Accept (Poster)