Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper study the problem of 6D pose estimation. It adopts a coarse-refine approach, where it first predicts a coarse pose hypothesis and use a refiner based render&compare to refine the pose. It can be applied to objects not seen in the training phase.
strengths_and_weaknesses: Strengths:
1. The idea of using classifier to acquire coarse pose is simple and robust and it can handle symmetry implicitly.
2. The experiments demonstrate the efficacy of the refiner module.

Weaknesses:
1. The efficacy of the coarse pose estimation model is not validated. In Table 1, the comparison between 8,9.10 and 11,12,13 is unfair for the proposed coarse module. It would be stronger if there is a baseline with other pose initialization techniques that can handle novel objects and use the proposed refiner.
2. Since both the coarse mode and refiner is based on rendering, how would rendering quality and the different lighting in the real world affect the performance?

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The general idea itself is pretty straightforward and not very interesting. But the performance of the proposed method is strong, especially on the real-world experiment. So I would vote for weak accept.
issues: 1. More results to demonstrate the advantage of the coarse model over other baselines.
2. Discuss how robust the model is to sim2real gap in terms of rendering.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This manuscript presented a method to estimate the 6D pose of objects unseen during training but only requires a CAD model at testing. It is achieved by learning two key components. A pose hypothesis generator and classification that initializes some pose guesses and whether they can be handled by the refiner, the second component. The refiner follows the render-and-compare idea similar to the existing CAD-model based methods with a difference that it can refine a novel object unseen during training. A large amount of training data is generated to gain such inductive bias.
strengths_and_weaknesses: Strengths:
- The paper is well written and the figures are presented nicely
- The motivation is clear and the application of this work is obvious for robotic manipulation
- The experiments are extensive, including multiple public benchmarks. The comparison baselines are also up to date.
- The performance is strong considering that it didn’t train on the testing objects. 
- To generalize to the testing object with a new coordinate frame definition provided in the CAD model, a novel and smart adaptation mechanism is designed by rendering multiple different vides to disambiguate.

Weakness:
- Is the performance equal when applied to different types of objects or is there any failure modes on certain types of objects? An illustration on this or discussion would be helpful. It’d be also interesting to have some detailed result decomposition on the objects to reflect their performance variations.
- The object detection is assumed to be available. While this is not the focus of this work, I’d imagine the accuracy of the 2D detection can play a crucial role. In addition, the current manuscript uses Mask-RCNN which needs training and inference on the same categories of objects. This contradicts with the claim of applying the method to out-of-class objects. Moreover, the Mask-RCNN is trained on the PBR data in this work, meaning many of the training instances are exactly the same as the testing data in BOP. This gives a huge advantage over the comparison methods (usually use natural detectors) from the beginning. It’d be more interesting to see a complete pipeline, by replacing with a class-agnostic object detector or at least a Mask-RCNN that is trained on some general dataset (eg. MS COO), or comparing with baselines that uses the same object detector to showcase the performance and compare.
- While the current manuscript claims to estimate 6D pose of novel objects, a CAD model is still needed during testing. The ease of accessibility to such a high-quality CAD model during testing (especially with high-fidelity texture) is questionable in real world scenarios.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: weak accept
issues: - In figure 2 bottom right, what does the green and blue axis mean and why is the T_CO2 and T_CO1 overlapped (redundant)?
- During testing for pose hypothesis generation, what if the object’s frame origin defined in the CAD model is far away from the object’s geometry center, and thus far away from the 2D bounding box center? Is the current initializing process still able to converge?
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: A method is proposed with which the pose of novel/unseen objects are estimated without requiring CAD models at test time, in contrast to the majority of methods that train with those models. During inference, the input is a region of interest (from a detector) as well as the CAD model from which a coarse pose is estimated then refined using two separate networks. The coarse pose estimation network is a binary classifier indicating if the synthetic rendering of the object can be successfully refined w.r.t. the observation. The input from a set of renderings with highest score is then the initial pose to start the refiner. The refiner network regresses the translation and rotation to iteratively bring the model into alignment with the observation. Experiments on the BOP core datasets as well as ModelNet show the proposed approach achieves best performance for novel objects and competitive results compared to methods that know the object models apriori i.e. seen objects.
strengths_and_weaknesses: **STRENGTHS**

The paper addresses a relevant problem in the robotics community and sufficiently motivates the design choices in the context of real limitations i.e. not always having access to object models before deployment. The authors cover the literature well, enabling them to position their work in the context of prior work.

Estimating the pose of unseen objects is very challenging. Currently there exist some solutions but they have limitations that are addressed in the proposed approach. Therefore, this paper offers novelty in terms of problem formulation and methodology.

The proposed approach achieves state-of-the-art performance on BOP and ModelNet in comparison to methods that also deal with unseen objects. Additionally, robotic grasping experiments are showcased in the supplementary video to demonstrate the real-world application.


**WEAKNESSES**

While the presented solution is fully-learned, I do not see this is a critical argument as to why the method is better than those discusses on lines 36-41. It is not a strong argument to say that non-learned components are unfavourable because they depend on hyperparameters, especially when the presented work also has its own set of tunable parameters e.g. M for the number of poses to render, k for the number of iterations and even the range of sampled standard deviations for the refiner dataset. A side not to this is that the method seems to be very sensitive to M, here large M means potentially better initial poses at the cost of computation time. It would be good to see an experiment to evaluate this trade-off to complement the statements made in Sec 4.4.

I question whether the method itself contributes to the high accuracy or if a significant performance gain is attributed to the enormous dataset used to train the networks. As table 3(b) reveals, fewer object models lead to a performance drop and under 500 + 10000, the method no longer outperforms the competitors. A better experiment would be to control for the dataset (i.e. train all methods with the new data or an equivalent number of training samples) in order to fairly compare the methods. The authors themselves make the claim that their method performs well because of the dataset in comparison to others that have a limited dataset to train in (lines 227-229).

While it is ok to declare the detection stage to be out of scope (and sufficiently handled by prior work), the experiments do not represent a realistic scenario of a fully unseen object pipeline since the detections (for BOP) come from a network trained on the relevant objects, hence they are known before. A more suitable method, in the context of the paper's goals, would be one of those mentioned on line 124. I suspect that the method could be sensitive to the initial detections, thus, it would be interesting to see an ablation on how the method performs w.r.t. detection inaccuracies. If the inaccuracy of zero-shot methods is below the tolerance of the proposed method, then a full pipeline is in fact not yet achieved. Lastly, it is not clear where the detections come from for the ModelNet experiments and the demonstrations in the supplementary video.

Table 1 is misleading since so few results are provided for the most relevant comparisons (rows 4-7). One can see that PPF + Zephyr + ICP is the best on LM-O, thus, it is questionable if the proposed method is the best overall without the missing results. (I understand the authors do not provide results on other datasets but code for their method is provided, thus it seems feasible to generate results for this publication). Furthermore, the authors state it is difficult to directly compare to OSOP due to the lack of a detector. Is it possible to use their detections to initialise your approach? This would make the comparison much more fair as the masks generated by Mask RCNN trained on the known objects is a huge advantage.

Finally, the demonstration of robotic grasping in the supplementary video is a good addition but it does not constitute a real "experiment" that evaluates the proposed method's performance. No details are provided about these demonstrations so it is difficult to draw any serious conclusions.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The authors have presented very interesting ideas that are novel for the purpose of unseen object pose estimation/refinement. Additionally, the method outperforms current SotA approaches with similar constraints. While the submission has some very solid strengths, unfortunately the evaluation has some flaws. Namely, there is a disparity in terms of the size of the dataset methods are trained on and the detections that are used for initialisation. Also, the most related methods are not evaluated on all datasets, which presents an incomplete picture. Thus, I consider some work still be done to fully convince readers of the practical significance of the work, which leads me to recommend a weak accept.
issues: The inability to train the coarse model that regresses instead of classifies is speculated to be caused by symmetries. Could symmetric objects be removed from the dataset, and the model retrained, to support this hypothesis?

The authors should state what hardware is used to train the models and how long training time is.

It would be good to see some qualitative failure cases.

It is not clear how the grasp poses are determined for the grasping demonstrations in the supplementary.
review_title: null

Meta-review: metareview: Main strengths:
1. The problem setting investigated (target CAD model given during test time only) seems relevant to practical robotics applications.
2. The proposed method (binary classfier to select best coarse pose as initialisation, render-and-compare to refine pose) appears to be sensible.
3. Results are comprehensive and overall convincing enough.

Main weaknesses:
1. The premise of the setting/method (bounding box and detailed CAD model of the target object are available during test time) is not beyond questioning.
2. Comparisons against other methods are not thorough; a few useful ablation tests are missing.

After the rebuttal and discussions, all reviewers thought that the concerns have been satisfactorily addressed. The paper has thus met the threshold to be accepted.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)