Venue: CoRL 2023 Conference

Reviews:

final_score: rating: null
review_content: I would like to thank the authors for their responses. However, the responses actually affirmed my concerns about the setup and I would be inclined to argue strongly against acceptance. Given the other reviewers are more positive, I wanted to raise this as a common discussion point and would like to hear from the other reviewers/authors if they can help address these fundamental concerns about the setup.

Specifically, the paper tackles a setting where K (say 20) images depicting an object in a specific scene (with known 6D object pose of the object in each image) are available.  Then, given a new image in the same scene (i.e. same background, unchanged configuration of surrounding objects), the proposed approach seeks to predict the 6D pose of the object in this new image. First, I am not sure why this is a valid/useful setup to consider. Secondly, this task can also be equivalently be framed as that of camera registration and I think the paper needs to compare to relevant baselines. I appreciate the authors point about possible generalization to new scenes, but without empirical evidence that this is feasible, I am inclined to only consider the 'within-scene' setup as one that the approach can be applied to and this raises some key issues:

1) I fail to see practical settings where this setup would be useful. The assumed input is several images of an object in a specific scene with known 6D poses of the object, and the query image is also from the same scene. This is in contrast to other setups which could be more useful e.g. a) given 3D mesh of object, estimate 6D pose in novel scenes or b) given reference images with known poses across a few scenes, estimate 6D pose in a novel scene. Given this approach requires reference and query images in the same scene, this implies that the user is be able to obtain 6D poses of the object for the K images in the scene of interest but not for the query (K+1st) image and I am not sure what scenario would arise with such constraints.

2) Given that the scene is unchanged, the knowledge of 6D poses of the object in the reference images implies known relative cameras for all the reference images. Thus, given a new image, inferring it’s camera pose can equivalently yield the 6D pose for the target object. In my view, this necessitates comparisons camera registration/SfM methods. While the authors are correct that COLMAP requires many more images for a dense reconstruction, I believe the recent advances in improved features/learned matching make it more robust (specifically if we only require cameras poses and not a reconstruction). Moreover, the task here is only registration/localization of a new camera given other calibrated cameras (as opposed to SfM where no camera extrinsics are known). Given the equivalence of the two tasks, I do not think this work can be accepted without such comparisons (e.g. using the localization code in https://github.com/cvg/pixel-perfect-sfm, https://github.com/cvg/Hierarchical-Localization).


review_title: Unconvinced about the setup

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: - Online self-adaptation module
  - Extract standard object model: Given reference images and pose of target, a segmentor is employed for segmenting the target objects. Then, by combining the point cloud of the target object, the "standard" object point cloud model is generated.
  - Generate a target-specific segmentor: Given the masks for target/non-target objects, contrastive learning is employed for finetuning the segmentor.
  - Segment test image: Test images are fed into both pretrained and finetuned segmentor for per-pixel label and dense feature, respectively. Target object is selected by comparing the feature with reference.
- Region Proposal Module: Estimate the size of target object based on the standard object model and crop the region of the same size on the target image. Then images are fed into Gen6D to generate a initial guess for the pose.
- Refinement module: Performs ICP between standard object model and segmented object from test image to refine the pose prediction. 
review: - Strength
  - The paper is well-written and easy to follow.
  - The topic of self-adaptative pose estimation is important to many practical applications
- Weakness
  - Need ablation study without online adaptation. Say both pixel label and dense feature are generated with the pretrained UCN.
  - Need baseline results for RePoNet, DualPoseNet on LineMod/LineMOD-OCC, etc.
  - Performance in uncontrolled environment is questionable. As shown in Table 3, on Wild6D, SA6D can hardly out-performs RePoNet, which has a much simpler inference pipeline.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: For now, a weak acceptance is given for the online adaptation. However, more results are required for supporting the effectiveness of the method, especially the effectiveness of the proposed online adaptation module.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This work tackles the task of 6D pose estimation of a generic object in occluded scene, given a few RGB-D images (under clutter) with known 6D poses for the target object as reference. The approach is to: a) adapt an off-the shelf instance segmentation system to better identify the target object, and b) build a object-specific 3D model using the 6D pose annotations and predicted segmentations. Given a test image, the adapted segmentation system can detect the object and an off-the-shelf 6D pose estimation system can yield initial 6D pose estimate (given object-centric crops from test and reference images), which is then refined via ICP. Experiments show that this approach performs better than prior methods across several examined datasets.
review: Strengths:
+ The paper tackles an interesting setting. Specifically, unlike some prior works which require either object meshes or reference images without clutter, this paper can leverage reference images in a cluttered setting. 

+ The empirical results are shown across datasets, and the importance of the contributions is  evaluated well.

Concerns:
- Please correct me if this is wrong, but it seems that reference and test images are from overlapping scenes (i.e. common background and object configurations across reference images and test image). In my view, this is a significant drawback of the setup as it’s not clear if the 6D pose estimation systems or segmentation adaptation actually generalizes (and pays attention to the object vs scene). Moreover, if the reference and test images are indeed the same scene, why do we even need to estimate 6D poses of objects? Instead, we can simply try to obtain relative camera poses between reference and test images (e.g. via COLMAP), and given 6D object pose in any reference image, can trivially obtain 6D pose of object is test image.

- While the paper presents itself as tackling the task of 6D pose estimation, the core technical contributions infact lie in adapting a segmentation model. The 6D pose estimation is actually done via a combination of a previous approach (Gen6D) followed by ICP. The key contribution here lies in improving the detection/segmentation so that clean inputs can be fed to Gen6D and ICP. While this is still a reasonable thing to do, there are no 2D detection/segmentation evaluations. I feel the paper would be better presented (and evaluated) as one for improving few-shot segmentation/detection for a generic object, with one possible application to 6D pose estimation.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: First and foremost, I have significant concerns regarding the setup (specifically, whether the train and test images come from common scenes). That said, even assuming these are addressed by the author response, I am still not overly positive about the work and would probably rate this as borderline because the primary contribution seems to be in few-shot adaptation of a segmentation system as opposed to any innovation in 6D pose estimation, which is done via Gen6D + ICP. On average, I am currently indicating my rating as a Weak Reject. 
---

Post-discussion update: The author responses (specifically the new experiment showing ability to test in different scenes) did address my concern, and as indicated in my comments in the discussion below, I'd be happy to lean towards acceptance.

Hopefully the authors will include this experiment in the final version.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper introduces an approach for 6D pose estimation of novel objects with heavy occlusions while not requiring any object information or object-centric images. 

More specifically, it fine-tunes the weights of a base image segmentation model using reference images with pose labels, which allows to better localize the target object. Moreover, it leverages the geometric features to improve the robustness against occlusion in comparison to Gen6D.

Experimental results on various benchmarks demonstrated the effectiveness of the proposed method, which could potentially facilitate generalizable 6D object pose estimation in robotic applications.
review: ### Strengthes
- The paper is well-written with nice figures and clear equations. 

- The proposed method achieves superior object pose estimation performance in various settings while being trained on synthetic data only.

- The proposed method builds upon a simple image segmentation model and basic point cloud registration methods, which is simple yet effective.

### Weaknesses

- The method assumes that the target object is always contained by the test image, what if the object is completely occluded in the test image? Can we tell the absence of the target object from the feature similarity, thus avoiding invalid pose estimation?

- How long does it take to complete the online adaptation? How will the adaptation time scale in the case of large-scene containing hundreds of objects?
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: This paper proposes a simple yet effective object pose estimation approach with good representation and convincing results. In particular, it achieves generalizable object pose estimation in heavy occlusions, with models trained on synthetic data only.
This leads to promising robotic applications where the robot needs to interact with novel objects in cluttered scenes.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary_of_paper: The paper proposes an RGB-D approach for few-shot 6-D pose estimation of novel objects, requiring as input a small set of posed RGB-D reference images and also handling severe occlusions. The method effectively combines pre-existing tools, including: 1. the object-agnostic segmentation module UCN, and an InfoNCE contrastive loss, used to extract object masks in both the reference images and in the test one and train an global object feature representation; 2. The Gen6D pose estimator, used to provide an initial 6-D pose estimate from an object crop; 3. ICP for point-cloud based pose refinement. This combination results in improved accuracy over state of the art and increased robustness to occlusions. Additional contributions include:
- The removal of the need for object-centric reference images or object information (e.g., 3-D model or object diameter).
- A method to extract the object crop in a test image which is more robust to occlusion and clutter, by using depth information.
review: ## Summary
### Strengths
- Improved performance and robustness to occlusion and clutter w.r.t. state-of-the-art methods.
- Thorough experiments and ablations.
- Very good quality of the manuscript.
- The supplementary material (particularly the video) is useful and facilitates the understanding of the main manuscript.

### Weaknesses
- The method is not extremely novel, in the sense that it combines pre-existing tools (although this combination is not trivial and results in improved performance).
- Compared to previous few-shot approaches, the method assumes the availability of depth information.

## Detailed review
### Quality and clarity
- The problem that the paper addresses is clearly stated and motivated, and an extensive list of related work is provided.
- The experiments are thorough, with evaluations on several standard datasets and with multiple types of baselines (both few-shot methods and RGB-D category-level methods) and ablations on the model components. All my main questions regarding the influence of the difference components were answered through these experiments.
- The quality of the manuscript, of the visualization and the supporting material is in general high, with a very limited amount of typos and mostly clear and linear language (cf. "Suggestions and minor questions" below).


### Originality and significance
- Although the method does not present groundbreaking new ideas, it combines pre-existing tools in a effective way, resulting in a clear improvement over previous methods.
- No experiments from a real-world setup are presented, but the method is evaluated in several real-world benchmarks.
- I find particularly interesting the simple, but smart idea, behind the selection of the mask for the object of interest in the reference images, by using the known extrinsics. I also appreciate the discussion in Sec. A.3 about how such a strategy might fail, but the confidence score can still compensate for it.

### Suggestions and minor questions
**Language**

- The abstract is probably the main part for which I would suggest rephrasing. In particular, the sentence "[...] meaningful robotic manipulation [...] one of the critical aspects" sounds vague and a bit clunky (e.g., what does "meaningful" mean exactly? "Critical" for what?). Similarly, the term "continuously [introduced]" (L4) can have a fairly specific (and diverse) meaning in the literature, so when reading it I would expect the paper to put emphasis for instance on the "open-set" problem, or potentially on some continual learning problems, which is not done in the rest of the paper.
- Minor: Throughout the paper, "i.e." could be replaced with the macro "\ie".
- L124: "copying $\phi$" -> "copying **the parameters of** $\phi$".
- L128: $\phi*$ -> $\phi^*$ or $\phi^\star$.
- Caption of Table 1: "HomwBrewedDB" -> "HomeBrewedDB".
- In all its occurrences (header of Table 1, L223, caption of Fig. 5, etc.), I would recommended replacing "reference number"/"reference" with "number of reference images".
- L203: Fix "cannot _even though without_ [...]". Also, on the same line, the tense of "found" is not coherent with the rest of the paragraph.
- L225: "[...] except that LatentFusion sometimes [...]" -> "[...] except LatentFusion, which sometimes [...]".
- L246: "replace" -> "tried replacing" or "experimented replacing" (tense coherency).
- L250: "costs" -> "is".
- Fig. 7 in the Suppl.: "Even though" -> "Nevertheless".
- L58 Suppl.: Missing "the case" after "It is often".
- L65 Suppl.: "exclude" -> "excluded", "We" -> "we"

**Notation and formatting**
- L123: Technically, $S^N$ should be defined as $\bigcup_{i=1}^M S^N_i$, since the $S^N_i$s are sets. A space is missing after the word "segments".
- L155-157: It could be clarified that $T^x_{init}$, $T^y_{init}$, $T^z_{init}$ are part of $T_{init}$.

**Additional details**
- The $n^\circ, m~\mathrm{cm}$ metric, used in Tab. 3, was not mentioned in the "Evaluation metrics" paragraph (L189-194).
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: While mostly based on a combination of pre-existing tools, the proposed approach advances the performance on the problem of few-shot 6-D pose estimation of novel objects. Furthermore, despite the additional requirement of depth information, it relaxes the assumptions of previous methods and further increases the robustness of these methods against occlusions and clutter, which are practical problems relevant for robotic applications. I therefore believe the paper would be a valuable contribution for the community and recommend acceptance.
review_title: null

Meta-review: The paper introduces a novel method on few-shot object pose estimation that can handle novel objects.

After the rebuttal, most concerns of the reviewers have been addressed, and all the reviewers agree on accepting the paper.

I recommend the paper as a poster presentation due to the lack of applying the pose estimation method to real robotic applications such as grasping or manipulation of objects.
Decision: Accept (Poster)