Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors proposed 6-DoF object pose estimation algorithms which does not require CAD models of target objects.  They require only video sequence of the target object with camera pose.  The keypoint-free SfM build 3D models in training and the keypoint-free 2D-3D matching network can estimate the correspondences between the models and image query. The proposed method handles low-textured objects and achieved state-of-the-art accuracy on public dataset.
strengths_and_weaknesses: Strengths:
- The proposed algorithm can handle low-textured objects and does not require CAD model for training, those are useful for real applications.
- The ablation study shows the effectiveness of each component.

Weaknesses:
- The proposed method needs off-the-shelf 2D object detector and real images for training.
questions: - How about processing time for training and testing?
- Is the pose estimation network trained for each object?
limitations: - CAD model or its equivalent can be reconstructed from the movie of target object with surrounded AR markers.  This might decrease the advantage of the proposed method.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper looks at the problem of one-shot object pose estimation on textureless objects where previous keypoint-based methods fail to perform well. The main contribution is using the keypoint-free SFM pipeline to create a repeatable semi-dense point cloud, which automatically helps improve 3D-2D correspondence to estimate the object pose. Comparisons show that the method performs better than a previous method which uses sparse point cloud reconstructions.
strengths_and_weaknesses: Strengths:
- Moving away from keypoint-based methods helps in automatic object pose estimation of objects in the wild.
- The semi-dense reconstruction of the objects seems to be very helpful in the pose estimation due to better 2D-3D correpondences.
- The results show that the method is more robust to occlusions than previous methods and a live demo helps show the method's accuracy.
- Ablation study shows the advantages of the refinement step and the attention module in the pose estimation network.
- Openpose-Hard dataset is useful for research in pose estimation of textureless objects.

Weaknesses:
- The major advantage or the difference between the proposed network and the previous baseline (onepose [30]) is the dense object reconstruction of the objects from videos. The contribution in the dense reconstruction from videos is well studied in the Multi-View Stereo frameworks. So I feel that the method doesn't have a lot of novelty in terms of reconstruction.
- The paper assumes that the object videos are given aprior, so one-shot object pose estimation might be a misleading term as the method will fail if the objects videos are not provided beforehand.
-  Literature review of Multi-view stereo needs to be well studied and the difference between these methods and the proposed keypoint-free methods need to be well established.
- The evaluations are not substantial as comparisons to other methods like CAD-model based pose estimation have not been well studied.
questions: How is the current method's dense reconstruction different from Multi-View Stereo paradigms. Why are evaluations not compared to these paradigms to show the accuracy improvement if the dense reconstruction is very good?


limitations: The limitations and potential negative impact are well studied in the paper.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The paper considers the problem of object pose estimation in scenarios where CAD model of the objects are not available. The paper describes OnePose++, a variant of the recently proposed OnePose approach that uses densely extracted descriptors (via LoFTR) rather than the SuperPoint keypoints used by OnePose. LoFTR provides matches between pairs of images, where the 2D positions of matching points vary depending on the image pair. As the resulting 2D positions are not repeatable, the paper uses a SfM approach designed to handle this scenario in order to build the 3D model of the object used for pose estimation. 2D-3D matches between a query image and the SfM model are established by directly matching descriptors against the 3D model in a coarse-to-fine manner. In addition to the OnePose++ method, the paper introduces a harder variant of the OnePose dataset, named OnePose-HARD. Experimental results show that OnePose++ outperforms most baselines by a wide margin (in particular, OnePose++ consistently outperforms OnePose).
strengths_and_weaknesses: The paper has multiple strengths:
S1) The proposed OnePose++ approach is a natural extension of OnePose that shows how to swap keypoint-based features with keypoint-free features. Since the latter have shown promise in challenging scenes, e.g., for weakly textured objects or under strong illumination conditions, this is interesting.

S2) OnePose++ clearly outperforms OnePose and also most of the baseline methods. The strong results are a strength of the paper.

S3) The paper provides a detailed ablation study that analyzes the impact of the individual components of OnePose++.

S4) The proposed dataset, OnePose-HARD, seems very challenging and thus has the potential to drive research in the field. It will be of interest to the community.

On the negative side, there are also multiple weaknesses:
W1) Both OnePose and OnePose++ are misclassified as being generalizable object pose estimation approaches. It is true that the underlying LoFTR features and the coarse and fine matching stages generalize beyond the data they were trained on. Yet, OnePose++ requires that "a video sequence with annotated poses is available for each object" is available and builds a SfM model for this particular type of object. I don't see how this is not an instance-level method. The SfM model for one object might give reasonable results for another object if both objects have very similar shapes (and textures). But a single model for one particular type of an object, e.g., a particular chair, will not generalize over the full class (e.g., all potential chairs).
Given that OnePose++ is an instance-level method, it should be more closely compared to other instance-level methods. It is not too surprising that it outperforms the generalizable baselines as it is able to train (in the form of building a SfM model) per object.

W2) The paper claims proposing a "keypoint-free SfM framework for accurate and complete semi-dense reconstruction" as one of its main contributions. It is not clear to my how the described framework contributes novelty to the literature:
a) Besides LoFTR, there are other keypoint-free descriptors, e.g., Patch2Pix [Zhou et al., Patch2Pix: Epipolar-Guided Pixel-Level Correspondences, CVPR'21], Sparse NCNet [Rocco et al., Efficient ´neighbourhood consensus networks via submanifold sparse
convolutions, ECCV'20], and Dual RCNet [Li et al., Dual-Resolution Correspondence Networks, NeurIPS'20], that are evaluated in a visual localization setting that requires an underlying SfM model. They thus also provide approaches for keypoint-free SfM. All of them are inherently applicable to object pose estimation (since they do not make any assumption on the type of scenes). Similarly, LoFTR also uses a keypoint-free SfM approach (based on Dual RCNet) (see https://github.com/zju3dv/LoFTR/issues/9). Another approach to keypoint-free SfM based on dense feature matching between images is [Widya et al., Structure from motion using dense CNN features with keypoint relocalization, IPSJ Transactions on Computer Vision and Applications 2018]. Yet, this prior work is not discussed. The differences between this prior work and the proposed approach should be clearly described. Furthermore, comparisons with other keypoint-free approaches, e.g., Patch2Pix or Dual RCNet, are missing.
b) As far as I can see, the coarse reconstruction stage for keypoint-free SfM is the same as for LoFTR (based on the description provided here: https://github.com/zju3dv/LoFTR/issues/9). The refinement stage seems identical to [18]. The paper states that "Note that our keypoint-free SfM framework is also related to PixSfM [18] but comes with different motivations. PixSfM improves keypoint-based SfM for more accurate 3D reconstructions by refining inaccurately-detected sparse local features with dense feature maps. Different from PixSfM, we aim to adapt the keypoint-free method LoFTR [29] to SfM for object pose estimation." However, I disagree with this statement. As stated in the paper, "Every pixel in the downsampled image can be regarded as a “keypoint” in the original image." The goal of the refinement stage is to "refine the object point cloud with sub-pixel correspondences." In other words, the motivation for the refinement stage is to obtain a more accurate 3D model by refining initially inaccurate keypoint positions. This is achieved using dense feature maps to detect more accurate keypoint positions.

W3) As in the case of keypoint-free SfM, there are other directions of highly related work that omitted:
a) Work on object pose estimation using SfM rather than CAD models certainly predates OnePose. Examples include [Gordon & Lowe, What and Where: 3D Object Recognition with Accurate Pose, Toward Category-Level Object Recognition, 2006], [Rothganger et al., 3D Object Modeling and Recognition Using Local Affine-Invariant Image Descriptors and Multi-View Spatial Constraints, 3D Object Modeling and Recognition Using Local Affine-Invariant Image Descriptors and Multi-View Spatial Constraints, IJCV 2006], [Hsiao et al., Making specific features less discriminative to improve point-based 3D object recognition, CVPR 2010[, [Bhat et al., Visual words for 3D reconstruction and pose computation, 3DIM/3DPVT 2011], and [Fenzi et al., 3D Object Recognition and Pose Estimation for Multiple Objects using Multi-Prioritized RANSAC and Model Updating, DAGM/OAGM 2012]. This prior work should be properly acknowledged.
b) The paper states that "HLoc is slow during pose estimation because it depends on multiple 2D-2D image matchings as the proxy for building 2D-3D correspondences." Yet, there is quite some literature on visual localization algorithms that do not use image retrieval but directly match 2D features against 3D points via associated feature descriptors. Examples include: [Arth et al., Wide Area Localization on
Mobile Phones. ISMAR 2009], [Li et al., Location Recognition using Prioritized Feature Matching, ECCV 2010], [Li et al., Worldwide Pose Estimation Using 3D Point Clouds, ECCV 2012], [Choudhary & Narayanan, Visibility probability structure from sfm datasets and applications, ECCV 2012], [Donoser & Schmalstieg, Discriminative featureto-point matching in image-based localization, CVPR 2014], [Cao & Snavely, Minimal scene descriptions from structure from motion models, CVPR 2014], [Lim et al., Real-time monocular image-based 6-dof localization, IJRR 2015], [Lynen et al., Get out of my lab: Largescale, real-time visual-inertial localization, RSS 2015], [Zeisl et al., Camera Pose Voting for Large-Scale Image-Based Localization, ICCV 2015], [Camposeco et al., Toroidal Constraints for TwoPoint Localization under High Outlier Ratios, CVPR 2017], [DuToit et al., Consistent map-based 3d localization on mobile devices, ICRA 2017], [Liu et al., Efficient Global 2D-3D Matching for Camera Localization in a Large-Scale 3D Map, ICCV 2017], [Sattler et al., Efficient & effective prioritized matching for large-scale image-based localization, PAMI 2017], [Svarm et al., City-scale localization for cameras with known vertical direction, PAMI 2017], and [Lynen et al., Large-scale, real-time visual-inertial localization revisited, IJRR 2019]. Many of these approaches are directly applicable to the object pose estimation setting based on SfM models and should thus be discussed.

The following additional comments did not affect my recommendation:
* References for RANSAC and the PnP solver used are missing.
questions: In order to consider raising my score, I would like to see the following points addressed in a rebuttal:
Q1) Please describe in detail how the proposed keypoint-free SfM approach differs from prior work in this area.
Q2) Why were Patch2Pix, Dual RCNet, etc. not considered as baselines?
Q3) Please describe the relation of OnePose++ to prior work on SfM-based object pose estimation and visual localization (see W3 above).
limitations: The paper adequately discusses limitations and potential negative social impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes an improvement of the one-shot pose estimation system OnePose to better estimate the pose of low-texture objects. In particular, the keypoint-based matching component of OnePose is replaced with the key-point free matching method LoFTR. For the evaluation of the improved functionality, the authors propose the new dataset OnePose-HARD, which contains low-texture objects along with their pose annotations.

strengths_and_weaknesses: Originality:
+ The idea of replacing keypoint-based matching with keypoint-free matching for improved low-texture performance is straightforward and a logical extension of the prior work.

Quality:
+ The paper is well written and structured.
+ The paper precisely adheres to the formatting requirements and the page limit.
+ Related work is described sufficiently and it is made clear how the proposed method is positioned in the existing landscape.
+ The proposed method is evaluated on appropriate datasets and compared to relevant state-of-the-art methods.

Clarity:
+ The language is clear and easy to follow.
+ Methods used in the paper, e.g., LoFTR, are explained briefly, but well understandable.
- The description of the proposed OnePose-HARD dataset is quite abstract. Example images of the new dataset are not included in the paper.
- In the evaluation, it is not always clear where the results of the other methods come from. E.g., in Table 1, the results of the OnePose method on the OnePose dataset is given as an overall single value for the whole dataset, while the original OnePose paper lists the results of the three categories "large", "medium", and "small" separately, but no overall single value. Similarly, in Table 3, the paper gives results for OnePose on the LINEMOD dataset, but the original OnePose paper contains no such evaluation.
- The authors claim that their method is "~10× faster" [line 290] than other methods, but the actual runtimes of the other methods are not listed in the paper.

Significance:
+ The addressed issue with low-texture objects is relevant in practice.
+ The results of the proposed method seem to be significantly better than comparable methods, particularly in low-texture scenarios.
questions: 
- Please consider including example images of the OnePose-HARD dataset in the paper.
- Please clarify the source of the results of other methods that you compare your method to: Where are these numbers taken from or did you evaluate the methods yourself? In particular, the results of OnePose on the OnePose dataset and on LINEMOD, as described above. Please also describe why some results are underlined in Table 1.
- Please consider including the runtimes of your method and the methods you compare to, in order to support the claim that your method is 10× faster.


Additional remarks (typos, suggestions etc.), no need to address in the rebuttal:
- line 49: "establish" -> "establishes"
- line 70: "OnePose-HARD ," -> "OnePose-HARD,"
- line 104: "keypoints ." -> "keypoints."
- Figure 2, caption: "a reference image sequences" -> "a reference image sequence"
- Figure 2, caption: "point cloud which are" -> "point cloud which is"
- line 131: "build" -> "builds"
- line 295: "number(~ 5000)" -> "number (~ 5000)"
- On the naming of the proposed dataset: The suffix "HARD" is very generic and does not tell what the difficulties actually are. Please consider a more informative suffix, such as "LowTexture".

limitations: Yes, limitations and impact were discussed where applicable.
review_title: null

Meta-review: metareview: This paper originally received slightly positive reviews overall, except for one review, which was plenty of requests of specific clarifications and comments. Main issues regarded just the need of clarifying some parts of the method and put better in context of the state of the art and former evaluations. Unclear novelty was another raised problem, as well as the need of off-the-shelf 2D object detector and real images for training, which might affect the general applicability of the method while weakening the "one/shot" claim of the work. A lot of concerns were also raised about missing baselines and prior work discussion. 
Authors provided detailed answers to the comments, also engaging in long discussions, especially with the most critical reviewer.
In the end, the most positive reviewers seem to be satisfied of the answers to their comments, maintaining the original positive ratings, and also the critical reviewer resulted convinced of the discussion with authors, raising his/her score to weak accept.
Overall, assuming that the comments and discussions could be included in the final version, this paper can be considered acceptable for NeurIPS 2022 publication.

confidence: Less certain
Decision: Accept