Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This work tackles the task of 6D pose estimation for novel objects given prior observations of a single object of the same category (where multiple RGB-D images from known relative poses are available for both, the template and the query object, and the goal is to find a rotation and translation to align the template and the query object).

The proposed approach is to: a) extract spatial features from all images using a pre-trained ViT (with dimensionality reduction using PCA) , b) find a rotation that aligns the template and query object using a set of (query image, template image) pairs that match well and are consistent with other pairs, and c) estimates a 7D alignment (R, t, scale) using patchwise matches between the selected pairs and leveraging RANSAC.

strengths_and_weaknesses: Strengths:
- The paper tackles a relevant problem and considers a natural setup for it. Given multiple views of a query object and a template object, computing alignment between the two is a great formulation for the task of one-shot pose estimation..

- The experiments are reported on the Co3D dataset which represents a challenging setup, and shows that the proposed method does indeed generalize across objects.

- A key implementation detail that this work highlights is the benefit of feature dimensionality reduction. This is well-ablated and leads to consistent improvements.

Weaknesses:

- One concern is that the paper relies rather heavily on the framework presented in [17], and the primary contribution here is to extend that framework to leverage multiple query views (in a slightly straightforward manner). While this obviously leads to significant empirical gains (as using multiple views should!), this does limit the technical contribution. Moreover, this should also necessitate an alternate baseline where one simply uses the approach from [17] to get multiple estimates and chooses the ‘mode’.

- While the work does use multiple query views, the precise methodology feels somewhat constrained. In particular, the only patch ‘correspondences’ that are allowed (and ultimately drive the optimization) are between batches at the same grid locations across images. This is suboptimal for two reasons as: a) it implicitly assumes that images from very similar viewpoints are available for matching and may not be a valid assumption, and b) cannot leverage observations of matching content from dissimilar viewpoints (the handle of a cup might match between a left-view and a right-view, and should drive the pose estimation).

- On a note slightly related to the above, I am curious why a more ‘traditional’ SfM like approach is not considered (as the method, or atleast as a baseline). In particular, given depth images, the patches can be considered as 3D points with associated (deep) features. One could then construct matches between the 3D points observed for the source and target objects, and using RANSAC, optimize the 7D pose to best align them. I would be interested to hear why this would not suffice (particularly as it is the ‘typical’ approach that drives large-scale structure from motion — and in this case the depth + known extrinsics within each object would only simplify the task further)?

- (comparatively minor concern) The continual learning aspect seems a bit weak and is more of a post-hoc application rather than something integral to the method. In particular:
a) the notion of a ‘category’ does not really keep evolving: if one sees multiple chairs, the category is still represented by a single template (or perhaps a collection of independent template — the text is unclear on this). In any case, multiple examples of a category do not lead to an evolving notion of the category. 
b) The determination of a ‘new’ vs seen category is simply done via average cosine similarity of the [CLS] token of the ViT model and does not have much in common with the approach for 6D pose alignment.
Considering both these, the continual learning aspect of the work, while tackling a good goal, seems a bit adhoc and technically disconnected.  

robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Overall, I am slightly torn about this work. On the one hand, it considers a relevant setup and shows promising results. However, this is currently outweighed by the concerns (similarity to [17], and possibilities of a simpler SfM-like solution).

Update:
Thanks for the response, and in particular for addressing my misunderstanding regarding the matching, which also addressed the concerns regarding a more 'regular' SfM-like method. I am happy to update the rating towards an accept and would recommend acceptance as a poster.
issues: - Please see the Weakness section. In particular, I would be interested in the author response to why directly reasoning in 3D space and simply searching for (robustly via RANSAC) a 7D pose that maximizes alignment between matching 3D points is not a good solution.
- As a minor concern, the only evaluation in the paper focuses on rotation estimation, and it would be good to see some results on the accuracy of the full pose estimated.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents an approach to object pose estimation at the category level by leveraging multiple views of a single reference object from the category. The approach combines advantages of different paradigms : deep models and feature matching.  High dimensional features from a pre-trained ViT is used for efficient descriptors through dimensionality reduction. These set of image descriptors from multiple views of an object are used to find correspondences between the target and the reference object of the same category. An average pose estimate is obtained by filtering best reference-target view pairs. Finally the filtered subset are aligned using a robust least square estimation to give a 6D object pose.
The paper builds upon [17] by using multiple reference object views which leads to considerable improvement in performance.
strengths_and_weaknesses: Strengths:

The paper is well written and supported with relevant media content.
The technical contribution is incremental yet considerable.
The proposed approach results in better performance in terms of speed and accuracy than existing category level technique.

Weakness:
Although the performnace improvement in Table 1 has promising results, the extensive evaluations with other techniques would be desirable.
For e.g., [14], You Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstration- RSS2022
Also, in line 38, I do not think its well argued.  Do you mean method in [14] is designed for single object instances? Please check
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I believe the approach is interesting and results in improved performance than recent approach.
However, I do want to see more comparative evaluations to be convinced.
issues: Comparative evaluations mentioned earlier.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: 
This paper proposes an approach for one-shot category-level pose estimation. Given M reference views of an object belonging to a specific category (which defines a category-level pose) and N views of a novel instance of that category the task is to define the category-level pose estimate. The authors propose to use a pre-trained DINO ViT to produce 2D-2D semantic correspondences between the reference and target images and then solve for the relative pose by (1)  lifting the 2D correspondences to 3D using provided depth images and (2) aligning the correspondence sets using RANSAC.
strengths_and_weaknesses: The paper is an extension of prior work [17] (note that this work is only on arxiv and I was unable to find any conference/journal publication of [17]). In particular the main innovation of this work seems to be in using multiple views $$I_{R_{1:M}}$$ of the reference object instead of just one view as in [17]. 


**Strengths**


The main strength of the paper is that it tackles an interesting and challenging problem of one-shot category-level pose estimation. They demonstrate that using multiple reference views leads to significantly improved performance over prior work [17].


**Weaknesses**



* The main experiments (Table 1, Table 2) assume 10+ views of the target object at inference time. This is typically not a realistic assumption in most robotics settings and thus requires more justification if this is to be the main test condition of the proposed approach. How would this affect the method in terms of the consensus and retrieval steps?
* The robot experiments are quite weak in the current paper. More qualitative examples of the pose estimates would be helpful. From the video it doesn’t seem that the pose estimates would be accurate enough to actually enable any meaningful robot manipulation tasks (e.g. category-level tasks such as those performed in [10, 11], category-level grasping, category-level pick and place, etc.)

* The authors should include at least some comparison the performance of this method vs. what can be achieved with a supervised category-level pose estimator such as [6, 8]. This would allow the reader to understand the tradeoffs between using a supervised approach (that has more requirements/restrictions) vs. a more general approach such as proposed one. A comparison on a standard category-level pose estimation benchmark such as the NOCS dataset from [6] would make sense.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: Overall this paper tackles the interesting and challenging problem of category-level pose estimation without category-level training, only relying on multiple reference views of an example from the category. The current approach leverages a pre-trained DINO ViT model to extract correspondences which are then used in a fairly standard RANSAC + Umeyama algorithm for producing pose estimates. The current paper lacks comparisons to some supervised category-level pose estimation baselines (such as [6,8]). Additionally it is not at all clear that these pose estimates are sufficiently accurate to actually be used to perform category-level tasks. The method also has several limitations. In particular it only considers a 7D pose transform (rigid + scale) for characterizing the category, which may fail to capture the intra-category shape variation in certain categories (e.g. comparing a sneaker to a tall boot).
issues: List issues to be addressed during the period of author response and revision. The authors have the opportunity to submit a revised manuscript during the period. (max 5000 characters)

* Can the authors expand on how category-level pose is actually defined in the extension of the CO3D dataset. It would be useful to have this in the appendix at least so the reader doesn’t need to refer to [17]. 
* The authors should include metrics on the translation errors of the pose estimates after orientation. Translation is just as important as orientation in terms of usefulness in a downstream robotic application.
* How would the method perform in a more cluttered setting that is typical of robotics scenarios? Would be helpful to have some quantitative and/or qualitative results here.
* Can you expand on why you get such a speedup compared to [17], the underlying algorithms seem to be the same with exception of the current work using multiple views.
* The authors should discuss how the method would handle more cluttered scenes (as is typically encountered in robotics scenarios). Would any significant changes be needed to the approach.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The paper presents a method for category-level object pose estimation based on RGB-D input. The approach does not require pose-labeled data, 3-D models, or category-specific networks, but only relies on few views of a target object and of a reference one of the same category. Similarly to [17], the paper shows that features extracted from a pre-trained ViT can be used as descriptors to find category-level correspondences for pose estimation. Compared to [17], the paper proposes to use multiple views from the reference object and shows that this has a significant impact on the estimated poses. Further, the method is applied in a continual learning setting for novel category discovery. The evaluations show that proposed approach significantly outperforms [17] and the baseline of [13], which however uses purely geometric correspondences. The method is demonstrated in a robotic setup to perform real-time pose estimation of a novel object based on a reference object from the same category.
strengths_and_weaknesses: The motivations and the research gaps addressed by the proposed paper are stated clearly. A detailed review of the most relevant related work is provided, with a sensible and targeted classification of the previous approaches.

The paper is generally very well written, with virtually no typos or notation inconsistencies. With very limited exceptions (mentioned in "Issues"), the language is clear and the explanations are thorough, making the paper easy to read.

The quality of the presentation is very high. Despite minor possible improvements, I find the visualizations professional and very informative, particularly Figure 1, which provides an effective overview of the method, and Figure 2, which illustrates clearly the process of finding correspondences and highlights interesting, not obvious points, such as the correlation between view-to-view similarities and true geodesic errors. The attention to the details is also evident in the video (e.g., the visualization of the actual reference-target views in the correspondence matrix, or the method-overview figure edited to visualize the moving robot arm).

A number of minor technical details, particularly in the choice of the hyperparameters, could be provided for completeness. Furthermore, for multiple explanations and additional results, though all supporting non-major points, the reader is referred to the supplementary material, but the latter does not include the aforementioned additions.

The main point that could be seen as a weakness is that the viewpoint estimation process is heavily based on the previous work of [17] (as repeatedly acknowledged), and that particularly one of the main differences, i.e, the use of multiple reference views as opposed of a single reference view (as in [17]) could render the setup more complicated. However, as shown in the experiments, this choice results in a large improvement of the pose estimation accuracy. Furthermore:

- A number of other relevant insights (e.g., the impact of feature dimensionality) or modifications (consensus filtering vs. RANSAC, cosine similarity vs L2 distance) are presented
- The method runs in real time and was demonstrated on a real robotic system
- The whole application in a continual category learning scenario for new category discovery is novel and of high relevance
- Several interesting ideas are introduced or investigated, for instance the use of the [CLS] token for category comparison, the evaluation of feature similarity in 3-D rather than in pixel space, or the use of the attention maps to produce a segmentation mask
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper presents a number of novel insights on the use of vision transformers for one-shot category level object pose estimation and novel category discovery, which could significantly question the need for training object- or category-specific feature networks. The approach runs in real time and is demonstrated on a robotic platform. The explored setup, i.e., category-level, without the assumption of CAD models, instance-level networks, or pose-labeled data, and only relying on a small number of reference and target views, has a high potential of concrete use in robotics. 

The experiments are solid and the presentation is professional and effective for illustrating the method and the results.

I believe this paper could be a strong contribution to this Conference, after addressing a few minor points.
issues: ### Technical details
- L144: In what sense "normalised"?
- Sec. 3: What values were used for $M$ and $N$? Was an ablation study performed to select them?
- Fig. 2: "Three consensus filters"/"iterations of filtering": Where does the "three" come from? Does this value always guarantee that the criteria mentioned in the text (L197) are met?
- Fig. 2: "good" -> "closest", as in the subfigure title of (b)? That is, what does "good" mean here?
- Sec. 3.4: What value of $K$ was used? What impact does this have on the resulting pose estimate?
- Sec. 3.4: The correspondence indices $u_k, v_k$ come from a $28\times 28$ feature map, whereas the RGB-D image has larger resolution. How do you select the depth of the 3-D point associated to $u_k, v_k$, given the difference in resolution? Have you investigated matching the RGB-D resolution, e.g., using interpolation? Intuitively, this could produce more accurate estimates of the 3-D point positions.
- L193-197: I find this explanation a bit articulated and hard to parse. If I interpret it correctly, for each pair $(i, j)\in \mathcal{Sel}\subset M\times N$ , where $\mathcal{Sel}$ are the retained pairs (Line 192), a mean pose $\mathbf{T}\overline{ij}$ is computed from all the other pairs $\{(m, n)\in \mathcal{Sel}\ \backslash(i, j)\}$, and then the "views" that are removed at each iteration are the _pairs_ $(i^\prime, j^\prime) := \arg\max_{(m, n)\in\mathcal{Sel}} \textrm{dist}(\mathbf{T}{mn}, \mathbf{T}\overline{mn})$, where $\textrm{dist}$ is the pose distance and $\mathbf{T}{mn}$ is the relative pose obtained from pair $(m, n)$. I find confusing the concurrent use of "view" and "view pair", and also the computation of the "mean" could be rephrased to be more clear (e.g., is the mean pair-specific, as I interpreted above?)
- L222: The idea of using [CLS] tokens for category comparison is interesting. Were other options considered, or any ablations performed?
- L297: It should be explained in the text what exactly is meant by "$15^\circ$". That is, how can a single angle parametrize a distance between poses
- L333 (Supp.): The concept of "(ViT) saliency" should be introduced
- From the real-world experiments shown in the video it seems that pose estimate is not very stable across frames. Did you evaluate quantitatively the extent of the pose robustness and its impact on manipulation (e.g., would inconsistencies lead to failures in grasping the object)?

### Missing details that should be in the Supplementary
- L226: Threshold on $\theta$ 
- L242-243: How was the threshold for TEASER++ chosen? This intuitively might have an impact on the results
- L252-253: "effect of reducing viewpoint overlap between the image sets of reference and target objects"
- L279-280: "a subsequent K-means cluster refinement approach" (no details on the category discovery seem to be provided in the Supp.)

### Suggestions/general questions
- What are the most important factors that produce the speed-up w.r.t. to [17] mentioned, e.g., on L75? Intuitively, by having to compute correspondences w.r.t. multiple instead of a single reference view, one would think that your method should if anything be slower than [17]
- The resizing results in the object being "stretched" in some reference views. Did you evaluate if this has an effect? Did you consider using, e.g., padding, to keep the object aspect ratio fixed, and see if this has an influence on the results?

### Typos, language and bibliography
- Figure 1, caption: Extra space before "10 s"
- L41-43: Repetition of "leverage"
- L99: Extra space before "15 object"
- L122: $\mathbb{T}_{\mathcal{R}_{1:N}}$ -> $\mathbb{T}_{\mathcal{T}_{1:N}}$
- L157: Dashes should be em dashes
- L179 and 239: "Certain" -> "A selection of the"?
- L180: "show" -> "shown"
- L188: $d$ is previously (Sec. 3.1) and later (Sec. 3.5) a function of feature maps $\Phi$, but here is defined as a function of descriptor indices
- Eq. 1: $\mathbf{u}_k$ and $\mathbf{v}_k$ are here written in bold letters, whereas the correspondences are indicated in the previous lines in the text as $u_k$ as $v_k$. There is a missing step to describe how to obtain $\mathbf{u}_k, \mathbf{v}_k$ from $u_k, v_k$
- L224-225 and Eq. 2: The meaning of the symbols $F$, $G$, and $x$ should be explicitly introduced in the text
- Eq. 2: For consistency with the previous notation (see Sec. 3.2), $\phi$ should be replaced with $\Phi$ or otherwise explained in the text
- Eq. 2: The brackets around the summation could be removed. Otherwise, I suggest adapting them so as to better fit their content ("\left(" and "\right)")
- L257-258: "not only serves [...], but also have" -> "not only serves [...], but also has" or "serves not only [...], but also _to_ have"
- L258: Unnecessary comma after "time"
- L259: "30 view" -> "30-view"
- L301: "work;" -> "work."
- L334: References [10], [27] and [39] are lacking their venue
- L339: "theWild" -> "the Wild"
- L341 (Supplementary): Missing brackets around "Section 4.3"
review_title: null

Meta-review: metareview: Reviewers felt positively that the proposed task is important and challenging and the proposed method is reasonable.  The reviewers also felt that the results are very nice:
- The dataset used for evaluation is challenging
- The performance is better than baselines in terms of speed and accuracy
- The results show nice generalization
- Nice ablations on dimensionality reduction
- The method runs in real-time
- The paper includes real-robot experiments 
Reviewers also felt that the paper is well-written and visualizations are clear and informative. Reviewers appreciated the response in the rebuttal and the subsequent improvements to the paper.

At the same time, reviewers felt that the paper had some weaknesses, especially a lack of comparison to more baselines (such as a supervised category-level pose estimator [6, 8, 14]).  Although the setting is different (one-shot vs training on a full dataset), it would still be helpful to show the gap in performance between such methods to get a sense of how far the performance of the proposed method is compared to methods that can train on more data. Other useful baselines include TEASER++ or a baseline that uses NOCS or NUNOCS for category-level generalization.

It would also be helpful to show the performance as the number of views at test-time is varied below 10. Reviewers also requested to see more qualitative examples of the pose estimates. Finally, reviewers would like to see a more clear explanation of how the ground-truth pose 7D pose is defined across instances of a category. 

The paper would be greatly improved by including these modifications in the final version of the paper.

confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)