DenseMatcher: Learning 3D Semantic Correspondence for Category-Level Manipulation from a Single Demo

Authors: Junzhe Zhu, Yuanchen Ju, Junyi Zhang, Muhan Wang, Zhecheng Yuan, Kaizhe Hu, Huazhe Xu
Primary area: applications to robotics, autonomy, planning
Keywords: robotics, correspondence, computer vision, 3D vision

Abstract:
Dense 3D correspondence can enhance robotic manipulation by enabling the generalization of spatial, functional, and dynamic information from one object to an unseen counterpart. Compared to shape correspondence, semantic correspondence is more effective in generalizing across different object categories. To this end, we present DenseMatcher, a method capable of computing 3D correspondences between in-the-wild objects that share similar structures. DenseMatcher first computes vertex features by projecting multiview 2D features onto meshes and refining them with a 3D network, and subsequently finds dense correspondences with the obtained features using functional map. In addition, we craft the first 3D matching dataset that contains colored object meshes across diverse categories. We demonstrate the downstream effectiveness of DenseMatcher in (i) robotic manipulation, where it achieves cross-instance and cross-category generalization on long-horizon complex manipulation tasks from observing only one demo; (ii) zero-shot color mapping between digital assets, where appearance can be transferred between different objects with relatable geometry. More details and demonstrations can be found at https://tea-lab.github.io/DenseMatcher/.

Decision: accept (spotlight)
Reviewer scores: 10, 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1730020844664

thread_type:
official_review

<thread_object>
creator: reviewer_se4d

<content>
summary:
Summary: This paper introduces DenseCorr3D, a 3D matching dataset featuring colored meshes and dense correspondence annotations. It addresses the limitations of existing datasets that predominantly emphasize geometry. The authors propose DenseMatcher, a model that integrates 2D foundation models with 3D networks to significantly enhance dense correspondence accuracy. The effectiveness of DenseMatcher is demonstrated through applications in robotic manipulation tasks and color transfer experiments.

strengths:
Strengths:

The authors have developed a dataset that is a valuable resource for the research community.

Despite its straightforward pipeline and principles, the proposed DenseMatcher effectively extracts semantic maps that facilitate subsequent tasks.

The introduction of the function map is promising, and the correspondence video demo on the accompanying website is impressive.

weaknesses:
Weaknesses:

The range of tasks and the diversity of object categories provided in the dataset are limited.

Line 853 mentions the total time expenditure without delving into specific details, such as the time required for rendering images, particularly the computation comsumption of the function map.

The paper lacks an ablation study for the DINO and SD components. Previous zero-shot methods shows that the features provided by SD VAE may not be optimal. An ablation analysis for the feature backbone should be included in the experimental tables.

There is no discussion on whether the model incorporates augmentations for the pose of the mesh. Research has shown that semantic features can easily overfit to spatial position-related scenarios. If the input mesh's position changes, the resulting semantic map may become inaccurate. Therefore, it would be beneficial to include experiments that apply random rotations to the mesh as input.

questions:
Additionally, it would be constructive to present examples of failure cases to provide a more comprehensive evaluation.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730619467809

thread_type:
official_review

<thread_object>
creator: reviewer_jn9k

<content>
summary:
This paper studies the problem of dense surface-point matching between objects, where similarity is understood as a user-defined semantic and matches can be between objects of the same, but also different category. The contributed method combines features that encode the visual appearance with features that encode local geometry. The method is evaluated and compared against baselines on a self-created dataset and real-world robotic imitation of human demonstrations.

strengths:
- The authors claim (and I am not aware otherwise, but also not super familiar with this subfield) to contribute the first method for 3D dense correspondences that combined visual appearance and geometric information. This very intuitively makes sense and makes especially the contributed dataset something that can have profound impact on the research on 3D correspondences.
- The method has directly been evaluated in a real-world application of mimicing human demonstrations with a robotic manipulator.
- The paper is very well written (the best in my review batch) and easy to follow.

weaknesses:
- The experimental evaluation is limited to a self-contributed dataset and very few qualitative runs on a robotic application (where it is unclear if the method difference is statistically significant).
- The method design contains a couple of non-straightforward design choices without justifications or experimental evidence to back up these choices [**update from discussion with authors: these points are mostly addressed now**]:
  - Using the XYZ coordinates of the mesh vertices makes the method sensible to random transformations on the input mesh. There is no experiment evaluating whether the model is able to learn invariance over such random coordinate system changes.
  - The choice of negative cosine similarity in $L_\textrm{semantic}$ is quite particular. The authors do not explain why they would choose this over e.g. L1 or L2 distances and also do not ablate this choice.
  - Similarly, for $L_\textrm{preservation}$, the choice of a single linear layer for reconstruction might hinder the encoder network to learn a more useful non-linear function. The more standard choice would probably be to mirror the encoder architecture like in an autoencoder, but this is neither discussed nor evaluated.
- The method requires supervised training with an expensive 3D annotation workflow.

questions:
- Section 4.1: I am not super familiar with the prior work on 3D dense matching, but this optimization formulation seems computationally expensive and as Section 4.4 shows also unstable. Why are other assignment and matching methods not compared as beaseline or ablation? e.g. Hungarian matching or the double-softmax used in [1]?
- line 200: The requirement of textured 3D assets is very limiting. It seems to me the method could also work from an untextured geometry asset and posed images, or am I missing something?
- line 242: Since the negative cosine distance is such an odd choice I suspect the authors were inspired here by related work? In that case it would be important to attribute this here with a reference.
- line 252: "object type and material" is misleading. Neither one of the frozen backbones captures this information, both are self-supervised encoders of visual appearance that might correlate with this information in some cases.
- line 254: What norm is used in the equation for $\mid\mid \cdot \mid\mid$? Why is that one choosen?
- Table 1: Please explain better the different ablation variants. Is "w/o Diffusion Net" directly matching the concatenation of $f_\textrm{multiview}$ and the HKS features? Or is it also using the XYZ features and therefore failing because of coordinate system change? 
- Section 6.2.3: I dont't think the comparison to Robo-ABC is entirely fair. It would be good to show both variants, with the full affordance memory and with the reduced form that is currently presented. The proposed method is very expensive in terms of the 3D data it requires, so really it needs to show that this additional information can compete with methods that are only based on cheaper and more abundant image data.
- Section 6.2.4: How is success determined in the experiments? Given the low number of overall trials, what level of statistical significance does the experiment currently have?



[1] Lindenberger, P., Sarlin, P.-E., & Pollefeys, M. (2023). LightGlue: Local Feature Matching at Light Speed. Retrieved from https://openaccess.thecvf.com/content/ICCV2023/html/Lindenberger_LightGlue_Local_Feature_Matching_at_Light_Speed_ICCV_2023_paper.html
</content>
</thread_object>

</thread>
<thread>
creation date:
1730685714626

thread_type:
official_review

<thread_object>
creator: reviewer_iy2b

<content>
summary:
This paper introduces DenseMatcher, an innovative method for computing dense 3D correspondences between objects with similar structures, geared towards applications in robotic manipulation. They propose that *semantic correspondence*—which aligns semantically similar parts across objects—provides more powerful generalization capabilities across categories compared to *shape correspondence*, which mainly focuses on geometry.

To facilitate the training and evaluation, they created *DenseCorr3D*, a new dataset comprising 589 colored object meshes across 23 categories, with dense correspondences organized into semantic group. DenseMatcher utilizes pre-trained 2D foundation models to extract multiview features, which are further refined using DiffusionNet. The enhanced features are then used to establish dense correspondences through a functional map. 

They provide comprehensive experiment results to demonstrate DenseMatcher’s effectiveness in 3D dense matching, zero-shot robotic manipulation, and color transfer tasks. DenseMatcher outperformed baseline methods on the DenseCorr3D benchmark and achieved a 76.7% success rate in real-world robotic manipulation, showcasing its robust generalization capabilities.

strengths:
- Integration of 2D and 3D: DenseMatcher effectively combines 2D foundation models, like SD-DINO, for multiview feature extraction with DiffusionNet to refine features with geometry. This fusion enhances semantic understanding and generalizability in 3D correspondence.

- New 3D matching dataset: The authors introduce DenseCorr3D, the first dataset with colored meshes and dense correspondences, featuring 589 textured meshes across 23 categories. It advances research by supporting methods that account for both appearance and geometry.

- Enhanced functional map for accuracy: A novel regularization scheme promotes sparsity in DenseMatcher’s functional map, achieving a 43.5% accuracy improvement over baselines.

- The paper is well-written and easy to understand. The experiment results are comprehensive and promising.

weaknesses:
- Limited analysis on varying topologies: While they analyze that previous methods struggle with different topologies, they do not deeply explore DenseMatcher's robustness on diverse object structures.

- Limitation to severe occlusion: The paper does not address how DenseMatcher handles significant occlusion. Since it relies on multiview feature extraction and functional maps, both susceptible to occlusion, further analysis of this limitation would strengthen the evaluation.

questions:
- Performance on Varying Topologies: How does DenseMatcher perform with objects of varying topologies? Are there specific object structures or topological variations where its performance significantly degrades?

- Handling Severe Occlusion: Is DenseMatcher able to be adapted or extended to handle severe occlusion more effectively? What potential modifications could mitigate its reliance on multiview feature extraction and functional maps in such cases?

- More Benchmark Validation: Are there any benchmarks or experiments that could further validate DenseMatcher’s robustness against topological diversity and occlusion? How might these additional evaluations impact its overall effectiveness and applicability in real-world scenarios?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730742622450

thread_type:
official_review

<thread_object>
creator: reviewer_w2z1

<content>
summary:
This paper proposes a framework and dataset for category-level object 3D dense matching. The DenseMatcher utilizes a 2D foundation model with 3D network refinement to reach generalization and 3D understanding. The author conducts robotic manipulation and zero-shot color mapping to validate the findings.

strengths:
1. This idea is novel and underexplored in relevant areas, especially in robotic manipulation learning. Instead of simply augmenting the data with numerous demos, this paper can address sample efficiency by embedding semantic information.

2. This paper's writing style is straightforward, and it is easy to catch the main topic.

3. Utilizing the existing 2D network (DINO in this paper) with 3D networks is a simple but promising approach.

4. Experiments can thoroughly reflect the model's ability. In robotic manipulation tasks, it covered pick-and-place, long-horizon, and dual arm.

weaknesses:
1. The statements of regularization terms in the methodology part are unclear and may cause ambiguity.

2. Some experiment details, like the description for each task, can be placed in the appendix and give a more precise visualization. The images in the robotic manipulation task are too undersized.

questions:
1. In Sec 4.1 Preliminary, Functional Map, please give a detailed justification about how to regularize the term C as isometric in your context. 

2. In the appendix, please provide a detailed explanation, with proofs, showing how previous constraint terms ensure that the output is minimized in the semantic distance function.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732546603994

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
# General Response

We thank the reviewers for their insightful comments and recognition of our work's strengths:

- **Novel Contribution:** Reviewers highlighted our work as "novel and underexplored," particularly in robotic manipulation learning (w2Z1), and the design of functional map is also noted as a "novel regularization scheme" (iy2b).
- **Effective Integration of 2D and 3D Models:** Our approach of combining 2D foundation models with 3D networks was praised as enhancing "semantic understanding and generalizability" (iy2b) and as a "simple but promising approach" (w2Z1).
- **Valuable Dataset Introduction:** The dataset we developed was recognized as a "valuable resource for the research community" (SE4d) and for "advancing research by supporting methods that account for both appearance and geometry" (iy2b), "profound impact on the research on 3D correspondences" (Jn9K).
- **Clear Presentation:** The paper was commended for being "straightforward and easy to catch the main topic" (w2Z1), "very well written and easy to follow" (Jn9K), and "well-written and easy to understand" (iy2b).
- **Strong Experimental Validation:** Reviewers appreciated that our experiments "thoroughly reflect the model's ability," covering various tasks (w2Z1), and demonstrating effectiveness in "zero-shot robotic manipulation" (iy2b). The real-world application and the "impressive" video demo were also noted (Jn9K, SE4d).

------

We have uploaded the detailed explanations requested by the reviewers, along with additional experiments, including the results of real-world robot experiments. We hope these experiments address the reviewers' concerns. All revisions are highlighted in **red** in the updated version.
- **Model performance for more diverse topologies(iy2b-Q1; Figure 3, Figure 4, Table 6):** Added figures to show qualitative results and training data examples. We also provided numerical results in the Appendix Table 6. 
- **Proofs and derivations of constraints for functional map(w2Z1-Q1, Q2; Section 4.1, Appendix A.5):** provided highly detailed derivations in appendix A.5 and referenced it in Section 4.1.
- **Justification of cosine similarity loss(Jn9K-Q2(2), Q6; Section 4.3.1, Appendix A.5.2):** proved in A.5.2 that after minimizing the cosine similarity objective described in 4.3.1, solving functional map is equivalent to minimizing total $D_\text{semantic}$ between matched vertices.
- **Explanation of ablation variants(Jn9K-Q9; Table 1, Section 6.4):** Referenced Section 6.4 in Table 1's caption. 
- **Comparison with Robo-ABC:(Jn9K-Q10; Secion 6.2.2 and Table 3):** Supplemented with experimental results of two variants of Robo-ABC.
- **Reorganize the paper structure(w2Z1-Q3; Line 789 and Figure 8):** Placed the task description section in the appendix and re-uploaded a clearer Figure 8.
- **Comparison with Hungarian matching(Jn9K-Q4; Secion 6.5 and Figure 11):** added a paragraph to explain the advantage of functional map and added a figure to compare results with Hungarian matching. 
- **Diversity of objects(iy2b-Q3, SE4d-Q1; Appendix A.2.1 and Table 4):** added to our dataset and updated the table to include more details about the diversity of our dataset.
- **Annotation Cost(Jn9K-Q3; Appendix A.2.4 and A.2.5):** added details about the annotation process and time consumption.
- **Training details(Jn9K-Q2, SE4d-Q4; Appendix A.4.2):** elaborated training procedure and augmentations details.
- **Runtime Analysis(SE4d-Q2; Appendix A.4.3 and Table 5):** provided detailed runtime analysis of our method and baselines
- **Handling Severe Occlusion(iy2b-Q2&Q3; A.6, Figure 12, and Fig13):** Supplemented visualizations of robotic and matching experiments under different severe occlusion conditions.
- **Examples of failure cases(SE4d-Q5；Our website):** Uploaded a failed robotics experiment video.
- **Update L2 norm notation(Jn9K-Q8)**: changed the notation of L2 norm from $|| \cdot ||$ to $|| \cdot ||_2$

------

Below, we address a common question raised by multiple reviewers, while detailed responses to specific reviewers are provided in separate posts.

**Common Question (iy2b-Q3, SE4d-Q1): Diversity of objects**

Appendex A.2.1 and Table 4 show that our dataset contains daily object categories spanning nearly two dozens of fruits & vegetables, vehicles, animals, backpacks, tools, and toiletries. Some categories actually contain more sub-types that were not listed(for example, “animals” contain 9 distinct species such as elephant, giraffe, cat, deer, dinosaur, etc). In addition, we have added a chairs category. We haved updated the category list in Appendix A.2.1 to include object subtypes.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces DenseMatcher to compute dense surface-point matching between objects, aiming at robotic manipulation applications. This proposed method significantly enhances semantic understanding and generalizability in dense 3D correspondence tasks, thereby improving the downstream manipulation performance. This paper is well written, supported by novel ideas, adequate experiments, and tangible contributions (dataset, method). Therefore, I recommend accepting this paper. I encourage authors to incorporate reviewers' comments in the final revision.