DenseMatcher: Learning 3D Semantic Correspondence for Category-Level Manipulation from a Single Demo
Authors: Junzhe Zhu
Yuanchen Ju
Junyi Zhang
Muhan Wang
Zhecheng Yuan
Kaizhe Hu
Huazhe Xu
Venue: ICLR 2025 Spotlight (2025)
Keywords: robotics
correspondence
computer vision
3D vision
Primary area: applications to robotics, autonomy, planning
TL;DR: We develop a dataset and a model for dense 3D correspondence on colored meshes, and perform robotic manipulation and color transfer experiments.

Abstract: Dense 3D correspondence can enhance robotic manipulation by enabling the generalization of spatial, functional, and dynamic information from one object to an unseen counterpart. Compared to shape correspondence, semantic correspondence is more effective in generalizing across different object categories. To this end, we present DenseMatcher, a method capable of computing 3D correspondences between in-the-wild objects that share similar structures. DenseMatcher first computes vertex features by projecting multiview 2D features onto meshes and refining them with a 3D network, and subsequently finds dense correspondences with the obtained features using functional map. In addition, we craft the first 3D matching dataset that contains colored object meshes across diverse categories. We demonstrate the downstream effectiveness of DenseMatcher in (i) robotic manipulation, where it achieves cross-instance and cross-category generalization on long-horizon complex manipulation tasks from observing only one demo; (ii) zero-shot color mapping between digital assets, where appearance can be transferred between different objects with relatable geometry. More details and demonstrations can be found at https://tea-lab.github.io/DenseMatcher/.

Published as a conference paper at ICLR 2025
DENSEMATCHER: LEARNING 3D SEMANTIC CORRE- SPONDENCE FOR CATEGORY- LEVEL MANIPULATION FROM A SINGLE DEMO
Junzhe 2 22.5 Yuanchen Ju Junyi Zhang 1.7, Muhan Wang! Zhecheng Yuan 13,66 Kaizhe Hu 33.63.6 Huazhe X,,1,3,6
IIs, Tsinghua University TTeaa Inc. Stanford University
Shanghai Qi Zhi Institute Shanghai Jiao Tong University
4UC Berkeley
"Shanghai AI Lab
(a)
(b)
Figure (a) Zero-shot color transfer between 3D assets. (b) In real-world robotic experi- ments, we use DenseMatcher to transfer a manipulation sequence to the robot from a single human demonstration. Circles represent the contact points in the human demo grasping points for robot manipulation.
ABSTRACT
Dense 3D correspondence enhance robotic manipulation by enabling the gen- eralization of spatial, functional, and dynamic information from one object to unseen counterpart. Compared shape correspondence, semantic correspon- dence more effective in generalizing across different object categories. To this end, present MenseMatcher, method capable of computing 3D correspon- dences between in-the-wild objects that share similar structures. DenseMatcher first computes vertex features by projecting multiview 2D features onto meshes and refining them with a 3D network, and subsequently finds dense correspon- dences with the obtained features using functional map. In addition, we craft the first 3D matching dataset that contains colored object meshes across diverse categories. We demonstrate (i) robotic manipulation, where it achieves cross-instance and cross-category gencralization on long-horizer horizon complex manipulation tasks from observing only demo; (ii) zero-shot color mapping between digital assets, where appearance can be transferred between different objects with relatable geometry. More de- tails and demonstrations can be found at https: //tees///eaaaeeeee lab. github. io/ Chese/
downstream effectiveness of Matherrtcher
INTRODUCTION
Correspondence plays pivotal role in robotics Wang (2019). By establishing correspondences, can enable the robot to identify semantically similar components between two objects, which is cru- cial for various day-to-day manipulation tasks. For instance, in robotic assembly, necessary to
"qqual contribution, Corresponding author.

---

Published as a conference paper at ICLR 2025
Figure 3: Predicted correspondences on few-shot categories. DenseMatcher can generalize across diverse topological variations, given only training examples per category. To ensure that the model is not reliant on canonical spatial poses, we randomly rotate the mesh before the test procedure.
determine the corresponding parts between the target and source objects. Furthermore, recent stud- ies Ju et al. (2024); Kuang et al. (2024) illustrate the capacity to infer the affordances of previously unseen objects through correspondences with known reference.
Correspondences can be classified along two axes: density and dimensionality. In 2D scenarios, sparse correspondence focuses on matching limited set of keypoints, while dense correspondence takes spatial proximity into account and aligns every pixel between images. Similarly, in 3D, sparse methods align key points of point clouds or meshes, whereas dense correspondence considers the entire structure for alignment.
Among these types, 3D dense correspondence is particu- larly advantageous for robotic manipulation, it ensures continuity by smooth mappings between surfaces. This crucial for tasks requiring precise multi-poin contact and positioning. Additionally, 3D correspondence avoids common 2D ambiguities like distortions from changing viewpoints or occlusions, enhancing a robot's ability to interact accurately with real-world objects.
The 4 types of correspon- dence. The reference image is on left, while the right side demonstrates 1) 3D dense, 2) 3D sparse. 3) 2D dense, and 4) 2D sparse correspondences.
Figure 2:
However, existing datasets and methods for 3D dense cor- respondence (Pratikakis et al., 2016; Dyke et al., 2019; Bogo et al., 2014;; Zuffi et al., 2017; Halimi et al., 2019;
Hedlin et al., 2023; Groueix et al., 2018b) often focus on geometry and ignore textures or color information. This limits the ability of models to effectively combine appearance and geometry information, both of which are essential for semantic understanding. In addition, they typically only contain single or few categories (c.g. humans, four-legged animals), which further limits the generalization ability of models. As result, prior methods generating dense 3D features can be divided into two categories: (1) 3D networks that only utilize geometry information and are trained on aategory-specifie datasets (Cao et al., 2023; Halimi et al., 2019), which do not generalize well to unseen objects, (2) models that naively average multiview appearance features from frozen 2D networks (Dutt et al., 2024) and do utilize any geometry information, which suffer from noise due varying visibility and pixel coordinates for each vertex, and lack global 3D consistency.
To address this, we release Cor3e,,, the first 3d matching dataset containing colored meshes with dense correspondence annotations, with 600 densely annotated assets across 24 categories. In addition, we develop DenseMatcher, a model framework that combines both the powerful gen- eralization capability of 2D foundation models with the geometric undertanding of 3D networks. DenseMatcher first computes per-vertex mesh features by projecting multiview features from 2D foundation models onto 3D meshes, before refining them with lightweight 3D network. It then calculates dense correspondences using the refined features via functional map, which we improve with several novel constraints.
We further demonstrate the downstream effectiveness of DenseMatcher by performing complex long-horizon robotic manipulation experiments based only a single demonstration of hand- object interaction. Finally, we further showcase the quality our correspondence by presenting several examples of color transfer from mesh to another without any additional supervision.
In summary, we make the following contributions: (i) novel 3d matching dataset that remedies the lack of texture information and categories in previous datasets, (ii) 3D dense correspondence model framework that bridges the gap between 2D and 3D neural networks (iii) comprehensive 31) matching, robotic manipulation, and color transfer experiments.
2

---

Published as a conference paper at ICLR 2025
Figure 4: Semantic group annotations examples of apple, banana, animals (deer, tiger. elephant), and chairs. Different colors represent different semantic groups across the same category. DenseC- orr3D contains objects of varying topologies and structures, both across and within categories.
2 RELATED WORKS
3D Correspondence. 3D correspondence, or shape correspondence, which focuses on establish- ing meaningful correspondences between shapes surfaces, be categorized into two main ap- proaches: deformation-aee bbased correspondence and mapping-based correspondence. Deformation- based methods focus tracking object its deformed version, often using canonical template. While simple and direct, these methods do not apply to surfaces under non-isometric transforma- tions (Groueix al., 201 8a;b; Luiten al., 2024). On the other hand, mapping-based methods such as functional map (Ovsjanikov et al., 2012) establish continuous mappings between two arbitrary surfaces, often leveraging spectral features such as Laplace- Beltrami eigenfunctions (Donati al., 2020; Roetzer & Bernard, 2024). However, most prior approaches focus on shape features and de- pend on carefully designed geometric descriptors like Wave Kernel Signature (WKS) (Aubry al., 2011), 1), or deep features learned from untextured shapes (Cao & Bernard, 2022; Cao et al., 2023), while ignoring semantic relationships between objects.
Laplace-Beltrami
Recently, powerful 2D foundation models such DINO (Oquab et al., 2023; Caron et al., 2021) and Stable Diffusion (Rombach et al., 2021) have enabled deep feature- 2D semantic correspon- dence (Amir et al., 2021; Zhang et al., 2023; Tang et al., 2023; Luo et al., 2024), offering powerful representations extendable to 3D. In particular, Diff3F Dutt et al. (2024) projects such 2D features onto 3D shapes and performs averaging across views. However, Diff3F focuses untextured shapes and additionally does not incorporate shape information, resulting in noisy and inconsistent 3D fea- tures. Our method addresses this by adding 3D neural network, DiffusionNet (Sharp al., 2022), to refine 2D features with 3D geometry, producing spatially consistent and informative features.
Semantic Correspondence for Robotics. Semantic correspondence helps robots to understand and reason about the relationships within scene. Florence al. (2018) utilizes correspondences to map human actions to robots. Recent work Ju et al. (2024) develops a method for few-shot transfer of affordances by querying retrieved objects, and Kuang al. (2024) extends (2023) infers poses from detected point cloud keypoints for transfering grasps to similar objects with arbitrary poses. Yuan et al. (2024) proposes a multi-view contrastive objective to capture the correspondence under different viewpoints. Notably, leveraging semantic correspondence obviates the need for collecting large amounts of demonstrations (Wang et al., 2024; Ze et al., 2024; Wang et al., 2023; Hu et al., 2024; Chi et al., 2024). Although certain approaches require only a single Liu et al. (2024) or zero demonstrations, they often cannot generalize across diverse object instances and categories.
3D. Xue et al.
3

---

Published as a conference paper at ICLR 2025
3 TASK: DENSE 3D MATCHING FOR TEXTURED OBJECTS
3.1 SEMANTIC GROUPS
To our knowledge, all previous benchmarks Pratikakis et al. (2016); Dyke et al. (2019); Bogo et al. (2014b); Zuffi et al. (2017); Bronstein et al. (2009) on 3d matching focus on category-specific syn- thetic shapes (e.g. humans, animals) with well-defined vertex-to-vertee correspondences, and lack crucial generalizability to daily objects. None them includes texture/color information. To remedy this, we develop semantic groups formal framework for defining category-level 3D correspon- dence, and release the first 3D matching dataset with textured assets.
The definition of correspondence is inherently subjective. For instance, elephant tusks and rhino tusks can corre- spond based on function, while an elephant's nose can correspond to rhino's tusk based on location. We for- malize this with semantic groups. we group the vertices of each mesh into unique seman- tic groups, where the group index of os is denoted n(us) and its group is defined as the set G. == {wy n(0y) n(u;)}. Meshes of the same category should share the same semantic groups, with vertices in the same group having the same learned features, while distinct groups have different features. As illustrated in Figure 5, the partitioning rules for semantic groups can be user-defined. Symmetric objects, like circular strip around an apple, form single semantic groups, while mirror-symmetrie but distinguish- able features, like cat ears, may belong different groups.
shown Figure 4,
Figure Two possible partitioning schemes for a hand are shown.
4 DENSEMATCHER MODEL
4.1 PRELIMINARY
Functional map (Ovsjanikov et al., 2012) is commonly used for dense 3D correspondences in syn- thetic meshes but is novel in robotics. We follow the notation from Nogneng & Ovsjanikov (2017) to introduce its formulation. Given source mesh M and target mesh N with aM and nn vertices, vertex features f E Ra x and g E IR" "yy xdua, and diagonal vertex area matrices AM E R X and AN E Raw xnx we compute the first & eigenfunctions of Saplace-Beltrant operator as a set of spectral bases &x E R and nN E Bey xk analagous sine waves in ID. Multiplication with these basis or their ascudo-inverses + = o A"A projects functions from the spectral domain to the manifold and back. The map from M to N is represented by sparse binary matrix II X ensuring 9 R ff for corresponding vertices.
Since II is large and takes combinatorial time solve for, functional map approximates II with low-rank representation II = x C' t where C' E R k is the functiona map matrix that we wish to find. We can translate the feature constraint to:
g x [T x C
N
GeR
FERExáge
where G and F are low-dimensional projections of and f onto the eigenfunction basis. Constraints have been proposed to regularize C. Ovsjanikov et al. (2012) shows that if C' is iso- metric, it should commute with the Aaplace-Beltrami operator (i.e. left/rght multiplying diag- onal matrices of eigenvalues AN and AM with C should be equivalent). To ensure C' approxi- mates a point-to-point mapping, Nogneng Ovsjanikov (2017) enforces that C' commutes with point-wise multiplication operators of each feature channel p: (() & (iag( ( and (p) & Diag(g (P)) (
Combining these contraints with scaling factors a and 33 results in the overall optimization objective:
dus
Cop arg min||CF - - alIANC - CAM
Σ ( - yOC
(1)
p=1
isometry constraint: commutativity with Laplace Beltrani operator
point-to-point constraint: commutativity with product operator
4

---

Published as a conference paper at ICLR 2025
High-res Features
mullivvew
Input Mesh
Fastane
Project Averggs
Remesh
DiffusionNet
f
Youppu
Functional Map
Nomulize
PenUUp
Render
MLP
Fenture
Frozen
Trainable
Sinusoidal Encoding
Propagation
Figure 6: DenseMatcher model architecture. SD-DINO (Zhang et al., 2023) fuses 2D features from DINOv2 and Stable Diffusion, which aggregated and fed into trainable DiffusionNet. Correspondences are computed from source and target features using functional map.
The detailed derivations can be found in A.4 and Nogneng & Ovsjanikov (2017).
4.2
ARCHITECTURE
4.2.1
MULTI-VIEW FOUNDATION MODELS (FROZEN 2D "BACKBONE")
We first render multiple views of the 3D asset and compute a 2D feature map for each view using SD-DINO (Zhang al., 2023), which extracts features with DINOv2 and Stable Diffusion and combines them using the post- post-processing module of Zhang et al. (2024). We then use Featup (Fu al., 2024) upsample the combined feature map. For each vertex Us we retrieve feature each view by projecting it into 2D image coordinate and performing bilinear interpolation on the feature map. We then average features from all visible views, or the feature zero vector if the vertex cannot be seen from any view. We dub this aggregated multiview feature fmultivie E R68
4.2.2 DIFFUSIONNET REFINER (TRAINABLE 3D "NECK")
We remesh our 3D asset into 2000 vertices to obtain its simplified geometry. We incorporate geometry information by concatenating Smultiview with the Heat Kernel Signature descriptor(HKSS (Sun al., 2009) and sinusoidal positional encoding from Mildenhall al. (2021) of each vertex's 's XYZ position. We feed this into DiffusionNet (Sharp et al., 2022), a 3D architecture that alternates between MIP layers and surface feature propagation layers, which serves the only trainable part of our model. The resulting output 512-dimensional per-vertex feature fouipa (r:) E IR512 which
we then unit-normalize as f(u,) = footuu (v:)
foouuu 11)
4.3 Loss FUNCTION
Our loss function consists of two components: L Lsemantic + Lpreservation- The former ensures that features are similar across nearby semantic groups and distinct across distant groups, while the latter ensures the feature retains rich information learned by 2D foundation models.
4.3. SEMANTIC DISTANCE LOSS LSSMANTIC
We define the semantic distance between two vertices Dsemanti (1;, 02) as the average geodesic dis- tance between vertices in their semantic groups. The formal definition can be found Appendix 1.2.1. We design our semantic distance loss to enforce the 7.2 distance between features of any two vertices to scale linearly with their semantic distance. To achieve this, we randomly sample pairs of v., Uj the same mesh and across different meshes, and minimize the negative cosine similarity between (1:) f(u))) and Dremantic (Us, U1)) across sampled pairs of i,
En, Ilf(v:)) f(v,)1l2 D Deemantic (U', 0.))
Liemmntic cos((0)
Eas (1s(ss) ((1s)12 En, D Deemantic (Vi, 12
5

---

Published as a conference paper at ICLR 2025
We prove in A.4.2 that after minimizing this training objective, solving for functional map results in minimal overall seemantic ('mmtch(j)>>>> F2y) across all matched pairs.
4.3.2 FEATURE PRESERVATION LOss
We can view DiffusionNet refiner nonlinear operater embedding features from Suuliview into foutput- Semantic distance loss ensures the feature space f(v:) is equipped with metric that approximates semantie. but might lose other information learned by 2D foundation models such as object type and material. Therefore, we train a linear layer to approximately invert DiffusionNet and reconstruct F m thereby preserving the rich information learned by SD-DINO:
IVII
Pereservatiin
Σ fmuttiview (w;) - Soupuu (1;)ls
where W E 768x512 is a learnable back-projection matrix that we optimize together with our DiffuusonNeet parameters.
4.4 IMPROVED FUNCTIONAL MAP
After obtaining the vertex features on pair of meshes, we calculate dense correspondences between them with functional map. Most previous methods using functional map focus on specific shape categories with distinct local geometry such as humans or four-legged animals, and used shape features HKS and WKS. Our approach, however, handles diverse array of daily objects such as fruits and jugs, which lack distinguishable local features. Despite our learned semantic features, we still observe that the objective of equation 1 insufficient. In particular, the lack of unique features and large deformations causes IIf to admit solutions where II is not sparse, leading to noisy correspondences. We therefore propose to add two extra regularization terms:
(1) We clamp the recovered point-to- point-to-point mapping matrix Cx between [0, 1]: N. max(0, min(1, II;;)) and penalize its entropy to promote sparsity:
AN
aM
H log Il.
(2) We enforce that each row of II sums to and each column sums to nx so that II a soft assignment matrix:
wMs
Σ T. 1) +
Σ
VAN
TMM
We scale those to terms and add them to the cost function in equation 1. The detailed optimization
procedure can be found in A.2.2.
THE DENSECORR3D DATASET AND BENCHMARK
To remedy the lack of textured data for the 3D matching task, we first filter Objaverse-XX XI. (Deitke ct al., 2023) and OmniObject3D (Wu et al., 2023) into 600 instances across 24 categories and split each into train, validation, and test, as specified in Tab.4. For categories with insufficient meshes, we skip the train and validation splits and use those as out-of-distributiin training samples.
5.1 FILTERING, ANNOTATION AND FORMAT
For fruits and vegetables, source meshes from Objaverse-XX XL. We label landmark points mesh surfaces, and interpolate them with separate algorithms for each category to acquire ground- truth semantic groups. (See Appendix A.1.2 and A.1.3 for details).
For other daily object categories, we pick assets from OmniObject3D. We use Blender's Vertex Brush functionality directly label all vertices in each semantic group.
Each instance contains the following: (i) original colored mesh (ii) remeshed geometry without texture, (iii) ground-truth semantic groups, and (iv) geodesic distance matrix for remeshed geometry

---

Published as a conference paper at ICLR 2025
Table 1: Performance comparison on DenseCorr3D shape matching benchmark. We report the results on both the full test set and the daily subset. Ablation studies are listed in Section 6.4.
All
Daily
Fruits & Veges
Methods
AUC
Err 4
AUC
Emr +
AUC
Err 4
ConsistFMap (DenseCorr3D (Cao & Bernard, 2022)
0.568
6,42
0.652
5.06
0.5404
6.87
URSSM (FAUST) (Cao et al., 2023)
0.485
8.19
0.602
5.90
0.445
8.96
URSSM (DenseCorrD) (Cao et al., 2023)
0.553
6.74
0.678
4.68
0,512
7.42
Diff3F (Dutt et al., 2024)
0.437
8.20
0.562
6.25
0.396
8.85
DenseMatcher (Ours) w/o Nef
0.737
3,49
0.725
3,71
0.740
3,41
0.708
4.02
0.726
3.75
0.702
4.11
w/o Preservation Loss w/o Constraint for FMap
0.603
5.38
0.651
4.93
0.587
5.54
0.706
3.84
0.729
3.72
0.698
3.88
5.2 EVALUATION CRITERIA
Given a pair meshes, we follow the convention of Cao et al. (2023); Cao & Bernard (2022) to compute the Normalized Geodesic Errors (Err) (Kim al., 2011), and the Atea-Under-Curve
(AUC) of the threshold-accuracy curve. Since our ground truth annotation is based on semantic groups, we need update the definition of matching distance be the distance of the predicted point to the nearest point the ground truth semantic group. We evaluate correspondence scores across all possible pairs within each category. For example, for categories with 6 test instances, predict correspondence for 6 pairs of instances.
6 EXPERIMENTS
We perform exhaustive evaluation across a spectrum of tasks, encompassing 3D Dense Matching, Color Transfer, and Zero-Shot Robot Manipulation. In addition, perform ablation studies individual components of our model.
6.1 3D DENSE MATCHING
6.1.1 BASELINES
We mainly compare with two training-based deep functional map methods, ConsistFMap (Cao & Bernard, 2022), and URSSM (Cao et al., 2023); and one 2d semantic feature-based method, Diff3F (Dutt et al., 2024). We mainly compare on our proposed DenseCorr3DDeenhharrk since our method requires texture as input.
ConsistFMapp (Cao & Bernard, 2022) utilizes cycle-consistency for robust multi-shape matching across shape collections, making it strong baseline in unsupervised shape matching. We evaluate its performance when respectively trained on FAUST (Bogo et al., 2014 4a) and DenseCorr3..
URSSM (Cao et al., 2023) is state-of-the- method which extends the functional map framework by coupling point-wise maps and functional maps during learning. We also evaluate both the version trained on FAUST (Bogo et al., 2014a) and on DenseCorr3D..
Diff3F (Dutt et al., 2024) projects 2D ceffusiinnbbased semantic features onto 3D shapes, focusing on semantic correspondence rather than purely geometric matching. We use the textured mesh as input to the diffusion model extract the semantic features. For details, refer A.3.1.
6.1.2 RESULTS
As shown in Tab. we found that our model achieves better AUC and Err compared to the baseline model. Additionally, due to the generalization capability of pre-trained 2D backbones, we achieve much higher accuracy on out-of-distribution test categories listed Tab. with zero training in- stances. We also observe surprising qualitative performance on categories with few examples, as shown in Fig. 3.
6.2 ZERO-So SHOT REAL WORLD ROBOTIC MANIPULATION
We create six real-world manipulation environments, exploring the performance of Mensheatthhe on daily life tasks by comparing the shape, size, material and category of the manipulated objects.

---

Published as a conference paper at ICLR 2025
Human Demo
Hand-Object Defector
Source Mesh
Target Mesh
AnyGrasp
Robot Manipulation
Densekka tchhe
Figure 7: Real-World Experiment Workflow. We obtain template mesh and contact points from a human demonstration with hand-object detector (Shan et al., 2020). We then transfer these contact points onto the target mesh with DenseMatcher. Finally, we use off-the-shelf AnyGrasp (Fang et al., 2023) to infer the grasping pose at contact point and proceed with the subsequent manipulation task.
Table 2: Task difficulty and classification of real world robot manipulation experiments. The classification of tasks is based on the differences between the objects manipulated in the human demonstration and manipulated by the robot. Here, "cross category" means template and target ob- jects are from different categories. "cross instance" means they are instances of the same category.
Cross
Cross Viewpoint
Cluttered Multiple Objets
Cross Category
Multiple
Long- Cross
Task Classification
Instance
Keypoints
term
Material
Peel a Banana
Flower Arrangement
Place Shoes
Decorate Chrismas Tree
Pull the Carrot
Point Object Parts with
Table 3: Real world robot manipulation experiment results. ory) and Robo-ABC* (with New Memory),
Robo-ABCt (with Original Mem-
Flower Arrangement
Place Shoes
Decorate Chrismas
Pull the Carrot
Object Parts
Task
Overall
Banana
Tree
with Pen
Robo-ABC
2/5
1/5
0/5
2/5
2/5
2/5
30%
Robo-ABC*
3/5
1/5
4/5
3/5
(enseMattcher(urr))
4/5
3/5
4/5
5/5
4/5
3/5
76.7%
Tasks difficulty and categorization are shown in Tab. 2. We RealSense L555 RGB-D camera and a URS robot arm to conduct all the real-world experiments. In this section, we use the term template mesh represent the mesh obtained from the human demo, and target mesh refer the mesh for robot manipulation.
6.2.1 1 GENERAL APPROACII
The workflow of the robotic experiment is shown in Fig. 7.
Obtaining Human Demonstrations. After recording RGB-D videos of human demonstrations, refer to the contact point collection process of VRB (Bahl et al., 2023) and Robo-ABC (Ju et al., 2024) to obtain contact points on the template mesh. For specific details, we recommend referring to the original papers. By using a hand-object detector Shan et al. (2020), we get the contact status between the hand and the object as well as their respective bounding boxes (bbox) each frame of the video. Then, in the contact frames, we sample the overlapping part of the two bboxes as the contact point. To avoid occlusion, we track the object and trace the contact points back to first frame, thereby obtaining the template keypoint on the template mesh
Contact Point Transfer. After obtaining the template mesh and keypoints, we calculate the dense descriptors for both the template mesh and the target mesh using DenseMatcher, and find dense mapping between their vertices using functional map with our proposed improvements. We transfer the keypoints through the dense mapping, thereby obtaining the grasp points the target mesh.
Grasp Pose and Post Grasp. After obtaining the grasp points, we use AnyGrasp to infer the corresponding grasp poses. We provide the waypoints of the trajectory after grasping and the final location to move to after completing the grasp. We use Movelt! (Coleman et al., 2014) to compute transformation from the target end-effector pose joint position trajectories.
8

---

Published as a conference paper at ICLR 2025
Figure 8: KeyFrames of 6 robotic tasks. The tasks from top to bottom are organizing the shoes, pointing object parts with pen, pulling out the carrot, putting flower into vase, peeling banana and decorating the the Christmas tree.
Source
Target
Transferred Color
Source
Target
Transferred Color
Figure 9: Color transfer results between (i) banana and eggplant, (ii) tomato and kabocha squash, and (iii) wine bottles (iii) gloves.
6.2.2 BASELINE
Robo-ABC (Ju et al., 2024) utilizes correspondences found in RGB images to transfer affordances. Since Robo-ABC has its collected affordance memory, compared two variants: with full memory capabilities and another where Robo-ABC's affordance memory only allowed to be collected from the corresponding human demos provide, while keeping Robo-ABC's original framework intact.
6.2.3 ROBOT MANIPULATION RESULTS
In Tab. 3, we compare the success rate of Robo-ABC with our method in the real world, and use task success rates the evaluation metric. For each task. measure the task success rates five trials. For tasks involving multiple objects and multiple keypoints, we calculate the success rate for each keypoint and object separately and determine the average success rate.

---

Published as a conference paper at ICLR 2025
Source
Ours
HKS
WKS
Source
Ours
HKS
WKS
(a) Dense correspondence results obtained using HKS and WKS features.
Functional Map (Ours)
Hungarian Matching
Nearest
Functional Map (Ours)
Hungarian Matching
Nearest Neighbor
Source
Source
Neighbor
(b) Dense correspondence results obtained with the same features but different matching methods.
Figure 10: Ablation study on dense correspondence results, (a) Effect using different features
(HKS, WKS) with functional maps. (b) Comparison of matching methods using the same features.
6.3 COLOR TRANSFER EXPERIMENTS
Ofri-Amar al. (2023) shows that dense correspondences can be used transfer object appeare- ances in 2D images. In Fig. 9, we show that our 3D dense correspondence scheme can transfer colors on 3D assets without any additional effort. To our knowledge, this has not been achieved before in the 3D generation literature.
Given pair of textured meshes and their corresponding simplified meshes, first color the vertices of the simplified mesh by copying the color from their nearest neighbor on the textured mesh. We then find the point-t--point mapping using DenseMatcher, and directly transfer the color over the corresponding vertices.
6.4 COMPARISON WITH SHAPE DESCRIPTOR FEATURES AND ABLATION STUDIES
As shown in Fig. 10a, we compare functional map outputs using our features to HKS and WKS, shape descriptor features commonly used by prior methods. As can be seen, the mapping obtained with our method significantly outperforms baselines terms of accuracy and continuity.
As shown in Tab. 1, we perform several ablation studies by (i) skipping DiffusionNet and directly feeding normalized fuultviww into functional map (ii) training our model without loss Lpreservatio- and comparing the difference in evaluation results (iii) removing the proposed entropy penalization and "sum regularization constraints from the functional map solver.
6.5 COMPARISON SPATIAL CONSISTENCY
One major advantage of functional map is that it preserves spatial consistency between points, estab- lishing smooth mapping between surfaces (Cheng et al., 2024). We compare functional map with two baselines in Fig. 10b; Hungarian matching and nearest neighbor retrieval, where we compute a pairwise feature distance matrix between vertices using the same feature from our model. Func- tional map produces a smooth mapping by preserving both woint-wise features and spatial relations between points, while the baselines only preserve the former and result in speckled mismatches.
7 CONCLUSION
summary, we make the following contributions this paper:
We create the first matching dataset with colored meshes, containing 600 assets span-
ning 24 categories with dense correspondence annotation.
We bridge the gap between 3D and multiview 2D correspondence methods by developing a framework that combines appearance and geometric information by refining features from
2D vision models with 3D geometric models.
We demonstrate the effectiveness of approach by performing two sets of experiments that generalize across objects from different categories: (i) real-world robotic manipulation experiments of long-horizon tasks requiring multiple grasps (ii) color transfer experiments.
10

---

Published as a conference paper at ICLR 2025
Acknowledgments. We extend sincere thanks colleagues and labmates the Shanghai Qi Zhi Institute for their contributions to the human demonstrations (in no specific order): Guangqi Jiang, Yuanliang Ju, Pu Hua, Lesi Chen, Huanyu Li and Tianming Wei. We deeply appreciate their beautiful and skillful hands. Special thanks to Galaxea for their hardware support of the A arm.
REFERENCES
Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep vit features as dense visual de- scriptors. arXiv preprint arXiv:2112.058144 2(3):4, 2021.
Mathieu Aubry, Ulrich Schlickewei, and Daniel Cremers. The wave kernel signature: A quantum mechanical approach to shape analysis. In ICCV workshops, pp. 1626-1633. IEEE, 2011.
Shikhar Bahl, Russell Mendonea, Lili Chen, Unnat Jain, and Deepak Pathak. Affordances from human videos as versatile representation for robotics. In CVPR, pp. 13778-13790, 2023.
Federica Bogo, Javier Romero, Matthew Loper, and Michael J Black. Faust: Dataset and evaluation for 3d mesh registration. In CVPR, pp. 3794-3801, 2014a.
Federica Bogo, Javier Romero, Matthew Loper, and Michael J. Black. Faust: Dataset and evaluation for 3d mesh registration. In CVPR, pp. 3794-3801, 2014b. doi: (CV))))44C))
Alexander M. Bronstein, Michael M. Bronstein, and Ron Kimmel. Numerical geometry of URL https://api.
non-rigid shapes. Monographs in Computer Science, 2009. semanticscholes Sttr://oopusDD:3355222
Dongliang Cao and Florian Bernard. Unsupervised deep multi shape matching. ECCV, pp. 55- 71. Springer, 2022.
Dongliang Cao, Paul Roetzer, and Florian Bernard. Unsupervised learning of robust spectral shape matching, arXiv preprint 1xXx::2304.14117 2023.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, pp. 9650- 9660, 2021.
Xinle Cheng, Congyue Deng, Adam Harley, Yixin Zhu, and Leonidas Guibas. Zero-shot image feature consensus with deep functional maps. In ECCV, 2024.
Cheng Chi, Zhenjia Xu, Chuer Pan, Eric Cousineau, Benjamin Burchfiel, Siyuan Feng, Russ Tedrake, and Shuran Song. Universal manipulation interface: In-the-wild robot teaching with- out wi-the-wild robots. arXiv preprint drXiv:2402.10329 2024.
David Coleman, Ioan Sucan, Sachin Chitta, and Nikolaus Conell. Reducing the barrier to entry of complex robotic software: a moveit! case study. arXiv preprint arXiv: 1404.378, 2014.
Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, Eli VanderBilt. Aniruddha Kembhavi, Carl Vondrick, Georgia Gkioxari, Kiana Ehsani, Ludwig Schmidt, and Ali Farhadi. Objaverse-xl: universe of 10m+ 3d objects, 2023. URL httpss///ewwvv////eeeee 2307.00 056633
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hicrarchical image database. In CVPR. pp. 248-255, 2009, doi: 10. (0.1109//VV 520664888888
Nicolas Donati, Abhishek Sharma, and Maks Ovsjanikov. Deep geometric functional maps: Robust feature learning for shape correspondence. In CVPR, pp. 8592-8601, 2020.
Niladri Shekhar Dutt, Sanjeev Muralikrishnan. and Niloy Mitra. Diffusion 3d features (diff3f): Decorating untextured shapes with distilled semantic features, 2024. URL https: //axxiv.. org/abs/ 1211..771........1..111 17024.
11

---

Published as a conference paper at ICLR 2025
Roberto M. Dyke, Caleb Stride, Yu-Kun Lai, and Paul L. Rosin. Shrec' 19: Shape correspondence with isometric and non-isometric deformations, 2019.
Hao-Shu Fang, Chenxi Wang, Hongjie Fang, Minghao Gou, Jirong Liu, Hengxu Yan, Wenhai Liu, Yichen Xie, and Cewu Lu. Anygrasp: Robust and efficient grasp perception in spatial and tem- poral domains. IEEE Transactions on Robolics, 2023.
Peter R Florence, Lucas Manuelli, and Russ Tedrake. Dense object nets: Learning dense visual object descriptors by and for robotic manipulation. arXiv preprini 1rXivv1806.08% 2018.
Stephanie Fu, Mark Hamilton, Laura E. Brandt, Axel Feldmann, Zhoutong Zhang, and William T. Freeman. Featup: A model-agnostic framework for features at any resolution. In ICLR, 2024.
URL
wtwperrrrreeeeee/oummmmmmmm wwtp:/wwwwwweeeeeeeeeeeeeeeeeeee forum? (o-oommidd-kuiN222000
Thibault Groueix, Matthew Fisher, Vladimir Kim, Bryan C Russell, and Mathieu Aubry. 3d-
coded: 3d correspondences by deep deformation. In ECCV, pp. 230-246, 2018a.
Thibault Groueix, Matthew Fisher, Vladimir G. Kim, Bryan C. Russell, and Mathieu Aubry, 3d- coded: 3d correspondences by deep deformation, 2018b. URL 1806. 05228.
abs/
Oshri Halimi, Or Litany, Emanuele Rodola, Alex M Bronstein, and Ron Kimmel. Unsupervised learning of dense shape correspondence. In CVPR, pp. 4370-4379 2019.
Eric Hedlin, Gopal Sharma, Shweta Mahajan, Hossam Isack, Abhishek Kar, Andrea Tagliasacchi, and Kwang Moo Yi. Unsupervised semantic correspondence using stable diffusion. In NeurlPS, 2023.
Kaizhe Hu, Zihang Rui, Yao He, Yuyao Liu, Pu Hua, and Huazhe Xu. Stem-ob: Generalizable visual imitation learning with stem-like convergent observation through diffusion inversion. arXiv preprint arXiv:2411. 1.0991, 2024.
Yuanchen Ju, Kaizhe Hu, Guowei Zhang, Gu Zhang, Mingrun Jiang, and Huazhe Xu. Robo-abe: Affordance generalization beyond categories via semantic correspondence for robot manipulation. ECCV, 2024.
Michael M. Kazhdan, Matthew Bolitho, and Hugues Hoppe. Poisson surface reconstruction. In Alla Sheffer and Konrad Polthier (eds.), Proceedings of the Fourth Eurographics Symposium on Geometry Processing, volume 256 of SGP '06, pp. 61-70, Aire-la-Ville, Switzerland, Switzer- land, 2006. Eurographics Association. ISBN 3-905673-36-3. URL http: //dl./dll//////// acm. org/ citaaionn cfm?id- seteiid-121191111 1281965.
Vladimir G Kim, Yaron Lipman, and Thomas Funkhouser. Blended intrinsic maps. TOG, 30(4): 1-12, 2011.
Diederik P. Kingma and Jimmy Ba. Adam: CoRR, abs/1442..6980, 2014. URL https:://saa /tass::/apls semantmantiiio semanticscholaat 6628106.
method for stochastic optimization.
Yuxuan Kuang, Junjie Ye, Haoran Geng, Jiageng Mao, Congyue Deng, Leonidas Guibas, He Wang, and Yue Wang. Ram: Retrieval-based affordance transfer for generalizable zero-shot robotic manipulation, 2024. URL http:::://erdi.oooo/aa///111111
24000//247..0 04689.
Yuyao Liu, Jiayuan Mao, Joshua Tenenbaum, Tomás Lozano-Pérez, and Leslie Pack Kael- bling. One-shot manipulation strategy learning by making contact analogics. arXiv preprint arXiv:2411.09627, 2024.
Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. In 3DV, 2024.
Grace Luo, Lisa Dunlap, Dong Huk Park, Aleksander Holynski, and Trevor Darrell. Diffusion hyperfeatures: Searching through time and space for semantic correspondence. NeurlPS, 36, 2024.
12

---

Published as a conference paper at ICLR 2025
Mark Meyer, Mathieu Desbrun, Peter Schröder, and Alan H Barr. Discrete differential siferential-geommery operators for triangulated 2-manifolds. In Visualization and mathematics III, pp. 35-57 Springer, 2003.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
Dorian Nogneng and Maks Ovsjanikov. Informative descriptor preservation via commutativity for shape matching. In Computer Graphics Forum, volume 36, pp. 259-267, Wiley Online Library, 2017.
Dolev Ofri-Amar, Michal Geyer, Yoni Kasten, and Tali Dekel. Neural congealing: Aligning images to joint semantic atlas. In CVPR, pp. 19403-19412, 2023.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023
Maks Ovsjanikov, Mirela Ben-Chen, Justin Solomon, Adrian Butscher, and Leonidas Guibas. Func- tional maps: a flexible representation maps between shapes. TOG, (4): 2012.
Ioannis Pratikakis, Michalis A. Savelonas, Fotis ArnaouTOGlou, George Alexios Ioannakis, Anestis Koutsoudis, Theoharis Theoharis, Minh-Triet Tran, Vinh-Tiep Nguyen, Viet-Khoi Pham, Hai- Dang Nguyen, Ho Lè. Ba-Huu Tran, Quan H. To, Minh-Bao Truong, Thuyen V. Phan, Minh D. Nguyen, Thanh-An Than, Khoi-Nguyen C. Mac, Minh N. Do, Anh Due Duong, Takahiko Furuya, Ryutarou Ohbuchi, M. Aono, Shoki Tashiro, David Pickup, X. Sun, Paul L. Rosin, and R. R. Martin. Shrec' 16 track: Partial shape queries for 3d object retrieval. 2016. URL https: / /api... wwwant ieschoilcscclaaa org/ httg/CorpsssI0 13810094
Emanuele Rodolà, Luca Cosmo, Michael M Bronstein, Andrea Torsello, and Daniel Cremers. Partial functional correspondence. In Computer graphics forum, volume 36, pp. 222-236. Wiley Online Library, 2017.
Paul Roetzer and Florian Bernard. Spidermatch: 3d shape matching with global optimality and geometric consistency. In CVPR, 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. CoRR, abs/21 12.1522 2021. URL
https:://// //arxiv./vivvv// org/oorgsss.....oooooooo...ooooo aass///1/a.. 12/2211.1100 10752.
Dandan Shan, Jiaqi Geng, Michelle Shu, and David F Fouhey, Understanding human hands in contact at internet scale. In CVPR, pp. 9869-9878, 2020.
Nicholas Sharp, Souhaib Attaiki, Keenan Crane, and Maks Ovsjanikov. Diffusionnet: Discretization agnostic learning on surfaces. TOG, 4163):1-16, 2022.
Jian Sun, Maks Ovsjanikov, and Leonidas Guibas. A concise and provably informative multi-scale signature based on heat diffusion. Computer graphics volume 28, pp. 1383-1392, Wiley Online Library, 2009.
Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emergent correspondence from image diffusion. NeurIPS, 36: 1363-1389, 2023.
Chen Wang, Linxi Fan, Jiankai Sun, Ruohan Zhang, Li Fei-Fei, Danfei Xu, Yuke Zhu, and An- ima Anandkumar. Mimicplay: Long-horizon imitation learning by watching human play, arXiv preprint drXiv2230..12422 2023.
Chen Wang, Haochen Shi, Weizhuo Wang, Ruohan Zhang, Li Fei-Fei, and C Karen Liu. Dexcap: Scalable and portable mocap data collection system for dexterous manipulation. arXiv preprint arXiv:2403.07788, 2024.
Xiaolong Wang.
Learning and Reasoning with Visual Correspondence in Time. PhD thesis, Careegie Mellon University, Pittshurgh, PA, September 2019.
13

---

Published as a conference paper at ICLR 2025
Tong Wu, Jiarui Zhang, Xiao Fu, Yuxin Wang. Jiawei Ren, Liang Pan, Wayne Wu, Lei Yang, Jiaqi Wang, Chen Qian, Dahua Lin, and Ziwei Liu. Omniobjecttd: vocabulayy object dataset for realistic perception, reconstruction and generation. CVPR. pp. 803- 814, 2023. URL https: / //api. semant cachola ar. org.or///ooooooo ptrg/CorrusTTii 255998491.
Zhengrong Xue, Zhecheng Yuan, Jiashun Wang, Xueqian Wang. Yang Gao, and Huazhe Xu. Useek: Unsupervised eqquiivaaa 3d keypoints for generalizable manipulation. In ICRA, pp. 1715- 1722. IEEE, 2023.
Zhecheng Yuan, Tianming Wei, Shuiqi Cheng, Gu Zhang, Yuanpei Chen, and Huazhe Xu. Learning to manipulate anywhere: A visual generalizable framework for reinforcement learning. arXiv preprint arXiv:2407.15815, 2024.
Yanjie Ze, Gu Zhang, Kangning Zhang, Chenyuan Hu, Muhan Wang, and Huazhe Xu. 3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations. In RSS, 2024.
Junyi Zhang, Charles Herrmann, Junhwa Hur, Luisa Polania Cabrera, Varun Jampani, Deqing Sun, and Ming-Hsuan Yang. tale of two features: Stable diffusion complements dino for zero-shot semantic correspondence. In NeurlPS, 2023.
Junyi Zhang, Charles Herrmann, Junhwa Hur, Eric Chen, Varun Jampani, Deqing Sun, and Ming- Hsuan Yang. Telling left from right: Identifying geometry-awan semantic correspondence. In CVPR, pp. 3076-3085, June 2024.
Silvia Zuffi, Angjoo Kanazawa, David Jacobs, and Michael J. Black, 3D menagerie: Modeling the 3D shape and pose of animals. In CVPR, July 2017.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 TASK DESCRIPTION
In this section, we elaborate details of our tasks and showcase our model's generalization capacity by illustrating differences between the objects manipulated by human demonstration and robots.
Peeling a Banana. Given an RGBD human demo of peeling a banana, we extract two keypoints: where the banana is initially grasped, and where the base of the banana is held. These points then mapped to the target banana that the robot will manipulate. Since this task that requires mul- tiple steps with multiple keypoints, we employ a collaborative dual-arm robot approach. We use Galaxea Al Arm and a UR5 arm to jointly complete the task of peeling the banana.
Flower Arrangement. We provide a human demonstration arranging flowers. Mimicking human contact points, the robot grasps the flowers by their stem and inserts them into a vase.
Placing Shoes. We provide a human demonstration of picking up shoes by the heel and arranging them. The robot operates on three different shoes in cluttered environment, and and arranges them sequentially throughout multiple steps.
Decorating the Christmas Tree. In the human demo for decorating a Christmas tree, the ornaments used are different in shape and color from the objects for robotic manipulation. The robot hangs the ornaments on the Christmas tree to complete the task.
Pulling Out the Carrot. The human demonstrator picks up kettle and manipulates plush carrot toy. The robot manipulates a real carrot, generalizing across different materials with correspondence. Finally, the robot first picks up the kettle to water the carrot and then pulls out the carrot.
Pointing Object Parts with Pen. In this task, we verify the spatial continuity of multiple corre- spondences on the same object. The template meshes we provide are a ballpoint pen, a plier, a screwdriver, racket, and panda toy, while the target objects are different ballpoint pens, pliers, screwdrivers, rackets, and a dog toy. The robot first grasps the ballpoint pen and then uses it to successively poke two keypoints each object.
A.1.1
ASSET CATEGORIES
A.1.2 DATASET FILTERING
For food items, We use Objaverse website's keyword search functionality narrow our search scope and visually pick meshes that belong to our listed categories. We remove all meshes that are bigger than 300MB in size. For assets from OmniOhject3DD we randomly pick meshes from the desired categories. Due to the large sizes of scans, we uniformly sample point clouds from the mesh surface and perform Poisson Reconstruction (Kazhdan et al., 2006) to save downsampled versions of colored meshes for rendering.
1.1.3 REMESHING
We first normalize object scales by the longest side and multiply them by 0.3, and center the center of bounding boxes at the origin. We cluster each asset into connected components and only keep the largest one. We then remove unreferenced faces and vertices, and merge vertices that closer than 1/100 of bounding box size. We use the isotropic explicit remeshing filter from MeMhs Lab, and iteratively increase the target edge length until the number of vertices goes below the desired vertex number.
A.1.4 SPARSE KEYPOINT ANNOTATION
For fruits and vegetables with simple geometry, we instruct the annotators to use the 3D annotation tool' from StrayRobots to label sparse landmark points, and compute dense vertex groups annota- tions by interpolation. Specifically, we first manually derive a graph to represent the relationships with keypoints, then, for each pair of connected landmarks, we use the shortest path function of igraph to compute waypoints along certain fractions of the way from one landmark to another. We
15

---

Published as a conference paper at ICLR 2025
Category
Train
Validation
Test
apple
65
banana
74
6
bell pepper
6
bread
52
broccoli
5
6
carrot
celery!
0
3
cucumber
0
0
6
egg
17
eggplant
0
mushroom
64
onion
peach
13
pear
potato
15
2
pumpkin
40
tomato
12
2
6
zucchini
animals
5
5
tools
vehicles
5
5
backpacks
5
toiletries
chairs
6
5
Table 4: DenseCorr3D Dataset for each category. indicates held-out test categories. Animals deer, elephant, mouse, cat, giraffe, tiger, panda, leopard, dinosaur. Tools include: kitchen knife, toy knife, pocket knife, hammer, mallet. Vehicles include: car, bus, truck, train head. Toi- letries include: shampoo, sun spray, face cream. Chairs include: wood plank chair, velvet chair, veneer chair, mahogany chair, office chair.
include:
then connect different waypoints with the shortest path between them to form dense vertex groups. The average mesh takes 10 seconds to annotate.
A.1.5 DENSE VERTEX ANNOTATION
For more complex daily object categories, create color code chart of semantic groups for each category, and instruct the annotators to use the vertex brush tool from Blender to paint the vertices accordingly. We then use the color codes parse the painted meshes into separate vertex groups. The average mesh takes 5 minutes to annotate.
A.2 METHOD DETAILS
A.2.1 CALCULATION SEMANTIC DISTANCE
For two vertices on the same mesh, we perform bipartite matching on the pairwise geodesic dis- tance matrix between vertices in their respective groups and compute the average distance between matched pairs vertices. the source and target groups are on different meshes, we find the corre- sponding group of the source group on the target mesh and compute its distance to the target group analogously. Formally, given vertex and Ug and their semantic groups Gr and G., semantic
https://www.blender.ogg
16

---

Published as a conference paper at ICLR 2025
groups with m, and vertices respectively, their semantic distance is defined as:
min(m,n)
min D Dremantic (Vi, Uj) min m, n.) ESSm
geodesscc (Gu, (T (k)); Go, (TZ (k)))
k=1
m G.,l n- = G
where mi and m2 are permutation functions encoding bipartite matching.
A.2.2 FUNCTIONAL MAP SOLVER
We set a = 10-22022000000 3 10-4 and weigh our added regularization function terms (entropy and sum- to-1) by 10-5 and 10 respectively. We use the first 10 eigenvectors of the cotangent Laplacian matrix as our bases and zero-initialized C. We modify the implementation from pyFM and compute the gradients of our added terms using Pytorch autograd with CUDA acceleration and use L-BFGS as our solver.
A.3 EXPERIMENT DETAILS
A.3.1 DIFF3F BASELINE
The original Diff3F pipeline first renders multiview depth maps, which are used as conditioning generate colored images with ControlNet model. then extracts DINOv2 features from the generated images and combines them with diffusion features to use as multiview features. Since our meshes are textured, we instead directly extract multiview features with Stable Diffusion and DINOv2 from rendered RGB images and concatenate them, before performing dimensionality re- duction with PCA. We then aggregate the resulting features onto vertices and feed them into the functional map.
A.3.2 TRAINING DENSEM
Our FeatUp module upsamples 16x16 features to 512x512 resolution. We pre-train FeatUp param- eters for 10,000 steps ImageNet (Deng et al., 2009). We freeze the 2D backhone models during training, and optimize a 4-block DiffusionNer with 512 channels on DenseCorr3Dfor 6000 steps with a batch size of 8 using Adam Kingma & Ba (2014). During training, we randomly rotate the meshes, and slice the meshes in half in random directions 50% of the time. We uniformly sample 5 cameras when the meshes sphere when the meshes are sliced. In order to make our model robust to the number of vertices, we randomly set the re-meshing target to between 500 and 2500 vertices during training. training for 50 epochs takes ~ 12h hours on 8xNvidia A100 GPUs. We do not observe any overfit- ting the lightweight DiffusionNet when using a default linear reconstructor, which contains -5M parameters. Note that in Dutt et al. (2024), 100 views rendered for each shape. which requires running the computationally heavy 2D extractor 100 times, consuming ~5 minutes for each mesh. Thanks to 3D network, found that using only 3 lateral views plus I top and bottom view during both training and inferencing is sufficient.
not sliced, and randomly sample I or 2 cameras in the same hemi-
total,
A.3.3 INFERENCE RUNTIME ANALYSIS
We performed runtime analysis during the inference stage of DenseMatcher on single A 1100 GPU. We directly render the original textured meshes to acquire posed images, and found the rendering time depend on the asset's meshing, varying between -0.05 seconds ~3 seconds and averag- ing to -0 -0.2 seconds per mesh. Computing 2D SD-DINO features for 5 views each consumes ~3.6 seconds, while performing DiffusionNet forward pass for each mesh consumes -0.01 seconds. Op- timizing the functional map consumes - -0.8 seconds for a pair of meshes with both 500 vertices, and consumes ~ -2.2 seconds for pair of meshes with both 2000 vertices. Overall, computing correspon- dences between pair of meshes with our algorithm consumes between 8.4 and 12.4 seconds single 1100 GPU, allowing memessenitive applications such as robotics planning.
In addition, vertex case and 2000-vertex case, purely matching features without accounting for spatial consis- tency. We found the runtime heavily depend on the sparsity of matrix values. Hungarian matching
Hungarian matching the pairwise vertex feature distance matrix for the 500-
17

---

Published as a conference paper at ICLR 2025
takes -0.01-0.4 seconds for the 500-vertex case, and 0.5-2.5 seconds for the 2000-verter case. We also derive theoretical runtime from SpiderMatch (Roetzer Bernard, 2024) and compare them below in Tab. 5.
Table 5: Runtime of functional map and baselines. All units are in seconds.
Method
500-vertex
2000-vertex
Functional Map (our implementation)
0.8
2.2
SpiderMatch (Roetzer & Bernard, 2024)
10
>:20
Hungarian Matching (no spatial consistency)
0.01-0.4
0.5-2.5
A.3.4 MODEL PERFORMANCE ON VARYING TOPOLOGIES
Table 6: 3D correspondence performance (Err }) categories with complex topologies.
Method
Chairs
Animals
Broccoli
Shampoo
URSSM (Cao al., 2023)
4.79
6.89
5.94
3,46
DenseMatcher (Ours)
3.74
3.29
3.18
0.73
A.3.5
MODEL PERFORMANCE HELD-OUT CATEGORIES
Table 3D correspondence performance on held-out categories.
Celery
Cucumber
Eggplant
Methods
AUC
Em
AUC
Err
AUC
Err
ConsistFMapp (DenseCorr3D) (Cao & Bernard, 2022)
0.957
0.44
0.596
4.73
0.707
3.47
URSSM (FAUST) (Cao al., 2023)
0.846
1.54
0.516
5.63
0.533
6.62
URSSM (DenseCor33) (Cao et al., 2023)
0.926
0.75
0.601
4.62
0.621
4.64
DiffF (Dutt et al., 2024)
0.676
3.28
0.600
4.85
0.444
7.87
DenseMatcher (Ours)
0.882
1.18
0.716
3.31
0.844
1.59
A.4 PROOFS
A.4.1 PRELIMINARY
We view source and target mesh manifold M discretized to nm vertices, and manifold N discretized to vertices, with diagonal area matrices AM apaxxxmx and Ay E X xnx denoting the area associated with each vertex. The inner product operator for scalar functions x E _ and E gn is defined as:
(x. = T = Σ ARaiyi
(2)
Given the area matrix and the contingent weight matrix of the mesh W E guxn (Meyer et al., 2003), the Laplace-Beltrami and computes its Laplacian, is defined as:
Beltrami operator A(-), which takes a scalar function x on the manifold as input
( (A WA--W)a.
(3)
18

---

Published as a conference paper at ICLR 2025
The first & Laplace- Belttrmm eigenfunctions 4M R dN E Ro are functions on M and N whose Laplacian is a scaled version of itself, obtained by solving the generalized eigenvalue problem:
1 = Ay
(4)
where J E IR" denotes the jth eigenfunction and X denotes the jth eigenvalue. The eigenfunctions orthonormal w.r.t. the inner produce operator:
0, i 1, i=j
22)
(5)
We compound the corresponding eigenvalues into diagonal matrices Ass E R and An R and re-write the 4 and 5 as:
WM M = AM = A( = AMM WN = AN
(6)
A(+N) AN.
Analogous to sine waves which are eigenfunctions of 1d Laplace operator, the Laplace-Beltram eigenfunctions can serve to project functions back and forth between the manifold and spectral domain.
The pscudo-inverse of the eigenfunction is defined as:
+ = Φ A.
(8)
Multiplying a function x with the pseudo-invent inverre of the bases:
X; = ( z), = ( (4" Ar), x).
(9)
equivalent to the inner product with the bases, and projects functions from the manifold to the spectral domain, where X is dubbed the "spectral coefficients" of the function, and X₃ E I&* corre- spondences to the ith eigenfunction.
To obtain function the manifold from its spectral coefficients, we can multiply the bases with the coefficients, since:
I In क ((Φ-.* (x) TX
(10)
A.4.2 FEATURE CONSTRAINT IN FUNCTIONAL MAP MINIMIZES SEMANTIC DISTANCE Loss
We show that the feature matching objective |CC Gl22 presented in 1 minimizes our proposed semantic distance between matched source and target vertices.
We define our source and target features as f E ar x and R
We can represent the vertex-to-vertex mapping from M with a sparse binary matrix Ray xnx, where:
0, i + match(j) Hy
(11)
1, i = match(j)
For pair corresponding features functions f R Kdia and E Fn x xdea, we can transport the source feature onto the target mesh with:
g - IIf (If), Új fmatch(j)>
(12)
where ý is the feature for jthe vertex target mesh obtained from its corresponding match(j)th vertex on the source mesh.
From our training objective in Eq 4.3.1, the feature distance should be linearly proportional the semantic distance function, assuming our training objective is fully optimized. We denote this linear constant
| 9;ll2 D semantti (Vi, By))
|| 9;ll2 sD (va, 0z)-
(13)
19

---

Published as a conference paper at ICLR 2025
Therefore, minimizing the sum of L2 distance between the transported source feature and the target feature is equivalent to minimizing the total semantic distance between matches:
|| 9
Σ [I(Tf)); 9j ll
(14)
Σ رقاا g3l12 mmmthb()) 95ll2
(15)
Σ semantic ( (Umatch(j)) 0z).
(16)
Provided with the first Laplace-Bellne Beltrami eigenvectors, we can decompose II into its low-rank "functional map matrix" representation C' R
Φ2 CD
(17)
Therefore,
Σ D semannicc (Umatchy U2)
||I g12
=1 g|1
Te T Ngl22
110 igll
I [2N112/CCF Gll2
where F fs f gixdss,
G= 2 kkdde
when using the full-rank bases, the inequality on the third line will become equality.
Thus, prove that minimizing the first term |ICF CFFGGy G in equation 1 is equivalent minimizing overall Dsemantic distance between matches in source and target match.
A.4.3 COMMUTATIVITY WITH LAPLACE OPERATOR INDUCES ISOMETRIC MAPPING
Lemma A.1. For function R", its Laplacian A(x) can be computed as 4AX from its full- rank spectral coefficients X + x E R²
Proof:
A(x) -
(18)
=A Σ D(X,)
(19)
Σ x.A(T;)
(20)
Σ X
(21)
PAX
(22)
-
(23)
The third line follows from the linearity of the Laplace-Beltrami Beltrami operator.
On Riemannian manifolds, diffeomorphism is isometric if and only if the Laplace operator is invariant under it. the discrete case, for vertex-to- to-rexxx mapping to be isometric, the equiva- lent statement is the mapping commutes with the Saplace-Beltrane operator for arbitrary function
20

---

Published as a conference paper at ICLR 2025
RTM defined source mesh M. Formally:
IA (IIx
(24)
ON + A (x) =1 Mrt)
(25)
M
T N M PM A MM X =A(
M P
(26)
NNCA -
$ N CX N
(27)
Multiplying both sides on the left by x obtain:
CAMX = ANCX.
(28)
Since this holds for any function roughly isometric mapping.
minimize ||ACC - CAM l1 as in equation to obtain
A.5
PERFORMANCE UNDER OCCLUSION
We study the performance our model under occlusion in two cases.
A.5.1 PARTIAL SOURCE AND PARTIAL TARGET
In the first case, both the source and target mesh are partially occluded. As shown in Fig. 11, our model is capable of matching partial meshes reconstructed from RGBD camera captures and finding the correct grasping points.
Source Scene RGB Image
Source Scene Point Cloud
Source Object
Contacl Point
Scene
Scene
Scene 3
Scene
Figure 11: Robot experiments visualization under occlusion conditions.
21

---

Published as a conference paper at ICLR 2025
A.5.2 PARTIAL SOURCE AND FULL TARGET
In the second case, the source mesh is a partial mesh, and the target mesh is full mesh. We follow the formulation of partial functional correspondence (Rodolà et al., 2017) and jointly optimize mask 7) E RN that indicates whether each vertex in the target mesh matched the source mesh. In addition to our proposed regularization constraints, we implemented the Mumford-Shh functional and area preservation constraints from Rodolà et al. (2017); in addition to penalizing the entropy of with- >in nN log 7/s- We showcase qualitative results in Fig. 12 below.
Figure 12: Correspondence between partial mesh with full mesh. DenseMatcher is capable of matching partial mesh to full mesh by utilizing the partial functional correspondence formulation.
22


Reviews:
[{"review_id": "YuPRFA91Xo", "reviewer": "Reviewer_w2Z1", "summary": "This paper proposes a framework and dataset for category-level object 3D dense matching. The DenseMatcher utilizes a 2D foundation model with 3D network refinement to reach generalization and 3D understanding. The author conducts robotic manipulation and zero-shot color mapping to validate the findings.", "questions": "1. In Sec 4.1 Preliminary, Functional Map, please give a detailed justification about how to regularize the term C as isometric in your context. \n\n2. In the appendix, please provide a detailed explanation, with proofs, showing how previous constraint terms ensure that the output is minimized in the semantic distance function.", "limitations": "", "rating": 10, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. This idea is novel and underexplored in relevant areas, especially in robotic manipulation learning. Instead of simply augmenting the data with numerous demos, this paper can address sample efficiency by embedding semantic information.\n\n2. This paper's writing style is straightforward, and it is easy to catch the main topic.\n\n3. Utilizing the existing 2D network (DINO in this paper) with 3D networks is a simple but promising approach.\n\n4. Experiments can thoroughly reflect the model's ability. In robotic manipulation tasks, it covered pick-and-place, long-horizon, and dual arm.", "weaknesses": "1. The statements of regularization terms in the methodology part are unclear and may cause ambiguity.\n\n2. Some experiment details, like the description for each task, can be placed in the appendix and give a more precise visualization. The images in the robotic manipulation task are too undersized.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vYmd6dnvAh", "reviewer": "Reviewer_iy2b", "summary": "This paper introduces DenseMatcher, an innovative method for computing dense 3D correspondences between objects with similar structures, geared towards applications in robotic manipulation. They propose that *semantic correspondence*—which aligns semantically similar parts across objects—provides more powerful generalization capabilities across categories compared to *shape correspondence*, which mainly focuses on geometry.\n\nTo facilitate the training and evaluation, they created *DenseCorr3D*, a new dataset comprising 589 colored object meshes across 23 categories, with dense correspondences organized into semantic group. DenseMatcher utilizes pre-trained 2D foundation models to extract multiview features, which are further refined using DiffusionNet. The enhanced features are then used to establish dense correspondences through a functional map. \n\nThey provide comprehensive experiment results to demonstrate DenseMatcher’s effectiveness in 3D dense matching, zero-shot robotic manipulation, and color transfer tasks. DenseMatcher outperformed baseline methods on the DenseCorr3D benchmark and achieved a 76.7% success rate in real-world robotic manipulation, showcasing its robust generalization capabilities.", "questions": "- Performance on Varying Topologies: How does DenseMatcher perform with objects of varying topologies? Are there specific object structures or topological variations where its performance significantly degrades?\n\n- Handling Severe Occlusion: Is DenseMatcher able to be adapted or extended to handle severe occlusion more effectively? What potential modifications could mitigate its reliance on multiview feature extraction and functional maps in such cases?\n\n- More Benchmark Validation: Are there any benchmarks or experiments that could further validate DenseMatcher’s robustness against topological diversity and occlusion? How might these additional evaluations impact its overall effectiveness and applicability in real-world scenarios?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Integration of 2D and 3D: DenseMatcher effectively combines 2D foundation models, like SD-DINO, for multiview feature extraction with DiffusionNet to refine features with geometry. This fusion enhances semantic understanding and generalizability in 3D correspondence.\n\n- New 3D matching dataset: The authors introduce DenseCorr3D, the first dataset with colored meshes and dense correspondences, featuring 589 textured meshes across 23 categories. It advances research by supporting methods that account for both appearance and geometry.\n\n- Enhanced functional map for accuracy: A novel regularization scheme promotes sparsity in DenseMatcher’s functional map, achieving a 43.5% accuracy improvement over baselines.\n\n- The paper is well-written and easy to understand. The experiment results are comprehensive and promising.", "weaknesses": "- Limited analysis on varying topologies: While they analyze that previous methods struggle with different topologies, they do not deeply explore DenseMatcher's robustness on diverse object structures.\n\n- Limitation to severe occlusion: The paper does not address how DenseMatcher handles significant occlusion. Since it relies on multiview feature extraction and functional maps, both susceptible to occlusion, further analysis of this limitation would strengthen the evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UKm1q8BDXh", "reviewer": "Reviewer_Jn9K", "summary": "This paper studies the problem of dense surface-point matching between objects, where similarity is understood as a user-defined semantic and matches can be between objects of the same, but also different category. The contributed method combines features that encode the visual appearance with features that encode local geometry. The method is evaluated and compared against baselines on a self-created dataset and real-world robotic imitation of human demonstrations.", "questions": "- Section 4.1: I am not super familiar with the prior work on 3D dense matching, but this optimization formulation seems computationally expensive and as Section 4.4 shows also unstable. Why are other assignment and matching methods not compared as beaseline or ablation? e.g. Hungarian matching or the double-softmax used in [1]?\n- line 200: The requirement of textured 3D assets is very limiting. It seems to me the method could also work from an untextured geometry asset and posed images, or am I missing something?\n- line 242: Since the negative cosine distance is such an odd choice I suspect the authors were inspired here by related work? In that case it would be important to attribute this here with a reference.\n- line 252: \"object type and material\" is misleading. Neither one of the frozen backbones captures this information, both are self-supervised encoders of visual appearance that might correlate with this information in some cases.\n- line 254: What norm is used in the equation for $\\mid\\mid \\cdot \\mid\\mid$? Why is that one choosen?\n- Table 1: Please explain better the different ablation variants. Is \"w/o Diffusion Net\" directly matching the concatenation of $f_\\textrm{multiview}$ and the HKS features? Or is it also using the XYZ features and therefore failing because of coordinate system change? \n- Section 6.2.3: I dont't think the comparison to Robo-ABC is entirely fair. It would be good to show both variants, with the full affordance memory and with the reduced form that is currently presented. The proposed method is very expensive in terms of the 3D data it requires, so really it needs to show that this additional information can compete with methods that are only based on cheaper and more abundant image data.\n- Section 6.2.4: How is success determined in the experiments? Given the low number of overall trials, what level of statistical significance does the experiment currently have?\n\n\n\n[1] Lindenberger, P., Sarlin, P.-E., & Pollefeys, M. (2023). LightGlue: Local Feature Matching at Light Speed. Retrieved from https://openaccess.thecvf.com/content/ICCV2023/html/Lindenberger_LightGlue_Local_Feature_Matching_at_Light_Speed_ICCV_2023_paper.html", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "- The authors claim (and I am not aware otherwise, but also not super familiar with this subfield) to contribute the first method for 3D dense correspondences that combined visual appearance and geometric information. This very intuitively makes sense and makes especially the contributed dataset something that can have profound impact on the research on 3D correspondences.\n- The method has directly been evaluated in a real-world application of mimicing human demonstrations with a robotic manipulator.\n- The paper is very well written (the best in my review batch) and easy to follow.", "weaknesses": "- The experimental evaluation is limited to a self-contributed dataset and very few qualitative runs on a robotic application (where it is unclear if the method difference is statistically significant).\n- The method design contains a couple of non-straightforward design choices without justifications or experimental evidence to back up these choices [**update from discussion with authors: these points are mostly addressed now**]:\n  - Using the XYZ coordinates of the mesh vertices makes the method sensible to random transformations on the input mesh. There is no experiment evaluating whether the model is able to learn invariance over such random coordinate system changes.\n  - The choice of negative cosine similarity in $L_\\textrm{semantic}$ is quite particular. The authors do not explain why they would choose this over e.g. L1 or L2 distances and also do not ablate this choice.\n  - Similarly, for $L_\\textrm{preservation}$, the choice of a single linear layer for reconstruction might hinder the encoder network to learn a more useful non-linear function. The more standard choice would probably be to mirror the encoder architecture like in an autoencoder, but this is neither discussed nor evaluated.\n- The method requires supervised training with an expensive 3D annotation workflow.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wkDcvIXMgd", "reviewer": "Reviewer_SE4d", "summary": "Summary: This paper introduces DenseCorr3D, a 3D matching dataset featuring colored meshes and dense correspondence annotations. It addresses the limitations of existing datasets that predominantly emphasize geometry. The authors propose DenseMatcher, a model that integrates 2D foundation models with 3D networks to significantly enhance dense correspondence accuracy. The effectiveness of DenseMatcher is demonstrated through applications in robotic manipulation tasks and color transfer experiments.", "questions": "Additionally, it would be constructive to present examples of failure cases to provide a more comprehensive evaluation.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strengths:\n\nThe authors have developed a dataset that is a valuable resource for the research community.\n\nDespite its straightforward pipeline and principles, the proposed DenseMatcher effectively extracts semantic maps that facilitate subsequent tasks.\n\nThe introduction of the function map is promising, and the correspondence video demo on the accompanying website is impressive.", "weaknesses": "Weaknesses:\n\nThe range of tasks and the diversity of object categories provided in the dataset are limited.\n\nLine 853 mentions the total time expenditure without delving into specific details, such as the time required for rendering images, particularly the computation comsumption of the function map.\n\nThe paper lacks an ablation study for the DINO and SD components. Previous zero-shot methods shows that the features provided by SD VAE may not be optimal. An ablation analysis for the feature backbone should be included in the experimental tables.\n\nThere is no discussion on whether the model incorporates augmentations for the pose of the mesh. Research has shown that semantic features can easily overfit to spatial position-related scenarios. If the input mesh's position changes, the resulting semantic map may become inaccurate. Therefore, it would be beneficial to include experiments that apply random rotations to the mesh as input.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)