Jamais Vu: Exposing the Generalization Gap in Supervised Semantic Correspondence
Authors: Octave Mariotti
Zhipeng Du
Yash Sanjay Bhalgat
Oisin Mac Aodha
Hakan Bilen
Venue: NeurIPS 2025 poster ()
Keywords: Semantic Correspondence
Evaluation
Generalization
Representation Learning
Primary area: applications
TL;DR: We show that supervised semantic correspondence methods fail to generalize well to unseen keypoints and we introduce geometric constraints during training  to address this.

Abstract: Semantic correspondence (SC) aims to establish semantically meaningful matches across different instances of an object category. We illustrate how recent supervised SC methods remain limited in their ability to generalize beyond sparsely annotated training keypoints, effectively acting as keypoint detectors. To address this, we propose a novel approach for learning dense correspondences by lifting 2D keypoints into a canonical 3D space using monocular depth estimation. Our method constructs a continuous canonical manifold that captures object geometry without requiring explicit 3D supervision or camera annotations. Additionally, we introduce SPair-U, an extension of SPair-71k with novel keypoint annotations, to better assess generalization. Experiments not only demonstrate that our model significantly outperforms supervised baselines on unseen keypoints, highlighting its effectiveness in learning robust correspondences, but  that unsupervised baselines outperform supervised counterparts when generalized across different datasets.

Jamais Vu: Exposing the Generalization Gap in Supervised Semantic Correspondence
Octave Mariotti'
Zhipeng Du
Yash Bhalgat?
Oisin Mac Aodha'
Hakan Bilen
1 University of Edinburgh
University of Oxford
https://github. http://a...oomm////ot/////////aa UomJaaiisJaaau
Abstract
The goal semantic correspondence (SC) estimation is to establish semantically meaningful matches across different instances of an object category. In this work, we illustrate how recent supervised SC methods generalize poorly beyond the anno- tated keypoints seen during training, thus effectively acting keypoint detectors. To address this, we propose new approach for learning dense correspondences by lifting 2D keypoints into canonical 3D space using monocular depth estimation. Our method constructs continuous canonical manifold that captures object geom- etry without requiring explicit 3D supervision or camera annotations. Additionally, we introduce SPair-U, an extension of SPair-71k Ik with novel keypoint annotations, better assess generalization. Experiments not only demonstrate that our model significantly outperforms supervised baselines on unseen keypoints, highlighting its effectiveness in learning robust correspondences, but that unsupervised baselines outperform supervised counterparts when evaluated across different datasets.
Introduction
Semantic correspondence (SC) estimation involves identifying semantically matching regions in images across different instances of the same object category. It remains challenging problem, as it requires recovering fine-grained details while maintaining robustness against variations in object appearance, shape, and viewing conditions. Recent advances in large-scale vision models, particularly self-supervised transformers [6, [39 and generative diffusion models [41], have led to notable improvements in SC. When employed backbones, these models have achieved over 20% gains in accuracy on the SPair-71k benchmark [34]. However, despite these advances, recent studies have highlighted that these powerful representations often struggle disambiguate symmetric object
parts due to their visual similarity [62] 32].
SC methods can be broadly categorized into two groups in terms of supervision: unsupervised models, which do not require correspondence annotations during training [
61 32,, and supervised mod- els 8 [g 62), which trained manually annotated correspondences. As expected, supervised models generally achieve higher performances when using the same backbone and same training unsupervised models. However, a key limitation of current benchmarks is that evaluation is typically performed on the same set of keypoints used for training, potentially inflating perceived generalization. As illustrated in Fig. [I] the performance of supervised models drops significantly when evaluated on unseen keypoints, while unsupervised models maintain their performance.
2
Building dense correspondences are key to fine-gras fine-grained object understanding and improving robust- ness in various recognition tasks, and essential to many applications including texture transfer [38] and robotic manipulation [4] In this work, examine the performance of sfate-of maaee-o-thhe-eee of-the-art SC
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Source
Ours
Geo-SC
Unsup
Seen
Unseen
Seen Keypoints
90
70
Unseen Keypoints
50
Ours
Geo-SC
Unsup
PCK@0.1
Figure 1: Illustration the generalization gap on unseen keypoints. (Left) Top row: when evaluated on known keypoints, both our model and Geo-SC 62 perform well, while the unsupervised DINOv2+SD [6] struggles to correctly disambiguate the legs of the horse. Bottom row: when presented with keypoints unseen training time. both model and DINO+SD predict noisy but reasonable correspondence, while Ge predictions noticeably degrade. (Right) Even though obtains strong performance on known keypoints, Geo-SC performs worse than unsupervised counterpart on our new benchmark unseen keypoints. comparison, our model still achieves competitive results.
models when evaluated on points that lie outside the set of annotated training keypoints. Under these conditions, observe that supervised pipelines often underperform their unsupervised counterparts, effectively reducing their function to that 'sparse keypoint detectors'. We attribute this limitation two main factors: (i) the sparsity of supervision, which typically focuses limited number of keypoints and (ii) the lack of evaluation on unseen points, which tends to favor models that bias their predictions toward the nearest
annotations.
We argue that an ideal SC method should be capable of matching arbitrary points, akin to the objectives in classical dense correspondence tasks (26). To move beyond sparse supervision, propose learning framework that predicts dense feature maps and supervises them using geometry from an aff-the-shelf depth estimator, thereby enabling training richer and more spatially diverse cues. Some unsupervised SC methods [46 (2] have leveraged 3D geometry to learn dense correspondences through mapping object pixels to spherical coordinate system where each coordinate corresponds to a different characteristic point of the object. However, this approach requires estimating the object shape and viewpoint from collection of 2D images, which limits the applicability of such models to synthetically generated datasets [6 and they can require additional camera viewpoint supervision
We propose a new approach that leverages existing 2D keypoint annotations and estimated 3D geometry to learn dense correspondences. We build on the idea of learning canonical representation of the object category, which is invariant to the object instance, viewpoint, and pose. We achieve this by lifting the 2D keypoints to 3D using monocular depth model, aligning them with set of canonical keypoints that are shared across all instances of the object category. Finally, by interpolating between them, learn continuous canonical manifold, that captures the underlying 3D shape of the object and incorporates geometric constraints into learning more effective and general feature representations. dataset for SC estimation, SPair-U, which extends the original SPair-71k
We also introduce test annotations with a set new keypoints, allowing us to evaluate the generalization of SC models on unseen keypoints. We show that supervised SC models trained on the original SPair-71k 1k dataset typically fail to generalize well to unseen keypoints, while our method is able to learn more general representation that can be applied unseen keypoints.
2 Related Work
Supervised methods. Supervised approaches rely on the availability of datasets with annotated keypoints such as CUB (51), PF-PASCAL [15), and SPair-71k [4 to learn corresponding points across instances of the same object class depicted in different images. This is typically using contrastive objectives minimizing distance between features coming from the same keypoints while

---

To obtain pushing other features away [6
61 62 A more computationally intensive option is to compute 44
dense 4D correlations maps between each source and target locations [8 33
[8] 22
stronger descriptors. it is also common to aggregate features from multiple network layers to form hypercolumns 35 62] Current state-of-the-ar supervised methods forego training from scratch and instead typically use large pretrained vision model as a backbone, the most popular options being DINOv2 [39 and Stable Diffusion [41]. While effective at matching instances of keypoints of the same type that have been observed during training, in our experiments we demonstrate that current supervised methods have a tendency to overfit to the set of keypoints observed during training and struggle to generalize to previously unseen keypoints (see Fig. I for example).
There have also been recent attempts to utilize the expressive power of the representations encoded in large multi-modal models for detecting sets of keypoints. Few-shot methods require supervision in the form of support set at inference time 288 29 17. Zero-shot methods forego the need for such supervision, but instead require that keypoints should be described via natural language Describing common keypoints (e.g.. 'the left eye") can be easily done via language, but how to best describe less salient points via text is not so clear. There have also been attempts to develop models that can take different various modalities (i.e., text and or keypoints) as input [30]. While promising, these methods make use of large multi-modal models and need large quantities of keypoint supervision data, spanning many diverse keypoints and categories, for pretraining.
prompts [63 GO
Unsupervised weakly-supervised methods. Methods that do not use correspondence supervision during training range from unsupervised approaches using general-purpose backbones [ [1], very weakly supervised methods that only assume an curated training set without labels, e.g., images of a single category [46, 45 21. zero-shot methods that only use test-time information about the relationships between keypoints [38] 62), dense methods that directly impose structure on the correspondence field 77]. and weakly supervised methods that use extra labels like segmentation masks or camera pose 220 32 4) Earlier unsupervised methods typically use self-supervised
objectives that make use of synthetic deformatiios//ugmeetaaiii
of the same image [46] by using
to provide pseudo ground truth correspondence. Later it was observed that large pretrained vision models naturally posses features that are very strong for SC, despite not being trained on this task explicitly. As a result, more recent unsupervised methods tend to not train their own backbones from scratch and instead explore ways to aggregate 61]. align [14 these features across images.
cycle consistencies 44 47 48
Geometry-aware methods. Inspired by the classic correspondence setting in vision that relies on geometric constraints to match the same 3D locations across views [10 42 utilizing geometry cues is an effective way to learn SC. For SC, the underlying assumption is that different instances from the same object category share similar spatial structure. Flow and rigidity constraints are often used in tracking (53 (88 and unsupervised SC 255 14 4]. Recent studies have shown that ambiguities caused One potential way to mitigate these errors is to develop 3D-aware methods. Initially, this has been explored by building correspondence across images by matching points along the surface of objects to 3D meshes 66 24 37 [6) II].
by symmetric objects are major source of errors in SC [62 32].
but the requirement for meshes greatly limits the applicability of these approaches. More recently, methods have been proposed to learn 3D shape from image collections [36] 57 3. However, these methods require solving multiple problems at once, i.e., estimating object shape, deformations, camera pose, and are therefore limited to specific types of object categories and tend to break easily when applied to more complex shapes. Recent advances in monocular depth [40 [5 21 55 [3] [54]
and geometry prediction (55) allows for reliable geometry estimation from a single image, which can be leveraged for imparting 3D-awareness in into SC methods.
Concurrent work [52 also proposes to geometrically align images in 3D using depth maps to build category prototypes. It is designed around a test-time alignment of the 3D prototype to the test image. This requires knowing the test category beforehand, having build a dedicated prototype for it, having a segmentation mask for the test instance, and solving a computationally expensive alignment problem, limiting its applicability to severely constrained scenarios where the category has been seen during training and throughput it not an issue. In comparison, while our model shares similar concept of building 3D category prototype, our goal is to design generalizable SC pipeline by training a correspondence head on top of backbone. This allows our model to compute correspondence in single feedforward pass and generalize to new categories all the while not requiring knowledge of the test category or segmentation.
3

---

Posed Pointoloud 2t
Depth
Backproject
B
Impult imaaee
Sogmantation
Canonical
Goordinate Prediction ta
Local
Predictor +
Learned Cananical Keypeinta(PP
Canonical Pointcloud 8
Figure 2: Overview of our approach. We extract segmentation masks and depths maps from training images and backproject object points produce the posed point clouds X We predict dense features with 4 and match them against our jointly learned sparse category prototype (P, 2) to produce canonical point clouds X The local geometric alignment between the two provides supervision for updating +.
3 Method
3.1 Overview
Let E #R be an RGB image depicting an object, defined over the image domain IR,,, lattice of size Hf x W. Our objective is to learn function (1,, u) - which maps each pixel coordinate u E A to a descriptor w E RM The descriptor w should be semantically consistent, be meaningfully aligned across different images of objects from the same category, and be invariant to changes in pose and shape. Once is learned, SC between two images and I', depicting the same object category, can be established by finding, for pixel u in image I, the most similar pixel u' in the other image I'. This done by querying nearest-neighbor matching in descriptor space according to distance d, typically the cosine distance, i.e., d(a b) 1 (a, ()/[||||22|bb
u' arg min di (1, u), 4
In the standard supervised SC task, where each image 7(m) is annotated with sparse set of semantic keypoints IC {ke, k where E A. A common strategy, adopted in recent works 18 descriptor function ([I, u) that produces local descriptor for each pixel in 7 such that the descriptors of corresponding keypoints in paired images are close in feature space. While this sparse keypoint supervision helps the model learn semantically meaningful descriptors for the annotated keypoints, it does not guarantee generalization to unlabeled regions of the object.
given training set,
(I ((N)))
9
31
61 62). to learn a
A promising direction to address this limitation is incorporate 3D geometry by assigning each pixel a coordinate in an object centric reference frame. Prior works [46] 32) explores this idea by projecting object surfaces onto spherical coordinate system, with each coordinate the sphere corresponding to a different characteristic point of the object. However, this necessitates inferring both object shape and the viewpoint from collection of 2D images, highly ill-posed problem, which requires generating pairs through synthetic warps resulting unrealistic shapes [46] or relies on viewpoint supervision which can be challenging to predict automatically. In the next section show how to combine sparse keypoints annotations with 3D geometry cues to learn dense and semantically consistent descriptors for every pixel in image. An overview of approach is shown in Fig.
3.2 Canonical Representation Learning
Similar function ( u + E I that maps pixel u in image to its 3D coordinates in the canonical object-centric coordinate system. Unlike spherical representations prior work [46] 332, we do not impose any topological constraints surface). We parameterized the canonical representation by a set of 3D keypoints P (pi...., }
46 32 we aim learn a 3D canonical representation for each object category. along with
the canonical representation (e.g., enforcing a spherical
where each p; corresponds to labeled keypoint k; in I. A crucial aspect of this parametrization

---

that unlike the labeled image keypoints K, P is shared across all instances of the category, and is invariant object instance. viewpoint, and pose, ensuring the canonicity of the representation.
To compute P. we first estimate the 3D coordinates of each keypoint & in image using monocular depth model (T, - IR+ and then backproject it using estimated camera intrinsics A E RR3x3 as follows:
k A:)
-1 [Re Ky. 1]
where & = (kr. ky) and k E IR3 represents the 3D 'posed' coordinate. We denote the set of backprojected coordinates as K (ki..... }
To align K with P. we compute a rigid transformation, comprising rotation R E SO(3) and translation T E R and scale IR such that M s|R|TT T R We optimize the canonical keypoints P by minimizing their alignment error across the training set by solving a generalized Procrustes problem:
min Σ P::
p. M M I1
where M
arg min M
Σ
p:
MR
112-
We use the Kabsch-Umeyama algorithm 19 50 to compute the optimal transformation M (x) between the canonical and posed coordinates keypoints. To prevent the degenerate solution where 8 0 leads to a global collapse, we modify the procedure to constrain 2 1. This ensures objects cannot shrink in size, effectively resizing them the size of the largest object in the training set. Even though alignments rely on only a sparse subset of visible keypoints per-image, i.c. occluded and out-of of-frame points not considered, we find this sufficient to recover globally consistent arrangement for P.
Next we associate each canonical keypoint in with learnable descriptor forming a set 2 = (.,,..., 2|z]]- We learn these jointly with 4, using cross-entropy loss over cosine similarities between extracted and canonical descriptors:
N
K
exp(sim(2; z;, (1(mm), (,("),,,))) )(T)
min
log
(4)
exp(sim 2j, 4 (m)) k. )/T)
i=1
with cosine similarity sim(:.) and learned temperature parameter This objective encourages & to
produce distinctive and semantically consistent features across object instances for each keypoint
3.3 Dense Geometric Alignment
So far, we have only modeled object geometry at the sparse level via P. We now extend this to dense correspondence by defining sp[I, u). function that maps every pixel u in I to a coordinate in the canonical space. We compute it as an sirention-weighte. sum over canonical keypoints:
[K]
exp(sim(z, ((1,
((I, u) =
=1 exp(sim(zz ((I, u))/) P;.
(5)
R
This is equivalent to computing descriptors via softmax attention over 2, using 407, u) as queries, z keys, and p as values. For labeled keypoints k,, minimizing Eq. (4) ensures qp(T kt) = Pt-
For each training image I, we can now estimate the dense canonical coordinates X over its pixels via Eq. (5). and the posed coordinates d via depth backprojection using Eq. ( In practice, X and 2 only consist of object points that are selected using an object segmentation mask. We aim to align these two representations so that X properly reflects the object geometry. However, our annotations are only sparse, thus we cannot directly supervise (p, u) for arbitrary coordinates u. Instead, we make the assumption that even though the posed and canonical shape different, they should be locally similar. We encourage the geometric alignment between a small neighborhood of points sampled in the posed space, and their corresponding locations in the canonical space.
For given point q E X we sample its nearest neighbors to obtain C2 in the canonical space, and the corresponding coordinates in the posed space B,, and minimize the alignment error between
5

---

two sets. We also sample neighbors of a given point r E 2 in the posed space, denoted as B,, and compute the loss in the other direction:
1 N N
min 2.2
Σ ==
|
- M
1 + | - 02c c26
1
(6)
where M.25 and M22. are the rigid transformations between the canonical and posed coordinates, computed using the Kabsch- Umeecma algorithm at each iteration as Eq. (3)
In the canonical space, neighbors are selected using a standard k-nearest neighbors strategy. However, this approach can be unreliable in the posed space due object deformations. For instance, in the case of a person eating, the hand might be close to the face in 3D space, and thus points belonging to the face might mistakenly be selected as neighbors of the hand. Instead, we use a seeuud--godesice sampling strategy that samples points along the surface of the object. Starting from seed point, we iteratively grow the neighborhood by selecting the next point with the shortest distance to the current set, effectively approximating surface-bases proximity rather than raw spatial closeness. Pseudocode provided in Alg.
We jointly optimize the descriptor learning loss in Eq. and the geometric consistency loss in Eq. (6) to learn and While these objectives suffice to learn SC model, in practice we build our implementation on Geo-SC [62 and optimize its parameters jointly over the sum of our objective and the original one. inference time, rather than simply querying nearest-neighhor in the descriptor space predicted by 4, we make use of the soft-argmax window matching strategy proposed in [62]. Unlike i,, which relies on category-specinn canonical coordinate set P and descriptors Z, O can be applied to previously unseen object categories directly.
4 SPair-U: A Benchmark for Evaluating Unseen Keypoints
As illustrated in Fig. [ the performance of the state-of-the-art supervised SC methods 62 degrades significantly when queried on keypoints that are not part of their training sets. We posit that this is caused by models only learning strong representations for these specific points, while largely ignoring the remaining pixels. We would like to assess the performance of SC methods when evaluated on keypoints that were previously unseen (i.e., not the labeled set) at training time. A possible solution is to exclusive sets of keypoints, seen and unseen, between training and evaluation. However, this strategy would reduce the supervision available, and require retraining previous techniques for evaluation.
existing dataset while splitting the annotations into mutually
Instead, we introduce new evaluation benchmark, SPair-U, by labeling additional keypoints from the SPair-7lk dataset 34. We added at least four new points for each of the 18 categories found in SPair-71k. For animals, we focused on additional joints on the limbs, and for vehicles we added semantic parts that were not already labeled, e.g., windshield or fenders. Boats, bottles, potted plants, and tv monitors keypoints are not semantic per se in SPair-7Ik, but are rather spread around on the outline of the object. Thus, we added midway points between those already defined. In total, we add 1,272 new individual test keypoint annotations resulting in 19,990 new keypoint pairs spread across 8,254 image pairs We illustrate some of these annotations in Fig. 3 and the full list with more details can be found in Table A33 As shown in Fig. [ current supervised methods tend to predict locations of keypoints seen during when queried on the new SPair-U U points.
SPain-U Datuset Statistics
8254
keypsints
1.272
keypsitt
19,000
images new points
freguent kp
occurrences trequent kp
average occumences
average points per image
Figure 3: Example keypoint annotations from SPair-U evaluation dataset. It utilizes the
images as the SPair-71k dataset (44, hut adds additional keypoints not present 7Pki This enables benchmarking of SC methods on the existing keypoints along with our new ones. On the right we summarize the main statisties of our new dataset.

---

Table 1: Results standard evaluation keypoints for SPair-7Ik. Per-image PCK reported. this table and the following: All models use the soft matching stralegy described in 62] except those followed by 9 Models with a dagger' benefit from AP-10K pretraining. Models in the K category use keypoint supervision. while A do not. Best results are bolded and second best are underlined.
scores
R
W
A
PT
W
mf
86
Q
JVg
K SD 6]
62.8
52.7
80.
31.2
43.4
19.1
35.6
76.0
32.0
57.6
30.9
59.7
51.0
47.3
48.6
43.8
61.8
52.9
32.0
DINOV2 (39
73.4
60.2
88.8
43.2
41.1
46.7
45.1
75.0
33.4
593
66.1
69.6
60.7
66.6
30.7
61.3
54.2
23.9
55.3
DINOv2+5D
m1
73.8
61.0
89.6
40.2
52.5
47.4
44.1
61.1
41.5
76.8
64.8
70.5
61.7
66.3
54.3
62.7
63.5
52.4
61.1
SphericalMaps
82
76.2
00.1
90.0
46.3
53.0
74,9
68.0
83.8
45.1
81.7
67.5
75.4
09.1
58.9
50.0
67.5
73.9
38.1
66.1
K SConSan
13
57.1
40.3
78.3
38.1
51.8
57.8
47.1
67.9
25.2
71.
63.9
49.3
45.3
49.5
48.8
40.3
77.7
69.7
55.3
CATS+*
60.6
46.9
82.5
56.8
72.8
75.8
65.4
50.9
48.2
74.9
59.8
DHF 81
87.2
54.3
64.9
76.5
Geo
70.7
95.8
64.8
91.0
91.8
87.9
82.3
96.6
Geo SCT
02
92.0
76.1
97.
70.4
70.5
91.4
89.7
92.2
13.4
95.0
90.5
87.7
$1.8
91.6
$2.3
83,4
96.5
85.3
85.6
Dars
86.
72.6
95.1
70.7
64.8
94.6
90.3
82.4
70.7
94.1
84.8
83.0
30.5
87.0
79.1
77.5
95.8
82.8
82.9
Ours!
92.2
76.3
96.5
72.0
08.
95.0
90.8
93.1
75.1
942
91.2
86.0
82.1
91.7
30.0
812
95.8
34.0
95.4
Table 2: Results unseen keypoints on our SPair-U U benchmark. Per-image @CK Labor scores unseen keypoints are reported.
6
R
A
PT
H
mf
*
avg
K. SD
6J
73.2
71.8
48.8
31.7
43.0
55.1
47.2
25.4
35.9
50.4
46.2
41.6
59.9
53.1
57.8
36.1
50.6
19.5
47.4
DINOv2 19
G
88.2
75.6
79.0
53.9
89.8
54.1
60.0
43.9
14.8
67.3
64.5
53.6
75.8
79
37.8
45.6
53.3
8.4
54.9
DINOv2+5D
68.0
50.4
22.3
48.2
47.9
62.3
61.5
64.8
15.0
73.0
64.7
58.2
25.5
80.0
62.7
46.1
55.9
59.4
SpharicalMaps
82
SConSan
18
26.9
25.0
31.4
16.6
15.4
27.3
20.1
32.7
CATS++*
14.0
35.9
50.4
56.5
51.6
43.3
DINGASD
73.6
35.8
67.7
64.3
67.9
86.8
60.9
70.1
55.8
57.8
42.7
60.0
Gev-SC 62
80.9
71.4
51.8
65.3
36.9
91.0
70.8
55.7
36.9
55.7
79.2
53.7
66.5
62.3
61.1
39.0
39.0
17.4
56.9
Geo
62
74.6
70.6
55.5
06.1
36.4
35.1
72.3
50.1
40.1
50.6
85.3
65.7
52.9
61.9
06.6
41.8
36.6
13.8
57.1
Durs
80.3
74.5
706
67.1
40.2
92.9
72.7
538
45.8
68.5
75.3
62.0
67.8
65.4
68.1
45.4
47.9
30.5
62.4
Ours!
66.1
5 Experimental Results
5.1 Implementation Details
Our 3D prototype approach is complementary to existing semantic correspondence architectures, thus we can add it as an additional objective on top established models. We base our experiments on Geo- SC [62), strictly following their provided hyperparameters, c.g., learning rate, batch size, optimizer, scheduler, and epoch count, simply adding our additional loss terms and jointly optimizing and Z alongside Geo-SC's feature extractor Q. We also preserve the contrastive Cypurse and dense Luense objectives with gaussian noise, as well as feature maps dropout and cose-variant augmentation. We refer to the original publication and official implementation for in-depth description of these features. While P and 2 are category-sper -specffic, a single & is trained on the full dataset, allowing generalization to new categories. Furthermore, gradients coming from Eq. (6) are not backpropagated to P, meaning is only optimized using Eq. 33.
Our complete loss term is Lyparse + Leense + Lp + 0.3 x L2 + Lgoom- where Lp, Lz, and fegoom correspond Eq. B), Eq. (4) and Eq. (6) respectively. The justification for setting the weight
Az = 0.3 is provided in Appendix B
In practice, t consists of additional bottleneck layers trained top of frozen DINOv2 and SD backbones. We extract depth maps and camera intrinsics using MoGe (54), and use Segment Anything 23 to obtain segmentation masks. Importantly, these only used during training. During evaluation, matches are computed only from the predictions of $ Additional implementation details can be found in Appendix A
5.2 Quantitative Results
We first compare our models other SC approaches on the SPair-71k bench- mark 34 which contains images from 18 categories. We use the standard PCK@0.1 which considers a match to be correct its prediction lies within distance 0.1 x max(h, w) of the ground truth location, where (h, w) is the height and width of the target object bounding box. Typically, supervised models report per-image PCK, i.e., the average score of each image per-category, while unsupervised ones use per point PCK, i.e., the average number of correct matches per-category. In order to properly compare results between the two families, we recompute per-image PCK for all models, which results in small drop for the unsupervised models.
Seen SPair-7lk.
7

---

Table 3: Cross- benssbbenchark evaluation held-out datasets. Scores reported using PCK with different
thresholds. Here, only keypoint supervision from Spair-7lk is used for supervised models.
Spuir 21k
SpairU
AP-10K
AP-10K C5
AP-10K CF
PF-PASCAL
threshold
0.10
0.01
0.10
0.01
0.05
0.10
0.05
0.10
0.15
+ SD
81 61
6.3
37.3
52.0
33
28,0
47.4
36,9
52.5
6.6
32.6
47.9
4.3
24.6
37.6
66.1
80.0
85.3
DINOV2
10 61
68
38.0
55.3
17
32.4
54.9
10.5
44.8
63.6
888
41.7
61.6
1.7
34.7
52.0
62.4
78.2
83.5
DDD 61
82
44.2
61.1
4.7
37.0
59.4
11.7
47.4
65.8
10.0
44.0
63.5
7.7
35.4
52.4
72.5
85.6
90.3
SphuticalMags
8.2
47.7
66.1
4.5
36.2
61.0
12.5
48.5
66.7
10.6
44,9
63.6
8.0
35.7
52.1
74.6
88,9
93.2
DINO+SD (S)
59.3
15.1
54.3
13.6
Geo-SC
20.0
72.2
83.2
1.6
35.5
56,9
16.6
55.8
70.5
15.2
52.4
67.7
11.9
45.9
59.6
25.3
87.0
90.7
Ours
20.5
22.1
$29
42
37.8
62.
10.3
55.8
21.
15.1
53.0
69.0
11.2
46.1
61.1
75.8
57.5
91.2
Results on seen keypoints in Table show that our model ranks competitively against other approaches, with marginal 0.2% performance drop on average against its backbone Geo-SC 62 Per-category results show small improvements in nine of the categories, the highest one being 3.6% on bus, and small drop on the other nine, the largest being 2.4% on bottle. Overall, the differences are minor, illustrating that adding our extra objective does not interfere with the original model.
Unseen SPair-U. To evaluate model's ability to generalize unseen semantic points, we assess its performance on our new SPair-U keypoints using per-image PCK 1 bbox for a like-for-like comparison. We exclude the Test-time Adaptive Pose Alignment from [62] since it requires prior knowledge of keypoint semantics to relabel flipped keypoints, which contradicts the assumption that evaluation keypoints are unknown.
As shown in Table 2 results on SPair-U reveal a stark contrast between supervised and unsupervised models. While unsupervised models see only modest performance drop. likely due to increased task difficulty, supervised models experience a significant decline. Many of the pre-existing approaches outperformed by the unsupervised DINO+SD baseline [61] and they are consistently beaten by the weakly-supervinn Spherical Maps 32). Notably, in eight categories, the best-performing model Has not seen keypoints during training, suggesting that supervised approaches behave more like keypoint regressors and fail to generalize to novel correspondences.
Our method also shows some performance degradation SPair-U, but the drop is smaller than that of its backbone Geo-SC. It achieves the highest overall performance, improving upon the best prior supervised model by 6.1%, indicating stronger generalization to unseen keypoints. Nevertheless, the substantial gap between results on SPair- Ik and SPair-U U underscores a broader limitation: despite recent progress, most models struggle to move beyond sparse keypoint supervision toward robust, general semantic correspondence.
Cross- bosshabncchmrk evaluation. We further evaluate our model on four benchmarks: SPair-71k, SPair-U, AP-10k 622, and PF-PASCAL [1S]. While most supervised SC methods train separate models for each benchmark, this setup encourages overfitting to the benchmark and is impractical for real-world use. Instead, we advocate for evaluating generalization by training a single model on one dataset and testing it across multiple benchmarks. We choose SPair- 7l for training due to its balanced mix of object and animal categories, making it suitable for generalization. To ensure fairness, we exclude models pretrained on AP-10K and standardize evaluation using the windowed soft-argmax protocol from 622]
As shown in Table 3 while Geo-SC 62 achieves the best performance on the standard SPair-714 k test
set, underperforms on all other benchmarks, highlighting its limited generalization. Notably, even a simple supervised DINO+SD [61 baseline outperforms Geo-SC at the standard O. 00 threshold when using the same soft window matching strategy. This stands in sharp contrast to the findings in (62) where Geo-SC consistently outperforms its baseline by 10% on the three AP-10K benchmarks, when both models are trained on AP-10K, indicating potential overfitting to that dataset.
Consistent with earlier observations, clear pattern emerges: models trained without keypoint supervision maintain stable rankings and performance gaps across datasets, whereas supervised models cluster more tightly in performance when evaluated out of their training distribution, revealing weaker cross-set generalization.
5.3 Qualitative Results
In Fig. we visualize PCA projections of object features produced by our model, Geo-SC, and the unsupervised DINO+SD. Our model produces descriptors that vary smoothly over the object surface while uniquely identifying each point. In comparison, the predictions of Geo-SC are noisier, with
8

---

Input
DINDASDCRS
Figure 4: PCA visualization of the feature maps from different models. Note that PCA is computed object features only. The inclusion of geometric constraints during training results in fewer high frequency artifacts in the predicted feature maps our approach.
Gco-SC
WATTEO St
GiIRED St
Figure 5: Visualization of keypoint matches for randomly selected object points. On each source image (left) we randomly sample points on the object of interest and compute their match on the target (right). Colored lines used as way distinguish the points.
sudden discontinuities (e.g., bus) and uniform descriptors regions that have no keypoints (e.g., the body of the cat and the cow). Meanwhile, the unsupervised features fail to separate repeated parts (e.g., plane engines) and produces noisy features in textureless areas (e.g., tv).
In Fig. 5 we further qualitatively evaluate our model's ability generalize unseen points. We randomly sample points on the source object and compute their matches on the target. Compared with Geo-SC, our approach exhibits better robustness against matching outside the target object, as well as better spatial awareness of points (c.g.. Geo-SC matching points from the bottom of the source bus the roof of the target), leading higher matching quality. Further examples, along with comparisons to the unsupervised DINOv2+SD backbone, are provided in Fig. A4
Following [52). we visualize our learned canonical shapes in Fig. by collecting predicted canonical coordinates of multiple objects in order to overlap their partial point clouds over the training data. We observe that the spatial organization of P, i.e., the large bold points, captures the general shape of the category, and the predicted coordinates densely span the object surface. Interestingly. our parametrization of the canonical shape Eq. (5) forces predicted coordinates lie within the convex hull of P. which explains the incomplete wheel on the motorbike. We also observe that very few points mapped towards the end of the train, which attribute the varying length of trains across instances and the bias toward frontal viewpoints. Note that contrary to [52]. these are simply visualizations and are not used for inference, meaning this limitation is unlikely to significantly affect performance.

---

Acroplane
Boat
Bottle
Car
Chair
Dog
Motorbike
Person
Train
Figure 6: Visualization learned canonical shapes. Large points correspond each being attributed distinctive color for visualization. Small points are predicted canonical coordinates objects, colored with PCA of the features predicted by go.
Limitations
While augmenting the test set of SPair-71k with new keypoints enables evaluate existing techniques with their provided models and code, our proposed benchmark SPair-U inherits some drawbacks from SPair-71k.
particular, the with over 8,000 test pairs, and the categories are restricted to only common objects and animals. Furthermore, some categories were already labeled with a high number of keypoints where it is potentially easier to detect the newly added ones by relating them to the existing ones. While our findings related to generalization issues in supervised SC techniques remain valid, a larger, higher- quality held-out set of images and keypoints would be beneficial for more extensive evaluation.
is small, consisting of only 481 images but
Compared to prior supervised methods [8 sion in the form of depth maps and segmentation masks, similar to [52], although in our case, they are only used during training. Furthermore, unlike [32]. which relies on camera viewpoint annotations that off-the-ste shelf models cannot reliably provide, we obtain all additional signals using existing pretrained models.
our approach incorporates additional supervi-
In Eq. we assume there exist global rigid transformation between the posed keypoints and their canonical counterpart, which in practice is not the case especially for deformable objects. However, the sole purpose of this step is to optimize P into a coarse spatial organization 3D) keypoints (e.g.. making sure that the left hand keypoint generally sits opposite of the right one) order allow the computation of local geometric alignment in Eq. (6). We show in Fig. 6 and Section (.2iion(52)) that despite this coarse assumption, our method is able recover reasonable 3D structure and performs well on deformable objects.
Finally, assumption that geometry is good proxy for semantics breaks down for complex object categories with diverse spatial part configurations. For example, cabinets may have different numbers of doors that open in various directions, leading to inconsistent placement of features like handles. Finally, we do not foresee any negative social impacts of our work.
7 Conclusion
We addressed the challenge of estimating semantic correspondences across different image instances of the same object category. Although recent supervised methods perform well on keypoints seen during training, show that they often struggle to generalize unseen keypoints. To overcome this, we introduced a new approach that incorporates geometric constraints during training by learning a continuous canonical manifold specific each category. Our method outperforms both supervised and unsupervised baselines, as demonstrated on SPair-U - new dataset we introduce with additional keypoint annotations for the widely used SPair-71k benchmark.
Acknowledgements. HB was supported by the EPSRC Visual AI grant EP/T028572/1.
10

---

References
[1] Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep vit features as dense visual descriptors. ECCV Workshops, 2022.
[2] Mehmet Aygün and Oisin Mac Aodha. Demystifying unsupervised semantic correspondence estimation. In ECCV, 2022.
[3] Mehmet Aygün and Oisin Mac Aodha. SAOR: Single-View Articulated Object Reconstruction. In CVPR, 2024.
[4] Nir Barel, Ron Shapira Weber, Nir Mualem, Shahaf E Finder, and Oren Freifeld. Spacejam: lightweight and regularization-fre method for fast joint alignment of images. In ECCV, 2024.
[5] Shariq Farooq Bhat, Reiner Birkl. Diana Wofk, Peter Wonka, and Matthias Müller, Zoedepth: Zero-shot transfer by combining relative and metric depth. arXiv preprint arXiv:2302.12288.
2023.
[6] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
[7] Xinle Cheng, Congyue Deng, Adam W Harley, Yixin Zhu, and Leonidas Guibas. Zero-shot image feature consensus with deep functional maps. In ECCV, 2024.
[8] Seokju Cho, Sunghwan Hong, Sangryul Jeon, Yunsung Lee, Kwanghoon Sohn, and Seungryong Kim. Cats: Cost aggregation transformers for visual correspondence. NeurIPS, 2021.
[9] Seokju Cho, Sunghwan Hong, and Seungryong Kim. Cats++: Boosting cost aggregation with convolutions and transformers. PAMI, 2022.
[10] Andrew J Davison, Ian D Reid, Nicholas D Molton, and Olivier Stasse. Monoslam: Real-time single camera slam. PAMI, 2007.
[11] Niladri Shekhar Dutt, Sanjeev Muralikrishnan, and Niloy J Mitra. Diffusion 3d features (diff3f): Decorating untextured shapes with distilled semantic features. In CVPR, 2024.
[12] Abhishek Dutta and Andrew Zisserman. The VIA annotation software for images, audio and vidco. In International Conference on Multimedia, 2019.
[13] Xiao Fu, Wei Yin, Mu Hu, Kaixuan Wang, Yuexin Ma, Ping Tan, Shaojie Shen, Dahua Lin, and Xiaoxiao Long. Geowizard: Unleashing the diffusion priors for 3d geometry estimation from single image. In ECCV, 2024.
[14] Kamal Gupta, Varun Jampani, Carlos Esteves, Abhinav Shrivastava, Ameesh Makadia, Noah Snavely, and Abhishek Kar. Asic: Aligning sparse in-the-wild image collections. In ICCV, 2023.
[15] Bumsub Ham, Minsu Cho, Cordelia Schmid, and Jean Ponce. Proposal flow: Semantic correspondences from object proposals. PAMI, 2017.
[16] Kai Han, Rafael S Rezende, Bumsub Ham, Kwan-Yee K Wong, Minsu Cho, Cordelia Schmid, and Jean Ponce. Scnet: Learning semantic correspondence. In ICCV, 2017.
[17] Or Hirschorn and Shai Avidan. A graph-based approach for category-agnostic pose estimation. In ECCV, 2024. 2024.
[18] Shuaiyi Huang, Luyu Yang, Bo He, Songyang Zhang, Xuming He, and Abhinav Shrivastava. Learning semantic correspondence with sparse annotations. ECCV, 2022.
[19] Wolfgang Kabsch. solution for the best rotation to relate two sets of vectors. Foundations of Crystallography, 1976.
[20] Angjoo Kanazawa, David Jacobs, and Manmohan Chandraker. Warpnet: Weakly supervised matching for single-view reconstruction. In CVPR, 2016.

---

[21] Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger. Rodrigo Caye Daudt, and Kon- rad Schindler. Repurposing diffusion-basee based image generators for monocular depth estimation. In CVPR, 2024.
[22] Seungwook Kim, Juhong Min, and Minsu Cho. Transformatcher: Match-to-match attention for semantic correspondence. In CVPR, 2022.
[23] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan-Yen Lo, et al. Segment anything. In ICCV, 2023.
[24] Nilesh Kulkarni, Abhinav Gupta, and Shubham Tulsiani. Canonical surface mapping via geometric cycle consistency. In ICCV, 2019.
[25] Svetlana Lazebnik, Cordelia Schmid, and Jean Ponce. Semi-local affine parts for object recognition. In BMVC, 2004.
[26] Ce Liu, Jenny Yuen, and Antonio Torralba. Sift flow: Dense correspondence across scenes and its applications. PAMI, 2010.
[27] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
[28] Changsheng Lu and Piotr Koniusz. Few-shot keypoint detection with uncertainty learning for unseen species. In CVPR, 2022.
[29] Changsheng Lu and Piotr Koniusz. Detect any keypoints: An efficient light-weight few-shot keypoint detector. In AAAI, 2024.
[30] Changsheng Lu, Zheyuan Liu, and Piotr Koniusz. Openkd: Opening prompt diversity for zero-and few-shot keypoint detection. In ECCV, 2024.
[31] Grace Luo, Lisa Dunlap, Dong Huk Park, Aleksander Holynski, and Trevor Darrell. Diffusion hyperfeatures: Searching through time and space for semantic correspondence. NewrIPS, 2023.
[32] Octave Mariotti, Oisin Mac Aodha, and Hakan Bilen. Improving semantic correspondence with viewpoint-guided spherical maps. In CVPR, 2024.
[33] Juhong Min and Minsu Cho. Convolutional hough matching networks. In CVPR, 2021.
[34] Juhong Min, Jongmin Lee, Jean Ponce, and Minsu Cho. Spair-71k: A large-scale benchmark for semantic correspondence. arXiv: 1908. 10543, 2019.
[35] Juhong Min, Jongmin Lee, Jean Ponce, and Minsu Cho. Learning to compose hypercolumns for visual correspondence. In ECCV, 2020.
[36] Tom Monnier, Matthew Fisher, Alexei Efros, and Mathieu Aubry. Share with thy neighbors: Single-view reconstruction by cross-instancs consistency. In ECCV, 2022.
[37] Natalia Neverova, David Novotny, Marc Szafraniec, Vasil Khalidov, Patrick Labatut, and Andrea Vedaldi. Continuous surface embeddings. In NeurlPS, 2020.
[38] Dolev Ofri-Amar, Michal Geyer, Yoni Kasten, and Tali Dekel. Neural congealing: Aligning images to a joint semantic atlas. In CVPR, 2023.
[39] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. TMLR, 2024.
[40] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. PAMI, 2020.
[41] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
12

---

[42] Johannes L Schonberger and Jan- MancMel Frahm. cenneture-ffo---eee motion revisited. In CVPR, 2016.
[43] Leslie N Smith and Nicholay Topin. Super-convergence: Very fast training of neural networks using large learning rates. In Artificial intelligence and machine learning for multi- operations applications, 2019.
[44] Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emer- gent correspondence from image diffusion. In NeurIPS, 2023.
[45] James Thewlis, Samuel Albanie, Hakan Bilen, and Andrea Vedaldi. Unsupervised learning of landmarks by descriptor vector exchange. In ICCV, 2019.
[46] James Thewlis, Hakan Bilen, and Andrea Vedaldi. Unsupervised learning of object frames by dense equivariant image labelling. NeurlPS, 2017.
[47] Prune Truong, Martin Danelljan, Fisher Yu, and Luc Van Gool. Warp consistency for unsuper- vised learning of dense correspondences. In ICCV, 2021.
[48] Prune Truong, Martin Danelljan, Fisher Yu, and Luc Van Gool. Probabilistic warp consistency for weakly-supervised semantic correspondences. In CVPR, 2022.
[49] Nikolaos Tsagkas, Jack Rome, Subramanian Ramamoorthy, Oisin Mac Aodha, and Chris Xi- aoxuan Lu. Click to grasp: Zero-shot precise manipulation via visual diffusion descriptors. In IROS, 2024.
[50] Shinji Umeyama. Least-squares estimation of transformation parameters between two point patterns. PAMI, 1991.
[51] C. Wah, S. Branson, P. Welinder, Perona, and S. Belongie. The caltech-ucsd birds-200-2011 dataset, 2011.
[52] Krispin Wandel and Hesheng Wang. Semalign3d: Semantic correspondence between rgb-images through aligning 3d object-class representations. In CVPR, 2025.
[53] Qianqian Wang. Yen- Yu Chang, Ruojin Cai, Zhengqi Li, Bharath Hariharan, Aleksander Holynski, and Noah Snavely. Tracking everything everywhere all at once. In ICCV, 2023.
[54] Ruicheng Wang, Sicheng Xu, Cassie Dai, Jianfeng Xiang, Yu Deng, Xin Tong, and Jiaolong Yang. Moge: Unlocking accurate monocular geometry estimation for open-domain images with optimal training supervision. In CVPR, 2025.
[55] Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made easy. In CVPR, 2024.
[56] Thomas Wimmer, Peter Wonka, and Maks Ovsjanikov. Back to 3d: Few-shot 3d keypoint detection with back-projected 2d features. In CVPR, 2024.
[57] Shangzhe Wu, Ruining Li, Tomas Jakab, Christian Rupprecht, and Andrea Vedaldi. MagicPony: Learning articulated 3d animals in the wild. In ICCV, 2023.
[58] Yuxi Xiao, Qianqian Wang, Shangzhan Zhang, Nan Xue, Sida Peng, Yujun Shen, and Xiaowei Zhou. Spatialtracker: Tracking any 2d pixels in 3d space. In CVPR, 2024.
[59] Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. NeurlPS, 2024.
[60] Hao Zhang, Lumin Xu, Shenqi Lai, Wenqi Shao, Nanning Zheng, Ping Luo, Yu Qiao, and Kaipeng Zhang. Open-vocabulary animal keypoint detection with semantic-feature matching. IJCV, 2024.
[61] Junyi Zhang, Charles Herrmann, Junhwa Hur, Luisa Polania Cabrera, Varun Jampani, Deqing Sun, and Ming-Hsuan Yang. tale of two features: Stable diffusion complements dino for zero-shot semantic correspondence. In NeurIPS, 2023.
13

---

[62] Junyi Zhang, Charles Herrmann, Junhwa Hur, Eric Chen, Varun Jampani, Deqing Sun, and Ming- Hsuan Yang. Telling left from right: Identifying geometry- aware semantic correspondence. In CVPR, 2024.
[63] Xu Zhang, Wen Wang, Zhe Chen, Yufei Xu, Jing Zhang, and Dacheng Tao. Clamp: Prompt- based contrastive learning for connecting language and animal pose. In CVPR, 2023.
[64] Tinghui Zhou, Philipp Krahenbuhl, Mathieu Aubry, Qixing Huang, and Alexei A Efros. Learning dense correspondence via 3d-guided cycle consistency. In CVPR, 2016.
14

---

Appendix
A Additional Implementation Details
General implementation. We base our model on Geo-SC 62]. reusing all default hyperparameters that come with the official implementation e.g., training for epochs using Adam [27 optimizer with 1.25 x 10-3 initial learning rate and 1.0 x 10 weight decay, coupled with one-cycle learning rate scheduler [43], with a batch size of 1. Every 5, 000 iterations, models evaluated the validation split, and the best performing model is retained. For evaluation, unless stated otherwise, a soft-argmax window of size 15 is used.
Experiments were performed on a single NVIDIA RTX 6000 Ada Generation, using pre-extracted DINOv2 and SD feature maps, depth maps, and segmentation masks. Training model on SPair-7Ik consumes roughly 4.3GB of VRAM over 8 hours, representing an increased memory cost over Geo-SC's 2.9GB, mainly due the many X₂ and X₂ we sample, and doubling of runtime from roughly hours. At inference time however, there is no impact as we estimate matches using features predicted with T in the exact same way Geo-SC does.
Point cloud sampling. When backprojecting image points using Eq. (2. we obtain a point cloud whose size depends on the number of visible object pixels. This can cause imbalance of the samples the final loss terms, with larger objects contributing more. Furthermore, these point clouds have very specific grid-like structure inherited from the bitmap format of images, which is dense on surfaces parallel to the image plane and gets sparser as the angle increases. Therefore, we first subsample each training point cloud to size of k 1024 points using farthest point sampling to obtain a fixed number of well-distributed samples.
When computing the local geometry loss Lgeam- we use neighborhoods of size k' = 64. An ablation study of this parameter is provided in TAble|AIc We also provide in Alg. [ the pseudocode for the sampling strategy used to obtain B, in the posed space.
Input: Point cloud X, seed point number of neighbors k
Output: neighbor set B
// Start with the seed
B {p]:
while [B] < l do
// Filter out already selected points
X' +- X B;
for x in X' do
// Compute the distance to the closest point in B
D2 + mingeB || y1ll
end
// Add the point with minimal distance to B B (arg minaex Dz);
end
return B;
Algorithm 1: Exeudo-geodest sampling
B Ablations
General ablations. We perform ablations of our designs in Table Al and report results on the the SPair-71k (34) validation set which helps chose the best performing model.
possible to ablate individual loss terms as they each have a distinct purpose without which the prototype cannot properly be learned: Lp optimizes P, Lz optimizes Z, and Zgomm provides a dense supervision signal, i.e., loss for (II, u) when u is an arbitrary object pixel, i.e., not keypoint. We can however examine the different effect of our and Geo-SC's specific loss terms on both SPair-7Ik and SPair-U UU by comparing a simple supervised approach using a DINOv2 and SD backbone trained only using Losparees adding only the Geo-SC specific losses, adding only our canonical prototype losses, and
htngs://gthuubooom///yyygggggoou
15

---

Table A1: Results of different ablations.
(a) Average PC on SPair- 7lk test set and SPair-U for different models. Geo-SC losses
DINOv2+SD
Canonical prototype
Spair-71k
Spair-U
76.5
60.0
85.6
57.1
75.9
66.0
85.4
66.1
(c) Average PCK on
(d) Average
PCK@0.1 on SPair-7Ik validation set for different rigidity constraints.
SPair-1lk validation set for different neighborhood size.
(b) Average @CK@0.1 SPair-
7lk validation set general ablations.
Neighborhood size
PCK
Ablation
PCK
4
85.5
Selected points
PCK
A =1
85.9
8
86.1
86.6
16
85.9
86.4
Az 0.1
86.1
K-nn sampling
85.9
32
86.5
5
86.1
Geodesic sampling
86.2
64
86.5
86.4
Full model
86.5
128
86.6
7
86.5
256
86.1
86.0
all
86.5
our proposed model that combines them. Results shown in Table Ala show a clear pattern, i.e., adding our canonical prototype loss results in very small drop in performance seen keypoints (i.e., SPair-71k), which we attribute to the models having less capacity to fully overfit the training keypoint supervision, but endows them with the ability to generalize much better to unseen points (i.e., SPair-U). Conversely, the Geo-SC losses allow models to perform really well on seen keypoints but its effect generalization ranges from harmful null (with vs. without Geo-SC losses). These results also demonstrate that our contributions do not require Geo-SC to work, as they also boost performance of the supervised baseline on unseen keypoints (with without Canonical prototype).
We show that setting λz to or 0.1 both negatively affect performance. We believe this is due to the interaction between Lg and goom as a high A would push & to collapse towards defaulting to predicting keypoint features for most points, while a weight too low prevents correct prediction on the keypoints. We also test different neighbor sampling strategies for X, and Xx, and show that sampling both spaces with either K-nearest neighbor or geodesic sampling is ineffective.
Neighborhood size in C. and B, We experiment with different neighborhood sizes when computing Egsmm and valite the different models on the SPair-7Ik validation set Table Results show little to no effect of the neighborhood size. which is consistent with our previous finding that our losses do not improve performance on semantics points that are present in the training set.
Number of points in Eq. (3) To compute Lp, we compute a global rigid transformation between the posed and canonical keypoints, which our qualitative analysis in Fig. 6 and Fig. Al shows to be reasonable despite being a coarse simplification of the problem. In order to evaluate its impact quantitatively, we evaluate altered version of this procedure where we learn not by globally aligning all keypoints but only randomly sampled local subset of them. Results in Table Ald show that only considering a local neighborhood does not impact the validation performance of our model.
C
Additional Results
C.1 Additional Metrics
Multiple recent works pointed out issues with evaluating using PCK, and proposed additional evaluation metrics to address its limitations.
PCK 2 PCK matches are counted correct even the prediction lies closer to keypoint that
not the target, which can lead to high scores when many points are grouped together, even though the system does not distinguish between them. The authors introduce PCKT which only considers match correct if it lies within the threshold and its closest annotated point is the target.
16

---

Table A2: Evaluation under robust metrics. All metrics per-image averaging. and models window soft-argmax. models are trained on SPair-7Ik, models double dagger' benefit from AP-10K pretraining. Models in the K category use keypoint supervision, while A. do not. Best results are bolded and second best are underlined.
Spain
Spair
Spain-7ik PCK!
Spair-U PCK!
Spain-7Ik
Threshold
0.01
0.05
0.10
0.01
0.05
0.10
0.01
0.05
0.10
0.01
0.05
0.10
0.01
0.05
0.10
0.01
0.05
0.10
# SD
44 [1
58.2
47.5
530
43.4
51.6
58.5
34.4
44.4
3.3
27.7
45.4
4.2
28.3
43.3
1.3
15.3
31.0
DINOV2 [9] 61
$7.8
47.1
52.8
43.3
52.4
60.6
6.7
34.4
46.0
3.7
52.1
52.1
3.6
26.3
43.4
2.3
25.6
47.0
DINDx2+SD
61
38.4
55.9
43.7
54.2
62.8
41.0
52.8
4.7
36.6
56.6
4.8
32.7
50.8
26.1
48.1
SphericalMags
55.4
K DINO+SD (5)
61
39.1
55.4
64.1
43.8
54.7
63.9
130
59.7
72.0
3.6
35.5
57.1
10.2
53.6
09.7
32.1
56.6
Gen-SC
39.8
39.3
67.8
43.8
54.2
62.8
200
69.8
78
4.6
35.1
54.9
17.2
65.6
78.0
3,7
33.6
$5.3
Gen-SC
40.1
61.0
69.2
43.8
54.1
62.8
12.0
73,0
80.9
4.3
35.8
55.3
20.0
70,9
82.3
56.1
Ours
40.0
60.3
69.0
442
56.0
66.0
20.8
72.1
50.7
4.5
41.3
64.2
70.7
82.4
KAP (32 PCK only considers matches when both ground-truth points are visible and does not penalize systems that predict strong similarities for points that do not correspond, for instance between the two opposite sides of a car. KAP reformulates
correspondence evaluation as a binary classification problem between the pixels that close to the target and those those that not. Crucially, penalizes high predictions when a source keypoint is invisible in the target.
Geo-aware subset (GA) 62 Finally, 2 and noted that SC pipelines especially unsuper-
vised ones often make mistakes because of repeated parts and object symmetries. evaluation on the Geo-aware subset of points only, e.g., the points for which there is a symmetric corresponding point.
62 proposed
Results in Table A2 confirm the patterns observed in Section 5.2 For all metrics, supervised models performances drop back down unsupervised-level or worse when evaluated outside their training labels. Interestingly, KAP scores do not widely vary between supervised and unsupervised models, indicating that supervised models are still likely to predict strong similarity between points when none exists.
C.2 Additional Visualizations
We visualize more canonical surfaces in Fig. A1 While the shapes are sensible, we observe some limitations in adequately modeling categories with extreme deformations like birds: points belonging to the wings are predicted close to the body when they are folded, and away when they are spread. However, this is consistent with SPair-7Ik labeling, where the tips are only labeled when the wings are spread.
Bicycle
pot
Figure A1: Visualization of extra canonical shapes. Large points correspond to P. each being attributed distinctive color for visualization. Small points are predicted canonical coordinates of objects, colored with PCA the features predicted by yp.
17

---

We also show some predictions for the unsupervised DINOv2+SD, Geo-SC, and our model on SPair- in Fig. A3 We observe some interesting failure cases: on the aeroplane, the unsupervised model correctly matches the door, while both supervised models incorrectly predict training keypoint. In two occasions, Geo-SC predicts points outside of the object when queried on points that are far from training annotations (cow and person). Finally, two very challenging cases are shown with the chair and the tv, illustrating that generic semantic correspondence still a particularly challenging task.
D Additional SPair-U Details
We annotated images using the VGG Image Annotator 12 We further post-processed the annotations into JSON files replicating the structure of SPair-7Ik annotations, i.e., per-image annotations and a list of testing pairs. This allows SPair-U to function drop-in replacement for SPair-71k evaluation in any semantic correspondence evaluation script. Note that it is designed to be benchmark of unseen semantic points intended for evaluating the generalization ability of SC models, therefore does not come with a training or validation split. We present the full list of keypoint semantics of SPair-U in Table A3 per-category statistics in Table and some keypoint visualization in Fig. A2
Table A3: List SPair-U keypoint semanties. front-left, front-right, rear-left, rear-right doors top and bottom of head tube; front brake; rear brake
Acroplane
Bicycle
Bird
back, chest; left wing wrist: right wing wrist midpoint of bow; front-left, front-right, rear-left, rear-right side midpoints center and corner points of label
Boat
Bottle
Bus
top-left, top-right, bottom-left, bottom-right front-left, front-right, rear-left, rear-right top of the wheel arches
of windshicld
Car
Cat
front-left, front-right. rear-left, rear-right hocks log midpoints; scat edge midpoints; left and right shoulder joints; left and right hip joints; left and right centers of the body: middle of back
Chair
Cow
Dog
front-left, front-right, rear-left, rear-right hocks
Horse
and right shoulder joints; left right hip joints
Motorbike
front fender midpoint: seat front edge, seat rear edge; engine compartment center forchead center; navel: neck base; left hip joint, right hip joint
Person
Plant
center of pot: midpoints edges; midpoints rim
Sheep
left and right shoulder joints; left and right hip joints
Train
locomotive top-left, top-right, bottom-left, bottom-righe right center point: top-left, top-right, bottom-left, bottom-right quadrant centers
Table A4: Per-category statisties for SPair-U U benchmark.
db
*
W
F
H
Y
*
G
Aag
Image count
27
27
23
25
26
23
27
27
Number pairs
256
576
656
338
200
510
552
438
420
488
746
218
314
600
Count new semantic lhbes
Tonal laheled points
81
116
Avenge romber visitle points
23
188
2.4
scmoky imagee
keyponnt
keypoint occurrence
keypontt
per pair
18

---

Acroplane
Bicycle
Bird
TET
Boat
Bottle
Bus
Car
Cat
Chair
Cow
Dog
Horse
Motorbike
Person
Plant Pot
Sheep
Train
TV
Figure A2: Visualization keypoint annotations from SPair-U. U, Colors represent keypoint IDs.
19

---

DINOv2+SD
Geo-SC
Ours
Figure A3: Visualization matches for SPair-U. Green lines correct, red incorrect.
20

---

DINOv2+SD
Geo-SC
Ours
Figure A4: Visualization matches for randomly selected object points. Colors are provided as a way to distinguish the points.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We illustrate the lack generalization supervised SC models by introducing novel benchmark of unseen points, while showing qualitatively and quantitatively that our proposed approach outperforms prior works in generalization settings
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We discuss our limitations in Section 6 where we highlight shortcomings in our proposed benchmark and discuss the key limitations associated the assumptions made when designing our model.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
22

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
We do not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provided detailed implementation details in the supplementary material. Code for reproducing our results will be provided upon publication.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
23

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We plan to release both code for our model and experiments as well as our new SPair-U benchmark upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://niis.cc.
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
7 //nips. //nips.cc/publll wwtpc//pppp.g..deeeeooeepepe httpa//cguuldee/ooeeeuubiiieses prrooeSubmissionool111.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Detailed implementation details and training settings are provided in the supplementary material. We plan to release the code detailing training and evaluation settings upon acceptance.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Retraining all methods evaluated on the multiple benchmarks we include in experiments would incur computational cost beyond our means. Instead, report performance using the standard protocol used in the semantic correspondence literature, where results averaged many categories and many images.
Guidelines:
The answer NA means that the paper does not include experiments.

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide information related to computational resources used our supple- mentary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We have reviewed the code of ethics and confirm paper conforms it.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
25

---

Justification: We briefly discuss broader impacts at the end of our limitations section. We do not anticipate any negative societal consequences stemming from our method. However, like any learnin-bbsee methods our approach is subject to potential limitations stemming from biases in the training data and there could be certain negative consequences in specific high stakes use cases if the outputs of the model are used without being validated first.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: As discussed in the previous answer, we do not anticipate any risks associated with the use of our method SPair-U dataset.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cite and discuss the relevant datasets for evaluation in Section 5
Guidelines:
26

---

The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paaassttssppersseaaee paperswi thcode. com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We use the same data format for SPair-U U as used in SPair- 1 1l to ensure that is drop in replacement for users. We describe the data annotation process in Section 4
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our research does not involve crowdsourcing research with human subjects. The authors themselves were responsible for annotating the correspondences in SPair-U.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
27

---

Answer:
[NA]
Justification:
Our research does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The topic of this paper does not overlap with LLMs. Beyond conventional spell checkers, no LLMs were used in the writing of text for this paper. We did use LLMs assist with visualizing the results.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy for what should or should not be described.
28


Reviews:
[{"review_id": "61Oa5PKjIk", "reviewer": "Reviewer_AwCB", "summary": "The authors proposed a method for semantic correspondence lifts sparse 2D keypoints into a shared canonical 3D space using monocular depth estimation. By learning a continuous, category-specific canonical manifold, the method aims to capture object geometry and guide dense correspondence learning without requiring explicit 3D supervision or camera pose information. Alongside the method, the authors introduce SPair-U, an extension of the SPair-71k benchmark, which includes additional keypoints not used during training. This allows for explicit evaluation of generalization to unseen points. Experiments show that the proposed method outperforms supervised baselines on these unseen keypoints, demonstrating improved generalization.", "questions": "Please see the weaknesses section above.", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- Shifting the focus from sparse, supervised alignment to dense, geometry-aware representation learning\n- Leveraging off-the-shelf components like monocular depth estimation (Moge) and segmentation masks (SAM), and is integrated into an existing model (Geo-SC) in a modular way\n- The introduction of the SPair-U benchmark;  exposing the overfitting tendency of standard SC models and enabling evaluation on unseen keypoints\n\nWeaknesses\n- The method introduces several additional sources of supervision (depth maps, segmentation masks, canonical 3D supervision) that are not available to baseline SC models, raising concerns about fairness in the performance comparison. The use of off-the-shelf modules like Moge and SAM means the observed gains could stem as much from auxiliary data as from the core method itself. This issue is not adequately discussed, and no control experiments are provided to isolate the contribution of each added component.\n- As for the benchmark, while SPair-U introduces new test keypoints, the extension is somewhat limited: only a few extra keypoints are added only for testing data. For the practical utility of the benchmark as a community resource,  it is desirable to provide dense annotations across all images, including those in the training set, by leveraging oracle-like 3D canonical representations.", "was_revised": false, "final_justification": "", "rebuttal": "We thank **AwCB** for their insightful review and for highlighting our shift toward geometry-aware learning and the value of SPair-U for revealing overfitting. As an initial remark, we would like to clarify that our paper does more than just introducing a new semantic correspondence approach and new benchmark dataset. Our goal is to expose a fundamental flaw in the way supervised semantic correspondence pipelines are trained and evaluated, i.e. **issues in the general task design** are the reason behind the generalization gap observed in models. As a result, GeoSC - the top performing model on SPair-71k at the time we wrote our paper - trains a supervised feature predictor on top of frozen DINOv2 and SD features. However, when evaluated on SPair-U, it performs *worse* than simply using the DINOv2 and SD features alone (see Tables 1 and 2). This indicates that **none** of the knowledge it has acquired transfers to new semantics, meaning it fails to  solve anything beyond sparse keypoint detection. We believe this finding is a key contribution of our work, with significant implications for future model and benchmark design.  \n\n**[AwCB-1] Extra segmentation and depth supervision.**  \nWe do acknowledge this additional supervision in our limitations section (see L308). Using the depth and segmentation predictions at training time from pre-trained models to build 3D-aware representations is “free” supervision and one or the other has been adopted before in related works [3, 27, 42, 46]. In particular, concurrent work [46] uses both, even requiring test-time segmentation masks, while in our case they are only used during training. While this regime could be considered unfair, our central claim is that the prevailing evaluation setup is itself biased: top-performing models on SPair-71k fail on SPair-U (Fig. 1). In that context, the purpose of our model is to demonstrate that using a 3D-informed representation can help bridge the generalization gap. \n\nAs stated in our supplementary PDF (L24), it is challenging to isolate the contributions of segmentation (SAM) and depth (MoGE) within our pipeline, as both are required to compute our geometric losses. Removing either of them would prevent the computation of Eq. 6 and revert our model back to Geo-SC. Conversely, as previous approaches generally rely on sparse 2D-based models, there is no straightforward way to directly incorporate depth or segmentation in their design.  Nonetheless, as requested, we performed an additional experiment to study the contribution of adding depth and segmentation by comparing our full model with Geo-SC (Zhang et al. CVPR 2024), as well as by ablating the Geo-SC-specific losses from our model, and comparing it to a simple supervised approach:\n\n| PCK@0.1$_{bbox}$ | SPair-71k| SPair-U|\n|-|-:|-:|\n|DINOv2+SD (unsupervised, for reference)| 61.1 | 59.4|\n|DINOv2+SD| 76.5 | 60.0|\n|DINOv2+SD+Geo-SC (i.e., Geo-SC)| 85.6| 57.1|\n|DINOv2+SD+canonical prototype| 75.9 | 66.0 |\nDINOv2+SD+Geo-SC losses+canonical prototype (i.e., Ours)| 85.4| 66.1|\n\nThe results show a clear pattern, i.e., adding our canonical prototype loss results in a very small drop in performance on seen keypoints (i.e., SPair-71k), which we attribute to the models having less capacity to fully overfit the training keypoint supervision, but endows them with the ability to generalize much better to unseen points (i.e., SPair-U). Conversely, the Geo-SC losses allow models to perform really well on seen keypoints but its effect on generalization ranges from harmful to null (see “DINOv2+SD” vs. “DINOv2+SD+Geo-SC”). These results also demonstrate that our contributions do not require Geo-SC to work, as they also boost performance of the supervised baseline on unseen keypoints (see “DINOv2+SD” vs. “DINOv2+SD+canonical prototype”). We will add this ablation to the revised text. Thanks for the suggestion. \n\n\n**[AwCB-2] SPair-U limitations.**  \nWe acknowledge the potential limitation regarding dataset size on L300. The existing SPair-71k dataset contains a total of 88,328 test keypoint pairs and our new SPair-U dataset contains 19,990, which is still significant. SPair-U is an evaluation tool to expose the lack of generalization of sparse supervised correspondence models, which we believe it does effectively. Importantly, it is not for training models, as making these points available during training would prevent assessing generalization. A particular advantage of SPair-U is that it can be used to easily test any model that supports evaluation on SPair-71k, as it simply requires swapping the annotation files, and provides a clear visualization of the gap, e.g., “seen” versus “unseen” points performance.   \n\nWe fully agree that a dense labeling with 3D canonical representation would be valuable for the community in the long run. However, this is not a trivial task, and is in fact an actively studied research problem. Collecting “oracle” dense annotations is extremely labour intensive and limited to few real object categories as it typically relies on using 3D meshes (see related works [21,31])."}, {"review_id": "xDWERs3vzj", "reviewer": "Reviewer_fqdv", "summary": "The authors propose a new method for semantic correspondence estimation. They first analyse prior supervised methods and find that they act like keypoint detectors by overfitting on the few annotated keypoint classes and thus do not generalise well.\nIn their approach, they propose to use pre-trained monocular depth estimation and semantic segmentation models to estimate the 3D geometry of object instances.\nUsing these point maps, they propose to learn a canonical object space per-category. They do so by finding the optimal rigid alignment between the sparse 3D point clouds stemming from the annotated keypoints in all training images.\nFurther, they optimise a latent embedding for each of these canonicalised 3D keypoints. Using the feature similarity of other pixels to these points, they obtain their respective location in the canonicalised 3D space by linear interpolation.\nThe local neighbourhoods of given points in the so-found 3D canonicalised point maps are then compared with the posed point clouds retrieved through the frozen depth estimation model as an additional loss.\nThe paper further introduces new labels to the SPair-71k dataset that are used to evaluate the generalisation against prior supervised methods.", "questions": "Just to make sure that this is not the case, can you confirm that the training/evaluation is not done per-category but over all classes at the same time? So, while category-specific P and Z are learned, the feature refinement is shared over all classes.\n\nIt would be good to show the performance of the proposed objectives without the Geo-SC losses, as the proposed losses should be sufficient for learning semantic correspondence, as mentioned by the authors.\n\nIs the loss in Eq.6 computed for all points q that lie within the object mask? What is the neighbourhood size that is used to compute the loss?\n\nIs gradient stop applied for the computation of Eq.6 wrt. the locations of canonicalised keypoints P or are they not only optimised via Eq.3 and also indirectly through the minimisation of Eq.6 (given the computation in Eq.5)?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\nThe paper generally follows a convincing story, where the motivation of overfitted supervised semantic correspondence models is clearly motivated (Fig. 1).\n\nThe related work section is generally also well written, but missing a few important references (see weaknesses).\n\nThe authors propose a method that almost maintains the \"overfitted\" quality of supervised SC methods but enable improved correspondence for other points through their 3D-aware loss, while not requiring any additional supervision besides the annotated 2D keypoints and the class labels through the use of pre-trained 2D models.\n\nIntroducing a new test dataset for further evaluation on the SPair-71k dataset is helpful for the community.\n\n**Weaknesses:**\n\nThe related work section includes references to some unsupervised semantic correspondence methods.\nFirst, the statement of these methods being \"completely unsupervised\" (l.87) is possibly not true for all of the referenced papers as class labels are still required as weak supervision signal for category-specific learning.\nFurther, it would be advisable to include more works doing weakly supervised correspondence detection, such as (P)WarpC [1,2] or also include methods that perform 3D shape matching (which is essentially semantic matching) by using features extracted with foundational vision models [3,4,5].\n\nSeveral details on the method/implementation are missing or not stated clearly, see the questions below for details.\n\nThe method is based on the assumption that annotated keypoints are geometrically salient (as all points must lie within the convex hull of the annotated keypoints), i.e., close to farthest points on the objects.\nWhile this is often true, e.g. in SPair, assuming this might reduce generality to other datasets.\n\nThe authors state that their local neighbourhood rigidity loss (Eq. 6) is only computed for local neighbourhoods because of the difference between posed and canonical object shapes (l.180).\nHowever, the alignment of posed annotated keypoints with the canonical coordinates in P (Eq.3) is based on exactly this assumption, where rigid transformations (+ isotropic scaling) are computed as transformations between posed and canonical objects.\nIt is unclear why this rigidity assumption should be valid for the alignment of sparse keypoints but not for dense points on the surface.\n\nThe paper is lacking important ablation experiments. The authors argue that their introduced \"objectives suffice to learn a SC model\" but only show the results of the method in combination with the objectives from Geo-SC.\nWhile leaving out single loss terms of the proposed new losses might not be possible (l. 24, suppl.), the results of this experiment should have been shown.\n\n---\n\n[1] Truong, Prune, et al. \"Warp consistency for unsupervised learning of dense correspondences.\" Proceedings of the IEEE/CVF international conference on computer vision. 2021.\n\n[2] Truong, Prune, et al. \"Probabilistic warp consistency for weakly-supervised semantic correspondences.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[3] Dutt, Niladri Shekhar, Sanjeev Muralikrishnan, and Niloy J. Mitra. \"Diffusion 3d features (diff3f): Decorating untextured shapes with distilled semantic features.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[4] Wimmer, Thomas, Peter Wonka, and Maks Ovsjanikov. \"Back to 3d: Few-shot 3d keypoint detection with back-projected 2d features.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[5] Uzolas, Lukas, Elmar Eisemann, and Petr Kellnhofer. \"Surface-Aware Distilled 3D Semantic Features.\" arXiv preprint arXiv:2503.18254 (2025).", "was_revised": true, "final_justification": "After reading the rebuttal by the authors, which addressed my questions and concerns, I remain positive about the impact and contributions of the paper. While the global rigidity assumption is fairly strong for inherently non-rigid objects (e.g. cats), it seems to be inevitable to make some sort of assumption about the global distribution of keypoints. The paper demonstrates that this assumption helps closing the generalization gap of supervised semantic correspondence models and proposes an additional set of keypoint annotations to measure this generalization on the SPair dataset, which, when made public, will help advance the field further and make other methods more robust as well.", "rebuttal": "We thank **fqdv** for their constructive review and for highlighting our clear motivation, coherent experimental narrative, and the value of SPair-U for testing generalisation. As an initial remark, we would like to emphasize that our paper goes beyond simply proposing a new semantic correspondence approach and new benchmark dataset. Our goal is to expose a fundamental flaw in the way supervised semantic correspondence pipelines are trained and evaluated, i.e. **issues in the general task design** are the reason behind the generalization gap observed in models. We believe this to be our work’s core contribution, of particular significance to the community as it carries strong implications towards future model and benchmark design. In particular, GeoSC - the top performing model on SPair-71k at the time we wrote our paper - trains a supervised feature predictor on top of frozen DINOv2 and SD features, yet when evaluated on SPair-U it performs *worse* than simply using the DINOv2 and SD features (see Tables 1 and 2). This means that **none** of what it has learned transfers to new semantics, effectively failing to solve anything beyond sparse keypoint detection. While the raised concerns about model limitations are valid, we are concerned that this contribution may have been overlooked and would like to highlight it.  \n\n**[fqdv-1] Additional weakly supervised approaches.**  \nWe will update the related works section accordingly and clarify that [1,50] do not make any assumption about the category, while [2,38,39] typically train and evaluate their models on single category datasets.  Warp-based matching is indeed an important precursor of geometry-aware methods, however, it is largely limited to instances in the same pose. While our model design is performing some form of 3D shape matching, we work only with images and do not require meshes, relying instead on pretrained segmentation and depth to obtain partial point clouds, which is a weaker supervision signal.\n\n**[fqdv-2] Assume that annotated keypoints are geometrically salient.**  \nWhile we agree it is a limitation (as discussed in section 6), to the best of our knowledge, it is unlikely to be a strong issue as semantics are by definition human-defined, and we have a bias towards geometrically salient points. In particular, it is unlikely that human annotators will not label object extremities, providing a good approximation of the convex hull.\n \n**[fqdv-3] Rigidity assumption.**    \nWe agree that making a global rigidity assumption in Eq.3 is a very coarse approximation, and can appear to contradict our goal of learning a correspondence model for deformable objects. However, this **global** assumption is only made in Eq.3, whose sole purpose is to optimize the set of 3D keypoints $\\mathcal{P}$ into a coarse spatial organisation of 3D keypoints, e.g., making sure that the left hand keypoint generally sits opposite of the right one, in order to provide a reasonable structure for our canonical prototypes. Importantly, $\\mathcal{P}$ is required to compute our geometric losses, *but is not used to predict correspondence*.  \n\nThe component of our model that predicts correspondence is the feature predictor $\\phi$, which is not directly involved in Eq.3. It is trained through Eq.5 and 6 in which only a **local** rigidity assumption is made. Though they happen simultaneously, optimizing $\\mathcal{P}$ and $\\phi$ can effectively be considered separate processes as $\\mathcal{P}$ is considered fixed in Eq.5 and 6.\n\nVisualizations of the learned $\\mathcal{P}$ (the large points in Fig 5) illustrate that the assumption of global rigidity *for $\\mathcal{P}$ only* allows our model to recover a reasonable 3D structure, even if in practice the estimated optimal rigid alignment $\\hat{\\mathbf{M}}^{(n)}$  is far from perfect and still incurs relatively large errors due to object deformations.\n\n\n**[fqdv-4] Ablation without GeoSC.**  \nThank you for pointing it out, it is indeed an important ablation that we overlooked. We trained a model purely from DINOv2 and SD features, disabling all components introduced in GeoSC. It obtained a PCK@0.1 of 75.9 on SPair-71k, 0.6 points lower than the simple supervised baseline trained only on these features. On SPair-U, it reaches a PCK of 66.0, only 0.1 points below what our complete model achieves. This demonstrates surprisingly well that what GeoSC learns and what our geometric losses learn are in fact almost orthogonal, and strengthens our observation that GeoSC fully overfits the training semantic, as removing it from our model does not harm the performance on SPair-U.\n\n**[fqdv-5] A single model for all categories.**   \nYes, as noted at  L267 we train our model on all categories jointly. Each category has its own $\\mathcal{P}$ and $\\mathcal{Z}$, while $\\phi$ is the same for all, allowing generalization to new categories as illustrated in Table 3. We will update the revised text to make this clearer. \n\n**[fqdv- 6] Additional implementation details.**   \nAfter performing backprojection, we first subsample 1024 points from the resulting point cloud using farthest point sampling in order to have a similar number of points per object instance. Then, for each of these points, we compute its 64 nearest neighbors to obtain $\\mathbf{C}_q$ . We selected a neighborhood size of 64 as it makes up 1/16 of the point cloud size, striking a good balance between being representative of the geometry (too few points carry little information about the object structure) while allowing deformations (too many and the rigidity assumption becomes global). We perform an ablation study of this in our response [Avym-3].\n\nYes, $\\mathcal{P}$ is only optimized using Eq. 3. They are considered fixed in Eq. 5 and 6 and no gradient is propagated to them at this stage. Apologies for not including these details, we will clarify them in the final manuscript."}, {"review_id": "go82twEZdX", "reviewer": "Reviewer_Avym", "summary": "Based on the observation that current supervised methods for category-specific 2D semantic correspondence fail to generalize to unseen keypoints not annotated during training, the authors proposed a new approach using geometric constraints at training time through a per-category, learned continuous canonical manifold.", "questions": "1. In the beginning of section 3.1, the authors introduced a notation S, which is never referred to in the following text. Also, from the whole section 3, it’s unclear whether the method wants to optimize dense feature descriptors for dense 3d points on the continuous canonical manifold or just optimizing feature descriptors for the |K| number of 3d keypoints.\n2. In the equation (3) in section 3.2, it’s unclear how to do the optimization, since optimizing M requires p, and optimizing p requires M. I guess the optimization is conducted alternatively, but it’s unclear in the text.\n3. For the equation (3) in section 3.2, the authors want to optimize a category-specific sparse 3d keypoint set P for all input. But objects within input images will have different articulation and arbitrary deformations, which will lead to \\bar{k} got by equation (2) from different images cannot be described accurately by only rigid transformation (with scaling) of just one 3d keypoint set P. If the authors’ idea is to only approximately learn a roughly correct rigid transformation (with scaling) M and a roughly correct 3d keypoint set P from equation (3), then ablation studies on different selection of 2d keypoints annotations need to be done to validate the method.\n4. In section 3.3, equation (6) defined an optimization based on local small neighbors of canonical space C and posed space B, but the details of how to select the neighbor size k and the ablation study of how the size k will influence the results (especially for different categories, since objects within difference categories will deform in difference sense) are not included in the paper.", "limitations": "1. The writing quality of the paper needs to be improved, since there are some technical details are unclear as mentioned in the questions section. \n2. The proposed method relies on the off-the-shelf segmentation and depth estimation methods and thus the performance heavily dependent on the quality as well as the consistency of the segmentation and depth estimation results. It is expected to evaluate how robust of the proposed method for highly articulated object such as human, animal, where the segmentation and depth estimation is typically less accurate due to the large deformation as well as occlusion.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "1. The authors proposed a method that leveraged pre-trained depth predictor to give depth estimation for sparse annotated 2d keypoints in image to lift them to 3d, and optimize a canonical 3d keypoints set as correspondence reference set. And the extra introduced 3d information will make 2d correspondence be more accurate.\n2. The authors also proposed a method that generate dense 3d keypoints from the optimized sparse 3d keypoints set, will make the optimized semantic correspondence prediction model could generalize to new 2d labelled keypoints unseen during training.\n3. The authors proposed a new dataset SPair-U, which will benefit future researchers in 2d semantic correspondence area.", "was_revised": true, "final_justification": "After reading the rebuttal, my most concerns are well addressed. Therefore, I would like to raise my final score.", "rebuttal": "We thank **Avym** for their detailed comments, and endorsing the use of monocular depth models to learn a dense generalization of sparse keypoint, and the benefits of introducing SPair-U. First and foremost, we would like to emphasize that our paper goes beyond simply proposing a new semantic correspondence approach and new benchmark dataset. Our goal is to expose a fundamental flaw in the way supervised semantic correspondence pipelines are trained and evaluated, i.e. **issues in the general task design** are the reason behind the generalization gap observed in models. We believe this to be our work’s core contribution, of particular significance to the community as it carries strong implications towards future model and benchmark design. In particular, GeoSC - the top performing model on SPair-71k at the time we wrote our paper - trains a supervised feature predictor on top of frozen DINOv2 and SD features, yet when evaluated on SPair-U it performs *worse* than simply using the DINOv2 and SD features (see Tables 1 and 2). This means that **none** of what it has learned transfers to new semantics, effectively failing to solve anything beyond sparse keypoint detection. While the concerns about unclear technical details and missing ablations are valid, we are concerned that this contribution may have been overlooked and would like to highlight it.  \n\n\n**[Avym-1] Unclear technical details.**  \nThank you for pointing these out, we will update our manuscript to clarify these points\n* We will remove the explicit definition of $S$.  \n* Our goal is to learn a dense feature predictor $\\phi$ that effectively predicts semantic correspondence. We do so by leveraging the sparse $|K|$ annotated keypoints to coarsely align all instances of an object category, then rely on fine-grained geometric constraints to densely supervise $\\phi$.  \n* In Eq. 3, only $\\mathcal{P}$ is optimized. $\\hat{\\mathbf{M}}^{(n)}$ is not optimized, it is obtained analytically by running an orthogonal procrustes solver relying on the Kabsch algorithm. The mention of a “nested optimization” on L159 was a misnomer on our part.  \n\n**[Avym-2] Learning the roughly correct rigid transformation.**  \nThe point of Eq. 3 is to guide $\\mathcal{P}$ towards a coarse category shape to serve as a reasonable basis for our canonical dense mapping. Because $\\hat{\\mathbf{M}}^{(n)}$ is rigid and objects can deform, it will indeed only be “roughly correct”. A detailed explanation of this point can be found in response [fqdv-3].  For each training image, *we use all visible keypoints*, and only these points, i.e., the summations in Eq. 3 are in fact not computed over all points in $\\mathcal{P}$, only those that correspond to the annotated points of $I$. As requested, we perform ablations by randomly selecting a subset of points when computing Eq. 3. Selecting fewer than 3 would lead to a degenerate alignment problem, and there are few image pairs in which more than 8 are visible at the same time.  \n\nHere are the different PCK@0.1$_{bbox}$ on the SPair-71k *validation* set. \n\n| #selected points|     3|      4|      5|     6|    7|      8|*all*|\n|:----------------------|-----:|------:|-----:|-----:|-----:|-----:|-----:|\n| PCK                  | 86.6| 86.4|86.1|86.4|86.5|86.0|86.5|\n\nSelecting a subset of points instead of using all of them only appears to cause small fluctuations in validation performance.  \n\n\n**[Avym-3] Ablation of neighborhood size.**  \nAfter performing backprojection, we first subsample 1024 points from the resulting point cloud using farthest point sampling in order to have a similar number of points per object instance. Then, for each of these points, we compute its 64 nearest neighbors to obtain $C_q$ . We selected a neighborhood size of 64 as it makes up 1/16 of the point cloud size, striking a good balance between being representative of the geometry (too few points carry little information about the object structure) while allowing deformations (too many and the rigidity assumption becomes global). As requested, we evaluated the impact of different neighborhood sizes, where we report PCK@0.1$_{bbox}$ on the SPair-71k *validation* set.\n\n| #NN|     4|      8|   16|   32|*64*|128| 256|\n|:------|-----:|------:|-----:|-----:|-----:|-----:|-----:|\n|PCK | 85.5| 86.1|85.9|86.5|86.5|86.6|86.1|\n\nExcept for extreme values, the neighborhood size appears to have little impact on the validation performance on the seen keypoints in SPair-71k. We will add these additional results to the revised paper.  \n\n\n\n\n\n**[Avym-4] Requires segmentation and depth which may be inaccurate.**      \nUsing the depth or segmentation predictions from pre-trained models to build 3D-aware representations has been adopted before in related works [3, 27, 42, 46]. In particular, concurrent work [46] uses both, and unlike us it even requires segmentation masks at *test-time*. In contrast, our approach only uses them during training. At inference time, correspondences are estimated solely using image features predicted by $\\phi$. It would be possible to filter out training instances where segmentation or depth is poorly estimated before training to remove any harmful impact of them. In practice however, we did not observe any catastrophic failures of these predictions. \n\n\n**[Avym-5] Results on highly articulated objects such as human and animal.**   \nOur model is evaluated on many deformable categories like people and animals from the SPair and AP-10K datasets, and performs well on them (Table 3, 4, and 5). Specifically on SPair-U, our model shows significantly improved generalization performance compared with its backbone Geo-SC on birds (+16.5 pts), cats (+11.1 pts) and humans (+7.0 pts)."}, {"review_id": "7IAwcnygdH", "reviewer": "Reviewer_UJnP", "summary": "This paper proposes a method for constructing a canonical 3D representation for object categories and using it to establish semantic correspondences.", "questions": "Please address the concerns expressed in the weaknesses part of the review above.\n\nAdditional questions:\n\nAs far as I know, Kabsch's algorithm (or its alternatives using quaternions) find a rigid transformation, not a similarity, please clarify.\n\nIt appears to be the case that the number of key points is fixed over all category instances. For a true 3D object see from different viewpoints, half of the object (its back) will always be invisible. How do you expect to handle this problem?\n\nI didn't understand your geodesic sampling strategy : what is the geodesic distance you use since you don't know the surface of the object or the true connectivity of the points? Please clarify.", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\nThe paper is well written, and the proposed method gives good results, in particular on data for which some keypoints are available for testing, but were not annotated during training.\n\nWeaknesses: \nThe presentation appears to implicitly assume that all instances of the same object category have the same 3D shape since Eq. (3) expresses the fact that their keypoints are related by similarity transforms. This is a very strong assumption that will not hold for many object categories (think ot cats or even cars for example), and a major limitation of the proposed approach. The necessity to relax global rigidity in  geometric representations of object categories was already noted in old papers such as Lazebnik et al., \"Semi-local affine parts for object recognition\", BMVC'04 for example.\n\nIn Eq. (3) there does not appear to be any constraint on the points p_i and matrices M. But  in this case what prevents the optimization from reaching a global optimum of 0 with p_i=0 and M=0?\n\nThe proposed method learns two descriptors z and Phi, one for the canonical keypoints, and one of image keypoints. But there does not appear to be any constraint on these either, except they should be similar. What makes these descriptors informative, and what prevents them from collapsing into constant values for example?", "was_revised": true, "final_justification": "I appreciate the efforts the authors have put in the rebuttal, but in the end my main concern, the implicit assumption that objects within a category have the same shape, stands, and I thus also stand with my rating.", "rebuttal": "We thank **UJnP** for their insightful comments, and recognizing the ability of our model to generalize to points that weren’t seen during training. As an initial remark, we would like to emphasize that our paper goes beyond simply proposing a new semantic correspondence approach and new benchmark dataset. Our goal is to expose a fundamental flaw in the way supervised semantic correspondence pipelines are trained and evaluated, i.e. **issues in the general task design** are the reason behind the generalization gap observed in models. We believe this to be our work’s core contribution, of particular significance to the community as it carries strong implications towards future model and benchmark design. In particular, GeoSC - the top performing model on SPair-71k at the time we wrote our paper - trains a supervised feature predictor on top of frozen DINOv2 and SD features, yet when evaluated on SPair-U it performs *worse* than simply using the DINOv2 and SD features (see Tables 1 and 2). This means that **none** of what it has learned transfers to new semantics, effectively failing to solve anything beyond sparse keypoint detection.  While the raised concerns about restricted model design and missing technical details are valid, we are concerned that this contribution may have been overlooked and would like to highlight it.  \n\n**[UJnP-1] Assuming all instances of a category have the same shape?**  \nWe agree that making a global rigidity assumption in Eq. 3 is a very coarse approximation, and can appear to contradict our goal of learning a correspondence model for deformable objects. However, this **global** assumption is only made in Eq. 3, whose sole purpose is to optimize the set of 3D keypoints $\\mathcal{P}$ into a coarse spatial organisation of 3D keypoints, e.g., making sure that the left hand keypoint generally sits opposite of the right one, in order to provide a reasonable structure for our canonical prototypes. Importantly, $\\mathcal{P}$ is required to compute our geometric losses, *but is not used to predict correspondence*.  \n\nThe component of our model that predicts correspondence is the feature predictor $\\phi$, which is not directly involved in Eq. 3. It is trained through Eq. 5 and 6 in which only a **local** rigidity assumption is made. Though they happen simultaneously, optimizing $\\mathcal{P}$ and $\\phi$ can effectively be considered separate processes as $\\mathcal{P}$ is considered fixed in Eq. 5 and 6.\n\nVisualizations of the learned $\\mathcal{P}$ (the large points in Fig 5) illustrate that the assumption of global rigidity *for $\\mathcal{P}$ only* allows our model to recover a plausible 3D structure, even if in practice the estimated optimal rigid alignment $\\hat{\\mathbf{M}}^{(n)}$  is far from perfect and still incurs relatively large errors due to object deformations.\n\n**[UJnP-2] Missing constraint in Eq. 3?**  \nApologies for this missing detail. In practice the estimated scale $s$ is constrained to be larger than 1, i.e., instances can only be scaled up, not down. This has the effect of rescaling all instances to the size of the largest one in the training set, and prevents collapse. We will update the text to make this clear. \n\n**[UJnP-3] Missing constraint between $z$ and $\\phi$?**   \nAs mentioned in Sec 5.1, we build our model on top of Geo-SC [51]. The main training objective of Geo-SC is a CLIP-style contrastive loss between source and target keypoints. This prevents collapse of the descriptors predicted by $\\phi$, which in turn prevents collapse of $\\mathcal{Z}$ through Eq.4. We will also update the text to make this clear. \n\n**[UJnP-4] Kabsch algorithm.**  \nYou are correct, this is a misuse of the term on our part. To be precise, estimating $M$ requires solving a *constrained orthogonal Procrustes problem*. Only $R$, the rotational part of $M$, is obtained through the Kabsch algorithm.\n\n**[UJnP-5] Occluded points.**   \nFor each training image, we use all visible keypoints, and only these points, i.e., the summations in Eq.3 are in fact not computed over all points in $\\mathcal{P}$, only those that correspond to the annotated points of $I$. At inference time, correspondences are computed using $\\phi$ only, i.e., $\\mathcal{P}$ is not involved.\n\n**[UJnP-6] Geodesic sampling.**   \nWe named this procedure “geodesic sampling\" as its goal is to aggregate neighbors along the object surface. However, it does not explicitly use the geodesic distance as we do not have access to the connectivity of the points. Below is a pseudocode for it, which we will add to the implementation details:\n\nInput: Point cloud $\\mathcal{X}$, seed point $p$, number of neighbors $k$;  \nOutput: neighbor set $B$;  \n$B \\leftarrow$ {$p$}  \nwhile $|B| < k$:  \n|&nbsp;$\\mathcal{X}' \\leftarrow \\mathcal{X}\\setminus B$   # We do not consider already selected points  \n|&nbsp;for $x$ in $\\mathcal{X}$':  \n|&nbsp;|&nbsp;$D_x \\leftarrow \\min_{y\\in B} \\left \\lVert x - y \\right \\rVert_{2}$ # Compute the distance to the closest point in B  \n|&nbsp;$B \\leftarrow B \\cup$ {$\\arg\\min_{x\\in \\mathcal{X}'} D_x$} # Add the point with minimal distance to B  \nreturn $B$  \n\nBy iteratively adding points that minimize the distance to the set of already selected neighbors, we minimize the risk of jumping over surface discontinuities.\n\n**[UJnP-7] Additional related work.**   \nThanks for pointing out Lazebnik et al. BMVC 2004. Their work is concerned with finding geometrically invariant parts for modeling 3D objects in the context of object recognition, and not our explored task of semantic keypoint correspondence. While their goal of matching image regions using local affine transform bears some high-level similarities with our local geometric objective defined by Eq. 5 and 6, ours is performed in 3D, endowing our model with the ability to deal with large viewpoint changes.  We will add a discussion to our related work."}, {"review_id": "nap6QPbmd5", "reviewer": "Reviewer_tgz1", "summary": "The paper proposes a semantic matching method, which establishes semantic correspondences between images by lifting the  2D keypoints into a canonical 3D space using monocular depth estimation.\nThe main motivation of the paper lies in the limited generalizability of existing semantic correspondence methods beyond the sparsely annotated training keypoints seen during training.\nTo better evaluate the generalizability of semantic matching methods on unseen keypoint categories, the paper further introduces introduce SPair-U, an extension of the existing SPair-71k benchmark with additional keypoint annotations.\nThe proposed method shows strong semantic matching performances across SPair-71k and SPair-U.", "questions": "Please refer to the weaknesses section. Specific questions below:\n\n1. How does the proposed method differ from SemAlign3D, and what additional technical / conceptual novelties does this paper hold?\n\n2. How would the proposed method perform without additional supervision (i.e., semantic meaning per keypoint), just using the keypoint positions as supervision?\n\n3. How do other existing methods (especially semantic matching methods which focus on correlation map refinement) perform on SPair-71k and SPair-U?", "limitations": "Yes.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 1, "strengths_and_weaknesses": "### Strengths:\n\n1. The paper addresses an important problem in semantic matching i.e., limited ability of existing methods in establishing dense semantic correspondences across viewpoints across the entire foreground of the objects.\n\n2. The paper is well written and easy to follow.\n\n\n### Weaknesses:\n\n1. The paper shares strikingly similar idea with SemAlign3D, a paper cited by the authors in L111 (Related Work: Geometry-Aware Methods). The core idea of building per-class 3D canonical representations based on monocular depth maps and segmentation masks overlap significantly. The authors mention that SemAlign3D \"requires segmentation masks and lacks the ability to generalize to new categories, as no prototype for them would exist for them (the new categories) in absence of training samples\", the same limitations exist for the proposed method. Moreover, the reported performance on SPair-71k in this paper falls behind SemAlign3D. At the time of submission of this paper (mid May), SemAlign3D had been accepted to CVPR 2025 and was already uploaded on Arxiv (late March). The authors were aware of SemAlign3D -- with that in mind, it is challenging to notice additional technical contribution beyond SemAlign3D in this paper.\n\n2. Disregarding the existence of SemAlign3D, the proposed pipeline is hard to agree with. Unlike existing semantic matching methods, the proposed method assumes that the object-wise keypoints are always on **semantically pre-defined positions**, and narrows the problem of semantic matching down to matching pre-defined semantic keypoints. The proposed method therefore aims to \"produce distinctive and semantically consistent features across object instances for each keypoint k (L 168)\", which, I believe, contradicts with the motivation of the paper which aims to address the \"limited generalizability of existing methods\". In clearer terms, I believe that assigning semantic meanings to each keypoint per object can be regarded as 'additional supervision' beyond simple keypoint positions, and this poses unfair comparisons with existing work. \n\n3. The proposed method lacks comparison with other existing work, which may show strong performances on SPair-71k and SPair-U. For example, semantic matching methods which refines correlation maps e.g., [1][2], can be expected to perform better in dense semantic matching scenarios.\n\n[1] Min et al., Convolutional Hough Matching Networks, 2021\n[2] Kim et al., TransforMatcher: Match-to-Match Attention for Semantic Correspondence, 2022", "was_revised": true, "final_justification": "The authors have cleared my major concerns during the rebuttal phase. While some questions remain regarding whether using semantically pre-defined keypoints can really handle the generalization gap, I hope the authors can handle this in the revision with additional analysis and visualizations.", "rebuttal": "We thank **tgz1** for their valuable feedback and for acknowledging our work addresses an important limitation in existing methods. To summarize, the primary concern noted by **tgz1** relates to the perceived similarity between our approach and the concurrent work SemAlign3D (Wandel et al., CVPR 2025). In our response below we go into detail describing why (1) according to the official NeurIPS guidelines it should be considered concurrent work and thus does not need to be compared against and (2) both works, while sharing similar high-level motivation, are actually significantly different in terms of their approaches (e.g., SemAlign3D is a test-time optimization method with no learnable components). As a reminder, in addition to proposing a new approach for semantic correspondence estimation, our core contribution relates to identifying a deeply rooted issue in the way supervised semantic correspondence pipelines are trained and evaluated.\n\n**[tgz1-1] Comparison to SemAlign3D.**   \nThe official NeurIPS 2025 call for papers states that *“papers that appeared online after March 1st, 2025 will generally be considered ‘contemporaneous’ in the sense that the submission will not be rejected on the basis of the comparison to contemporaneous work.”*\n\nSemAlign3D was published at CVPR 2025 and first made public on arXiv on 28th of March 2025, which clearly identifies it as contemporaneous. We would like to clarify that we developed our work independently from SemAlign3D. We only became aware of it shortly before the NeurIPS deadline, we did our best to acknowledge it by citing it as concurrent work (L111) and submitted our work in good faith. Note, the code for SemAlign3D was only released on July 10th, but this is only a subset of the code needed to recreate the results in their paper. Below is a breakdown the main conceptual and technical differences between our works:\n\n*[tgz1-1.1] Paper contributions.*   \nSemAlign3D proposes a new supervised semantic correspondence system and introduces a new approach based on matching objects through 3D category prototypes built by lifting a segmentation and monocular depth model. In contrast, our primary goal is to expose a critical flaw in recent semantic correspondence benchmarks and model designs. We do so by showing that recent supervised models are overfitted to the training set and merely act as keypoint detectors (see Table 1 vs 2). In order to demonstrate this, we introduce new evaluation points that do not appear in the SPair-71k training set. Finally, we hypothesize that incorporating 3D awareness is an efficient solution to this generalization gap, and introduce a model following this principle that shows better generalization on SPair-U to validate it. The only point of confluence is that both our model and SemAlign3D leverage 3D category prototypes. However, other prior works have also previously explored the use of 3D prototypes for correspondence learning, e.g., [27,39]. Please refer to L97-115 in our paper for a discussion of geometry aware methods. \n\n*[tgz1-1.2] Conceptual model differences and associated limitations.*  \nThe way SemAlign3D and our approach use the 3D prototypes is also conceptually different, incurring different limitations. SemAlign3D does not train any neural network. It builds the 3D prototypes using the training images, and aligns them at inference time as a test-time optimization procedure. As a result, their approach has several limitations:\n1. To align the correct category prototype on the test image, it needs to have seen the exact object category during training, e.g. build a chair prototype to be able to evaluate correspondence on test chairs, preventing generalization to new categories.  \n2. It needs to know the object category at test time, to select the corresponding prototype.  \n3. The test-time optimization process relies on a **test** segmentation mask (Eq. 11 in SemAlign3D).   \n4. Finally, test-time optimization is costly, requiring 10-30 seconds to align the prototype on a single image, so up to a full minute for a pair (Sec. 4.3, paragraph 1 in SemAlign3D). It is also prone to failure and requires category-specific hyperparameters (Optimization paragraph in Sec. 4.1 and Supplementary Tab. 5 in SemAlign3D) \n\nIn comparison, our model aims to learn a function $\\phi$, parametrized by a neural network, that enables general 2D correspondence prediction. The 3D prototypes we create are only used during training to allow us to compute our geometric loss (Eq. 6). At inference time, features predicted by $\\phi$ are directly used to compute correspondence, which means:\n1. We can apply $\\phi$ to images of any category and estimate correspondence on them. We do so in our cross-benchmark evaluation (L264) where we evaluate models trained on Spair-71k on other datasets, including AP-10K which contains animal categories very different from those seen during training, e.g., monkeys, bears and squirrels. Results in Table 3 show that our model generalizes better than its competitors in this challenge regime.\n2. We train a single $\\phi$ shared across all categories and thus do not require knowing the test category beforehand. \n3. We compute correspondence without requiring the aid of a test-time segmentation mask. \n4. Estimating correspondence between two images only requires computing two forward passes and querying nearest neighbors. On our hardware which is roughly similar to that used in SemAlign3D, it takes only 50 ms per pair, meaning our model is roughly **three orders of magnitude faster** than theirs (x400 up to x1200).\n\n*[tgz1-1.3] Technical model differences.*   \nWhile both approaches rely on segmenting the object and using a monocular depth model to obtain a point cloud, the way we build our 3D prototypes is also different. Semalign3D relies on estimating the distribution of relative distance and angles of keypoint quadruplets, then explicitly learns a dense prototype by clustering points coming from different object point clouds. In contrast, we only learn a coarse category prototype using Kabsch’s algorithm. Our dense mapping is implicitly learned through geometrically aligning the predictions of  $\\phi$ to with the backprojected point clouds. Thus, both 3D prototypes are learned in a completely different way, and only share some high-level conceptual similarity.\n\nWe hope this clearly shows that our paper shares a similar motivation in its model design with SemAlign3D simply because the idea of developing a 3D category prototype is an intuitive solution to solving correspondence. We are happy to discuss these points further during the discussion period should any clarification be needed. We will revise our discussion of SemAlign3D, which is indeed very interesting work, in the updated paper to summarize these key differences.\n\n**[tgz1-2] Keypoint semantics as extra supervision?**     \nIrrespective of how models were trained, we *precisely* show that previous models overfit to semantically predefined positions, as they are unable to generalize to new semantic points that do not appear in the training annotations (see Fig. 1). Even if such an assumption is not explicitly made in their design, they still end up only being able to match predefined keypoints due to the limitation of the task design, i.e., sparse supervision and semantic overlap between the training and evaluation points, **which is a central point of our paper**.\n\nMoreover, gathering all points that belong to the same semantic part, even if the keypoint semantics are not provided, can easily be done by iterating through the training images and grouping points by equivalence class, e.g., point A in image 1 corresponds to point B in image 2 and point C in image 3, etc…, hence all these points share the same semantics. Therefore, we respectfully disagree that this constitutes ‘additional supervision’ as we use exactly the same labels as other models, albeit under slightly different assumptions.\n\nIn practice, all semantic correspondence datasets are made up of instances labeled with a predefined keypoint set. That is because semantics are by definition human-defined, and we have a bias towards salient points common to all instances, therefore expecting anything else is unrealistic and would constitute a particularly contrived setting.\n\nFinally, similar or even more constraining assumptions are made in recent related works. As previously discussed, SemAlign3D makes the same assumptions as us, while Geo-SC explicitly requires additional semantic knowledge to perform its flipping based data augmentation (when flipping the image of a person, a left hand becomes a right hand).\n\n**[tgz1-3] Comparison to CHMN and TransfoMatcher.**   \nThanks for pointing out these papers, we will add them to our related works section. While their dense objectives could indeed make them less prone to overfitting, their designs lack 3D awareness, which we posit is an important component for generalization, especially under the large viewpoint changes that SPair exhibits.  \nWe evaluated both approaches on SPair-71k and SPair-U. For CHMN, we used the code and checkpoint provided by the official github repo. For TransforMatcher, we used the code provided by the official github repo. Since no checkpoint was released, we retrained the model ourselves.\n\n| PCK@0.1$_{bbox}$ | SPair-71k| SPair-U|\n|-|-:|-:|\n|CHMN| 46.4 | 28.0|\n|TransfoMatcher| 50.7 | 29.7|\n|ScorrSan|55.3|32.7|\n|Ours| 85.4| 66.1|\n\nBoth models exhibit a significant generalization gap, very consistent with that of the other related works. The closest performing model we evaluated in the paper, SCorrSan, reaches similar numbers. Interestingly, SCorrSan also aims to learn dense descriptors, though it uses pseudo labels to achieve it. Generally speaking, this confirms our observation that many current methods overfit to the seen keypoints in SPair-71k and fail to generalize well to unseen ones. We will add these news results to the revised paper."}]

Author rebuttal:

Meta-review: This paper investigates the generalization ability of semantic correspondence methods beyond sparsely annotated training keypoints and proposes a novel approach for learning dense correspondences by lifting 2D keypoints into a canonical 3D space using monocular depth estimation. Additionally, the authors introduce a new benchmark, SPair-U, with novel keypoint annotations to better assess the issue of generalization. There were several concerns regarding motivation, assumptions, analyses, and comparisons, and there were a rebuttal and follow-up discussion. After post-rebuttal discussions, the paper received mixed reviews, one accept, two borderline accepts, and two borderline rejects. The main remaining concerns were the lack of in-depth analyses/comparison (reviewer AwCB) and the implicit assumption of the same shape for the same category (reviewer UJnP). After carefully reading the review, the rebuttal, the discussions, and the paper, AC finds that while the reviewers’ points are valid and reasonable, the authors’ rebuttal addresses them relatively well, and the remaining issues are not critical enough to warrant rejection. For example, the global rigidity assumption is fairly strong for non-rigid objects indeed, but not a completely unreasonable assumption, and the authors demonstrated well that it helps close the generalization gap of existing supervised semantic correspondence methods; the effect of the implicit assumption may motivate further development into a more relaxed assumption of shapes in future work. AC also feels positive about the impact and contributions of the work, which reveal and investigate the interesting issue of existing work on the important vision task. Therefore, AC recommends accepting this work. AC strongly encourages the authors to incorporate the reviewers’ constructive comments and discussions into their final manuscript to clarify their contributions and include missing related work, relevant analyses, and discussions.
Decision: Accept (poster)