Title: Jamais Vu: Exposing the Generalization Gap in Supervised Semantic Correspondence
Venue: NeurIPS 2025
Accepted: yes
Keywords: Semantic Correspondence

Evaluation

Generalization

Representation Learning
Abstract: Semantic correspondence (SC) aims to establish semantically meaningful matches across different instances of an object category. We illustrate how recent supervised SC methods remain limited in their ability to generalize beyond sparsely annotated training keypoints, effectively acting as keypoint detectors. To address this, we propose a novel approach for learning dense correspondences by lifting 2D keypoints into a canonical 3D space using monocular depth estimation. Our method constructs a continuous canonical manifold that captures object geometry without requiring explicit 3D supervision or camera annotations. Additionally, we introduce SPair-U, an extension of SPair-71k with novel keypoint annotations, to better assess generalization. Experiments not only demonstrate that our model significantly outperforms supervised baselines on unseen keypoints, highlighting its effectiveness in learning robust correspondences, but  that unsupervised baselines outperform supervised counterparts when generalized across different datasets.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a semantic matching method, which establishes semantic correspondences between images by lifting the  2D keypoints into a canonical 3D space using monocular depth estimation.\nThe main motivation of the paper lies in the limited generalizability of existing semantic correspondence methods beyond the sparsely annotated training keypoints seen during training.\nTo better evaluate the generalizability of semantic matching methods on unseen keypoint categories, the paper further introduces introduce SPair-U, an extension of the existing SPair-71k benchmark with additional keypoint annotations.\nThe proposed method shows strong semantic matching performances across SPair-71k and SPair-U.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths:\n\n1. The paper addresses an important problem in semantic matching i.e., limited ability of existing methods in establishing dense semantic correspondences across viewpoints across the entire foreground of the objects.\n\n2. The paper is well written and easy to follow.\n\n\n### Weaknesses:\n\n1. The paper shares strikingly similar idea with SemAlign3D, a paper cited by the authors in L111 (Related Work: Geometry-Aware Methods). The core idea of building per-class 3D canonical representations based on monocular depth maps and segmentation masks overlap significantly. The authors mention that SemAlign3D \"requires segmentation masks and lacks the ability to generalize to new categories, as no prototype for them would exist for them (the new categories) in absence of training samples\", the same limitations exist for the proposed method. Moreover, the reported performance on SPair-71k in this paper falls behind SemAlign3D. At the time of submission of this paper (mid May), SemAlign3D had been accepted to CVPR 2025 and was already uploaded on Arxiv (late March). The authors were aware of SemAlign3D -- with that in mind, it is challenging to notice additional technical contribution beyond SemAlign3D in this paper.\n\n2. Disregarding the existence of SemAlign3D, the proposed pipeline is hard to agree with. Unlike existing semantic matching methods, the proposed method assumes that the object-wise keypoints are always on **semantically pre-defined positions**, and narrows the problem of semantic matching down to matching pre-defined semantic keypoints. The proposed method therefore aims to \"produce distinctive and semantically consistent features across object instances for each keypoint k (L 168)\", which, I believe, contradicts with the motivation of the paper which aims to address the \"limited generalizability of existing methods\". In clearer terms, I believe that assigning semantic meanings to each keypoint per object can be regarded as 'additional supervision' beyond simple keypoint positions, and this poses unfair comparisons with existing work. \n\n3. The proposed method lacks comparison with other existing work, which may show strong performances on SPair-71k and SPair-U. For example, semantic matching methods which refines correlation maps e.g., [1][2], can be expected to perform better in dense semantic matching scenarios.\n\n[1] Min et al., Convolutional Hough Matching Networks, 2021\n[2] Kim et al., TransforMatcher: Match-to-Match Attention for Semantic Correspondence, 2022\n\nQUESTIONS:\nPlease refer to the weaknesses section. Specific questions below:\n\n1. How does the proposed method differ from SemAlign3D, and what additional technical / conceptual novelties does this paper hold?\n\n2. How would the proposed method perform without additional supervision (i.e., semantic meaning per keypoint), just using the keypoint positions as supervision?\n\n3. How do other existing methods (especially semantic matching methods which focus on correlation map refinement) perform on SPair-71k and SPair-U?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-06-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method for constructing a canonical 3D representation for object categories and using it to establish semantic correspondences.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThe paper is well written, and the proposed method gives good results, in particular on data for which some keypoints are available for testing, but were not annotated during training.\n\nWeaknesses: \nThe presentation appears to implicitly assume that all instances of the same object category have the same 3D shape since Eq. (3) expresses the fact that their keypoints are related by similarity transforms. This is a very strong assumption that will not hold for many object categories (think ot cats or even cars for example), and a major limitation of the proposed approach. The necessity to relax global rigidity in  geometric representations of object categories was already noted in old papers such as Lazebnik et al., \"Semi-local affine parts for object recognition\", BMVC'04 for example.\n\nIn Eq. (3) there does not appear to be any constraint on the points p_i and matrices M. But  in this case what prevents the optimization from reaching a global optimum of 0 with p_i=0 and M=0?\n\nThe proposed method learns two descriptors z and Phi, one for the canonical keypoints, and one of image keypoints. But there does not appear to be any constraint on these either, except they should be similar. What makes these descriptors informative, and what prevents them from collapsing into constant values for example?\n\nQUESTIONS:\nPlease address the concerns expressed in the weaknesses part of the review above.\n\nAdditional questions:\n\nAs far as I know, Kabsch's algorithm (or its alternatives using quaternions) find a rigid transformation, not a similarity, please clarify.\n\nIt appears to be the case that the number of key points is fixed over all category instances. For a true 3D object see from different viewpoints, half of the object (its back) will always be invisible. How do you expect to handle this problem?\n\nI didn't understand your geodesic sampling strategy : what is the geodesic distance you use since you don't know the surface of the object or the true connectivity of the points? Please clarify.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nBased on the observation that current supervised methods for category-specific 2D semantic correspondence fail to generalize to unseen keypoints not annotated during training, the authors proposed a new approach using geometric constraints at training time through a per-category, learned continuous canonical manifold.\n\nSTRENGTHS AND WEAKNESSES:\n1. The authors proposed a method that leveraged pre-trained depth predictor to give depth estimation for sparse annotated 2d keypoints in image to lift them to 3d, and optimize a canonical 3d keypoints set as correspondence reference set. And the extra introduced 3d information will make 2d correspondence be more accurate.\n2. The authors also proposed a method that generate dense 3d keypoints from the optimized sparse 3d keypoints set, will make the optimized semantic correspondence prediction model could generalize to new 2d labelled keypoints unseen during training.\n3. The authors proposed a new dataset SPair-U, which will benefit future researchers in 2d semantic correspondence area.\n\nQUESTIONS:\n1. In the beginning of section 3.1, the authors introduced a notation S, which is never referred to in the following text. Also, from the whole section 3, it’s unclear whether the method wants to optimize dense feature descriptors for dense 3d points on the continuous canonical manifold or just optimizing feature descriptors for the |K| number of 3d keypoints.\n2. In the equation (3) in section 3.2, it’s unclear how to do the optimization, since optimizing M requires p, and optimizing p requires M. I guess the optimization is conducted alternatively, but it’s unclear in the text.\n3. For the equation (3) in section 3.2, the authors want to optimize a category-specific sparse 3d keypoint set P for all input. But objects within input images will have different articulation and arbitrary deformations, which will lead to \\bar{k} got by equation (2) from different images cannot be described accurately by only rigid transformation (with scaling) of just one 3d keypoint set P. If the authors’ idea is to only approximately learn a roughly correct rigid transformation (with scaling) M and a roughly correct 3d keypoint set P from equation (3), then ablation studies on different selection of 2d keypoints annotations need to be done to validate the method.\n4. In section 3.3, equation (6) defined an optimization based on local small neighbors of canonical space C and posed space B, but the details of how to select the neighbor size k and the ablation study of how the size k will influence the results (especially for different categories, since objects within difference categories will deform in difference sense) are not included in the paper.\n\nLIMITATIONS:\n1. The writing quality of the paper needs to be improved, since there are some technical details are unclear as mentioned in the questions section. \n2. The proposed method relies on the off-the-shelf segmentation and depth estimation methods and thus the performance heavily dependent on the quality as well as the consistency of the segmentation and depth estimation results. It is expected to evaluate how robust of the proposed method for highly articulated object such as human, animal, where the segmentation and depth estimation is typically less accurate due to the large deformation as well as occlusion.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a new method for semantic correspondence estimation. They first analyse prior supervised methods and find that they act like keypoint detectors by overfitting on the few annotated keypoint classes and thus do not generalise well.\nIn their approach, they propose to use pre-trained monocular depth estimation and semantic segmentation models to estimate the 3D geometry of object instances.\nUsing these point maps, they propose to learn a canonical object space per-category. They do so by finding the optimal rigid alignment between the sparse 3D point clouds stemming from the annotated keypoints in all training images.\nFurther, they optimise a latent embedding for each of these canonicalised 3D keypoints. Using the feature similarity of other pixels to these points, they obtain their respective location in the canonicalised 3D space by linear interpolation.\nThe local neighbourhoods of given points in the so-found 3D canonicalised point maps are then compared with the posed point clouds retrieved through the frozen depth estimation model as an additional loss.\nThe paper further introduces new labels to the SPair-71k dataset that are used to evaluate the generalisation against prior supervised methods.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\nThe paper generally follows a convincing story, where the motivation of overfitted supervised semantic correspondence models is clearly motivated (Fig. 1).\n\nThe related work section is generally also well written, but missing a few important references (see weaknesses).\n\nThe authors propose a method that almost maintains the \"overfitted\" quality of supervised SC methods but enable improved correspondence for other points through their 3D-aware loss, while not requiring any additional supervision besides the annotated 2D keypoints and the class labels through the use of pre-trained 2D models.\n\nIntroducing a new test dataset for further evaluation on the SPair-71k dataset is helpful for the community.\n\n**Weaknesses:**\n\nThe related work section includes references to some unsupervised semantic correspondence methods.\nFirst, the statement of these methods being \"completely unsupervised\" (l.87) is possibly not true for all of the referenced papers as class labels are still required as weak supervision signal for category-specific learning.\nFurther, it would be advisable to include more works doing weakly supervised correspondence detection, such as (P)WarpC [1,2] or also include methods that perform 3D shape matching (which is essentially semantic matching) by using features extracted with foundational vision models [3,4,5].\n\nSeveral details on the method/implementation are missing or not stated clearly, see the questions below for details.\n\nThe method is based on the assumption that annotated keypoints are geometrically salient (as all points must lie within the convex hull of the annotated keypoints), i.e., close to farthest points on the objects.\nWhile this is often true, e.g. in SPair, assuming this might reduce generality to other datasets.\n\nThe authors state that their local neighbourhood rigidity loss (Eq. 6) is only computed for local neighbourhoods because of the difference between posed and canonical object shapes (l.180).\nHowever, the alignment of posed annotated keypoints with the canonical coordinates in P (Eq.3) is based on exactly this assumption, where rigid transformations (+ isotropic scaling) are computed as transformations between posed and canonical objects.\nIt is unclear why this rigidity assumption should be valid for the alignment of sparse keypoints but not for dense points on the surface.\n\nThe paper is lacking important ablation experiments. The authors argue that their introduced \"objectives suffice to learn a SC model\" but only show the results of the method in combination with the objectives from Geo-SC.\nWhile leaving out single loss terms of the proposed new losses might not be possible (l. 24, suppl.), the results of this experiment should have been shown.\n\n---\n\n[1] Truong, Prune, et al. \"Warp consistency for unsupervised learning of dense correspondences.\" Proceedings of the IEEE/CVF international conference on computer vision. 2021.\n\n[2] Truong, Prune, et al. \"Probabilistic warp consistency for weakly-supervised semantic correspondences.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[3] Dutt, Niladri Shekhar, Sanjeev Muralikrishnan, and Niloy J. Mitra. \"Diffusion 3d features (diff3f): Decorating untextured shapes with distilled semantic features.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[4] Wimmer, Thomas, Peter Wonka, and Maks Ovsjanikov. \"Back to 3d: Few-shot 3d keypoint detection with back-projected 2d features.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[5] Uzolas, Lukas, Elmar Eisemann, and Petr Kellnhofer. \"Surface-Aware Distilled 3D Semantic Features.\" arXiv preprint arXiv:2503.18254 (2025).\n\nQUESTIONS:\nJust to make sure that this is not the case, can you confirm that the training/evaluation is not done per-category but over all classes at the same time? So, while category-specific P and Z are learned, the feature refinement is shared over all classes.\n\nIt would be good to show the performance of the proposed objectives without the Geo-SC losses, as the proposed losses should be sufficient for learning semantic correspondence, as mentioned by the authors.\n\nIs the loss in Eq.6 computed for all points q that lie within the object mask? What is the neighbourhood size that is used to compute the loss?\n\nIs gradient stop applied for the computation of Eq.6 wrt. the locations of canonicalised keypoints P or are they not only optimised via Eq.3 and also indirectly through the minimisation of Eq.6 (given the computation in Eq.5)?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors proposed a method for semantic correspondence lifts sparse 2D keypoints into a shared canonical 3D space using monocular depth estimation. By learning a continuous, category-specific canonical manifold, the method aims to capture object geometry and guide dense correspondence learning without requiring explicit 3D supervision or camera pose information. Alongside the method, the authors introduce SPair-U, an extension of the SPair-71k benchmark, which includes additional keypoints not used during training. This allows for explicit evaluation of generalization to unseen points. Experiments show that the proposed method outperforms supervised baselines on these unseen keypoints, demonstrating improved generalization.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- Shifting the focus from sparse, supervised alignment to dense, geometry-aware representation learning\n- Leveraging off-the-shelf components like monocular depth estimation (Moge) and segmentation masks (SAM), and is integrated into an existing model (Geo-SC) in a modular way\n- The introduction of the SPair-U benchmark;  exposing the overfitting tendency of standard SC models and enabling evaluation on unseen keypoints\n\nWeaknesses\n- The method introduces several additional sources of supervision (depth maps, segmentation masks, canonical 3D supervision) that are not available to baseline SC models, raising concerns about fairness in the performance comparison. The use of off-the-shelf modules like Moge and SAM means the observed gains could stem as much from auxiliary data as from the core method itself. This issue is not adequately discussed, and no control experiments are provided to isolate the contribution of each added component.\n- As for the benchmark, while SPair-U introduces new test keypoints, the extension is somewhat limited: only a few extra keypoints are added only for testing data. For the practical utility of the benchmark as a community resource,  it is desirable to provide dense annotations across all images, including those in the training set, by leveraging oracle-like 3D canonical representations.\n\nQUESTIONS:\nPlease see the weaknesses section above.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}