Title: Weak-shot Keypoint Estimation via Keyness and Correspondence Transfer
Venue: NeurIPS 2025
Accepted: yes
Keywords: weak-shot learning

pose estimation

transfer learning

unsupervised learning

few-shot learning
Abstract: Keypoint estimation is a fundamental task in computer vision, but generally requires large-scale annotated data for training. Few-shot and unsupervised keypoint estimation are prevalent economical paradigms, but the former still requires annotations for extensive novel classes while the latter only supports for single class. In this paper, we focus on the task of weak-shot keypoint estimation, where multiple novel classes are learned from unlabeled images with the help of labeled base classes. The key problem is what to transfer from base classes to novel classes, and we propose to transfer keyness and correspondence, which essentially belong to comparing entities and thus are class-agnostic and class-wise transferable. The keyness compares which pixel in the local region is more key, which can guide the keypoints of novel classes to move towards the local maximum (i.e., obtaining keypoints). The correspondence compares whether the two pixels belongs to the same semantic part, which can activate the keypoints of novel classes by reinforcing the consistency between corresponding points on two paired images. By transferring keyness and correspondence, our framework achieves favourable performance for weak-shot keypoint estimation. Extensive experiments and analyses on large-scale benchmark MP-100 demonstrate our effectiveness.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the concept of weak-shot learning for pose estimation, wherein the model is trained using labeled examples from base classes and unlabeled examples from novel classes. The proposed method processes pairs of images belonging to the same class, encoding them through a shared encoder and a latent diffusion model to estimate keypoints. Additionally, the approach generates keyness maps, indicating the direction toward the next keypoint, and establish correspondences between the images in each pair. For labeled images from base classes, a supervised loss is employed, while an unsupervised loss is proposed for unlabeled images from novel classes. The effectiveness of the method is demonstrated on various splits of the MP-100 dataset, with comparisons made against both unsupervised and few-shot learning approaches.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\nThe authors are first to apply the concept of weak shot learning on pose estimation, which is of high relevance due to the sparsity of labelled images in this field. \n\nThe method outperforms existing unsupervised and few-shot approaches by a large margin under the given data availability. The method even outperforms few-shot methods that have additional labels of novel classes. \n\nThe paper shows evidence that the addition of the keyness maps substantially enhances the performance in comparison to prior works. \n\nWeaknesses: \n\nMissing details on evaluation.   \n\nTo enable fair comparisons between methods, it is necessary to specify the number of labeled/unlabeled training samples and classes used for each method.  \n\nAdditionally, the authors should elaborate on the degree of similarity between the base and novel classes; while Appendix A4 provides some information, this discussion should be extended and included in the main text. \n\nThe overall clarity of the paper can be improved due to missing details and misleading terminology (see below).\n\nQUESTIONS:\nTable 1 in the supplementary material indicates that shuffling the keypoint order significantly degrades the performance of Hedlin et al., despite this method being unsupervised. What factors might account for this observation? \n\nHow computationally expensive is the computation of the correspondence map, considering that the approach requires calculating the cosine similarity between every pair of pixels? \n\nIs the \"reference image\" truly a reference in the conventional sense? According to my understanding, it simply refers to another image belonging to the same class. If so, this terminology may be misleading.In Equation 5, M_i* does not denote ground truth, whereas the same symbol is used to represent the ground-truth keyness map in Equation 4. This inconsistency could lead to reader confusion.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a weak shot pose estimation, where multiple novel classes are learned from unlabeled images from the labeled base classes.\n\nThe proposed method leverages keyness (gradients of the keypoint that indicate the direction to a more important key location) and correspondence from the base class by transferring them to a novel class to form the unsupervised learning of the novel classes. Keyness and correspondences are estimated from the diffusion model. Novel classes are trained from offset guiding loss, consistency loss, and localization loss, while the base class is trained with keypoint loss, keyness loss, and correspondence loss.\n\nThe paper shows an experiment comparing the results on the baselines with similar setups as well as a fully supervised setup, and shows that the proposed method outperforms the baseline with the similar setup, while performs comparable results on the fully supervised setup.\n\nSTRENGTHS AND WEAKNESSES:\n1. Strength:\n\n- The paper is not my field, but the proposed method covers a less explored aspect of keypoint estimation, and technically sound & the experiment results look reasonable. I don't find any noticeable errors in the paper that need to be fixed. Also, the proposed method has great potential to be used in other computer vision fields.\n\n2. Weakness:\n\n- It's possible that this issue is coming from a terminology difference between different fields, but as a person from the 6D pose estimation field, the term \"Pose Estimation\" is somewhat misleading. A better term to be used is \"Keypoint Estimation\". The paper is mainly focused on keypoint estimation, and no where to find actual \"pose\" estimation.\n\nQUESTIONS:\n- Found one typo in line 130. I assume it's \"single-class\" instead of \"sing-class\"?\n- Estimating keypoints is not necessarily a pose estimation. It would be great to avoid using the term \"pose estimation\" too many times in the paper.\n- It would be great to have actual name on each loss term when they are introduced, for example in line 153: correspondence via XXX, XXX, XXX loss in base class (L_BP, L_BK, and L_BC), and line 154: keypoints via XXX, XXX, XXX loss in novel class (L_LG, L_NC, and L_NU)\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 1, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a weak-shot keypoint detection approach, which aims to detect semantic-rich keypoints on new classes without labeled training data.\nThe method is first trained on labeled data and then reuses the pretrained prompt tokens to detect keyness maps on new classes. Then, in new classes, the keyness map is used to guide the pixel movement to high-intensity change regions, where the semantic correspondences across images are used to provide consistency loss. Experiments are conducted on the MP-100 dataset.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n(1) The proposed keyness constraint and correspondence constraint on the new class are reasonable and seem effective\n\n(2) Experiments and analysis are well conducted. The paper provides comprehensive ablation studies to validate the effectiveness of the proposed method.\n\n# Weaknesses\n\n(1) Definition of \"class\". It is not clear to me whether new class means new category (cat to dog) or new instance (cat 1 to cat 2). The paper seems to focus on a new instance. Otherwise, the reuse of the pretrained prompt tokens on different categories may introduce severe bias towards the new category.\n\n(2) Task definition. The paper seems to estimate a set of keypoints that can be matched across different instances. However, the reviewer thinks this work may not belong to the pose estimation task, which aims to detect the predefined set of semantic keypoints, and can benefit the downstream tasks like action recognition, skeleton-driven animation, etc. \nIf the reviewer does not misunderstand, the detected keypoints on the new class in this work are only \"semantic-rich\", but don't have specific (known) semantic meaning. \nTherefore, it should be a semantic keypoint detection work instead of a pose estimation work.\n\nQUESTIONS:\n(1) Can authors provide a clearer definition of \"new class\"?\n\n(2) Given that the prompt tokens are pretrained exclusively on the base categories, could using them during the unlabeled training phase for novel categories inadvertently introduce bias?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new task, named weak-shot pose estimation, and designs a framework to transfer keyness and correspondence from base classes to novel classes for the focused task. To address the lack of annotations for novel classes,  an offset guiding loss and a consistency loss are introduced to guide the learning of heatmap, keyness, and correspondence. Experiments are conducted on MP-100 benchmark to demonstrate the effectiveness of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\n##  Strength\n\n- The paper proposes the new task of weak-shot pose estimation, and designs a framework to transfer keyness and correspondence from base classes to novel classes.\n\n- Self-supervised losses on novel classes, e.g., offset guiding loss and consistency loss, are introduced to guide the learning of  heatmap, keyness, and correspondence.\n\n- The method significantly outperforms the established baselines by a large margin on MP-100 benchmark.\n\n- The paper is well-organized.\n\n\n## Weakness\n\n- How is the keyness map $M$ used to identify the corresponding keypoint for each pixel?\n\n- For the base classes, how is the visibilities $V$ supervised?\n\n- The description of the offset-guiding loss (Lines 179–184) is somewhat unclear.  How is the visibilites $V$ obtained via the argmax operation? Is there any thresholding involved? How exactly is Eq. (5) used to supervise visibility? Does $M_i^*[P_i[k]]$ refer to the keyness at pixel $P_i[k]$? What is the precise operation of dt()? How does the gradient propagate through $P$ given the use of argmax operations?\n\nQUESTIONS:\n- What's the meaning of K In Eq. (3) and Eq. (5)? Why not M? Is K the exact number of keypoints?\n\n- No illustrations of $S$ and $w_{l,s}$ in Line141.\n\n- In Table 5, does the “Basic” result in the first row refer to unsupervised setting? It still outperforms other methods under the weak-shot setting by a large margin. If so, what advantages does the proposed framework have over existing approaches?\n\n- Typo: Line133 \"\\Phi_^c_l () and \\Phi_^c_l () $ as the c-th ...\"; Line183 \"detach and block\" -> \"detaches and blocks\"\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}