Weak-shot Keypoint Estimation via Keyness and Correspondence Transfer
Authors: Junjie Chen
Zeyu Luo
Zezheng Liu
Wenhui Jiang
Li Niu
Yuming Fang
Venue: NeurIPS 2025 poster ()
Keywords: weak-shot learning
pose estimation
transfer learning
unsupervised learning
few-shot learning
Primary area: applications

Abstract: Keypoint estimation is a fundamental task in computer vision, but generally requires large-scale annotated data for training. Few-shot and unsupervised keypoint estimation are prevalent economical paradigms, but the former still requires annotations for extensive novel classes while the latter only supports for single class. In this paper, we focus on the task of weak-shot keypoint estimation, where multiple novel classes are learned from unlabeled images with the help of labeled base classes. The key problem is what to transfer from base classes to novel classes, and we propose to transfer keyness and correspondence, which essentially belong to comparing entities and thus are class-agnostic and class-wise transferable. The keyness compares which pixel in the local region is more key, which can guide the keypoints of novel classes to move towards the local maximum (i.e., obtaining keypoints). The correspondence compares whether the two pixels belongs to the same semantic part, which can activate the keypoints of novel classes by reinforcing the consistency between corresponding points on two paired images. By transferring keyness and correspondence, our framework achieves favourable performance for weak-shot keypoint estimation. Extensive experiments and analyses on large-scale benchmark MP-100 demonstrate our effectiveness.

Weak-Shot Keypoint Estimation via Keyness and Correspondence Transfer
Junjie Chen'
Zeyu Luo'
Zezheng Liu'
Wenhui Jiang'
Li Niu Yuming Fang'
Jiangxi University of Finance and Economics
"Shanghai Jiao Tong University
(chenjunjie, 2202320619, 1202200661,,,, 2202426552, jQj edu. edu. en, jstcnewlyssjtu. edu. cn, fangyuming@jxufe xufe edu cn
Abstract
Keypoint estimation is fundamental task in computer vision, but generally re- quires large-scale annotated data for training. Few-shot and unsupervised keypoint estimation are prevalent economical paradigms, but the former still requires anno- tations for extensive novel classes while the latter only supports for single class. In this paper, we focus on the task of weak-shot keypoint estimation, where multi- ple novel classes are learned from unlabeled images with the help of labeled base classes. The key problem is what to transfer from base classes to novel classes, and we propose to transfer keyness and correspondence, which essentially belong to comparing entities and thus are class-agnostic and class-wise transferable. The keyness compares which pixel in the local region is more key, which can guide the keypoints of novel classes to move towards the local maximum (i.e., obtaining precise keypoints). The correspondence compares whether the two pixels belongs to the same semantic part, which can activate the keypoints of novel classes by reinforcing the consistency between two paired images. Extensive experiments and analyses large-scale benchmark MP- 100 demonstrate our effectiveness.
Introduction
Keypoint estimation is a fundamental computer vision task and has extensive applications in real world, including intelligent interaction [65]], behavioural analysis [D]] and augmented reality [[6]] Although existing keypoint estimation methods [ML. EL 533 have achieved great success, they usually require large-scale annotated data of all classes for fully supervised learning. As consequence, the expensive annotation dramatically limits category-wise expansion and wider application.
Few-shot learning and unsupervised learning are two prevalent paradigms to economize annotations. As illustrated in Fig. M (a), few-shot keypoint estimation [69, 59, EI] greatly reduces the number of labeled images of novel classes (i.e., novel object categories), and unsupervised keypoint estimation [64. (79. I reduces the annotations of target class. However, few-shot methods still requires non- negligible annotations for each novel class, and thus the annotation cost can become substantially high with large number of novel classes. Unsupervised methods only focuses on single class, and is difficult to promise that the discovered keypoints are desired. Thus, it would be more practicable we can learn desired keypoints from unlabeled images for multiple novel classes.
In this paper, we follow the paradigm of weak-shot learning [7] 38 H. 26] and explore the task of weak-shot keypoint estimation. Weak-shot learning has achieved promising success in economizing annotations for various tasks, including classification [[]], detection [38. 80, 32]], segmentation [&
27 266 B. X]]] and so on [61]] Specifically in keypoint estimation, we would like learn keypoints
"Correspondngg author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Offict Guiding Lovs
Inpul Image X
Base Classes
Novel Classes
Keynexs Map
Supervised Learning:
Keypoint Hestmmaps Hi
Diffusion Model
Few-shot Learning:
Consiseency Loss
Unsupervised Leamnnn:
(single class only
Our Setting:
annokated image
unannotated image
Keypoint Heatruus Hr
Paired Imugee Xy
Keyness Map M,
Offset Guiding Loss
(a) Comparison of various task seltings.
(b) Overview of our framework.
Figure 1: (a) Data comparison among fully supervised learning, few-shot learning, unsupervised learning and our weak-shot learning. Our setting is more economical and practicable. (b) Our proposed framework transferring keyness and correspondence from base classes to support the un- supervised learning of multiple novel classes with offset guiding loss and consistency loss.
for novel classes from unlabeled images with the support of labeled images of base classes, as shown in Fig. (a). Intuitively, we leverage the knowledge transferred from base classes to facilitate the unsupervised learning of multiple novel classes. Therefore, our setting integrates few-shot and unsupervised keypoint estimation to alleviate their drawbacks (i.e., annotations on novel classes, single-class discovery, and uncontrollable discovery).
In weak-shot keypoint estimation, the key problem is what to transfer to facilitate the unsupervised learning of novel classes. A representative paradigm [63. [ in previous works is to transfer the comparing entities (or relation), which is class-agnostic and class-wise transferable. Different from the image-level comparison in [63, M] here explore compare pixels to facilitate keypoint local- ization. Specifically, we conduct the following two types of comparisons: intra-image comparison (i.e., keyness) and inter-image comparison (i.e., correspondence).
Respectively, for intra-image comparison, we propose to compare the pixels in local regions, since "keypoint" can be understood as the most key point within local region the point that are more key compared to nearby ones (dubbed as "keyness"). Given a kcypoint Gaussian map, we compute the two-dimensional derivative as its keyness map, where each pixel is a 2-D vector indicating the more key direction. We learn such keyness from the annotations of base classes, and employ the transferred keyness the keypoints of novel classes to move towards the local maximum (i.e., by offset guiding loss). For inter-image comparison, we propose to compare the pixels in two images belonging the same class, since "keypoint" should be semantic consistency across images. Given two images from the same class, we compute the bipartite pairwise similarities to derive correspondences as in [24]]. Analogous to keyness transfer, class-wise transfer correspondences to activate the keypoints of novel classes (i.e., by consistency loss).
Based on aforementioned keyness and correspondence transfer, design well-tailored and effec- tive model for weak-shot keypoint estimation, as shown in Fig. M (b). In particular, we follow [[]] and build our model upon pretrained diffusion model [55]] because unsupervised keypoint estima- tion quite challenge, let alone in the multi-class scenario. The training data contains image pairs from both base classes and novel classes. For the image pair from base classes, enable the loss terms to learn keyness and correspondence. Otherwise, we switch the loss terms and apply estimated keyness and correspondence to support the unsupervised learning of novel classes.
We conduct comprehensive experiments and in-depth analyses on the large-scale multi-category pose dataset MP-100 100 [69]. Because the model has never accessed the GT keypoints of novel classes, we adopt the matching-bass based evaluation [[L3]] and regression- based evaluation [[X]] for quantitative analysis. Our contributions could be summarized as follows:
(1) We are the first explore weak-shot keypoint estimation, where we could use the knowledge transferred from base classes to facilitate the unsupervised learning of multiple novel classes.
(2) We propose well-tailored framework learn keyness and correspondence from base classes, and transfer them to provide effective supervision for the unsupervised learning of novel classes.
(3) Extensive experiments the large-scale dataset demonstrate the effectiveness of our method.

---

2 Related Works
Fully-Supervised Pose Estimation. Most existing methods focus on fully-supervised learning for specific class, including human [01, 29), animals [SS 27 and vehicles
60]] Technically, these 6G,
[53
I5I, heatmap-baae basedd [[55 methods. For example, DEKR US employed adaptive convolutions through spatial transformer to activate keypoint regions and learn representations for more accurate keypoints. Above works have achieved great success for estimating pose of specific classes, but they require abundant annotations and are inapplicable for novel classes. In this paper, focus estimating poses for multiple novel classes.
methods could be roughly classified into regression- based [S5, ZZC 5%
28,
G5.
m.. 20 2T1
2a). E.]. and transformer-basee
DX
re!
Zero-Shot and Few-Shot Keypoint Estimation. Zero-shot keypoint estimation locates keypoints for novel classes using descriptions, and existing methods generally learn mapping from descrip- tion to keypoints| [ZT] 72 55. 92, Ki, 77i 778). Our task setting closer to few-shot keypoint esti- mation, which locates keypoints for novel classes using a few examples. Early methods focus on specific domains, e.g., facial images [L. [i] clothing images [[=]] or animal images [[22]. Recently, POMNet [69] introduced the task of category-agnostic keypoint estimation, and proposed match- ing framework to retrieve results based on few-shot. Later, CapeFormer (59] further enhanced the similarity modeling and proposed to refine the coarse keypoints via transformer decoder. Mean- while, Lu et al. [a0]] explored a more flexible few-shot scenario to learn novel/base classes and novel/base keypoints. Although existing methods [BC] ET, [T AS. RI BT,, 541, 5 have great pro- moted few-shot keypoint estimation, they requires non noglnnglii annotations for novel classes. In contrast, we propose to harness diffusion models and learn novel classes from unannotated images.
Unsupervised Keypoint Estimation. Unsupervised keypoint estimation only require unlabeled images to discovery keypoints for single class. Overall, the high-level idea of most existing method to design keypoint bottleneck in image reconstruction [641, IZ, I6. RD] or video reconstruction [23, 57]. Recently, Hedlin et al. [I8) proposed to employ attention maps of text embeddings in pretrained diffusion models as keypoint heatmaps, and optimize text embeddings with localization loss and cquivariance loss for unsupervised keypoint estimation. Although above methods could discover keypoints from unlabeled images, most of them are difficult to discover desired keypoint for multiple classes. In this work, we take the inspiration of [[S] and propose well-tailored framework to transfer keypoint prompts and correspondences for weak-shot keypoint estimation.
Weak-Shot Learning Deep learning methods generally require large-scale labeled data, and thus the demand of reducing annotations is extensive various tasks. To this end, weak-shot learning. i.e., learning weakly labeled novel classes with the support of fully labeled base classes, has been explored in classification [77] 511, (50], detection [B&, 80), semantic segmentation [M. 8J], instance seg- mentation [22, 26]], and so on [61]]. To name few, Sim'Trans [ proposed to transfer the pair-wise similarity to de-noise the web data of novel classes. RETAB [X]] proposed to transfer affinity and boundary to expand seed mask to semantic mask. Here we explore weak-shot keypoint estimation, and propose to transfer keyness and correspondence to support multiple novel classes.
3 Method
3.1 Task Setting of Weak-Shot Keypoint Estimation
In weak-shot keypoint estimation, we would like to learn keypoints for novel classes from unlaheled images with the support of labeled images of base classes. Specifically, in the training stage, there are labeled samples from base classes available, and the i-th labeled sample consists of image æ; e R x wai and GT keypoints P. E IR K x2, Note that, Ki is the keypoint number, and the images from different classes could have different keypoint numbers. For ease of process, we pad all keypoints to unified number К with visibilities, i.e. [P:: V.*] IR (2+11, There are also unlabeled samples for novel classes available, which have no overlap with base classes. In the test stage, the model should locate the keypoints for images from multiple novel classes.
3.2
The Forward Pipeline of Our Framework
As shown in Fig. we propose a compact framework for learning and transferring keyness and correspondence. In the training stage, each mini-batch contains images randomly sampled from
3

---

Sub-Fiiguee
Keyness Map Mj
(b
Fas
Procept
Office Candines Leg ANo
Iapor Image
Narty Keynee
Agarcazic
from (base or novel)
Exc
Sub-Fgurn
Iqqez Iragge
Paincl Imsge
Prompts
Condimeey Loss
Keyness Map M,
Conopeaalnn
Figure 2: (a) The detailed illustration of our framework in the training stage. Given image pairs from the same base or novel class, we employ keypoint prompts to estimate keypoint heatmaps and aggregate feature maps to estimate keyness and extract correspondences. For labeled image pairs from base classes, we learn keypoint prompts, keyness and correspondence via LBP, LBK and LBC. For unlabeled image pairs from novel classes, we transfer them to learn valid keypoints via LNG. LnC and LNU. The sub-figures (b) and (c) illustrate the insight of LxG and LNC.
both base classes and novel classes. Additionally, we pair each training image æ; with a reference image z, (i.e., paired image) from the same class, and apply respective supervisions image pairs from base or novel classes. To estimate keypoints, we follow [[]] and maintain learnable keypoint prompts E E R xD which correspond to text/query embeddings and set N 200 by default. Different from the single-class scenario in [[8], our keypoint prompts are shared by all images and all classes, and thus could learn transferrable knowledge under
match-based supervision.
Firstly, for each image in the input pair, we obtain keypoint heatmaps according to keypoint prompts E. We denote the (-) and (-) as the e-th head and the (-th linear layers of the U-Net in the trans- former part of pretrained Diffusion model. We compute the query Qf 46 (=
R H W DD,
where z the latent embedding mapped from image æ, and we use We compute the key from keypoint prompts Kf
as suggested by [[8].
(E((E) Nx Then we obtain keypoint
heatmaps H E HxWNN by collecting cross-attention maps from various layers:
H =
Ex softmax..
Ki/VDi)]
(I)
In this way, we can obtain the estimated keypoint heatmaps H, E #xWN and H, E BR for images æ; and x, using keypoint prompts E. More details could be found [I8].
Secondly, we aggregate the feature maps from above diffusion process and obtain hyperfeatures F Hxx x for estimating keyness. Concretely. F ΣS E/ w1,- f((Ft.), where (F....) squeezed feature map, S the number of subsampled timesteps selected from the diffusion timesteps, and Wi.s denotes the mixing weights for the 1-th layer at the i-th timestep, and we use the architecture of [E Since keyness compares the pixels local regions, we intuitively estimate it via conv. layers with act. functions. We denote the estimated keyness map as M E x where each pixel M|x, [z, y] a 2D vector indicating the direction to the most key pixel.
Finally, we use the hyperfeatures of the paired images to compute the semantic correspondences, denoted as F; and F, respectively for images æ; and x,. Specificallyy, we compute the mapping Φ(-) from keypoint to its corresponding keypoint in the other image as:
p(p)) arg max S [p.), F.(p.)). Pr
where and Pr are points two paired images and ((: denotes cosine similarity. As in [+]]
we only need to compute the similarity between keypoints on æ; and all pixels on ør, and thus the computation is relatively efficient and affordable.

---

Given above estimated keypoint heatmaps (H, Hr,,, keyness maps (M, M,), and correspondence ((-) for image pair (.). we switch the loss terms for learning from base classes and transferring to novel classes. For labeled image pairs from base classes, we learn keypoint prompts, keyness and correspondence via LBP, LBK and LBC. For unlabeled image pairs from novel classes, we transfer them to learn valid keypoints via LG LN and LNC. We introduce the loss terms as following.
3.3 Learning from Base Classes
To learn keypoint prompts from base classes, we apply following loss over the keypoint heatmaps:
K - H1 + H,1000)
1 K
CBP
H;
(3)
where [:] is the k-th GT keypoint Gaussian map generated as in [69]] Besides, the max value for H; [k] is 1 if the i-th keypoint is visible, otherwise 0. 8(-) is the matching function solved as in [R,, IIL РI. Although is feasible to directly apply the matching-loss to each image respectively as in 19 III Ei], propose conduct joint matching over two paired images in light of the semantic consistency of keypoints. Specifically, we calculate the cost C E R X by considering the cost over paired two images i.e., C(n, k] H [n] H; [k]]] + [H,[n]
[K] Such loss can adaptively embed knowledge into the same matched prompt, leading to transferability.
To learn valid keyness, we apply the loss analogous to Eqn. B due to the same "map" representation:
CBK
M. M
+
M, M;
where M E 2x2 indicate the GT keyness map of input image or paired image, where each pixel M I y| is a 2D vector indicating the direction to the most key pixels. We compute the deriva- tive of GT keypoint Gaussian map to obtain GT keyness map, i.c., the GT of [rr y) is the derivative of the GT keypoint Gaussian map at [ar yl. To learn valid correspondences, we compute cosine sim- ilarity between every possible pair of points and apply symmetric cross entropy loss according to GT keypoints as [4 which is denoted as LBC.
Note that, the names of above loss terms (i.e., LBP, LBK and CBc) all begin with B, indicating that they are enabled only when the image pair comes from Base classes.
3.4 Transferring Novel Classes
For the unlabeled image pair æ; and σ, belonging to novel classes, we transfer and reuse the key- point prompts E to estimate keypoint heatmaps H, and H,. Note that such estimations without accessing any data of novel classes satisfyingly locate keypoints for multiple novel classes, which will be demonstrated in Sec. B. Nevertheless, the domain gap between base classes and novel classes inevitably matters. To bridge the domain gap, we propose to apply offset guiding loss and consistency loss according to the transferred keyness and correspondences.
Specifically, we firstly convert the heatmaps H, and H, to the keypoint coordinates and visibili- ties [P;; ViJ and [P,; V.] E Nx++1) via the soft argmax and max operators in [59]. Given the transferred keyness map, we guide the coordinates with following loss:
N
.NG = N > P.N M| + P. di(P + + M,P., (5)
where []]]][K]] is an offset vector indicating the direction for P. be more key, and dt(-) detaches and blocks the gradient to provide stable supervision. By such offist guiding loss, the estimated keypoints could be supervised gradually offset to more key positions.
For the consistency loss, we reuse the aggregation network obtain pseudo-points P. Nx2 by corresponding P; from image 2; to image Fy,, i.e. P. (():))rr- We obtain pseudo-points P. similarly. Then, we use below consistency loss to facilitate the keypoint learning of novel classes:
N
Lxc = Σ V [n]
H.[m] - H M(P)))))] + Vinn] H. - P((P.|n))
(6)
a
5

---

where Vn is the visibility of the n-th pseudo-pon point blocking wrong supervision from invisible pseudo-point, and H((-) is function mapping keypoint coordinates to Gaussian heatmap [69]] By our consistency loss, the keypoint heatmaps estimated on the unlabeled images of novel classes are regularized to approximate to the corresponding keypoints on the paired images.
Besides, we apply the equivariance and localization loss to complement our unsupervised learning on novel classes, which are proposed in [TS] for single-class unsupervised learning. We use the same configuration and denote as LNU. Note that, the names of above losses (i.e., LNG. LxC and LNU) all begin with N. indicating that they are enabled when the image pair comes from Novel classes.
3.5 Summary of Training and Inference
Our framework is end-to-end learnable, which can jointly learn keypoint prompts, keyness and cor- respondence from base classes and transfer them to novel classes. In the training stage, our full loss for the i-th image æ; can be summarized as:
(7) where ( is the indicator function, B (resp., .N) denotes the image set of base classes (resp., novel classes). Hence, we respectively apply the supervisions for the base images and novel images. As for the loss balancing, we find that LBP balance well with LAK. may due to the same map representation. Besides, a 0.1, B = 0.2, 7 = 0.1 and À = 0.5 are hyper-parameters for balancing the other losses.
LFULI = EB EB (LBP + LBK + a LBC) + 1,,EN' (3 LNG + 7 LNC + A LNN)-
In the test stage, we remove the additional modules estimating keyness and correspondence, which are only used to provide effective supervisions for learning novel classes without annotations. Com- pared with the related work [[]] in unsupervised keypoint estimation, our model keeps the same ar- chitecture for inference and has extra capacity to estimate keypoints for multiple unlabeled classes.
Experiments
4.1 Dataset and Implementation Details
We follow the related works [6% 59 M] to conduct our experiments on MP-100 dataset [6%]] which the most prevalent benchmark dataset and covers 100 classes within 8 super-classes. MP-100 dataset contains keypoint numbers ranging from 8 to 68 across different classes. We follow the dataset splits in [6%]] i.e., all classes are split into non-overlapping train/val/test sets with the ratio of 70 10 20, and there are five random splits (S1-S5) to reduce the impact of randomness. Considering that our model is built pretrained diffusion model, focus learning from a small-scale training set in this paper, similar to large model adaption [66]] Unless otherwise stated, we use 3 labeled images per base class and 30 unlabeled images per novel class by default.
In practice, the keypoint orders of different classes matter transfer learning. Specifically, the default orders in MP-100 are related, e.g., the first keypoint of base/novel classes always corresponds to "left-eye"" On the one hand, such relation requires that all class annotations keep explicit relation and thus severely limits class-wise extension. On the other hand, models may class-wiss transfer by overfitting such unpractical relation. Therefore, we use the shuffled orders of each class experi- ments, which has less limitation and thus more practical.
4.2 Evaluation and Metric
Similar to unsupervised keypoint estimation [1K], our model have never accessed the GT annotations of novel classes, and thus we cannot directly compare the estimated keypoints with GT keypoints. Thus, we employ the following two kinds of evaluations.
Matching- evaluation Considering that the number of GT keypoints varies across classes, we follow unsupervised semantic segmentation || to compare the estimation with GT based on matching. We employ Average Precision (AP) to compute the metric, because unmatched key- points will reduce recall rate and thus reduce AP. Specifically, for the test image I; with padded GT keypoints| IP;; V."] E (2++)+1) come from novel class, the model produces padded keypoints [Ps V.J (2++11) Then, compute the matching cost over all test images in each class by:
C,, = [i'kk] 1 ((P.M)) ((VI))>)))
(8)

---

Table 1: Comparison with prior works. and regression-bant evaluation.
We report results 5 dataset splits with matching-basee baasdd
MP-100
Splitl L2
MP-100
Split2 124
MP-100 Split3
MP-100 Split4 L2
MP-100
Split5 L2
AVG
Method
MAP1
mAP
mAP 1
L2
mAP 1
mAP
mAP
L2
Sim.Base. [X
14.4
60.8
12.9
65.3
13.3
63.5
13.1
64.4
14.2
61.3
13.6
63.1
GroupPose [
17.4
58.2
17.0
57.1
15.9
58.4
18.2
52.1
17.0
He et al. [[66]
18.3
51.5
16.2
56.6
18.6
51.7
17.8
54.3
19.5
50.2
18.1
52.9
Hedlin er al. [[X]
27.0
38.3
22.1
50.3
23.5
38.6
22.1
44.2
24.1
47.1
23.8
43.7
MetaPoint [B]]
31.5
33.5
36.2
31.7
30.1
22.4
26.8
25.9
36.9
39.6
32.3
30.6
Ours
70.4
26.2
61.7
21.3
24.3
62.6
22.0
Oracle*
86.1
14.4
78.3
21.6
72.8
16.9
69.6
19.4
76.0
20.3
76.6
18.5
100
Image
2) Keypoints
MetaPoint
4) MetaPoint-8())
5) Ours
Ouxx-6())
Figure 3: The qualitative comparison against the most competitive baseline. The first two columns show the image and GT keypoints. The mid two columns display the localized keypoints and matched keypoints of MetaPoin([]). Note that we use transparency show the visibility, the right- bottom number indicates the Fl-score of keypoint of this sample, and the black arrows show the deviations to matched GT. The right two columns display the same results of our method.
where 1(.) is the indicator function, Tр is distance threshold in PCK A and T. is score threshold for computing AP. We follow [[5]] to adopt T, = 0.2, and use 9 scores from 0.1 to 0.9 to calculate AP. Finally, obtain overall metric by averaging the AP of all classes, i.e., mAP1.
Regression- evaluation We also follow related work [[S. IZ mation with GT based linear regression for better comprehensiveness. Specifically, linear regression model is firstly trained to map estimated keypoints to padded GT keypoints on val im- ages. Afterwards, in the formal evaluation, all the estimated keypoints on test images are mapped by above regressor, and averaged L2, error to GT is employed as overall metric.
69 compare esti-
4.3 Comparison with Prior Works
Comparable baselines. far as we know, there no method designed for weak-shot keypoint estimation, and thus we adapt representative methods to our baselines, including Sim.Base. [68].
GroupPose [}7], J
al. [16],, Hedlin el al. [[X] and MetaPoint [D]. Specifically, Sim.Base. [68],
He al. [16], and Hedlin el al. [0S] estimate the heatmaps of padded GT keypoints directly.
GroupPose [i regresses the coordinates of padded GT keypoints directly. MetaPoint [9] contains
the first stage in [X]] which learn potential keypoints with matching-bait based supervision. We supervise

---

all methods with heatmap loss or coordinates loss on labeled images of base classes, and comple- ment all methods with unsupervised loss (i.e. equivariance loss and localization loss as in [TS] on unlabeled images of novel classes. We also set baseline named Oracle" to show our upper-bound by training our model with all labeled images from both hase class and novel classes.
Quantitative comparison. We summarize the results of above baselines and our method on five dataset splits in Tab. M Firstly, we could find that our method outperforms baselines by large mar- gins in both evaluation metrics. Although Hedlin et al. [18] could utilize pretrained diffuuson model, cannot adaptively learn transferrable keypoint prompts and thus only achieves dissatisfactory per- formances. MetaPoint [9] can learn transferrable keypoints, but employs common matching- -bsed loss on single image and also requires abundant annotated images for training. By contrast, our model can leverage pretrained diffusion model to adaptively learn and transfer keypoint prompts and correspondences, resulting preferable performances. Besides, our model can reach about 75% of the upper-bound mAP represented by Oracle", showing promising potential. Overall, our proposed model has relatively robust mAPs on different splits using both evaluation protocols.
Qualitative comparison. We select the most competitive baseline (i.e. MetaPoint [9] for quali- tative comparison in Fig. C. With threshold T. = 0.5, we show the Fl-score (harmonic mean of precision and recall) of matched keypoints in the right-bottom of sub-figures. In the top two rows, the baseline may locate right keypoints for horse, but fails to keep semantic consistency across im- ages (by comparing two rows). In the bottom two rows, the baseline seems to fail to locate keypoints for bird, may due to the challenging domain gap between base and novel classes. Overall, the key- points located by model could better fit the object structures and keep semantic consistency across images, which is general requirement in keypoint estimation. Therefore, our method could learn preferable keypoints in weak-shot keypoint estimation.
4.4 Comparison with Other Settings
Our weak-shot keypoint estimation is related with two prevalent settings (i.e., few-shot and unsu- pervised setting) as shown in Fig, II. Here we conduct setting-wise comparison to investigate our potential.
Comparison with few-shot setting. We set representative few-shot baselines including PoseAny- thing [[9]] MetaPoint [9]] and SCAPE [[4|]. Considering that Peng et al. [522] requires test-time optimization, we don't include it as baseline. As shown in Tab. Q few-shot methods [T9, M, [E]] generally achieve inferior performances in our experiments, due to the limited training data (3 la- beled images per base class described in Sec. I...). Although few-shot methods may outperform our method by using extra labeled support images from novel classes, the support images are not always available for all novel classes and will have expensive cost if we annotate for a large num- ber of novel classes. Therefore, our method has the unique advantage on transferring knowledge upon diffusion models to facilitate the unsupervised learning of multiple novel classes, which is ineelaceable against few-shot methods.
Comparison with unsupervised setting. We set representative unsupervised baselines including AutoLink [07]] and Hedlin er al. [[и]] Considering that existing unsupervised methods [[7, [8] mostly focus on single-class scenario, we conduct the comparison using two splits. In the split "Novel Bird", the novel classes are all the bird classes in MP-100, where the unsupervised methods [IZZ [S] could be well applied. In the "MP-100 Splitl", the novel classes contain multiple classes, e.g. Bed and Lion. As shown in Tab. EL the unsupervised methods Hedlin et al. [[X] dramatically degrades when changing the split from "Novel Bird" to "MP-100 Splitl", probably because that they require pre-define keypoint number and thus are not intuitive to tackle multiple classes having different keypoint numbers. Our method outperforms unsupervised method Hedlin et al. [[]] dramatically in both splits, because method could simultancously tackle multiple novel classes and leverage the transferred knowledge.
4.5 Method Analysis
Analysis on keyness transfer. We learn keyness from labeled base classes and transfer to unlabeled novel classes, and thus the transferability of keyness is key factor of our method. We quantitatively evaluate the estimated keyness map on base classes and novel classes to investigate its transferability. Tab, A summarizes the averaged L2 distances (after multiplying 100 as in [[S]]] for better readability)
8

---

relatively lower because Table 2: Comparison with few-shot baselines.
Table Comparison with unsupervised baselines.
Extra annckated
MP- 100 Split
MP-100
Spla2 L21
Off-hhe-hell Base Labels
Novel Bird
MP-100
Split]
Method
mAPt
L2.
mAPt
Method
mAPt
L21
mAP?
L21
PostAnything [T]
41.7
22.2
32.7
36.0
AutoLink IM
14.5
60.7
8.2
70.1
MetaPoint [+]
42.3
27.2
33.1
35.3
SCAPE
Our
56.1
22.1
70.4
18.4
Ours
(1) Image
(2) GT Keypoints (3) Estimated Keypoints; GT Keyness Map (5) Estimated Keyness Map
Figure 4: Qualitative analysis of estimated keypoints and keyness. The col (2,3) show the GT and estimated keypoints, while col (4,5) show the GT and estimated keyness. The keyness is shown by colors, where the hue indicates the direction angle and the saturation denotes the magnitude.
between estimated keyness maps and GT keyness maps for both base classes and novel classes different splits. The averaged L2 distance before training is about 80 (i.e., the lower bound of performance), and can that the L2 distances of base classes of the full supervision. The 1.2 distances of novel classes without full supervision satisfactorily approximate the results of base classes, indicating that the keyness class-wise transferrable.
To further explore the transfcrability of keyness, we visualize the estimated keypoints and keyness map in Fig. @ in the intermediate training stage to whether the keyness could provide beneficial guidance. Generally, the estimated keyness maps are more approximate to their GT than the esti- mated keypoints. Therefore, some biased keypoints (e.g., the right eye of horse in the top row) could be guided to offset to the most key pixels, leading to precise keypoints of novel classes. Although some cases (e.g., the sofa in the 2-nd row) still imperfect, learning of multiple classes, which is quite challenging and has never been explored before. Overall, the transferred keyness maps are beneficial, and provide valid supervision to learn precise keypoints.
are the first to explore unsupervised
Analysis on correspondence transfer. The learned and transferred correspondences provide benefi- cial regularization for the unsupervised learning of novel classes. We evaluate their PCKs according to the GT keypoints on æ; and the GT keypoints corresponded from æ, to æ;, As shown in Tab. the PCKs of base classes are high enough due to full supervision. And the PCKs of novel classes are relatively satisfactory, indicating that the transferred correspondences can provide valid regular- ization.
Ablation study. study the contributions of our loss terms, we evaluate the combinations of our basic model, LNU.LNC and LNG- As shown in Tab. EX enabling the unsupervised loss LxU could improve the mAP dramatically, indicating the primitively gains of using unannotated images. Solely enabling the consistency loss LNC promotes the performance more significantly. Furthermore, incor-

---

Table
The performance of keyness (L2J) and correspondence (PCKst) evaluated for base classes and novel classes on five splits.
Table 5: Ablation results of our proposed frame- work on two splits.
MP-100 mAPt
Splitl L2]
MP-100
Split2
Method
Split
S1
Basic
Lsu
Lxc
LxG
MAPt
L2]
Base
47.2
42.2
44.8
60.2
22.4
51.5
29.0
Keyness
Novel
53.0
49.6
51.6
48.7
50.9
V
65.8
20.1
56.7
27.5
66.5
20.2
21.9
Base
89.1
94,0
93.3
94,8
93.5
67.7
19.5
58.2
27.3
Corespondence
Novel
73.5
74,8
69.8
69.9
71.2
Table 6:
The statistical analysis of our method against the strongest baseline (i.e., MetaPoint(II). We summarize the "mean std" of mAPf and report the p-values against the base- line.
(e) Our model may be indistinct localize the keypoints.
Dataset Split
MctaPoint [
Ours
P- P-Value
SI
31.5+0061
70.4+-0.53
0.0
S2
36.2+0.72
61.3=0.89
0.0
S3
30.110..3
60.910.78
0.0
S4
26.8+0.69
61.7+0.62
0.0
(1) langee
Eatnetsl Kerkos Map
S5
36.9=0.71
58.9
0.0
may
fin-grninodd keyness
Figure 5: The illustrations of two limitations.
porating LxG alone also demonstrates notable performance improvements. With all losses enabled, our method achieves the best results. Thus, our loss terms are effective and complementary.
4.6 Limitation Discussion
Our weak-shot keypoint estimation is prone to suffer from various issues due to insufficient anno- tations multiple novel classes. We summarize three major problems found in practice. Firstly, model is sometimes difficult precisely localize keypoints via argmax heatmaps, shown in Fig. 5S (a). For example, in the first sub-figure, there are two maximums, and thus we may need localize keypoints from heatmaps. Secondly,
estimated keyness map may fail to provide the directions to key directions, as shown in Fig. E (b). Our model is difficult to esti- mate precise fine-grained keyness map the chest area of the "long_sleeved" sleeved_outwear" probably because the GT keypoints are too dense and crowded. Thirdly, we method may suffer from the im- balance between the image numbers of base classes and novel classes. Nevertheless, are the first to explore such challenging cases, and we would like to address above limitations in future works.
better strategies
4.7 Significance Test
In this section, we statistically analyse our framework against the most competitive baseline Meta- Point [X]] on splits of MP-100 dataset using 3 labeled images per base class and 30 unlabeled images per novel class. With random seeds ranging from to 10, we run both methods for 10 times. Under the significance level 0.05, conduct the significance
to show that method superior than MetaPoint [ We summarize the "mean + std" of mAPst results and p-values in various dataset splits in Tab. R. where could see that the p-values are all far below the significance level 0.05. Therefore, this statistical analysis demonstrates that the improvements of our proposed framework is statistically significant.
Conclusion
In this paper, we have explored weak-shot keypoint estimation, where multiple novel classes are leamned from unlabeled images with the support of labeled base classes. We have proposed novel framework transferring keyness and correspondence to facilitate the unsupervised learning of novel classes. By transferring keyness and correspondence, our framework has achieved favourable per- formance for weak-shot keypoint estimation. We have conducted comprehensive experiments on MP-100 dataset to demonstrate effectiveness.
10

---

Acknowledgements
This work was supported in part by the National Key Research and Development Program of China under Grant FE0311000, in part by the National Natural Science Foundation of China under Grants 6244 1203, 6231 1530101, 62132006, 62471287, 62402201, 62161013, in part by the Natural Science Foundation of Jiangxi Province of China under Grants 20252BAC2401 197, in part by the China Postdoctoral Science Foundation under Grant 2025 1495, and in part by the Early-Career Young Scientists and Technologists Project of Jiangxi Province under Grant 20244BCE52070.
References
[1] Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele. 2d human pose estimation: New benchmark and state of the art analysis. In CVPR, pages 1886-3693, 2014.
[2] Tadas Baltrušaitis, Peter Robinson, and Louis-Philippe Morency. Openface: an open source facial behav- ior analysis toolkit. In WACV, pages 1-10, 2016.
[3] Vighnesh Birodkar, Zhichao Lu, Siyang Li, Vivek Rathod, and Jonathan Huang. The surprising impact of head architecture on novel class segmentation. In ICCV, 2021.
[4] Bjorn Browatzki and Christian Wallnaven. 3fabrec: Fast few-shot face alignment by reconstruction. In CVPR, pages 61 10-6120, 2020.
[5] Jinkun Cao, Hongyang Tang, Hao-Shu Fang, Xiaoyong Shen, Cewu Lu, and Yu- Yu-Wing Tai. Cross-domain adaptation animal pose estimation. ICCV, pages 9498-9507, 2019.
[6] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, pages 213-229, 2020.
[7] Junjie Chen, Li Niu, Liu Liu, and Liqing Zhang. Wcak-shot fine-grained classification via similarity transfer. In NeurlPS, 2021.
[8] Junjie Chen, Li Niu, Siyuan Zhou, Jianlou Si, Chen Qian, and Liqing Zhang. Weak-shot semantic seg- mentation via dual similarity transfer. In NeurlPS, pages 32525-32536, 2022.
Junjic Chen, Jiebin Yan, Yuming Fang, and Li Niu. Meta-point learning and refining for category-agnostic pose estimation. In CVPR, 2024.
[10] Yilun Chen, Zhicheng Wang, Yuxiang Peng, Zhiqiang Zhang, Gang Yu, and Jian Sun. Cascaded pyramid network for multi-person pose estimation. In CVPR, pages 1, 2018.
[11] Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. NeurlPS, 34:17864-178755 2021.
[12] MMPose Contributors. Openmmlab pose estimation toolbox and benchmark. https://github. com/open- mmlab/nmpose, 2020.
[13] Shanghua Gao. Zhong-Yu Yu Li, Ming-Ilsuan Yang, Ming-Ming Cheng, Junwei Han, and Philip Torr. Large- scale unsupervised semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelli- gence, 2022.
[14] Yuying Ge, Ruimao Zhang, and Ping Luo. Metacloth: Learning unseen tasks of dense fashion landmark detection from a few samples. IEEE Transactions on Image Processing, 31:1120- 1130, 2021.
[15] Zigang Geng, Ke Sun, Bin Xiao, Zhaoxiang Zhang, and Jingdong Wang. Bottom-up human pose estima- tion via disentangled keypoint regression. In CVPR, pages 14676-14686, 2021.
[16] Xingzhe He, Gaurav Bharaj. David Ferman, Helge Rhodin, and Pablo Garrido. Few-shot geometry keypoint localization. In CVPR, pages 21337-21343 2023.
[17] Xingzhe He, Bastian Wandt, and Helge Rhodin. Autolink: Self-supervised learning of human skeletons and object outlines by linking keypoints. In NewrlPS, pages 36123-300 331411,, 2022.
[18] Eric Hedlin, Gopal Sharma, Shweta Mahajan, Xingzhe He, Hossam Isack, Abhishek Kar Helge Rhodin, Andrea Tagliasacchi, and Kwang Moo Yi. Unsupervised keypoints from pretrained diffusion models. In CVPR, 2024.
11

---

[19] Or Hirschorn and Shai Avidan. Pose anything: A graph-based approach for category-agnosle pose esti- mation. In ECCV, 2024.
[20] Jian Hu, Jiayi Lin, Shaogang Gong, and Weitong Cai. Relax image-specitte prompt requirement in sam: A single generie prompt for segmenting camouflaged objects. In AAAI, 2024.
[21] Jian Hu, Jiayi Lin, Junchi Yan, and Shaogang Gong. Leveraging hallucinations to reduce manual prompt dependency in promptable segmentation. NeurlPS, 37, 2024.
[22] Ronghang Hu, Piotr Dollár, Kaiming He, Trevor Darrell, and Ross Girshick. Learning to segment every thing. In CVPR, 2018.
[23] Tomas Jakab, Ankush Gupta, Hakan Bilen, and Andrea Vedaldi. Unsupervised learning of object land- marks through conditional image generation. In NewrlPS, 2018.
[24] Sheng Jin, Wentao Liu, Wanli Ouyang, and Chen Qian. Multi-person articulated tracking with spatial and temporal embeddings. In CVPR, pages 5664 2613,2019.
[25] Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. ICLR, 2015.
[26] Weicheng Kuo, Anelia Angelova, Jitendra Malik, and Tsung-Yt Yi Lin. Shapemask: Learning to segment novel objects by refining shape priors. In ICCV, 2019.
[27] Rollyn Labuguen, Jumpei Matsumoto, Salvador Blanco Negrete. Hiroshi Nishimaru, Hisao Nishijo, Masahiko Takada, Yasuhiro Go, Ken-ichi Inoue, and Tomohiro Shibata. Macaquepose: novel in the wild macaque monkey pose dataset for markerless motion capture. Frontiers in behavioral neuroscience. 14:581 154, 2021.
[28] Jiefeng Li, Siyuan Bian, Ailing Zeng, Can Wang. Bo Pang, Wentao Liu, and Cewu Lu. Human pose regression with residual log- likeliiel estimation. ICCV, pages 11025-10044 2021.
[29] Jiefeng Li, Can Wang. Hao Zhu, Yihuan Mao, Hao-Shu Fang, and Cewu Lu. Crowdpose: Efficient crowded scenes pose estimation and a new benchmark. In CVPR, pages 10863-10872, 2019.
[30] Shuo Li, Fang Liu, Zehua Hao, Xinyi Wang, Lingling Li, Xu Liu, Puhua Chen, and Wenping Ma. Logits deconfusion with clip for few-shot learning. In CVPR, 2025.
[31] Shuo Li, Fang Liu, Zehua Hao, Kaibo Zhao, and Licheng Jiao. Unsupervised few-shot image classifica- tion by learning features into clustering space. In ECCV, 2022.
[32] Yan Li, Junge Zhang, Kaiqi Huang, and Jianguo Zhang. Mixed supervised object detection with robust objectness transfer. JEEE transactions pattern analysis and machine intelligence, 41(3):639-655 -533, 2018.
[33] Yanjie Li. Shoukui Zhang, Zhicheng Wang, Sen Yang, Wankou Yang. Sau-Tao Xia, and Erjin Zhou. Tokenpose: Learning keypoint tokens for human pose estimation. In CVPR, pages 11313-11322 2021.
[34] Yujia Liang, Zixuan Ye, Wenze Liu, and Hao Lu. Scape: A simple and strong category-agnosti pose estimator. In ECCV, pages 342-360, 2024.
[35] Zhaohe Liao, Jiangtong Li. Li Niu, and Liqing Zhang, Align and aggregate: Compositional reasoning with video alignment and answer aggregation for video auestionannswering. In CVPR, 2024.
[36] Zhaohe Liao, Jiangtong Li, Siyu Sun, Qingyang Liu, Fengshun Xiao, Tianjiao Li, Qiang Zhang, Guang Chen, Li Niu. Changjun Jiang, et al. Divide and conquer: Exploring language-centric tree reasoning for video question-answering In ICML., 2025.
[37] Huan Liu, Qiang Chen, Zichang Tan, Jiang-Jiang Liu, Jian Wang. Xiangho Su, Xiaolong Li, Kun Yao, Junyu Han, Errui Ding, et al. Group pose: A simple baseline for end-to-end multi-person pose estimation. In ICCV, pages 15029-15053 2023.
[38] Yan Liu, Zhijic Zhang, Li Niu, Junjic Chen, and Liqing Zhang. Mixed supervised object detection by transferring mask prior and semantic similarity. In NeurlPS, 2021.
[39] Dominik Lorenz, Leonard Bereska, Timo Milbich, and Bjom Ommer. Unsupervised bast- disentan- gling of object shape and appearance. In Proceedings of the IEEE/CVF conference computer vision and puttern recognition, pages 10965-10964, 2019.
[40] Changsheng Lu and Piotr Koniusz. Few-shot keypoint detection with uncertainty learning for unseen species. In CVPR, pages 19416-19426, 2022.
12

---

[41] Changsheng Lu and Piotr Koniusz. Detect any keypoints: An efficient light-weight few-shot keypoint detector. In AAAI, pages 3882-3890, 2024.
[42] Changsheng Lu, Zheyuan Liu, and Piotr Koniusz. Openkd: Opening prompt diversity for zero-and few- shot keypoint detection. In ECCV, 2024.
[43] Changsheng Lu, Hao Zhu, and Piotr Koniusz. From saliency to dino: Saliency- guided vision transformer for few-shot keypoint detection. arXiv preprint 2304v:2004003140, 2023.
[44] Grace Luo, Lisa Dunlap. Dong Huk Park, Aleksander Holynski, and Trevor Darrell. Diffusion hyperfea- tures: Searching through time and space for semantic correspondence. In NeurlPS, 2023.
[45] Weian Mao, Yongtao Ge, Chunhua Shen, Zhi Tian, Xinlong Wang, Zhibin Wang, and Anton van den Hengel. Poscur: Direct human pose regression with transformers. In ECCV, pages 72-88. Springer, 2022.
[46] Eric Marchand, Hidcaki Uchiyama, and Fabien Spindler. Pose estimation for augmented reality: a hands- survey. IEEE transactions on visualization and computer graphics, 22(12):2633 2651, 2015.
[47] Alejandro Newell, Kaiyu Yang, and Jía Deng. Stacked hourglass networks for human pose estimation. In ECCV, pages 483-49449 Springer, 2016.
[48] Khoi Due Nguyen, Chen Li, and Gim Hee Lee. Escape: Encoding super-keypoints for category-agnostic pose estimation. In CVPR, pages 23491-23500 2024.
[49] Xuecheng Nie, Jiashi Feng. Jianfeng Zhang, and Shuicheng Yan. Single-stage multi- pose ma- chines. In ICCV, pages
2019.
[50] Li Niu, Wen Li, and Dong Xu. Visual recognition by learning from web data: weakly supervised domain generalization approach. In CVPR, 2015.
[51] Li Niu, Ashok Veeraraghavan, and Ashutosh Sabharwal. Webly supervised learning meets zero-shot lcarning: A hybrid approach for fine-grained classification. In CVPR, 2018
[52] Duo Peng. Zhengbo Zhang, Ping Hu, Qiuhong Ke, David KY Yau, and Jun Liu. Harnessing text-to-image diffusion models for dategory-agnostir pose estimation. In ECCV, pages 342-360, 2024.
[53] N Dinesh Reddy, Minh Vo, and Srinivasa G Narasimhan. Carfusion: Combining point tracking and part detection for dynamic reconstruction of vehicles. CVPR, pages 1906-1915, 2018
[54] Pengfei Ren, Yuanyuan Gao, Haifeng Sun, Qi Qi. Jingyu Wang, and Jianxin Liao. Dynamic support information mining for antegory-agnostic pose estimation. In CVPR, pages 1921-1930 2024.
[55] Robin Rombach, Andreas Blattmann. Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pages 10684-10695, 2022.
[56] Matan Rusanovsky, Or Hirschomn, and Shai Avidan. Capex: Category-agnostic pose estimation from textual point explanation. arXiv preprint 24l::22006003884 2024.
[57] Luca Schmidtke, Athamasios Vlontzos, Simon Ellershaw, Anna Lukens, Tomoki Arichi, and Bernhard Kainz. Unsupervised human pose estimation through transforming shape templates. In CVPR, pages 2484-2494, 2021.
[58] Dahu Shi, Xing Wei, Liangqi Li, Ye Ren, and Wenming Tan. End-to-end multi-person pose estimation with transformers. In CVPR, pages
2022.
[59] Min Shi, Zihao Huang, Xianzheng Ma, Xiaowei Hu, and Zhiguo Cao. Matching is not enough: A two- stage framework for category-agnostic pose estimation. In CVPR, pages 7308-7317, 2023.
[60] Xibin Song, Peng Wang, Dingfu Zhou, Rui Zhu, Chenye Guan, Yuchao Dai, Hao Su, Hongdong Li, and Ruigang Yang. Apollocar3d: A large 3d car instance understanding benchmark for autonomous driving. In CVPR, pages 1452-5462, 2019.
[61] Jinghan Sun. Dong Wei, Liansheng Wang, and Yefeng Zheng. Lesion guided explainable few shot medical report generation. In International Conference on Medical Image Computing and Computer- Assisted Intervention, pages Springer, 2022.
13

---

[62] Meiqi Sun, Zhonghan Zhao. Wenhao Chai, Hanjun Luo. Shidong Cao, Yanting Zhang, Jenq-Neng Hwang. and Gaoang Wang. Uniap: Towards universal animal perception in vision via few-shot learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 5008-5016, 2024.
[63] Flood Sung. Yongxin Yang. Li Zhang. Tao Xiang, Philip HS Torr and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 119--1208, 2018.
[64] James Thewlis, Hakan Bilen, and Andrea Vedaldi. Unsupervised learning of object landmarks by factor- ized spatial embeddings. In ICCV, pages 5916-5925, 2017.
[65] Bo Wan, Desen Zhou, Yongfei Liu, Rongji Li, and Xuming He. Pose-aware multi-level feature network for human object interaction detection. In ICCV, pages 9469-9478, 2019.
[66] Ziteng Wang. Zizhao Zhang. Chen-Yu Lee, Han Zhang. Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Plister. Learning to prompt for continual learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 139-149, 2022.
[67] Zhen Wei, Bingkun Liu, Weinong Wang, and Yu-Wing Wing Tai. Few-shot model adaptation for customized fa- cial landmark detection, segmentation, stylization and shadow removal. arXiv preprint arXiv:2104.09457 2021.
[68] Bin Xiao, Haiping Wu, and Yichen Wei. Simple baselines for human pose estimation and tracking. In ECCV, pages 466-481, 2018. 2018.
[69] Lumin Xu, Sheng Jin, Wang Zeng, Wentao Liu, Chen Qian, Wanli Ouyang, Ping Luo, and Xiaogang Wang. Pose for everything: Towards agggsor pose estimation. In ECCV, pages 398-416, 2022.
[70] Yufei Xu, Jing Zhang, Qiming Zhang, and Dacheng Tao. Vitpose: Simple vision transformer baselines for human pose estimation. NeurIPS, 35:38571-3888 2022.
[71] Yang, Ailing Zeng, Ruimao Zhang, and Lei Zhang. X-pose: Detecting any keypoints. In European Conference on Computer Vision, pages 249 268. Springer, 2025.
[72] Yang, Wang Zeng, Sheng Jin, Lumin Xu, Wentao Liu, Chen Qian, and Ruimao Zhang. Kptllm: Un- veiling the power of large language model for keypoint comprehension. arXiv preprint axXiv:2111.01844 2024.
[73] Yi Yang and Deva Ramanan, Articulated human detection with flexible mixtures of parts. IEEE transac- tions on pattern analysis and machine intelligence, 35(12):2888-2890 2012.
[74] Wang Zeng, Sheng Jin, Wentao Liu, Chen Qian, Ping Luo, Wanli Ouyang. and Xiaogang Wang. Not all tokens are equal: Human-centric visual analysis via token clustering transformer. In CVPR, pages 11101-11111, 2022.
[75] Feng Zhang, Xiatian Zhu, Hanbin Dai, Mao Ye, and Ce Zhu. Distribution-aware coordinate representation for human pose estimation. In CVPR, pages 7093-7102, 2020.
[76] Hao Zhang, Lumin Xu, Shenqi Lai, Wengi Shao, Nanning Zheng, Ping Luo, Yu Qiao, and Kaipeng Zhang open-vocabulary animal keypoint detection with semantic-feame matching. International Journal of Computer Vision, pages 1-18, 2024.
[77] Hao Zhang, Kaipeng Zhang, Lumin Xu, Shenqi Lai, Wenqi Shao, Naning Zheng, Ping Luo, and Yu Qiao. Language-driven open-vocabulary keypoint detection for animal body and face. arXiv preprint arXiv:2310.05056, 2023.
[78] Xu Zhang, Wen Wang. Zhe Chen, Yufei Xu, Jing Zhang, and Dacheng Tao. Clamp: Prompt-based con- trastive learning for connecting language and animal pose. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 23272-23281, 2023.
[79] Yuting Zhang, Yijie Guo, Yixin Jin, Yijun Luo, Zhiyuan He, and Honglak Lee. Unsupervised discovery of object landmarks as structural representations. In Proceedings of the IEFF Conference on Computer Vision and Pattern Recognition, pages 2694-2703, 2018.
[80] Yuanyi Zhong, Jianfeng Wang, Jian Peng, and Lei Zhang. Boosting weakly supervised object detection with progressive knowledge transfer. In ECCV, 2020.
[81] Siyuan Zhou, Li Niu, Jianlou Si, Chen Qian, and Liqing Zhang. Weak-shot semantic segmentation by transferring semantic affinity and boundary. BMVC, 2022.
14

---

NeurIPS Paper Checklist
The checklist designed to encourage best practices for responsible machine learning research, ad- dressing issues of reproducibility, transparency. research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (12 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evalu- is perfectly acceptable answer "[No] provided proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased binary way, we acknowledge that the true amswer is often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to question, in the justification please point to the section(s) where related material for the question can be found.
ation. While "[Yes] " generally preferable to "[No]
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Check- list".
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our abstract makes the main claims and our introduction summarize con- tributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
fine to include aspirational goals motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
15

---

Justification:
We discuss the limitations in appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach ad- dress problems of privacy and faireess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: We describe our method clearly and fully in the main text, and we conduct experiment using public dataset. We will release the code after the acceptance of paper.
Guidelines:
16

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language modell, releasing of model checkpoint, other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We conduct experiment using public dataset, and will release code after the acceptance of paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so No an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
//mips.cc/publi///uuii
isgionolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
17

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We specify all the training and test details in the main text and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We conduct significance test in appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors verified.
For asymmetric distributions, the authors should careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the details of computer resources in appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
18

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https://neuri:: httpp////..aars.c.ccc//////taaa/
Answer: [Yes]
Justification: The research conducted in our paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the potential positive societal impacts and negative societal im- pacts in appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
19

---

Answer:
[NA]
Justification: We conduct experiment using public dataset, and will release model after the accpetance of paper.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere usage guidelines or restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes)
Justification:
We cite all the original papers that produced the code package or dataset.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should provided.
assets are released, the license, copyright information, and terms use in the pack- age should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [NA]
Justification: We conduct experiment using public dataset, and will release code after the accpetance of paper.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
20

---

14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information the supplemental material is fine, but the main contri- bution of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, cura- tion, or other labor should paid at least the minimum wage the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper deseribe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: Our core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.M policy [cTe:s///eeerppppccc////oeerreee for what should or should not be described.
21


Reviews:
[{"review_id": "oQecaxclGY", "reviewer": "Reviewer_iFGi", "summary": "This paper proposes a new task, named weak-shot pose estimation, and designs a framework to transfer keyness and correspondence from base classes to novel classes for the focused task. To address the lack of annotations for novel classes,  an offset guiding loss and a consistency loss are introduced to guide the learning of heatmap, keyness, and correspondence. Experiments are conducted on MP-100 benchmark to demonstrate the effectiveness of the proposed method.", "questions": "- What's the meaning of K In Eq. (3) and Eq. (5)? Why not M? Is K the exact number of keypoints?\n\n- No illustrations of $S$ and $w_{l,s}$ in Line141.\n\n- In Table 5, does the “Basic” result in the first row refer to unsupervised setting? It still outperforms other methods under the weak-shot setting by a large margin. If so, what advantages does the proposed framework have over existing approaches?\n\n- Typo: Line133 \"\\Phi_^c_l () and \\Phi_^c_l () $ as the c-th ...\"; Line183 \"detach and block\" -> \"detaches and blocks\"", "limitations": "Yes.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "##  Strength\n\n- The paper proposes the new task of weak-shot pose estimation, and designs a framework to transfer keyness and correspondence from base classes to novel classes.\n\n- Self-supervised losses on novel classes, e.g., offset guiding loss and consistency loss, are introduced to guide the learning of  heatmap, keyness, and correspondence.\n\n- The method significantly outperforms the established baselines by a large margin on MP-100 benchmark.\n\n- The paper is well-organized.\n\n\n## Weakness\n\n- How is the keyness map $M$ used to identify the corresponding keypoint for each pixel?\n\n- For the base classes, how is the visibilities $V$ supervised?\n\n- The description of the offset-guiding loss (Lines 179–184) is somewhat unclear.  How is the visibilites $V$ obtained via the argmax operation? Is there any thresholding involved? How exactly is Eq. (5) used to supervise visibility? Does $M_i^*[P_i[k]]$ refer to the keyness at pixel $P_i[k]$? What is the precise operation of dt()? How does the gradient propagate through $P$ given the use of argmax operations?", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer’s positive feedback and constructive suggestions. Below, we provide point-by-point responses to address each concern raised.\n\n---\n\n> **Q1:** How is the keyness map $M$ used to identify the corresponding keypoint for each pixel?\n\n**A1:**  Actually, the keyness map $M\\in \\mathbb{R}^{2\\times H\\times W}$ presents the directions towards more key positions, and we employ it to guide the estimated keypoints to move to more key coordinates. For example, given coordinates $p=[x,y]$, the 2D offset vector $M[p]=[\\Delta x,\\Delta y]$ fetched from the keyness map $M$ indicates the more key direction. Therefore, the primary role of keyness map $M$ is to guide the keypoint coordinates, rather than identifying the corresponding keypoint for each pixel.\n\n---\n\n> **Q2:** For the base classes, how is the visibilities $V$ supervised?\n\n\n**A2:** In our model, the visibilities $V$ are the max values fetched from heatmaps (Line 179-180), and thus are indirectly supervised by the loss on heatmaps (i.e., $\\mathcal{L}_{BP}$ at Eq. (3)). For the base classes, the matched heatmaps are supervised to approaching the GT keypoint Gaussian maps (max=visibility=1), while the unmatched heatmaps are supervised to approaching all-zero maps (max=visibility=0). Considering that the supervision on unmatched estimations is trivial and similar to [6,11,9], we omit it for brevity. Thanks for the considerate comments, and we will add more explanations in our revision.\n\n---\n\n> **Q3:** The description of the offset-guiding loss (Lines 179–184) is somewhat unclear. ① How is the visibilites $V$ obtained via the argmax operation? ② Is there any thresholding involved? ③How exactly is Eq. (5) used to supervise visibility? ④Does $M_i^*[P_i[k]]$ refer to the keyness at pixel $P_i[k]$? ⑤What is the precise operation of dt()? ⑥How does the gradient propagate through $P$ given the use of argmax operations?\n\n**A3:** **The answers to ①②④⑥**: Actually, we employ \"soft\" argmax operation to obtain coordinates and employ max operation to obtain visibilities, as stated in Line 179-180. \n\nSpecifically, given one keypoint heatmap $H$, we compute the keypoint coordinates $\\tilde{\\mathbf{p}}\\in \\mathbb{R} ^{2}$ by following \"soft\" argmax: $\\tilde{\\mathbf{p}}= \\sum_{\\mathbf{p}} w(\\mathbf{p}) \\mathbf{p}$, where   $w(\\mathbf{p}) = \\frac{\\exp(H[\\mathbf{p}])}{\\sum_{\\mathbf{p}} \\exp(H[\\mathbf{p}])}.$ Here, the index $\\mathbf{p}$ denotes the normalized pixel coordinates (all spatial grid points), and $H[\\mathbf{p}]$ denotes fetching the value at $\\mathbf{p}$ from $H$. Similarly, $M_i^*[P_i[k]]$ refers to the keyness at pixel $P_i[k]$. \n\nFor keypoint visibility, we compute by $v_n = \\underset{\\mathbf{p}}{\\mathrm{max}} H[\\mathbf{p}]$. We do not involve thresholding for computing visibility, because soft values are compatible for supervision and evaluation. The above \"soft\" argmax operation is differentiable, and identical operations could be found in [18,57].\n\n**The answers to ③**: As also stated in our Abstract and Introduction, the primary role of Eq. (5) is providing direct guidance for keypoint coordinates. We sincerely apologize for the rigor of expression of \"and visibilities\" at Line 181, and we will remove it in the final version.\n\n\n**The answer to ⑤**: The precise operation of dt() is torch.detach().\n\n---\n\n> **Q4:** What's the meaning of $K$ In Eq. (3) and Eq. (5)? Why not $M$? Is $K$ the exact number of keypoints?\n\n**A4:** As described in Line 118-120, $K$ is the exact number of keypoints (including padded and invisible keypoints). Besides, we do not employ $M$ to represent a certain number, but employ $N$ to represent the number of keypoint prompts and thus estimate $N$ keypoint heatmaps for each image.\n\nThe loss term in Eq. (3) supervises on $K$ matched heatmaps, because there are $K$ GT keypoints for matching. As for Eq. (5), the loss term should average over $N$, i.e., $\\frac{1}{N}\\sum_k^N$, because the guidance is applicable for all estimated coordinates, which can also be inferred from the dimensional notation $N$ at Line 180. We sincerely apologize for this symbol mistake, and will revise in the final version.\n\n---\n\n\n> **Q5:** No illustrations of $S$ and $w_{l,s}$ in Line141.\n\n**A5:** Specifically, $S$ denotes the number of subsampled timesteps selected from the diffusion timesteps, and $w_{l,s}$ denotes the mixing weights for the $l$-th layer at the $s$-th timestep. We will add the illustrations in our final revision.\n\n\n---\n\n> **Q6:** In Table 5, does the “Basic” result in the first row refer to unsupervised setting? It still outperforms other methods under the weak-shot setting by a large margin. If so, what advantages does the proposed framework have over existing approaches?\n\n**A6:** In the ablation study presented in Table 5, the default learning paradigm is weak-shot learning, with various loss terms activated. Thus, our basic model (the first row in Table 5) is not in unsupervised setting.\nSpecifically, our basic model disables the loss terms for novel classes (i.e., $\\mathcal{L}\\_{NU}$,$\\mathcal{L}\\_{NC}$,$\\mathcal{L}\\_{NG}$), and learns from base classes and directly estimates for novel classes. Existing unsupervised approaches are designed for single-class scenario, and thus cannot: (1) well tackle different keypoint numbers of multiple classes; (2) well transfer across classes. Our basic model is trained with matching-based loss ( $\\mathcal{L}\\_{BP}$ at Eq. (3)). In this way, different number of keypoints could adaptively and dynamically match to keypoint prompts, and thus our basic model can well tackle multiple classes. Besides, the keypoints of novel classes and base classes may share the same keypoint prompts, and thus our prompts learned from base classes can also estimate precise keypoints for novel classes. Other loss terms (i.e., $\\mathcal{L}\\_{NU}$,$\\mathcal{L}\\_{NC}$,$\\mathcal{L}\\_{NG}$) can further improve.\n\n---\n\n\n> **Q7:** Typo: Line133 \"\\Phi_^c_l () and \\Phi_^c_l () $ as the c-th ...\"; Line183 \"detach and block\" -> \"detaches and blocks\"\n\n**A7:** Sincerely thanks for the considerate comments, and we will correct the typos and revise the paper more carefully."}, {"review_id": "HWI66V2Kiw", "reviewer": "Reviewer_YayZ", "summary": "The paper proposes a weak-shot keypoint detection approach, which aims to detect semantic-rich keypoints on new classes without labeled training data.\nThe method is first trained on labeled data and then reuses the pretrained prompt tokens to detect keyness maps on new classes. Then, in new classes, the keyness map is used to guide the pixel movement to high-intensity change regions, where the semantic correspondences across images are used to provide consistency loss. Experiments are conducted on the MP-100 dataset.", "questions": "(1) Can authors provide a clearer definition of \"new class\"?\n\n(2) Given that the prompt tokens are pretrained exclusively on the base categories, could using them during the unlabeled training phase for novel categories inadvertently introduce bias?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n(1) The proposed keyness constraint and correspondence constraint on the new class are reasonable and seem effective\n\n(2) Experiments and analysis are well conducted. The paper provides comprehensive ablation studies to validate the effectiveness of the proposed method.\n\n# Weaknesses\n\n(1) Definition of \"class\". It is not clear to me whether new class means new category (cat to dog) or new instance (cat 1 to cat 2). The paper seems to focus on a new instance. Otherwise, the reuse of the pretrained prompt tokens on different categories may introduce severe bias towards the new category.\n\n(2) Task definition. The paper seems to estimate a set of keypoints that can be matched across different instances. However, the reviewer thinks this work may not belong to the pose estimation task, which aims to detect the predefined set of semantic keypoints, and can benefit the downstream tasks like action recognition, skeleton-driven animation, etc. \nIf the reviewer does not misunderstand, the detected keypoints on the new class in this work are only \"semantic-rich\", but don't have specific (known) semantic meaning. \nTherefore, it should be a semantic keypoint detection work instead of a pose estimation work.", "was_revised": true, "final_justification": "After reading other reviewers' comments and discussions, the reviewer decided to maintain the original positive score.", "rebuttal": "We sincerely thank the reviewer for the positive evaluation of our work and for acknowledging the insights of our method. Below, we provide detailed responses to address each of the concerns raised.\n\n---\n\n> **Q1:** Definition of \"class\". It is not clear to me whether new class means new category (cat to dog) or new instance (cat 1 to cat 2). The paper seems to focus on a new instance. Otherwise, the reuse of the pretrained prompt tokens on different categories may introduce severe bias towards the new category. \n\n> Can authors provide a clearer definition of \"new class\"?\n\n**A1:** We will add explicit definition, i.e., new class means new category, which follows the definition in related works [71,57,9]. Exactly, the cross-category bias is one of the major issues to address in our task. We will further explain our contributions to this issue in the following Q3-A3.\n\n---\n\n\n> **Q2:** Task definition. The paper seems to estimate a set of keypoints that can be matched across different instances. However, the reviewer thinks this work may not belong to the pose estimation task, which aims to detect the predefined set of semantic keypoints, and can benefit the downstream tasks like action recognition, skeleton-driven animation, etc. If the reviewer does not misunderstand, the detected keypoints on the new class in this work are only \"semantic-rich\", but don't have specific (known) semantic meaning. Therefore, it should be a semantic keypoint detection work instead of a pose estimation work.\n\n\n**A2:** We intuitively follow the related works Category-Agnostic Pose Estimation [71,57,9] to use the term of \"Pose Estimation\", and we will revise our task name to more precise one, i.e., Keypoint Detection or Keypoint Estimation. \n\nAs for the downstream tasks, our estimated keypoints have similar properties as the ones in unsupervised keypoint detection [21,19,83], which can also benefit the downstream tasks such as action recognition. In particular, optical flow is semantic-free, but is beneficial for action recognition. Our estimated keypoints may be regarded as sparse optical flow, and the keypoint trajectories across video frames could depict the motion pattern. Besides, as in the regression-based evaluation [21,19,83] described in Line 237-241, the estimated keypoints can be regressed to known semantic meaning. \n\nDiscovering keypoints with known semantic meaning is an interesting yet challenging problem, and we will explore more in future works. Our primary focus in this paper is on discovering keypoints for multiple classes, which also has received relatively limited exploration.\n\n---\n\n\n> **Q3:** Given that the prompt tokens are pretrained exclusively on the base categories, could using them during the unlabeled training phase for novel categories inadvertently introduce bias?\n\n**A3:** Indeed, and addressing this cross-category domain gap (bias) is one of the major issues. Preliminarily, reusing the prompt tokens for novel categories is effective due to the keypoint sharing between base categories and novel categories. For example, \"bird\" and \"lion\" both have eye keypoints. Moreover, we propose to employ transferred keyness and correspondence to further bridge the domain gap (bias). As explained in Line 41-46, both keyness and correspondence belong to comparing entities(or relations), which are highly transferable. By above designs, our model outperforms existing methods dramatically in learning keypoints for multiple classes without labels."}, {"review_id": "uHFB6u6b7E", "reviewer": "Reviewer_zjHs", "summary": "This paper proposes a weak shot pose estimation, where multiple novel classes are learned from unlabeled images from the labeled base classes.\n\nThe proposed method leverages keyness (gradients of the keypoint that indicate the direction to a more important key location) and correspondence from the base class by transferring them to a novel class to form the unsupervised learning of the novel classes. Keyness and correspondences are estimated from the diffusion model. Novel classes are trained from offset guiding loss, consistency loss, and localization loss, while the base class is trained with keypoint loss, keyness loss, and correspondence loss.\n\nThe paper shows an experiment comparing the results on the baselines with similar setups as well as a fully supervised setup, and shows that the proposed method outperforms the baseline with the similar setup, while performs comparable results on the fully supervised setup.", "questions": "- Found one typo in line 130. I assume it's \"single-class\" instead of \"sing-class\"?\n- Estimating keypoints is not necessarily a pose estimation. It would be great to avoid using the term \"pose estimation\" too many times in the paper.\n- It would be great to have actual name on each loss term when they are introduced, for example in line 153: correspondence via XXX, XXX, XXX loss in base class (L_BP, L_BK, and L_BC), and line 154: keypoints via XXX, XXX, XXX loss in novel class (L_LG, L_NC, and L_NU)", "limitations": "Yes", "rating": 5, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "1. Strength:\n\n- The paper is not my field, but the proposed method covers a less explored aspect of keypoint estimation, and technically sound & the experiment results look reasonable. I don't find any noticeable errors in the paper that need to be fixed. Also, the proposed method has great potential to be used in other computer vision fields.\n\n2. Weakness:\n\n- It's possible that this issue is coming from a terminology difference between different fields, but as a person from the 6D pose estimation field, the term \"Pose Estimation\" is somewhat misleading. A better term to be used is \"Keypoint Estimation\". The paper is mainly focused on keypoint estimation, and no where to find actual \"pose\" estimation.", "was_revised": true, "final_justification": "Therer weren't any major reason to change the final rating", "rebuttal": "We sincerely thank the reviewer for the insightful comments and constructive feedback. Below, we address each concern and question raised.\n\n---\n\n> **Q1:** It's possible that this issue is coming from a terminology difference between different fields, but as a person from the 6D pose estimation field, the term \"Pose Estimation\" is somewhat misleading. A better term to be used is \"Keypoint Estimation\". The paper is mainly focused on keypoint estimation, and no where to find actual \"pose\" estimation.\n\n**A1:** We intuitively follow the related works Category-Agnostic Pose Estimation [71,57,9] to use the term of \"Pose Estimation\", and we will revise it to \"Keypoint Estimation\" for better understanding.\n\n---\n\n\n> **Q2:** Found one typo in line 130. I assume it's \"single-class\" instead of \"sing-class\"?\n\n**A2:**  Sincerely thanks for the meticulous comments, and we will correct the typo and revise the paper more carefully.\n\n---\n\n\n> **Q3:** Estimating keypoints is not necessarily a pose estimation. It would be great to avoid using the term \"pose estimation\" too many times in the paper.\n\n**A3:** Thanks for the insightful comments, and we will employ the precise term, i.e., keypoint estimation.\n\n---\n\n\n> **Q4:** It would be great to have actual name on each loss term when they are introduced, for example in line 153: correspondence via XXX, XXX, XXX loss in base class ($\\mathcal{L}\\_{BP}$, $\\mathcal{L}\\_{BK}$, and $\\mathcal{L}\\_{BC}$), and line 154: keypoints via XXX, XXX, XXX loss in novel class ($\\mathcal{L}\\_{LG}$, $\\mathcal{L}\\_{NC}$, and $\\mathcal{L}_{NU}$)\n\n**A4:** Thanks for the beneficial comments, and we will add the actual name in the descriptions to improve the readability."}, {"review_id": "X6WwBbymoR", "reviewer": "Reviewer_4nj5", "summary": "This paper introduces the concept of weak-shot learning for pose estimation, wherein the model is trained using labeled examples from base classes and unlabeled examples from novel classes. The proposed method processes pairs of images belonging to the same class, encoding them through a shared encoder and a latent diffusion model to estimate keypoints. Additionally, the approach generates keyness maps, indicating the direction toward the next keypoint, and establish correspondences between the images in each pair. For labeled images from base classes, a supervised loss is employed, while an unsupervised loss is proposed for unlabeled images from novel classes. The effectiveness of the method is demonstrated on various splits of the MP-100 dataset, with comparisons made against both unsupervised and few-shot learning approaches.", "questions": "Table 1 in the supplementary material indicates that shuffling the keypoint order significantly degrades the performance of Hedlin et al., despite this method being unsupervised. What factors might account for this observation? \n\nHow computationally expensive is the computation of the correspondence map, considering that the approach requires calculating the cosine similarity between every pair of pixels? \n\nIs the \"reference image\" truly a reference in the conventional sense? According to my understanding, it simply refers to another image belonging to the same class. If so, this terminology may be misleading.In Equation 5, M_i* does not denote ground truth, whereas the same symbol is used to represent the ground-truth keyness map in Equation 4. This inconsistency could lead to reader confusion.", "limitations": "N/A", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths: \n\nThe authors are first to apply the concept of weak shot learning on pose estimation, which is of high relevance due to the sparsity of labelled images in this field. \n\nThe method outperforms existing unsupervised and few-shot approaches by a large margin under the given data availability. The method even outperforms few-shot methods that have additional labels of novel classes. \n\nThe paper shows evidence that the addition of the keyness maps substantially enhances the performance in comparison to prior works. \n\nWeaknesses: \n\nMissing details on evaluation.   \n\nTo enable fair comparisons between methods, it is necessary to specify the number of labeled/unlabeled training samples and classes used for each method.  \n\nAdditionally, the authors should elaborate on the degree of similarity between the base and novel classes; while Appendix A4 provides some information, this discussion should be extended and included in the main text. \n\nThe overall clarity of the paper can be improved due to missing details and misleading terminology (see below).", "was_revised": true, "final_justification": "My concerns have been clarified, and I maintain the score.", "rebuttal": "We sincerely thank the reviewer for the positive feedback and constructive suggestions. Below, we provide detailed responses to address each of the concerns raised.\n\n---\n\n> **Q1:** To enable fair comparisons between methods, it is necessary to specify the number of labeled/unlabeled training samples and classes used for each method.\n\n**A1:** Below, we firstly specify the number of base/novel classes, and then specify the number of labeled/unlabeled training samples.\n\nThe number of classes is determined by the dataset split. \nFor the standard splits (i.e., MP-100 Split1-Split5), we follow [71,57,9] and employ 70/20 for base/novel classes, as introduced in Line 212-214. For the cross super-class splits in Table 4 of Appendix, we follow [71,57,9] and employ 99/1, 98/2, 95/5 and 97/3 for base/novel classes in Body, Face, Fur. and Veh. splits. For the Novel Bird split in Table 3, we employ 92/8 for base/novel classes.\n\nThe numbers of samples and labels for base/novel classes are determined by the learning paradigm. Specifically, all methods in weak-shot learning (e.g., all methods in Table 1 and Ours in Table 2 and Table 3) employ 3/30 labeled/unlabeled training samples for each base/novel class, as introduced in Line 214-217. For the few-shot methods (with 1-shot) in Table 2, we employ 3/1 labeled/labeled training samples for each base/novel class. For the unsupervised methods in Table 3, we employ 30 unlabeled training samples for each novel class.\n\nThanks again for the beneficial comments, and we will add the specifications in our revision.\n\n---\n\n> **Q2:** Additionally, the authors should elaborate on the degree of similarity between the base and novel classes; while Appendix A4 provides some information, this discussion should be extended and included in the main text.\n\n**A2:** We measure the similarity via the GT keypoints and a ResNet-50 backbone pre-trained on ImageNet. Specifically, we firstly extract an image feature vector for each image, by averaging the feature vectors of GT keypoints on the backbone feature map. Then, we extract a class feature vector, by averaging the image feature vectors belonging to the same class. After that, for each novel class, we compute its similarity to base classes by averaging its similarities to all base classes using class feature vectors. Finally, we obtain the similarity from novel classes to base classes by averaging the similarities of all novel classes. \n\nIn the table below, we summarize the similarities for all dataset splits. We could see that, the similarities of standard splits (i.e., Split1-Split5) are robust and intermediate due to random splitting. The similarities of cross super-class splits are significantly lower than the standard splits. \n\n\n|         | Split1 | Split2 | Split3 | Split4 | Split5 | Body   | Face   | Vehicle | Furniture |\n| ------------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------- | --------- |\n| **Cosine Similarity**  | 0.65 | 0.64 | 0.65 | 0.65 | 0.63 | 0.62| 0.61 | 0.54  |  0.59  |\n\n\n> **Q3:** Table 1 in the supplementary material indicates that shuffling the keypoint order significantly degrades the performance of Hedlin et al., despite this method being unsupervised. What factors might account for this observation?\n\n**A3:** Firstly, we adapt related methods (e.g., Hedlin et al. [21]) into weak-shot learning paradigm for fair comparison, as stated in Line 243-251. Secondly, our default learning paradigm is weak-shot learning, including the experiments in Table 1 of Supplementary. Therefore, in Table 1 of Supplementary, Hedlin et al. is an adapted weak-shot method, which significantly degrades due to overfitting unpractical relation, as explained in Line 218-223. The comparison to original unsupervised methods (e.g., Hedlin et al. [21]) is presented in Table 3 of main text, which are inferior due to the various keypoint numbers of multiple classes. Thanks again, and we will introduce distinguishing markers to differentiate the original method from its adapted version.\n\n---\n\n> **Q4:** How computationally expensive is the computation of the correspondence map, considering that the approach requires calculating the cosine similarity between every pair of pixels?\n\n**A4:** Overall, we follow the implementation of [44] to compute correspondences. In practice, we only need to compute the similarity between all keypoints on image1 and all pixels on image2. In this way, the computation of such correspondence is relatively efficient and affordable, i.e., 0.66 ms and 0.37 GFLOPs. Specifically, the implementations with tensor shapes annotated are shown below:\n```\n\nimg1_feature_map = normalize_feats(img1_feature_map) # shape: (B, K, C)\nimg2_feature_map = normalize_feats(img2_feature_map) # shape: (B, W*H, C)\ncorrespondence_map = torch.matmul(img1_feature_map, img2_feature_map.permute((0, 2, 1))) # shape: (B, K, W*H)\n\ndef normalize_feats(feats):\n    feats = feats / torch.linalg.norm(feats, dim=-1)[:, :, None]\n    return feats\n```\nSpecifically, for the time cost, the spent time is 0.66 ms, which is averaged over 500 training iterations. For the theoretical cost, the GFLOPs is 0.37, which is measured by the fvcore package through aggregating all floating-point operations of feature normalization to unit vectors and matrix multiplication.\n\nThanks again for the beneficial comments, and we will add more specifications in our revision.\n\n---\n\n> **Q5:** Is the \"reference image\" truly a reference in the conventional sense? According to my understanding, it simply refers to another image belonging to the same class. If so, this terminology may be misleading. In Equation 5, $M_i^*$ does not denote ground truth, whereas the same symbol is used to represent the ground-truth keyness map in Equation 4. This inconsistency could lead to reader confusion.\n\n**A5:** Our \"reference image\" just means another image belonging to the same class, and we may employ \"paired image\" for better understanding. In Equation 5, we regard the transferred keyness map as pseudo ground truth, and we will remove the marker * for better consistency and readability."}]

Author rebuttal:

Meta-review: This paper proposes a weak-shot pose estimation framework that transfers knowledge from base classes to novel classes through keyness and correspondence. The method demonstrates innovation, and the experiments and analyses are thorough, showing improvements over SOTA approaches.

​​Strengths:​​
1. The method is innovative and addresses a practically relevant problem.
2. The experimental results and analyses are comprehensive, validating the effectiveness of the approach.

​​Weaknesses:​​
1. As noted by the reviewers, further elaboration is needed on certain technical details, conceptual definitions, and experimental specifics.
2. Limitations and failure cases could be included in the main text to enhance structural completeness.

During the rebuttal phase, reviewers acknowledged the method's novelty and experimental validity, with discussions primarily focusing on clarifications regarding technical details and experiments (Reviewers iFGi, 4nj5, zjHs). Additionally, Reviewers YayZ and zjHs suggested refining the terminology, particularly replacing "pose estimation" with "keypoint estimation." The authors provided detailed responses in the rebuttal, committing to supplementing explanations in the paper, including technical details and experiments, as well as considering the terminology adjustment. The reviewers expressed satisfaction with the rebuttal, with final recommendations consisting of 3 ​​Borderline Accept​​ and 1 ​​Accept​​. The AC concurred with the reviewers' suggestions.
Decision: Accept (poster)