CapeX: Category-Agnostic Pose Estimation from Textual Point Explanation

## Abstract

Abstract Conventional 2D pose estimation models are constrained by their design to specific object categories. This limits their applicability to predefined objects. To overcome these limitations, category-agnostic pose estimation (CAPE) emerged as a solution. CAPE aims to facilitate keypoint localization for diverse object categories using a unified model, which can generalize from minimal annotated support images.
Recent CAPE works have produced object poses based on arbitrary keypoint definitions annotated on a user-provided support image. Our work departs from conventional CAPE methods, which require a support image, by adopting a text-based approach instead of the support image.
Specifically, we use a pose-graph, where nodes represent keypoints that are described with text. This representation takes advantage of the abstraction of text descriptions and the structure imposed by the graph.
Our approach effectively breaks symmetry, preserves structure, and improves occlusion handling.
We validate our novel approach using the MP-100 benchmark, a comprehensive dataset spanning over 100 categories and 18,000 images. Under a 1-shot setting, our solution achieves a notable performance boost of 1.07%, establishing a new state-of-the-art for CAPE. Additionally, we enrich the dataset by providing text description annotations, further enhancing its utility for future research.

## 1 Introduction

Pose estimation deals with the prediction of semantic parts’ positions within objects depicted in images, a task crucial for applications like zoology, autonomous driving, virtual reality, and robotics [].
Previous pose estimation methods were typically constrained by their reliance on category-specific datasets for training. Consequently, when confronted with novel objects, these methods often exhibit limited efficacy due to their lack of adaptability.

To address this challenge, recent research has introduced category-agnostic pose estimation (CAPE) [], a paradigm capable of localizing semantic parts across diverse object categories, based on a single or few support examples. All previous CAPE works require a small set of support images annotated with the keypoints of interest. These support images are used in order to find the best spatial arrangement of the keypoints in the query image, based on latent visual correspondence to the annotated support keypoints.

This raises two challenges. First, the need to provide annotated support image(s) is cumbersome. Second, relying solely on visual correspondence between keypoints in different images, even from the same category, may lead to suboptimal results. This is because no two distinct images share parts with the exact same appearance. Still, both images should share parts with the same semantic meaning. For example, all cats have a head, legs, and a tail, but they never look the same. This idea is even more crucial when the objective is to estimate poses of objects in images from novel categories (i.e., dogs), as in CAPE.

We cope with these limitations by adopting a holistic approach to pose estimation, based on a support graph as input, with open-vocabulary textual descriptions on its nodes. No support images are needed. Instead of exclusively relying on visual support data, we leverage the abstraction power of textual data. This comprehensive view enables us to match the query keypoints’ appearance to the textual description of the support keypoints, eliminating the need for support images altogether.
Furthermore, following Pose Anything [], instead of treating the input keypoints as isolated entities, we treat them as structure-aware connected nodes of a graph.
By doing so, we effectively leverage the inherent relationships and dependencies between keypoints, enhancing the overall performance, breaking symmetry, preserving structure, and better handling occlusions. Figure 1 demonstrates our approach.

To evaluate the efficacy of our proposed method, we utilize the extended version [] of the CAPE benchmark, MP-100 []. This dataset consists of more than 18,000 images spanning 100 categories, encompassing diverse subjects such as animals, vehicles, furniture, and clothes.
As some of the categories miss the keypoints’ text descriptions, we collected and unified the text descriptions of the keypoints in all categories.
Our method is evaluated against previous CAPE methodologies. Notably, our approach surpasses the performance of existing methods, showcasing a new state-of-the-art performance under the 1-shot setting.

In summary, our contributions can be outlined as follows:

- •
We propose modeling the support keypoints using connected graph nodes coupled with text descriptions as opposed to previous methods that rely on visual signals. This methodology matches the support to the query keypoints, thanks to the abstraction power of text and graphs. Furthermore, this approach does not require support images for either training or inference.
- •
We provide an enhanced version of the MP-100 dataset with textual annotations for the keypoints in all categories, enriching the benchmarking capabilities for category-agnostic pose estimation.
- •
We establish new benchmarks in category-agnostic pose estimation, showcasing state-of-the-art performance on the MP-100 dataset, without finetuning the support feature extraction.

## 2 Related Work

### 2.1 Category-Agnostic Pose Estimation

The primary aim of pose estimation is to localize the semantic keypoints of objects or instances precisely. Traditionally, pose estimation methods have been largely tailored to specific categories, such as humans [, , ], animals [, ], or vehicles [, ]. However, these prior works are constrained to object categories encountered during training.

An emerging aspect in this field is category-independent pose estimation, as introduced by POMNet []. This few-shot approach predicts keypoints by comparing support keypoints with query images in the embedding space, addressing the challenge of object categories not seen during training.
POMNet employs a transformer to encode the support keypoints and query images. It uses a regression head to predict similarity from the extracted features.
CapeFormer [] extends this matching paradigm to a two-stage framework, refining unreliable matching outcomes to improve prediction precision.
Pose Anything [] presented a significant departure from previous CAPE methods, which refer to keypoints as isolated entities, by treating the input pose data as a graph.
It utilizes Graph Convolutional Networks (GCNs) to leverage the inherent object’s structure to break symmetry, preserve the structure, and better handle occlusions. However, similar to previous CAPE models, it relies solely on visual features.
Our work builds upon Pose Anything, utilizing its structure-aware architecture, while introducing the abstraction power of text.

### 2.2 Open-Vocabulary Models

A growing area in computer vision called Open-Vocabulary learning is being explored in various vision tasks. These new methods aim to localize and recognize categories beyond the labeled space. The open-vocabulary approach is broader, more practical, and more efficient compared to weakly supervised setups [].
Large-scale vision-language models (VLMs) like CLIP [] and ALIGN [] have shown promise in handling both visual and text data, and proved useful for open-vocabulary tasks.

Open-vocabulary object detection (OVOD) using VLMs was utilized by performing object-centric alignment of language embeddings from the CLIP model []. Zang et al. [] suggested a DETR (common transformer-based architecture) based detector, able to detect any object given its class name. In addition, LLMs were also used to generate informative language descriptions for object classes and construct powerful text-based annotations [].
Another task that recently achieved significant progress is open-vocabulary semantic segmentation (OVSS), which aims to segment objects with arbitrary text. One line of research [, , ] combines powerful segmentation models like MaskFormer [] with CLIP [] while others [] utilize foundation segmentation models like SAM [].
Recently, Wei et al. [] suggested a new benchmark for Open-Vocabulary Part Segmentation, to further enhance open-vocabulary capabilities.

Yet, there’s still limited exploration into open-vocabulary keypoint detection.
Recently, CLAMP [] leveraged CLIP to prompt animal keypoints. They found that establishing effective connections between pre-trained language models and visual animal keypoints is challenging due to the substantial disparity between text-based descriptions and keypoint visual features. CLAMP attempts to narrow this gap by using contrastive learning to align the text prompts with the animal keypoints during training.
Our approach aims for general keypoint estimation of any category while taking advantage of structure as a prior for localization by treating the input prompts as a graph.

## 3 Method

### 3.1 Open-Vocabulary Keypoint Detection

Figure: (a)
Refer to caption: extracted/5636531/Figures/horse/part_detection.png

Open-vocabulary learning seeks to localize and recognize categories beyond those included in annotated labels. While open-vocabulary object detection and segmentation have gained attraction, keypoint detection has largely been overlooked.
Segmentation offers pixel-level details about semantic regions, whereas object detection identifies specific objects and their locations. Keypoint detection lies between these two, offering finer semantic localization than object detection, yet being more lightweight and practical for parts localization compared to segmentation. Figure  2 demonstrates the differences between the three tasks.

Figure: Figure 3: Architecture overview: Our framework uses image and text backbones benefiting from both accurate and abstract descriptions respectively. The extracted feature descriptors are forwarded into the transformer encoder that refines them. The refined features are passed into the proposal generator alongside the graph transformer decoder, utilizing the graph structure within the data.
Refer to caption: x2.png

Open-vocabulary keypoint detection aims to use natural language to identify any keypoints in images, even if those key categories were not part of the training data. Advances in vision-language models such as CLIP allow keypoint detectors to harness powerful language models to perform language-driven tasks.
We introduce a new open-vocabulary keypoint detector inspired by CAPE, a few-shot task of localizing keypoints in unseen categories using a few annotated images.
The core idea of our work is that for the task of CAPE, it is more beneficial to describe the searched points in the query image using text description instead of relying only on the visual features of the support images. This is because text allows a higher level of abstraction and offers a looser restriction to the support request. This is true even when the support and query images are from the same category - for example, no two cats share visually the exact same front left leg, but both cats have a part within them that follows the same text description: front left leg. This distinction is even more significant when dealing with images from different categories as in CAPE. We present in the supplemental Figure 10 how a support image-based CAPE solution might suffer from incorrect pose estimation due to visually inconsistent support images.
Bottom left in Figure 3 is an example of a support text-graph that our system utilizes.

### 3.2 Text Prompts as Visual Queues

Our framework extracts visual features from the query image and matches them to the textual features that are extracted from the support text-graph. We incorporated this notion by introducing text comprehension into Pose Anything’s framework [].

A pre-trained and fine-tuned SwinV2-S [] is utilized for extracting image features from the input query image producing the feature map $\hat{\mathcal{I}}_{q}\in\mathbb{R}^{hw\times C_{i}}$, where $hw$ is the total number of patches and $C_{i}$ is the image embedding dimension. Then $\hat{\mathcal{I}}_{q}$ is passed through a 1x1 convolutional layer, resulting in $\mathcal{I}_{q}\in\mathbb{R}^{hw\times C}$.

The support keypoint text descriptions are embedded in our model using a pre-trained gte-base-v1.5 []. The text embeddings of all $K_{s}$ keypoints of the provided support sample are then normalized. The normalized keypoints are padded with zeros, effectively resulting in $K$ keypoints, where $K$ is defined to be the maximum amount of possible keypoints in the dataset. The final text feature map is of the form $\hat{\mathcal{T}}_{s}\in\mathbb{R}^{K\times C_{t}}$ where $C_{t}$ is the text embedding dimension. Then $\hat{\mathcal{T}}_{s}$ is passed through a linear layer resulting in $\mathcal{T}_{s}\in\mathbb{R}^{K\times C}$.
During training, the text backbone is frozen.
This approach also offers a lighter optimization procedure, as the gradients of the text features are ignored. An architecture overview is presented in Figure 3.

The extracted query image features and the support descriptions features are then refined using the transformer encoder. This encoder comprises three transformer blocks. Since the embedding spaces of the support text and query image differs, the support and query features are first fused together and then separated again. This practice aids in closing the gap between their representations [] using self-attention layers.
Then, similarity heatmaps between the query and support features are formed, using the proposal generator. The proposal generator utilizes a trainable inner-product mechanism [] to explicitly represent similarity. Peaks are then chosen from these maps to act as the basis for similarity-aware proposals.
A graph transformer decoder network receives these initial proposals, processes them using a combination of attention and Graph Convolutional Network (GCN) layers, and predicts the final estimated keypoints locations.
Utilizing GCN layers allows for the explicit consideration of semantic connections between keypoints, thereby benefiting CAPE tasks.
We visualize cross-attention maps from the decoder trained with text prompts compared to visual prompts in the supplemental (Figure 11).

To train our end-to-end method we use two loss terms: $\mathcal{L}_{heatmap}$ and $\mathcal{L}_{offset}$. The former penalizes the similarity metric while the latter penalizes the localization output:

$$ $\mathcal{L}_{heatmap}=\frac{1}{(K\cdot H\cdot W)}\sum_{i=1}^{K}{||\sigma(M_{i} )-H_{i}||}$ (1) $$

$$ $\mathcal{L}_{offset}=\frac{1}{L}\sum_{i=1}^{L}{\sum_{i=1}^{K}{|P_{i}^{l}-\hat{ P_{i}}|}}$ (2) $$

where $\sigma$ is the sigmoid function, and for each point $i$, $M_{i}$ is the output similarity heatmap of the proposal generator, $H_{i}$ is the ground truth heatmap, $P_{i}^{l}$ is the output location from layer $l$ and $\hat{P_{i}}$ is the ground truth location. The overall loss is:

$$ $\mathcal{L}=\lambda_{heatmap}\cdot\mathcal{L}_{heatmap}+\mathcal{L}_{offset}$ (3) $$

## 4 Experiments

Table: Table 1: MP-100 results: PCK0.2 performance under the 1-shot setting. Our approach outperforms other methods on average.

In line with prior CAPE studies, we utilize the MP-100 dataset [] as both our training and evaluation dataset, which comprises samples sourced from existing category-specific pose estimation datasets [, , , , , , , , , , , , ]. This dataset consists of over 18K images spread across 100 distinct sub-categories and 8 super-categories (human hand & face & body, animal face & body, clothes, furniture and vehicle), featuring varying numbers of keypoints, ranging from 8 to 68 keypoints.

The dataset is divided into five separate splits for training and evaluation. Importantly, each split ensures that the categories used for training, validation, and testing are mutually exclusive, ensuring that the categories used for evaluation are unseen during the training phase.

The original dataset comes with partial skeleton annotations in different formats, including variations in the keypoint indexing. We use the updated version of Pose Anything [] that includes unified skeleton definitions for all categories. The updated version predominantly featured brief text sentences describing each point within most categories. However, certain categories exhibited text descriptions with distinct characteristics, such as the use of underscores between words instead of spaces, while others lacked any text descriptions altogether.
We annotated and standardized the text descriptions of all points in all categories, offering a new supervision capability to the updated version of [] of the original MP-100.

To assess our model’s performance, we employ the Probability of Correct Keypoint (PCK) metric [], setting a PCK threshold of 0.2, following the conventions established by Pose Anything [], POMNet [] and CapeFormer [].
More design choices and evaluations are in the supplementary.

##### Implementation Details

To ensure a fair comparison, except for the text backbone, the configuration settings remain consistent with Pose Anything [] and CapeFormer []. The trainable features of the framework remain exactly the same as in Pose Anything (except for the new linear layer) since the text backbone is frozen during training in our framework. However, we also evaluate and present the performance of the framework with an unfrozen text backbone in the supplemental Table 2. $C_{i}$ is 768 in SwinV2-S, $C_{t}$ is 768 in gte-base-v1.5. $C$ and $K$ are set to $256$ and $100$, respectively.
The architecture is implemented within the MMPose framework [], trained using the Adam optimizer for 200 epochs with a batch size of 16. The initial learning rate is $10^{-5}$, reducing by a factor of 10 at the 160th and 180th epochs. All experiments in our work were carried out using a machine equipped with an NVIDIA RTX A5000 GPU. Our model required $10$ GB of GPU memory and took roughly $20$ hours to train for each split.

### 4.1 Benchmark Results

Figure: Figure 4: Qualitative results: From left to right: support images that are used by the competitors, CapeFormer-S, Pose Anything-S, our model, and the GT. Support text descriptions used by our model are not shown. Main differences are pointed out using arrows.
Refer to caption: x3.png

We conduct a comparative analysis of our approach with gte-base-v1.5 [] as the freezed text backbone, against Pose Anything [], as well as prior CAPE methodologies such as CapeFormer [] and its enhanced version CapeFormer-T from [], POMNet [], ProtoNet [], MAML [], and Fine-tuned []. For a comprehensive understanding of these models’ performance, additional details can be found in [].

Our evaluation is based on the MP-100 dataset, considering the 1-shot scenario. While traditionally 1-shot refers to a single required support image, our framework uses a single text-graph instead. We do not report the 5-shot results, because we do not use 5 different support images. The results are presented in Table 1. Notably, our text-based approach outperforms Pose Anything on most splits, with an average improvement of 1.07% under the 1-shot setting. These results establish a new state-of-the-art result, showcasing the efficacy of utilizing text-graphs for CAPE.

A qualitative comparison of our model against CapeFormer-S and Pose Anything-S in presented in Figure 4. Our model performs well given the support text-graph input (not shown), while the support image-based techniques are sensitive to the inconsistencies between the
support and query images.

### 4.2 Ablation Study

##### Text Modifications

Figure: Figure 5: Modified text descriptions: Top is the support keypoints text descriptions. Left is a synonym words test, middle is a translation test and right is typo test. Below each description, query output(s) are presented. Each node in the presented graph is the average positions between the original and modified text descriptions. The diameter represents the distance between the positions.
Refer to caption: x4.png

The fact that the text backbone was not fine-tuned during the training of our model, keeps it from overfitting to text descriptions from the training set. On the contrary, the model demonstrates its effectiveness across modified text inputs, while preserving similar estimated poses overall. We test the robustness of our model on different types of modifications for the keypoint descriptions in Figure 5. Specifically, we test the adaptability of the model to synonym descriptions (left), to translation to another language and back to English (middle), and to typos (right).
Notably, all average keypoints are placed in acceptable positions.
The main differences in the two synonym test examples are in keypoints 7, 8 (’elbow’ $\rightarrow$ ’middle of the arm’) and 13, 14 (’knee’ $\rightarrow$ ’middle of the leg’). In the translation test, the main differences are in keypoints 5-7 and 9-11 (’top/bottom side of the right/left jaw/cheek’ $\rightarrow$ ’upper/lower side of the right/left jaw/cheek’) and 27 (’top side of the nose’ $\rightarrow$ ’upper side of the nose’). In the typos test, the significant inconsistencies are in the head (keypoints 0,2 and 3), while minor differences are spotted also in the leg joints (keypoints 5,6 and 12).
All these differences are compatible with the discrepancy imposed by the different descriptions.

Figure: Figure 6: Text Abstractions: Model performance over different levels of text-pose abstractions.
Refer to caption: x5.png

##### Occlusions and Levels of Abstraction

We test the robustness of our model on keypoints that are described using different levels of text and pose-graphs abstractions. Results are in Figure 6. Although not trained with the prompted text descriptions and pose-graphs, the model presents satisfactory results in both examples.

Figure: Figure 7: Masking the query image: PCK0.2 performance as a function of the masking percentage.
Refer to caption: x6.png

Furthermore, we assess the effectiveness of text-graphs in handling occlusions within query images by applying random masks to them before estimating the support keypoints. Quantitative results are in Figure 7, and a qualitative comparison is presented in the supplemental Figure 13. Our method demonstrates superior performance over Pose Anything-S and CapeFormer-S in the entire presented occlusion range while maintaining similar degradation levels between 0% to 25%. The improved performance at lower masking percentages can be attributed to the text-graphs’ abstraction capability and their ability to estimate missing keypoints relative to the visible ones. However, as our approach does not utilize a support image as input, performance significantly drops and matches the competitors, when a substantial portion of the image is occluded (50%). This is because the absence of the query image leaves the model with insufficient information to operate effectively. This stands in contrast to traditional CAPE methodologies that incorporate a support image, which provides crucial structural cues. In such frameworks, the support image aids the model in hallucinating and extrapolating matching keypoints within the query, particularly when considering graph structures as in Pose Anything.

##### Out of Distribution Query Images

Figure: Figure 8: Out of distribution performance: Top is the support keypoints text descriptions. Below each description, we present the query output.
Refer to caption: x7.png

We evaluate the resilience of our model to out-of-distribution query images generated via diffusion models. In Figure 8, we examine novel styles, categories, poses, and even imaging methods. While the estimated poses generally align coherently with both the query and the support text-graph, there are notable inconsistencies. For example, the model appears to inaccurately localize ’knees’ in both two left query outputs and fails to localize the ’front paws’ in the zebra query output.

## 5 Limitations

Figure: Figure 9: Failure cases: From left to right: a category outside of the dataset, introducing vastly new keypoint descriptions, and cross-category descriptions.
Refer to caption: x8.png

We stress that although our model strives for full open-vocabulary performance, it is still trained on a relatively small training set over arguably a short time period, compared to the state-of-the-art large vision-language foundation models. We present in Figure 9 a few failure cases that may be addressed in future research. Our model does not handle new categories with novel text-graphs well, as can be seen in the plane example on the left. In addition, prompting with vastly new parts may lead to incorrect localizations as can be seen in the car example on the middle (for example, driver/passenger-side door). Lastly, the model incorrectly executes semantically challenging descriptions. For example, the model can not localize a ’seat’ in a horse, even though riders may seat on it. Instead, it hallucinates a pose of a chair that it has seen in the training set.

## 6 Broader impact

Advancements in pose estimation technology can revolutionize fields such as autonomous driving, smart cities, sports, etc., by enabling precise movement analysis. However, when a pose estimation tool is used, specifically in fields such as surveillance, it is crucial to address privacy concerns and establish ethical guidelines to protect sensitive personal data and ensure responsible use.

## 7 Conclusions

CapeX is a Category Agnostic Pose Estimation (CAPE) approach that is based on text input. In particular, CapeX takes a pose-graph, where text descriptions are attached to its nodes, and finds these keypoints in a query image. This stands in contrast to previous CAPE approaches that require support image with annotated pose-graph as part of the input. CapeX can be viewed as an Open Vocabulary Kepoint Detection algorithm, closing the gap between Open Vocabulary Object Detection and Open Vocabulary Segmentation.

CapeX was tested on the standard MP-100 dataset and achieves a new state of the art result, surpassing previous CAPE methods that rely on support image as input instead of text.

## Acknowledgments and Disclosure of Funding

This work was partly funded by the Weinstein Institute.

## Appendix A Appendix / Supplemental Material

### A.1 Disadvantages of Visual Prompts in CAPE

Figure: Figure 10: Visual Prompts Inconsistencies: We show different results using Pose Anything model, for the same query image using different support images. Keypoints definitions and skeletons are the same. Using visual features impairs the ability to describe abstract semantic parts.
Refer to caption: extracted/5636531/Figures/same_query/000000043182_0.png

Figure: (a)
Refer to caption: extracted/5636531/Figures/attention_maps/helen_face_attn_GRAPH_top_right_side_of_the_left_eye.png

We exemplify the key disadvantage of support image-based CAPE approaches in Figure 10. Specifically, Pose Anything-S, suffers from incorrect pose estimations when prompted with visually inconsistent support images.

We also compare our models with regards to localization and symmetry breaking. We visualize cross-attention maps from the decoder trained with text prompts compared to visual prompts in Figure 11. Our model breaks symmetry and distinguishes better between left and right, compared to Pose Anything. This can be seen by the lower attention to the right eye, when prompted with the keypoint ’top right side of the left eye’.

### A.2 Additional Experiments

#### A.2.1 Different Architectures

Table: Table 2: Ablation experiments: Tuning (T) VS. Freezing (F) the text backbone in model training, utilizing the graph transformer decoder or the original mlp transformer decoder. PCK0.2 performance under 1-shot setting, with gte-base-v1.5 or CLIP ViT-B/32 as the text backbone.

We assess the framework’s performance with or without fine-tuning applied to the text backbone. We explore two potential text backbones: gte-base-v1.5 [] and CLIP ViT-B/32 [].
The optimal configuration appears to be the frozen gte-base-v1.5 as the text backbone, yielding superior performance. Interestingly, although gte-base-v1.5 boasts approximately 139 million trainable parameters compared to the 63 million parameters in the text module of CLIP ViT-B/32 (totaling 150 million parameters), training with either as a frozen text backbone consumed similar execution times, lasting roughly $20$ hours. Memory usage for loading both models required a similar volume of $10$ GB.
However, fine-tuning both models incurred substantial costs in terms of memory: $15$ in gte and $30$ GB in CLIP, as well as in execution time: $35$ hours for both architectures, without yielding any performance improvements in both text backbones.
The drop in performance can possibly be attributed to the fact that the text backbones suffer from overfitting during their tuning. This is somewhat expected as language models usually train on larger datasets over longer training sessions.

We also tested the original MLP transformer decoder architecture as in [] with the best performing setting. Memory consumption and execution time using this transformer decoder were comparable to the graph transformer decoder. We find that utilizing the graph structure via the graph transformer decoder as in [] slightly boosts the performance. Full results are presented in Table 2.

Figure: Figure 12: Modified text descriptions: Top is the support keypoints text descriptions. Left is a synonym words test, middle is a translation test and right is typo test. Below each description, query output(s) are presented. Each node in the presented graph is the average positions of the original and modified text descriptions. The diameter represents the distance between the positions.
Refer to caption: x9.png

#### A.2.2 Adaptability to Support Text Modifications

We provide additional examples of the ability of our model to adapt to different types of text modifications in Figure 12.

#### A.2.3 Occlusions and levels of abstraction

Figure: Figure 13: Comparison to Pose Anything-S: Qualitative comparison between our model and Pose Anything-S on masked queries. CapeX does not require support images and can handle masked occlusions. Support images and text-graphs are not shown.
Refer to caption: x10.png

Figure: Figure 14: Text Abstractions: Model performance over different levels of text-pose abstractions.
Refer to caption: x11.png

We present qualitative comparison between our support text-based framework and Pose Anything-S’s support image-based framework in Figure 13. Our model demonstrates better performance due to the abstraction power of text-graphs, compared to the use of support image which may be more restrictive.

We include additional results of our model’s performance on different levels of abstractions in Figure 14.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please see weaknesses.
soundness: 3
strengths: This work focuses on the interesting and important task of category-agnostic pose estimation (CAPE). The proposed CapeX utilizes the abstract textual description for keypoint detection to improve the human-computer interaction. Graph structure is applied to capture the relationship between keypoints. The proposed CapeX achieves the state-of-the-art performance on the MP-100 dataset.
summary: This paper proposes to employ textual information connected in a graph structure to tackle the problem of category-agnostic pose estimation (CAPE). The textural keypoint descriptions are annotated on the MP-100 dataset. The experimental results on the MP-100 dataset demonstrate the effectiveness of the proposed method.
weaknesses: Pose Anything has designed the graph structure to capture the keypoint correlations, and the textual prompts have been explored for pose estimation in recent works such as KDSM. X-Pose has also provided textual annotations on the MP-100 dataset. Therefore, the contribution of this paper is somewhat incremental.

The graph construction is critical for graph-based approach. Also, manual design of node connections may introduce extra empirical knowledge, leading to unfair comparison. To avoid the above misgiving, the details of graph construction are expected to be clarified.

The cross super-category evaluation on the MP-100 dataset is absent. The cross super-category results would further validate the robustness of the proposed method.

---

confidence: 5
contribution: 3
initial_rating: 8
presentation: 4
questions: See Weaknesses.
soundness: 3
strengths: 1. This paper addresses a very important issue: relying on visual information from a support image to locate keypoints in a test image is not reliable.

2. The use of a graph and text representation is intriguing, and I believe it provides a simple yet effective solution to the challenges associated with relying solely on visual information for localization.

3. The paper expands the MP-100 dataset, enabling graph-based CAPE.

4. The experiments in the paper are thorough, and the comparisons are comprehensive.
summary: This paper introduces a novel approach that diverges from traditional CAPE methods, which typically rely on a support image, by utilizing a text-based framework instead. By implementing a pose-graph in which nodes signify keypoints described through textual annotations, this method leverages the abstraction inherent in text and the structural organization provided by the graph. This strategy effectively mitigates symmetry issues, maintains structural integrity, and enhances the handling of occlusions. The authors enrich the dataset by incorporating text description annotations for both the training and testing phases. They also provide alternative text annotations to specifically evaluate the model's ability to generalize across varying textual descriptions.
weaknesses: I don’t have many questions regarding this paper, just a few minor issues. 

1. Were all the keypoint texts in the dataset generated by off-the-shelf foundation models? When many keypoints are very close to each other and have highly similar semantics, how are these very similar keypoints distinguished in the text? Could you provide some examples?

2. The authors could provide more analysis regarding the limitations of the paper, explaining why certain failure cases occur.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. It seems that both visual prompt (support images) and text prompt (keypoint name) have their own limitations. For example, human face may have 68 keypoints, and it seems hard for textual prompts to precisely describe the keypoints along the edge of the face. How to solve this? Could you please give some examples about how to describe these keypoints in language?

2. Figure 6 seems interesting. If I understand correctly,  the experiments about "Modified text descriptions" are evaluated based on the same model without retraining. Is it correct?

3. In "Modified text descriptions", could you add experiments with more detailed description (longer description) about the keypoint. The reviewer is curious about the generalization ability of the system.
soundness: 3
strengths: 1. The motivation is clear  and the paper reads smooth. 

2. The design of the presented method is reasonable.

3. Experiments are conducted on the standard MP-100 dataset, establishing a new state-of-the-art for CAPE. The presented method shows superiority to previous CAPE models. Especially, the experimental analysis (e.g. Figure 6 and 7) is very interesting.
summary: Recent CAPE works have produced object poses based on arbitrary keypoint definitions annotated on a user-provided support image. Our work departs from conventional CAPE methods, which require a support image, by
adopting a text-based approach instead of the support image. Specifically, we use
a pose-graph, where nodes represent keypoints that are described with text. This
representation takes advantage of the abstraction of text descriptions and the structure imposed by the graph. 

Our approach effectively breaks symmetry, preserves
structure, and improves occlusion handling. We validate our novel approach using the MP-100 benchmark, a comprehensive dataset covering over 100 categories
and 18,000 images. MP-100 is structured so that the evaluation categories are unseen during training, making it especially suited for CAPE. Under a 1-shot setting,
our solution achieves a notable performance boost of 1.27%, establishing a new
state-of-the-art for CAPE. Additionally, we enhance the dataset by providing text
description annotations for both training and testing. We also include alternative
text annotations specifically for testing the model’s ability to generalize across
different textual descriptions, further increasing its value for future research.1
.
weaknesses: 1. The novelty of the presented paper is a little bit concerning. (1) Open-vocabulary (textual prompt) keypoint estimation is not new, there are works such as CLAMP (Zhang et al. 2023) and KDSM (Zhang et al. 2024). (2) Graph representation for few-shot/zero-shot keypoint estimation is not new, as there are works like GraphCape (Hirschorn & Avidan, 2023). Could you highlight the difference between these methods? 

2.  Pose-Graph seems to be the most important contribution. However, experimental analysis about the pose-graph is insufficient. For example, there are different graph structures, what if using a fully-connected graph AND different graph structural settings (e.g. Tree-structure)?  

3. The reviewer suggests adding "Cross Super-Category Pose Estimation" on MP-100 evaluation, by training the model on all but one super-categories, and evaluate the performance on the remaining one super-category. This is to further evaluate the generalization ability. 

4. Since there already exists an open-vocabulary keypoint detection work like CLAMP (Zhang et al. 2023), what is the performance when comparing your method to CLAMP on Animal pose dataset in the setting of five-leave-one-out problem? Namely training on four species while testing on the left one species. The results of PCK @ 0.1 should be presented in paper for comparisons.

[Zhang et al. 2023] CLAMP+Prompt-based Contrastive Learning for Connecting Language and Animal Pose @ CVPR'23

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: See Weaknesses*. Overall, the idea of paper seems interesting, and I would like to see more future works. My rating could be increased if my concerns are solved.
soundness: 3
strengths: - The paper proposes to make full use of text to boost CAPE, which seems novel and interesting.
- The proposed method seems resonable and works well for CAPE.
- The achieved performances are significant against existing works of CAPE.
- The paper is well-written.
summary: The paper focuses on the category-agnostic pose estimation task, which learns to estimate the keypoints of novel classes based on a few support images and text descriptions.
Specifically, the paper proposes a pose-graph and use nodes to represent keypoints that are described with text, which could take advantage of the abstraction of text descriptions and the structure imposed by the graph.
On the MP-100 benchmark dataset, the paper achieves a notable performance boost of 1.27% against SOTA methods of category-agnostic pose estimation.
weaknesses: - The description about the related work MetaPoint could be improved, which actually first assigns and then refines.
- The paper composing could be improved. E.g., Fig. 3 and Fig. 4 waste too much space and provide limited information.
- The limitations on using text could be discussed more deeply, because some keypoints are hard to describe using text.
- The experiments could be enhanced with cross super-class transfer as in MetaPoint, CapeFormer and POMNet.

Meta-review: This work proposes to make full use of text to enhance Category-Agnostic Pose Estimation (CAPE), and the method seems to be novel and interesting. The proposed scheme is reasonable and works for this task. Most of the reviewers pointed out that this work is novel and recommended acceptance. There are some weaknesses in the work, e.g., the limitations of using text need to be discussed more deeply, 
and the experiments need to be enhanced with cross super-class transfer. Authors have addressed most of the concerns. AC thus recommend acceptance.

Decision: Accept (Poster)