KptLLM: Unveiling the Power of Large Language Model for Keypoint Comprehension

Authors: jie  yang
wang  zeng
sheng  jin
lumin  xu
wentao  liu
chen  qian
ruimao  zhang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 11:11:36

Abstract: Recent advancements in Multimodal Large Language Models (MLLMs) have greatly improved their abilities in image understanding. However, these models often struggle with grasping pixel-level semantic details, e.g., the keypoints of an object. To bridge this gap, we introduce the novel challenge of Semantic Keypoint Comprehension, which aims to comprehend keypoints across different task scenarios, including keypoint semantic understanding, visual prompt-based keypoint detection, and textual prompt-based keypoint detection. Moreover, we introduce KptLLM, a unified multimodal model that utilizes an identify-then-detect strategy to effectively address these challenges. KptLLM underscores the initial discernment of semantics in keypoints, followed by the precise determination of their positions through a chain-of-thought process. With several carefully designed modules, KptLLM adeptly handles various modality inputs, facilitating the interpretation of both semantic contents and keypoint locations. Our extensive experiments demonstrate KptLLM's superiority in various keypoint detection benchmarks and its unique semantic capabilities in interpreting keypoints.

Main paper: 
 introduction: 
 Recent advancements in deep learning and natural language processing have facilitated the rise of Large Language Models (LLMs) that display human-like fluency in text comprehension and generation [1, 2, 3, 4, 5, 6, 7] . By incorporating visual information, researchers have developed Multimodal Large Language Models (MLLMs) [8, 9, 10, 11, 12] , specifically designed for visuallanguage tasks, showcasing remarkable abilities in image understanding. However, these models encounter difficulties in capturing fine-grained semantic details, particularly at the point level, which are crucial for various real-world applications. The exploration of MLLMs for keypoint comprehension remains under-explored in the literature.Keypoint detection is a fundamental aspect of computer vision that supports various applications such as controllable image/video generation [13, 14, 15] , human-centric perception [16, 17] , and AR/VR systems [18, 19, 20] . Initially, research in this field focused on closed-set problems, aiming to predict the locations of predefined semantic keypoints of a certain object category (e.g. human body). As the demand for generalization grew, researchers started investigating the detection of keypoints for novel objects by providing visual prompts (i.e., a support image of a novel object with its keypoint definitions) [21, 22] or utilizing textual prompts (i.e., keypoint names) [23] . Despite significant progress in these areas, existing models still fall short of achieving genuine semantic comprehension of keypoints akin to humans. These models primarily rely on direct learning of visual patterns for keypoint localization through extensive data fitting, while neglecting semantic understanding of the keypoints, thus leading to misinterpretation of the prompts and inaccurate predictions. Moreover, the input-output structures are designed in fixed and predefined formats, restricting their usage to predetermined methods and impeding the flexibility required for interfacing with users.Motivated by the aforementioned challenges, this paper delves into a more comprehensive problem of Semantic Keypoint Comprehension to evaluate the model capability of comprehensively understanding keypoints both visually and semantically. As shown in Fig. 1 , we investigate three distinct capabilities via different task instructions: (a) Keypoint Semantic Understanding aims to infer the desired keypoint semantics, given the target image and a keypoint prompt (i.e., the position of the target keypoint) as inputs. It provides the potential for an AI model with high-level visual understanding and analytical capabilities, crucial for tasks such as structural comprehension, action recognition, and medical image analysis. (b) Visual Prompt-based Keypoint Detection, also referred to as category-agnostic pose estimation, takes a query image and a labeled support image with the keypoint annotation as inputs and then outputs the corresponding keypoint positions in the query image. This capability requires the model to acquire keypoint definitions from visual prompts, enabling it to perform cross-class and cross-keypoint localization tasks using sample images provided by users. (c) Textual Prompt-based Keypoint Detection, also known as open-vocabulary keypoint detection, aims to utilize detailed descriptions of keypoints through extensive text for keypoint localization. The keypoint detectors directly receive the human language guidance, facilitating keypoint localization on arbitrary object and keypoint categories in a zero-shot manner.We introduce KptLLM, a novel framework that utilizes an identify-then-detect strategy to address the challenging problem of semantic keypoint comprehension. It formulates all three capabilities depicted in Fig. 1 , by first identifying the semantic meaning of keypoints and then detecting their positions via a chain-of-thought approach, akin to human cognition. KptLLM is a unified framework that comprises four key components designed to accommodate various modality inputs and infer both the semantics and location of the keypoint. Specifically, we first extract visual features of both query and support images to obtain query visual tokens and support image features. Secondly, we encode the support keypoint prompt, which describes the position of keypoint on the support image, to generate keypoint prompt embedding. Thirdly, prompt-oriented features are derived by integrating support image features with keypoint prompt embedding, and are utilized to form keypoint prompt tokens. Lastly, LLMs take query visual tokens, keypoint prompt tokens, and task-related language tokens as input, and then generate the semantic description of the target keypoint and its corresponding position on the query image. By harnessing commonsense knowledge in LLMs, KptLLM can assist in keypoint localization of novel object categories, potentially leading to enhanced generalizability in performance. In addition, the chain-of-thought design elicits the powerful keypoint understanding capabilities of LLMs, which helps to distinguish visually ambiguous keypoints (e.g. left and right arms). Extensive experiments demonstrate KptLLM's superiority on semantic keypoint comprehension, showcasing its unique semantic understanding capabilities in interpreting keypoints and state-of-the-art performance in various keypoint detection benchmarks.In summary, the contributions of this work are three-fold: (1) We pioneer the investigation of a novel problem in semantically interpretable keypoint analysis, termed Semantic Keypoint Comprehension, which aims to enhance MLLMs with improved image understanding at a finer-grained keypoint level;(2) We introduce KptLLM, a unified multimodal model that utilizes an identify-then-detect strategy to effectively address three tasks of semantic keypoint comprehension. KptLLM underscores the initial discernment of semantic significance in keypoints, followed by the precise determination of their positions through a chain-of-thought process. (3) We demonstrate KptLLM's superiority in various existing keypoint detection benchmarks and its unique semantic capabilities in interpreting keypoints. We hope our work could inspire future research on keypoint understanding and localization, while also fostering enhanced human-AI interface in fine-grained visual understanding.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: All claims are validated by experiments.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
keypoint detection: 
 Keypoint detection, also referred to as pose estimation, focuses on localizing the 2D keypoints of objects in the image. Traditional models for keypoint detection are typically designed for a single category, e.g., human [24, 25, 26, 27, 28, 29, 30] , animal [31, 32, 33] and clothes [34] . Based on the localization strategy, existing methods are generally divided into regression-based methods [35, 36, 37, 38, 39, 40] and heatmap-based methods [41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51] . More recently, methods that can recognize and localize keypoints for unseen object categories in the training datasets, are gaining increasing attention from the community. Category-agnostic pose estimation [21, 22] , also referred to few-shot keypoint detection, aims to estimate the pose of any category in query images with visual prompts (i.e., a few support images of a novel class and its corresponding keypoint annotations). Another line of research explores open-vocabulary keypoint detection [23, 52] , which aims to localize keypoints based on text prompts in zero-shot settings. In this work, we investigate semantic keypoint comprehension and propose a novel unified framework to comprehend keypoints across three different task scenarios, including (a) keypoint semantic understanding; (b) visual prompt-based keypoint detection; (c) textual prompt-based keypoint detection.Inspired by the success of Large Language Models (LLMs) [1, 2, 53, 3, 4, 54, 5, 6, 7] , researchers are exploring ways to transfer the formidable capabilities of LLMs into the realm of vision, developing Multimodal Large Language Models (MLLMs) [55, 8, 9, 10, 56, 57, 11, 12, 58, 59, 60, 61, 62, 19] . These models exemplify an autoregressive mechanism predicated on a transformer decoder architecture [63] . The integration of visual representation from vision encoders [64, 65] into the domain of LLMs ushers a new era of visual comprehension and reasoning. Such integration is predominantly facilitated through a Multilayer Perceptron (MLP) that seamlessly transforms visual features into the input embedding space of LLMs [10, 56, 57] , or via a cross-attention mechanism that attends to visual contents through a series of attention layers [55, 8, 11] . However, most of these VLMs can only provide text outputs, inhibiting the complex applications requiring detailed visual perception. VisionLLM [66] The left ear is situated on the left side of the head, specifically on the lateral aspect.Where is the left ear of the object in this image? tuning LLMs. However, it may fall short of fully leveraging the comprehensive reasoning faculties of LLMs. Kosmos-2 [67] , Qwen-VL [68] and DetGPT [69] further exploit the power of LLMs to enable user-guided detection. Moreover, GPT4RoI [70] , Ferret [71] , Shikra [72] , and PerceptionGPT [73] innovates by incorporating spatial boxes or masks as inputs and training with region-text pairs, offering region-level visual comprehension. Notably, a concurrent work, LocLLM [74] , utilizes LLMs for human keypoint localization via textual description. In contrast, we take a step further by enabling LLMs to comprehend keypoints of various objects via multi-modal (e.g., textual or visual) prompts under different task formulations. This advancement not only broadens the utility of MLLMs for keypoint detection but also enhances interpretive depth, allowing for a more comprehensive understanding and grounding across a wider range of visual information.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Please refer to Sec. 6.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
methodology: 
 This section introduces our proposed unified framework, referred to as KptLLM, which effectively addresses three semantic keypoint comprehension scenarios. As illustrated in Fig. 2 , KptLLM accepts multiple images (query and support images) along with a support keypoint prompt (i.e., the position of the target keypoint in the support image) and textual user instructions as the input. The output comprises both the response text and the desired keypoint position. Specifically, KptLLM comprises four key architectural components: (1) A visual encoder that extracts features from both query and support images (see Sec. 3.1); (2) A prompt encoder that converts support keypoint prompts into prompt embeddings (see Sec. 3.2); (3) A prompt feature extractor that derives prompt-oriented features from the corresponding image features (see Sec. 3.3); (4) A pre-trained LLM that processes multimodal tokens for keypoint comprehension (see Sec. 3.4).The Visual Encoder is designed to process two types of images in parallel: query and support images. Generally, it receives an input image I ∈ R H×W ×3 and generates a feature map F = V(I) ∈ R h×w×d . Here, d represents the feature dimension, and h and w are the spatial dimensions obtained by downsampling the original image dimensions H and W .Query Image. The query image represents the image that is to be analyzed. We extract its spatial features through the vision encoder V, resulting in F q . Following LLaVA [10] , we apply a linear layer to project F q into language space: z q = Linear(F q ). As a result, query visual tokens aligned with the LLM dimension are obtained and fed to the LLM.Support Image. The support image serves as a reference example. We extract its spatial features, which are represented as F s . Unlike the query image features, F s is not directly input into LLM. Instead, it is processed by the prompt feature extractor to derive prompt-oriented features.In addition to processing images, we need to incorporate an additional prompt consisting of 2D coordinates x ∈ R 2 , which describes the keypoint location within the image. Inspired by SAM [75] , we introduce a prompt encoder to adapt this prompt input to be aligned with the image feature space F. The prompt encoder encodes the keypoint coordinates using a sine-cosine position embedding (PE), followed by a Multi-Layer Perceptron (MLP):EQUATIONThe Prompt Feature Extractor is designed to extract the prompt-specific features from image features.As illustrated in Fig. 2 , the semantics of the keypoint prompt directly correspond to the support image. We initialize the prompt feature extractor with a two-layer transformer that incorporates the cross-attention mechanism (CrossAttnLayers). This mechanism employs F p as the query and F s as the key and value to extract keypoint-specific visual features indicated by the prompt:EQUATIONwhere z p denotes the keypoint-specific visual features. In essence, compared with average poolingbased feature extraction method [21] , the prompt feature extractor is trainable and capable of incorporating global image features to enhance keypoint identification. This is particularly beneficial for distinguishing mirror-symmetric keypoints, such as the left and right eyes, which can be highly ambiguous when relying solely on local image features. Our ablation study demonstrates the performance improvements achieved through the utilization of this component.Given a query image and an optional prompt specifying the keypoint of interest, our goal is to generate textual descriptions and keypoint locations that convey fine-grained keypoint information within the image. Recognizing the exceptional ability of LLMs in handling multimodal tokens for different perception tasks [76, 10, 77, 73, 78] , we further leverage LLM for keypoint comprehension, which could effectively process various inputs: (1) the visual tokens z q of the query image, (2) the prompt tokens z p , and (3) a sequence of language tokens t, which depend on the three semantic keypoint comprehension scenarios. Keypoint Semantic Decoding. We design the model to directly generate textual descriptions that interpret keypoint semantics, following the standard approach used by LLMs for text generation. Generally, the architecture of an LLM typically comprises Transformer layers (TransformerLayers) followed by a final Feed Forward Network (FFN). The latent embedding u, which captures the fused multimodal information, can be computed as:EQUATIONwhere [z q , z p , t] denotes the concatenation of the visual, prompt, and language tokens. This embedding u is then passed through the FFN and a Softmax function to generate the probability distribution p over the vocabulary for the next token:EQUATIONKeypoint Position Decoding. Inspired by [78, 73] , we introduce a special token, <keypoint>, into the vocabulary. Consequently, the 2D keypoint position y can be computed from the output latent embedding u kpt of the special token using another FFN prediction head:EQUATION)Indentify-then-Detect (ItD) Strategy. Instead of relying on extensive data fitting to learn fixed keypoint localization patterns, we adopt an approach where the LLM initially identifies the semantics of keypoints. Subsequently, a chain-of-thought process is employed to accurately detect the locations of these keypoints. Experiments demonstrate improved performance compared to alternative baselines.To retain the learned general knowledge of the pre-trained LLM, we employ LoRA [79] for efficient fine-tuning of LLM, while fully fine-tuning other modules of the framework. The training and inference processes for different tasks are outlined below.Keypoint Semantic Understanding. As shown in Fig. 1-(a) , this task focuses on extracting semantic textual information associated with specific keypoints within an image. The training objective is to minimize the language modeling loss, computed as the cross-entropy loss over the vocabulary of the LLM's tokenizer. Specifically, the loss function is defined as:EQUATION)where a is the text response predicted by the model, and â is the ground-truth text response. During inference, given an image provided by the user and the corresponding keypoint position as a prompt, our model comprehends and generates the semantic meaning of the specified keypoint.Visual Prompt-based Keypoint Detection. This task involves simultaneously comprehending the semantics of keypoints, generating textual descriptions of this understanding, and precisely localizing the keypoint coordinates, as shown in Fig. 1-(b ). The overall training function further incorporates the L1 loss for keypoint regression:EQUATION)where ŷ is the ground-truth keypoint position, and λ is the loss weight that balances the learning of keypoint regression and text generation (λ = 2 in our implementation). During inference, the user provides two images: one as the query image for testing and the other as the support image for reference. Additionally, the keypoint definition for the support image should be provided as the support keypoint prompt. Our model then comprehends the semantics of the desired keypoint and detects its corresponding position in the query image. (2) The AP-10K dataset [32] for Textual Prompt-based Keypoint Detection: The dataset comprises 23 animal families and 54 species, totaling 10,015 images. Each image is annotated with 17 keypoints, including two eyes, one nose, one neck, two shoulders, two elbows, two knees, two hips, four paws, and one tail. We follow CLAMP [23] to assess the models' ability to generalize to previously unseen animal species within a zero-shot learning paradigm. We establish two experimental scenarios based on the taxonomic relationship between the species in the training and test sets-specifically, whether they belong to the same animal order. Species within the same order typically share similar visual characteristics, whereas those from different orders exhibit greater diversity in appearance. These scenarios enable us to assess how different methods perform when generalizing to unseen species under varying conditions. Following CLAMP, we assign Bovidae and Canidae as the training and test sets for the different order setting, while Canidae and Felidae are chosen as the training and test sets for the same order setting.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]Answer: [Yes] Justification: Our code will be released at https://kptllm.github.io.Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
evaluation & metrics: 
 Different evaluation methods and metrics are used for different tasks of semantic keypoint comprehension. (1) Keypoint Semantic Undertanding: We use the MP-100 dataset [21] (Split-1), with the keypoint semantic labels adopted from X-Pose [52] . Some keypoints are excluded from the evaluation due to ambiguity or inadequacy in their descriptions, such as those used to describe the collar in the clothing category. By aggregating the results of tested keypoints, we derive corresponding accuracy rates (%). ( 2) Visual Prompt-based Keypoint Detection: We employ the Probability of Correct Keypoint (PCK) metric, which is the standard evaluation measure for this task. Consistent with POMNet [21] , we uniformly set the PCK threshold to 0.2 across all categories. Additionally, we compute and report the average PCK over all five data splits to provide a comprehensive indication of the model's overall effectiveness. (3) Textual Prompt-based Keypoint Detection: Following CLAMP, we employ average precision (AP) as the primary metric for AP-10K. This metric is computed based on the object keypoint similarity (OKS). For detailed protocol definitions, please refer to [24] .Architecture. We utilize LLaVA-V1.5-7B [10] as our base model, which incorporates the ViT-based visual encoder of CLIP for image encoding and Vicuna-7B (fine-tuned from Llama-2) as the LLM backbone. We employ LoRA for efficient fine-tuning LLM. Instead, all other modules, including the visual encoder, prompt encoder, prompt feature extractor, and a series of linear layers and feed forward networks, undergo full fine-tuning. The input image only contains a single object of interest, cropped according to the ground-truth bounding box and resized to 336×336, consistent with CLIP-ViT-L.Training Details. LoRA parameters are configured with a rank of 128 and an alpha of 256. Optimization is conducted using AdamW, with a learning rate of 2e-4 and weight decay of 0. We utilize 8 NVIDIA A100-80G GPUs for training, and use the DeepSpeed engine to enhance training efficiency. Each GPU operates with a batch size of 16, and we employ a gradient accumulation step of 1.Table 1 : Keypoint Semantic Understanding on MP-100 (Split-1) [21] .* means LLaVA is finetuned using LoRA.Accuracy LLaVA [10] 3% LLaVA * [10] 72% KptLLM 83%As depicted in Tab. 1, we present the accuracy for keypoint semantic understanding on MP-100 [21] Split-1 set. To facilitate a comprehensive comparison, we highlight the keypoint area in the image and feed the processed image, along with the task instruction, into LLaVA [10] . We report the performance of both the original LLaVA model and a version fine-tuned on the MP-100 dataset. The original LLaVA performs notably poorly in grasping keypoint semantics, indicating the inadequacy of traditional multimodal large language models in capturing fine-grained semantic details. Conversely, the fine-tuned LLaVA demonstrates significantly enhanced performance, thereby validating the efficacy of our training pipeline. Furthermore, our KptLLM surpasses the fine-tuned LLaVA by a substantial margin, particularly in terms of keypoint accuracy (83% vs 72%). It demonstrates the effectiveness of our keypoint prompt token in guiding attention to the fine-grained keypoint area. 1-shot & 5-shot Evaluation. We compare our method with the previous visual prompt-based methods ProtoNet [80] , MAML [81] , Fine-tune [82] , POMNet [21] , and CapeFormer [22] . Tab. 2 presents the PCK results of different approaches on the MP-100 dataset under both 1-shot and 5-shot settings.Compared with previous methods, KptLLM showcases the potential of MLLM in detecting keypoints through the use of visual prompts, consistently outperforming across all settings and data splits. More importantly, we integrate keypoint semantic understanding into the output response, introducing novel functionalities for comprehending the semantic aspects of support image keypoints.Cross Super Category Evaluation. To thoroughly assess generalization across markedly different categories, we conduct a cross-supercategory evaluation following the protocol of POMNet [21] .While the MP-100 dataset ensures that training, validation, and test categories are non-overlapping, some categories may still exhibit similar features, e.g., body characteristics commonly shared among different quadruped animals. To address this, we designate four supercategories-human face, human body, vehicle, and furniture-from the MP-100 dataset as test categories. The remaining categories are utilized for training, allowing us to better evaluate the model's ability to generalize across significantly diverse categories. As shown in Tab. 3, KptLLM consistently outperforms previous methods, highlighting the robustness and excellent generalization ability of our proposed method.Qualitative Results. For the visual prompt-based keypoint detection task, the input necessitates a support image of the object to be tested, as well as keypoint positions that represent the definitions of those keypoints. In this study, we examine how the visual disparity between support images and query images affects the model's performance. As depicted in Fig. 3 , our model is capable of effectively detecting keypoints in various query images when provided with the same support image and its corresponding keypoints. This effectiveness is maintained even in the presence of differences in object poses, appearances, and environmental conditions. In this subsection, we perform ablation study on the design choices of our model. The experiments are conducted on the visual prompt-based keypoint detection task using the MP-100 Split-1 setting with the PCK metric reported.Indentify-then-Detect (ItD) Strategy. KptLLM follows the identify-then-detect paradigm, where the model learns to first interpret the semantic information of the keypoint to be detected, and then predict the precise location of the keypoint. In Tab. 5, we validate the effectiveness of our ItD strategy. We observe notable enhancement, which can be attributed to the inter-task synergy that arises from the ItD mechanism.Prompt Feature Extractor. In Tab. 6, we compare our prompt feature extractor (Sec. 3.3) with the average pooling based feature extraction method [21] . The results show that our prompt feature extractor significantly outperforms the baseline method (91.66 vs 89.78), which validates the efficacy of our prompt feature extractor in enhancing focus on fine-grained keypoint areas.Combining Visual and Textual Prompts. In Tab. 7, rather than relying solely on the visual prompt for localization, we further incorporate the textual prompt to demonstrate the effect of this combination. The improved results indicate that the textual prompt could provide valuable high-level and semantically rich guidance, enhancing keypoint localization. 
conclusion: 
 This paper introduces the novel challenge of Semantic Keypoint Comprehension, which aims to comprehend keypoints across different task scenarios. To address this challenge, we present KptLLM, a novel and unified multimodal large language model designed to adeptly process various modality inputs, facilitating the interpretation of both semantic contents and keypoint locations. Extensive experiments show the superiority of our model in three different tasks for comprehending keypoints, including keypoint semantic understanding, visual prompt-based keypoint detection, and textual prompt-based keypoint detection. We hope this work can open up new possibilities for more finegrained multimodal vision-language understanding and provide valuable insights for future research. 
discussion: 
 Limitations. (1) A major limitation of our work is the model's size and computational efficiency, which is a common challenge for MLLMs compared to traditional vision models. However, this is acceptable because, as a pioneering effort in utilizing LLMs for keypoint comprehension, our main contribution is demonstrating the potential of LLMs to understand and locate pixel-level details at keypoints. (2) Additionally, the datasets used in our experiments have constraints in the diversity of object and keypoint categories for both training and testing. This highlights the need to expand these datasets to validate the model's applicability in more diverse, real-world scenarios.Future Work. (1) Improving the Capacity of the Vision Encoder: Our work follows LLaVA [10] by employing a CLIP-based ViT as the vision encoder. However, some studies [83, 84] have demonstrated that stronger vision encoders can lead to more significant improvements, e.g., DINOv2 [85] .(2) Refining Keypoint Decoding Strategy: Inspired by previous MLLMs for perception tasks [78, 73] , we introduce a special token <keypoint> into the model's vocabulary. When the model generates this <keypoint> token, its hidden embedding is decoded to the corresponding keypoint position.Although this strategy has shown promising results, it remains sub-optimal for user interaction. A more direct approach is to output the keypoint coordinates as textual descriptions. However, training a model to express numerical values in text using cross-entropy loss is challenging because slight deviations in numerical values can lead to significant differences in the generated text. Therefore, it intuitively requires more data for effective training. (3) Expanding Data Scale and Category Diversity: The datasets used in our experiments follow standard benchmarks. However, both the MP-100 dataset [21] for visual prompt-based keypoint detection and the AP-10K dataset [32] for textual prompt-based keypoint detection contain only a small amount of data, which limits the model's generalization performance. Furthermore, the limited diversity of object and keypoint categories greatly reduces the model's applicability, making it insufficient for handling open-world scenarios. A promising direction is to leverage large-scale keypoint datasets for training, such as UniKPT [52] , which could further explore the upper bounds of MLLMs for keypoint comprehension.Broader Impact. The study aims to enhance MLLMs for understanding images at a more granular keypoint level. We also propose a new challenge of keypoint semantic understanding, which holds promise for benefiting tasks such as structural understanding, action recognition, and medical image analysis. Nevertheless, recognizing the potential negative impacts that are common to many MLLMs, our model also carries risks, including the amplification of societal biases and concerns regarding privacy and ethics. To address these issues, we are committed to implementing safeguards, including strict access controls and the establishment of clear usage policies and agreements.NeurIPS Paper Checklist 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: Error bars are not reported because it would be too computationally expensive.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: Please refer to Sec. 6.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes] Justification: Please refer to Sec. 6.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The assets are properly cited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix:  


Decision: **Summary:** This work introduces KptLLM, an MLLM that addresses the challenges of semantic keypoint comprehension, where a model must reason about fine-grained semantic visual information through keypoint. KptLLM shows that using a support set and LoRA on a frozen LLM leads to improvements over other baseline models on keypoint comprehension.

**Strengths/Weaknesses:** The reviewers comment on the strength of the approach for an interesting problem and that KptLLM leads to noticeable improvements over baselines. Most weaknesses focus on a lack of clarity. The wording/figures/tables can be confusing and the authors do not explicitly say the approach using a support set is an instance of few-shot prompting. This can be addressed in the camera ready. Some other weaknesses (e.g. selecting less similar support sets to test generalization) were addressed in the rebuttal. The remaining comments from Reviewer j2Jb should be included more thoroughly in the camera ready as well.

**Recommendation:** I recommend this paper for acceptance. The reviews are positive overall and the authors appear to have thoughtfully addressed reviewers’ questions and concerns through additional experiments and examples. Reviewers have requested more clarity in the Figure 2 and Tables 5 and 6) As acknowledged during the rebuttal, I’d like to see all of the additional experiments and suggestions for language shifts included in the final version. A more in-depth analysis of Llava (expanding on the discussion with Reviewer j2Jb) should be included as well to further justify the contributions of KptLLM.

Reviews: review: paper_summary: paper_summary: The paper proposes to build a MLLM for keypoint detection. They propose three main points to achieve this. First, the paper introduces a new benchmark that measures various types of keypoint detection including both text and visual-prompt-based, as well as traditional keypoint understanding. Next, they propose to build a multimodal LLM, which is designed specifically for keypoint detection that is shown to perform quite well on this task.
strength_weakness: strength_weakness: [W1] Not a lot of novel insights: This is my main, and mostly only, weakness for the paper, but it take a few different forms. Firstly, this is a straightforward "building" of MLLM formula; i.e., convert the task at hand into "instruction format" -> train a vision to LLM connector --> fine-tune LLM; that is used by a lot of recent works. As such, this would be mostly of interest to the domain performance, Keypoint comprehension in this case. This already puts some doubt as to whether this is of interest enough to a broad community. Next, there are also not a lot of modular components that might be broadly useful that hasn't already been studied by other works. Some details below:

W1.1 LLava does almost as good (for supported tasks): The paper shows that, once fine-tuned, the popular LLAVA model does almost as good as the proposed method. The significance of this finding is not thoroughly discussed. Why build a new model at all? Furthermore, there are more capable MLLMs baseline such as R1, R2, R3 that might be more suitable targets for fine-tuning. See 1.2 below as well

W1.2 The proposed method of generating a special token that can be decoded to a key point is also explored before (R1). Besides, there are also various MLLMs that are capable of doing region comprehension (R2, R3...). Those are not sufficiently discussed. If fine-tuned, like LLAVA, how would they do on this task?

W1.3 Is visual prompting a good paradigm for keypoint detection? What is the importance of visual prompting for this task? [EDIT: Clarified in post-rebuttal]

References:
R1: https://arxiv.org/abs/2311.06612
R2: https://arxiv.org/abs/2306.15195
R3: https://openreview.net/forum?id=2msbbX3ydD
questions: questions: Please address the weaknesses above, which also contain questions.
limitations: limitations: N/A
paper_summary: paper_summary: The authors aim to enhance multi-modal LLMs with semantic keypoint comprehension. They introduce a hybrid visual prompting approach using a query and a support input image. In their pipeline, visual features from both images are extracted via a vision encoder model. Additionally, a support keypoint prompt, indicating keypoint positions on the support image, is used. The embeddings from this prompt are combined with the support image embeddings and fed into the LLM. This mixed embedding helps the model detect and understand the keypoint in the input query image, generating the desired keypoint location information.
strength_weakness: strength_weakness: The proposed support image and support keypoint prompt essentially act as few-shot examples for the model. It would be beneficial to explicitly acknowledge this in the paper.

Another limitation of the proposed method is the requirement for a visually similar support image with keypoint locations for each input query image. Have the authors tested the model with support images that are visually different from the query images? How much visual difference can the model handle without a significant drop in accuracy? Conducting an ablation study on this would be crucial to determine the extent of semantic (high-level) or texture (low-level) distribution shifts the method can tolerate without a notable accuracy decline.
questions: questions: The authors should compare their approach to GPT-4V as a baseline to better understand the benefits of the additional steps they have added to their proposed pipeline.

It is unclear where the support image and support keypoint prompts come from during inference. Are these data available to other methods in the experiments?

Figure 2 is somewhat confusing. (Z_q) is fed to the LoRA section, while the rest of the information is fed to the rest of the LLM. Is the figure designed this way on purpose? If so, could you please clarify this?
limitations: limitations: The authors have discussed the limitations of their work.
paper_summary: paper_summary: This paper proposes an LLM-based keypoint localization method that can detect keypoint by visual or text prompt and classify the keypoint category by given its coordinates. Experiments on several public benchmarks demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1.	It seems that the Visual Prompt-based Keypoint Detection and Textual Prompt-based Keypoint Detection are trained separately. However, it is more useful if we can complete these tasks using a single model. So how about jointly training above two tasks? Can they benefit from each other or not?
2.	What’s the meaning of keypoint identifiers in Table 2 and 3? It would be confusing if these is no specific illustration.
3.	Some typos. “Texual”->Textual in line217, 234, 254, etc.
4.	Some claims are not proper. For instance, in line 57-58 “To the best of our knowledge, this is the first work that equip LLMs with the point-level visual perception capabilities.” [A] already introduces LLM into keypoint localization and this paper should emphasize the different aspect such as  visual prompt-based keypoint localization.

[A] D. Wang, S. Xuan, S. Zhang. LocLLM: Exploiting Generalizable Human Keypoint Localization via Large Language Model. CVPR 2024.
questions: questions: In Weakness.
limitations: limitations: This paper discusses the limitation in A.3.
paper_summary: paper_summary: The paper introduces and studies the problem of Semantic Keypoint Comprehension to evaluate the capability of Multimodal Large Language Models (MLLMs) in tackling fine-grained perception and comprehension. It does so by defining three related tasks: (1) keypoint semantic understanding, (2) Visual prompt-based keypoint detection and  (3) Textual Prompt-based Keypoint Detection. The first task targets the semantic interpretation capabilities of objects and keypoints, while the other two tasks involve position detection and semantic comprehension from visual or textual descriptions. In the context of this paper, keypoint detection (a.k.a object pose estimation) is the task of localizing the semantic keypoints of objects.

The authors further present KptLLM, an MLLM for Semantic Keypoint Comprehension based on LLaVA. The model employs an identify-then-detect strategy, where the model first captures the required semantics and then detects the precise keypoint position. The model employs attention to fuse features from image and keypoint encoded prompts. The fused encoding and the encoding of visual inputs are used as additional latent contexts for an LLM, which is finetuned with LORA to generate the prompt. The keypoint position is further regressed with an MLP head. The authors evaluate their proposed model on the MP-100 (diverse and category agnostic) dataset and the AP-10K dataset and compare its performance to state-of-the-art methods for keypoint detection and to pretrained and finetuned LLaVA models.
strength_weakness: strength_weakness: - The properties (and potential limitations) of the MLLM-based approaches in terms of runtime, memory, and scalability to multiple key points, compared to existing state-of-the-art methods, are not discussed
- Lack of details on the methods chosen for cross-comparison and their key properties/assumptions makes it harder to evaluate the contribution
While the paper is overall well structured, the evaluation protocol and details are not always easy to follow
questions: questions: - Can you please provide more details on runtime, memory, and scalability?
- Is the visual encoder trained/fine-tuned or frozen? The textual description mentions a pretrained frozen encoder, while the image shows it to be fine-tuned
limitations: limitations: It is worth elaborating on memory, runtime and scalability aspects of the proposed method.

Comments: title: Post rebuttal comments
Rebuttal by Authors
Official Comment by the Authors of Submission 9290
comment: Thanks for your response. 

- In addressing Q1, we emphasize the significance and potential applications of the proposed semantic keypoint detection task. However, these downstream applications are beyond the scope of our current work and are intended for future exploration. We will adjust some of our claims accordingly.

- Regarding Q2, fine-tuning existing MLLMs for keypoint detection is a non-trivial task, as it involves challenges such as keypoint data preparation and necessary architectural modifications. This highlights the importance of our work, which pioneers the advancement of MLLMs for fine-grained keypoint understanding and localization. We will add some discussions to ensure clarity and avoid any potential misinterpretations.
Thank you for your detailed reply. 

> Response to Q1
I agree with some of the points and am unclear on others. Even the points that sounds good. E.g., "For instance, semantic keypoint understanding equips MLLMs with enhanced visual comprehension and analytical capabilities, essential for tasks such as structural analysis, action recognition, and medical image interpretation." is not shown in the paper. A stronger submission would indeed do a generic eval to highlight these properties as well.

> Response to Q2
I kinda disagree that existing MLLMs cannot be fine-tuned for this task. However, I do not think that is a weakness by itself. MLLMs are very versatile and they could be fine-tuned to a broad range of tasks, that does not, by itself pose a weakness. However, I am worried about a slight overclaim in treating the current work as more than what it does, i.e., enabling a new decoding to take place similar to perception GPT. 

> Response to Q3
Thank you for your response. I understand that it is indeed an established benchmark and I was able to verify that. So, I stand corrected from my earlier comments. 

Overall, some of my concerns have been clarified but I think a major rewrite with additional experiments and discussion is needed. So, I will slightly raise my score (also in light of other reviews and responses) but I am still leaning towards a major revision is needed.
Considering the additional details and clarifications provided by the authors in their rebuttal, along with their commitment to incorporate the rebuttal and reviewers’ comments in the camera-ready version, I believe the manuscript meets the acceptance criteria for a poster presentation. Therefore, I increase my rating to 6.
Thank you for your response. We greatly appreciate all the valuable feedback from the reviewers, which will help us improve our paper. We promise that all reviewers' feedback, including yours, will be incorporated into our final camera-ready version. Specifically, your comments, along with the visualizations and additional experiments in response to Q2 and Q3, will further clarify the advantages of our method for visual prompt-based keypoint detection. Additionally, we appreciate your suggestions regarding the framework diagram, and we will revise it to prevent any misunderstandings.
I appreciate the authors’ rebuttal.

Q5: I did not ask “What is LoRA,” which I already know. I asked why the figure is designed in that particular way; the figure is confusing for the average reader. However, this question and the answer do not affect my rating.

Considering the other reviewers’ feedback and the authors’ rebuttal, I plan to increase my overall rating. Before doing so, I would like to know if the authors plan to include the reviewers’ feedback in their final camera-ready version (specifically for my reviews, Q2 and Q3), as I did not see any promises being made by the authors in their comments.
Thank you for addressing my questions. Overall, the soundness and presentation of the presented work are good: the paper describes an interesting problem and a method for tackling it while demonstrating superior results and performing ablations, so I keep my original rating for these criteria. However, after reading through the comments and concerns made by the other reviewers and the authors' responses regarding the contribution of the task presented, I have decided to change my original rating from excellent to good.
Dear AC and Reviewers,

Thank you for taking the time to review our paper. We are delighted to see that the reviewers recognized the strengths of our work. Specifically, all the reviewers (j2Jb, Hwow, wNmk, 7RyA) acknowledged the great results we achieved. Moreover, Reviewers wNmk and 7RyA appreciated our contributions to advancing the semantic comprehension of MLLMs at a fine-grained level and the adoption of LLMs for in-context learning of keypoint localization.

However, the reviewers also raised some concerns. For instance, Reviewer j2Jb questioned the broader impact of our work, while Reviewer Hwow requested more detail on the visual prompt-based keypoint detection. We have provided a detailed rebuttal addressing these concerns and kindly ask the reviewers to consider our responses when convenient.

Thank you once again for your valuable feedback.

Sincerely,

The Authors of Submission 9290