KptLLM: Unveiling the Power of Large Language Model for Keypoint Comprehension
Authors: Jie Yang
Wang ZENG
Sheng Jin
Lumin Xu
Wentao Liu
Chen Qian
Ruimao Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: Keypoint Detection
Pose Estimation
Multimodal Large Language Model
Primary area: machine_vision

Abstract: Recent advancements in Multimodal Large Language Models (MLLMs) have greatly improved their abilities in image understanding. However, these models often struggle with grasping pixel-level semantic details, e.g., the keypoints of an object. To bridge this gap, we introduce the novel challenge of Semantic Keypoint Comprehension, which aims to comprehend keypoints across different task scenarios, including keypoint semantic understanding, visual prompt-based keypoint detection, and textual prompt-based keypoint detection. Moreover, we introduce KptLLM, a unified multimodal model that utilizes an identify-then-detect strategy to effectively address these challenges. KptLLM underscores the initial discernment of semantics in keypoints, followed by the precise determination of their positions through a chain-of-thought process. With several carefully designed modules, KptLLM adeptly handles various modality inputs, facilitating the interpretation of both semantic contents and keypoint locations. Our extensive experiments demonstrate KptLLM's superiority in various keypoint detection benchmarks and its unique semantic capabilities in interpreting keypoints.

KptLLM: Unveiling the Power of Large Language Model for Keypoint Comprehension
Jie Yang 125 Wang Zeng45
125
Sheng Jin Lumin Xu* Wentao Liu Chen Qian Ruimao Zhang
Sun Yat-sen University "The Chinese University of Hong Kong, Shenzhen 'The University of Hong Kong *The Chinese University of Hong Kong
Sense'Time Research and Ae
https://kptllm..iibba github..... io
Abstract
Recent advancements in Multimodal Large Language Models (MLLMs) have greatly improved their abilities image understanding. However, these models often struggle with grasping pixel-level semantic details, e.g., the keypoints of an object. To bridge this gap, we introduce the novel challenge of Semantic Keypoint Comprehension, which aims to comprehend keypoints across different task scenarios, including keypoint semantic understanding, visual prompt- based keypoint detection, and textual prompt- based keypoint detection. Moreover, we introduce KptLLM, a unified multimodal model that utilizes an identify-then-detes strategy to effectively address these challenges. KptLLM underscores the initial discernment of semantics keypoints, followed by the precise determination of their positions through chain-of-thouge process. With several carefully designed modules, KptLLM adeptly handles various modality inputs, facilitating the interpretation of both semantic contents and keypoint locations. Our extensive experiments demonstrate KptLLM's superiority in various keypoint detection benchmarks and its unique semantic capabilities in interpreting keypoints.
1 Introduction
Recent advancements in deep learning and natural language processing have facilitated the rise of Large Language Models (LLMSs) that display human-like fluency in text comprehension and generation [ 2
3 5 77. By incorporating visual information. researchers have developed Multimodal Large Language Models (MLLMs) [8 9
I0 [T] specifically designed for visual- language tasks, showcasing remarkable abilities in image understanding. However, these models encounter difficulties in capturing fine-grained semantic details, particularly at the point level, which crucial for various real-world applications. The exploration of MLLMs for keypoint comprehension remains under-explored the literature.
Keypoint detection is fundamental aspect of computer vision that supports various applications such as controllable image/video generation [3 14 15), human-centric perception [16 [7], and AR/VR systems 18 [9 20 Initially, research in this field focused on closed-set problems, aiming to predict certain object category (e.g. human body). As the demand for generalization grew, researchers started investigating the detection of keypoints for novel objects by providing visual prompts (i.e., a support image novel object with its keypoint definitions) 211 22) utilizing textual prompts (i.e., keypoint names) 23). Despite significant
the locations of predefined semantic keypoints
progress in these areas, existing models still fall short of achieving genuine semantic comprehension
"Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

What the keypoint <keypoint> name of the object image?
keypoint name": left ear;
(a) Keypoint Semantic Understanding
g
What keypoint <keypoint> name the object the image? Please accurstely locate the position of the keypoint.
keypointt name": left ear: "keypoint position": <keypoints;
(b) Visual Prompt-based Keypoint Detection
The Jeft situated the side
"keypoint name": ear;
head, specifically the lateral aspect.
"keypoint position': <keypoints;
Where is the left ear of the object in this image?
(c) Textual Prompt-based Keypoint Detection
Figure 1: This work aims to address the problem of semantic keypoint comprehension, which aims to understand keypoints across different task scenarios: (a) Keypoint Semantic Understanding takes the object image and a keypoint prompt (i.e., the position of the target keypoint) as inputs, then generate responses that interpret keypoint semantics; (b) Visual Prompt-based Keypoint Detection takes query image and support image with a keypoint prompt as inputs and then outputs the corresponding keypoint positions and semantics of the query image; (c) Textual Prompt-based Keypoint Detection utilizes detailed descriptions of keypoints through extensive text, to perform more generalizable keypoint detection.
of keypoints akin to humans. These models primarily rely on direct learning of visual patterns for keypoint localization through extensive data fitting, while neglecting semantic understanding of the keypoints, thus leading to misinterpretation of the prompts and inaccurate predictions. Moreover, the input-output structures are designed in fixed and predefined formats, restricting their usage to predetermined methods and impeding the flexibility required for interfacing with users.
Motivated by the aforementioned challenges, this paper delves into a more comprehensive problem of Semantic Keypoint Comprehension to evaluate the model capability of comprehensively understanding keypoints both visually and semantically. As shown Fig. [I] we investigate three distinct capabilities (a) Keypoint Semantic Understanding aims to infer the desired keypoint semantics, given the target image and keypoint prompt (i.e., the position of the target keypoint) as inputs. It provides the potential for an Al model with high-level visual understanding and analytical capabilities, crucial for tasks such as structural comprehension, action recognition, and medical image analysis.
via different task instructions:
(b) Visual Prompt-based Keypoint Detection, also referred to as category-agnosse pose estimation, takes a query image and labeled support image with the keypoint annotation as inputs and then outputs the corresponding keypoint positions in the query image. This capability requires the model to acquire keypoint definitions from visual prompts, enabling it to perform cross-class and cross-keypon keypoit localization tasks using sample images provided by users. (c) Textual Prompt-based Keypoint Detection, also known as open-vocabulary keypoint detection, aims to utilize detailed descriptions of keypoints through extensive text for keypoint localization. The keypoint detectors directly receive the human language guidance, facilitating keypoint localization on arbitrary object and keypoint categories in zero-shot manner.
We introduce KptLLM, a novel framework that utilizes an identify-then-detee strategy to address the challenging problem of semantic keypoint comprehension. It formulates all three capabilities depicted in Fig. [ by first identifying the semantic meaning of keypoints and then detecting their positions via a chain-of-tte -fhohuugttthhht approach, akin to human cognition. KptLLM is unified framework

---

that comprises four key components designed to accommodate various modality inputs and infer both the semantics and location of the keypoint. Specifically. we first extract visual features of both query and support images to obtain query visual tokens and support image features. Secondly, we encode the support keypoint prompt, which describes the position of keypoint on the support image, to generate keypoint prompt embedding. Thirdly, prompt-oriented features are derived by integrating support image features with keypoint prompt embedding, and are utilized to form keypoint prompt tokens. Lastly, LLMs take query visual tokens, keypoint prompt tokens, and task-related language tokens as input, and then generate the semantic description of the target keypoint and its corresponding position on the query image. By hamessing commonsense knowledge in LLMs, KptLLM can assist in keypoint localization of novel object categories, potentially leading to enhanced generalizability in performance. In addition, the chain- of of-thhught design elicits the powerful keypoint understanding capabiliites of L.LMs, which helps to distinguish visually ambiguous keypoints (e.g. left and right arms). Extensive experiments demonstrate LptL.LM's superiority on semantic keypoint comprehension, showcasing its unique semantic understanding capabilities in interpreting keypoints and ofate-of-the-art performance in various keypoint detection benchmarks.
In summary, the contributions of this work are three-fold: (I) We pioneer the investigation of a novel problem in semantically interpretable keypoint analysis, termed Semantic Keypoint Comprehension, which aims to enhance MLLMs with improved image understanding at a finer-graine -rainer keypoint level; (2) We introduce KptLLM, unified multimodal model that utilizes effectively address three tasks of semantic keypoint comprehension. KptLLM underscores the initial discerment of semantic significance in keypoints, followed by the precise determination of their positions through a chain-of-thouuht process. (3) We demonstrate KptLLMMs superiority in various existing keypoint detection benchmarks and its unique semantic capabilities in interpreting keypoints. We hope our work could inspire future research on keypoint understanding and localization, while also fostering enhanced human-Al interface in fine-grained visual understanding.
thent-etett strategy to
2 Related Work
2.1 Keypoint Detection
Keypoint detection, also referred to pose estimation, focuses on localizing the 2D keypoints of objects in the image. Traditional models for keypoint detection are typically designed for a single
category, e.g., human (24 22 26 27, 28 29 30]. animal [3]
32
33 and clothes (34]. Based on the
localization strategy, existing methods are generally divided into regression
methods [35] 36
37
38
39 40 and heatmap-based methods [41 42
45
44
45
46
49 50 [1]. More recently,
methods that can recognize and localize keypoints for unseen object categories in the training datasets, gaining increasing attention from the community.
aategory-agnostte pose estimation 21 22
also referred to few-shot keypoint detection, aims to estimate the pose of any category in query images with visual prompts (i.e., a few support images of novel class and its corresponding keypoint annotations). Another line of research explores open-vocabulary keypoint detection [23 [2], which aims to localize keypoints based on text prompts in shot settings. In this work, we investigate semantic keypoint comprehension and propose a novel unified framework to comprehend keypoints across three different task scenarios, including (a) keypoint semantic understanding; (b) visual prompt-base based keypoint detection; (c) textual prompt-based keypoint detection.
2.2 Multimodal Large Language Model
Inspired by the success of Large Language Models (LLLMs) 1
53
4
54 5 6. 7 researchers are
exploring ways to transfer the formidable capabilities of LLMs into the realm of vision, developing Multimodal Large Language Models (MLLMs) (55]
8 9 I0 56 57
I1] 12 58 59
60
61
62]
19. These models exemplify an autoregressive mechanism predicated on a transformer decoder
architecture 663 The integration of visual representation from vision encoders [64
65 into the
domain of LLMs ushers a new era of visual comprehension and reasoning. Such integration is predominantly facilitated through a Multilayer Perceptron (MLP) that seamlessly transforms visual features into the input embedding space of LLMS [10 56 57), or via a cross-attention mechanism that attends to visual contents through a series of attention layers [55 g However, most of these VLMS can only provide text outputs, inhibiting the complex applications requiring detailed visual perception. VisionLLM [66] tackles a range of conventional vision-centric tasks by instruction
3

---

Linear
C00
LoRA
heypolnt name" left ear: heypornt postion:: <oeypoint>
Visual Encoder
Fq
Zq
Query Image 1
Fg
Prampt Feature Extractor
Mkpt
Support Image Ig
Fp
2p
Pre-trained LLM
FFN
Prompt Encoder A
What the kespoint <keypoint> name of the dbject in the image?
Please also acourately locala position of the keypoint.
Support Keypaint X
The situsted on the side head, specifcclly on the lateral espect. Where the (eft oar the object in image?
Figure 2: We introduce KpLLLM, unified framework designed to address three tasks of semantic keypoint comprehension: ① Keypoint Semantic Undertanding, which processes a support image I, and support keypoint prompt generate responses that interpret the semantic information of the specified keypoint; Visual Prompt-based Keypoint Detection aims to detect the corresponding keypoint in the query image Ig based on the understanding of the support keypoint prompt: 3 Textual Prompt-based Keypoint Detection leverages textual keypoint descriptions to directly infer the corresponding keypoint positions in the query image.
tuning LLMs. However, it may fall short of fully leveraging the comprehensive reasoning faculties of LLMs. Kosmos-2 [67], Qwen-VI VL [68] and DetGPT (69 further exploit the power of LLMs to enable
user-guided detection. Moreover, GPT4Rol 770, Ferret [71], Shikra (72), and PerceptionGPPT [73 innovates by incorporating spatial boxes or masks as inputs and training with region-text pairs, offering region-level visual comprehension. Notably, concurrent work, LocLLM (74, utilizes LLMs for human keypoint localization via textual description. In contrast, we take a step further by enabling LLMs comprehend keypoints of various objects via multi-modal (e.g., textual visual) prompts under different task formulations. This advancement not only broadens the utility of MLLMs for keypoint detection but also enhances interpretive depth, allowing for a more comprehensive understanding and grounding across wider range of visual information.
3 Methodology
This section introduces our proposed unified framework, referred KptLLM, which effectively addresses three semantic keypoint comprehension scenarios. As illustrated in Fig. 2 multiple images (query and support images) along with a support keypoint prompt (i.e., the position of the target keypoint in the support image) and textual user instructions as the input. The output comprises both the response text and the desired keypoint position. Specifically, KptLLM comprises four key architectural components: (1) A visual encoder that extracts features from both query and support images (see Sec. 3.1 T (2) A prompt encoder that converts support keypoint prompts into prompt embeddings (see Sec. 3.2 (3) A prompt feature extractor that derives prompt-oriented
KptLLM accepts
features from the corresponding image features (see Sec. 3.3): (4) A pre-trained LLM that processes
multimodal tokens for keypoint comprehension (see Sec. 3.4
3.1 Visual Encoder
The Visual Encoder is designed process two types of images in parallel: query and support images. Generally, it receives an input image I E IR #WW x3 and generates a feature map F = V(I) e Rhxwxd xxd Here, d represents the feature dimension, and h and w are the spatial dimensions obtained by downsampling the original image dimensions H and W.
Query Image.
The query image represents the image that to be analyzed. We extract its spatial features through the vision encoder V, resulting in F,. Following LLAVA [10, we apply linear layer to project F, into language space: n = Linear
Linear(F,)) (F₂). As a result, query visual tokens aligned with the LLM dimension obtained and fed to the LLM.

---

Support Image. The support image serves as a reference example. We extract its spatial features, which are represented F., Unlike the query image features, F, not directly input into LLM. Instead, it is processed by the prompt feature extractor to derive prompt-oriented features.
3.2 Prompt Encoder
In addition to processing images, we need to incorporate an additional prompt consisting of 2D coordinates x E R?? which describes the keypoint location within the image. Inspired by SAM (75), we introduce prompt encoder to adapt this prompt input to be aligned with the image feature space F. The prompt encoder encodes the keypoint coordinates using a sine-cosine position embedding (PE), followed by Multi-Layer Perceptron (MLP):
Fp MLE(()(x)).
3.3 Prompt Feature Extractor
The Prompt Feature Extractor is designed to extract the prompt-specific features from image features. As illustrated Fig. 2 the semantics of the keypoint prompt directly correspond to the support image. We initialize the prompt feature extractor with two-layer transformer that incorporates the cross-attention mechanism (CrossAttnlayere)) This mechanism employs F, as the query and F, as the key and value to extract keypoint-specific visual features indicated by the prompt:
2p =
Fs),
(2)
where Zy denotes the syypoint-speciffee visual features. In essence. compared with average pooling- based feature extraction method [21]. the prompt feature extractor is trainable and capable of incorporating global image features enhance keypoint identification. This is particularly beneficial for distinguishing mirror-symmetric keypoints, such as the left and right eyes, which can be highly ambiguous when relying solely on local image features. Our ablation study demonstrates the performance improvements achieved through the utilization of this component.
3.4 Multimodal LLM for Keypoint Comprehension
Given query image and an optional prompt specifying the keypoint of interest, goal is to generate textual descriptions and keypoint locations that convey fine fine-grained keypoint information within the image. Recognizing the exceptional ability of LLMs in handling multimodal tokens for different perception tasks [76 10 73, 78), we further leverage LLM for keypoint comprehension, which could effectively process various inputs: (1) the visual tokens 2, of the query image, (2) the prompt tokens Zp. and (3) a sequence of language tokens t, which depend on the three semantic keypoint comprehension scenarios.
Keypoint Semantic Decoding. We design the model to directly generate textual descriptions that interpret keypoint semantics, following the standard approach used by LLMS for text generation. Generally, the architecture of an LLM typically comprises Transformer layers followed by final Feed Forward Network (FFN). The latent embedding which captures the fused multimodal information, can be computed as:
Transf Tr ormerLayers Zp: t)):
where [Zy. Zp. t] denotes the concatenation of the visual, prompt, and language tokens. This embed- ding u then passed through the FFN and a Softmax function to generate the probability distribution over the vocabulary for the next token:
p = Sof Softmax(FFF(u)))
Keypoint Position Decoding, Inspired by 77 73
we introduce a special token, <keypoint>, into the vocabulary. Consequently, the 2D keypoint position y can be computed from the output latent embedding Ukpe of the special token using another FFN prediction head:
y FFN Ukpt) E IR?.
Indentify-thent then--etentt (ItD) Strategy. Instead of relying on extensive data fitting to learn fixed key- point localization patterns, we adopt an approach where the LLM initially identifies the semantics of keypoints. Subsequently, chain-of-thought process employed to accurately detect the locations of these keypoints. Experiments demonstrate improved performance compared to alternative baselines.
5

---

3.5 Training and Inference Details
To retain the learned general knowledge of the pre-trained LLM, we employ LoRA 779 for efficient fine-tuning of LLM, while fully fine-tuning other modules of the framework. The training and inference processes for different tasks are outlined below.
Keypoint Semantic Understanding. As shown in Fig. (a), this task focuses on extracting semantic textual information associated with specific keypoints within image. The training objective is to minimize the language modeling loss, computed as the cross-entropy loss over the vocabulary of the LLM's tokenizer. Specifically, the loss function defined as:
L = Lim (a, å),
(6)
where a is the text response predicted by the model, and is the ground -ruth text response. During inference, given an image provided by the user and the corresponding keypoint position prompt, our model comprehends and generates the semantic meaning of the specified keypoint.
Visual Prompt-based Keypoint Detection. This task involves simultaneously comprehending the semanties of keypoints, generating textual descriptions of this understanding, and precisely localizing the keypoint coordinates, as shown in Fig. (b). The overall training function further incorporates the L1 loss for keypoint regression:
C= A 3 + Lim (a, a),
where ÿ is the ground-truth keypoint position, and À is the loss weight that balances the learning of keypoint regression and text generation (λ in our implementation). During inference, the user provides two images: one as the query image for testing and the other as the support image for reference. Additionally, the keypoint definition for the support image should be provided as the support keypoint prompt. Our model then comprehends the semantics of the desired keypoint and detects its corresponding position in the query image.
Textual Prompt-based Keypoint Detection. As illustrated in Fig. (c), this task aims to accurately localize keypoints based on the detailed keypoint descriptions. The loss function aligns with that of the visual prompt-based keypoint detection. During inference, users have the option to provide detailed descriptions of the desired keypoints or simply the keypoint names, based on which our model can detect the corresponding keypoints.
4 Experiments
4.1
Experimental Setup
4.1.1 Datasets
In our experiments. we employ two datasets to evaluate the semantic keypoint comprehension in three scenarios: (1) The 1P- dataset (21) for both Keypoint Semantic Undertanding and Visual Prompt- based Keypoint Detection: This dataset is a pioneering dataset for
pose estimation, which encompasses 100 different object categories with over 20,000 instances. The number of keypoints varies across categories, ranging from 8 to 68. Following the protocols established by POMNet (2I) the dataset is divided into five distinct splits to ensure comprehensive coverage across different model training and validation scenarios. Each split contains all 100 categories, with 70 for training, 10 for validation, and for testing. The splits are carefully designed to avoid category overlap, maintaining the independence and integrity of training and testing scenarios.
The dataset comprises 23 animal families and 54 species, totaling 10,015 images. Each image is annotated with 17 keypoints, including two eyes, one nose, one neck, two shoulders, two elbows, two knees, two hips, four paws, and one tail. We follow CLAMP 23 to assess the models' ability to generalize to previously unseen animal species within zero-shot learning paradigm. We establish two experimental scenarios based on the taxonomic relationship between the species in the training and test sets sttissssciiiillly,. whether they belong to the same animal order. Species within the same order typically share similar visual characteristics, whereas those from different orders exhibit greater diversity in appearance. These scenarios enable us to assess how different methods perform when generalizing to unseen species under varying conditions. Following CLAMP, we assign Bovidae and Canidae as the training and
(2) The AP-10K dataset 32 for Textual Prompt-based Keypoint Detection:

---

Table 2: Visual Prompt-based Keypoint Detection MP-100 [2 dataset. Performance (PCK)
under 1-shot and 5-shot settings.
1-shot
5-shot
Methods
Split
Split2
Split3
Spli14
Split5
Meun
Split1
Spli12
Split3
Split4
Split5
Meun
ProtoNet 330
46.05
40.84
49.13
43.34
44.54
44.78
60.31
53.51
61.92
58.44
58.61
58.56
MAML 81
68.14
54.72
64.19
63.24
57.20
61.50
70.03
55.98
63.21
64.79
58.47
62.50
Finetune 82
70.60
57.04
66.06
65.00
59.20
63.58
71.67
57.84
66.76
66.53
60.24
64.61
POMNet 21
84.23
78.25
78.17
78.68
79.17
79.70
84.72
79.61
78.00
80.38
80.85
80.71
CapeFormer
22
89.45
84.88
83.59
83.53
85.09
85.31
91.94
88.92
89.40
88.01
88.25
89,30
Kpd.LMM
91.66
86,58
86.19
84.76
86,32
87.10
93.17
89,45
90.08
88.74
89.52
90,19
test sets for the different order setting, while Canidac and Felidae are chosen as the training and test sets for the same order setting.
4.1.2 Evaluation & Metries
Different evaluation methods and metrics are used for different tasks of semantic keypoint compre- hension. (1) Keypoint Semantic Undertanding: We use the MP-100 dataset (21) (Split-1), with the keypoint semantic labels adopted from X-Pose [2). Some keypoints are excluded from the evaluation due to ambiguity or inadequacy in their descriptions, such as those used to describe the collar in the clothing category. By aggregating the results of tested keypoints, we derive corresponding accuracy rates (%). (2) Visual Prompt-based Keypoint Detection: We employ the Probability of Correct Keypoint (PCK) metric, which the standard evaluation measure for this task. Consistent with POMNet 21, we uniformly set the PCK threshold to 0.2 across all categories. Additionally, compute and report the average PCK over all five data splits to provide a comprehensive indication of the model's s overall effectiveness. (3) Textual Prompt-based Keypoint Detection: Following CLAMP, we employ average precision (AP) as the primary metric for AP-10K. This metric is computed based on the object keypoint similarity (OKS). For detailed protocol definitions, please refer to (24)
4.1.3 Implementation Details
Architecture. We utilize LLaVA-VI.5-7B [0] as our base model, which incorporates the ViT-based visual encoder of CLIP for image encoding and Vicuna-7B (fine-tuned from Llama-2) the LLM backbone. We employ LoRA for efficient fine-tuning LLM. Instead, other modules, including visual encoder, prompt encoder, prompt feature extractor, and series of linear layers and feed forward networks, undergo full fine-tuning. The input image only contains a single object of interest, cropped according to the ground-truth bounding box and resized to 336 636,.36. consistent with
Training Details. LoRA parameters are configured with a rank of 128 and an alpha of 256. Optimiza- tion is conducted using Adam W. with learning rate of 2e-4 and weight decay of O. We utilize 8 NVIDIA A 1100-80G GPUs for training, and use the DeepSpeed engine to enhance training efficiency, Each GPU operates with batch size of 16, and employ gradient accumulation step of 1.
4.2 Keypoint Semantic Understanding
As depicted in Tab. we present the accuracy for key- point semantic understanding on MP-100 21 Split- I set. To facilitate comprehensive comparison, highlight the keypoint area in the image and feed the processed image. along with the task instruction, into LLaVA [10]. We report the performance of both the original LLaVA model and version fine-tuned on the MP-100 dataset. The original LLaVA performs notably poorly in grasping keypoint semanties, indicating the inadequacy of traditional multimodal large language models in capturing tne-grained semantic details. Conversely, the fine-tuned L.LVA VA demonstrates significantly enhanced performance, thereby validat- ing the efficacy of our training pipeline. Furthermore, our Kpt.L.M surpasses the fine-tuned L. by substantial margin, particularly terms of keypoint accuracy (83% vs 72%). demonstrates the effectiveness of our keypoint prompt token in guiding attention to the fine-grain fine-grained keypoint area.
Table 1: Keypoint Semantic Understand- ing on MP-100 (Split-1) [21] * means
LLaVA is finetuned using LoRA.
Methods
Accuracy
LLaVA [10
3%
LLaVA
10
72%
KptLLM
83%
7

---

Table 3: Visual Prompt-based Keypoint Detection for super-category evaluation MP..
100 221] Experiments are conducted under the I-shot setting.
Method
Human Body
Human Face
Vehicle
Furniture
ProtoNet 80
37.61
57.80
28.35
42.64
MAML 8
51.93
25.72
17.68
20.09
Fine-tune
52.11
25.53
17.46
20.76
POMNet 21
73.82
79.63
34.92
47.27
CapeFormer 22
83.44
80.96
45.40
52.49
KptLLM
83,91
83.37
46.23
54.05
(a) Support with Keypoints
(b) Query Image with Model P Predictions
Figure 3: Using the same support image with support keypoints, our model could effectively detect different query images with various poses, object appearances, and environments.
4.3 Visual Prompt-based Keypoint Detection
1-shot & 5-shot Evaluation. We compare our method with the previous visual prompt- bbsse methods
ProtoNet [80), MAML [8I), Fine-tune [22, POMNet [21], and CapeFormer [22]. Tab. presents PCK results of different approaches on the MP-100 dataset under both 1-shot and 5-shot settings. Compared with previous methods, KptLLMM showcases the potential MILM in detecting keypoints through the use of visual prompts, consistently outperforming across all settings and data splits. More importantly, we integrate keypoint semantic understanding into the output response, introducing novel functionalities for comprehending the semantic aspects of support image keypoints.
Cross Super Category Evaluation. To thoroughly assess generalization across markedly different categories. we conduct cross superccuurra evaluation following the protocol of POMNet [21]. While the MP-100 dataset ensures that training, validation, and test categories are non-overlapping. some categories may still exhibit similar features, e.g., body characteristics commonly shared among different quadruped animals. To address this, we designate four supercategories human face, human body, vehicle, and furniture- from the MP-100 dataset test categories. The remaining categories are utilized for training. allowing us to better evaluate the model's ability to generalize across significantly diverse categories. As shown in Tab. 3 KptLLM consistently outperforms previous methods, highlighting the robustness and excellent generalization ability of our proposed method.
Qualitative Results. For the visual prompt-based based keypoint detection task, the input necessitates support image of the object to be tested, as well as keypoint positions that represent the definitions of those keypoints. In this study, we examine how the visual disparity between support images and query images affects the model's performance. As depicted in Fig- 3 our model capable of effectively detecting keypoints in various query images when provided with the same support image and its corresponding keypoints. This effectiveness is maintained even in the presence of differences in object poses, appearances, and environmental conditions.

---

Table 4: Textual Prompt-based Keypoint Detection AP-10K [2].
Method
Train
Test
AP
APso
AP
APM
APr.
AR
SimpeBasellnee
Bovidae
Canidae
41.3
79.4
36.4
26.8
41.3
49.1
CLAMP 23
Bovidae
Canidae
46.9
84.4
45.6
30.3
46.9
53.8
KptLLM
Bovidae
Canidae
62.2
94.5
68.7
26.0
62.3
65.8
SimpleBaseline 48
Canidae
Felidac
39.6
74.1
34.5
9.5
40.2
46.6
CLAMP 23
Canidae
Felidae
48.4
85.7
44.0
13.6
48.9
55.1
KptLLLM
Canidae
Felidae
70.2
97.8
82.2
49.2
70.6
74.7
Table 5: Ablation study of se- mantic understanding.
Table 6:
Ablation study of prompt feature extraction.
Table The effect on combin- ing visual and textual prompts.
Strategies
PCK
Strategies
PCK
Visual
Textual
PCK
w/o
87.68
Average Pooling
89.78
91.66
w/ ltD
91.66
Ours
91.66
92.18
4.4 Textual Prompt-based Keypoint Detection
The results are presented in Tab. Compared to previous textual prompt-based model-CLAMP 23
KptLLM achieves superior cross-species generalization. Specifically, our model demonstrates a 15.3 average precision (AP) improvement in the different order setting and a 21.8 AP increase in the same order setting. Notably, our model performs better in the same order setting, where species often share similar visual characteristics. Overall, we show that leveraging detailed keypoint descriptions through comprehensive text, combined with commonsense knowledge from LLMs, effectively enhances generalizable performance in keypoint localization.
4.5
Ablation Study
In this subsection, we perform ablation study on the design choices our model. The experiments are conducted on the visual prompt-based keypoint detection task using the MP-100 Split-1 setting with the PCK metric reported.
Podentify-then-Deeeee (ItD) Strategy. KptLLM follows the Deentify-then-detee paradigm, where the model learns to first interpret the semantic information the keypoint to detected, and then predict the precise location of the keypoint. In Tab. 5 we validate the effectiveness of our Itl) strategy. We observe notable enhancement, which can be attributed the inter-task synergy that arises from the ItD mechanism.
Prompt Feature Extractor. In Tab. we compare our prompt feature extractor (Sec. 3.3 with the average pooling based feature extraction method [21] The results show that our prompt feature extractor significantly outperforms the baseline method (91.66 89.78). which validates the efficacy of our prompt feature extractor in enhancing focus on fine-grained keypoint areas.
Combining Visual and Textual Prompts. In Tab. 7 rather than relying solely the visual prompt for localization, we further incorporate the textual prompt to demonstrate the effect this combination. The improved results indicate that the textual prompt could provide valuable high-level and semantically rich guidance, enhancing keypoint localization.
5 Conclusion
This paper introduces the novel challenge of Semantic Keypoint Comprehension, which aims to comprehend keypoints across different task scenarios. To address this challenge, we present KptLLM, a novel and unified multimodal large language model designed to adeptly process various modality inputs. facilitating the interpretation of both semantic contents and keypoint locations. Extensive experiments show the superiority of our model in three different tasks for comprehending keypoints, including keypoint semantic understanding, visual prompt- based keypoint detection, and textual prompt-based keypoint detection. We hope this work can open up new possibilities for more fine- grained multimodal vision- langugee understanding and provide valuable insights for future research.

---

Discussion
Limitations. (1) A major limitation of our work is the model's size and computational efficiency, which is a common challenge for MLLMs compared to traditional vision models. However, this is acceptable because, as a pioneering effort in utilizing LLMs for keypoint comprehension, our main contribution demonstrating the potential of LLMs to understand and locate pixel-level details at keypoints. (2) Additionally, the datasets used in our experiments have constraints in the diversity of object and keypoint categories for both training and testing. This highlights the need to expand these datasets to validate the model's applicability in more diverse, real-world scenarios.
Future Work. (1) Improving the Capacity of the Vision Encoder:
Our work follows LLaVA [0]
by employing CLIP-based ViT as the vision encoder. However, some studies 83 84 have demon- strated that stronger vision encoders can lead to more significant improvements, e.g. DINOv2
(2)
Refining Keypoint Decoding Strategy: Inspired by previous MLLMs for perception tasks 78 7
we introduce a special token <keypoint> into the model's vocabulary. When the model generates this <keypoint> token, its hidden embedding is decoded to the corresponding keypoint position. Although this strategy has shown promising results, it remains sub-optimal for user interaction. A more direct approach is output the keypoint coordinates as textual descriptions. However, training a model to express numerical values in text using cross-entropy loss is challenging because slight deviations in numerical values can lead to significant differences in the generated text. Therefore, it intuitively requires more data for effective training. (3) Expanding Data Scale and Category Diversity: The datasets used in our experiments follow standard benchmarks. However, both the MP-100 dataset [21] for visual prompt- keypoint detection and the AP-10K dataset [32) for textual prompt-based keypoint detection contain only small amount of data, which limits the model's generalization performance. Furthermore, the limited diversity of object and keypoint categories greatly reduces the model's applicability, making insufficient for handling open-world scenarios. A promising direction is to leverage large-scale keypoint datasets for training, such as UniKPT [52). which could further explore the upper bounds of MILMs for keypoint comprehension.
Broader Impact. The study aims to enhance MLLMs for understanding images at a more granular challenge of keypoint semantic understanding, which holds
keypoint level. We also propose promise for benefiting tasks such as structural understanding, action recognition, and medical image analysis. Nevertheless, recognizing the potential negative impacts that common many MLLMs, our model also carries risks, including the amplification of societal biases and concerns regarding privacy and ethics. To address these issues, we committed to implementing safeguards, including strict access controls and the establishment of clear usage policies and agreements.
Acknowledgements
The work is partially supported by the Young Scientists Fund of the National Natural Science Foundation China under grant No.62106154, by the Natural Science Foundation of Guangdong Province, China (General Program) under grant and by the Guangdong Provincial Key Laboratory Technology Program ICYJ202208181 of Big Data Computing, The Chinese University of Hong Kong (Shenzhen).
1515011524, and by Shenzhen Science and
References
[1] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-19011 1901, 2020.
[2] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08874, 2023.
[3] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint 2rXiv:2302.13971 2023.
10

---

[4] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint urXiv:2307.092860 2023.
[5] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-
113, 2023.
[6] Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2334.03227,, 2023.
[7] Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rívière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403. 000.:2990 2024.
[8] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742. PMLR, 2023.
[9] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. Advances Neural Information Processing Systems, 36, 2024.
[10] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023.
[11] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304. 14178, 2023.
[12] Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang. Fanyi Pu, Jingkang Yang. Chunyuan Li, and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint wrXiv:2306.05425, 2023.
[13] Jianzhu Guo, Dingyun Zhang, Xiaoqiang Liu, Zhizhou Zhong, Yuan Zhang, Pengfei Wan, and Di Zhang. Liveportrait: Efficient portrait animation with stitching and retargeting control. arXiv preprint arXiv:2407.03168. 2024.
[14] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 3836-3847, 2023.
[15] Xuan Ju, Ailing Zeng, Chenchen Zhao, Jianan Wang, Lei Zhang, and Qiang Xu. Humansd: native skeleton-guie guided diffusion model for human image generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15988-15998, 2023.
[16] Jie Yang. Chaoqun Wang, Zhen Li, Junle Wang, and Ruimao Zhang. Semantic human parsing via scalable semantic transfer over multiple label domains. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19424-19433, 2023.
[17] Jie Yang, Bingliang Li, Fengyu Yang, Ailing Zeng, Lei Zhang, and Ruimao Zhang Boost- ing human-object interaction detection with text-to-image diffusion model. arXiv preprint arXiv:2305. /2252, 2023.
[18) Ling-Hao Chen, Shunlin Lu, Ailing Zeng, Hao Zhang, Benyou Wang, Ruimao Zhang, and Lei Zhang. MotionlIm: Understanding human behaviors from human motions and videos. arXiv preprint arXiv:2405.203-40, 2024.
[19] Jie Yang, Xuesong Niu, Nan Jiang, Ruimao Zhang, and Siyuan Huang. F-hoi: Toward fine- grained semantic-aligned 3d human-object interactions. arXiv preprint arXiv:2407. 12435, 2024.
11

---

[20] Shunlin Lu, Ling-Hao Chen, Ailing Zeng, Jing Lin, Ruimao Zhang, Lei Zhang, and Heung. Yeung Shum. Humantomato: Text-aligned whole-body motion generation. arXiv preprint arXiv:2310. 12978, 2023.
[21] Lumin Xu, Sheng Jin, Wang Zeng, Wentao Liu, Chen Qian, Wanli Ouyang, Ping Luo, and Xiaogang Wang. Pose for everything: Towards category-agnostic pose estimation. In European conference computer vision, pages 398-416 416.4 Springer. 2022.
[22] Min Shi, Zihao Huang, Xianzheng Ma, Xiaowei Hu, and Zhiguo Cao. Matching is not enough:
A two-stage framework for category-agnostie pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7308-7317, 2023.
[23] Xu Zhang, Wen Wang, Zhe Chen, Yufei Xu, Jing Zhang, and Dacheng Tao. Clamp: Prompt- based contrastive learning for connecting language and animal pose. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages
2023.
[24] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Eur: Conf. Comput. Vis., 2014.
[25] Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele. 2d human pose estimation: New benchmark and state of the art analysis. In IEEE Conf. Comput. Vis. Pattern Recog., 2014.
[26] Jiefeng Li, Can Wang. Hao Zhu, Yihuan Mao, Hao-Shu Fang, and Cewu Lu. Crowdpose: Efficient crowded scenes pose estimation and new benchmark. In IEEE Conf. Comput. Vis. Pattern Recog., 2019.
[27] Sheng Jin, Lumin Xu, Jin Xu, Can Wang. Wentao Liu, Chen Qian, Wanli Ouyang, and Ping Luo. boole-body human pose estimation in the wild. In Eur. Conf. Comput. Vis., 2020.
[28] Christos Sagonas, Epameinondas Antonakos, Georgios Tzimiropoulos, S. Zafeiriou, and M. Pan- tic. 300 faces in-the-wild challenge: database and results. Image and Vision Computing, 2016.
[29] Wayne Wu, Chen Qian, Shuo Yang, Quan Wang, Yici Cai, and Qiang Zhou. Look at boundary: A boundary-aware face alignment algorithm. In IEEE Conf. Comput. Vis. Pattern Recog., 2018.
[30] Gyeongsik Moon, Shoou-I Yu, He Wen, Takaaki Shiratori, and Kyoung Mu Lee. Interhand2.co 6m: dataset and baseline for 3d interacting hand pose estimation from single rgb image. Eur. Conf. Comput. Vis., 2020.
[31] Jinkun Cao, Hongyang Tang, Hao-Shu Fang, Xiaoyong Shen, Cewu Lu, and Yu- Wi-ging Tai. Cross-domain adaptation for animal pose estimation. In Int. Conf. Comput. Vis., 2019.
[32] Hang Yu, Yufei Xu, Jing Zhang, Wei Zhao, Ziyu Guan, and Dacheng Tao. Ap-10k: A benchmark for animal pose estimation in the wild. arXiv preprint arXiv:2108. 12617, 2021.
[33] Muhammad Haris Khan, John McDonagh, Salman Khan, Muhammad Shahabuddin, Aditya Arora, Fahad Shahbaz Khan, Ling Shao, and Georgios Tzimiropoulos. Animalweb: A large- scale hierarchical dataset of annotated animal faces. In IEEE Conf. Comput. Vis. Pattern Recog.. 2020.
[34] Yuying Ge, Ruimao Zhang, Xiaogang Wang, Xiaoou Tang, and Ping Luo. Deepfashion2: versatile benchmark for detection, pose estimation, segmentation and re-identification of clothing images. In IEEE Conf. Compur. Vis. Pattern Recog., 2019.
[35] Jiefeng Li, Siyuan Bian, Ailing Zeng, Can Wang, Bo Pang, Wentao Liu, and Cewu Lu. Human pose regression with residual log- likelhhodd estimation. In Int. Conf. Comput. Vis., 2021.
[36] Xuecheng Nie, Jiashi Feng, Jianfeng Zhang, and Shuicheng Yan. Single-stage multi-person pose machines. Int. Conf. Comput. Vis., 2019.
[37] Xiao Sun, Jiaxiang Shang, Shuang Liang, and Yichen Wei. Compositional human pose regres- sion. In Int. Conf. Comput. Vis., 2017.
12

---

[38] Alexander Toshev and Christian Szegedy. Deeppose: Human pose estimation via deep neural networks. In IEEE Conf. Comput. Vis. Pattern Recog., 2014.
[39] Jie Yang, Ailing Zeng, Shilong Liu, Feng Li, Ruimao Zhang, and Lei Zhang. Explicit box detection unifies end-to-end multi-person pose estimation. arXiv preprint arXiv:2332.0159: 2023.
[40] Jie Yang, Ailing Zeng, Feng Li, Shilong Liu, Ruimao Zhang, and Lei Zhang. Neural interactive keypoint detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15122-15132, 2023,
[41] Yilun Chen, Zhicheng Wang. Yuxiang Peng, Zhiqiang Zhang, Gang Yu, and Jian Sun. Cascaded pyramid network for multi-person pose estimation. In IEEE Conf. Comput. Vis. Pattern Recog., 2018.
[42] Bowen Cheng, Bin Xiao, Jingdong Wang, Honghui Shi, Thomas S Huang, and Lei Zhang. Higherhrnet: Scale-aware representation learning for bottom-up human pose estimation. In IEEE Conf. Comput. Vis. Patlern Recog., 2020.
[43] Sheng Jin, Wentao Liu, Wanli Ouyang, and Chen Qian. Multi-person articulated tracking with spatial and temporal embeddings. In IEEE Conf. Comput. Vis. Pattern Recog., 2019.
[44] Sheng Jin, Wentao Liu, Enze Xie, Wenhai Wang, Chen Qian, Wanli Ouyang, and Ping Luo. Differentiable hierarchical graph grouping for multi-person pose estimation. In Eur. Conf. Comput. Vis., 2020.
[45] Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hourglass networks for human pose estimation. In Eur: Conf. Compur. Vis., 2016.
[46] Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose estimation. In IEEE Conf. Comput. Vis. Pattern Recog., 2019.
[47] Shih-En Wei, Varun Ramakrishna, Takeo Kanade, and Yaser Sheikh. Convolutional pose machines. In IEEE Conf. Comput. Vis. Pattern Recog., 2016.
[48] Bin Xiao, Haiping Wu, and Yichen Wei. Simple baselines for human pose estimation and tracking. In Eur. Conf. Comput. Vis., 2018.
[49] Yuhui Yuan, Rao Fu, Lang Huang, Weihong Lin, Chao Zhang, Xilin Chen, and Jingdong Wang. Hrformer: High-resolution transformer for dense prediction. arXiv preprint 2021.
[50] Yanjie Li, Shoukui Zhang, Zhicheng Wang, Sen Yang, Wankou Yang, Shu-Tao Xia, and Erjin Zhou. Tokenpose: Learning keypoint tokens for human pose estimation. arXiv preprint arXiv:2104.03516, 2021.
[51] Yufei Xu, Jing Zhang. Qiming Zhang, and Dacheng Tao. Vitpose: Simple vision transformer baselines for human pose estimation. Advances in Neural Information Processing Systems, 35:38571-38584, 2022.
[52] Jie Yang, Ailing Zeng, Ruimao Zhang, and Lei Zhang. Unipose: Detecting any keypoints. arXiv preprint 10X.022310.085300 2023.
[53] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.00066 2022.
[54] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b, arXiv preprint arXiv:2310.0682. 2023.
[55] Jean-Baptiste Alayrac Jeff Donahue, Pauline Lue, Antoine Miech, Iain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances neural information processing systems, 35:23716-23736 2022.
13

---

[56] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023.
[57] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024.
[58] Ji Lin, Hongxu Yin, Wei Ping. Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models, arXiv preprint arXiv:2312.0755.3 2023.
[59] Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng. Floris Weers, et al. Mml: Methods, analysis & insights from multimodal IIm pre-training. arXiv preprint arXiv:2403.096// 2024.
[60] Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang. Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, and Chong Ruan. Deepseck-vl: Towards real-world vision-language understanding, 2024.
[61] Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXhv:2403.18814 2023.
[62] Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models? arXiv preprint arXiv:2405.02246, 2024.
[63] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Adv. Neural Inform. Process. Syst., 2017.
[64] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Int. Conf. Mach. Learn., 2021.
[65] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19955-11886, 2023.
[66] Wenhai Wang, Zhe Chen. Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. Advances Neural Information Processing Systems, 36, 2024.
[67] Zhiliang Peng, Wenhui Wang. Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. 14824, 2023.
[68] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. 120.:26608.1666 2023.
[69] Renjie Pi, Jiahui Gao, Shizhe Diao, Rui Pan, Hanze Dong, Jipeng Zhang, Lewei Yao, Jianhua Han, Hang Xu, and Lingpeng Kong Tong Zhang. Detgpt: Detect what you need via reasoning. arXiv preprint arXiv: 14167, 2023.
[70] Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuning large language model on region-of-interest. arXiv preprint arXivv2307.03661, 2023.
[71] Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint 23X 2023.
[72] Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal Im''s referential dialogue magic. arXiv preprint arXiv:2306. 15/55 2023.
14

---

[73] Renjie Pi, Lewei Yao, Jiahui Gao. Jipeng Zhang, and Tong Zhang. Perceptiongpt: Effectively fusing visual perception into Ilm. arXiv preprint arXiv:2311.06612, 2023.
[74] Dongkai Wang, Shiyu Xuan, and Shiliang Zhang. Locllm: Exploiting generalizable human keypoint localization via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 614-622 2024.
[75] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yer Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015-4026, 2023.
[76] Yuhang Zang, Wei Li, Jun Han, Kaiyang Zhou, and Chen Change Loy, Contextual object detection with multimodal large language models. arXiv preprint arXiv:2305. /8279, 2023.
[77] Jiarui Xu, Xingyi Zhou, Shen Yan, Xiuye Gu, Anurag Arnab, Chen Sun, Xiaolong Wang, and Cordelia Schmid. Pixel-aligned language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13030-13039, 2024.
[78] Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li. Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. arXiv preprint orXiv:2388.00692 2023.
[79] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv: 106.08850, 2021.
[80] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Adv. Neural Inform. Process. Syst., 2017.
[81] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adapta- tion of deep networks. In Int. Conf. Mach. Learn., 2017.
[82] Akihiro Nakamura and Tatsuya Harada. Revisiting fine-tuning for few-shot learning. arXiv preprint arXiv: 19/0,00200, 2019.
[83] Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, al. Cambrian-1: A fully open, vision-centric exploration of multimodal 1lms. arXiv preprint 11XXv:2200.168660 2024.
[84] Dongsheng Jiang, Yuchen Liu, Songlin Liu, Xiaopeng Zhang, Jin Li, Hongkai Xiong, and Qi Tian. From clip dino: Visual encoders shout in multi-modal large language models. 2023.
[85] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.071933 2023.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
All claims are validated by experiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer to Sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Implementation details can be found in Sec. 4.1.3
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [Yes]
Justification:
Our code will be released at https://kptllmm waw.lll....mtttt github... io
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/dde/eeee Cooeeeoodeesssseeeeee 7/nips.com htte:/uulllllliiieeee//eeeueaasl ssiniioiicyp for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Training details can be found in Sec. 3.5 and experimental setups can be found in Sec. 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported because it would be too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
18

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We have carefully reviewed the NeurlPS Code of Ethics. And the research conform with it in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer to Sec.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Please refer to Sec. 6
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The assets properly cited.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
20

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not introduce any assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
21


Reviews:
[{"review_id": "FVzhhM0mP2", "reviewer": "Reviewer_j2Jb", "summary": "The paper proposes to build a MLLM for keypoint detection. They propose three main points to achieve this. First, the paper introduces a new benchmark that measures various types of keypoint detection including both text and visual-prompt-based, as well as traditional keypoint understanding. Next, they propose to build a multimodal LLM, which is designed specifically for keypoint detection that is shown to perform quite well on this task.", "questions": "Please address the weaknesses above, which also contain questions.", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "[S1] Good results: The paper boasts quite good results for the task at hand. The experiments are thorough and clearly outline the contribution of various components of the model.\n\n[S2] New benchmark: The paper proposes a new benchmark for keypoint understanding, which goes beyond the traditionally available keypoint understanding benchmarks by bringing visual and text prompting. While this is mainly in service to the current paper, it could be independently useful for posterity.", "weaknesses": "[W1] Not a lot of novel insights: This is my main, and mostly only, weakness for the paper, but it take a few different forms. Firstly, this is a straightforward \"building\" of MLLM formula; i.e., convert the task at hand into \"instruction format\" -> train a vision to LLM connector --> fine-tune LLM; that is used by a lot of recent works. As such, this would be mostly of interest to the domain performance, Keypoint comprehension in this case. This already puts some doubt as to whether this is of interest enough to a broad community. Next, there are also not a lot of modular components that might be broadly useful that hasn't already been studied by other works. Some details below:\n\nW1.1 LLava does almost as good (for supported tasks): The paper shows that, once fine-tuned, the popular LLAVA model does almost as good as the proposed method. The significance of this finding is not thoroughly discussed. Why build a new model at all? Furthermore, there are more capable MLLMs baseline such as R1, R2, R3 that might be more suitable targets for fine-tuning. See 1.2 below as well\n\nW1.2 The proposed method of generating a special token that can be decoded to a key point is also explored before (R1). Besides, there are also various MLLMs that are capable of doing region comprehension (R2, R3...). Those are not sufficiently discussed. If fine-tuned, like LLAVA, how would they do on this task?\n\nW1.3 Is visual prompting a good paradigm for keypoint detection? What is the importance of visual prompting for this task? [EDIT: Clarified in post-rebuttal]\n\nReferences:\nR1: https://arxiv.org/abs/2311.06612\nR2: https://arxiv.org/abs/2306.15195\nR3: https://openreview.net/forum?id=2msbbX3ydD", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1: Novelty and Contribution\n\n**To Keypoint Detection Community:** To the best of our knowledge, this is the first work to \naddress the problem of semantic keypoint comprehension, which aims\nto understand keypoints within different human-AI interaction contexts.\nPrevious keypoint detectors primarily rely on directly learning visual or textual prompts for keypoint localization through extensive data fitting, often neglecting the semantic understanding of keypoints. This oversight often leads to misinterpretations of the prompts and inaccurate predictions. In contrast, our work proposes the \"identify-then-detect'' strategy, which requires the model to first comprehend the semantics of keypoints and then accurately determine their positions through a chain-of-thought process. This paradigm could enhance both interpretability and localization accuracy.\n\n\n**To MLLM Community:**\nCompared with previous works (R1, R2, R3) that focus on region-level understanding, our model takes a further step to achieve more fine-grained, pixel-level understanding and localization, which is critical for various real-world applications. For instance, **semantic keypoint understanding** equips MLLMs with enhanced visual comprehension and analytical capabilities, essential for tasks such as structural analysis, action recognition, and medical image interpretation. **Visual prompt-based keypoint detection** allows MLLMs to acquire keypoint definitions from visual prompts, enabling cross-class and cross-keypoint localization using sample images provided by users. Additionally, **textual prompt-based keypoint detection** enables MLLMs to follow human language guidance, facilitating keypoint localization on arbitrary objects and keypoint categories in a zero-shot manner.\n\n\n### Q2: Compared with LLaVA, R1, R2, R3\n\nThe preliminary results in our paper indicate that LLaVA can address the keypoint semantic understanding task through fine-tuning, primarily because this task only involves answering questions based on images. Given the architecture of LLaVA and similar models like R1, R2, and R3, these models can be directly fine-tuned to support this task. \n**However, they all lack capabilities for fine-grained keypoint localization, a critical aspect that our article emphasizes.** For example, LLaVA, R1, R2, and R3 cannot support the input of visual prompts (support images along with support keypoint definitions) for keypoint detection, which could enhance the generalization capabilities of in-context learning within LLMs. In addition, although R1 employs a special token to extract features from LLMs and uses an additional decoder to decode corresponding masks or boxes, it does not address the new challenge of keypoint detection, which remains unhandled by existing models.\n\n### Q3: The importance of visual prompt-based keypoint detection\nActually,  visual prompt-based keypoint detection is a task that has been widely studied in keypoint detection community. It is also supported by a well-established benchmark, which further demonstrates its significance and practice. \nMoreover,  visual prompt-based keypoint detection is important and practical in various real-world scenerios. For example, real-world applications across various fields often require detecting keypoints on a variety of unseen objects. \nTo address this, it is a good choice to use a visual prompt-based keypoint detection model, which can detect any poses of unseen objects. In such scenerio, compared to textual prompt-based keypoint detection, visual prompts could allow the model to more accurately detect keypoints that are difficult to describe semantically, such as cloth.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "d3iLBU6bNM", "reviewer": "Reviewer_Hwow", "summary": "The authors aim to enhance multi-modal LLMs with semantic keypoint comprehension. They introduce a hybrid visual prompting approach using a query and a support input image. In their pipeline, visual features from both images are extracted via a vision encoder model. Additionally, a support keypoint prompt, indicating keypoint positions on the support image, is used. The embeddings from this prompt are combined with the support image embeddings and fed into the LLM. This mixed embedding helps the model detect and understand the keypoint in the input query image, generating the desired keypoint location information.", "questions": "The authors should compare their approach to GPT-4V as a baseline to better understand the benefits of the additional steps they have added to their proposed pipeline.\n\nIt is unclear where the support image and support keypoint prompts come from during inference. Are these data available to other methods in the experiments?\n\nFigure 2 is somewhat confusing. (Z_q) is fed to the LoRA section, while the rest of the information is fed to the rest of the LLM. Is the figure designed this way on purpose? If so, could you please clarify this?", "limitations": "The authors have discussed the limitations of their work.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is well written and easy to follow. \n\nThe proposed method shows promising results albeit its scope of usability and also generalization should be better examined.", "weaknesses": "The proposed support image and support keypoint prompt essentially act as few-shot examples for the model. It would be beneficial to explicitly acknowledge this in the paper.\n\nAnother limitation of the proposed method is the requirement for a visually similar support image with keypoint locations for each input query image. Have the authors tested the model with support images that are visually different from the query images? How much visual difference can the model handle without a significant drop in accuracy? Conducting an ablation study on this would be crucial to determine the extent of semantic (high-level) or texture (low-level) distribution shifts the method can tolerate without a notable accuracy decline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1: Visual prompt-based keypoint detection is equal to few-shot keypoint detection\n\nThe visual prompt-based keypoint detection can be considered as the few-shot keypoint detection.\nIn our main article, we have explicitly illustrated the input requirements of visual prompt-based keypoint detection. \n\n\n\n### Q2: The impact of visual differences between support images and query images on model performance\n\nIn fact, visual prompt-based keypoint detection does not require a visually similar support image with keypoint locations for each input query image.\nIn the MP-100 dataset, objects within the same novel category are not always similar, and there are many challenging cases. To illustrate this, we select the tern body as a novel category, which exhibits significant species variation, to demonstrate the impact of visual differences between support images and query images on model performance. As shown in the visualizations in the PDF, using the same support images with support keypoints, our model can effectively detect different query images, despite variations in poses, object appearances, and environments.\n\n### Q3: Compared with GPT-4V\n\nWe compare our model with GPT-4V for visual prompt-based keypoint detection. Given the high cost of utilizing the GPT-4V API, we randomly sample 10 paired images for each category in the test set. For the split-1 set, which includes 20 test categories, this results in a total of 200 image pairs used for evaluation. As shown in the table below, our model effectively detects novel objects given the support image and support keypoints, outperforming GPT-4V in these scenarios.\n\n| Method | MP100-Split1-Sub |\n|--------|------------------|\n| GPT-4V | 25.23            |\n| Ours   | 91.58            |\n\nTable: Compared with GPT-4V for visual prompt-based keypoint detection.\n\n### Q4: The evaluation protocol is consistent across our model and other comparable methods\n\n\nOur model and other comparable methods all use the standard benchmark MP-100 for the evaluation of visual prompt-based keypoint detection task. This benchmark ensures that the species categories used in the training and testing phases are non-overlapping, meaning that the test objects are entirely unseen during training. Following the standard evaluation protocol used by comparable methods, we sample 3,000 random pairs (one as the support image and another as the query image) for each novel category during testing. With 20 test categories for each split, we construct a total of 60,000 pairs for comprehensive and effective evaluation.\n\n\n### Q5: About the figure of our framework\n\nWe apologize for any misunderstanding. \nLoRA (Low-Rank Adaptation) is an efficient fine-tuning technique that modifies existing layers of a Large Language Model (LLM) by introducing additional low-rank matrices. These matrices enhance the model's ability to comprehend keypoints without the need for extensive retraining of the original weights. As a result, all multimodal inputs ($Z_q$ and other inputs) are processed through both the LLM and the LoRA weights.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0swblmja2F", "reviewer": "Reviewer_wNmk", "summary": "This paper proposes an LLM-based keypoint localization method that can detect keypoint by visual or text prompt and classify the keypoint category by given its coordinates. Experiments on several public benchmarks demonstrate the effectiveness of the proposed method.", "questions": "In Weakness.", "limitations": "This paper discusses the limitation in A.3.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Adopting LLM to perform in-context learning of keypoint localization is interesting and this paper demonstrates that this pipeline is feasible and can get superior performance.", "weaknesses": "1.\tIt seems that the Visual Prompt-based Keypoint Detection and Textual Prompt-based Keypoint Detection are trained separately. However, it is more useful if we can complete these tasks using a single model. So how about jointly training above two tasks? Can they benefit from each other or not?\n2.\tWhat’s the meaning of keypoint identifiers in Table 2 and 3? It would be confusing if these is no specific illustration.\n3.\tSome typos. “Texual”->Textual in line217, 234, 254, etc.\n4.\tSome claims are not proper. For instance, in line 57-58 “To the best of our knowledge, this is the first work that equip LLMs with the point-level visual perception capabilities.” [A] already introduces LLM into keypoint localization and this paper should emphasize the different aspect such as  visual prompt-based keypoint localization.\n\n[A] D. Wang, S. Xuan, S. Zhang. LocLLM: Exploiting Generalizable Human Keypoint Localization via Large Language Model. CVPR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1: The mutual benefit between textual prompt and visual prompt\n\nIn the main article, since textual prompt-based and visual prompt-based keypoint detection have their own benchmarks, combining both prompts for training would result in unfair comparisons. Here, we supplement the experiment by using both textual and visual prompts. The results on MP100 split-1 show that the textual prompt could provide valuable high-level and rich semantic guidance to enhance the keypoint localization.\n\n| Visual  Prompt    | Textual  Prompt   | MP100-Split1 |\n|------------|------------|--------------|\n| &#10004; |            | 91.66        |\n| &#10004; | &#10004; | 92.18        |\n\nTable: The ablation study of the mutual benefit between textual prompt and visual prompt\n\n\n### Q2: The explanation of keypoint identifier\n\nThe keypoint identifier used in the Capeformer can alleviate the ambiguity of keypoints with similar appearances when compared to using only visual features for low-level matching. However, the keypoint identifier requires additional effort to annotate keypoints across different categories, which limits real-world applications where keypoint identifiers are not available.\n\n\n### Q3: About typo\n\nThanks for pointing out the typo errors. We will correct them in the revised version.\n\n\n### Q4: About related work\n\nLocLLM [A] was publicly available on June 7, 2024, which is later than our submission date. Therefore, our work can be considered concurrent. LocLLM only uses textual prompts for human keypoint localization using LLM. In contrast, we explore the use of LLM for more comprehensive keypoint comprehension of diverse objects. We will revise some claims and add discussions to address this.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pEWp6SbXkZ", "reviewer": "Reviewer_7RyA", "summary": "The paper introduces and studies the problem of Semantic Keypoint Comprehension to evaluate the capability of Multimodal Large Language Models (MLLMs) in tackling fine-grained perception and comprehension. It does so by defining three related tasks: (1) keypoint semantic understanding, (2) Visual prompt-based keypoint detection and  (3) Textual Prompt-based Keypoint Detection. The first task targets the semantic interpretation capabilities of objects and keypoints, while the other two tasks involve position detection and semantic comprehension from visual or textual descriptions. In the context of this paper, keypoint detection (a.k.a object pose estimation) is the task of localizing the semantic keypoints of objects.\n\nThe authors further present KptLLM, an MLLM for Semantic Keypoint Comprehension based on LLaVA. The model employs an identify-then-detect strategy, where the model first captures the required semantics and then detects the precise keypoint position. The model employs attention to fuse features from image and keypoint encoded prompts. The fused encoding and the encoding of visual inputs are used as additional latent contexts for an LLM, which is finetuned with LORA to generate the prompt. The keypoint position is further regressed with an MLP head. The authors evaluate their proposed model on the MP-100 (diverse and category agnostic) dataset and the AP-10K dataset and compare its performance to state-of-the-art methods for keypoint detection and to pretrained and finetuned LLaVA models.", "questions": "- Can you please provide more details on runtime, memory, and scalability?\n- Is the visual encoder trained/fine-tuned or frozen? The textual description mentions a pretrained frozen encoder, while the image shows it to be fine-tuned", "limitations": "It is worth elaborating on memory, runtime and scalability aspects of the proposed method.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces new tasks for studying the semantic comprehension of MLLMs at a fine-grained level\n- The paper presents an MLLM-based method for tackling the introduced tasks, which outperforms current keypoint detection methods across datasets and tasks by notable margins\n- The proposed method also outperforms state-of-the-art MLLM (pre trained and finetuned) on the target tasks\n- The authors ablate their proposed detection strategy and other architectural choices.", "weaknesses": "- The properties (and potential limitations) of the MLLM-based approaches in terms of runtime, memory, and scalability to multiple key points, compared to existing state-of-the-art methods, are not discussed\n- Lack of details on the methods chosen for cross-comparison and their key properties/assumptions makes it harder to evaluate the contribution\nWhile the paper is overall well structured, the evaluation protocol and details are not always easy to follow", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1: The runtime, memory, and scalability of our model\n\nDuring inference, our model processes multiple keypoints by stacking them into a batch. For the standard 17 keypoints, our model obtains their positions in 3.45 seconds, consuming 25,100 MiB of memory. It is important to note that in our implementation, we did not utilize LLM-related acceleration and optimization techniques, which could potentially enhance the model's efficiency.\n\n### Q2: Details of comparable methods and evaluation protocol\n\n**Details of comparable methods:** For visual prompt-based keypoint detection, all the methods take the visual prompt and query image as the inputs, and output the corresponding keypoints. These methods only rely on matching support keypoint features with query image features. This matching is semantic-agnostic, which can fail when there is insufficient similarity between the support and query instances, especially when they differ significantly in poses, textures, or styles. Additionally, these methods struggle when dealing with similar keypoints that exhibit symmetrical appearances, such as the left eye and right eye. For textual prompt-based keypoint detection, all the methods take the textual prompt and test image as the inputs, and output the corresponding keypoints.\n\n**Details of evaluation protocol:** For **keypoint semantic understanding** and **visual prompt-based keypoint detection**, we employ the standard benchmark-MP100 for evaluation. This benchmark ensures that the species categories used in the training and testing phases are non-overlapping, meaning the test objects are entirely unseen during training. Following the standard evaluation protocol used by comparable methods, we sample 3,000 random pairs (one as the support image and another as the query image) for each novel category during testing. With 20 test categories for each split, we construct a total of 60,000 pairs for evaluation. For **textual prompt-based keypoint detection**, we follow CLAMP to evaluate the models’ generalization ability on unseen animal species in the zero-shot learning paradigm. Two experimental settings are defined based on whether the animal species in the training set and test set belong to the same animal order or not. Species belonging to the same order tend to have similar appearances, while species from different orders exhibit more diverse appearances.\n\n### Q3: About Visual Encoder\n\nWe have finetuned the visual encoder, as the original pre-trained CLIP image encoder cannot adequately capture the pixel-level fine-grained visual information, which is essential for the keypoint detection tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate the efforts of all reviewers in reviewing our paper and providing insightful comments and valuable suggestions. The supplementary visualization results (To \# Reviewer Hwow) have been included in the rebuttal PDF.

Meta-review: **Summary:** This work introduces KptLLM, an MLLM that addresses the challenges of semantic keypoint comprehension, where a model must reason about fine-grained semantic visual information through keypoint. KptLLM shows that using a support set and LoRA on a frozen LLM leads to improvements over other baseline models on keypoint comprehension.

**Strengths/Weaknesses:** The reviewers comment on the strength of the approach for an interesting problem and that KptLLM leads to noticeable improvements over baselines. Most weaknesses focus on a lack of clarity. The wording/figures/tables can be confusing and the authors do not explicitly say the approach using a support set is an instance of few-shot prompting. This can be addressed in the camera ready. Some other weaknesses (e.g. selecting less similar support sets to test generalization) were addressed in the rebuttal. The remaining comments from Reviewer j2Jb should be included more thoroughly in the camera ready as well.

**Recommendation:** I recommend this paper for acceptance. The reviews are positive overall and the authors appear to have thoughtfully addressed reviewers’ questions and concerns through additional experiments and examples. Reviewers have requested more clarity in the Figure 2 and Tables 5 and 6) As acknowledged during the rebuttal, I’d like to see all of the additional experiments and suggestions for language shifts included in the final version. A more in-depth analysis of Llava (expanding on the discussion with Reviewer j2Jb) should be included as well to further justify the contributions of KptLLM.
Decision: Accept (poster)