Talking Points: Describing and Localizing Pixels
Authors: Matan Rusanovsky
Shimon Malnick
Shai Avidan
Venue: ICLR 2026 Poster (2026)
Keywords: Keypoint Description
Keypoint Localization
Pixel-Level Grounding
Reinforcement Learning
Vision-Language Model
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: A bidirectional framework: Point Descriptor generates natural language from pixels, Point Localizer regresses coordinates from descriptions. RL optimizes descriptions for improved localization accuracy.

Abstract: Vision-language models have achieved remarkable success in cross-modal understanding. Yet, these models remain limited to object-level or region-level grounding, lacking the capability for pixel-precise keypoint comprehension through natural language. We introduce a novel framework for pixel level grounding. The framework consists of two complementary components: a Point Descriptor that generates rich, contextual descriptions of individual keypoints, and a Point Localizer that regresses precise pixel coordinates from these descriptions. Unlike prior work that relies on templated prompts or keypoint names, our approach produces free-form, coarse-to-fine descriptions that situate keypoints within their visual context.
Since there is no available dataset to train such a system, we introduce LlamaPointInPart, a carefully curated dataset of 20K+ image-keypoint-description triplets synthesized from multiple vision-language models, capturing multi-scale information from scene-level context to visual features around the keypoint. 
For cross-category generalization, we optimize the Point Descriptor on AP-10K via GRPO, using the frozen Point Localizer as a reward model to produce descriptions that maximize localization accuracy.
To evaluate our results we establish a new evaluation protocol. Instead of comparing the text description produced by our method to the ground truth, we use the localizer to determine how close is the predicted point generated to the ground truth point.
Experiments demonstrate superior performance compared to baseline models on LlamaPointInPart. 
The bidirectional nature of our framework enables applications in both keypoint-guided image understanding and language-guided precise localization. Our code and dataset are publicly available at https://matanr.github.io/Talking_Points .

Published as a conference paper at ICLR 2026
TALKING POINTS: DESCRIBING AND LOCALIZING PIXELS
Matan Rusanovsky, Shimon Malnick and Shai Avidan
Tel Aviv University (matanru, mainick] /eaammmll.........aam...la wta.ii.tau.e.. .il and avidan@eng. tau. ac.
ABSTRACT
Vision-language models have achieved remarkable success in cross-modal under- standing. Yet, these models remain limited to object-level or region-level ground- ing, lacking the capability for pixel-precise keypoint comprehension through nat- ural language. We introduce novel framework for pixel level grounding. The framework consists of two complementary components: Point Descriptor that generates rich, contextual descriptions of individual keypoints, and Point Lo- calizer that regresses precise pixel coordinates from these descriptions. Unlike prior work that relies on templated prompts or keypoint names, our approach pro- duces free-form, coarse-to- to-ffeto--iie descriptions that situate keypoints within their vi- sual context. Since there no available dataset to train such system, we in- troduce LlamaPoim/lnPurl, carefully curated dataset of 20K+ image-keypoint- description triplets synthesized from multiple vision-language models, captur- ing multi-scale information from scene-level context to visual features around the keypoint. For cross-category generalization, we optimize the Point De- scriptor on AP-10K via GRPO, using the frozen Point Localizer a reward model to produce descriptions that maximize localization accuracy. To evalu- ate our results we establish a new evaluation protocol. Instead of comparing the text description produced by our method to the ground truth, we use the localizer to determine how close is the predicted point generated to the ground truth point. Experiments demonstrate superior performance compared base- line models on LlamaPoimtInPart. The bidirectional nature our framework should enable future applications in both keypoint-guide image understanding and language-guided precise localization. Our code and dataset are publicly avail- able at https: //mtaar / / Talking_Point
INTRODUCTION
A central challenge in multi-moda. learning is bridging the gap between dense pixel-level vi- sual features and semantic natural language. Although recent models have greatly improved vision-language alignment, they predominantly reason at the image object scale, leaving fine-grained, pixel-level grounding largely unexplored. As illustrated in Figure [1] this task of pre- cisely describing and localizing individual pixels proves remarkably challenging: while our method doubles the performance of our baseline (OMG-LLaVA) and outperforms the state-of-the-an foun- dation model ChatGPT-5, human annotations surprisingly perform worse than ChatGPT-5, under- scoring the inherent difficulty of this new lixel-level grounding task.
The advent of Vision Latguag Models (VLMs) such as LLaVA (Liu tially advanced cross-modal integration by treating visual patches and linguistic tokens uniformly within a transformer. These VL.Ms excel tasks like image captioning, visual dialogue, and question answering. Recent grounding works have extended these capabilities bidirectionally. Models like SAM (Kirilovv et al., 2023), Semantic-SAM (Li et al. 2023a), and Grounding- -INNO (Liu et al. 2023b) accept spatial prompts (points, boxes) or language queries to generate segmentation masks and bounding boxes. Conversely, recent VLMs enable both grounded conversation and spatial output generation: DAM (Lian et al. 2025) produces rich descriptions from visual prompts, Groundhog (Zhang et al. 2024c) generates segmentations from textual descriptions, while OMG-LLLVA (Zhang al.) 2024h unifies both directions, accepting visual prompts (bound-
2023al has substan-

---

Published as a conference paper at ICLR 2026
The point is located the leftmest horse the image, which positioned close to the left edge and slightly below the center, specifically on its head, near the right and above the center of the head, where dark, elongated shape, possibly nose or part of the horse's 's head, is visible,
Localize
Describe
100
MPCK
NPCK Score (%)
78
60
56
20
Description
ChatGPT5
ONG-LLaVA
Figure 1: Talking Points: Describing and Localizing Pixels. Given an image and keypoint (left, red star), generate rich descriptions (top right) progressing from scene-level object localization, through part identification, to position within part, and local visual features. We evaluate the descrip- tions by localizing them back to pixels (green point image). Our TalkingPoints (TP) achieves near GT performance (bottom right), doubling our baseline (OMG-LLaVA) and outperforming ChatGPT- and human annotations, which surprisingly perform worst, highlighting the challenge of pixel-level grounding, Evaluation uses mPCK (mean Percentage of Correct Keypoints), measuring the fraction of predictions within a normalized distance threshold of ground truth, averaged across fine and coarse thresholds.
ing boxes, masks, points) for region-specifee conversations and producing segmentation tokens that decode into spatial outputs through specialized heads. Yet, despite this growing flexibility in bridg- ing vision and language through spatial grounding, all these methods still operate on entire segments or regions rather than reasoning over individual pixels.
Recent efforts such as KptLLMM YYang al. 2024b) and LocLLM (Wang al. 2024a) attempt to move beyond object-level prompts toward keypoint comprehension. However, both rely on rigid, template-based textual descriptions tied to predefined anatomy or part labels, falling short of rich, language-grounded localization.
To address these limitations, we introduce two complementary components: a Point Descriptor and Point Localizer. The Point Descriptor, given image and single pixel (Figure left), generates richly expressive, free-form language that specifies the object's placement within the scene, the part's location within that object, the keypoint's position within that part, and salient visual cues immediately surrounding the keypoint (Figure [ top right). The Point Localizer then consumes this description to regress the exact pixel coordinate, achieving higher localization accuracy than models relying on templated or name-only prompts. This bidirectional capability, describing pixels in natural language and localizing them back, enables precise pixel-level grounding that significantly outperforms existing approaches.
We first train both components on carefully curated dataset of 20K+ image- boanec-kypooit-desrrpeee triplets. Our construction pipeline combines part-level annotations with vision-language models operating different scales, one processing the full image for object-level context and another ana- lyzing masked regions around keypoints for local detail. A large language model synthesizes these complementary perspectives into rich, coherent descriptions that connect precise pixel locations with their semantic context.
We report the results based on a Point Descriptor and a Point Localizer that were trained separately curated dataset. Our descriptor-through-lee through-localizer evaluation measures description quality through localization accuracy, providing a novel metric for pixel-level language grounding. Addi- tionally, explore reinforcement learning promising direction for extending our approach to novel categories without ground-trun truth descriptions.
follows: (1) We construct dataset of 20,000 image- keygen-e description triplets with rich natural language capturing multi-scale spatial context; (2) We intro- duce Point Descriptor and Point I Localizer for language-to-pixel mapping: (3) We explore rein-
Our contributions
2

---

Published as a conference paper at ICLR 2026
forcement learning using GRPO promising direction for adapting the Point Descriptor to novel categories without ground truth descriptions; and (4) We propose a novel evaluation methodology measuring descriptor quality through localization accuracy.
RELATED WORK
2.1
KEYPOINT DETECTION AND COMPREHENSION
Traditional keypoint detection methods focus on dategory-specifie models for humans (Lin al.
2014
Andriluka et al. 2014), animals (Cao et al.
2019
Yu et 2011], or objects (Ge et al.,
employing either regression- based (Li et al. based (Xiao et al., 2018) Xu et al., 2022b) approaches. Recent work extends to category-agnostic settings through few-shot keypoint detection (Xu et al. 2022a Shi et al. 2023) using visual prompts. Several methods, CLAMP (Zhang et al.) 2023), X-Pose (Yang al.] 2024a), and CapeX (Ru- sanovsky et al. 2025), use textual prompts or point explanations for aategory-agnostie pose esti- mation. However, these approaches rely predefined keypoint names and templates rather than free-form descriptions.
2019
2021;
Toshev & Szegedy (014) or heatmap-
The emergence of VLMS has enabled new approaches to keypoint understanding. LocLLMM (Wang et al. 2024a) pioneered LLMM-based keypoint localization but is trained exclusively on human key- points, where the model receives textual prompts describing body parts (c.g., "left shoulder," "right ankle")) and directly regresses pixel coordinates. While LocLLM incorporates some descriptive con- text through instruction templates, these remain formulaic and category-spectno limited human anatomy. KptLLM (Yngg et al., 2024b) introduces semantic keypoint comprehension across three tasks: semantic understanding keypoint names, visual prompt-based detection using support im- ages, and textual prompt-based detection from part names. However, its textual descriptions are generated through a fixed template that combines object category, part name, and keypoint name (e.g., "the left eye of the cat"), lacking free-form, context-rich language that captures the visual appearance or spatial context surrounding the keypoint.
Our work introduces bidirectional pixel locations and inversely localizing keypoints from these descriptions, enabling true pixel-level language grounding through learned visual context rather than predefined templates.
grounding: generating rich descriptions from
2.2 VISION-LANGUAGE GROUNDING
Vision-language models have evolved from image-level understanding to sophisticated spatial grounding capabilities. Early VLMs like LLaVA (Liu et al. 2023a), BLIP-2 (Li et al.) 20236), and Flamingo (Alayrac et al.. (022) excel at image captioning, visual dialogue, and question answering by treating visual patches and linguistic tokens uniformly within transformers. Recent grounding works have extended these capabilities bidirectionally.
Models like SAM (Kiilllov et af.) 2023), Semantic-SAM (Li et al. 2023a), and Grounding- DINO (Liu al.) 2023b) accept spatial prompts (points, boxes) or language queries to generate segmentation masks and bounding boxes. Conversely, recent VLMs enable both grounded conver- sation and spatial output generation: Kosmos-2 (Peng et al. 2023) and Shikra (Chen et al., (023) innovate by incorporating spatial boxes as inputs and training with region-text pairs for region- level comprehension. DAM (Lian al.) 2025) produces rich descriptions from visual prompts, Groundhog (Zhang et al. 2024c) generates segmentations from textual descriptions, while OMG- LLaVA (Zhang et al. (2024) unifies both directions, accepting visual prompts (bounding boxes, masks, points) for region-specific conversations and producing segmentation tokens that decode into visual outputs. Ferret (You et al. (223) and GPT4Rol (Zhang et al. 2024a) further advance legion-level visual comprehension through referring and grounding capabilities.
However, these approaches operate at object or segment scales rather than true keypoint-level com- prehension. Our work adapts OMG-LLaVA's architecture but fundamentally shifts from object- centric to pixel-centric grounding through Gaussian attention masks, enabling description and local- ization of individual keypoints rather than entire regions.
3

---

Published as a conference paper at ICLR 2026
2.3 REINFORCEMENT LEARNING FOR VISION-LANGUAGE MODELS
Recent advances in reinforcement learning for vision-language models have shifted from subjec- tive human preferences (RLHF) toward spatially-grounde reward signals that provide verifiable, automatic evaluation metrics. Several works establish closed-loop training between description generation and localization. RL-VLM-F (Wang al., 2024b) uses vision-language foundation models as reward signals based on semantic alignment between descriptions and visual observa- tions. SpatialVLM (Chen al.) 2024) enables dense reward annotation through quantitative spatial understanding, while SE-GUI (Du et al. 2025) implements GRPO (Shao et al. (024) with self- evolutionary training, computing rewards based on coordinate prediction accuracy.
However, all existing work operates at object or region scales using bounding boxes segmentation masks as grounding primitives. Our approach uniquely employs pixel-level keypoint localization accuracy as the reward signal, where the Point Descriptor is fine-tuned via GRPO with the Point Localizer serving as reward model. This pixel-centric reward mechanism optimizes for exact co- ordinate accuracy rather than regional overlap, establishing the first closed-loop training paradigm between keypoint description and localization at the individual pixel level.
(c) Chair seat (a) "The point is the bicycle, which spans most of the horizontal and vertical axis in the image, specifically on its handlebar; situated to the left and close to the top edge of bike, and within the handlebar, keypoint located slightly above and to the right of the center, in a region that features curved metal bar with grip area."
(a) Bicycle handlebar
(b) Snake eye
(b) "The point is located on the snake, which is positioned close to the left edge and slightly above the vertical center of the image, and within this snake, it is situated on the head, specifically near the right edge and below the vertical center of the head, in region that features dark, oval-shaped reflective surface, likely the pupil of snake's 's eye."
with
(c) "The point located on the chair that is closest to the viewer and positioned on the left side of the dining table, specifically on the chair's seat, which is near the right edge and slightly above the center of the seat, and in the region around the keypoint, there is small, dark spot standing out against the lighter background."
Figure/ 2: LlamaPointInPart dataset examples demonstrating diverse objects and parts across three source datasets: (a) PascalPart1 16, (b) PartImageNet, (c) ADE22KParr234, Red circles indicate keypoints with corresponding coarse-to-fin descriptions that progress from scene-level object lo- calization, through part identification, to keypoint position within the part, and finally visual features around the keypoint, enabling accurate language-guider keypoint localization.
METHOD
3.1 DATASET CONSTRUCTION
LlamaPointInPart Dataset We construct LlamaPointInPart high-quality dataset of 20K+ triplets, through a multi-stage pipeline leveraging complementary langunel models (Figure 3). Starting from PascalPartI ADE20KPar234 (Wei et al. 2023), and PartImageNet (He et al. 2021). we extract images with lert-level bounding box an- notations. For each image, compute SIFT (Lowe) 1999) features and select the highest-response keypoint within annotated parts (excluding background). All keypoints in our dataset are semantic by construction: every keypoint corresponds meaningful part of object rather than arbitrary
4

---

Published as a conference paper at ICLR 2026
I5 dise up the The erel het and the
OFF
Biped head
Serple
OVGLLIN
A small norkeyy vibble
biped head of snall monkey specifioonll nery the cetter of the bipad head, in region with shorp focus the eyeleshes and surrounding skin.
peeking out from bahind
partial
Datasets
Point Descrption
kmndatan
Patton
*
Merticah close edge
Image
Image, Point, Description
Figure 3: LlamaPointInPart dataset construction pipeline: We start with part-annotated datasets, and select the highest-scoring SIFT keypoint within some part (red star). LLaVA generates fine- grained local descriptions from cropped regions around keypoints, while OMG-LLaVA provides object-centric context from full images. Llama3.3 3 synthesizes these multi-scale perspectives with hierarchical spatial annotations into coherent, coarse-to- descriptions, forming our final image- point-description triplets.
background location. Moreover, although SIFT is used to rank candidate points, the final keypoints span both textured and smooth within-part regions, since selection is restricted to annotated semantic parts rather than gradient-rich areas. We determine the keypoint's 's relative position within its con- taining part (e.g., "near top edge"), explicitly encoding spatial relationships and instance ordering for disambiguation.
To ensure dataset diversity, we maintain equal proportions across the three source datasets when sampling keypoints, resulting in balanced representation across 64 unique object categories and 297 unique part categories (with some semantic overlap, e.g., "biped" encompassing multiple animal types). Appendix A.I (Figure ( visualizes this distribution, with the inner rings showing equal sampling from each source dataset and the outer rings displaying the variety of objects and parts covered. This balanced strategy ensures comprehensive coverage across diverse semantic categories, from animals and vehicles to furniture and household objects.
To capture multi-scale context, query two VLMs: (1) OMG-LLaVA ZZang al. 2024b) ceives the image and keypoint to generate object-centric descriptions, and (2) LLaVA (Liu et al. 2023a) processes Gaassian-masker region centered the keypoint extract fine-grained local features. This dual approach captures details beyond part annotations, for instance, identifying a keypoint near bird's eye despite lacking explicit eye annotations. We synthesize these descriptions via a quantized LLaMA3.3 (Dubey et al. 2024) through a two-stage process (generation followed by refinement) produce coherent, coarse-to tone keypoint descriptions. Our descriptions follow deliberate hierarchical progression: (1) object location within image, (2) part location within object, (3) keypoint position within part, and (4) local visual features. While some descriptions Fig- ure are abbreviated for space constraints, this coarse-to-fine structure is consistently maintained throughout our dataset.
Our dataset encompasses diverse keypoint types: semantically salient features like the snake's eye pupil (Figure[p). functional components such as the bicycle handlebar grip (Figure 2h), and seem- ingly ordinary surface points like the chair seat marking (Figure D This diversity, ranging from visually distinctive landmarks to unremarkable surface locations, ensures our models generalize beyond prototypical keypoints arbitrary pixel locations, critical capability for true pixel-level comprehension. We split LlamaPointInPart into 17K training and 4K test examples, maintaining proportional representation across source datasets. We manually tested 5% of the and verified that more than 91 % of the keypoints can be casily localized using the point descriptions.
samples,
AP-10K Adaptation To evaluate cross-category generalization capabilities, we leverage AP- 10K Yu et al. 2021). following the experimental split configuration of CLAMP (Zhang al. 2023) and KptLLM Yang et al., 20244b). Specifically, we adopt their different order setting, where models trained super-category and tested on another assess generalization to visually distinct
5

---

Published as a conference paper at ICLR 2026
Image features
Region features
Keypoint features
Instruction prompt features
The keypoint is located on
OMG-Seg
head,
Output Keypoint Description
Descriptor
Point
Ximagez In Please segment The keypoint is located on the cat's s head....
region1:
LLM Generate
LoRA
cp keypoint s/p <EE.>
Input Keypoint "Conversation"
Input Image
OMG-Seg
Output Keypoint (0.51. 0.43]
Point Localizer
LLM - Forward Pass
LoRA
Projection
<EG>'s Hidden State
Input Gaussian Keypoint Mask
Input Image
Figure 4: Talking Point Architecture. The Point Descriptor (blue) generates textual descriptions from image-keyy imagc-keypoint pairs using Gaussian masks centered at keypoints using different type of token features. See legend at top of figure. The Point Localizer (green) regresses keypoint coordinates from image-descriptin the localizer can be used standalone, it can also be applied to the generated descriptions (red dashed arrow) to evaluate localization performance.
pairs through a special <SEG> token that encodes visual information. While
animal families. We evaluate bidirectionally: training on Bovidae (22.6K keypoint-image pairs) and testing Canidae (17K pairs), well as training Canidae and testing on Bovidad with both categories annotated with up to 17 keypoints per instance. This setup provides a systematic evaluation of our model's ability generalize keypoint understanding unseen taxonomically and visually distinct groups. Since AP-10K provides only keypoint annotations without descriptive con- text, we utilize these pairs exclusively for reinforcement learning-baad -based fine-tuning (Section (.4), where our Point Descriptor learns to generate descriptions that maximize localization accuracy.
3.2 POINT DESCRIPTOR
Our Point Descriptor adapts OMG-LLaVA's object grounding architecture for pixel-level keypoint description generation (blue part in Figure D. While OMG-LLaVA predicts segmentation masks to describe entire objects, we replace these with fixed Gaussian attention masks centered at keypoints, fundamentally shifting focus from object-level to pixel-level comprehension.
Given an image and keypoint coordinates (), we generate Gaussian mask M centered at the keypoint. The coordinates undergo two parallel transformations through OMG-Seg's 's decoder: learnable prompt embeddings generate initial semantic queries, while sinusoidal positional encoding of (x, y) followed by linear projection provides spatial information. These initial representations then interact with multi-scale visual features through transformer decoder layers, with the Gaussian mask controlling the attention pattern.
The Gaussian mask constrains the attention mechanism in each decoder layer by defining a boolean attention mask for the cross-attention operation: queries only attend image features (keys and values) within the Gaussian region around the keypoint. This differs fundamentally from OMG- LLaVA, where predicted object masks allow attention across entire object boundaries. Our fixed masks force the queries to gather information exclusively from the keypoint's 's immediate neighbor- hood.
'Bovidae includes antelope, argali sheep, bison, buffalo, cow, and sheep. Canidae includes arctic fox, dog, fox, and walf.

---

Published as a conference paper at ICLR 2026
As result, rather than encoding global object semanties, the queries now capture fine- grained infor- mation at the specific keypoint; accordingly, we denote them as featuin In OMG-LLaVA, these representations were termed "object features" because they captured object-level information; we rename them here to reflect the shift introduced by our masked-attention modification toward keypoint-level focus. Similarly, the positional encodings, which serve query positional embed- dings throughout the attention operations, become region features that anchor spatial reasoning at the keypoint location.
features.
Crucially, while the Gaussian mask constrains attention within the OMG-Seg decoder, the full image features are provided to the LLM alongside the keypoint and region features during description generation (see Figure 4 This ensures the model has access to complete visual context while maintaining focus on the specific keypoint location.
This architectural modification proves essential. Without Gaussian masks (Table 3). performance catastrophically drops: mPCK drops from 78.13 to 23.63, as the model loses the ability to connect specific pixel locations with their descriptions. The refined keypoint and region features are then projected to the language model's embedding space description generation. We optimize using LoRA adapters (Hu et al. (022) while freezing the vision encoder, training with standard language modeling loss on LlamaPointInPai descriptions.
3.3
POINT LOCALIZER
The Point Localizer inverts the description task: given image / and textual description D, it regresses keypoint coordinates (green part in Figure Following OMG-LLaVA's grounding formulation, we structure inputs as: "<image>\nP lease segment regionl: Descriptien DI". followed by the response: "<p> keypoint </p> <SEG>." .", where the special token <SEG> encodes visual information. The image is encoded via the vision encoder and projected to the language space through learned projection using OMG-Seg. These projected features combine with the tokenized prompt and pass through the language model.
We perform single forward pass and extract the hidden state corresponding to <SEG>, h Rd. This representation passes through a text-to-vision projection layer, followed by a multi-layer per- ceptron that maps to normalized coordinates (#, 9) 1]2.
Training minimizes the mean squared error between predicted and ground-trui coordinates:
Lioe MSE( Pyt)
where أ (é, !) and Pot represent predicted and ground-truth normalized coordinates respectively. We jointly optimize LoRA adapters (Hu et al., 2022) the language model, the vision-to-text projection layer, and the coordinate regression head. As demonstrated in our ablations (Table the LoRA adaptation of the language model is crucial for effective keypoint understanding.
3.4 REINFORCEMENT LEARNING FOR MUTUAL ENHANCEMENT
To enable keypoint comprehension across diverse categories without annotated descriptions, we employ reinforcement learning where the Point Localizer provides reward signals for optimizing the Point Descriptor.
Given an image-keypoint pair (l. we sample G descriptions (0), 02, ,G) from the descriptor policy Te, where p)). For each generated description, the frozen localizer predicts coordinates p.. The reward function measures localization accuracy:
MME(
We optimize the descriptor via modified Group Relative Policy Optimization (GRPO) (Shao et al, 2024). For the sampled descriptions, we compute normalized group-relative advantages:
A, r; mean (r)
(rd
where r = {ri, (21, } and apply clipping for numerical stability. Following GRPO, we assign each sequence's advantage to all its tokens and normalize by sequence length. Unlike standard
7

---

Published as a conference paper at ICLR 2026
GRPO, do not employ importance sampling. The policy gradient objective becomes:
G Σ 1=1
lod log To(00,e00, 1,))
policy
A
(4)
where |o:| denotes the length of description o; and oi, represents the 1-th token the i-th descrip- tion. This formulation ensures gradient updates are invariant to sequence length.
To prevent policy drift, we incorporate KL regularization against the reference policy Te.. Following the GRPO formulation, compute the KL divergence using an unbiased estimator (Schulman 2020) at the token level:
1
ID KL To [rerl
lo.1 Σ (exp(r:) - 1]
(5)
where = log 0orr(0,,lo0,<4: I,p) - log To (or, I.)) is the log-ratio for token t. To prevent gradient instability from extreme probability ratios, we apply conservative clamping to the log-ratio: clamp(ri, -5, 5). This bounds the exponential term to manageable range while preserving the KI. signal. The per-sequence KI. divergence computed by averaging over valid tokens, then averaged across all samples in the batch. The complete training objective combines policy gradient and KL regularization:
L Lpllicy + 9KL. DpL
We implement selective fine-tuning by optimizing LoRA adapters (Hu et al. 2022) while updating only the final two transformer blocks, preserving general linguistic capabilities while adapting high- level representations for keypoint description.
This closed-loop paradigm creates mutual enhancement: the descriptor learns to generate descrip- tions that maximize localization accuracy, effectively adapting its outputs to the localizer's capa- bilities. By optimizing descriptions for localizability, improve the alignment between generated descriptions and the localizer's 's expected input distribution, enabling strong localization performance on ressriptor-geenrraie text.
EXPERIMENTS
We quantify performance using the Percentage of Correct Keypoints (PCK) metric, where keypoint coordinates are first normalized by image dimensions to [0, 1). then compute the Fuclidean dis- lance between predicted and ground-truth points, counting a prediction as correct if this distance falls below threshold. We follow (Chen al. 2025) and use mean PCK (mPCK) that is defined as follows. We average PCK @0.1 and PCK @0.2 to capture both fine-graines accuracy (0.1) and coarse regional localization (0.2) in unified measure, with full breakdowns Appendix [A.4] [p This ciriipooo-ttroug-llocccceo
evaluation extends existing semantic keypoint comprehension tasks by measuring descriptor quality through localization accuracy, emphasizing both expressive language and precise grounding.
4.1 SUPERVISED FINE-TUNING
Point Descriptor and Localizer Training. We initialize from OMG-LLaVA's pretrained weights and fine-tune the Point Descriptor on LlamaPointInPar'ss training set for 10 epochs, using a batch size of 8, and learning rate of - optimizing only the language modeling loss Liest under the same LoRA configuration as OMG-LLaVA (rank 512, effective scaling 0.5, dropout 0.05, no bias). The Point Localizer trains for 15 epochs with learning rate 10-5 and batch size 8. We optimize LoRA adapters (same as above) on the language model, the vision-to-text projection layer, and the coordinate regression MLP, while freezing all other parameters
All training runs in this work were carried out on single NVIDIA H100 (80 GB) GPU.
8

---

Published as a conference paper at ICLR 2026
Table 1: Performance comparison on LlamaPointInPart test set. Our ap- proach substantially outperforms the OMG-ILaVA baseline, with the Point Descriptor achieving near ground-truth performance.
Originaal
ChataPTS (Ptuu)))
Deactiption
Method
mPCK
OMG-LLaVA (Zhang et al. 2024b)
31.03
DAM (Lian et al. 2025)
42.87
TP (Ours)
78.13
Description
78.83
Figure 5: Qualitative keypoints visualization.
LlamaPointInPart Results. Table presents localization accuracy on LlamaPointInPar' test set, evaluated using our Point Localizer. Our Point Localizer with ground-trutt test descriptions achieves 78.83% mPCK. While with the OMG-LLaVA (Zhang et al. 2024b) and DAM (Lian et al. 2025) baselines, our Point Localizer achieves only 31.03% and 42.87% respectively. Using predicted descriptions from our Point Descriptor maintains robust performance 178., achieving x x2.5 performance boost compared OMG-LLaVA baseline. This demonstrates that at the test time of our Point Localizer, we can replace ground-truth descriptions with generated ones while maintaining localization performance.
Extended Evaluation with Foundation Model and Human Annotations. We extended evalu- ation to ChatGPT-5 and human descriptions on 100 test samples. As shown in Figure [ (bottom right), our approach achieves ground-truth performance (78%), consistently maintaining perfor- mance boost more than x2.5 compared to OMG-LLaVA's baseline and surpassing both ChatGPT- 5 (62%) and surprisingly, human annotations (56%). Note that these evaluations trained on LlamaPointInPar dataset. The lower human performance may reflect differences in description style from training data rather than humans' inability to accurately describe keypoint locations. F shows a qualitative example. Additional examples of generated descriptions and localizations presented in Appendix A.2 Table 5
fixed localizer
4.2 LOCALIZATION BASED REINFORCEMENT LEARNING
We evaluate our RL approach for generalizing novel categories without ground-truth descriptions on the cross- category generalization setup described in Section 3.1
Table 2: Cross-uuper-aateorry gener- alization on AP-10K. RL adaptation shows promising improvements over zero-shot performance.
Table reports performance before and after RL adap- tation. Although absolute accuracy remains limited due to the challenge of localizing keypoints with rich text on visually distinct dataset, RL fine-tuning yields consis- tent improvements. Training on Bovidae and testing on Canidae shows modest gains (~0.4%%, while the reverse setup demonstrates larger improvements (~-.3%% Over- all, these results suggest that localization-based RL rep- resents a promising direction for scaling keypoint under- standing by exploiting abundant keypoint-image pairs without costly description annotations.
Test Set
Method
mPCK
TP (zero-shot)
29.85
Canidac
TP+RL (on Bovidae)
29.96
TP (zero-shot)
28.56
Bovidac
TP+RL (on Canidae)
30.36
4.3 ABLATION STUDIES
Gaussian Mask Guidance. We investigate the importance of providing explicit Gaussian masks around keypoints during Point Descriptor training. When relying solely on OMG-LLaVA's standard
Due computational constraints, we conducted these RI. experiments subset of the data. See Ap- pendix A.3 for RI. setup and computational considerations.

---

Published as a conference paper at ICLR 2026
Table 3: Impact of explicit Gaussian masks. Without visual guidance, the model fails to con- nect keypoints with descriptions.
Table 4: Impact of language model adaptation. Freezing the LLM and training only projection layers severely degrades performance.
Configuration
mPCK
Configuration
mPCK
w/o Gaussian mask
23.63
w/o LLM adaptation
47.60
Point Descriptor 78.13
Point Localizer
78.83
mechanism to predict object masks from input keypoints, without the Gaussian visual guidance, the model completely fails to learn the connection between keypoints and their descriptions (Table B) This demonstrates that explicit visual marking is crucial for the model to establish spatial-semantic correspondence. This setup also corresponds to fine-tuning the original LMG--LLVV architecture on LlamaaPintUnPar since it retains OMG-LLaVA's object-mask prediction mechanism without our Gaussian modification. Despite being trained on the same data, this adapted OMG-LLaVA VA variant achieves only 23.63 mPCK (compared to 78. 13 mPCK when using the Gaussian mask), indicating that the Gaussian mask formulation is essential for pixel-level grounding.
Language Model Adaptation. Fine-tuning the language model proves essential for keypoint com- prehension. Without LoRA adaptation, keeping the LLM frozen while training only projection lay- and the regression head, performance drops substantially (Table This confirms that keypoint- specific language understanding requires adaptation of the language model's 's representations.
5 DISCUSSION
Conclusions. We presented a framework for pixel-level keypoint comprehension through natural language, introducing a Point Descriptor that generates rich contextual descriptions and Point Localizer that regresses precise coordinates. Our approach moves beyond templated prompts to produce free-form, coarse-to-fine descriptions that capture multi-scale spatial context. Through reinforcement learning using the frozen Point Localizer as reward model, we optimize the Point Descriptor to generate descriptions that maximize localization accuracy.
Our method achieves near ground-truth performance on our new LlamaPointInPal and significantly outperforms baseline models, demonstrating the effectiveness of task saec-fp architectures for pixel- level understanding. The reinforcement learning approach shows promising improvements when generalizing across taxonomically distinct categories AP-10K. Importantly, this RL approach particularly promising for scaling, as keypoint-image pairs are substantially easier to collect than triplets required for supervised training, opening a path towards training on larger and more diverse datasets.
the complete
Limitations and Future Work. Our descriptions currently rely heavily on spatial context, requir- ing the image remain unchanged, which limits applicability scenarios like stereo matching or multi-view settings. The analysis revealed that human- annotated descriptions also exhibited similar spatial dependencies, suggesting this may be an inherent characteristic of pixel-level localization tasks where context is crucial to uniquely identify specific points. More challenging still is the task of image correspondence: generating description from a point in one image that can identify the corresponding point in an entirely different image.
practical limitation of our current evaluation is that relies on a single Localizer trained on Llama- PointInPart. As result, the metric is most reliable for comparing methods that operate within a similar descriptive style. The goal of this work not to position this Localizer as a universal evaluator, but offer consistent, reproducible protocol for the emerging task of pixel-level de- -ocaliztoo.c.iiztionn. Developing style-robust or human-trained localizers would further expand the generality of the evaluation.
Future work should explore developing multi-view datasets and descriptions that emphasize seman- tic and appearance- apaseed features spatial relationships. We hope that releasing our dataset and framework will encourage the community to build upon this direction, ultimately driving progress toward even finer-grained and more reliable localization capabilities in the future.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
Part of this research was supported by ISF grant 2132/23.
REFERENCES
cean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in neural information processing systems. 35:23716- 23736, 2022.
Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele. 2d human pose estima- tion: New benchmark and state of the art analysis. In Proceedings of the IEEE Conference computer Vision and Pattern Recognition, pp. 3686-3693, 2014.
Jinkun Cao, Hongyang Tang, Slao-Shu&e Fang, Xiaoyong Shen, Cewu Lu, and Yu-Wing Tai. Cross- domain adaptation for animal pose estimation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1498-9507,220
Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvim: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14455- 14465, June 2024.
Junjie Chen, Weilong Chen, Yifan Zuo, and Yuming Fang. Recurrent feature mining and keypoint mixup padding for catesory-agnostic pose estimation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 22035-22044 2025.
Keqin Chen, Zhao Zhang, Weili Zeng. Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal I1m's 's referential dialogue magic. arXiv preprint arXiv:2306. 15195, 2023.
Yong Du, Yuchen Yan, Fei Tang. Zhengxi Lu, Chang Zong, Weiming Lu, Shengpei Jiang, and Yongliang Shen. Test-time reinforcement learning for gui grounding via region consistency, 2025.
URL
05615
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, al. The llama 3 herd of models. arXiv e-prints, pp. arXiv-2409 2024.
Yuying Ge, Ruimao Zhang, Xiaogang Wang, Xiaoou Tang, and Ping Luo. Deepfashion2: A versatile benchmark for detection, pose estimation, segmentation and re-identification of clothing images. In Proceedings of the IEFF/CVF conference on computer vision and patlern recognition, pp- 5337-5345, 2019.
Ju He, Shuo Yang, Shaokang Yang, Adam Kortylewski, Xiaoding Yuan, Jie-Neng Chen, Shuai Liu, Cheng Yang, and Alan Yuille. Partimagenet: A large, high-quality dataset of parts. arXiv preprint /2.0:22//.000933.. 2021.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan- Yen Lo, et al. Segment anything. In Proceed- ings of the IEFF/CVF international conference on computer vision, pp. 4015-4026, 2023.
Feng Li. Hao Zhang, Peize Sun, Xueyan Zou, Shilong Liu, Jianwei Yang, Chunyuan Li, Lei Zhang, and Jianfeng Gao. Semantic-sam: Segment and recognize anything any granularity. arXiv preprint arXivv2307.047677 2023a.
Jiefeng Li, Siyuan Bian, Ailing Zeng, Can Wang, Bo Pang. Wentao Liu, and Cewu Lu. Human pose regression with residual log- likelihoood estimation. In Proceedings of the IEEE/CVE international conference on computer vision, pp. 11025-11034 2021.

---

Published as a conference paper at ICLR 2026
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image image pre-training with frozen image encoders and large language models. In International conference machine learning. pp. 19730-19742. PMLR, 2023b.
Long Lian, Yifan Ding, Yunhao Ge. Sifei Liu, Hanzi Mao, Boyi Li, Marco Pavone, Ming- Yu Liu, Trevor Darrell, Adam Yala, and Yin Cui. Describe anything: Detailed localized image and video captioning. arXiv preprint arXiv:2504. /607,, 2025.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023a.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang. Jie Yang. Chunyuan Li, Jianwei Yang. Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499 2023b.
David G Lowe. Object recognition from local scale-invariant features. In Proceedings of the seventh IEEE international conference on computer vision, volume pp. 1150-1157. Icee, 1999.
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models the world. arXiv preprint arXiv:2306. 14824, 2023.
Matan Rusanovsky, Or Hirschorn, and Shai Avidan. Capex: aatesory- pose estimation from textual point explanation. In The Thirteenth International Conference on Learming Representa- tions, 2025. URL htttps::
net forum? id- sc
John Schulman. Approximating kl divergence. Online, 2020. k1 k1-apppox.her html
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y.K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024.
abs////b//aaa 2402.4220 03300
Min Shi, Zihao Huang, Xianzheng Ma, Xiaowei Hu, and Zhiguo Cao. Matching not enough:
A two-stage framework for category-agnosti pose estimation. In I Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7308-7317, 2023.
Alexander Toshev and Christian Szegedy. Deeppose: Human pose estimation via deep neural net- works. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1653-1660, 2014.
Dongkai Wang, Shiyu Xuan, and Shiliang Zhang. Loellm: Exploiting generalizable human keypoint localization via large language model. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 614-623, 2024a.
Yufei Wang, Zhanyi Sun, Jesse Zhang, Zhou Xian, Erdem Biyik, David Held, and Zackory Erick- son. Rl-vim-f: Reinforcement learning from vision language foundation model feedback. In Proceedings of the 41th International Conference on Machine Learning, 2024b.
Meng Wei, Xiaoyu Yue, Wenwei Zhang, Shu Kong, Xihui Liu, and Jiangmiao Pang. Ov-parts: Towards open- vocabulary part segmentation. In seirth- Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
Bin Xiao, Haiping Wu, and Yichen Wei. Simple baselines for human pose estimation and tracking. In Proceedings of the European conference on computer vision (ECCV), pp. 666-481, 2018
Lumin Xu, Sheng Jin, Wang Zeng, Wentao Liu, Chen Qian, Wanli Ouyang, Ping Luo, and Xiaogang Wang. Pose for everything: Towards category-agnoste pose estimation. In European conference computer vision, pp. 398-419--698 Springer, 2022a.
12

---

Published as a conference paper at ICLR 2026
Yufei Xu, Jing Zhang, Qiming Zhang, and Dacheng Tao. Vitpose: Simple vision transformer base- lines for human pose estimation. Advances in neural information processing systems, 35:38571- 1- 38584, 2022b.
Jie Yang, Ailing Zeng, Ruimao Zhang, and Lei Zhang. X-pose: Detecting any keypoints. In Euro- pean Conference on Computer Vision, pp. 249-268. Springer, 2024a.
Jie Yang, Wang Zeng, Sheng Jin, Lumin Xu, Wentao Liu, Chen Qian, and Ruimao Zhang. Kptilm: Unveiling the power of large language model for keypoint comprehension. Advances Neural Information Processing Systems, 37: 17:140766-140788 2024b.
Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint arXiv:2310.07700 2023.
Hang Yu, Yufei Xu, Jing Zhang, Wei Zhao, Ziyu Guan, and Dacheng Tao. Ap-10k: A benchmark for animal pose estimation in the wild. arXiv preprint arXiv:2108, 10n:22008.1211,, 2021.
Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Yu Liu, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuning large language model on region-of-interest In European conference computer vision, pp. 52-70. Springer, 2024a.
Tao Zhang. Xiangtai Li, Hao Fei, Haobo Yuan, Shengqiong Wu, Shumping Ji, Chen Change Loy, and Shuicheng YAN. OMG-LLaVA: Bridging image-level, object-level, pixel-level reasoning and understanding. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024b. URL (tth:tts: https//////ppeeeee..eeoo net itt.:r/iwwweeeeooo..iiiii.e. WeeoodPPPsS
Xu Zhang. Wen Wang, Zhe Chen, Yufei Xu, Jing Zhang, and Dacheng Tao. Clamp: Prompt-based contrastive learning for connecting language and animal pose. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 23272-23281, 2023.
Yichi Zhang, Ziqiao Ma, Xiaofeng Gao, Suhaila Shakiah, Qiaozi Gao, and Joyce Chai. Groundhog: Grounding large language models to holistic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14227-14238, 2024c.
13

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1
ADDITIONAL DETAILS ON LLAMAPOINTINPART CONSTRUCTION
Figure presents the compositional distribution of objects and parts, and Figure 2 provides repre- sentative dataset examples.
Parce
partimargu gaarggeett
Pasmitrrrti
PartinaggeNet
LlamaPointinPart
LiamaPolntinPart
ADE20KPar1234
(a) Object category distribution
(b) Part category distribution
Figure 6: LlamaPointInPart dataset composition showing (a) 64 object categories and (b) 297 part categories across our 20K+ keypoint- despripiissiiiiionn pairs. Inner rings indicate source datasets (Pascal- Part 16, ADr20KPart234, PartImageNet), outer rings show sampled objects and parts.
14

---

Published as a conference paper at ICLR 2026
A.2 ADDITIONAL EXAMPLES
We provide further qualitative examples in Table 5 accompanied by keypoint descriptions from different sources. Each example shown in separate column: the top row displays the image with localizations from each source, distinguished by different markers, while the bottom rows present the corresponding textual descriptions from each source.
Table 5: Keypoint Localization Descriptions Comparison.
eat
Chompet
Human oveLLoc
Driinal
Charcen iDunl
LLvk
GT De- scription
The point is located on the cat that spans most of the image, specifically situated to the right and close to the top edge of cat, and within this paw, point slightly to the right and above the center, in region featuring a small, dark, circular area with lighter center, possibly pupil a small opening.
The point is located on the rightmost dog in the image, which is close the right and bot- tom edges of the picture, and it positioned on dog 's nose, specifically slightly above and to the right of the center of the nose, with the region around the keypoint being a small, dark, and somewhat circular shape. likely the nose itself.
its paw, which
TP
The point is located the cat, which spans of the horizontal and vertical axis of the image, specifically on the cat's paw, which is near the right and top edge of the cat, and within the paw, the point is slightly to the left and above the center, in region that features small, dark, curved shape with smooth texture.
The point is located on the dog that posi- tioned closer the right and bottom edges of the image, and it is second dog from the left, on its eye, which is situated slightly to the left and above the center of the dog head, and within the eye, the point is slightly to the left and below the center, that features a small, dark, circular area with lighter center, possibly pupil.
(Ours)
region
ChatGPT-
The keypoint located on the raised front paw of the cat, precisely at the outer edge of the paw's 's upper surface where the fur is dark and contrasts with the bright light shin- ing from the window behind. sits just above the slightly curved claws that are silhouetted against the background, directly opposite the cat's open mouth and extended tongue. The paw is angled upward, and the keypoint lies on the side facing the camera. slightly the right of the cat's nose and whiskers, frramed by the surrounding purple feather boa that drapes across the cat's 's chest and lower body. This placement marks a distinct point on the paw's contour, highlighted by the sharp out- line against the sunlit backdrop.
The keypoint is located on the smaller brown puppy lying in the snow, specifically at the center of its head where the snout meets the forchead, just above the nose. It sits directly opposite the larger light-colored dog, which is bending its head downward as sniffing toward the smaller one. The keypoint is posi- tioned slightly right of the image's 's center, the puppy whose body is stretched out hori- zontally on the snow. Surrounding cues clude the textured snow surface beneath both dogs, the large blue metal gate in the back- ground, and the clear contrast between the smaller puppy's reddish fur and the white snow, This placement highlights the midpoint of the puppy's 's face, precisely where its head is directed toward the approaching dog.
5
Human
The point is located on the hand of the cat in the iamge, on the same horizontal line that crosses the middle of the cat's nose, the middle part of the paw. The window is clear.
the point is located on the tip of the nose of the right (and smaller) dog
OMG- LLaVA
The dog is brown.
15

---

Published as a conference paper at ICLR 2026
A.3 REINFORCEMENT LEARNING SETUP AND CONSTRAINTS
Point Descriptor Reinforced Learning Fine-tuning. We set the group size G = 3, KL-penalty coefficient 0x1. 0.1, and learning rate of 10-6 Fine-tuning is conducted for 3 epochs with batch size 10, reusing the same LoRA configuration. This stage explicitly optimizes the descriptor towards producing localization-focsse descriptions, complementing the language-only supervised objective.
Scope and Computational Constraints. RL training requires generating multiple descriptions per sample in every forward pass, which introduces substantial computational overhead. For this reason, our RL experiments focus on a targeted cross-category setting rather than large-scale do- main transfer. While Bovidae and Canidae share similar body plans, this setup isolates whether the descriptor can improve using only (image, keypoint) pairs without ground-truth descriptions. Despite the limited visual divergence, the consistent gains indicate that the pixel-level localization reward is effective for adapting the descriptor beyond its supervised domain. This represents the first application of localization accuracy reward for keypoint description, offering scalable direction for future work on broader category shifts.
16

---

Published as a conference paper at ICLR 2026
A.4
EVALUATIONS USING PCK @0.1 AND PCK @0.2
100
35
PCX Score (99)
82
POKEO..
60
37
20
Description TP (Durs)
Cha16PT-5
Fummnn
OMG LLM
Figure 7: PCK @0.1 and PCK@02 breakdown for the mPCK results presented in Fig.
Table 6: PCK @0.1 and PCK @0.2 breakdown for the mPCK results presented in Table
Method
PCK@0.1
PCK @0.2
OMG-LLaVA
17.26
44.80
DAM
28.24
57.49
TP (Ours)
63.93
92.33
GT Description
65.60
92.05
Table 7: PCK @0.1 and PCK @0.2 breakdown for the mPCK results presented in Table
PCK
Test Set
Method
@0.1
@0.2
(zero-shot)
15.97
43.72
Canidac
TP+RL (on Bovidae)
16.16
43.76
(zero-shot)
15.49
41.62
Bovidae
TP+RI. (on Canidae)
16.63
44.09
Table 8: PCK 10.1 and PCK @0.2 breakdown for the mPCK results presented in Table
Table 9: PCK @0.1 and PCK@002 breakdown for the mPCK results presented in Table 4
Configuration
PCK 0.1
PCK 0.2
Configuration
PCK
PCK @.0.2
w/o Gaussian mask
11.54
35.72
w/o LLM adaptation
27.90
67.30
Point Descriptor
63.93
92.33
Point Localizer
65.60
92.05
A.5 COMPUTATIONAL ANALYSIS
Model Size: The Point Descriptor shares OMG-LLaVA's architecture (7B LLM backbone). The Point Localizer adds only a lightweight MI.P head 5512-128-2 parameters), representing negligible overhead.
Training Time (single H100 80GB GPU): Point Descriptor fine-tuning: ~11 hours on LlamaPointIn- Part (17K samples, epochs); Point Localizer: ~-6 hours (15 epochs); RL fine-tuning: ~22.5 hours on Bovidae subset (22.6K samples), 16.8 hours on Canidae subset (17K samples).
Inference Time: Full pipeline achieves approximately 0.35 samples/second on Bovidae (~~18 hours) and 0.39 samples/second on Canidae (~~12 hours).
A.6 THE USE OF LARGE LANGUAGE MODELS (LLMs)
The text in this paper was refined with the help of LLMs to improve clarity and style. They helped polish the writing.
17


Reviews:
[{"review_id": "ozq2QWBlR2", "reviewer": "Reviewer_psBk", "summary": "This paper proposes Talking Points (TP), a bidirectional framework for pixel-level grounding that (i) generates free-form, coarse-to-fine keypoint descriptions from an image + keypoint (Point Descriptor) and (ii) localizes a keypoint from an image + description (Point Localizer). It introduces LlamaPointInPart, a 20k+ triplet dataset (image, keypoint, description) synthesized from multiple VLMs and fused by an LLM, and evaluates descriptions via localization accuracy using mPCK. The system attains ~78 mPCK, substantially exceeding OMG-LLaVA/DAM baselines on the new test set, and explores GRPO-style RL that optimizes the descriptor using the localizer as a reward model. The authors report modest cross-category gains on AP-10K with RL and discuss limitations around description style bias and spatial dependency.", "questions": "1. Does TP have the ability to conduct semantic segmentation? How is the performance?\n2. Why does the point localizer only use one kind of features of the OMG-Seg?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. LlamaPointInPart composes multi-scale descriptions by querying OMG-LLaVA for object context and LLaVA for local appearance, then LLM-synthesizing coherent coarse-to-fine text; balanced sampling spans 64 objects/297 parts.\n2. Unlike prior region/part-level grounding, TP tightly couples language-point mapping with a description-through-localizer metric and architecture tailored to individual pixels.", "weaknesses": "1. OMG-LLaVA is capable of segmenting object masks, since its training data is for segmentation. The LlamaPointInPart dataset and TP architecture are tailored for point localization. The direct comparisons between OMG-LLaVA and TP are unfair. Comparing TP with a fine-tuned OMG-LLaVA on LlamaPointInPart is valuable.\n2. For keypoint selection, the authors use SIFT with the highest response for keypoint generation. The keypoint validation is necessary but not explained. SIFT detector may choose a point without concrete semantics.  The ratio of semantically rich keypoint in the dataset could be provided.\n3. The Python-generated description of the point location is based on the relative position to the part segmentation. This may be confused with the relative position to the edge of an image.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z0evWzeI97", "reviewer": "Reviewer_YhME", "summary": "This paper introduces a novel framework for bidirectional, pixel-level language grounding, a task that aims to bridge natural language with precise pixel coordinates. The authors propose a two-component system: a Point Descriptor that generates rich, multi-scale descriptions for a given pixel, and a Point Localizer that regresses a pixel's coordinates from such a description. Key contributions include the framework itself, a new synthetic dataset of 20K+ image-keypoint-description triplets named LlamaPointInPart, and a reinforcement learning approach for generalizing the descriptor to new categories without textual annotations.", "questions": "1. Regarding the evaluation bias: Have you considered fine-tuning separate “Point Localizer” models on human-generated or GPT5-generated descriptions (or any other evaluation protocol)? This would create a fairer comparison by evaluating each description source with a model adapted to its specific linguistic style and could provide a more accurate measure of their relative effectiveness.\n2. Regarding the RL approach: Could you comment on the scalability of the RL method for generalization? How do you expect it would perform when fine-tuning on a category like \"animals\" and testing on a visually disparate category like \"vehicles\" or \"furniture\"?\n3. The dataset construction relies on SIFT keypoints, which are inherently biased towards high-texture regions. How does the model perform when tasked with describing arbitrary points on smooth or textureless surfaces, and how might this dataset bias affect its real-world applicability?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper introduces a novel problem formulation by shifting vision-language grounding from the common object/region level to the more precise pixel level. This addresses a clear gap in the literature. The creation and open-sourcing of the LlamaPointInPart dataset is an appreciated contribution to enable research on this new task.\n- The proposed two-component framework is technically sound and logically structured for the bidirectional task. The adaptation of an existing architecture via a fixed Gaussian attention mask is a straightforward method to focus the model on pixel-level features. The empirical results show that the model performs well on the proposed dataset.\n- The paper is structured clearly, with a well-defined problem statement, methodology, and experimental setup. The paper is easy to follow, and the provided figures help to illustrate the proposed architecture and data generation process.", "weaknesses": "1. **Critical Evaluation Bias:** The primary weakness lies in the evaluation methodology. The \"Descriptor-Through-Localizer\" protocol is inherently biased, as it measures a description's quality based on its compatibility with a single Point Localizer model. This localizer was trained exclusively on the synthetic LlamaPointInPart dataset, learning its specific \"machine dialect.\" This bias is further compounded by the RL fine-tuning, which explicitly optimizes the proposed descriptor to align with this specific localizer. Consequently, the claims of superiority over other models are not justified. This is also illustrated by the surprisingly low performance of human and ChatGPT-5 descriptions (Fig 1 & 7), which likely reflects a stylistic mismatch with the localizer rather than an inherent inferiority of the descriptions themselves.\n2. **Limited Generalization in RL Experiments:** The experiments on cross-category generalization, while a promising direction, have significant limitations.\n    - **Scope:** The generalization task was performed between Bovidae and Canidae—two categories of four-legged mammals with similar body plans. This does not provide strong evidence that the RL method can generalize to truly novel and visually dissimilar categories (e.g., from an animal to a bicycle).\n    - **Performance:** The reported mPCK improvements from RL are modest, therefore questioning the robustness and scalability of the approach for more challenging generalization scenarios.\n3. (minor)**Architectural Constraint of Fixed Attention:** A potential architectural limitation is the reliance on a fixed, symmetric Gaussian attention mask in the Point Descriptor. This assumes the most relevant context is always in an isotropic region around a keypoint. This assumption may not hold for points on elongated structures, thin objects, or part boundaries, where a more flexible and content-aware attention mechanism could be more effective.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hoBm3if0DW", "reviewer": "Reviewer_1BTH", "summary": "This paper addresses the core problem of ​​pixel-level vision-language understanding​​ by proposing a novel bidirectional framework, ​​TalkingPoints​​. The framework aims to bridge the gap between dense pixel features and natural language semantics and consists of two complementary components: a ​​Point Descriptor​​, which generates rich, contextual natural language descriptions given an image and a keypoint, and a ​​Point Localizer​​, which regresses precise pixel coordinates from text descriptions. Due to the lack of readily available training data, the authors constructed a large-scale dataset, ​​LlamaPointInPart​​, comprising over 20,000 image-keypoint-description triplets. A key methodological innovation is the use of ​​Gaussian attention masks​​ to achieve pixel-level focus. The paper also explores a ​​Reinforcement Learning (RL)​​ strategy using the localizer as a reward model to optimize the descriptor's generalization to unseen categories. Experiments demonstrate that the method achieves performance close to ground-truth annotations (78.13% mPCK) on the proprietary dataset, significantly outperforming baseline models (e.g., OMG-LLaVA at 31.03%), and shows promise in cross-category generalization tasks. The primary contributions are the proposal of a new task for pixel-level language grounding, a novel method, a new dataset, and a new evaluation protocol.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Novelty of Task and Framework:​​ This is the first systematic exploration of a bidirectional task involving free-form language description and localization of pixel-level keypoints. The proposed TalkingPoints framework is cleverly conceived, with the two components being highly complementary and logically coherent.\n- High-Quality Dataset Construction:​​ The pipeline for creating the LlamaPointInPart dataset is rigorous and innovative, synthesizing multi-scale VLMs and a large language model to ensure the richness and diversity of descriptions. The dataset's balance and quality (over 91% localizability) form a solid foundation for reliable experiments.\n- Comprehensive Experiments and Analysis:​​ The evaluation design is reasonable. It not only compares performance but also provides in-depth validation of the core components' effectiveness through ablation studies. The inclusion of comparisons against strong baselines and human annotations strengthens the persuasiveness of the results.", "weaknesses": "- ​​Generalization and Spatial Dependency:​​ The paper correctly identifies the strong dependence on image spatial context as a major limitation. This severely restricts its applicability in scenarios requiring view or appearance invariance, such as stereo matching or cross-image correspondence. \n\n- ​​Sufficiency of RL Experiments:​​ As mentioned, the RL experiments were conducted on a data subset, making their results preliminary and indicative. Full-scale experiments on larger datasets, with reports of statistical significance, are needed to firmly demonstrate the effectiveness of this approach.\n\n- ​​Computational Cost Analysis:​​ The framework involves two large models (Descriptor and Localizer). The inference speed and computational cost are not discussed, which is an important consideration for practical applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)