Grasp Any Region: Towards Precise, Contextual Pixel Understanding for Multimodal LLMs
Authors: Haochen Wang
Yuhao Wang
Tao Zhang
Yikang Zhou
Yanwei Li
Jiacong Wang
Jiani zheng
Ye Tian
Jiahao Meng
Zilong Huang
Guangcan Mai
Anran Wang
Yunhai Tong
Zhuochen Wang
Xiangtai Li
Zhaoxiang Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: image caption
benchmark
region understanding
Primary area: foundation or frontier models, including LLMs

Abstract: While Multimodal Large Language Models (MLLMs) excel at holistic understanding, they struggle with the dense world, i.e., complex scenes requiring fine-grained analysis of intricate details and object inter-relationships. Region-level MLLMs have been a promising step. However, previous attempts are generally optimized to understand given regions in isolation, neglecting crucial global contexts. To address this, we introduce Grasp Any Region (GAR) for comprehensive region-level visual understanding. Empowered by an effective RoI-aligned feature replay technique, GAR supports (1) precise perception by leveraging necessary global contexts, and (2) modeling interactions between multiple prompts. Together, it then naturally achieves (3) advanced compositional reasoning to answer specific free-form questions about any region, shifting the paradigm from passive description to active dialogue. Moreover, we construct GARBench, which not only provides a more accurate evaluation of single-region comprehension, but also, more importantly, measures interactions and complex reasoning across multiple regions. Empirically, GAR-1B not only maintains the state-of-the-art captioning capabilities, e.g.,
outperforming DAM-3B +4.5 on DLC-Bench, but also excels at modeling relationships between multiple prompts with advanced comprehension capabilities, even surpassing InternVL3-78B on GARBench-VQA. More importantly, our zero-shot
GAR-8B even outperforms in-domain VideoRefer-7B on VideoRefer-BenchQ, indicating its strong comprehension capabilities can be easily transferred to videos. Code and data will be released to the community.

Published as a conference paper at ICLR 2026
GRASP ANY REGION: TOWARDS PRECISE, CON- TEXTUAL PIXEL UNDERSTANDING FOR MULTIMODAL
LLMS
Haochen Wangl.2+ Yuhao Wang Tao Zhang*
Yikang Zhou' Yanwei Li Jiacong Wang Zilong Huang" Guangean Mai Anran Wang"
Jiani Zheng" Ye Tian
Jiahao Meng
Yunhai Tong Zhuochen Wang Xiangtai Li Zhaoxiang Zhang
NNww Laboratory of Pattern Recognition (NLPR).
State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS).
Institute of Automation, Chinese Academy of Sciences (CASIA) University Chinese Academy of Sciences *Peking University "Wuhan University *ByteDance [wanghacchen222, zhaoxiang. zhang) 8 ac..ac.cn
ABSTRACT
While Multimodal Large Language Models (MLLMs) excel at holistic under- standing, they struggle in capturing the dense world with complex scenes, quiring fine-grained analysis of intricate details and object inter-relationships. Region-level MLLMs have been a promising step. However, previous attempts are generally optimized to understand given regions in isolation, neglecting cru- cial global contexts. To address this, we introduce Grasp Any Region (GAR) for comprehensive region-level visual understanding. Empowered by an effec- tive Rol-aligned feature replay technique, GAR supports (1) precise perception by leveraging necessary global contexts, and (2) modeling interactions between multiple prompts. Together, then naturally achieves (3) advanced compositional reasoning to answer specific free-form questions about any region, shifting the paradigm from passive description to active dialogue. Moreover, we construct GAR-Bench, which not only provides
accurate evaluation of single-region comprehension, but also, more importantly, measures interactions and complex reasoning across multiple regions. Extensive experiments have demonstrated that GAR-1B not only maintains the state-of-the-art captioning capabilities, e.g. outper- forming DAM-3B +4.5 on DLC-Bench, but also excels at modeling relationships between multiple prompts with advanced comprehension capabilities, even sur- passing Intern 7LB on GAR-Bench-Ve VQA. More importantly, our zero-shot GAR-8B even outperforms in-domain VideoRefer-7B on VideoRefer- Bench indicating its strong capabilities can be easily transferred to videos. The code is available https: //gi/ sighthut thub com Owt.hheennan440//////
INTRODUCTION
The ambition of Multimodal Large Language Models (MLLMs) is to endow machines with human- like abilities perceive, interpret, and reason about the dense visual world (Yuan al., 2024; Lian et al.. 2025; Li et al., 2025). To date, renowned state-of-thc-and models (Bai et al., 2025; Wu al., 2024; Wang et al., 2025e: DeepMind, 2025b: OpenAL, 2024a;b; 2025) have made remarkable strides, excel in generating holistic descriptions and answering general questions about an entire image. However, this global-level perception struggles with the dense understanding of cluttered environments, intricate object details, and the complex interplay between multiple entities.
To address the limitation of global perception, several previous works (Chen al., 2023; Yuan et al., 2024; Zhang et al., 2024a; Lian et al., 2025; Lin et al., 2025b) argue for paradigm shift to region-level MI.LMs. Specifically, they equip MI.LMs with promptable and fine grained interactions
"Equal contribution. 'Corresponding authors.

---

Published as a conference paper at ICLR 2026
(a) Precise and Detailed Captioning
(b) Multiple Prompts Interaction
(c) Compositional Reasoning
Osesiiea: Deverit he regie
repio in deal
Quatter: What sdatecal betwoce oepr2ot
Queiliaa Art chrooptts and chroest2) koanal within iie eiroe
GAR--B iB:
drag-llupe slpper with ammostl, ceaaded badby sada vidi, spen soooth The pocaggecta has henepun tip
strecppito is ssling shroogtt's eopert
shroepts to
DaMa-t viteast
fre wit smach, alway sctuus loch wondedliaall walllaa
D. Daayy onr pproaposo la cairer
ridge mininn fiunii of at had rowa to Pehool in eyes anc chocchocd, mouth lipiil apess toredings lf sublic along back.
C-stranpnto is cmilagigg chrouptzs and about
Only cPramptis la niirrer
diraspoto
d.drrappooo is bakinn dhremptio lodding away from ohroigttb
Figure Illustration of our GAR, which is superior at leveraging necessary global context to (a) generate precise captions, where green is correct and red means wrong, (b) model complex interactions among multiple prompts, and perform reasoning such (c) recognizing non-entities. Colors of <PromptO>, <Promptls 1>, and <P <oommt22> correspond to masks with respective colors. Image source: (a) Shao al. (2019), (b) Lin al. (2014), and (c) Mei al. (2021).
to achieve targeted region-level understanding, using boxes (Zhang et al., 2024a; Chen et al., 2023) masks (Yuan et al., 2024; Lian al., 2025). This mechanism transforms the model from passive observer of the entire scene into an active participant capable of deep, localized analysis. However, effectively balancing global scene context with fine-grained local details remains challenging, which serves as a fundamental trade-off in region-level MLLMs. Conventional methods (Yuan et al., 2024; You et al., 2023) that employ pooled local features suffer from insufficient details, while recent models (Lian et al., 2025; Lin et al., 2025b) mainly focus on the ability to generate descriptive caption for a single region, and thus model architectures are generally optimized to understand a given region in isolation. This design often neglects crucial global context, e.g., misidentifying frog-shaped slipper as real frog in Figure 1a. To this end, we propose Grasp Any Region (GAR) for comprehensive and detailed region understanding. As shown in Figure 1, key features include:
(1) Precise Perception. Thanks to the leverage necessary global contexts, GAR achieves a more precise perception of given regions, which is the fundamental capability for region MLLMs. As shown in Figure la by aggregating information from the broader, unmasked scene, our GAR manages to generate much more accurate descriptions than previous crop-based approaches (Lian et al., 2025).
(2) Interactions between Multiple Prompts. GAR moves beyond the prevailing single-prone prompt paradigm (Lian et al., 2025), which treats every region of interest isolated entity. As illustrated in Figures 1 and 1c, GAR manages to model relationships between an arbitrary number of prompts.
(3) Advanced Compositional Reasoning Capabilities. Empowered with the aforementioned features, GAR is naturally equipped with advanced compositional reasoning capabilities, allowing it answer any specific free-form questions.
Permi-danch
MDWH Bunch
To achieve these capabilities, effectively encoding global con- texts becomes equally crucial as local detailed features. To this end, we propose an Rol-aligned feature replay technique. Specifically, GAR first encodes the full, uncropped image (to- gether with the mask prompt) with AnyRes (Liu et al., 2024). Subsequently. Rol-Align (He et al., 2017) employed gather relevant features directly from the global feature map. Those gathered features are inherently context-aware, providing suffi- cient local details while maintaining global information simul- tancously. Please refer to Figure for the detailed pipeline.
GaRDeech-CCp
GARBeTchANDA
PACO
RM
GAR [Dum)
Furthermore, we introduce GAR-Bench, which not only pro- vides a more accurate evaluation of single-region compre- hension by constructing multiple-choice questions, but also, more importantly, measures interaction and complex reasoning across multiple regions. It includes test cases that require
Performance compari- son. GAR achieves strong perfor- mances not only region-level un- derstanding, but also excels in gen- eral multimodal benchmarks.
Figure 2:
2

---

Published as a conference paper at ICLR 2026
model to aggregate information from multiple visual regions to arrive at correct conclusion, thereby quantifying the ability to interpret the whole scene rather than independent parts.
Empirically, shown in Figure 2, our GAR-1B not only outperforms DAM-3B (Lian et al., 2025) and PAM-3B (Lin et al., 2025b) on detailed captioning benchmarks (Lian et al., 2025; You et al., 2023; Lin al., 2025a), but also excels in general multimodal benchmarks (Wu & Xie, 2024; Tong al., 2024b; xAI, 2024; Chen et al., 2024a). Interestingly. even outperforms large-scale models like Intern VL33 78B (Zhu et al., 2025) on GAR-Bench, demonstrating its advanced comprehension capability in modeling interactions between multiple prompts. More importantly, our zero-shot GAR-8B even outperforms in-domain VideoRefer-7B on VideoRefer- Benchh, indicating its strong comprehension capabilities can be easily transferred to videos. We hope our work inspires the community to develop MLLMMs that can perceive and understand the dense visual world more effectively.
2 RELATED WORKS
Multimodal Large Language Models (MLLMs). Typical MLLMs (Liu et al., 2023; Li et al., 2024; Liu et al., 2024; Bai al., 2025; Zhu et al., 2025; Wu et al., 2024; Lei et al., 2025; Wang al., 2025b;e; Tong et al., 2024a; Yang et al., 2024a;b; 2025b;a; 2026) project visual features extracted from pre-trained visual encoders (Radford et al., 2021; Zhai et al., 2023) to LLM for understanding multimodal contents. However, these models usually lack precise localization capabilities (Lian et al., 2025; Lin et al., 2025b) and struggle to understand specific regions. One potential solution is to "think with images" (OpenAI, 2025; Wang et al., 2025d;a). But these agentic models require complex multi-turn conversations, while we mainly focus on precise perception within a single-turn dialogue.
Region-Level MLLMs. Different from conventional image-level comprehension, localized un- derstanding requires MLLMS to capture regional attributes. Previous methods either utilize visual markers (Yang et al., 2023), bounding boxes (Zhang al., 2024a; Chen et al., 2023; You et al., 2023; Rasheed et al., 2024; Lee et al., 2024; Ma et al., 2024), or segmentation masks (Yuan et al., 2024; Lian et al., 2025), represent regions-of-ine of-ittrsssssss within an image. We simply regard masks as visual prompts, since masks have less ambiguity than other representations. Beyond prompt representations, effectively balancing global scene context with local details remains an open problem. Existing methods struggle to master both. Methods like DAM (Lian et al., 2025) excel at local perception but lack a holistic view of the global context. Conversely, earlier works like GPT4Rol (Zhang et al., 2024a) and GLaMM (Rasheed et al., 2024), while incorporating the global image, tend to lose crucial local details by pooling region features into single vectors. GAR is designed specifically to solve this dilemma. Based on this, GAR excels in modeling the relationship between arbitrary number of visual prompts while effectively maintaining crucial global context and sufficient local details.
Benchmarks for Region-Level Understanding, Typical region-level benchmarks only evaluate the caption quality for single prompt using conventional language- based captioning metrics (You et al., 2023; Yuan al., 2024; Zhang et al., 2024b; Guo al., 2024; Rasheed al., 2024), model-basee similarities (Chen et al., 2025; Yuan et al., 2024), and LLM-Judged accuracies without the need for reference captions (Lian al., 2025). GAR-Bench is systematically evaluate the comprehension capabilities with multiple visual prompts. It contains caption protocol to measure the correctness of descriptions for the relation between visual prompts, and a VQA protocol to evaluate both the basic understanding capability for specific regions, e.g., color and shape, and advanced compositional reasoning abilities for multiple regions.
3 GRASP ANY REGION
We start from the task formulation in Section 3.1. Subsequently, we introduce our model architecture and training data pipeline in Section 3.2 and Section 3.3, respectively. Finally, we introduce our benchmark designs in Section 3.4 to systematically evaluate region-level comprehension capabilities.
3.1 TASK FORMULATION
The task of grasping any region is hierarchical challenge from basic perception to complex, compositional reasoning about specific visual regions. Specifically, given an image I E x x3 where x W indicates the resolution, and a set of V binary visual prompts, e.g., masks (M) where M; E {0, x>W the objective is to generate precise text response R that demonstrates
3

---

Published as a conference paper at ICLR 2026
Preservation of Global Context: The entire image that captures the entire scene.
<Prampt0> is swinging <Pronpt2> and is
High-Fidelity Local Detoils: Rol-Aligned Feature Replay emphasizes specific regions interest. Efficient and Powerful Understanding
about to hit <Prompt1>
Large Language Model
Visual Encoder Projector
Rol-Aligned Feature Replay
Embedding
Mask Embedding
Rol-Align
Question: What relationship between <Prompt@>, <Pronpt1>, and <Prompt2>?
<Box0>, <Box1>, and <Box2>.
Figure 3: Illustration of our GAR. It leverages single-pass visual encoder to create a holistic feature map of the entire scene, thus preserving global context. Simultaneously. an "RoI-Aligned Feature Replay" mechanism extracts high-fidelity features for specific objects of interest. Both the global context features and the detailed local features are then fed into an L.LM to accurately infer complex relationships and interactions between multiple objects within the image.
multi-layered comprehension of the scene, e.g., detailed attributes description and relational caption, based on the given text instruction T::
R = RegionModel (M.
Specifically, this task is structured in three ascending levels of capability: (1) Generating detailed descriptions for a single region is the foundation, e.g., "describe <F <Frompt 1> in detail", where <Prompt1> actually denotes binary mask and specified by the user. It requires the model to accurately perceive and articulate the fine-graine attributes contained strictly within the boundaries of given prompt. (2) The next stage requires understanding the given region with the necessary global contexts. This moves beyond isolated analysis, requesting to aggregate information from the broader, unmasked scene. This capability critical for advanced reasoning tasks such as position identification (i.e., locating object "the second from the left in the third row")) and non-entity recognition (e.g.. correctly identifying a reflection in a mirror versus physical object), where the prompt itself is insufficient for correct interpretation. (3) Finally, the task culminates in the ability to perceive, understand, and describe the relationship between multiple regions. This assesses the capacity for true compositional reasoning by requiring it to articulate the spatial, functional, or interactive connections between different prompts.
3.2 MODEL ARCHITECTURE
The task definition above requires overcoming the contextual blindness inherent in models that analyze prompted regions isolation. As established, this myopic focus can lead to fundamental reasoning errors, such as misidentifying a frog- shaped slipper as a real frog because the surrounding bedroom context is ignored. Therefore, our architectural design of Grasp Any Region (GAR) guided by a central principle: achieve simultaneously preserving and leveraging the global context of the entire scene. Illustrated Figure we introduce two new components into the architecture: (1) simple yet effective prompt encoding scheme, and (2) novel Rol-aligned feature replay technique.
understunding of the prompted region while
Prompt Encoding and Integration. To integrate spatial guidance into the vision backbone, introduce a lightweight prompt encoding mechanism similar to Lian et al. (2025) and Sun et al. (2024). The input binary mask, which specifies the region(s) of interest, is first processed by simple convolutional block (LeCun et al., 1989) to produce a mask embedding. This zero pero-initialized (Zhang al., 2023) mask embedding is then added ViT's (Dosovitskiy et al., 2021) patch embeddings.
4

---

Published as a conference paper at ICLR 2026
Seed Captioner
Seed Dataset Desorche Anyching-
Fine-Grained Dataset
Fine-Grained Captioner
Fine-Grained Captioner
ofompbh th maakm
<boeetl-llhe soug sokiee
Final Dataset
LLM-Merger Qw
Colegery Ausotation
L.M--Merger 8
Graup Any Region-2.5M 2.5M
chremstii hote nold chrepttc abkic
Rokerian thure Oerokee-Aneeeiig
Whrt eniiii fettee
and
Detailed Description
sodighng show
GAR-IB 8B
Retatlas-tran Cigtke
Compositional Interaction
choraeti batting chamaazz
The aivariii........ latniig raader iusta cucktt hill Be tonal bull
Figure Illustration of our training data pipeline, which mainly includes two rounds of captioning and judging. Specifically, (1) starting from using the seed dataset to train a seed captioner, we first construct 456K fine-grand ene-grained descriptions. Subsequently, (2) we utilize both datasets to obtain fine-grained captioner, and leverage the annotations of the Panoptic Scene Graph (PSG) dataset (Yang et al., 2022) to provide sufficient relation-aware captions and question-answente pairs. Finally, our GAR models are trained with all three parts.
RoI-aligned Feature Replay. To simultaneously provide sufficient local details and maintain necessary global context, we introduce the Rol-aligned feature replay technique. Specifically, our model processes the full, uncropped image (with the encoded mask prompt) with AnyRes (Liu et al., 2024), producing global feature map that is rich in contextual information. Based on the input mask, we then derive a corresponding bounding box for the region of interest and employ Rol-Align (He et al., 2017) to gather the relevant feature vectors directly from the global feature map. Because the features are extracted from feature map that was computed over the entire image, they are inherently contexl-aware, which elegantly avoids the pitfalls of local-only processing in Lian et al. (2025). At the same time, provides the subsequent language model with a sufficiently detailed, high-resolution representation of the prompted region, enabling to perform fine-grained understanding. This replay of context-rich features allows GAR to simultancously "zoom in" on detail without "losing sight" of the bigger picture. Ablations of this design can be found in Table 8, where we demonstrate that this design is capable of both (1) providing sufficient local details and (2) preserving global contexts.
3.3 TRAINING DATA PIPELINE
To enhance model capabilities from basic object recognition with single region to complex relational reasoning with multiple regions, we design a multi-stage process to generate a large-scale, high- quality dataset, as illustrated in Figure 4. Ablations of each round can be found in Table 10. Prompts for each stage can be found in Appendix G.
Round 1: Enhance Recognition Capability. Initially, we start from the Deseribe Anything- -1. 1.5M dataset (Lian et al., 2025). However, we observe deficiencies in its fine-grained recognition capability, limiting the quality of generated captions for more complex scenarios. To address this, we integrated images and masks provided by Sun et al. (2024), which is a subset of ImageNet-21K (Deng et al., 2009), an extremely fine-grained classification dataset and renowned for its detailed and extensive category labels. We employ the seed captioner to generate descriptions and then utilize an LLM to validate these generated captions against the ground-truth categories, resulting in refined fine-grained dataset of 456K samples. We utilize both datasets to train a fine-graines captioner.
Round 2: Supporting Multiple Prompts. To further enable understanding multiple prompts, we incorporated the Panoptic Scene Graph (PSG) dataset (Yang et al., 2022). which is rich in relational information. We first query the fine-grained captioner to generate a detailed description for each region. Subsequently, we regard @wen2.5-72D (Team, 2024) as the LLM-Merger, together with the original annotations provided by the PSG dataset (Yang et al., 2022), to generate: (1) 144K rich object descriptions that explicitly integrate relational context, (2) 144K guestion-answering pairs designed to probe the understanding of complex relationships, and (3) 126K multiple-choice questions. We construct relation dataset with 41 414K samples in total during this stage.
5

---

Published as a conference paper at ICLR 2026
3.4 GAR-BENCH
Finally, we introduce GAR-Bench, a comprehensive benchmark suite designed systematically evaluate the region-level comprehension capabilities of MLLMs beyond simply describing a single region. Specifically, it is structured into two primary components: (GAR-Bench-Can Canp) and a multifaceted visual question answering task (GAR- (GAR-Bench- Bench--QA).. The captioning component is designed to assess model's ability describe the complex relationships and interactions between multiple visual prompts in a cohesive narrative. The VQA component further dissects a model's understanding into two key areas: (1) its ability to perceive basic attributes for a given prompt, and (2) its capacity for advanced, region- compositional reasoning that requires synthesizing information from the prompt and its surrounding context.
captioning task
GA GAR-Bencher CaR-Bench-Cap goes beyond isolated object descriptions and measures the ability to perform compositional scene understanding. In this task, a model is provided with an image and two or more distinct visual prompts. It contains two sub-tasks: (1) simply describe the relationship, and (2) generate detailed captions including necessary relationships. For the "simple" protocol, models are directly asked with "what is the relationship between <Prompt 1> and <Prompt2>" and are required to answer the question simply. For the "detailed" protocol, for instance, <Prompt1s 1> highlights person and <Prompt 2> is bike, the model is not evaluated on its ability to describe each independently, but rather on its capacity to generate an accurate description of their relation like, "<Prompt is riding <Prompt2>". The models need to perform spatial reasoning, action recognition, and semantic integration across disparate image regions, thereby quantifying its ability to interpret scene as a cohesive whole rather than collection of independent parts.
MARRBench--QL is designed to shift the evaluation from static description to dynamic, interactive dialogue. This task assesses the ability to answer specific questions about one or more prompted regions, directly measuring its comprehension rather than its descriptive fluency. To provide a comprehensive and but complementary sub-tasks: "perception" and "reasoning".
evaluation of the reasoning abilities, we divide it into two distinct
Perception evaluates the model's foundational ability to recognize basic visual attributes of single object, serving as a litmus test for its core visual acuity. This task quantifies the ability to perceive the foundational details. Specifically, for given visual prompt, the model asked targeted questions about its intrinsic visual properties, specifically focusing on color, shape, material, and texture/pattern.
Reasoning is designed to probe higher-order cognitive abilities. This component challenges the model to synthesize information from local prompts, global context, and the relationships between multiple prompts to arrive at logical conclusions. It is composed of several sub-tasks, each targeting unique and challenging aspect of visual reasoning:
Position evaluates the model's grasp of spatial arrangement and ordinal logic within global context. A model is presented with a mask on a single object within a larger group and asked to identify its precise position in complex, grid-like structure. Answering correctly requires the model to not only recognize the masked object but also to process the entire scene structure.
Non-Entity Recognition designed to test this specific capability by requiring the model to leverage sufficient global context. For instance, the given prompt might highlight reflection in mirror, the shadow of a person, a face depicted on a television screen, and so on. The model is then queried to determine if the prompted region corresponds to physical entity. Success in this task demonstrates that the model is performing sophisticated context-aware reasoning rather than simple pattern matching on the masked pixels alone.
Relation measures the capacity for complex compositional reasoning across multiple prompts. In this challenging setup, the model is presented with several visual prompts and must deduce the intricate spatial or logical relationship between them. A key challenge is the inclusion of redundant prompts. To arrive at the correct answer, the model must ignore the potentially distracting informa- tion. requires the model to build mental "scene graph", which is essential for comprehending complex object assemblies and interactions in cluttered, real-world environments.
For more benchmark details, including the annotation pipeline and statistics, please refer to Ap- pendix B.1 and Appendix B.2, respectively.

---

Published as a conference paper at ICLR 2026
Table I: Comparison on GAR-Bench-VOL VQA. indicates this subtask evaluates the interaction between multiple visual prompts. i means evaluated with the thinking mode. Our GAR-1B even outperforms Intern VL3-78B. Moreover, GAR-8B surpasses private state-of-the-ar non non-thinking model GPT-4o.
Perception (198)
Reasoning (226)
Method
Overall
Color (69)
Shape (64)
Texture
Material (36)
Position (64)
Non-Entity" (61)
Relation (101)
(29)
Private General MLLMs
GPT-4o
53.5
34.8
65.3
48.3
52.8
57.8
60.2
61.4
031
61.3
58.0
70.3
55.2
63.9
54.7
49.2
71.3
Gemini-2.5-Prol
64.2
62.3
68.8
58.6
66.7
64.1
64.9
70.3
Public Generul MLLMs
Qwen2.5-VL-3B
34.4
29.0
25.0
34.5
30.6
43.8
26.2
Owen2.5-VL-7B
41.7
40.6
44.8
59.4
Owen2.5-VI--327
50.9
46.4
53.1
41.4
30.6
71.9
36.1
58.4
Qwen2.5-V1.-72B
52.8
46.4
50.0
65.5
33.3
68.8
44.3
57.4
InternVL3-2B
35.1
30.4
21.9
48.3
38.9
48.4
26.2
38.6
InternVL3-8B
36.2
37.5
58.6
51.6
InternVL3-38B
46.5
39.1
40.6
51.7
55.6
60,9
36.1
47.5
Intern 1
50.5
44.9
54.7
58.6
61.1
53.1
47.5
45.5
Region MLLMs
Sa2VA-8B
45.3
29.6
54.7
VP-SPHINX-13B
37.5
33.3
25.0
44.8
38.9
60.9
34.3
32.7
DAM-3B
38.2
55.1
39.1
41.4
36.1
31.3
36.1
31.7
PAM-3B
2.9
6.9
GAR-1B
50.6
55.1
46.9
69.0
47.2
21.9
62.3
56.4
GAR-8B
59.9
59.4
54.7
75.9
52.8
48.4
60.7
68.3
Table 2: Comparison of localized relational cap- tioning on our GAR-Bench-Cap We utilize GPT-4o (OpenAI, 2024a) with cropped images and masks to judge the correctness of the answer.
Table 3: Comparison on detailed localized cap- tioning on DLC-Bench (Lian et al., 2025). f indicates using GPT-4o (OpenAl, 2024a) with extra cropped images as judge, otherwise per- forming text-only judging, where discussions can found Appendix F, $ means our evaluation with the official checkpoint.
Overall (204)
Simple (97)
Detailed (107)
Method
Private General MLLMs
GPT-4o
51.5
39.2
62.6
Method
Avg.
Pos.
Neg.
03
56.9
37.1
74,8
Private General MILLMs
Gemini-2.. 2.5-PPo
59.3
51.6
66.4
Gemini-2.5-Pry
55.8
36.5
75.2
Public General MLLMs
GPT-4o
61.5
43.4
79.6
0wen2.5-VL-30
22.5
9.3
34.6
ol
62.5
46.3
78.8
Qwen2.5-VL--7B
32.4
12.4
50.5
Region MLLMs
Owen2.5-V1L-322
36.8
17.5
54.3
Shikra-7B
22.2
2.7
41.8
Intem VL3-2B
29.4
14.4
43.0
Ferret-7B
22.4
6.4
38.4
Intern L3--B
33.8
11.3
54.2
RegionGPT-7B
27.2
Intern
45.1
29.9
58.9
VP-SPHINX-L3B
22.5
11.7
33.2
Region MLLMs
DAM-3B
64.5t
47.2
81.81
DAM-3B
13.1
17.5
10.3
GAR-IB
67.9
48.9
PAM-3B
21.1
39.3
GAR-SB
67.4
50.2
84.6
VP-SPHINX-13B
32.3
27.8
39.3
Sa2VA-8B 8B
45.6
46.4
44.9
DAM-3B'
72.6
61.83
83,4
GAR-IB
57.5
56.7
63.6
GAR-IB
77.1
66.2
88.0
GAR-SB
62.2
66.0
64.5
GAR-8B
77.0
68.0
86.0
EXPERIMENTS
Owing to page limitations, we only present the key properties this section. For implementation details, comparative baselines, and ablation studies, please refer Appendix C.
7

---

Published as a conference paper at ICLR 2026
Table 4: Zero-shot results on region-level detailed image cap- tioning on Ferret-Bench (You et al., 2023) and and MDVP- Bench (Lin al., 2025a). We adopt SAM (Kirillov et al., 2023) to produce masks conditioned on bounding boxes for MDVP- Bench (Lin et al., 2025a). All results are our reproduction using the official checkpoint, the original judger GPT-4V is no longer available, and we take GPT-4o as the judge.
Table 5: Results of category-level image recognition on LVIS (Gupta et al., 2019) and PACO (Ra- manathan et al., 2023).
LVIS
PACO
Method
Sim.
IoU
Sim.
loU
Ferret-Bel Bench
MDVP-Bent (Box Caption)
Shikra-7B
49.7
19.8 43.6
11.4
Method
GPT4Rol-7B
51.3
12.0
48.0
12.1
Refer. Desc.
Natural
OCR Multi Pantil Sceenshot
Ferret-7B
63.8
36.6
58.7
26.0
Osprey-7B
107.7
99.4
70.0
81.3
Osprey-7B
65.2
38.2 73.1 52.7
PAM-3B
52.2
71.4
94.3
86.8
84.5
DAM-8B
89.0
77.7 84.2
73.2
DAM-3B
55.0
87.0
127.7
79.4
76.4
PAM-3B
88.6 78.3 87.4
74.9
GAR-1B
56.0
152.6
149.6
103.7
115.3
GAR-1B
91.0 68.2
93.2
72.4
GAR-SB
64.8
178.6
149.1
117.2
123.0
GAR-8B
93.6 88.7
95.5
91.8
Table 6; Zero-shot comparison of detailed localized video captioning VideoRefer-Bencht (Yuan et al., 2025b). For "single-frame", we select the target frame and apply AnyRes with max num tiles=16. For "multi-frame", we uniformly sample 16 frames and turn off AnyRes.
Single-Frame
Multi-Frame
Method
Avg.
TD
HDD
Avg.
SC
AD
HD
General MLLMs
LLaVA-One Vison-BB
2.12
2.62
2.19
2.07
2.48
2.50
2.41
Qwen2-VL-7B
2.39
2.97
2.24
2.03
2.31
2.55
3.30
2.54
2.22
2.12
IntemVL2-26B
2.84
3.55
2.99
2.57
2.25
3,20
4.08
3.35
3,08
2.28
GPT-4o
2.95
3.34
2.96
3.01
2.50
3.25
4.15
3.31
3.11
2.43
Region MILMS
Elysium-7B 7B
1.57
2.35
0.30
0.02
3,59
Ferret-7B
2.18
3.08
2.01
1.54
2.14
2.23
2.38
1.38
Osprey-7B
2.34
3.19
2.16
1.54
2.45
2.41
3.30
2.66
2.10
1.58
Artemis-7B
2.26
3.42
1.34
1.39
2.90
DAM-8B
3.34
4.45
3.30
3.03
2.58
GAR-1B
2.72
4.41
2.98
1.09
2.40
2.83
4.38
3.01
1.61
2.30
GAR-SB
2.75
4.41
2.96
1.58
2.45
3.25
2.57
3.42
Advanced comprehension requires precisely modeling complex relationships between multiple prompts. To evaluate this capability, we conducted comprehensive comparison on our GAR- Bench-VQA. As demonstrated in Table 1, GAR-8B achieves impressive overall score of 54.5, surpassing even the powerful, private, state-of-the-art non-thinking model, GPT-4o (OpenAI, 2024a). Furthermore, the efficiency and effectiveness of approach highlighted by GAR-IB. Despite its significantly smaller size, it scores 50.6 overall, outperforming large-scale public models like Intern VL.3771 (Zhu al., 2025). This advantage is particularly evident in fine-grained perception tasks, where GAR-1B 1B and GAR-8B achieve "Texture" scores 69.0 and 75.9, respectively.
Detailed localized captioning requires generating detailed descriptions for given regions with multiple sentences. We benchmark our GAR models on series of challenging datasets, and the results consistently demonstrate their state-of-the-an capabilities. As shown in Table 2, our GAR- Bench-Cap, GAR-IB and GAR-SB achieve the highest overall scores of 57.5 and 62.2, respectively, even exceeding that powerful private models like Gemini-2.5-PHr (DeepMind, 2025b). This superiority is further confirmed on the DLC-Bench (Lian et al., 2025) in Table 3, where GAR-IB and GAR-8B again outperform top models like DAM-3B using either I.LaMA3.1 (Grattafiori et al., 2024) or GPT--4o (OpenAI, 2024a) as the judge. The zero-shot performance of our models on Ferret- Bench (You et al., 2023) and MDVP-Bench (Lin et al., 2025a), detailed in Table 4, is particularly noteworthy. On both benchmarks, our GAR emerges single category. Specifically on MDVP-Bench our models show commanding lead, with GAR-8B achieving score of 178.6 6 natural images, result that is substantially higher than any competitor. Collectively, these comprehensive evaluations across multiple benchmarks unequivocally establish GAR the oate-off-the-ar for producing rich, accurate, and detailed localized captions.
the top-performing model across every
8

---

Published as a conference paper at ICLR 2026
Table 7: zero-shot comparison of detailed video understanding on VideoRefer- (Yuan al., 2025b). i indicates trained in in-domain VideoRefer- with regard to CeeoRefer-BBnnche Notably, our zero-shot GAR-8B even outperforms in in-oomain VideoRefer-7B.
Basic Questions (235)
Sequential Questions (256)
Relationship Questions (252)
Reasoning Questions (143)
Future Predictions
Overall (1000)
Method
(114)
General MLLMs
IntemVL2-266
65.0
58.5
63.5
53.4
88.0
78.9
Qwen2-VL VL-7B
66.0
62.0
69.6
54.9
87.3
74.6
LLaVA-One OieVinon--
67.4
58.7
62.9
64.7
87.4
76.3
GPT-4o
71.3
62.3
74.5
66.0
88.0
73.7
Region MLLMs
Osprey-7B
39.9
45.9
30.0
48.6
23.7
Ferret-7B
48.8
35.2
44.7
41.9
70.4
74.6
VideoRefer-7B'
71.9
75.4
68.6
59.3
89.4
78.1
GAR-1B
69.9
75.0
69.9
59.7
83.2
63.7
GAR-8B
72.0
77.2
71.0
61.7
86.6
68.1
Open-class category-level image recognition requires the model to recognize the category of the object and part entities. We evaluate this capability in Table 5. Our GAR-8B demonstrates significant leap in performance, establishing a new statc-of-the-art. It consistently outperforms all prior methods across every metric, achieving top scores of 93.6 semantic similarity and 88.7 semantic IoU on LVIS (Gupta et al., 2019), and 95.5 semantic similarity and 91.8 semantic IoU on PACO (Ramanathan al., 2023). This indicates its superior ability in both semantic understanding and precise localization. These results demonstrate the effectiveness of GAR for complex recognition tasks, showcasing its robust performance in identifying a diverse range of object categories.
Extension to videos is straightforward. Similar to Lian et al. (2025), we simply extend our GAR models to videos and evaluate them on Bench (Yuan et al., 2025b) in Table 6 and Table 7, respectively, We uniformly sample 16 frames to represent a video. Our GAR-SB surpasses DAM-SB (Lian et al., 2025) under the zero-shot setting. zero-shot GAR-8B
CetoReffr-Bencch (Yuan et al., 2025b) and VidcoRefer-
More importantly, demonstrated in Table 7, VideoRefer-755 demonstrating its strong comprehension capabilities can be easily transferred to videos. However, as our models actually trained with images, they get reasonably low scores on temporally related tasks, e.g., temporal description (TD) in Table 6 and future predictions in Table 7.
outperforms in-domain
We provide qualitative comparisons between GAR-8B with DAM-3B (Lian et al., 2025) on detailed localized captioning on DLC-Bench (Lian et al., 2025) in Figure 5. As demonstrated in the figure, our GAR-SB is more capable of generating precise descriptions, especially when the category of the given prompt can be determined only when understanding sufficient global contexts. More comparisons can be found in Appendix
Qualitative Results.
5 CONCLUSION
This paper introduces Grasp Any Region (GAR), a family of MILMs for region understanding, and GAR-Bench, systematic evaluation framework that not only provides more accurate evaluation of single-region comprehension, but also for multi poalippprompt interaction and advanced compositional reasoning. On detailed captioning benchmarks (Lian al., 2025; You et al., 2023; Lin al., 2025a), GAR demonstrates superior performance over DAM (Lian et al., 2025). More importantly, our GAR achieves advanced comprehension capability in modeling interactions between multiple prompts. Specifically, on GAR-Bench-VOR GAR-1B even surpasses Intern LL3-78B (Zhu et al., 2025). On VideoRefer-Bench (Yuan al., 2025b), our zero-shot GAR-8B even outperforms in-domain VideoRefer-775 (Yuan et al., 2025b). We hope our work inspires the community to develop MLLMs that can perceive, interrogate, and understand the dense visual world more effectively.
LCKNOWLED
This work was supported by the Beijing Natural Science Foundation (No. L257015) and the National Natural Science Foundation of China (No. 62320106010)

---

Published as a conference paper at ICLR 2026
Question: Describe the masked region in detail.
Question: Desurbbe the masked region detall.
GAR-ER: A black leather handhag with smooth, slightly finss. The visible shows rectungular shape with subdle along the
GAR-SR:
The mierowave oven las white exterior with restangulat
shape. features peuminent, curved handle the front dour, which alsu white, control
right buttons and a small display screem. The top the microwave has vented section for verilation.
JB: A black leather jacluet with a smooth testure and sightly glossy faish. The festures classic collar design and visiblle vaa running horizontally acroes the upper arm
DAM-3B: A wouden cabinet with vertical grain putteen, festuring single visible door handle right side, which cubinet has flat top and appears be part of larger piece furniture.
matallic.
Figure 5: Qualitative comparisons on DLC-Bench (Lian et al., 2025), where green indicates correct descriptions and red means errors.
ETHICS STATEMENT
Our research is grounded in ethical practices, with particular attention paid to the responsible use of data. All datasets employed in this study are publicly available and well-established within the computer vision community. Specifically, our training data includes ImageNet-211 (Deng et al., 2009) and the PSG (Yang et al., 2022) dataset (with image sources from COCO (Lin et al., 2014), while benchmarking 2021), and SA-IB (Kirillov et al., 2023). Our use of this data is in accordance with their provided licenses and intended academic purpose.
conducted FSC-147 (Ranjan et al., 2021), RGBD-Mirror (Mei et al.,
REPRODUCIBILITY STATEMENT
We are committed to ensuring the reproducibility of the research presented in this paper. To this end, comprehensive implementation details for our models and experiments are provided Appendix C, including the training procedures and all hyperparameters used. Furthermore, upon acceptance of this paper, all source code, datasets, and trained model checkpoints will be made publicly available.
REFERENCES
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang. Jun Tang, al. Qwen2.5-vl technical report. arXiv preprint arXiv: 13923, 2025.
Jierun Chen, Fangyun Wei, Jinjing Zhao, Sizhe Song, Bohuai Wu. Zhuoxuan Peng, S-H Gary Chan, and Hongyang Zhang, Revisiting referring expression comprehension evaluation in the of large multimodal models. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recoggnttion (CVPR). pp. 513 513-524, 2025.
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Zhao. Shikra: Unleashing multimodal I1m's referential dialogue magic. arXiv preprint arXiv:2306. /5/95, 2023.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang. Yu Qiao, Dahua Lin. et al. Are arXiv:2403.22 20330, 2024a.
the right way for evaluating large laniunel models? arXiv preprint
Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Fun Jeon, Yuwei Fang, Hsin- Ying
Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, et al. Panda-70m: Captioning 70m videos with
10

---

Published as a conference paper at ICLR 2026
multiple cross-modality teachers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13331, 2024b.
Zhe Chen, Weiyun Wang. Hao Tian, Shenglong Ye. Zhangwei Gao. Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are to gpt-4v? closing the gap to commercial multimodal models with open-source suites, Science China Information Sciences, 67(12))20011, 2024c.
Jang Hyun Cho, Andrea Madotto, Effrosyni Mavroudi. Triantafyllos Afouras. Tushar Nagarajan, Muhammad Maaz, Yale Song. Tengyu Ma, Shuming Hu. Suyog Jain, et al. Perceptionlm: Open-access data and models for detailed visual understanding. arXiv preprint arXiv:2504. 13180, 2025.
XTuner Contributors. Xtuner: A toolkit for efficiently fine-tuning Ilm.
xtuner, 2023.
Google DeepMind. Gemini-2.5-flash. https://deepminn http:s///eeeennnh.ogooooo///o//a 2025a.
Google DeepMind. 2:5-pi--..--rr... https::/ http://waaa.....ooooo..///aaaaaa
Jia Deng, Wei Dong, Richard Socher, Ii-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 248-255. leee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition scale. International Conference on Learning Representations (ICLR). 2021.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, al. The Ilama herd of models. arXiv preprins arXiv:2407. 217883 2024.
Qiushan Guo, Shalini De Mello, Hongxu Yin, Wonmin Byeon, Ka Chun Cheung, Yizhou Yu, Ping Luo, and Sifci Liu. Regiongpt: Towards region understanding vision language model. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 17796-13866 2024.
Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.
5356-5364, 5364, 2019.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2961-2969, 2017.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg. Wan-Yen Lo. et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4015-4026, 2023.
Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4): 541-551, 1989.
Jungbeom Lee, Sanghyuk Chun, and Sangdoo Yun. Toward interactive regional understanding in vision large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). pp. 6416 6429, 2024.
Weixian Lei, Jiacong Wang, Haochen Wang, Xiangtai Li, Jun Hao Liew, Jiashi Feng, and Zilong Huang. The scalability of simplicity: Empirical analysis of languu-la learning with single transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2025.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.0m 03326, 2024.
Xiangtai Li, Tao Zhang, Yanwei Li, Haobo Yuan, Shihao Chen, Yikang Zhou, Jiahao Meng, Yucyi Sun, Shilin Xu, Lu Qi, et al. Denseworld- Im: Towards detailed dense grounded caption in the real world. arXiv preprint arXiv:2506.22 24 1420 2025.
Long Lian, Yifan Ding, Yunhao Ge, Sifei Liu, Hanzi Mao, Boyi Li. Marco Pavone, Ming- Yu Liu, Trevor Darrell, Adam Yala, et al. Describe anything: Detailed localized image and video captioning. arXiv preprint 250iv:2504. 16072, 2025.
11

---

Published as a conference paper at ICLR 2026
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. European Conference on Computer Vision (ECCV), pp. 740-755, 2014.
Weifeng Lin, Xinyu Wei, Ruichuan An, Peng Gao, Bocheng Zou, Yulin Luo, Siyuan Huang. Shanghang Zhang, and Hongsheng Li. Frrawaand-udeenee you want. In International Conference on Learning Representations (ICLR), 2025a.
Leveraging visual prompts to enable milms to comprehend what
Weifeng Lin, Xinyu Wei, Ruichuan An, Tianhe Ren, Tingwei Chen, Renrui Zhang, Ziyu Guo, Wentao Zhang, Lei Zhang, and Hongsheng Li. Perceive anything: Recognize, explain, caption, and segment anything in images and videos. arXiv preprint 25Xiv:2500.05332,, 2025h.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in Neurul Information Processing Systems (NeurIPS), 36:34822-33916, 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Ilava- next: Improved reasoning, ocr, and world knowledge. 2024-01-30- Lavaa-neett 2024.
wal.giitub..... wwt.tuu.../bbaa ic/blog/
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03w00 2016.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, arXiv preprint arXiv: 17X.v:1111.05001, 2017.
Chuofan Ma. Yi Jiang, Jiannan Wu, Zehuan Yuan, and Xiaojuan Qi. Groma: Localized visual tokenization for grounding multimodal large language models. In European Conference on Computer Vision (ECCV), pp. 417-435. Springer, 2024.
Haiyang Mei, Bo Dong. Wen Dong, Pieter Peers, Xin Yang. Qiang Zhang, and Xiaopeng Wei. Depth-aware mirror segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3044-3053, 2021.
OpenAI. Openai-gpte -ponai-gpt-4o. httph::t/////c
2024a.
OpenAL. Openai-ol. https: httpps///oppaaa..ccm/////////// int roducing- -4nniii/
OPenAL. Openai-o3,
Jihao Qiu, Yuan Zhang, Xi Tang. Lingxi Xie, Tianren Ma, Pengyu Yan, David Doermann, Qixiang Ye, and Yunjic Tian. Artemis: Towards referential understanding in complex vidcos. Advances in Neural Information Processing Systems (NeurIPS), 37:1 141111111111111 114447, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICMI.), pp- 8748-8763, 2021.
Vignesh Ramanathan, Anmol Kalia, Vladan Petrovic, Yi Wen, Baixue Zheng, Baishan Guo, Rui Wang, Aaron Marquez, Rama Kovvuri, Abhishek Kadian, et al. Paco: Parts and attributes of common objects. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7141- 7151, 2023.
Viresh Ranjan, Udbhav Sharma, Thu Nguyen, and Minh Hoai. Learning to count everything. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 3394-3403, 2021.
Hanoona Rasheed, Muhammad Maaz, Sahal Shaji. Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing. Ming-Hsuar Yang. and Fahad S Khan. Glamm: Pixel grounding large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13009-13018, 8, 2024.
Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng. Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 8430-8439, 2019.
Zeyi Sun, Ye Fang, Tong Wu, Pan Zhang, Yuhang Zang, Shu Kong, Yuanjun Xiong, Dahua Lin, and Jiaqi Wang. Alpha-clip: A clip model focusing on wherever you want. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13019-13029, 2024.
Qwen Team. Qwen2.5 technical report. arXiv preprint arXiv:2412. 2024.
12

---

Published as a conference paper at ICLR 2026
Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang. Manoj Middepogu, Ziteng Wang, et al. Cambrian-1: fully open, vision- centric exploration multimodal llms, Advances in Neural Information Processing Systems (NeurIPS), 37: 87310-87535 87356, 2024a.
Shengbang Tong. Zhuang Liu, Yuexiang Zhai, Yi Ma. Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal Ilms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9568-9578 2024b.
Han Wang. Yongjie Ye, Yanjie Wang. Yuxiang Nie, and Can Huang. Elysium: Exploring object-level perception in videos via mllm. In European Conference Computer Vision (ECCV), pp. 166-185. Springer, 2024a.
Haochen Wang. Xiangtai Li. Zilong Huang, Anran Wang. Jiacong Wang. Tao Zhang, Jiani Zheng, Sule Bai, Zijian Kang, Jiashi Feng, et al. Traceable evidence enhanced visual grounded reasoning: Evaluation and methodology. arXiv preprint arXiv:2507.0790 2025a.
Haochen Wang, Yucheng Zhao, Tiancai Wang, Hanqiang Fan, Xiangyu Zhang, and Zhaoxiang Zhang. Ross3d: Reconstructive visual instruction tuning with 3d-awareness. Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), 2025b.
Haochen Wang, Anlin Zheng, Yucheng Zhao, Tiancai Wang, Zheng Ge, Xiangyu Zhang, and Zhaoxiang Zhang. Reconstructive visual instruction tuning. In International Conference Learning Representations (ICLR), 2025c.
Jiacong Wang. Zijiang Kang. Haochen Wang, Haiyong Jiang, Jiawen Li, Bohong Wu, Ya Wang, Jiao Ran, Xiao Liang, Chao Feng. et al. Vgr: Visual grounded reasoning. arXiv preprint arXiv:2506.1199 2025d.
Peng Wang, Shuai Bai, Siman Tan, Shijic Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, et al. -1: Enhancing rision-langueen model's perception of the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024b.
Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing. Shenglong Ye. Jie Shao, et al. Internv13.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508. / 18265, 2025e.
Penghao Wu and Saining Xie. V+: Guided visual search a core mechanism in multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13084-13094, 2024.
Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai. Huazuo Gao, Yiyang Ma. Chengyue Wu, Bingxuan Wang. al. Deepseck-vit Mixture-of- expeerts vision- vision-language models for advanced multimodal understanding. arXiv preprint arXiv:2412.1030,, 2024.
xAI. Grok. 2024.
Jianwei Yang, Hao Zhang, Feng Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v, arXiv preprint 1rXiv:2310.11441, 2023.
Jingkang Yang, Yi The Ang, Zujin Guo, Kaiyang Zhou, Wayne Zhang, and Ziwei Liu. Panoptic scene graph generation. In European Conference on Computer Vision (ECCV), pp- 178-196. Springer, 2022.
Ruofeng Yang. Bo Jiang, Cheng Chen, Baoxiang Wang, Shuai Li, et al. Few-shot diffusion models escape the curse of dimensionality. Advances in Neural Information Processing Systems (NeurIPS), 37:68528 68558, 2024a.
Ruofeng Yang, Zhijic Wang, Bo Jiang, and Shuai Li. Leveraging drift to improve sample complexity of variance exploding diffusion models. Advances in Neural Information Processing Systems (NeurIPS). 37: 107662-107702. 2024b.
Ruofeng Yang. Jiang, Shuai The polynomial iteration complexity variance exploding diffusion models: Elucidating sde and ode samplers. In The 28th International Conference on Artificial Intelligence and Statistics, 2025a.
Ruofeng Yang, Zhaoyu Zhu, Bo Jiang, Cheng Chen, and Shuai Li. Elucidating rectified flow with deter- ministic sampler: Polynomial discretization complexity for multi and one-step models. arXiv preprint 2xxi::5508.08735. 2025b.
Ruofeng Yang, Yongcan Li, Bo Jiang, Cheng Chen, and Shuai Li. Multi-subspace multi-modal modeling for diffusion models: Estimation, convergence and mixture of experts. arXiv preprint arXiv:2601.0444 2026.
13

---

Published as a conference paper at ICLR 2026
Нaoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao. Shih-Fu Chang, and Yinfci Yang. Ferret: Refer and ground anything anywhere any granularity. arXiv preprint arXiv:2310.07704, 2023.
Haobo Yuan. Xiangtai Tao Zhang, Zilong Huang, Shilin Xu, Shunping Ji, Yunhai Tong, Lu Qi, Jiashi Feng. and Ming- Hsuan Yang. Sa2va: Marrying sam2 with llava for dense grounded understanding of images and videos. arXiv preprint 2r00v:2501.004000 1,4001, 2025a.
Yuqian Yuan, Wentong Li, Jian Liu, Dongqi Tang, Xinjie Luo, Chi Qin, Lei Zhang, and Jianke Zhu. Osprey: Pixel understanding with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), pp. 28202-28211, 2024.
Yuqian Yuan, Hang Zhang, Wentong Li, Zesen Cheng, Boqiang Zhang, Long Li, Xin Li, Deli Zhao, Wenqiao Zhang, Yucting Zhuang, al. Videorefer suite: Advancing spatial-temports object understanding with vidco llm. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), pp. 18970-18980, 2025b.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp- 11975-11986, 2023.
Lymin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the TEEE/CVF International Conference on Computer Vision (ICCV), pp. 3836-3847, 2023.
Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang. Yu Liu, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuming large language model on region-of-interest. In European Conference on Computer Vision (ECCV), pp. 52-70. Springer, 2024a.
Tao Zhang, Xiangtai Li, Hao Fei, Haobo Yuan, Shengqiong Wu, Shunping Ji, Chen Change Loy, and Shuicheng Yan. Olaval Bridging image-level, object-level. pixel-level leve reasoning and understanding. Advances in Neural Information Processing Systems (NeurIPS), 37:71737-71767, 2024b.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhanyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, al. Interv13:: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504. 10479, 2025.
14

---

Published as a conference paper at ICLR 2026
APPENDIX
A OVERVIEW
Here is the table of contents of this appendix:
In Appendix B, we introduce details of our GAR-Bench, including the annotation pipeline and statistics.
In Appendix C, provide more implementation details as well experimental results. Detailed ablations of each component can be found in this section.
In Appendix D, we provide qualitative results on both detailed image captioning and understanding, and localized video captioning and understanding.
In Appendix E, we discuss potential limitations and analyze failure cases.
In Appendix F, we discuss some underlying issues towards the evaluation protocols of DLC-Bench (Lian al., 2025).
In Appendix G, we provide all prompts utilized to construct our dataset.
* Finally in Appendix H, we discuss the use of LLMs in preparing this paper.
B DETAILS OF GAR-BENCH
B. ANNOTATION PIPELINE
The construction of GAR- follows rigorous, semi-automated pipeline designed generate high-quality, diverse, and challenging data. This process combines the strengths of advanced foundation models for initial data generation with the nuanced judgment of a team of MLLM experts for curation, annotation, and quality control.
Image Selection. To ensure the relevance and challenge of our sub-tasks, we begin by carefully curating source images from existing datasets known to contain specific visual patterns. For the "relation" tasks, we source images from the Panoptic Scene Graph (PSG) dataset (Yang et al., 2022), which is rich in complex scene graphs and explicit object relationships, providing natural foundation for multi-prompt interaction queries. For the "non-entity recognition" task, we utilize the RGBD- Mirror dataset (Mei et al., 2021), it specifically contains scenes with mirrors and reflections, allowing us to create unambiguous test cases for distinguishing real objects from illusory ones. For the "position" task, select images from the FSC- 147 dataset (Ranjan et al., 2021), which features images with numerous countable objects often arranged in grid-like patterns, making it ideal for evaluating spatial and ordinal reasoning. Other images are from SA-1B (Kirillov et al., 2023).
Mask Labeling, Following image selection, we generate high-quality segmentation masks for all potential objects of interest. This stage is similar Li al. (2025), which decomposes complex scenes into different objects. while not containing numerous meaningless, trivial objects like those in the SA-IB (Kirillov et al., 2023) dataset.
Object Selection and Annotation. With a high-quality pool of object masks generated, the annotation team performs the critical tasks of selection and annotation. The experts first reviewed the masks, selecting only those with high segmentation quality that are also qualified for the target sub-task. Concurrently, they are responsible for annotating the ground-trute -ttth information required for the benchmark. Specifically, for the "reasoning" protocol of GA G-R-BBnnh--OO annotate the correct answers for relation, ordering, and entity status. For L-AR-Bench-ao Cap, they annotate the ground-truth captions describing the interactions between the selected masked objects.
they meticulously
Automated Attribute Generation. For the "perception" protocol of GA GAR-Bench- VQA, we leverage the advanced capabilities of Gemini-2.5-Pro (DeepMind, 2025b). For each selected and verified object mask, we prompt the model to generate a list of its basic perceptual attributes, including its primary color, shape, material, and any discernible texture pattern.
Quality Control and Formatting. The raw, annotated data then underwent a meticulous, multi- stage quality control process. First, human experts review all machine machine-generated attributes from the previous step to verify their factual correctness and filter out any ambiguous or inaccurate labels. Following this verification, the experts transform the raw annotations into the final benchmark formats.
15

---

Published as a conference paper at ICLR 2026
Reluiie (201)
252
220
SAgp (54
660
59%
47%
2 1615
(274)
64.8%
Pontton
30
Matrii
(28)
8.10
(a) Distribation each disciplate.
(b) Distribation of number of prompts.
(6) Distribution prompts..
Figure 6: Statistics of our GAR-Bench. We (a) slightly prioritize reasoning over perception, and build challenging questions through (b) multiple visual prompts (even have questions with prompts and 9 prompts) and (c) small areas of cach prompt with an average of 4.4%.
For all VQA tasks, they rewrite the question-answer pairs into standardized multiple-choice format, ensuring consistent and objective evaluation. For the captioning task, the ground-trum truth data was structured for compatibility with LLM-as- evaluation protocols similar to Lian et al. (2025).
Difficulty Filtering. As a final quality assurance measure, we implement difficulty filtering process to ensure the benchmark remains challenging for even the most advanced models. Specifically, any question answered correctly by all four state-of-the-eee 72B (Bai et al., 2025), Intern VL3-78B (Zhu et al., 2025), GPT-4o (OpenAI, 2024a), and Gemini-2.. Flash (DeepMind, 2025a), was excluded from the final benchmark.
non-thinking MLLMs, i.e., Qwen2.5-VV VL-
B.2 STATISTICS
Distribution of Each Discipline. As demonstrated in Figure 6a, GAR-Bench slightly prioritizes advanced reasoning (53%) over basic perception (47%) with relatively balanced distribution. In addition, prioritizes complex relational reasoning with multiple prompts in the "relation" protocol.
Distribution of Number of Prompts. As illustrated in Figure 6b, our GAR-Bench even contains questions with prompts and prompts, respectively, leading to an advanced requirement of modeling complex relationships between multiple visual prompts.
Distribution of Areas of Prompts. We compute the relative area of each visual prompt in Figure 6c, where the majority of prompts in GAR-Bence are extremely small, with sharp peak near 0.0. The mean area across all questions 4.4%. This distribution highlights the importance of addressing small-scale and fine-grained understanding.
C MORE EXPERIMENTS
We adopt PerceptionI.A .M series (Cho al., 2025) base model, as demonstrates strong perception capabilities among several open-source MILMMs We perform supervised fine-tuning of the model on our GAR-2.5M using Xtuner (Contributors, 2023) with the AdamW optimizer (Loshchilov & Hutter, 2017) with a global batch size of 64 and a learning rate of le-5 with a cosine decay (Loshchilov & Hutter, 2016).
Implementation Details.
Comparison Baselines. We mainly compare our GAR with both general MLLMs, including state-of-the-art private models (OpenAI, 2024a: 2025: DeepMind, 2025b), and representative pub- lic models (Bai et al., 2025; Zhu et al., 2025; Liu et al., 2023), and region-level MLLMs, in- cluding GLaMM (Rasheed et al., 2024), GPT4Rol (Zhang al., 2024a), Osprey (Yuan al., 2024), Shikra (Chen et al., 2023), Ferret (You et al., 2023), RegionGPT (Guo et al., 2024), OMG- LLaVA (Zhang et al., 2024b), VP-SPHINX (Lin et al., 2025a), Sa2VA (Yuan et al., 2025a), DAM (Lian al., 2025), and PAM (Lin et al., 2025b). We transform masks to boxes for box- level MLLMs, e.g., (Zhang et al., 2024a; Chen et al., 2023: You et al., 2023; Lin et al., 2025b), as our further compare with LLaVA-One VisVin (Li et al., 2024), Qwen2 VL (Wang et al., 2024b), Intern (Chen et al.,
GAR-Bench provides segmentation masks by default. On video benchmarks, Vision
2024c), Elysium (Wang et al., 2024a), Artemis (Qiu al., 2024), and VideoRefer ( (Yuan et al., 2025b).
16

---

Published as a conference paper at ICLR 2026
Table Ablations across different model architectures with PerceptionLLM-IB. indicates using GPT-4o (OpenAI, 2024a) with extra cropped images as the judge, instead of text-only judging. Our proposed Rol-aligned feature replay strategy effectively preserves necessary global contexts. We also report the average latency (ms) to generate the first token and the maximum number of tokens ViT (Dosovitskiy et al., 2021) 1) By default, we set max_num__aa coumtiles-__ for AnyRes (Liu et al., 2024), resulting in maximum of 17 crops in total for one global image.
GAR-Bench
OLC-Benchh
Inference Speed
Global
Local
Caption
VQA
Avg.
Pos.
Neg.
Latency
ViT Tokens
image + mask
20.1
37.8
69.3
60.2
78.4
36.1
256
image mask cross-attention
19.1
40.0
68.8
57.3
80.3
57.1
4,608
image + mask
image mask
28.4
36.6
77.4
70.1
84.8
93.1
4,608
image mask
Rol-aligned feature replay
57.5
50.6
77.1
66.2
88.0
87.7
4,352
image
Rol-pooled feature
39.3
39.1
50.0
37.3
62.7
78.2
4,352
image Rol-aligned feature replay
42.1
40.1
67.1
55.8
78.4
90.8
4,608
image
17.8
35.4
65.0
59.3
70.7
30.4
256
image mask
32.7
40.4
51.7
40.4
63.0
75.0
4,352
Table 9: Ablations across different model architectures with different base models. f indicates using GPT-4o (OpenAI, 2024a) with extra cropped images as the judge, instead of text-only judging. Our proposed Rol-aligned feature replay strategy effectively preserves necessary global contexts.
GAR-Bench
DLC-Bench
Global
Local
Caption
VQA
Avg.
Pos.
Neg.
Base Model: Qwen2.5-VL-3
image mask
24.5
30.7
52.2
38.0
66.4
image mask cross-attention
27.9
30.0
55.7
46.8
64.6
image mask
image mask
34.3
32.1
62.1
50.7
73.5
image mask
Rol-aligned feature replay
41.2
40.8
69.2
58.1
80.3
Base Model: IntemVL3-28
image mask
24.6
33.0
65.6
48.5
82.6
image mask + cross-attention
29.4
31.8
68.8
56.7
80,9
image + mask image mask
image mask Rol-aligned feature replay
32.8
36.1
70.3
61.6
79.0
43.1
44.6
73.0
63.8
82.2
Ablations Architecture Designs. We first claborate aligned feature replay in Table 8. Other baselines include: ! only local images, 2 DAM-like architectures (Lian al., 2025) which preserves context via zero-initialized gated cross-attention, A simply cropping local images as a supplement of global images, and 4 our Rol-aligned feature replay design. As demonstrated in Table 8, both D 2 and 3 struggle modeling nulti-prompt relations, leading to poor results on GAR-Bench, although b is superior at precise description on DLC-Bench (Lian al., 2025). However, our proposed Rol-aligned feature replay strategy effectively preserves necessary global contexts while achieving competitive performances on DLC-Bench.
key architecture design, i.e., Rol-
In Table 9, we further extend our ablations on model architectures more base models, including Qwen2.5-V 25.2.55v 5-BL (Bai et al., 2025) and IntemVL3-22 (Zhu et al., 2025). As demonstrated in the table, our proposed Rol-aligned feature replay consistently brings significant improvements over different base models.
Alltiios on Data Pipeline. We study the effectiveness of our data in Table 10. Starting from the seed dataset, i.e., Describe-Anythingg- 1.5M (Lian et al., 2025), we first add our Fine-Grained Dataset-456K, and then add our Relation D 4144 By introducing our Fine-Grained Dataset- 456K, model is able to produce more accurate recognition, leading to improvement of +3.1 on DLC-Bench (Lian et al., 2025). By further combining our proposed Relation Datasct-414K 14K, the model is finally equipped with compositional reasoning capabilities with multiple prompts this time, resulting in significant improvements on our
Performances General Multimodal Benchmarks. We compare GAR-8B with other region-level models, i.e.. DAM-3B (Lian et al., 2025) and PAM-3B (Lin et al., 2025b), on general vision-centric multimodal benchmarks, including V (Wu & Xie, 2024), MMVP (Tong al., 2024b),
17

---

Published as a conference paper at ICLR 2026
Table 10: Ablations on each component of our data with IB model size. i indicates using GPT- 4o (OpenAI, 2024a) with extra cropped images as the judge, instead of text-only judging. Each component of our data plays significant role.
GAR-Bench
DIC-Benchh
Data
Caption
VQA
Avg.
Pos.
Neg.
Sced Datasct-1 1.3M
13.8
41.5
74.4
63.0
85.8
Fine- Grainer Dataset-456K
14.2
44.1
77.5
67.6
87.4
Relation Dataset-414K
57.5
50.6
77.1
66.2
88.0
Table 11: Performance general multimodal benchmarks (Wu & Xie, 2024; Tong al., 2024b; xAI, 2024; Chen et al., 2024a), where we set mask = for evaluation. Our GAR maintains the most general performance. Combining general VQA datasets would be effective.
Method
V*
MMVP
RealWorldQA
MMStar
DAM-3B (Lian et al., 2025)
45.0
60.7
54.3
39.7
PAM-3B (Lin et al., 2025b)
1,4
4.3
1.7
2.7
Base Model: PerceptionLM-881 PercetionL.M-B8 (Cho et al., 2025)
69.1
76.0
75.0
57.1
GAR-8B
59.2
78.0
58.7
43,9
GAR-8B (w/ 600K General Data (Li et al., 2024))
62.3
79.7
61.8
51.6
Table 12: Robustness analysis of GAR-Bench- Table 13: Robustness analysis of GAR-Bench- VQA, where we randomly sample subset of Cap, where randomly sample subset of each each subtask. The relative ordering is stable.
subtask. The relative ordering remains stable.
Full (424)
1/2 (212)
1/4 (106)
Full (214)
1/2 (107)
1/4 (53)
Model
Model
Overall Rank Overall Rank Overall Rank
Overall Rank Overall Rank Overall Rank
PAM-3B
4.3
0.9
DAM-3B
13.1
VP-SPHINX-I3B
37.5
40.0
8
33.3
PAM-3B
21.1
8
18.8
20.1
DAM-3B
38.2
7
48.6
7
41.9
7
VP-SPHINX-IST
32.3
7
29.7
7
20.0
GAR-1B
50.6
6
51.4
6
49.5
5
Qwen2.5-VL-32B
36.8
6
32.7
26.1
Qwen2.5-VL-322
50.9
$2.4
5
48.6
6
GPT-4o
51.5
5
45.5
5
52.1
GPT-4o
$3.5
56.7
4
57.1
4
o3
56.9
4
50.6
50.3
GAR-8B
59.9
60.0
63.8
GAR-IB
57.5
3
51.5
53.9
63.3
1
58.1
Gemini-2.5-Pro
59.3
54.4
Gemini-2.5-PPr
64.2
61.0
2
60.0
GAR-8B
62.2
57.4
61.8
RealWorldQA (xAI, 2024), and MMStar (Chen et al., 2024a). As illustrated in Table 11, our GAR-8B outperforms them by a large margin.
FEAR-Bench. We conducted subsampling stability analysis. We randomly subsampled -AR-Bench- VQA and SaAR-Bench-Cae Caph to 50% and 25% of their original sizes (for each subtask) and re-evaluated the full suite of models in Table 12 and Table 13, respectively. Our goal was test whether the relative performance rankings remained consistent even with significantly fewer samples. The results, presented in the tables, demonstrate a high degree of ranking stability. As the results show, the relative ordering of models remarkably stable. For example, in the top 3 models (GAR-8B, Demini-2.5-Phr (DeepMind, 2025b), GAR-IB) and bottom 3 models (VP-SPHINX (Lin et al., 2025a), PAM (Lin al., 2025b), DAM (Lian et al., 2025)) maintain their general ranking group even at 1/4 sample size. While minor fluctuations exist among the top-tier models in the VQA 1/4 split (e.g., GAR-8B jumping from 3rd Ist), the overall performance tiers are preserved.
Robust Analysis
Robust Analysis of LLM-Judges. To directly investigate the consistency and potential bias of the LLM judge, we conducted new cross-judge validation experiment. We re-evaluated all models on CAR-Bench-Caa using four different powerful LLMs as judges: GPT-4o (OpenAI, 2024a) (our original judge), o3 (OpenAI, 2025), Gemini-2.5-Flast (DeepMind, 2025a), and Gemini-2.5. Pro (DeepMind, 2025b). The results, presented in Table 14, demonstrate high degree of consistency in model rankings. While the absolute scores fluctuate across different judges, which reflects their inherent stylistic preferences or scoring strictness, the relative ranking of models is remarkably stable. Most importantly, GAR-8B consistently ranks 1st across all four distinct judges, and our other
18

---

Published as a conference paper at ICLR 2026
Table 14: Robustness analysis of the LLM-judges utlized by GAR-Bench-Cap The relative ordering remains stable across different judges.
GPT-4o
Cominii-2.5-Fi Gemini-2 2.5-i
Model
Score
Rank
Score
Rank
Score
Rank
Score
Rank
DAM-3B
13.1
10
9.3
10
9.4
10
0.3
10
PAM-3B
21.1
21.5
25.3
9
31.3
VP-SPHINX-13B
32.3
27.5
30.4
8
32.7
Qwen2.5-VL-32B
36,8
7
28.1
37.4
7
41.8
InterVL33-3BB
45.1
6
30.8
6
41.9
6
47.4
6
GPT-4o
51.5
5
31.8
5
46.3
5
50.6
5
o3
56.9
37.8
52.8
60.7
GAR-IB
57.5
37.4
55.1
3
59.8
Gemini-2.5-Pro
59.3
2
44.5
2
57.4
2
61.3
GAR-8B
62.2
46.7
1
60.3
1
62.6
Table 15: Analysis for general models using four different region-specification formats. "VQA" and "Cap" represent "GAR-Bench- and "GAR-Bennchee respectively. Powerful, general- purpose VLMs consistently struggle across all four settings.
GPT-4o
o3
Gemini-2.5
Input Type
VQA
Cap
DLC-Bench
VQA
Cap
DLC-Bench
VQA
Cap
DLC-Bench
Type I
53.5
51.5
41.0
61.3
56.9
48.0
64.2
59.3
48.4
Type? 2
56.4
52.0
38.4
63.4
54.9
47.8
61.9
58.1
52.9
Type
51.2
31.9
25.4
57.8
40.7
34.9
61.9
61.3
54.6
Type 4
48.3
38.2
28.5
59.0
47.1
41.3
66.0
44.1
47.8
Type
Image
Imagea
Type Outline
Type Bounding
Type
<Pronst
<Prorpt1>> Question: Dowarbe (Pronptoo in detall including fee nhutimhiip with chromatio,
Global image
<Pranptoo ouilined in roll <Prongtit is indiented by red box: cieneptis indicated
<Promptit indkonal by crees chropptss indicoted by aeyan enoea.
<Bronptis continee
Figure Illustration of different input types.
GAR models also consistently place in the top tier. This experiment suggests that our primary claims about GAR's superior performance are robust and not an artifact single judge's bias.
Input Type Analysis for General Models. To investigate whether general models lack the ability to interpret masks or are genuinely deficient understanding local details and relationships, we conduct analysis using four different input types illustrated in Figure 7: (1) Type 1: Separated global image and local masks (our original setting). (2) Type 2: Drawing mask outlines directly onto the image. (3) Type 3: Using bounding boxes derived from masks. (4) Type 4: Using center-point crosses derived from masks. Empirical results are presented in Table 15, where powerful models consistently struggle across all four settings. This crucial finding reveals fundamental deficiency in fine-grained perception and relational reasoning, regardless of how the regions of interest are indicated.
D QUALITATIVE RESULTS
D.1 QUALITATIVE RESULTS ON BENCH
"Relation" of MAR--Benc--VOA In Figure 8, provide qualitative comparisons the "relation" protocol of our VQA, including two failure cases (the last row). As demonstrated the figure, GAR-SB manages not only effectively model relationships but also leverage crucial local
19

---

Published as a conference paper at ICLR 2026
details for choosing the best answer. For instance, in the right example of the middle row, the person (<Prompt 0>) is actually not reading the book (<Prompti>). since she is looking at the camera. Our GAR-8B manages to recognize such details and thus select "<Prompt 0> is holding <P <Prompt>>" instead of "reading", while both Gemini-2.5-PP (DeepMind, 2025b) and o3 (OpenAl, 2025) fail.
However, illustrated in the last two examples in Figure 8, current models still sometimes struggle to understand complex relationships with more than two objects. Constructing such complicated training data and keeping the correctness of relation annotations could be a potential solution.
"Non-Entity Recognition" of GAR-Bench- MARR-Bench-VQA. VQA. In Figure 9, we provide qualitative comparisons including two failure cases (the
on the "non-entity recognition" protocol of our MRR-eenc--VV last row). As demonstrated in the figure, GAR-8B is able to correctly recognize objects in the mirror without any depth prior, thanks its encoded global contexts.
However, as demonstrated in the right case in the last row, current models still struggle to distinguish whether the reflection actually comes from the mirror (<Prompt2>) or other reflective surfaces (<Prompto and <Promptl>).
D.2 QUALITATIVE RESULTS ON VIDEOREFER-BENE
Detailed Localized Video Captioning. In Figure 10, we provide qualitative results of extending GAR-8B generate detailed video descriptions on VideoRefer-Bench (Yuan et al., 2025b). In most cases, where videos usually remain static, GAR-8B manages to generate detailed, specific, and precise descriptions. However, as demonstrated in the last example, GAR-8B fails capture detailed temporal differences among frames, leading to a low score on "temporal description". This because our GAR models are actually trained with only images and lack tie-ggaieed temporal comprehension capabilities.
Detailed Video Understanding, Figure 11, we provide qualitative results of extending GAR- 8B to detailed video understanding on VideoRefer- Bentee GAR-8B is capable of understanding basic motions under a zero-shot setting, e.g., the sequential question, the relation question, and the reasoning question. However, on the "future prediction" protocol, GAR-8B sometimes fails to choose the best choice with significant motion changes.
E LIMITATION AND FAILURE CASES
One potential limitation is that our GAR is limited to static images. Although it can be successfully extended to video and even achieves competitive results compared with video models (please refer to Tables 6 and 7 for detailed experimental results), it sometimes fails when input videos contain signifi- cant motion changes. Specifically, as demonstrated in the failure cases in Figures 10 and 11, GAR-8B is superior at comprehending and describing static videos, and is also capable of understanding basic motions. However, with significant motion changes, GAR-8B sometimes fails. Carefully collecting video training data could be a potential solution.
F DISCUSSION ON DLC-BENCH
Our analysis in Figure 12 reveals significant weakness in the original judger of DLC-Bench (Lian et al., 2025), which relies on a text-only LLM, i.e., LLaMA3.1-8B (Grattafiori et al., 2024), for automated scoring. Specifically, a fundamental flaw in the original DLC-Bench (Lian et al., 2025) evaluation lies in its assumption that semantic categories can be accurately adjudicated within the abstract confines of language space alone. However, Figure 12 demonstrates that this text-only approach is inherently unreliable due to the ambiguity of linguistic labels without visual contexts. We argue that the image is the only ground truth capable of resolving this ambiguity. Therefore, we provide the image for valid evaluation. To truly assess a model's 's descriptive power, the judge must be multimodal, capable of grounding the generated caption in the visual reality it purports describe.
G PROMPT TEMPLATES
We provide all of our prompts utilized in building our data in Figures 13, 14, 15, and 16.
20

---

Published as a conference paper at ICLR 2026
Question: the spatial relationship <Prorptl> <Prospte>?
Question: What the relationship between <Promp 18> and
<Prerptis?
holding <Prorpt1>.
A. <Prorpt1> in front <Prospt8>.
B. <Prorpt0> beside <Prorpti>.
B. <Prorptias front of <Prorpti>,
C. <Pronpt1>> is silting on <Prompt2> with <Prorpto <Prospt@> over <Prompt1>.
C. <Prompt0> is looking at <Prorpt1>,
<Prompt@> eating <Prompt1>.
Question:
Which statement accurately describes the interaction
Question: Which of following best describe the relationship between <Prospte> and <Promot1>>
between <Prorpté>> and <Prospti>>
A. <Prorpte lying on <Prorpti>.
<Prospt reading <Prompt1>,
B. dProrptl cleaning <Prorpt@).
B. <Prorptt>> holding <Prompti>>
C. dProrpto> is cleaning <Prorpt.>.
C, <Prorpti>>> is attached to <Prompt0>.
D. <Prorpt0> wearing <Prorpt1>.
D. <Prompteo sitting <Prompti>.
Budweiser
NDO UDS.COM
Crow Ferm
Question: relationship <Prompte>, <Proept2s, and <Prompti>?
position of cProrpt2: relation to <Prompt0> and <Prompti>?
Question:
<Promptid> wearing 2Prompt and carrying <Prompti>.
A, It is beside <Pronpt1> and uttached to <Prompte..
B. <Prospt@> carrying <Proept2> and wearing <Prorpti>.
is over <Prompt0> and under <Prorpt1>. is beside <Prospté> and attached to <Prospti>,
C. <Prorpts> carrying both <Prorpt2> and <Prorptis.
D. It is under <Proret3s and beside cPromptos.
D. <Prorptas wearing both oProretto and cProrptis..
Figure 8: Qualitative comparisons on the "relation" protocol our MRR-Bench-VQA.. including two failure cases (bottom). Notably, in the right case of the middle row, the person (<P 00mp:0>) actually not reading the book (<Prompti>). since she is looking at the camera. Our G-R-8B manages to recognize such details while both Gemini-2.5-Par (DeepMind, 2025b) and OpenAl- o3 (OpenAI, 2025) fail. From the last two cases, we can tell that models are still struggling with understanding complex relationships with more than two objects. Image source: Shao al. (2019).
H USE OF LLMS
In preparing this paper. LLMs are utilized as a general-purpose purpose assistive tool. Specifically, the use of LLMs is strictly limited to proofreading the author-writte text for grammatical errors, spelling
21

---

Published as a conference paper at ICLR 2026
corrections, and improvements to language clarity. This application is consistent with the use of conventional Haammar-checkinn software and did not extend to research ideation, data analysis, or the generation of any substantive content.
22

---

Published as a conference paper at ICLR 2026
Question: you tell <Promptl> <Pronpti>
Question: Which among <Prorpt@), <Pronpt1>,
inside the mimor?
<Prorpt2> <Prorpt is in the mirror?
A. <Prorpti> and <Prorpt8> are both in the mirror.
A. <Prompti>.
Only <ProrptB> in the mirror.
B. <Prompt1>,
Only <Proupti> is the mirror.
C. <Prorpt2>,
Neither <ProrptB> and <Prompti> the mirror.
D. <Prorpt3>.
Question:
Among cPronpti8,, <Prompti>, and dPrompt2>,
Question: Are <Prorpt0: and (Prorpti> located within the mirror?
which ene in the mirror?
A. <Prorptts.
A. Both <Prorpt@> and <Prompt1> are in the mirror.
B. <Prompt1>.
B. Only <Prorpt@> is the mirror.
<Proupt2>.
B. Only <Prorpti>
mirror.
None mirror.
D. None
are mirror.
Question: <Prorpt@> and <Prorpti> located within
Among <ProrptO), <Prorpti>, and <Prorpt2>, which one is in the mirror?
Question:
mirror?
A. Both <Prospt0> and <Prorpt1> are in the mirror.
A. <Proupte>.
B. Ouly <Prompta> in the mirror.
B. <Prorpti>>
B. Only <Prospt>>> is in the mirror.
C. <Proupt2>.
None of the above are mirror.
above are
Figure 9: Qualitative comparisons on the "non-entity recognition" protocol of our GAR-Bench- VQA, including two failure cases (bottom). From the last two cases, tell that models are sometimes still struggling with recognizing non-entities, especially distinguishing reflection from the mirror (<Prompt2>) and other surfaces (<Promp=0> and <Promptl>).
23

---

Published as a conference paper at ICLR 2026
Death 26
Question:
region
Question: Deserihe region
white polica dots. sleeveless and has black belt around the waist and worn woman's body.
GAR-8R:
wenring a checkered singing with the people around him. He keeps smiling and looks very happy:
cup,
Judgement:
Judgement:
Subject Correspondence
Subject Comespondence
Appearance Description
Appearance Description
Temporal Description
Temporal Descripsion
Hallucination Detection
Hallucination
Question:
Describe the masked region detail.
Question:
Deseribe the masked region in detail.
GAR-SB:
The sports in the vidco blue Porsche. parked under a covered area, has slock design with glossy finish, black rims, and visible Porsche emblem on hood. The car stationary and in motion.
GAR-BB: A blue bus with carteonish design featuring
driver's side dlor open. The
windshicld and windows, through which
colorful characiers visible. The bus rounded front with two circular headlights
small, rectangular grille below the windshield. The side of the bus displays row of windows with body adorrned with a
visible door towards the froet. partern of small, colorful shapes and designs.
GPT-4o Judgement:
Subject Corespondence
Appearince Description
CPT-4o Judgement:
Temporal Description
Subject Conreponnance
Hallucination Detection
Appearacce Description
Temporal Description
Hallucination Detection
Figure 10: Qualitative results of detailed video captioning VideoRefer Beeech (Yuan al., 2025b), including one failure case with a low "temporal description" score. Video source: (Chen et al., 2024b).
24

---

Published as a conference paper at ICLR 2026
Basic Question
Sequential Question
Question: What <Prompt0> holding?
What action does <Pronpte: perform during the sequence?
Question:
A.
She forward with her rollerblades
B. phone
B. She jumps with her rollerblades
C. Leashes ropes
C. She stops and takes off her rollerblades
D. A camera
D. She falls down
Relation Question
Reasoning Question
Question:
What is the relationship between <Prorptos, <Prompt1> and <Prampt2> during the sequence?
Given that <Prompt@> and <Prorpti> are engaged in a playful tussle, what can be inferred about their re relationship?
Question:
A. <Prorptl and <Prorpt2> are both confronting <Prorpt@>
B. <Prompt0s, <Prompt1> and <Prompt2> ignoring each other
A. They are likely unfamiliar with each other
B. They are likely aggressive towards each other
C. <Prompti> is chasing <Pronpt@> and <Pronpt2>
C. They are likely scared each other They likely friends familiar
D. <Prorpt0>, <Prorptl> and <Prorpt2> are playing
together
Future Prediction
Failure Case
Question: If <Prorpté>
remuins on the tree, what is likely future event?
Question: What is likely future event of <Prorpte>?
A. <Prorpt0> will change color match the tree
She will sit down rest
<Proupt@> will jump into the water
She attempt unother vault
<Prompto> will fly away
C. She will leave the field
<Prosptas will dig a hole
She celebrate successful jump
Figure 11: Qualitative results of detailed video understanding on VideoRefer-Benchh (Yuan et al., 2025b), including one failure in the "future prediction" protocol.
25

---

Published as a conference paper at ICLR 2026
Judge Prompt
Answer the sultiple-chatec question based on the text description of object in an Inago. You need to follow these rules:
the choices 2. There is no need for exect natching, Please choose the closest option besed on the description.
output reasoning.
perfom correction, output exactly
each question. Do not repeat the question.
description cartion)
Fron the description above, please answer the following question with one of the choices:
likely objects description (class_nare) objects sixilar type? Again, does an exact natch.
False Negative
Category: Van
Category: Sign Hanner
Description: A black taxi with the word TAXI in
Description: A white price tag with handwritten text in blue ink. The text reuds L64 at the top. followed by 100g and 100g again below it. the bottom, there is a large 6390
yellow on the rear window, featuring a yellow license plate and red tail lights.
GPT-4o
GPT-4o: Yes
8 LLaMA3.1-8B: No
LLaMA3.1-8BB
Category: Microwave
Category: Truck
Description: A wooden cabinet with a vertical grain pattem, featuring a single visible door handle on the right side, which is curved and metallic. The cabinet has top appears part larger piece furnitur..
Deseription: The bus is predominantly blue with sleek, modern design. It features black and white logo on the side, and the word AMBULANCE is visible in white letters on black background. The bus has large, curved windshield side mirror extending from front.
GPT-40:
GPT-4o:
LLaMA3.1-8BP Yes
LLaMA3.1-8B. Yes
Figure 12: Incorrect text-only judging results using LLaMA3.1-2 1-88 (Grattafiori al., 2024) on DLC-Bench (Lian et al., 2025). The model is required to judge whether the description is consistent with the ground-truth category name, We illustrate both correct and wrong results. Providing extra cropped images and masks to GPT-4o (OpenAI, 2024a) effectively eliminates this issue.
26

---

Published as a conference paper at ICLR 2026
Prompt for Judging Descriptions and Ground- Truth Categories
Answer the nultiple-choice question based on the text description of object in inage. You need to follow these rules:
not output any reasoning, not perform correction. output exactly answer from choices oach question. do not repeat the question.
2. There is no need for exact watching. Please choose the closest option based on the description.
The description is: pprod_caption)
description above, pleese answer following questiin
choices:
Is it likely that the objects in the description are (class_nand) nare, liste or objects of sinilar type? Again, it not have
natch.
Figure 13: Prompt for judging the description and the ground-truth category.
Prompt for Generating Relation-Awar Captions
You given the following Information:
Subject (dbject_näne) (ancct_nne))
(subject_nane)
Object name:
Predicate (relation): (predicate nane)
Subject description: (sub caption)
Object description: (obi_caption)
Instructions:
'Subfect description' (subjectt_nam objects type. It does not have to be an exact natch. If it does not, output only: False.
Judgo
objects
The 'anject description' does not need to notch the "dbject nase" 'Object
'Object description
you ray
does not natch, ignore
only use the "Object nase".
Gonerate fluent caption focusing mainly on the Subject.
Preserve the relation ([predicate, (
detail from the subject description possible.
object (usting either 'Object description
valid, or the 'Object name").
4. Output only the Final caption, without explanations reasoning.
Figure 14; Prompt for generating relation-aware caption.
27

---

Published as a conference paper at ICLR 2026
Prompt for Generating Question-Answering Pairs
You are profess ional Visuel Question Answering (VQA) expert. Your task is to create high-quelity, direct question pairs about virtual based provided ground truth data.
Input Fomat:
I w1ll provide with the ground truth for scene two 350N formats:
tags for dofects (e.g., <Prorpt)>), their corresponding category (category, _ane)), and a detailed led text description (caption).
captions: dictionary containing
relations: A dictionary that describes the relationships between objects using the fornat <subject>, cobject>, <predicato>.
Task & Output Fornat:
Your task is to this ground truth data to generate J50N containing 1-3 question-answer pairs. Your output nust single,
array nothing Do include any explanations, corments, text autside of the I50N structure. The foraat should be as follows:
json
"question": "The of questiion...,,
"answar": direct,
santence or phrase."
Core Generation Rules:
Relationships:
All questions rust prinerily test the spatial, action-based, or state-based relationships defined in the relations data.
Formulate Concise Factual Answers:
The answer_text must directly and accurately respond to the question. The
be short, complete sentence or descriptive phrase based only the provided relations and captions.
Exarple:
Q: "What is the relationship between <Prorpte> and <Pronpti>2"
A: "<Pronpte> is on top of <Prompt>>."
Diverse Questioning Styles (Cructal):
Your questions rust varied. Enulate the following styles: RellesionshipAArrageeeett
spatial relationship between <Prorpti>> <Prompti>?' "Describe arrangenent invelving <Prouptl>, (Prongt>,, and <Proupts>." or "Which statement accurately describes the positions of <Pronpt0>, <Prorpt>>, and <Proupt>>?"
Comprehensive Statements: "Can you describe the arrangenet involving <Prompt1>, <Pronpt2> <Prompt>>?"
Location: 'where <Prorpt2> located relative <Prompt>>? or "How are <Prompt2> and <Prompti> pos itiondd relative to <Pronota>?"
Actian State: "What is primany activity of <Premtt8)" "What
<Pronpto)
(Pronpt4)?"
statenent synthesizes relationships involving <Prorpts> <Pronptis?"
Attribute-based (using caption details): "Wat is the back of the giraffe <Pronpt2>?" Direct Relationshio: 'What is the spetial relationship between <Prompto> <Prorpt2>?"
<Prorpti>?" or "How <Prorpt
interacting (Proptl>
Ask for proupt: "Wiich are/is described as driving on ePronpti>?" ar "which object is located between (Ponptay and dProrpti>?" 2 (the answer should be like "crooppto" on "<Pranpts> and (Pronpt2>")
You can vary your question
styles use styles appear
4. Synthesize Inforeation for Reasoning:
Whenever possible, answer. answer
questions synthesis.
synthesizing niltiple relationships
5. Intelligent Use captions:
Utilize category_nane caption details formulate specific, context-aware questions
6. Strict Fornatting and Wording (Crucial): Innersive Phrasing:
visual scene, Crucially, you phrases like "Based on the provided relationships, or "According to the information." Tag-Only References: You must use the <PromptX> tags to refer to objects. Do not add descriptions to the tags <Frompto>, the <Prorpt)>).
questions
asking
thenselves
Input:
captions: {captions)
relations: (relations)
Figure 15: Prompt for generating question-answering pairs.
28

---

Published as a conference paper at ICLR 2026
Prompt for Generating Multiple Choices Questions
You are professional Visual Question Answering (VQ4) expert. Your task is to create high-quality, diverse, multiple-choice questions about virtual scere based provided ground truth data.
Input Fomaa:
I sill provide with the ground truth for scene in two 150N formats and some inages:
captions: dictionary containing reference tags objects (e-g., <Pronpté>), their corresponding categony jattegor__aane), and detatled
description (caption).
relations: dictionary that describes the relationships between objects using the fornat csubject>, spredicctt..
images:
image and mask crop images stand specific <Proeptx)
Task & Output Fornat:
task is Your output nust be a single, valid 150N array and nothing else. po not include any explamations, comnents, or text outside of the 150N structure. The fornat should be follows:
ground
generate 350N array containing multiple-choice questions.
json
"question": "options": ["A. ...", "D, ".", "C ...", '0 ".."},
question...",
"answer": "A"
Core Generation Rules:
1. Core Focus on Relationships:
questions prinarily tast the spatial, action based, state-basee relationshiip doffned in the relations
The correct answer sust be directly verifiable fron the provided ground truth.
Questioning Styles (Crucial):
Do not overuse singlo question format, Your questions must be varied. Enulate the following styles based on the exarples provided in the user's file:
Corprehensive Statements: 'which the following statements accurately descrites arrangenent involving (Pronpt>>, <Prospt2>, and (rompt3>?"
Location á Belonging: "which of the following objects are all located on(beside, parkee on...) <Prorpt4>?"
Action State: "What
prinary activity of <Prorpt@s?" "What drrorptis and <Prompti> doling
(Pronptas?
Attribute-based (using caption details): "Which object, described as having an illustration of cat, is located <Prorpt>>? "which surface is the giraffe <Prompt2> lying on?"
<Prorpte <Prompti>?" "Hoa (Pooptts interacting with <Proppis and dPropp22>" or "Which objects are located beside <Prompti>?",
telationship(eeinly::
spatial relationship
Synthesize Inforsation Reasoning:
Whenever possible, design questions that require synthesizing miltiple relationships to arrive at the correct answer. For exarple, a question night test <Prospto>'s relationship to both <Prospti>> and <Prortt>>
ligent
captions:
Utilize the sategory_nane and caption details not just for creating distractors, but to formulate rore specific, nuanced, wwere questions and answers.
5. Plausible Distractors:
Each question must have one correct answer and 2-3 plaucible but incorrect distractors.
these by
subject, object, or predicate from scene to create false but believable statement.
relationship, by using other objects
Use sumary options like "Both <Prosptd> and <Prospti>" or "None of the above" where appropriate.
Fornatting and Wording (Crucial):
Innersive Phrasing: Frate questions if asking about a real visual scene. Crucially, you must not phrases on provided relationships," "According
information," reference
sources any
way.
Tag-Only References: You rust the <Propptx tags refer to objects in both questions and options. not add descriptions
(e.g., <Pronptes, Category-Agnos Questions: When asking "Which..", you rust use general phrasing. For exarple, always use "Which of the following is..." instead of 'Which person is..." " to ensure the question renains valid for all possible types.
tags
the <Prorpté)).
Input:
captions :ccaptions) relations:(relation::
Figure 16: Prompt for generating multiple-choice questions.
29


Reviews:
[{"review_id": "S2sbybBZs8", "reviewer": "Reviewer_uvNs", "summary": "This work addresses the limitation of existing MLLMs in fine-grained regional understanding, global context integration, and multi-prompt interaction. It proposes GAR, a model empowered by RoI-aligned feature replay to preserve global context while capturing local details, and GARBench, a benchmark for evaluating single-region comprehension, multi-prompt interaction, and compositional reasoning. GAR outperforms state-of-the-art models on multiple benchmarks and achieves strong zero-shot transfer to video tasks.", "questions": "RoI-Align’s necessity could be further strengthened, as no comparison is provided in the ablation study (e.g., Table 8) with a simpler alternative: directly cropping the local image region and feeding it through the ViT encoder to extract features. It would be helpful to clarify how RoI-Align outperforms this more straightforward approach.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. Proposes an innovative RoI-aligned feature replay technique that seamlessly integrates global context and local details, solving a key limitation of existing region-level MLLMs.\n2. Develops GARBench, the first benchmark to systematically evaluate multi-prompt interaction and compositional reasoning, filling an evaluation gap.\n3. Demonstrates strong generalization, including zero-shot transfer to video tasks, highlighting the model’s practical utility.\nConducts comprehensive experiments (ablation, cross-benchmark, qualitative analysis) that rigorously validate the method’s effectiveness.", "weaknesses": "1. RoI-Align’s context binding validity could benefit from additional validation, as the paper does not include targeted experiments for complex scenes where misbinding irrelevant context might occur. Additionally, the ablations in Table 8 do not test whether shielding irrelevant global regions impacts the accuracy of extracted local features.\n2. The Grasp Any Region-2.5M dataset and GARBench do not provide clear annotations or statistics on which tasks specifically depend on global context. This may lead to potential over-reliance on global context during training, and it remains unclear if GAR’s performance gains stem from global-local fusion or merely improved local feature capability.  \n3. GARBench appears statistically limited for drawing robust conclusions, with only 204 samples for GARBench-Cap and 424 samples for GARBench-VQA—sizes that may not fully account for result variability.  \n4. RoI-Align’s necessity could be further strengthened, as no comparison is provided in the ablation study (e.g., Table 8) with a simpler alternative: directly cropping the local image region and feeding it through the ViT encoder to extract features. It would be helpful to clarify how RoI-Align outperforms this more straightforward approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c6iXyaEtkp", "reviewer": "Reviewer_SknR", "summary": "This paper aims to enhance the capability of Vision-Language Models (VLMs) in understanding dense visual scenes, particularly in capturing fine-grained local details and the relationships among multiple local regions. The authors propose the GAR method, whose core lies in the ROI-aligned feature replay technique, which enhances local visual feature representation through ROI alignment during the visual feature extraction process. Furthermore, the paper introduces a large-scale GAR-2.5M training dataset and GARBench benchmark for training and evaluating the GAR-1B/8B models, respectively. Experimental results across multiple datasets demonstrate the superior performance of the proposed GAR-1B and GAR-8B models.", "questions": "See Weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Understanding fine-grained details and object inter-relationships is critical for real-world applications of VLMs. This paper provides important contributions in training dataset, evaluation benchmark and model architechture. \n- The paper is clearly written and easy to read.", "weaknesses": "- Understanding the dense world is an important capability. However, the proposed GAR task represents only a specific task formulation within this direction. In particular, the paper constrains the use of masks as indicators of objects, which may introduce bias when evaluating a model’s true understanding of dense visual scenes. The compared models might simply lack the ability to interpret masks, rather than being genuinely deficient in understanding local details and relationships.\n- The proposed method in this paper is designed for a specific task type, incorporating a task-specific module — the ROI-Aligned Feature Replay Module. Moreover, the corresponding training dataset GAR-2.5M is constructed using the same methodology as GARBench, which weakens the assessment of the method’s effectiveness and generalizability. In addition to comparisons with mainstream open-source models such as Qwen2.5-VL and InternVL3 on GARBench, evaluations on general benchmarks would further strengthen the validation of the proposed method and dataset in terms of their effectiveness and general applicability.\n- The ablation study in this paper is not sufficiently comprehensive. In particular, it would be beneficial to include an additional variant that uses only global patch embeddings and mask embeddings, without the ROI-Aligned Feature Replay Module, and train this variant on the GAR-2.5M dataset. This result would help highlight the effectiveness of the ROI-Aligned Feature Replay Module. Furthermore, comparisons with the pretrained model, namely the PerceptionLM baseline, should also be added across all datasets to provide a more complete evaluation.\n- The implementation details of GAR are not clearly described. In particular, how are the RoI-Aligned features integrated with the global context features and question tokens?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2hKAYFiURY", "reviewer": "Reviewer_n7Lw", "summary": "This paper proposes a vision-language model that enhances region-level visual understanding by combining global context with fine-grained local detail. Using a RoI-aligned feature replay mechanism, GAR processes full-image features while focusing on specific regions, enabling accurate reasoning about multiple prompts (e.g., spatial and relational understanding). The authors also introduce GARBench, a new benchmark that evaluates both single-region captioning and multi-region reasoning. Experiments show GAR surpasses previous models like DAM-3B, Ferret, and even larger models such as InternVL3-78B on both captioning and VQA tasks.", "questions": "- Scalability to the number of regions:\nThe paper shows results for up to a few regions. What happens when the number of regions scales to tens or hundreds—does reasoning complexity or memory usage grow linearly?\n\n- Judge consistency: How LLM judges' results are consistent? Are they consistent enough for multiple runs?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ Clear motivation and reasonable architectural solution:\nGAR tackles the important problem to understand regional information in the vision-language model, and proposes the reasonable solution with a RoI-aligned feature replay strategy. This design allows global context retention while focusing on high-resolution local features.\n\n+ Focus on inter-region relationships:\nUnlike prior works, which primarily handle single-object or localized descriptions, GAR explicitly models multi-prompt interactions such as spatial relations and entity reflection recognition. This makes it suitable for compositional reasoning tasks previously underexplored by region-level MLLMs\n\n+ Comprehensive benchmark contribution (GARBench):\nI appreciate the GAR Benchmark contribution. It evaluates not only region captioning but also relational VQA, multi-prompt reasoning, and non-entity discrimination—offering a more complete view of region-level comprehension capabilities", "weaknesses": "- Evaluation dependence on LLM judges:\nThe authors rely on LLM-based evaluators (e.g., GPT-4) for qualitative assessment. This introduces potential bias due to stylistic or verbosity differences among models. Cross-validation with human ratings or standardized metrics would strengthen the claims.\n\n- Insufficient data transparency:\nThe training pipeline involves multiple stages—seed captioner, LLM merger, and relational caption generation—but lacks detail on data deduplication and leakage checks. Since datasets like COCO or PSG may overlap with benchmarks, more rigorous data hygiene reporting is necessary\n\n- Marginal novelty compared to related works:\nThe combination of RoI-Align with contextual replay is technically effective but not radically novel; similar ideas of multi-scale or global-local fusion appear in Ferret and GPT4RoI.\n\n- Missing references: The paper misses recent region-understanding and grounding works such as [ref1] and [ref2], and would benefit from a more detailed comparison and discussion of how GAR differs from these grounded vision-language models.\n\n[ref1] Toward Interactive Regional Understanding in Vision-Large Language Models, NAACL 2024\n\n[ref2] Groma: Grounded Multimodal Assistant, ECCV 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fXkK9UkOaB", "reviewer": "Reviewer_ip5b", "summary": "The paper proposes Grasp Any Region (GAR), a region-level MLLM that aims to combine precise region perception with access to global scene context, support interactions among multiple region prompts, and enable region-centric compositional reasoning in a single-turn conversation. The core technical idea is an RoI-aligned feature replay pipeline: the model encodes the full image with prompt masks to obtain a global, context-aware feature map, then re-pools prompt-specific features with RoI-Align so the LLM receives both prompt-aware global tokens and high-fidelity local tokens. The paper also proposes a dataset of 2.5M-scale assembled in two rounds to strengthen fine-grained recognition and relational understanding, and GARBench, a benchmark suite with a captioning protocol for multi-prompt relations and VQA that separates basic perception from reasoning such as position, non-entity recognition, and relations. The results section reports strong numbers on DLC-Bench and competitive results on GARBench, with qualitative examples", "questions": "Could you precisely differentiate GAR from GPT4RoI-style architectures beyond the use of RoI-Align? Please explain whether GAR’s mask changes the vision encoder’s computation to create prompt-aware global features, how this differs from methods that inject regions only at the language side after a prompt-agnostic image encoding, and what effect this has on using global context.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "– The data contribution is good and will likely be useful to the community. The paper constructs a large training set that starts from Describe-Anything, augments with fine-grained category supervision, and then adds relation-aware descriptions and QAs from PSG using an LLM merger. This staged pipeline is clearly described with concrete numbers and roles for each round, and it targets the specific capability gaps claimed by the model, like fine-grained region recognition and multi-region relations. The ablation helps understand what each round adds. \n\n– The benchmark contribution is also meaningful. GARBench decomposes evaluation into captioning of relations across prompts and a VQA protocol that cleanly separates perception (color, shape, material, texture) from reasoning (position, non-entity recognition, relations). The paper also uses multiple-choice designs in places where open-ended judging is brittle, and it includes tasks that force models to use global context, like mirror reflections and grid positions. Together, this looks like a well-scoped benchmark for region-aware reasoning.", "weaknesses": "- The major weakness is that the paper has not positioned its novelty in the literature with enough precision. The paper frames the novelty around inclusion of global context for precise perception, interactions between multiple prompts, and reasoning capabilities, but the related works section states that “previous approaches only support a single visual prompt, and often neglect global context”(L123). This statement is too strong, given that prior works referred in the same section (GPT4RoI and GLaMM) already support multiple regions and attempt to preserve global context, so the current phrasing risks overstating the novelty. \n\n- Regarding the specific point about inclusion of global context would benefit from a clear technical comparison to GPT4RoI‑style designs. \nGPT4RoI encodes the entire image once to obtain a global visual representation, then extracts RoI‑aligned features for user‑specified regions (via boxes) and interleaves those region features with the text sequence so the LLM can answer region‑centric queries while still using whole‑image context, it supports multiple regions and treats each region as an additional visual cue alongside the global image feature. The paper follows essentially the same recipe - one global image pass + RoI‑aligned region features fed to the LLM for multi‑region reasoning, with the only material differences being that (i) it takes masks rather than boxes and injects a learned mask embedding into the vision backbone before computing the global feature map (so the map itself is prompt‑aware), and (ii) instead of pooling each RoI down to a single vector, it replays a small set of RoI‑aligned tokens per region taken from that prompt‑aware map. In other words, apart from mask conditioning and unpooled multi‑token RoI replay, the overall architecture and goal, joining global context with local RoI features to enable fine‑grained, multi‑prompt understanding, are very similar.\n\n- Comparisons currently center on DAM, but GPT4RoI is the key reference for multi-region prompting and should anchor the novelty discussion. A simple ablation in a GPT4RoI style would make the technical contribution clear, keeping the LLM and data fixed. This ablation would show whether the gains come from the architecture rather than data or evaluation choices.\n\n- The only clear novelty that remains well supported is the region‑specific reasoning emphasis, which is indeed useful since even strong models like Gemini-2.5-Pro still fail on spatial reasoning. GARBench’s design and the strong results there are aligned with this direction, yet the novelty narrative should reflect that emphasis more, rather than attributing novelty to global context or single‑prompt limitations in prior work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)