FOCUS: Internal MLLM Representations for Efficient Fine-Grained Visual Question Answering
Authors: Liangyu Zhong
Fabio Rosenthal
Joachim Sicking
Fabian Hüger
Thorsten Bagdonat
Hanno Gottschalk
Leo Schwinn
Venue: NeurIPS 2025 poster ()
Keywords: vision-language models
large language models
visual question answering
Primary area: deep_learning
TL;DR: We propose a training-free visual cropping method that leverages MLLM-internal representations for VQA tasks focusing on small details, achieving strong performance with significantly higher efficiency than prior methods.

Abstract: While Multimodal Large Language Models (MLLMs) offer strong perception and
reasoning capabilities for image-text input, Visual Question Answering (VQA)
focusing on small image details still remains a challenge. Although visual cropping
techniques seem promising, recent approaches have several limitations: the need
for task-specific fine-tuning, low efficiency due to uninformed exhaustive search,
or incompatibility with efficient attention implementations. We address these
shortcomings by proposing a training-free visual cropping method, dubbed FOCUS,
that leverages MLLM-internal representations to guide the search for the most
relevant image region. This is accomplished in four steps: first, we identify the
target object(s) in the VQA prompt; second, we compute an object relevance map
using the key-value (KV) cache; third, we propose and rank relevant image regions
based on the map; and finally, we perform the fine-grained VQA task using the top-ranked
region. As a result of this informed search strategy, FOCUS achieves strong
performance across four fine-grained VQA datasets and three types of MLLMs. It
outperforms three popular visual cropping methods in both accuracy and efficiency,
and matches the best-performing baseline, ZoomEye, while requiring 3 – 6.5 ×
less compute.

FOCUS: Internal MLLM Representations for Efficient Fine-Grained Visual Question Answering
Liangyu Zhong 1,3 Fabio Rosenthal Joachim Sicking Fabian Hüger
Joachim Thorsten Bagdonat', Hanno Gottschalk Leo Schwinn
Technical University of Berlin, Techhicall University of Munich, &CARIAD SE, Volkswagen AG
Project page: http:://ooeu-mmllln.vggggghhhhhu
Abstract
While Multimodal Large Language Models (MLLMs) offer strong perception and reasoning capabilities for image-text input, Visual Question Answering (VQA) focusing on small image details still remains challenge. Although visual cropping techniques seem promising, recent approaches have several limitations: the need for task-specific fine-tuning, low efficiency due to uninformed exbaustive search, incompatibility with efficient attention implementations. We address these shortcomings by proposing a training-free visual cropping method, dubbed FOCUS, that leverages M...M-inneera.. representations guide the search for the most relevant image region. This is accomplished in four steps: first, we identify the target object(s) in the VQA prompt; second, compute an object relevance map using the key-value (KV) cache; third, we propose and rank relevant image regions based on the map; and finally, we perform the fine-grate ane-grained VQA task using the top- ranked region. As result of this informed search strategy, FOCUS achieves strong performance across four fine-gr ane-grained VQA datasets and three types of MLLMs. It outperforms three popular visual cropping methods in both accuracy and efficiency, and matches the best-performing baseline, ZoomEye, while requiring 6.6.x less compute.
Introduction
Multimodal Large Language Models (MLLMs) exhibit compelling cross-modal perception and reasoning capabilities, particularly on image-text data [8 (2) However, standard MLLM architec- not well suited perceive and reason about small visual details in high-resolution images as they typically downscale their inputs, leading to loss of information. Examples of these so-called global-view MLLMs include Instruct-BLL. BLIP and LLaVA-1.5 24), which only support low-resolution inputs of 224 x 224 px and 336 x 336 px, respectively. As consequence, global- view MILMs perform poorly on Visual Question Answering (VQA) tasks involving small-scale visual details 38
37 42
Recent MILMM architectures such I.LaVA-Onel Vision 22 and Gemma-3 19 address this limita- tion by processing both a downsampled global view and local crops extracted from the original image. This dual-view strategy enables them to handle high-resolution inputs with reduced information loss compared to global-view MLLMs. However, despite having access to fine-grained visual details from all local crops, these so-called global localllcal-laae local-view MLLMs struggle to identify the few visual tokens that are relevant for fine- fine-graing grained VQA amid the large number of local crop tokens. While these
Equal contribution
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

global-local-ye architectures outperform global-view MLLMs, their effectiveness on line-graines VQA tasks still remains limited [37].
An orthogonal research direction to address the limitations of MILMS in capturing fine details in which seek to pass only relevant image regions to the MLLM. However, popular visual cropping techniques like SEAL (38),
high-resolution images are visual cropping approaches 31 37
38 22
DC [37]. ZoomEye BII and ViCrop 42 suffer from one more of the following limitations: (I) reliance on task-specific fine-tuning of MLLMs for fine-grained VQA, (2) use of inefficient, exhaustive search algorithms, and (3) incompatibility with efficient attention implementations such as FlashAttention [10] (see Tab. I and Fig.
and 6 for visual comparison of the methods). We propose a visual cropping method, termed Fine- grained visual Object Cropping Using cached token Similarity (FOCUS), that addresses these issues as is outlined in the following.
Fine-grained
To tackle limitation (1), FOCUS leverages the in- ternal representations of MILMs, specifically their key-value (KV) caches (27], localize question-relevant image regions in a training-
Table 1: Comparison of visual cropping methods w.r.f. desirable properties. Unlike previously suggested methods, FOCUS is training-free, algorithmically effi- cient in scarch, and compatible with efficient attention implementations.
free manner-unlike the SEAL K8 technique. Moreover, to mitigate limitation (2), our method includes textual clues to enable object-aware localization without exhaustive cropping of the image, thereby improving the algorithmic ef- ceiency-different from DC [37 and Zoom- Eye B1. To overcome limitation (3), FOCUS utilizes the cached value features readily avail- able during inference, making it natively com- patible with efficient attention implementations
Training- free
Efficient search algo,
Compatible w/ efficient attention
Method
SEAL
38
DC
27
ZyomEye
H1
x
ViCrop +2
X
FOCUS (Ours)
V
unlkeei ViCrop 42] that depends on full attention weights. Specifically, FOCUS combines these components as follows: for each VQA question, we first identify the target object(s) in the question prompt. Second, we construct an object relevance map using cosine similarity between the cached text tokens of the target object(s) and the cached image tokens, and then propose relevant regions based on this map. Third, we rank the proposed image regions based on the existence confidence of the target object in each region. Finally, we perform VQA solely based the image region with the highest confidence. Note that FOCUS is compatible with both global- and global-local-vie MLLMs.
We evaluate FOCUS the fine-grained VQA datasets V*Bench (38), HRBench-4K [37]. HRBench- 8K [37 and MME-RealWorid-Lite 43 Across the first three datasets, our method achieves on average 42% higher accuracy over the vanilla MLLMs when using -L. and 17% when using LLa VA-One while improving -LaVA-OOeeVision by 6% on the multi-domain MME- RealWorld-Lite dataset. Moreover, FOCUS achieves comparable or superior performance w.r.t. the of baseline ZoomEye (31] while being 3.5 - 4.5x more efficient with LLaVA-1.5 and 3 x more efficient with VA--AA--
Our key contributions are as follows: First, we propose FOCUS, a training-fres free visual cropping method for HiLLM-based fine-grained VQA that identifies relevant image regions using internal representations of the MLLM. Second, we provide extensive empirical evidence for FOCUS's 's favorable accuracy-efficieee trade-off's compared to previous visual cropping methods for fine-grained VQA. Third, we conduct an ablation study that provides insights on how FOCUS leverages MLLM-internal knowledge for efficient visual cropping.
2
Related work
VQA involves answering a question based on the visual content of an image. While various types of machine learning models can be applied this task, MLLMS have become the de facto standard due to their strong cross- moddl reasoning capabilities variant of VQA, where the MILM is expected to select the correct answer from a fixed set of options. In the following paragraphs, we describe the datasets commonly used to evaluate MILM performance on fine-grained VQA tasks, i.e., tasks that require focus on visual details. Further, we provide a technical overview of recent visual cropping methods.
24]. Here, we focus on the multiple-chote -hoice
2

---

Fine-grained VQA datasets In common VQA datasets such as Tex 33 and Ref-COCO 20), the relevant objects are typically prominent within the image. In con- trast, fine-grained VQA tasks focus on much smaller visual targets. The V*Bench dataset exemplifies this, containing significantly smaller question-relevant objects compared to the aforementioned datasets (see Fig. T). Additional fine-grained VQA datasets include [43]. Among these, V*Bench is the only dataset that provides Ground Truth (GT) annotations for question-relevant objects.
HRBench-4K (37), HRBench-8K [37, and
Visual cropping for fine-grained VQA In this emerging area of research, different visual crop- ping methods [3 37 38 42] have been pro- posed to improve MILLM performance on fine- grained VQA tasks (see TTb. 1). SEAL [3 employs dual-MILM setup: one MILM for visual search and another one for the actual VQA task. The visual search model with ad- ditional decoders is fine-tuned to predict object heatmaps and coordinates. It performs a top- down hierarchical search, generating contextual cues to iteratively locate the target object based confidence scores. This approach requires task-specific fine-tuning and is rather inefficient due to its complex, multi-module design and recursive search process. In contrast, DC² [37] constructs a hierarchical image region tree from the global view down to regions, matching the base resolution of the vision encoder. Each re- gion is captioned using the MLLM and those containing the target object are merged for actual VQA. While training- free. 2C² is inefficient due to the extensive tree traversal and costly captioning process. Similar to DC ZoomEye B1 employs a hierarchical tree search, but instead of captioning, predicts confidence score for each image region. This score computed through complex mechanism involving three sequential MILM forward passes, each using different question prompt. As result, the process cannot be simplified or shared across regions, making the method inefficient due to both the hierarchical search and the high cost of confidence estimation. ViCrop [2) is an efficient, training-free visual cropping method that avoids hierarchical search by directly computing a question-guided heatmap to localize the target object. However, its best- pest-performing variants depend on Q-K attention weights and answer gradients, making the method incompatible with efficient atten- tion implementations such FlashAttention [1O]. Unlike these methods, our method is training free visual cropping approach without additional modules besides the MLLM. Rather than relying on recursive search, captioning, Q-K attention weights, employ informed search guided by internal representations of the MILMM that is compatible with efficient attention implementations. This design enables our method to achieve significantly higher efficiency without sacrificing accuracy.
V*Bench
100-
Tox:VVA
Rel-COCD
Relative accurrence [%6]
60
40-
20.03k.3
273
20.3
5.8
0.0
11
0.0
0.0
0.0
Relative sire of questior-relevart object(s)
Figure I: Many VQA datasets focus large instead of tiny objects. This figure shows the relative area of question-relevant objects w.r.t. the image. V*Bench contains various tiny VQA-relevant objects.
3 FOCUS: Fine-grained visual object cropping using cached token similarity
We first provide relevant background information for method in Sec. 3.1 Next, we describe in detail how our method proposes relevant image regions based on the KV-cache in /ecc. 3.2) Finally, we explain how these image regions are used for fine-grained VQA tasks in Sec. 3.3 visualization of FOCUS Fig.
We provide
3.1 Background
MLLMS typically comprise three core components: vision encoder, modality projector, and Large Language Model (LLM). The LLM receives an input sequence of tokens X (#1...., (.,)) and predicts the next token xn++1 auto-regressively (28). This sequence can be viewed as concatenation of visual tokens X,иx = (æ)..... .x,) ) and instruction textual tokens X = (Iga41...... Hn), where is the grid size of the vision encoder (for simplicity, we ignore system prompt tokens here). In
many models, the visual tokens from the vision encoder are extracted from downscaled global view of the image and projected into the LLM's feature space via the modality projector. The resizing
3

---

Identify target object using in-context learning
(II) Generate pseudo-attention using cached token similarity from MLLMs
(III) Construct object map
(context)
there a paraglider in the image? cosine smilarity
color of car?
need the info about car.
What the color the paraglider?
Element-wiie multiplication
Image tokens
need the info about paraglider.
Question tokens
Value cache from loyer? of MLLMs
(IV) Propose regions of interest
(V) Rank regions of interest based on
(VT) Final VQA with the selected region
existence confidence
What is the color of the paroglider?
there paraglider in the image?
R5
(without FOCUS)
LLM
Red / Unknown
X
What is the color of the paraglider?
"Yes" (+0.97)
"No" (-0.71)
"No" (-0.99)
locate anchor points (b) propose the regions of interest con-malimum suppression
(with rOCUs step (-V) Blue
(the selected region)
Figure 2: Overview of FOCUS. The method identifies the target objects mentioned in the question (I) and constructs their object relevance maps using cosine similarity between cached tokens (II III). Then, it proposes regions of interest and ranks those by the existence confidence the target object in each region (IV + V). Finally, the selected region is used to perform VQA (VI).
aligns the input image with the base resolution expected by the vision encoder, e.g., 336 336 for CLIP [29] and 384 x 384 for SigLIP [11. While this regime is effective in many scenarios, it
tends fail on ame-grained VQA tasks B77 42 To address this limitation, many global-local-vin
MLLMs [16 22 23 40) partition the original (unresized) image into local crops in addition to the resized global view. These crops are encoded into additional visual tokens, extending the visual input to Xvis (ZI...... ,.....,,.... (b21(b+))) This results in rapid growth of computational cost due to the quadratic complexity of self-attention in transformer layers [36] We aim to improve the performance of both giobal-view and global-local-vier MILMss on fine-grained VQA tasks by constructing map that efficiently localizes the image regions most relevant to target object mentioned in question. This map is referred to as the object relevance map. Once the relevant regions are identified, we restrict processing to these areas for the final VQA prediction, thereby improving accuracy and computational efficiency.
3.2 Constructing object relevance maps from cached token similarity
Localizing objects in images remains a challenge for MLLMs (6. While many models are fine-tuned on predicting bounding boxes (20), they often fail when prompted directly for the location of small objects, frequently producing hallucinated or imprecise coordinates [6]. Instead of relying explicit prompting, we propose to localize target objects by leveraging value features of cached tokens of MLLMs. Recent work
(25) shows that visual tokens in the sequence largely preserve spatial correspondence to their originating image regions across transformer layers [36]. In fine-grained questions typically involve one or more specific objects in the image, which we refer to as the target object(s). Since the text tokens corresponding to these targets co-cxist alongside visual tokens in the token sequence, estimate object relevance map by computing the cosine similarity between the text and visual tokens.
VQA datasets 3 38 43
To construct this object relevance map, we first identify the text tokens corresponding to the target object(s) as shown in Fig. (I). Inspired by ZoomEye [31]. we use the in-context learning (ICL) capability of MLLMs I2 to extract the target object(s) by providing a few examples in the prompt. This might extract one or multiple target object(s) from the question. For each target object, we apply a generic prompt template "Is there ftarget object} in the image?" to query the MILM alongside the image. Due to the tokenization (36] of the MILMM, the target object can result sequence of target text tokens Xet (fo...., G. ..... 2.). We calculate the cosine similarity between

---

the target tokens and the visual tokens in the sequence to construct the object relevance map. While one might consider using standard query-key (Q-K) attention weights [66] for this purpose, many recent MILMSs employ efficient attention implementations such as FlashAttention [T), which do not generate Q-K attention weights explicitly. We address this issue by using value features preserved in the K V-cache mechanism [27]. which is commonly used to accelerate autoregressive inference by storing intermediate representations. Leveraging this, we propose an alternative value-value (V-V) pseudo-attention approach as shown in Fig. (II. For each target token f i E {O....,5) 5} in the (-th layer, we compute pseudo-attention map A E Raxa by measuring its cosine similarity (cos) with the visual tokens (zi..... zz)), where value features are available via the KV-cache:
A = =o=cos (i), =f),, for j a
and reshape A into an X a matrix. Alternatively, one can also compute A! using key features, see SSec. 4.3 for details. In preliminary experiments, we empirically find that the pseudo-attention map A from a single layer might be noisy. Therefore, we aggregate maps from (-th layer to L-th layer using attention rollout (1. incorporating residual connections to better preserve information flow as follows:
A, = Σ (A} 11/2,
km/
where I denotes the identity matrix. We aggregate the pseudo-attention maps A, for each individual target token ét by element-wise multiplication to capture consensus as shown in Fig. 2 (III):
A = A A
@ A.
This operation allows only regions that are consistently highlighted across all target tokens to remain prominent. For example, the token red may highlight many red objects, but when combined with car, only regions corresponding to red cars will be emphasized. We refer to A as the object relevance map corresponding the target object. A normalization of A is performed after every matrix addition multiplication to ensure numerical stability.
For global-local-view MLLMs, instead of calculating cosine similarity between the target tokens and the visual tokens from the global view, we use visual tokens extracted from the local crops. We empirically find that these local tokens can better capture fine-grained details. We compute the pseudo-attention map A using local visual tokens:
A = cos(( (o), 2y),, for j = a² + (bb
and reshape A into h x w matrix, where / and w denote the spatial dimensions of the local visual tokens arranged to closely preserve the original image's 's aspect ratio, so that h w a² b. To reduce noise and enhance spatial coherence, we empirically apply a Gaussian filter to A, followed by downsampling, yielding a cleaner object relevance map.
3.3 Ranking of proposed regions of interest (ROIs) for fine-grained VQA
Given an object relevance map, we define in the following a relevance score for each of its elements. As shown in Fig. 2 (IV), once the object relevance map obtained, we extract the locations of the top-k highest scores as anchor points on A, which represent regions likely containing the target object. To ensure sufficient spatial coverage,
select relatively large k and enforce a minimum distance Sdist between anchor points. Anchor points that are too close are discarded. Then, we generate an initial symmetric ROI of minimal size Smin per anchor point.
Each initial ROI is then expanded up to the maximal size Smax. stopping when the average relevance score within the ROI falls below predefined threshold. We rank the resulting ROIs based on the relevance score at their respective anchor points. To eliminate redundancy, we apply non mon-maximum- suppression [14 [7]. using low threshold to promote diversity among selected regions. This encourages broader spatial coverage.
The resulting object relevance map can be noisy due to spurious high-activation tokens 11 that do not correspond to the target object. a result, a ROI with a high relevance score may not actually contain the target object. Therefore, we verify whether the ROI contains the target (see App. A.5
5

---

for details). Similar to ZoomEye [31), the ROI is provided to the MLLM together with an existence prompt, and an existence confidence is computed from the model's response, as shown in (V). Then, we rerank the top Rsteps ROIs according to their existence confidence, with the number of steps Dsteps controlling the Forward Pass (FP) budget.
Final VQA inference In the previous paragraphs, we demonstrated how to obtain the most relevant ROIs for each target object. These ROIs are now passed the MLLM for the final VQA prediction. We follow the inference strategy from ZoomEye [I which categorizes questions in fine-grained VQA datasets into two types. Type-I questions involve single-object instances and type-2 questions concern multiple instances of an object type. These categories can be automatically inferred using ICL or keyword-bare wwsee heuristics, without requiring prior knowledge of the question. For type- questions, we select the ROI with the highest confidence score for each target object. If the question involves multiple target objects, we combine the regions covering all relevant objects for the final VQA. For type-2 questions, we iterate over all proposed ROIs and select those with confidence scores higher than a threshold. This step not constrained by the nsspps In the case of (lobal-local-viee MLLMs, this process slightly differs. Instead of combining the visual crops into single area, utilize the model's ' image view-where all target objects are visually highlighted-allngside the top-confidence ROIs for each target object. Please check App. A.5 for details.
capabilities 12 22 23). Specifically, we provide one global
4 Experiments
We first describe the implementation details of our experiments in Sec. 4. and provide the results on fine-grained VQA datasets in [Sec. 4.2 Further, we conduct an ablation study and hyperparameter sensitivity analysis in Secc. 4.3 and provide additional results (i) on VQA datasets with larger objects
and (ii) with Qw5n-2.5-Vi [4] in Sec, 4.4. Finally, we show some qualitative examples and discuss the limitations of FOCUS in Sec. 4.5 and Sec. 4.6 respectively.
LLaVA.1.5 5-78 V'Bench
1:78 HRBandh-4K
LLaVA-1 5-78 HRBanch-SK
88
3:3h more ofident
more
58
39
Accurscy [%6]
48
19
LLaVA-OnettsonnTB V'Banch
LLaWA-OneMfslon-TBB HRBench-4K
LLaVA-CreVisinn-7 HRBendh-aK
02
70
86
à 57x more-ceTeed
80
74
66
50
16
21
28
25
37
Avg. Forward Passes (FPs) for visual searching per Question [M]
Vanilia baseline
DC2
-
ViCrop
SEAL
ZoomEye
FOCUS (Ours)
Figure 3: FOCUS is the Pareto front fine-grained VQA benchmarks. Given the same computation budget. FOCUS (purple crosses) significantly outperforms other visual cropping methods, on three different datasets and for two model architectures. It achieves 6.5x higher efficiency than the best-performing bascline ZoomEye. Note that we show only a limited set of data points for each method to ensure a clear visualization. The full results available in App. C.I
4.1 Implementation details
Following prior work BT
37 38), we evaluate FOCUS on several fine-grained VQA benchmarks:
V*Bench [88] HRBench 37 and MME-Real HMME-RealWorldLiit [43]. We report accuracy on multiple- choice questions as the primary performance metric. Another critical consideration the trade-off between performance and efficiency, as inference with MLLMs can be computationally expensive. In
the case multiple-choice VQA, inference speed is largely determined by the sequence length during the prefill phase 32 36 Since the sequence length remains relatively consistent across different searches, we quantify efficiency using the number of Forward Passes (FPs) required for the visual search. We provide details of used hardware and on the caleulation of all metries in App. A.I and

---

App. A.2 We evaluate our method using two types of MLLMs: LLaVA-1.5-7B [4], global-view MLLM. As described in Sec. 3.2] we utilize representations from multiple layers to compute the object relevance map. For LLaVA-1.55 we use representations from the 14th to the 32nd layer (1 14, I, = 32), and for LLaVA-One Visiionon from the 14th to the 28th layer (1 = 14, I. = 28). To evaluate performance under varying computational budgets, we adjust only the number of steps, setting sstpps E {1, 3, 4, 6, 8). We describe the configuration of the remaining hyperparameters of FOCUS in App. A.3 for analysis of FOCUS's s hyperparameter sensitivity, see SSc. 4.3
MLLM, and LLe Vision-8 [22], a global-local-viee
For ZoomEye, we vary the number of crops per layer and the cropping depth. For ViCrop, we report results from its best best-performing variants, i.c. rel-attn and attn- attn-grad. For DC we determine the FPs via the base resolution of the vision encoder. For SEAL, we evaluate the publicly available pre-trained model without modifications. Additional implementation details are provided in App. A.4 Note that SEAL and ZoomEye use a different inference scheme on V*Bench compared to the open-ended generation approach of FOCUS. With this alternative scheme, we observe significantly improved performance with ZoomEye on V*Bench. Further, we observe notable gap between our ZoomEye results and those reported in the original paper. Detailed results are provided in App. C.5
4.2 Results on fine-grained VQA datasets
We conduct experiments with LLa on V*Bench, HRBench-4K, and HRBench-8K (see Fig. 3). Overall, FOCUS outperforms the four other visual cropping methods on a relatively small computa- tional budget of fewer than 17 FPs. FOCUS achieves an accuracy of 72. 72.77% on V*Bench, 51.75% on HRBench-4K, and 45. 45.00% on HRBench- -... SEAL achieves slightly higher accuracy than FOCUS on V*Bench, but at the cost of significantly lower efficiency due to its multi-module design and recursive visual search. On high-resolution datasets like HRBench, its performance is only on par with the vanilla MILM. We suspect this decline stems from SEAL'S training data, which is specifically optimized for resolutions below 2K. ZoomEye achieves the highest accuracy on V*Bench (77.48%) but only with an extremely deep tree search, resulting in substantial computational overhead. At our top accuracy of 72.77% FOCUS is 3.43x more efficient than ZoomEye. On HRBench-4K, our method not only surpasses ZoomEye with a better top accuracy but also uses 4.39x fewer FPs. On HRBench-8K while ZoomEye attains higher top accuracy of 49.00% FOCUS achieves 44.88% with 4.72x greater efficiency. This performance gap on HRBench- 8K is likely due to the limitations of the 24 x 24 object relevance map produced by VL. Smaller objects often remain unde- tected, limiting our model's ability to improve accuracy even when increasing nsteps to allocate more computation.
With LLa Vision,,,,,,,,, we conduct experiments on the previously introduced datasets (see Fig. 3) and additionally evaluate on AMME-Rea/World-LLe (see Tab. 2). We compare FOCUS only to Zoom- Eye, as SEAL does not provide any models based on LLa VA-VA-One Vision. Moreover, neither Vi- Crop nor DC supports evaluation with VL- Vision. FOCUS significantly benefits from the higher-resolution object relevance map generated based on the local crops. As a result, our method outperforms ZoomEye both in terms of accuracy and efficiency on the three datasets. Although Vision natively supports resolutions up to 2K, FOCUS can still boost the accuracy on V*Bench (2K) from 74.46% to 92.15% This be attributed and reducing irrelevant background regions. Furthermore, we perform additional evaluation on MME- Real RealWorld-Lite, see Tab. FOCUS outperforms the vanilla baseline on most sub-tasks. ZoomEye and our method have strengths in different domains. FOCUS is better for reasoning, while ZoomBye is
method isolating target objects
Table 2: Results on ReM--RaalWorld-iit We provide the accuracy for each task. Further, we report the average accuracy and FPs per subset for efficiency comparison. The dataset includes the domains OCR (Optical Character Recognition). RS (Remote Sensing), DT (Diagram and Table), MO (Monitoring), and AD (Autonomous Driving).
Perception
Reasoning
Sub-tusk necurncy [%]
Average
Sub-task accurncy
[%]
Average
Model
OCR
RS
DT
MO
AD
Acc. (96) 1
FP [+]]
OCR
DT
MO
AD
Acc, [%]
FP [+]
LLeVA CLeVA-OV-20 OV-7B
81.60
52,00
65.00
34.48
43.14
52.01
72.00
40.00
41.00
32.35
40.93
w/ ZoomEye
81.20
51.33
74.00
38.87
$1.43
$6.29
41.60
64.00
49.00
46.00
35.50
43.20
45.95
FOCUS (Ouns)
83.60
46.67
51.15
51.33
33.50
7

---

better at perception tasks. Still, our method is on average 5.47x more efficient than ZoomEye. We provide the full numerical results in App. C.I
Comparison with ZoomEye ZoomEye's strong performance on fine-grained VQA is largely driven by its exhaustive tree search, which includes high number of image regions. This leads substantial number of FPs, as each region requires three FPs with different prompts for confidence prediction-making the overall process computationally expensive. In contrast, FOCUS leverages an object relevance map derived from internal representations to identify target object locations with just a single FP explaining its superior efficiency compared to ZoomEye. We report additional efficiency metrics, including execution time, FLOPs and memory usage. in Appp.. C.4, Crucially, our analysis shows that reducing FPs not only lowers theoretical computation but also leads to significantly shorter execution times, confirming the practical efficiency of FOCUS compared to ZoomEye.
4.3 Ablation studies and hyperparameter sensitivity analysis
In this subsection, we validate the design choices of FOCUS on V*Bench and HRBench-4K, using LLaVA-1.5 (see Tab. 3) Fig. 4). V*Bench the only dataset that provides GT region annotations, allowing us to report both accuracy and recall. We calculate recall by checking whether one of the proposed ROIs overlaps with the GT region by at least 50%. We use the same hyperparameters across all ablation studies, with the number of steps set to Nsteps 8 unless stated otherwise. Furthermore, we analyze the hyperparameter sensitivity of FOCUS on V*Bench.
Component analysis We first verify the contributions of the object relevance map (see Sec. 3.2 and the ROI ranking (see[Sec. 3.3). To assess the impact of the object relevance map, we replace with randomly generated map. This substitution leads to a substantial decrease in accuracy on both datasets interestingly, however, FOCUS still performs well above random guessing (35.99%). This indicates that even without an accurate global understanding of the object's 's location, our method can identify likely regions through confidence-baat based ROI selection, demonstrating our ranking mechanism's 's robustness. Next, we assess the effect of discarding ROI ranking by directly selecting the ROI with the highest object relevance score for the final VQA. This yields a recall of 38.48% on V*Bench, with accuracy improvements of 3 percentage points (pp.) on V*Bench and 5 pp. on HRBeneh-4K compared to the vanilla baseline. These results suggest that object relevance maps, even without proc-proc,,,ing, serve strong prior for identifying meaningful visual regions.
100
Table 3: Ablation studies of FOCUS. We evaluate the influences of design choices of our method based on accuracy and recall. "rel." is short for "relevance".
R Accuraccy [56
80
70
Ablation
VeBench
HRBench-4K
Object rel. map
Proposal
Acc.
Recall
Acc.
ranking
[X6] f
[%[] 1
[%] 1
Recall -Lawk 1.77
Component
LLaVA
48.68
18.37
36.13
Racall Onevision
x
51.30
38.48
41.13
40
Pseudo-attn.
K K. (wlo RoPE)
69.10
63.47
45.63
12
Avg. Forward Pesses (FPs) per Question
Layers
U-14 14
66.49
76.17
47.38
32
71.20
75.56
49.38
Figure 4: Ablation studies on the search steps. We analyze how the number of search steps influences the recall and accuracy on V*Bench. A positive correlation between accuracy and recall can be observed.
Original design choice
72.77
77.49
51.75
Vanilla baseline
47.64
36.13
Random guess
35.99
25.00
Value features vs. key features in pseudo-attention One might argue that key features- central to standard attention-encon could be used to generate more precise object relevance maps. However, directly substituting key features for value features in FOCUS results in degraded performance. This is mainly due to the use of Rotary Positional Embedding (RoPE) [3] in recent MLLMS 4
richer semantic information [2] compared value features and therefore
9 22 24), which injects position-dependent rotations into the key features. As a result, RoPE causes nearby tokens to exhibit artificially high cosine similarity due to positional proximity, rather than semantic alignment [3 34) In this ablation, we remove RoPE from the key features to isolate semantic content before computing object relevance maps. Despite this, both accuracy and recall remain lower than those achieved
8

---

using value features (see Tab. 3). We hypothesize that removing RoPE disrupts the semantic integrity of the key features, as they are trained to operate with positional encoding, resulting in noisier relevance maps.
Later layers vs. earlier layers Apart from using the default later layers (14 -32) for LLaVA-1.55
we also experiment with representations from earlier layers (0- 14) or all layers (0 (0-32). 32). We observe that the representations from later layers yield the best performance across both datasets. This is consistent with findings from the Logit Lens technique [26]. which suggests that later layers encode more predictive and semantically coherent representations.
Number of steps In FOCUS, we vary only the number of steps of the ROI ranking in our method (see (Sec. 4.1). For this ablation, we report accuracy and recall on V*Bench using LLaVA-1.5 and LLaVA-One Vision. Across both models, recall increases with more steps, as additional lower-priority ROIs are explored and more of the image is covered (see Fig. 4). Similarly, accuracy also improves with more steps but begins to plateau beyond a certain point. This saturation might occur because the model fails identify the correct region among a considerable number of proposed ROIs during ranking. Another reason could be that even when the correct region is provided, the MLLM is unable give the correct answer, due object size, ambiguity, or other limitations.
Hyperparameter analysis We investigate FOCUS's s sensitivity on key hyperparameters and find it to be robust across wide range of parameter choices. In particular, do not observe any accuracy degradation larger than 4.71 pp. for LLaVA-1.5 and 2.62 pp. for LLaVA-OV. The complete results of this hyperparameter analysis are provided in App. C.3 Please note that this study conducted post-hoc and not used to optimize the performance of FOCUS.
4.4 Additional results
In this subsection, we analyze how FOCUS performs on VQA questions involving large-size objects. Moreover, we provide results for FOCUS when using the state-of-the-ar model Qwen-2.5-VL. as the base MILM.
Results on VQA datasets with large objects While the previously used datasets focus on fine-
grained VQA, we also evaluate the performance of FOCUS datasets featuring large-scale objects to
assess its robustness across varying object sizes. We compare FOCUS on the datasets A-OKVQA (30] and GQA 18] using VA-15. and VA-OV with the vanilla model and ViCrop the latter is the only henchmark method providing respective results (see Tab. 4). Overall, FOCUS demonstrates strong robustness datasets with large objects, maintaining competitive performance compared to the baseline models.
Results with Qwen-2.5-VL Qwen-2.5-VL 4 processes high-resolution images with native resolu- tion, thereby preserving spatial details more effectively. We evaluate FOCUS with Owen-2255VL--7 (see Tab. 5) and find state-of-the-an accuracy HRBench-4K and HRBench- 8K. This confirms the compatibility of FOCUS with different MLLM architectures.
Table 4: Results on VQA datasets with large objects. We find only minor performance degradation of FOCUS w.r.t the base model.
Table 5: Results of FOCUS with Qwen-2.5-VL FOCUS significantly boosts the performance of the base model, consistent with previous results for
VL and LLaVA-OV.
A-OKVQA Acc. [%]
GQA Acc. [%]
Model
A
Bench (%)
HRBench-4K
HRBench-8K [%]
Model
[%]
LLaVA-1.5
77.99
61.97
ViCrop
60.66
-17.33
60.98
-0.99
Owen-2.5.8 VI.
79.06
71.62
68.62
FOCUS
74.76
-3.23
60.34
-1.63
FOCUs
90.58
79.25
76.25
LLaVA-OV
91.44
62.01
FOCUS
91.00
-0.44
51.02
-10.99
4.5 Qualitative examples
We provide two qualitative examples that show how FOCUS improves performance of LLa for single-target tasks and LLLVVA-Ore One for multiple-target tasks (see Fig. 5). In both examples, the accurate visual crops generated by FOCUS enable the respective MIL.M to answer the question correctly. The detected location of the target objects is highlighted in the object relevance map. One

---

can see that VLe Vision provides bigher-resolution and cleaner object relevance map. We provide more examples App.
Question: ih the color of the tissue box? (A) gray (B) white (C) black (D) blue
(I)
Label: D A (LLaVA- x nAnwer (LLaVA-1.5w w/ FOCUS):
Object relevance map
Original Image
GT region
Selected RO1
tissue box Selectedd: Rg
R
R
RS
PA
R2
R
R
R
MAN
Question: Is the soccer ball the left right the water dispenser ? (A) left (B) right (II)
Label: A Answer
A X
nnwer (LLaVA-OneVision) w/ FOCUS):
Original image
Combined region
Object relevance map water dispenser Selected:: R
Object relevance map soccer ball Selected: R
R
Rio
RR
Figure 5: Qualitative examples FOCUS. provide some exemplary inferences with our method single- target tasks with LLaVA-1.5 (I) and multiple-target tasks with LLaVA-One SaAAA-OneViionn Vision (II). The Ground Truth (GT) locations are highlighted in red in the original image. Further, we show the detected image regions and their locations in the object relevance map. Note that the object relevance maps corresponds to the original images.
4.6 Limitations
One limitation of our method is its reliance on the resolution of the object relevance map. When the input image is high-resolution (e.g., 8K), but the internal representations of the MILM can only produce low-resolution relevance map, accurately localizing fine- tne-ggained objects becomes difficult. This limitation partly explains the reduced effectiveness of our method with LLaVA-1.5 on HRBench-8K. A potential solution is to construct the object relevance map in sliding-window manner over the image, allowing finer spatial resolution. Moreover, FOCUS inherits the typically limited understanding of spatial relationships 33) from the base MLLM, as it is training-free method. Thus, FOCUS struggles with spatial concepts such "on the left/right of the image". We leave these shortcomings for future work.
5 Conclusion
In this work, we proposed FOCUS, an efficient, training-free visual cropping method for fine-grained VQA tasks, where identifying small objects essential. Our method constructs object relevance map from cached token representations to localize image regions relevant to the question, enabling detail-focused VQA inference. FOCUS achieves performance on par with better than existing meth- ods across multiple fine-grained VQA benchmarks, while requiring significantly fewer computational overhead. These results highlight the potential of training-free, high-resolution VQA systems that both effective and computationally efficient. Moreover, the central idea of FOCUS harnessing the hidden spatial capabilities of MLLMs via an inference-ting time method holds significant promise for spatial reasoning tasks well beyond VQA.
10

---

Acknowledgment
The authors gratefully acknowledge financial support by the German Federal Ministry for Economic Affairs and Energy under the grant numbers 19A24004U and 19A24004I part of the Safe Al Engineering consortium.
DISCLAIMER: The results, opinions and conclusions expressed in this publication are not necessarily those Volkswagen Aktiengesellschaft.
References
[1] Samira Abnar and Willem H. Zuidema. Quantifying attention flow in transformers. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 4190 4197. Association for Computational Linguistics, 2020. doi: 10. 18653/VV1 A020.ACL-MAIN.330
httts:s/::///o org/ 10. 18653/v1/2020 acl- main.man... 385
[2] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Has- son, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L. Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan. Flamingo: a visual language model for few-shot learning. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 Decem- pappr/2022/nar
ber 9, 2022, 2022. URL http:://papess. nips cc wwtpcpaper__teeeeeeeeeeeeee files/ wwt.ff/eppeeeeeeeeeee/ 9000177607600117cccb444111146400 Abstroct oat.aa/eeeeeoottttttttaoooooorra Conference html
[3] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruy Batra, C. Lawrence Zitmick, and Devi Parikh. VQA: visual question answering. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pages 2425-2433. IEEE Computer Society, 2015. doi: 10. 2..10//CCCV.2015.7700 URI. https
D.tol..c//...110/ICCVV..... 2015. 279
[4] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang. Humen Zhong, Yuanzhi Zhu, Ming- Hsuan Yang. Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng. Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. 5wen2.5-vl technical report. CoRR, abs/2502. 13923, 2025. doi: 10.48550/ARXIV.2500.. 13923. URL https://doi. org/10. /0/0000000 48550 18550/arXiv 2502. 13923
[5] Boyuan Chen, Zhuo Xu, Sean Kirmani, Brian Ichter, Dorsa Sadigh, Leonidas J. Guibas, and Fei Xia. Spatialvim: Endowing vision-language models with spatial reasoning capabilities. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 14455-14466 IEEE, 2024. doi: 10.1 10.1109/VPP52733.2044 01370. URL /ttpss://oof
11tp:/....100////2//1/00 2024. 01370
[6] Jierun Chen, Fangyun Wei, Jinjing Zhao, Sizhe Song, Bohuai Wu, Zhuoxuan Peng, S.-H. Gary Chan, and Hongyang Zhang. Revisiting referring expression comprehension evaluation in the of large multimodal models. CoRR, abs/2406. 16866, 2024. doi: 10.48550/ARXIV.2406.. 16866.
URL httph:t/t/oiiiii org/ 10. 48550/ arXiv 2406 16866
[7] Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-ple inference acceleration for large vision-language models. In Ales Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol, editors, Computer Vision ECCV 2024 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part LXXXI, volume 15139 of Lecture Notes Computer Science, pages 95. Springer, 2024. doi: 10.1007/978-3-031- wwW.ttts: https:://doo org/ 10. 1007 0-0---3----------77777111------- 7_2
URL

---

[8] Zhe Chen, Weiyun Wang. Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong. Kongzhi Hu, Jiapeng Luo, Zheng Ma, Ji Ma, Jiaqi Wang, Xiaoyi Dong, Hang Yan. Hewei Guo, Conghui He, Botian Shi, Zhenjiang Jin, Chao Xu, Bin Wang, Xingjian Wei, Wei Li, Wenjian Zhang, Bo Zhang. Pinlong Cai, Licheng Wen, Xiangchao Yan, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites, Sci. China Inf. Sci., (72))), 2024. doi: 10. 10.1007/S11432-02--42 //doi. org/ 1007 s11422/1112222111 423--231
URL https
[9] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning. In Alice Oh, Tristan Nau- mann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Ad- vances in Neural Information Processing Systems 36: Annual Conference on Neural In- formation Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023. URL http::/papperrs nips. cc wttpppperfillee/////eeeeeeeeeeee pwppp/ppp//pp Seaper/2023/nae 2023/hash/ 000644356754119836ffff....
A Conference html
[10]
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficiet exact attention with io-awareness. In Sanmi Koyejo, Mo- hamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neu- ral Information Processing Systems 35: Annual Conference on Neurul Information Pro- cessing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- Decem- ber 9, 2022, 2022. URL. http::/ www.. cc/ wwpprpppppppppppppppeepeeeeeeeea Saaperffileee files/ files/paper/2200 pappr/20222222/ 2apppr/222///ahhh / http://22222200000222222222221..
wwweeedddd-Dsttttrteeeee Ceteeraact-ooffeeencee hhtel .ttml
[11] Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL http:tt/oopereriiiss. net/forum?id 2ddmmmmiiddddmn111111
[12] Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma. Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui. A survey on in-context learning. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16. 2024, pages 1107-1128. Association for Computational Linguisties. 2024. URL
acl http://aaaaaatooog...o.oo org/ 2024. emnlp main.ainn... 64
[13] Mark Endo, Xiaohan Wang, and Serena Yeung-Levy. Feather the throttle: Revisiting visual token pruning for vision- vision-language model acceleration. CORR, abs/241 2. 13180, 2024. doi: 10.485 1044855/ARXIII2212.1118
https://:///oi org/10. www.://///......////////////..// arxiv 2412. 13180
[14] Ross B. Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2014, Columbus, OH. USA, June 23-28, 2014, pages 580-587. IEEE Computer Society, 2014. doi: 2.1109/CVVR.2014..00 URL httLhhtps://ot org/10. /OV.. 2014.81 81
[15] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Flevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
[16] Zonghao Guo, Ruyi Xu, Yuan Yao, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, and Gao Huang. Llava-uhd: An LMM perceiving any aspect ratio and high- resolution images. In Ales Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol, editors, Computer Vision ECCV 2024 18th European Conference, Mi- lan, Italy, September 29- 29-October 4, 2024, Proceedings, Part LXXXIII, volume 15141 of Lecture Notes in Computer Science, pages 406 Springer, 2024. doi: 10. 100110779788-3031-330000
23. URL httpst 10t.
12

---

[17] Jan Hendrik Hosang, Rodrigo Benenson, and Bernt Schiele. Learning non maximmm suppres- sion. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 6469-6477 IEEE Computer Society, 2017. doi: N0.119/CVVR.2017..66
URL StRl.htps::// htop:://doi..oo orggoogg///oo 1oorr///..... 10. 1100..119111 1109/1100PPPP 2017.6017..
[18] Drew A. Hudson and Christopher D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[19] Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean-Bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Eti- enne Pot, Ivo Penchev, Gaël Liu, Francesco Visin, Kathleen Kenealy, Lucas Beyer, Xiaohai Zhai, Anton Tsitsulin, Róbert Busa-Fekete, Alex Feng, Noveen Sachdeva, Benjamin Coleman, Yi Gao, Basil Mustafa, Jain Barr, Emilio Parisotto, David Tian, Matan Byal, Colin Cherry, Jan-Thorsten Peter, Danila Sinopalnikov, Surya Bhupatiraju, Rishabh Agarwal, Mehran Kazemi, Dan Malkin, Ravin Kumar, David Vilar, Idan Brusilovsky, Jiaming Luo, Andreas Steiner, Abe Friesen, Abhanshu Sharma, Abheesht Sharma, Adi Mayrav Gilady, Adrian Goedeck- emeyer, Alaa Saade, Alexander Kolesnikov, Alexei Bendebury, Alvin Abdagic, Amit Vadi, András György, André Susano Pinto. Anil Das. Ankur Bapna, Antoine Miech, Antoine Yang. Antonia Paterson, Ashish Shenoy, Ayan Chakrabarti, Bilal Piot, Bo Wu, Bobak Shahriari, Bryce Petrini, Charlie Chen, Charline Le Lan, Christopher A. Choquette-Chove CJ Carey, Cormac Brick, Daniel Deutsch, Danielle Eisenbud, Dee Cattle, Derek Cheng, Dimitris Pa- paras, Divyashree Shivakumar Sreepathihalli, Doug Reid, Dustin Tran, Dustin Zelle, Eric Noland, Erwin Huizenga, Eugene Kharitonov, Frederick Liu, Gagik Amirkhanyan, Glenn Cameron, Hadi Hashemi, Hanna shal Tushar Lehri, Hussein Hazimeh, Ian Ballantyne, Idan Szpektor, and Ivan Nardini. Gemma 3 technical report. CoRR, 11w/2503.197866 2025. doi: 10.4550/ARXXV2250..9999 URL 2503. 19786
Harman Singh, Harsh Mehta, Har-
https://doi. org/10.
[20] Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Refer- ring to objects in photographs of natural scenes. In Alessandro Moschitti, Bo Pang. and Walter Daelemans, editors, Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACI. pages 787-798. ACL., 2014. doi: htt.:http:::/d/e org/ 10. 31155v1/11441066
[21] Mengcheng Lan, Chaofeng Chen, Yiping Ke, Xinjiang Wang, Litong Feng, and Wayne Zhang. Proxyclip: Proxy attention improves CLIP for open-vocabulary segmentation. In Ales Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol, editors, Computer Vision ECCV 2024 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part LXVIII, volume 15126 of Lecture Notes in Computer Science, pages 70-88. doi: 978-3-031-73113-5
Springer, 2024.
URL https://doi org/ 10.1007/
[22] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. LLa LLaVA-onevision Easy visual task transfer. Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URL
[23] Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li. Zejun Ma, and Chunyuan Li. Llava-next-imterleave Tackling multi-image, video, and 3d in large multimodal models. CoRR, abx/240..0889995 2024. doi: 10.48550/ARXIV.24007.
[ttp/ttts:////a org/
10. 148550/arXi10 2ww.22270707737
[24] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual in- struction tuning. In IEEF/CVE Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 86286-26296 IEFE, 2024. doi: 10. 109/ 01152233.2024022444 URL https: //ooi org/ 10.. 1 /1009///2252333... 2024. 02484
13

---

[25] Clement Neo, Luke Ong, Philip Torr, Mor Geva, David Krueger, and Fazl Barez. Towards interpreting visual information processing vision-language models. The Thirteenth Inter- national Conference on Learning Representations, 2025. URL. (tttps//oppnreviiie net/
Horum?id=chanJGGozZ.
[26] Nostalgebraist. Interpreting gpt: The logit lens. https://:
posts/
gpt- the- opt-the-logit-111 logit-lens
[27] Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer infer- ence. In Dawn Song, Michael Carbin, and Tianqi Chen, editors. Proceedings of the Sixth Conference on Machine Learning and Systems, MLSys 2023, Miami. FL, USA. June 4-8, 2023. mlsys.org, 2023. URL. hash/ http:///00000eee1606660000000000
http://wwww....aaaaaaaaa.arrrr/a - wttpooopppp.pppeeeee/ooppppeeeee wiwesllleppeeee pealeeepaper/2222/0 html
[28] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018. URL https://cdn. openai com/
research- horprrh--ccovere/aaaaggaaa baaaapprriiee///angggggggueeeeer OpenAl technicall
pdf
[29] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervi- sion. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning. ICML. 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 8748 8763. PMLR, 2021. URL http: htttt:///prcceedngg.. mlr. wwwpppresssl.... http://1/3rraaffd22222 html
[30] Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: benchmark for visual question answering using world knowledge. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision - ECCV 2022, pages 146-162, Cham, 2022. Springer Nature Switzerland. ISBN 978-3-031-200744
[31] Haozhan Shen, Kangjia Zhao, Tiancheng Zhao, Ruochen Xu, Zilun Zhang, Mingwei Zhu, and Jianwei Yin. ZoomEye: Enhancing multimodal lims with human-like zooming capabilities through tree- image exploration. In Conference on Empirical Methods Natural Language Processing 2025, 2025. doi: 10.4550/ARXIV.241...6 16044. URL prttps://dae org/10 18550/arXiv 2411 16044
[32] Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, lon Stoica, and Ce Zhang. Flexgen: throuhhhr generative inference of large language models with single GPU. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 31094-31116, PMLR, 2023. mlr. press/v202) v202/ sheng23a.hto html
URL http::ttt///ooooeo
[33] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards VQA models that can read. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 20/9, Long Beach, CA, USA, June 16-20, 2019, pages 87 8326. Computer Vision Foundation IEEE, 2019. doi: 10.1109/ CVPR.2019.0085/1 URL http:
thecwf cww http///oteettttooe CV_p____C_______ CWt_CCVVRRRCCCEEEEe 2019/htm1/ Singh_Tooardae VQA MatMAMMMeees___ That_hhttaaaa Tht_aaaaCCaa_aed___PPP...0000000
html
[34] Jianlin Su, Murtadha H. M. Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024. doi: 10. NEOOO///EEC....222........ neucom. 2023 127063
URL https://////o orw.ei...rgg/1..1. 1016/
[35] Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal 1lms. In IEEE/CVF Conference
14

---

on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024. pages 9568 9578. IEEE, 2024. doi: N0.1009CVPRS223332224.000 //doi. 1100..110510 1109/ C0009/CPP52733.... 2024.00914 00914
URL https
[36] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30; Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017. Long Beach, CA. USA, pages
2017. URL
neur neurips ps cc/ wc/paper//////eec/pp//cep/ea 221r//0/////////
[37] Wenbin Wang. Liang Ding, Minyan Zeng, Xiabin Zhou, Li Shen, Yong Luo, and Dacheng Tao. Divide, conquer and combine: A training-free framework for high-resolution image perception in multimodal large language models. CoRR, abs/2408. 155566 2024. doi: 10.48550/ARXIV. 00g/10. 48550/ 2408. 15556
2408. 15556. httpps::///doi
[38] Penghao Wu and Saining Xie. V*: Guided visual search as a core mechanism in multimodal llms. In IEEEF/CV Conference on Computer Vision and Puttern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 13084-13094, IEFE, 2024. doi: 10. N..110/CCVPS22733.2000
01243. (RLLhhts::/ddee httsps://doioorg/1000 1119//1VV//2//// 2024. 01243
[39] Long Xing, Qidong Huang, Xiaoyi Dong, Jiajie Lu, Pan Zhang. Yuhang Zang, Yuhang Cao, Conghui He. Jiaqi Wang, Feng Wu, and Dahua Lin. Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction. CoRR, abs/2410. 17247, 2024. doi: 1048550/ARXIV.2410.. 17247. (tLphtts:://dot org 10. 48550/ arxiv 2410 17247
[40] Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, Qianyu Chen, Huarong Zhou, Zhensheng Zou, Haoye Zhang, Shengding Hu, Zhi Zheng, Jie Zhou, Jie Cai, Xu Han, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. Miniepm-v: A GPT-4V level MILMM on your phone. CORR, abs/2408.01 000000 2024. doi: 10,48550/ARXIV224000 01 01800. 01800
org/ 10. 48550/ arviv 2408
[41] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In /EEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pages 11941-11952. IEEE, 2023, doi: 10,1109/ ICCV51070.2023.011 100. httphhttp:////ot org/ 10. 1109/ ICCV5100V 2023. 01100
[42] Jiarui Zhang. Mahyar Khayatkhoei, Prateek Chhikara, and Filip Ilievski. MILMMs know where to look: Training-free perception of small visual details with multimodal LLMs. In The Thirteenth International Conference on Learning Representations, 2025. URL https
/ wwwparrrwiee.eeefoommiieeeeeeeee
[43]
YiFan Zhang, Huanyu Zhang. Haochen Tian, Chaoyou Fu, Shuangging Zhang, Junfei Wu, Feng Li, Kun Wang, Qingsong Wen, Zhang Zhang, Liang Wang, and Rong Jin. MME-realworld: Could your multimodal LLM challenge high-resolution real-world scenarios that are difficult for humans? In The Thirteenth International Conference on Learning Representations, 2025. wtt.ne//oooummiiccssiiiiieee
URL htttps://operreeee
[44] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, edi- tors, Advances in Neural Information Processing Systems 36: Annual Conference ral Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL httphhhtp:p:pp/eee nips cc/ pptppr_flles/pppeeeeeepeepeeeeee 2ttp//paaare222/ashhhhh 2203//aah// Abstracttb-ttttttt.aa...a-aaatt. Datasets, and, Benchmarks html
Neu-
15

---

A Implementation details
In this section, we provide additional technical details of our work. We begin by describing the hardware setup and report the total GPU hours required to reproduce our main results in App. A. Next, we explain in detail how the reported metries are calculated, see App. A.2 We then provide the hyperparameters used for FOCUS and the baseline visual cropping methods in App. A.3 and App. A.4 respectively. Finally, we present further details on the processing scheme of FOCUS in App. A.5
A.1 Hardware specifications
We run all experiments presented in Sec. 4 and App. C on identical hardware, namely compute instances equipped with NVIDIA A700 80GB GPU, an AMD EPYC 7V13 CPU, and 220 GB of RAM. FOCUS's s software environment includes CUDA 12.2, PyTorch 2.6.0, and the HuggingFace transformers library in version 4.46.0.
We also provide an estimation of the GPU hours needed to reproduce our results the entire project. In total, our reported results require approximately 132 GPU hours the hardware configuration described above (see Tab. 6). Additionally, we report the GPU hours required to run the experiments that achieve the best-performing variants for each method on LLaVA-1.5 and LLa LLaVA- VA-One Vision.
Table 6: Estimated GPU hours.
Best-perf. variant w/ LLaVA-1.5
Best-perf. variant w/ LLaVA-OV
Model
All variants
SEAL
ViCrop
ZoomEye
30
FOCUS (Ours)
40
Sum
16
39
132
A.2 Metrics
In this subsection, provide detailed description of our performance and efficiency metrics.
Performance metrics The main performance metric is the accuracy of visual cropping methods, which is typically reported as percentage. We follow the standard evaluation protocol and do not apply any post-processing to the answers generated by the visual cropping methods using the MILM. This means that if the ground-truth label is "A" and the model outputs "The answer is A" or (A) ", we do not extract or normalize the answer to match the label. As result, such responses are counted as incorrect. However, this type of mismatch is of minor importance in our experiments: across all models and three datasets (V*Bench, HRBench-4K, HRBench-8K), we did not observe any irregular
non-standard response formats. We compute the average accuracy the unweighted mean over all N samples dataset, i.c., as EN (9: y:)/N, where 9i is the predicted answer and y is the ground-truth label for the i-th sample.
Efficiency metries An important metric is the number of Forward Passes (FPs) required for the visual search. Note that we exclude the FPs needed for the final VQAs prediction, as different inference schemes can generate different forward passes, see App. C.5] We compute this by tracking how often the generate, forward, or _call__ methods of the respective MLLM are invoked per question. For methods that use multiple MILMs (e.g., SEAL), we report the total number of FPs of all MI..Ms.
Another key metric we report is the efficiency improvement of FOCUS relative to the baseline methods. Among all evaluated approaches, FOCUS demonstrates the highest efficiency. To ensure a fair comparison, we consider the top accuracy achieved by FOCUS and the best-performing baseline, and select the lower the two as the reference accuracy. For both methods, we determine the number of Forward Passes (FPs) required to reach this reference accuracy- either by taking the exact value
'SEAL uses customized MLLMS based LLaVA-7B.
16

---

interpolating between data points to estimate the FPs. This yields ouu for FOCUS and Fr for the pest-perfonming baseline. The relative efficiency improvement is then computed as FP Ferer/FPonn ours.
Furthermore, we report two additional metrics to provide a practical comparison of efficiency, i.e., execution time and peak GPU memory usage (see Tab. 13p. Execution time is recorded per sample, and we report the average time per question across a dataset. Peak memory is measured using torch. cuda www. wwwamaaaeememeeaee watss_mmor__allocttttteee _llccated () and converted to GB by dividing the result by 1024*.
(see
A.3 Hyperparameters of FOCUS
This subsection outlines the hyperparameters used in our method, FOCUS. These parameters are applied consistently across all experiments and correspond to the results reported in [Sec. As noted in the main paper, the only parameter we vary is the number of steps, Rsteps- Most other hyperparameters remain fixed across experiments for both LLaVA-1.5 and LLaVA-One Vision. The exceptions are three parameters: i and Sdisi (specific to LLaVA-1.5), and Smax (specific to LLaVA-
One Vision), as detailed in Tab. 7
For VL. we use a smaller value of l when Rsteps is low to reduce the number of proposed ROIS. Additionally, we increase dist on HRBench to ensure broader spatial distribution of the ROIs across the image. In contrast, for LLaVA-One Vision, we set a larger due to its higher-resolution object relevance map. Morcover, we Smax for V*Bench and max 5 for the other datasets, as V*Bench contains lower-resolution (2K) images.
The hyperparameters used in FOCUS are generally robust and transferable across wide range of use cases. For users applying our method to new datasets, we recommend adjusting Smax, which determines the maximum size of each proposed ROIs, based on both the resolution of the input images and the spatial resolution of the object relevance map. In particular, Smax should be chosen so that the corresponding region in the original image spans approximately 1-2x the base resolution of the vision encoder. For example, the object relevance map has a spatial resolution 60 x 30, and the input image resolution is 7680 3840, then each grid element corresponds to area of 128 x 128. Setting mmax = 5 yields a maximum crop size of 640 x 640, which falls within the recommended range of 384 x 384 768 768 for the SigLIP encoder. Additionally, if one considers migrating our method to another MILM, we recommend selecting the last 25% - 60% of the layers.
Table 7: Hyperparameters FDCUS.
Hyper- parameter
Description
LLaVA-1.5
LLa VA-OOeeissin
15
if Tsspss <
Number anchor points
k
30
30
otherwise
Smin
Minimum size of each ROT
Amin =
8min 3
9 for V Bench
mmax
Maximum size each ROI
mmax 5
Smax
5
otherwise
for V*Bench
Sdist
Minimum Euclidean distance be- tween anchor points
Adist
otherwise
Adst 2
Start layer of the used MLLM- internal representations
1 = 14
= 21
L
End layer of the used MLLM- internal representations
L 32
28
faype2
Threshold for inclusion of ROIs for type-2 questions
fxyp2 0.6
kype2 0.5
fobiddist dist
Threshold for merging ROIS of nearby objects (see App. A.5)
toby_dis dist 1200
A.4 Hyperparameters of recent visual cropping methods
This subsection outlines the hyperparameters used for the baseline methods, i.e., for DC2, SEAL, ViCrop, and ZoomEye. Further, we provide a visual comparison of these baseline methods in Fig. 6
17

---

fine-tuning
Exhaustive search
Importance map
Image tree constructiin
VOA MLLM
VOA MILM
Visual Search d MLLM
VOA
Selection via captioning
ottertion weight
uncertainty fittrring
Response
Response
Response
SEAL (CVPR 2024)
DCF (AAA) 2025), ZoomEyo (EMNLP 2025)
ViCrop (CLR 2025)
Figure 6: Comparison of recent visual cropping methods. We categorize the baseline methods SEAL., DC ZoomEye, and ViCrop based on whether they: (I) require task-specific fine-tuning, (2) employ exhaustive tree search to identify relevant image regions, or (3) compute importance maps using attention matrices.
For DC the full evaluation code is not publicly available, and were unable to reproduce their results using the provided demo code. Therefore, we report the performance metrics as stated their paper and estimate the number of Forward Passes (FPs) based on the available demo. Specifically, follow the procedure described the paper: splitting the image into patches using the base resolution of the vision encoder (i.e., 336 x 336 for LLaVA-1.55 and merging patches via hierarchical clustering to improve efficiency. Although the FPs for DC are estimated and may carry a high margin of error, the method remains less efficient than other baselines. This is further exacerbated by its region-wiss captioning step, which makes it more computationally intensive than other baselines, even when the number FPs is comparable. Furthermore, DC² consistently underperforms compared to the other methods across all three datasets- Bencc,, HRBench-4K, and HRBench-8K- trend also noted in ZoomEye's ' evaluation.
For SEAL, we use the hyperparameters described in their paper and the default configuration provided in their code. Specifically, we set the minimum search size to 224 and the minimum search scale to For the visual search, we use a confidence lower bound of 0.3 and a confidence upper bound of 0.5. Regarding the target cue, threshold to 4. Note that these parameters were originally configured for the V*Bench dataset, and did not adjust them for the other datasets.
set the threshold to 6, the decay factor to 0.7, and the minimum
For ViCrop, we select only the two best- pesst-erforming variants from their work: att set-grad-hee -ighh and rel-att-high Notably, both methods employ the high-resolution processing scheme high, which divides high-resolution images into grid of 1K sub-images and computes importance maps for each sub-image individually. As result, the computational overhead increases significantly with higher input resolutions.
For ZoomEye, we report more results per dataset-model combination than those presented in the original paper to offer a more comprehensive view of its afficiency-accuraa trade-offs. Specifically, we vary two key parameters: the number of sub-regions into which each region is split (2 or the default 4 crops), and the depth of the search tree (1, and the default 5). All other hyperparameters are specified in the ZoomEye paper.
A.5 Additional implementation details of FOCUS
We provide additional implementation details of FOCUS ensure reproducibility, including how to construct the object relevance maps and how to perform the final VQA prediction.
Constructing object relevance maps We provide PyTorch-style pseudocode in Fig. For detailed motivation and description of this method, see Sec. 3
Verification of object existence in ROI Inspired by ZoomEye 31, we use yes/no prompt "Is there a {target object} the image?" to verify whether the target object exists within a ROI. We compute a confidence score cyes based on the softmax-normalinn logits lyes INo corresponding to the responses "Yes" and "No". Specifically, eyes defined as:
CYcs = 2- (softmax(4xxss, Nos))os 0.5), CYes E [-1,1]
(5)
18

---

value features total number of LIM layers
total number of attention heads/kv-cachs heads sequence length
D: hidden size of each token embedding
s: total number of target tokens
bmm: batch matrix multiplication grid size of vision encoder
1: start of the selected layer
end of the selected layer
x visuel: indices the visual tokens, [a**2,]
x target: indices the target tokens, [s,]
x __layer: indices of selected layers, [L-1,]
X = 12 nonnomllizee (X, dim=-1) # G, E, N, D
reshape (X) O, N, H+D
calculate cosine similarity bxm(X, X.permute 2, 1)) # o, N,
In' Initialize rel relevance map cejeet_re___nee map ones like (x visual)
for target in x Larget:
initialize pseudo-atta for each target token
@aeudo_attn zeros_ (x_ visual) [a**2]
for layer in x lalayer: identity eye (N) $ IN, N1
add residual connection X layer (X[layer] identity) / [N, N]
# row-wi se
X layer /= x
[N, N]
pseudo attn X |aayer x_visual] # [a**2]
pseudo attn layer fecer__sedd__tte attn [a**2]
gl glabal normal normalization
pseudo_ed attn /- pseudo_attn.as attn..uttng....
element-wise multiplication of pseudo-attentions object_rell rel map == pseudo_atte attn
object rel map reshape (objectt_ rel map)
[a, a]
Figure 7: PyTorch-style pseudocode for constructing object relevance maps in FOCUS.
we differentiate between type-1 1 and type-2 questions. Type-I questions involve single-object instances, e.g., "What is the color of the car?" or "What is the relative position of the ball to the bench?". Type-2 ques- tions concern multiple instances of object type, e.g., "How many bikes are in the image?". We follow the inference strategy introduced ZoomEye [31] when passing the selected ROIs to the MLLMs. In this appendix, elaborate the strategy used for global-view MILMs, which typically accept only a single low-resolution image input.
Final VQA prediction As explained in Sec. 3.3
For type-1 questions, select the ROI with the highest confidence score for each target object. If the question involves multiple target objects, we compute the combined image region covering targets and use this as the final VQA input. However, this approach may fail when the target objects
19

---

are far apart, as the resulting combined region can become excessively large and include irrelevant background. To address this, a fallback strategy taken. If the Euclidean distance between any pair of target objects exceeds a threshold (oobj_ disi (e.g., 1200 pixels), we instead resize each individual ROI to a lower resolution and paste them onto an empty canvas based on their relative positions. This avoids including unnecessary background while maintaining spatial relationships. The resulting composite image is then used for the final VQA prediction. For type-2 questions, we first select all ROIs whose confidence scores exceed a predefined threshold. Overlapping regions are then merged. The merged ROIs are subsequently placed on a canvas using the pasting strategy described above.
In the case of global-loallview MLLMs, this workaround is unnecessary, as these models natively support multi-image reasoning without requiring canvas composition. Fine-grained details in local regions can be preserved by supplying them as separate image inputs. We leverage the model's s text-image interleaved capabilities [2 22 23 by providing a global view with all ROIs visually highlighted, alongside the individual high-confidence local crops.
Proposal ranking of ROIS for fine-grained VQA Further, we present pseudocode in Algorithm that outlines the creation and ranking of ROIS to identify the most relevant ROI for the final inference,
as described in Sec. 3.3
Algorithm Pseudocode for proposal ranking of ROIs for the-ggained VQA
Input: Object relevance map A, number of anchor points k. number of steps natpps ROI parameters:
Smin. Smax, Sdist; threshold for NMS: NMSthreshodd; model MLLM
1: anchor_points +- B k-)
2: initial_rois <- Gennerate-Syymmeerrrr--O0LOooooo
8min)
3: expanded_ rois <- Expandd-ROIssmitiilroosss A, 8max: Sdist)
4: filtered rris + 4pplyy-WSSeexpanded rois, NMSthreshold)
5: ranked rois - -e
rris, Nsstep))
6: answer <- FinnllIInferenceMLLLL. ranked_roin rois)
Output: MLLM answer based on top-ranked ROI
B Dataset statisties
This section provides a summary of the datasets used in our experiments. For each dataset, we report the average image resolution, the number of images, and the number of question-answer pairs Tab. 8
Table 8: Statistics of the employed VQA datasets
Attribute
Benchh
HRBench-4K
HRBench-SS 8K
MME-Real
Avg. width
2,246
4,024
7,431
2,836
Avg. height
1,582
3,503
5,358
1, 1,566
Images [+]]
191
200
200
1,543
QA-pairs [#]
191
800
800
1,919
V*Bench 38 comprises images with an average resolution 2K, specifically 2,256 1,582 pixels. The dataset includes two types of tasks: direct altribute and spatial relation. The direct attribute task involves identifying visual properties of a single object (c.g., color), making them single-target tasks. In contrast, the spatial relation task requires predicting the spatial relationship between two objects, thus constituting multiple-target tasks. In total, the dataset contains 191 images, each paired with a single question, resulting in 191 crestion-answed (QA) pairs.
HRBench [7] comprises two sub-datasets with average image resolutions of 4K and 8K, respectively. Both sub-datasets include two task types: FSP (Fine- grained Single- inngll-instanceeee Perception) and FCP (Fine-grained Cross-instance Perception). FSP questions are single-target and focus on identifying fine-grained attributes of individual objects, whereas FCP questions are multiple-target and involve reasoning about spatial relationships between target objects. HRBench-SK contains full-resolution images with an average size of 7,431 x 5,358 pixels, while HRBench-4K provides cropped versions
20

---

of these images, on average with 4,024 x 3,503 pixels. Each dataset contains one question per image. To improve evaluation robustness, HRBench permutes the positions of the answer options, yielding total of 800 question-answer pairs across 200 images.
Real-dealWordd-Litee 4 is a real-world, fine-grained VQA dataset with average image resolu- tion of 2,836 x 1,566 pixels. It includes a diverse set of subtasks designed to evaluate the perception and reasoning capabilities of MILMs across various domains, such as Autonomous Driving and OCR. The dataset is divided into two subsets: Perception and Reasoning. Each subset contains questions from multiple domains, including OCR (Optical Character Recognition), RS (Remote Sensing), DT (Diagram and Table). MO (Monitoring), and AD (Autonomous Driving), resulting in a total of nine tasks, as shown in Tab. It comprises 1,543 images, with some images associated with multiple questions, leading to a total of 1,919 questions. The number of QA pairs per task is detailed in Tab.
Table 9: Number of question-answer (QA) pairs in MME-Realworkee
per domain
Perception
Reasoning
OCR
RS DT
MO
AD
OCR
DT
MO
AD
QA-pairs [#]
250
150
100
319
350
100
100
150
400
C
Additional results
We include full numerical results in App. C.I results on open-ended VQA in App. C.2 and an analysis of the hyperparameter influence FOCUS in App. C.3 Further, provide additional efficiency metrics in inpppp. C.4jand comparison of different inference schemes as well as performance discrepancies between reported and reproduced results in App. C.5
C.1 Full results
We present the full results of ZoomEye [3I, ViCrop 42] DC 37,, SEAL 38] and our method
FOCUS on V*Bench, HRBench-4K, and HRBench-8K in Tab. 10 For MME-Rea World--r (43 we compare ZoomEye, the vanilla baseline, and FOCUS, see Tab. 11]
As described in Sec. 4.1 reasoning capabilities of MILMM under higher computational budgets, we allow overrun mode: the logit Yye. is lower than IN.-.e.. the MI.LM responds "No" to the existence prompt for all of the top-Nsteps ROIs- the model continues evaluating additional ROIs until it receives "Yes" response. This mechanism improves the efficiency trade-off in many cases. For Rsteps E 21, we report results both with and without the overrun mechanism to provide complete comparison in the low-computation setting.
we run experiments with Vistess E 2, 4,6, 6,8 To better leverage the
In general, accuracy improves with increased computation budget for methods that support config- urable computation budget-ssun such as ZoomEye and FOCUS. ZoomEye exhibits exponential scaling in the number of evaluated regions due its hierarchical tree structure, leading to significantly higher number of Forward Passes (FPs) when targeting high accuracy. In contrast, FOCUS constructs an object-aware relevance map and directly retrieves the most relevant regions, resulting in linear scaling with respect to the number of FPs.
C.2 Open-ended VQA
While the primary results in the main paper are concerned with multiple-choice VQA, here we focus on the evaluation on open-ended VQA. As we are not aware of fine-grained datasets focusing on open-ended VQA, we reuse V*Bench for this task. While it follows the multiple-choice format, it also provides the ground-truth answer natural language format, e.g., "The color of the dog is white." ". To explore the open-ended VQA capabilities of FOCUS with LLa we provide it with VQA questions without answer options, e.g., "What is the color of the dog?". Then, we compare the responses with the ground-truth sequence using an LLM--as--juddee framework [44) leveraging Qwen-2.5-7B (. Moreover, we manually review Qwen-2.5's judgments and correct any misclassifications. The results of this analysis for LLa LLa VA-1.5 clearly show that FOCUS substantially
21

---

Table 10: Full results of different models on fine-grained VQA benchmarks. V*Bench comprises two tasks, namely direct attribute (Attr) and spatial relationship (Spatial) Similarly, HRBench consists of two tasks, ie.. aene-graieed Single-instance Perception (FSP) and Fine-grained Cross-instance Perception (FCP). The highest accuracy for each method-model combination is highlighted in bold. As DC does not provide the complete evaluation code, we report the accuracy from the original paper and estimate the number of FPs following the procedure described in App. A.4
BeBehc
HRBench-4K
HRBench-8K
Model
Overall Acc. f (Attr I Spatial) [%]
Overall Acc. 1 (FSP (FP)FF))]S)
1
Overall Acc. 1 (FSP [CP))
EP
ZoomEye
L.5VA-1.5-7B
50.26
36,25
Depth-1 (2 crops)
(41.74
12.50
| (33225)
11.54
(32.25 135.5)
11:55
50,78
39.13
32.88
Depth-1 (4 crops)
(41.74 I 64.47)
20.37
(44.75
17.46
(32.75 33.00)
18. 18
71.20
47.75
42.13
Depth-2 (4 crope)
(67.83 76.32)
44.54
(57.25 (3.255)
35,60
(49. 199.001 36,25)
39.15
77.48
50,00
49.00
D5pth-5 (4 crops)
(80.87 (72.37)
48.63
(63.25 (36.75)
49.38
(61.75 36.25)
59.64
LLVAA-V-77
75.92
61.38
59.00
Depth-1 (2 crops)
(77.39 173.688
10.52
(70.00 (52775)
(66.50 (1.50)
9.02
81.15
65.75
59.63
Depth-1 (4 crops)
(81.74 I 80.26)
13.34
(80.75 50075)
11:43
(59.00
12.59
90.05
66.13
65.63
2 (4 crops)
(93.04 185,53)
21.08
(80.25 (,2.00)
20.42
(811.75 (49.5)
22.31
Depth-5 (4 crops)
91.10
23.98
69.38
29.19
69.00
36.75
(93.91 I 86.84)
(84.5154.25
(86.75 (51.25)
ViCrop
T.VA--L5-7B
59.16
42,50
39.38
rel-att-bigh
(58.26 60.53)
12.26
(51.50 33.50)
30.59
(48.00 (3..5)
78.60
54.97
44.25
38.38
gred-att-bigh
1 17.990
6.63
(53.75 (34775)
15,80
(44.25 32.500
39.80
DC
1.55A.x1.5.77
57.00
18.18
42.30
48.55
39.50
77.02
SEAL
73.68
25.53
34.50
18.05
33.50
16.96
FOCUS (Ours)
LLVVA-1.5-B
51,30
41.13
40.63
Steps-1 (no-overrun)
(46.95157.88
1.47
(49.75 13.55)
3.14
(46.00 13.255)
3.14
57.07
46.5
40.63
Steps-2 (no-overrun)
(
4.25
(56.00 (37000)
5.41
(44.75 36.50)
5.41
64.40
42,25
42.38
1 (overrun)
(63.48 65.79)
4,86
(51.501 (3,000
4.99
(48.50 36.25)
5.08
66.49
45.88
42,15
Steps-2 (overrun)
(66.09 (1)
5.70
(55.75 (6,00)
5.95
(46.75 37,50)
6.04
67.01
47.13
44.13
S (overrun)
(66.09 I 68.42)
(56.50 (37.55)
(48.00 40.25)
9.07
68.06
49.25
45.00
Steps-4 (overrun)
(66.96 69.74)
8.27
(59.75 38.75)
10.14
(50.25 39.75)
10.10
70.68
50.63
45.00
Steps-6 6 (overrun)
770.43171.00
10.71
(62.25 (39.00)
12.31
(52.00 38.00)
12:23
72.77
51.75
44.13
Steps-8 (overrun)
(72.17173.68)
13.28
(64.00 193.500)
14.49
(52.25 1 36.00)
14.4]
LLAVA-OV-78
83.24
69.00
65.75
Steps-1 (no-overrun)
(87882 76.31)
1.47
(82.25 (55775)
3.84
(77.00 (4,50)
3.86
89.01
70.38
66.88
Steps-2 (no-everrun)
(92.17 84,21)
4.23
1.5.00))6.755)
6.07
(78.00 (5.755)
6.09
90.57
69.88
68.75
Steps-1 (overrun)
1
4.05
(85.75 55.00)
6.55
(81.0 (5.50)
7.35
91.62
70.25
67.63
Steps-2
(overrun)
(93.93 18.15)
5.16
(86.50) 54.00)
7.41
(79.00 29.0011 156.55)
8.06
91.62
70.00
66.88
Stapa-3 (overrun)
(93.91 (88.11)
6.37
(85.75 15.255
8.34
178.001550 155.00)55.75)
8.93
Steps-4 (overrun)
(93.91 1 19.47)
92.15
7.63
(85.75 55.75)
70.75
9.32
(80.75 156.00)
68.38
9.86
Steps-6 (overrun)
(93.91 (89.47)
92.15
10.22
(85.75 55,50)
70.63
11.33
155.25))
11.81
91.62
71.13
69.63
Staps-8 (ovarrun)
(93.04 189447)
12.84
(86.75 (55,50)
13.41
(83.50 55.75)
13.83
22

---

Table 11: Detailed results on the AMME-RealWorld-Liee dataset. Accuracy is reported both per subtask and on average subset. The highest accuracy for each subtask is highlighted in bold.
Task
LLVVA-V-7B
Vanilla
ZoomEye
FOCUS (Ours)
Acc. [96]
Acc. [%]]
Acc. [%]]
Moti -uulti-padtttrii
22.00
28.00
34.00
Moti Mamiti vehicles
46,00
38.00
40.00
Moti opddestrian
24.00
46.00
44.00
AD
Moti Motionyyiicie
24.00
54.00
34.00
Objecteomm:
36.00
42.00
40.00
Objectidmtity
78.00
74.00
70.00
Visualtraffticnnggm
62.00
78.00
68.00
Diagram
70.00
80.00
60.00
DT
Table
60.00
68.00
56.00
Personeolor
32.00
40.00
56.00
PersON:oomting
32.00
38.00
38.00
Perception
PersoNgrimmatton
10.53
15.79
10.53
MO
Vehiclepolor
46.00
60.00
52.00
Vehicle @pointing
Vehiclepumting
56.00
56.00
58.00
Vehicle 1lcattion Vehiclelacationn
38.00
28.00
28.00
Vehicle.rimttnn
12.00
20.00
26.00
Advert & product
82.00
82.00
88.00
Book map poster
78.00
70.00
74.00
OCR
License
88.00
86.00
88.00
Phone address
82.00
96.00
90.00
Text recognition Color Count
78.00
72.00
80.00
60.00
64.00
64.00
RS
34.00
40.00
20.00
Position
62.00
50.00
56.00
Average
52.01
56.29
54.15
Attentioneaafficaat
74.00
72.00
74.00
Intentionge
26.00
24.00
22.00
Intentio Intentionppdernt Oppdesrrian
48.00
50.00
54.00
Intention.aticla
26.00
42.00
40.00
AD
20.00
28.00
22.00
Interactioneg: interacttionng---rrrtft 2-troaooo--ttt--eeeet eighal
28.00
30,00
22.00
Reasoning
Interactiongp-2-waa
26.00
22.00
26.00
Intoractiofaataen222a00
10.00
12.00
8,00
Diagram
40.00
54.00
58.00
DT
Table
40.00
44.00
46.00
Calculate
42.00
46.00
50.00
MO
Intention
26.00
26.00
42.00
Property
64.00
66.00
62.00
Character identification
72.00
64,00
74,00
OCR
Scene understanding
72.00
64.00
68.00
Average
40.93
43.20
44.53
23

---

improves the fine- gree-graind ended VQA performance, increasing accuracy from 44.50% for the vanilla LLaVA-1.5 model 65.97%.
C.3 Analysis of hyperparameter influence
We further investigate how variations in the hyperparameters of FOCUS affect its performance. This analysis is conducted using FOCUS with LLaVA-1.5 and LLaVA-OV on V*Bench, focusing on five key hyperparameters, namely the number of anchor points (k), the ROI expansion threshold, the maximum ROI size (Smax), the minimal Euclidean distance between anchor points ($aist and the NMS threshold. As shown in Tab. 12] using alternative hyperparameter settings reduces accuracy by at most 4.7 pp. compared to the original configuration, demonstrating our method's ' low sensitivity to hyperparameters. During the sensitivity analysis, discover some alternative hyperparameter configurations that achieve even higher accuracy than our default settings.
Table 12: Hyperparameter analysis of FOCUS. We assess the impact of five hyperparameters on the performance of FOCUS. indicates the original hyperparameter.
(a) Variation of numbers of anchor points k.
(d) Variation of minimum distance between ROI
anchor points &dist
Model
k
Accuracy [%]]
30
72.77
Model
Sdist
Accuracy [%]
LLaVA-1.5
4(10,50)
72.77 1.55
2
72.77
LLaVA-1.5
3
72.25
30
92.15
ILVVA-OV
4
69.11
50)
92.03 + 1.37
2
92.15
3
91.62
LLaVA-OV
(b) Variation of ROI expansion threshold.
92.15
5
91.62
ROI expans. threshold
Model
Accuracy [%]
Variation of NMS threshold.
0.5
72.77
LLaVA-1.5
(.0.3,0.7)
72.77 0.00
NMS threshold
Model
Accuracy [%]
0.5
92.15
LLaVA-OV
0.0.3,0.7)
92.70 + 0.25
0.3f
72.77
0.1
70.16
LLaVA-1.5
0.5
72.25
(c) Variation of maximum ROI size Smas.
0.7
72.25
Model
Smax
Accuracy [%]
0.3
92.15
0.1
92.15
5
72.77
LLaVA-OV
0.5
92.15
LLaVA-1.5
69.63
0.7
92.15
68.06
gi
92.15
94.24
LLaVA-OV
91.01
11
89.53
For and the ROI expansion threshold, we randomly sample 50 values from 24(10, 50) and 20..3, 0.7), respectively, where U indicates a uniform distribution. Across both LLaVA-1.5 and LLaVA-OV we observe only minor performance variations when adjusting k: an accuracy of 72.77 + 1.55 for LLaVA-1.5 and 92.03 1.37 for LLaVA,OV, as shown Tab. 12a For the ROI expansion threshold, we observe an even smaller impact on the performance of FOCUS, with an accuracy of 72.77 + 0.00 for LLaVA-1.5 and 92.70 0.25 for LLaVA-OV, as shown Tab. 126 For Sdists Smax, and the NMS threshold, we vary the values in discrete steps and analyze their influence on performance. Across both V.L. and LLaVA-OV, we find that 8max has the largest impact on FOCUS among all analyzed hyperparameters, resulting in an accuracy drop of 4.71 pp. for -A- and 2.62 pp. for LLaVA-OV, as shown Tab. 12c Interestingly, for FOCUS with VA-VV, we observe accuracy improvement of 2.09 pp. over the baseline when setting Smax 5. For Sdist,

---

observe maximum accuracy degradation of 3.66 pp. for LLaVA-1.5 and 0.53 pp. for LLaVA-OV, as shown in Tab. 12d For the NMS threshold, we find only minor impact, with a maximum accuracy degradation of 2.61 pp. for LLaVA-1.5 and no observable influence for LLaVA-OV, as shown in Tab. 12e] This is likely because FOCUS with VA- generates higher-resolution object relevance maps, reducing its reliance on NMS.
C.4 Additional efficiency metrics
We report additional efficiency metrics-includdin average execution time and peak GPU memory usage per sample- on V*Benc. We compare SEAL with a customized LLaVA-7B, vanilla LLaVA- 1.5-7B, vanilla Viaii-Oneeissionn7,, and three training-fres ffre methods (ViCrop, ZoomEye, and FOCUS) using LLa 1.5 models.
As shown in Tab. 13 among the waining-free methods, ZoomEye achieves the highest accuracy but suffers from poor efficiency due to complex confidence prediction mechanism, which involves multiple question prompts and hierarchical tree structure. This is reflected by its high number of Forward Passes (FPs) and long execution times. FOCUS, by contrast, leverages an object relevance map built from cached token similarities to directly identify relevant image regions. As a result, it requires only 25% of ZoomEye's FPs and execution time to reach comparable accuracy. ViCrop is slightly more efficient than FOCUS in terms of execution time and FPs, but it achieves lower accuracy and incurs the highest peak memory usage due to its incompatibility with efficient attention mechanisms. SEAL differs architecturally from LLa 1.5 and LLa Make-nneian Vision. Its dual-MLLM design makes it significantly slower and more memory -intenive than most methods in the comparison. In general, lower number of FPs is associated with reduced execution time.
Table 13: Additional performance and efficiency metries on Bench,. performing method is highlighted in bold and the runner-up is underlined.
last three rows, best-
Model
Acc. [%]4
FP [+]
Avg. execution time [s]]
Avg. peak memory [GB]
SEAL
73.68
25.53
9.16
27.34
LLaVA-OV-7B
74.46
1.30
19.64
LLaVA-1.5-7B
47.64
0.25
13.57
+ w/ ViCrop
59.16
12,26
1,36
19.93
+ w/ ZoomEye
77.49
48.63
11.26
14.24
+ w/ Ours
72.77
13.28
2.19
14.91
Additionally, we compare the efficiency of FOCUS and ZoomEye with LLaVA-115 across multiple configurations (see Tab. 14). We evaluate them on V*Bench in terms of accuracy, average inference time, average FPs and average FLOPss The latter ones are estimated based on a calculation scheme applied in prior MLLM work 7 39]. We ran the evaluations the same hardware to ensure result comparability.
Notably, the lowest-complexin configuration of ZoomEye exhibits a higher inference time and nearly identical FPs and FLOPs compared the gighest-complexiis configuration FDCUS. Despite this, FOCUS outperforms ZoomEye by 22.51 pp. in accuracy under this configuration. Moreover, ZoomEye shows a steep increase in complexity pageexxttymmasureeee by inference time, FPs, and FLOPs as the search depth in its tree structure increases. Its maximum configuration achieves an accuracy of 77.48%, with an inference time of 11.96 seconds, 48.63 FPs, and a computational cost of 217 TFLOPs
In summary, the efficiency gains reported in terms of FPs are consistently reflected in inference time and FLOPs. FOCUS achieves competitive or superior accuracy with significantly lower inference time and fewer FLOPs compared to ZoomEye. For instance, at Steps-6 (overrun), FOCUS reaches 70.68% accuracy in just 2.00 seconds and 51.26 TFLOPs, whereas ZoomEye (Depth-2) requires nearly 4x more FLOPs and 5x longer inference time to reach comparable accuracy. These results underscore the efficiency of FOCUS for fine-graine fine-grained visual reasoning tasks in practice.
C.5 Inference scheme and performance discrepancy
We describe the following the comparison between different inference schemes and the discrepancy between reported and reproduced performance.
25

---

Table 14: In-depth efficiency comparison of FOCUS and ZoomEye. Across different configurations,
compare efficiency in terms of accuracy, average inference time, average FPs and average FLOOPs V*Bench.
Model
Accuracy [%]
Inference time [8]
FP [#]
TFLOPs [#]
ZoomEye
LLaVA-1.5-7B
Depth-1 (2 crops)
50.26
3.78
12.50
55.95
Depth-1 (4 crops)
50.78
4.76
20.37
91.03
Depth-2 (4 crops)
71.20
9.73
44.54
199.21
Depth-5 (4 crops)
77.48
11.96
48.63
217.00
FOCUS
LlVVA-1.5-BB
Steps-1 (no overrun)
51.30
0.99
1.47
10.98
Steps-2 (no overrun)
57.07
1.28
4.25
23.11
Steps-1 (overrum)
64.40
1,36
4.86
25.73
Steps-2 (overrun)
66.49
1.44
5.70
29.43
Steps-3 (overrun)
67.01
1.55
6.79
34.15
Steps-4 (overrun)
68.06
1.73
8.27
40.61
Steps-6 (overrun)
70.68
2.00
10.71
51.26
Steps-8 (overrun)
72.77
2.27
13.28
62.46
Table 15: Comparison of different inference schemes models. Each result includes accuracy, execution time, and number forward passes for visual search (FP). The highest accuracy a column highlighted in bold.
Vanilla
ZoomEye
FOCUS (Ours)
Exec. time[s]
Exec. time|s]
Inference Scheme
Acc. [96]
Acc. [%]]
FP
[+]
Acc. [%]]
FP
[#]
Logits matching
48.16
77.49 72.25
11.26
48.63
74.35
2.76
13.28
Open-ended generation
47.64
9.26
36.94
72.77
2.19
13.28
Inference scheme: logits matching open-ended generation Multiple-choice VQA requires selecting one of several fixed answer options. A common approach is open-ended generation [5].), where the prompt includes the question and options (e.g.. "(A) Red"), and the MLLM generates the corresponding option letter. In contrast, SEAL [38] and ZoomEye [1] adopt an alternative scheme called logits matching on V*Bench (38). In this method, the model is prompted multiple times, once for each answer option. Specifically, each answer option is reformulated as a sentence (e.g.. " (A) the original question.
is red.") which is then appended The model is prompted with these reformulated question-option pairs and the image, and the answer option yielding the highest logit score for its target tokens selected as the final prediction.
Red" "The color of the
We noticed that ZoomEye utilizes logits matching on V*Bench but uses open-ended generation on the other three datasets, prompting us investigate the impact of different inference schemes. We evaluate LLaVA-1.5 on V*Bench using both open-ended generation and logits matching across three methods: the vanilla baseline, ZoomEye, and FOCUS (ours). SEAL is excluded from this comparison, as it is not a training-free method. All other hyperparameters are kept constant.
As shown in Tab. 15 both the vanilla model and FOCUS achieve 0.5 pp. and 1.6 pp. higher accuracy. respectively, when using logits matching. This improvement likely stems from the fact that logits matching climinates the need for strong instruction-fol followiin. models longer need to explicitly generate the option letter, but instead compare the semantic content of full answer statements. This makes the inference process more robust, particularly for models with weaker generative alignment.
In contrast, ZoomEye's accuracy drops by over pp. and its execution time decreases by approx- imately 2 seconds when switching to open-ended generation. Given that FOCUS shows a 1.6 pp. accuracy difference and a 1e57-d reduction execution time under the same scheme change, we attribute that portion of ZoomEyye's decline to the scheme itself. The remaining gap-bodt both in accuracy and runtime- can likely be attributed to suboptimal tuning the open-ended setting, as all hyperparameters were held constant. The changed inference mode likely alters the model confidence.
26

---

Table 16: Reported reproduced accuracy across fine-grained VQA datasets, A dash indicates that no evaluation that henchmark was performed the original work.
V*Bench
HRBench-4K
HRBench-8K
Reported Acc. [%)]
Reprod. Acc. [%]]
Reported
Reprod.
Reported
Reprod. Acc. [%]]
Model
Acc. [%]]
Acc. [%]]
Acc. [%
ZoomEye
LLaVA-1.5-7B
83.25 90.58
77.48 91.10
53.25
50.00 69.38
50.75
49.00
LLaVA-OV-7B
69.63
69.25
69.00
ViCrop
LLaVA-1.5-7B
rel-att-high
62.30
59.16
42.50
39.38
grad-att-high
57.07
54.97
44.25
38.38
SEAL
75.39
73.68
34.50
33.50
which may lead to premature termination of tree search (indicated by the lower number of FPs) and increased prediction instability.
Despite the potential accuracy gains of logits matching, we adopt open ende-e generation for FOCUS across all datasets. This is done for two reasons. First, the effectiveness of logits matching depends heavily on the quality of option reformulations, which are not always available consistent across datasets. This limits its generalizability and makes cross- datsset comparisons less reliable. Second, logits matching requires one Forward Pass (FP) per answer option, compared to only single FP in open-ended generation. Although some acceleration of the logits matching scheme implemented by caching question tokens, it still increases inference time (see Tab. 15). Therefore, we report FOCUS results using open-ended generation their original inference schemes. Their respectively reported V*Bench performance is based on logits matching.
main paper. For SEAL and ZoomEye, we preserve
Discrepancy between reported and reproduced performance For the baseline methods, i.c., SEAL 38,, ViCrop 42 and ZoomEye (31), we use the official implementations provided in their respective repositories. We strictly follow the original configurations, including software environments and data structures, as specified by each work. As shown in Tab. 16) we observe some discrepancies between the reported and reproduced performance, most notably for LLaVA-1.5 with ZoomEye. We hypothesized this may be linked to the use of different efficient attention backends (e.g., FlastAAteenionn22 [10] vs. PyTorch's ' SDPA However, the observed deviation with different attention implementations is small, less than 1% on V*Bench- thus, further root causes of this deviation seem to exist, that however remain unclear. To ensure fair and consistent comparisons under a unified evaluation setup, we always report the reproduced results for these benchmark methods in the main paper.
D Further qualitative examples
This section provides additional qualitative examples that highlight both the strengths and limitations of FOCUS when applied to L.L (see (seeFFig. 9) and LLLVVA-Oee Vision
Vision (see Fig. 10.. For improved clarity in visualization, we use a reduced A = 10, differing from the values specified in App. A.3 Further, provide qualitative examples for failure cases of FOCUS with LLaVA-1.5 on high- resolution images in Fig. to highlight the limitations of low-resolution object relevance maps (see Sec. 4.6p. In these examples, use an increased & 50, enabling FOCUS cover larger portion of the image space; nevertheless, FOCUS still fails to identify the relevant image region.
Fig. 9 (I) showcases type-1 single-targer task with FOCUS and LLaVA-1.5. By leveraging the MLLM's internal representations, FOCUS identifies a relevant crop that highlights the color of small candles, correcting the model's initial VQA response. The ROI ranking mechanism demonstrates robustness to noise in the object relevance map by assigning the highest confidence to the originally
"See http:://sagggcccaahhcccchccllltt tutorial.html
27

---

nourth-ranked region. Moving to a type-1 multiple-traee target task, Fig. 9 (II) illustrates how FOCUS identifies both the person the red jacket and the large tree using in-context learning. The object relevance maps clearly localize both targets. Since L.L.V--1.5 accepts only single-image inputs, the selected ROIs are stitched together-a strategy detailed in App. A.5 avoid excessive image size or irrelevant content. Despite this, the model fails to answer correctly, likely due to limited spatial reasoning, A type-2 counting task is depicted in Fig. (II). Here, FOCUS successfully locates both chairs in the image. As in the previous example, the regions are combined to form a single input for LL. This enables the model to correctly answer the VQA query, which it could not do using the global view. Fig. 9 (TV). finally, presents another failure case with LLaVA- underscoring the limitation discussed in Sec. 4.6) The example, drawn from the HRBench -&K dataset (see Tab. 8). involves a hiso-resolttion image where the sign is too small to be detected via the low-resolution object relevance map. Consequently, FOCUS selects an incorrect region and fails to improve the VQA result.
Turning to Visiinn 10 (I) features a type-1 single-target task. While vanilla LLaVA- Vneion fails to answer the question about the speed limit sign, FOCUS successfully identifies the relevant region using internal representations. By isolating this region (see selected ROI), the model is able to generate the correct VQA response. Fig. 10 explores type-1 multiple-target task. Despite the relatively large size of the relevant regions, vanilla LLaVA-One Vision does not answer correctly. FOCUS identifies the appropriate areas, generates combined image region, and creates one local crop per relevant object. The relevant regions are highlighted in the image of the combined ROIs with rectangles, helping reduce background noise and enabling the model to answer the VQA task correctly. Next, a type-2 counting task is shown in Fig. 9 (III) with L.La VA-One Vanilla VA-VA-One Vision fails count the number of computers accurately. FOCUS identifies five computers in total, assisting the model in producing the correct answer. However, only detects four morsectly-mssingg one and mistakenly counting another one twice. Finally, Fig. (IV) illustrates a failure case with LLaVA-One Vision. Despite access to a high-resolution object relevance map, FOCUS fails detect the region associated with the umbrella. As a result, it does provide the necessary input for the MLLM to answer the VQA example correctly.
28

---

What's the color of the car? (A) White (R) Pink (C) Yellow (D) Blue Label: A Answer (I.L.V w/ FOCUS):
Question:
Object relevance map car SSelected: R4
Original image
GT region
Selected ROI
Ra
Ris
Feres
RER
Bag
Rg
R
Question:
What's 's the color of the flag? (A) White (B) Yellow (C) Black (D) Red
Label: A Answer (LLaVA-115 w/ FOCUS): D x
Object relevance
Original image
region
Selected
flag Selected: R4
Reg
Rad
Rag
Re
Figure 8: Further failure cases of FOCUS with LLaVA-1.5. We provide examples for failure cases of FOCUS using LL. corresponding to the resolution limitation mentioned in Sec. 4.6) Note that we manually highlight the relevant regions the original image to facilitate casier localization the ground truth area for the reader and these annotations are not included in the input for the MIL.M.
29

---

Question: color the candles? (A) (B) yellow (C) gray (D) white
Label: Answer LLV A-1.5); | Answer (LLaV A-1.5 w POCUS): v
Object relevance map
Original Image
GT region
Selected ROI
candles Selected: R₃
Question: What relative position of the person the red jacket compared large tree? Behind the large tree (B) Right the large tree (C) In front the large (D) Left of the large tree
Label: Answer (LLaV A-1.): 5): x Answer we (LLaV A-1.5 w FOCUS):
*
Object relevance map person in the red jacket Seeected: R
Object relevance map
Original image
Combined region
large tree |eelecte:: Rg
|II
Question: How many chairs there in the image? (A) One (B) Four (C) Two (D) Three
L Label: C Answer (LLaVA- A-55): Answer (LLaVA-13. A.5 w/ FOCUS):
Original image
Combined region
O relevance map chairs Selected: Rg R₃
Q Question: W hat the color the sign in the image? (A) Green and white (B) Y Vellow and white (C)
(IV
Y elloo and green (D) W hite and red
L Label: B Answer (LLaVA-) A-5.): *
Answer L-1 w/ FOCUS): C
relevance map sign Selected: R
0 riginal image
GT region
Selected ROI
Figure 9: Further qualitative examples of FOCUS with LLaVA-1.5. We provide examples for single-object (I), multi-object (II), type-2 question (III), and failure (IV). Note that do adjust the aspect ratio of the images for L.a.V--155. Therefore, there some padding areas the object relevance maps. Additionally, we manually highlight relevant regions the original image facilitate casier localization of ground truth area for the reader and these annotations are not included in the input for the MI..M.
30

---

Question: W hat the speed limit the sign the image? (A) 20 (B) 40 (C) 60 (D) 30 L Lbbe:: D Answer (LLaV A--neVisionk x Answer (LLaV AaeAA--Oniiiii ision FOCUS: D
0 Object relevance map speed limit on the sign Selected: Rg
Original image
GT region
Selected ROI
P
Question: W is the position of the totem pole in relation to the bear statue? (A) To the (B) To the right (C) Behind the bear statue (D) In front
Label: Answer (LLaV A-OneV ision): Answer (LLaV A-OneVissoon
FOCUS): A
Objectelevvann relevance map totem pole Selected: R
Object relevance map bear statue Selected:
Original image
Combined region
R
Bi
Question: How many computers are on the table? (A) Three (B) Five (C) Two (D) Four L Label: B Answer (LLaVA A OneVision): C x Answer (LLaVA A---ne-iiiionn w/ FOCUS B Object relevance map computers Selected: R, R, & R2 & R, & R, & Rg
Original image
Combined region
Question: is the color of the umbrella? (A) blue black (D) purple Label: Answer (LLaVA OnneVision): x Answer (LLaVA A-OnVVion ision FOCUS): C x
Objett map umbrella Selected: R,
O Original Image
GT region
Selected ROI
Figure Further qualitative examples FOCUS with LLaVA- -ne One Vision. provide examples single- object (I), multi-ohject (II), type-2 question (III), and failure case (IV). Note that we manually highlight the relevant regions in the original image to facilitate easier localization of the ground truth area for the reader and these annotations are included in the input for the MLLM.
31

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The paper substantiates its main claims by demonstrating that internal repre- sentations of MILMMs can be effectively leveraged to guide visual cropping, enabling both efficient and high-performing inference on fine-grained VQA tasks. SnfS. and 2] we identify key limitations of existing visual cropping approaches. In See. 3J we introduce our method, which uses cosine similarity between cached text and image token representations to construct an object relevance map and propose relevant regions. Finally, in Snsc... 4| we present a large-scale empirical evaluation across four datasets and three types of MLLMs, demonstrating substantial improvements in computational efficiency and competitive or superior accuracy compared to prior methods.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations of our work in a dedicated paragraph in Sec. 4.6
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
32

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: Our work does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
we fully describe our efficient visual cropping method FOCUS. We list experimental setups in Sec. 4.1 and
Justification: In Sec. 3
App. A.3
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
(b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
33

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: All datasets and MLLMs are publicly available. We are waiting for internal clearance to release our code upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
public/ poolic/guides// poooeesubmsssoonool11.
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https wttp:/ic/ggii////eeeeeeeeeeeeeet htte:/gguuddee/ooCSuubbiiiii www.oodeuuiiiisonnnee..
/ //nips.
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We list all experimental setups in Sec. 4.1 and App. A.3
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: In the MLLM domain, many evaluation especially those using deter- low variability. our case, experiments are conducted using pre-trained MILMSs in evaluation mode without stochastic components such as sampling or RAG. As such, our method produces consistent results across runs, and there is no randomness that would necessitate reporting error bars confidence intervals.
ministie decoding-
34

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We report the utilized computation hardware, including GPU hours for the experiments App. A.I
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported didn't ' make it into the paper).
paper (e.g., preliminary or failed experiments that
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httpge////orrppp..c////ll///eaas
Answer: [Yes]
Justification: We do not conduct any research involving human subjects or participants. Further, we follow all aspects listed in the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
35

---

Answer: [Yes]
Justification: Our main contribution is improving the efficiency of MLLMs in perceiving and reasoning over fine visual details. This directly reduces energy and resource utilization. We hope that our efficient visual cropping method will contribute to positive environmental impact, as discussed in Sec.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Our work does not pose significant risks of misuse. We rely exclusively on publicly available pre-trained MLLMs and standard benchmark datasets, and we do perform any task-specific fine-tuning or release new models or data. Thus, there is no need for additional safeguards.
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
36

---

Justification: All external assets used in this work work-including, datasets, models, and code are properly cited with references to their original publications. We have ensured that all assets are used in accordance with their respective licenses and terms of use. We relied solely on publicly available resources that permit academic usage, and no proprietary or restricted components were used.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We still waiting for clearance publicly release code (see above) which we documented well. All of our code written by the authors of this paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsourcing and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
37

---

15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our work does not involve research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: Our work uses the internal representations of freely available pre-trained models and uses the MLLMs for VQA tasks.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy hutt::///oorrgsscc//ooofrrrrre// for what should or should not be described.
38


Reviews:
[{"review_id": "HD7VGfj851", "reviewer": "Reviewer_LaD3", "summary": "This paper presents FOCUS, a training-free method for fine-grained visual question answering (VQA) that uses the internal token features of Multimodal Large Language Models (MLLMs) to find the most relevant parts of an image. Different from other methods, instead of exhaustive search or extra training, FOCUS quickly locates the target object by measuring cosine similarity between cached value tokens of the target word and image areas. It then selects, checks, and ranks possible regions, so the final VQA is only done on the most likely area.", "questions": "1. The paper uses several hyperparameters, like the minimum distance between ROI anchors and the ROI size threshold. Did you test how sensitive FOCUS is to these choices? Is there an ablation on these hyperparameters?\n2. In the Limitations section, you mention that FOCUS struggles with very small objects in high-res images. Can you show more failure cases or examples to help understand this problem?", "limitations": "Can FOCUS handle questions that involve spatial relationships, such as \"the red ball on the left,\" where token similarity maps may not uniquely identify the correct region?", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The method is entirely training-free and operates only at inference time, allowing it to generalize well across models and datasets without any retraining or task-specific adaptation.\n2. FOCUS achieves high computational efficiency, reducing the number of forward passes and inference time compared to previous visual cropping approaches, while maintaining or improving accuracy.\n3. FOCUS  is practical to integrate with a wide range of modern Multimodal Large Language Models.\n\nWeaknesses:\n1. The current experiments focus only on multiple-choice VQA tasks. The method has not been evaluated on generative or open-ended VQA, and it remains unclear how well FOCUS would generalize to these settings.", "was_revised": true, "final_justification": "My questions are answered by the authors. I'll maintain my rating.", "rebuttal": "We sincerely appreciate your detailed assessment of our work and value the thoughtful questions, which provide meaningful direction for improvements of our paper.\n\n### Open-ended VQA\n> \"The current experiments focus only on multiple-choice VQA tasks. The method has not been evaluated on generative or open-ended VQA, and it remains unclear how well FOCUS would generalize to these settings.\"\n\nWe agree with you that this is a valuable consideration, especially when assessing FOCUS for real-world usage. To our knowledge, we are not aware of fine-grained datasets focusing on open-ended VQA. Therefore, we reuse V\\*Bench for open-ended evaluation. While it primarily follows the multiple-choice format, it also provides the ground-truth answer in a natural language format, e.g., `\"The color of the dog is white.\"`. To explore the open-ended VQA capabilities of FOCUS and baseline models, we provide them with VQA questions without answer options, e.g., `\"What is the color of the dog?\"`. Then we compare their responses with the respective ground-truth sequence. We evaluate the responses of the MLLMs using an LLM-as-a-judge framework [R1], leveraging Qwen-2.5-7B. Moreover, we manually review Qwen-2.5’s judgments and correct any misclassifications. The results of this analysis for LLaVA-1.5 clearly show that **FOCUS substantially improves the fine-grained *open-ended* VQA performance**, increasing accuracy from **44.50%** for the vanilla LLaVA-1.5 model to **65.97%**:\n\n| Model                | Accuracy (%) |\n|----------------------|--------------|\n| LLaVA-1.5            | 44.50        |\n| LLaVA-1.5 + FOCUS    | 65.97        |\n\n### Sensitivity to hyperparameters\n\n> \"The paper uses several hyperparameters, like the minimum distance between ROI anchors and the ROI size threshold. Did you test how sensitive FOCUS is to these choices? Is there an ablation on these hyperparameters?\"\n\nWe thank you for pointing out the need for more clarity regarding hyperparameter sensitivity. To address your question, we conduct a systematic sensitivity analysis using both LLaVA-1.5 and LLaVA-OV on V\\*Bench. We examine the impact of key hyper-parameters, including the number of anchor points ($k$), the minimal Euclidean distance between anchor points ($s_\\text{dist}$), the maximum ROI size ($s_\\text{max}$), the ROI expansion threshold, and the NMS threshold. As shown in the tables below, the accuracy obtained using alternative hyperparameter choices shows only **marginal degradation** of maximum 4.7% compared to the accuracy achieved with the original settings, demonstrating our method's **low sensitivity to hyperparameters**. During the sensitivity analysis, we discover some alternative hyperparameter configurations that achieve even higher accuracy than our default settings. In the following, we present the results of our hyperparameter sensitivity analysis in detail. For the number of anchor points ($k$), we uniformly sampled 20 values and report the mean accuracy along with a 95% confidence interval (corresponding to 2 standard deviations).\n\n| Model     | $k$   | Accuracy (%)   | Comment        |\n|-----------|---------------|--------------|----------------|\n| LLaVA-1.5 | 30               | 72.77          | original choice| \n| LLaVA-1.5 | random(10,50)               | 72.77 ± 1.55  ($\\Delta$ = +0.00)        | alternative    | \n| LLaVA-OV  | 30               | 92.15          | original choice| \n| LLaVA-OV  | random(10,50)               | 92.03 ± 1.37   ($\\Delta$ = -0.12)       | alternative    | \n\n\nFor the minimum distance between ROI anchor points $s_{\\text{dist}}$, we varied the value from 2 to 4 for LLaVA-1.5, and from 2 to 5 for LLaVA-OV.\n\n | Model     | $s_\\text{dist}$ | Accuracy (%)              | Comment         |\n|-----------|---------------|-------------------------|-----------------|\n| LLaVA-1.5 | 2               | 72.77                     | original choice |\n| LLaVA-1.5 | 3               | 72.25 ($\\Delta$ = −0.52)  | alternative     |\n| LLaVA-1.5 | 4               | 69.11 ($\\Delta$ = −3.66)  | alternative     |\n| LLaVA-OV  | 2               | 92.15                     | original choice |\n| LLaVA-OV  | 3               | 91.62 ($\\Delta$ = −0.53)  | alternative     |\n| LLaVA-OV  | 4               | 92.15 ($\\Delta$ = +0.00)  | alternative     |\n| LLaVA-OV  | 5               | 91.62 ($\\Delta$ = −0.53)  | alternative     |\n\nFor the maximum ROI size $s_\\text{max}$, we explored alternative values ranging from 5 to 9 for LLaVA-1.5, and from 5 to 11 for LLaVA-OV.\n\n| Model     | $s_\\text{max}$ | Accuracy (%)              | Comment         |\n|-----------|--------------|-------------------------|-----------------|\n| LLaVA-1.5 | 5              | 72.77                     | original choice |\n| LLaVA-1.5 | 7              | 69.63 ($\\Delta$ = −3.14)  | alternative     |\n| LLaVA-1.5 | 9              | 68.06 ($\\Delta$ = −4.71)  | alternative     |\n| LLaVA-OV  | 9              | 92.15                     | original choice |\n| LLaVA-OV  | 5              | 94.24 ($\\Delta$ = +2.09)  | alternative     |\n| LLaVA-OV  | 7              | 91.01 ($\\Delta$ = −1.14)  | alternative     |\n| LLaVA-OV  | 11             | 89.53 ($\\Delta$ = −2.62)  | alternative     |\n\n\nFor the ROI expansion threshold, we uniformly sampled 20 values and reported the mean accuracy along with a 95% confidence interval (corresponding to 2 standard deviations).\n\n| Model     | ROI expansion threshold   | Accuracy (%)   | Comment        |\n|-----------|---------------|--------------|----------------|\n| LLaVA-1.5 | 0.5               | 72.77          | original choice| \n| LLaVA-1.5 | random (0.3, 0.7)               | 72.77 ± 0.00  ($\\Delta$ = +0.00)        | alternative    | \n| LLaVA-OV  | 0.5               | 92.15          | original choice| \n| LLaVA-OV  | random (0.3, 0.7)               | 92.70 ± 0.25 ($\\Delta$ = +0.55)         | alternative    | \n\nMoreover, we report accuracy using NMS thresholds ranging from 0.1 to 0.7 to demonstrate robustness w.r.t. this parameter.\n\n| Model     | NMS threshold   | Accuracy (%)              | Comment         |\n|-----------|---------------|-------------------------|-----------------|\n| LLaVA-1.5 | 0.3             | 72.77                     | original choice |\n| LLaVA-1.5 | 0.1             | 70.16 ($\\Delta$ = −2.61)  | alternative     |\n| LLaVA-1.5 | 0.5             | 72.25 ($\\Delta$ = −0.52)  | alternative     |\n| LLaVA-1.5 | 0.7             | 72.25 ($\\Delta$ = −0.52)  | alternative     |\n| LLaVA-OV  | 0.3             | 92.15                     | original choice |\n| LLaVA-OV  | 0.1             | 92.15 ($\\Delta$ = +0.00)  | alternative     |\n| LLaVA-OV  | 0.5             | 92.15 ($\\Delta$ = +0.00)  | alternative     |\n| LLaVA-OV  | 0.7             | 92.15 ($\\Delta$ = +0.00)  | alternative     |\n\nIn addition to demonstrating insensitivity to hyperparameter changes, we also validate the robustness of FOCUS across different model architectures. Specifically, we integrate Qwen-VL-2.5-7B with FOCUS using hyperparameters similar to those applied to LLaVA-OV-7B. As shown in the table below, FOCUS substantially improves performance on various datasets compared to the vanilla Qwen model. These results further demonstrate the robustness of FOCUS across different MLLMs. We will include these results in the revised version of the paper.\n\n| Model              | Dataset     | Accuracy (%)            |\n|--------------------|-------------|-------------------------|\n| Qwen-2.5-VL        | V\\*Bench    | 79.06                     |\n| Qwen-2.5-VL + FOCUS| V\\*Bench    | **90.58** ($\\Delta$ = +11.52) |\n| Qwen-2.5-VL        | HRBench-4K  | 71.62                     |\n| Qwen-2.5-VL + FOCUS| HRBench-4K  | **79.25** ($\\Delta$ = +7.63)  |\n| Qwen-2.5-VL        | HRBench-8K  | 68.62                     |\n| Qwen-2.5-VL + FOCUS| HRBench-8K  | **76.25** ($\\Delta$ = +7.63)  |\n\nOur analysis of FOCUS's robustness with respect to hyperparameter selection shows that the method's performance is stable across a range of settings and thus broadly applicable.\n\n### Limitations regarding extremely small objects\n\n> \"In the Limitations section, you mention that FOCUS struggles with very small objects in high-res images. Can you show more failure cases or examples to help understand this problem?\"\n\nWe would like to refer you to Figure 7 (IV) on page 25 of the appendix, which illustrates this issue. Due to the updated rebuttal guidelines, we are unable to include additional qualitative examples at this stage. In a revised version of the paper, we will incorporate further examples to more clearly demonstrate this limitation.\n\n### VQA regarding spatial concepts\n\n> \"Can FOCUS handle questions that involve spatial relationships, such as \"the red ball on the left,\" where token similarity maps may not uniquely identify the correct region?\"\n\nThis question highlights an important limitation of current MLLMs. These models exhibit poor understanding of spatial concepts such as \"on the left\" and \"on the right\" in the image [R2-4]. While targeted fine-tuning can partially mitigate this issue [R3], it is not part of the standard training of MLLMs. Consequently, our FOCUS method inherits this limitation and may produce ambiguous object relevance maps when applied to off-the-shelf MLLMs. We will explicitly acknowledge this limitation in a revised version of the paper.\n\n**References:**\n\nR1: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, Zheng et al, NeurIPS 2023\n\nR2: Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs, Tong et al, CVPR 2024\n\nR3: SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities, Chen et al, CVPR 2024\n\nR4: Beyond Semantics: Rediscovering Spatial Awareness in Vision-Language Models, Qi et al, arXiv 2025"}, {"review_id": "OnMdmB92KT", "reviewer": "Reviewer_DZ3H", "summary": "This paper proposes a novel approach for fine-grained VQA called FOCUS, based on visual cropping techniques, where the objective is to capture relevant regions of an image depending on a question. The authors propose a training-free method that leverages the self-attention mechanism of an MLLM to build object relevance maps. Their system has the following steps: i) identify target objects on the question using in-context learning, ii) generate pseudo-attention using cached token similarity from MLLMs, iii) build object relevance maps, iv) propose regions of interest, v) rank regions of interest, and vi) prompt the MLLM with the selected region for answer generation. FOCUS is evaluated on three fine-grained VQA datasets and two MLLMs, one global-view MLLM (Llava-1.5) and another global-local-view MLLM (Llava-One-Vision). In general, FOCUS presents similar or superior performance compared with SOTA techniques, being significantly more efficient. Specially strong results are obtained with Llava-One-Vision. In addition to the main results, authors also present an ablation study and some qualitative examples.", "questions": "I have no further questions or suggestions besides those formulated in the weaknesses.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strengths:**\n1. The use of cached token similarity to build object relevance maps, since it is a technique that could be applied to different problems, not only fine-grained VQA.\n2. The obtained results are strong, considering performance and efficiency, specially for Llava-One-Vision.\n3. The training-free nature of FOCUS, since it shows that current MLLMs are more capable than immediately obvious and that there is much room for improvement working on inference-time techniques.  \n4. The paper is well structured and written. Authors motivate their work very well, providing a comprehensive analysis of the state-of-the-art. The method is clearly explained and the experiments are meaningful.\n\n**Weaknesses:**\n1. FOCUS has many hyperparameters, as it is quite a complex inference-time algorithm. Authors show the values for those hyperparameters in the appendices, but they don't specify how they set them. There are also many design decisions that I suppose they had to test, such as the prompts used, applying a Gaussian filter to the pseudo-attention map, normalizing confidence scores... Do you use a development dataset to take all those decisions? Do you use one of the MLLMs only to find those hyperparameters and then apply them for the other also? Those details are very important in my opinion and I couldn't find much information about them.\n2. From the paper text, I understand that the ablation studies of Section 4.3 are done on the evaluation datasets (V*Bench and HRBench-4K). This is not a good practice, since ablation studies and development experiments can be connected. Any reader may suspect that ablation studies are actually used to develop the system, optimizing for the test datasets. This weakness is related with the previous. I think authors have to make those points clear.", "was_revised": true, "final_justification": "I think the authors have addressed all my concerns suitably. Given the contributions of the paper, the clear writing and the new results provided by the authors during the rebuttal, I recommend the paper for acceptance.", "rebuttal": "We are grateful for your feedback and pleased to hear that you see broad application possibilities for core concepts of FOCUS, not only for fine-grained VQA but a wide spectrum of visual reasoning tasks!\n\n### Design choices\n\n> \"FOCUS has many hyperparameters, as it is quite a complex inference-time algorithm. Authors show the values for those hyperparameters in the appendices, but they don't specify how they set them. There are also many design decisions that I suppose they had to test, such as the prompts used, applying a Gaussian filter to the pseudo-attention map, normalizing confidence scores...\n\nWe thank you for pointing out the need for more clarity regarding hyperparameter selection and design choices. We first explain our design choices. Several components of FOCUS are based on best practices from prior work rather than iterative tuning: \n\n* Gaussian filter on pseudo-attention: We apply a Gaussian filter with σ = 1 as a standard denoising step commonly used in image processing [R1]. Empirically, we observe that this results in cleaner and more coherent object relevance maps.\n* Confidence normalization: This is adopted from ZoomEye and used without tuning the normalization function.\n* Existence prompt: This is inspired by ZoomEye, with only minor wording adjustments to better fit our task context.\n\n### Development set and robustness w.r.t. hyperparameters\n\n> Do you use a development dataset to take all those decisions? Do you use one of the MLLMs only to find those hyperparameters and then apply them for the other also? Those details are very important in my opinion and I couldn't find much information about them.\"\n\nand\n\n> \"From the paper text, I understand that the ablation studies of Section 4.3 are done on the evaluation datasets (VBench and HRBench-4K). This is not a good practice, since ablation studies and development experiments can be connected. Any reader may suspect that ablation studies are actually used to develop the system, optimizing for the test datasets. This weakness is related with the previous. I think authors have to make those points clear.\"\n\nWe acknowledge that we did not use a dedicated development set. One practical challenge is that high-quality, fine-grained VQA data is still quite scarce. We select hyperparameters based on common values from literature, and did not systematically tune them to optimize the performance on evaluation datasets. Regarding the ablation studies in Section 4.3, we confirm that they were conducted strictly post-hoc, after the method was finalized and all hyperparameters were fixed. We exclusively use them to interpret and validate design choices, not to guide system development.\n\nTo directly address your concern regarding potential overfitting or reliance on hyperparameter tuning, we conduct a systematic robustness analysis using both LLaVA-1.5 and LLaVA-OV on V\\*Bench. We examine the impact of key hyperparameters, including the number of anchor points ($k$), the minimal Euclidean distance between anchor points ($s_\\text{dist}$), the maximum ROI size ($s_\\text{max}$), the ROI expansion threshold, and the NMS threshold. As shown in the tables below, the accuracy obtained using alternative hyperparameter choices shows only **marginal degradation** of maximum 4.7% compared to the accuracy achieved with the original settings, demonstrating the stability of our method. During the robustness analysis, we discover some alternative hyperparameter configurations that achieve even higher accuracy than our default settings.\n\nIn the following, we present the results of our hyperparameter robustness analysis in detail. For the number of anchor points ($k$), we uniformly sampled 20 values and report the mean accuracy along with a 95% confidence interval (corresponding to 2 standard deviations).\n\n| Model     | $k$   | Accuracy (%)   | Comment        |\n|-----------|---------------|--------------|----------------|\n| LLaVA-1.5 | 30               | 72.77          | original choice| \n| LLaVA-1.5 | random(10,50)               | 72.77 ± 1.55  ($\\Delta$ = +0.00)        | alternative    | \n| LLaVA-OV  | 30               | 92.15          | original choice| \n| LLaVA-OV  | random(10,50)               | 92.03 ± 1.37   ($\\Delta$ = -0.12)       | alternative    | \n\n\nFor the minimum distance between ROI anchor points $s_{\\text{dist}}$, we varied the value from 2 to 4 for LLaVA-1.5, and from 2 to 5 for LLaVA-OV.\n\n | Model     | $s_\\text{dist}$ | Accuracy (%)              | Comment         |\n|-----------|---------------|-------------------------|-----------------|\n| LLaVA-1.5 | 2               | 72.77                     | original choice |\n| LLaVA-1.5 | 3               | 72.25 ($\\Delta$ = −0.52)  | alternative     |\n| LLaVA-1.5 | 4               | 69.11 ($\\Delta$ = −3.66)  | alternative     |\n| LLaVA-OV  | 2               | 92.15                     | original choice |\n| LLaVA-OV  | 3               | 91.62 ($\\Delta$ = −0.53)  | alternative     |\n| LLaVA-OV  | 4               | 92.15 ($\\Delta$ = +0.00)  | alternative     |\n| LLaVA-OV  | 5               | 91.62 ($\\Delta$ = −0.53)  | alternative     |\n\nFor the maximum ROI size $s_\\text{max}$, we explored alternative values ranging from 5 to 9 for LLaVA-1.5, and from 5 to 11 for LLaVA-OV.\n\n| Model     | $s_\\text{max}$ | Accuracy (%)              | Comment         |\n|-----------|--------------|-------------------------|-----------------|\n| LLaVA-1.5 | 5              | 72.77                     | original choice |\n| LLaVA-1.5 | 7              | 69.63 ($\\Delta$ = −3.14)  | alternative     |\n| LLaVA-1.5 | 9              | 68.06 ($\\Delta$ = −4.71)  | alternative     |\n| LLaVA-OV  | 9              | 92.15                     | original choice |\n| LLaVA-OV  | 5              | 94.24 ($\\Delta$ = +2.09)  | alternative     |\n| LLaVA-OV  | 7              | 91.01 ($\\Delta$ = −1.14)  | alternative     |\n| LLaVA-OV  | 11             | 89.53 ($\\Delta$ = −2.62)  | alternative     |\n\n\nFor the ROI expansion threshold, we uniformly sampled 20 values and reported the mean accuracy along with a 95% confidence interval (corresponding to 2 standard deviations).\n\n| Model     | ROI expansion threshold   | Accuracy (%)   | Comment        |\n|-----------|---------------|--------------|----------------|\n| LLaVA-1.5 | 0.5               | 72.77          | original choice| \n| LLaVA-1.5 | random (0.3, 0.7)               | 72.77 ± 0.00  ($\\Delta$ = +0.00)        | alternative    | \n| LLaVA-OV  | 0.5               | 92.15          | original choice| \n| LLaVA-OV  | random (0.3, 0.7)               | 92.70 ± 0.25 ($\\Delta$ = +0.55)         | alternative    | \n\nMoreover, we report accuracy using NMS thresholds ranging from 0.1 to 0.7 to demonstrate robustness w.r.t. this parameter.\n\n| Model     | NMS threshold   | Accuracy (%)              | Comment         |\n|-----------|---------------|-------------------------|-----------------|\n| LLaVA-1.5 | 0.3             | 72.77                     | original choice |\n| LLaVA-1.5 | 0.1             | 70.16 ($\\Delta$ = −2.61)  | alternative     |\n| LLaVA-1.5 | 0.5             | 72.25 ($\\Delta$ = −0.52)  | alternative     |\n| LLaVA-1.5 | 0.7             | 72.25 ($\\Delta$ = −0.52)  | alternative     |\n| LLaVA-OV  | 0.3             | 92.15                     | original choice |\n| LLaVA-OV  | 0.1             | 92.15 ($\\Delta$ = +0.00)  | alternative     |\n| LLaVA-OV  | 0.5             | 92.15 ($\\Delta$ = +0.00)  | alternative     |\n| LLaVA-OV  | 0.7             | 92.15 ($\\Delta$ = +0.00)  | alternative     |\n\nIn addition to demonstrating robustness to hyperparameter changes, we also validate the robustness of FOCUS across different model architectures. Specifically, we integrate Qwen-VL-2.5-7B with FOCUS using hyper-parameters similar to those applied to LLaVA-OV-7B. As shown in the table below, FOCUS substantially improves performance on various datasets compared to the vanilla Qwen model. These results further demonstrate the robustness of FOCUS across different MLLMs. We will include these results in the revised version of the paper.\n\n| Model              | Dataset     | Accuracy (%)              |\n|--------------------|-------------|-------------------------|\n| Qwen-2.5-VL        | V\\*Bench    | 79.06                     |\n| Qwen-2.5-VL + FOCUS| V\\*Bench    | **90.58** ($\\Delta$ = +11.52) |\n| Qwen-2.5-VL        | HRBench-4K  | 71.62                     |\n| Qwen-2.5-VL + FOCUS| HRBench-4K  | **79.25** ($\\Delta$ = +7.63)  |\n| Qwen-2.5-VL        | HRBench-8K  | 68.62                     |\n| Qwen-2.5-VL + FOCUS| HRBench-8K  | **76.25** ($\\Delta$ = +7.63)  |\n\nOur analysis of FOCUS's robustness with respect to hyperparameter selection shows that the method's performance is stable across a range of settings and thus broadly applicable.\n\n**References:**\n\nR1: Digital Image Processing, Gonzalez and Woods, 2008"}, {"review_id": "j2wmswx1sJ", "reviewer": "Reviewer_VuY3", "summary": "This paper proposes a new method for fine-grained image recognition on high-resolution images. Standard vision-language models such as LLaVA often struggle with recognizing small objects due to limitations in their vision encoder architecture. Although several training-free methods have been proposed to address this issue, they typically incur high inference costs because they require multiple forward passes over many local crop regions. To improve efficiency, this paper proposes a method that leverages attention maps obtained during inference to reduce the number of forward passes. The proposed approach is evaluated on standard benchmarks for fine-grained recognition with high-resolution images and demonstrates a favorable accuracy-efficiency trade-off.", "questions": "As mentioned above, discussing the trade-off between accuracy and inference time is crucial. Could you include a plot showing accuracy versus inference time with the number of steps (n_step) for both the proposed method and existing baselines? This would provide a clearer view of the practical impact and efficiency of your approach.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Quality and Significance\n\n- The proposed method is intuitive and demonstrates a promising accuracy-efficiency trade-off, making it a valuable contribution in this domain.\n\n- While the number of forward passes is used as the primary metric for efficiency, it would be important to also report actual inference time. This is because efficiency is affected by various factors such as memory access patterns, hardware-level optimizations, and framework-specific implementation details. Although inference time is reported in the appendix, it should be discussed in the main text, and a plot showing accuracy versus number of steps (n_step) would help better illustrate the actual trade-off.\n\n- It is unclear how reliable the number of forward passes is as an indicator of computational cost, as it can vary significantly depending on implementation. FLOPs or MACs may serve as more objective metrics for computational complexity.\n\n\nClarity\n\n- The paper is generally easy to follow. However, providing the full algorithm (e.g., in pseudocode or step-by-step form) in the main text would improve clarity—particularly since Section 3.3 is somewhat difficult to follow.\n\n- Some equations use Python- or PyTorch-specific notations (e.g., reshape), which are not appropriate in mathematical equations. These should be replaced with standard mathematical notation.\n\n\nOriginality\n\n- While the use of attention maps itself is not novel, applying them in this way to improve efficiency for high-resolution fine-grained recognition is relatively underexplored and shows good empirical results. This gives the work sufficient novelty within its scope.", "was_revised": true, "final_justification": "Thank you for conducting the additional experiments on inference cost. The results clearly demonstrate the effectiveness of the proposed method. Although the best accuracy achieved by the proposed approach does not surpass that of ZoomEye, its inference efficiency is significantly superior. As my concerns have been sufficiently addressed, I am willing to raise my score.", "rebuttal": "We sincerely thank you for your constructive feedback and greatly appreciate the emphasis on real-world efficiency metrics, such as inference time, which aligns closely with the core motivation of our work.\n\n### Additional efficiency metrics, e.g., FLOPs\n\n> \"While the number of forward passes is used as the primary metric for efficiency, it would be important to also report actual inference time. This is because efficiency is affected by various factors such as memory access patterns, hardware-level optimizations, and framework-specific implementation details. Although inference time is reported in the appendix, it should be discussed in the main text, and a plot showing accuracy versus number of steps (n_step) would help better illustrate the actual trade-off.\"\n\nand \n\n> \"It is unclear how reliable the number of forward passes is as an indicator of computational cost, as it can vary significantly depending on implementation. FLOPs or MACs may serve as more objective metrics for computational complexity.\"\n\nand\n\n> \"It is unclear how reliable the number of forward passes is as an indicator of computational cost, as it can vary significantly depending on implementation. FLOPs or MACs may serve as more objective metrics for computational complexity.\"\n\nIn our paper, we select the number of forward passes (FPs) as the primary indicator of computational cost due to its simplicity. To provide a more comprehensive efficiency analysis, we additionally report both inference time and FLOPs for FOCUS and ZoomEye. We calculate the FLOPs for LLaVA-1.5 based on the estimation formulas introduced in FastV [R1] and PyramidDrop [R2]. Following these prior works, we exclude the embedding, projection layers and the output head of the MLLM from our FLOPs estimation:\n\n```\ndef flops_estimate(seq_length):\n    \"\"\"\n    Estimate the FLOPs of one LLaVA-1.5 forward pass given the sequence length\n    Using the estimation formula from FastV/PyramidDrop paper\n    Per LLM layer: 4nd**2+2n**2*d+3ndm\n    Per ViT layer: 4nd**2+2n**2*d+2ndm\n    ViT-CLIP-L outputs 576 + 1 (CLS) = 577 visual tokens\n    Number of visual tokens in the sequence is always 576 \n    \"\"\"\n    llm_dim = 4096\n    llm_num_layers = 32\n    llm_ffn_dim = 11008\n    vit_dim = 1024\n    vit_num_layers = 24\n    vit_ffn_dim = 4096\n    flops_llm = llm_num_layers * ((4 * seq_length * (llm_dim**2)) + (2 * (seq_length**2) * llm_dim) + (3 * seq_length * llm_dim * llm_ffn_dim))\n    flops_vit = vit_num_layers * ((4 * 577 * (vit_dim**2)) + (2 * (577**2) * vit_dim) + (2 * 577 * vit_dim * vit_ffn_dim))\n    return flops_llm + flops_vit\n```\n\nDue to the new rebuttal guidelines, we are unable to include new plots. Therefore, we present the results in table format. For the revised version of the paper, we plan to include more detailed efficiency plots, including inference time, FLOPs, and number of steps.\n\nWe conduct our experiments for LLaVA-1.5 with FOCUS on V\\*Bench and report the average inference time and FLOPs per image across the full dataset:\n\n| Config of FOCUS           | Accuracy (%)   | Inference Time (s)    | FPs       | TFLOPs    |\n|---------------------------|--------------|---------------------|---------|---------|\n| n_steps=1 (no_overrun)    | 51.30          | 0.99                  | 1.47      | 10.98     |\n| n_steps=2 (no_overrun)    | 57.07          | 1.28                  | 4.25      | 23.11     |\n| n_steps=1                 | 64.40          | 1.36                  | 4.86      | 25.73     |\n| n_steps=2                 | 66.49          | 1.44                  | 5.70      | 29.43     |\n| n_steps=3                 | 67.01          | 1.55                  | 6.79      | 34.15     |\n| n_steps=4                 | 68.06          | 1.73                  | 8.27      | 40.61     |\n| n_steps=6                 | 70.68          | 2.00                  | 10.71     | 51.26     |\n| n_steps=8                 | 72.77          | 2.27                  | 13.28     | 62.46     |\n\nAs these are newly conducted experiments, the results for `n_steps=8` slightly differ from those reported in the appendix. For comparison, we also provide the results for LLaVA-1.5 with ZoomEye on V\\*Bench:\n\n| Config of ZoomEye         | Accuracy (%) | Inference Time (s)  | FPs     | TFLOPs  |\n|---------------------------|--------------|---------------------|---------|---------|\n| Depth-1 (2 crops)\t        | 50.26        | 3.78                | 12.50   | 55.95   |\n| Depth-1 (4 crops)\t        | 50.78        | 4.76                | 20.37   | 91.03   |\n| Depth-2 (4 crops)\t        | 71.20        | 9.73                | 44.54   | 199.21  |\n| Depth-5 (4 crops)\t        | 77.48        | 11.96               | 48.63   | 217.00  |\n\nIn summary, the efficiency gains reported in terms of FPs are **consistently reflected** in **real-world improvements**. FOCUS achieves competitive or superior accuracy with **significantly lower inference time and fewer FLOPs** compared to ZoomEye. For instance, at `n_steps=6`, FOCUS reaches 70.68% accuracy in just 2.00 seconds and 51.26 TFLOPs, whereas ZoomEye (depth-2) requires nearly 4× more FLOPs and 5× longer inference time to reach a comparable accuracy. These results underscore the practical efficiency of FOCUS for fine-grained visual reasoning tasks.\n### Pseudocode\n> \"The paper is generally easy to follow. However, providing the full algorithm (e.g., in pseudocode or step-by-step form) in the main text would improve clarity—particularly since Section 3.3 is somewhat difficult to follow.\"\n\nWe agree with you that presenting the full FOCUS algorithm in pseudocode would enhance clarity. Due to the updated rebuttal guidelines, we are unable to upload PDFs containing properly formatted LaTeX-style pseudocode. Nevertheless, we have made an effort to represent the core steps of Section 3.3 using simplified pseudocode within the rebuttal. A complete and properly formatted version of the pseudocode will be included in the revised paper.\n\n| **Pseudocode for Section 3.3** |\n|------------------------|\n| **Input**: Object relevance map $\\mathbf{A}$, number of anchor points $k$, number of steps $n_{\\text{steps}}$, ROI parameters: $s_\\text{min}$, $s_\\text{max}$, $s_\\text{dist}$, threshold for NMS: $\\text{nms}_{\\text{threshold}}$, model MLLM |\n| **Step 1**: anchor_points ← ExtractTopKAnchors($\\mathbf{A}$, $k$) |\n| **Step 2**: initial_rois ← GenerateSymmetricROIs(anchor_points, $s_\\text{min}$) |\n| **Step 3**: expanded_rois ← ExpandROIs(initial_rois, $\\mathbf{A}$, $s_\\text{max}$, $s_\\text{dist}$) |\n| **Step 4**: filtered_rois ← ApplyNMS(expanded_rois, $\\text{nms}_{\\text{threshold}}$) |\n| **Step 5**: ranked_rois ← RankROIsByConfidence(filtered_rois, $n_{\\text{steps}})$ |\n| **Step 6**: answer ← FinalInference(MLLM, ranked_rois) |\n| **Output**: MLLM answer based on top-ranked ROI |\n\n> \"Python- or PyTorch-specific notations (e.g., reshape), which are not appropriate in mathematical equations. These should be replaced with standard mathematical notation.\"\n\nWe agree that replacing Python- or PyTorch-specific notations with standard mathematical notation would enhance clarity. We will revise the equations accordingly in the updated version of the paper.\n\n**References:**\n\nR1: An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models, Chen et al, ECCV 2024\n\nR2: PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction, Xing et al, CVPR 2025"}, {"review_id": "D7NpoD9Eji", "reviewer": "Reviewer_BXxL", "summary": "This study introduces a visual cropping technique for fine-grained visual question answering. The proposed method leverage the representation from MLLM to search for the most relevant image region. Specifically, it first identify the target objects in order to compute an object relevance map. It then pick the top rank regions to make the prediction. The paper shows that the proposed cropping method improves the efficiency significantly.", "questions": "See Weaknesses above.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**:\n1. The motivation is clear.\n2. It is interesting to explore the visual cropping to make MLLM more efficient.\n\n**Weaknesses**:\n1. The proposed method introduces many hand-crafted components such as ROI, non-maximum suppression which contains more hyper-parameters. This potentially makes the method less robust.\n2. The proposed method is only evaluated with limited datasets. Beyond the fine-grained VQA, I would like to see the evaluation on broader range of vision-language datasets.\n3. While becoming more efficient, the proposed method does not outperform another cropping method, ZoomEye. It seems like the method trades performance for efficiency.", "was_revised": true, "final_justification": "Most of my concerns have been addressed in the rebuttal.", "rebuttal": "We sincerely thank you for your questions and feedback on our paper. \n\n### Robustness w.r.t. hyperparameters\n\n> \"The proposed method introduces many hand-crafted components such as ROI, non-maximum suppression which contains more hyper-parameters. This potentially makes the method less robust.\"\n\nWe understand your concern regarding robustness of FOCUS w.r.t. components such as region of interest (ROI) proposals and non-maximum suppression (NMS). We conduct additional experiments using both LLaVA-1.5 and LLaVA-OV on V\\*Bench to systematically examine the impact of key hyperparameters, including the number of anchor points ($k$), the minimal Euclidean distance between anchor points ($s_\\text{dist}$), the maximum ROI size ($s_\\text{max}$), the ROI expansion threshold, and the NMS threshold. As shown in the tables below, the accuracy obtained using alternative hyperparameter choices shows only **marginal degradation** of maximum 4.7% compared to the accuracy achieved with the original settings, demonstrating the stability of our method. During the robustness analysis, we discover some alternative hyperparameter configurations that achieve even higher accuracy than our default settings.\n\nIn the following, we present the results of our hyperparameter robustness analysis in detail. For the number of anchor points ($k$), we uniformly sampled 20 values and report the mean accuracy along with a 95% confidence interval (corresponding to 2 standard deviations).\n\n| Model     | $k$   | Accuracy (%)   | Comment        |\n|-----------|---------------|--------------|----------------|\n| LLaVA-1.5 | 30               | 72.77          | original choice| \n| LLaVA-1.5 | random(10,50)               | 72.77 ± 1.55  ($\\Delta$ = +0.00)        | alternative    | \n| LLaVA-OV  | 30               | 92.15          | original choice| \n| LLaVA-OV  | random(10,50)               | 92.03 ± 1.37   ($\\Delta$ = -0.12)       | alternative    | \n\n\nFor the minimum distance between ROI anchor points $s_{\\text{dist}}$, we varied the value from 2 to 4 for LLaVA-1.5, and from 2 to 5 for LLaVA-OV.\n\n | Model     | $s_\\text{dist}$ | Accuracy (%)              | Comment         |\n|-----------|---------------|-------------------------|-----------------|\n| LLaVA-1.5 | 2               | 72.77                     | original choice |\n| LLaVA-1.5 | 3               | 72.25 ($\\Delta$ = −0.52)  | alternative     |\n| LLaVA-1.5 | 4               | 69.11 ($\\Delta$ = −3.66)  | alternative     |\n| LLaVA-OV  | 2               | 92.15                     | original choice |\n| LLaVA-OV  | 3               | 91.62 ($\\Delta$ = −0.53)  | alternative     |\n| LLaVA-OV  | 4               | 92.15 ($\\Delta$ = +0.00)  | alternative     |\n| LLaVA-OV  | 5               | 91.62 ($\\Delta$ = −0.53)  | alternative     |\n\nFor the maximum ROI size $s_\\text{max}$, we explored alternative values ranging from 5 to 9 for LLaVA-1.5, and from 5 to 11 for LLaVA-OV.\n\n| Model     | $s_\\text{max}$ | Accuracy (%)              | Comment         |\n|-----------|--------------|-------------------------|-----------------|\n| LLaVA-1.5 | 5              | 72.77                     | original choice |\n| LLaVA-1.5 | 7              | 69.63 ($\\Delta$ = −3.14)  | alternative     |\n| LLaVA-1.5 | 9              | 68.06 ($\\Delta$ = −4.71)  | alternative     |\n| LLaVA-OV  | 9              | 92.15                     | original choice |\n| LLaVA-OV  | 5              | 94.24 ($\\Delta$ = +2.09)  | alternative     |\n| LLaVA-OV  | 7              | 91.01 ($\\Delta$ = −1.14)  | alternative     |\n| LLaVA-OV  | 11             | 89.53 ($\\Delta$ = −2.62)  | alternative     |\n\n\nFor the ROI expansion threshold, we uniformly sampled 20 values and reported the mean accuracy along with a 95% confidence interval (corresponding to 2 standard deviations).\n\n| Model     | ROI expansion threshold   | Accuracy (%)   | Comment        |\n|-----------|---------------|--------------|----------------|\n| LLaVA-1.5 | 0.5               | 72.77          | original choice| \n| LLaVA-1.5 | random (0.3, 0.7)               | 72.77 ± 0.00  ($\\Delta$ = +0.00)        | alternative    | \n| LLaVA-OV  | 0.5               | 92.15          | original choice| \n| LLaVA-OV  | random (0.3, 0.7)               | 92.70 ± 0.25 ($\\Delta$ = +0.55)         | alternative    | \n\nMoreover, we report accuracy using NMS thresholds ranging from 0.1 to 0.7 to demonstrate robustness w.r.t. this parameter.\n\n| Model     | NMS threshold   | Accuracy (%)              | Comment         |\n|-----------|---------------|-------------------------|-----------------|\n| LLaVA-1.5 | 0.3             | 72.77                     | original choice |\n| LLaVA-1.5 | 0.1             | 70.16 ($\\Delta$ = −2.61)  | alternative     |\n| LLaVA-1.5 | 0.5             | 72.25 ($\\Delta$ = −0.52)  | alternative     |\n| LLaVA-1.5 | 0.7             | 72.25 ($\\Delta$ = −0.52)  | alternative     |\n| LLaVA-OV  | 0.3             | 92.15                     | original choice |\n| LLaVA-OV  | 0.1             | 92.15 ($\\Delta$ = +0.00)  | alternative     |\n| LLaVA-OV  | 0.5             | 92.15 ($\\Delta$ = +0.00)  | alternative     |\n| LLaVA-OV  | 0.7             | 92.15 ($\\Delta$ = +0.00)  | alternative     |\n\nIn addition to demonstrating robustness to hyperparameter changes, we also validate the robustness of FOCUS across different model architectures. Specifically, we integrate Qwen-VL-2.5-7B with FOCUS using hyperparameters similar to those applied to LLaVA-OV-7B. As shown in the table below, FOCUS substantially improves performance on various datasets compared to the vanilla Qwen model. These results further demonstrate the robustness of FOCUS across different MLLMs. We will include these results in the revised version of the paper.\n\n| Model              | Dataset     | Accuracy (%)              |\n|--------------------|-------------|-------------------------|\n| Qwen-2.5-VL        | V\\*Bench    | 79.06                     |\n| Qwen-2.5-VL + FOCUS| V\\*Bench    | **90.58** ($\\Delta$ = +11.52) |\n| Qwen-2.5-VL        | HRBench-4K  | 71.62                     |\n| Qwen-2.5-VL + FOCUS| HRBench-4K  | **79.25** ($\\Delta$ = +7.63)  |\n| Qwen-2.5-VL        | HRBench-8K  | 68.62                     |\n| Qwen-2.5-VL + FOCUS| HRBench-8K  | **76.25** ($\\Delta$ = +7.63)  |\n\n### Evaluation beyond fine-grained VQA\n> \"The proposed method is only evaluated with limited datasets. Beyond the fine-grained VQA, I would like to see the evaluation on broader range of vision-language datasets.\"\n\nIn our work, we primarily focused on fine-grained VQA, as this research area remains underexplored and lacks efficient, high-performing methods. To assess the broader applicability of FOCUS, we conduct additional experiments on the A-OKVQA dataset, which features questions involving more commonly sized or large objects. Notably, this dataset is also used to evaluate ViCrop's performance in large-object scenarios [R1].\n\nBelow, we report the performance of LLaVA-1.5 and LLaVA-OV with and without FOCUS on A-OKVQA, employing the same hyperparameters as used for V\\*Bench:\n\n| Model           | A-OKVQA Accuracy (%)|\n|-----------------|--------------------|\n| LLaVA-1.5       | 77.99                |\n| LLaVA-1.5 + FOCUS| 74.76 ($\\Delta$ = −3.23)|\n| LLaVA-OV        | 91.44            |\n| LLaVA-OV + FOCUS| 91.00 ($\\Delta$ = −0.44)|\n\nWe observe **only marginal performance degradation**, no more than 3.3% compared to the vanilla counterparts. This indicates that FOCUS **remains compatible** with VQA benchmarks **involving larger objects**, highlighting its **adaptability** across different vision-language datasets.\n\n### Comparison to ZoomEye\n\n> \"While becoming more efficient, the proposed method does not outperform another cropping method, ZoomEye. It seems like the method trades performance for efficiency.\"\n\nWe thank you for highlighting this concern. We would like to take this opportunity to put your assessment into perspective. As shown in the table below, FOCUS consistently achieves **substantial efficiency gain** over ZoomEye, often by 2× to 5×, **without compromising accuracy**. In **5 out of 8** dataset-model combinations, such as LLaVA-1.5 on HRBench-4K and LLaVA-OV on V\\*Bench and HRBench datasets, FOCUS even **outperforms** ZoomEye in terms of accuracy.\n \n| Model     | Dataset    | FOCUS's performance gain relative to ZoomEye (%) | FOCUS's efficiency gain relative to ZoomEye (%) |\n|-----------|------------|---------------------|--------------------|\n| LLaVA-1.5 | V\\*Bench   | -6.08                 | +266.19         |\n| LLaVA-1.5 | HRBench-4k | **+3.50**            | +240.79           |\n| LLaVA-1.5 | HRBench-8K | -8.17                 | +490.49         |\n| LLaVA-OV  | V\\*Bench   | **+1.15**            | +214.29           |\n| LLaVA-OV  | HRBench-4K | **+2.52**            | +117.67           |\n| LLaVA-OV  | HRBench-8K | **+0.91**            | +165.73           |\n| LLaVA-OV  | MME-RW-P   | -3.80                 | +439.56           |\n| LLaVA-OV  | MME-RW-R   | **+3.08**            | +459.68           |\n\nIn summary, FOCUS does not compromise performance for efficiency. Instead, it achieves a **balanced and optimized trade-off**, outperforming or roughly matching ZoomEye in accuracy while being significantly more efficient. At a time when the computational demands of AI models are steadily increasing, we see our work as a contribution toward addressing resource constraints and reducing the energy consumption associated with large-scale vision-language systems.\n\n**References:**\n\nR1: MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs, Zhang et al, ICLR 2025"}]

Author rebuttal:

Meta-review: The paper was reviewed by four experts who provided detailed comments. Among the weaknesses raised, a common concern was the large number of hyperparameters that the method required. The authors addressed this concern and others in their response, to the satisfaction of the reviewers and area chair. In the end, the reviewers unanimously recommend accepting the paper and the area chair sees no reason to override their recommendation.
Decision: Accept (poster)