MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs
Authors: Jiarui Zhang
Mahyar Khayatkhoei
Prateek Chhikara
Filip Ilievski
Venue: ICLR 2025 Poster (2025)
Keywords: Multimodal Large Language Models
Visual Details
Attention
Gradients
Bias
Perception
Localization
Primary area: generative models
TL;DR: We study the perception limitation of Multimodal LLMs and propose automatic visual cropping as a scalable and training-free solution to mitigate this limitation.

Abstract: Multimodal Large Language Models (MLLMs) have experienced rapid progress in visual recognition tasks in recent years. Given their potential integration into many critical applications, it is important to understand the limitations of their visual perception. In this work, we study whether MLLMs can perceive small visual details as effectively as large ones when answering questions about images. We observe that their performance is very sensitive to the size of the visual subject of the question, and further show that this effect is in fact causal by conducting an intervention study. Next, we study the attention patterns of MLLMs when answering visual questions, and intriguingly find that they consistently know where to look, even when they provide the wrong answer. Based on these findings, we then propose training-free visual intervention methods that leverage the internal knowledge of any MLLM itself, in the form of attention and gradient maps, to enhance its perception of small visual details. We evaluate our proposed methods on two widely-used MLLMs and seven visual question answering benchmarks and show that they can significantly improve MLLMs' accuracy without requiring any training. Our results elucidate the risk of applying MLLMs to visual recognition tasks concerning small details and indicate that visual intervention using the model's internal state is a promising direction to mitigate this risk. Our code is available at: https://github.com/saccharomycetes/mllms_know.

Published as a conference paper at ICLR 2025
MLLMS KNOW WHERE TO LOOK: TRAINING-FREE PERCEPTION OF SMALL VISUAL DETAILS WITH MULTIMODAL LLMs
Jiarui Zhang e. Mahyar Khayatkhoei & Prateek Chhikara e Filip Ilievski f
& University of Southern California, USA
Vrije Universiteit Amsterdam, The Netherlands
ABSTRACT
Multimodal Large Language Models (MLLMs) have experienced rapid progress in visual recognition tasks in recent years. Given their potential integration into many critical applications, it is important to understand the limitations of their visual perception. In this work, we study whether MILMSs can perceive small visual details as effectively as large ones when answering questions about images. We observe that their performance very sensitive to the size of the visual subject of the question, and further show that this effect is in fact causal by conducting an intervention study. Next, we study the attention patterns of MLLMs when answering visual questions, and intriguingly find that they consistently know where to look, even when they provide the wrong answer. Based on these findings, we then propose training-frre frre visual intervention methods that leverage the internal knowledge of any MLLM itself, in the form of attention and gradient maps, to enhance its perception of small visual details. We evaluate our proposed methods on two widely-used MLLMs and seven visual question answering benchmarks and show that they can significantly improve MILMs' accuracy without requiring any training. Our results elucidate the risk of applying MLLMs to visual recogni- tion tasks concerning small details and indicate that visual intervention using the model's internal state is a promising direction to mitigate this risk.'
INTRODUCTION
Multimodal large language models (MLLMs) (Hurst et al., 2024; Team al., 2024; Anthropic, 2024; Wang et al., 2024; Li et al., 2024a; Team et al., 2025; Chen et al., 2025) have greatly progressed the state of multimodal reasoning and planning, and are rapidly being integrated into various downstream applications, ranging from robotics (Li et al., 2024b; Chen et al., 2024), biomedicine (Li et al., 2023a), autonomous driving (Xu et al., 2024b; Zhang et al., 2023a) to visual mathematical reasoning (Gao et al., 2023; Zhang et al., 2024c;b) and even food recipe generation (Chhikara et al., 2024). Given the rapidly growing application of MILMs, especially in critical domains such as biomedicine and security, it crucial to study the limitations of their visual perception to elucidate the potential risks that may affect their downstream applications.
To motivate the limitation that will be the focus of this work, we start by presenting three revealing vi- sual question answering examples in Fig. 1, in which we ask a popular MLLM BLIP-2 (FlanT5xx.) (Li al., 2023b) identify an object's presence or type in each image we vary the size of the object. In the absence of any prior evidence, we might reasonably expect the MLLM's 's answer to be invariant to the size of the object, because of the MLLM's large representational capacity and pretraining on wide variety of images containing objects of various sizes. To the contrary, in Fig. (left), we observe that initially the model does not recognize the existence of a small street sign and assigns a lower probability to the correct answer; however, zooming into the image (via more focused visual cropping) towards the street sign gradually increases the probability assigned to the correct answer, suggesting that the model gradually perceives more and more relevant details of the street sign.
Our code is available at wttptttt/////. //g: thub. com/ http://ccccaaocccceeeeescco
know.

---

Published as a conference paper at ICLR 2025
Q: Are there any street signs the picture?
What kind of bird this?
Q: What brand clock is this?
Smaller doop sizel
Smaller srop size
Smalle 2rop sizo
MoMA
Figure I: The effect of visual cropping on the probability of answers predicted by BLIP-2 Fl zero-shot VQA model. The x-axis labels are indices to the respective cropped images displayed under each plot that the model sees at each step. The model gradually finds the correct answer.
In Fig. (middle), we observe further evidence of this difficulty in perceiving small details: the model initially predicts white as the type of the bird, but when we zoom into the image towards the bird, without changing the question in any way, observe that the model gradually assigns higher probability to the correct bird type of egret. This suggests that the model was not making a semantic error of misunderstanding what type means, rather it was unable to perceive sufficient details to discriminate egret from other white birds, which is mitigated by visual cropping. Similarly, in Fig. I (right), we observe that the model's initial answer is not entirely irrelevant ("ama" compared the correct answer "moma"), suggesting that the model knows where to look based on the question but cannot accurately perceive the actual word, which is again mitigated by visual cropping.
In this work, we will study the limitation observed in Fig. extensively, elucidate its cause, and propose potential solutions to mitigate its consequences. In Sec. 3, we quantitatively show that there indeed exists a difficulty in perceiving small visual concepts across various widely-used MLLMs. Our findings are consistent with prior works on evaluating the text-image matching in vision-language joint embedding models, which have observed reverse correlation between visual object size in images and the text-image matching score (Zhao et al., 2022), but we further establish a causal connection between visual concept size and MLLMs' perception ability through intervention study, In Sec. 4, we study whether the MLLMs' difficulty in perceiving small visual concepts stems from a difficulty in perceiving visual details, or from difficulty in locating the concept due to its small size. We quantitatively show that MILLM consistently know where to look, even when they fail to answer the question correctly. In Sec. 5, we propose three automatic visual cropping methods- the attention maps and gradients of the MLLM itself-as scalable and training-free solutions to the visual perception limitation. Finally, in Sec. 6, we apply our proposed methods to two popular MLLMss and evaluate them on seven visual question answering (VQA) benchmarks, showing their efficacy enhancing MILMs' accuracy, especially on detail-sensitive benchmarks.
2 RELATED WORKS
Multimodal Large Language Models (MLLMs). MLLMs foundation models capable of handling diverse language and vision tasks. These models fall into two categories: end-to-end pretrained and modular pretrained. End-to-end models process joint image-language data through architectures such as dual-encoder (Radford et al., 2021), fusion-encoder (Li et al., 2021), encoder- decoder (Cho et al., 2021), and unified transformer (Wang et al., 2022), using objectives like image-text matching, contrastive learning, and masked language modeling. Modular pretrained approaches. avoid costly full pretraining by adapting existing components: BLIP2 (Li et al., 2023b) and InstructBLIP (Dai et al., 2023) train a Transformer- based connector between a frozen pretrained VIT (Dosovitskiy al., 2021) image encoder and a frozen L.M,, which transforms VIT output tokens into fixed set of image tokens in the input space of the LLM; Qwen-VL (Bai et al., 2023), similarly uses fixed-length token connector (a single cross-attention layer), but trains both the connector and the LLM; LLaVA (Liu et al., 2023b) and LLaVA-1.5 (Liu et al., 2023a) instead use a linear projection and a two-layer MLP as their connectors, respectively, and train both. Our work will contribute better understanding of the perception limitations of MLLM and improve their perception scalably and without training, offering orthogonal benefits existing approaches.
models, which dominate recent state-of-the-ee
2

---

Published as a conference paper at ICLR 2025
Visual Localization Methods. Dedicated visual localization methods, such YOLO (Redmon et al., 2016), SAM (Kirillov et al., 2023), and GLIP (Li et al., 2022b), rely heavily on dense spatial annotations for identifying salient image regions. Native approaches, such Grad-CAM (Selvaraju et al., 2017), localize regions by analyzing gradients from classifier decisions without spatial supervi- sion. Prior work adapts Grad-CAM to BLIP (Li al., 2022a) leveraging its dedicated image-text similarity computation neural network called the Image-Tex: Matching network (Tiong et al., 2022; Guo et al., 2023). In this work, we derived more general way for localizing the attention of MLLMS to images, not relying on the specific BLIP architecture. Several recent works have explored ways improve the visual localization capability of MILMs for visual question answering. including thain-of-thought (Shao et al., 2024; Liu et al., 2024b), tool-using (Wu and Xie, 2023), and visual programming approaches (Surís al., 2023; Gupta and Kembhavi, 2023). In contrast, we demonstrate that MILMS can often effectively localize the visual subject of a question in their internal states, and propose training-foe free methods to leverage their internal states for improving their visual perception.
Visual Perception Limitations in MLLMs. The difficulty of answering questions about small objects in images has been observed by several prior and concurrent works (Zhang et al., 2023b; 2024a; Liu al., 2024a; Wu and Xie, 2023), which have explored mitigating solutions based high-resolution fine-tuning (Liu et al., 2024a; Dehghani et al., 2023; Wang et al., 2024), multi-agent pipelines (Wu and Xie, 2023), and use of visual cropping (Zhang al., 2023b). In this work, we provide more extensive evidence for this difficulty, establish its causal effect on MILMs performance, and show that it is rooted in a failure to observe small visual details as opposed to a failure to locate small objects. Several works have also shown that MLLMS suffer from object hallucination (Li et al., 2023c; Yu et al., 2024). Furthermore, Zhang et al. (2024a) have shown visual blind spots in MLLMs-i.e. locations on the image where the MLLMs' perception degrades- as well as their sensitivity to visual quality, presence of visual distractors in the image, and even local object location perturbations.
3 MLLMs' SENSITIVITY TO THE SIZE OF VISUAL CONCEPTS
In this section, our goal is to quantitatively study our qualitative observations in Fig. that MILMs struggle with describing small visual details images. To that end, we consider the TextVQA dataset, in which for each question we can find the image ground-truth bounding box that contains the correct textual answer. We partition its validation set into three groups based on the relative size of the Acocal where Ass denotes the area of the ground-truth bounding box, and Asotal the total area of the image: S < 0.005 (small) 1) consisting of 773 question-iman 2) 0.005 S < 0.05 (medi um) consisting of 2411 question-image pairs, and 3) 5 2 0.05 (large) consisting of 1186 question-image pairs. We chose TextVQA for this study because it contains
ground-truth bounding box $ =
Ass
pairs,
Table I: Sensitivity of the accuracy of MLLMs to the size of visual concepts in TexIVQA. As the relative visual size of the answer decreases (right to left in each row), observe a decline in the accuracy of the original models (no cropping) in answering questions, whereas visual cropping (buman-CRO?) significantly improves accuracy on smaller objects.
Answer Bbox Size (S)
Model
Method
ama 11
medium
large
no cropping
12.13
19.57
36.32
BLIP-2 (FlanT5xx.)
human-CROP
55.76
52.02
45.73
no cropping
21.79
30.58
45.30
InstructBIIIP (Vicunaa-7)
numan-CROP
69.60
61.56
53.39
no cropping
39.38
47.74
50.65
LLaVA-1.5 (Vicuna-7B)
human -COP
69.95
65.36
56.96
no cropping 56.42
65.09
68.60
Qwen-VL (Qwen-7B)
human human-CROP CRRP
70.35
75,49
71.05
no cropping 65.76
72.81
69.17
GPT-4o
human-CROF 75.63
81.32
71.72
3

---

Published as a conference paper at ICLR 2025
significant number of questions about small visual concepts, and textual answers are harder for MLLMs to guess from other side information in the image (compared to object types and attributes).
Sensitivity Study. If model's perception is not sensitive to the size of visual concepts, we expect it to have similar accuracy in all three partitions. In Tab. 1, we observe that the accuracy of all MLLMs declines as the ground-truth bounding box becomes relatively smaller (right to left on the no cropping rows). BLIP-2 and InstructBLIP are not trained on TextVQA (i.e., are zero-shot models), and their accuracy declines by 24 and 23 absolute percentage points between the 1 large and small 1 partitions, respectively. VA and Qwen-VL are trained on the training set of TextVQA, yet, their accuracy also declines by 11 and 12 between the large and small 1 partitions, respectively. Lastly, even the most recent commercial GPT-4o, with an unknown training set that might include all of TextVQA, is suffering from a percentage point decline in accuracy between small and medium partitions. These findings suggest that MILMs have a bias against perceiving smaller visual concepts.
Intervention Study. The perceptual limitation we observed above might be merely correlated with size. To study whether this limitation is causally related to size, we conduct an intervention study where we provide the MLLMs with visually cropped images based on the ground-trutt bounding boxes, denoted as human-CROP. More specifically, for each image-question pair and each MLLM, we crop the smallest square-shaped region containing the ground ground-truth bounding box from the image, and resize it to the input image resolution of the MLLM (the square-shapeds cropping prevents extreme deformation of the cropped image when resizing). The cropped image is then provided to the MILM in addition to the original image-question pair (see more details in Fig. 4). We observe in Tab. 1 that human -RomannCRPPPP-- significantly improves the accuracy of all MLLMS on the small 1 and medi um partitions, and lesser extent on the large partition. These findings show that the perception limitation is indeed caused by the size of the visual concepts, and that visual cropping can be a promising direction to mitigate this limitation.
Do MLLMS KNOW WHERE TO LOOK?
The limitation in perceiving small visual concepts can have two primary reasons: 1) they are hard to locate in the larger image, and 2) their small details are hard to perceive correctly. In Fig. 1, we observed that the MLLM's incorrect answer may contain partially correct information, hinting that quantitatively study that observation
it might know where to look in the image. In this section, to answer whether MLLMs' sensitivity to size is rooted in a perception limitation or a localization limitation. To that end, we first utilize the attention maps computed inside the Transformer layers of an MLLM to quantify its spatial attention over the image and then compare the total amount of this attention inside the ground-trutt bounding box to other bounding boxes of the same size.
MLLMs' Setup. The considered MILMS process a given image-question pair (It, q) in four steps (depicted in Fig. 4): 1) the image is divided into N x N non-overlapping patches and processed by the ViT image encoder into N x N output tokens; 2) the ViT output tokens are transformed into Transformer connector (BLIP-2, InstructBLII and Qwen- VL) VL)-which we refer to as image tokens; 3) the image tokens are then prepended to the question tokens and a predefined starting answer token, and fed to the LLM; 4) the LLM is sampled regressegryy following the starting answer token (we use greedy sampling).
the input space of the backbone LLM- either an MLP (LLaVA-1.5)
Quantifying MILMs' Spatial Attention over the Image. We first measure how important each image token is to the MILM's decision (answer-to-tone tooken attention) by extracting the softmax cross- attention of the starting answer token to all image tokens in all layers of the backhone L.M,, resulting in Agr (z, 9) E I ILLHHxIXX where L, H are the number of layers and heads -eee--eer- in the LLM, and T is the number of image tokens provided to the LLM. We then take the average over attention heads to arrive at the answer-to-token attention Ast(r. q) मे Σ Ast(a, q). Next, we measure how important each image region is to each image token (token-to tepeen-to-mmaae to-image attention). For the MLLMs that use a Transformer connector to resample VIT output tokens into a fixed number of image tokens (BLIP-2, InstructBIIII and Qwen-VI.) we extract the softmax cross- attention of each image token to where Le, He
all ViT output tokens in all layers of the connector, resulting in Ars E R H,xxx.xxxxx in the connector, T the number of learnable query
the number of layers and heads-per-layy tokens (that become input image tokens to the LLM), and 2 the number of image patches of the ViT image encoder. We then take the average over attention heads to arrive at the token-to-ine io-ag attention
4

---

Published as a conference paper at ICLR 2025
DEDICATE 94408-971
LLaVA-1.5
What number foatball plyer?
Q: What number cal?? 202- 002-553-2000
G: the color bicyde? (A) blue (B)
white siver (D)
EXIT 225
InstructBUp
number need
the number?
Q: there car image?
Figure Examples MILMS knowing where to look despite answering incorrectly. The right panel in each example displays relative attention to the image (defined in Sec. 4) of one layer in the MLLM.
A=(x)= = = EK (r).).. For LLaVA-1.5 which uses an MLP to transform ViT output tokens to image tokens, we set (n(x) to the identity tensor. Finally, we compute the answer-to-inage attention by computing the tensor product of the answer-to-token and token-to-image attention, resulting in (z, q) R Lexxxx where A (a, q) Ám (x) (superscripts m and & denote layer indices on the LLM and the connector, respectively).
Relative Attention. One issue with using the softmax cross-altention is that all highly attended tokens are semantically relevant to the input question. For example, recent work has observed that Transformers may use several tokens as registers aggregate global information (Darcet al., 2023). To emphasize semantically relevant attention, we propose to normalize the answer-to-imale -oae- attention of an image-question pair (Ir, 4) by its value on a generic instruction g'. Specifically, we consider a fixed instruction g' =Wrrite a general description of the image.", and compute relative attention as (2, q) (arqg))
Asc(zty) under element-wise division. Fig. 2 shows examples of relative attention for
LLaV--1.5 and InstructBLIP (A layers 14,
and 15, k respectively).
MLLMs Know Where to Look? Equipped with relative attention, we now return to our question of whether MILMs have localization limitation perception limitation. To that end, we consider the validation set of TextVQA again. For each image-question pair, we first compute the relative attention. We then define attention ratio the ratio of the total (sum) relative attention inside the answer ground-truth bounding box to its average across all bounding boxes of the same size
BLIP-2 (FlanT5x.)
InstructBLIP (Vicuna-7B)
3.0
Ostention Rabio
1.3
Attention Ragio
1.2
2.0
1.1
1.5
W/h
0.9
24
64
160
fn
LLaVA-1.5 1.5 (Vicuna-78)
Qwen-VL (Qwen- 7B)
Attention Patio
Correctly Answered Incorrectly Arswered
Attantion
16
24
12
20
32
Layer
in Layer
Figure MLLMs' attention ratio across all layers (average with 95% CI over TextVQA). The attention ratio measures how significantly the MLLM is attending to the ground-truth bounding box (defined in Sec. 4). We observe that it greater than I most layers, showing that the MLLMs know where to look in the image when they fail to answer correctly.
5

---

Published as a conference paper at ICLR 2025
as the ground-truti truth the image. This ratio provides measure of how significantly the MLLM is attending to the ground-truth bounding box (in the sense of Markov's s inequality). In Fig. 3, we plot the average (with 95% confidence interval) of the attention ratio, over the validation set of TextVQA for all layers in the considered MI.MMs The horizontal axis shows the combined layer index 7 + l x L for m E {0... .0.... L. }} spanning the number of cross-attention layers in the backbone LLM, and & E {0... .... Le - }} spanning the number of cross-attention layers in the connector (BLIP-2:
24, Le 6; InstructBLIP: L 32, Le = Qwen-VL: L 32, Le LLaVA-1.5: L = 32, Le 1). In all MILMs, we observe significantly larger than 1 attention ratio most layers, suggesting that the models are attending significantly to the ground-truthh bounding box region on the image. Intriguingly, the models show similarly strong attention to the correct region regardless whether they can answer the question correctly or incorrectly. These observations show that the MLLMs tend to know where to look, even when they answer incorrectly.
5 AUTOMATIC VISUAL CROPPING (VICROP)
We observed in Sec. 4 that the sensitivity of MLLMs to visual concept size is primarily perception limitation (rather than a localization limitation). Therefore, one solution to mitigate this limitation to simply train MLLMs with a larger number of image patches while maintaining per-patch resolution (hence increasing the image resolution of MLLMs). However, increasing the input image resolution by factor of a, increases the number of VIT input patches (and output tokens) from N2 to a N², which in turn increases the softmax attention computation complexity on the order of a e'N Given that this solution is not scalable for current Transformer-base based MILMs,, alternative solution that does not require any training and is scalable to any image resolution. We note that several concurrent works have explored the first direction of training MLLMs with higher resolution image patches (Li et al., 2024c; Sun et al., 2024; Li et al., 2024d; McKinzie et al., 2024: Xu al., 2024a; Luo et al., 2024), and notably LLaVA-Nex: (Liu al., 2024a) has achieved the VQA state-of-the parallel these efforts in the following sense: rather than training higher and higher resolution MLLMS to enable them to see all resolutions (which is inevitably upper bounded), we explore how to smartly adjust the input image towards what an MLLM already can see without any additional training. We provide evidence showing that our training-frree method can provide orthogonal benefits to the training- based methods in Appendices D and E.
choose explore
in several VQA benchmarks at the time of writing. We believe our work
Inspired by our findings that MLLMs tend know where look (Sec. 4) and that visual cropping can mitigate the perception limitation (Sec. 3), in this section we construct three automatic visual cropping methods in order to mitigate the perception limitation of MLLMs. These methods seek to use the internal information an MILMM itself-in the form of attention maps and gradients-to find the approximate region of interest in images (i.e., the region containing the subject of question), and then to zoom into that region via visual cropping, One potential drawback of visual cropping is that some questions might need to have global view of the image. To address this issue, we utilize the fact that MLLMs typically convert the image into series of tokens. This allows us to directly
LLaVA-1.5
Answer
NxN
Image Encoder
MLP
LLM
NxN
Question
5
Visual Cropping
InstructBLIP
Answer
Image Encoder
Transformer
LLM
NxN
Quesstion
Figure 4: Illustration of the proposed visual cropping approach applied to two MLLMs.

---

Published as a conference paper at ICLR 2025
extend the original image tokens by concatenating the visually cropped image tokens, as illustrated in Fig. 4. We use this concatenation approach when applying all our methods to MLLMs.
Relative Attention ViCrop (zel-att). In this method, we directly compute the relative attention (2, 9) defined in Sec. 4 for each image-question pair (x, q). We then select a target layer (in LLM and connector) based on a small held-out set of samples in TextVQA and use its relative attention as the importance map for visual cropping (discussed below). We ablate on the choice of layer in Sec. 6.
Gradient- -eifiete-W Attention ViCrop (grad-att). The relative attention runs additional generic instruction through the MILM to normalize the answer-to-image attention and emphasize semantically relevant attention. As an alternative that does not require second forward pass, we consider using the gradients to normalize attention, because the gradient of the model's decision with respect to an attention score shows how sensitive the decision semantically relevant the attention is for answering the question. To get differentiable representation of the model's decision, we consider the logarithm of the maximum output probability at the starting answer token, e = log softmax q)):- E R, where E Rº' is the output logit of the LLM at the starting answer position, D the vocabulary size, and t* = arg max; 2t Then, following our notation in Sec. 4, we can compute the gradient- weighted versions of answer-to-token attention Azc q) = Ase (z, q) @ A., v(x,)))) 4)) and token-to-image attention Ass(2, q) = As( O AV.... v(2 9)). where is element-wise product and (ww) max(0, w). We remove negative gradients because they correspond to tokens that if attended to will reduce the model certainty, hence often distracting locations Selvaraju al. (2017). Finally, we compute the gradient-weighted answer-to-image attention as the following tensor product (s( q) = Ast (z, q) @ An, q) E REXL<XIXN?, We will select the same target layer identified in rel-att from Az,(z, 9) as the importance map for visual cropping.
changes in that attention, hence how
Input Gradient ViCrop (pure-grad), In this method, we seek to find the relevant regions on the image directly using the gradient of the MILM's decision with respect to the input image. Compared to the previous attention based methods, pure-grad is more versatile because it does not rely on the Transformer-basee
architecture. Specifically, for each image-question pair (I. q), will compute G(x, q) = V (x, 1)1l2, where v(x,4) q) is the logarithm of the maximum output probability of the LLM at the starting answer token (as defined in grad att above), and the L.2-norm is taken over the image channel dimension. However, gradients sometimes show high values in entirely constant-color regions (e.g.. blue skies). Given that these non-edge regions do not contain any visual details, we will explicitly diminish them in G. To that end, we first apply a 3 x 3-size Gaussian high-pass filter to the image, followed by a median filter of the same size to reduce salt-and -ee-pe noise. The resulting filtered image is then thresholded at its spatial median value to become a binary mask and is element-wise multiplied by G. Finally, the edge-emphasized G is spatially verage-pooled into the N x N patches of the MILM to become an importance map for visual cropping.
Bounding Box Selection for Visual Cropping. To convert the importance map (from each of the methods described above) to a bounding box, we use sliding windows of different sizes inspired by object detection literature Redmon et al. (2016). Specifically, for each MLLM, we define a set of windows with sizes equal to multiple of the input image resolution of the MLLM. The multiples in 11,1.2.... 2}. We slide each window over the image with a stride of 1 and find its best position where the sum of the importance map inside the window maximized. After selecting the best position per window, we choose the window whose internal sum has the largest difference from the average internal sum of its adjacent positions. This latter step a heuristic to avoid choosing too small or too large windows (notice that in both cases, moving the window slightly left/right or up/down will not change its internal sum significantlly. The chosen window then cropped from the image, resized to the input image resolution of the MILM,, and provided to the MI.LM in addition to the image-question pair.
High-Resolution Visual Cropping. In one of the benchmarks we consider in this work, V* Wu and Xie (2023), the images are of very high resolution (always more than IK) and consequently, the resized input image provided to the MLLM might completely lose the visual concept of interest for question. To mitigate this, on this benchmark, we employ a two-stage strategy. In the first stage, we divide images into non non-overlapping blocks of smaller than 1024 x 1024 with aspect ratio close to 1, compute the importance map separately for each block according to the ViCrop methods, and then re-attach the blocks back together. In the second stage, we find the bounding box for visual
7

---

Published as a conference paper at ICLR 2025
LIBE
LIBROS
Q: What is the color of the clock? (A) black (B) yellow (C) green (D) red
à LLaVA 1.5: A
P LLaVA 1.5 (wl ViCrop):
N
cn
Wh last on the lady pointing at?
What is name of player?
A LLaVA 1.5:
è LLaVA 1.5 (w/ ViCrop): Use numbers
InstruciBLIP Rudolph
InstruciBLIP (w/ ViCrop): Holland
Figure 5: Examples of -att helping MLLMS correct their mistakes (cyan-colored bounding box shows cropped region by rel ta zoom-in insets are displayed for better readability).
cropping on this re-attached importance map exactly as described before and provide the original image-question pair with the resized cropped image to the MLLM.
6 VICROP METHOD ANALYSIS
In this section, we apply our proposed visual cropping methods to two open-source SOTA MLLMs, InstructBLIP (Vicuna-7B) (Dai et al., 2023) and LLaVA-1.5 (Vicuna-7B) (Liu et al., 2023a). We evaluate their effectiveness in improving the perception of smaller visual concepts on 4 detail- sensitive datasets (TextVQA 2 (Singh et al., 2019), V (Wu and Xie, 2023), POPE (Li al., 2023c), VOA (Mathew et al., 2021)). and their ability to maintain performance on larger visual concepts in general-purpose datasets containing mostly large objects (GQA (Hudson and Manning, 2019), AOKVQA (Schwenk et al., 2022), VQAv2 (Goyal et al., 2017)). InstructBLIP uses the hyper- parameters
16, m = 15, A = and input image resolution of 224 x 224. LLaVA-1.5 uses N 24, 14 and input image resolution of 336 336. When reporting accuracy, compute VQA-score for all benchmarks except GQA. For GQA, we compute accuracy using the official code.". See Appendices A to C for further details about implementation, datasets, and prompts.
ViCrop Improves VQA Accuracy. In Fig. 5, we show a few examples of the ViCrop helping the MILLM correct itself (more examples in Appendix G), and in Tab. 2, we report the accuracy of the proposed ViCrop methods on the VQA benchmarks. We observe that all methods significantly improve the accuracy of the original MLLMs (no cropping) on detail-sensitive benchmarks, without requiring any training, while maintaining the MLLMs' performance on benchmarks with larger visual concepts. Thus, the accuracy gain on fine details (most notably in TextVQA and V*) does not seem to come at the cost of accuracy on larger visual details and relations. We also observe that the accuracy gain for L.a more substantial than for InstructBIIIP This can be explained by the
In TextVQA evaluation, do not provide extemally extracted OCR tokens the MLLMM since want measure its true perception, this differs from the setup the original paper. See more discussion in Appendix A. http:/uuullggoogglvluuaainnooooa thups://cs. wttp:/csssstaaood.ccoooooo. wttp://///moccooooooooo/oooaaaae
8

---

Published as a conference paper at ICLR 2025
Table Accuracy of the proposed ViCrop methods on visual question answering benchmarks.
Smaller Visual Concepts
Larger Visual Concepts
Model
TextVQAT
Ve
POPE
DocVQA
AOKVQA
GQA
VQAv2
no cropping
47.80
42.41
85.27
15.97
59.01
60.48
75.57
rel rel-att
55.11
62.30
87.25
19.63
60.66
60.97
76,51
LLaVA-1.5
grad-att
56.06
57.07
87.03
19.84
59.94
60.98
76.06
pure-grad
51.67
46.07
86.06
17.70
59.92
60.54
75.94
no cropping
33.48
35.60
84.89
9.20
60.06
49.41
76.25
rel-att 45.44
42.41
86.64
9.95
61.28
49.75
76.84
InstructBLIP
grad-att 45.71
37.70
86.99
10.81
61.77
50.33
76.08
pure-grad
42.23
37.17
86.84
8.99
61.60
50.08
76.71
Table 3: Ablation study on the choice of layer and the of high-resolution visual cropping.
Choice of Layer
High-Resolution ViCrop
Model
Selective
Average
High-Res
w/o High-Rec Res
cropping
47.80
42.41
42.41
rel-att 55.17
55.45 56.26
+0.28
62.30
47.64
-14.66
LLaVA-1.5
grad-att
56.06
+0.20
57.07
49.74
-7.33
pure-grad
51.67
46.07
45.03
-1.04
no cropping
33.48
35.60
35.60
rel-att 45.44
44.40
-1.04
42.41
38.74
-3.67
InstructBLIP
grad-att 45.71
44.98
-0.73
37.70
42.41
+4.71
pure-grad 42.23
37.17
42.41
+5.24
fact that InstructBLIP only trains its connector and not its backbone LLM during tuning- the LLM does not adapt to use the image tokens, rather the image tokens are adapted to optimally prompt the LLM-and therefore the LLM cannot effectively use the additional (cropped) image tokens provided through visual cropping. Nonetheless, the results show that ViCrop can be effectively applied to different MLLMs, and is promising enference-timas solution for mitigating the perception limitation observed in Sec.
Ablation Study on the Choice of Layer. To understand the importance of the choice of an informative layer for rel-att and grad-att (as discussed in Sec. 5), Tab. 3 we compare the accuracy of these methods when simply taking the average of all layers in Arel and Agi respectively, on TextVQA. We observe that rel-att is robust to this choice and grad-at declines about 3.5 percentage points in accuracy. Importantly, both methods still improve the MLLMs' accuracy even when using the layer average, suggesting that averaging is suitable choice in the absence of any data for selecting a layer.
Ablation Study on the Hiigh-Resolution ViCrop. In Sec. proposed two-stage strategy for processing the very high-resolution images in the V benchmark. To see how effective this strategy is, in Tab. compare the accuracy of ViCrop methods with and without this high-resolution strategy on V*. We observe that while this strategy is very beneficial to LLaVA-1.5. it declines the performance grad-at and pure-grad for InstructBLIP. However, all methods, with and without this strategy, still improve the MILMs* accuracy.
ViCrop with External Tools. In addition to the internal ViCrop methods, we also considered the use of external off-the-sheii models to find the region of interest in image for visual cropping. Specifically, we utilized SAM (Kirillov et al., 2023), YOLO (Redmon et al., 2016), and CLIP (Radford et al., 2021) to find the most relevant part of an image
given question (details of these external ViCrop methods are provided in Appendix F). In Tab. 4, we compare the accuracy of external ViCrop methods to the internal methods TextVQA. While external models also effective in improving the accuracy of MLLMs, they are weaker than all the proposed internal ViCrop methods, thus we did not explore them further.

---

Published as a conference paper at ICLR 2025
Table 4: Accuracy of ViCrop using external tools compared attention/gradient (on TextVQA); and the inference time overhead of ViCrop methods (in seconds). Original's 's time is per answer token.
Model
Original
SAM
YOLO
CLIP
rel-att grad-ast
pure-grad
Accuracy
LLaVA-1.5
47.80
49.42
48.84
48.55
55.17
56.06
51.67
(TextVQA)
InstructBLIP
33.48
39.23
36.49
39.61
45.44
45.71
42.23
LLaVA-1.5
2.26
14.43
11.33
29.86
CPU Time
91.53
0.97
5.46
InstructBLIP
0.66
4.35
3.78
7.04
LLaVA-1.5 Instruc(BLIP
0.17
1.16
0.89
2.36
GPU Time
3.33
0.35
1.07
0.06
0.28
0.29
0.60
Inference Time Overhead. In Tab, 4, we report the average inference-time overhead of the proposed visual cropping methods on GPU (NVIDIA RTX A6000) and CPU (Intel(R) Gold 5317 CPU @ 3.00GHz) and compare with the per-answer-token processing time of the MI.MMs. We see that all proposed methods (except SAM) are reasonably fast (1 to 2 seconds overhead on GPU). For example, computing the visual cropping with rel rel-att takes the time of generating only 5 tokens by the MLLM. Note that our methods' time overhead will not scale with the number of answer tokens and is constant regardless of how long the answer is because our external methods do not need any answer token, and internal methods only need the starting answer token (see Sec. 5). In contrast, MLLMs' inference time scales approximately linearly with the number of answer tokens.
CONCLUSION
In this work, we qualitatively and quantitatively showed that there exists a perception bias against small visual details in widely-used MI..Ms. Then we found that MLLMS often know where to look even if they fail to answer the question, indicating that the bias toward small visual details is rooted in a perception limitation rather than localization limitation. To mitigate this limitation, we proposed multiple automatic visual localization methods as scalable and training -rreinnggeeeeeeeeeee solutions based on models' internal dynamics while answering the visual questions. Through evaluation of multiple multimodal benchmarks, we showed that our method can significantly improve MLLMs' accuracy without requiring any training, especially in detail-sensitive scenarios. Our findings suggest that MLLMs should be used with caution detail-sensitive applications, and that visual cropping/localiiation with the model's own knowledge is a promising direction to enhance their performance.
Limitations and Future Work. The proposed ViCrop methods do not enhance all types of questions equally. We have observed that questions concerning relations and counting are particularly difficult for ViCrop methods to help answer. This is expected as the proposed ViCrop can only focus on one region in the image. We leave extending ViCrop to focus multiple regions simultaneously for future work. Another limitation of the proposed methods is their time overhead and the addition of visual tokens. While the overhead is reasonable (a few seconds), believe it can be significantly optimized as an inference- time mechanism, for example by utilizing lower precision, and weight quantization. Furthermore, Matryoshka Query Transformer (MQT) (Hu et al., 2024) enables MLLMs to have varying visual context size during inference. In our current results, we have shown that our methods can work with two different MILMSs with distinct visual context sizes, so seems entirely possible that our method can still work with varying visual context size under MQT, which can further reduce our computational cost through rescaling the cropped image. We leave these inference cost optimizations to future works. Lastly, we have observed that the proposed methods tend to have some complementary benefits, and therefore exploring ways to combine them (for example based on the prediction uncertainty) is also very interesting direction for future research.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We thank Jinyi Hu and Joe Mathai for their very useful insights. We also express our gratitude to anonymous reviewers for their valuable feedback. This research was supported in part by the National Science Foundation under Contract No. IIS-2153546.
REFERENCES
Anthropic. The claude 3 model family: Opus, Sonnet, Haiku, March 2024. URL https: // www- edn. anthropic com/ de (at.0000.......aafaf333300000000 Model [bodel_Card_Claudee pddf.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang. Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint 220i:::308.19666 2023.
Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. SpatialvIm: Endowing aision-language models with spatial reasoning capabilities. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14455- 14465, June 2024.
Liang Chen, Lei Li, Haozhe Zhao, Yifan Song, and Vinci. R1-v: Reinforcing super generalization ability in vision-language models with less than $3. https: /gitttb..tt..g wtw.ttttb.oooooooooo Deep- Agent/ R1-V, 2025. Accessed: 2025-02-02.
Prateek Chhikara, Dhiraj Chaurasia, Yifan Jiang, Omkar Masur, and Filip Ilievski. Fire: Food image to recipe generation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 8184 8194, 2024.
Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying vision- aaddo--ad-language tasks via text generation. In International Conference Machine Learning, pages 1931-1942. PMLR, 2021.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towaard general- purpose vision-language models with instruction tuning, 2023.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. arXiv preprint arXiv:2309. 16588, 2023.
Mostafa Dehghani, Basil Mustafa, Josip Djolonga, Jonathan Heek, Matthias Minderer, Mathilde Caron, Andreas Steiner, Joan Puigcerver, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Patch pack: Navit, vision transformer for any aspect ratio and resolution. Advances in Neural Information Processing Systems, 36:2252-2211 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
Jiahui Gao, Renjie Pi, Jipeng Zhang, Jiacheng Ye, Wanjun Zhong, Yufei Wang, Lanqing Hong, Jianhua Han, Hang Xu, Zhenguo Li, et al. G-llava: Solving geometric problem with multi-modal large language model. arXiv preprint arXiv:2312.113770 2023.
Yash Goyal, Tejas Khot, Douglas Summers-Star Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 6904-69133 2017.
Jiaxian Guo, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Boyang Li, Dacheng Tao, and Steven Hoi. From images to textual prompts: Zero-shot visual question answering with frozen large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10867-10877, 2023.

---

Published as a conference paper at ICLR 2025
Tammay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 14953-14998 1493314962.22 2023.
Wenbo Hu, Zi-Yi Dou, Liunian Harold Li, Amita Kamath, Nanyun Peng, and Kai- Kai Chang. Matryoshka query transformer for large rision-language models. arXiv preprint arXiv:2405.19315 2024.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 6700-6709, 2019.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, al. Gpt-4o system card. arXiv preprint arXiv:2410.21276 2024.
Glenn Jocher, Ayush Chaurasia, and Jing Qiu. YOLO by Ultralytics. January 2023. URL https: //github. com/ www.taatticcaaaoeoooao huwss//uutaaaatitsesaaaaaa
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et Segment anything. arXiv preprint arXiv:2304.02643, 2023.
Bo Li, Yuanhan Zhang. Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408. 03326, 2024a.
Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training large anguagg-andd-visin assistant for biomedicine in one day. arXiv preprint arXiv:2306.0053 2023a.
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694-9705, 2021.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. In International Conference on Machine Learning, pages 2888-12900 PMLR, 2022a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023b.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-imas image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10965-10975, 2022b.
Xiang Li, Cristina Mata, Jongwoo Park, Kumara Kahatapitiya, Yoo Sung Jang, Jinghuan Shang, Kanchana Ranasinghe, Ryan Burgert, Mu Cai. Yong Jae Lee, et al. Llara: Supercharging robot learning data for vision-language policy. arXiv preprint arXiv:2406.20000
2024b.
Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403. 18814, 2024c.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji- Ji-Rong Wen. Evaluating object hallucination large lingual models. arXiv preprint arXiv:2305. 10355, 2023c.
Zhang Li. Biao Yang. Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi-modal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26763-26773, 2024d.
12

---

Published as a conference paper at ICLR 2025
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings. Part V 13, pages 740-755. Springer, 2014.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.0377 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint 3rXiv:2304.08444 2023b.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, oer, and world knowledge, January 2024a. URL https:
//llava- w.l.laaaall.giaaaa...... wig.....u..... wtw.th...oooo. 100lo////2///oo///222/222 -014--0---1 1001-30-110 alava-
Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang. Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang, Hang Su, Jun Zhu, et al. Llava-plus: Learning use tools for creating multimodal agents. In European Conference on Computer Vision, pages 126-142. Springer, 2024b.
Gen Luo, Yiyi Zhou, Yuxin Zhang, Xiawu Zheng, Xiaoshuai Sun, and Rongrong Ji. Feast your eyes: Mixture-of-resolution adaptation for multimodal large language models. arXiv preprint arXiv:2403.03003, 2024.
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200-2209, 2021.
Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, et al. Mml: Methods, analysis & insights from multimodal IIm pre-training, arXiv preprint arXiv:2403.090 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages PMLR, 2021.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 779-788, 2016.
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In Computer Vision- ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part VIII, pages 146-162. Springer. 2022.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localiza- tion. In Proceedings of the IEEE international conference computer vision, pages 2017.
Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang. Yu Liu, and Hongsheng Li. Visual cot: Unleashing tfain-of-thouuht reasoning in multi-modal language models, 2024.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317-8326, 2019.
Hai-Long Sun, Da- Wei Zhou, Yang Li, Shiyin Lu, Chao Yi, Qing-Gur Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, De-Chuan Zhan, al. Parrot: Multilingual visual instruction tuning. arXiv preprint arXiv:2406.025399 2024.
Dídac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. arXiv preprint arXiv:2303.08/22 028, 2023.
13

---

Published as a conference paper at ICLR 2025
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang. et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXhv:2403.05530, 2024.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. 5: Scaling reinforcement learning with Ilms. arXiv preprint arXiv:2501.
2025.
Anthony Meng Huat Tiong Junnan Li, Boyang Li, Silvio Savarese, and Steven CH Hoi. Plug-and- play vqa: Zero-shot vqa by conjoining large pretrained models with zero training. arXiv preprint arXiv:2210.08773. 2022.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-languae model's 's perception of the world at any resolution. arXiv preprint
2024.
Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208. 10442, 2022.
Penghao Wu and Saining Xie. v*; Guided visual search core mechanism in multimodal Ilms. arXiv preprint Stt::2312.11133.. 2023.
Ruyi Xu, Yuan Yao, Zonghao Guo, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, Maosong Sun, and Gao Huang. Llava-uhd: Imm perceiving any aspect ratio and high-resolution images. arXiv preprint 2403.:2003.11003 2024a.
Zhenhua Xu, Yujia Zhang, Enze Xie, Zhen Zhao, Yong Guo, Kwan- Yee K Wong, Zhenguo Li, and Hengshuang Zhao. Drivegpt4: Interpretable end-to-end autonomous driving large language model. IEEE Robotics and Automation Letters, 2024b.
Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. Rlhf-v: Towards trustworthy milms via behavior alignment from fine-grained correctional human feedback. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 13807-1381 2024.
Jiarui Zhang, Filip Ilievski, Kaixin Ma, Aravinda Kollaa, Jonathan Francis, and Alessandro Oltramari. A study of situational reasoning for traffic understanding. KDD, 2023a.
Jiarui Zhang, Mahyar Khayatkhoei, Prateek Chhikara, and Filip llievski. Visual cropping improves zero-shot question answering of multimodal large language models. In RO-FoMo: Robusiness of Few-shot and Zero-shot Learning in Large Foundation Models, 2023b.
Jiarui Zhang, Jinyi Hu, Mahyar Khayatkhoei, Filip Ilievski, and Maosong Sun, Exploring perceptual limitation of multimodal large language models. arXiv preprint arXiv:2402.07338 2024a.
Jiarui Zhang, Ollie Liu, Tianyu Yu, Jinyi Hu, and Willie Neiswanger. Euclid: Supercharging multimodal Ilms with synthetic high-fidelity visual descriptions. arXiv preprint arXiv:2412.08777, 2024b.
Renrui Zhang, Xinyu Wei, Dongzhi Jiang, Ziyu Guo, Shicheng Li, Yichi Zhang, Chengzhuo Tong, Jiaming Liu, Aojun Zhou, Bin Wei, al. Mavis: Mathematical visual instruction tuning with an automatic data engine. arXiv preprint arXiv:2407.0873) 2024c.
Tiancheng Zhao, Tianqi Zhang, Mingwei Zhu, Haozhan Shen, Kyusong Lee, Xiaopeng Lu, and Jianwei Yin. Vl-checklist: Evaluating pre-trained vision-language models with objects, attributes and relations. arXiv preprint arXiv:2207.00221, 2022.
14

---

Published as a conference paper at ICLR 2025
A IMPLEMENTATION DETAILS
We use python 3.10.6, transformers 4.29.1 and torch 2.1.2 for all the experiments. Our environment consists of an Intel(R) Gold 5317 CPU @ 3.00GHz with 48 cores and 756 GB of RAM, and utilize NVIDIA RTX A6000 GPUs for our experiments. We use the huggingface implementations of all studied MLLMs with the recommended typer-parameters according the respective papers. For GPT-4o, we use the official public API, which is available at the time of submission.
Regarding the evaluation setting of the Tex dataset in Tab. our setting slightly different from the one used by the LLaVA-1.5 original paper Liu et al. (2023a). They report accuracy on Tex by using externally extracted OCR tokens to enrich its text prompt. This is text- specific trick that essentially out-sources the perception of text to an external OCR model. This text- specific trick is not mentioned in their paper or supplementary material, but see their clarification in response to GitHub issue here: https: /tt:://gittttl
www/hamc/ccooc an aannnluu/aae /illLLLLLA. LLaVA/ / / suues/ 176377767777 93411 In contrast, we treat Tex the same as any other vision
515tis seriissuuccmmmem dataset in our experiments, that is, we do not provide any OCR extracted tokens to MLLMs when applying them to TextVQA (only image and question, in the evaluation prompt format specified in their respective papers). This results in slightly lower accuracy compared to the one reported in the original paper, but instead, this number shows the true perception ability of LLaVA-1.5 on also
TextVQA, not confounded by the ability of external OCR model. For completeness, measured TextVQA accuracy in the presence of OCR tokens, which results in 59.8 for LLaVA-1.5 without any visual cropping, and 63.95 with rel- sel-att, showing that our proposed visual cropping can still be beneficial even when OCR tokens are provided to the MLLM.
B DATASET STATISTICS
In this section, we present the details of the datasets used for evaluation in this paper. We report the average height and weight of the images in the dataset. We also report the number of images and questions in each dataset. For VQAv2, we run our experiment on a random 50K subset of the official validation set. We the entire validation set in all other datasets.
Table 5: Average width (W) and height (H) of images, number of images, and number of questions on all datasets.
V*
DoA
TextVQa
POPE
AOKVQA
GQA
VQAv2
W 2246
1776
954
584
581
578
577
H
1582
2084
818
478
480
482
485
# Images
191
1286
3166
500
1122
398
14206
# Questions
191
5349
5000
8910
1145
10781
50000
For our analysis presented in Table and Figure 3, we focused on TextVQA dataset, which includes bounding box annotations for OCR-detected text within images. However, this dataset does not specify which bounding boxes correspond to the regions where answers are located, necessitating a manual annotation process. The TextVQA dataset comprises 5000 questions and 3166 images. We manually annotated these question-ing -mmage pairs, ensuring accurate bounding boxes over all the regions of interest where the answers could be found. This manual annotation process was essential for our analysis, allowing us to provide precise and reliable ground-truth data for the study. Given that some questions were associated with multiple bounding boxes in their corresponding images, we undertook a filtering process to isolate the question-imet of 4370 -imaie pairs, where there is only one instance of the subject of the question in the image. For example, if the question is "what type of drink is sold here?" and there are two different cans of drinks in the image, remove this image-question pair.
pairs. This effort resulted in a refined set
15

---

Published as a conference paper at ICLR 2025
C PROMPT FORMAT FOR ZERO-SHOT INFERENCE
In this section, we provide details about the prompt format used in models for zero-shot inference. We different prompt format for LLaVA and InstructBLIP which we adapt from the original papers, as shown below.
LLaVA-1.5
<image> USER: (question) Answer the question using single word or phrase. ASSISTANT:
InstructBLIP
<image> Question. (question) Short Answer:
D ORTHOGONAL BENEFITS TO LLAVA-NEXT
We apply our proposed rel -at visual cropping method an additional newer MLLM LLaVA- NeXT (Liu et al., 2024a) current SOTA in several VQA benchmarks that has support for higher- resolution compared to LLaVA-1.5. performance, without requiring any training. This provides further evidence for the generalizability of our proposed visual cropping and its orthogonal benefits to training MILMs with higher image patch resolution.
Tab. 6, we observe that our method can still boost the MLLM's
Table Orthogonal benefits of visual cropping when applied to LLaV-NeXT that trained adapt to processing high-resolution images.
Model
TextVQA
V*
LLaVA-NeXX (Mistral-7B)
65.17
58.11
LLaVA-NeXT (Mistral-7B) + rel-att
68.65
61.78
E COMPARISON WITH THE V METHOD (SEAL)
The V* method (SEAL) (Wu and Xie, 2023) proposes multi-agen fine-tuning approach to enhance ability of an underlying MLLM to answer questions about small visual concepts. However, SEAL requires substantial training and finetuning of several neural networks, whereas our methods are completely training-fie free, so a direct comparison would not be fair. Nonetheless, to provide an idea of how method compares to SEAL in an "as-is" fashion (i.e. if user just wants to pick one method as-is off-the-shelf), we report the accuracy of SEAL compared to L.L 1.5+eel -att in Tab. 7. We observe that method outperforms SEAL except the V* benchmark. We think this might be because SEAL is designed and tuned specifically toward high-resolution images in its V* benchmark. We also note that the inference time of SEAL is slower than our method (4.44s compared 1.88s on average per question, tested on the same random 100 Tex samples with one A6000 GPU). That being said, we note that our methods and SEAL can both help enhance MLLMs, and our methods can be integrated into SEAL or other multi- pipelines.
Table 7: Performance comparison between our rel latt applied on I.I. VA-1.5 and SEAL (Wu and Xie, 2023) across multiple vision-language benchmarks.
Model
TextVQA
V*
POPE
DocVQA
AOKVQA
VQAV2 60.66
GQA
SEAL
36.30
75.30
82.40
5.31 19.63
55.34
50.18 60.97
65.35
LLa LLaVA--.erre-aat
55.17
62.30
87.25
76.29
16

---

Published as a conference paper at ICLR 2025
F EXTERNAL TOOLS VICROP
In this section, we present three automatic question-guided localization methods based on popular off-the-shelf vision-based models, namely CLIP Radford et al. (2021), YOLO Redmon al. (2016), and SAM Kirillov et al. (2023). These three methods utilize external vision-based knowledge for the localization process through multimodal encoding, object detection, and semantic segmentation, respectively, See Tab. 4 for their results compared to internal ViCrop methods.
CLIP ViCrop. The intuition of this method is to progressively refine the image towards the region of highest relevance to a given question using CLIP Radford et al. (2021). CLIP consists of an image encoder and text encoder, which are trained on large dataset of image-caption pairs to map each image (caption) close to its caption (image) and far from all other captions (images). The result is an aligned shared space where various images can be directly compared with various texts. To find the region of interest, given an image-question pair, we first crop the image from the four sides (top, bottom, left, and right) at a cropping ratio of 0.9 to produce four overlapping cropped images. We then use CLIP to assess the semantic similarity between these cropped images and the question. The highest-scoring crop is chosen as the input for the next iteration. This process is repeated for 20 iterations, and the cropped image with the highest CLIP similarity to the question is selected for visual cropping.
YOLO ViCrop. Instead of a progressive approach finding the region of interest, in this method we select candidate regions based on a state-of-the-arr object detection method: YOLOv8 (Jocher et al., 2023) pretrained on COCO Lin al. (2014). Using YOLO, we filter out regions that contain no salient objects i.e., regions for which CLIP could mistakenly assign high similarity. More concretely, for each question-image pair, we first use YOLO to collect bounding boxes for all predicted objects with confidence higher than 0.25 (the recommended default). Then, for each predicted bounding box, we crop its corresponding image and compute its similarity to the question using CLIP. Finally, the bounding box with the highest similarity score is selected the region of interest for visual cropping.
SAM ViCrop. A limitation of YOLO is that it only provides bounding boxes corresponding to fixed number of object classes. To overcome this issue, we use the segment anything model (SAM) Kirillov al. (2023), which has shown state-of-the-atr an extensive set of segmentation masks for each image, thus providing a more granular set of salient candidate regions compared to YOLO. More concretely, for each image-question pair, feed the image into SAM, which provides an extensive set of segmentation masks corresponding to all objects and object parts. Then, we translate these masks into bounding boxes by computing the smallest bounding box that covers each segmentation mask. Finally, the bounding box with the highest CLIP similarity to the question is selected as the region of interest for visual cropping.
shot segmentation performance. SAM can provide
Finally, for each method, we crop the smallest covering square (so that the cropped image is not deformed when resized to the input resolution of the MLLM), and provide it to the MLLM in addition to the original image-question pair (as depicted in Fig. 4).
17

---

Published as a conference paper at ICLR 2025
G ADDITIONAL EXAMPLES ON MODEL'S PREDICTIONS
Q: What the bread of
the dog?
(A] Husky
congi
Daimation
golden
LLaVA 1.5:
LLaVA
ViCrop):
the blue muck or right side of the while validde?
LLaVA 1.E:
LLaVA 1.5 ViCrop):
C: What the color the handbag?
(A] white
red
black
LLAVA 1.5:
LLaVA 1.5 ViCrop):
C: What the color
the parachute??
(A) Blue
(B) yellow
(C) Green
red
LLaVA 1.5:
L 1.5
Figure 6: Success (first 3) and failure (last) examples of LLa (rel-att on the V benchmark (cyan-colored bounding box shows cropped region by rel rel-att; zoom-in insets displayed for better readability).
18

---

Published as a conference paper at ICLR 2025
PERUCTE ENGE
right
d'blllard curope?
LLais 1.5 Parrya
LLaw 1.5 owl ViGrop):
Cottrel
LLaMA 1.5: Vericnes
LLav 1.5 twi ViCroph:
LLWUA 1.5:
LLaM ViCrop):
Riverdogs
MUNCHKIN BRICKS
Last HOURSS
company that gamee
LLaVA
LLaw 1.5 ViCrop):
Postivo
LLaMA 1.5:
LL 1.5 ViDrap):
LLaV 1.5
LLaV ViCropk
brick
sign areswalk say?
Jersey??
country
LLaw 1.5:
LLaV
LLawi Usa
LLaVA 1.5 ViCrop)
10av
LLaVA 1.5 ViCrop):
LLaVA 1.5 (wi ViCrop).
Canada
Tell MARINES
GRAYSOS
what does the bubible lest say the woman?
O: what is alyped in the sear ber?
Q. who picuure
LLaVA
LLavA 7.5 Cherles grayson
ViCrop):
ViCrop):
Buscar
ViCrop):
SAB
house
what the number the paayer in the middle?
what is the whe brand?
Q: what kind exchange shows on the banner?
ViCrop)
ViCrap):
ViCrop)
Figure 7: Success (first 9) and failure (last 6) examples of L on the TextVQA benchmark (cyan-colored bounding box shows cropped region by rel-att).
19

---

Published as a conference paper at ICLR 2025
OUI
0
DU respect
Q what is the name of the group the battom right of poster?
carton?
InstrucBLIP (wr ViCrap):
InstrccBLIn IP: F
T
InstructiLL. JP:
InstrudBLIP (wi ViCropl:
Pubric
sodialittss
InstrucdEl IP: Milk
InstructELL (wf ViCrop)
L. foroed europa
sign ssy
sharing
InstrudBLPP: InstrucBLIP
Pacanii
InstrunBUP
InsirusfBUPP (we ViCrop)
Reatromm
InstrudBLIS InstructLL
Options
very right
right comnrr?
degroe angle drown?
InstrucBBLIP IP ViCropl Newsome
InstrucIBLPP: Shanesseear
InstructBUP:
InstrucBLIP ViCrop):
InstructEL
InstrutBLIP (wl ViCropl:
45
what brand are the guy's shors?
C: who the service provider?
middlee
ViCrop):
InstrucBBPPP (wl ViCrop):
Apple
ViCrop):
Mensil
EARATY
menufackuee
InttrccBBLIP IP: Hardiast
uot
InstrciBLPP.
Ins.uuULLIP ViCrop):
Flextale
Inst UPP ViCrop):
Calffmia
(we ViCrap):
Remington
Figure Success (first 9) and failure (last 6) examples of InstructBLIP (rel-att) on the TextVQA benchmark (cyan-colored bounding box shows cropped region by rel-att).
20


Reviews:
[{"review_id": "Gm4mvoWZ0h", "reviewer": "Reviewer_1vjv", "summary": "This paper studies the perception limitations of Multimodal Large Language Models (MLLMs) when dealing with small visual details in images, and proposes training-free visual cropping methods to mitigate these limitations. The key contributions are: (1) demonstrating that MLLMs struggle with perceiving small visual details despite knowing where to look in images, (2) showing this limitation is causal through intervention studies with visual cropping, and (3) developing automatic visual cropping methods that leverage MLLMs' internal attention and gradient information to improve their perception without any additional training.\nThe paper makes a valuable contribution by rigorously analyzing an important limitation of MLLMs and providing practical solutions. However, there are some points that need clarification and potential improvements that could strengthen the work.", "questions": "* Like mentioned in the first point of  weakness, How will traditional grounding algorithms help address the problem  under similar setting? Why viscrop instead of that? \n\n* How does the method handle cases where the visual subject spans multiple regions of interest?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* This paper propose an interesting problem that MLLM seems know which areas should be focused but still fails to solve the related problem successfully all the time. By conducting intervention study, the problem is validated.\n\n* The proposed methods are intuitive and straightforward to the problem mentioned and make good use of the finding from the pilot study.", "weaknesses": "* The evaluation part is not so well done given the baseline is just no croping without any other fair comparison. V* (star) may be a good baseline since they tried to address a similar problem like this paper brought about. \n\n* The existing approach seems achieve limited improvement by up to ~4% even compared to no-crop setting across all 8 benchmarks.\n\n* The role of model architecture in determining perception limitations isn't explored", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kMgPZFn4wm", "reviewer": "Reviewer_J4zc", "summary": "This paper identifies a drawback of MLLMs in VQA when questions concern small-sized objects in the images. It shows empirically that MLLMs perform worse on such questions, and that their performance is due to perception issues but often they are able to attend correctly to the relevant regions of the image (localization). With these two insights, they propose a method based on visual cropping which computes attention maps of the MLLM through various methods, and uses these attention maps to derive a cropped version of the image which is appended to the original image and passed through the MLLM. They show empirically that on VQA datasets involving fine-grained questions (e.g. TextVQA), this inference-time approach outperforms the base MLLM, while maintaining performance on general VQA datasets.", "questions": "(Sec. 3) Human-crop is a very strong intervention where the crop is tight around the ground-truth object. It would be nice to see how accuracy improves for more realistic crops. For example, one experiment -- crop around objects in the \"small\" dataset s.t. the proportion of its size in the image is equal to the average proportion value in the \"medium\" and \"large\" datasets. Does the accuracy improve, maybe matching the accuracy of the medium and large splits?\n\n(Sec. 4) Is this a standard way to calculate attention in transformers? If I am understanding right -- I am not sure how interpretable attention values are in later layers, where the input tokens have already been transformed significantly. How does this method compare to prior work on attention in MLLMs/vision transformers? Also, it would be nice, if feasible, to evaluate variants of GradCam for transformers as another attention method. These are mentioned in the related work. \n\n(Sec. 4) It'd be interesting to get more intuition on when the MLLM produces correct attention when giving an incorrect answer, and when it does not. I imagine that in the exit or bicycle number case in Fig. 2, it seems reasonable to produce correct attention. In the car question I'm a bit surprised this works since it's just asking about existence of the object -- proper localization seems to imply that the model also perceived the car correctly.\n\n(Sec. 6) A simple baseline here would be to provide a random crop; crop the image at a random location, perhaps half the size of the original image, and provide this as additional input to the MLLM. This would reinforce the analysis in Sec. 4, and show that the accurate localization of the MLLM provides informative crops. It would be nice to show the results of such a baseline.\n\nMinor questions\n- What is the \"input image resolution\" in line 356? Is that the size of the patches input to the MLLM, or the image resolution? Clarifying since the multiples are >= 1.\n- What is the reason for the oscillatory behavior in Fig. 3?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This paper identifies an important problem in MLLMs, specifically their lack of sensitivity to small details in images in a VQA context. Moreover, it pairs this with an original and useful insight -- that often the MLLM can attend correctly to the relevant region of the image, even if it produces the wrong answer. This observation may not have been recognized previously. Moreover they turn this analysis into a practical method which generates an attention map and uses this to crop the image. The method has several advantages -- it is intuitive, fits well into the framework of pretrained MLLMs, and works at inference-time without further training.\n\nEmpirical results on Table 2 are convincing and show that their method significantly outperforms the base MLLM on detail-sensitive datasets. \n\nThe paper is also written well and has good clarity.", "weaknesses": "I wonder whether the drawback of MLLMs for small-sized objects for vision-language tasks has been noted before. For example, some work is mentioned in Lns 287-288 on training MLLMs with higher-resolution patches. It would be good to understand related work that has addressed the problem of small-sized objects for MLLMs generally and/or VQA specifically. This could potentially be added to the related work. \n\nThe method assumes that there is a single relevant location of attention in the image; this is not always true,  for example in questions that ask about spatial relationships between objects. The authors note this when they discuss limitations. It is not inconceivable that the approach could be extended to such questions if attention maps are sufficiently informative.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8YDoCkfvg9", "reviewer": "Reviewer_bmba", "summary": "This paper studies the attention patterns of MLLMs when answering visual questions, and investigate into do MLLMs know where to look at, i.e. the perception problem v.s localization problem. Based on these findings, this paper introduces an automatic visual cropping methods leveraging attention and gradient maps, to help it better perceive the small visual subjects. The proposed methods is evaluated on two MLLMs and seven VQA benchmark and demonstrates significant improvement.", "questions": "1. I'm curious to see, have the authors tried and explore methods to add the cropped visual object to the original image instead of concatenating two images together. Would that bring to on par or better performance with current approach and save the computation cost. \n2. I incline to accept this paper unless I saw critical drawbacks that I missed from other reviewers.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. This paper studied the perception v.s localization problem for small visual objects and introduced insightful findings. \n2. It introduced a training-free method to help MLLMs better perceive the small visual subject of any question.\n3. Experimentally, it demonstrated significant improvement in 7 benchmarks and on two MLLMs.", "weaknesses": "1. To help the model keep the global visual information, the cropped object introduced extra tokens. As illustrated in paper's Table 4, it indeed introduced some computational latency. But I suggest the author could try this approach (Matryoshka Query Transformer for Large Vision-Language Models (NeurIPS 24)) on the cropped object to save the visual tokens, as the re-scaled cropped objects intuitively don't have much visual detailed information. \n2. The improvement on general VQA benchmarks such as large visual concepts are not as significant as on small visual concepts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)