Exploring Perceptual Limitations of Multimodal LLMs on Small Visual Objects
Authors: Jiarui Zhang
Jinyi Hu
Mahyar Khayatkhoei
Filip Ilievski
Maosong Sun
Venue: Accepted by TMLR ()

Abstract: Multimodal Large Language Models (MLLMs) have recently achieved remarkable performance in various multimodal benchmarks. However, general benchmarks often do not reveal the specific aspects of their visual perception limits due to the lack of controllability. In this work, we quantitatively study the perception of small visual objects in several widely-used MLLMs and reveal a pervasive limitation in answering questions about small objects in images. We then conduct a controlled study of MLLMs' perception, using text-reading as a surrogate task for general visual perception to understand how quality, size, distractors, and location of an object can independently affect the ability of MLLMs to perceive it in images. Through this controlled study, we find that lower object quality, smaller object size and the presence of visual distractors can both independently reduce MLLMs' ability to answer visual questions. More surprisingly, even local perturbations of an object by a few pixels can cause a drastic decline in the ability of MLLMs to perceive it. Our study provides a better understanding of the perceptual limitations of MLLMs and contributes new evaluation protocols for analyzing, enhancing perception of future MLLMs.

Published in Transactions on Machine Learning Research (02/2026)
Exploring Perceptual Limitations of Multimodal LLMs on Small Visual Objects
Jiarui Zhang!** Jinyi Hu
Mahyar Khayatkhoei"
Filip Ilievski3
Maosong Sun
University of Southern California, Los Angeles, California, USA TTinghua University, Beijing, China
Vrije Universiteit Amsterdam, Amsterdam, Netherlands
*Equal contribution. $Corrosponding author: jzhang370usc.edu.
Reviewed on OpenReview: http:s///ppppeerrrrrr.ee////orod
Abstract
Multimodal Large Language Models (MLLMs) have recently achieved remarkable performance in various multimodal benchmarks. However, general benchmarks often do not reveal the specific aspects of their visual perception limits due to a lack of controllability. In this work, we quantitatively study the perception of small visual objects in several widely-used MLLMs and reveal a pervasive limitation answering questions about small objects in images. We then conduct a controlled study of MLLMs' perception, using text-reading as a surrogate task for general visual perception, to understand how quality, size, distractors, and location of an object can independently affect the ability of MLLMs to perceive it in images. We find that lower object quality, smaller object size and the presence of visual distractors can independently reduce MLLMs' ability to answer visual questions. More surprisingly, their accuracy sensitive to the location of the object in the image, and even local perturbations of an object by a few pixels can cause a drastic decline the ability of ML.LMs to perceive it. Our study provides a better understanding of the perceptual limitations of MLLMs and contributes new evaluation protocols for analyzing and enhancing perception of future MLLMs. To facilitate further investigations, release our code and data here.
1 Introduction
The development of Multimodal Large Language Models (MLLMs) (OpenAI, 2023; Gemini Team et al., 2023; Liu et al., 2023b; Dai et al., 2023) has significantly broadened the capabilities of Large Language Models (LLMs) (OpenAI, 2023; Touvron et al., 2023), enabling them to navigate and interpret the visual domain. Leveraging pre-trained visual encoders like CLIP-ViT (Dosovitskiy et al., 2021; Radford et al., 2021), MLLMs have extended the powerful textual understanding of LLMs to multimodal scenarios, such as visual question answering (Li al., 2023a), visual conversations (Liu et al., 2023b), non-verbal reasoning (Ahrabian al., 2024), and multimodal in-context learning (Alayrac el. al., 2022; Zhao al., 2023). To serve multimodal agents (Yang et al., 2023; Hong et al., 2023) and accomplish complex embodied tasks (Driess et al., 2023; Mu et al., 2023), MLLMs need to recognize and interpret visual information with different quality, size, and location, including large central objects and small peripheral pieces of text.
Despite the remarkable advancements of current MLLMs, accurately identifying small objects within images seems to remain a challenge. As Fig. shows, the widely-used GPT-4V (OpenAI, 2023) struggles to discern specific details like small textual descriptions. Prior research suggests that increasing the resolution of input images can generally enhance the accuracy of the response to visual questions (Bai al., 2023; Yu al., 2023a). Furthermore, some works (Zhang et al., 2023; Wu & Xie, 2023; Shao et al., 2024; Zhang et al., 2025) have introduced methods for image cropping and visual scarching to aid MLLMs in recognizing finer

---

Published in Transactions on Machine Learning Research (02/2020)
Input Image to GPT4V: Amazon webpage
What is the weight green barbell?
GPT4V: The image does not provide weight the green barbell
When will the watch's coupon end?
09:04
Wrenlng Fich
GPT4V: coupon for the watch ends in 06:04
Figure 1: Failure cases of GPT-4V OpenAI (2023) in perceiving small objects when serving as web agents. Our research studies this perceptual limitation in several recent MLLMs.
details. However, the extent of this limitation and the underlying factors contributing to it have not been systematically studied.
To bridge this gap, we quantitatively study MLLMs' perceptual sensitivity to relative object sizes and identify various visual factors that contribute to this sensitivity. We first conduct a comprehensive experiment with seven widely-used MLLMs on two common visual question-answering datasets, GQA (Hudson & Manning, 2019) and TextVQQ (Singh et al., 2019), and observe a significant performance drop with a decrease in object sizes in both benchmarks, a trend that persists in all MLLMs. We then conduct an extensive controlled experiment to study the independent. effect of four visual factors that can contribute to limitation in perceiving small objects, namely, object quality, object size, object distractors, and object location. For these experiments, Section 4) because these tasks allow us to unambiguously control the aforementioned properties of visual objects. Our controlled experiments seek to answer the following questions:
focus text-reading and object detection MashionMNIST (described in
Do widely-used MLLMs have a bias against perceiving smaller visual objects? (Section 3)
Does object quality (sharpness) and size (number of pixels it occupies) independently impact widely-used MLLMs' ability to sce/read it? (Sections 4.2 and 4.3)
How does the existence of similar objects in the image impact the ability of widely-used MLLM's to sco/read specific target object? (Section 4.4)
How does changing the location of an object in the image impact widely-used MLLM's ability to see/read it? (Section 4.5)
Can slightly moving object (a few pixels vertically horizontally) impact widely-used MLLM's ability to see/read it? (Section 4.5)
The significance of our findings is threefold. First, results suggest that MLLMs should be used with caution, especially when the task relies on accurately identifying small visual details. Second, our findings provide novel insights for developing more reliable MLLMs, especially when dealing with data of lower quality, objects of smaller sizc, various distractors, and specific object positions. Third, we provide a new evaluation protocol for studying future MLLMs. This protocol can be applied, for example, to measure the robustness of an MLLLM response different positions by showing the difference between maximum and minimum performance across different object locations.
2 Related Works
Multimodal Large Language Model. MILMs like GPT-4V (OpenAI, 2023) and Cemini-pro-visiie (Gem- ini Team al., 2023) demonstrate strong capability for visual understanding. MLLMM typically have three
2

---

Published in Transactions on Machine Learning Research (02/2026)
primary components: a vision encoder, a bridge module, and an LLM backbone (Yu et al., 2023a). (1) Vision Encoder: Commonly, MLLLMs utilize CLIP-ViT and its variants (Radford et al., 2021; Dehghani et al., 2023; Zhai et al., 2023) as the vision encoder, which divides the input image into patches and feeds them into Transformer blocks sequentially in a raster-scan order. (2) Bridge Module: The resulting visual features from the vision encoder are then either linearly projected (Liu et al., 2023b) or condensed inlo a fixed-sized representation (Ti et al., 2023a) to align with the textual representation space. (3) LLM Backbone: The transformed visual features are then prepended to the text embedding within the LLM. We consider seven widely used MLLMs in this work. GPT-4V and Demini-pro-viisiee are included proprietary closed-source representatives. Both BLIP-2 (Li et al., 2023a) and InsI (Dai et al., 2023) adopt a Q-Former as the bridge module: BLIP-2 extracts visual features without instruction conditioning, whereas InstructBLIP incorporates instruction signals to achieve instruction-awar feature extraction. LLaVA-1.5 (Lin et al., 2023a) projects the visual feature from ViT into the LLM space with an MLP layer. Qwen- VL-Chat (Bai et al., 2023) chooscs a larger vision encoder ViT-bigG and a one-layer cross-attention module to perceive visual features. Fuyu-8B (Bavishi el al., 2023) uniquely removes the external vision encoder, directly incorporating pixel information into the language decoder. More recent architectures such as Qwen3-VI VL (Bai et al., 2025) introduce any-resolution Vision Transformer (Dehghani et al., 2023), combined with visual token pooling and multimodal positional embeddings, to improve both computational efficiency and visual understanding performance. The training of MLLMs typically undergoes an initial pre-training on extensive image-text datasets such as LAION (Schuhmann et al., 2022), followed by specialized multimodal instruction tuning (Liu et al., 2023b). Enhancements in MLLMs have been pursued through various means, including increasing image resolution (Yu et al., 2023a), scaling data and model size (Wang et al., 2023), extending to multilingual context (Hu et al., 2023), and introducing interleaved data formats (Lin et al., 2023b).
Robustness Analysis of MLLMs. The capabilities of MLLMs have been evaluated using general benchmarks like the traditional VQA benchmark VQAv2 (Antol et al., 2015) and GQA (Hudson & Manning, 2019), alongside newer benchmarks such as MM-Bench (Liu et al., 2023c) and MMMU (Yue et al., 2023). Some works have shown that MLLMs suffer from object hallucination (Li et al., 2023b; Yu et al., 2023b) and a lack of robustness in processing visual details (Zhang et. al., 2023; 2025). The MMVP benchmark (Tong et al., 2024) further highlights these visual shortcomings, particularly emphasizing the discrepancy between the embedding spaces of CLIP and the vision-only self-supervised space of DINOv2. The V* algorithm (Wu & Xie, 2023) offers an innovative approach with its LLM-guided visual search method, specifically largeting the focus on visual details. Our paper builds upon these insights, quantitatively exploring MLLMs' performance in handling visual details.
3 Can MLLMs Perceive Small Objects?
The recent study by Zhang et al. (2023; 2025) suggests that MLLMs face challenges in perceiving small visual details compared to larger ones. Inspired by this research, we conduct an extensive quantitative experiment to study the sensitivity to size of recent widely-used MLLMs on two standard VQA benchmarks. We evaluate the seven representative models on two prominent visual muestion-answering datasets, GQA (Hudson & Manning, 2019) for world objects reasoning and VQA (Singh et. al., 2019) for reading and comprehending texts presented in the real-world image. Both datascts offer the advantage of bounding box annotations, pinpointing areas of interest within images. For GQA, we aggregate bounding boxes encompassing all related objects. For Text VQA, we focus on the bounding box with the highest textual similarity to the ground-truth answer. To facilitate a nuanced assessment, we categorize these datasets into quintiles based on the relative size of the target area. The accuracy is measured via inclusion match (Liu et al., 2023d).
Our findings, depicted in Fig. 2, demonstrate a consistent issue across all models: a marked decline in processing accuracy for smaller visual elements. Such a trend is most notable in BLIP-2, whose performance gap across different quantiles is 16.719 71% and 21.83% on GQA and TCx respectively. In addition, the two leading closed-source API models, GPT-4V and Gemini-pro-vision, have 7.32% and 6.39% performance gap on GQA and 9.05% and 3.31% on TextVQA, respectively, also exhibiting a performance decline.
Given these observations, the next natural question is to explore what visual factors may contribute to the observed difficulty of perceiving small objects. The most immediate factor is of course quality, since smaller
3

---

Published in Transactions on Machine Learning Research (02/2026)
BUP-2 (FlanTSyu)
Owen-VL-Chat
InstructBLIP (Vicuna-130)
GPT-4V
LLavA 1.5 (Vicuna 139)
Gemini-pr vision
Fuyu-8B
GQA
TextVQA
75
70
655
60-
Accuracy (%)
60
50-
40
50-
45
30
20
Quantile by Object Relative Size
The performances of multiple popular ML.LMs on GQA and Text show a clear positive correlation with relative size of target objects. The accuracy is computed with inclusion match. *A small part of questions skipped due to safety policy of APIs. The models showing stable performance on GQA are reported to having been trained on such dataset.
Figure 2:
objects are typically sampled at lower inherent resolution. However, it could be possible that the size of the object, independent. from its quality, might have an affect the MLLMs' ability to perceive it. Another less obvious factor is the number of potential distractors, since the smaller object is relative to the total image size, the more room there is for distractors to appear. Finally, small objects also have more room to move around on the image, compared to a larger object that is limited to the central locations on the image. It challenging to predict how each of these visual factors independently affect the perception of MLLMMs, and thereby their ability to perceive small objects. In the remainder this work, our goal is to conduct controlled intervention experiments to shed light on the independent effects of the aforementioned visual factors on MLLLMs perception.
4 What Factors Affect MLLMs' Perception of Small Objects?
We seek to study the independent effect of the following four visual factors on the perception of MLLMs: object quality, object size, object distractors, and object location. While these identified factors by no means exhaustive, they aim to illuninate some of the fundamental perceptual limitations of current MLLMs, thereby informing both practical applications and future enhancements of these models. In our experiments, we mainly focus on the text-reading ability of MLLMs, as surrogate for their general perception ability on small objects. This decision is driven by the idea that text reading involves recognizing diverse shapes and their spatial ionships,,,,, providing clear and definitive framework for assossment of perception. Compared to other visual tasks like identifying object colors or types, text recognition offers reduced ambiguity in evaluation. To facilitate controlled comparisons, we use synthetic digital texts, rendered in the ten widely used fonts and colors, and overlaid on ten seleeee light color backgrounds. Note that digital text reading widely appears in web-scraped datasets used to pretrain MLLMs, and is therefore a realistic (in-domain) setting. We confirm similar trends when repeating all our studies on FashionMNIST in Appendix F,, where we replace the digits with 10,000 MNSS objects and pose the task as a multiple-choice object recognition problem to probe whether the observed sensitivities persist beyond text.
Object Quality. We define quality as the original sampling rate of an object (in pixels per inch, or pixels per vector graphic range), that is, the original resolution of object in given image. To vary object quality, we adopt wownsampl-upsampee is illustrated in the upper part of Fig. 3. Starting from an original 300-pixel by 300-pixel raster image of a
strategy an original high-resolution image of the object, which
4

---

Published in Transactions on Machine Learning Research (02/2026)
vector graphic digit (Dorig): we reduce its quality six times by down-sampling that raster image to 50 pixels by 50 pixels (Ddown), Then we upsample the Daown six times, and the resulting Daown_ reaches the same image size with Dorig, but six times lower sampling rate. Note that image upsampling does not inherently change the sampling rate of the object despite the increase in pixel values. In this paper, we use the terms "sampling rate" and "quality" interchangeably.
Object Size. The object size is defined as the number of pixels that belong to an object in the input image to MLLMs. Note that object to the desired size. To this end, adopt a crop-upsample strategy, as is illustrated in Fig. 3 (lower). Given a 300-pixel by 300-pixel raster image of digit (of a particular quality due to the original sampling rate), we crop the Dorig at the center to 100 pixels by 100 pixels (Dorop). Then we upsample the Derop three times, resulting Derop_up up with the same sampling rate and image pixel size with Dorigs while having a three times larger object size.
modify the object size while keeping its quality constant by upsampling the
Object Distractors. Object distractors are objects that belong to the same conceptual distribution as a target object of interest (e.g., other numbers when the object of interest is particular number in the image).
Object Location. Current MLLMs share the same manner for image processing, where complete image is divided into numerous patches, which are subsequently transformed into individual image tokens. Formally, the imput image x E &RAWxC with spatial dimensions (H, W) and C' color channels is first reshaped into 2D patches Xp E NNP22c and the resulting N image patches are mapped to N token embedding as the input of the Transformer architectures. Therefore, an input object could be cut by image patch boundaries and divided into different image patches. In light of this, we investigate two complementary location-related factors: the global location on the image and the local patch boundary cut on the target object.
300 X 300
300 X 300
Down sample
Up sample
50 X
613
613
D Doven
Davig
D Dannaa
300
X 300
100 100
Crop
sample
613
613
613
Doria
D
Deroyoap
Figure 3: An illustration of the Downsample-ppsample (upper) and Crop-Upsample (lower) procedure described in Section 4.2 and 4.3. The upper process reduces object quality 6 times while keeping the same size and position. The lower increases object size three times while keeping the object quality.
4.1 Experimental Setup
Text Reading Objective. During the evaluation, the accuracy of the MLLMs responses is assessed against the actual text in the images using Gestalt Pattern Matching (GPM) (Ratcliff et al., 1988). This metric is a widely used smooth metrie for OCR task assessments.. The sampling rate is defined in terms of the font size used during text creation, which correlates with the vertical pixel count of the text characters.
Evaluated Models. Due to the prohibitive cost of running granular experiments on commercial MLLMs, we will consider six open-source models as representative examples of current MLLMs: BLIP2 (Li et al., 2023a), TnstructBI.IP (Dai et al., 2023), LLaVA-1.5 (Tiu et al., 2023a), Qwen-VT-Chat (Bai et al., 2023), Fuyu-8B (Bavishi et al., 2023) and Qwen3- VL-8B (Bai et al., 2025). The architectures of these models are introduced in Section 2. Notably, BLIP-2 has not been explicitly trained on OCR-oriented tasks, relying instead image-text, pairs with text annotations within the images. InstructBIIIP and V.a1.5 have undergone training on several OCR-oriented tasks, including OCR-VQA (Mishra et al., 2019) and TextCaps (Sidorov et al., 2020). Sowen-VL-Chan having been trained on substantial 25M OCR-oriented
5

---

Published in Transactions on Machine Learning Research (02/2026)
DUP-2 (Tanl + InstradBLPP (Vicuna-138)
+
LLVA--1 15 (Mcuna-138)
+
Fuyu-BB
Chon-V-Chak
Reading digits
Reedir 5 gits
Reeding 7 digits
1.00
130
Averigg String
60
0.2
40
23
12
14
16
18
20
10
12
14
16
20
20
Sampling rate
9404549
5934549
5934549
5934549
5934549
5934549
5934549
5934549
5934549
The effect of changing text sampling rate (quality) on model's performance of reading texts while keeping the size of the text. is noticeable that from the sampling rate of 8 (marked as red), the image starts to become fully recognizable as '59345499'
Figure 4:
dataset, demonstrates enhanced OCR capabilities, and thus referred as an OCR-enhanced- in our analysis. The training specifics for Fuyu-8B and Qwen3VI-SB are not publicly disclosed, but based on its performance, presume its OCR training to be similar to that of Qwen-VL.
Performance Measurement and Uncertainty Quantification. For each experiment, model accuracy is reported as the mcan performance over all samples within the corresponding evaluation group. To quantify statistical reliability, we additionally provide the 95% confidence interval computed from the standard error of the mean:
Mean 1.96
where a is the unbiased sample standard deviation and is the number of samples. These intervals are visualized as vertical error bars in the figures, ensuring that all reported accuracy trends are accompanied by a clear estimate their statistical uncertainty.
4.2 Quality Sensitivity Study
Our goal in this section is to study the ability of MLLMs in reading small text of varying quality (sampling rates). We adopt the Dwwnsample- Sapple-Uppaapllee strategy which is described in Fig. 3 and construct a dataset with a sampling rate from 2 to 20 in increments 2, examples are shown at the bottom of Fig. 4. Our experimental tasks involve reading 3, 5, and digits, signifying three tiers task complexity, placed at the center of image. Each tier includes 500 random numbers read. We prompt MLLMs with the question "What is the number the image?".
MLLMs' response to object quality is threshold-dependenn.. As shown in Fig. 4, we observed a significant improvement in the MLLMs performance as the sampling rate increased from 4 to 8. However, after this point, the performance stabilized with increasing sampling rate, indicating a threshold-deependen.. trend in the MLLMs' ability to read text of varying qualities.
The threshold is consistent across the evaluated models and aligns with human perception. Remarkably, threshold of sampling rate of 6-8 is consistently observed across all MLLM models, irrespective of their text recognition capabilities and the varying levels of task complexity. This threshold seems to be consistent with human perceptual ability, as it becomes hard to read text below this threshold for our own eyes. These findings suggest that the MLLMs response to image quality is more influenced by the intrinsic properties of the images rather than the internal differences among MLLMs. Considering this threshold-depeendent performance, the continuous improvement in performance within image size observed

---

Published in Transactions on Machine Learning Research (02/2026)
in Fig. 2 cannot be solely attributed to image quality improvements. In the following sections, we conduct further experiments to investigate other factors that can affect MLLM''s perception of small objects.
4.3 Size Sensitivity Study
In the preceding section, we observed that the sampling rate of text does not significantly challenge MLLMs after certain threshold. This leads to inquire about the impact of object size MLLMs performance with a fixed sampling rate (quality). To explore this, we follow the in Fig. 3. Specifically, for Darig: we place an 8-font size text. in the center the image, then in Derop up the original text enlarged to 5.5 times, with a step of 0.5, illustrated at the bottom of Fig. 5. The tasks include recognizing 500 random numbers with 3, 5, and digits following Section 4.2. We prompt MLLMs with the question "What is the number on the image?".
strategy described
0UP-2 (1an)15ou) +
InstructBII MMuna-138
+ LLaVA-1 (Mauna-13E)
+
Fuys-8B
+
Qwen-MCCrat
+
Qwenn-VL-BE-iit
Reading 3 digits
Readir 5 gits
Reading 7 digits
(1) Aoerrge Sming Siiltarty
100
00
00
70
60
60
40
10
155
2.0
2.5
3.0
35
40
4.5
5.0
5.5
1.0
1.5
2.5
3.0
35
4.0
45
5.5
0
1.
2.0
25
30
4.0
5.0
Text Size Ratio
75:00 NRR
7391608
7391600
7391688
7391688
7391688
7391688
Figure 5: The effect of changing lext size on model's performance while keeping the sampling rate of the text.
At a fixed object quality, most MLLMs perform better at recognizing larger objects. As shown in Fig. 5, except for the OCR-enhanced model Qwen-VL-Chat, the performance of MLLMs improves with the increase of object size while maintaining a constant quality (sampling rate). Notably, the performance trajectory of Fuyu-SB exhibits a signíficant enhancement the early stages of size increase. In contrast, BL.IP- and InstructBLIP show a more gradual improvement in performance with increasing object size. LLaVA-1.5, however, demonstrates relatively stable performance across varying sizes, indicating losser scnsitivity to changes in object size. Furthermore, we observe that for tasks with greater complexity (recognizing more digits), the increase in object size has larger impact on the models' accuracy. This phenomenon may be attributed to two reasons. First, larger object sizes occupy more image patches. These patches translate into transformer tokens, which, during the self-attention mechanisms of the transformer architecture, allow for a more extensive fusson of information. Second, the majority of MLLM image-text matching data for pro-training, only present textual descriptions for the main visual components in the image which are often larger, diminishing their capability of perceiving smaller objects. The second reason is supported by the fact that the OCR-enhanced model Qwen-VL-Chatt VL-Chat, which is trained on large-scale synthetic data with 41 English fonts and 11 Chinese fonts, maintains its accuracy when processing smaller objects.
4.4 Distractor Sensitivity Study
Small objects in image, in addition the inherent effects of their size we observed in the previous scctions, can also affect MLLMs perception by allowing for the presence of more distractors in the image. Our goal in this section is to study the effect of distractors on MLLMs' perception of small objects. To that end, we place the answer text (number) at the center of the image, then we introduce k distractor numbers, positioning them at random locations throughout the image. The answer digit text assigned to the variable
7

---

across varying Published in Transactions on Machine Learning Research (02/2026)
BLIP-2 (F anT3xx))
Fuyu-8B
InstructBLIP IP (Vicuna-138)
Qwen-VL-Chat
LLaVA-1.5 (Vicuna-138)
Qwen3-VL-8B-nnstruc
Font size 8
Font size 12
100
100
(%% Average String Simi arity
80
80
60
40
20
3
Number of Distractors
Figure The effect of changing the number of distractors on MLLMs' text reading.
'a', while the distractor numbers are assigned to 'b' and subsequent letters ('c', 'd), We vary the number of distractors from 0 to 9, and prompt MLLMs with "What is the number assigned to variable 'a' in the image?". We experiment with text font sizes 8 and 12 without resampling to gain 2 tiers of task difficulty, each tier including 100 random numbers (3 digits) to read, and the random position of distractors for cach number is varied 5 times.
Increasing the number of distractors makes perception harder for MLLMs. As shown Fig. 6, the increase in the number of distractors consistently decreases MILMs' performance regardless of their overall performance. Specifically, the OCR-enhanced MLLM Qwen-VL-Chat reaches perfect distractor numbers font size 12, while facing 10-point performance drop during the increase of distractor numbers on font size 8. In contrast, Qwen3-VL-8B VL-8B demonstrates perfect performance across all distractor settings and font sizes. This result suggests that improved caining-potentially including higher-resolution visual pretraining, stronger OCR alignment, or more diverse cluttered-scene supervision cusiiinnnuusttnnilllyy. enhances perceptual robustness. Among the other models, Fuyu-SB, InstructBLIP, and BLIP-2 present heightened sensitivity to the additional distractors while LLaVA keeps a relatively minor performance drop. It is worth noting that although Fuyu-8B has superior performance over LLL in Fig. 5, it appears to lack robustness when facing distractors.
4.5 Location Sensitivity Study
Another factor that can significantly vary for small objects their location in the image, which can in turn affect MLLMs' perception. We study two complementary location-related factors in this section: the global location the image and the local patch boundary cut the target object (described in detail at the start of Section 4).
8

---

Published in Transactions on Machine Learning Research (02/2026)
Table The input image patch number and patch size of the MLLMs considered in our experiment. *MLLMs that uses any resolution visual encoders.
Patch Number
Patch Size
Model
Resolution
BLIP-2
16x16
14x 14
224x224
InSIructBLIP
16x
14
224x224
LLaVA-1.5
24x24
14x14
336
Qwen- -h.
32x32
14x14
448x448
Fuyu-SB*
10x10
30x30
300
Qwen3- VL-8B*
32x32
16x 16
512x512
0
11
24 24
Figure The effect of text (number) location in the image on MLLMs' ability to read the text correctly, with and without distractors (bottom and top, respectively). Higher values in lighter colors.
4.5.1 Global location
Table 1 outlines the patch sizes and counts of the MLLMs evaluated in our study. To augment patch capacities, we merge every four adjacent 14 14 image patches from models like BLIP-2, InstructBLIP, LLaVA-1.5, and Qwen-VI--Cha into single 28 28 patch. Texts are centrally placed within cach merged patch, maintaining consistent sampling rate of 8. In this experiment, following the setting of Section 4.4, we examine MLLLSs text recognition and localization performance under variations in distractor presence and global text positioning. For assessing MLLMs' capabilities, we introduce scenarios with zero and k distractors-zew zero distractors that evaluate pure text recognition ability across different image locations and k distractors that require localizing the target text. Specifically, the OCR-enhanced Voven3-VL-SL model tested with nine distractors, while all other models with one distractor. We include 100 random numbers (3 digits) placed all through the image patches. We prompt MLLMs with the question "What is the number assigned to variable 'a' in the image?" during evaluation.
and
MLLMs exhibit inconsistent text recognition and localization performance across different global locations. In Fig. 7, we observe that the majority of models, except LLa encounter challenges in recognizing or localizing text on the right side of an image. Moreover, BLIP2 and InstructBLIP also experience difficulties with text on the left side. Notably, the OCR-enhanced model Qwen-VT--Chat, despite obtaining near-perfect score in most locations, demonstrates a significant performance disparity of 58 points across different locations. Also, Fuyu-&B experiences sharp decrease in its performance in the first row. This observation suggests that MLLMs are susceptible to positional bias when processing images, and while including more training datascts can lead to much better overall performance (as in Qwen), performance declines on certain image positions still persist.
9

---

Published in Transactions on Machine Learning Research (02/2020)
BLIP-2 anT5
BLIP-2 (F
25
InSTRUCEE IP VIL 130)
20
1n557 DISTPUCTEI SC (Vicuna-1 361
60
50
4.-
LLaVA-1
Seinng
60-
FLay
Fuyu-86
60-
VL
90
00-
UWIT SVL 36
Qwee WW VL Instrcc
100
90-
Taxt Horizontal Position-
Text Vertical Position
435
435
435
435
435
435
435
435
435
435
435
435
435
435
435
Figure 8: The performance of MLLMs in text recognition tasks demonstrates notable variability when textual content is vertically (left) and horizontally (right) cut by image patch boundaries. Gray area indicates that the target texts are cut by patch boundary. We provide two local illustrations below showing that text is shifted between two adjacent image palches. Due to space constraints, we only present the middle part of the entire shifting (range ratio from 0.25 to 0.75), the complete plots are presented the Fig. 9.
4.5.2 Local Patch Boundary Cut
We construct dataset where the generated digital text gradually crosses an image boundary. For vertical patch boundary cut, the digit text is anchored at a predetermined vertical location, while being horizontally moved across the full span of the image. For horizontal cuts, the digit text is fixed at specific horizontal position and moved vertically. An illustrative example of vertical cut shown at the bottom of Fig. 8. We determine the number of reading digits depending on the maximum digit capacity for single image patch, specifically setting at six digits for Fuyu-8B and three digits for the remaining models. We include 100 random numbers for each experiment. We prompt MLLMs with "What is the number the image?" during evaluation.
Model's performance is lower when target objects remain undivided by patch boundaries. For image patch boundary vertical cutting, as observed Fig. (left), a common trend among all models the performance decline at the center of the patch, where texts remain undivided by patch boundaries (white background). Notably, although presenting near-perfect score, Sowen-VL-CHe 10 percent gap between different patch boundary cuts. The only model that does not show this trend is Fuyu-8B assume this is due to its enlarged patch size, making the performance inside an image patch more robust. This phenomenon indicates that contrary to intuitions, texts divided across multiple patches may be more effectively recognized by MLLMs. Therefore, even with the same size and quality, small objects soem to be more recognizable by MLLMs when they are divided into different image patches.
still presents an around
Horizontal cuts hurt the performance more than vertical cuts. Fig. (right) demonstrates the performance of the MLLMs when the target text horizontally cut by patch boundary. Consistent with vertical cuts, in LLa-15, we observe notable performance peak the boundary cuts. However, the remaining models do not show such a trend. We hypothesize two factors contributing to this observation. First, at the horizontal cut, all characters presented are divided into two separate parts, while the vertical cut divides at most only one character into different patches. This effect potentially diminishes the completeness
10

---

Published in Transactions on Machine Learning Research (02/2026)
of shape information. Second, for the horizontal cut, the two resulting image tokens positioned further apart after the image is translated into a sequence tokens for the Transformer; for the vertical cut, the two corresponding image patches remain adjacent.
5
Discussion
5.1
Contributing Factors to Small-Object Performance
In this section, we summarize the key connections between visual factors, observed behaviors, and our corresponding hypothesizes follows.
First, with respect to object quality, we observe clear threshold effect: performance increases rapidly as the sampling rate grows, but saturates once the rate exceeds approximately cight. This pattern suggests that the limiting factor is not the model architecture itself, but rather an intrinsic resolution constraint of the visual signal, consistent with known human perceptual limits.
Second, regarding object size, performance exhibits a strong positive correlation with pixel count even when visual quality controlled. We hypothesize that larger objects span more image patches and thus produce greater number of visual tokens, enabling richer feature aggregation in Transformer layers compared to cases where small objects are compressed into a single token.
Third, for visual distractors, performance degrades steadily as the number of competing elements increases. This indicates a failure of grounding and selective attention: the model struggles to localize and bind the queried target among competing visual signals, pointing limitations in spatial reasoning rather than basic recognition.
Fourth, for global object location, find systematic positional biases, with substantial performance gaps across regions such center versus periphery and left versus right. These disparities are likely driven by biases in the training data distribution, where salient objects are more frequently centered or arranged according to left-to-right reading conventions.
Finally, concerning local location at patch boundaries, we observe a counerintuitive boundary benefit: small objects tend to yield better performance when split vertically across patch boundaries, but a horizontal split tends to be detrimental. We hypothesize that this is because splitting an object to multiple visual tokens provides larger encoding capacity for the object, in contrast to being encoded into single token, except when such split would result in the dispersion of the object across far away token positions (as occurs in horizontal split).
5.2 Countermeasures and Practical Implications
Our experimental findings suggest three promising directions for future development of MLLMs. First, equipping MLLMs with automated perception limitation reports can improve transparency by quantifying to the
where their perceptual capabilities remain reliable. Second, perceptual safety rule-based safeguards in text-only LLMs can down-weight overconfident predictions when the perceived object size or quality falls below confidence threshold. Lastly, aggregating complementary perceptual strengths through ensembles of heterogeneous MLLMs may mitigate individual blind spots and yield more robust small-object recognition.
5.3 Scope and Limitations
Our study focuses text reading and single-object detection (detailed in Appendix F) surrogates for general perception, driven by the difficulty of reliably manipulating arbitrary objects in images. As a result, the current experiments do not fully capture several perceptual challenges that arise in natural scenes, such as complex texture/matent changes, and 3D understanding (e.g., depth perception and spatial geometry). Extending our controlled interventions to broader object categories and more realistic scene variations potttssall via generative data augmentation ien important future work. Furthermore, we do not provide mechanistie analysis that.
recognition, heavy clutter and occlusion, illumination and viewpoint
11

---

Published in Transactions on Machine Learning Research (02/2026)
links the observed limitations to specific architectural optimization choices. Understanding those causal factors is essential for principled improvements of MLLMss architecture, and we leave this invest investigation to future research.
6 Conclusion
In this paper, we exposed notable limitations of current MLLMs in perceiving small visual concepts, and explored four visual factors that contribule to them: object quality, size, distractors, and location. We extensively explored each of the factors by conducting controlled intervention studies, concluding that 1) object quality does not pose an additional obstacle for MILMs after a certain threshold; 2) most MLLMs fall short in perceiving small objects, even with enough object quality; 3) MLLMs' perception ability is prone the existence of distractors, and changes significantly based the object's both global and local locations in the image. In addition, study also provides new evaluation protocol for automatic benchmarking of MILMs' perception and their blind-spots, which are crucial in real world applications.
References
Kian Ahrabian, Zhivar Sourati, Kexuan Sun, Jiarui Zhang, Yifan Jiang, Fred Morstatter, and Jay Pujara. The curious case of nonverbal abstract reasoning with multi-modal large language models. arXio preprint
arXi::440.122117 2024.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Tain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, al. Flamingo: visual language model for few-shot learning. Advances in Newral Information Processing Systems, 35:23716 627366, 2022.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: Visual Question Answering. In International Conference Computer Vision (ICCV), 2015.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: versatile vision-languago model for understanding, localization, text reading, and beyond. arXiv preprinl Xrvv:2230.. 12966, 2023.
Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, et al. Qwen3-vl technical report. arXiv preprint arXiv:2511.2163 2025.
Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sagnak Taşrlar. Introducing our multimodal models, 2023. URL attps://wwiz adept. wwwpaadeppppaaeeeeeeeeee ai/blo//f/ouuuuttt////
Wenliang Dai, Junnan Li, Dongxu Li, Authony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards general-purpose vision-language models with instruction tuning, 2023.
Mostafa Dehghani, Basil Mustafa, Josip Djolonga, Jonathan Heek, Matthias Minderer, Mathilde Caron, Andreas Steiner, Joan Puigcerver, Robert Geirhos, Ibrahim M Alabdulmohsin, et al. Patch n n'pack: Navit, a vision transformer for any aspect ratio and resolution. Advances in Neural Information Processing Systems, 36:2252 2274, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Representations, 2021. URL
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-c: An embodied multimodal language model. aixiv preprint XxX0:2303.03377. 2023.
12

---

Published in Transactions on Machine Learning Research (02/2026)
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean- -ean-ttptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXio:2312.11800 2023.
Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. ar preprint arXiv:2312.0881, 2023.
Jinyi Hu, Yuan Yao, Chongyi Wang, Shan Wang, Yinxu Pan, Qianyu Chen, Tianyu Yu, Hanghao Wu, Yue Zhao, Haoye Zhang, Xu Han, Yankai Lin, Jiao Xue, Dahai Li, Zhiyuan Liu, and Maosong Sun. Large multilingual models pivot zero-shot multimodal learning across languages. arXiv preprint 2rxw::2088. 12038, 2023.
Drew A Hudson and Christopher D Manning. Gqa: A new datasct for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVE conference on compuler vision and 2019.
pattern recognition, pp.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730 19742. PMLR, 2023a.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 292-393 30-,,, Singapore, December 2023b. Association for Computational Linguistics. doi: (0.11663/11/2023.mmmm main.20. URL https://aclanthollo org/2023. emnlp-main. 20.
Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Pong Jin, and Li Yuan. Vidco-llava: Learning united visual representation by aligument before projection. arXiv preprinl Xii::2311.1012 2023a.
Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. arXiv preprint Ar.iv:2312. 075333 2023b.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee, Improved baselines with visual instruction tuning. arXiv preprint 251::3300037444 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Loo. Visual instruction tuning. ar preprint 20X 2023b.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbennh: Is your multi-modal model an all-around player? arx preprinl w:Xiv:237.002881 2023c.
Yuliang Liu, Zhang Li, Hongliang Li, Wenwen Yu, Mingxin Huang, Dezhi Peng, Mingyu Liu, Mingrui Chen, Chunyuan Li, Lianwen Jin, and Xiang Bai. On the hidden mystery of oer in large multimodal models. ArXio, abs/2305.007 07895, 2023d. URL attps://apis https://api.seeanttaat wwwpaaaiiisccooaaooorrrooooa :2:.ppu8IID:288552222 258685422.
Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Oer-vqa: Visual question answering by reading text in images. In 2019 international conference on document analysis and recognition (ICDAR), pp.
IEFE, 2019.
Yao Mu, Qinglong Zhang, Mengkang Hu, Wenhai Wang, Mingyu Ding, Jun Jin, Bin Wang, Jifeng Dai, Yu Qiao, and Ping Luo. EmbodiedGPT. Vision-language pre-training via embodied chain of thought. In Thirth-seventh Conference on Neural Information Processing Systems, 2023.
OpenAL. GPT-4 technical report. arXio preprint orXiw:203300877, 2023.
13

---

Published in Transactions on Machine Learning Research (02/2026)
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PmLR, 2021.
John W Ratcliff, David Metzener, et al. Pattern matching: The gestalt approach. Dr. Dobb's 's Journal, 13(7):
46, 1988.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. LAION-5b: An open large-scale dataset for training next generalion image-lext models. Advances in Newral Information Processing Systems, 35:25278 25294, 2022.
Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hongsheng Li. Visual cot: Advancing multi-modal language models with comprehensive dataset and benchmark for chain-of-thought reasoning. Advances in Neural Information Processing Systems, 37:8612 8642, 2024.
Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image captioning with reading comprehension. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part II 16, pp. 742 152.758. Springer, 2020.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8887888 8326, 2019.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal lmss. arXiv preprint XXxx22401006200 2024.
Hugo Touvron, Thibaut Lavril, Gautier Tzacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Tacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and efficient foundation language models. arXiv preprint ar arxiv:2002. 13971, 2023.
Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. Cogvlm: Visual expert for pretrained language models, 2023.
Penghao Wu and Saining Xie. v*: Guided visual search as core mechanism in multimodal Ilms. arXiv preprint arXi::2312. 14135, 2023.
Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. Appagent: Multimodal agents as smartphone users. arXiv preprint 2rXi:22312.133 13177771, 2023.
Tianyu Yu, Jinyi Hu, Yuan Yao, Haoye Zhang, Yue Zhao, Chongyi Wang, Shan Wang, Yinxv Pan, Jiao Xue, Dahai Li, et al. Reformulating vision-language foundation models and datasets towards universal 2023a.
multimodal assistants. arXie preprint arXiv:2310.000
Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. Rlhf-v: Towards trustworthy milms via behavior alignment from fine-grained correctional human feedback. arXio preprinl arXiv:2312.000 008449 2023b.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Steveus, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi disti-diseipliee multimodal understanding and reasoning benchmark for expert agi. arXiv preprint 1rXio:2311.16500 2023.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre- training. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 11975 11986, 2023.
Jiarui Zhang, Mahyar Khayatkhoci, Pratock Chhikara, and Filip Nievski. Visual cropping improves zero-shot question answering of multimodal large language models. In R0-FoMo Robustness of Few-shot and Zero-shot Learning in Large Foundation Models, 2023.
14

---

Published in Transactions on Machine Learning Research (02/2026)
Jiarui Zhang, Mahyar Khayatkhoei, Prateek Chhikara, and Filip Ilivvski. MLLMs know where to look: Training-free perception of small visual details with multimodal LLMss In The Thirteenth International Conference on Learning Representations, 2025. URL https://openrevier net ne//forrunidetDeeeoonn....
Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang, Wenjuan Han, and Baobao Chang. Mmicl: Empowering vision-language model with multi-modal in-context. learning. ar Xiv preprinl orXiv:2300007913 2023.
15

---

Published in Transactions on Machine Learning Research (02/2026)
A Implementation Details
We use python 3.10.6, transformers 4.29.1 and torch 2.1.2 for all the experiments. Our environment consists of an Intel(R) Gold 5317 CPU @ 3.00GHz with 48 cores and 756 GB of RAM, and we utilize NVIDIA RTX A6000 GPUs for our experiments. We use the huggingface implementations of all studied MLLMs with the recommended hyper-parameterr according to the respective papers. For GPT-4V, we use the official public API as available at the time of submission.
B Complete result of patch boundary cut.
Fig- shows the complete result of horizontal (upper) and vertical (lower) cuts, the overall trend stays the same.
OLIP2 (Flan T
40
LEEVA-15 IVIELna- 38
Fayu-88
OWen--V--Chhtt
90
-02
OWANT VALL 86 BB RSSTUCT
892
Vertical
90
Fuyu-
COPETT OF VL D6 RESFULE
90
-Text Horizontal Position-
Figure 9: Complete result. of vertical and horizontal cut.
16

---

Published in Transactions on Machine Learning Research (02/2026)
C Justifying the Factors via Regression Analysis on Real-World Experiments
To investigate whether factors beyond object size significantly influence model performance, we fit regression models using the predictions evaluated in Fig. 2. Since "object quality" and "distractors" are not annotated in existing datasets, we focus on two reliably available factors: object size and object location.
Specifically, we compare two polynomial linear regression models to isolate the contribution of Location:
1. Baseline: Accuracy
Object Size
2. Full: Accuracy Object Size + Object Location (x, y)
The regression models result is detailed in Table 2. From the result, Object Location provides statistically significant improvement in predictive power for the vast majority of models, even after controlling for size, indicating that spatial positioning contributes meaningfully to performance.
Table ANOVA comparison between the two polynomial linear regression models.
Model
Dataset
R² (size)
R² (size+tpos)
p-value
GQA
0.0165
0.0260
17.258
6.81 x 10 23
BLIP-2 (FlanT5xxi)
TextVQA
0.0275
0.0321
3.425
1.17 10
GQA
0.0057
0.0091
6.127
3.61 10 -7
InstructBLIP (Vicuna-13B)
TextVQA
0.0210
0.0249
2.832
6.03 x 10
GQA
0.0003
0.0035
5.610
1.78 x 10-6
LLaVA-1.5 (Vicuma-13B)*
TextVQA
0.0084
0.0311
16.764
5.10 x 10 22
GQA
0.0062
0.0085
4.074
1.79 x 10
Fuyu-8B
TextVQA
0.0027
0.0066
2.806
6.46 x 10
GQA
0.0000
0.0024
4.169
1.36 10
Qwen-VL-CCat
TexVQQ
0.0165
0.0196
2.243
2.81 X 10
GQA
0.0027
0.0036
1,535
0.150
GPT-4V"
TextVQA
0.0091
0.0105
1.004
0.426
GQA
0.0029
0.0053
4.272
1.00 x 10
Gemini-pro-vision"
Tex
0.0064
0.0114
3.629
6.57 x 10
D Why does Fuyu-8B have a noticeable low performance in its first row?
In Fig. 7, we notice a sharp decrease in Fuyu-8B's performance score within the first row. We assume this unexpected phenomenon is related to its unique pure transformer decoder architecture. To this end, we choose several images and present the attention map of Fuyu-SB, providing observations for further investigation.
In Fig. 10, we provide the attention map for each of the 36 layers of Fuyu-8B. The input image is the synthotic image we construct in the location study in Section 4.5, where single 'a=665' is placed in an image patch's center. The position of the patch is: 0, 4, 9, 19, 49, 99, in the raster scan order of the original image (with 10 x 10 image tokens), the input position can also be seen from the yellow attention outlier in Layer 1. The attention map is computed for the next token after promoting Fuyu-8B with 'Question: What is the number in the image? Short answer:' and we track the attention of the next. token with respect to each image patch. From the attention map, we can tell that ranging from approximately 13-27 layers, for the image whose text is placed in deh position, there are consistently high attention values in the first ls tokens, where i = if S otherwise 9. Such result could be linked to the low performance observed in the first row since the high attention among those layers stays consistent within the tokens in the first row. For the deeper reason behind the phenomenon, we leave them as open future works.
17

---

Published in Transactions on Machine Learning Research (02/2026)
700
189
Live4
USM
12
Live's
Lrve
65626
1aye7
100 8
Laye
Up 13
SXCENG
3
Figure 10: The attention map of Fuyu-8B on six different input images. Detailed descriptions in Appendix D.
18

---

Published in Transactions on Machine Learning Research (02/2026)
E
Folor&Font augmentation detail
All of our experiments are conducted on randomly selected colors and font-styles. The text colors used in the experiments are:
black, navy, dark green, maroon, dark violet, crimson, chocolate, dark orange, teal, indigo
The background colors used in the experiments are:
#f7f9ff, #fff7e6, #f2fff6, #19f0ff, #e8f8ff, #fff0f4, #f0fff9, #fdf7ff, #f4faf0, #fffaf2
The font styles used in the experiments are:
Arial, Courier New, Impact, Palatino Linotype, Trebuchet MS, Comic Sans MS, Georgia, Lucida Console, Times New Roman, Verdana
F Experiment on FashionMNIST
To provide some evidence that our discovered sensitivities can generalize to perception of other objects, we repeat all our experiments with the FashionMNIST dataset in Fig. 11. We formulate the task as multiple choice question, where the MLLM asked to pick the correct class of the object in the image (the class names are the official labels and provided using 10,000 objects from the testing of FashionMNIST placed on white background, the setup of all experiments are the same in the paper). We observe that while the absolute performance of all MLLMs change, their sensitivities to quality, size, object global/local location, and distractors still present. A few notable points:
the text-prompt to the MLLMMs; we measure the model's accuracy
BLIP family shows much better absolute performance than other MLLMs on MNIHT, we hypothesis that this is because they are trained on much larger multimodal datasets (e.g., LLa VA-1.5 only trained on about 1M multimodal instruction tuning dataset, while BLIP2 is trained on 129M dataset for multimodal alignment).
While all sensitivities that we observed in text-reading are similarly observed FashionMNIST, their exact trends can be slightly different.
An issue with any synthetic object manipulation is that it causes certain degree of inevitable distribution shift, since the models are not trained on such data. Most notably, we see that Fuyu is almost completely unable to understand the assignment of FashionMNIST objects to letters, resulting in near chance performance in presence of distractors. In contrast, out text-reading benchmark does not suffer from distribution shift because all MLLMs have been trained on scanned documents containing black text. white background, hence our choice in the main paper.
19

---

Published in Transactions on Machine Learning Research (02/2026)
BLIP-2 (FlanT5xul)
InstructBLIP (Vicuna-138)
LLaVA-1.5 (Vicuna-138)
Fuyu-8B
Qwen-VL-Chat
Q=2
9-1
Q-5
S=2
Q=10
Quality (Q=10 is original quality)
Size (all images have Q=5)
Number of Distractors
an))
W
OOjject Norbontal Position
-Object Vertical Position-
ObjectVerticalCut
Object Horizoataaz
Charn
Global Location (top without distractor, bottom with distractors)
Example with 8 Distractors
Figure 11: All experiments repeated on FashionMMIST, as is described in Appendix F.
20


Reviews:
[{"review_id": "RZnbHPVqyI", "reviewer": "Reviewer_YpFP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The manuscript \"Exploring Perceptual Limitations of Multimodal LLMs on Small Visual Objects\" investigates a fundamental limitation of MLLMs on their ability to perceive & reason over small visual objects. This work mainly focuses on measuring the perception sensitivity of MLLMs. The authors conduct comprehensive experiments on: 1. The four visual factors, namely, object quality (sampling rate), object size, number of distractors, and object location, that limit the models' performance under a controlled synthetic benchmark (mainly OCR and FashionMNIST related tasks). 2. Real-word evidence from GQA and TextVQA benchmarks. The baselines involve seven representative MLLMs. The findings show that lower target quality, smaller object size, distractors, global positional bias and patch-boundary sensitivity contribute to MLLMs' perception limitation with small objects.\n\nStrengths:\n+ The research question addresses a real and impactful limitation across multiple MLLMs. The finds will bring insights to OCR, small object detection and medical imaging tasks where small object perception plays a vital role.\n+ The experiments are comprehensive including real dataset and synthetic controlled text/FashionMNIST datasets.\n\nWeakness:\n- Although the paper positions itself as a study of small visual objects, the experimental design is dominated by text-centric synthetic stimuli (digits, multi-digit combinations, fonts, colors). FashionMNIST experiments partially alleviate this, but they use very simple, low-resolution, centered grayscale objects, which do not reflect the complexity of real small-object perception (e.g. remote sensing - tiny vehicles; e-commerce - logos, icons; robotics - multi-object dense clutter). Thus, the general term “small object perception” may be overly broad relative to the experimental domain.\n- Limited discussion of how the findings transfer to downstream applications. The four factors identified (quality, size, distractors, location) are intuitive from a vision perspective, but the paper does not clearly connect these quantitative findings to specific downstream tasks where such insights could substantially improve performance.\n- While not central to the authors’ focus, the study does not consider text rotation, skew or arbitrary orientation, which are major failure modes for OCR-heavy MLLMs.\n- The selected models—BLIP-2, InstructBLIP, LLaVA-1.5, Fuyu-8B, Qwen-VL-Chat—represent 2023–2024 era architectures. However, modern models such as: Qwen2.5-VL/Qwen3-VL/InternVL3 would potentially strengthen claims.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper provides comprehensive experimental evidence on the four key aspects affecting the perceptual capability of MLLMs on small object. The synthetic controlled experiments shows that: the four axes—quality, size, distractors, location—are carefully examined, and the observed trends are highly consistent across models, task complexities, OCR vs non-OCR settings, and font/color variations. As a backup ablation study, the FashionMNIST experiments confirm that sensitivities persist beyond digit/text recognition, reinforcing generality. The real-world datasets GQA and TextVQA also supports claim.", "audience_yn": "Yes", "audience_explanation": "This work is relevant to many TMLR’s audience. Understanding MLLMs' failure modes in small object perception is essential for OCR and document-understanding related tasks. Meanwhile, small-object perception is crucial for robotics/embodied AI in manipulation tasks (e.g., reading labels, locating fasteners, identifying small tools).", "requested_changes": "1. The experiments make an implicit claim that “Synthetic small-object perception is a good proxy for real-world metrics.” However, the synthetic controlled tasks are relatively simple. I believe more explanations are required here to further strength the claim.\n\n2. For OCR-based tasks especially OCR recognition, we can actually use a strong OCR model to extract the small fonts and then feed the recognition results together with the original image and questions to the MLLMs to obtain the answer. In that case, experiments on distractors also seem to be simple. As long as the distractors could be extracted, the reasoning for answer would be straightforward. Some explanations/experiments are expected here.\n\n3. Patch-grid experiments are insightful but not fully investigated. The baselines using different positional embedding schemes, patching strategies, as well as resizing mechanisms that will potentially contributes to the shown results. Some explanations or add-on points will be appreciated for Fig. 8.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Q7saETpt7H", "reviewer": "Reviewer_dVMh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors evaluate the perception of small visual objects in 7 MLLMs on GQA and TextVQA, and conduct a controlled study of 5 MLLMs' perception, using text reading as a surrogate, to show that lower object quality, smaller object size, the presence of visual distractors, and global/local locations of the object, can independently reduce MLLMs' ability to answer visual questions.\n\n**Strengths**\n1. Comprehensive evaluation on different factors, model, and datasets.\n1. Systematic controlled/ablated design which clearly isolates difference factors, e.g., as illustrated in Figure 3.\n1. Proposes a new evaluation protocol for future MLLMs\n1. The paper is clearly written, and has nice illustrations\n\n**Weaknesses**\n1. Text reading is a questionable surrogate: while the authors claim that \"text reading involves recognizing diverse shapes and their spatial relationships, providing a clear and definitive framework for assessment of perception\", there are a lot of \"small visual objects\" other than texts as those presented in GQA.\n    1. Moreover, the authors only use \"synthetic digital texts, rendered in the widely used Arial sans-serif font, and overlaid on plain white backgrounds\", which limits generalizability.\n1. Only covered models released in 2023. I usually review conference papers, so I'm not sure if it is common for journal/TMLR papers to be (slightly?) outdated.\n1. No explanation of observed phenomenon and no solutions proposed", "claims_and_evidence_yn": "Yes", "claims_explanation": "1. Controlled methodology with clear manipulation of individual factors\n1. Comprehensive experiments across 5 models on 4 factors with tons of test cases\n1. Consistent patterns across models\n1. Extra validation with varied colors/fonts and FashionMNIST", "audience_yn": "Yes", "audience_explanation": "Fits TMLR's scope of \"experimental [...] studies yielding new insight into the design and behavior of learning in intelligent systems\":\n1. Discovered 4 factors affecting the MLLMs’ perception of small objects: object quality, object size, object distractors, and object location.\n    1. Novel findings like global positional bias (Figure 7) and local patch boundary cut (Figure 8) are especially interesting/surprising.\n1. Controlled evaluation protocol valuable for future work", "requested_changes": "1. Release code/data for reproducibility.\n1. Fit a regression model `accuracy ~ object quality + object size + object distractors + object location` with the data used to produce Figure 2. Then do some basic diagnostics, e.g. report $r^2$, and run ANOVA to compare this model against the nested model `accuracy ~ object size`, to help us understand to which extend can the factors (other than size) you have proposed explain the variation in accuracy on standard benchmark datasets. I think \"object quality\" might correlate strongly with \"object size\" in practice, so you can simply omit one of them or run LASSO.\n1. If that's not too much work, use other font/color/backgrounds for synthetic texts, and add error bars to your plots.\n1. This is also optional, but it would be really great if you could run your experiment on some newer models. It's almost 2026, after all.", "broader_impact_concerns": "Not required. The work identifies MLLM weaknesses in fine-grained visual understanding, which benefits responsible deployment.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xdWiljvg6A", "reviewer": "Reviewer_CstT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose an empirical study about the limitations of Multimodal Large Language Models (MLLMs) in perceiving visual objects. In particular, they explore how different factors: object quality, object size, object location and presence of distractors, each independently affect performance. Different pre-trained models are evaluated, both open and closed source. Experimental results focus on Optical Character Recognition (OCR), however additional results on FashionMNIST are provided in the Appendix. Overall the paper concludes that the considered factors can degrade MLLMs performance in correctly retrieving object content from an image, and the proposed evaluation protocol can enhance automatic benchmarking of MLLMs performance in the considered task.\n\n**Strengths:**\n- The paper is well organized and very easy to follow.\n- The controlled experimental environment is well designed and allows evaluating the effect of different factors while reducing confounders.\n- The experiment regarding patch boundaries shows counterintuitive results which can help pinpointing issues with current modeling approaches and/or biases in training data.\n\n**Weaknesses:**\n- While the paper claims to answer very general questions, the provided evidence is mostly limited to OCR and a simple synthetic settings based on FashionMNIST.\n- The OCR enhanced model evaluated in the paper seems to be the least affected by changes in object size and location. This may suggest that the observed limitations can be task dependent and may not lead to similar behavior when considering more complex visual inputs.\n- The discussion section is hard to parse. As such, the paper's insights on the reasons behind the observed effects are not clearly conveyed to the reader. Moreover there is no discussion on approaches to counter these limitations.\n- Aside from the behavior when crossing patch boundaries, most of the observed effects might be already intuitively understood by researchers in the field.\n- The work claims that their study \"provides a new evaluation protocol for automatic benchmarking of MLLMs' perception and their blind-spots\". However, it is unclear how this can be done beyond OCR tasks or simple synthetic settings like FashionMNIST. Moreover, it is not mentioned whether code to implement this protocol will be released as part of this work.", "claims_and_evidence_yn": "No", "claims_explanation": "The reasons why I am not convinced of the alignment between claims and evidence is summarized by the first two weaknesses I highlighted.  \nThe claims of the paper would be aligned with the provided empirical evidence if said claims where to be limited to the specific task of OCR.  \nAs the paper's experiments are mostly centered around OCR tasks and otherwise around a very simple object classification task (FashionMNIST), I would not say this is sufficient evidence to answer the questions posed in the paper's introduction. At least in their current form which is very broad.  \nThe authors claim that \"text reading involves recognizing diverse shapes and their spatial relationships, providing a clear and definitive framework for assessment of perception\". However, I think there is no sufficient evidence to support this claim.", "audience_yn": "Yes", "audience_explanation": "While part of the paper's findings might be already intuitively understood by TMLR's audience (in particular those about object quality, size and distractors), I think some others (those about object global and local location) could come up as unexpected.  \nMoreover, as these are packaged within a well thought and systematic study, I would expect the paper to be an interesting read for at least a part of TMLR's audience.", "requested_changes": "The following are my requested changes to the authors. Major changes indicate critical aspects, while minor ones are mostly improvements to the text. Note that I am not stating that all changes here reported as \"major\" are a strict requirement to me, rather that they are the most relevant. They are loosely in importance order.\n\n**Major:**\n  - I think the paper's main claims should be narrowed to the specific context of OCR. This should be reflected in the questions posed in the introduction, in the paper's title and, where appropriate, in the body of text. Nonetheless, I think discussing how the observed phenomenons may apply also to other, more complex tasks is relevant to the contribution of this paper, given it is clarified that the accompanying empirical evidence is limited in providing a definitive answer. Of course the authors could also improve empirical evidence in more complex scenarios as an alternative way to address this point.\n  - Plots in the paper should also report the performance variance in the different settings. This would provide an additional dimension of analysis, which I think might be relevant to this kind of systematic study.\n  - Tracing connections between causes and effects, as the discussion section aims to do, should be done in a clearer and more systematic way in order to convey clear insights to the reader.\n  - Discussing possible countermeasures to the found limitations could enhance the contribution of this work.\n  - I encourage the authors to release code implementing the proposed evaluation protocol as part of this work, in order to allow other researchers to integrate this kind of analysis in model evaluation pipelines.  \n**Minor:**\n- In the related works, the phrase \"while InstructBLIP integrates instructions into the Q-Former for an instruction-awarding visual feature\" seems to be missing a second clause about what BLIP-2 does instead.\n- In related works, in the discussion about the LLM Backbone, it is said \"We consider seven widely-used MLLMs in this work\". However, only 5 are then discussed.\n- The second paragraph of the related works, \"Robustness analysis to MLLMs\", seems to contain a typo, should it be \"of MLLMs\"?\n- Figure 2: I would comment, in the text, that the models showing stable performance on GQA are reported to having been trained on such dataset.\n- Figure 6: the sample images should be reported with a bigger scale, otherwise they are hard to read.\n- Appendix D: there is a hyperlink to section D, which points actually to Figures 11-12. I imagine it is a typo in the hyperlink's label.\n- Appendix F should refer to the relevant figure in the text.\n- In section 4.2, the third paragraph's title mentions a \"universal threshold\". I would narrow the statement to the threshold being consistent across the evaluated models.", "broader_impact_concerns": "I do not have any concerns about the broader impact of this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)