QLIP: A Dynamic Quadtree Vision Prior Enhances MLLM Performance Without Retraining
Authors: Kyle R. Chickering
Bangzheng Li
Muhao Chen
Venue: ICLR 2026 Poster (2026)
Keywords: MLLMs
mesoscopic bias and interpolation bias
dynamic quadtree
vision encoders
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: QLIP addresses mesoscopic bias and interpolation bias of CLIP with a novel dynamic quadtree visual prior, and seamlessly enhances MLLMs without re-training.

Abstract: Multimodal Large Language Models (MLLMs) encode images into visual tokens, aligning visual and textual signals within a shared latent space to facilitate cross-modal representation learning. The CLIP model is a widely adopted foundational vision language model whose vision encoder has played a critical role in the development of MLLMs such as LLaVA. However, the CLIP vision encoder suffers from notable limitations including being constrained to only handling fixed input resolutions and a failure to produce separated embeddings for dissimilar images. Replacing the vision encoder of an existing model typically incurs substantial computational costs because such a change often necessitates retraining the entire model pipeline.

In this work, we identify two factors which underlie the limitations of the CLIP vision encoder: mesoscopic bias and interpolation bias. To address these issues, we propose QLIP, a drop-in replacement for CLIP that can be seamlessly integrated with existing MLLMs with only a few lines of code and can enhance both coarse-grained and fine-grained visual understanding, without re-training. QLIP is designed around an image quadtree which replaces the standard uniform grid patches with a novel content aware patchification.

Our experimental results demonstrate that QLIP improves the general visual question answering accuracy of the LLaVA-1.5 model series across various model sizes—without requiring retraining or fine-tuning of the full MLLM. Notably, QLIP boosts detailed understanding performance on the challenging $V^*$ benchmark by up to 13.6%.

Published as a conference paper at ICLR 2026
QLIP: A DYNAMIC QUADTREE VISION PRIOR EN- HANCES MLLM PERFORMANCE WITHOUT RETRAIN- ING
Kyle R. Chickering, Bangzheng Li, & Muhao Chen (krchicke,
bahli., bahli, muhchen, @ucdavis. edu
ABSTRACT
Multimodal Large Language Models (MLLMs) encode images into visual tokens, aligning visual and textual signals within a shared latent space to facilitate cross- modal representation learning. The CLIP model is widely adopted foundational vision language model whose vision encoder has played critical role in the development of MI..Mss such as LLa However, the CL.IP vision encoder suffers from notable limitations including being constrained to only handling fixed input resolutions and failure to produce separated embeddings for dissimilar images. Replacing the vision encoder of existing model typically incurs substantial computational costs because such a change often necessitates retraining the entire model pipeline.
In this work, we identify two factors which underlie the limitations of the CLIP vision encoder: mesoscopic bias and interpolation bias. To address these issues, we propose QLIP, a lightweight adaptation of CLIP that can be seamlessly inte- grated with existing MLLMs with only few lines of code and can enhance both coarse-grained and fine-grained visual understanding, without retraining the vision encoder or LLM weights. QLIP is designed around an image quadtree which replaces the standard uniform grid patches with novel content aware patchifica- tion. Our experimental results demonstrate that QLIP improves the general visual question answering accuracy of the LLaVA-1.5 model series across various model sizes-withous without requiring retraining of the vision encoder or LLM. Notably, QLIP boosts detailed understanding performance the challenging V benchmark by to 13.6%. Code is available at https: //gittb..b.../.. com/ kyroChi/qlip
CLIP
X Random Guess: Lent
Question: with wirle trousers on loft right side of the penaoni Miar?
Vanits LLaVA
QLIP
+ Correct:
Right
Figure 1: QLIP is lightweight adaptation of CLIP which allows models like LLLVVA to perform inference on arbitrarily large images. In our experiments we find that vanilla LLaVA + QLIP gives +13.6% % accuracy on the challenging V benchmark without retraining the vision encoder LLM. The example in the figure above demonstrates an instance where CLIP cannot correctly get the answer because (a) in the cropped version of the image the person question not present, and (b) if use padded image the person will be too small to provide meaningful signal to the model.

---

Published as a conference paper at ICLR 2026
1 INTRODUCTION
Multimodal Large Language Models (MLLMs) have shown impressive multi-modaa question answer- ing ability, yet recent work has highlighted a deficiency whereby these models struggle to answer questions about fine-grained visual details (Shi et al. 2024 Wu & Xie 2024). MILMMs, like the popular LLaVA family (Liu et al., 2024a 2023), use a vision encoder and visual projector to embed visual information into shared -inguui- embedding space before passing these tokens to downstream LLM. This strategy is not without its flaws. Firstly, authors have observed that a high number of visual input tokens can be removed without significantly affecting performance, indicating redundancy (Hu et al. 2024 Li et al.) Sun et al. 2025). Secondly, it has been shown that models like LLaVA overly rely on information from the vision encoder's [CLS] token, which captures global semantics, to answer questions (Zhng et al. 2024b.
We posit that this failure on fine-grained VQA tasks is neither deficiency in the training process of the MLLM nor deficiency in the representations which can be encoded by the vision encoder. Prior works that have aimed at modifying the vision encoder or projector have implicitly assumed that the failure mode is caused by CLIP itself, but this is only partially true. Li et al. show that the vanilla LLaVA architecture with the CLIP encoder is capable of much better VQA performance, but requires the "correet" tokens to be fed to the language model. Similarly, [i et al. (2024) show that the information from the CLIP encoder is often sufficient for certain vision tasks or VQA, however, the models often do not adequately use the given information. Thus, there remains room to improve MLLM performance by focusing on better use of the available tokens.
We argue that the failures incurred while using the CLIP encoder can be attributed two specific biases induced by the inductive priors implicitly assumed during CLIP training. Mesoscopic Bias occurs because CLIP uses uniform sedd-pacchiiicttion (UGP) strategy (Dosovitskiy et al. 2020 Radford et al. 202T) and manifests as downstream models implicitly treating uniform grid cells at a specific image scale as the fundamental unit of semantic meaning. Interpolation Bias arises as a consequence of CLIP being trained with fixed positional embeddings on fixed-resolution images and prevents CL.IP from natively handling high-resolution images.
This work addresses the biases inherent to standard priihpatchiiicationn in MLLMs. While newer vision encoders are similarly able to handle high-resolution images, the biases we explore here remain relevant broad class of encoders which see continued use. Our method is specifically designed to improve the semantic content of the image tokens for the purposes of VQA.
Previous work has focused on training new vision encoders to replace CLIP (Guo et al. 2024 Liu ct al. 2024a Luo et al.] 2024; Shi et al. 2024). but these proposals require re- re-training the entire MLLM, which is expensive and often not feasible. In this work, we take minimally invasive approach and carefully reason through the consequences of updated vision priors. This leads us to lightweight, content-aware, drop-in modification to the CLIP encoder which we call QLIP, portmanteau of "quadtree" and "CLIP".
QLIP empowers CLIP based MILMS to automatically process arbitrary resolution input images, while adaptively scaling the number of input tokens based on the semantic content of the image. We find that reducing the number of input tokens has beneficial effects beyond faster computation: token reduction can decrease model hallucination and improve fine-grained VQA. To assess both the effec- tiveness and efficiency of QLIP, we apply our ideas to the LLa family of MILMS for VQA. Our method is particularly suited for fine-grand fine-grained visual tasks like the challenging V benchmark (Wu & Xie, 2024). Our method achieves a 13.6% improvement on V*, reduces hallucination rates as measured by the POPE Fl score (Li et al. (023) by 5.2, and yields improvements across other multi-modal benchmarks including MME (Fu et al. 2023) and RealWorld-O -QA (xAI Team 2024h.
We accomplish this by using two novel strategies. First, to address the mesoscopic bias we introduce a non-uniform patchification scheme based on image quadtrees (Hunter & Steiglitz (179)). Our quadtree patchification is adaptive, tunable, and training-free, and implicitly treats semantically similar regions of the image as the fundamental unit of semantic meaning instead of UGP. Second, to address the interpolation bias, we train a small MLP network to interpolate the fixed positional CLIP embeddings while maintaining usable positional signals for downstream models.
Our key contributions follows:
2

---

Published as a conference paper at ICLR 2026
An example of the same semantic feature (animal: : elephant) at three different spatial scales. These photos could be accompanied by the question What animal is shown in this photo? For the leftmost image the elephant fits into single patch. Without memorization unlikely for any classifier to be able to accurately identify the pixelated blob as an elephant instead of, for example, horse a buffalo.
Figure 2:
1. We identify two fundamental biases the CLIP vision encoder, i.e. mesoscopic bias and interpolation bias, and propose quantitative measures of both.
2. We introduce QLIP, lightweight, drop-in modification for CLIP that supports arbitrary image resolutions and adaptively seales the number of the image tokens based on image content. QLIP directly mitigates the aforementioned biases without retraining the vision encoder LLM.
3. We empirically validate the effectiveness of QLIP by integrating it into the LLaVA model family (Liu et al. 2024al 2023) and demonstrate substantial performance improvements. Our results are achieved without retraining the vision encoder or LLM. For the challenging V* benchmark, we achieve significant improvement of +13.6% accuracy using LLaVA 13B with QLIP, outperforming the previous SoTA CLIP-based LLaVA results by +3.1%
(Shi et al. 2024)
WHY CLIP FAILS AT HIGHER RESOLUTIONS
The CLIP vision encoder is trained at a fixed input resolution using learned absolute positional encodings (Radford et al. 202T). This design introduces two notable and consequential biases. First, because the positional encodings are absolute rather than relative, they do not generalize beyond the spatial grid used during training. Second, the encoder is trained exclusively on fixed-scale images, which biases the encoder towards only recognizing features at a specific mesoscopic spatial scale. For an exaggerated example, consider the elephants in Figure 2 The CLIP encoder is most likely to understand the middle (mesoscopic) image as containing an elephant, rather than the left or right images. This is because during training is unlikely that the leftmost image would be labeled as having an elephant in it and the rightmost image may be too zoomed to distinguish it from other concepts. In practice the bias is not this extreme, but as show in Figure 4 below, changing the image resolution by only a few pixels already substantially decreases the model's ''s ability to recognize the semantic content of image.
Quantification of Interpolation Bias Consider a single image rendered at two different resolu- tions, R3 (H1, Wi) and R₂ = (H2, W2). We denote the corresponding resized images as In, and IR respectively. Since both images originate from the same source and contain identical (or nearly identical) semantic content, remain invariant or at least approximately constant across these resolutions, especially when R1 and R2 are only slightly different. Under this assumption, the cosine similarity between the corresponding CLIP embeddings, & = CLIP(In,) ) and & CLIP(Inr), serves as a measure of the deviation introduced by resolution changes. quantify the extent to which positional embeddings contribute to this deviation, we define the interpolation bias as:
would reasonably expect the CLIP [CLS] token embedding to
Bineep (I) =
where P denotes the additive positional encodings applied to patch embeddings during the CLIP encoding process (Radford et 202T) and CS is cosine similarity.
Quantification of Mesoscopic Bias Mesoscopic bias is casier to quantify because we can simply remove the positional encodings and look at the cosine similarity of the [CLS]] token embeddings at
3

---

Published as a conference paper at ICLR 2026
Figure 3: (a) An example of the quadtree patchification (QtP) applied to high-resolution image. QtP uses only 25% of the original number of tokens yet retains a high-degree of semantic information. Photo courtesy of first author. (b) A schematic of 4 x patch image being decomposed into 7 leaf patches using quadtree. Leaves which consist of more than a single patch are downsampled to the patch size.
different image sizes. To this end, consider an image I with resolution N x N and then consider the same image rescaled 336 x 336, which we denote Z336. Let 8 CLIP (Z), E CLIP (Z336) be the respective [CLS] embeddings after setting the positional encodings to zero. Then
- == CS(E* E
captures the degree to which the overall embedding has changed as an effect of the mesoscopic scale of the input images.
3 ADDRESSING THE MESOSCOPIC AND INTERPOLATION BIASES
We identify and address two implicit inductive priors underlying the CLIP encoder, noting that these assumptions were likely adopted primarily for engineering practicality.
The first prior is that UGP captures fundamental units of semantics. We address by replacing UGP with content-aware quadtree patchification (QtP). The second prior is that images can be effectively represented by center-cropp croppig and rescaled to a fixed resolution which we address by training small interpolation network.
3.1 VISION QUADTREE MITIGATES THE EFFECTS OF MESOSCOPIC BIAS
Natural images do not contain uniformly distributed information throughout their sub-images. In general, semantic information can continue to be extracted even when large portions of the image are subjected to extreme levels of information degradation at the pixel level (see Figure 5 This is why compression algorithms like JPEG work (Wallace 1992).
We derive strategy for adaptively merging adjacent patches in an attempt to increase the quality of the visual signal coming from the vision encoder. This strategy is based on the intuition that many pixels in given image do not contribute to the representation of the semantic content of the image. We propose using a quadtree (Hunter& & Steiglitz 1979) structure to adaptively select tokens based on some property intrinsic to the sub-images themselves. Quadtrees, applied in image processing, hierarchical image representation trees which generalize a binary tree into two dimensions. At the root of the is the original image, and each level we subdivide the image into four, until we reach leaf nodes which represent patches (see Figure 3 (b)). We can then prune the tree according to some selection criteria and the resulting leaf-nodes will consist of all sub-images which satisfy some maximal condition. We apply downsampling to the leaf-nodes which are larger than the CLIP encoder's 's patch size obtain a sequence of patches that can be fed to the CLIP vision encoder. In theory, semantically irrelevant portions of the image are downsampled back to the mesoscopic scale that CLIP expects, and important tokens which represent a small portion of the visual field are effectively upsampled into the same scale (see Figure (a)).
In what follows, we use the following quadtree selection criteria, which can be thought of as the maximum of the average gradient over patch. Thus, an image I is leaf-node if it cannot be sub-divided or if
D(I) = max 0,1 0,1) < <a I.y
4

---

Published as a conference paper at ICLR 2026
where a is pre-chosen selection constant. We also test random selection strategy ablation for our selection strategy. More details are contained in Appendix F
3.2 COORDINATE-BASE MLP MITIGATES THE EFFECTS OF INTERPOLATION BIAS
The CLIP vision encoder consists of two mechanisms that work in concert to map information from the pixel space into the embedding space.
Let P = {ps) be a set of patches with coordinates X' = {(1,, /= (Iz. E [-1,1]²
The CLIP encoder can be understood as taking P and X and producing sequence of tokens S = {s; := (p;) + M (zi, y:) along with a [CLS] token E cctss (P) which captures global information about the image.
CLIP is trained on 336 x 336 images decomposed into series 14 x 14 patches using the standard UGP @Dssoviskiyy et al., 2020 Radford et al.) 202T). There will then be 24 24 576 patches and to each of these patches CLIP associates positional embedding E E R 1024 where 1 & i, & 24 respectively index the rows and columns of both and the grid of patches. For this patchification we 21 = Eig. We will extend M the entire square [-1, 1]2 so that we can 23
have M 23 natively handle images of any resolution and apply our QtP. We choose to train an MLP using our new inductive priors. Choosing an MLP for this task gives us a high-degree of expressivity.
We make the assumption that the [CLS] token should remain invariant when CLIP is applied to a 336 x 336 image and the same image at its native resolution Thus, if is the standard UGP associated to the image /336 and P is a patchification associated to the image 1N. then we expect that
LICis: E (eas) (G) Eicis) (P) ||| small.
(3)
This provides target for training the MLP. However, in practice Lecus is insufficient for training since the transformer pooling which generates the [CLS] token means that as long as Eij Eij = Σ; M (Iz, Vi). then the [CLS] embedding will be constant. Because to train a drop-in modification for CLIP and because downstream MLLMs utilize the positional infor- mation from CLIP, we must ensure that the MLP positional embeddings match the CLIP positional embeddings the standard 24 24 grid. We add residual L error
are attempting
24 24
2i
2j
R(M
M
-1
Eij
576
23
23
Thus we arrive suitable loss function for the MLP training:
Loss LICLs] + /RR,
(5)
where is hyperparameter to balance the relative effects of the two components of the loss. Training is stable and we include additional training details in Appendix
3.3 TRAINING THE INTERPOLATION NETWORK
The coordinate-basee MIP must be trained; however, training cost is small relative to MILMM retraining (11 hours on four NVIDIA L40S GPUs). We train the MLP for 100 epochs with the Adam optimizer (Kingma| 2014) on the training split of the Imagenette dataset (Howard) 2019). This dataset small subset Imagenet (Deng et al., 2009) with only 10 classes, and consists of about 10k images. We argue that the choice of dataset does not matter much for the MLP training because the embedding function M is independent of the image content. We train with a batch size of 14, with images kept at either their native resolution or at a resolution with the smallest edge length being 560, whichever smaller. We kept y = 1. For our MLP architecture we use four hidden layers and pass the input features through a Fourier features layer (Tancik et al. 2020) with 48 Fourier features. See Appendix C for a discussion about how we chose model hyperparameters
This assumption is better justified when the native resolution very close 336 336. However find that generalizing this assumption to arbitrary image resolutions leads to favorable results We found that 7,7 loss was better than 1,2 loss since we aim to get R to he smaller than 5 x 10-?, See Appendix C for more details.
5

---

Published as a conference paper at ICLR 2026
Erdic Mappicon
Figure 4: The first two panels compare our MLP interpolation with bicubic interpolation. We plot Cf in the first panel as a measure of mesoscopic bias and Biteer in the middle panel as measure of interpolation bias. The third panel shows comparison between the [CLS] tokens of various image sizes with (blue) and without (red) QtP. All data is collected and averaged over the images from the V benchmark.
EXPERIMENTAL RESULTS
Recall that the parameter a from equation controls the amount of pruning done to the quadtree. For natural images, users can reuse the trained MLP and tune only a: default values work reasonably well across similar domains. As Figure 6 gillustrates, performance exhibits non-monotonic behavior with respect a and image size. We recommend a > 0 for fine- mne-graindd tasks and a = 0 for general VQA. We perform sweeps in and image size, over suite of multi-moda benchmarks. in an attempt to understand the dynamics of our proposed methodology. For some benchmarks we additionally sweep native image resolution vs. cropped image resolutions. We report the best score from our sweeps Table [ We choose look at the performance on V (Wu & Xie 2024), MM-Bench (Liu al. 2024b), POPE (Li et al. 2023). CV-Bench (Tong et al., 2024), the visual portion of ScienceQA (Lu el al. 2022h. MME (Fu et al. 2023), and the Real QA benchmark (xAI Team 2024p. We VLM Eval (Dan al. 2024) to do the evaluations MM-Bench, POPE, ScienceQA, MME, and RealWorld-QA We use a custom evaluation script to evaluate V and CV-Bench. More details of experimental setup contained in Appendix D.1 and instructions reproduce our experiments are contained in Appendix
QLIP Reduces Measured Interpolation and Mesoscopic Bias: In Figure we plot a comparison between QLIP and the vanilla CLIP encoder using bicubic interpolation, which we found outper- formed bilinear interpolation. We see that MLP training successfully reduces interpolation bias as measured by Bieerp (Figure middle panel), and brings the cosine similarity between the [CLS] tokens together as predicted by our theoretical assumptions. Next, we observe that the quadtree selection mechanism mitigates the effects of mesoscopic bias by slowing the rate at which the cosine similarity of the CLIP [CLS] tokens diverge as a function image size (Figure rightmost panel).
Table 1: Performance comparison between LLaVA-QLIP and baseline LLaVA models. Bold high- lights the better-performing variant of the same base model. Underlining denotes the best result across all models. An asterisk (*) indicates results obtained using cropped images. Performance increases and decreases are annotated in green and red, respectively.
Model
V*
MM-Bench
POPE F1
CV-Bench
Sci-QA
MME
RW-QA
VQA
LLaVA-1.5-7b
42.4
62.5
74.4
39,9
64.0
1207
49.0
+ QLIIP
53.4
59.7
79.6
40.2
63.5
1241
47.3
(+11.0)
(-2.8)
(45.2)
(40.3)
(-0.5)
(+34)
(-1.7)
LLLVA-1.5-13B
45.0
67.4
82.4
61.6
67.8
1390
48.0
QLIP
58.6
67.9*
83.6
60.7*
67.9
1388
49.4
(+13.6)
(+0.5)
(+1.2)
(-0.9)
(+0.1)
(-2)
(+1.4)
QLIP Significantly Improves the Detailed Visual Grounding on High-Resolution Images: The V benchmark (Wu & Xie 2024) is challenging, vision centric benchmark focused fine-grained image understanding. This benchmark is particularly challenging for CLIP-based based vision encoders because the questions designed to be answered with access to the full-resolution image (see

---

Published as a conference paper at ICLR 2026
Figure D. guessing.
Without access to all of the appropriate visual information the model is often reduced to
Figure demmontrattss that in the absence of the quadtree selection method, our MLP interpolation network already allows the model to effectively utilize all the image tokens from the original image. We note that the 7B parameter is more robust image sizes which were not seen during training than the 13B model. These results already indicate that there is a large performance gap that can be closed with minimal interventions, indicating that significant portion of the poor performance on high-resolution image tasks can be explained simply by lack of access to high-quality visual input signal (c.f. (Li et al.) ). This result indicates that the CLIP encoder and LLaVA weights possess sufficient capacity to do VQA, but lack high-quality inputs.
Figure shows the full sweep over image size and a, plotted with cubic best-fit lines. The r-axis is measured in percentage of tokens seen compared to the baseline model, on logarithmic scale. We see a clear trend where increasing a increases performance with maximal performance occurring for > O. This indicates that the QIP mechanism is complementing the MLP interpolation to boost VQA performance, either by reducing the number of image tokens and sending stronger attention signal to the LLM (Levy et al., 2024, Veličković et al. 2024), by reducing noise by combining redundant image patches through merging, or both. Our ablations in Section below suggest that the latter is more likely.
Refforancc Function Sbn 70
Perfermance Function
0.73
0413
0.40
0.00
0.50
250
Figure 5: The performance V using re-scaled and cropped images with no quadtree selection mechanism and our MLP interpolation. The red line is with bicubic interpolation and the orange line with bilinear interpolation. The black line represents performance of the base CLIP model with 336 x 336 cropping. The 7B model is plotted on the left, and the 13B model on the right. We see that neither bilinear nor bicubic interpolation suitable for extending CLIP larger resolutions.
Figure 6: The compute vs. accuracy curves for our sweep of V with the LLaVA-QLIP-13B model. The z-axis is logarithmic scale. The green- shaded region highlights experiments where model surpasses the baseline with fewer visual tokens.
Improved Token Efficiency: Previous studies which reduce token counts have aimed matching MLLM performance with fewer tokens (Cao et al. 2023) Chen et al. 2024 Hu et al. 2024 Li et al.
Tang ct al. 2022 Our work is largely orthogonal to the aforementioned works, however we note in Figure 6 that we can achieve higher than baseline accuracy fewer image tokens than the baseline model. This is shown in the figure by the top left region, shaded in green, which represents higher than baseline accuracy with lower than baseline numbers of tokens. This reveals that the quadtree selection method prunes tokens in such a way that higher-quality visual signal is provided to the LLM. The work (Li et al.) demonstrated that such improved performance with reduced tokens is
7

---

Published as a conference paper at ICLR 2026
theoretically possible, but to our knowledge this work is the first time such result has been achieved in practice.
QLIP Matches or Improves Performance Across a Range of MLLM Benchmarks: Because our method is trained to be both minimally invasive and does require re-training of the MLLM, we can adjust the model parameters to fit the task at hand without re- re-trriningg Because our training program was oriented towards matching CLIP outputs on images which are the same size as CLIP was trained on, we can nearly achieve baseline performance for any benchmark by using 336 x 336 images with a = 0. Any loss in performance beyond that can attributed to the error interpolating the CLIP embeddings with our MLP network. Notably we find little to no change in performance on MM-Bench, CV-Bench, Sci-QA, MME, or RealWorld QA.
QLIP is designed to perform well on images with fine-grained, high-resolution semantic detail. Because of this global or coarse cues useful for general VQA may attenuated. For example, V*, the 7B model shows regressions on MM-Bench and RealWorld- -A (Table The method is intentionally specialized for fine aine-griied tasks and reveals latent capacity in existing models rather than aiming to replace full retraining approaches.
when tuned for fine-graid une-grained performance
On three of the seven benchmarks the 13B model attained its best performance when the input images were cropped to be square at the original image resolution of 336 336 with 0.1. We found that performance quickly dropped off for these three benchmarks when we varied image size or increased a. We suspect that the 13B parameter version of LLaVA is much more sensitive to deviations the [CLS] token, and the drop-off in performance seems correlated with the change in cosine similarity of the [CLS] token plotted in Figure E We did not observe the same trend in the 7B model, nor did we observe this trend on V*, where the content of the [CLS] token is not helpful for answering the questions.
LLaVA 13B is Sensitive to Image Aspect Ratio:
Table 2: Comparison of LLaVA- LLLII with other models which improve fine-detail grounding. We report the numbers from the authors' papers. Note that S2 requires pre-training and instruction tuning of the LLM [Shi et al. (2024), and that SEAL requires fully re-placing the vision encoder before pre-training and instruction tuning Wu & Xie (2024).
Model
Att
RCl
V Overall
POPE F1
Fine- eraingr grounding
QLIP-7B
50.4
60.5
53,4
79.6
52-7B Shi et 2024
51.3
61.8
55.5
QLIP-13B
53.9
65.8
58.6
83.6
S 13B Shi et 2024
50.4
63.2
55.5
SEAL (7B) Wu & Xie 2024
74.8
76.3
75.4
82.4
Hallucination can be Mitigated by Reducing the Number of Image Tokens: The POPE dataset was designed to measure the hallucination proclivity of MILMs (Li et al. 2023). The proposed measurement of model performance for POPE is the Fl score. For both the 7B and 13B QLIP models we increased performance POPE, with more significant gains for the 7B model. In fact, QL.IP even outperforms SEAL (Wu & Xie 2024) which a heavily optimized version of LLaVA designed specifically address fine- uinengraiedd VQA (see Table 2b. We found that peak POPE performance occurred with the smallest image size we tested (shortest edge is 224 pixels), and an 0.7, corresponding to slightly less than 50% of the baseline image tokens.
ABLATIONS
We ablate our design decisions along two axes. The first axis is along interpolation strategy. where show that our MLP network vastly outperforms bilinear and bicubic interpolation. Next, demonstrate that our performance improvements from the quadtree mechanism are predicated on selection strategy and due solely to a reduced token counts. More detailed ablations are contained in Appendix G
8

---

Published as a conference paper at ICLR 2026
224
280
336
392
0.5
0.5
0.5
0.5
0.4
0.4
0.4
0.4
448
504
560
616
0.5
0.5
0.5
0.45
0.4
0.4
0.4
0.40
0.35
100
200
100
200
100
700
300
100
700
300
Figure 7: Ablation on V* with QLIP-13B. The black curves are QLIP, with derivative pruning, and the green curves are QLIP with random pruning. The green curves are plotted with min/max lightly shaded, and the first standard deviation more darkly shaded. Each of the evaluations with the random selection strategy was run 10 times to compute the average and standard deviation. The z-axis is the percentage of image tokens seen compared to baseline, and the y-axis is accuracy. Each pane is labeled with its image size and the vertical and horizontal blue dashed lines represent baseline number of image tokens and baseline accuracy respectively.
MLP Interpolation is Essential for Generalizing to Arbitrary Image Sizes: We experiment with using bicubic interpolation to scale evaluation with image size. We find that across all of our benchmarks bicubie and bilinear interpolation under-perform our MLP interpolation. This is clearly demonstrated for V benchmark in Figure where the bicubic and bilinear interpolation schemes under-perform even the baseline model performance average.
Performance Gains are not Solely a Result of a Reduced Number of Image Tokens: We verify that the derivative selection strategy provides meaningful information signal the downstream LLM by comparing it to using a random selection strategy which prunes quadtree branches at some random rate. We compare the performance of these two selection strategies on the V benchmark in Figure / where keeping precise semantic information about particular regions of the image is critically important. We find that on average there are large gaps in performance between random selection and derivative selection, indicating that our derivative selection strategy provides a more meaningful visual signal to the model.
6 RELATED WORK
Improved Vision Encoders and MLLMs The observation that grid patchification at a fixed image resolution is a poor inductive bias is not new. This has led to litany of proposed replacements for
CLIP
Radford et al.
202T) and ViT (Dosovitskiy et al. 2020). For example, the studies (Bolya et al.
2022
Darcet et al.] 2023: Dehghani et 2023 Duggal al. 2024; Fan et al.] 202T, Haurum et al.
2023
Kong et al.
2022, Lee et al.) [2022] Marin et al.) 2023 Meng et al. 2022 Oquab et al.) 2023
Yang al. 2022 Zhang et al. (022) propose modifications to the ViT architecture which provide
better visual signal. These studies do not attempt to train an attendant LLM to create an MLLM. The
studies (Bigverdi et al. 2024; Guo et all 2024 Liu et al. 2024a Lu et al., 2022, Luo et al.) 2024
Shi et al. [024 Thapa et al. 2024 Tong al. 2024 Wang et al. 2024 Wu et al. 2024 Yang et al.
2022] Zhang et al.] 2025) introduce new vision encoders specifically ly in the context of MLLM, but
require pre- tra-iiainin and instruction tuning. The most closely related result work ours is by Shi et
al. (Shi et al.) 2024) who show that LLaVA performance can be increased substantially by feeding the [.LM visual tokens from different scales while keeping the CLIP encoder frozen. We go beyond all of these studies by obtaining improved performance using the same underlying MILM backbone, with no pre-training, instruction-tuning or supervised fine-tuning of the language model.
Newer vision encoders such as Intern or Qwen-V employ integrated token reduction, multi- resolution training, or relative/rotary position encodings, which address interpolation and resolution handling in different way than we have here. These architectural choices make direct integration of QLIP non- con-trivial, and we leave adaptation to such encoders as future work.
Token Pruning and Merging Many MILM studies have been directed at reducing the number of visual input tokens, either by pruning tokens or merging them. Such reductions are well-motivated. (Levy et al. 2024 Velicković et al. 2024) show that in addition to being computationally expensive, feeding an LLM too many tokens can harm performance. Recent work has also demonstrated that MLLMs rely heavily the [CLS] token during VQA (Zhang et al.] 2024), which helps explain
why previous authors have been able to remove up to 95% of the visual tokens and nearly maintain
MILM performance (Coo al. 2023
Chen al. 2024 Hu et al. 2024 Sun et al. 2025
Tang et al.

---

Published as a conference paper at ICLR 2026
prune tokens across video frames while maintaining performance (Choudhury et al. 2024). key distinction is that QLIP operates before encoding, at the image patchification stage, whereas most pruning and merging methods operate after encoding and typically trade accuracy for efficiency. QLIP instead improves the signal quality during patchification rather than pruning encoded tokens. Additionally, all of these studies require expensive pre-training and fine-tuning stage to align the
2022),
LLM with the vision encoder. Furthermore, our work is orthogonal to the studies (Cao et al. 2023
Hu al. 2024 Sun al. 2025 Tang al. 2022 since these models rely on training LLaVA family models while using the CLIP encoder, which can be replaced in their studies by QLIP.
CONCLUSION
We have proposed QLIP, a drop-in, adaptive, and content-aware replacement for the CLIP encoder. We defined mesoscopic bias and interpolation bias, argued that these biases are responsible for performance difficulties on fine-graing fine-grained VQA, and shown that QLIP satisfactorily addresses these biases. We achieve + +13.6% accuracy on the challenging V* benchmark without retraining the vision encoder or LLM. We are also able to exceed baseline performance on V while using fewer image tokens. On other benchmarks, we show that can nearly match exceed baseline performance.
REFERENCES
Mahtab Bigverdi, Zelun Luo, Cheng- Hsieh, Ethan Shen, Dongping Chen, Linda G Shapiro, and Ranjay Krishna. Perception tokens enhance visual reasoning in multimodal language models. arXiv preprint arXiv:2412.03544, 2024.
Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffmann. Token merging: Your vit but faster. arXiv preprint arXiv:2210.094611 2022.
Qingqing Cao, Bhargavi Paranjape, and Hannaneh Hajishirzi. Pumer: Pruning and merging tokens for efficient vision language models. arXiv preprint arXiv:2305.17530 2023.
Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision- language models. In European Conference on Computer Vision, pp. 19-35. Springer, 2024.
Rohan Choudhury, Guanglei Zhu, Sihan Liu, Koichiro Niiuua,, Kris Kitani, and László Jeni. Don't 't look twice: Faster video transformers with run-length tokenization. Advances in Neural Information Processing Systems, 37:28127-299
2024.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. arXiv preprint 2rX.v:2309.16588, 2023.
Mostafa Dehghani, Basil Mustafa, Josip Djolonga, Jonathan Heek, Matthias Minderer, Mathilde Caron, Andreas Steiner, Joan Puigcerver, Robert Geirhos, Ibrahim M Alabdulmohsin, et al. Patch n'pack: Navit, a vision transformer for any aspect ratio and resolution. Advances in Neural Information Processing Systems, 36:2252-2221 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. leee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly. et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 19929, 2020.
Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang. Pan Zhang, Jiaqi Wang, al. VImevalkit: An open-source toolkit for evaluating large multi-modality models. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 1 1118-11111 1201, 2024.
10

---

Published as a conference paper at ICLR 2026
Shivam Duggal, Phillip Isola, Antonio Torralba, and William T Freeman. Adaptive length image tokenization via recurrent allocation. In First Workshop on Scalable Optimization for Efficient and Adaptive Foundation Models, 2024.
Haoqi Fan, Bo Xiong. Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In Proceedings of the IEEF/CVF international conference on computer vision, pp. 6824-6835, 2021.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang. Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 13344, 2023.
Zonghao Guo, Ruyi Xu, Yuan Yao, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, and Gao Huang. Llava-uhd: an Imm perceiving any aspect ratio and high-resolution images. In European Conference on Computer Vision, pp. 390-406. Springer, 2024.
Joakim Bruslund Haurum, Sergio Escalera, Graham Taylor, and Thomas Moeslund. Which tokens to use? investigating token reduction in vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 773-783, 2023.
Byeongho Heo, Song Park, Dongyoon Han, and Sangdoo Yun. Rotary Position Embedding for Vision Transformer, July 2024. URL http:://axii..orrtt/// http://bb/2203332222 arXiv:2403.13298
Jeremy Howard. Imagenctte: A smaller subset of 10 easily classified classes from imagenet, March 2019. URL http:pee//.iitlto.ccc//aaaaaaaat
Wenbo Hu, Zi-Yi Dou, Liunian Li, Amita Kamath, Nanyun Peng, and Kai-Wei Chang. Matryoshka query transformer for large vision-language models. Advances in Neural Information Processing Systems, 37:50168 501888, 2024.
Gregory M Hunter and Kenneth Steiglitz. Operations on images using quad trees. IEEE Transactions on Pattern Analysis and Machine Intelligence, (2):145-153, 1979.
Diederik Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.698,, 2014.
Zhenglun Kong, Peiyan Dong, Xiaolong Ma, Xin Meng, Wei Niu, Mengshu Sun, Xuan Shen, Geng Yuan, Bin Ren, Hao Tang, et al. Spvit: Enabling faster vision transformers via latency-aware soft token pruning. In European conference on computer vision, pp. 620-640. Springer, 2022.
Youngwan Lee, Jonghee Kim, Jeffrey Willette, and Sung Ju Hwang. Mpvit: Multi-path vision transformer for dense prediction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7287-7296. 2022.
Mosh Levy, Alon Jacoby, and Yoav Goldberg. Same task, more tokens: the impact of input length on the reasoning performance of large language models. arXiv preprint arXiv:2402. 14848, 2024.
Bangzheng Li, Fei Wang, Wenxuan Zhou, Nan Xu, and Ben Zhou. Semantic-clipping: Efficient vision-language modeling with semantic-guided visual selection. arXiV.
Siting Li, Pang Wei Koh, and Simon Shaolei Du. Exploring how generative milms perceive more than clip with the same vision encoder. htpps//ariixorrgiis22411..0000 2024.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang. Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355 2023.
Haotian Liu, Chunyuan Li. Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in
neural information processing systems, 1834892-34916 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IÉEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26296-26306, 2024a.
11

---

Published as a conference paper at ICLR 2026
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? Springer. 2024b.
In European conference computer vision, pp.
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song- Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurlPS), 2022.
Gen Luo, Yiyi Zhou, Yuxin Zhang, Xiawu Zheng, Xiaoshuai Sun, and Rongrong Ji. Feast your eyes: Mixture-of-resolution adaptation for multimodal large language models. arXiv preprint arXiv:2403.03003 2024.
Dmitrii Marin, Jen-Hao Rick Chang, Anurag Ranjan, Anish Prabhu, Mohammad Rastegari, and Oncel Tuzel. Token pooling in vision transformers for image classification. In Proceedings of the IEEE/CVF winter conference applications of computer vision, pp. 12-21, 2023.
Lingchen Meng, Hengduo Li, Bor-Chun Chen, Shiyi Lan, Zuxuan Wu, Yu-Gang Jiang, and Ser-Nam Lim. Adavit: Adaptive vision transformers for efficient image recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12309-12318, 2022.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PmLR, 2021.
Baifeng Shi, Ziyang Wu, Maolin Mao, Xin Wang, and Trevor Darrell. When do we not need larger vision models? In European Conference on Computer Vision, pp. 444-462. Springer, 2024.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputting 568: 127063, 2024.
Yizheng Sun, Yanze Xin, Hao Li, Jingyuan Sun, Chenghua Lin, and Riza Aatista-Navarro Lypruning: An effective yet simple language-guidee vision token pruning approach for multi-modal large language models. arXiv preprini 2501::2501.1365. 2025.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in neural information processing systems, 33:7537-7547, 2020.
Shitao Tang, Jiahui Zhang. Siyu Zhu, and Ping Tan. Quadtree attention for vision transformers. arXiv preprint arXiv:2201.0222 002667, 2022.
Rahul Thapa, Kezhen Chen, Ian Covert, Rahul Chalamala, Ben Athiwaratkun, Shuaiwen Leon Song, and James Zou. Dragonfly: Multi-resolution zoom-in encoding enhances vision-language models. arXiv preprini 2rXiv:2406.00077 2024.
Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu, Ziteng Wang, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal 1mms. Advances in Neural Information Processing Systems, 37:87310-87355 87356.7 2024.
Petar Veličković, Christos Perivolaropoulos, Federico Barbero, and Razvan Pascanu. softmax is not enough (for sharp out-of-distributionn) arXiv preprint arXiv:2410.01 104, 2024.
Gregory K Wallace. The jpeg still picture compression standard. IEEE transactions on consumer electronics, -xvixxxii,, 1992.
12

---

Published as a conference paper at ICLR 2026
Yonghui Wang, Wengang Zhou, Hao Feng, and Houqiang Li. Adaptvision: Dynamic input scaling in milms for versatile scene understanding. arXiv preprint arXiv:2408. 16986, 2024.
Penghao Wu and Saining Xie. V Guided visual search as core mechanism in multimodal Ilms. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 13084-13094, 2024.
Shengqiong Wu, Hao Fei, Xiangtai Li, Jiayi Ji, Hanwang Zhang, Tat-Seng Chua, and Shuicheng Yan. Towards semantic equivalence of tokenization in multimodal Ilm. arXiv preprint arXiv:2406.05117, 2024.
xAI Team. Grok-1.5 vision preview, 2024. URL /tL.tttp:://....///ee///o.o/
An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang. Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:241..15115. 2024.
Tao Yang, Yuwang Wang, Yan Lu, and Nanning Zheng. Visual concepts tokenization, Advances in Neural Information Processing Systems, 35:31571-31582, 2022.
Jiaxin Zhang, Wentao Yang, Songxuan Lai, Zecheng Xie, and Lianwen Jin. Dockylin: A large multi- modal model for visual document understanding with efficient visual slimming. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pp. 9923-992233 2025.
Qiming Zhang, Yufei Xu, Jing Zhang, and Dacheng Tao. Vsa: Learning varied-size window attention in vision transformers. In European conference on computer vision, pp. 466-483. Springer, 2022.
Qizhe Zhang, Aosong Cheng, Ming Lu, Zhiyong Zhuo, Minqi Wang, Jiajun Cao, Shaobo Guo, Qi She, and Shanghang Zhang. [cls] attention is all you need for training-fiee visual token pruning: Make vlm inference faster. arXiv preprint arXiv:2412.01822 2024.
13


Reviews:
[{"review_id": "5IBcOIDGXs", "reviewer": "Reviewer_y52t", "summary": "This paper targets the limitations of the standard CLIP vision encoder used in Multimodal Large Language Models (MLLMs), specifically its fixed-resolution training and uniform patch grid1. The authors diagnose two key issues: \"mesoscopic bias\" (a preference for features at a specific training scale ) and \"interpolation bias\" (an inability to generalize positional encodings to new resolutions ). They propose QLIP, a \"drop-in\" replacement that combines two components: a content-aware \"quadtree patchification\" (QtP) mechanism to dynamically merge low-information patches based on image gradients , and a small, trained MLP to interpolate positional encodings for arbitrary coordinates. The central claim is that QLIP can be integrated into existing MLLMs like LLaVA without retraining the full model , leading to significant performance gains, most notably a +13.6% accuracy boost on the fine-grained $V^{*}$ benchmark.", "questions": "1. On Performance Inconsistency: The performance on $V*$ is excellent, but the model gets worse on MM-Bench. Do the authors have a hypothesis for this trade-off? Does the QtP's gradient-based pruning accidentally discard global, \"mesoscopic\" context that is necessary for general VQA benchmarks but less important for $V^{*}$?\n\n2. On Hyperparameter Sensitivity: Given the extreme sensitivity to $\\alpha$ and image size shown in Figure 6, how do you justify the \"plug-and-play\" claim? What concrete, general-purpose default values would you recommend to a user who cannot afford to run the \"sweeps\" you performed?On MLP Training: The MLP was trained on Imagenette. \n\n3. How confident are you that this MLP generalizes to image domains outside of natural images (e.g., medical scans, line drawings, or thermal imagery)? Does the \"drop-in\" claim only apply if the target domain is similar to Imagenet?\n\n4. On Inference Overhead: What is the latency overhead of the quadtree patchification step during inference? Calculating the gradient map and recursively building the tree seems computationally non-trivial compared to a simple uniform grid patch.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Excellent Problem Diagnosis: The paper provides a clear and valuable conceptual framework by identifying and naming \"mesoscopic bias\" and \"interpolation bias\". This diagnosis of why CLIP fails at high resolutions is a useful contribution to the field.Strong Target-Task Performance: The +13.6% gain on the $V^{*}$ benchmark is extremely significant. \n\nThis is a challenging benchmark designed to test the exact fine-grained failures of MLLMs, so this result strongly suggests the method is effective for its intended purpose.Token Efficiency: The quadtree mechanism (QtP) not only prunes tokens but can also improve performance with fewer tokens than the baseline. \n\nThis is a highly desirable outcome, demonstrating a successful, content-aware pruning strategy that removes noise/redundancy rather than just information.Low-Cost Intervention (in theory): The core idea of replacing a small part of the vision encoder without retraining the entire MLLM is highly practical and computationally appealing.", "weaknesses": "1. The paper's components are not fundamentally new. Quadtrees are a classic data structure, and their application to vision transformers and dynamic tokenization has been explored. Similarly, the token pruning/merging field is already well-established. The idea of replacing static positional embeddings with a dynamic, coordinate-based MLP is also a known technique (e.g., in NeRF). The novelty lies in the combination for this specific MLLM problem, but the components themselves are iterative.\n\n2. The method's impressive gains are not general. The +13.6% boost on $V*$ is met with a performance decrease on MM-Bench (-2.8% for 7B) and stagnation or minor losses on several other benchmarks (e.g., Sci-QA, CV-Bench). This strongly suggests the method is an overspecialization, it trades general VQA capability for fine-grained acuity. The gains do not generalize.Misleading Baselines: The paper highlights outperforming the $S^{2}$ model. However, it also cites SEAL, which achieves a much higher 75.4% on $V^{*}$ overall. While SEAL requires full retraining, this massive 17-point performance gap shows that QLIP is far from the state-of-the-art in solving fine-grained VQA, even if its method (drop-in) is cheaper.\n\n3. \"Plug-and-Play\" Claim is Misleading: The method is not \"plug-and-play.\" It introduces highly sensitive new hyperparameters, namely the image size and the quadtree pruning threshold $\\alpha$. The paper states that the reported results are the \"best score from our sweeps\". Figure 6 clearly shows a chaotic and non-monotonic relationship between $\\alpha$, image size, and performance. A user cannot simply \"drop in\" QLIP; they must perform an expensive, multi-axis hyperparameter sweep for their specific task to have any hope of replicating the results, which undermines the entire premise of a simple, no-cost intervention.\"\n\n4. No Retraining\" Claim is Misleading: The paper repeatedly claims \"no re-training\". However, the core MLP interpolation network must be trained. The authors trained it for 11 hours on four NVIDIA L40S GPUs. While this is vastly cheaper than retraining an MLLM, it is not \"no retraining.\" It is a separate, required training step that introduces a dependency on a new dataset (Imagenette) and process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IQEnjPJ5BT", "reviewer": "Reviewer_cr9P", "summary": "CLIP is employed as the vision encoder for many MLLMs, yet its absolute positional encoding and fixed-resolution pre-training severely degrade performance when images are supplied at non-native resolutions.\nThe paper introduces QLIP, a drop-in replacement that can be inserted into any CLIP-based MLLM without architectural changes.  Instead of uniform patchification, QLIP first performs a content-aware partition that keeps salient regions at higher resolution and fuses the resulting variable-length token sequence into the original embedding dimension.  Extensive experiments show up to 13.6 % absolute improvement on a battery of cross-resolution benchmarks.", "questions": "No", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Simple, training-free at inference time, and demonstrably effective.", "weaknesses": "**Rapidly moving baseline.**  The community is already shifting from CLIP to newer vision backbones (e.g. InternVL, SigLIP-2) that use RoPE or 2-D absolute + relative encoders and are pre-trained with native multi-resolution recipes.  It is unclear whether QLIP retains any advantage when the underlying encoder itself is resolution-robust.  A head-to-head comparison with such models is missing.\n\n**Limited to CLS-level bias.**  QLIP is optimised to deliver a single, high-quality CLS token; it does not guarantee per-token fidelity.  For dense-prediction tasks (segmentation, object detection) that require spatially accurate patch-wise features, the content-aware patchification may discard positional details and harm downstream accuracy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RtgFcf1mj5", "reviewer": "Reviewer_a7Xx", "summary": "This paper introduces QLIP, a novel, lightweight, and drop-in modification for the CLIP vision encoder designed to enhance the performance of Multimodal Large Language Models (MLLMs), such as LLaVA, without requiring expensive retraining of the entire MLLM backbone. The authors attribute limitations of the standard CLIP encoder—specifically its fixed-resolution nature and poor handling of fine-grained details—to two biases: Mesoscopic Bias and Interpolation Bias. QLIP addresses these issues via two components: Quadtree Patchification (QtP) and Coordinate-Based MLP. Experiments show that QLIP achieves substantial performance improvements, notably a +13.6% accuracy boost on the challenging $V^{*}$ benchmark for fine-grained VQA, and successfully mitigates model hallucination.", "questions": "The QtP process involves a dynamic computation (quadtree traversal and gradient calculation). Can the authors quantify the computational overhead of the QtP stage in terms of wall-clock time? How does this overhead scale with very high input resolutions? Is the total prefill time (QtP + CLIP encoding) still significantly faster than a simple, high-resolution baseline that uses linear interpolation?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Practical & Cost-Effective Drop-in Solution: QLIP’s design as a \"drop-in replacement\" is highly impactful. By significantly enhancing the visual signal without necessitating the re-training or fine-tuning of the entire MLLM pipeline, it offers a practical, low-cost path to upgrading existing MLLMs.\n\n2. Clear Theoretical Motivation: The paper clearly and quantitatively identifies two specific, fundamental biases in the CLIP vision encoder (mesoscopic and interpolation bias). The proposed solutions are elegantly tailored to directly address these theoretical deficiencies.\n\n3. Content-Aware Patchification: The use of Quadtree Patchification (QtP) with a simple gradient-based criterion provides an effective, training-free mechanism for adaptively merging semantically similar regions. This mechanism also has the beneficial side effect of reducing input token count, which is shown to decrease hallucination.", "weaknesses": "1. The base model is out of data. LLaVA suffers limited performance. I suggest the authors to conduct evaluation on more SOTA models, such as InternVL-3.5 or QwenVL2.5 to truly demonstrate the effectiveness.\n\n2. I wonder the performance on visual grounding benchmarks, such as refcoco, since it also requires fine-grained region information.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l7ZW5ql4U7", "reviewer": "Reviewer_5fie", "summary": "This work proposes Q-LIP, a plug-and-play CLIP replacement that integrates into MLLMs with minimal code (no full retraining) and uses content-aware quadtree patchification to boost visual understanding.​ Experiments show QLIP improves LLaVA-1.5’s performance in several benchmarks, without full MLLM retraining/fine-tuning.", "questions": "Refer to weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. QLIP fixes CLIP’s mesoscopic/interpolation biases with quadtree patchification and a small MLP, handling arbitrary resolutions without encoder weight changes.\n2. As a drop-in CLIP replacement, it boosts LLaVA’s  benchmark accuracy by 13.6% without MLLM retraining/fine-tuning.\n3. QLIP improves token efficiency (higher accuracy with fewer tokens) and reduces hallucination, performing well on several benchmarks.", "weaknesses": "1. The application of the proposed method is limited. Currently, many large models natively support input image at arbitrary resolutions.\n2. The generalization of the method is not validated. This study only conducts experiments on LLaVa-1.5 (7B and 13B), and experiments on additional models should be added.\n3. It is advisable to add comparisons between the proposed quadtree patchification and some token merging/pruning methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)