Dragonfly: Multi-Resolution Zoom-In Encoding Enhances Vision-Language Models

Authors: Rahul Thapa, Kezhen Chen, Ian Connick Covert, Rahul Chalamala, Ben Athiwaratkun, Shuaiwen Leon Song, James Zou
Primary area: foundation or frontier models, including LLMs
Keywords: Multimodel Language Model, Visual Instruction Tuning, Biomedical multimodal model, foundation model

Abstract:
Recent advancements in vision-language models (VLMs) have highlighted the benefits of processing images at higher resolutions and leveraging multi-crop features to retain native resolution details. However, current vision transformers (ViTs) often struggle to capture fine-grained details from non-dominant objects, charts, and embedded text, limiting their effectiveness in certain tasks. In this paper, we push beyond the conventional high-resolution and multi-crop techniques by not only preserving but also zooming in past the native resolution of images. This enhancement allows our model to better extract fine-grained details, overcoming the limitations of current ViTs. To manage the increased token count and computational complexity, we show that a simple mean-pooling aggregation over tokens is effective. Our model, Dragonfly, achieves competitive performance on general tasks such as ScienceQA and AI2D, and excels in tasks requiring fine-grained image understanding, including TextVQA and ChartQA. On average, across ten general-domain benchmarks, Dragonfly ranks at the top, outperforming models that are significantly larger or trained on much larger datasets. Notably, Dragonfly sets new benchmarks on several biomedical tasks, achieving 91.6\% accuracy on the SLAKE (compared to 84.8\% for Med-Gemini) and a 67.1\% token F1 score on Path-VQA (compared to 62.7\% for Med-PaLM M). On biomedical image captioning tasks, Dragonfly attains state-of-the-art results majority of the performance metrics.

Decision: reject
Reviewer scores: 5, 5, 6

Reviews and discussion:
<thread>
creation date:
1730431849920

thread_type:
official_review

<thread_object>
creator: reviewer_etvr

<content>
summary:
The manuscript presents Dragonfly, a novel Vision-Language Model (VLM) that employs a multi-resolution zoom-in encoding strategy to enhance fine-grained visual understanding. Unlike conventional Vision Transformers (ViTs) that downsample images to fixed, low resolutions—thereby losing critical details—Dragonfly processes images at higher resolutions and employs a multi-crop technique that exceeds the native resolution. This approach allows the model to capture intricate details from non-dominant objects, charts, and embedded text, which are often challenging for existing ViTs. To address the computational complexity arising from the increased token count, Dragonfly utilizes a mean-pooling aggregation strategy. The model demonstrates competitive performance across ten general-domain benchmarks and sets new benchmarks in several biomedical tasks, outperforming larger models trained on significantly more extensive datasets.

strengths:
1. Dragonfly introduces an innovative multi-resolution zoom-in encoding strategy that surpasses native image resolutions, enabling the capture of intricate details from non-dominant objects, charts, and embedded text.
2. It implements a simple yet effective mean-pooling aggregation method and achieves good performance across a diverse set of benchmarks.

weaknesses:
1. Novelty. The proposed method builds upon existing multi-resolution and multi-crop techniques without offering substantial novel contributions. The idea of processing images at higher resolutions and using multi-crop strategies has been explored in prior works, and Dragonfly does not sufficiently differentiate itself beyond these established methods.
Model Comparison: Dragonfly is developed using the more advanced Llama3 model, whereas comparable methods utilize less capable language models, such as Llama2 and Qwen2. This discrepancy raises concerns about the fairness of comparisons. How does Dragonfly's performance measure up when evaluated against these models?
Data Influence: It is unclear whether the observed performance improvements with Dragonfly stem from the curated data or from the model's design. How does Dragonfly perform when tested with a commonly used dataset?

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730555824686

thread_type:
official_review

<thread_object>
creator: reviewer_wn6b

<content>
summary:
This paper introduces multi-crop techniques beyond the native resolution for high-resolution images. To handle the huge number of numbers, the authors employ the average pooling startegy on each crop. Except for general domain of benchmarks on fine-grained image understanding, this paper also introduce the contributioni on biomedical tasks. They also curate a SFT dataset including 2.4M general images and 1.4M biomedical images for training.

strengths:
1. The ablation studies proves their proposed strategy.
2. Biomedical domain is considered by this paper.
3. The motivation is nature and easy to follow.
4. A SFT dataset with different domains and huge number of images is built.

weaknesses:
1. The novelty is limited. The proposed strategy is only an extension of any-resolution technical. Compared to any-resolution which uses two levels of rosulotions, they only resize the image, crop more patches and use three levels of resolutions of image.
2. The visualizations only shows the response of the proposed Dragonfly, other MLLM's responses are encouraged to be listed for better comparision.
3. To balance the computational costs and performance, they use mean pooling within each crop. The paper lacks the dicussion about how to choose a proper compressing ratio for trade-off.

questions:
Please see weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730718376425

thread_type:
official_review

<thread_object>
creator: reviewer_njjw

<content>
summary:
The paper introduces DragonFly to enhance vision-language models. The main idea is to combine the multi-cropping with mean pooling, so that the VLM can use high-resolution image encoders and work on images' native resolution, while ensuring the efficiency of the model. The proposed enhanced VLM performs well in tasks needing finer details, such as biomedical tasks.

strengths:
1. Dragonfly uses multi-crop techniques to process images at high resolutions, thus enhancing the model’s capability to capture fine details. Meanwhile, it uses a simple mean-pooling strategy to reduce visual tokens effectively, which retains the efficiency of the model.
2. The proposed method achieves competitive performance across multiple benchmarks and shows strong generalizability across both general and specialized biomedical tasks.

weaknesses:
The paper uses a very simple strategy that combines multi-crop and mean-pooling to enhance VLM. However, the motivation for choosing mean-pooling instead of other compression techniques is not clearly stated. It's like an experimental report that simply states this method is effective. So why does mean-pooling outperform other strategies? Why do you choose such a pooling window? Will this direct pooling harm the extraction of the fine details? A more comprehensive analysis is expected.

questions:
Please refer to the weaknesses part. Additionally, is there a comparison of the inference time or flops of different methods?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732219144557

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to clarify Motivation

comment:
We thank the reviewer for taking their time to review our paper and provide insightful feedback. Below, we show result for an additional ablation that we ran to clarify our motivation.

### **Additional Ablations to Clarify Relative Importance of Zooming In**

To further clarify the source of our improvements and differences from prior work,, we conducted a controlled experiment to disentangle the benefits of zooming in and preserving native-resolution details. Our original results in **Tables 1-2** show that zooming beyond native resolution enhances performance by capturing fine-grained image details. To assess how much gain comes primarily from solely zooming in, we ran the following new experiment, the results of which are shown in **Table 3**. 

| **Metric**               | **Low-Resolution** | **Medium-Resolution from Low-Resolution** |
|----------------------|--------------------|-------------------------------------------|
| AI2D                      | 60.6                    | 62.9                                                    |
| ScienceQA            | 76.0                    | 77.6                                                    |
| ChartQA                | 21.6                    | 52.4                                                    |
| POPE                    | 83.4                    | 85.1                                                    |
| GQA                      | 49.5                    | 54.7                                                    |
| TextVQA                | 40.0                    | 57.4                                                    |
| VizWiz                   | 57.4                    | 58.0                                                    |
| MME Perception   | 1205.3                | 1398.9                                                |


For the first run, we rescaled all images to a low resolution of $336 \times 336$ and generated 576 tokens from that single image, represented as "Low-Resolution" in Table 3 (and identical to our "low resolution" entry in Table 2). From this setup, we then conducted an experiment where we zoomed in $2 \times$, generating images of size $672 \times 672$ and producing four crops from the rescaled image. Each crop was passed through the ViT, generating 576 tokens, which we then pooled down to 144 tokens per crop, for a total of 576 tokens across all crops. Now, this approach has the exact same raw image information and final image tokens as the "Low-Resolution" baseline, and its only benefit is zooming in. In Table 3, this is the "Medium-Resolution from Low-Resolution" column, and it outperforms the "Low-Resolution" model in all benchmarks, particularly in tasks like ChartQA and TextVQA where fine-grained information is critical. This shows that solely zooming in and featurizing specific sub-regions, without requiring additional native-resolution image information, significantly improves performance.

For further details, please refer to **Section 4.4** in our updated submission.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732555990248

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Gentle Reminder: Feedback on Rebuttal

comment:
Dear Reviewers, 

I hope this message finds you well. I am writing to kindly remind you of the rebuttal submitted for our paper in response to the initial reviews. We have addressed the concerns raised and provided additional clarifications and results to improve the manuscript.

We would greatly appreciate it if you could take a moment to review our responses and share any further feedback or questions you may have. Your insights are invaluable to us in refining our work.

Thank you for your time and consideration. We look forward to your comments.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This submission received two negative sores and a positive score after rebuttal. After carefully reading the paper, the review comments, the AC can not recommend the acceptance of this submission, as the average score is under the threshold bar and the concerns about the proposed approach remain. The AC also recognizes the contributions confirmed by the reviewers, and encourages the authors to update the paper according to the discussion and submit it to the upcoming conference.