Feast Your Eyes:  Mixture-of-Resolution Adaptation for Multimodal Large Language Models
Authors: Gen Luo
Yiyi Zhou
Yuxin Zhang
Xiawu Zheng
Xiaoshuai Sun
Rongrong Ji
Venue: ICLR 2025 Poster (2025)
Keywords: high-resolution adaptation
multimodal large language models
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: In existing multimodal large language models (MLLMs), image resolution plays a significant role for  granular visual recognition.  However, directly increasing image resolution leads to expensive computational cost for MLLMs.  In this paper, we  reveal that a combination of low- and high-resolution visual features can efficiently mitigate this shortcoming.  Based on this principle, we propose a novel and efficient method for MLLMs, termed Mixture-of-Resolution Adaptation (MRA). In particular, MRA adopts two visual pathways for  images of different resolutions, where  high-resolution visual information is embedded into the low-resolution pathway via the novel mixture-of-resolution adapters (MR-Adapters). This design also   greatly  reduces the input sequence length of MLLMs. To validate MRA, we apply it to a recent MLLM called LLaVA, and term the new model LLaVA-HR. We conduct extensive  experiments on 17 vision-language (VL) tasks, which show that LLaVA-HR outperforms existing MLLMs on 15 VL tasks, e.g., +5.2\% on TextVQA.  More importantly,    both training and inference  of LLaVA-HR remain efficient with MRA, e.g.,  20 training hours and  faster inference speed  than LLaVA-NeXT.  Source codes are  released at: https://github.com/luogen1996/LLaVA-HR.

Published as a conference paper at ICLR 2025
FEAST YOUR EYES: ADAPTATION FOR MULTIMODAL LARGE LANGUAGE MODELS
Gen Luo Yiyi Zhou', Yuxin Zhang', Xiawu Zheng', Xiaoshuai Sun', Rongrong Ji "eey Laboratory of Multimedia Trusted Perception and Efficient Computing. Ministry of Education of China, Xiamen University, 361005, P.R. China. 2OpenGVLab, Shanghai Al Laboratory.
ABSTRACT
In existing multimodal large language models (MILMs), image resolution plays a significant role for granular visual recognition. However, directly increasing image resolution leads to expensive computational cost for MLLMs. In this paper, we reveal that a combination of low- and high-resolution visual features can efficiently mitigate this shortcoming. Based on this principle, we propose a novel and efficient method for MILMs, termed Mixture-of- Resolltti Rf Adaptation (MRA). In particular, MRA adopts two visual pathways for images of different resolutions, where high-resolution visual information is embedded into the low- resolution pathway via the novel mixture-of resolution adapters (MR- -dd-tteeers). This design also greatly reduces the input sequence length of MLLMs. To validate MRA, apply it to recent MILM called LLa LLaVA, and term the new model LLaVA-HR. We conduct extensive experiments on 17 vision-language (VL) tasks, which show that LLaVA outperforms existing MLLMs on 15 VL tasks, e.g., +5.2% on TeXIVQA. More importantly, both training and inference of LL-VA-HR remain efficient with MRA, e.g., 20 training hours and faster inference speed than LLaVA-NeXT. Source codes are released at: LLaVA-HR
INTRODUCTION
Driven by the remarkable success of large language models (LLMs) TTuvron et al. 2023 Chen ct al. 2020), research on multi-modal large language models (MLLMs) also receives an influx of interest both academia and industry (Liu al.. 2023b Luo al. 2023 Alayrac et al. 2022
Chen et al. 2022 2023c). Numerous efforts have been recently devoted to extending LLMs to more modalities, achieving breakthroughs on various linguael tasks (Goyal et al. 2017 Singh et al.)
2019: Hudson & Manning 2019). Despite their success, existing MILMMs still fall short of granular Visual recognition. For instance, the powerful GPT4-V also suffers from visual hallucinations when identifying small and occluded objects (Tong et al. 2024). This shortcoming inevitably limits the practical use of MLLMs.
To compensate for this shortcoming, early practitioners often resort to scaling up model size and increasing per-training data size (Alayrac et al.] 2022] Li et al.] 2023b Bai et al. 2023). For instance, InstructBLIP (Dai et al. 2023) adopts over 129M image-text pairs for lasion-language (VL) alignments, showing that a larger visual encoder is beneficial for MI.LMs. Similarly, Qwen-VI. (Bai al. 2023) also increases the parameters of visual encoder 1.9 billion and uses 1.5 billion image- text pairs for pre-training. Despite effective, this paradigm is prohibitively expensive, which often consumes about thousands of GPU hours.
Orthogonal to these works, we study the visual shortcoming of MLLMs from the perspective of image
resolutions. As revealed in previous VL research (Jiang et al., 2020 Tong et al. 2024), increasing the resolution of input images is straightforward solution for visual recognition, which becomes more important for MLLMs that involve fine- ann-ggained visual reasoning (Rose et 2023). As shown
Correspoondngg author.

---

Published as a conference paper at ICLR 2025
OKYQA
POPE
65
768 piv
1024 pix
pix
TeteVQ
144S pix
T.aVA-H (ours)
Acciaay
60
pixi
336 pix.
37.80
1024 pis
SFED
QwenVI-Chhtt
Owf2-7B
VQAv2
pix
LLaVA-IR (ouns)
InstructBLIP
555
LLaVA-1.5
HLVA-NoXT
BLsI
QuenVVL
MatiVista
Mini-Geminl
224 pix
50
mPLUG-Owl2
LLAVA-15
MM-Vet
400
500
600
700
800
Inference Speed (ms)
MMMM
SQA-
Figure I: Comparison between existing MLLMs and VL- on TextVQA (left) and various benchmarks (right). Increasing image resolution is effective yet expensive for fine- fine-grained visual understanding. In contrast, LLaVA-HR can efficiently adapt high resolution to boost performance.
in Fig. 1 increasing the resolution of LLa (Liu et al. 2023a) from 384 384 to 672 672 can bring obvious performance gains (+4.6%) on TextVQA (Singh et al. 2019). However, the use of high-resolution images will greatly exacerbate the already high computational cost of MILLMs. For instance, 448 x 448 resolution will increase the computation complexity of LLaVA by about 1.4 times compared with the default 336 x 336. In addition, the training will become unstable as the resolution is greatly increased] e.g., sharp drop at 1, 022 x 1, 022 resolution in Fig. Although such issue can be overcome by dividing high-resolution images into small patches via the dynamic slicing strategy Liu et al. (2024a), its computational cost still remains expensive for MLLMs.
In this paper, we focus on the efficient high-resolution image adaptation of MLLMs and propose a novel method called mfxxurreof-resoluttin adaptation (MRA). As shown in Fig. 2 MRA adopts an innovative dual visual pathway design to process the input images of high- and low-resolutions simultaneously. Specifically, one pathway aims to encode global information of low-resolution images, while the other one serves to capture fine-grained semanties from high-resolution images. Meanwhile, these two pathways are closely interacted via the novel mixture-of- resolution reiiturroo adapters (MR-Adapters), which embeds the higo-resolution visual information into the low-resolution modeling. In this way, we can use a much fewer number of visual tokens to represent the input images from macro- to micro-views. With the careful design of -uat-wyt structure, MRA can easily scale the image resolution up to 1,024 x 1,024 pixels while maintaining high efficiency.
To validate MRA, we apply it to a recent MLLM called LLa VA (Liu et al. 2023b a), and term the new model as VLR We conduct extensive experiments on 17 vision-languag (VL) tasks, including common VL tasks like VQAv2 (Goyal et al. 2017) and MLLM benchmarks such as POPE (Li et al. 2023c Experimental results show that LLa VA-HR outperforms existing MLLMs on 15 of 17 VL tasks, e.g., +9.6% over VA- on TextVQA. More importantly, the training and inference of LLLVA-HR are cost-effective. In particular, the pre-training and instruction tuning of LL-VA-HR (7B, 1,024 x 1,024) only take total of 20.7 hours on 8 A800 GPUs, which is hundreds of times cheaper than InstructBLIP (Dai et al., 2023) and Qwen-VL (Bai al.) 2023). Under the same high-resolution setting, its inference speed is consistently faster than LLaVA-I. 1.5 and LLaV Liu al. (2024a).
Liu et al. 2023a)
In summary, our contributions are three folds:
We propose novel and efficient adaptation scheme, termed mixture-of -ffsuuuu-o-- adaption (MRA), which adopts novel dual visual pathway design to obtain the benefits of high- resolution visual information while keeping training and inference efficient.
We propose novel mixture--of-resluttee adapter (MR-Adapter) for MRA, which can embed the high-resolution information into the low-resolution visual pathway to improve visual descriptive power.
Based MRA, we propose a powerful MLLM, coined LLaVA- HR, which outperforms
existing MILMS on 15 of 17 VL tasks and are much more efficient than most ML.MMs.
Visual encoders like CIT are pre-trained with low resolution, and the significant increase of resolution may hurt feature representations.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
2.1
MULTTMODAL. LARGE LANGUAGE MODELS
Driven by the great successes of large language models (LLMs) Gilardi et al. 2023 Touvron et al.
2023:
Chen et al. 2020), growing interest has been aroused in building end to- to-end mu multimodal large
language models (MLLMs) (Liu et al.
2023b Zhu et al. 2023 Luo et al. 2023 Bai et al. 2023 Fuyu-
8B 2023
Peng et al. 2023
Luo et al.
2024a b). In particular, most existing MILMs adopt modular
structure Luo et al. 2023 Liu al.
2023b), which utilizes intermediate network project the
visual features into the word embedding space of the LLM. Then, the LLM is used to accomplish various VL tasks in an autoregressive manner. Based on the modular structure, existing MLLMs can be distinguished by the designs of the intermediate network. Popular MLLMs represented by LLaVA (Liu et al. 2023b) often adopt linear projection layer or an MLP layer connect the visual encoder and the L.M (L.iu et al. 2023b Chen et al.) 2023a e Peng et al. 2023). 'he other works employ sampler-bases modules to bridge the gap between the visual encoder and the LLM (Bai et al.) 2023 Alayrac et al.) 2022] Li et al. (023b). These sampler-based modules can effectively reduce the number of visual tokens, but often requires a large-scale pre-training to achieve a promising performance (Bai et al.
2023 Li et al. 2023b). Despite the effectiveness, the low resouusonttion visual perception still limits the performance of existing MLLMs in fine fine-graine grained tasks.
2.2 HIGH-RESOLUTEE MULTIMODAL, LARGE LANGUAGE MODELS
To improve the perception ability of MLLMs, increasing attentions have been focused on high- resolution MLLMs (Liu et al. 2024a Li et al. 2024c Liu et al. 2024b Li et al. 2024b Chen
et al. 2024b). Among them, most methods (Li et al. 2024c Liu al. 2024a) adopt the dynamic slicing strategy to divide a high-resolution image into multiple low-resolution patches. By doing so, pre-trained visual encoders can maintain their default resolutions for adapting high-resolution processing, and support images with flexible aspect ratio. For example, Monkey (Li et al. (024c) and LLaVA-Next (Liu et al., 2024a) divide input images into set of 448 x 448 patches for high- resolution visual understanding. Based on this framework, Chen et al. (2024b) and (Dng et al. (2024) further explore the strategy realize the optimal image division. Despite the effectiveness, their computational cost is still expensive as the image resolution increases. Orthogonal to these works, we aim to improve image resolution in an efficient way, which still lacks extensive explorations.
2.3
VISUAL REPRESENTATIONS FOR MULTIMODAL LARGE LANGUAGE MODELS
The pursuit of better visual representations has been popular research trend in the VI. community LLu al. 2019 Jiang et al. 2020 Radford et al. (02T). Early endeavors mainly explore the leeect-level features for VL models (Lu et al. 2019] Zhang et al. 2021). Driven by the large-scale image-text
pre-training, grid features from CLIP (Radford et al. 2021 ) have demonstrated the great efficiency and generalization in MLLMs (Liu et | 2023b Chen et al. 2022 Alayrac et al. 2022). Based on grid features, existing researchers mainly improve visual representations by scaling up the visual encoder. For example, PaLI (Chen al. (202)) increases the parameters of visual encoder to billions and shows the significant performance boost of MI.MMs. In contrast to these works, we improve the visual representations for MILLMs from the perspective of dual-branch network interactions, and propose a novel and efficient solution, namely mixture-of-rrsolluioo adaptation.
PRELIMINARY
We first recap the structure of multimodal large language models (MLLMs)), which consists of an image encoder Fz(.), ), an intermediate network F-(() and an LLM Fz(.).
In particular, given input image I #x3 and a textual instruction T R.L, the visual tokens F. E R [xxxx)) w)xx are obtained via the image encoder, and the text tokens ft E IRxd are represented by the corresponding word embeddings. Based on the visual and textual tokens, the LLM will decode the target word step by step, formulated as
S+1
II FP( B0.-1)-
(1)
3

---

Published as a conference paper at ICLR 2025
Low-resotution Image
Low-resottion Patlwway (Marro View) 32x
Text Instruction:
"describe this image in short."
VIT stage
VIT stage
ViT stage
ViT
herd elophants and deer are gathered around a watering hole. The elephants are of various sizes, including baby elophantt The doer are also of different sizees uppearing be young.
Output:
MLP
Tokenizer
448 x 448
MR- Adapter
MR- Adapteri
Adapter;
llgghresolution Image
Multi-head Attention
Cony Stage
Conv slage
Conv
Feed- corwwad Network
slage
slage
LLaMA2-7B
High-resolution Pathway (Micro View)32
1024 1024
Figure 2:
Illustration of Mixture-of-Resolutie Adaptation (MRA) and its deployment LLaVA-HR. MRA employs dual visual pathways to process high-resolution and low-resolution images, respectively. High-resolution information is embeded into the fast pathway via novel mixture-of-resolution adapter (MR-Adapter).
Here, Pt E I denotes the probabilities of the predicted word and is the size of word vocabulary.
In some MLLMs (Liu et al. 2023b a). (.).) is often a stack of simple linear layers, which are used directly project the visual tokens onto the semantic space of LLMs. Although simple and effective, this strategy inevitably leads to longer visual sequence as the resolution increases, e,g. 5,329 tokens for 1,022 x 1,022 resolution LLaVA-1.5 1.5. In practice, processing such large number of tokens is computationally expensive in MLLMs. To further reduce the number of visual tokens, recent advances adopt the sampler-basee basee module for F-(-) (-) e.g., @Former (Li al. 2023b). which aggregates visual features into several query tokens that LLM can directly handle. Nevertheless, these methods often require large-scale pre-training achieve VL alignments (Bai al. 2023).
Based on the above analyses, we conclude that the main difficulty of high- righ-resslution image adaptation lies the rapidly growing visual sequence. This issue motivates us further explore how to efficiently encode richer visual information with fewer visual tokens.
MINTURE-OF-RISOLUTION ADAPTATION
4.1 OVERVIEW
To address the above issues, we propose novel and efficient method for MILMs, termed mixture-of- resolution adaptation (MRA). As shown in Fig. 2 into the low-resolution via a dual pathway design. In this case, MRA can keep smaller number of visual tokens while encoding richer visual information.
MRA aims to embed high-resolution information
In particular, given the input images of two resolutions 4 R Wxxx and In F x3 the process of MRA can be formulated as
F. = FI, (11, (Fubi 02); where Foh FI, (Ini OI,)
(2)
Here, Feh e Rhe Xar, xdn and F, E hxwxx denote the nigh-resolution features and the final visual features, respectively. And Fz, (-) and FI, are the visual encoders for high- and low- resolution images, respectively. FA denotes the coxturre-ofrrsoluute adapter (MR-Adapter). Based on Eq. 2 the obtained visual features will be further processed by the LLMM based on Eq.
4.2 DUAL VISUAL PATHWAYS
As shown in Fig. dual visual pathways, i.e., (.) and Fz, are the key design of MRA. To maximize their benefits, we consider the heterogeneous dual-branch design from two aspects.
Visual functionality. Firstly, the dual visual pathways process images from macro- and micro-views which is inspired by the visual system of human being (Merigan & Maunsell [993] Robertson & Lamb 1991). Particularly, Robertson & Lamb (1991) find that the visual system processes local
4

---

Published as a conference paper at ICLR 2025
and global semantics via different pathways. Similar mechanisms in computer vision are not new. Previous works (Chen et al. 2021; ; Peng et al. 2021) like CrossViT (Chen et al. 2021 typically incorporate this feature into their network design for image classification.
However, the exploration of dual visual pathways in high-resolution adaptation for MLLMs can still bring new insights beyond previous works, Le., fewer visual tokens can also result in stronger visual understanding. Specifically, one visual pathway aims to capture fine- grained semantics from high-resolution images i.e., processing images from local view. The other pathway designed to encode global information from low-resolution images for a larger receptive field. In this case, MRA can not only efficiently process high-resolution images, but also greatly benefits from two complementary visual semantics.
Visual alignment. The alignment of two pathways also challenging MLLMs, which typically requires additional fusion layers like cross- voos-attentionss (Vaswani et al., 2017). Due to different resolutions, these two pathways often produce visual features of different shapes, impeding their quick alignments (Yu al. 2019). To overcome this limitation, we adopt different downsampling rates for the low- and high-resolution pathways, respectively. Thus, their output features can keep the same spatial shape.
Based on the above motivations, (.) and Fz, (.) are designed as a vision transformer (ViT) (Dooo vitskiy et al. 2020) and convolutional network (CNN) (Liu et al. 2022), respectively. Specifically, CNN is equipped with a downsampling stride of 32 to process high-resolution images. VIT encodes low-resolution images with downsampling stride of 14. Notably, such designs also ensure the efficiency of MLLMs, where the high-resolution images are processed by the efficient CNN, and the number of visual tokens is also kept small via the large downsampling stride.
4.3 MIXTURE-OF-RESOLUUE
ADAPTER
To better collaborate the feature learning of two pathways, we propose a mixturre-of-resoluiie udapter (MR-Adapter) to embed high-resolution information of CNN into different stages of VIT. This early fusion strategy can leverage ViT's deep Transformer layers to excavate fine-graid mne-grained context from different visual sources.
In particular, given the visual features Fyh E wxx of the high-resolution image, we embed them into the low-resolution visual pathway by
High- Resolueeoo Features
ViT Stage
Fen
F
F = FiF 0:) + + (Fon:
(3)
Mapping
Mapping
Here, Fi, E RRxxwxd are features from the i-th stage of VIT. Fi(.) is lightweight convolution layer with residual connection. F.(.)) denotes an MLP layer. g is a dynamic score to control the weights of high-resolution information, defined by
Layer
I
Layer
Gate
G
(4)
ViT Stage
Here, fo E 2d is the global average pooling of visual features (F w. Fr (F eh)). where [-] denotes the concate- nation operation. W, E R2dx and W2 E & are two projection matrices. and 5 denote the activation function of GELU and Tanh, respectively.
Figure 3: Illustration of MR-Adapter. MR-Adapter can dynamically embed the res-uuesolution features into the low- resolution pathway.
As shown in Fig. high-resolution information can be fused with the features in each block of ViT. In this case, the low-resolution features of ViT also contain rich semantics, improving the visual descriptive power of MLLMs.
4.4 THE DEPLOYMENT ON MLLM
We apply MRA to LLaVA-1.5 Liu et al. 2023a) and construct a new model, namely LLaVA-HR. Its training consists of two stages, Le., www-leolution pre-training and high-resolution instruction tuning.
5

---

Published as a conference paper at ICLR 2025
Stage 1: Low-resolution pre-training. Similar to LLaVA (Liu et al. 2023b and LLaVA-1.5 LLu et al. 2023a), this stage aims to optimize the projector to align the visual features with the word embedding space of LLM. Therefore, the image encoder and the LLM are frozen during pre-training. Besides, we adopt low resolutions for two pathways, Lee. 384 x 384 and 336 x 336. In this stage, the MR-Adapter is not inserted, and output features of dual pathways are upsampled to the same size and directly combined.
Stage 2: High-resolution instruction tuning. During instruction tuning, we increase the resolution of the high-resolution pathway, egg, from 384x 384 to 1,024x 1,024. And the low-resolution one is also accordingly adjusted to ensure the visual alignment of two pathways, e.g. from 336x 336 to x 448. Mcanwhile, the MR-Adapter is then applied to connect two visual pathways. Different from the first training stage, the entire MILM will be fully optimized to better accommodate high-resolution images.
5 EXPERIMENTS
5.1 EVALUATIONS AND METRICS
Multimodal benchmarks for MLLM. We evaluate LLaVA-HR on six emerging multimodal bench- marks for MLLMs, including MME (Fu et al.] 2023), POPE (Li et al.] 2023c), SEED (Li et al.] 2023a), MM-VET YYu al. 2023b), MMMU Yue et 2023) and Math Vista (Lu al. (023)..
In particular, MME and MM-VET evaluate the multimodal perception and cognition abilities of MLLMs. SEED extends the modalities of evaluation to images and videos. POPE aims evaluate the visual hallucinations of MLLMs. MMMU and Math Vista aim to evaluate the multi muuti-discipline and math understanding ability, respectively. The metries used in our paper follow their default settings.
General visual question answering benchmarks. We also evaluate LLaVA-HR on seven VL datasets, including VQAv2 (Goyal et al. 2017). GQA (Hudson & Manning, 2019), OKVQA (Marino et al. 2019) OCRVQA (Mishra et al.) 2019), ScienceQA (Lu et al. (022a). VizWiz (Gurari et al. 2018) and TextVQA. In particular, ScienceQA (Lu et al., 2022a) and VizWiz (Gurari et al. (0.1)(01)) are two zero-shot tasks, and their samples are not appeared in our training data. We report the accuracy on the test set of OCRVQA, the test set of VizWiz, and the val set of OKVQA. We organize samples of these tasks in instruction formats of LLaVA-115 (Liu et al. 2023a).
OCR-related benchmarks. To validate the fine-grained recognition ability of LLR we further evaluate it on five text-rich image understanding tasks, including TextVQA (Singh al. 2019), Doc (Mathew et al., 2021). InfoVQA (Mathew et al. 2022), A12D (Kembhavi et al. 2016) and ChartVQA (Masry et al. 2002). For DocVQA and VQA, we use the metric of ANLS. For remaining benchmarks, we use the accuracy as the metric. Results of VA- on OCR-related benchmarks are evaluated by the VLMEvalKit Duan et al. (2024).
5.2 IMPLEMENTATION DETAILS
Liu et al. 2022 as the dual visual paths to encode low- and high-resolution images, respectively. The In HL, use CLIP-ViT-L (Radford et al., 2021 Ilaarco et al. 202I) and CLiP-ConvNeXt-
In LLaVA- the CLIP-ConvNe-X
is replaced with the stronger
MR-Adapter is applied into the last three stages of ViT. Following LLaVA-1.5, we first pre-train LLaVA-HR on LCS-558K (Liu et al. 2023b), which contains 558k image-text pairs. During the pre-training stage, both the visual encoder and the LLM are frozen, and only the MLP projector is fine-tuned, AdamW (Kingma & Ba 2014) used as the optimizer, and the learning rate and batch size are set to 1e-3 and 256, respectively. Visual resolutions are set to 336x 336 and 284x384 for the ViT and the CNN, respectively. During instruction tuning, we follow LLaVA- to use 665A VL instruction data. When fairly comparing with recent MILMS like MM1 (McKinzie et al. 2024).
we use additional 1.6M instruction data including ShareGPT4V (Chen et al. 2023b), LAION-GI GPT-
4V (laion| 2023), ALLAVA (Chen et al. 2024a). LIMA (Zhou et al. 2024). OpenAssistant2 (Köpi et al. 2024), Tabmwp (Lu et al. (022b)). MathQA Yuu et al.) 2023a), KVQA (Shah et al. 2019). Geometry (Lu et al. 2021), STVQA (Biten et al., 2019), ChartQA (Masry al. 2022), DVQA Kafle ct al.) 2018), A12D (Kembhavi et al. 2016). LLa ALaVA-Med (Li et al. (2244)), Inf (Mathew et al. 2022) and Math RathV3600 V360k Shi al. (2024). At this stage, the entire model is updated with a learning

---

Published as a conference paper at ICLR 2025
Table 1: Performance and efficiency comparisons of existing high-resolution adaptation solutions. All experiments are conducted based on LLaVA-1.5. The training and inference costs are measured on NVIDIA A800s. "Res." " and "V-Token" denote image resolutions and the number of visual tokens, respectively. "v/s" denotes the number of generated tokens per second. "N/A" means that GPU
memory overflows, so we reduce the batch size.
Vision-Languee Tasks
Training Time 4
GPU
Inference
Methods
Res.
V-Token
VQAv2
TVQA
MME
POPE
Memory
Speed f
LLaVA-1.5 (Luu al. 2023a
336 pix
576
80.4
59.4
1461
86.2
15.6h
28G
23.8 t/s
+Resize
448 pix
1024
81.1
62.1
1493
87.2
19.4h
49G
19.9 t/s
+Resize
672 pix
2304
81.5
64.2
1498
87.9
31.8h
79G
12.7 t/s
+Resize
1022 pix
5329
74.2
37.8
1266
84.4
69.4h
N/A
5.6 Us
+Av.. Pooling
756 pix
729
80.6
59.6
1480
86.5
37.3h
45G
23.9 t/s
+CNN Encoder [Luu al. 2022
768 pix
576
80.3
64.6
1415
86.6
17.6h
37G
23.7 us
+Resampler (Jaegle et al. 2021
756 pix
64
79.8
58.9
1403
85.8
36.5h
40G
27.6 t/s
+AnyRes (Liu 2024a
1088 pix
2888
81.7
65.1
1487
87.7
33.5h
65G
14.8 t/s
+MRA (ours)
768 pix
576
81.8
64.3
1524
88.0
18.2h
38G
23,5 t/s
+MRA (ours)
1024 pix
1024
81.9
67.1
1554
87.6
20.7h
40G
19.7 t/s
rate of 2e-5, Besides, we increase the resolution of ViT and CNN to 448 448 and 1,024 1,024, respectively. The training epoch to I for pre-training and instruction tuning.
5.3 EXPERIMENTAL RESULTS
5.3.1 QUANTITATIVE ANALYSIS
Comparison with high-resolution baselines. In Tab. [ we compare the performance and efficiency of MRA and existing high-resolution solutions on LLaVA-1.5 (Liu et al. 2023a). In this table, "Resize" aims to directly increase the image resolution. 'CNN Encoder" replaces the visual backbone with ConvNeXt (Liu et al., 2022). which uses larger downsampling rate to reduce the number of visual tokens. "Avg. Pooling" and "Resampler" refer the two pooling strategies for reducing the number of visual tokens. For "Resampler", we follow Qwen and reduce the number of visual tokens to 64. "AnyRes" divides a high-resolution image into several sub-images (Liu et al. 2024a). From this table, we observe that directly increasing image resolution obviously improves the performance of two models on four tasks, e.g., +4.8% of I.LaVA-1.5 on TextVQA. However, the performance of LLaVA-1.5 drops significantly at the resolution of 1,024x 1,024. To explain, the number visual tokens greatly exceeds the pre-trained context length of the LLM, which easily causes the instability during training, Besides, we can also see that although several baselines can well maintain the inference efficiency, their benefits to performance are not obvious.
In particular, "Resampler" even hurts the model performance on four bench- mark datasets, which often requires large-scale pre-training achieve promising performance. In contrast, as the most popular solution in exist- ing literature (Liu et al. 2024a Gao al. 2002)), "AnyRes" can effectively bring obvious performance gains on TextVQA and POPE. Nevertheless, the number of visual token increases significantly, leading to extremely high computational complexity. Compared to these methods, the performance of MRA is consistently improved from 768 x 768 resolution to 1,024 1,024 resolution. Besides, the total gain of MRA is more obvious than that of all compared methods, egg., +2.0% against AnyRes (Liu et al. 2024a) on TextVQA.
Table 2: Ablation Study of MRA on LLaVA-1.5. "Tune
vision" means that the image encoder is fine-tuned.
Methods
VQAv2
TVQA
MME
POPE
ILLVA-115 1.5 (Lau al. 2023a
78.5
58.2
1510.7
85.9
+Tune vision
80.4 .09 59.4 -1.2
1461.2 885
86.2
+03
DDuul-pttway
81.3 +18
62.8
1513.1
+2.4
87.2 +13
-dR-Adapter
81.8 423
64.4
162
1524.8
1141
88.0 v2.1
+1024 resolution
81.9
67.1
1554.9
87.6
+13 LLM
82.3 128
68.1
1999
1540.9
+30.2
87.8 +19
+IB Vision
82.6
70.9
1487.3
88.0
In addition to performance, the expenditure of LLaVA-HR is also cost-effective. In particular, increasing resolution from 336 x 336 1,022 x 1,022 slows down the training and inference of
7

---

Published as a conference paper at ICLR 2025
Table Comparison with existing methods on four MLLM benchmarks. "Param.", "Res." and "Data" refer to the parameters, the resolution and the training data, respectively. "V/s" refers to tokens per second. CogVLM-Cher and Intern VL-1.2 use more data and parameters, so we mark in gray.
Settings
General MLLM Benchmarks
Inference Speed
Method
Param,
Res.
Data
MMB
POPE
SEED SEED
MM Vet MMMU Vist
BLIP-2 Li
2023b
14B
224
129M
1293.8
85.3
46.4
49.7
22.4
130M
78.9
QwenVI-Chip [5at
10B
448
1.4B
1487.5
58.2
65.4
35,9
17.0
Fuyu-8B (Fuyu-XB 2025
8B
600
728.6
74.1
21.4
15.6 t/s
mPLUG-OW12 Ye Ye
2023
448
400M
1450.2
57.8
36.2
32.7
19.6
I-MoF [Tong
13B
336
1.2M
86.7
34.6
LLaVA-T.3 (T.u ai.
21233a
7B
336
1.2M
1510.7
85,9
58.6
66.1
30.5
23.8 1/s
LLaVA-1.5 [1.00
20250
13B
336
1.2M
1531.3
85.9
61.6
68.2
35.4
36.4
27.6
16.1
LLaVA-HR
1024
1.2M
1554,9
87.6
64.2
70.6
31.5
35.2
28.5
19.7
LLaVA-HR
13B
1024
1.2M
1540,9
87.8
64.5
70.9
35.5
35.7
27.7
15.0 1/s
LLaVA-HR-X
14B
1.2M 1024
1487.3
88.0
65.3
71.4
40.3
36.6
28.1
12,9 t/s
More Instruction Data:
AlaVA--NeX (Lau 2024g
1344
1.6M
1519.0
86.5
70.2
43.9
35.8
34.6
14.8 t/s
SPHINX-interd intern2 Gao of 2024
7B
448
16M
1260.4
86.9
36.5
35.5
InternL--XC Zhang atal 2023
1.BB
1528.4
35.2
29.5
Mini-Gemini eL el 2024b
2.7M
1546.0
41.3
36.8
32.2
16.2
MM1 McKiinzee et
2024
7B
1792
1B
1529.3
86.6
64.0
69.9
42.1
37,0
35.9
CogV M- (Wang cl al. 2023
17B
490
1.518
51.1
41.1
34.5
11.5 i/s
InternVL-1.2 1.2
Chen
40B
450M
1687.0
51.6
47.7
11.3
LLaVA-HRI
7B
1024 2.7M
1490.5
86,9
64,9
71.9
45.1
38.4
46.0
19.7 1/s
LLaVA-1.5 by 344.8% and 325%, respectively. However, these costs are reduced to only 17.6% and 20.8% in LL. Despite better performance, the training and inference speeds of LA- are three times faster than LLaVA-1.5. Besides, the costs of GPU memory also remain cheap for LLaVA-HR. For example, adapting the resolution of 1,024 x 1,024 for LLaVA-HR only consumes 40G GPU memory, but the same settings for LLaVA-1.5 will cause GPU memory overflow. These results greatly confirm the efficiency of our MRA and LLa
Ablation studies. In Tab. 2 and B] we
conduct comprehensive ablation studies for MRA on four benchmarks. Firstly, we val- idate each design of our MRA in Tab. 2 From these results, we find that each com- ponent obviously contributes performance. For example, the dual vi- sual pathways and the MR-Adapter pro- vide +3.4% and +1.6% performance gains on TextVQA, respectively. After increas- ing the resolution 1 1,024 x 1,024. the performance on TextVQA VQA further boosts by +2.7%. In the second block of Tab. we also ablate the parameter scale of the LLM and the visual encoder. Experimental results show that larger visual backbone or LLM will consistently improve the model performance, further confirming the scala- bility of MRA.
Table 3: Different choices MRA on LLaVA-HR. "L- Res Path.", "H-Res Path." and "Fusion Direct." denote the low-resolution pathway, the high-resolution pathway and the fusion direction, respectively. Our final setting is colored in gray.
the final
Settings L-Res
Choices
VQAv2
TVQA
MME
POPE
VIT-L
81.8
64.4
1524.8
88.0
Path.
ViT-G
81.7
65.3
1469.7
87.9
H-Res
ConvXt-L XXL
81.8
64.4
1524.8
88.0
Path.
82.3
66.5
1479.2
87.9
Fusion
High to Low
81.8
64.4
1524.8
88.0
Direct.
Low to High
81.0
62.8
1463.5
87.3
last 3 stages
81.8
64.4
1524.8
88.0
Insert Position
last stage
81.3
62.8
1513.1
87.2
In Tab 3 we compare different designs in MRA. From these results, we find that a larger high-resolution visual encoder typi- cally brings more gains than larger low- resolution one. Besides, the fusion direction of MRA A is also significant. Specifically, changing the fusion direction obviously degenerates the performance, e.g., -61.3 on MME. Such results also con- finm our design principle of MRA, i.e., embedding high-resolution information in low-resolution pathway. Mcanwhile, the best choice of the insert position of MRA is the last 3 stages of ViT. These ablations further confirm the designs of MR-Adapter.
last 2 stages
81.6
63.8
1508.4
87.5
last 4 stages
81.4
63.1
1461.6
87.5
8

---

Published as a conference paper at ICLR 2025
Table Comparison with existing methods on seven general visual question answering tasks. SQA' refers to the IMG subset of ScienceQA.
Settings
General Visual Question Answering
Infer.
Method
Param.
Res.
Data
VQAv2
GQA OKVQA OCRVQA SQA'
VizWiz
TVQA
Speed
BLIP-2 ct al. 2023b
14B
224
129M
41.0
41.0
45.9
40.6
61.0
19.6
42.5
InstructBLIP (Dai et
2023
130M
49.5
44.8
63.1
33.4
50.7
Shikra Chen al.
13B
224
6.1M
77,4
IDEFICS-9B (DEFFCC
2023
9B
224
354M
50.9
38.4
35.5
25.9
30.5 Us
IDEFICS-80E TIC
2023
80B
224
354M
60.0
45.2
36,0
30,9
QwenVI.-Chat [Ba et A123
10B
448
1.4B
78.2
$7.5
56.6
70.5
68.2
38.9
61.5
17.0 d/
Fuyu-8B (Fuyu-S85 14/2
8B
600
74.2
60.6
15.6
mPLUG-OWIZ YY 2023
400M
79.4
56.1
57.7
68.7
54.5
58.2
19.6
I-MoF (Tong at a.. [2124
1.2M
79.3
58.7
LLAVA-T.5 al 2023a
7B
336
1.2M
78.5
62.0
66.8
50.0
58.2
23.8 U/
LLaVA-1.5 Liu ct 20458
13B
336
1.2M
80.0
63.3
71.6
53.6
61.3
16.1 t/s
LLaVA-HR
7B
1024
1.2M
81,9
64.2
58.9
68.4
67.9
48.7
67.1
19.7
LLaVA-HR
13B
1024
1.2M
82.3
64.8
60.7
67.7
70.1
57.9
68.1
15.0 t/s
LLAVA-HR-X
14B
1024
1.2M
82.6
65.2
61.5
69.0
69.7
56.6
70.9
12.9 ds
Table 6: Comparison with existing MLLMs on five multimodal OCR-related benchmarks.
Method
Param.
Res.
Data.
TexVVQA
DocVQA
InfoVQA
A12D
ChartQA
QwenVL Bai ct al. 2023
10B
336
1.4B
63.8
65.1
35.4
65.7
Monkey er 20240
10B
1344
1.4M
67.6
66.5
36.1
62.6
LaaV--NeXL Lau 2024n
7B
1344
1.6M
64.9
66.6
54.8
TextMonkey el 20244
1344
2.5M
73.0
28.6
65.5
DocOwl-1.5 (Hu 2024
4032
4M
68.6
82.2
50.7
70.2
CogAgent Hong et 2025
Hong
18B
1120
3300
76.1
81.6
44.5
68.4
LLaVA-HRi
7B
1024
2.7M
73.8
85.8
52.3
75.3
77.6
Comparison with existing MLLMs. In Tab. and we compare LLa VA-HR with existing MLLMs on 13 tasks. On the six MLLM benchmarks, we observe comprehensive advantages of LLa VA-HR against existing MLLMs. In particular, LLaVA-HR achieves 1554.9 scores in MME benchmark, outperforming LLaVA-1.5 by +23.6. On POPE, a benchmark including video evaluations, LLaVA- HR-X still outperforms existing MILMs by large margin, i.e,, +3.7% gains. Besides, LLaVA-HR achieves the best performance on the benchmark for visual hallucinations, iee,, POPE, suggesting that its visual hallucinations are greatly alleviated. Meanwhile, we also compare the recently proposed particular, we still observe the better performance LLaVA- HR against LLaVA-NeXT (Liu al. 2024a), SPHINX-inteni (Gao et al. 2024), Mini-Gemini (Li
MLLMs in the second block of Tab.
et al. 2024b) and MM1 (icKin et al. 2024). e.g., +3.0% on MM Vet.
Tab. gives the performance comparison on common VL tasks. On in-domain tasks, LLaVA-HR achieves the best results on three tasks, e.g., 82.6 on VQAv2 and 61.5 on OKVQA. On OCRVQA, Owen-VL-Chat collects more in-domain data for training. it performs better than LLaVA-HR. Under the zero-shot setting, we can observe more significant advantages of LLaVA-HR on the fine-grained tasks, e.g., VizWiz. Most notably, even Qwen-VL-Chat is pre-trained with 24.8M OCR samples, it still performs worse than LLaVA-HR-X on TexIVQA. These results suggest the significance of high resolution for these tasks. In contrast, most images of ScienceQA are synthetic and of low resolution, so the advantages of LLaVA HR are not obvious. Overall, these results greatly confirm the effectiveness and generalization of LLaVA-HE and our MRA.
Tab. O compares LLaVA-HE and existing MLLMs on text-rich image understanding tasks. Compared to common MILMM benchmarks and VQA benchmarks, these reR-related benchmarks pose a higher requirement for image resolution. As shown in Tab. 6 low-resolution MLLMs like Qwen VL. often perform inferior to high-resolution ones, e.g., -4.8% on TexiVQA compared to DocOwl-1. 1.50w-115--C 1.5-Chat Hu et al. 22024). However, we still observe that VA-HR greatly outperforms existing MLLMs on five benchmarks. For example, although DocOwl-1.5-CH and data size, LLaVA-B also demonstrates superior fine-grained text recognition ability, e.g., +3.6 on DocVQA and +1.6 on IQA. These results further validate the effectiveness of our mixture-of-resolutton design text-rich image understanding tasks.
has larger model size, input resolution

---

Published as a conference paper at ICLR 2025
User: How many boats are fully
User: what are the three words on page?
de wedee
LLaVA-HIR (384 2
LLaVA-HR (384 pio:: and other sturies.
LLaVA-HR (1021 pix:
LLaVA-HR (1022 poo:: Wo ich welne.
User: color is The tiniest"
User: what page this book on?
in the forefront?
LLaVA- HRR (384 pio): This is the first
LLaVA-HR (384 pio: Red
page of a book.
LLaVA-HR pR)) W hitee
L.L.V. (1024 pio); This book is on
page 67.
Please provide the bounding box coor
User:
User: Please provide the hounding courdinate
the
region
region sentence describes the far right.
weteecce deseribes: sliding down
clothes.
LLaVA-HR (384 pix):
L.L.VA-HR (384 pix):
0.97, 0.72)
[0.4, 0.6, 0.51, 0.78)
LLaVA-HHR (10234 pix): (0.66, 0.68, 0.74, 0.73]
LLaVA-HR (1024 pio):
[0.53, 0.57, 0.71]
Ground-truth: [0.66, 0.74,
Ground-truthe [0.32, 0.56, 0.6, 0.7)
(a) Comparison of LLaVA- HR with different resolutions
the text in image return information the fallowing JSON format (note axx placeholder, if information not available "DOB": Xxx, Namee XXX, "Address" XXX, "EXP": XXX, "ISS": XXX. "SEX" XXX, HGT XXX, WGT"; XXX, "EYES": XXX, "HAIR", XXX,
User: Please
NEW YORK STATE
image, put
instrad). class: XXX.
"DONOR": XXX)
INHAACEE
DRIVER LICENSE
GPT dlass: ; "D",, "DLN"; 97542054 "DOB"; "15-03-1984' "Name": "G VEERE*..>* " "Address": 52 KLINK RD ROCHESTER 4
'EXP'; "10-01-66", "ISS": * 09-30-08" "SEX"; "M", "HGT": "5-09", "WGT": "NAA", "EYES" BR", - "HAIR": NNA", "DONOR": "YES")
97542964
CLASS D
KLINK
LLVA--155: d' Class D. 'DIN N"; 97745434, "DOB": 03/15/108, "Name": Gopal Veerendlaar, 'Address": 54 Klinck Rd, Rochester, NY 1460,, "EXP"; 1091/2016
DB:
15-03-1984
DIE BR 10500
10/0/2017, "IS"" New York State Department of Motur Vehicles "SEX": Male, "HGT": 5' 10", "GGT": 170, BY ES": Brown, "HAIR": Black, "DONOR" N/A]
Songh Zone Brwndd
LLAVA-HR-X: f d'aass: Class D. "DEN": 97542954***4 "DOB": 15/03/1964, Namee' GP Veare "A Adrress: 52 Klink Rd, Rochester *- exe.,, "EXP": 10/01/2016, "ISS": New York State, "SEX"; "EVES": Brown, "HATR": Black, DDONOR" N(A) -1.5 and GPT4-V in visual information extraction
"HGT": NiA, "WGT": NAA,
(b) Comparison of LLLVA-HR-XX
Figure 4: Visualizations of LLa and existing MLLMs. Subfig-(a) shows that high image resolution greatly improves the capability of MLLMs on fine-graines VL tasks. In Subfig-(b), LLaVA- HR-X demonstrates the comparable ability with GPT4-V in visual information extraction. Correct and incorrect answers are colored in green and red, respectively.
5.3.2 QUALITATIVE EXPERIMENTS
In Fig (a), we compare the predictions of L.LVAAHR with different resolutions. The visualizations show that higher image resolution obviously improves the capability of MLLMs on fine-grained tasks. For example, LL- with resolution of 1,024 x 1,024 can well capture granular visual content, e.g., the tiny boat in the first example. Besides, high image resolution also enables LLa VA-HR stronger ability of text recognition. For instance, the small and blurred phrase of "wo ich wohne" in the second example are correctly identified by the high-resolution LLa VA-H These results greatly confirm the significance of high image resolution in addressing visual shortcoming. In Fig (b), we further compare the predictions of LLaVA-HR-X. -LL (Liu et al.) (2023a) and GPT4. V (OpenAll 2023) in visual information extraction. Notably, LLaVA-HR-X shows comparable ability with GPT4-V on this challenging task. As shown in Fig 4 (b), LLaVA-HR- - and GPT4-V can correctly extract almost all visual content of the driver license and organize it in JSON format. Compared to GPT4-V, LLaVA- -A----X--- also correctly identifies the hair color of the person, which requires fine gra-nna visual reasoning. In contrast, VL. can only recognize simple visual content like "class" and "SEX", and fail to extract most visual information. These results further validate the effectiveness of MRA in addressing visual shortcoming of MI.LMs.
CONCLUSION
In this paper, we focus on the efficient high-resolution adaptation for MLLMS and propose a novel method, namely mixture-of-resolutee process images of both high and low resolutions, where high-resolution information is embeded into the low-resolution modeling via the novel mixture-of of resolution adapters MMR-Adapters). We apply MRA to a popular MLLM called LLaVA-11. and construct new high-resolution MLLM, termed LLa VA-HR. Experimental results not only validate the effectiveness of LLa in addressing visual shortcoming, but also confirm its remarkable efficiency against existing ML..Ms.
adaptation (MRA). MRA adopts dual visual pathways to
10

---

Published as a conference paper at ICLR 2025
Acknowledgments. This work supported by the National Science Fund for Distinguished Young Scholars (No.62056033 the China Postdoctoral Science Foundation (No. 2024M7615480 the National Natural Science Foundation of China (No. U21B2037, No. U22B2051, No. 623B2088, No. U23A20383, No. U21 A20472, No. 62176222, No. 62176223, No. 62176226, No. 62072386, No. 62072387, No. 62072389, No. 62002305 and No. 62272401), the Natural Science Foundation of Fujian Province of China (No. 2021J06003 No.2022106001 and the Fundamental Research Funds for the Central Universities (Xiamen University: No. 20720240053).
REFERENCES
Sean-Baptisse Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204. 14198, 2022.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. 110222088.1966.. 12966, 2023.
Ali Furkan Biten, Ruben Tito, Andres Maila, Lluis Gomez, Marçal Rusinol, Ernest Valveny, CV Jawa- har, and Dimosthenis Karatzas. Scene text visual question answering. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4291-4301, 2019.
Chun-Fu Richard Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attention multi-scale vi- sion transformer for image classification. In Proceedings of IEEE/CVF international conference on computer vision, pp. 357-366, 2021.
Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Hamnsssing gptdv-synthesized data for a lite vision-language model. arXiv preprint arXiv: 1684, 2024a.
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal Ilm's 's referential dialogue magic. arXiv preprint arXiv:2306.15195, 2023a.
Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793, 20233...
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong semi-supervised learners. Advances in neural information processing systems (NeurIPS), 33:22243-223 22255, 2020.
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209.06794 2022.
Xi Chen, Xiao Wang, Lucas Beyer, Alexander Kolesnikov, Jialin Wu, Paul Voigtlaender, Basil Mustafa, Sebastian Goodman, Ibrahim Alahdulmohsin, Piotr Padlewski, al. Pali-3 vision language models: Smaller, faster, stronger. arXiv preprint arXiv:2310.09199 2023c.
Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing, Zhong Muyan, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv preprint arXiv: 14238, 2023d.
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404. 16821. 2024b.
Wenliang Dai, Jumnan Li. Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general- purpose languaee models with instruction tuning. arXiv preprint arXiv:2305.066 06500, 2023.
11

---

Published as a conference paper at ICLR 2025
Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Songyang Zhang, Haodong Duan, Wenwei Zhang, Yining Li, et al. dxemll-xcomposer/2 large vision-language model handling resolutions from 336 pixels 4k hd. arXiv preprint arXiv:2404.065 12, 2024.
A pioneering
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, Dahua Lin, and Kai Chen. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models, 2024. URL http:: ptttpp://arxiv.coo /
11691
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint wrxXv:2306.13394, 2023.
Fuyu-8B,
www.:acppp...iiaaaaooo aw/blog////ooooo... fuyu 8b 2023.
Peng Gao, Renrui Zhang, Chris Liu, Longtian Qiu, Siyuan Huang, Weifeng Lin, Shitian Zhao, Shijie Geng. Ziyi Lin, Peng Jin, et al. Sphinx-x: Scaling data and parameters for a family of multi-modal large language models. arXiv preprint arXiv:2402.059333 2024.
Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. Chatgpt outperforms crowd-workers for text-annotation tasks. arXiv preprint arXiv:2303.. 1506, 2023.
Yash Goyal, Tejas Khot, Douglas Summers-Star Dhruv Batra, and Devi Parikh. Making the in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6904 0913,9913222 2017.
Danna Gurari, Qing Li, Abigale Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3108-36177 2018.
Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, and Jie Tang. Cogagent: visual language model for gui agents, 2023.
Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan. Liang Zhang, Bo Zhang, Chen Li, Ji Zhang, Qin Jin, Fei Huang, et mplug-docowl 1.5: Unified structure learning for ocr-free document understanding. arXiv preprint arXh::2403.12899 2024.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. CVPR, 2019.
IDEFICS. Introducing idefics: An open reproduction of state-of-the-art visual language model. 2023.
https://huggingface http:///iaa.aaee.oooooo////eeeee
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip. July 2021. doi: htt.///oooo..1777700 URL If you use this software, please cite
httpps://doi. http:///....1//oo..ooo 5143773 as below.
Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. International conference on machine learning, pp. 4651 16644. PMLR, 2021.
Huaizu Jiang, Ishan Misra, Mareus Rohrbach, Erik Miilee- and Xinlei Chen. In defense of grid features for visual question answering. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 10267-10276, 2020.
12

---

Published as a conference paper at ICLR 2025
Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. Dvqa: Understanding data visual- izations via question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5648-56566 2018.
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth dozen images. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pp- 235-251. Springer, 2016.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv: 1412.69800 2014.
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte. Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Due Nguyen, Oliver Stanley, Richard Nagyfi, et al. Openassistant coaversations-democraiiinn large language model alignment. Advances in Neural Information Processing Systems, 36, 2024.
laion. laion 2023.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Bench- marking multimodal Iims with generative comprehension. arXiv preprint arX1v:230..16125, 2023a.
Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language-and-vision assistant for biomedicine in day. Advances in Neural Information Processing Systems, 36, 2024a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre- training with frozen image encoders and large language models. arXiv preprint arXiv:2301. 125977 2023b.
Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403. 18814, 2024b.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305. 10355, 2023c.
Zhang Li. Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi-modal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26763-26773, 2024c.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprini arXiv:2310.03744 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurlPS, 2023b.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. nexta-next: Improved reasoning, ocr, and world knowledge, January 2024a. URL https
7/1 wlllaavvl.gittae hot.h...bbbbb....... ho/bolll/oo.o ht///blop
ava-next/
Yuliang Liu, Biao Yang. Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, and Xiang Bai. Textmonkey: An ocr-free large multimodal model for understanding document. arXiv preprint arXiv:2403.0444 044733 2024b.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11976-1 1986, 2022.
13

---

Published as a conference paper at ICLR 2025
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task agnostic visiolinguistic representations for saiio-and-languuae tasks. arXiv preprint arXiv:I90e 10X.::1908.02265, 2019.
Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu. Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. arXiv preprint arXiv:2105.0411: 0466,, 2021.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 2022a.
Pan Lu, Liang Qiu, Kai-We Chang, Ying Nian Wu, Song-Chun
Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. arXiv preprint arXiv:2209. 14610, 2022b.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02253 2023.
Gen Luo, Yiyi Zhou, Tianhe Ren, Shengxin Chen, Xiaoshuai Sun, and Rongrong Ji. Cheap and quick: Efficient vision-language instruction tuning for large language models. Advances in neural information processing systems (NeurIPS), 2023.
Gen Luo, Xue Yang, Wenhan Dou, Zhaokai Wang. Jifeng Dai, Yu Qiao, and Xizhou Zhu. Mono- internvl: Pushing the boundaries of monolithic multimodal large language models with endogenous visual pre-training. arXiv preprint 100.022410.08202. 2024a.
Yaxin Luo, Gen Luo, Jiayi Ji, Yiyi Zhou, Xiaoshuai Sun, Zhiqiang Shen, and Rongrong Ji. 7- mod: Exploring mixture-of-depe adaptation for multimodal large language models. arXiv preprint arXiv:2410.13859, 2024b.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: visual question answering benchmark requiring external knowledge. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A bench- mark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203. 10244, 2022.
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference applications of computer vision, pp. 2200-2209, 2021.
Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographievga. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1697-1706, 2022.
Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, et al. Mml: Methods, analysis & insights from multimodal Ilm pre- pre-training, arXiv preprint arXiv:2403.00611 2024.
William H Merigan and John HR Maunsell. How parallel are the primate visual pathways? Annual review of neuroscience. 16(1):369 402, 1993.
Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In 2079 international conference on document analysis and recognition (ICDAR), pp. 947-952. IEEE, 2019.
OpenAl. Gpt-4v(ision) system card. /nttps://col http://cc.ppraaaao..oooo/ae////a StyssyyteeePSSSSttemmmmee System fard.pdi 2023.
Zhiliang Peng, Wei Huang, Shanzhi Gu, Lingxi Xie, Yaowei Wang, Jianbin Jiao, and Qixiang Ye. Conformer: Local features coupling global representations for visual recognition. In Proceedings of the IEFF/CVF international conference computer vision, pp. 367-376, 2021.
14

---

Published as a conference paper at ICLR 2025
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306.14824, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2 10.2200030000000 00020, 2021.
Lynn C Robertson and Marvin Lamb. Neuropsychological contributions to theories of part/whole organization. Cognitive psychology, 23(2):299-330, 1991.
Daniel Rose, Vaishnavi Himakunthala, Andy Ouyang, Ryan He, Alex Mei, Yujie Lu, Michael Saxon, Chinmay Sonar, Diba Mirza, and William Yang Wang. Visual chain of thought: Bridging logical gaps with multimodal infillings. arXiv preprint Sidv::2305.02317. 2023.
Sanket Shah, Anand Mishra, Naganand Yadati, and Partha Pratim Talukdar. Kvqa: Knowledge- aware visual question answering. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 8876-88844 8884,888442 2019.
Wenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu, Yang Yang, See-Kiong Ng. Lidong Bing, and Roy Ka-Wei Lee. Math-llava: Bootstrapping mathematical reasoning for multimodal large language models. arXiv preprint arXiv:2406.1 17294, 2024.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pp. 8317-8326. 2019.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie, Eyes wide shut? exploring the visual shortcomings of multimodal 1lms. arXiv preprint 4rX1v:2401.06209. 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 139771 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
Weihan Wang. Qingsong Lv, Wenmeng Yu, Wenyi Hong, Qi, Yan Wang, Junhui Ji, Zhuoyi Yang. Lei Zhao, Xixuan Song, et al. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.000 1.00079, 2023.
Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-ow12: Revolutionizing multi-modal large language model with modality collaboration. arXiv preprint arXiv:23/ 1,04257, 2023.
Jun Yu, Jing Li, Zhou Yu, and Qingming Huang. Multimodal transformer with multi-view visual rep- resentation for image captioning. IEEE transactions on circuits and systems for video technology, 30(12):446744480 2019.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint (tt.:2309.122844 2023a.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang. and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308 02490, 2023b.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi musti-dissciplne multimodal understanding and reasoning benchmark for expert agi. arXiv preprint 1.Xiv:2311.165/// 2023.
15

---

Published as a conference paper at ICLR 2025
Pan Zhang, Xiaoyi Dong Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Shuan- grui Ding, Songyang Zhang, Haodong Duan, Hang Yan, et al. Internlm language large model for advanced text-image comprehension and composition. arXiv preprint arXiv:2309. 151 12, 2023.
Internlm-xcopposee xcomposer: vision-
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations vision- vision-language models. In CVPR, 2021.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10521 2023.
16


Reviews:
[{"review_id": "FxDVard2bM", "reviewer": "Reviewer_TWQ4", "summary": "This paper aims to enhance MLLM by enlarging resolution of input images. By combining features from ViT and a CNN encoder through an adapter, performances of MLLM are improved a lot. Meanwhile, fusing high-resolution features from convolution-based encoder into low-resolution features from transformer-based encoder does not increase vision tokens to LLM decoder, so that additional computational cost is low. Proposed LLaVA-HR increases effective resolution for MLLM to 1024 and outperforms concurrent MLLMs.", "questions": "1. In section4.3(line 258), the statement, global average pooling is confusion, is the features are pooled into 1 global token? If so, it seems to be not consistent with figures. Please clarify the exact dimensions of fv after global average pooling.\n2. In Table 1, resizing LLaVA-1.5 to 672 pix achieves close performance with 768pix version of LLaVA-HR, is there a direct comparision between 768-pix version of them?\n3. In table 2, there is an ablation of \"tune vision\" referring to finetune vision encoder. However, I think the vision encoder in LLaVA-1.5 is fixed, can you provide a detailed description about this. For example, implementation and aim of tuning vision encoder.\n4. LLaVA-HR is proposed to process input resolution of 1024, what if input images larger than 1024. Is there any extended experiments for even larger images such as 4K ones.\n5. What do you mean by \"stages\" in vision transformers? And, currently only final features from ConvNext is utilized, is there any experiments of multi-stage feature integration for that of CNN encoder?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This work proposed a novel method to increase resolutions of MLLMs, which is an important problem in the field and critical in fine-grained  vision tasks. Without large modification of training recipe and computational cost of its baseline, LLaVA-1.5. \nEvalutions are conducted on many existing benchmarks and performance of LLaVA-HR is quite impressive. Besides, the computational cost involved is quite small compared with related works.", "weaknesses": "Please see as in questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M53SECk0So", "reviewer": "Reviewer_b6Tu", "summary": "In this paper, the authors propose the Mixture-of-Resolution Adaptation method to embed the high-resolution features into the low-resolution pathway. The MRA enhances the visual perception ability in MLLMs, and allow them to benefit from high-resolution visual inputs with reduced computational cost. Extensive experiments demonstrate the effectiveness of the MRA.", "questions": "1. There are several micro-designs in the Mixture-of-Resolution Adapter, including $\\mathcal{F}_l$, $\\mathcal{F}_h$, and the gate function. Why do we choose a conv layer for $\\mathcal{F}_l$, an MLP layer for $\\mathcal{F}_h$? Are these layers and functions necessary? Please provide some analyses.\n\n2. In the Mixture-of-Resolution Adapter, the authors choose the addition operation to fuse features of different resolutions. (Deformable) Cross Attention is also an option. I wonder which method is better?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to follow.\n2. The comparison of MRA and other high-resolution adaptation solutions is clear, highlighting the effectiveness of the dual visual pathways.\n3. The experiments are well-conducted and quite comprehensive.\n4. The study demonstrates strong performance on most datasets compared with other MLLMs.", "weaknesses": "1. In Table 1, the MRA is compared to other high-resolution adaptation methods that use a single visual pathway. However, the introduction of a new visual encoder in the MRA raises concerns about the fairness of this comparison. Could the authors provide a baseline that uses dual visual pathways without the MR-Adapter?\n2. The analyses of the MRA’s architecture and design details are insufficient, particularly regarding $\\mathcal{F}_l$, $\\mathcal{F}_h$, and the gate function. Could the authors provide ablation studies on these components?\n3. The main novelty of the paper appears to be the Mixture-of-Resolution Adapter. While the application of dual visual pathways for high-resolution adaptation in MLLMs is innovative, the overall contribution of the paper seems somewhat insufficient. If MR-Adapter could integrate a wider variety of low- and high- resolution visual encoders, its contribution would be significantly enhanced.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XEvJhqxUrr", "reviewer": "Reviewer_Mfi4", "summary": "This paper presents a new approach for efficient multimodal large language models (MLLMs) by addressing the high computational cost of processing high-resolution images. The authors introduce Mixture-of-Resolution Adaptation (MRA), a method that combines both low- and high-resolution visual features to enhance model efficiency without compromising visual recognition quality. MRA uses two visual pathways: one for low-resolution and one for high-resolution images, with novel mixture-of-resolution adapters (MR-Adapters) that embed high-resolution information into the low-resolution pathway. This design significantly reduces input sequence length and computational load.\n\nThe authors apply MRA to the LLaVA model, resulting in an improved version called LLaVA-HR, which demonstrates superior performance across 15 out of 17 vision-language (VL) tasks, including a 5.2% increase in accuracy on TextVQA. Furthermore, LLaVA-HR maintains efficient training and inference times, showing improvements over LLaVA-NeXT.", "questions": "> ### 1. Clarification on Visual Encoder Notation\n\nIn line 206, it states that $F_{I_l}$ and $F_{I_h}$ are visual encoders for high- and low-resolution images, which seems to be a typo. The correct notation should reflect that $F_{I_l}$ and $F_{I_h}$ correspond specifically to low- and high-resolution encoders, respectively.\n\n> ### 2. MR-Adapter Placement in ViT Architecture\n\nFigure 2 shows the MR-Adapter is applied starting from the second stage of the ViT architecture. Does this mean the initial stage of the ViT does not utilize high-resolution features? Clarifying this could help illustrate the feature extraction flow more clearly.\n\n> ### 3. Implementation of LLaVA-1.5-448\n\nFor LLaVA-1.5-448, only the image resolution is modified at the fine-tuning stage. Have you considered modifying the visual backbone from ViT-336 to ViT-448 and retraining it for both pre-training and fine-tuning? This comparison could provide insight into performance differences when using higher resolution throughout the model’s entire training process.\n\n> ### 4. $SEED^{img}$ Performance Comparison\n\nCould you provide the $SEED^{img}$ performance for LLaVA-1.5, LLaVA-1.5-448, and LLaVA-NeXT? This metric would help evaluate relative image-processing capabilities across these models.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to follow.\n\n2. Figures 2 and 3 are effectively designed and enhance understanding of the framework.\n\n3. The ablation study is solid to reveal the contribution of component.", "weaknesses": "> ### 1. LImited performance imprvement.\n\nThe performance gains with MRA are modest. The low-resolution branch operates at 448×448, so the appropriate baseline is LLaVA-1.5 with 448-pixel resizing. Compared to this baseline, the improvements MRA achieves are minimal (e.g., +0.7 on VQA v2, +31 on MME, and +0.8 on POPE). Training cost and inference speed are also similar between MRA and LLaVA-1.5-448, reducing the practical benefit.\n\n> ### 2. Limited novelty\n\nThe dual-pathway, high-and-low-resolution approach isn’t particularly new. Similar strategies have been explored in other works, such as Mini-Gemini and CogAgent, yet the authors do not compare their method with these models. Explicitly differentiating MRA from these approaches would help clarify its unique contributions.\n\n> ### 3. Limited generalizability\n\nThe authors apply MRA solely to LLaVA-1.5. Expanding the evaluation to other MLLMs, like Qwen-VL, would strengthen claims of the method’s generalizability across architectures.\n\n\n[1] CogAgent: A Visual Language Model for GUI Agents\n[2] Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gcYHRauKr9", "reviewer": "Reviewer_HwDF", "summary": "This paper focuses on the efficient high-resolution adaptation for multimodal large language models (MLLMs) and proposes a mixture-of-resolution adaptation (MRA) method for MLLMs. To be specific, the proposed MRA employs two visual pathways for images of different resolutions, where high-resolution visual information is embedded into the low-resolution pathway via the mixture-of-resolution adapters. Besides, the paper conducts extensive experiments to verify the effectiveness of the proposed model.", "questions": "As mentioned, in Table 1, it seems that there is no significant gap between ‘Avg. Pooling’ and the proposed MRA for the VQAv2 task, which is perplexing. The paper should explain the experimental phenomenon.\n2. The paper should carry out a qualitative experiment between the proposed MRA and the model variant in Table 2.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper aims to explore the high-resolution adaptation for MLLMs, which is crucial and engaging.\n2. The paper is well written and easy to follow.\n3. The paper is well motivated and the proposed MRA appears reasonable.", "weaknesses": "1. As demonstrated in Table 1, it seems that there is no significant gap between ‘Avg. Pooling’ and the proposed MRA for the VQAv2 task, which is perplexing. The paper should explain the experimental phenomenon.\n2. The paper should carry out a qualitative experiment between the proposed MRA and the model variant in Table 2.\n3. The paper fails to clarify the version of LLaVA-1.5 used in Figure 4.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BGcdyhqGmc", "reviewer": "Reviewer_Yq9F", "summary": "This paper introduces an novel high-resolution adaptation method for multimodal large language models (MLLMs), termed Mixture-of-Resolution Adaptation (MRA). MRA employs a dual visual pathway design to process high- and low-resolution images simultaneously from both macro and micro perspectives, while integrating high-resolution information into the low-resolution pathway through the Mixture-of-Resolution Adapter (MR-Adapter). This approach reduces the number of visual tokens while preserving rich visual semantics, significantly enhancing the model's visual descriptive power.", "questions": "See Weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Unlike previous strategies that divide high-resolution images into sub-images, this paper introduces an innovative dual visual pathway structure, offering a fresh perspective for high-resolution adaptation. The MR-Adapter effectively embeds high-resolution information into the low-resolution pathway, introducing a new adaptation mechanism within the visual processing framework of MLLMs. This design overcomes the efficiency limitations of traditional high-resolution processing.\n\n- The paper conducts extensive experiments across multiple vision-language tasks, providing a range of comparisons, with promising results.\n\n- The writing is clear and easy to follow. It effectively highlights MRA's performance gains and efficiency advantages across different tasks, helping readers fully understand the model’s effectiveness and strengths.", "weaknesses": "1. The processing of both low-resolution and high-resolution images in the paper is mainly square-based, such as 448x448 and 1024x1024. Is there any adaptation mechanism for handling images with different aspect ratios? Would processing high-resolution images in a way that matches the input image's aspect ratio lead to better performance?\n\n2. For high-resolution image inputs, we are more focused on improvements in OCR-related tasks. The results for OCRVQA in Table 5 don’t seem to be the best. Additionally, Table 6 only presents results for LLaVA-HR+, but it lacks results for LLaVA-HR-7B, LLaVA-HR-13B, and LLaVA-HR-X with less training data. It would be helpful to include these results to better illustrate the impact of MRA on OCR-related tasks.\n\n3. Could the authors further explain why the MR-Adapter is inserted in the last 3 stages? What is the design principle behind this decision? Could it be inserted in the earlier stages instead?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)