InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD
Authors: Xiaoyi Dong
Pan Zhang
Yuhang Zang
Yuhang Cao
Bin Wang
Linke Ouyang
Songyang Zhang
Haodong Duan
Wenwei Zhang
Yining Li
Hang Yan
Yang Gao
Zhe Chen
xinyue zhang
Wei Li
Li Jingwen
Wenhai Wang
Kai Chen
Conghui He
Xingcheng ZHANG
Jifeng Dai
Yu Qiao
Dahua Lin
Jiaqi Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Vision Language Model (LVLM)
Primary area: machine_vision
TL;DR: This paper represents InternLM-XComposer2-4KHD, a groundbreaking exploration into elevating LVLM resolution capabilities up to 4K HD (3840 × 1600) and beyond.

Abstract: The Large Vision-Language Model (LVLM) field has seen significant advancements, yet its progression has been hindered by challenges in comprehending fine-grained visual content due to limited resolution. Recent efforts have aimed to enhance the high-resolution understanding capabilities of LVLMs, yet they remain capped at approximately 1500 $\times$ 1500 pixels and constrained to a relatively narrow resolution range. This paper represents InternLM-XComposer2-4KHD, a groundbreaking exploration into elevating LVLM resolution capabilities up to 4K HD (3840 × 1600) and beyond. Concurrently, considering the ultra-high resolution may not be necessary in all scenarios, it supports a wide range of diverse resolutions from 336 pixels to 4K standard, significantly broadening its scope of applicability. Specifically, this research advances the patch division paradigm by introducing a novel extension: dynamic resolution with automatic patch configuration. It maintains the training image aspect ratios while automatically varying patch counts and configuring layouts based on a pre-trained Vision Transformer (ViT) (336 $\times$ 336), leading to dynamic training resolution from 336 pixels to 4K standard. Our research demonstrates that scaling training resolution up to 4K HD leads to consistent performance enhancements without hitting the ceiling of potential improvements. InternLM-XComposer2-4KHD shows superb capability that matches or even surpasses GPT-4V and Gemini Pro in 10 of the 16 benchmarks.

IntemL/N-XCompssez-kkDD:: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD
Xiaoyi Dong Pan Zhang"?, Yuhang Zang"), Yuhang Cao Bin Wang', Linke Ouyang',
Songyang Zhang", Haodong Duan', Wenwei Zhang", Yining Lit, Hang Yan', Yang Gaot, Zhe Chen Xinyue Zhang', Wei Li", Jingwen Li Wenhai Wang Kai Chen', Conghui He Xingcheng Zhang?
Jifeng Dai Yu Qiao', Dahua Lin Jiaqi Wang!. 1.E
Shanghai Artificial Intelligence Laboratory, The Chinese University of Hong Kong, SenseTime Group, "Tsinghua University, CPII under InnoHK internlmppjaab org.cn
Abstract
The Large Vision-Language Model (LVLM) field has seen significant advance- ments, yet its progression has been hindered by challenges in comprehending fine-grained visual content due to limited resolution. Recent efforts have aimed enhance the high-resolution understanding capabilities of LVLMs, yet they remain capped at approximately 1500 x 1500 pixels and constrained to a relatively narrow resolution range. This paper represents IntemLM--XCoppose22 XComposer2-4KHHD a groundbreaking exploration into elevating LVLM resolution capabilities up to 4K HD (3840 1600) and beyond. Concurrently, considering the ultra-high reso- lution may not be necessary in all scenarios, it supports a wide range of diverse resolutions from 336 pixels to 4K standard, significantly broadening its scope of applicability. Specifically, this research advances the patch division paradigm by introducing novel extension: dynamic resolution with automatic patch configu- ration. It maintains the training image aspect ratios while automatically varying patch counts and configuring layouts based on pre-trained Vision Transformer (ViT) (336 x 336), leading to dynamic training resolution from 336 pixels to 4K standard. Our research demonstrates that scaling training resolution up 4K HD leads to consistent performance enhancements without hitting the ceiling of poten- tial improvements. InternI.M- XCCompser2-44KH shows superb capability that matches or even surpasses GPT-4V and Gemini Pro in 10 of the 16 benchmarks. The InternI. Mow...-MXoooooore22-HH00 model series with 7B parameters are publicly available https: www.o///wwnneeeeeeeeoeoooeeeeeee Iwt.///InteenLLM Xattremm/l//ntterrr/.Kooooeeer
Introduction
In recent years, the progress in Large Language Models (LLMs) (73 92 93 39
29 2
has provoked the development of Large Vision-Language Models (LVLMss.. These models have demonstrated proficiency in tasks such image captioning (17 14) and couuullqustion-answeeeee
(VQA) (57) 31 (07). Nevertheless, due to their limited resolution, they struggle with processing images containing fine details, such charts (68). tables (87). documents (70), and infographics (69). This limitation constrains their practical applicability in real-world scenarios.
Recent advancements have aimed at enhancing the resolution of Large Vision-Language Models (LVLMs). Some approaches (66) 36, 97: 48) involve adapting high-resolution vision encoders
indicates equal contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

DocVQA
ChartQA
OCRBench
90.0
MathVista
81.0
67.5
TextV QA
57.8
77.2
Benchmark
Min
Median
Max
InfoVQA
355
1444
8541²
A12D
InfoVQA
80.9
Doc
246
1948²
8243²
OCRBench
20
408
5899²
HallBench
147a
724
4898²
Qbenchy
312a
8732
4750²
MME
2204.9
74.7
SEED;
Vista
582
444
4275
TextQQ
5122
8832
3990
MMMU
462
414
31662
71.8
77.7
MMStar
862
418
2598²
Qencch
MMBenchox
A12D
152a
480
1.000
54.9
39.9
ChartQA
245
668
11992
40.9
80.2
SEED,
3622
432²
5122
MM
1812
422
512²
MM-Vet
MMBench
MME
1142
420a
5122
34.1
MMBenchew
682
4172
5122
HallB
MMMU
MMBench
682
417²
5122
MMStar
GPT-4V
Gemini-B Peo
IXC2-4KHD
Previous open- Source SOTA
Figure 1: (a) Overview of Internl.M-XComposer with different resolutions. Our model based on InternL (91) matches or even surpasses GPT-4V (74) and Gemini Pro (90) in 10 of the 16 benchmarks.
(IXC-4KHD) performance on benchmarks
(b) Image resolution statistic of 16 benchmarks. We report the minimum (Min), median, and maximum (Max) image area (resolution). Both the milerfintra-benchmaae resolution diversity are large, and we sort them by the maximum resolution.
directly. However, the Vision Transformer (ViT) architecture falls short when dealing with images of varying resolutions and aspect ratios, thereby restricting its ability to handle diverse inputs effectively. Alternatively, some methods (50 59
55 46) maintain the vision encoder's resolution, segmenting high-resolution images into multiple low-resolution patches. Yet, these methods are constrained by an inadequate resolution, typically around 1500 X 1500, which does not satisfy the demands of daily content, e.g., website screenshots (85). document pages (70), and blueprints (69). Furthermore, they are confined either few predefined righ-uesolttion
37
51
99
settings (36) 977 48 50 51] [5] 46; 66) (9) or limited range of resolutions (701) 99). thereby restricting their utility across a variety of applications.
In this work, we introduce expands the resolution capabilities of Large VinguanLengu Models (LVLMs) to 4K HD and even higher, thereby setting a new standard in high-resolution ussion-languag understanding. Designed to handle broad range of resolutions, MitemLLM-CCompssr22--55 ratio from 336 pixels up to 4K HD, facilitating its deployment in real-world contexts.
Mo-XCooosse22-4XHD, a pioneering model that for the first time
supports images with any aspect
InternL MILLLM--XCCOO XComposer2 4KHD follows patch division (50) 46) paradigm and enhances it by incorpo- rating an innovative extension: dynamic resolution with automatic patch configuration. To be specific, scaling the resolution of Large Vision-Language Models (LVLMs) 4K HD and even higher standard is far beyond merely increasing the number of patches. involves a nuanced approach to overcoming specific challenges: (1) Dynamic Resolution and Automatic Patch Configuration: Addressing the scarcity of high-resolution training data, our framework introduces a strategy that dynamically adjusts resolution alongside an automatic layout configuration. During training, it maintains the original aspect ratios of images while adaptively altering patch (336 x 336) layouts and counts. This results in a training resolution that exceeds the original image resolutions, reaching up to 4KHD, addressing the shortfall of high-resolution data. (2) Handling Variability in Patch Layouts: Despite the apparent simplicity of dynamic resolution training. the variability in patch layouts can heavily confuse LVLMs. To mitigate this, we introduce newline token after each row of patch tokens to clearly delineate patch layouts, reducing training ambiguity and significantly boosting performance. (3) Inference Beyond 4K Resolution: Our observations reveal that, even when trained on images up to 4K resolution, the model can achieve additional performance improvements during inference by processing images at higher resolutions.

---

SturGPPee Inpecsiag Large Captikes
Anslyee this figure detail
Dhe tteg!
The imag sceeffi paper sited "ShamGPT4V Inpeosing Large Mudti-Mdil Madds Uaivecity of Sciezce and Techaslogy of China and Shanghai Al Labcestury The paper dicusses lhe compamanr betwenn the eaptions in their penponed Stand (v datame and those stilied by nocer large malt: moda) mobels (1 (LMMs) The capliaas are compaare in temms of tazir cualty and perfommanee. The paper asso highlighss the remaalable performance of the propoeed LMM losad on simple
Capiiona
MMMBcads
architcture the bos renarkoble performance acmus negurity ct the mutti mostl benctmatiss
MMBleach-C
ShareGPT4V-77
Waich model has the best peroemaace on the MMiBeach and what its esect rumber?
Qucn-VL-Chat-78
Accoting to the piger, ShvmGPP44V-7 his perfocmazce the MMBercc with score of 68.5.
image size: 2550 3300
sechaeds This prder o
Where can gut fhis project?
coall cottle Chcnsti li optte mad adgen seme
You fris project the following GitHubbliik link: Attp:: gittuhio
Illustrate the fine details present in image.
2021
The image vibrant and colorful infographic that provides an overview of 7 graphic design trends that are expected to dominate 2021. The infographic is divided into 7 sections, each representing different trend.
At the top, there's section titled "Muted Color Palettes", which introducce the concept muted colors as trend for 2021.
This is followed by a section on "Simple Data Visualizations" highlighting the goal of making complex data easy understand.
The third section focuses on "Geometric Shopes Everywhere", showcasing how designers are using geometric shapes their designs. The fourth section delves into 'Flat Icons and Illustations', explaining how brands are using flat and Illustrations their social media graphics, website design, more.
The fifth section introdaces CCassscc Senf Ponts", discussing how these fonts are seen classic, elegant, and trustwortyy.
The sixth sistion explore "Social Media Slide Decks", illustrating how slide decks have become a visual way of communicating longur messagus on Instagrum and LinkedIn. Finally, the seventh section discusses 'T Hasvy Videos", predicting that people will lisely continue to work remotely well into 2021, leading to an increese in video contect that uses text on- screen to communicate messages.
Overall, the infographhc provides comprehensive overview of the graphic design trends that expected to shage the design landszape in 2021.
What the detailed explonation the third part?
The third part of the image Is titled "Geometric Shapes Everywhere'. It explains that last year, desigmers used lot flowing and abstract shapes in their designs. However, this year, they have been replaced with rigid, hard edged geometric shapes and patterns. The hard edges of geometric shape create a great, contrest against muted colors.
Image shoe: 816x 5133
Figure 2: Chat with MiternLM-XCoonie &Composer2-4KHH Some regions of the input HD images are zoomed in for better visualization. For more results please refer to the Supplementary materials.
Furthermore, scaling the training resolution up to 4K standard results a consistent improvement in performance, highlighting the potential for training even beyond 4K resolution. This underscores the capacity for further enhancing model capabilities and suggests a promising trajectory for advancing the frontiers of high-resolution image processing within the domain of large vision-language models.
We evaluate our MetemmLM-XCooposeer
on 16 diverse benchmarks spanning various do- DocVQA OCRBench(58),
mains, including 5 challenging OCR datasets (InfographicVQAAGG)) TextVQA and ChartQA(68), Compared to previous open-source LVLM models and closed- source APIs, our approach achieves SOTA results in 6 of 16 benchmarks, demonstrating competitive performance despite only 7B parameters. As shown in Figure T InternLM- Hww..LM--ooooosr--KHHHt even surpasses the performance of GPT4V (74) and Gemini Pro (90) across ten benchmarks. Notably, our method exhibits excellent performance on 5 challenging OCR datasets, over existing open-source LVLMs by a substantial margin.
3

---

2 Related Works
Large Vision-Languee Models (LVLMs). Large Language Models (LLMs) 9
76
73
23
92 93 39,
10) have gained significant attention due to their impressive performance in various lantuage-related tasks such as text generation and question answering. Following this
91
108
78
enthusiasm, recent Large Vision-Language Models (LVLMs) have emerged(74) 19
16
[8]
28: 32
113 25 110
47
77 102; 4), combining LLMs with vision encoders (79)
109 89)
leverage
the complementary strengths of language and vision modalities. By fusing textual and visual representations, LVLMs can ground language in visual contexts, enabling more comprehensive
understanding and generation of multimodal content (14 20 51
5
95
27
60).
LVLMs for RighlResolution Understanding. Large Rision-Language Models (LVLMs) often employ -ITP-ViT as the visual encoder for vision-dependent tasks. However, the visual encoder's reliance on low resolutions, such 224 224 or 336 336 pixels, limits its effectiveness for high- resolution tasks like OCR and document/chart perception. To enhance high-resolution understanding, recent works have primarily employed the following strategies: (1) High-resolution (HR) visual encoders or dual encoders catering to high-resolution (HR) and low-resolution (LR) inputs (66) 997] 48). For instance, Vary (97) introduces new image encoder supporting HR inputs, which then concatenated with LR embeddings from the original CLIP visual encoder. Similarly, Cog Agent (36) and Mini-Gemini (88) also separate HR and LR images using distinct vision encoders, subsequently merging their features using a cross attssttttntiion module. In contrast, our approach offers a more simplified solution and shows ad advantages for varying resolutions and aspect ratio inputs. (2) Cropped image patches (50) 59; 99, Tor, 37 51 46). For example, Monkey (50) employs sliding windows to segment images into patches, subsequently processing them with LoRA fine-tuning. TextMonkey (59) further proposes shifted window attention and token resampler to consider the connections among different patches. Fuyu ( eliminates the need for the image encoder by directly processing a raw image patch sequence. These approaches are confined to either a few predefined high-resolution settings (36 97 48
55 46 66, 59) or a limited range of resolutions (37 99). Conversely, our method devises dynamic resolution and automatic path configuration strategy to support the scaling from 336 pixels to 4K resolution, and the maximum resolution is larger than previous approaches
50
51
(e.g., 1.5k for Monkey (50) and 1.2k for UReader (TOI). For the first time, approach discussed the challenges and solutions for handling variability in image feature patch layouts, ensuring effective training with dynamic high resolutions.
LVLMs for Document Understanding, Document understanding involves analyzing and compre- hending various digital documents, such as figures, tables, and academic papers. Many document understanding tasks require models to handle high-resolution inputs, complex layouts, various aspect ratios, and diverse document formats. To enhance the capabilities of LVLMs for document under- standing, several works have collected and constructed high-quality document instruction tuning data, including VAR (122) mPL.U-DDocOw (TOO) and TGDoc (96). DocPediaDoc processes document inputs in the frequency domain. Some previous works have improved document understanding ability by designing special modules for high-resolution inputs, such as HR and LR encoders (3,997) or cropped image patches (IOT) 59, (9). Our InternLM MiternLM-Xoomee XCCompoee22-4KHH first scales 4K resolution inputs and demonstrates strong document understanding ability OCR- related benchmarks. Also, our approach also achieves comparable results to state-of-the-ar open- sourced LVLMs other general LVLM benchmarks like perception and reasoning (61 57 33 15).
(30)
3
Method
3.1
Model Architecture.
The model architecture of XComposer2(27) (XComposer2 / IXC2 in the following for simplicity ), including light- Vision Encoder OpenAl ViT-Large/14, Large Language Model InternLM2-7B, and Partial LoRA for efficient alignment.
mainly follows the design of InternLM-
3.2 High Resolutoo Input.
Dynamic Patch Configuration. Utilizing static input image size for processing high-resolution images, particularly those with varying aspect ratios, is neither efficient nor effecttve. To overcome

---

PROTECTING
Resiine
Encade Merge
Lo
Lo
Dymamiic Image
Confieuurtion
POOTECTING
in
Encode & Marge
In
LLM
The image presents vibrant and informative infographic titled "Protecting Our Planet Starts With You". The infographic is divided into various sections, each dedicated to a different aspect of environmental protection. In the top left corner,"
Tokenize
Figure 3:
The framework of InterLLM M- XCoomossr2-4KHHD Our model processes the high-resolution image with Dynamic Image Partition strategy and concatenates the image tokens with text tokens as LLM input.
this limitation, we introduce dynamic patch configuration approach via image partitioning, as shown in Figure 3 Our method strategically segments the image into smaller patches, while maintaining the integrity the original image's 's aspect ratio.
Given a maximum patch number H. the image æ with size [h, w| is resized and padded to the new image & with size (ph. 336, pw 336). This process is subject the following constraints:
Pw x S H: Ph. = [Pw x h/wl
(1)
here Pi and Ph represent the number of patches in each row and column, respectively. We then split the å into Ph Pw non-overlapped patches. Each patch is small image with 336 x 336 size and we treat these patches as individual inputs for the ViT.
In the following, we use 'HD-H' to represent our high-resolution setting with the constraint of H patches. For example, the 'HD-9' allows up 9 patches, including range of resolutions such as 1008 x 1008, 672 x 1344, 336 x 3024, etc.
Global-Local Format. For each input image, present it to the model with two views. The first is the global view, where the image is resized to a fixed size (in our case, 336 x 336). This provides a macro understanding of the image. Empirically, we have found this to be crucial for the LVLM to correctly understand the image. The second view is the local view. We divide the image into patches using the previously mentioned Dynamic Image Partition strategy and extract features from each patch. Following feature extraction, the patches are reassembled into a large feature map. The feature map is then flattened to the final local features after straightforward token merging process.
Patch Layout Indicator. Given that an image will have dynamic patch layout in our method, number of tokens for each row can vary across different images. This variation will confuse the LVLM, making it difficult to determine which tokens belong to the same row of the image and which ones belong the next row. This confusion hinders the LVLM's ability to understand the 2D structure of the image, which crucial for comprehending structural image content such as documents, charts, and tables. To address this issue, introduce a learnable newline (")n') token the end of each row of the image features before the flattening. Finally, we concatenate the global and local views, inserting special separate ('sep') token between them to distinguish the two views.
3.3 Pre-Training
During the pre-training phase, the L.L frozen while both the vision encoder and Partial LoRA are fine-tuned to align the visual tokens with the LLM following (C)mposer2(2)). The pre-training data
5

---

Table 1: Pre-Training Datasets. The data collected from diverse sources for the three objectives.
Task
Dataset
General Semantic Aligament
ShareGPTav PT [4]
COCO 177. Nocaps T.
TextCaps (66 SBU
75
LATON400M
N0
CC
3M
S
World Knowledge Alignment
Concept Data
Vision Capability Enhanccment
WanJuan (5) Flicker 103 MMC-Inst 54], RCTW-1764] CTW [162
LSVT
RcCT+III.
Ar
Table 2: Supervised Fine--Tuning Datasets. We collect data from diverse sources to empower the model with different capabilities. The image resolution is also different for different tasks.
Task
Resolution Setting
Dataset
Caption
HD-25
ShareGPTAV (14) COCO (17) NNccpps (1
General QA
HD-25
VQAv2 GQA (38] OK-VQA (67]
VD 26
RD VSR
Science
HD-25
AI2D [2).SQA (63). TQA IsonQA
Chart QA
HD-25
DVQA [40] ChariQA, CharQA-AUU (68
Math QA
HD-25
MatQQA (104] Geometry3
T54 CLEVR-MATH [Spper
World Knowledge QA
HD-25
A-OKVQA KVQJ (82). [IQuAE[44
OCR QA
HD
TexIVQA(7] OCR VQA VQA
HD-OCR QA
HD-55
InfoVQA (ocVQA(10]
Conversation
LLaVA-150k (56), LVIS-InstuerVV (4). ShueGGTTennázh
21] InternLM-Cha(911
mainly follow the design in XComposer2 which is curated with three objectives in mind: 1) general semantic alignment, 2) world knowledge alignment, 3) vision capability enhancement. In this paper, we focus on high-resolution and structural image understanding. So collected OCR and chart data from diverse sources to enhance this specific capability, as shown in Table I
In practice, we employ the OpenAI CLIP ViT-L-14 14-336 as the vision encoder. We keep the VIT resolution as 336 x 336 while adopting Dynamic Patch Configuration to handle higher-resolution images. For pretraining, we use the 'HD-25' configuration, which resizes the input image to random larger resolution to generate more patches, with the constraint that the total number of patches does not exceed 25. For each image patch tokens, the image token number is decreased to 1/4 with simple merge operation. We concatenate the nearby 4 tokens into a new token through the channel dimension, then align it with the LLM by an MLP. The 'sep" and "\n" tokens are randomly initialized. For the Partial LoRA, we set rank of 256 for all the linear layers in the LLM decoder block. Our training process involves batch size of 4096 and spans across 2 epochs. The learning rate linearly increases to 2 x 10 within the first 1% of the training steps. Following this, it decreases to 0 according cosine decay strategy. To preserve the pre-existing knowledge of the vision encoder, we apply a layer-wise learning rate (LLDR) decay strategy, and the decay factor is set to 0.90.
3.4 4KHD Supervised Fine-tuning (SFT)
After pre-training, we empower the model to understand high-resolution images and solve diverse challenges. Different from conventional perception tasks (e.g.. VQAv2, GQA) which typically answer questions based on the noticeable object in the image. OCR-related tasks depend on a detailed understanding of text within a high-resolution image. For instance, in Info InA. the length of the longer side of 50% of the images exceeds 2000 pixels. Low-resolution inputs can distort the dense text information, causing the model to fail in its understanding. However, we have observed a resolution saturation phenomena with perception tasks, where a higher resolution makes minor gains.
To address this, we introduce a mixee--esoluttion strategy for more efficient training. For tasks requiring high resolution, we employ the 'HD-55' setting during training. This allows for the input of 4K (3840 x 1600) images without necessitating additional image compression. These tasks are referred to as the HD-OCR QA tasks in Table 2 For the other tasks, we apply the 'HD-25' resolution setting for them. As in pre-training, we adopt the dynamic-resolution strategy during SFT, images are resized to fall within a range between their original size and the size specified by the 'HD' setting. This dynamic approach enhances the robustness of the LVLM against differences in input resolution, thereby enabling the LVLM to utilize a larger resolution during inference. For instance, we have observed that using the 'HD30' setting yields better results on most OCR-related tasks when the LVLM is trained under the 'HD25' setting.
In practice, we jointly train all the components with a batch size of 2048 over 3500 steps. Data from multiple sources are sampled a weighted manner, with the weights based on the number of data from each sourced (See Appendix B.1 for more details). As the 'HD-55' setting has double image tokens than the 'HDD25', we adjust the data loader to enable different batch sizes for them and adjust their weight accordingly. The maximum learning rate is set to 5 x 10-3 and each component has its

---

Table 3: Comparison with closed-source APIs and previous open-source SOTAs. Our InternLM-- XComposer2-4KKB gets SOTA results in of the benchmarks with only 8B parameters, showing competitive
results with current closed-source APIs. The best results are bold and the second-best results are underlined.
Doc
Chart
Info
Text
OCR
MM
Math Vista
MMB
MMB
SEED
QBench
MM-
Hall
Method
VQA
VQA
VQA
Bench
AI2D
MMMU
MME
EN
CN
Image
Tost
Bench
Open-Source Source
37
[37
57]
36
[6]
155
[55
[55]
20
[55
[55
[55]
[110]
[95]
(50
Previous SOTA
8R
8B
8B
18B
18B
35B
35B
35B
40R
34B
35B
35B
35B
8B
178
10B
822
70.2
44.5
76.1
59.0
52.1
39.0
78.9
51.6
2050.2
81.1
79.0
75.7
64.4
54.5
39.3
Closed-source API
GPT-4V
88.4
78.5
75.1
78.0
51.6
57.1
47.8
56.8
1,926.5
77.0
74.4
69.1
74.1
56.8
Gemini-Pro
88.1
74.1
75.2
74.6
68.0
42.6
45.8
70.2
47.9
1,933,3
73.6
74.3
70.7
70.6
59.2
45.2
LXC2-VL
27
57.7
72.6
34.4
70.1
53.2
55.4
57.6
81.2
41.4
2,220.4
80.7
79.4
74.9
72.5
46.7
41.0
IXC2-4KHD
90.0
81.0
68.6
72.2
67.5
54.1
57.8
39.7
2.204.9
80.2
77.7
74.7
71.8
54.9
40.9
Table 4: Comparison with open-source SOTA methods. IXC2-4KHD outperforms competitors in most benchmarks. The best results are bold the second- best results are underlined.
Method
LLM
MMStar
Mist
AI2D
MME"
MME
MMB
MMB
SEED
QBench""
MM- Vet
Qwen Bren-VL--Cie VL--Chat
Qwen-7B
37.5
33.8
63.0
1,487.5
360.7
60.6
56.7
58.2
61.7
47.3
ShareGPT4V
Vicuna-7B
33.0
25.8
58.0
1,567.4
376.4
68.8
62.2
69.7
37.6
Monkey
Qwen-7B
38.3
34,8
62.5
1,522.4
401.4
12.4
67.5
68.9
33.0
CogVLM-17B
Vicuna-7IB
36.5
55.9
68.8
LLaVA XTTuner
InemL.M2 20B
24.6
65.4
75.1
73.7
70,2
37.2
LLaVA-1.5
Vicuna-133
32.8
1,5313
295.4
67.7
63.6
68.2
35.4
LLaVA Next
Vicuna 13B
38.3
32.4
72.2
1,445.0
296.0
70.0
68.5
71.4
44,9
Interni
Tneml M-7B
56.9
1,528.4
391.1
72.4
IXC2-VL
InemLM2-78
55.4
57.6
81.2
1,712.0
$30.7
80.7
79.4
74,9
72.5
46.7
IXC2-IKHD
InerLLM2-77 MB
54.1
57.8
80.9
1,655.9
548.9
80.2
77.7
74.7
71.8
54.9
own unique learning strategy. For the vision encoder, we set the LLDR to 0.9, which aligns with the pretraining strategy. For the LLM, we employ a fixed learning rate scale factor of 0.2. This slows down the update of the LLM,, achieving balance between preserving its original capabilities and aligning it with vision knowledge. takes almost 40 hours with 256 A100 GPUs.
4
Experiments
In this section, we validate the benchmark performance of our InterL NiemLL-XXComposr2244HH (IXC2- 4KHD in the following for simplicity) after supervised fine-tuning.
4.1 LVLM Benchmark results.
In Table and Table 4 we compare our IXC2-4KHD a list of benchmarks with both SOTA open-source LVLMs and closed-source APIs. Here we report results in DocVQA ChartQA(68) Infographic VQA 69.. TexIVQA(87), OCRBench
155,, Math (61, (MMU(T07))
A12D(42). MME (3I) MMBench (MMB) (57). MMBench- Chinese (MMBCM CNN (57), SEED- Bench Image Part (SEED' (55), QBnnch-Testeer (QBench' (998) MM-Vet (105), HellusionBenchh
(HallB)(4)) The evaluation is mainly conducted on the OpenCompass VLMEvalKi (4) for the unified reproduction of the results.
Comparison with Closed-Source APIs. As demonstrated in Table IXC2-4KHD exhibits com- petitive performance across variety of benchmarks, rivaling that of Closed-Source APIs. Owing to its high-resolution input, IXC2-4KHD achieves a score of 90.0% on Doc and 81.0% on ChartQA, thereby surpassing GPT-4V and Gemini-Pro with non-trivial margin. In the challenging Infographic VQA task, our model is the first open-source model that is close to the performance of Closed-Source APIs, exceeding the performance of previous open-souree models by nearly 20%. In addition to OCR-related tasks, IXC2-4KHD is general-purpose Large Vision- Vision-Language Modal that excels in semantic-level tasks, demonstrating competitive results.
Comparison with Open-Source Models. We also conduct a comprehensive comparison with open- source LVLMs under similar model scale. As shown in Table our model significantly outperforms existing open-source models, achieving competitive results across all benchmarks.
High-resolutinn Understanding Evaluation. Then we compare IXC2-4KKD with models that are specifically designed for high-resolution understanding tasks. We report the results of 5 high- resolution benchmarks in Table 5 general LVLM. IXC2-4KHD shows superb performance on
7

---

Table 5: High-resolution Evaluation. IntenrL.M-X -eemL..--XCCommpeer---k1i has the largest input resolution and outperforms open-source LVLMs which are specifically tuned for document understanding.
Model
Model Size
Max Resolution
DocVQA
CharqQA
InfoVQA
TexiVQA Vul OCRBench
TextMonkey(59
9B
896x896
73.0
66,9
28.6
65.6
55.8
LLaVA-UHD 99
13B
1008x672
67.7
CogAgent [6
17B
1024x
81.6
68.4
44.5
76.1
59.0
UReader [10]
7B
1120x896
65.4
59.3
42.2
57.6
DocOwl 1,5 37
8B
1344x TEAAX1344 1344
82.2
70.2
50.7
68.6
IXC2-4KHD
8B
3840x
90.0 (+7.8)
81.0
(+10.8)
68.6
(+17.9)
77.2 (+1.2)
67.5 (+8.5)
Table 6: Influence of Inference Resolution. The model achieves better performance on text-rclated tasks when the inference resolution higher than its training resolution.
Train
Eval
Doc
Info
Text
Chart
MMB
MME
SEED
HD9
79.4
50.5
73.8
78.2
79.5
2,201
76.6
HD9
HD16
58.6
75.8
79.3
2,198
76.7
HDI6
84.9
60.8
75.7
80.1
80.2
2,129
75.7
HD16
HID25
85.9
62.1
75.8
79.1
80.1
2,100
75.4
HD25
87.0
63.6
76.0
80.3
78.5
2,209
74.9
HD25
HD30
87.4
64.6
76.2
79.4
78.9
2.173
74.3
65
60
55
DocVQg Ve
TextVO4 AM
CheriQ
MMBench Pelee
MME
Seed!
A12D Tnt
Talans)
Tokey)
Figure 4: Influence of Training Resolution. High-resolution training is critical for HD-OCR tasks, while its gain other tasks is minor.
these tasks and outperforms competitors with a large margin. For example, IXC2-4KHD gets 68.6% on Infographic ViciciiQQA VQA, surpassing recent DocOwl 1.5 with +17.9%. For the OCRBench, IXC2-4KHD gets 67. 67.5%, outperforms CogAgent with +8.5%.
4.2 Dive into Resolution
Nligh-Resolution Training is Critical for HD-OCR tasks. study four resolution settings: HD-9 (1561 image tokens at most, we simply the statement in the following). HD-16 (2653 tokens), HD-25 (4057 tokens), and 4KHD (8737 tokens). Here we report the validation set of InfoVQA, DocVQA, and TextVQA, test set of ChartQA and AI2D, MMBench EN-Test, and a 2k subset of SEEDBench (we denote it as SEED*) ) In the following, we report results on the above benchmarks by default.
As illustrated in Fig we note a significant improvement in the HD-OCR tasks as the resolution increases. For instance, the model achieves only a 50.5% score on the infographicVOA with the HD-9 setting. However, when we switch to the HD-16 setting, we observe a performance gain of +10.2%. The performance continues to improve as the resolution increases. with saturation not observed even for the 4KHD setting. Due to computational constraints, we defer the exploration of the upper bound of improvement future work. In terms of other OCR-related tasks, the performance gain attributable to increased resolution is relatively minor. For the perception-relater benchmarks, performance is saturated on the resolution that only has negligible difference between the four settings.
Higher Inference Resolution Leads to better results on Text-related Tasks. An intriguing observation from our experiments is that our model, when inferring with higher resolution. tends to yield improved results on text-related tasks. We present the results of HD-9, HD-16, and HD-25 in Table
6 For instance, IXC2-HDS achieves a 50.5% score on Infographic When we infer with HD16, 16,,,, we see a performance gain of +8.1%, without additional training. Similar improvements are also observed with IXC2-HDI6 and IXC2-HD25. We posit that dynamic image token length used in training enhances the robustness of the LVLM, leading to better results when the text in the image is more 'clear' in the higher-resolution input. Conversely, the results on ChartQA consistently degrade under this setting. This could be due to the model becoming confused about the chart structure when

---

Table 7: (a) Influence of Indicator "nn' in the Image Features. "(n' helps LVLM understand structural images when the input resolution dynamic and large. (b) Ablation on Token Merging Operation. Both the simple concatenation operation and the C-Abstractor works well.
Model
"/n'
Doc
Info
Text
Chart
MMB
MME
SEED
Strategy
Doc Info Text Chart MMB MME
SEED'
HD9
x 79.5 50.3
74,0
78.2
79.1
2206
75.9
Re- Sampler
86.2 67.1 75.3 78.8
79.6
2124
74.2
HD9
79.4
50.5
73.8
78.2
79.5
2201
76.6
C-Abstractor 88,6 69.5 77.1
80.6
80.4
2236
76.7
4KHD
x
88.1
67.4
75.9
80.4
79.9
2232
76.4
Concat
89.0 69.3 77.2
81.0
80.2
2205
76.2
4KHD
89.0 69.3
77.2
81.0
80.2
2205
76.2
Table 8: Influence of Global- View in the Input. Global-vies view is critical for most benchmarks.
Model
Doc
Info
Text
Chart
MMB
MME
SEED
HD9
79.4
50.5
73.8
78.2
79.5
2201
76.6
w/o global-view
78.1
47.9
71.2
77.9
75.1
2019
76.2
Table Strategy-Level comparison between LLaVA-Next and our IXC2-4KHD. Our strategy
reaches better performance under a similar image token number constrain.
Model Nmae
HD Strategy
Image Tokens
Max Resolution
DocVQA
TeXtVQA
CharQA
MMBench
LLeVA-Next
LLaVA-Next
2880
672x672
78.2
52.0
69.5
72.1
IXC-LLaVA
LLaVA-Next
2880
672x672
78.9
71.5
74.1
77.6
IXC-4KHD
HD9
1440
1008x1008
79.4
73.8
78.2
79.5
IXC-4KIID
IID16
2448
1344x1344
84.9
75.7
80.1
80.2
the resolution increased. Additionally, similar to the observation from Figure 4 the impact of resolution on perception-related benchmarks appears to be quite minor.
4.3 Hligh-Resolution Strategy Ablation
The Role of Global-View We first examine the impact of the global view in our Global-Local Format. As indicated in Table a), a), we find that the global view is essential for the LVLM to accurately comprehend the input image. When it is removed, the model performs worse across all benchmarks. For instance, the model experiences a -4.4 -4.4% drop in performance on the MMBench EN-Test without the global view. We contend that the global view offers general macro understanding of the image, which the model struggled to derive from the large number of tokens in the local view.
The Role of the Newline Token. We incorporate a special newline token the end of each row of the image features before the flattening operation. This token serves as an indicator of the image's 2D structure. We examine its impact on both the HD-9 and 4KHD strategies in Table 7[) a). When a fixed high-resolution strategy HD-9 is employed. we observe that the benefit derived from the newline token is minor. This could be attributed to the LVLM's ability to handle limited differences in image ratios after training. However, when we implement a more challenging 4KHD (HD-25 + HD-55) strategy, which exhibits significant diversity in both image ratio and token number, the LVLM demonstrates notable decline in performance on OCR-relatee tasks without the newline indicator. This finding supports our hypothesis that the LVLM struggles to comprehend the shape of the image when the image tokens are directly flattened into a ID sequence. The newline token can assist the model in better understanding the structure the image.
Influence of Token Merging Operation. In practice, we employ a simple merging operation that concatenates four adjacent tokens along the channel dimension. We have found this approach to be effective in reducing the number of image tokens efficiently. Here we study the influence of different token-merging operations under the 4KHD setting. In Table (bb) we study two additional strategies: Re- Re-Sampler(5) and C-Abstractor(12) with their default setting and the same compressing rate 0.25, i.e., reducing an image with 576 tokens to 144 tokens. Results show that both concatenation and C-Abstractor work well and get similar results on most benchmarks, this observation is also consistent with the study in MM-1 (II) that the influence of the connector is minor. However, the Re-Sampler performs worse than the other methods with noticeable margin. We argue this is caused by the
learnable queries used for gathering information requiring a great number of data for training, our pre-training data is somewhat lightweight for it to converge fully.

---

Table 10: Inference Efficieny Analysis. The image token number mainly inference the prefix speed. and their difference in the decoding part is neglectable.
HD
image tokens
prefix encoding time
per-token decoding speed time generation 2048 new tokens
HID
1440
0.2845
0.0982
201.4
HDI6
2448
0,3966
0.0983
201.7
HD25
3744
0,5513
0.0981
201.5
Strategy-Lewe Comparison. For a fair comparison, we trained a new model with the identical architecture, training strategy, and dataset as our original model, but with one key modification: we adopted the high-resolution strategy from LLaVA-Next. We name it as IXC-LLaVA and compare it with 1) LLa VA-Next 0530 official results and 2) IXC2-4KHD under HD-9/16 setting. The results in Table demonstrate that IXC-LLaVA achieves promising performance across all six benchmarks, leveraging the benefits of additional training data and advanced IXC architecture design. However, is outperformed by IXC2-4KHD HD-99, which utilizes fewer image tokens yet yields better results. This fair comparison underscores the efficiency and effectiveness of our proposed high-resolution strategy, highlighting its advantages over the LLa VL-Ve-Next approach.
Inference Efficiency Analysis. Our model processes high-resolution images with numerous image tokens, and here we study its inference efficiency in real-world usages. The model inference process consists of two stages: encoding the prefix (model input) and autoregressively decoding new tokens (model output). Correspondingly. the inference efficiency considers two parts: time to encode the prefix and speed to decode each token. Here we report the prefix encoding time and per-token decoding speed under different HD settings. We test the speed with Nvidia-A 100 80G. With the results in Table TO we have three observations: 1) Prefix encoding time increases linearly with the number of prefix tokens. 2) Decoding speed remains relatively constant, regardless of prefix length, thanks to optimizations on transformers from research communities and companies, including kv-cache and flash-attention. 3) When generating 2048 tokens, total inference time usage nearly identical across HD9 to HD55, as encoding time is much smaller. Based on the above analysis, believe the inference efficiency of model is acceptable. Besides, we believe some targeted designs can further improve efficiency, while our paper focuses on enabling LVLM to understand nigh-resolution images with general and effective solution, and we would leave the efficiency exploration in future work.
5 Conclusion
In this paper, we propose the InternL--XXommm MM--coooosee2-44HHH that exceeds the performance of previous open-source models OCR-related tasks and also achieves competitive results general-pus purrpsse LVLM benchmarks. Thanks to our dynamic resolution and automatic patch configuration, our model supports maximum training resolution of up to 4K HD. We also integrate global view patch to support the macro understanding and a learnable newline token to handle the various input image resolutions. Our model's performance continues to improve for HD-OCR tasks. Notably, we do not observe any performance saturation even for the 4KHD setting, and we have explored the upper bound due to the computational burden increasing with higher-resolution inputs. In future work, we plan to explore efficient solutions for accurate LVLM training and inference, enabling our model to handle even higher resolutions while maintaining computational efficiency.
the training resolution increases
6
Acknowledgment
This project funded in part by Shanghai Artificial Intelligence Laboratory, the National Key R&D Program of China (20227D016000)) the Centre for Perceptual and Interactive Intelligence (CPII) Ltd under the Innovation and Technology Commission (ITC)'s InnoHK. Dahua Lin is of CPI under the InnoHK
10

---

References
[1] Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. Nocaps: Novel object captioning at scale. In Proceedings of the IRFF/CVF international conference on computer vision, pages 9948-8957, 2019.
[2] 01. AI, :. Alex Young. Bei Chen, Chao Li, Chengen Huang. Ge Zhang. Guanwei Zhang, Heng Li, Jiangeheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang. Shiming Yang. Tao Yu, Wen Xic, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang. Yuxuan Cai,, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai. Yi: Open foundation models by 01 ai, 2024.
[3]
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick. and Devi Parikh. Vqa: Visual question answering. In International Conference on Computer Vision
(ICCV), 2015.
[4] Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, Jenia Jitsev, Simon Komblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. Openflamingo: An open-source framework for training
large autoregressive vision-language models. arXiv.org, 2023.
[5]
Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang. Sinan Tan, Peng Wang. Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv.org,
2023.
[6] Baichuan. Baichuan 2:
Open large-scale language models. arXiv.org, 2023,
[7] Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sagnak Taşırlar. Introducing our multimodal models, 2023.
[8] Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marçal Rusinol, Emest Valveny, CV Jawahar. and Dimosthenis Karatzas. Scene text visual question answering. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4291-4301, 2019.
[9] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.
Advances in Neural Information Processing Systems (NewrIPS), 33:1877-1900 2020.
[10] Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo, Qipeng Guo, Conghui He, Yingfan Hu, Ting Huang, Tao Jiang, Penglong Jiao, Zhenjiang Jin, Zhikai Lei, Jiaxing Li, Jingwen Li, Linyang Li, Shuaibin Li, Wei Li, Yining Li, Hongwei Liu, Jiannning Liu, Jiawei Hong. Kaiwen Liu. Kuikun Liu, Xiaoran Liu. Chengqi Lv, Haijun Lv, Kai Lv, Li Ma. Runyuan Ma, Zerun Ma. Wenchang Ning. Linke Ouyang. Jiantao Qiu, Yuan Qu, Fukai Shang. Yunfan Shao, Demin Song, Zifan Song, Zhihao Sui, Peng Sun. Yu Sun. Huanze Tang. Bin Wang. Guoteng Wang, Jiaqi Wang. Jiayu Wang, Rui Wang, Yudong Wang, Ziyi Wang, Xingjian Wei, Qizhen Weng, Fan Wu, Yingtong Xiong, Chao Xu, Ruiliang Xu, Hang Yan, Yirong Yan, Xiaogui Yang, Haochen Yc, Huaiyuan Ying, Jia Yu, Jing Yu, Yuhang Zang, Chuyu Zhang, Li Zhang, Pan Zhang, Peng Zhang, Ruijie Zhang, Shuo Zhang, Songyang Zhang, Wenjian Zhang, Wenwei Zhang, Xingcheng Zhang, Xinyue Zhang, Hui Zhao, Qian Zhao, Xiaomeng Zhao Fengzhe Zhou, Zaida Zhou, Jingming Zhuo Yicheng Zou, Xipeng Qiu, Yu Qiao, and Dahua Lin. Internlm2 technical report. arXiv preprint arXiv:2403. 17297, 2024.
[11] Yuhang Cao. Pan Zhang, Xiaoyi Dong, Dahua Lin, and Jiaqi Wang. DualFocus: Integrating macro and micro perspectives in multi-modal large language models. arXiv preprint arXiv:2402. 14767, 2024.
[12] Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. Honeybee: Locality-enhaneer projector for multimodal Ilm. arXiv preprint arXiv:2312.0611 02.4422,2222 2023.
[13] Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao, Shikra: Unlcashing multimodal I1m's 's referential dialogue magic. arXiv.org, 2023.
[14] Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi- modal models with better captions. arXiv preprint arXiv:2311.1277 2023.
[15] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang. Yu Qiao, Dahua Lin, and Feng Zhao. Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.2033 20333, 2024.
[16] Xi Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Carlos Riquelme Ruiz, Sebastian Goodman, Xiao Wang, Yi Tay, Siamak Shakeri, Mostafa Dehghani, Daniel Salz, Mario Lucic, Michael Tschannen, Arsha Nagrani, Hexiang Hu, Mandar Joshi, Bo Pang, Ceslce Montgomery, Paulina Pietrzyk, Marvin Ritter, AJ Piergiovanni, Matthias Minderer, Filip Pavetic, Austin Waters, Gang Li. Ibrahim Alabdulmohsin. Lucas Beyer. Julien Amelot. Kenton Lee, Andreas Peter Steiner, Yang Li. Daniel Keysers. Anurag Arnab, Yuanzhong Xu, Keran Rong, Alexander Kolesnikov, Mojtaba Seyedhosseini, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut. Pali-x: On scaling up a multilingual vision and language model, 2023.
[17] Xinlei Chen, Hao Fang. Tsung-Yi Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollar, and
C. Lawrence Zitnikk Microsoft coco captions: Data collection and evaluation server, 2015. [18] Xi Chen. Xiao Wang, Lucas Beyer. Alexander Kolesnikov, Jialin Wu, Paul Voigtlaender. Basil Mustafa, Sebastian Goodman, Ibrahim Alabdulmohsin. Piotr Padlewski, Daniel Salz, Xi Xiong, Daniel Vlasic.
11

---

Filip Pavetic, Keran Rong, Tianli Yu, Daniel Keysers, Xiaohua Zhai, and Radu Soricut. Pali-3 vision language models: Smaller, faster, stronger, 2023.
[19] Xi Chen, Xiao Wang. Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding. Keran Rong, Hassan Akhari, Gaurav Mishra, Linting Xue, Ashish Thapliyal, James Bradbury, Weicheng Kuo, Mojtaha Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme, Andreas Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut. Pali: A jointly-caled scaled multilingual language-image model, 2023.
[20] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Intervvl: Sealing up vision foundation models and aligning for generic -ssau--in tasks. arXiv preprint arXiv:2312. 14238.
2023.
[21] Wei-Lin Chiang. Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang. Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez. lon Stoica, and Eric P. Xing. Vicuna: An open- spen-source chatbot impressing gpt-4 with 90%* a chatgpt quality, March 2023.
[22] Chee Kheng Chag. Yuliang Liu. Yipeng Sun. Chun Chet Ng. Canjie Luo, Zihan Ni, ChuanMing Fang. Shuaitao Zhang. Junyu Han, Errui Ding. al. lcdar2019 robust reading challenge on arbitrary-shaped lext-rrc-art. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages
1571-1576. IEEE, 2019.
[23] Aakanksha Chowdhery. Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways, arXiv.org. 2022.
[24]
OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models.
https://githhb. com/open- compass/ pppppppps//ppccompasss.... 2023.
[25] Wenlrang Dat, Junnan L1, Dongxu LI, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang. Boyang
Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with
instruction tuning, 2023,
[26] Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José M.F. Moura, Devi Parikh, and Dhruv Batra. Visual Dialog. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[27] Xiaoyi Dong, Pan Zhang. Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao. Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Xinyue Zhang, Wei Li, Jingwen Li, Kai Chen, Conghui He, Xingeheng Zhang. Yu Qiao, Dahua Lin, and Jiaqi Wang. Internlmxccoppont xcomposer2: Mastering free-form ext-image composition and comprehension in vision-language large model. arXiv preprini arXiv:240I.
2024.
[28) Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong. Tianhe Yu, Wenlong Huang. Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Mare Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. Palm-e: An embodied multimodal language model. In arXiv preprint arXiv:2.033.03.33
2023.
[29] Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu. Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pages 320-335, 2022.
[30] Hao Feng, Qi Liu, Hao Liu, Wengang Zhou, Houqiang Li, and Can Huang. DocPedia: Unleashing the power of large multimodal model in the frequency domain for versatile document understanding. arXiv preprint 2311:23311118111
2023.
[31] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulci Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 13394, 2023.
[32] Chaoyou Fu. Renrui Zhang, Zihan Wang, Yubo Huang. Zhengye Zhang, Longtian Qiu, Gaoxiang Ye, Yunhang Shen, Mengdan Zhang, Peixian Chen, Sirui Zhao, Shaohui Lin, Deqiang Jiang. Di Yin, Peng Gao, Ke Li, Hongsheng Li, and Xing Sun. A challenger to gpt-4v? early explorations of gemini in visual expertise. arXiv preprint arXiv:23 /2. 12436, 2023.
[33] Yuying Ge. Yixiao Ge. Ziyun Zeng. Xintao Wang, and Ying Shan. Planting a seed of vision in large 2023.
language model. arXiv preprint arXiv:2307.08000
[34] Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang. Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. Hallusionbench: An advanced diagnostic suite for entangled language hallucination & visual illusion in large vasion-language models, 2023.
[35] Conghui He. Zhenjiang Jin, Chaoxi Xu, Jiantao Qiu, Bin Wang, Wei Li, Hang Yan, Jiaqi Wang. and Da Lin. Wanjuan: A comprehensive multimodal dataset for advancing english and chinese large models. ArXiv, abs/2308. 10755, 2023.
[36] Wenyi Hong, Weihan Wang, Qingsong Lv. Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. arXiv preprint 132.0231,..0814,, 2023.
12

---

[37] Anwen Hu, Haiyang Xu, Jiabo Ye. Ming Yan, Liang Zhang, Bo Zhang, Chen Li, Ji Zhang, Qin Jin, Fei Huang. al. mplug-docowl 1.S: Unified structure learning for ocr-free document understanding. arXiv preprint urXiv:2403./2899
2024.
[38] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. Conference on Computer Vision and Pattern Recognition (CVPR),
2019.
[39] Albert Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William EI Sayed. Mistral 7b, 2023.
[40] Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. Dvqa: Understanding data visualizations via question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 5648-5656, 2018.
[41] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.00000 00061.0....2.00 2020.
[42] Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Sco, Hannanch Hajishirzi, and Ali Farhadi.
A diagram is worth dozen images. In Computer Vision- ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pages 235-251. Springer, 2016.
[43] Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi. Ali Farhadi, and Hannaneh Hajishirzi. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. In Proceedings of the IEEE Conference on Computer Vision and Pottern recognition, pages 4999-5007, 2017.
[44] Paul Lemer, Olivier Ferret, Camille Guinaudeau, Hervé Le Borgne, Romaric Besançon, José G Moreno, and Jesús Lovón Melgarejo. Viquae, dataset for knowledge-basse visual question answering about named entities. Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 3108-3120, 2022.
[45] Bohao Li, Rui Wang. Guangzhi Wang. Yuying Ge, Yixiao Ge, and Ying Shan. Seed-hench: Benchmarking multimodal llms with generative comprehension, 2023.
[46] Bo Li, Peiyuan Zhang, Jingkang Yang. Yuanhan Zhang, Fanyi Pu, and Ziwei Liu. Otterhd: A high- resolution multi- mottlityyy model, 2023.
[47] Bo Li, Yuanhan Zhang. Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi- modal model with in-context instruction tuning. arXiv.org, 2023.
[48] Yanwei Li. Yuechen Zhang, Chengyao Wang. Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-Gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403. 18814, 2024.
[49] Zhuowan Li. Xingrui Wang, Elias Stengel-Esking Adam Kortylewski, Wufei Ma, Benjamin Van Durme, and Alan Yuille. Super-clevr: A virtual benchmark to diagnose domain robustness visual reasoning. In Proceedings of the /EEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14963- 14973, 2023.
[50] Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yaho Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi-modal models.
arXiv preprint arXiv:2311.00 1.06607, 2023.
[51] Ziyi Lin, Chris Liu, Renrui Zhang. Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal
large language models. arXiv preprint orXiv:2311.0755
2023.
[52] Adam Dahlgren Lindström and Savitha Sam Abraham. Clevr-math: A dataset for compositional language.
visual and mathematical reasoning. arXiv preprint S..iv:2200.0535555 2022.
[53] Fangyu Liu, Guy Edward Toh Emerson, and Nigel Collier. Visual spatial reasoning. Transactions of the
Association for Computational I Linguistics, 2023.
[54] Fuxiao Liu, Xiaoyang Wang, Wenlin Yao. Jianshu Chen, Kaiqiang Song, Sangwoo Cho, Yaser Yacoob, and Dong Yu. Mme: Advancing multimodal chart understanding with large-scale instruction tuning. arXiv preprint arXiv:2311. 100774, 2023.
[55] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning. ocr, and world knowledge. January 2024.
[56] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lec. Visual instruction tuning. arXiv.org, 2023. Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhnag, Wangbo Zhao, Yike Yuan, Jiaqi Wang. Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an 2023.
[57]
all-around player? arXiv:2307.0623
[58] Yuliang Liu, Zhang Li, Biao Yang. Chunyuan Li, Xucheng Yin. Cheng lin Liu, Lianwen Jin, and Xiang Bai. On the hidden mystery of ocr in large multimodal models, 2024.
[59] Yuliang Liu, Biao Yang. Qiang Liu, Zhang Li, Zhiyin Ma. Shuo Zhang, and Xiang Bai. Textmonkey: An ocr- ocr-free large multimodal model for understanding document. arXiv preprint arXiv:2403.0444 2024. [60] Ziyu Liu, Zeyi Sun, Yuhang Zang, Wei Li, Pan Zhang, Xiaoyi Dong, Yuanjun Xiong, Dahua Lin, and Jiaqi Wang. RAR: Retrieving and ranking augmented milms for visual recognition. arXiv preprint arXiv:2403.1 13805, 2024.
13

---

[61] Pan Lu, Hritik Bansal, Tony Xia. Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi. Hao Cheng. Kai- Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representations (ICLR). 2024,
[62] Pan Lu, Ran Gong. Shibiao Jiang. Liang Qiu, Siyuan Huang. Xiaodan Liang, and Song-Chun Zhu. Inter-gps: Interpretable geometry problem solving with formal language and symbalic reasoning. In The 59th Annual Meeting of the Association for Computational I Linguistics (ACL), 2021.
[63] Pan Lu. Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai- Wei Chang. Song-Chur Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-2521 2022.
[64] Pan Lu, Liang Qiu, Kai-Wei Chang. Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. arXiv preprint arXiv:2209. 14610, 2022.
[65] Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu. Iconqa: A new henchmark for abstract diagram understanding and visual language
reasoning. arXiv p arXiv:2 10. 14, 2021.
[66] Tengchao Lv, Yupan Huang, Jingye Chen, Lei Cui, Shuminng Ma, Yaoyao Chang, Shaohan Huang, Wenhui Wang. Li Dong, Weiyao Luo, Shaoxiang Wu, Guoxin Wang, Cha Zhang, and Furu Wei. Kosmos-2.5; A multimodal literate model, 2023,
[67] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring externa knowledge. In Proceedings of the IEEE/cyf conference on computer vision and pattern recognition, pages 3195-3204, 2019.
[68] Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: benchmark for question answering about charts with visual and logical reasoning. arXiv preprint 2rx: 10244, 2022.
[69] Minesh Mathew, Viraj Bagal, Ruben Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvqa. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1697-1766 1706, 2022.
[70] Minesh Mathew. Dimosthenis Karatzas, and CV Jawahar. Doevqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2209-2209,
2021.
[71] Brandon McKinzie, Zhe Gan, -can-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng. Floris Weers, et al. Mml: Methods, analysis & insights from multimodal IIm pre-training, arXiv preprint arXiv:2403.09%
2024.
[72] Anand Mishra, Shashank Shekhar, Aject Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In ICDAR, 2019.
[73]
OpenAI. Chatgpt. hattps://opent httpa////ppeaaa...oolooo//////// http/bllogcchatgp/ 2022.
[74] OpenAL. Gpt-4 techniial report, 2023.
[75]
Vicente Ordonez, Girish Kulkarni, and Tamara L., Berg. Im2text: Describing images using million
captioned photographs. In Neural Information Processing Systems (NIPS), 2011.
[76] Long Ouyang. Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems (Neur/PS), 35:27730-27744, 2022.
[77] Zhiliang Peng. Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang. Shuming Ma, and Furu Wei. Kosmos-
Grounding
B multimodal large language models to the world. arXiv.org, 2023.
[78] Owen. Introducing qwen-7h: Open foundation and human-aligned models (of the state-of-the-artss, 2023.
[79] Alec Radford, Jong Wook Kim. Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry. Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine learning (ICML), pages 8748-8763 8763. PMLR, 2021.
[80] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filterdd
400 million image-text pairs. arXiv preprint arXiv:2111.0211
2021.
[81] Dustin Schwenk. Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In European Conference on Computer Vision, pages 146-162. Springer, 2022.
[82] Sanket Shah, Anand Mishra, Naganand Yadati, and Partha Pratim Talukdar. Kvqa: Knowledge-aware visual question answering. In Proceedings of the AAAI conference on artificial intelligence, 2019. [83] Piyush Sharma, Nan Ding, Sehastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hyperymeed image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 2556-2565, 2018.
[84] Baoguang Shi, Cong Yao, Minghui Liao, Mingkun Yang, Pei Xu, Linyan Cui, Serge Belongie, Shijian Lu, and Xiang Bai. ledar2017 competition on reading chinese text in the wild (retw-17). In 2017 14th iapr international conference on document analysis and recognition (ICDAR), volume 1, pages 1429- 1434. IEEE, 2017.
14

---

[85]
Chenglei Si. Yanzhe Zhang, Zhengyuan Yang. Ruibo Liu, and Diyi Yang. Design2eode: How far are we from automating front-end engineering?. 2024.
[86] Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanprect Singh. Textcaps: a datasct for image captioning with reading comprehension. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Part 16, pages 742-758. Springer, 2020.
[87] Amanpreet Singh. Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 831 7-332, 2019.
[88] Yipeng Sun, Zihan Ni, Chee-Kheng Kheng Chng, Yuliang Liu, Canjie Luo, Chun Chet Ng, Junyu Han, Errui Ding, Jingtuo Liu, Dimosthenis Karatzas, et al. Icdar 2019 competition on large-scale street view text with partial labeling-rre-sstt. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 1557 1562. IEEE, 2019.
[89] Zeyi Sun, Ye Fang. Tong Wu, Pan Zhang, Yuhang Zang, Shu Kong, Yuanjun Xiong, Dahua Lin, and Jiaqi Wang. Alpha-CI.IP. A clip model focusing on wherever you want, arXiv preprint arXiv:2112.088 2023.
[90]
Gemini Team. Gemini: A family of highly capable multimodal models, 2023.
[91]
InterL.M Team. Internim: A multilingual language model with progressively enhanced capabilities. com/ InternLM/ InternLM 2023.
https://gitthn
[92]
Hugo Louvron, Thibaut Lavnil Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv.org. 2023.
[93] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models, 2023.
[94] Junke Wang. Lingehen Meng, Zejia Weng, Bo He, Zuxuan Wu, and Yu-Gang Jiang. To see is to helieve: Prompting gpt-4v for better visual instruction tuning. arXiv preprint 3xx::2311.0755 2023.
[95] Weihan Wang. Qingsong Lv, Wemmeng Yu, Wenyi Hong. Qi, Yan Wang. Junhui Ji. Zhuoyi Yang. Lei
Zhao, Xixuan Song. Jiazheng Xu, Bin Xu. Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. Cogylm: Visual expert for pretrained language models, 2023.
[96] Yonghui Wang, Wengang Zhou, Hao Feng, Keyi Zhou, and Houqiang Li. Towards improving document understanding: An exploration on texi-grounding via milms.. arXiv preprint arXiv:2311.131 2023. [97] Haoran Wei, Lingyu Kong. Jinyue Chen, Liang Zhao, Zheng Ge. Jinrong Yang. Jianjian Sun, Chunrui Han, and Xiangyu Zhang. Vary: Scaling up the vision vocabulary for large vision-language models. arXiv preprint arXiv:2312.061 00,00,00000000000 2023.
[98] Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang. Chunyi Li, Wenxiu Sun, Qiong Yan, Guangtao Zhai, et al. Q-bench: benchmark for general-purpose foundation models on low-level vision. arXiv preprint arXiv:2309, 14181, 2023.
[99] Ruyi Xu, Yuan Yao, Zonghao Guo, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, Mansong Sun, and Gao Huang. Llava-uhd: an Imm perceiving any aspect ratio and high-resolution
images. arXiv preprint arXiv:2403. /0t.//7032 /703, 2024.
[100] Jiabo Ye, Anwen Hu. Haiyang Xu. Qinghao Ye, Ming Yan, Yuhao Dan. Chenlin Zhao, Guohai Xu, Chenliang Li, Junfeng Tian, al. D.L.UG-DecOwl: Modularized multimodal large language model for document understanding. arXiv preprint arXiv:2307.022
2023.
[101] Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Zhang, et al. Ureader: Universal ocr-free visually-situated language understanding with multimodal large language model. arXiv preprint arXiv:2310.0512
2023.
[102] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengeheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimddaityy. arXiv.org, 2023.
[103] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaicr. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78. 2014,
[104] Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo
Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309. 12284, 2023.
[105] Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.0244 02490, 2023.
[106] Tai- Yuan, Zhe Zhu, Kun Xu, Cheng- Jun Li, Tai-Jiang Mu, and Shi-Min Hu. A large chinese text dataset in the wild. Journal of Computer Science and Technology. 34(3):509 521, 2019.
[107] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang. Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin. Boyuan Zheng. Zhenzhu Yang. Yibo Liu. Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. arXiv preprint arXiv:231 1. 16502, 2023.
[108] Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Zhiyuan
15

---

Liu, Peng Zhang, Yuxiao Dong, and Jie Tang. GLM-130b: An open bilingual pre-trained model. In The Eleventh International Conference Leurning Representations (ICLR). 2023.
[109]
Beichen Zhang. Pan Zhang, Xiaoyi Dong. Yuhang Zang, and Jiaqi Wang. Long-CLIP: Unlocking the
long-text capability of clip. arXiv preprint arXiv:2403. 15378, 2024.
[110]
Pan Zhang, Xiaoyi Dong Bin Wang. Yuhang Cao, Chao Xu, Linke Ouyang. Zhiyuan Zhao, Shuangrui Ding. Songyang Zhang, Haodong Duan, Hang Yan,
vision- lssionllaguaag large model for advanced text-image comprehension and composition. arXiv preprint arXiv:2309.15 1511122.9.111...... 2023.
Internlm peeenll-xcomposer
[111] Rui Zhang, Yongsheng Zhou, Qianyi Jiang, Qi Song, Nan Li, Kai Zhou, Lei Wang, Dong Wang, Minghui Liao, Mingkun Yang, et al. Icdar 2019 robust reading challenge on reading chinese text on signboard. In
2019 international conference on document analyxis and recognition (ICDAR), pages 1577-1581. IECE, 2019.
[112] Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. LLaVAR: Enhanced visual instruction tuning for text-rich image understanding. arXiv preprint a3Xiv:2306. 17107, 2023.
[113] Deyao Zhu, Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models. arXiv.org, 2023.
16

---

A More examples chat with our IntenLN--XCompose244KHH
Give an in-depth description of the visual.
Public Heath England
The image is infographic from Public Health England that explains the concept of contact tracing in the context of preventing the spread of coronavinus (COVID-19). Contact tracing is fundamental part of outbreak control used by public health professionals around the world to prevent the spread of infections.
WHAT contact tracing? ?
Contact tracing fundammental outtreak control 's professionals to prevent the spread of infections
for coronavirus [COWID 10)
The infographic is divided into several sections, each providing different aspect the contact tracing process. The first section explains that a contact is assessed to see how close they confirmed case, which could include someone living with the case, someone who has been in direct contact with them their bodily fluids. Following this assessment, the contacts categorised into low high risk and provided with advice on w hat they should do.
idonely
sepene how best clese contatt
Following this scasuam we can cotngerin provido odvice on what mney oiooudt
New risk
14
ligher rbalk infictionn
direarth puff tociuation
ends inakens, renoiinng
ittuing away from voors.
The second section explains that contact is at high risk of infection, they may asked self-isolate, remaining in their home for 14 days and staying away from work, school public places. If contact is low risk, they don't require self- ooff-isolation.
contlact
We contact then daily on Dhay
inicht rears person loving maci
the olean
penson done
theee prodd thoh wilh The roed cononerics
The third soction explains that if the person develops symptoms, they would be tested and provided with specialist care if they have the novel coronavirus (COVID-19).
Recovery timto
Finally, the infographic emphasizes the importance of recovery time, as contact displays symptoms of coronavinus (COVID-19) the patient makes a full recovery, they can back daily lives as normal and pose no risk to others.
of coronantrua (C0VD--1
they can baick tu dely lile
Image size; 986 2560
Figure 5: Chat with InternI. Internl---CComposer--eee
17

---

Detect all the wond in given image
VENGAGE
Develop Healthy Eating Habits
Herrl VemGAAGE.
dils
How Develop Healthy Habits. And Reduce Risk Heart Disease.
Your best defense is healthy lifestyle filled with physical activity and nutritious meals, cutting avoiding drisking smoking.
Lil
The Current Trend
The Cornen Trrnd
75% of the US population low in its thuit and vegetables consumption, 1/10 the recommendation young adulls in general cat the fewest fruits and vegetables compared to any other group.
vegetable consumption. Men
Steps Eating Healthy 1. produce of various colors your diet divensfy your produce consumption. 2. and fruils your moil, and throughout the day. 3. Take sieps to improve acciss vegetables
incosporase more veggies
1. Adding Color Your Diet
Eating 2% cups (4 to servings) dhill of fruits and veguiables may reluce and srake
your risk for candiovseuuler disense, including
Adding Color Your
Examples of single serving of fruit: medium apple, small banana, I avocado, large strawberriss Examples single serving vegetables: 5- broczali floreta, 2/3 large sweet potato, 6 buhy curroty, cup of spinach.
Adding variery of produce to your dier has number of health benefiss, helps your Manage your weight, digestion, Redure the risk some cancers, such colon cancer, Reduce the risk chronic health problems, such diabetes.
your pressure, Support healthy
2. Use Healthy Eating Hacks
1. Fresh, frueon, camned and dried all count toward your daily servings, plenty of vegetables. Choose whole fruits and vegetables. 4. Try to eat a variety vegetables, instead of same thing all the time. S. Look for fruit packed its own fhuit juice 100% vegetable juice, low'nu sodium options for canned vegetables, and 100% vegetable juice.
Always
your plate fruits
Bating
addod sugar
3. Take Action for Better Access
many communities, available in comer stores, convenience stures, bodegas and gas stations.
vegetables are limited limited/evpensivv expenssive only
Lack of access can profoundly impact diet. Here's how: Contact your city and state leaders let them know what food access
community.
Find ways to spread the word about nutrition assistance program, such SNAP, WIC school meals.
Create petition for signage ppiccement.
improve affordability and advocatic for better
an afler-schuul daycare program expresemtatvne more fruits and vegetables for snacks. Organize letter-writing campaign and up meeting with state leaders.
For example, ask for funding to host farmers market in an underserved communityy. Sign up "Yoe're the Cure" Congresaperson advocuting for healthier meals at school.
your
American Sbart Miaadh
American Heat Mosth! Share infographic with your fimily, friends and network today. You can carry out all of these actions or just a few. But whichever actions you choose, they pave way greater food for your communiry.
The HD image (816 3813)
Figure 6: Chat with InternI Ditern/MXCComoor2244100
18

---

Plysc
Suppott
Enghnering Raciitée France Sale
dept: tnalysis
The line chart presents the openation cost by deportment US dollrr for catrgorie from May 2018 Mar 2013. The cstegoris include Humon
(HR)) Information Techusology Payroll, Support, Enginesting, Facilitiess Finence, Sales, Each oxegorr regeeeeria differett color and line on grapi. be y aads regresents the oast troosands of US dellams while the s-arts regroemtts the months from May 2018 to Mar 2018, Te grah shoves Euctustions the cost for forwach oxisgory Shruugout the months, with some categotiss experienzing pesks and trougts different times.
W Wcch depoctment has the highest cast most cases?
May Months
Jan 2019
Mar
Payzell has the lughest cost most cases, as inchcated by the purple line the gragh.
The HD image (1475 x 675)
anzlyne this flow chort by step
Ordering A Book From amazon
This flow chart outlines the process of oedering book from Amazon. Heces' srp-hy-sstop anilyss
1. process begtns with
inmaainng
2. Enter webeite from the browsen The user not igates the Aaozon website.
3. Registered The system checks if the user slready zegistered, If nol, they am prompped to rigaserr
Sign in the user elready registeend,
promppeed to sign
desined page
On
for book your choice The soorthes for
your preferne page sourth resuls..
user selects preferrod
7. Buy now?: The system adks if the user wants to buy a new book. If the asswerts "V the process continues to step
B. the anewer is "No" the proces continues step 9.8. Selectia book:
The user selects bock.
Select old hook: nser sclects book
book:
cart.
order The user reviews thiir oeterr.
Change infomation: The user las the option to change their information.
Iaey chouse not to. the process coalinues step 13. If they chouse to. Lse pmicess retums to shop 1t.
13. Enter shipping information: The user enters their shipping information
Selort payment mathod: The selects their preforred payment method. order The process
Place
The HD image (1045x1442) 1428)
with the orter being placed
hat the algonthm of thas flow thart?
The flow chart the image is an alponthm Soc finding the muximum elenent in on army.. Here is detailed explatation of the elgurthm:
Declaro isteger amy having size rumbers
anmy. Initialize max firs elenent each olement if the current clement is greater than max, set max the curent element
amay, stomag from
the and of the loop, mox will hold the mocimum elament in the ammy.
6.
Outpul the matimum element.
This slgocithm iterates thrugh the companing each element the current If langer element element
At and long, max, then outpomod
The HD image (1888 x 3172)
Figure 7: Chat with InternLAAXComposer2--tI
19

---

B Experiment Details
B.1 Data sampling strategy.
We maintain different dataloaders for image-text data and pure text data, sampling them in weighted manner. The pure language dataloader is sampled with fixed weight of 0.1 and the image-text dataloader is sampled with weight of 0.9.
Within each dataloader, the training data comes from multiple sources, and sample them in weighted manner. In detail, for K datasets and the data number the ken dataset 7g.- its weight wg min(100, na/ /1000) and the normalized weight Wa wx/ Σ wa- For each gel_ dala operation within the dataloader, we use weighted sampling to choose a dataset and randomly choose training sample from it.
C Broader Impacts
On the positive side, our model introduces novel solution that significantly enhances the ability of large language models to comprehend high-resolution images. This innovative approach is expected to be highly beneficial for the research community. It paves the way for new explorations and discoveries in the field of MILM and image processing. The potential applications of this model are vast.
Moreover, have plans make our model open-source. By sharing our work with the public, aim to foster an environment of shared learning and progress. Users, researchers, and developers can utilize our model, adapt it to their needs, and even contribute to its further development. This open- open-source approach will accelerate the pace of innovation and bring about more rapid advancements in the field.
On the negative side, like any powerful tool, there is potential for misuse of our model. There is risk that individuals with malicious intent may exploit the capabilities of the model for unethical or harmful purposes. This is a challenge that we, as researchers, must acknowledge and address. However, it is also crucial for ethically. We believe that through collective effort, we can mitigate these risks and ensure that the benefits of our model are realized while minimizing potential harm.
and the wider community use our model responsibly and
D Limitation
In this paper, we study the influence of training resolution in wide range, from HD-9 to 4KHD (HD-25 HD-55). Our results show that high-resolution OCR-related tasks rely on the training resolution heavily, and get significant performance gains with increased resolution. Till our largest setting 4KHD, its gain is not saturated. If we keep increasing the resolution, it may get better results. However, due to computational constraints, failed to fully explore the potential improvements from further increasing training resolution, and we have to leave it as future work.
20

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We write the paper carefully and confirm that the main claim in the abstract and introduction accurately reflects the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
21

---

Justification: We mentioned the limitation of not fully exploring the upper bound of im- provement by increasing training resolution, due to the computational constraints, and we would leave it to future work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:
[NA]
Justification:
We do not have theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results. the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The training details and datasets are provided in the paper, we also plan to open-source model.
22

---

Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel urchitecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification:
We plan to open-source the code and data after the final decision of the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
hhipss.c//ublic/uuddds wttpeese/oooeSeeeeooooeoeoe
www.obiiiiiiioooooooiiooo for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
23

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We have provided the details in Sec.3
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
is computationally expensive to report the error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have provided it in Sec 3.4
Guidelines:
The answer NA means that the paper does not include experiments.

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https::/ httpp//p/ppp/ttticccccciieeeeeee
Answer: [Yes]
Justification: Our research conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed it in the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
If the authors
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
25

---

Our work focus on high-resolution image understanding and does not has such risks.
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
All the assets are properly cited in the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwwpssppppssseeeeeeeeeeeee thcode wwt.coo///aaaa comddatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide a new model with the capability to understand high-resolution images, and we have provided the details of the model in the paper.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
26

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "C21y5kdvOP", "reviewer": "Reviewer_Lqsg", "summary": "This paper aims to explore the high resolution scenes of multimodal large language models. The authors find that the performances are largely improved when the model is equipped with 4K resolution. This finding will greatly inspire subsequent research works, which is of great value to the research community. The experimental results in the popular multimodal datasets, such as MME and MathVista, are impressive. The proposed dynamic resolution and automatic patch configuration will become a common approach, due to its simplicity and effectiveness.", "questions": "None", "limitations": "None", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The motivation sounds good. High resolution is the inevitable development direction of multimodal large language models, and we urgently need to know what effects high resolution will bring to model training and performance. This paper stands on a high ground.\n\n2. The method looks simple yet effective. Dynamic resolution and automatic patch configuration could not only maintain the aspect ratio of the image, but also indirectly enlarge the resolution of the image. \n\n3. The experimental results on many datasets are impressive. The proposed method achieves SOTA results in 6 of the 16 benchmarks with only 8B parameters.\n\n4. The writing of this paper is good, and the structure is easy to follow.", "weaknesses": "1. Considering that high resolution is a very meaningful direction, it is suggested that the authors provide more details of the training to help other researchers, such as the loss figure of the training. Because training at high resolution is expensive, releasing more intermediate results from training will help researchers conduct further studies.\n\n2. The current method seems to largely increase the number of visual tokens while increasing the resolution. One question is whether it is reasonable to use so such many visual tokens for ONE image. I would love to hear the authors' thoughts on this point. This is an open question.\n\n3. It is suggested that the authors could enlarge the LLM parameters synchronously in subsequent studies if conditions permit, which will make the study more complete. It seems that image resolution and LLM parameters are the two most important aspects affecting model performance, in addition to the training data of course.\n\n4. I am curious about the generalization performance of the proposed model on multimodal video evaluation dataset, such as Video-MME [1], which applies to both image MLLMs, i.e., generalizing to multiple images, and video MLLMs. The sub-figures in the split-patch strategy has some similarities with video frames. It would be great if the authors could update the results of the Video-MME into the paper.\n\n[1] Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis. Arxiv, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1: More training details\n\nA1: Thanks for your valuable comments, here we show the training loss in the Pretrain/SFT stage. Please refer to Figure 1 of the rebuttal pdf.\n\n### Q2: Is it reasonable to use so such many visual tokens for ONE image?\n\nA2:\nThis is a great and interesting question, and we address the reasonableness of using more tokens for HD images by considering the following:\n1. **Can an LLM understand an image with so many tokens?** Our paper reveals that the LVLM model can effectively leverage a large number of image tokens. It captures image details from thousands of image tokens, leading to significant gains on various high-resolution benchmarks.\n2. **Is an image really worth thousands of tokens?** The aim of our paper is to develop a universal solution for LVLMs to analyze images with varying resolutions, ranging from 336 pixels to 4K resolution. Further token compression may result in unavoidable information loss and impact our study. As discussed in Figure 4 of the main paper, we observed that more visual tokens lead to consistent performance gains on benchmarks requiring fine-detailed understanding, such as DocVQA and InfographicVQA. This implies the necessity of scaling up the image resolution for a single image. We leave the task of increasing image resolution with efficient image token numbers as a future research direction.\n\n### Q3: The influence of  LLM parameters \n\nA3: Thanks for your valuable idea! Due to the rebuttal period limitation, there is no sufficient time for us to train a 20B LVLM. Instead, we conduct the experiment on a smaller model InternLM2-1.8B and we found a clear performance improvement with LLM parameter increasing. We believe if we keep increasing the LLM size, the model will get better performance.\n\n| Model    | LLM Size | DocVQA  | InfoVQA | TextVQA | ChartQA | MMBench | MME     | SEED  |\n|----------|------|---------|---------|---------|---------|---------|---------|-------|\n| IXC-4KHD | 1.8B | 81.7    | 54.9    | 70.1    | 72.4    | 70.6    | 1,973.0 | 70.5  |\n| IXC-4KHD | 7B   | 90.0    | 68.6    | 77.2    | 81.0    | 80.2    | 2,204.9 | 74.7  |\n\n### Q4: Performance on Video Benchmarks\n\nA4: Thanks for your valuable comments, we tried to concatenate the video frames into a long image and finetuned it with the SFT data used in Video-ChatGPT [1]. Here we report the results on three widely used video benchmarks. As shown below, our model performs on par with GPT-4V on the challenging MME-Video, and significantly better than GPT-4V on the MVBench and MLVU.\n\n| Model    | MME-Video w/o subtitle | MVBench | MLVU  |\n|----------|------------------------|---------|-------|\n| GPT-4V   | 56.0                   | 43.5    | 49.2  |\n| IXC-4KHD | 54.2                   | 56.7    | 65.1  |\n\n[1] Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3UrB7QPbNC", "reviewer": "Reviewer_kpFC", "summary": "The paper presents InternLM-XComposer2-4KHD, a high-resolution MLLM that performs better than existing methods on various benchmarks.", "questions": "My main concern is the novelty and the experimental setting. The main idea has been widely explored in previous works.", "limitations": "See weakness.", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well written and easy to understand.\n2. The performance of  InternLM-XComposer2-4KHD is good, which is valided on various benchmarks.", "weaknesses": "1. The paper has limited novelty. The high-resolution solution is not new. Starting from Monkey, there are a lot of works that adopt the similar idea, e.g., LLaVA-NeXT and InternVL-1.5.  Compared to these methods, the technical contribution of the paper is trivial. \n\n\n2. From the experimental results, I cannot tell whether the proposed high-resolution solution is better than other solutions. Since the setting of the paper are not comparable to others,  it seems that the data plays a more significant role. Maybe more comparisons or ablations should be provided.\n\n\n3. There still be a lot of MLLMs that have similar settings to  IXC2-4KHD, e.g., InternVL-1.5.  Why not compare with these models in Tab 3-4.\n\n4. The implementation details are not clear. It's still confusing for me that how much data is used for training.   In Tab 3, the previsous SoTA LLaVA-NeXT uses less than 2M data. So It's not clear whether the comparison is fair.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1: Limited novelty...\n\nA1: Thanks, we would like to highlight the novelty of our work in the following aspects:\n1. As discussed in the related work (Lines 95-101), our approach is the first to address the challenges and propose solutions for handling variability in image feature patch layouts, enabling effective training with dynamic high-resolution images.\n2. Our method introduces a dynamic resolution and automatic path configuration strategy, allowing for seamless scaling from 336 pixels to 4K resolution - a range that exceeds previous approaches. In contrast, existing methods for high-resolution understanding are limited to either predefined high-resolution settings or a narrow range of resolutions. For instance, Monkey and LLaVA-Next employ sliding windows and image splitting, respectively, but are restricted to segmenting images into a maximum of 6 and 4 patches, respectively. This limitation hinders their ability to handle images with extreme aspect ratios, commonly found in infographics. Our IXC2-4KHD, on the other hand, supports up to 55 local image patches, making it capable of tackling these challenging cases.\n3. Our work also presents a thorough analysis of high-resolution strategy design, examining the impact of global image, 'newline' token, and training/inference resolution - aspects that were underexplored in previous studies. Through extensive experimentation, we have gleaned valuable insights, including:\n    - The LVLM struggles to comprehend the 2D structure of a simply flattened image, but incorporating a global image and 'newline' tokens could solve this problem with clear performance improvement.\n    - For tasks requiring high-resolution images, our model demonstrates improved performance with larger images, showing no signs of saturation even at 4KHD resolution. This suggests potential for further scaling and improved results.\n    - Conversely, for tasks insensitive to resolution, model performance plateaus as image size increases.\n\n4. Finally, we note that InternVL-1.5 is a concurrent, unpublished work that has not undergone peer review or been accepted by any reputable conference or journal. Similarly, LLaVA-Next is a GitHub blog post, rather than a rigorously evaluated academic publication.\n\n### Q2: Experimental... More comparisons or ablations.\n\nA2: Thanks for your valuable advice. For a fair comparison, we trained a new model with the identical architecture, training strategy, and dataset as our original model, but with one key modification: we adopted the high-resolution strategy from LLaVA-Next. We name it as IXC-LLaVA and compare it with 1) LLaVA-Next 0530 official results and 2) IXC2-4KHD under HD-9/16 setting. The results below demonstrate that IXC-LLaVA achieves promising performance across all six benchmarks, leveraging the benefits of additional training data and advanced IXC architecture design. However, it is outperformed by IXC2-4KHD HD-9, which utilizes fewer image tokens yet yields better results. This fair comparison underscores the efficiency and effectiveness of our proposed high-resolution strategy, highlighting its advantages over the LLaVA-Next approach.\n Model Nmae         | HD Strategy | Image Tokens | Max Resolution | DocVQA  | TextVQA | ChartQA | MMBench \n-|-|-|-|-|-|-|-\n LLaVA-Next(LLaMA3) | LLaVA-Next  | 2880         | 672x672        | 78.2    | 52.0    | 69.5    | 72.1    \n IXC-LLaVA          | LLaVA-Next  | 2880         | 672x672        | 78.9    | 71.5    | 74.1    | 77.6    \n IXC-4KHD           | HD9         | 1440         | 1008x1008      | 79.4    | 73.8    | 78.2    | 79.5    \n IXC-4KHD           | HD16        | 2448         | 1344x1344      | 84.9    | 75.7    | 80.1    | 80.2    \n\n### Q3: Compare with other MLLMs, e.g., InternVL-1.5.\n\nA3: \nThank you for your comments. \n1. As discussed in A1, InternVL-1.5 is not a peer-reviewed publication that has not been accepted by any conference or journal yet. Moreover, it is a concurrent work that is posted on Arxiv within one month to the NeurIPS submission deadline. Therefore, we do not feel obligated to compare our work with it at this time. \n2. In Table 3, we have already reported the performance of InternVL-1.2, which was a previously released version.\n3. In Table 4, as mentioned in line 214, we conducted a comprehensive comparison with open-source LVLMs of similar scale. However, InternVL-1.5 has a significantly larger model size (26B parameters) compared to our model (8B parameters), making a direct comparison unfair and meaningless.\n\n### Q4: The implementation details are not clear...\n\nA4: Sorry for the confusion. We list the detailed data usage in the following. We use almost 3M data for the pre-training and 1.2M data for the SFT stage.\n\n| General Semantic Alignment    | ShareGPT4V-PT (1.2M), COCO(50K), Nocaps(20K), TextCaps (50K), SBU(200K)，LAION400M (1.2M), CC 3M  (0.3M) |\n|-|-|\n| World Knowledge Alignment     | Concept Data (500K)                                                                                     |\n| Vision Capability Enhancement | WanJuan (400K), Flicker(30K), MMC-Inst(100K), RCTW-17(8K), CTW(2K), LSVT(40K), ReCTs (20K), ArT (6K)    |\n\n| Caption| ShareGPT4V(100K), COCO (50K), Nocaps(5K)|   |   |   |   |   |   |\n|-|---|---|---|---|---|---|---|\n| General QA| VQAv2(100K), GQA (100K), OK-VQA(10K), VD(100K), RD(2K), VSR(5K)|   |   |   |   |   |   |\n| Science QA| AI2D(15K), SQA(10K), TQA(6K), IconQA(10K)|   |   |   |   |   |   |\n| Chart QA| DVQA(30K), ChartQA(7K), ChartQA-AUG(21K)|   |   |   |   |   |   |\n| Math QA| MathQA (25K), Geometry3K (2K), TabMWP (20K), CLEVR-MATH (10K)/Super (10K)|   |   |   |   |   |   |\n| World Knowledge QA | A-OKVQA (17K),KVQA (5K), ViQuAE (3K)|   |   |   |   |   |   |\n| OCR QA| TextVQA (20K), OCR-VQA (10K), ST-VQA (26K)|   |   |   |   |   |   |\n| HD-OCR QA| InfoVQA (24K), DocVQA (40K)|   |   |   |   |   |   |\n| Conversation| LLaVA-150k (100K), LVIS-Instruct4V (100K), ShareGPT-en&zh + InternLM-Chat (100K) |   |   |   |   |   |   |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zupzLCHGX5", "reviewer": "Reviewer_339k", "summary": "This paper introduces a large vision-language model named InternLM-XComposer2-4KHD, which is designed to process images up to 4K HD resolution. The model incorporates a dynamic resolution adaptation mechanism that maintains the aspect ratios of input images, thus enhancing the model's ability to understand complex image-text interactions. This approach allows for scalable resolution adjustments during training, supporting a wide range of resolutions. The model even outperforms well-known GPT-4V and Gemini Pro in 10 out of 16 benchmarks.", "questions": "1. The substantial computational resources required for training at 4K resolutions and the lack of detailed analysis on inference efficiency are main concerns.\n2. The paper needs clearer problem statements and motivational examples.\n3. The paper demonstrates enhancements in traditional OCR capabilities but leaves some uncertainty regarding the model's ability to process more complex visual content. If the authors can offer further insights or evidence, it would significantly strengthen the paper’s impact and might merit a higher evaluation.", "limitations": "The paper addresses its limitations and potential negative societal impacts in the appendices. The explanation of limitations lacks depth. For more detailed concerns, please refer to the questions section.", "rating": 5, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. This paper introduces a substantial enhancement in the field of large vision-language models. It demonstrates significant performance improvements across multiple benchmarks, outperforming well-known models such as GPT-4V and Gemini Pro, thereby potentially exerting a broad impact on the field.\n\n2. The dynamic resolution design adopted in this study is particularly noteworthy. It enables the model to adapt to a variety of image resolutions while preserving critical detail.\n\n3. The writing in the paper is commendable, with informative charts and examples that effectively showcase the model's capabilities.", "weaknesses": "1. The paper frequently mentions training with 4K resolution, yet it lacks optimizations for accelerating high-resolution training. The heavy reliance on substantial computational resources limits the broader applicability of the proposed methods.\n2. There is a lack of analysis regarding the inference efficiency of the model. While the paper validates that high-resolution training significantly enhances performance for text-based vision-language tasks, it does not adequately address the computational costs incurred by high-resolution inputs. This omission makes it challenging to assess whether the additional computational expenses are justified by the performance gains.\n3. The introduction repetitively emphasizes adaptability to 4K resolution without providing clear problem statements or motivational examples. For instance, the discussion on \"variability in patch layouts\" (Line 52) lacks an intuitive explanation, and it would benefit from specific examples illustrating this variability. Besides, the paper highlights at Line 56 a critical aspect concerning the model's performance at resolutions beyond 4K with 'even when...'. It remains unclear whether this suggests that traditional methods trained at 4K resolution might fail to gain additional performance at lower resolutions. Clarifying this ambiguity is crucial for understanding the challenges the study addresses, reducing the need for readers to infer details solely from experimental results.\n4. This work notably advances vision-language understanding by enhancing traditional OCR capabilities. However, it prompts inquiries about the model's proficiency with complex visual content. The examples presented do not conclusively demonstrate the model's ability to accurately interpret intricate visuals such as hand-written mathematical formulas, circuit diagrams, or chemical structures. These types of content demand detailed parsing for complex problem-solving, an aspect not fully illustrated by the simpler cases in the appendices. While these considerations may exceed the current study's scope, they do not impact the overall evaluation for acceptance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1: The paper frequently...\nA1: Thanks for your valuable comments. \n1. Enabling LVLM to understand high-resolution images remains a challenging and open problem in the field. The primary goal of our paper is to explore a general and effective strategy for high-resolution image understanding LVLMs, and further study how the image resolution influences LVLM performance.\n2. Efficiency is an important under-explored topic, but in most cases, there is a trade-off between performance and efficiency. In this work, we focus on pushing the performance upper bound without introducing additional variables, and we would leave the study on efficiency in future work.\n3. Our proposed weighted multi-dataloader sampling strategy (Please see Appendix B.1) is designed to optimize the training process. By enabling each GPU to process batches with either high-resolution images only or low-resolution images only, we reduce the need for padding tokens to handle unbalanced sample lengths caused by varying image sizes. This approach speeds up the training process, making it more efficient and scalable\n\n### Q2: lack of analysis regarding the inference efficiency...\n\nA2: The model inference process consists of two stages: encoding the prefix (model input) and autoregressively decoding new tokens (model output). Correspondingly, the inference efficiency considers two parts: time to encode the prefix and speed to decode each token. \nHere we report the prefix encoding time and per-token decoding speed under different HD settings. We test the speed with a Nvidia-A100 80G.\nWith the results below, we have three observations\n1. Prefix encoding time increases linearly with the number of prefix tokens.\n2. Decoding speed remains relatively constant, regardless of prefix length, thanks to optimizations on transformers from research communities and companies, including kv-cache and flash-attention.\n3. When generating 2048 tokens, total inference time usage is nearly identical across HD9 to HD55, as encoding time is much smaller.\n\nBased on the above analysis, we believe the inference efficiency of our model is acceptable\n\nWe believe some targeted designs can further improve efficiency, but as we discussed in A1, our paper focuses on enabling LVLM to understand high-resolution images with a general and effective solution, and we would leave the efficiency exploration in future work.\n HD | image tokens | prefix encoding time | per-token decoding speed | time to generation 2048 new tokens  \n-|-|-|-|-\n HD9  | 1440| 0.2845| 0.0982| 201.4\n HD16 | 2448| 0.3966| 0.0983| 201.7\n HD25 | 3744| 0.5513| 0.0981| 201.5\n HD55 | 8064| 1.1272| 0.0983| 202.4 \n\n### Q3: The introduction repetitively emphasizes......\nA3: Sorry for the confusion. \n1. Regarding the variability in patch layouts, we illustrate this challenge with Fig.2 in the rebuttal PDF. Images a and b are distinct, yet when flattened into a one-dimensional input sequence, they become identical inputs (Fig. 2(c)) for the LLM. In practice, images also exhibit diverse aspect ratios, which can lead to confusion in object positioning. To address this, it is crucial for LVLM to accurately comprehend the 2D structure. We propose two core designs to achieve this: Global-Local Format and Patch Layout Indicator. The global image provides a concise overview, while the indicator precisely conveys the 2D structure to the model.\n\n2. For 'Even when' in Line 56, this is an inference drawn from observations in Fig. 4 (model performance is not saturated even with HD55) and Table 6 (model performance improves with higher inference resolution) in our paper. We provide quantitative results on a challenging subset of InfoVQA to support this. Specifically, we selected images with resolutions exceeding 1680x3696 (HD55) pixels from the InfoVQA validation set, resulting in a 15.6% subset. We evaluated IXC2-4KHD's performance under HD55 and HD65 settings. The results, presented below, demonstrate that even though the model is trained with the 4KHD setting, it still benefits from larger inference resolutions when applied to images larger than the training cases.\n InfoVQA HD55 subset | 55    | 65 \n-|-|-\n HD55| 70.41 | 71.64 \n\n3. For the 'methods trained at 4K resolution might fail to gain additional performance at lower resolutions,' problem, we observe that when comparing MME (at most 512x512) and MMBench (at most 512x512) results between HD9 and HD25 in Fig. 4, the performance is nearly identical. This suggests that model performance saturates when the training resolution exceeds the image's original resolution, and increasing the training resolution yields no further gain. \nIn contrast, when the training resolution is smaller than the image's original resolution, model performance continues to improve with increasing training resolution, as seen in DocVQA (at most 5367×7184) and InfoVQA (at most 2921×24978) in Fig. 4. \nThese observations underscore the necessity of our goal: exploring a general and effective strategy to enable LVLM to understand images across a broad range of resolutions, from 336 pixels to 4KHD.\n### Q4: It prompts inquiries.... \n\nA4: We appreciate your insightful suggestion and have investigated this issue using the college-level benchmark MMMU. Specifically, we curated a subset of high-resolution images (with resolutions exceeding 1000x1000 pixels) from the MMMU validation set, which accounts for approximately 10.5% of the dataset. We then compared the performance of IXC2 and IXC2-4KHD on this subset. \nThe results demonstrate a notable improvement in performance by IXC2-4KHD, indicating its capability to not only handle dense OCR tasks on large images but also tackle complex, college-level tasks that require high-resolution input. \nAdditionally, we have included two exemplary cases in the rebuttal PDF Fig.3, showcasing our model's ability to solve intricate reasoning problems based on the visual information provided.\nModel |  MMMU-val HD subset\n-|-\n IXC2      | 46.32\n IXC2-4KHD | 52.63", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank the efforts of all the reviewers and the AC. We are encouraged by the acknowledgment of our strong performance by  Reviewer Lqsg, Reviewer kpFC, and Reviewer 339k, and the noteworthy and effective model design by Reviewer Lqsg and Reviewer 339k. We answered all the questions in the rebuttal and attached the necessary images in the pdf. We hope our rebuttal could solve the Reviewers' concerns.

Meta-review: This work proposes a novel large vision-language model capable of handling a dynamic range of resolutions from 336 to 4K pixels. To achieve this, it combines various existing design ideas, including the use of global and local patches, learned new line tokens and token merging along with new ideas for dynamic layout prediction. The proposed method is shown to outperform the existing open source methods in various VLM tasks requiring inference with high resolution images, while maintaining accuracy on general purpose VLM tasks.

Reviewers raised concerns about the novelty of the proposed ideas used to construct the method; the lack of comparisons to several recent SOTA approaches and the inference and training speed of the method. Several of the reviewers' concerns were addressed in the rebuttal, while requested comparisons are to recent and unpublished works. While the AC agrees that several of the ideas used to build the proposed method are known from prior works, they feel that the current work does employ them in a novel manner to achieve scalable and scalable inference at various image resolutions. In terms of quality this work presents a significant step forward for open source models. Since the authors have promised to release the data and code as well, it will a valuable resource for the research community to advance open source models further along the dimension of processing high resolution images and will form a strong foundation for the community to build upon. How to process higher resolution images in LVLM is a significant and important unsolved problem. Hence, all things considered the AC leans towards accepting this work and recommends that the authors incorporate the changes that they have promised in the rebuttal into their final manuscript.
Decision: Accept (poster)