Title: InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD
Venue: NeurIPS 2024
Accepted: yes
Keywords: Large Vision Language Model (LVLM)
Abstract: The Large Vision-Language Model (LVLM) field has seen significant advancements, yet its progression has been hindered by challenges in comprehending fine-grained visual content due to limited resolution. Recent efforts have aimed to enhance the high-resolution understanding capabilities of LVLMs, yet they remain capped at approximately 1500 $\times$ 1500 pixels and constrained to a relatively narrow resolution range. This paper represents InternLM-XComposer2-4KHD, a groundbreaking exploration into elevating LVLM resolution capabilities up to 4K HD (3840 × 1600) and beyond. Concurrently, considering the ultra-high resolution may not be necessary in all scenarios, it supports a wide range of diverse resolutions from 336 pixels to 4K standard, significantly broadening its scope of applicability. Specifically, this research advances the patch division paradigm by introducing a novel extension: dynamic resolution with automatic patch configuration. It maintains the training image aspect ratios while automatically varying patch counts and configuring layouts based on a pre-trained Vision Transformer (ViT) (336 $\times$ 336), leading to dynamic training resolution from 336 pixels to 4K standard. Our research demonstrates that scaling training resolution up to 4K HD leads to consistent performance enhancements without hitting the ceiling of potential improvements. InternLM-XComposer2-4KHD shows superb capability that matches or even surpasses GPT-4V and Gemini Pro in 10 of the 16 benchmarks.

InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD

Abstract

The Large Vision-Language Model (LVLM) field has seen significant advancements, yet its progression has been hindered by challenges in comprehending fine-grained visual content due to limited resolution. Recent efforts have aimed to enhance the high-resolution understanding capabilities of LVLMs, yet they remain capped at approximately 1500 × 1500 pixels and constrained to a relatively narrow resolution range. This paper represents InternLM-XComposer2-4KHD, a groundbreaking exploration into elevating LVLM resolution capabilities up to 4K HD (3840 × 1600) and beyond. Concurrently, considering the ultra-high resolution may not be necessary in all scenarios, it supports a wide range of diverse resolutions from 336 pixels to 4K standard, significantly broadening its scope of applicability. Specifically, this research advances the patch division paradigm by introducing a novel extension: dynamic resolution with automatic patch configuration. It maintains the training image aspect ratios while automatically varying patch counts and configuring layouts based on a pre-trained Vision Transformer (ViT) (336 × 336), leading to dynamic training resolution from 336 pixels to 4K standard. Our research demonstrates that scaling training resolution up to 4K HD leads to consistent performance enhancements without hitting the ceiling of potential improvements. InternLM-XComposer2-4KHD shows superb capability that matches or even surpasses GPT-4V and Gemini Pro in 10 of the 16 benchmarks. The InternLM-XComposer2-4KHD model series with 7B parameters are publicly available at https://github.com/InternLM/InternLM-XComposer .

1 Introduction

In recent years, the progress in Large Language Models (LLMs) (73; 92; 93; 39; 91; 10; 78; 29; 21) has provoked the development of Large Vision-Language Models (LVLMs). These models have demonstrated proficiency in tasks such as image captioning (17; 14) and visual-question-answering (VQA) (57; 31; 33; 107). Nevertheless, due to their limited resolution, they struggle with processing images containing fine details, such as charts (68), tables (87), documents (70), and infographics (69). This limitation constrains their practical applicability in real-world scenarios.

Recent advancements have aimed at enhancing the resolution of Large Vision-Language Models (LVLMs). Some approaches (66; 36; 97; 48) involve adapting high-resolution vision encoders

,

GPT-4V

Gemini-Pro

IXC2-4KHD

Previous open-source SOTA

DocVQA

InfoVQA

MMStar

MMBench

MMBenchCN

SEEDI

QBench T

MM-Vet

HallB

81.0

90.0

68.6

2204.9

71.8

40.9

39.9

80.2

74.7

80.9

77.2

MME

MathVista

ChartQA

AI2D

MMMU

TextVQA

OCRBench

57.8

54.9

54.1

77.7

67.5

Benchmark

Min

Median

InfoVQA

355

2

1444

2

8541

2

DocVQA

246

2

1948

2

8243

2

OCRBench

20

2

408

2

5899

2

HallBench

147

2

724

2

4898

2

Qbench

T

312

2

873

2

4750

2

MathVista

58

2

444

2

4275

2

TextVQA

512

2

883

2

3990

2

MMMU

46

2

414

2

3166

2

MMStar

86

2

418

2

2598

2

AI2D

152

2

480

2

1500

2

ChartQA

245

2

668

2

1199

2

SEED

I

362

2

432

2

512

2

MMVet

181

2

422

2

512

2

MME

114

2

420

2

512

2

MMBench

CN

68

2

417

2

512

2

MMBench

68

2

417

2

512

2

directly. However, the Vision Transformer (ViT) architecture falls short when dealing with images of varying resolutions and aspect ratios, thereby restricting its ability to handle diverse inputs effectively. Alternatively, some methods (50; 59; 37; 51; 99; 55; 46) maintain the vision encoder's resolution, segmenting high-resolution images into multiple low-resolution patches. Yet, these methods are constrained by an inadequate resolution, typically around 1500 × 1500, which does not satisfy the demands of daily content, e.g. , website screenshots (85), document pages (70), and blueprints (69). Furthermore, they are confined to either a few predefined high-resolution settings (36; 97; 48; 50; 51; 55; 46; 66; 59) or a limited range of resolutions (101; 37; 99), thereby restricting their utility across a variety of applications.

In this work, we introduce InternLM-XComposer2-4KHD, a pioneering model that for the first time expands the resolution capabilities of Large Vision-Language Models (LVLMs) to 4K HD and even higher, thereby setting a new standard in high-resolution vision-language understanding. Designed to handle a broad range of resolutions, InternLM-XComposer2-4KHD supports images with any aspect ratio from 336 pixels up to 4K HD, facilitating its deployment in real-world contexts.

InternLM-XComposer2-4KHD follows patch division (50; 46) paradigm and enhances it by incorporating an innovative extension: dynamic resolution with automatic patch configuration. To be specific, scaling the resolution of Large Vision-Language Models (LVLMs) to 4K HD and even higher standard is far beyond merely increasing the number of patches. It involves a nuanced approach to overcoming specific challenges: (1) Dynamic Resolution and Automatic Patch Configuration : Addressing the scarcity of high-resolution training data, our framework introduces a strategy that dynamically adjusts resolution alongside an automatic layout configuration. During training, it maintains the original aspect ratios of images while adaptively altering patch (336 × 336) layouts and counts. This results in a training resolution that exceeds the original image resolutions, reaching up to 4KHD, addressing the shortfall of high-resolution data. (2) Handling Variability in Patch Layouts : Despite the apparent simplicity of dynamic resolution training, the variability in patch layouts can heavily confuse LVLMs. To mitigate this, we introduce a newline token after each row of patch tokens to clearly delineate patch layouts, reducing training ambiguity and significantly boosting performance. (3) Inference Beyond 4K Resolution: Our observations reveal that, even when trained on images up to 4K resolution, the model can achieve additional performance improvements during inference by processing images at higher resolutions.

The image is a vibrant and colorful infographic that provides an overview of 7 graphic design

trends that are expected to dominate in 2021. The infographic is divided into 7 sections, each

representing a different trend.

At the top, there's a section titled "Muted Color Palettes", which introduces the concept of

muted colors as a trend for 2021.

This is followed by a section on "Simple Data Visualizations", highlighting the goal of making

complex data easy to understand.

The third section focuses on "Geometric Shapes Everywhere", showcasing how designers are

using geometric shapes in their designs.

The fourth section delves into "Flat Icons and Illustrations", explaining how brands are using

flat icons and illustrations in their social media graphics, website design, and more.

The fifth section introduces "Classic Serif Fonts", discussing how these fonts are seen as

classic, elegant, and trustworthy.

The sixth section explores "Social Media Slide Decks", illustrating how slide decks have

become a visual way of communicating longer messages on Instagram and LinkedIn.

Finally, the seventh section discusses "Text Heavy Videos", predicting that people will likely

continue to work remotely well into 2021, leading to an increase in video content that uses text

on-screen to communicate messages.

Overall, the infographic provides a comprehensive overview of the 7 graphic design trends that

are expected to shape the design landscape in 2021.

image size: 2550 x 3300

Analyze this figure in detail

The image is a scientific paper titled "ShareGPT4V: Improving

University of Science and Technology of China and Shanghai

AI Laboratory. The paper discusses the comparison between the

captions in their proposed ShareGPT4V dataset and those

utilized by recent large multi-modal models (LMMs). The

captions are compared in terms of their quality and

performance. The paper also highlights the remarkable

performance of the proposed LMMs based on a simple

architecture that has remarkable performance across a majority

of the multi-modal benchmarks.

Which model has the best performance on the

MMBench and what is its exact number?

According to the paper, ShareGPT4V-7B has the best

performance on the MMBench with a score of 68.8.

Where can I get this project?

You can get this project from the following GitHub link:

https://ShareGPT4V.github.io

image size: 816 x 5133

Illustrate the fine details present in the image.

What is the detailed explanation of the

third part?

The third part of the image is titled "Geometric Shapes Everywhere". It explains that last year,

designers used a lot of flowing and abstract shapes in their designs. However, this year, they

have been replaced with rigid, hard-edged geometric shapes and patterns. The hard edges of a

geometric shape create a great contrast against muted colors.

ShareGPT4V: Improving Large Multi-Modal Models with Better Captions

Lin Chen*§1.2, Jinsong Li*52, Xiaoyi Dong2, Pan Zhang², Conghui He², Jiaqi Wang²,

University of Science and Technology of China²Shanghai AILaboratory

60.6

64.3

68.8

56.7

MMBench

58.3

62.2

MMBench-CN

ShareGPT4V-7B

malti-modal benchmarks.This project is available at

https://ShareGPT4V.github.io to serve as a

Figure 1. (a) We showcase a comparison between the caption in our proposed ShareGPT4V dataset and those utilized by recent large

multi-modal models (LMMs). Unlike COCO-Caption [7] involves brief human-made captions on the main subject. LLaVA-Instruct

[31] combines human-made captions, bounding boxes, and GPT4 [39] to imagine° the image details, which leads to inevitable er-

rorlripimarkdduhingllyigpmlgwidil

the advanced GPT4Vision [40] and the descriptions aremoredetailed and accurate (marked inblue)(b）Wehighlight the remarkable

performance of the proposed LMM, ShareGPT4V-7B, developed with the asistance of the ShareGPT4V dataset.

2021

雪

雪

Furthermore, scaling the training resolution up to 4K standard results in a consistent improvement in performance, highlighting the potential for training even beyond 4K resolution. This underscores the capacity for further enhancing model capabilities and suggests a promising trajectory for advancing the frontiers of high-resolution image processing within the domain of large vision-language models.

We evaluate our InternLM-XComposer2-4KHD on 16 diverse benchmarks spanning various domains, including 5 challenging OCR datasets (InfographicVQA(69), DocVQA(70), OCRBench(58), TextVQA(87), and ChartQA(68)). Compared to previous open-source LVLM models and closedsource APIs, our approach achieves SOTA results in 6 of 16 benchmarks, demonstrating competitive performance despite only 7B parameters. As shown in Figure 1, InternLM-XComposer2-4KHD even surpasses the performance of GPT4V (74) and Gemini Pro (90) across ten benchmarks. Notably, our method exhibits excellent performance on 5 challenging OCR datasets, over existing open-source LVLMs by a substantial margin.

2 Related Works

Large Vision-Language Models (LVLMs). Large Language Models (LLMs) (9; 76; 73; 23; 41; 92; 93; 39; 91; 108; 6; 78; 10) have gained significant attention due to their impressive performance in various language-related tasks such as text generation and question answering. Following this enthusiasm, recent Large Vision-Language Models (LVLMs) have emerged(74; 19; 16; 18; 28; 32; 113; 25; 110; 7; 47; 77; 102; 4), combining LLMs with vision encoders (79; 109; 89) to leverage the complementary strengths of language and vision modalities. By fusing textual and visual representations, LVLMs can ground language in visual contexts, enabling a more comprehensive understanding and generation of multimodal content (14; 20; 51; 5; 95; 27; 11; 60).

LVLMs for High-Resolution Understanding. Large Vision-Language Models (LVLMs) often employ CLIP-ViT as the visual encoder for vision-dependent tasks. However, the visual encoder's reliance on low resolutions, such as 224 × 224 or 336 × 336 pixels, limits its effectiveness for highresolution tasks like OCR and document/chart perception. To enhance high-resolution understanding, recent works have primarily employed the following strategies: (1) High-resolution (HR) visual encoders or dual encoders catering to high-resolution (HR) and low-resolution (LR) inputs (66; 97; 36; 48). For instance, Vary (97) introduces a new image encoder supporting HR inputs, which are then concatenated with LR embeddings from the original CLIP visual encoder. Similarly, CogAgent (36) and Mini-Gemini (48) also separate HR and LR images using distinct vision encoders, subsequently merging their features using a cross-attention module. In contrast, our approach offers a more simplified solution and shows advantages for varying resolutions and aspect ratio inputs. (2) Cropped image patches (50; 59; 99; 101; 37; 51; 46). For example, Monkey (50) employs sliding windows to segment images into patches, subsequently processing them with LoRA fine-tuning. TextMonkey (59) further proposes shifted window attention and token resampler to consider the connections among different patches. Fuyu (7) eliminates the need for the image encoder by directly processing a raw image patch sequence. These approaches are confined to either a few predefined high-resolution settings (36; 97; 48; 50; 51; 55; 46; 66; 59) or a limited range of resolutions (37; 99). Conversely, our method devises a dynamic resolution and automatic path configuration strategy to support the scaling from 336 pixels to 4K resolution, and the maximum resolution is larger than previous approaches ( e.g. , 1.5k for Monkey (50) and 1.2k for UReader (101)). For the first time, our approach discussed the challenges and solutions for handling variability in image feature patch layouts, ensuring effective training with dynamic high resolutions.

LVLMs for Document Understanding. Document understanding involves analyzing and comprehending various digital documents, such as figures, tables, and academic papers. Many document understanding tasks require models to handle high-resolution inputs, complex layouts, various aspect ratios, and diverse document formats. To enhance the capabilities of LVLMs for document understanding, several works have collected and constructed high-quality document instruction tuning data, including LLaVAR (112), mPLUG-DocOwl (100) and TGDoc (96). DocPediaDocPedia (30) processes document inputs in the frequency domain. Some previous works have improved document understanding ability by designing special modules for high-resolution inputs, such as HR and LR encoders (36; 97) or cropped image patches (101; 59; 99). Our InternLM-XComposer2-4KHD first scales to 4K resolution inputs and demonstrates strong document understanding ability on OCRrelated benchmarks. Also, our approach also achieves comparable results to state-of-the-art opensourced LVLMs on other general LVLM benchmarks like perception and reasoning (61; 57; 33; 15).

3 Method

3.1 Model Architecture.

The model architecture of InternLM-XComposer2-4KHD mainly follows the design of InternLMXComposer2(27) (XComposer2 / IXC2 in the following for simplicity ), including a light-weight Vision Encoder OpenAI ViT-Large/14, Large Language Model InternLM2-7B, and Partial LoRA for efficient alignment.

3.2 High-Resolution Input.

Dynamic Patch Configuration. Utilizing a static input image size for processing high-resolution images, particularly those with varying aspect ratios, is neither efficient nor effective. To overcome

Encode

& Merge

Encode

& Merge

Resize

Dynamic Image

Configuration

\n

\n

\n

\n

\n

\n

LLM

The image presents a vibrant and informative infographic titled

"Protecting Our Planet Starts With You". The infographic is

divided into various sections, each dedicated to a different

aspect of environmental protection.

In the top left corner,…………

Tokenize

PROTECTINGOURPLANETSTARTSWITHYOU

DRIVE LESS

BIKE MORE

7

recycle

reduce

choose sustainable

A TREE

PLANT

EDUCATE

WATER

CONSERVE

-SHOP-

ATTSIM

Don't send

chemicals

inte our

waterways.

BRIGHT

IDEA

AREA

11

11

sep

11

n1

PROTECTNG OURPUAET STARTSWITH YOL

PROTECTINGOU

RPLANETSTARTS

WITHYOU

DRIVE LESS

BIKE MORE

reduce

recycle

choose

A TREE

PLANT

-SHOP-

AT3SIM

into our

chemlcals

waterways

BRIGH

IDE

this limitation, we introduce a dynamic patch configuration approach via image partitioning, as shown in Figure 3. Our method strategically segments the image into smaller patches, while maintaining the integrity of the original image's aspect ratio.

Given a maximum patch number H , the image x with size [ h, w ] is resized and padded to the new image ˆ x with size [ p h × 336 , p w × 336] . This process is subject to the following constraints:

here p w and p h represent the number of patches in each row and column, respectively. We then split the ˆ x into p h × p w non-overlapped patches. Each patch is a small image with 336 × 336 size and we treat these patches as individual inputs for the ViT.

In the following, we use 'HDH ' to represent our high-resolution setting with the constraint of H patches. For example, the 'HD-9' allows up to 9 patches, including a range of resolutions such as 1008 × 1008 , 672 × 1344 , 336 × 3024 , etc.

Global-Local Format. For each input image, we present it to the model with two views. The first is the global view, where the image is resized to a fixed size (in our case, 336 × 336). This provides a macro understanding of the image. Empirically, we have found this to be crucial for the LVLM to correctly understand the image. The second view is the local view. We divide the image into patches using the previously mentioned Dynamic Image Partition strategy and extract features from each patch. Following feature extraction, the patches are reassembled into a large feature map. The feature map is then flattened to the final local features after a straightforward token merging process.

Patch Layout Indicator. Given that an image will have a dynamic patch layout in our method, the number of tokens for each row can vary across different images. This variation will confuse the LVLM, making it difficult to determine which tokens belong to the same row of the image and which ones belong to the next row. This confusion hinders the LVLM's ability to understand the 2D structure of the image, which is crucial for comprehending structural image content such as documents, charts, and tables. To address this issue, we introduce a learnable newline (' \ n') token at the end of each row of the image features before the flattening. Finally, we concatenate the global and local views, inserting a special separate ('sep') token between them to distinguish the two views.

3.3 Pre-Training

During the pre-training phase, the LLM is frozen while both the vision encoder and Partial LoRA are fine-tuned to align the visual tokens with the LLM following XComposer2(27). The pre-training data

mainly follow the design in XComposer2 which is curated with three objectives in mind: 1) general semantic alignment, 2) world knowledge alignment, 3) vision capability enhancement. In this paper, we focus on high-resolution and structural image understanding. So we collected OCR and chart data from diverse sources to enhance this specific capability, as shown in Table.1.

In practice, we employ the OpenAI CLIP ViT-L-14-336 as the vision encoder. We keep the ViT resolution as 336 × 336 while adopting Dynamic Patch Configuration to handle higher-resolution images. For pretraining, we use the 'HD-25' configuration, which resizes the input image to a random larger resolution to generate more patches, with the constraint that the total number of patches does not exceed 25. For each image or patch tokens, the image token number is decreased to 1 / 4 with a simple merge operation . We concatenate the nearby 4 tokens into a new token through the channel dimension, then align it with the LLM by an MLP. The 'sep' and ' \ n' tokens are randomly initialized. For the Partial LoRA, we set a rank of 256 for all the linear layers in the LLM decoder block. Our training process involves a batch size of 4096 and spans across 2 epochs. The learning rate linearly increases to 2 × 10 -4 within the first 1% of the training steps. Following this, it decreases to 0 according to a cosine decay strategy. To preserve the pre-existing knowledge of the vision encoder, we apply a layer-wise learning rate (LLDR) decay strategy, and the decay factor is set to 0 . 90 .

3.4 4KHD Supervised Fine-tuning (SFT)

After pre-training, we empower the model to understand high-resolution images and solve diverse challenges. Different from conventional perception tasks ( e.g. , VQAv2, GQA) which typically answer questions based on the noticeable object in the image. OCR-related tasks depend on a detailed understanding of text within a high-resolution image. For instance, in InfoVQA, the length of the longer side of 50% of the images exceeds 2000 pixels. Low-resolution inputs can distort the dense text information, causing the model to fail in its understanding. However, we have observed a resolution saturation phenomena with perception tasks, where a higher resolution makes minor gains.

To address this, we introduce a mixed-resolution strategy for more efficient training. For tasks requiring high resolution, we employ the 'HD-55' setting during training. This allows for the input of 4K ( 3840 × 1600 ) images without necessitating additional image compression. These tasks are referred to as the HD-OCR QA tasks in Table 2. For the other tasks, we apply the 'HD-25' resolution setting for them. As in pre-training, we adopt the dynamic-resolution strategy during SFT, images are resized to fall within a range between their original size and the size specified by the 'HD' setting. This dynamic approach enhances the robustness of the LVLM against differences in input resolution, thereby enabling the LVLM to utilize a larger resolution during inference. For instance, we have observed that using the 'HD30' setting yields better results on most OCR-related tasks when the LVLM is trained under the 'HD25' setting.

In practice, we jointly train all the components with a batch size of 2048 over 3500 steps. Data from multiple sources are sampled in a weighted manner, with the weights based on the number of data from each sourced (See Appendix B.1 for more details). As the 'HD-55' setting has double image tokens than the 'HD-25', we adjust the data loader to enable different batch sizes for them and adjust their weight accordingly. The maximum learning rate is set to 5 × 10 -5 , and each component has its

own unique learning strategy. For the vision encoder, we set the LLDR to 0 . 9 , which aligns with the pretraining strategy. For the LLM, we employ a fixed learning rate scale factor of 0 . 2 . This slows down the update of the LLM, achieving a balance between preserving its original capabilities and aligning it with vision knowledge. It takes almost 40 hours with 256 A100 GPUs.

4 Experiments

In this section, we validate the benchmark performance of our InternLM-XComposer2-4KHD (IXC24KHD in the following for simplicity) after supervised fine-tuning.

4.1 LVLM Benchmark results.

In Table 3 and Table 4, we compare our IXC2-4KHD on a list of benchmarks with both SOTA open-source LVLMs and closed-source APIs. Here we report results in DocVQA(70), ChartQA(68), InfographicVQA(69), TextVQA(87), OCRBench(58), MMStar(15), MathVista(61), MMMU(107), AI2D(42), MME (31), MMBench (MMB) (57), MMBench-Chinese (MMB CN ) (57), SEEDBench Image Part (SEED I )(45), QBench-Testset (QBench T )(98), MM-Vet (105), HallusionBench (HallB)(34). The evaluation is mainly conducted on the OpenCompass VLMEvalKit(24) for the unified reproduction of the results.

Comparison with Closed-Source APIs. As demonstrated in Table 3, IXC2-4KHD exhibits competitive performance across a variety of benchmarks, rivaling that of Closed-Source APIs. Owing to its high-resolution input, IXC2-4KHD achieves a score of 90 . 0% on DocVQA and 81 . 0% on ChartQA, thereby surpassing GPT-4V and Gemini-Pro with a non-trivial margin. In the challenging InfographicVQA task, our model is the first open-source model that is close to the performance of Closed-Source APIs, exceeding the performance of previous open-source models by nearly 20% . In addition to OCR-related tasks, IXC2-4KHD is a general-purpose Large Vision-Language Modal that excels in semantic-level tasks, demonstrating competitive results.

Comparison with Open-Source Models. We also conduct a comprehensive comparison with opensource LVLMs under a similar model scale. As shown in Table 4, our model significantly outperforms existing open-source models, achieving competitive results across all benchmarks.

High-resolution Understanding Evaluation. Then we compare IXC2-4KHD with models that are specifically designed for high-resolution understanding tasks. We report the results of 5 highresolution benchmarks in Table 5, as a general LVLM, IXC2-4KHD shows superb performance on

Table 6: Influence of Inference Resolution. The model achieves better performance on text-related tasks when the inference resolution is higher than its training resolution.

50

55

60

65

70

75

80

85

90

InfoVQA

DocVQA

TextVQA

ChartQA

MMBench

MME

Seed

AI2D

HD-9 (1561 Tokens)

HD-16 (2653 Tokens)

HD-25 (4057 Tokens)

4K HD (8737 Tokens)

*

Val

Val

Val

Test

Test

EN-Test

these tasks and outperforms competitors with a large margin. For example, IXC2-4KHD gets 68 . 6% on InfographicVQA, surpassing recent DocOwl 1.5 with +17 . 9% . For the OCRBench, IXC2-4KHD gets 67 . 5% , outperforms CogAgent with +8 . 5% .

4.2 Dive into Resolution

High-Resolution Training is Critical for HD-OCR tasks. We study four resolution settings: HD-9 (1561 image tokens at most, we simply the statement in the following), HD-16 (2653 tokens), HD-25 (4057 tokens), and 4KHD (8737 tokens). Here we report the validation set of InfoVQA, DocVQA, and TextVQA, test set of ChartQA and AI2D, MMBench EN-Test, and a 2k subset of SEEDBench (we denote it as SEED ∗ ). In the following, we report results on the above benchmarks by default.

As illustrated in Fig.4, we note a significant improvement in the HD-OCR tasks as the resolution increases. For instance, the model achieves only a 50 . 5% score on the InfographicVQA with the HD-9 setting. However, when we switch to the HD-16 setting, we observe a performance gain of +10 . 2% . The performance continues to improve as the resolution increases, with saturation not observed even for the 4KHD setting. Due to computational constraints, we defer the exploration of the upper bound of improvement to future work. In terms of other OCR-related tasks, the performance gain attributable to increased resolution is relatively minor. For the perception-related benchmarks, performance is saturated on the resolution that only has negligible difference between the four settings.

Higher Inference Resolution Leads to better results on Text-related Tasks. An intriguing observation from our experiments is that our model, when inferring with a higher resolution, tends to yield improved results on text-related tasks. We present the results of HD-9, HD-16, and HD-25 in Table 6. For instance, IXC2-HD9 achieves a 50 . 5% score on InfographicVQA. When we infer with HD16, we see a performance gain of +8 . 1% , without additional training. Similar improvements are also observed with IXC2-HD16 and IXC2-HD25. We posit that the dynamic image token length used in training enhances the robustness of the LVLM, leading to better results when the text in the image is more 'clear' in the higher-resolution input. Conversely, the results on ChartQA consistently degrade under this setting. This could be due to the model becoming confused about the chart structure when

the resolution is increased. Additionally, similar to the observation from Figure 4, the impact of resolution on perception-related benchmarks appears to be quite minor.

4.3 High-Resolution Strategy Ablation

The Role of Global-View. We first examine the impact of the global view in our Global-Local Format. As indicated in Table 8(a), we find that the global view is essential for the LVLM to accurately comprehend the input image. When it is removed, the model performs worse across all benchmarks. For instance, the model experiences a -4 . 4% drop in performance on the MMBench EN-Test without the global view. We contend that the global view offers a general macro understanding of the image, which the model struggled to derive from the large number of tokens in the local view.

The Role of the Newline Token. We incorporate a special newline token at the end of each row of the image features before the flattening operation. This token serves as an indicator of the image's 2D structure. We examine its impact on both the HD-9 and 4KHD strategies in Table 7(a). When a fixed high-resolution strategy HD-9 is employed, we observe that the benefit derived from the newline token is minor. This could be attributed to the LVLM's ability to handle limited differences in image ratios after training. However, when we implement a more challenging 4KHD (HD-25 + HD-55) strategy, which exhibits significant diversity in both image ratio and token number, the LVLM demonstrates a notable decline in performance on OCR-related tasks without the newline indicator. This finding supports our hypothesis that the LVLM struggles to comprehend the shape of the image when the image tokens are directly flattened into a 1D sequence. The newline token can assist the model in better understanding the structure of the image.

Influence of Token Merging Operation. In practice, we employ a simple merging operation that concatenates four adjacent tokens along the channel dimension. We have found this approach to be effective in reducing the number of image tokens efficiently. Here we study the influence of different token-merging operations under the 4KHD setting. In Table 7(b), we study two additional strategies: Re-Sampler(5) and C-Abstractor(12), with their default setting and the same compressing rate 0 . 25 , i.e. , reducing an image with 576 tokens to 144 tokens. Results show that both concatenation and C-Abstractor work well and get similar results on most benchmarks, this observation is also consistent with the study in MM-1(71) that the influence of the connector is minor. However, the Re-Sampler performs worse than the other methods with a noticeable margin. We argue this is caused by the learnable queries used for gathering information requiring a great number of data for training, our pre-training data is somewhat lightweight for it to converge fully.

Strategy-Level Comparison. For a fair comparison, we trained a new model with the identical architecture, training strategy, and dataset as our original model, but with one key modification: we adopted the high-resolution strategy from LLaVA-Next. We name it as IXC-LLaVA and compare it with 1) LLaVA-Next 0530 official results and 2) IXC2-4KHD under HD-9/16 setting. The results in Table.9 demonstrate that IXC-LLaVA achieves promising performance across all six benchmarks, leveraging the benefits of additional training data and advanced IXC architecture design. However, it is outperformed by IXC2-4KHD HD-9, which utilizes fewer image tokens yet yields better results. This fair comparison underscores the efficiency and effectiveness of our proposed high-resolution strategy, highlighting its advantages over the LLaV A-Next approach.

Inference Efficiency Analysis. Our model processes high-resolution images with numerous image tokens, and here we study its inference efficiency in real-world usages. The model inference process consists of two stages: encoding the prefix (model input) and autoregressively decoding new tokens (model output). Correspondingly, the inference efficiency considers two parts: time to encode the prefix and speed to decode each token. Here we report the prefix encoding time and per-token decoding speed under different HD settings. We test the speed with a Nvidia-A100 80G. With the results in Table 10, we have three observations: 1) Prefix encoding time increases linearly with the number of prefix tokens. 2) Decoding speed remains relatively constant, regardless of prefix length, thanks to optimizations on transformers from research communities and companies, including kv-cache and flash-attention. 3) When generating 2048 tokens, total inference time usage is nearly identical across HD9 to HD55, as encoding time is much smaller. Based on the above analysis, we believe the inference efficiency of our model is acceptable. Besides, we believe some targeted designs can further improve efficiency, while our paper focuses on enabling LVLM to understand high-resolution images with a general and effective solution, and we would leave the efficiency exploration in future work.

5 Conclusion

In this paper, we propose the InternLM-Xcomposer2-4KHD that exceeds the performance of previous open-source models on OCR-related tasks and also achieves competitive results on general-purpose LVLM benchmarks. Thanks to our dynamic resolution and automatic patch configuration, our model supports a maximum training resolution of up to 4K HD. We also integrate a global view patch to support the macro understanding and a learnable newline token to handle the various input image resolutions. Our model's performance continues to improve as the training resolution increases for HD-OCR tasks. Notably, we do not observe any performance saturation even for the 4KHD setting, and we have not explored the upper bound due to the computational burden increasing with higher-resolution inputs. In future work, we plan to explore efficient solutions for accurate LVLM training and inference, enabling our model to handle even higher resolutions while maintaining computational efficiency.

6 Acknowledgment

This project is funded in part by Shanghai Artificial lntelligence Laboratory, the National Key R&D Program of China (2022ZD0160201), the Centre for Perceptual and Interactive Intelligence (CPII) Ltd under the Innovation and Technology Commission (ITC)'s InnoHK. Dahua Lin is a PI of CPII under the InnoHK.

References

Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. Nocaps: Novel object captioning at scale. In Proceedings of the IEEE/CVF international conference on computer vision , pages 8948-8957, 2019.

01. AI, :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai. Yi: Open foundation models by 01.ai, 2024.

Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In International Conference on Computer Vision (ICCV) , 2015.

Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. Openflamingo: An open-source framework for training large autoregressive vision-language models. arXiv.org , 2023.

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv.org , 2023.

Baichuan. Baichuan 2: Open large-scale language models. arXiv.org , 2023.

Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sa˘ gnak Ta¸ sırlar. Introducing our multimodal models, 2023.

Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marçal Rusinol, Ernest Valveny, CV Jawahar, and Dimosthenis Karatzas. Scene text visual question answering. In Proceedings of the IEEE/CVF international conference on computer vision , pages 4291-4301, 2019.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems (NeurIPS) , 33:1877-1901, 2020.

Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo, Qipeng Guo, Conghui He, Yingfan Hu, Ting Huang, Tao Jiang, Penglong Jiao, Zhenjiang Jin, Zhikai Lei, Jiaxing Li, Jingwen Li, Linyang Li, Shuaibin Li, Wei Li, Yining Li, Hongwei Liu, Jiangning Liu, Jiawei Hong, Kaiwen Liu, Kuikun Liu, Xiaoran Liu, Chengqi Lv, Haijun Lv, Kai Lv, Li Ma, Runyuan Ma, Zerun Ma, Wenchang Ning, Linke Ouyang, Jiantao Qiu, Yuan Qu, Fukai Shang, Yunfan Shao, Demin Song, Zifan Song, Zhihao Sui, Peng Sun, Yu Sun, Huanze Tang, Bin Wang, Guoteng Wang, Jiaqi Wang, Jiayu Wang, Rui Wang, Yudong Wang, Ziyi Wang, Xingjian Wei, Qizhen Weng, Fan Wu, Yingtong Xiong, Chao Xu, Ruiliang Xu, Hang Yan, Yirong Yan, Xiaogui Yang, Haochen Ye, Huaiyuan Ying, Jia Yu, Jing Yu, Yuhang Zang, Chuyu Zhang, Li Zhang, Pan Zhang, Peng Zhang, Ruijie Zhang, Shuo Zhang, Songyang Zhang, Wenjian Zhang, Wenwei Zhang, Xingcheng Zhang, Xinyue Zhang, Hui Zhao, Qian Zhao, Xiaomeng Zhao, Fengzhe Zhou, Zaida Zhou, Jingming Zhuo, Yicheng Zou, Xipeng Qiu, Yu Qiao, and Dahua Lin. Internlm2 technical report. arXiv preprint arXiv:2403.17297 , 2024.

Yuhang Cao, Pan Zhang, Xiaoyi Dong, Dahua Lin, and Jiaqi Wang. DualFocus: Integrating macro and micro perspectives in multi-modal large language models. arXiv preprint arXiv:2402.14767 , 2024.

Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. Honeybee: Locality-enhanced projector for multimodal llm. arXiv preprint arXiv:2312.06742 , 2023.

Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal llm's referential dialogue magic. arXiv.org , 2023.

Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793 , 2023.

Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330 , 2024.

Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Carlos Riquelme Ruiz, Sebastian Goodman, Xiao Wang, Yi Tay, Siamak Shakeri, Mostafa Dehghani, Daniel Salz, Mario Lucic, Michael Tschannen, Arsha Nagrani, Hexiang Hu, Mandar Joshi, Bo Pang, Ceslee Montgomery, Paulina Pietrzyk, Marvin Ritter, AJ Piergiovanni, Matthias Minderer, Filip Pavetic, Austin Waters, Gang Li, Ibrahim Alabdulmohsin, Lucas Beyer, Julien Amelot, Kenton Lee, Andreas Peter Steiner, Yang Li, Daniel Keysers, Anurag Arnab, Yuanzhong Xu, Keran Rong, Alexander Kolesnikov, Mojtaba Seyedhosseini, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut. Pali-x: On scaling up a multilingual vision and language model, 2023.

Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollar, and C. Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server, 2015.

Xi Chen, Xiao Wang, Lucas Beyer, Alexander Kolesnikov, Jialin Wu, Paul Voigtlaender, Basil Mustafa, Sebastian Goodman, Ibrahim Alabdulmohsin, Piotr Padlewski, Daniel Salz, Xi Xiong, Daniel Vlasic,

Filip Pavetic, Keran Rong, Tianli Yu, Daniel Keysers, Xiaohua Zhai, and Radu Soricut. Pali-3 vision language models: Smaller, faster, stronger, 2023.

Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thapliyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme, Andreas Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut. Pali: A jointly-scaled multilingual language-image model, 2023.

Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv preprint arXiv:2312.14238 , 2023.

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.

Chee Kheng Chng, Yuliang Liu, Yipeng Sun, Chun Chet Ng, Canjie Luo, Zihan Ni, ChuanMing Fang, Shuaitao Zhang, Junyu Han, Errui Ding, et al. Icdar2019 robust reading challenge on arbitrary-shaped text-rrc-art. In 2019 International Conference on Document Analysis and Recognition (ICDAR) , pages 1571-1576. IEEE, 2019.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv.org , 2022.

OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models. https://github.com/open-compass/opencompass , 2023.

Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023.

Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José M.F. Moura, Devi Parikh, and Dhruv Batra. Visual Dialog. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2017.

Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Xinyue Zhang, Wei Li, Jingwen Li, Kai Chen, Conghui He, Xingcheng Zhang, Yu Qiao, Dahua Lin, and Jiaqi Wang. Internlm-xcomposer2: Mastering free-form text-image composition and comprehension in vision-language large model. arXiv preprint arXiv:2401.16420 , 2024.

Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. Palm-e: An embodied multimodal language model. In arXiv preprint arXiv:2303.03378 , 2023.

Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 320-335, 2022.

Hao Feng, Qi Liu, Hao Liu, Wengang Zhou, Houqiang Li, and Can Huang. DocPedia: Unleashing the power of large multimodal model in the frequency domain for versatile document understanding. arXiv preprint arXiv:2311.11810 , 2023.

Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394 , 2023.

Chaoyou Fu, Renrui Zhang, Zihan Wang, Yubo Huang, Zhengye Zhang, Longtian Qiu, Gaoxiang Ye, Yunhang Shen, Mengdan Zhang, Peixian Chen, Sirui Zhao, Shaohui Lin, Deqiang Jiang, Di Yin, Peng Gao, Ke Li, Hongsheng Li, and Xing Sun. A challenger to gpt-4v? early explorations of gemini in visual expertise. arXiv preprint arXiv:2312.12436 , 2023.

Yuying Ge, Yixiao Ge, Ziyun Zeng, Xintao Wang, and Ying Shan. Planting a seed of vision in large language model. arXiv preprint arXiv:2307.08041 , 2023.

Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. Hallusionbench: An advanced diagnostic suite for entangled language hallucination & visual illusion in large vision-language models, 2023.

Conghui He, Zhenjiang Jin, Chaoxi Xu, Jiantao Qiu, Bin Wang, Wei Li, Hang Yan, Jiaqi Wang, and Da Lin. Wanjuan: A comprehensive multimodal dataset for advancing english and chinese large models. ArXiv , abs/2308.10755, 2023.

Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. arXiv preprint arXiv:2312.08914 , 2023.

Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Chen Li, Ji Zhang, Qin Jin, Fei Huang, et al. mplug-docowl 1.5: Unified structure learning for ocr-free document understanding. arXiv preprint arXiv:2403.12895 , 2024.

Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. Conference on Computer Vision and Pattern Recognition (CVPR) , 2019.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.

Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. Dvqa: Understanding data visualizations via question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5648-5656, 2018.

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 , 2020.

Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14 , pages 235-251. Springer, 2016.

Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Hajishirzi. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. In Proceedings of the IEEE Conference on Computer Vision and Pattern recognition , pages 4999-5007, 2017.

Paul Lerner, Olivier Ferret, Camille Guinaudeau, Hervé Le Borgne, Romaric Besançon, José G Moreno, and Jesús Lovón Melgarejo. Viquae, a dataset for knowledge-based visual question answering about named entities. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages 3108-3120, 2022.

Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Benchmarking multimodal llms with generative comprehension, 2023.

Bo Li, Peiyuan Zhang, Jingkang Yang, Yuanhan Zhang, Fanyi Pu, and Ziwei Liu. Otterhd: A highresolution multi-modality model, 2023.

Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multimodal model with in-context instruction tuning. arXiv.org , 2023.

Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-Gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403.18814 , 2024.

Zhuowan Li, Xingrui Wang, Elias Stengel-Eskin, Adam Kortylewski, Wufei Ma, Benjamin Van Durme, and Alan L Yuille. Super-clevr: A virtual benchmark to diagnose domain robustness in visual reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1496314973, 2023.

Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi-modal models. arXiv preprint arXiv:2311.06607 , 2023.

Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv preprint arXiv:2311.07575 , 2023.

Adam Dahlgren Lindström and Savitha Sam Abraham. Clevr-math: A dataset for compositional language, visual and mathematical reasoning. arXiv preprint arXiv:2208.05358 , 2022.

Fangyu Liu, Guy Edward Toh Emerson, and Nigel Collier. Visual spatial reasoning. Transactions of the Association for Computational Linguistics , 2023.

Fuxiao Liu, Xiaoyang Wang, Wenlin Yao, Jianshu Chen, Kaiqiang Song, Sangwoo Cho, Yaser Yacoob, and Dong Yu. Mmc: Advancing multimodal chart understanding with large-scale instruction tuning. arXiv preprint arXiv:2311.10774 , 2023.

Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv.org , 2023.

Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhnag, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player? arXiv:2307.06281 , 2023.

Yuliang Liu, Zhang Li, Biao Yang, Chunyuan Li, Xucheng Yin, Cheng lin Liu, Lianwen Jin, and Xiang Bai. On the hidden mystery of ocr in large multimodal models, 2024.

Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, and Xiang Bai. Textmonkey: An ocr-free large multimodal model for understanding document. arXiv preprint arXiv:2403.04473 , 2024.

Ziyu Liu, Zeyi Sun, Yuhang Zang, Wei Li, Pan Zhang, Xiaoyi Dong, Yuanjun Xiong, Dahua Lin, and Jiaqi Wang. RAR: Retrieving and ranking augmented mllms for visual recognition. arXiv preprint arXiv:2403.13805 , 2024.

Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representations (ICLR) , 2024.

Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu. Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. In The 59th Annual Meeting of the Association for Computational Linguistics (ACL) , 2021.

Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems , 35:2507-2521, 2022.

Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. arXiv preprint arXiv:2209.14610 , 2022.

Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. arXiv preprint arXiv:2110.13214 , 2021.

Tengchao Lv, Yupan Huang, Jingye Chen, Lei Cui, Shuming Ma, Yaoyao Chang, Shaohan Huang, Wenhui Wang, Li Dong, Weiyao Luo, Shaoxiang Wu, Guoxin Wang, Cha Zhang, and Furu Wei. Kosmos-2.5: A multimodal literate model, 2023.

Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition , pages 3195-3204, 2019.

Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244 , 2022.

Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvqa. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 1697-1706, 2022.

Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision , pages 2200-2209, 2021.

Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, et al. Mm1: Methods, analysis & insights from multimodal llm pre-training. arXiv preprint arXiv:2403.09611 , 2024.

Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In ICDAR , 2019.

OpenAI. Chatgpt. https://openai.com/blog/chatgpt , 2022.

OpenAI. Gpt-4 technical report, 2023.

Vicente Ordonez, Girish Kulkarni, and Tamara L. Berg. Im2text: Describing images using 1 million captioned photographs. In Neural Information Processing Systems (NIPS) , 2011.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems (NeurIPS) , 35:27730-27744, 2022.

Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos2: Grounding multimodal large language models to the world. arXiv.org , 2023.

Qwen. Introducing qwen-7b: Open foundation and human-aligned models (of the state-of-the-arts), 2023. [79] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine learning (ICML) , pages 8748-8763. PMLR, 2021.

Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 , 2021.

Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In European Conference on Computer Vision , pages 146-162. Springer, 2022.

Sanket Shah, Anand Mishra, Naganand Yadati, and Partha Pratim Talukdar. Kvqa: Knowledge-aware visual question answering. In Proceedings of the AAAI conference on artificial intelligence , 2019.

Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 2556-2565, 2018.

Baoguang Shi, Cong Yao, Minghui Liao, Mingkun Yang, Pei Xu, Linyan Cui, Serge Belongie, Shijian Lu, and Xiang Bai. Icdar2017 competition on reading chinese text in the wild (rctw-17). In 2017 14th iapr international conference on document analysis and recognition (ICDAR) , volume 1, pages 1429-1434. IEEE, 2017.

Chenglei Si, Yanzhe Zhang, Zhengyuan Yang, Ruibo Liu, and Diyi Yang. Design2code: How far are we from automating front-end engineering?, 2024.

Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image captioning with reading comprehension. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16 , pages 742-758. Springer, 2020.

Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 8317-8326, 2019.

Yipeng Sun, Zihan Ni, Chee-Kheng Chng, Yuliang Liu, Canjie Luo, Chun Chet Ng, Junyu Han, Errui Ding, Jingtuo Liu, Dimosthenis Karatzas, et al. Icdar 2019 competition on large-scale street view text with partial labeling-rrc-lsvt. In 2019 International Conference on Document Analysis and Recognition (ICDAR) , pages 1557-1562. IEEE, 2019.

Zeyi Sun, Ye Fang, Tong Wu, Pan Zhang, Yuhang Zang, Shu Kong, Yuanjun Xiong, Dahua Lin, and Jiaqi Wang. Alpha-CLIP: A clip model focusing on wherever you want. arXiv preprint arXiv:2312.03818 , 2023.

Gemini Team. Gemini: A family of highly capable multimodal models, 2023.

InternLM Team. Internlm: A multilingual language model with progressively enhanced capabilities. https://github.com/InternLM/InternLM , 2023.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv.org , 2023.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models, 2023.

Junke Wang, Lingchen Meng, Zejia Weng, Bo He, Zuxuan Wu, and Yu-Gang Jiang. To see is to believe: Prompting gpt-4v for better visual instruction tuning. arXiv preprint arXiv:2311.07574 , 2023.

Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. Cogvlm: Visual expert for pretrained language models, 2023.

Yonghui Wang, Wengang Zhou, Hao Feng, Keyi Zhou, and Houqiang Li. Towards improving document understanding: An exploration on text-grounding via mllms. arXiv preprint arXiv:2311.13194 , 2023.

Haoran Wei, Lingyu Kong, Jinyue Chen, Liang Zhao, Zheng Ge, Jinrong Yang, Jianjian Sun, Chunrui Han, and Xiangyu Zhang. Vary: Scaling up the vision vocabulary for large vision-language models. arXiv preprint arXiv:2312.06109 , 2023.

Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang, Chunyi Li, Wenxiu Sun, Qiong Yan, Guangtao Zhai, et al. Q-bench: A benchmark for general-purpose foundation models on low-level vision. arXiv preprint arXiv:2309.14181 , 2023.

Ruyi Xu, Yuan Yao, Zonghao Guo, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, Maosong Sun, and Gao Huang. Llava-uhd: an lmm perceiving any aspect ratio and high-resolution images. arXiv preprint arXiv:2403.11703 , 2024.

Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Yuhao Dan, Chenlin Zhao, Guohai Xu, Chenliang Li, Junfeng Tian, et al. mPLUG-DocOwl: Modularized multimodal large language model for document understanding. arXiv preprint arXiv:2307.02499 , 2023.

Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, et al. Ureader: Universal ocr-free visually-situated language understanding with multimodal large language model. arXiv preprint arXiv:2310.05126 , 2023.

Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv.org , 2023.

Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics , 2:67-78, 2014.

Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12284 , 2023.

Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490 , 2023.

Tai-Ling Yuan, Zhe Zhu, Kun Xu, Cheng-Jun Li, Tai-Jiang Mu, and Shi-Min Hu. A large chinese text dataset in the wild. Journal of Computer Science and Technology , 34(3):509-521, 2019.

Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. arXiv preprint arXiv:2311.16502 , 2023.

Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Zhiyuan

Liu, Peng Zhang, Yuxiao Dong, and Jie Tang. GLM-130b: An open bilingual pre-trained model. In The Eleventh International Conference on Learning Representations (ICLR) , 2023.

Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang, and Jiaqi Wang. Long-CLIP: Unlocking the long-text capability of clip. arXiv preprint arXiv:2403.15378 , 2024.

Pan Zhang, Xiaoyi Dong Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Shuangrui Ding, Songyang Zhang, Haodong Duan, Hang Yan, et al. Internlm-xcomposer: A vision-language large model for advanced text-image comprehension and composition. arXiv preprint arXiv:2309.15112 , 2023.

Rui Zhang, Yongsheng Zhou, Qianyi Jiang, Qi Song, Nan Li, Kai Zhou, Lei Wang, Dong Wang, Minghui Liao, Mingkun Yang, et al. Icdar 2019 robust reading challenge on reading chinese text on signboard. In 2019 international conference on document analysis and recognition (ICDAR) , pages 1577-1581. IEEE, 2019.

Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. LLaVAR: Enhanced visual instruction tuning for text-rich image understanding. arXiv preprint arXiv:2306.17107 , 2023.

Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv.org , 2023.

A More examples chat with our InternLM-XComposer2-4KHD

Give an in-depth description of the visual.

The image is an infographic from Public Health England that explains the concept of contact tracing in the context of preventing the spread of coronavirus (COVID-19). Contact tracing is a fundamental part of outbreak control used by public health professionals around the world to prevent the spread of infections.

The infographic is divided into several sections, each providing a different aspect of the contact tracing process. The first section explains that a contact is assessed to see how close they were to a confirmed case, which could include someone living with the case, or someone who has been in direct contact with them or their bodily fluids. Following this assessment, the contacts are categorised into low or high risk and provided with advice on what they should do.

The second section explains that if a contact is at high risk of infection, they may be asked to self-isolate, remaining in their home for 14 days and staying away from work, school or public places. If a contact is at low risk, they don't require self-isolation.

The third section explains that if the person develops symptoms, they would be tested and provided with specialist care if they have the novel coronavirus (COVID-19).

Finally, the infographic emphasizes the importance of recovery time, as contact displays no symptoms of coronavirus (COVID-19) the patient makes a full recovery, they can go back to daily lives as normal and pose no risk to others.

Image size: 986 x 2560

雪

PublicHealthEngland

WHATIS

contact tracing

Contacttracingisa fundamentalpartofoutbreakcontrol

that'susedbypublichealthprofessionalsaround theworld

toprevent thespreadof infections

Person

testspositive

for coronavirus

(COVID-19)

？

B

they were toaconfirmedcase.

is assessed to see how

A contact

close

whichcouldincludesomeone

living with the case，or

with themduring thetimetheyareinfectious

anyonewhohashadclosecontact

speak to thepatient toidentify

directcontactwiththemortheir

someone who hasbeenin

bodily fluids

Following this assessment,we cancategorise

themintoloworhighriskandcontactthemto

provideadviceonwhat they shoulddo

If webelievea contact isat

lowrisk

14

higher risk of infection

contact

days

theymaybeaskedto

doesn'trequire

self-isolate,remainingin

self-isolation

theirhomefor14daysand

stayingawayfromwork,

school orpublicplaces

contacton

We contactthem daily

passive followup

providethemwith advice

whichmeanspersonbeing

or

on what todoif they

monitored but wedon'tnecessarily

become unwelluntithey

contact themeveryday

can be given the all-clear

Ifthepersondoes

wewouldtest

themandprovidethemwith

specialistcareif theyhave

thenovelcoronavirus

(COVID-19)

Recovery

time

Contact displaysno symptoms

of coronavirus(COvID-19)

the patientmakesafullrecovery

they can go back to daily lives as

normal and pose no risk to others

Detect all the word in the given image

American Heart Association. VennGAGE.

How to Develop Healthy Eating Habits. And Reduce the Risk of Heart Disease.

Your best defense is a healthy lifestyle filled with physical activity and nutritious meals, while cutting down or avoiding drinking and smoking.

The Current Trend

75% of the US population is low in its fruit and vegetables consumption. 1/10 adults meets the recommendation for fruit and vegetable consumption. Men and young adults in general eat the fewest fruits and vegetables compared to any other group.

3 Steps to Eating Healthy 1. Add produce of various colors to your diet to diversify your produce consumption. 2. Use hacks to incorporate more veggies and fruits to your meal, and throughout the day. 3. Take steps to improve access to fruits and vegetables for all.

1. Adding Color to Your Diet

Eating 2 to 2% cups (4 to 5 servings) daily of fruits and vegetables may reduce your risk for cardiovascular disease, including heart attack and stroke.

Examples of a single serving of fruit: 1 medium apple, 1 small banana, 1 medium avocado, 4 large strawberries. Examples of a single serving of vegetables: 5 - 8 broccoli florets, 2/3 large sweet potato, 6 baby carrots, 1 cup of raw spinach.

Adding a variety of produce to your diet has a number of health benefits. It helps you: Manage your weight, Control your blood pressure, Support healthy digestion, Reduce the risk of some cancers, such as colon cancer, Reduce the risk of chronic health problems, such as diabetes.

2. Use Healthy Eating Hacks

1. Fresh, frozen, canned and dried all count toward your daily servings, so you have plenty of options. 2. Always reserve half of your plate for fruits and vegetables. 3. Choose whole fruits and vegetables. 4. Try to eat a variety of vegetables, instead of the same thing all the time. 5. Look for fruit packed in its own fruit juice and 100% vegetable juice, with no added sugar. 6. Look for low/no-sodium options for canned vegetables, and 100% vegetable juice.

Take Action for Better Access

In many communities, fruits and vegetables are limited/expensive and only available in corner stores, convenience stores, bodegas and gas stations.

Lack of access can profoundly impact diet. Here's how: Contact your city and state leaders to let them know what food access is in your community.

Find ways to spread the word about nutrition assistance programs, such as SNAP, WIC and school meals.

Create a petition for more variety, improve affordability and advocate for better signage/placement.

Meet with an after-school or daycare program representative to discuss serving more fruits and vegetables for snacks. Organize a letter-writing campaign and set up a meeting with state leaders.

For example, ask for funding to host a farmers market in an underserved community. Sign up for "You're the Cure" and send a note to your Congressperson advocating for healthier meals at school.

It's American Heart Month! Share this infographic with your family, friends and network today. You can carry out all of these actions or just a few. But whichever actions you choose, they pave the way for greater access to nutritious food for your community.

The HD image (816 x 3813)

VENNGAGE

How to Develop

HealthyEating Habits

And Reduce the Risk of Heart Disease

Your best defense is a healthy lifestyle filled with physical activity and

The Current Trend

75%

1/10

Men and young adults

vegetables consumption.

of the US population is

low in its ruit and

recommendation for fruit and

vegetable consumption.

adults mets the

compared to any other group.

in general eat the feest

fruits and egetables,

3 Steps to Eating Healthy

2

Ad produe of vanying colrs

1 pu fet to 6ieiyour

poduoe consumpfion.

mgjes and tuts t yor mel,

Use hadks to incorporte more

and throeghout the dlay.

Take taps to improt

accest fuits and

wegrtaeles for al.

路

1.AddingColor toYour Diet

Eating 2 to 2% cups (4 to 5 servings) daily of fruits and vegetables each may

reduce your risk for cardiovascular disease, including heart attack and stroke.

Examples of a single serving of fruit:

Examples of a tingle serving of vegetables:

• 1 medium apple

+ 1 small banana

• 4 large strzeberies

• % medium avocado

+ 6 baby carrots

• 5 - 8 broccoli flonets

• 1 cup of raw spinach

er of helth beefits. Ihelp you

yor blood pm

cancers, sch as col cancer

Reduoe the risk of some

protlens, such as diabetes

1

Frest, f, caed and dred al t

p dl g, y pty

of aptoss.

2

Als ee lrr

and vegetables.

3

sageade.pue tenq ape a0og

4

Tryto et a vty of veetes, itd o e

same ting all fe time.

5

Lookfrtkdiits nt nd

10% ft j,hs.

6

Lok for odu options for canned

vegetatet, and 100% egetatle jlot.

3.TakeActionforBetterAccess

In many communities, fruits and vegetables are limited/expensive and only

available in cormer stores, convenience stores, bodegas and gas stations.

Lack of Access Can Profoundly Impact Diet

There are waystht yocaactively supo prove acoess0

higher qualty foods for neighbourhoods. Here's how:

wht food acoes is lke i

yoer community.

Ceetact yoer eity and state

leaders to let them know

Find ways 10 sprad the word

about nutrion assistance

Cnest a petion for no

sigpage/placenent,

Met wih an afe-school er

dayare progran reesentatiee

egies for sucks.

put seniq aou fuues ssnotg 0l

Foreagleas f hdigt

Orgnizea later-eiing

cnpeip d s upmting

undersened community.

with state leaders.

host a farmers'marker in an

and the send a note to you

Heart Month!

It'sAmerican

Now is the time to raise awareness around heart health and wellness.

Share this infographie with your family, friends and network today.

che,they he wyops s ffory

雪

Conduct an in-depth analysis of the chart.

The line chart presents the operations cost by department in US

dollars for various categories from May 2018 to Mar 2019. The

categories include Human Resources (HR), Information

Technology (IT), Payroll, Support, Engineering, Facilities,

Finance, and Sales. Each category is represented by a different

color and line on the graph. The y-axis represents the cost in

thousands of US dollars, while the x-axis represents the months

from May 2018 to Mar 2019. The graph shows fluctuations in

the cost for each category throughout the months, with some

categories experiencing peaks and troughs at different times.

Which department has the highest cost in most cases?

Payroll has the highest cost in most cases, as indicated by

the purple line on the graph.

HRIT● Payroll Support ● Engineering●Facilities Finance ● Sales

Operations Cost by Department by Mo

↑ May 2018

Jul 2018

Sep 2018

Nov 2018

0

May2018

Jan2019

Mar20

Months

雪

雪

This flow chart outlines the process of ordering a book from Amazon. Here's a step-by-step analysis:

Start: The process begins with the user initiating the order.

Enter the website from the browser: The user navigates to the Amazon website.

Registered?: The system checks if the user is already registered. If not, they are prompted to register.

Sign in: If the user is already registered, they are prompted to sign in.

On home page search for book of your choice: The user searches for the desired book on the home page.

Select your preferred book: The user selects their preferred book from the search results.

Buy new?: The system asks if the user wants to buy a new book. If the answer is "Yes", the process continues to step

If the answer is "No", the process continues to step 9.8. Select a new book: The user selects a new book.

Select an old book: The user selects an old book.

Add to cart: The user adds the selected book to their cart.

Review order: The user reviews their order.

Change information: The user has the option to change their information. If they choose not to, the process continues to step 13. If they choose to, the process returns to step 11.

Enter shipping information: The user enters their shipping information.

Select payment method: The user selects their preferred payment method.

Place order: The user places their order.16. End: The process concludes with the order being placed

The HD image (1045x1428)

analyze this flow chart step by step

Ordering A Book From

amazon

雪

The flow chart in the image is an algorithm for finding the maximum element in an array. Here is a detailed explanation of the algorithm:

Declare an integer array having size m.

Input the m numbers into the array.

Initialize max as the first element of the array.

For each element in the array, starting from the second element:

If the current element is greater than max, set max to the current element.

At the end of the loop, max will hold the maximum element in the array.

Output the maximum element.

This algorithm iterates through the array, comparing each element to the current maximum. If a larger element is found, it becomes the new maximum. At the end of the loop, the maximum element is stored in the variable max, and it is then outputted

What is the algorithm of this flow chart ？

Moel

dasloet.on

hskalize

mox.phd

Zouolopauu

pe(i-1;icn;i+)

Srpay(lmue)

n ti

slax

？

yes

de

doo10pu

pinhrrox

stop

雪

B Experiment Details

B.1 Data sampling strategy.

We maintain different dataloaders for image-text data and pure text data, sampling them in a weighted manner. The pure language dataloader is sampled with a fixed weight of 0 . 1 and the image-text dataloader is sampled with a weight of 0 . 9 .

Within each dataloader, the training data comes from multiple sources, and we sample them in a weighted manner. In detail, for K datasets and the data number of the k th dataset is n k , its weight w k is min (100 , n k // 1000) and the normalized weight is ˆ w k = w k / ∑ w k . For each get _ data operation within the dataloader, we use weighted sampling to choose a dataset and randomly choose a training sample from it.

C Broader Impacts

On the positive side, our model introduces a novel solution that significantly enhances the ability of large language models to comprehend high-resolution images. This innovative approach is expected to be highly beneficial for the research community. It paves the way for new explorations and discoveries in the field of MLLM and image processing. The potential applications of this model are vast.

Moreover, we have plans to make our model open-source. By sharing our work with the public, we aim to foster an environment of shared learning and progress. Users, researchers, and developers can utilize our model, adapt it to their needs, and even contribute to its further development. This open-source approach will accelerate the pace of innovation and bring about more rapid advancements in the field.

On the negative side, like any powerful tool, there is a potential for misuse of our model. There is a risk that individuals with malicious intent may exploit the capabilities of the model for unethical or harmful purposes. This is a challenge that we, as researchers, must acknowledge and address. However, it is also crucial for users and the wider community to use our model responsibly and ethically. We believe that through collective effort, we can mitigate these risks and ensure that the benefits of our model are realized while minimizing potential harm.

D Limitation

In this paper, we study the influence of training resolution in a wide range, from HD-9 to 4KHD (HD-25 + HD-55). Our results show that high-resolution OCR-related tasks rely on the training resolution heavily, and get significant performance gains with increased resolution. Till our largest setting 4KHD, its gain is not saturated. If we keep increasing the resolution, it may get better results. However, due to computational constraints, we failed to fully explore the potential improvements from further increasing training resolution, and we have to leave it as future work.

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We write the paper carefully and confirm that the main claim in the abstract and introduction accurately reflects the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We mentioned the limitation of not fully exploring the upper bound of improvement by increasing training resolution, due to the computational constraints, and we would leave it to future work.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: We do not have theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The training details and datasets are provided in the paper, we also plan to open-source our model.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We plan to open-source the code and data after the final decision of the paper.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have provided the details in Sec.3

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: It is computationally expensive to report the error bars.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have provided it in Sec 3.4

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research conforms with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We have discussed it in the appendix.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our work focus on high-resolution image understanding and does not has such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All the assets are properly cited in the paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We provide a new model with the capability to understand high-resolution images, and we have provided the details of the model in the paper.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a large vision-language model named InternLM-XComposer2-4KHD, which is designed to process images up to 4K HD resolution. The model incorporates a dynamic resolution adaptation mechanism that maintains the aspect ratios of input images, thus enhancing the model's ability to understand complex image-text interactions. This approach allows for scalable resolution adjustments during training, supporting a wide range of resolutions. The model even outperforms well-known GPT-4V and Gemini Pro in 10 out of 16 benchmarks.\n\nSTRENGTHS:\n1. This paper introduces a substantial enhancement in the field of large vision-language models. It demonstrates significant performance improvements across multiple benchmarks, outperforming well-known models such as GPT-4V and Gemini Pro, thereby potentially exerting a broad impact on the field.\n\n2. The dynamic resolution design adopted in this study is particularly noteworthy. It enables the model to adapt to a variety of image resolutions while preserving critical detail.\n\n3. The writing in the paper is commendable, with informative charts and examples that effectively showcase the model's capabilities.\n\nWEAKNESSES:\n1. The paper frequently mentions training with 4K resolution, yet it lacks optimizations for accelerating high-resolution training. The heavy reliance on substantial computational resources limits the broader applicability of the proposed methods.\n2. There is a lack of analysis regarding the inference efficiency of the model. While the paper validates that high-resolution training significantly enhances performance for text-based vision-language tasks, it does not adequately address the computational costs incurred by high-resolution inputs. This omission makes it challenging to assess whether the additional computational expenses are justified by the performance gains.\n3. The introduction repetitively emphasizes adaptability to 4K resolution without providing clear problem statements or motivational examples. For instance, the discussion on \"variability in patch layouts\" (Line 52) lacks an intuitive explanation, and it would benefit from specific examples illustrating this variability. Besides, the paper highlights at Line 56 a critical aspect concerning the model's performance at resolutions beyond 4K with 'even when...'. It remains unclear whether this suggests that traditional methods trained at 4K resolution might fail to gain additional performance at lower resolutions. Clarifying this ambiguity is crucial for understanding the challenges the study addresses, reducing the need for readers to infer details solely from experimental results.\n4. This work notably advances vision-language understanding by enhancing traditional OCR capabilities. However, it prompts inquiries about the model's proficiency with complex visual content. The examples presented do not conclusively demonstrate the model's ability to accurately interpret intricate visuals such as hand-written mathematical formulas, circuit diagrams, or chemical structures. These types of content demand detailed parsing for complex problem-solving, an aspect not fully illustrated by the simpler cases in the appendices. While these considerations may exceed the current study's scope, they do not impact the overall evaluation for acceptance.\n\nQUESTIONS:\n1. The substantial computational resources required for training at 4K resolutions and the lack of detailed analysis on inference efficiency are main concerns.\n2. The paper needs clearer problem statements and motivational examples.\n3. The paper demonstrates enhancements in traditional OCR capabilities but leaves some uncertainty regarding the model's ability to process more complex visual content. If the authors can offer further insights or evidence, it would significantly strengthen the paper’s impact and might merit a higher evaluation.\n\nLIMITATIONS:\nThe paper addresses its limitations and potential negative societal impacts in the appendices. The explanation of limitations lacks depth. For more detailed concerns, please refer to the questions section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-06-12"}

comments: SUMMARY:
The paper presents InternLM-XComposer2-4KHD, a high-resolution MLLM that performs better than existing methods on various benchmarks.

STRENGTHS:
1. The paper is well written and easy to understand.
2. The performance of  InternLM-XComposer2-4KHD is good, which is valided on various benchmarks.

WEAKNESSES:
1. The paper has limited novelty. The high-resolution solution is not new. Starting from Monkey, there are a lot of works that adopt the similar idea, e.g., LLaVA-NeXT and InternVL-1.5.  Compared to these methods, the technical contribution of the paper is trivial. 


2. From the experimental results, I cannot tell whether the proposed high-resolution solution is better than other solutions. Since the setting of the paper are not comparable to others,  it seems that the data plays a more significant role. Maybe more comparisons or ablations should be provided.


3. There still be a lot of MLLMs that have similar settings to  IXC2-4KHD, e.g., InternVL-1.5.  Why not compare with these models in Tab 3-4.

4. The implementation details are not clear. It's still confusing for me that how much data is used for training.   In Tab 3, the previsous SoTA LLaVA-NeXT uses less than 2M data. So It's not clear whether the comparison is fair.

QUESTIONS:
My main concern is the novelty and the experimental setting. The main idea has been widely explored in previous works.

LIMITATIONS:
See weakness.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to explore the high resolution scenes of multimodal large language models. The authors find that the performances are largely improved when the model is equipped with 4K resolution. This finding will greatly inspire subsequent research works, which is of great value to the research community. The experimental results in the popular multimodal datasets, such as MME and MathVista, are impressive. The proposed dynamic resolution and automatic patch configuration will become a common approach, due to its simplicity and effectiveness.\n\nSTRENGTHS:\n1. The motivation sounds good. High resolution is the inevitable development direction of multimodal large language models, and we urgently need to know what effects high resolution will bring to model training and performance. This paper stands on a high ground.\n\n2. The method looks simple yet effective. Dynamic resolution and automatic patch configuration could not only maintain the aspect ratio of the image, but also indirectly enlarge the resolution of the image. \n\n3. The experimental results on many datasets are impressive. The proposed method achieves SOTA results in 6 of the 16 benchmarks with only 8B parameters.\n\n4. The writing of this paper is good, and the structure is easy to follow.\n\nWEAKNESSES:\n1. Considering that high resolution is a very meaningful direction, it is suggested that the authors provide more details of the training to help other researchers, such as the loss figure of the training. Because training at high resolution is expensive, releasing more intermediate results from training will help researchers conduct further studies.\n\n2. The current method seems to largely increase the number of visual tokens while increasing the resolution. One question is whether it is reasonable to use so such many visual tokens for ONE image. I would love to hear the authors' thoughts on this point. This is an open question.\n\n3. It is suggested that the authors could enlarge the LLM parameters synchronously in subsequent studies if conditions permit, which will make the study more complete. It seems that image resolution and LLM parameters are the two most important aspects affecting model performance, in addition to the training data of course.\n\n4. I am curious about the generalization performance of the proposed model on multimodal video evaluation dataset, such as Video-MME [1], which applies to both image MLLMs, i.e., generalizing to multiple images, and video MLLMs. The sub-figures in the split-patch strategy has some similarities with video frames. It would be great if the authors could update the results of the Video-MME into the paper.\n\n[1] Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis. Arxiv, 2024.\n\nQUESTIONS:\nNone\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-14"}