InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD

Authors: xiaoyi  dong
pan  zhang
yuhang  zang
yuhang  cao
bin  wang
linke  ouyang
songyang  zhang
haodong  duan
wenwei  zhang
yining  li
hang  yan
yang  gao
zhe  chen
xinyue  zhang
wei  li
jingwen  li
wenhai  wang
kai  chen
conghui  he
xingcheng  zhang
jifeng  dai
yu  qiao
dahua  lin
jiaqi  wang
min  benchmark
max  median
  infovqa
  docvqa
  ocrbench
  hallbench
  qbencht
  chartqa
  mmvet
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-09 15:13:07

Abstract: The Large Vision-Language Model (LVLM) field has seen significant advancements, yet its progression has been hindered by challenges in comprehending fine-grained visual content due to limited resolution. Recent efforts have aimed to enhance the high-resolution understanding capabilities of LVLMs, yet they remain capped at approximately 1500 $\times$ 1500 pixels and constrained to a relatively narrow resolution range. This paper represents InternLM-XComposer2-4KHD, a groundbreaking exploration into elevating LVLM resolution capabilities up to 4K HD (3840 × 1600) and beyond. Concurrently, considering the ultra-high resolution may not be necessary in all scenarios, it supports a wide range of diverse resolutions from 336 pixels to 4K standard, significantly broadening its scope of applicability. Specifically, this research advances the patch division paradigm by introducing a novel extension: dynamic resolution with automatic patch configuration. It maintains the training image aspect ratios while automatically varying patch counts and configuring layouts based on a pre-trained Vision Transformer (ViT) (336 $\times$ 336), leading to dynamic training resolution from 336 pixels to 4K standard. Our research demonstrates that scaling training resolution up to 4K HD leads to consistent performance enhancements without hitting the ceiling of potential improvements. InternLM-XComposer2-4KHD shows superb capability that matches or even surpasses GPT-4V and Gemini Pro in 10 of the 16 benchmarks.

Main paper: 
 introduction: 
 In recent years, the progress in Large Language Models (LLMs) (73; 92; 93; 39; 91; 10; 78; 29; 21) has provoked the development of Large Vision-Language Models (LVLMs). These models have demonstrated proficiency in tasks such as image captioning (17; 14) and visual-question-answering (VQA) (57; 31; 33; 107) . Nevertheless, due to their limited resolution, they struggle with processing images containing fine details, such as charts (68) , tables (87) , documents (70) , and infographics (69) . This limitation constrains their practical applicability in real-world scenarios.Recent advancements have aimed at enhancing the resolution of Large Vision-Language Models (LVLMs). Some approaches (66; 36; 97; 48) involve adapting high-resolution vision encoders 91) matches or even surpasses GPT-4V (74) and Gemini Pro (90) in 10 of the 16 benchmarks. (b) Image resolution statistic of 16 benchmarks. We report the minimum (Min), median, and maximum (Max) image area (resolution). Both the inter-/intra-benchmark resolution diversity are large, and we sort them by the maximum resolution.directly. However, the Vision Transformer (ViT) architecture falls short when dealing with images of varying resolutions and aspect ratios, thereby restricting its ability to handle diverse inputs effectively. Alternatively, some methods (50; 59; 37; 51; 99; 55; 46) maintain the vision encoder's resolution, segmenting high-resolution images into multiple low-resolution patches. Yet, these methods are constrained by an inadequate resolution, typically around 1500 × 1500, which does not satisfy the demands of daily content, e.g., website screenshots (85) , document pages (70) , and blueprints (69) . Furthermore, they are confined to either a few predefined high-resolution settings (36; 97; 48; 50; 51; 55; 46; 66; 59) or a limited range of resolutions (101; 37; 99) , thereby restricting their utility across a variety of applications.In this work, we introduce InternLM-XComposer2-4KHD, a pioneering model that for the first time expands the resolution capabilities of Large Vision-Language Models (LVLMs) to 4K HD and even higher, thereby setting a new standard in high-resolution vision-language understanding. Designed to handle a broad range of resolutions, InternLM-XComposer2-4KHD supports images with any aspect ratio from 336 pixels up to 4K HD, facilitating its deployment in real-world contexts.InternLM-XComposer2-4KHD follows patch division (50; 46) paradigm and enhances it by incorporating an innovative extension: dynamic resolution with automatic patch configuration. To be specific, scaling the resolution of Large Vision-Language Models (LVLMs) to 4K HD and even higher standard is far beyond merely increasing the number of patches. It involves a nuanced approach to overcoming specific challenges: (1) Dynamic Resolution and Automatic Patch Configuration: Addressing the scarcity of high-resolution training data, our framework introduces a strategy that dynamically adjusts resolution alongside an automatic layout configuration. During training, it maintains the original aspect ratios of images while adaptively altering patch (336 × 336) layouts and counts. This results in a training resolution that exceeds the original image resolutions, reaching up to 4KHD, addressing the shortfall of high-resolution data. (2) Handling Variability in Patch Layouts: Despite the apparent simplicity of dynamic resolution training, the variability in patch layouts can heavily confuse LVLMs. To mitigate this, we introduce a newline token after each row of patch tokens to clearly delineate patch layouts, reducing training ambiguity and significantly boosting performance.(3) Inference Beyond 4K Resolution: Our observations reveal that, even when trained on images up to 4K resolution, the model can achieve additional performance improvements during inference by processing images at higher resolutions.The image is a vibrant and colorful infographic that provides an overview of 7 graphic design trends that are expected to dominate in 2021. The infographic is divided into 7 sections, each representing a different trend.At the top, there's a section titled "Muted Color Palettes", which introduces the concept of muted colors as a trend for 2021. This is followed by a section on "Simple Data Visualizations", highlighting the goal of making complex data easy to understand. The third section focuses on "Geometric Shapes Everywhere", showcasing how designers are using geometric shapes in their designs. The fourth section delves into "Flat Icons and Illustrations", explaining how brands are using flat icons and illustrations in their social media graphics, website design, and more. The fifth section introduces "Classic Serif Fonts", discussing how these fonts are seen as classic, elegant, and trustworthy. The sixth section explores "Social Media Slide Decks", illustrating how slide decks have become a visual way of communicating longer messages on Instagram and LinkedIn. Finally, the seventh section discusses "Text Heavy Videos", predicting that people will likely continue to work remotely well into 2021, leading to an increase in video content that uses text on-screen to communicate messages.Overall, the infographic provides a comprehensive overview of the 7 graphic design trends that are expected to shape the design landscape in 2021. What is the detailed explanation of the third part?The third part of the image is titled "Geometric Shapes Everywhere". It explains that last year, designers used a lot of flowing and abstract shapes in their designs. However, this year, they have been replaced with rigid, hard-edged geometric shapes and patterns. The hard edges of a geometric shape create a great contrast against muted colors. Furthermore, scaling the training resolution up to 4K standard results in a consistent improvement in performance, highlighting the potential for training even beyond 4K resolution. This underscores the capacity for further enhancing model capabilities and suggests a promising trajectory for advancing the frontiers of high-resolution image processing within the domain of large vision-language models. We evaluate our InternLM-XComposer2-4KHD on 16 diverse benchmarks spanning various domains, including 5 challenging OCR datasets (InfographicVQA(69), DocVQA (70) , OCRBench (58) , TextVQA (87) , and ChartQA(68)). Compared to previous open-source LVLM models and closedsource APIs, our approach achieves SOTA results in 6 of 16 benchmarks, demonstrating competitive performance despite only 7B parameters. As shown in Figure 1 , InternLM-XComposer2-4KHD even surpasses the performance of GPT4V (74) and Gemini Pro (90) across ten benchmarks. Notably, our method exhibits excellent performance on 5 challenging OCR datasets, over existing open-source LVLMs by a substantial margin.Eating 2 to 2% cups (4 to 5 servings) daily of fruits and vegetables may reduce your risk for cardiovascular disease, including heart attack and stroke.Examples of a single serving of fruit: 1 medium apple, 1 small banana, 1 medium avocado, 4 large strawberries. Examples of a single serving of vegetables: 5 -8 broccoli florets, 2/3 large sweet potato, 6 baby carrots, 1 cup of raw spinach.Adding a variety of produce to your diet has a number of health benefits. It helps you: Manage your weight, Control your blood pressure, Support healthy digestion, Reduce the risk of some cancers, such as colon cancer, Reduce the risk of chronic health problems, such as diabetes. 
related works: 
 Large Vision-Language Models (LVLMs). Large Language Models (LLMs) (9; 76; 73; 23; 41; 92; 93; 39; 91; 108; 6; 78; 10) have gained significant attention due to their impressive performance in various language-related tasks such as text generation and question answering. Following this enthusiasm, recent Large Vision-Language Models (LVLMs) have emerged (74; 19; 16; 18; 28; 32; 113; 25; 110; 7; 47; 77; 102; 4) , combining LLMs with vision encoders (79; 109; 89) to leverage the complementary strengths of language and vision modalities. By fusing textual and visual representations, LVLMs can ground language in visual contexts, enabling a more comprehensive understanding and generation of multimodal content (14; 20; 51; 5; 95; 27; 11; 60) .LVLMs for High-Resolution Understanding. Large Vision-Language Models (LVLMs) often employ CLIP-ViT as the visual encoder for vision-dependent tasks. However, the visual encoder's reliance on low resolutions, such as 224 × 224 or 336 × 336 pixels, limits its effectiveness for highresolution tasks like OCR and document/chart perception. To enhance high-resolution understanding, recent works have primarily employed the following strategies: (1) High-resolution (HR) visual encoders or dual encoders catering to high-resolution (HR) and low-resolution (LR) inputs (66; 97; 36; 48) . For instance, Vary (97) introduces a new image encoder supporting HR inputs, which are then concatenated with LR embeddings from the original CLIP visual encoder. Similarly, CogAgent (36) and Mini-Gemini (48) also separate HR and LR images using distinct vision encoders, subsequently merging their features using a cross-attention module. In contrast, our approach offers a more simplified solution and shows advantages for varying resolutions and aspect ratio inputs. (2) Cropped image patches (50; 59; 99; 101; 37; 51; 46) . For example, Monkey (50) employs sliding windows to segment images into patches, subsequently processing them with LoRA fine-tuning. TextMonkey (59) further proposes shifted window attention and token resampler to consider the connections among different patches. Fuyu (7) eliminates the need for the image encoder by directly processing a raw image patch sequence. These approaches are confined to either a few predefined high-resolution settings (36; 97; 48; 50; 51; 55; 46; 66; 59) or a limited range of resolutions (37; 99) . Conversely, our method devises a dynamic resolution and automatic path configuration strategy to support the scaling from 336 pixels to 4K resolution, and the maximum resolution is larger than previous approaches (e.g., 1.5k for Monkey (50) and 1.2k for UReader (101)). For the first time, our approach discussed the challenges and solutions for handling variability in image feature patch layouts, ensuring effective training with dynamic high resolutions.LVLMs for Document Understanding. Document understanding involves analyzing and comprehending various digital documents, such as figures, tables, and academic papers. Many document understanding tasks require models to handle high-resolution inputs, complex layouts, various aspect ratios, and diverse document formats. To enhance the capabilities of LVLMs for document understanding, several works have collected and constructed high-quality document instruction tuning data, including LLaVAR (112) , mPLUG-DocOwl (100) and TGDoc (96) . DocPediaDocPedia (30) processes document inputs in the frequency domain. Some previous works have improved document understanding ability by designing special modules for high-resolution inputs, such as HR and LR encoders (36; 97) or cropped image patches (101; 59; 99) . Our InternLM-XComposer2-4KHD first scales to 4K resolution inputs and demonstrates strong document understanding ability on OCRrelated benchmarks. Also, our approach also achieves comparable results to state-of-the-art opensourced LVLMs on other general LVLM benchmarks like perception and reasoning (61; 57; 33; 15) . The model architecture of InternLM-XComposer2-4KHD mainly follows the design of InternLM-XComposer2(27) (XComposer2 / IXC2 in the following for simplicity ), including a light-weight Vision Encoder OpenAI ViT-Large/14, Large Language Model InternLM2-7B, and Partial LoRA for efficient alignment.1. Fresh, frozen, canned and dried all count toward your daily servings, so you have plenty of options. 2. Always reserve half of your plate for fruits and vegetables. 3. Choose whole fruits and vegetables. 4. Try to eat a variety of vegetables, instead of the same thing all the time. 5. Look for fruit packed in its own fruit juice and 100% vegetable juice, with no added sugar. 6. Look for low/no-sodium options for canned vegetables, and 100% vegetable juice. 
high-resolution input.: 
 Dynamic Patch Configuration. Utilizing a static input image size for processing high-resolution images, particularly those with varying aspect ratios, is neither efficient nor effective. To overcome this limitation, we introduce a dynamic patch configuration approach via image partitioning, as shown in Figure 3 . Our method strategically segments the image into smaller patches, while maintaining the integrity of the original image's aspect ratio.Given a maximum patch number H, the image x with size [h, w] is resized and padded to the new image x with size [p h × 336, p w × 336]. This process is subject to the following constraints:EQUATIONhere p w and p h represent the number of patches in each row and column, respectively. We then split the x into p h × p w non-overlapped patches. Each patch is a small image with 336 × 336 size and we treat these patches as individual inputs for the ViT.In the following, we use 'HD-H' to represent our high-resolution setting with the constraint of H patches. For example, the 'HD-9' allows up to 9 patches, including a range of resolutions such as 1008 × 1008, 672 × 1344, 336 × 3024, etc.Global-Local Format. For each input image, we present it to the model with two views. The first is the global view, where the image is resized to a fixed size (in our case, 336 × 336). This provides a macro understanding of the image. Empirically, we have found this to be crucial for the LVLM to correctly understand the image. The second view is the local view. We divide the image into patches using the previously mentioned Dynamic Image Partition strategy and extract features from each patch. Following feature extraction, the patches are reassembled into a large feature map. The feature map is then flattened to the final local features after a straightforward token merging process.Patch Layout Indicator. Given that an image will have a dynamic patch layout in our method, the number of tokens for each row can vary across different images. This variation will confuse the LVLM, making it difficult to determine which tokens belong to the same row of the image and which ones belong to the next row. This confusion hinders the LVLM's ability to understand the 2D structure of the image, which is crucial for comprehending structural image content such as documents, charts, and tables. To address this issue, we introduce a learnable newline ('\n') token at the end of each row of the image features before the flattening. Finally, we concatenate the global and local views, inserting a special separate ('sep') token between them to distinguish the two views.During the pre-training phase, the LLM is frozen while both the vision encoder and Partial LoRA are fine-tuned to align the visual tokens with the LLM following XComposer2 (27) . The pre-training data (35) , Flicker (103) , MMC-Inst(54), RCTW-17 (84) , CTW (106) , LSVT (88) , ReCTs (111) , ArT (22) Table 2: Supervised Fine-Tuning Datasets. We collect data from diverse sources to empower the model with different capabilities. The image resolution is also different for different tasks. mainly follow the design in XComposer2 which is curated with three objectives in mind: 1) general semantic alignment, 2) world knowledge alignment, 3) vision capability enhancement. In this paper, we focus on high-resolution and structural image understanding. So we collected OCR and chart data from diverse sources to enhance this specific capability, as shown in Table .1.In practice, we employ the OpenAI CLIP ViT-L-14-336 as the vision encoder. We keep the ViT resolution as 336 × 336 while adopting Dynamic Patch Configuration to handle higher-resolution images. For pretraining, we use the 'HD-25' configuration, which resizes the input image to a random larger resolution to generate more patches, with the constraint that the total number of patches does not exceed 25. For each image or patch tokens, the image token number is decreased to 1/4 with a simple merge operation. We concatenate the nearby 4 tokens into a new token through the channel dimension, then align it with the LLM by an MLP. The 'sep' and '\n' tokens are randomly initialized.For the Partial LoRA, we set a rank of 256 for all the linear layers in the LLM decoder block. Our training process involves a batch size of 4096 and spans across 2 epochs. The learning rate linearly increases to 2 × 10 -4 within the first 1% of the training steps. Following this, it decreases to 0 according to a cosine decay strategy. To preserve the pre-existing knowledge of the vision encoder, we apply a layer-wise learning rate (LLDR) decay strategy, and the decay factor is set to 0.90.After pre-training, we empower the model to understand high-resolution images and solve diverse challenges. Different from conventional perception tasks (e.g., VQAv2, GQA) which typically answer questions based on the noticeable object in the image. OCR-related tasks depend on a detailed understanding of text within a high-resolution image. For instance, in InfoVQA, the length of the longer side of 50% of the images exceeds 2000 pixels. Low-resolution inputs can distort the dense text information, causing the model to fail in its understanding. However, we have observed a resolution saturation phenomena with perception tasks, where a higher resolution makes minor gains.To address this, we introduce a mixed-resolution strategy for more efficient training. For tasks requiring high resolution, we employ the 'HD-55' setting during training. This allows for the input of 4K (3840 × 1600) images without necessitating additional image compression. These tasks are referred to as the HD-OCR QA tasks in Table 2 . For the other tasks, we apply the 'HD-25' resolution setting for them. As in pre-training, we adopt the dynamic-resolution strategy during SFT, images are resized to fall within a range between their original size and the size specified by the 'HD' setting. This dynamic approach enhances the robustness of the LVLM against differences in input resolution, thereby enabling the LVLM to utilize a larger resolution during inference. For instance, we have observed that using the 'HD30' setting yields better results on most OCR-related tasks when the LVLM is trained under the 'HD25' setting.In practice, we jointly train all the components with a batch size of 2048 over 3500 steps. Data from multiple sources are sampled in a weighted manner, with the weights based on the number of data from each sourced (See Appendix B.1 for more details). As the 'HD-55' setting has double image tokens than the 'HD-25', we adjust the data loader to enable different batch sizes for them and adjust their weight accordingly. The maximum learning rate is set to 5 × 10 -5 , and each component has its own unique learning strategy. For the vision encoder, we set the LLDR to 0.9, which aligns with the pretraining strategy. For the LLM, we employ a fixed learning rate scale factor of 0.2. This slows down the update of the LLM, achieving a balance between preserving its original capabilities and aligning it with vision knowledge. It takes almost 40 hours with 256 A100 GPUs.In many communities, fruits and vegetables are limited/expensive and only available in corner stores, convenience stores, bodegas and gas stations.Lack of access can profoundly impact diet. Here's how: Contact your city and state leaders to let them know what food access is in your community.Find ways to spread the word about nutrition assistance programs, such as SNAP, WIC and school meals.Create a petition for more variety, improve affordability and advocate for better signage/placement.Meet with an after-school or daycare program representative to discuss serving more fruits and vegetables for snacks. Organize a letter-writing campaign and set up a meeting with state leaders.For example, ask for funding to host a farmers market in an underserved community. Sign up for "You're the Cure" and send a note to your Congressperson advocating for healthier meals at school.It's American Heart Month! Share this infographic with your family, friends and network today. You can carry out all of these actions or just a few. But whichever actions you choose, they pave the way for greater access to nutritious food for your community.The HD image (816 x 3813) This flow chart outlines the process of ordering a book from Amazon. Here's a step-by-step analysis:1. Start: The process begins with the user initiating the order. 2. Enter the website from the browser: The user navigates to the Amazon website.3. Registered?: The system checks if the user is already registered. If not, they are prompted to register. 4. Sign in: If the user is already registered, they are prompted to sign in. 5. On home page search for book of your choice: The user searches for the desired book on the home page. The flow chart in the image is an algorithm for finding the maximum element in an array. Here is a detailed explanation of the algorithm:1.Declare an integer array having size m. 2.Input the m numbers into the array. 
experiments: 
 In this section, we validate the benchmark performance of our InternLM-XComposer2-4KHD (IXC2-4KHD in the following for simplicity) after supervised fine-tuning.In Table 3 High-resolution Understanding Evaluation. Then we compare IXC2-4KHD with models that are specifically designed for high-resolution understanding tasks. We report the results of 5 highresolution benchmarks in Table 5 , as a general LVLM, IXC2-4KHD shows superb performance on these tasks and outperforms competitors with a large margin. For example, IXC2-4KHD gets 68.6% on InfographicVQA, surpassing recent DocOwl 1.5 with +17.9%. For the OCRBench, IXC2-4KHD gets 67.5%, outperforms CogAgent with +8.5%.High-Resolution Training is Critical for HD-OCR tasks. We study four resolution settings: HD-9 (1561 image tokens at most, we simply the statement in the following), HD-16 (2653 tokens), HD-25 (4057 tokens), and 4KHD (8737 tokens). Here we report the validation set of InfoVQA, DocVQA, and TextVQA, test set of ChartQA and AI2D, MMBench EN-Test, and a 2k subset of SEEDBench (we denote it as SEED * ). In the following, we report results on the above benchmarks by default.As illustrated in Fig. 4 , we note a significant improvement in the HD-OCR tasks as the resolution increases. For instance, the model achieves only a 50.5% score on the InfographicVQA with the HD-9 setting. However, when we switch to the HD-16 setting, we observe a performance gain of +10.2%. The performance continues to improve as the resolution increases, with saturation not observed even for the 4KHD setting. Due to computational constraints, we defer the exploration of the upper bound of improvement to future work. In terms of other OCR-related tasks, the performance gain attributable to increased resolution is relatively minor. For the perception-related benchmarks, performance is saturated on the resolution that only has negligible difference between the four settings.Higher Inference Resolution Leads to better results on Text-related Tasks. An intriguing observation from our experiments is that our model, when inferring with a higher resolution, tends to yield improved results on text-related tasks. We present the results of HD-9, HD-16, and HD-25 in Table 6 . For instance, IXC2-HD9 achieves a 50.5% score on InfographicVQA. When we infer with HD16, we see a performance gain of +8.1%, without additional training. Similar improvements are also observed with IXC2-HD16 and IXC2-HD25. We posit that the dynamic image token length used in training enhances the robustness of the LVLM, leading to better results when the text in the image is more 'clear' in the higher-resolution input. Conversely, the results on ChartQA consistently degrade under this setting. This could be due to the model becoming confused about the chart structure when the resolution is increased. Additionally, similar to the observation from Figure 4 , the impact of resolution on perception-related benchmarks appears to be quite minor.The Role of Global-View. We first examine the impact of the global view in our Global-Local Format. As indicated in Table 8 (a), we find that the global view is essential for the LVLM to accurately comprehend the input image. When it is removed, the model performs worse across all benchmarks.For instance, the model experiences a -4.4% drop in performance on the MMBench EN-Test without the global view. We contend that the global view offers a general macro understanding of the image, which the model struggled to derive from the large number of tokens in the local view.The Role of the Newline Token. We incorporate a special newline token at the end of each row of the image features before the flattening operation. This token serves as an indicator of the image's 2D structure. We examine its impact on both the HD-9 and 4KHD strategies in Table 7 (a). When a fixed high-resolution strategy HD-9 is employed, we observe that the benefit derived from the newline token is minor. This could be attributed to the LVLM's ability to handle limited differences in image ratios after training. However, when we implement a more challenging 4KHD (HD-25 + HD-55) strategy, which exhibits significant diversity in both image ratio and token number, the LVLM demonstrates a notable decline in performance on OCR-related tasks without the newline indicator. This finding supports our hypothesis that the LVLM struggles to comprehend the shape of the image when the image tokens are directly flattened into a 1D sequence. The newline token can assist the model in better understanding the structure of the image.Influence of Token Merging Operation. In practice, we employ a simple merging operation that concatenates four adjacent tokens along the channel dimension. We have found this approach to be effective in reducing the number of image tokens efficiently. Here we study the influence of different token-merging operations under the 4KHD setting. In Table 7 (b), we study two additional strategies: Re-Sampler(5) and C-Abstractor (12) , with their default setting and the same compressing rate 0.25, i.e., reducing an image with 576 tokens to 144 tokens. Results show that both concatenation and C-Abstractor work well and get similar results on most benchmarks, this observation is also consistent with the study in MM-1(71) that the influence of the connector is minor. However, the Re-Sampler performs worse than the other methods with a noticeable margin. We argue this is caused by the learnable queries used for gathering information requiring a great number of data for training, our pre-training data is somewhat lightweight for it to converge fully. Strategy-Level Comparison. For a fair comparison, we trained a new model with the identical architecture, training strategy, and dataset as our original model, but with one key modification: we adopted the high-resolution strategy from LLaVA-Next. We name it as IXC-LLaVA and compare it with 1) LLaVA-Next 0530 official results and 2) IXC2-4KHD under HD-9/16 setting. The results in Table .9 demonstrate that IXC-LLaVA achieves promising performance across all six benchmarks, leveraging the benefits of additional training data and advanced IXC architecture design. However, it is outperformed by IXC2-4KHD HD-9, which utilizes fewer image tokens yet yields better results. This fair comparison underscores the efficiency and effectiveness of our proposed high-resolution strategy, highlighting its advantages over the LLaVA-Next approach.Inference Efficiency Analysis. Our model processes high-resolution images with numerous image tokens, and here we study its inference efficiency in real-world usages. The model inference process consists of two stages: encoding the prefix (model input) and autoregressively decoding new tokens (model output). Correspondingly, the inference efficiency considers two parts: time to encode the prefix and speed to decode each token. Here we report the prefix encoding time and per-token decoding speed under different HD settings. We test the speed with a Nvidia-A100 80G. With the results in Table 10 , we have three observations: 1) Prefix encoding time increases linearly with the number of prefix tokens. 2) Decoding speed remains relatively constant, regardless of prefix length, thanks to optimizations on transformers from research communities and companies, including kv-cache and flash-attention. 3) When generating 2048 tokens, total inference time usage is nearly identical across HD9 to HD55, as encoding time is much smaller. Based on the above analysis, we believe the inference efficiency of our model is acceptable. Besides, we believe some targeted designs can further improve efficiency, while our paper focuses on enabling LVLM to understand high-resolution images with a general and effective solution, and we would leave the efficiency exploration in future work. 
conclusion: 
 In this paper, we propose the InternLM-Xcomposer2-4KHD that exceeds the performance of previous open-source models on OCR-related tasks and also achieves competitive results on general-purpose LVLM benchmarks. Thanks to our dynamic resolution and automatic patch configuration, our model supports a maximum training resolution of up to 4K HD. We also integrate a global view patch to support the macro understanding and a learnable newline token to handle the various input image resolutions. Our model's performance continues to improve as the training resolution increases for HD-OCR tasks. Notably, we do not observe any performance saturation even for the 4KHD setting, and we have not explored the upper bound due to the computational burden increasing with higher-resolution inputs. In future work, we plan to explore efficient solutions for accurate LVLM training and inference, enabling our model to handle even higher resolutions while maintaining computational efficiency.A More examples chat with our InternLM-XComposer2-4KHDGive an in-depth description of the visual.The image is an infographic from Public Health England that explains the concept of contact tracing in the context of preventing the spread of coronavirus (COVID-19). Contact tracing is a fundamental part of outbreak control used by public health professionals around the world to prevent the spread of infections.The infographic is divided into several sections, each providing a different aspect of the contact tracing process.The first section explains that a contact is assessed to see how close they were to a confirmed case, which could include someone living with the case, or someone who has been in direct contact with them or their bodily fluids. Following this assessment, the contacts are categorised into low or high risk and provided with advice on what they should do.The second section explains that if a contact is at high risk of infection, they may be asked to self-isolate, remaining in their home for 14 days and staying away from work, school or public places. If a contact is at low risk, they don't require self-isolation.The third section explains that if the person develops symptoms, they would be tested and provided with specialist care if they have the novel coronavirus (COVID-19).Finally, the infographic emphasizes the importance of recovery time, as contact displays no symptoms of coronavirus (COVID-19) the patient makes a full recovery, they can go back to daily lives as normal and pose no risk to others.Image size: 986 x 2560 Your best defense is a healthy lifestyle filled with physical activity and nutritious meals, while cutting down or avoiding drinking and smoking.The Current Trend 75% of the US population is low in its fruit and vegetables consumption. 1/10 adults meets the recommendation for fruit and vegetable consumption. Men and young adults in general eat the fewest fruits and vegetables compared to any other group.3 Steps to Eating Healthy 1. Add produce of various colors to your diet to diversify your produce consumption. 2. Use hacks to incorporate more veggies and fruits to your meal, and throughout the day. 3. Take steps to improve access to fruits and vegetables for all.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification:We plan to open-source the code and data after the final decision of the paper.Guidelines:• The answer NA means that paper does not include experiments requiring code. • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We have provided the details in Sec.3Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: It is computationally expensive to report the error bars.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We have provided it in Sec 3.4Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: We have discussed it in the appendix.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [NA] Justification: Our paper does not involve crowdsourcing nor research with human subjects. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Initialize max as the first element of the array. 4.For each element in the array, starting from the second element: If the current element is greater than max, set max to the current element. 5.At the end of the loop, max will hold the maximum element in the array. 6.Output the maximum element.This algorithm iterates through the array, comparing each element to the current maximum. If a larger element is found, it becomes the new maximum. At the end of the loop, the maximum element is stored in the variable max, and it is then outputted What is the algorithm of this flow chart？The HD image (1888 x 3172) We maintain different dataloaders for image-text data and pure text data, sampling them in a weighted manner. The pure language dataloader is sampled with a fixed weight of 0.1 and the image-text dataloader is sampled with a weight of 0.9.Within each dataloader, the training data comes from multiple sources, and we sample them in a weighted manner. In detail, for K datasets and the data number of the k th dataset is n k , its weight w k is min(100, n k //1000) and the normalized weight is ŵk = w k / w k . For each get_data operation within the dataloader, we use weighted sampling to choose a dataset and randomly choose a training sample from it.On the positive side, our model introduces a novel solution that significantly enhances the ability of large language models to comprehend high-resolution images. This innovative approach is expected to be highly beneficial for the research community. It paves the way for new explorations and discoveries in the field of MLLM and image processing. The potential applications of this model are vast.Moreover, we have plans to make our model open-source. By sharing our work with the public, we aim to foster an environment of shared learning and progress. Users, researchers, and developers can utilize our model, adapt it to their needs, and even contribute to its further development. This open-source approach will accelerate the pace of innovation and bring about more rapid advancements in the field.On the negative side, like any powerful tool, there is a potential for misuse of our model. There is a risk that individuals with malicious intent may exploit the capabilities of the model for unethical or harmful purposes. This is a challenge that we, as researchers, must acknowledge and address. However, it is also crucial for users and the wider community to use our model responsibly and ethically. We believe that through collective effort, we can mitigate these risks and ensure that the benefits of our model are realized while minimizing potential harm.In this paper, we study the influence of training resolution in a wide range, from HD-9 to 4KHD (HD-25 + HD-55). Our results show that high-resolution OCR-related tasks rely on the training resolution heavily, and get significant performance gains with increased resolution. Till our largest setting 4KHD, its gain is not saturated. If we keep increasing the resolution, it may get better results. However, due to computational constraints, we failed to fully explore the potential improvements from further increasing training resolution, and we have to leave it as future work.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 


Decision: This work proposes a novel large vision-language model capable of handling a dynamic range of resolutions from 336 to 4K pixels. To achieve this, it combines various existing design ideas, including the use of global and local patches, learned new line tokens and token merging along with new ideas for dynamic layout prediction. The proposed method is shown to outperform the existing open source methods in various VLM tasks requiring inference with high resolution images, while maintaining accuracy on general purpose VLM tasks.

Reviewers raised concerns about the novelty of the proposed ideas used to construct the method; the lack of comparisons to several recent SOTA approaches and the inference and training speed of the method. Several of the reviewers' concerns were addressed in the rebuttal, while requested comparisons are to recent and unpublished works. While the AC agrees that several of the ideas used to build the proposed method are known from prior works, they feel that the current work does employ them in a novel manner to achieve scalable and scalable inference at various image resolutions. In terms of quality this work presents a significant step forward for open source models. Since the authors have promised to release the data and code as well, it will a valuable resource for the research community to advance open source models further along the dimension of processing high resolution images and will form a strong foundation for the community to build upon. How to process higher resolution images in LVLM is a significant and important unsolved problem. Hence, all things considered the AC leans towards accepting this work and recommends that the authors incorporate the changes that they have promised in the rebuttal into their final manuscript.

Reviews: review: paper_summary: paper_summary: This paper aims to explore the high resolution scenes of multimodal large language models. The authors find that the performances are largely improved when the model is equipped with 4K resolution. This finding will greatly inspire subsequent research works, which is of great value to the research community. The experimental results in the popular multimodal datasets, such as MME and MathVista, are impressive. The proposed dynamic resolution and automatic patch configuration will become a common approach, due to its simplicity and effectiveness.
strength_weakness: strength_weakness: 1. Considering that high resolution is a very meaningful direction, it is suggested that the authors provide more details of the training to help other researchers, such as the loss figure of the training. Because training at high resolution is expensive, releasing more intermediate results from training will help researchers conduct further studies.

2. The current method seems to largely increase the number of visual tokens while increasing the resolution. One question is whether it is reasonable to use so such many visual tokens for ONE image. I would love to hear the authors' thoughts on this point. This is an open question.

3. It is suggested that the authors could enlarge the LLM parameters synchronously in subsequent studies if conditions permit, which will make the study more complete. It seems that image resolution and LLM parameters are the two most important aspects affecting model performance, in addition to the training data of course.

4. I am curious about the generalization performance of the proposed model on multimodal video evaluation dataset, such as Video-MME [1], which applies to both image MLLMs, i.e., generalizing to multiple images, and video MLLMs. The sub-figures in the split-patch strategy has some similarities with video frames. It would be great if the authors could update the results of the Video-MME into the paper.

[1] Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis. Arxiv, 2024.
questions: questions: None
limitations: limitations: None
paper_summary: paper_summary: The paper presents InternLM-XComposer2-4KHD, a high-resolution MLLM that performs better than existing methods on various benchmarks.
strength_weakness: strength_weakness: 1. The paper has limited novelty. The high-resolution solution is not new. Starting from Monkey, there are a lot of works that adopt the similar idea, e.g., LLaVA-NeXT and InternVL-1.5.  Compared to these methods, the technical contribution of the paper is trivial. 


2. From the experimental results, I cannot tell whether the proposed high-resolution solution is better than other solutions. Since the setting of the paper are not comparable to others,  it seems that the data plays a more significant role. Maybe more comparisons or ablations should be provided.


3. There still be a lot of MLLMs that have similar settings to  IXC2-4KHD, e.g., InternVL-1.5.  Why not compare with these models in Tab 3-4.

4. The implementation details are not clear. It's still confusing for me that how much data is used for training.   In Tab 3, the previsous SoTA LLaVA-NeXT uses less than 2M data. So It's not clear whether the comparison is fair.
questions: questions: My main concern is the novelty and the experimental setting. The main idea has been widely explored in previous works.
limitations: limitations: See weakness.
paper_summary: paper_summary: This paper introduces a large vision-language model named InternLM-XComposer2-4KHD, which is designed to process images up to 4K HD resolution. The model incorporates a dynamic resolution adaptation mechanism that maintains the aspect ratios of input images, thus enhancing the model's ability to understand complex image-text interactions. This approach allows for scalable resolution adjustments during training, supporting a wide range of resolutions. The model even outperforms well-known GPT-4V and Gemini Pro in 10 out of 16 benchmarks.
strength_weakness: strength_weakness: 1. The paper frequently mentions training with 4K resolution, yet it lacks optimizations for accelerating high-resolution training. The heavy reliance on substantial computational resources limits the broader applicability of the proposed methods.
2. There is a lack of analysis regarding the inference efficiency of the model. While the paper validates that high-resolution training significantly enhances performance for text-based vision-language tasks, it does not adequately address the computational costs incurred by high-resolution inputs. This omission makes it challenging to assess whether the additional computational expenses are justified by the performance gains.
3. The introduction repetitively emphasizes adaptability to 4K resolution without providing clear problem statements or motivational examples. For instance, the discussion on "variability in patch layouts" (Line 52) lacks an intuitive explanation, and it would benefit from specific examples illustrating this variability. Besides, the paper highlights at Line 56 a critical aspect concerning the model's performance at resolutions beyond 4K with 'even when...'. It remains unclear whether this suggests that traditional methods trained at 4K resolution might fail to gain additional performance at lower resolutions. Clarifying this ambiguity is crucial for understanding the challenges the study addresses, reducing the need for readers to infer details solely from experimental results.
4. This work notably advances vision-language understanding by enhancing traditional OCR capabilities. However, it prompts inquiries about the model's proficiency with complex visual content. The examples presented do not conclusively demonstrate the model's ability to accurately interpret intricate visuals such as hand-written mathematical formulas, circuit diagrams, or chemical structures. These types of content demand detailed parsing for complex problem-solving, an aspect not fully illustrated by the simpler cases in the appendices. While these considerations may exceed the current study's scope, they do not impact the overall evaluation for acceptance.
questions: questions: 1. The substantial computational resources required for training at 4K resolutions and the lack of detailed analysis on inference efficiency are main concerns.
2. The paper needs clearer problem statements and motivational examples.
3. The paper demonstrates enhancements in traditional OCR capabilities but leaves some uncertainty regarding the model's ability to process more complex visual content. If the authors can offer further insights or evidence, it would significantly strengthen the paper’s impact and might merit a higher evaluation.
limitations: limitations: The paper addresses its limitations and potential negative societal impacts in the appendices. The explanation of limitations lacks depth. For more detailed concerns, please refer to the questions section.

Comments: comment: Thanks for the response, which has addressed most of my concerns. Since I am not an expert in this field, I carefully read other reviewers' comments and the literature mentioned by Reviewer kpFC. I noticed that the methods in this paper have many similarities to existing methods. Therefore, the paper should clarify these similarities in the method section. Considering that these designs are not original in this paper, I have to adjust my score accordingly.
Dear Reviewer 339k,

Thank you for the time and effort you have dedicated to reviewing our submission. Your appreciation for our work, along with the questions and suggestions you provided, has greatly helped us improve the quality of our work. We hope we have addressed the concerns raised in your initial reviews.

As the author-reviewer discussion period for NeurIPS 2024 only has two days, please let us know if you require additional information or clarification. We are ready to engage in further discussions to enhance and elevate our work.

Best regards and thanks,

Paper 3325 Authors
Thank you so much for appreciating the novelty and contribution of IXC2-4KHD to the community. We are also grateful for your recognition of IXC2-4KHD is distinguished from existing works. We appreciate your suggestions and will add more experimental details in our revision. Additionally, our data and code will be made public to benefit the community.
Thanks for your valuable feedback. We hope the following explanations may further answer your concerns:

1. Indeed, the context window of Vicuna is 4096 tokens. In the meantime, the LLaVA-NexT team has officially released their LLaVA-NexT-Mistral-7B model, which utilizes Mistral-based LVLMs and features a 32K context window. However, the challenge of high-resolution understanding remains unsolved in LLaVA-NexT-Mistral-7B, indicating that handling diverse resolutions up to 4K is still non-trivial.           
&nbsp;      
2. We would like to highlight that IXC2-4KHD contributes new insights to the community, enabling LVLMs to handle diverse resolutions from 336 to 4K. These insights are crucial for bridging the significant performance gap between the research community and proprietary, closed-source models. For example, IXC2-4KHD delivers a 10.8% gain on ChartQA and a 17.9% improvement on InfoVQA. For the first time, we demonstrate that LVLMs with only 8B parameters can also compete with GPT-4V on various high-resolution understanding benchmarks.    
&nbsp;   
On the technical aspect, we have acknowledged the previous works on patch division (Lines 43-44 in the paper) and would like to highlight our innovations in dynamic resolution with automatic patch configuration design, including our training and inference strategies. These innovations mitigate the critical obstacles posed by the scarcity of high-resolution training data and provide key insights into training and inference across extremely diverse resolutions, which have not been fully addressed in previous works yet.

In summary, we sincerely appreciate your comments. However, we would like to clarify that the contribution of IXC2-4KHD encompasses much more than merely increasing the number of patches from 4 to 55. We are genuinely grateful for any further discussion or additional comments you may have.
Thanks for authors' further response. I don't think that there are any misunderstanding. 4096 is the maximum context window of vicuna used in LLaVA-NexT, which is the main bottleneck for its training with more visual tokens, e.g., 25 patches or more. This paper adopts InternLM2 with a longer context window, which of course supports larger resolution.   

Back to my concernes,  the patch division is no obviously different to monkey and llava-next. I cannot agree that increasing the  maximum number of patch from 4 patches to 55 patches  will be a big contribution to the community.  Expect for this, other contributions highlighted in the paper are all present  in recent methods.
Thanks for your comments. We are delighted that our rebuttal has solved part of your concerns. For the newly posted comments, we shall clarify them as follows:

### **1. The key contribution and novelty of IXC2-4KHD lie in successfully enabling LVLM to understand images across a vast range of resolutions (336~4K).**
The mentioned related works are impressive that have been discussed in the related work section of our submission. It is important to note that they are explored on limited resolutions (i.e., Monkey at 1344x896, LLaVA-Next at 672x672, and LLaVA-UHD at 672x1088). Therefore, they are not able to answer the question of how to let LVLM tackle an extreme diversity of image size and aspect ratios. Moreover, IXC2-4KHD examines the impact of global image, 'newline' token,and other factors in diverse resolutions, revealing insights into their effectiveness in a unified framework for understanding dynamic high resolutions— insights not explored previously. In the final version, we will talk more about the detailed designs in Monkey, LLaVA-Next, and LLaVA-UHD and discuss our insights.

### **2. We would like to clarify two misunderstandings:**
(1) First, regarding the training of IXC2-4KHD, it was not limited to a maximum token window of 4096. Instead, IXC2-4KHD was trained with up to 8737 image tokens, which is about twice the size of 4096 tokens, as illustrated in Figure 4 of the main paper.

(2) Second, it is our proposed dynamic patch layout strategy, rather than the token merge strategy, that contributes to the high-resolution advantages. It is worth noting that Monkey also employed a token resampler, resulting in 256 tokens for each patch (up to 16 patches with a 4096 window). Therefore, it is not the maximum token window that limits Monkey from scaling up its resolution. In summary, it is not the token merge strategy that enables high-resolution understanding.

### **3. We are glad the reviewer agrees that InternVL-1.5 is a concurrent work with IXC2-4KHD and should not be used to dispute our work.**
 We notice that InternVL-1.5 employs a similar token merge strategy, where we concatenate the nearby 4 tokens in channels, and InternVL-1.5 adopts pixel shuffle. Generally, these two designs should not make noticeable performance differences.
I have carefully read the rebuttal. The rebuttal solved my questions and I think the study about adaptive resolution LVLM is novel and beneficial to the research community. The results of video benchmarks also show the potential of the proposed solution.

I also read the review comments from other reviewers, I think from both study depth and scale, it provides novel insight and understanding about how to enable LVLM to understand images with high resolution, it is distinguished from existing works such as LLaVA-Next.

In conclusion, I would keep my score as 8, strong accept. If the author could open-source data, code, and more experiment details about this work in the future, it would be helpful for the following study.
I appreciate authors' response, which partly addresses my concerns.  However, I still think that the novelty and contribution of this paper are still below the acceptance bar of NeurIPS.

Expect for the  patch layout,  all designs of the paper exist in recent methods. For example, The global-local format and the patch division is proposed in MonKey. The Newline Indicator has been adopted  in many works like LLaVA-NeXT and LLaVA-UHD.  By the way, one thing that might be unacceptable is that the authors did not even mention these methods in their methods, which might mislead other reviewers.

Most importantly, the authors attribute the high-resolution advantages to their patch layout, which is not rigorous. IXC2-4KHD is able to handle such high-resolution input not because of its patch layout, but because the token merge strategy can reduce the number of tokens so that it can be trained with a maximum token window of 4096.  This explain why Monkey and LLaVA-Next are restricted to 4-6 patches. Unfortunately, the token merge strategy is not firstly proposed in the paper, so I still doubt the novelty of the paper.

Based on this concern, I think InternVL-1.5, which also applies the same token merge strategy,  is a good comparable method to prove the patch layout design.   I also understand that InternVL-1.5 may be  a concurrent work to this paper. So, maybe the authors should strictly control the same token merge strategy to prove its patch layout design.

Overall, I am still inclined to reject this paper.