HiRes-LLaVA: Restoring Fragmentation Input in High-Resolution Large Vision-Language Models

## Abstract

Abstract High-resolution inputs enable Large Vision-Language Models (LVLMs) to discern finer visual details, enhancing their comprehension capabilities.
To reduce the training and computation costs caused by high-resolution input, one promising direction is to use sliding windows to slice the input into uniform patches, each matching the input size of the well-trained vision encoder.
Although efficient, this slicing strategy leads to the fragmentation of original input, i.e. , the continuity of contextual information and spatial geometry is lost across patches, adversely affecting performance in cross-patch context perception and position-specific tasks.
To overcome these shortcomings, we introduce HiRes-LLaVA , a novel framework designed to efficiently process any size of high-resolution input without altering the original contextual and geometric information. HiRes-LLaVA comprises two innovative components: (i) a SliceRestore adapter that reconstructs sliced patches into their original form, efficiently extracting both global and local features via down-up-sampling and convolution layers, and (ii) a Self-Mining Sampler to compresses the vision tokens based on themselves, preserving the original context and positional information while reducing training overhead.
To assess the ability of handling context fragmentation, we construct a new benchmark, EntityGrid-QA, consisting of edge-related and position-related tasks. Our comprehensive experiments demonstrate the superiority of HiRes-LLaVA on both existing public benchmarks and on EntityGrid-QA, particularly on document-oriented tasks, establishing new standards for handling high-resolution inputs.

## 1 Introduction

Recent progress in Large Vision-Language Models (LVLMs) ; ; ; ; ; has significantly enhanced capabilities in vision-language tasks, fostering improved understanding, reasoning, and interaction.
Early LVLMs ; ; processed images at low resolutions, typically $224\times 224$, which hindering their ability to capture detailed visual information. This limitation often results in inaccurate recognition of objects and their contextual relationships within images ; .

Enhancing the high-resolution capabilities of LVLMs presents substantial challenges, i.e., training visual encoders to handle high-resolution inputs requires significant computational resources as well as struggling with handling arbitrary image sizes ; .
Recent advances have introduced resource-efficient methods to improve the input resolution of LVLMs. One effective strategy involves using a sliding window technique ; ; to segment high-resolution images into smaller patches. These patches are then processed by a visual encoder that has been trained on lower-resolution inputs, maintaining computational efficiency while enhancing detail capture.

Although effective, this slicing approach leads the fragmentation of the original input, leading to the problems such as: (i) Disruption of Context. Slicing the entire image can alter the original context, particularly when an object is located at the edge of two slices; (ii) Impairment of Spatial Geometry. Slicing disrupts the inherent spatial geometry, complicating tasks that rely on positional relationships, such as identifying objects adjacent to specific regions; see details in Fig. 1 (a).
Furthermore, existing approaches ; generally use a sampler, such as Q-Former , to reduce the long context caused by high-resolution input.
However, this Q-Former like sampler utilizes a fixed number of queries to compress and capture visual features through a cross-attention mechanism, suffering from problems, e.g., lacking position information and high training overhead.

Figure: Figure 1: (a) Fragmentation of the original input. Current slicing-based LVLMs would disrupt the context and spatial geometry. (b) Visualization of the accuracy of objects in different positions. We test images with a bear positioned in nine different locations using various LVLMs. Note that the green circles with number here are only for illustration and not presented for LVLMs. By comparing the average accuracies for these positions, results indicate that bears straddling slices typically show reduced accuracy with existing SOTA models ; ; . In contrast, our method maintains consistent accuracy across all positions.
Refer to caption: x1.png

In this paper, we propose HiRes-LLaVA, an efficient approach to integrating high-resolution data into LVLMs without disrupting the original context and spatial geometry. As illustrated in Fig.1(b), our method maintains consistent accuracy even when objects are positioned across different slices.
HiRes-LLaVA utilizes a new SliceRestore Adapter to combine sliced low-resolution patch features into a high-resolution feature map, preserving the image’s complete context and spatial information. This map is processed through dual parallel fusion modules to capture both global and local information. The enhanced high-resolution map is then segmented back into small patches.
The SliceRestore Adapter is a lightweight module that can be seamlessly integrated into any attention layer of the low-resolution vision encoder, enabling efficient fine-tuning without altering pre-trained parameters. Furthermore, we introduce a self-mining sampler that uses average pooled sliced patches as queries. Unlike fixed learnable query-based methods, our self-mining sampler preserves the original context and positional information while optimizing efficiently.

To evaluate our proposed method, besides 11 public benchmarks, we also introduce a new benchmark named EntityGrid-QA.
This benchmark is specifically designed to include identification, position, and counting tasks, which are critical for further assessing the ability of VLMs to handle context fragmentation resulting from slicing approaches.
Comprehensive experiments show that our HiRes-LLaVA not only achieves superior performance on existing public benchmarks but also significantly outperforms state-of-the-art LVLMs on our newly introduced EntityGrid-QA.

## 2 Related Works

### 2.1 Large Vision-Language Model

Leveraging pre-trained Large Language Models (LLMs) such as LLaMA  and Vicuna , Large Vision-Language Models (LVLMs) have demonstrated substantial advancements across various domains including general image/video understanding ; ; ; ; ; ; , medical analysis , and autonomous driving ; .
These models primarily encode vision features to be understand by LLMs using tools like CLIP-ViT ; , pre-trained via contrastive learning with vast image-text pairs  to align visual embeddings with language.
Subsequently, visual embeddings are transformed to match the dimensionality of LLMs through a visual projector, categorized into: (i) learned queries such as the perceiver resampler  or Q-Former ; that use fixed queries for capturing features via cross-attention, and (ii) MLP modules, exemplified by the LLaVA series . Recent efforts have explored combining diverse visual features from sources like DINO-V2  and SAM  with CLIP-ViT to enrich visual representation ; . However, the dependency on CLIP-ViT, which requires images of fixed resolution (e.g., $336\times 336$), restricts handling higher resolutions or varying aspect ratios, thereby limiting performance in fine-grained tasks.

### 2.2 High Resolution Large Vision-Language Model

To discern fine-grained visual details from high-resolution inputs, an intuitive approach is to split images into patches and project them using linear layers, treating these as a sequence for input into Large Vision-Language Models (LVLMs) ; . While this eliminates the need for an image encoder, it often results in insufficient visual representation, leading to increased training costs and suboptimal performance.
Alternatively, Up-Resize methods such as Qwen-VL  adapt the positional embeddings of Vision Transformers (ViT) from $224\times 224$ to $448\times 448$ and include an additional training phase to fine-tune the ViT. However, this adaptation may alter the original visual position encoding from CLIP-ViT , potentially degrading visual representation.
Dual-branch approaches introduce a high-resolution branch with lightweight convolutional networks to manage high-resolution inputs but require additional training data and parameters ; ; ; .
Slicing-based methods offer a compromise by using slicing windows to divide the high-resolution image into patches that match the input size of a pre-trained vision encoder, maintaining efficiency in parameter use and training data while still achieving competitive performance ; . However, they suffer from "Context Fragmentation", where the continuity of contextual information across patches is damaged, impacting tasks that require cross-patch context and spatial relationships.
In this paper, we propose HiRes-LLaVA, a novel technique designed to seamlessly integrate global-local high-resolution details into LVLMs without disrupting the original context or spatial geometry, effectively addressing the issue of Context Fragmentation.

## 3 Method

The overall framework of HiRes-LLaVA is shown in Fig. 2(a). First, the original image is resized and padded to a low resolution (typically $224\times 224$) and processed by the pre-trained CLIP-ViT, producing global features. To capture high-resolution details, the image is split into slices, each matching the CLIP-ViT input size. These slices are processed by a shared CLIP-ViT with the proposed SliceRestore adapter (Section 3.1), yielding slice features.
The slice features are then fed into a shared self-mining sampler to reduce token length, resulting in compressed features. These compressed features, along with text instruction embeddings, are concatenated and fed into the LLM to generate the response. Importantly, we keep the vision encoder (CLIP-ViT) and LLM parameters frozen, training only the proposed SliceRestore adapter, the self-mineing sampler and LoRA.

Figure: Figure 2: (a) Overall framework of HiRes-LLaVA. The vision encoding consists of two branches: one for low-resolution images processed by CLIP-ViT to extract global features, and another for high-resolution images to capture fine-grained details. (b) SliceRestore Adapter. This component restores sliced features into a whole feature by capturing both local and global information, then splits the whole feature back into slices. (c) Self-Mining Sampler. This component compresses vision token numbers to reduce computation and memory costs by using downsampled sliced features as queries and the original sliced features as keys and values. Both low=resolution image input and each high-resolution slice are compressed by the same self-mining sampler.
Refer to caption: x2.png

### 3.1 SliceRestore Adapter

As depicted in Fig. 2 (b), the SliceRestore adapter is integrated into the self-attention layer of CLIP-ViT. We denote the slice features in the $l$-th layer of ViT as $\{\mathbf{P}_{i}\}_{i=1}^{N}$ with $\mathbf{P}_{i}\in\mathbb{R}^{L\times D}$, where $N$ is the number of slices, $L=H\times W$ is the token length, and $D$ is the feature dimension. Each slice feature is processed individually by the self-attention layer, $\text{Self-Attn}(\mathbf{P}_{i})$, which can fragment context and disrupt image geometry (see Fig. 1 (a)). Although low-resolution inputs contain the overall information, small objects in slices are still difficult to perceive. A naive approach would be concatenating slice features for self-attention, but this incurs quadratic computation costs.

In this paper, we propose the SliceRestore adapter to efficiently capture complete information from high-resolution inputs. This is formulated as:

$$ $\{\hat{\mathbf{P}}_{i}\}_{i=1}^{N}=\{\text{Self-Attn}(\mathbf{P}_{i})\}_{i=1}^ {N}+\{\overline{\mathbf{P}}^{l}_{i}\}_{i=1}^{N},$ (1) $$

where:

$$ $\{\overline{\mathbf{P}}^{l}_{i}\}_{i=1}^{N}=\text{SRA}(\{\mathbf{P}_{i}\}_{i=1 }^{N}),$ (2) $$

and SRA represents the proposed SliceRestore adapter.

The SliceRestore adapter has three main steps to restore complete semantics from slice features:

1. Merging: Each slice feature $\mathbf{P}_{i}$ is first reshaped into $\mathbf{H}_{i}\in\mathbb{R}^{H\times W\times D}$. These reshaped slice features, $\{\mathbf{H}_{i}\}_{i=1}^{N}$, are then merged to form the original input’s features $\mathbf{F}\in\mathbb{R}^{(m*W)\times(n*H)\times D}$. $m$ and $n$ indicates the number of rows and columns of slices, respectively. $N$ is equal to $m*n$

2. Capturing: We use two fusion modules for extracting both local and global information from $\mathbf{F}$. The local fusion module uses a single layer depth-wise convolution with $3\times 3$ kernel to efficiently capture local details and retain image-related biases. The global fusion module employs self-attention to capture global context. Given the quadratic computation cost of self-attention, we first dowsample $\mathbf{F}^{l}$ to a smaller size, $(m*W_{1})\times(n*H_{1})$ ($W_{1}<W$ and $H_{1}<H$), perform self-attention, and then up-sample back to the original size. The enhanced whole feature $\overline{\mathbf{F}}$ is obtained by element-wise addition of the outputs from the local and global fusion modules:

$$ $\overline{\mathbf{F}}=\underbrace{\text{Depth-Wise Conv}(\mathbf{F})}_{\text{ local fusion}}+\underbrace{\text{Up}(\text{Self-Attn}(\text{Down}(\mathbf{F})) )}_{\text{global fusion}}.$ (3) $$

3. Slicing: Finally, the enhanced whole feature $\overline{\mathbf{F}}$ is sliced back into the original slice format, resulting in $\{\overline{\mathbf{P}}_{i}\}_{i=1}^{N}$, where $\overline{\mathbf{P}}_{i}\in\mathbb{R}^{L\times D}$.

This process allows model to capture the complete semantics from high-resolution inputs while maintaining computational efficiency.

### 3.2 Self-Mining Sampler

High-resolution images necessitate processing significantly more visual tokens, contributing to a substantial part of the computational load.
Existing solutions, such as Q-Former , utilize a fixed number of queries to compress and capture visual features through a cross-attention mechanism. While this method effectively captures visual information regardless of image resolution in a computationally affordable manner, it suffers from several limitations:
(i) Lacking Positional Information. Learned queries lose positional information, degrading performance in tasks requiring spatial relationships and precise localization.
(ii) High Training Overhead. Training Q-Former-like resamplers requires more data and longer training times to convert visual features into learnable queries, posing challenges in data-scarce domains.

To address the issues, we propose a self-mining sampler, as shown in Fig. 2(c). Specifically, we reshape the one-dimensional output vision tokens of the vision encoder (e.g., CLIP-ViT), $\mathbf{P}\in\mathbb{R}^{L\times D}$, into a two-dimensional form, ${H\times W\times D}$, where $L=H\times W$. After applying average-pooling with kernel size $S\times S$, we obtain $\mathbf{P}^{c}\in\mathbb{R}^{H_{2}\times W_{2}\times D}$, where $W_{2}<W$ and $H_{2}<H$.
Next, we compute the final compressed tokens using cross-attention, $\text{Cross-Attn}(\mathbf{P}^{c},\mathbf{P})$, with $\mathbf{P}^{c}$ as the query and $\mathbf{P}$ as the key and values.
Compared with fixed learnable query-based methods, our self-mining sampler compresses the vision tokens based on themselves, preserving the original context and positional information while reducing training overhead.

Figure: Figure 3: Construction process of our proposed EntityGrid-QA benchmark. There are three steps: (a) Entity Sampling. Select one or two entities from the pre-defined entity set; (b) Image Generation. Put the selected entities in one position sampled from the nine pre-defined positions of the blank image, we can obtain the generated images. Note that the dash and solid lines in (b) are for illustration purposes only, and not presented to models. (c) QA pairs Generation. Based on the generated images, entity category and positions, we can automatically generate the question-answer pairs (QAs). We totally construct three different tasks, i.e., identification, position and counting.
Refer to caption: x3.png

### 3.3 Dynamic High Resolution Slicing

Pre-trained visual encoders, utilized across supervised, self-supervised, and vision-language pretraining, predominantly reshape images into squares by cropping or padding. This approach, however, struggles with non-square images prevalent in real-world scenarios, particularly failing to preserve detail or loss information in images with extreme aspect ratios. The current solution of high-resolution images input in large vision-language models involves slicing these images into pre-trained model-compatible slices based on pre-defined aspect ratio candidates, which does not accommodate the diversity of real-world imagery effectively ; ; .

In contrast, HiRes-LLaVA dynamically slices high-resolution images without predefining aspect ratio candidates which enables any aspect ratio image inputs. We set a maximum slice count $M$, allowing an image to automatically select an optimal bounding box by calculating the necessary $m$ rows and $n$ columns based on the base resolution:

$$ $m=\left\lceil\frac{H}{r}\right\rceil,n=\left\lceil\frac{W}{r}\right\rceil.$ $$

Where $r$ is the base resolution in pretrained vision encoder. $\left\lceil x\right\rceil$ denotes the ceiling function of $x$, which round up the $x$ to the smallest integer greater than itself if $x$ is decimal.

This approach adapts to the image’s aspect ratio, only quadrupling the number of slices by scaling $2\times$ of $m$ and $n$ if “$4*m*n$” does not exceed $M$, ensuring detailed preservation without overwhelming the model. Our input to the language model includes a low-resolution overview and multiple high-resolution slices, differentiated by three types of separators to maintain clarity in (1) between the low-resolution image and high-resolution slices, (2) between high resolutions slices and (3) the end of each slice row.

### 3.4 EntityGrid-QA Benchmark

Existing benchmarks, particularly document-related datasets, can evaluate the fine-grained understanding of LVLMs. However, these benchmarks are inadequate for assessing the ability to handle fragmented inputs, as filtering slicing-related questions is time-consuming and labour-intensive. Therefore, we introduce a new benchmark named EntityGrid-QA to better assess LVLMs’ ability to handle fragmentation.

#### Construction Process.

As shown in Fig. 3, the construction process of EntityGrid-QA consists of three main steps: Entity Sampling, Image Generation, and QA Pairs Generation. Each step is detailed as follows:

(a) Entity Sampling.
We first construct one entity set which includes various types such as Text (e.g., "apple"), Digit (e.g., "0.596"), Object (e.g., a teddy bear), Shape (e.g., triangle, circle), and Relative Position (e.g., traffic lights, stop sign), as shown in Fig. 3 (a).
Then, we select several entities from a predefined entity set, which can be denoted as
$\mathcal{E}=\{e_{i}\}_{i=1}^{M}$, where $e_{i}$ is the $i$-th entity and $M$ is the total number selected entities.

(b) Image Generation. The selected entities $\mathcal{E}$ are positioned in nine predefined positions (labeled 1 to 9) within a blank image $I$ using a 3x3 grid layout, as shown in Fig. 3 (b).
The resolution of the blank image is set to $2R$, where $R$ is the base resolution for existing LVLMs, e.g., $224\times 224$.
In this way, each $I$ would be divided into four slices during inference, and each slice would match the input size of well-pretrained vision encoder, without the requirement of additional operations, e.g., resize and padding.
Note that our HiRes-LLaVA can process any number of slices, however some existing LVLMs, i.e., LLaVA-1.6  can only receive four slices as input.
Hence, for a fair comparison, we only generate the images with a fixed resolution $2R$.
We denote the selected positions as $\mathcal{P}=\{p_{i}\}_{i=1}^{M},p_{i}\in[1,9]$, where $p_{i}$ is the position for $e_{i}$.

(c) QA Pairs Generation.
For more comprehensive evaluation, we define three different tasks, i.e., Identification, Position and Counting, where each task has a specific question prompt, i.e., $Q_{\text{ident}}$, $Q_{\text{pos}}$ and $Q_{\text{count}}$, as follows: $Q_{\text{ident}}=\texttt{"What is the object in the picture?"}$, $Q_{\text{pos}}=$ "Where is $e_{i}$ at $e_{j}$ in the picture?" and $Q_{\text{count}}=$ "How many $e_{i}$ are in the picture?".
Based on the generated images $I$ as well as its corresponding entity $\mathcal{E}$ and the question prompt $Q$, we can automatically generate questions, and the answer can be obtain by positions $\mathcal{P}$, e.g., the relative position of two objects $e_{i}$ and $e_{j}$ can be caculated by their positions $p_{i}$ and $p_{j}$.
As shown in Fig. 3 (c), the generated question-answer pairs are in a multiple choice questions format.
Examples of our benchmark are provided in Appendix.

#### Evaluation Metric.

To evaluate the ability to handle the fragmentation, we introduce a new metric that measure the precision discrepancies between entity located at the edge positions ($\mathcal{P}_{\text{edge}}=\{2,4,5,6,8\}$) and other locations ($\mathcal{P}_{\text{center}}=\{1,3,7,9\}$), defined as:

$$ $\text{Discrepancy}_{1}=\frac{\sum_{p\in\mathcal{P}_{\text{edge}}}A_{p}/| \mathcal{P}_{\text{edge}}|}{\sum_{p\in\mathcal{P}_{\text{center}}}A_{p}/| \mathcal{P}_{\text{center}}|},$ (4) $$

$$ $\text{Discrepancy}_{2}=\frac{\sum_{p\in\mathcal{P}_{\text{edge}}}A_{p}/| \mathcal{P}_{\text{edge}}|-\sum_{p\in\mathcal{P}_{\text{center}}}A_{p}/| \mathcal{P}_{\text{center}}|}{\sum_{p\in\mathcal{P}_{\text{center}}}A_{p}/| \mathcal{P}_{\text{center}}|},$ (5) $$

where $A_{p}$ is the average accuracy of three tasks when entities located at the position $p$, $|\cdot|$ is the number of elements in the set.

## 4 Experiment

### 4.1 Implementation Details

We utilize the CLIP-ViT-L/14-224px and CLIP-ViT-L/14-336px as the vision encoders, and Vicuna-7B-v1.5 as LLM. We adopt a two-stage training approach following LLaVA , including the pre-training stage and the instruction tuning stage. During pre-training, only the self-mining sampler is trainable. The batch size is 256, and the learning rate is 1e-3. In the instruction tuning stage, the SliceRestore adapter, self-mining sampler, and LoRA parameters  in LLM are trained. Four SliceRestore adapterare applied in the 19th to 22nd layers of the vision encoder. The batch size of the instruction tuning stage is 128. The LoRA has a rank of 128 and an alpha of 256. The learning rate of the LoRA parameters is 2e-4, and the learning rate of the rest of the trainable parameters, including the SliceRestore Adapter, is 2e-5. We adopt AdamW  as the optimizer with $\beta_{1}=0.9$ and $\beta_{2}=0.95$ to stabilize the training in the instruction tuning stage. In both two stages, the learning rates are warmed up for the first 0.03 epochs and then adjusted by a cosine scheduler in the remaining training. We don’t apply any weight decay in the training. The maximum number of slices $M$ is set to 16. All models are trained two days on 64 NVIDIA V100 GPUs under FP16.
Regarding the training data, we gather public datasets to augment the original LLaVA dataset. In the pretraining stage, we collect 1.6M data, including 1.2M caption data and 0.4M OCR data. To construct a large instruction tuning dataset that covers a wide range of tasks, we also collect 1.5M data from open-source datasets. Please refer to the Appendix for more details.

### 4.2 Experimental Setting

We introduce experimental settings including the benchmarks and the compared LVLMs.

Benchmarks. We adopt 11 popular benchmarks to evaluate our models on five document-related VQA benchmarks, including VQA-text, ChartQA test set , DocVQA test set , InfoVQA test set , STVQA test set , two general VQA benchmarks ; , including AI2D , ScienceQA , and four comprehensive benchmarks, including LLaVA-in-the-wild (LLaVA-W) , MMbench , POPE  and HallusionBench .

LVLMs. We compare our model with SOTA LVLMs. (1) General baselines, i.e., InstructBLIP , Qwen-VL , LLaVA-1.5 , mPLUG-Owl2 , Monkey , Mini-Gemini , LLaVA-UHD  and LLaVA-NeXT , as representative general baselines.
(2) Document LVLMs, i.e., Donut , DocPedia , UReader , mPLUG-Docowl , TextMonkey .

**Table 1: Quantitative results on 11 popular benchmarks. The suffix ‘-224px’ and ‘-336px’ mean the vision encoders are CLIP-L/14-224px and CLIP-L/14-336px. ‘MaxRes’ means the maximum resolution supported. ‘Doc’, ‘General’ and ‘Comprehensive’ indicate the documant-related VQA, general VQA and comprehensive benchmarks. *’ denotes the results evaluated by lmms-eval . Methods with 13B LLMs are marked in gray. ‘$\dagger$’ means using LoRA to finetune LLM.**
|  |  |  | Doc | General | Comprehensive |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | LLM | MaxRes | VQA-text | ChartQA | DocVQA | InfoVQA | STVQA | AI2D | ScienceQA | LLaVA-W | MMBench | POPE | HallBench |
| General LVLMs (normal resolution) |  |  |  |  |  |  |  |  |  |  |  |  |  |
| InstructBLIP | Vicuna-7B | 224×224 | 50.1 | 5.3 | 4.5 | 16.4 | - | - | - | - | 36.0 | - | - |
| Qwen-VL | Qwen-7B | 448×448 | 63.8 | 65.7 | 65.1 | 35.4 | 59.1 | 55.9 | 67.1 | - | 38.2 | - | - |
| Qwen-VL-Chat | Qwen-7B | 448×448 | 61.5 | 66.3 | 62.6 | - | - | 57.7 | 68.2 | - | 60.6 | - | - |
| LLaVA-1.5 | Vicuna-7B | 336×336 | 58.2 | 18.2* | - | - | 38.1 | 54.8* | 66.8 | 59.6* | 64.3 | 85.9 | - |
| LLaVA-1.5 | Vicuna-13B | 336×336 | 61.3 | 18.2* | - | - | - | 59.5* | 71.6 | 66.1* | 67.7 | 85.9 | - |
| mPLUG-Owl2 | LLaMA-7B | 224×224 | 58.2 | - | - | - | - | - | 68.7 | - | 64.5 | 86.2 | - |
| Document LVLMs |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Donut | - | 2560×1920 | 43.5 | 41.8 | 67.5 | 11.6 | 43.5 | - | - | - | - | - | - |
| DocPedia | Vicuna | 2560×2560 | 60.2 | 46.9 | 47.1 | 15.2 | 45.5 | - | - | - | - | - | - |
| mPLUG-DocOwl  $\dagger$ | Vicuna | - | 52.6 | 57.4 | 62.2 | 38.2 | - | - | - | - | - | - | - |
| UReader | LLaMA-7B | 896×1120 | 57.6 | 59.3 | 65.4 | 42.2 | 57.6 | - | - | - | - | - | - |
| TextMonkey | QWen-7B | 896 | 65.9 | 65.5 | 71.5 | 28.2 | 68.0 | - | - | - | - | - | - |
| TextMonkey+ | QWen-7B | 896 | 64.3 | 66.9 | 73.0 | 28.6 | 65.6 | - | - | - | - | - | - |
| General LVLMs (high resolution) |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Monkey  $\dagger$ | QWen-7B | 1344×896 | 67.6 | 65.1 | 66.5 | 36.1 | 67.7 | 57.9 | 69.4 | - | - | - | - |
| Mini-Gemini | Vicuna-7B | 672 | 68.4 | - | - | - | - | - | - | - | 65.8 | - | - |
| LLaVA-UHD | Vicuna-13B | 672×1008 | 67.7 | - | - | - | - | - | 72.0 | - | 68.0 | 89.1 | - |
| LLaVA-NeXT | Vicuna-7B | 672 | 64.9 | 54.8* | - | - | - | 66.6 | 70.2 | 72.3* | 67.4 | 86.5 | 41.5 |
| HiRes-LLaVA-224px $\dagger$ | Vicuna-7B | 896×896 | 62.5 | 61.2 | 68.0 | 43.5 | 62.8 | 68.9 | 84.2 | 76.8 | 70.4 | 86.5 | 41.7 |
| HiRes-LLaVA-336px $\dagger$ | Vicuna-7B | 1344×1344 | 65.4 | 61.5 | 74.7 | 48.0 | 65.8 | 69.7 | 84.1 | 78.2 | 70.5 | 87.3 | 42.6 |

Figure: Figure 4: The visualization comparison with the state-of-the-art methods. Dash lines are only illustrated for the slice clarify, not presented to LVLMs.
Refer to caption: x4.png

### 4.3 State-of-the-art Comparison

General Benchmarks. Table 1 reports the performance comparison of our methods against state-of-the-art approaches on 11 benchmarks.
Unexpectedly, our method utilizing LoRA fine-tuning  surpasses well-established LVLMs that require substantial data and extensive full fine-tuning, underscoring our model’s efficiency and effectiveness.
Notably, although both our model and Monkey  employ LoRA, Monkey is initialized from the pre-trained Qwen model , while our model is trained from scratch, which further proves our model’s efficiency. Furthermore, our method demonstrates competitive performance against specialized document-centric LVLMs such as TextMonkey , proving its capability to manage document-related tasks effectively.

Fig. 4 shows a visual comparison of results generated by LLaVA-1.6 , Monkey , and our method, highlighting our superior performance, especially when the region of interest spans across slices.
For example, the number $1.14$ in Fig. 4 (b) is split into two slices, causing Monkey to misrecognize it as $1.4$. Additionally, the slicing operation separates the year and percentage values into different slices, leading LLaVA-1.6 to incorrectly associate the 2017 percentage with 2014 due to the lack of global information.
Our method, with the SliceRestore adaptercapturing complete global high-resolution information, correctly predicts the answers.

Table: Table 2: Comparison with the state-of-the-art methods on EntityGrid-QA. ‘$\downarrow$’ indicates lower scores are better, while ‘$\uparrow$’ means higher scores are better. The suffix ‘-224px’ and ‘-336px’ mean the vision encoders are CLIP-L/14-224px and CLIP-L/14-336px, respectively. ‘Accuracymean’ and ‘Accuracystd’, representing the mean and standard deviation of the average accuracy across three tasks. ‘Accuracyedge’ and ‘Accuracycenter’ show the average accuracy for entities at $\mathcal{P}_{\text{edge}}$ and $\mathcal{P}_{\text{center}}$, respectively. Discrepancy1 and Discrepancy2 are calculated using Eq. 4 and Eq. 5.

EntityGrid-QA.
To evaluate the ability to address input fragmentation, we compare four SOTA slicing-based LVLMs with our HiRes-LLaVAand present the results in Table 2.

From Table 2, we observe two key findings:
(i) Our method performs competitively on tasks with entities at $\mathcal{P}_{\text{center}}$. For instance, our method scores $0.6400$ in accuracymean and $0.6650$ on Accuracycenter, compared to the best prior SOTA scores of $0.6311$ and $0.6799$.
(ii) Our method significantly outperforms SOTAs in handling entities at $\mathcal{P}_{\text{edge}}$. For example, the previous SOTA, Xcomposer , shows a notable difference between Accuracyedge and Accuracycenter, with $0.5919$ vs. $0.6799$. In contrast, our method achieves a smaller difference, with $0.6200$ in accuracyedge and $0.6650$ on Accuracycenter.
Additionally, the values of Discrepancy1 and Discrepancy2 further reflect the consistent performance of our method for both edge and center cases, surpassing existing SOTAs.
In summary, our HiRes-LLaVA demonstrates superior ability to handle input fragmentation while maintaining competitive performance in center cases.

Table: Table 3: The ablation study of different proposed modules. ‘SMS’ and ‘SRA’ represent the Self-Mining Sampler and SliceRestore Adapter, respectively. The downsample rate of all downsampler are $4\times$.

Table: Table 4: Effect of different downsample kernel sizes in the self-mining sampler. ‘Downsample Kernel Size’ is $S\times S$ defined in Section 3.2. ‘Base Resolution’ indicates the base resolution of the vision encoder. ‘Max # Token’ indicates the maximum number of visual tokens, i.e., $H_{2}\times W_{2}$, as the maximum number of sclices is 16.

Figure: Figure 5: (a) Ablation on data efficiency of HiRes-LLaVA. We sample the training data mixture at ratios of 20%, 60%, and 100% and report the performance of our HiRes-LLaVAon seven benchmarks. (b) Data efficiency comparison with Q-former and our proposed self-mining sampler (SMS). The performance on ‘Doc QA’ is averaged from DocVQA, ChartQA and InfoVQA. The performance on ‘General QA’ is averaged from the other four benchmarks. Our SMS can use $40\%$ fewer data to achieve competitive performance compared with Q-former, which indicates the efficiency of our method. Note that both Q-former and our SMS apply one cross-attention block.
Refer to caption: x5.png

### 4.4 Ablation Study

In this section, we conduct ablation studies to evaluate the effect of our proposed modules. In our ablation study, we conduct the experiments on the LLaVA 1.2M data  with additional 79K document-oriented data in the instruction tuning stage, i.e., DocVQA , ChartQA  and InfoVQA .

Effect of different proposed modules.
We ablate the two main components of our HiRes-LLaVA, specifically the SliceRestore adapter (SRA) and the self-mining sampler (SMS), as shown in Table 3. Our findings are as follows:
Our SMS demonstrates superior performance compared to other samplers, notably outperforming Resampler  by $6.9\%$ on the average score across four benchmarks.
Integrating the model with SRA leads to further improvements across these benchmarks.
Additionally, the introduction of learnable queries to isolate slice representations, referred to as Separator, results in a $1.8\%$ enhancement in the average score.

Ablation study of different kernel sizes in the self-mining sampler. Here we conduct the ablation study of the self-mining sampler.
In Table 4, we compare the performance of the average pooling with different kernel sizes, i.e., $s\times s$ in Section 3.2.
The results show that as the kernel size increases, i.e., the fewer vision tokens, the performance would degrade, since the information loss.

Data efficiency analysis.
We evaluated the data efficiency of our method, HiRes-LLaVA, by subsampling the training data mixture at ratios of 20%, 60%, and 100%. Results in Fig. 5 (a) show that using the entire dataset achieves optimal performance. Remarkably, with only 60% of the data, performance remains above 90% of the full dataset’s level, highlighting the potential for improved data efficiency.
Additionally, we compared our self-mining sampler’s efficiency against the commonly used Q-former in LVLMs. As depicted in Fig. 5 (b), our method performs competitively with Q-former even with only 20% of the data, demonstrating its effectiveness and efficiency.

## 5 Conclusion

In this paper, we present HiRes-LLaVA, a large visual-language model (LVLM) designed to efficiently address input fragmentation caused by current slicing-based high-resolution LVLMs. To evaluate this capability, we introduce a new benchmark, EntityGrid-QA, which includes identification, position, and counting tasks. Comprehensive experimental results on 11 popular existing benchmarks and EntityGrid-QA demonstrate the effectiveness of HiRes-LLaVA. Analytical evaluation and visualization results are provided for a deeper understanding of the model’s performance.

Limitations. The samples in our constructed EntityGrid-QA are simple, lacking complex backgrounds, and the categories of entities and tasks are limited. In the future, we aim to create a more diverse dataset to better evaluate the performance of LVLMs in handling fragmented input.

## Appendix A Appendix

### A.1 Implementation Details

#### Training Datasets.

Table 5 shows the details construction of the pretraining data of HiRes-LLaVA. Specifically, it has 1.2M captioning including original LLaVA-Pretrain datasets  with 558K images which is the short caption data and 708k long captioning data from ALLAVA . The 0.4M OCR data are all sampled from SynthDoG  including 300k English OCR data and 100k Chinese OCR data.

Table 6 shows the detailed construction of the instruction tuning dataset. First, we remove 23K caption data from original LLaVA-158K  and include GPT4V-generated caption data, i.e., LAION-GPT4v  and ShareGPT4V . We also sampled 406K instruct data from ALLAVA instruction data . To enhance the common knowledge of our model, we convert the visual dialog , visual spatial reasoning , AI2D , and Science QA  training set into the instruct-tuning data. Finally, we collect document-oriented data from diverse datasets, includes ChartQA , DVQA  , PlotQA  , OCRVQA , ST-VQA , DocVQA , InfoVQA , DeepForm , TAT-DQA , TableFact  and WebSRC .

#### Module Design Details.

The self-mining sampler consists of one cross-attention block with an output layer norm. The cross-attention block has a cross-attention layer and a FFN. Both of them apply the shortcut. The cross-attention layer has two layer norm for the query and key/value, respectively. As for the SliceRestore Adapter, the parameters of the self-attention layer with the layer norm are initialized from the pretrained CLIP self-attention at the same layer. To provide the positional information between slices, we apply a 2D RoPE ; on the global fusion module.

#### Evaluation Details.

We utilize the open-source evaluation tools, lmms-eval , to align our evaluation method to LLaVA-Next . Note that the LLaVA-W is evaluated under GPT4-0613 because of the deactivation of GPT4-0314.

**Table 5: Datasets in the pretraining stage.**
| Task | Dataset | # Samples |
| --- | --- | --- |
| Caption | LLaVA-Pretrain | 558k |
|  | ALLAVA | 708k |
| OCR | SynthDog-en | 300k |
|  | SynthDog-zh | 100k |
| Total | - | 1.6M |

**Table 6: Summary of datasets used in the instruction tuning stage.**
| Task | Dataset | # Sample |
| --- | --- | --- |
| Captioning | ShareGPT4V | 91K |
| TextCaps | 22K |  |
| Laion-GPT4V | 11K |  |
| General QA | VQAv2 | 83K |
| GQA | 72K |  |
| OKVQA | 9K |  |
| A-OKVQA | 66K |  |
| Visual Spatial Reasoning | 12K |  |
| VisualDialog | 123K |  |
| Science | AI2D | 4K |
| ScienceQA | 12K |  |
| Doc QA | ChartQA | 28K |
| DVQA  , | 10K |  |
| PlotQA | 10K |  |
| OCRVQA | 80K |  |
| ST-VQA | 18K |  |
| DocVQA | 49K |  |
| InfoVQA | 14K |  |
| DeepForm | 5K |  |
| TAT-DQA | 2K |  |
| TableFact | 14K |  |
| WebSRC | 5K |  |
| Grounding | RefCOCO ; | 48K |
| VG | 86K |  |
| Conversations | LLaVA | 135K |
| ALLaVA | 400K |  |
| HTML Code Generation | WebSight | 50K |
| Text-only | ShareGPT | 40K |
| Total | - | 1.5M |

### A.2 More Visualization

#### Samples from EntityGrid-QA Eenchmark.

We illustrate three examples from our proposed EntityGrid-QA benchmark in Fig. 6. These three samples visualize examples of the three tasks in the benchmark we proposed. For each task, we write or paste the digital number or object directly onto each position of an empty image, and ask questions to the models.

#### More Qualitative Results.

To further validate the effectiveness of our model, we illustrate the more qualitative results of InfoVQA, ChartQA and V* Benchmark in Fig. 7 and Fig. 8. Moreover, we give two qualitative examples to present the HiRes-LLaVA’s capability of generating HTML code when given a website image in Fig. 9.

Figure: Figure 6: Examples of our proposed EntityGrid-QA Benchmark.
Refer to caption: x6.png

Figure: Figure 7: Qualitative results from InfoVQA .
Refer to caption: x7.png

Figure: Figure 8: Qualitative results from ChartQA and Vstar Benchmark . We use the red circle to highlight the answer target in the image.
Refer to caption: x8.png

Figure: Figure 9: Qualitative results on Image2HTML task . We visualize convert the generated html code to website image and compare to the input image.
Refer to caption: x9.png

### A.3 Broader Impacts

The development of HiRes-LLaVAadvances the field of vision-language models and has broad implications for various applications, including document analysis, medical imaging and remote sensing. However, alongside these potential benefits, there are considerable concerns.

HiRes-LLaVA, not having undergone rigorous safety training, might generate harmful or inappropriate content, leading to legal and ethical issues. Furthermore, its enhanced ability to process high-resolution inputs could be misused for creating misleading news, contributing to disinformation. These potential negative impacts highlight the need for careful management and ethical guidelines in the deployment of such technologies.

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. what is the stride of the depth-wise convolution in L211? In table 6, it looks that the stride equals the kernel size. But a more natural way is to set stride=1 regardless of the kernel size.
2. How many entities are selected in a generated image in L274-L282. It is unclear what would the model performs with only 1 object and multiple objects.
3. The training pipeline is very complicated with three stages, compared to LLaVA. Can the authors make it more clear of the training dataset, tunable parameters in each stage?
soundness: 3
strengths: 1. The problem of high-resolution images encoding is very important for Large Vision-Language Models. A new method and benchmark are proposed to improve the accuracy and enrich the evaluation datasets, which would be interesting to the LVLM community
2. The two new modules, SliceRestore adapter and Self-Mining Sampler, are both well-motivated, and clearly-designed to improve the vision encoding of cross-patch regions, and improve the training efficiency for visual token compression.
3. The proposed method achieves strong performance not only on the newly-proposed benchmark, but also on many existing public datasets. The performance improvements are very impressive.
summary: This work aims to improve the vision encoding module of Large Vision-Language Models for high-resolution images. A naive method is to slice the input image into uniform patches using sliding windows, and then encode each patch using the vision encoder. But this method has negative impacts on the cross-patch regions. This paper propose a new module, SliceRestore adapter, to introduce cross-patch interaction for joint feature extraction of multiple patches. The Self-Mining Sampler introduces better initializations for object query in Q-Former, which is easier for training. A new benchmark is also introduced to evaluate the model's ability for cross-patch region understanding.
weaknesses: The major concerns is about the novelty of proposed modules, and insufficient literature review on similar approaches. (1) The key idea of SliceRestore adapter is to introduce the local and global fusion operations for multiple patches. However, this problem has been well studies in the vision-transformer paper before, like swin-transformer [1], PVT [2], Twins [3] and many others. The key of these work is also how to improve the communication of multiple small patch during multiple layer of feature extraction. A more through comparison should be provided. (2) The idea of using pooled image feature as object queries or learning object queries from local regions has been well studied in many DETR-style works before, Conditional DETR [3], DAB-DETR [4] and many others. The novelty of the proposed two modules is limited, considering the broader related literatures.

[1] Liu, Ze, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. "Swin transformer: Hierarchical vision transformer using shifted windows." In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012-10022. 2021.

[2] Wang, Wenhai, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. "Pyramid vision transformer: A versatile backbone for dense prediction without convolutions." In Proceedings of the IEEE/CVF international conference on computer vision, pp. 568-578. 2021.

[3] Chu, Xiangxiang, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. "Twins: Revisiting the design of spatial attention in vision transformers." Advances in neural information processing systems 34 (2021): 9355-9366.

[4] Meng, Depu, Xiaokang Chen, Zejia Fan, Gang Zeng, Houqiang Li, Yuhui Yuan, Lei Sun, and Jingdong Wang. "Conditional detr for fast training convergence." In Proceedings of the IEEE/CVF international conference on computer vision, pp. 3651-3660. 2021.

[5] Liu, Shilong, Feng Li, Hao Zhang, Xiao Yang, Xianbiao Qi, Hang Su, Jun Zhu, and Lei Zhang. "Dab-detr: Dynamic anchor boxes are better queries for detr." arXiv preprint arXiv:2201.12329 (2022).

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: * Table 5 ablates the SMS module, Token number vs performance. Could you also compare to other downsampler modules in Table 1, compare their token number and performance? 

* Table 2 shows LLaVA baseline achieves best Discrepancy 1 and 2, it worth a explanation about why.

* When compare with other methonds on your EntityGrid-QA benchmark, do you fine-tune these models on your EntityGrid-QA datasets? How about the performance of HIRES-LLAVA without EntityGrid-QA trainingset?

* Compare Table 1 and Table 4 third row, we can see dataset has a very large gains on the performance. How about the performance of training LLaVA baseline, LLaVA-1.5-336 and LLaVA-Next on the same dataset as yours?

* You compare 4KHD in Table 2, but why don't you compare 4KHD in the Table 1?
soundness: 3
strengths: 1. The idea sounds and the paper is easy to follow.

2. Figures 2 and 3 are helpful for understanding.

3. Resolving fragment problem in the current dynamic slicing strategy for high-resolution input is important and valuable.
summary: The paper presents HiRes-LLaVA, a framework designed to efficiently handle high-resolution image inputs in Large Vision-Language Models while maintaining contextual and geometric integrity. High-resolution inputs improve visual detail comprehension but increase computational costs and disrupt context when sliced into patches. HiRes-LLaVA introduces the SliceRestore Adapter (SRA) for reconstructing images from patches and the Self-Mining Sampler (SMS) for compressing vision tokens without losing context. The EntityGrid-QA benchmark is developed to test handling of context fragmentation, particularly in edge-related tasks. Experimental results demonstrate HiRes-LLaVA's superior performance, with a ∼9% improvement over existing LVLMs and enhanced efficiency in visual token downsampling using SMS.
weaknesses: 1. Table 3 does not provide the performance metrics for the SRA alone, SMS + SRA (L)

2. The main experiments is based on LLaVA-224, which is kind of outdate. It may be better and convicine to use LLaVA-1.5-336 as the baseline.

3. No computation cost and parameter flops analysis. It may be more convicine to provide a comparison between inference time, accuracy and number of tokens on different methods.

4. Please refer to my questions.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. Does the SliceRestore adapter (SRA) meaningfully contribute to solving the fragmentation issue, or is the performance gain primarily due to the self-mining sampler (SMS)? From my perspective, SMS is not designed to solve the fragmentation issue but to be provided as a better substitution for Q-former.

2. Could the performance gains in Table 9 be due to higher resolution rather than the SMS and SRA modules?

3. Is the EntityGrid-QA benchmark too simplistic, allowing models with an overlapping sliding window to bypass its intended challenges?
soundness: 2
strengths: 1. The proposed self-mining sampler demonstrates higher data efficiency compared to Q-former and enhances the model's performance.

2. The newly introduced EntityGrid-QA serves as a valuable synthetic benchmark, offering a more comprehensive evaluation of MLLMs in handling information at the fragmentation boundary.

3. The model has a sota performance on various tasks including doc related tasks and science related tasks.
summary: This paper introduces HiRes-LLaVA, aiming at restoring fragmentation input for high resolution VLMs. Two important components are proposed: the SliceRestore adapter to fuse the global and local features, and the self-mining sampler to better initialize the query and narrow the receptive field that per query needs to compress for faster congerence and preserving the positional information. Additionally, a new benchmark, EntityGrid-QA, is created to evaluate the model's ability to address input fragmentation. The proposed model outperformed prior works on various benchmarks including general benchmarks as well as the proposed EntityGrid.
weaknesses: 1. My primary concern lies with the contribution of the SliceRestore adapter (SRA), which is intended to address the fragmentation issue. Based on the ablation study, it appears that the main performance improvement stems from the self-mining sampler (SMS) rather than the SRA. This raises uncertainty about the actual utility of the SRA; if it proves ineffective, the fragmentation issue that this paper aims to address remains unresolved.

2. My second concern is the fairness of the comparison between the proposed model and the baselines. In Table 1, the proposed model was trained on different data from that of the other models, making it challenging to draw conclusions about whether the architecture itself outperforms others. While Table 9 attempts a comparison using a fixed dataset and contrasts the proposed model with LLaVA-1.5, this remains problematic as LLaVA-1.5 is not a high-resolution model. It is unclear if the performance gains are due to higher resolution or the proposed SMS and SRA modules. A more appropriate comparison would involve a baseline model that uses the same number of input slices (i.e., matching the resolution of the proposed model) without SMS and SRA, and trained on the same dataset.

3. The constructed EntityGrid-QA benchmark may be too simplistic, and I suspect that a model utilizing an overlapping sliding window for input slices could easily circumvent the challenges this benchmark is designed to test.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. In section 4.3, it is mentioned that this work uses the LoRA fine-tuning. At which stage of training is this applied? Is HiRes-LLaVa trained on a pretrained model or from scratch?
soundness: 3
strengths: 1. Theoretically, The method proposed in this paper can handle outputs of any resolution and shows a fast convergence in training and significant performance improvement, outperforming the baseline on 8 benchmark results.
2. A new and simple benchmark to evaluate the model for handling fragmented inputs is proposed, demonstrating the effectiveness of the SRA module in this paper.
3. The ablation experiments are comprehensive, showing the effectiveness of the proposed method across multiple models, with sustained gains as the resolution gradually increases.
summary: This paper proposes the framework HiRes-LLaVa to address the issues of high training cost and poor image fragment continuity in LVLM under high-resolution image inputs. HiRes-LLaVa supports arbitrary resolution inputs while maintaining the original image's contextual and geometric continuity. HiRes-LLaVa incorporates the SlicerRestore adapter (SRA) in the vision encoder to extract global and local information, reconstructing image fragment features into the original size. To reduce the computational cost, the paper also introduces the image token compression method Self-Mining Sampler (SMS) to replace the Q-former. Across 9 image-text benchmarks, HiRes-LLaVa supports higher image resolutions and consistently outperforms or compares to baseline models. This work also introduces a new benchmark, Entity-QA, to evaluate the model's ability to handle fragmented inputs. On Entity-QA, HiRes-LLaVa demonstrates superior fragment handling capabilities compared to current state-of-the-art models. The paper conducts thorough ablation experiments to validate the proposed new modules, showing that the method not only improves the performance of models under different vision encoders or llms, but also achieves higher data efficiency and faster training convergence compared to existing models.
weaknesses: 1. The experimental setup description is unclear. Section 4.1 mentions a 3-stage training process, but section 4.3 gives out a LoRA finetuning is applied, and also claims the model is trained from scratch in comparison with Monkey.