Mini-Monkey: Alleviating the Semantic Sawtooth Effect for Lightweight MLLMs via Complementary Image Pyramid

## Abstract

Abstract Recently, scaling images to high resolution has received much attention in multimodal large language models (MLLMs). Most existing practices adopt a sliding-window-style cropping strategy to adapt to resolution increase. Such a cropping strategy, however, can easily cut off objects and connected regions, which introduces semantic discontinuity and therefore impedes MLLMs from recognizing small or irregularly shaped objects or text, leading to a phenomenon we call the semantic sawtooth effect. This effect is particularly evident in lightweight MLLMs. To address this issue, we introduce a Complementary Image Pyramid (CIP), a simple, effective, and plug-and-play solution designed to mitigate semantic discontinuity during high-resolution image processing. In particular, CIP dynamically constructs an image pyramid to provide complementary semantic information for the cropping-based MLLMs, enabling them to richly acquire semantics at all levels. Furthermore, we introduce a Scale Compression Mechanism (SCM) to reduce the additional computational overhead by compressing the redundant visual tokens. Our experiments demonstrate that CIP can consistently enhance the performance across diverse architectures (e.g., MiniCPM-V-2, InternVL2, and LLaVA-OneVision), various model capacity (1B → \rightarrow 8B), and different usage configurations (training-free and fine-tuning). Leveraging the proposed CIP and SCM, we introduce a lightweight MLLM, Mini-Monkey, which achieves remarkable performance in both general multimodal understanding and document understanding. On the OCRBench, the 2B-version Mini-Monkey even surpasses the 8B model InternVL2-8B by 12 12 score. Additionally, training Mini-Monkey is cheap, requiring only eight RTX 3090 3090 GPUs. The code is available at https://github.com/Yuliang-Liu/Monkey .

## 1 Related Work

### 1.1 Multimodal Large Language Models

In recent years, Large Language Models (LLMs) have made significant progress [, , , ]. Drawing from this advancement, many efforts have been made to integrate a vision encoder into Large Language Models for vision-language understanding. A commonly employed approach is the linear projector method[, ], which maps the output of the vision encoder to the same feature space as the text features of the Large Language Models. Some methods, such as Q-Former [], Perceiver Resampler [], or Abstractor [], introduce a set of learnable queries to facilitate this integration. Despite these notable advances, previous methods often struggle with detailed scene understanding due to limitations in resolution.

To address this issue, recent research has adopted two primary strategies: 1) The direct use of visual encoders that support high-res input. Some methods [, ] utilize two vision encoders, one for processing high-res images and another for low-res images. Others [, ] leverage a single vision encoder. However, these methods require additional training data and parameters, or processing attention over high-res images significantly increases computational demands. 2) An alternative, more resource-efficient method is the cropping strategy, which divides the high-res image into multiple lower-resolution sub-images for processing. Some methods apply a fixed-size cropping scheme [, ]. Others adopt a dynamic cropping approach [, , ] to keep the aspect ratio of the original image. Although the cropping strategy achieves promising results on several multimodal benchmarks, it will inevitably result in a semantic sawtooth effect: 1) If an object or character is divided, it may not be recognized; 2) If the word or sentence is segmented, the semantic damage of the segmented word will be caused. For example, the word ‘Breakdown’ may be divided into ‘Break’ and ‘down’, causing semantic damage to the segmented word. This will limit the model’s ability to understand the detailed scene. Although some methods [, ] attempt to address this issue by introducing new modules, they introduce additional parameters to the original model and require training this module from scratch. In contrast, the proposed CIP is designed to be seamlessly integrated without introducing additional parameters, offering a plug-and-play solution. A recent work [] also attempts to utilize the multi-scale image to enhance the model’s capacity. However, they employ a fixed resolution, lacking the ability to dynamically generate complementary semantics based on the resolution of input images. On the contrary, The CIP dynamically constructs an image pyramid that provides complementary semantic features after cropping for the MLLMs, enabling them to richly acquire semantics at all levels.

### 1.2 Lightweight Multimodal Large Language Models

Due to the substantial computational costs associated with multimodal large language models (MLLMs), some recent efforts have focused on developing more efficient models for rapid development and real-world applications. For instance, LLaVA-Phi [] and Imp [] integrate a lightweight large language model with a vision encoder to develop a powerful multimodal system. MobileVLM [] further conserves resources by integrating a lightweight downsampling projector that reduces the number of visual tokens. Bunny [] advances efficiency through an effective data compression technique, which minimizes the required pretraining dataset. TinyGPT-V [] adopts a multi-stage training process specifically designed for lightweight multimodal models. The above models support only low-res input. To improve detailed scene understanding in lightweight MLLMs, one of the most commonly used methods is the cropping strategy. For instance, MiniCPM-V series [] d employs an adaptive visual encoding method to accommodate high-res images of varying aspect ratios. Similarly, InternVL2-2B [] enhances the performance of lightweight MLLMs by adopting a dynamic high-res cropping strategy. Despite these advancements, the cropping strategy will introduce a semantic sawtooth effect, which significantly limits the performance of lightweight multimodal large language models. Larger MLLMs with enhanced comprehension capabilities can alleviate this issue to some extent, as discussed in Sec. 3.4.

Figure: Figure 2: Overall architecture of Mini-Monkey. H-Attn represents high attention weights. L-Attn represents low attention weights. The tokens with low attention weights will be filtered. The shared LLM’s Layer represents using the block layer from LLM in SCM.
Refer to caption: x2.png

## 2 Mini-Monkey

The overall architecture is illustrated in Fig. 2. Mini-Monkey consists of a CIP, a vision encoder, an MLP layer, a Scale Compression Mechanism, and a Large Language Model (LLM). Initially, CIP dynamically generates an image pyramid based on the resolution of input images. Then, we divide these images into a set of sub-images. These sub-images are then processed by the vision encoder and MLP layer to extract image tokens. The Scale Compression Mechanism adjusts these image tokens and forwards them to the LLM, which subsequently generates the final answers.

### 2.1 Complementary Image Pyramid

Existing cropping strategy [, ] directly divides the high-res images into a set of sub-images that will lead to a semantic sawtooth effect. To address this issue, we propose a plug-and-play method, termed complementary image pyramid (CIP), to promote synergy among images at varying scales to alleviate the semantic sawtooth effect. The process of CIP is shown in Fig. 2 (b).

Grouping Operation. We first generate a set of pre-defined aspect ratios. During testing, we limit the maximum number of tiles to 24. A greater number of tiles corresponds to images with higher resolution. These aspect ratios are then categorized into three groups through a grouping operation, including a detailed group $D_{g}$, an adaptive group $A_{g}$, and a global group $G_{g}$. The classification is based on the following criteria: (1) Aspect ratios that result in more than 9 tiles being allocated to the detailed group, enabling the largest possible image size and thus a clearer depiction of the objects within. (2) For aspect ratios producing between 3 and 8 tiles, we classify them into the adaptive group, which is responsible for enhancing the fine details at the borders of the crops. (3) The 1:1 aspect ratio is designated to the global group, providing a low-res, comprehensive view of the whole image. The grouping operation generates three groups of different aspect ratios for generating the image.

Dynamically Generating Images. After the grouping operation, we will select one aspect ratio from each group to produce a total of three images. For the detailed group, we calculate the aspect ratio of the input image and then compare it with the aspect ratios within the detailed group by calculating the absolute differences. The aspect ratio that best matches the input image, denoted as $D_{h},D_{w}$, where $D_{h}$ is the number of height tiles and $D_{w}$ is the number of width tiles, is then used to resize the image and divide the resized image into a series of sub-images. After obtaining the detailed image, the adaptive group will dynamically generate an aspect ratio based on $D_{h},D_{w}$, ensuring that the cropping lines on the detailed group and those on the adaptive group do not overlap. To achieve this, we ensure that the aspect ratios of the adaptive and detailed groups are not integer multiples of one another. Consequently, we will dynamically remove any aspect ratios that are exact multiples of $D_{h},D_{w}$.
This can be formulated as follows:

$$ $\forall k\in\mathbb{Z},\,D_{h}\neq k\cdot A_{h}\quad\text{and}\quad D_{w}\neq k\cdot A_{w}$ (1) $$

where $A_{h}$ and $A_{w}$ denote the height and width components of the aspect ratios in the adaptive group, respectively. Then, we will resize the image by selecting the ratio closest to the aspect ratio of the original image from the remaining aspect ratio.

Due to the independent processing of sub-images in the vision encoder, previous cropping-based MLLMs have not facilitated feature interaction across different sub-images. In our method, adaptive group employ a distinct aspect ratio to partitioning windows compared to detailed group, thereby simulating cross-window feature interactions within the detailed group. The detailed image provides detailed information about the image. The adaptive image enhances the fine features at the cropping boundaries of the detailed image. The global image captures the global information of the entire image for the model. Different from previous method [, ], the proposed CIP alleviates the semantic sawtooth effect from the perspective of the image, bringing several advantages: (i) it is plug-and-play, requiring no additional parameters; (ii) it seamlessly integrates with existing MLLMs that utilize cropping strategies, leading to consistent performance improvements; and (iii) it can be utilized without training and its effectiveness can be further improved through fine-tuning.

### 2.2 Scale Compression Mechanism

Although the proposed CIP significantly enhances model performance, certain scenarios may restrict the level of computational resources available. To tackle this challenge, we introduce a parameter-free token compression method called the Scale Compression Mechanism (SCM), which is used to reduce the visual tokens, as shown in Fig. 2 (c). The detailed group provides tokens with lower information density, whereas the adaptive and global groups yield tokens that are more information-dense. Therefore, we primarily focus on compressing the tokens from the detailed group. Specifically, a well-trained LLM from MLLM can effectively select the necessary visual features based on the input question. Consequently, SCM utilizes the first and second layers of the LLM to select visual tokens without generating any additional parameters. The input visual token including $\mathbf{V_{d}}\in\mathbb{R}^{L_{1}\times C}$, $\mathbf{V_{a}}\in\mathbb{R}^{L_{2}\times C}$, and $\mathbf{V_{g}}\in\mathbb{R}^{L_{3}\times C}$, and the textual token $\mathbf{T_{t}}\in\mathbb{R}^{T\times C}$ will be sent into an LLM’s Layer. $\mathbf{V_{d}}$ represents the tokens from the detailed group. $\mathbf{V_{a}}$ represents the tokens from the adaptive group. $\mathbf{V_{g}}$ represents the tokens from the global group. Notable, we reuse the layer of the LLM as this LLM’s Layer. The LLM’s Layer will output an attention map. We use the visual token from the adaptive group, global group, and textual token to attend to the visual token from the detailed group. The calculation of the attention can be formulated as follows:

$$ $\mathbf{Q}={\tt cat}(\mathbf{V_{a}},\mathbf{V_{g}},\mathbf{T_{t}})\,,$ (2) $$

$$ $\mathbf{Attn_{w}}={\tt softmax}(\frac{\mathbf{Q}+{\rm PE}(\mathbf{Q}))(\mathbf{V_{d}}+{\rm PE}(\mathbf{V_{d}}))^{T}}{\sqrt{D}})\,.$ (3) $$

where $\rm PE$ represents the position encoding and $D$ denotes the dimension of the LLM. $Cat()$ represents the sequence concatenation operation. After computing the attention mechanism, we average the first dimension of the attention map $\mathbf{Attn_{w}}\in\mathbb{R}^{(L_{2}+L_{3}+T)\times L_{1}}$ to obtain a weight vector $\mathbf{W_{a}}\in\mathbb{R}^{L_{1}}$. Subsequently, we select the top $K$ visual features from detailed layers based on this weight vector $\mathbf{W_{a}}$. These selected tokens, along with tokens from the adaptive group, global group, and textual token, are input into the LLM to generate the results. Compared to FastV [], SCM works in conjunction with the CIP and is more targeted by using tokens with high relative information density to compress tokens with low information density.

## 3 Experiments

### 3.1 Implementation Details

We use InternVL2-2B [] as the Baseline to develop the Mini-Monkey. Following previous work [], we use the (448, 448) as the input resolution of InternViT. The training datasets used to train the model include DocVQA [], ChartQA [], DVQA [], AI2D [], GeoQA+ [], and LLaVA-150K (zh) [].
We use the AdamW [] as the optimizer. The base learning rate is 4e-8.

Evaluation. Following the previous work [, ], we evaluate Mini-Monkey on eleven general multimodal understanding benchmarks, including MathVista testmini [], SEED Image [], RealWorldQA [], AI2D test [], POPE [], CCBench [], MME [], and HallusionBench []. For document understanding, following the previous work [], we employ two distinct types of metrics to verify the performance of Mini-Monkey. Initially, we leverage the standard metrics provided by the benchmarks to evaluate Mini-Monkey. We utilize benchmarks such as ChartQA [], DocVQA [], InfoVQA [], TextVQA [], STVQA [], FUNSD [], SROIE [], POIE [] and OCRBench []. We also apply the accuracy metric to verify the performance. For this metric, a response from Mini-Monkey that fully captures the ground truth is considered a true positive. Further details on this metric and the used benchmarks can be referenced in appendix LABEL:more_implementation.

**Table 1: Comparison with SoTA models on 8 multimodal benchmarks. General multimodal benchmarks encompass: MME [], RealWorldQA [], AI2D test [], CCBench [], SEED Image [], HallusionBench [], and POPE []. Additionally, the math dataset includes MathVista testmini []. The MME results we report are the sum of the perception and cognition scores. ^§ represents the results from the OpenCompass leaderboard [].**
|  |  | General Multimodal Benchmarks | Math |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| model | #param | MME | RWQA | AI2D | CCB | SEED | HallB | POPE | MathVista |
| QWEN-VL [] | 7B | 1848.3 | 49.3^§ | 63^§ | 65.7^§ | 52.5^§ | 29.9^§ | 70^§ | 34.9^§ |
| Mini-Gemini [] | 35B | 2141.0 | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | 43.3 |
| LLaVA-NeXT [] | 35B | 2028.0 | $-$ | 74.9 | 49.2 | 75.9 | 34.8 | 89.6^§ | 46.5 |
| InternVL 1.2 [] | 40B | 2175.4 | 67.5 | 79.0 | 59.2 | 75.6 | 47.6 | 88.0 | 47.7 |
| InternVL 1.5 [] | 26B | 2187.8 | 66.0 | 80.7 | 69.8 | 76.0 | 49.3 | 88.3 | 53.5 |
| DeepSeek-VL [] | 1.7B | 1531.6 | 49.7^§ | 51.5^§ | 37.6^§ | 43.7^§ | 27.6^§ | 85.9^§ | 29.4 |
| Mini-Gemini [] | 2.2B | 1653.0 | - | - | - | - | - | - | 29.4 |
| Bunny-StableLM-2 [] | 2B | 1602.9 | - | - | - | 58.8 | - | 85.9 | - |
| MiniCPM-V-2 [] | 2.8B | 1808.6 | 55.8^§ | 62.9^§ | 48.0^§ | - | 36.1^§ | 86.3^§ | 38.7 |
| InternVL 2 [] | 2B | 1876.8 | 57.3 | 74.1 | 74.7 | 70.9^§ | 37.9 | 85.2^§ | 46.3 |
| Mini-Monkey (ours) | 2B | 1884.2 | 57.9 | 74.8 | 75.5 | 71.3 | 38.8 | 88.0 | 47.3 |

**Table 2: Comparison to state-of-the-art MLLMs on OCR-related Tasks. Mini-Monkey achieves the best results among the 2B-parameter MLLMs. ^§ represents the results from the OpenCompass leaderboard [].**
| Model | Model Size | DocVQA^Test | ChartQA^Test | InfoVQA^Test | TextVQA^Val | OCRBench |
| --- | --- | --- | --- | --- | --- | --- |
| TextMonkey [] | 9B | 73.0 | 66.9 | 28.6 | 65.6 | 558 |
| TextHawk [] | 7B | 76.4 | 66.6 | 50.6 | — | — |
| DocKylin [] | 7B | 77.3 | 46.6 | 66.8 | — | — |
| HiRes-LLaVA [] | 7B | 74.7 | 61.5 | 48.0 | 65.4 | — |
| LLaVA-UHD [] | 13B | — | — | — | 67.7 | — |
| CogAgent [] | 17B | 81.6 | 68.4 | 44.5 | 76.1 | 590 |
| UReader [] | 7B | 65.4 | 59.3 | 42.2 | 57.6 | — |
| DocOwl 1.5 [] | 8B | 82.2 | 70.2 | 50.7 | 68.6 | — |
| HRVDA [] | 7B | 72.1 | 67.6 | 43.5 | — | — |
| TextSquare [] | 7B | 84.3 | 79.4 | 51.5 | 66.8 | 622 |
| IXC2-4KHD [] | 8B | 90.0 | 81.0 | 68.6 | 77.2 | 675 |
| InternVL 1.5 [] | 26B | 90.9 | 83.8 | 72.5 | 80.6 | 724 |
| InternVL 2 [] | 8B | 91.6 | 83.3 | 74.8 | 77.4 | 794 |
| GLM4-V [] | 9B | - | - | - | - | 786 |
| Vary-toy [] | 1.8B | 65.6 | 59.1 | - | - | - |
| MiniCPM-V 2.0 [] | 2.8B | 71.9 | 55.6^§ | - | 74.1 | 605 |
| InternVL 2 [] | 2B | 86.9 | 76.2 | 58.9 | 73.4 | 784 |
| Mini-Monkey (Ours) | 2B | 87.4 | 76.5 | 60.1 | 76.0 | 806 |

**Table 3: Quantitative accuracy (%) comparison of our model with existing multimodal large language models (MLLMs) on several benchmarks. Following TextMonkey [], we use the accuracy metrics to evaluate our method.**
| Method | Scene Text-Centric VQA | Document-Oriented VQA | KIE |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| STVQA | TextVQA | DocVQA | InfoVQA | ChartQA | FUNSD | SROIE | POIE |  |
| BLIP2-OPT-6.7B [] | 20.9 | 23.5 | 3.2 | 11.3 | 3.4 | 0.2 | 0.1 | 0.3 |
| mPLUG-Owl [] | 30.5 | 34.0 | 7.4 | 20.0 | 7.9 | 0.5 | 1.7 | 2.5 |
| InstructBLIP [] | 27.4 | 29.1 | 4.5 | 16.4 | 5.3 | 0.2 | 0.6 | 1.0 |
| LLaVAR [] | 39.2 | 41.8 | 12.3 | 16.5 | 12.2 | 0.5 | 5.2 | 5.9 |
| BLIVA [] | 32.1 | 33.3 | 5.8 | 23.6 | 8.7 | 0.2 | 0.7 | 2.1 |
| mPLUG-Owl2-8 [] | 49.8 | 53.9 | 17.9 | 18.9 | 19.4 | 1.4 | 3.2 | 9.9 |
| LLaVA1.5-7B [] | 38.1 | 38.7 | 8.5 | 14.7 | 9.3 | 0.2 | 1.7 | 2.5 |
| TGDoc [] | 36.3 | 46.2 | 9.0 | 12.8 | 12.7 | 1.4 | 3.0 | 22.2 |
| UniDoc [] | 35.2 | 46.2 | 7.7 | 14.7 | 10.9 | 1.0 | 2.9 | 5.1 |
| DocPedia [] | 45.5 | 60.2 | 47.1 | 15.2 | 46.9 | 29.9 | 21.4 | 39.9 |
| Monkey-8B [] | 54.7 | 64.3 | 50.1 | 25.8 | 54.0 | 24.1 | 41.9 | 19.9 |
| InternVL-8B [] | 62.2 | 59.8 | 28.7 | 23.6 | 45.6 | 6.5 | 26.4 | 25.9 |
| InternLM-XComposer2-7B [] | 59.6 | 62.2 | 39.7 | 28.6 | 51.6 | 15.3 | 34.2 | 49.3 |
| TextMonkey-9B [] | 61.8 | 65.9 | 64.3 | 28.2 | 58.2 | 32.3 | 47.0 | 27.9 |
| InternVL2-2B [] | 65.6 | 66.2 | 76.7 | 46.8 | 67.6 | 42.0 | 68.0 | 66.8 |
| Mini-Monkey-2B (Ours) | 67.2 | 68.8 | 78.4 | 50.0 | 67.3 | 43.2 | 70.5 | 71.2 |

**Table 4: Ablation study of Complementary Image Pyramid. We compare our method with the existing cropping strategy and the overlay cropping strategy.**
| Model | Resolution Strategy | TextVQA | OCRBench | MME | HallB | POPE |
| --- | --- | --- | --- | --- | --- | --- |
| Baseline | Dynamic high-res Strategy [] | 73.4 | 784 | 1876.8 | 37.9 | 85.2 |
| Baseline | Fixed Size high-res Strategy [] | 74.2 | 772 | 1824.5 | 37.6 | 85.0 |
| Baseline | Overlapping Cropping Strategy | 70.6 | 758 | 1874.1 | 36.8 | 83.5 |
| Baseline | Multi-Scale Strategy [] | 74.8 | 776 | 1846.8 | 38.1 | 85.3 |
| Mini-Monkey (Ours) | Complementary Image Pyramid | 76.0 | 806 | 1884.2 | 38.8 | 88.0 |

### 3.2 Comparison with State-of-the-art Methods

General Multimodal Understanding. We evaluate Mini-Monkey on general multimodal understanding following  [, ]. The results are shown in Tab. 1. Mini-Monkey surpasses other 2B-parameter models on 8 benchmarks. On the MathVista and POPE, Mini-Monkey outperforms the previous state-of-the-art method InternVL2-2B by 1% and 2.8%, respectively. On the HallusionBench, Mini-Monkey outperforms MiniCPM-V-2 by 2.7%. These results showcase the ability of Mini-Monkey to handle general multimodal understanding and reasoning tasks.

Document Understanding. For the first type of metric, the results are presented in Tab. 2. We use a relaxed accuracy measure for ChartQA, ANLS for DocVQA and InfoVQA, and the VQA score for TextVQA. The results indicate that Mini-Monkey achieves state-of-the-art performance among 2B-parameter multimodal large language models. Compared to InternVL2-2B, our method outperforms it by 2.6%, 1.2%, and 22 for TextVQA, InfoVQA, and OCRBench, respectively. Due to the small original resolution of ChartQA, it is less impacted by cropping operations, resulting in a minor improvement from our method. Notably, in the OCRBench, Mini-Monkey even surpasses the 8B-parameter Large Multimodal Model InternVL2-8B and the 9B-parameter Large Multimodal Model GLM4-V by 12 and 20, respectively. These results demonstrate the advantages of a Complementary Image Pyramid in enhancing document understanding. For the accuracy metric, the results are shown in Tab. 3. Mini-Monkey shows an average performance improvement of 15.6% compared to TextMonkey-9B [], demonstrating the effectiveness of our method. Mini-Monkey also outperforms the state-of-the-art methods on multiple text-related benchmarks. Specifically, Mini-Monkey outperforms the InternVL2-2B by 2.6%, 3.2%, and 4.4% on TextVQA, InfoVQA, and POIE, respectively. These results further indicate the great potential of Mini-Monkey for document understanding tasks.

### 3.3 Ablation Study

In this section, we perform ablation studies on both general multimodal understanding and document understanding benchmarks to validate the effectiveness of our method. We adopt the TextVQA [], OCRBench [], HallusionBench [], MME [], and POPE [] to conduct ablation studies.

Complementary Image Pyramid.
We conducted ablation studies to investigate the effectiveness of the CIP. We compared our method with several alternatives: The dynamic high-res strategy [], which maintains aspect ratios to increase resolution. The fixed-size high-res strategy [], which uses a fixed size to increase resolution. The overlapping cropping strategy uses a high-res approach but crops with overlay. The multi-scale strategy [], which introduces a multi-scale strategy to the MLLM. As presented in Tab. 4, the proposed CIP achieved the best results. Our method outperforms the previous multi-scale strategy [] on both general multimodal understanding and document understanding. Notably, the over-overlay cropping strategy, instead of improving the model’s performance, actually degraded it.

Various Model Capacity. We performed ablation studies to assess the impact of CIP on models with varying model capacity. As illustrated in Table 6, CIP consistently improves the performance of varying model capacity.

Different Usage Configurations. To further validate the improvements introduced by CIP, we performed experiments on usage configurations: a training-free configuration and a fine-tuning configuration. As shown in Tab. 5, CIP demonstrates improvements in performance even when applied without training. The performance can be further improved with fine-tuning. Additionally, we surprisingly find that CIP can even facilitate the model fine-tuning process. As presented in the second line in Tab. 5, direct fine-tuning of the Baseline model not only failed to improve performance but, in some cases, led to a decline. Conversely, incorporating CIP during the fine-tuning of the Baseline resulted in substantial improvements in both general multimodal understanding and document understanding, as evidenced in the fourth line of Tab. 5.

Incorporating CIP to various MLLMs. The proposed complementary image pyramid (CIP) can be seamlessly integrated into crop-based methods. To demonstrate its effectiveness, we incorporated CIP into various structures of MLLM, such as MiniCPM-V-2 [], InternVL 2 [], LLaVA-OV []. As shown in Tab. 6, CIP consistently enhances the performance across different MLLM structures, thereby validating the effectiveness of our approach.

The Number of Sub-Images. To investigate whether the performance enhancement is attributed to an increase in the number of sub-images, we performed an experiment by incrementally raising the sub-image count for the Baseline. The findings, summarized in Table 8, indicate that increasing the number of sub-images does not lead to better performance; instead, it may result in a decline. In contrast, CIP can effectively improve the performance of the model. These results further demonstrate the effectiveness of the CIP.

**Table 5: Exploring different usage configurations of CIP. Train represents fine-tuning the model.**
| Model | CIP | Train | TextVQA | OCRBench | MME | HallB | POPE |
| --- | --- | --- | --- | --- | --- | --- | --- |
| InternVL2-2B | $\times$ | $\times$ | 73.4 | 784 | 1876.8 | 37.9 | 85.2 |
| InternVL2-2B | $\times$ | $\checkmark$ | 73.3 (-0.1) | 787 (+3) | 1858.3 (-18.5) | 37.3 (-0.6) | 85.3 (+0.1) |
| InternVL2-2B | $\checkmark$ | $\times$ | 75.2 (+1.8) | 800 (+17) | 1881.9 (+5.1) | 38.7 (+0.8) | 86.7 (+1.5) |
| InternVL2-2B | $\checkmark$ | $\checkmark$ | 76.0 (+2.6) | 806 (+22) | 1884.2 (+7.4) | 38.8 (+0.9) | 88.0 (+2.8) |

**Table 6: Ablation study of incorporating complementary image pyramid (CIP) to other MLLMs. ^$\mathsection$ represents the results from the OpenCompass leaderboard [].**
| Model | CIP | TextVQA | OCRBench | MME | HallB | POPE |
| --- | --- | --- | --- | --- | --- | --- |
| MiniCPM-V-2-2.8B | $\times$ | 74.1 | 605 | 1808.6 | 36.1^$\mathsection$ | 86.3^$\mathsection$ |
| MiniCPM-V-2-2.8B | $\checkmark$ | 76.0 (+1.9) | 627 (+22) | 1819.5 (+10.9) | 36.5 (+0.4) | 87.1 (+0.8) |
| LLaVA-OV-0.5B | $\times$ | 65.3 | 577 | 1478.0 | 28.1 | 86.7 |
| LLaVA-OV-0.5B | $\checkmark$ | 66.2 (+0.9) | 600 (+23) | 1482.6 (+4.6) | 28.8 (+0.7) | 87.7 (+1.0) |
| InternVL2-1B | $\times$ | 70.5 | 754 | 1794.4 | 33.4 | 84.9^$\mathsection$ |
| InternVL2-1B | $\checkmark$ | 72.3 (+1.8) | 772 (+18) | 1801.5 (+7.1) | 34.3 (+0.9) | 85.7 (+0.8) |
| InternVL2-2B | $\times$ | 73.4 | 784 | 1876.8 | 37.9 | 85.2 |
| InternVL2-2B | $\checkmark$ | 76.0 (+2.6) | 806 (+22) | 1884.2 (+7.4) | 38.8 (+0.9) | 88.0 (+2.8) |
| InternVL2-8B | $\times$ | 77.4 | 794 | 2210.3 | 45.0^$\mathsection$ | 84.2^$\mathsection$ |
| InternVL2-8B | $\checkmark$ | 79.3 (+1.9) | 809 (+15) | 2226.4 (+16.1) | 45.4 (+0.4) | 84.8 (+0.6) |

**Table 7: Ablation study of the scale compression mechanism. We used different compression ratios to compare with FastV []. (0.5) represents 50% compression and (0.9) represents 90% compression.**
| Model | Resolution Strategy | TextVQA | OCRBench | MME | HallB | POPE |
| --- | --- | --- | --- | --- | --- | --- |
| Mini-Monkey | Pooling (0.5) | 47.6 | 256 | 1765.2 | 31.5 | 84.5 |
| Mini-Monkey | Random (0.5) | 63.5 | 503 | 1805.5 | 36.2 | 85.9 |
| Mini-Monkey | FastV [] (0.5) | 73.4 | 781 | 1848.0 | 38.3 | 83.9 |
| Mini-Monkey | FastV [] (0.9) | 73.9 | 792 | 1866.1 | 37.5 | 85.8 |
| Mini-Monkey | SCM (0.5) | 74.7 | 794 | 1886.0 | 38.7 | 86.1 |
| Mini-Monkey | SCM (0.9) | 75.2 | 801 | 1884.7 | 38.6 | 86.2 |

Scale Compression Mechanism. We compared the proposed Scale Compression Mechanism with the related work FastV []. For different methods, we compress the number of visual tokens by 50%. For our method and FastV, we further conduct an experiment with 90% compression. Following FastV’s paper, we set the K in FastV as 2. As illustrated in Tab. 7, when using 50% compression and 90% compression, our method outperformed FastV by 21.5% and 4.4%, respectively, demonstrating its effectiveness. FastV compresses input tokens, including both visual and textual tokens, within Transformer blocks. In contrast, our method works in conjunction with the CIP and more targeted by using tokens with high relative information density to compress tokens with low information density.

**Table 8: Ablation study on the number of sub-images. The number denotes the sub-image count.**
| Model | Number | TextVQA | OCRBench | MME | HallB | POPE |
| --- | --- | --- | --- | --- | --- | --- |
| Baseline | 18 | 74.2 | 782 | 1851.7 | 37.0 | 85.8 |
| Baseline | 24 | 74.4 | 783 | 1857.6 | 36.9 | 85.8 |
| Baseline | 32 | 74.3 | 782 | 1845.0 | 36.9 | 85.9 |
| Baseline | 48 | 74.0 | 767 | 1841.6 | 36.2 | 85.7 |
| CIP | 32 | 76.0 | 806 | 1884.2 | 38.8 | 88.0 |

Figure: Figure 3: Qualitative results of Mini-Monkey. (a) Input Image and Ground Truth. (b) The results of using overlapping cropping strategy. OSC represents the overlapping cropping strategy. (c) The results of InternVL2-2B and InternVL2-26B. (d) The results of Mini-Monkey.
Refer to caption: x3.png

### 3.4 Qualitative Results

In this section, we provide some qualitative results to demonstrate the effectiveness of our method. First, we verify that the semantic sawtooth effect is particularly evident in lightweight MLLMs, which adopt InternVL2-2B and InternVL2-26B. As shown in Fig. 3(c), InternVL2-26B can answer the questions correctly. However, due to the word ‘classrooms’ and ‘school’ being cropped, InternVL2-2B gives a wrong answer that addresses the text in the bottom left corner of the original image. Mini-Monkey can overcome this semantic sawtooth effect and provide the correct answer, as presented in Fig. 3(d). Comparing Fig. 3(b) and Fig. 3(d), we can see that the overlapping cropping strategy introduces some hallucinations and cannot answer questions accurately based on the image, whereas our methods can effectively address the semantic sawtooth effect.

## 4 Conclusion

In this study, we introduce a Complementary Image Pyramid (CIP) designed to alleviate the semantic sawtooth effect for MLLMs, thereby enhancing their capability to process high-resolution images effectively. CIP is plug-and-play and can be seamlessly integrated into various multimodal large language models at a low cost. We demonstrate the effectiveness of the proposed CIP across diverse architectures, various parameters, and different usage configurations, leading to consistent performance improvements. Besides, we present a Scale Compression Mechanism (SCM) to compress the visual tokens for computational efficiency. CIP not only enhances the general multimodal understanding performance but also shows consistent improvements in document understanding tasks. Furthermore, our experimental results demonstrate that 2B-parameter MLLM equipped with CIP even surpasses larger 8B-parameter state-of-the-art models like InternVL2-8B on the OCRBench. Limitations: To ensure the seamless application of our method across various architectures, we adopt an image-centric approach to construct an image pyramid without introducing additional parameters. In future work, we will explore the use of trainable Feature Pyramid Network (FPN) for MLLMs, aiming to more efficiently leverage multi-scale features.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: 1. There is a lack of explanation for the setting of the Adaptive Group. While both detail and global are easy to understand, the paper does not explicitly state the benefits of detail group or provide experimental evidence to support it.
2. The experiments lack ablation studies, such as removing the detail, adaptive, or global components. What would be the impact if one of these were removed? Which component is most critical? Secondly, what impact does the number of tiles in CIP have? How does the performance of CIP change with the number of tiles in the detail component? Also, what aspect ratios are set by default for CIP?
3. The motivation behind SCM does not align well with the experiments. The paper mentions that "certain scenarios may restrict the level of computational resources available," but the experimental part does not provide experiments on how different compression rates of SCM affect model acceleration and computational cost.

If the authors could supplement their experiments, I would be willing to raise the score to above borderline.
soundness: 3
strengths: 1. The proposed CIP is logically clear and reasonable. Experiments show that CIP outperforms other cropping methods.
2. The experiments are comprehensive, showing significant improvements across various model families and sizes, as well as multiple datasets, which demonstrates the effectiveness of the proposed CIP.
3. The paper is well-written and clear.
summary: The paper introduces the "semantic sawtooth effect" caused by common cropping strategies in high-resolution image scaling for MLLMs. To tackle this issue, they propose a Complementary Image Pyramid (CIP), a flexible and easy-to-integrate approach aimed at reducing semantic discontinuity by providing rich semantic information across different scales. Alongside CIP, they also introduce a Scale Compression Mechanism (SCM) to minimize computational overhead by compressing unnecessary visual tokens. These enhancements improve performance across various MLLM architectures and capacities, leading to the development of a lightweight model called Mini-Monkey, which shows notable improvements in multimodal and document understanding tasks.
weaknesses: 1. The insight and inspiration of proposed Adaptive Group is not clear.

2. Lack ablation studies on proposed CIP and SCM

See below for details.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: 1、I would like to inquire why the paper does not mention the maximum resolution of the images that the model supports, as well as the corresponding comparative experiments.
soundness: 3
strengths: 1、The paper describes CIP for dynamic segmentation of images and SCM for compression of visual tokens to address the semantic sawtooth effect in MLLM high-resolution image processing, demonstrating innovations in addressing specific challenges.
2、In the CIP module, the model focuses on the feature interactions of different sub-images
3、In the SCM module, the model selectively compresses visual tokens. The interaction information of different types of visual tokens is also considered.
summary: This paper proposes Mini-Monkey, a lightweight multimodal large language model that effectively mitigates the semantic sawtooth effect in high-resolution image processing through a Complementary Image Pyramid (CIP) and a Scale Compression Mechanism(SCM), achieving superior performance across various benchmarks.
weaknesses: 1、In Figure 2a, the pixel shuffle operation appears, but the paper does not reflect the transformation of the image features before and after this operation.
2、According to the formula in line 241, the aspect ratios of the adaptive and detailed groups are not integer multiples. But for Figure 2b, the final selection of Ah is 1 and Dh is 3, which seems to be a contradiction.
3、In the CIP module, the paper does not present a clear picture of how the predefined slice ratios appropriate to the size of the image are selected, i.e., what principle is it based on.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: Please refer to the Weaknesses.
soundness: 3
strengths: 1. The use of the complementary image pyramid (CIP) to replace high-resolution input images with sub-images of various scales is an excellent idea. Compared to existing methods that introduce multi-scale visual signals through models, this approach is simpler and more effective, without requiring additional parameters or training.

2. The Scale Compression Mechanism (SCM) reasonably and interestingly reduces the extra computational load brought by multi-scale input sub-images by compressing visual tokens.

3. Both the proposed CIP and SCM do not require the introduction of additional parameters or training, making them applicable to different MLLMs.

4. The method proposed in this paper achieves promising results in various general multimodal understanding and document understanding benchmarks.
summary: Existing multimodal large language models (MLLMs) often use cropping when processing high-resolution images (divides the high-res image into multiple lower-resolution as the input). However, Non-Overlapping Cropping can lead to semantic discontinuity and semantic damage, referred to by the authors as the "semantic sawtooth effect." On the other hand, Overlapping Cropping results in redundant visual information.

To address this, the paper proposes a complementary image pyramid, which aims to alleviate the semantic sawtooth effect in the context of Non-Overlapping Cropping. To mitigate the additional computational burden introduced by this module, the authors propose a Scale Compression Mechanism. This mechanism leverages the attention weights of the LLM and the proposed multi-scale image semantics in a training-free and parameter-free manner to compress redundant tokens.

The proposed approach achieves promising results on 8 general multimodal understanding benchmarks and 9 document understanding benchmarks.
weaknesses: 1. The authors claim in lines 265-266 that "a well-trained LLM from MLLM can effectively select the necessary visual features based on the input question," which seems to differ from the conclusions of existing MLLM works [1,2]. This discrepancy makes me question the effectiveness of the proposed method. If MLLMs cannot truly understand images, how can their attention weights be used here to compress visual tokens?

2. The rationale for selecting only the first and second layers of the LLM to choose visual tokens is not sufficiently explained, and no ablation studies have been conducted. How would the results differ if more LLM layers were selected, only one LLM layer was chosen, or the selection was done randomly without using LLM attention priors? In conclusion, ablation studies have only been conducted on the Resolution Strategy, lacking ablation experiments on the compression of visual tokens.

3. For the complementary image pyramid, the authors need to manually preset a set of predefined aspect ratios, which seems somewhat tricky. How these aspect ratios are set and why these specific values are chosen remains unclear. A better solution might be to perform K-means clustering on the resolution ratios of images and use the clustering results as the predefined aspect ratios.

4. When comparing with other methods and conducting ablation studies, only the number of parameters and performance are shown, lacking comparisons on FLOPs. Although your proposed multi-scale input does not introduce new parameters, it does increase actual computational load. Therefore, in the ablation studies of Table 4, the actual computational load and inference overhead should also be compared.

Reference 
* [1] Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs, NeurIPS 2024
* [2] Are We on the Right Way for Evaluating Large Vision-Language Models?, NeurIPS 2024

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: Please see the weaknesses.
soundness: 3
strengths: 1. Mini-Monkey tackles an important problem in MLLM: scaling images to high resolution. The Complementary Image Pyramid (CIP) introduces the pyramid structure, which is an interesting idea. Besides, CIP and SCM are plug-and-play, which can be easily integrated into different MLLMs.
2. The experiments are sufficient.
summary: The paper focuses on the issue of semantic discontinuity in MLLM when scaling images to high resolution, particularly through a sliding-window cropping strategy that can misidentify small or irregularly shaped objects. To tackle this problem, the paper proposes the Complementary Image Pyramid (CIP), which dynamically constructs an image pyramid to enhance semantic information. Besides, the authors introduce a Scale Compression Mechanism (SCM) to minimize computational overhead by compressing redundant visual tokens. Experimental results show the proposed method achieves the best performance across diverse benchmarks.
weaknesses: 1. Implications of this research. Whether the semantic sawtooth effect mentioned in the paper is a necessary issue to investigate, common Crop-based methods (such as LLaVA-UHD [1] and InternVL [2]) put all cropped regions into a sequence, which does not affect semantic continuity.
2. The work is incremental. The core crop strategy has been widely used in other approaches. CIP is an incremental improvement and doesn't mean much to the community.
3. The architecture is highly sophisticated. The global group is enough to solve the loss of fine-grained features caused by the detailed group. This brings the question of whether the proposed adaptive group is necessary for CIP.
4. The writing needs further improvement. Authors are suggested to improve the readability of the paper. For example, it is hard to understand "For the detailed group, we calculate the aspect ratio of the input image and then compare it with the aspect ratios within the detailed group by calculating the absolute differences." in L231-L233. How to compare? Another example: L270 "We reuse the layer of the LLM as this LLM's Layer". What's the difference between two LLMs?
5. Incomplete experimental analysis. Experimental analysis should include analysis of reasons and not just a list of indicators.

[1] Xu R, Yao Y, Guo Z, et al. Llava-uhd: an lmm perceiving any aspect ratio and high-resolution images[J]. arXiv preprint arXiv:2403.11703, 2024.

[2] Chen Z, Wu J, Wang W, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 24185-24198.

Meta-review: This paper tackles the high resolution image requirement in MLLM, and propose a simple, plug-and play method named Complementary Image Pyramid (CIP) to mitigate the semantic sawtooth effect of patch tileing. The reviewers approve for the effectiveness of the proposed method, and detailed ablation study may help the MLLM community to help for image preprocessing. The AC	recommend for accept based on the reviewers’ opinion.

Decision: Accept (Poster)