Mini-Monkey: Alleviating the Semantic Sawtooth Effect for Lightweight MLLMs via Complementary Image Pyramid

Authors: Mingxin Huang, Yuliang Liu, Dingkang Liang, Lianwen Jin, Xiang Bai
Primary area: foundation or frontier models, including LLMs
Keywords: Multimodal Large Language Model, Document Understanding

Abstract:
Recently, scaling images to high resolution has received much attention in multimodal large language models (MLLMs). Most existing practices adopt a sliding-window-style cropping strategy to adapt to resolution increase. Such a cropping strategy, however, can easily cut off objects and connected regions, which introduces semantic discontinuity and therefore impedes MLLMs from recognizing small or irregularly shaped objects or text, leading to a phenomenon we call the semantic sawtooth effect. This effect is particularly evident in lightweight MLLMs. To address this issue, we introduce a Complementary Image Pyramid (CIP), a simple, effective, and plug-and-play solution designed to mitigate semantic discontinuity during high-resolution image processing. In particular, CIP dynamically constructs an image pyramid to provide complementary semantic information for the cropping-based MLLMs, enabling it rich acquire semantics at all levels. Furthermore, we introduce a Scale Compression Mechanism (SCM) to reduce the additional computational overhead by compressing the redundant visual tokens. Our experiments demonstrate that CIP can consistently enhance the performance across diverse architectures (e.g., MiniCPM-V-2, InternVL2, and LLaVA-OneVision), various model capacity (1B$\rightarrow$8B), and different usage configurations (training-free and fine-tuning). Leveraging the proposed CIP and SCM, we introduce a lightweight MLLM, Mini-Monkey, which achieves remarkable performance in both general multimodal understanding and document understanding. On the OCRBench, the 2B-version Mini-Monkey even surpasses the 8B model InternVL2-8B by 12 score. Additionally, training Mini-Monkey is cheap, requiring only eight RTX 3090 GPUs. Code and models are available at 
https://github.com/Yuliang-Liu/Monkey.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729155073990

thread_type:
official_review

<thread_object>
creator: reviewer_q5hv

<content>
summary:
The paper focuses on the issue of semantic discontinuity in MLLM when scaling images to high resolution, particularly through a sliding-window cropping strategy that can misidentify small or irregularly shaped objects. To tackle this problem, the paper proposes the Complementary Image Pyramid (CIP), which dynamically constructs an image pyramid to enhance semantic information. Besides, the authors introduce a Scale Compression Mechanism (SCM) to minimize computational overhead by compressing redundant visual tokens. Experimental results show the proposed method achieves the best performance across diverse benchmarks.

strengths:
1. Mini-Monkey tackles an important problem in MLLM: scaling images to high resolution. The Complementary Image Pyramid (CIP) introduces the pyramid structure, which is an interesting idea. Besides, CIP and SCM are plug-and-play, which can be easily integrated into different MLLMs.
2. The experiments are sufficient.

weaknesses:
1. Implications of this research. Whether the semantic sawtooth effect mentioned in the paper is a necessary issue to investigate, common Crop-based methods (such as LLaVA-UHD [1] and InternVL [2]) put all cropped regions into a sequence, which does not affect semantic continuity.
2. The work is incremental. The core crop strategy has been widely used in other approaches. CIP is an incremental improvement and doesn't mean much to the community.
3. The architecture is highly sophisticated. The global group is enough to solve the loss of fine-grained features caused by the detailed group. This brings the question of whether the proposed adaptive group is necessary for CIP.
4. The writing needs further improvement. Authors are suggested to improve the readability of the paper. For example, it is hard to understand "For the detailed group, we calculate the aspect ratio of the input image and then compare it with the aspect ratios within the detailed group by calculating the absolute differences." in L231-L233. How to compare? Another example: L270 "We reuse the layer of the LLM as this LLM's Layer". What's the difference between two LLMs?
5. Incomplete experimental analysis. Experimental analysis should include analysis of reasons and not just a list of indicators.

[1] Xu R, Yao Y, Guo Z, et al. Llava-uhd: an lmm perceiving any aspect ratio and high-resolution images[J]. arXiv preprint arXiv:2403.11703, 2024.

[2] Chen Z, Wu J, Wang W, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 24185-24198.

questions:
Please see the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729970857893

thread_type:
official_review

<thread_object>
creator: reviewer_hrbu

<content>
summary:
Existing multimodal large language models (MLLMs) often use cropping when processing high-resolution images (divides the high-res image into multiple lower-resolution as the input). However, Non-Overlapping Cropping can lead to semantic discontinuity and semantic damage, referred to by the authors as the "semantic sawtooth effect." On the other hand, Overlapping Cropping results in redundant visual information.

To address this, the paper proposes a complementary image pyramid, which aims to alleviate the semantic sawtooth effect in the context of Non-Overlapping Cropping. To mitigate the additional computational burden introduced by this module, the authors propose a Scale Compression Mechanism. This mechanism leverages the attention weights of the LLM and the proposed multi-scale image semantics in a training-free and parameter-free manner to compress redundant tokens.

The proposed approach achieves promising results on 8 general multimodal understanding benchmarks and 9 document understanding benchmarks.

strengths:
1. The use of the complementary image pyramid (CIP) to replace high-resolution input images with sub-images of various scales is an excellent idea. Compared to existing methods that introduce multi-scale visual signals through models, this approach is simpler and more effective, without requiring additional parameters or training.

2. The Scale Compression Mechanism (SCM) reasonably and interestingly reduces the extra computational load brought by multi-scale input sub-images by compressing visual tokens.

3. Both the proposed CIP and SCM do not require the introduction of additional parameters or training, making them applicable to different MLLMs.

4. The method proposed in this paper achieves promising results in various general multimodal understanding and document understanding benchmarks.

weaknesses:
1. The authors claim in lines 265-266 that "a well-trained LLM from MLLM can effectively select the necessary visual features based on the input question," which seems to differ from the conclusions of existing MLLM works [1,2]. This discrepancy makes me question the effectiveness of the proposed method. If MLLMs cannot truly understand images, how can their attention weights be used here to compress visual tokens?

2. The rationale for selecting only the first and second layers of the LLM to choose visual tokens is not sufficiently explained, and no ablation studies have been conducted. How would the results differ if more LLM layers were selected, only one LLM layer was chosen, or the selection was done randomly without using LLM attention priors? In conclusion, ablation studies have only been conducted on the Resolution Strategy, lacking ablation experiments on the compression of visual tokens.

3. For the complementary image pyramid, the authors need to manually preset a set of predefined aspect ratios, which seems somewhat tricky. How these aspect ratios are set and why these specific values are chosen remains unclear. A better solution might be to perform K-means clustering on the resolution ratios of images and use the clustering results as the predefined aspect ratios.

4. When comparing with other methods and conducting ablation studies, only the number of parameters and performance are shown, lacking comparisons on FLOPs. Although your proposed multi-scale input does not introduce new parameters, it does increase actual computational load. Therefore, in the ablation studies of Table 4, the actual computational load and inference overhead should also be compared.

Reference 
* [1] Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs, NeurIPS 2024
* [2] Are We on the Right Way for Evaluating Large Vision-Language Models?, NeurIPS 2024

questions:
Please refer to the Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730095459728

thread_type:
official_review

<thread_object>
creator: reviewer_bhqc

<content>
summary:
This paper proposes Mini-Monkey, a lightweight multimodal large language model that effectively mitigates the semantic sawtooth effect in high-resolution image processing through a Complementary Image Pyramid (CIP) and a Scale Compression Mechanism(SCM), achieving superior performance across various benchmarks.

strengths:
1、The paper describes CIP for dynamic segmentation of images and SCM for compression of visual tokens to address the semantic sawtooth effect in MLLM high-resolution image processing, demonstrating innovations in addressing specific challenges.
2、In the CIP module, the model focuses on the feature interactions of different sub-images
3、In the SCM module, the model selectively compresses visual tokens. The interaction information of different types of visual tokens is also considered.

weaknesses:
1、In Figure 2a, the pixel shuffle operation appears, but the paper does not reflect the transformation of the image features before and after this operation.
2、According to the formula in line 241, the aspect ratios of the adaptive and detailed groups are not integer multiples. But for Figure 2b, the final selection of Ah is 1 and Dh is 3, which seems to be a contradiction.
3、In the CIP module, the paper does not present a clear picture of how the predefined slice ratios appropriate to the size of the image are selected, i.e., what principle is it based on.

questions:
1、I would like to inquire why the paper does not mention the maximum resolution of the images that the model supports, as well as the corresponding comparative experiments.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730185299413

thread_type:
official_review

<thread_object>
creator: reviewer_cdq9

<content>
summary:
The paper introduces the "semantic sawtooth effect" caused by common cropping strategies in high-resolution image scaling for MLLMs. To tackle this issue, they propose a Complementary Image Pyramid (CIP), a flexible and easy-to-integrate approach aimed at reducing semantic discontinuity by providing rich semantic information across different scales. Alongside CIP, they also introduce a Scale Compression Mechanism (SCM) to minimize computational overhead by compressing unnecessary visual tokens. These enhancements improve performance across various MLLM architectures and capacities, leading to the development of a lightweight model called Mini-Monkey, which shows notable improvements in multimodal and document understanding tasks.

strengths:
1. The proposed CIP is logically clear and reasonable. Experiments show that CIP outperforms other cropping methods.
2. The experiments are comprehensive, showing significant improvements across various model families and sizes, as well as multiple datasets, which demonstrates the effectiveness of the proposed CIP.
3. The paper is well-written and clear.

weaknesses:
1. The insight and inspiration of proposed Adaptive Group is not clear.

2. Lack ablation studies on proposed CIP and SCM

See below for details.

questions:
1. There is a lack of explanation for the setting of the Adaptive Group. While both detail and global are easy to understand, the paper does not explicitly state the benefits of detail group or provide experimental evidence to support it.
2. The experiments lack ablation studies, such as removing the detail, adaptive, or global components. What would be the impact if one of these were removed? Which component is most critical? Secondly, what impact does the number of tiles in CIP have? How does the performance of CIP change with the number of tiles in the detail component? Also, what aspect ratios are set by default for CIP?
3. The motivation behind SCM does not align well with the experiments. The paper mentions that "certain scenarios may restrict the level of computational resources available," but the experimental part does not provide experiments on how different compression rates of SCM affect model acceleration and computational cost.

If the authors could supplement their experiments, I would be willing to raise the score to above borderline.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732198784330

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
Dear reviewers and AC,

We would like to express our gratitude to the reviewers for their diligent efforts in reviewing our work.

As highlighted by the reviewers, we believe our paper proposes an interesting (q5HV, HRBu) and effective (cDq9, HRBu, bHqc) method that alleviates the issue in the exsisting cropping strategies, which can be easily integrated into different MLLMs (cDq9, HRBu, q5HV).

We appreciate your helpful suggestions on our manuscript. In accordance with your comments, we have carefully revised the manuscript with the following additional discussions and experiments:

For experiments:

* We added the results of removing different components in CIP (Table 5);

* We added the results of the various aspect ratio settings in the CIP (Table 11 and Table 12);

* We added the results of ablation studies of SCM (Table 6 and Table 13);

We hope the added experiments will provide more insights. For discussion and analysis:

* We added more details about the CIP to make it clearer (Section 3.1).

* We provide a more intuitive example in Section A.4 of the appendix to demonstrate the effectiveness of the adaptive group (Figure 4);

* We provide a example of the matching process of CIP in Section A.5 of the appendix (Figure 5);

We highlighted the revised contents in blue for your convenience to check. We sincerely believe that Mini-Monkey will be of strong interest to the ICLR community, especially, as the revision allows us to better deliver the effectiveness of our method.

For other concerns, we addressed them in our responses.

Thank you very much!

Authors.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper tackles the high resolution image requirement in MLLM, and propose a simple, plug-and play method named Complementary Image Pyramid (CIP) to mitigate the semantic sawtooth effect of patch tileing. The reviewers approve for the effectiveness of the proposed method, and detailed ablation study may help the MLLM community to help for image preprocessing. The AC	recommend for accept based on the reviewers’ opinion.