Mini-Monkey: Alleviating the Semantic Sawtooth Effect for Lightweight MLLMs via Complementary Image Pyramid
Authors: Mingxin Huang
Yuliang Liu
Dingkang Liang
Lianwen Jin
Xiang Bai
Venue: ICLR 2025 Poster (2025)
Keywords: Multimodal Large Language Model
Document Understanding
Primary area: foundation or frontier models, including LLMs

Abstract: Recently, scaling images to high resolution has received much attention in multimodal large language models (MLLMs). Most existing practices adopt a sliding-window-style cropping strategy to adapt to resolution increase. Such a cropping strategy, however, can easily cut off objects and connected regions, which introduces semantic discontinuity and therefore impedes MLLMs from recognizing small or irregularly shaped objects or text, leading to a phenomenon we call the semantic sawtooth effect. This effect is particularly evident in lightweight MLLMs. To address this issue, we introduce a Complementary Image Pyramid (CIP), a simple, effective, and plug-and-play solution designed to mitigate semantic discontinuity during high-resolution image processing. In particular, CIP dynamically constructs an image pyramid to provide complementary semantic information for the cropping-based MLLMs, enabling it rich acquire semantics at all levels. Furthermore, we introduce a Scale Compression Mechanism (SCM) to reduce the additional computational overhead by compressing the redundant visual tokens. Our experiments demonstrate that CIP can consistently enhance the performance across diverse architectures (e.g., MiniCPM-V-2, InternVL2, and LLaVA-OneVision), various model capacity (1B$\rightarrow$8B), and different usage configurations (training-free and fine-tuning). Leveraging the proposed CIP and SCM, we introduce a lightweight MLLM, Mini-Monkey, which achieves remarkable performance in both general multimodal understanding and document understanding. On the OCRBench, the 2B-version Mini-Monkey even surpasses the 8B model InternVL2-8B by 12 score. Additionally, training Mini-Monkey is cheap, requiring only eight RTX 3090 GPUs. Code and models are available at 
https://github.com/Yuliang-Liu/Monkey.

Published as a conference paper at ICLR 2025
MINI- MONKEY. ALLEVIATING THE SEMANTIC SAW- TOOTH EFFECT FOR LIGHTWEIGHT MLLMS VIA COM- PLEMENTARY IMAGE PYRAMID
Mingxin Huang', Yuliang Liu', Dingkang Liang", Lianwen Jin Xiang Bai*2
1 South China University of Technology HHuzhoog University of Science and Technology 3 SCUT-ZZuuai Institute of Modern Industrial Innovation, Zhuhai, China INTSIG-SCUT Joint Lab on Document Analysis and Recognition
ABSTRACT
Recently, scaling images to high resolution has received much attention in multi- modal large language models (MLLMs). Most existing practices adopt sliding- window-style cropping strategy to adapt to resolution increase. Such cropping strategy, however, can easily cut off objects and connected regions, which in- troduces semantic discontinuity and therefore impedes MLLMs from recogniz- ing small or irregularly shaped objects or text, leading to a phenomenon we call the semantic sawtooth effect. This effect is particularly evident in lightweight MLLMs. To address this issue, we introduce a Complementary Image Pyramid (CIP), a simple, effective, and plug-and-play solution designed to mitigate se- mantic discontinuity during high-resolution image processing. In particular, CIP dynamically constructs an image pyramid to provide complementary semantic in- formation for the cropping-basee MLLMs, enabling them to richly acquire se- mantics at all levels. Furthermore, we introduce a Scale Compression Mechanism (SCM) to reduce the additional computational overhead by compressing the re- dundant visual tokens. Our experiments demonstrate that CIP can consistently enhance the performance across diverse architectures (e.g., MiniCPM-V-2 In- Ve.2 and L. various model capacity (1B-+BB), and differ- ent usage configurations (training-free and fine-tuning). Leveraging the proposed CIP and SCM, we introduce a lightweight MILM, Mini-Monkey, which achieves remarkable performance in both general multimodal understanding and document understanding. On the DCRBench, the 2B-version Mini-Monkey even surpasses the 8B model Intern L2-8B by 12 score. Additionally, training Mini-Monkey is cheap, requiring only eight RTX 3090 GPUs. Code and models are available at https::/gittii com Yulll ang Liu/Monkey
INTRODUCTION
Recently, Large Language Models (LLMs) (Zhang al. 2022 Brown al. 2020 Touvron et al. 2023 OpenAl] 2023) have received significant attention for their robust text understanding and generation capabilities. Researchers are actively exploring ways integrate vision encoders into LLMs to upgrade them to multimodal large language models (MLLMs) (Li et al. 2023b Liu et al., 2023a Bai al.] 2023). Some approaches employ Q-former (Alayrac
2022 Li et al. 2023b), while others (Liu et al.] [022d]] Waag et al., (023a) use linear projection. Despite the promising results, they are constrained processing low-res images, which limits their ability to execute detailed scene analysis.
To address this limitation, much recent effect aims to enable MILMs to process high-res images. One straightforward solution is to adopt visual encoder that can tackle high-res images. However, developing high-quality visual encoder demands substantial training resources (Bai et al.) 2023]
Chen et al. 2023 An alternative, more resource-efficient strategy is the non-overlapping crop-
"Corresponding authors.
This work was done when M. Huang was visiting Huazhong University of Science and Technology.

---

Published as a conference paper at ICLR 2025
Impad Image
Overlapping Cropping
(d) Complementary Image Pyranid
Figure 1:
Comparison of different image cropping strategies. (a) Input image; (b) Non-
overlapping cropping; (c) Overlapping cropping; (d) Ours: complementary image pyramid.
ping (Lin et al.
2023b Liu et al. 2024c et 2023a et al. 2024h Chen 2024b). which splits a high-res image into a set of low-res sub-images.
While the non-overlapping cropping strategy has shown promising results, it inevitably cuts off ob- jects and connected regions, rendering difficulty for the MLLM in recognizing small or irregularly shaped objects due to semantic discontinuity, particularly the context of document understanding. This mainly leads to two types of consequences: i) semantic ambiguity: if an object character divided, may be misidentified; the nose of guinea pig in Fig. [[b) looks much like a monkey after cropping, for instance; 2) semantic damage: if a word sentence is segmented, the meanings of the segmented word will be changed completely; if the word "breakdown' is divided into 'break' and 'down', the segmented words will have nothing do with the original one (Liu al.] 20240 Zhang et al. 2024). For simplicity, we call these phenomena the semantic sawtooth effect in this paper. To alleviate this effect, rather straightforward idea adopt overlapping cropping. However, this strategy will result in the processing of much duplicate information, as presented in Fig. [Tc). This redundancy could potentially cause hallucinations in MILMs. Moreover, can even deteriorate per- formance according to our ablation studies in Sec. 4.3 Additionally, the semantic sawtooth effect can observed more evidently lightweight MILMMs. Larger MILMSs with enhanced compre- hension capabilities and feature extraction capabilities often can alleviate this issue to some extent. Even when the object is segmented, these models can understand the objects through their powerful feature extraction.
To alleviate the semantic sawtooth effect more explicitly, we propose a plug-and-play approach, termed Complementary Image Pyramid (CIP). CIP can be easily integrated into variety of cropping-beed based MLLMs, allowing them to tackle high-res images with reduced semantic sawtooth effect. CIP dynamically constructs image pyramid that provides complementary semantic fea- tures for the MILMs, enabling it rich acquire semantics at all levels. If object semantics are lost at one scale, they can be compensated by those from another scale. Different from previous work (Liu) et al. our approach tocuses on enriching the image semantics per se. Consequently, CIP can be easily in- tegrated into variety of MLLMs, allowing them to tackle high-res images with reduced semantic sawtooth effect. Considering that the CIP introduces some additional computational overheads, we further propose a Scale Compression Mechanism (SCM) for use in situations with limited compu- tational resources. The SCM is both training-free and parameter-free. attention layers of the LLM and the multi-scale information generate attention weights, which in turn are used to compress redundant tokens. Utilizing the proposed CIP and SCM, we introduce a lightweight MILM, Mini-Monkey.
a 2024f Huang et al. (0224) that addresses this issue by modifying the architecture of the model,
leverages the well-trained
Our experiments demonstrate the effectiveness of the proposed method: 1) 2B- 2B-parameter Mini- Monkey outperforms the IntemVL2-22 by an average of 2.4% across 17 benchmarks in terms of evaluation metrics; 2) Mini-Monkey achieves a score 806 on the OCRBench, outperforming the 8B 8B-parameter model Intern by 12 score. Moreover, observe that directly fine-tuning
2

---

Published as a conference paper at ICLR 2025
well perl-perfo pre- pre-trained MLLM does not enhance, but rather degrades its performance. In con- trast, fine-tuning with CIP can facilitate the training process to improve performance. In conclusion, the contributions of this work be summarized as follows:
CIP: a blug-and-play complementary image pyramid designed to alleviate the semantic sawtooth effect for multimodal large language models;
Mini-Monkey: a lightweight, effective, and training-efficien multimodal large language model that integrates the complementary image pyramid and the scale compression mech- anism;
Our method achieves promising results on 8 general multimodal understanding benchmarks and 9 document understanding benchmarks, demonstrating the benefits of alleviating the semantic sawtooth effect.
RELATED WORK
2.1 MULTIMODAL LARGE LANGUAGE MODELS
Res-Resolttion Input. In recent years, Large Language Models (LLMs) have made significant
progress (Zhang et al. 2022 Brown al. 2020 Touvron et al., 2023 OpenAl 2023 Drawingg from this advancement, many efforts have been made to integrate vision encoder into Large Lan- guage Models for vision-languaae understanding. A commonly employed approach is the linear projector mLth al. 2024d: Wang al. 2023a), which maps the output of the vision en- coder to the same feature space as the text features of the Large Language Models. Some methods, such Q-Former (Li et al.] 2023b), Perceiver Resampler (Alayrac al. (002)), or Abstractor (Ye) et al. 2023b), introduce a set of learnable queries to facilitate this integration. Despite these notable advances, previous methods often struggle with detailed scene understanding due to limitations in resolution.
Naïve High-Resolution Strategy vs. Cropping Strategy To address this issue, recent research has adopted two primary strategies: 1) Naïve High-Resolution Strategy. This Strategy leverages the vanilla ViT to handle images of any resolution and aspect ratio (Wang et al. 2024 Liu et al. 2024g). However, these methods require additional training data and parameters, or processing attention over high-res images significantly increases computational demands. 2) An alternative, more resource- efficient method is the cropping strategy, which divides the high-res image into multiple lower- resolution sub-images for processing (Li et al.) 2024f Lin et al. 2023b Ye et al. [2023a] Chen et al.) 2024b. Dubey et al.] 2024). At resolution of 2240x2240, the Naive High-Resolune requires around 50GB of GPU memory, whereas the cropping strategy needs only about 16GB. These results demonstrate the cropping strategy's 's efficiency in managing high-resolution images, making it a more practical choice for resource-constn constraiied environments.
Strategy
Although the cropping strategy achieves promising results several multimodal benchmarks, it will inevitably result in a semantic sawtooth effect: 1) If an object or character divided, it may not be recognized; 2) If the word sentence is segmented, the semantic damage of the segmented word will be caused. For example, the word 'Breakdown' may be divided into 'Break' and 'down', causing semantic damage to the segmented word. This will limit the model's ability to understand the detailed scene. Although some methods (Liu et al., 2024f Huang et al. 2024) attempt to address this issue by introducing new modules, they introduce additional parameters the original model and require training this module from scratch. In contrast, the proposed CIP is designed to be seamlessly integrated without introducing additional parameters, offering plug-and-play solution.
2.2 LIGHTWEIGHT MULTIMODAL LARGE LANGUAGE MODELS
Due to the substantial computational costs associated with multimodal large language models (MLLMs), some recent efforts have focused on developing more efficient models for rapid devel- opment and real-world applications. For instance, LLaVA-Phi (Zhu et al. 2024) and Imp (Shao) et al. 2024) integrate a lightweight large language model with a vision encoder to develop a pow- erful multimodal system. Mobile (Chu al. (023) further conserves resources by integrating a lightweight downsampling projector that reduces the number visual tokens. Bunny (He et al. 2024] advances efficiency through an effective data compression technique, which minimizes the
3

---

Published as a conference paper at ICLR 2025
Mini-Monkey
Complementary Image Pyramid
Shuffic
Project
Compression
Mechanism
(b) Complementary Image Pyramid
Scale Compression
Grouping Operation
Tslamury
Tokens
Groaict
H-Amt
3:4
14:1
LLM's Leyer
1:2
1A11
Adtiotive GroupdA,)
dynamically Select
Giobal
Giobal
1:
Figure 2: Overall architecture of Mini-Monkey. H-Attn represents high attention weights. L-Attn represents low attention weights. The tokens with low attention weights will be filtered. The shared LLM's Layer represents using the block layer from LLM in SCM. The pixel shuffle operation is utilized to reduce the number of visual tokens one-quarter the original.
required pretraining dataset. TinyGPT-V (Yuan et al. (023) adopts multi-stage training process specifically designed for lightweight multimodal models. The above models support only low-res input. To improve detailed scene understanding, one of the most commonly used methods is the cropping strategy. For instance, InternVL2-2B (Chen et al. 2024b) enhances the performance of lightweight MLLMs by adopting dynamic high-res cropping strategy. Despite these advance- ments, the cropping strategy will introduce a semantic sawtooth effect, which significantly limits the performance of lightweight multimodal large language models. Larger MLLMs with enhanced comprehension capabilities can alleviate this issue to some extent, as discussed in Sec. 4.4
3 MINI-MONKEY
The overall architecture is illustrated in Fig. Mini-Monkey consists of a CIP, vision encoder, an MLP layer, a Scale Compression Mechanism, and a Large Language Model (LLM). Initially, CIP dynamically generates image pyramid based the resolution of input images. Then, divide these images into a set of sub-images. These sub-images are then processed by the vision encoder and MLP layer extract image tokens. The Scale Compression Mechanism adjusts these image tokens and forwards them to the LLM, which subsequently generates the final answers.
3.1 COMPLEMENTARY IMAGE PYRAMID
Existing cropping strategy (Li et al. 2024f Chen et al. (024b) directly divides the high-res images into set of sub-images that will lead to semantic sawtooth effect. To address this issue, propose a plug-and-play method, termed complementary image pyramid (CIP), to promote synergy among images at varying scales to alleviate the semantic sawtooth effect. The process of CIP shown in Fig. (b).
Grouping Operation. We begin by generating a of pre-defined aspect ratios, which define as follows: [g = (nz, X Rw)) N Nmin S niş Th < N Nmaz: nty E N, Ru NJ. The n,, and nuw represent the number of height and width of the grid g. N Wax the maximum number of tiles. Nmin is the minimum number of tiles. These aspect ratios are then categorized into three groups through grouping operation, including a detailed group Dy. an adaptive group Ag. and global group Gg- The classification based on the following criteria: (1) Aspect ratios that are between 3 Nmaz and Nmaz tiles being allocated to the detailed group, enabling the largest possible image size and thus clearer depiction
the objects within. (2) For aspect ratios producing between & -Nmaz and & * N Naaz tiles, we classify them into the adaptive group, which responsible for enhancing the fine details at the borders of the crops. (3) The 1:1 aspect ratio is designated the global group,

---

Published as a conference paper at ICLR 2025
providing low-res, comprehensive view of the whole image. The grouping operation generates three groups of different aspect ratios for generating the image.
Dynamically Generating Images. After the grouping operation, will generate three images from each group. First, we calculate the absolute differences between the aspect ratio of the input image and the aspect ratios within the detailed group. Then, the ratio that has the smallest absolute difference from the input image's s aspect ratio is selected as the matched ratio, denoted as Da. D D₁ the number of height tiles and D.. is the number of width tiles. Once a matched aspect ratio is determined, the image is resized to the corresponding resolution. For example, 1288x 1257 image would be resized to 1344x1344, 134.. The resized image is then divided into tiles of 448x448 pixels. After obtaining the detailed image, the adaptive group will dynamically generate an aspect ratio based on DR, Dac ensuring that the cropping lines on the detailed group and those on the adaptive group do not overlap. First, we eliminate any aspect ratios in the adaptive group that are exact multiples of D,, Dw- When the A, 1, means that there is no required cropping. Therefore, they can be integer multiples in such cases. This process can be formulated as follows:
th ViE (h,w), w], {D. = k- As, if A,
(I)
+ k. As, otherwise.
where A, and A denote the height and width components of the aspect ratios in the adaptive group, respectively. Then, we will resize the image by selecting the ratio closest to the aspect ratio of the original image from the remaining aspect ratio.
Because the vision encoder processes each tile independently, existing cropping-based MLLMs fail to capture feature interactions between different tiles. In our method, the adaptive group employs distinct aspect ratio to partitioning windows compared to the detailed group, thereby simulat- ing cross-tile interaction features and providing the cropping positions information for the detailed group. Similarly, the global group provides the cross-tile interaction features and the cropping po- sitions information for the adaptive component. Three groups provide complementary semantic information and multi-scale information for the model, enabling the model to better capture finer details and handle objects of different sizes in images. Different from the previous method (Liu al., 2024f HHuang al. 2024), the proposed CIP alleviates the semantic sawtooth effect from the perspective of the image, bringing several advantages: (i) it is plug-and-play, requiring no addi- tional parameters; (ii) it seamlessly integrates with existing MLLMs that utilize cropping strategies, leading to consistent performance improvements; and (iii) it can be utilized without training and its effectiveness can be further improved through fine-tuning.
3.2 SCALE COMPRESSION MECHANISM
Although the proposed CIP significantly enhances model performance, certain scenarios may restrict the level of computational resources available. To tackle this challenge, introduce a parameter- free token compression method called the Scale Compression Mechanism (SCM), which is used to reduce the visual tokens, as shown in Fig. 2 (c). The detailed group provides tokens with lower information density, whereas the adaptive and global groups yield tokens that are more information- dense. Therefore, we primarily focus on compressing the tokens from the detailed group. Previous work demonstrates that a well-trained LLM from MLLM can effectively select the necessary visual features based on the input question (Chen et al., 2024a). Consequently, SCM utilizes the layers of the LLM from well- rell-trained MLLM select visual tokens without generating any additional parameters. The input visual token including Va E I A1 XC V a E RlaxC and Vg "g E R La xC, and the textual token T, E TT'C' will be sent into an LLM's Layer. Va represents the tokens from the detailed group. Va represents the tokens from the adaptive group. V represents the tokens from the global group. We utilize the first and second layers of LLM to compress the tokens. The LLM's Layer will output an attention map. We use the visual token from the adaptive group, global group, and textual token to attend to the visual token from the detailed group. The calculation of the attention can be formulated as follows:
Q cat(Va. Vg. TT),
(2)
Q + PE Q) Va + PE(Va))
Attn = softmax
(3)
VD
5

---

Published as a conference paper at ICLR 2025
Table 1: Comparison with SoTA models on 8 multimodal benchmarks. General multimodal bench-
marks encompass: MME (Fu et al. ). RealWoridQA (X.ail 2024), AI2D test (Kembhavi et al.
2016. CCBench (Liu et al. 2023b), SEED Image (Li al. 2023a). HallusionBench (Guan al.
2023 and POPE (Li et al. 2023c). Additionally, the math dataset includes MathVista testmini (Lu
al. ).23). The MME results we report are the sum of the perception and cognition scores.
represents the results from the OpenCompass leaderboard (Contributors) 2023).
General Multimodal Benchmarks
Math Vash
Aparan
MME RWQA A12D CCB SEED HallB POPE
QWEN-VL
2023
70
34,91
Mini-Gemini
Get al.
2024g
35B
2141.0
43.3
LLaVA-NeX1
Liu et
202
35B
2028.0
74,9
49.2
75.9
34.8
89.6
46.5
InterVLL 1.2
Len
20244c
40B
2175.4
67.5
79.0
59.2
75.6
47.6
88.0
47.7
InterVL 1.5
Chen et
162
26B
2187.8
66.0
80.7
69.8
76.0
49.3
88.3
53.5
DeepSect-Vi
2024
1.78
1531.6
49.7
51.51
37.6
85.0
29,4
Mini-Gemine
Li
202
2:2B
1653.0
29,4
Bunny-Sulel
M-
M.T
2024
28
1602.9
58.8
85.9
Min
Yau ct
2024
2:8B
55.8
62.91
48.0
1
86.35
InterVL. 2
Chon al
202
2B
1876.8
57.3
74.1
74.7
70.9
37.9
85.2
46.3
Mini-Monkey (ours)
2B
1884.2
57.9
74.8
75.5
71.3
38.8
88.0
where PE represents the position encoding and D denotes the dimension of the LLM. Cat() rep- resents the sequence concatenation operation. After computing the attention mechanism, we aver- age the first dimension of the attention map Attn E +L(++t++I)xx to obtain a weight vector Wa E R Subsequently. we select the top K visual features from detailed layers based on this weight vector Wa. These selected tokens, along with tokens from the adaptive group, global group, and textual token, are input into the L.LM to generate the results. Compared to FastV (Chen al,, 2024a), SCM works in conjunction with the CIP and is more targeted by using tokens with high relative information density to compress tokens with low information density.
4 EXPERIMENTS
4.1
IMPLEMENTATION DETAILS
We use Intern VL2-2B (Chen et al. previous work (Chen et al. 2024c). we use the (448, 448) as the input resolution of InternViT. The
2024b) the Baseline to develop the Mini-Monker Following
training datasets used train the model include VOA (Mathew al. 202T). ChartQA (Masry
et al. 2022), DVQA (Kafle et al.) 2018). A12D (Kembhavi et al., 20 GeoQA+ (Cao & Xiao 2022 and LLa VL- (zh) (Liu et 2024d). We use the Adam W (Loshchilov & Hutter 2017) as the optimizer. The base learning rate is 4e-8. We limit the maximum number N to 24 and the minimum number Nmin is 1.
Evaluation. Following the previous work (He et al. 2024 Chen et al.) 2024b). we evaluate Mini- Monkey on eleven general multimodal understanding benchmarks, including Math Vista testmini (Lu et al. 2023), SEED Image (Li et al. 2023a), WorldOA (X.ii (0224) A12D test (Kebbhavi et al.) 2016 POPE (Li et al. 2023c), CCBench (Liu et al.] (023b), MME (Fu et al. 2023), and Hallu- sionBench (Guan et al. 2023 For document understanding. following the previous work (Liu (ct al. 2024f). we employ two distinct types of metrics to verify the performance of Mini-Monkey. Initially, we leverage the standard metrics provided by the benchmarks to evaluate Mini-Monkey.
We utilize benchmarks such as ChartQA (Masry et al.) 2022), DocVQA (Mathew et al.
2021
InfoVQA (Mathew al., 2022). TextVQA (Singh al. 2019). STVQA (Biten et al. 2019
FUNSD (Jaume et al. 2019). SROIE (Huang et al.) (201). POIE (Kuang et al. 2023) and OCR-
Bench (Liu al.] 2023c). We also apply the accuracy metric to verify the performance. Further
details on this metric and the used benchmarks can be referenced in appendix A.8
4.2 COMPARISON WITH STATE-OF OF-THE METHODS
General Multimodal Understanding. We evaluate Mini-Monkey on general multimodal under- standing following (He et al.) 2024 Chen et al. 2024b). The results shown in Tab. T Mini- Monkey surpasses other 2B -3raammetter models on benchmarks. The results indicate that CIP en- hances Mini-Monkey's perception ability, thereby improving its capability handle general multi-

---

Published as a conference paper at ICLR 2025
Table 2: Comparison to state-of-the-art MLLMS OCR-related Tasks. Mini-Monkey achieves the best results among the 2B P-paametter MLLMs. represents the results from the OpenCompass
leaderboard (Contributors 2023).
Model
Midid Sioe
DoeVQA
CharqQ
InfoVQA
TesVVA
OCRBBench
TexiMookey
et popar
13.0
66.9
28.6
65.5
558
ToaHHokk
50.6
Zingece
HiRes
Tooas 2114
48.0
MaVA-1HHD
Kurrar
67.7
CogApent
17B
81.6
68.4
48.5
76.1
590
UReader
TE Filetn
78
63.4
39.3
42.2
$7.6
ThcOul
8R
82.2
70.2
50.7
68.5
HRVDA
REveIR RUACS1
7B
72.1
67.5
43.5
TexiSquare
84.3
79.4
51.5
66.8
622
IXC2-4KII)
$1,0
68.6
77.2
675
IntenVL
26B
83.8
50.6
IntenVI.
774
Vay-toy
1.88
MinCPAC
2ar Vaneral
2004
2,88
71.9
55.67
74.1
605
InterVI.
Chereer 20140)
28
86.9
76.2
58.9
73.4
754
Mini Monkey (025)
28
87.4
76.5
60.1
76.0
906
Table 3: Quantitative accuracy (%) comparison of model with existing multimodal large lan- guage models (MLLMs) on several benchmarks. Following TextMonkey (Liu et al. 2024f), we use the accuracy metrics to evaluate our method.
Scrre Tinn-Catticc VIJ
Drcamei-Orrmaa VQA
FUNSD
BUPT-GPP4
namal NaPTh
20.9
13.5
3.2
11.3
3.4
0.2
6.1
0.3
HPLUGIOW!
30.5
34.0
7.4
20.0
0.5
1.7
2.5
IJ IVA
12.1
54
216
0.3
0.7
2.1
O
49.5
$3.9
175
13.
194
1,4
32
0.9
LLA
38.1
35.7
14.
02
1.7
255
45.5
47.1
45.9
20,9
21.4
36.9
Morkey
14.7
64.3
50.1
25.8
$1.0
24.1
41.
1599
InzeVV
62.2
26.7
45.6
6.5
26.6
259
Inten XCimXCamaaaaee
20240
39.7
65.5
76.7
430
080
66.
Misi-Morbnyy (1yer1111))
67.2
68.8
784
50.
67.3
4322
765
71.2
modal understanding tasks. Additionally, the POPE benchmark, which evaluates hallucinations in MILMs, Mini-Monkey outperforms the Baseline InternVL2-2B by 2.8%, demonstrating that CIP can also mitigate hallucinations in MLLMs.
Document Understanding, For the first type of metric, the results are presented in Tab. 2 Com- pared to Baseline Intern VI.2-2B, our method outperforms it by 2.6%, 1.2%, and 22 for TextVQA, IQA, and OCRBench, respectively. The CIP provides the model with complementary semantic and multi-scale information, enhancing its ability to perceive fine-grained and varying-sized text. Due to the small original resolution of ChartQA, it is less impacted by cropping operations, result- ing minor improvement from our method. With these complementary semantic and multi-scale information, on the OCRBench, Mini-Monkey even surpasses the 3B-parameter Large Multimodal Model Intern LL2-8B and the 9B-parameter Large Multimodal Model GLM4-V by 12 and 20, re- spectively. For the accuracy metric, the results are shown in Tab. 3 Mini-Monkey outperforms the InternVL2228 by 2.6%. 3.2%, and 4.4% on TextVQA. IQA, and POIE, respectively. OCR- related tasks are utilized to evaluate the fine-graid fine-grained recognition capabilities of the MLLM. The results from these tasks demonstrate the effectiveness of CIP in enhancing such capabilities.
4.3 ABLATION STUDY
In this section, we perform ablation studies on both general multimodal understanding and doc- ument understanding benchmarks to validate the effectiveness of method. We adopt the TextVQA (Singh et al. 2011). OCRBench (Liu et al. 2023c), HallusionBench (Guan et al. 2023). MME (uu et al. 2023). and POPE (Li et al. 2023c) to conduct ablation studies.
Complementary Image Pyramid. We conducted ablation studies to investigate the effectiveness of the CIP. We compared method with several alternatives: The dynamic high-res strategy (Chen) jet al. 2024b), which maintains aspect ratios to increase resolution. The fixed-size high-res strat- Li al. 2024f), which fixed size increase resolution. The overlapping cropping
egy
7

---

Published as a conference paper at ICLR 2025
Table 4: Ablation study of Complementary Image Pyramid. We compare our method with the
existing cropping strategy and the overlay cropping strategy.
Medil
Roseltiion Stralgy
TouVQA
OCRBfonch
MMF
HeiliB
POPE
Flopo()
Latincy/Fssmple
Beseline
Dynamic high-res Stratge
(Chn scal 202ap
37.9
349.4
Biseline
Fixed high Scrtege
772
37.6
510.9
Reseline
Oveilagping Cropping Stntery
738
1.3s
Biseline
Multi-Scale Stratggy
74.8
776
58.1
599.2
1.6s
Mini Mini-Mekkyy (Oums) Complementary Imagh Pymmm
76.0
806
1884.2
38.8
88.0
531.3
1.36
Table 5: Ablation study of the impact of different components in CIP.
Devalled Componses Compoatts Adaptive Composeer.
TeuVQA
OCRBesch
MME
HalB
62.5
355
1686.2
34.8
1ecm/112-28 2B
473
1905.]
60.8
6.34
1943.3
85.3
64
1msemV12-20 28
74.5
752
1874,2
39.0
$7.5
13
74.6
785
1851.5
17.6
87.6
06
ImeeVL2222 28
76.0
306
1884.
38.8
$8.0
Table 6: Ablation study of the different compression rates of SCM.
Compression Rate 0.00
0.1
0.2
0.3
0.4
0.5
0.7
0.9
MME
1884.2
1884.7
1879.8
1878.5
1876.3
1886.0
1871.7
1870.2
Flops
446.9
383.6
323.0
293.7
237.0
171.4
Latency/Excuple
0.83
0.78%
0.736
0.67s
0.63
0.5%
0.51s
0.4g
strategy uses high-res approach but crops with overlay. The multi-scale strategy (Shi et al. 2024). which introduces multi-scale strategy the MLLM. As presented in Tab. the proposed CIP achieved the best results on both general multimodal understanding and document understanding without significantly increasing lateney or computational load. The over-overlay cropping strategy, instead of improving the model's 's performance, actually degraded it.
Various Model Capacity. We performed ablation studies to assess the impact of CIP on models with varying model capacities. As illustrated in Table 8 CIP consistently improves the performance of varying model capacities, illustrating the general applicability of our approach.
Different Usage Configurationss To further validate the improvements introduced by CIP, we performed experiments on usage configurations: a training-free free configuration and fine-tuning con- figuration. As shown in Tab. 7 CIP demonstrates improvements in performance even when applied without training. The performance can be further improved with fine-tuning. Additionally, we sur- prisingly find that CIP can even facilitate the model fine-tuning process. As presented in the second line in Tab. | direct fine-tuning of the Bascline model not only failed to improve performance but, in some cases, led decline. Conversely, incorporating CIP during the fine-tuning of the Base- line resulted substantial improvements in both general multimodal understanding and document understanding, evidenced in the fourth line of Tab.
Incorporating CIP to various MLLMs. The proposed CIP can be seamlessly integrated into crop- based MILMSs To demonstrate its effectiveness, we incorporated CIP into various structures of MLLM, such as MiniCPM-V-2 Yao et al. 2024), Intern 2 (Chen et al. 2024b), VLaVA-OV (Li al. 2024b). The results shown in Tab. show that CIP can be seamlessly integrated into various MLLMs and consistently improves their performance.
The Number of Sub- To investigate whether the performance enhancement is attributed to an increase in the number of sub-images, we performed an experiment by incrementally raising the sub-image count for the Baseline. The findings, summarized in Tab[T]] indicate that increasing the number of sub-images does not lead to better performance; instead, it may result in decline. In contrast, CIP can effectively improve the performance of the model, demonstrating its effectiveness.
The impact of different components in CIP. To evaluate the importance of each component in the CIP, we performed ablation studies using the InternVL2-2B. As presented in Tab. 5 the results indicate that using only the global component or only the detailed component results in significant performance drop, as shown in rl and r3 of Tab. By comparing the rl and r2, as well as r3 and r4, in Tab. [] we find that adding Adaptive component significantly improves the performance. When using both the detailed component and the global component, adding the adaptive component leads to further improvements, shown in r5 and r6 of Tab. Furthermore, the removal of any one of the
8

---

Published as a conference paper at ICLR 2025
Table 7: Exploring different usage configurations of CIP. Train represents fine-tuning the model.
Model
CIP
Train
TexVQQ
OCRBeach
MME
HallB
POPE
Intera VL2228
X
73.4
784
1876.8
37.9
85.2
InterVII 2-
73.3 (-0.1)
787 (+3)
1858,3 (-18.5)
(-0.6)
85.3 (+0.1)
InterVVI 2BB
75.2 (+1.8)
800 (+17)
1881.9 (+5.1)
38.7 (+0.8)
86.7 (+1.5)
LL22mVL2-28
76.0 (+2.6)
806 (+22)
1884.2 (+7.4)
38.8 (40.9)
(8.0
Table 8: Ablation study of incorporating complementary image pyramid (CIP) to other MLLMs. represents the results from the OpenCompass leaderboard (Contributors 2023)
Model
TextQQA
OCRBencc
HallB
POPE
MiniCPPM-V.22-222
X
74.1
6015
1808.6
36,1
86.33
MiniCPM- V.2-2.8B
(+1.9)
627 (+22)
1819.5 (+10.9)
36.5 (+0.4)
87.1 (+0.8)
LLaVA-OV-0.58
65.3
577
1478.0
28.1
86.7
LLaVA LLaVA-OV-0.5B OV-0.5B
V
66.2 (+0.9)
600 (+23)
1482.6 (+1,6)
28.8 (+0.7)
87.7 (+1.0)
70.5
IntemnVL--1B
72.3 (+1.8
772 (+18)
1801.5 (+7.1)
34.3 (+0.9)
85.7 (+0.8)
InternVL2-2P
73.4
784
1876.8
37,9
85.2
L22em
76.0
806 (+22)
184.2 (+7.4)
38.8 (+0.9)
88.0 (+2.8)
InternV1. 2-88
77.4
794
2210.3
45.0
84.27
IntemVL2-88
79.3 (+1.9)
809 (+15)
2226.4 (+16.1)
45. (+0.4)
84.8 (+0.6)
Table Ablation study of the scale compression mechanism. We used different compression ratios to compare with FastV (Chen et al. 2024a). (0.5) represents 50% compression and (0.9) represents 90% compression.
Model
Compression Stratgyy
ToxVQQA
OCRBench
MME
HallB
POPE
Mini Monkey
Pooling (0.5)
47.6
256
1765.2
31.5
84,5
Mini-Monkeyy
Random (0.5)
1805.5
36.2
Mini-Monkey
FastV Chen ct 2024g
(0.5)
73.4
781
1848.0
38.3
83.9
Mini- Monkey
FastV [Chon
73.9
792
1866.1
37.5
85.8
Mini-Moukey
SCM (0:3)
74.7
1886.0
38.7
86.1
Mini- Monkey
SCM (0.9)
75.2
801
1884.7
38.6
86.2
three components leads to a decline in performance (r2, r4, 15, and r6). The removal of the global component results in the most significant performance drop (r2 and r6). This is because VL2- 2B was pretrained with both the detailed and global components. Removing the global component or detailed component will result in a significant performance drop. The adaptive component can to some extent compensate for the information provided by the detailed group, thus the impact of removing the detailed component is less significant than the global component. However, to achieve optimal performance, the synergy among all three components is indispensable. These results demonstrate the effectiveness of our method.
Scale Compression Mechanism (SCM). We compared SCM with the related work FastV (Chen iet al.] (2024)). For different methods, we compress the number of visual tokens by 50%. For our method and FastV, we further conduct experiment with 90% compression. Following FastV's paper, we set the K in FastV as 2. As illustrated Tab. 9 when using 50% compression and 90% compression, our method outperformed Fast' by 21.5% and 4.4%, respectively, demonstrating its effectiveness. FastV compresses input tokens, including both visual and textual tokens, within Transformer blocks. In contrast, our method works in conjunction with the CIP and more target by using tokens with high relative information density to compress tokens with low information density.
The different compression rates of SCM. We conduct an ablation experiment on the MME to show how the compression rates of SCM influence the acceleration and computational cost. Following FastV (Chen et al. 2024a), we consider the computation of multi-head attention (MHA) and feed- forward network (FFN) modules in the FLOPs estimation. The total FLOPSs are estimated by L (4 * n * dl + n² * + 2 n * * m)) where n is the token number, d is the hidden state size, m is the intermediate size of FFN, L is the number of transformer layer. The latency experiments are conducted on A6000 GPU. As presented Tab. 6 we can find that as the compression ratio increases, the computational load continues to decrease. and the speed keeps improving, without significant drop in performance. More ablation studies about the selection of hyperparameters are presented in appendix A

---

Published as a conference paper at ICLR 2025
Table 10: Ablation study the number of sub-images. The number denotes the sub-image count.
Model
Number
TextVQ
OCRBench
MME
HalIB
POPE
Baseline
18
74.2
782
1851.7
37.0
85.8
Baseline
24
74.4
783
1857.6
36.9
85.8
Baseline
32
74.3
782
1845.0
36.9
85.9
Baseline
74.0
767
1841.6
36.2
85.7
CIP
32
76.0
806
1884.2
38.8
88.0
(a) Input Image
(b) The results ofs
47 die INSTALLED
Artiiiia Blimpes
125 Cles Frwar vabora
47
25
INSTI ALLECO
Tiyk
words are below
Intera L22-2B OCS: The words located helow the 'children! are 'school groupe..
children!?
GT: classrooms home school groups
(c) The results of LnternVL2-2B and 26B
(d) The results of Mini-Monkey
INS 125 the
125 2
DAGAA BUTOO
INSTAL LED
Intern LL2-2B: The words located below the words chilkren' are CCrriculum- based Field Tripe'.
Adaptive Select
25
TALLIO
LitennVL2-26B:. The words located below 'lo children' are "classrooms home school groups".
Ours: The words located below the wurds to children' are 'chssrromma and 'hame school groups!
Qualitative results of Mini-Monkey, (a) Input Image and Ground Truth. (b) The results of using overlapping cropping strategy. OSC represents the overlapping cropping strategy. (c) The results of Intern VL2-2B and Intern 2LB- (d) The results of Mini-Monkey.
Figure 3:
4.4 QUALITATIVE RESULTS
In this section, we provide some qualitative results to demonstrate the effectiveness of our method. First, we verify that the semantic sawtooth effect is particularly evident in lightweight MILMs, which adopt Intern VL2-2B and InternVL2-26B. As shown in Fig. 3(.), InternVL2-26B can an- swer the questions correctly. However, due to the word 'classrooms' and 'school' being cropped, Intern gives a wrong answer that addresses the text in the bottom left corner of the original image. Mini-Monkey can overcome this semantic sawtooth effect and provide the correct answer, as presented in Fig. d).. Comparing Fig. (3(b) and Fig. Bid), we can see that the overlapping cropping strategy introduces some hallucinations and cannot answer questions accurately based on the image,
whereas our methods can effectively address the semantic sawtooth effect. More qualitative results are presented appendix A.7
5 CONCLUSION
In this study, we introduce a Complementary Image Pyramid (CIP) designed to alleviate the seman- tic sawtooth effect for MLLMs, thereby enhancing their capability to process nigh-resolution images effectively. CIP plug-and-play and can be seamlessly integrated into various multimodal large lan- guage models at low cost. We demonstrate the effectiveness of the proposed CIP across diverse architectures, various parameters, and different usage configurations, leading to consistent perfor- mance improvements. Besides, we present Scale Compression Mechanism (SCM) to compress the visual tokens for computational efficiency. CIP not only enhances the general multimodal un- derstanding performance but also shows consistent improvements in document understanding tasks. Furthermore, our experimental results demonstrate that 2B 2--parameter MLLM equipped with CIP surpasses larger (B--parmetter state-of-the-art models like Intern VL.2-BB the OCRBench.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This research is supported in part by National Natural Science Foundation of China (Grant No.: 62476093, 62441064, 62206104, 62225603, 62441604).
REFERENCES
Jean-Bayy San-Baptiste Alayrac. Jeff Donahue, Pauline Lue, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716-
23736, 2022.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2388. 12966, 2023.
Ali Furkan Biten, Ruben Tito, Andres Mañla, Lluis Gomez, Marçal Rusinol, Ernest Valveny, CV Jawahar, and Dimosthenis Karatzas. Scene text visual question answering. In Proceedings of the JEEE/CVF international conference on computer vision, pp. 4291-4301 2019.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 2020.
Jie Cao and Jing Xiao. An augmented benchmark dataset for geometric question answering through dual parallel text encoding. In Proceedings of the 29th International Conference on Computa- fional Linguistics, pp. 1511-1520, 2022.
Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang- An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision- language models. arXiv preprint MxXiv:2403.06660 2024a.
Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Car- los Riquelme Ruiz, Sebastian Goodman, Xiao Wang, Yi Tay, al. Pali-x: On scaling up multilingual vision and language model. arXiv preprint arXiv:2305. 185655 2023.
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo. Zheng Ma, et al. How far are we to gpt-4v? closing the gap to com- mercial multimodal models with open-source suites. arXiv preprint arXiv:2404.166 16821, 2024b.
Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguisti tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14185-241999 2024c.
Xiangxiang Chu, Limeng Qiao, Xinyang Lin, Shuang Xu, Yang Yang. Yiming Hu, Fei Wei, Xinyu Zhang, Bo Zhang, Xiaolin Wei, et al. Mobilevlm: A fast, reproducible and strong vision language assistant for mobile devices. arXiv preprint arXiv:2312.16880 16886, 2023.
OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models. 2023.
pettps://qithine http://ttha..comooppp.e.....
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards general-purpose vision- language models with instruction tuning. In Thiriy-sevntth Conference on Neural Information Processing Systems, 2023.
Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Xinyue Zhang, Wei Li, Jingwen Li. Kai Chen, Conghui He. Xingcheng Zhang, Yu Qiao, Dahua Lin, and Jiaqi Wang. Internlm -coomooexcomppse2::: Mastering free-form text-image composition and comprehension in langunne large model, 2024a.
11

---

Published as a conference paper at ICLR 2025
Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang. Songyang Zhang, Haodong Duan, Wenwei Zhang. Yining Li, et al. Internlm hoaolll-xooooser2-222 A pioneering large vision-language model handling resolutions from 336 pixels to 4k hd. arXiv preprint arXiv:2404.065 12, 2024b.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Lerman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen. Mmbench-vides A long-form multi-shot benchmark for holistic video understanding. arXiv preprint arXiv:2406. 14515, 2024.
Hao Feng, Qi Liu, Hao Liu, Wengang Zhou, Houqiang Li, and Can Huang. Docpedia: Unleashing the power of large multimodal model in the frequency domain for versatile document understand- ing. arXiv preprint arXiv:2311.11810 2023a.
Hao Feng, Zijian Wang, Jingqun Tang, Jinghui Lu, Wengang Zhou, Houqiang Li, and Can Huang- Unidoc: A universal large multimodal model for simultaneous text detection, recognition, spotting and understanding. arXiv preprint arXiv:2308. 1192,, 2023b.
Chaoyou Fu, Pcixian Chen, Yunhang Shen. Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, et al. Mme: A comprehensive evaluation benchmark for multi- modal large language models. arXiv preprint arXiv:2306. 13394, 2023.
Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evalua- tion benchmark of multi-modal lims in video analysis. arXiv preprint arXiv:2405. 2005,, 2024.
Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie Huang, Peng Zhang. Qinkai Zheng, Rui Lu, Shuaiqi Duan, Shudan Zhang, Shulin Cao, Shuxun Yang, Weng Lam Tam, Wenyi Zhao, Xiao Liu, Xiao Xia, Xiaohan Zhang, Xiaotao Gu, Xin Lv, Xinghan Liu, Xinyi Liu, Xinyue Yang, Xixuan Song, Xunkai Zhang, Yifan An, Yifan Xu, Yilin Niu. Yuantao Yang, Yueyan Li, Yushi Bai, Yuxiao Dong, Zehan Qi, Zhaoyu Wang, Zhen Yang, Zhengxiao Du Zhenyu Hou, and Zihan Wang. Chatglm: A family of large language models from glm- glm-130b to glm-4 all tools, 2024.
Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: An advanced diagnostic suite for en- tangled language hallucination & visual illusion in large vision-language models. arXiv preprint arXiv:2310. 14566, 2023.
Muyang He, Yexin Liu, Boya Wu, Jianhao Yuan, Yueze Wang. Tiejun Huang, and Bo Zhao. Efficient multimodal learning from data-centric perspective. arXiv preprint arXi: 11530, 2024.
Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: visual language model for gui agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14281-14290, 2024.
Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan. Liang Zhang, Bo Zhang, Chen Li, Ji Zhang, Qin Jin, Fei Huang, et al. mplug-docom 1 1.5: Unified structure learning for ocr-free document understand- ing. arXiv preprint arXiv:2403./28
2024a.
Wenbo Hu, Yifan Xu, Y Li, W Li, Z Chen, and 7. Tu. Bliva: A simple multimodal Ilm for bet- ter handling of text-rich visual questions. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024b.
Runhui Huang, Xinpeng Ding, Chunwei Wang, Jianhua Han, Yulong Liu, Hengshuang Zhao, Hang Xu, Lu Hou, Wei Zhang, and Xiaodan Liang. Hires-llava: Restoring fragmentation input in high- resolution large vision-language models. arXiv preprint 24X 2024.
12

---

Published as a conference paper at ICLR 2025
Zheng Huang. Kai Chen, Jianhua He. Xiang Bai, Dimosthenis Karatzas, Shijian Lu, and C. V. Jawahar. ICDAR 2019 competition on scanned receipt oct and information extraction. In 2019 International Conference Document Analysis and Recognition (ICDAR), pp. 1pp.5510 1516-1520, 2019. doi: 10.1199//DAR 2019.00244.
Guillaume Jaume, Hazim Kemal Ekenel, and lean-Philippe Thiran. Funsd: dataset for form under- standing in noisy scanned documents. In 2019 International Conference on Document Analysis and Recognition Workshops (ICDARW), volume 2, pp. 1-6, 2019. doi: 10.1 109/CI//R.2001, 10029.
Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13700- 13710, 2024.
Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. Dvqa: Understanding data visual- izations via question answering. In Proceedings of the IEEE conference on computer vision and patter recognition, pp. 5648-5656, 2018.
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth dozen images. In ECCV, pp. 235-251, 2016.
Jianfeng Kuang, Wei Hua, Dingkang Liang, Mingkun Yang, Deqiang Jiang, Bo Ren, and Xiang Bai. Visual information extraction in the wild: practical dataset and end-to-end solution. In International Conference on Document Analysis and Recognition, pp. 36-53. Springer, 2023.
Bo Li, Kaichen Zhang. Hao Zhang, Dong Guo, Renrui Zhang, Feng Li, Yuanhan Zhang, Ziwei Liu, and Chunyuan Li. Llava-next: Stronger lims supercharge multimodal capabilities in the wild, 2024a.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408. 03326, 2024b.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Bench- marking multimodal Ilms with generative comprehension. arXiv preprint arXiv:2307.16125 2023a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 2023b.
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mybench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 02195-22206, 2024c.
Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In European Conference on Computer Vision, pp. 323-340. Springer, 2024d.
Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi molliityd vision language models. arXiv preprint arXiv:2403 188144 2024c.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao and Ji-Rong Wen. Evaluating object hallucination in large laiionl models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 292-305, 2023c.
Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi- modal models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattenn Recognition, pp. 26763-26773 2024f.
13

---

Published as a conference paper at ICLR 2025
Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv23311.11122, 2023a.
Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv preprint urXiv:2311.07522 2023b.
Chaohu Liu, Kun Yin, Haoyu Cao, Xinghua Jiang, Xin Li, Yinsong Liu, Deqiang Jiang, Xing Sun, and Linli Xu. Hrvda: High-resolution visual document assistant. arXiv preprint arXiv:2404. 06918,99988. 000.0148.. 2024a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurlPS, 2023a.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 26296-26306, 2024b.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: next: Improved reasoning, ocr, and world knowledge, January 2024c. URL https://
Ilava- v1. www.ith.b................ wt/blog//////ooo/ooo 2024 01 30- llava-next/
Haotian Liu, Chumyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024d.
Ruyang Liu, Chen Li, Haoran Tang, Yixiao Ge, Ying Shan, and Ge Li. St-llm: Large language models are effective temporal learners. In European Conference on Computer Vision, pp. 1-18. Springer, 2024e.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: your multi-modal model an all-around player? arXiv preprint arXiv:2307.0628/ 2023b.
Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xucheng Yin, Cheng lin Liu, Lianwen Jin, and Xiang Bai. On the hidden mystery of ocr in large multimodal models. arXiv preprint arXiv:2305.0785 2023c.
Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, and Xiang Bai. Textmonkey: An ocr-free large multimodal model for understanding document. arXiv preprint arXiv:2403.04473. 2024f.
Zuyan Liu, Yuhao Dong, Ziwei Liu, Winston Hu, Jiwen Lu, and Yongming Rao. Oryx mllm: On- demand spatial-temporal understanding at arbitrary resolution. arXiv preprint arXiv:2409. 12961, 2024g.
llya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2017.
Haoyu LU, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, al. Deepseek-vl: towards real-world vision-language understanding. 2024.
arXiv preprint arXiv:
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arxiv preprint arXiv:2310.02255, 2023.
Ahmed Masry, Xuan Long Do Jia Qing Tan, Shafiq Joty, and Enamul Hoque Chartqa: A bench- mark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 2263-2277 2022.
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Doevqa: A dataset for vqa document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 2200-2209, 2021.
14

---

Published as a conference paper at ICLR 2025
Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Emest Valveny, and CV Jawahar. Infographicvga. In Proceedings the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1697-1706, 2022.
OpenAl. Gpt-4 technical report, 2023.
Rui Qian, Xiaoyi Dong, Pan Zhang. Yuhang Zang. Shuangrui Ding, Dahua Lin, and Jiaqi Wang. Streaming long video understanding with large language models. arXiv preprint arXiv:2405.16009, 2024.
Zhenwei Shao, Zhou Yu, Jun Yu, Xuecheng Ouyang. Lihao Zheng. Zhenbiao Gai, Mingyang Wang, and Jiajun Ding, Imp: Highly capable large multimodal models for mobile devices. arXiv preprint arXiv:2405. 107 2024.
Baifeng Shi, Ziyang Wu, Maolin Mao, Xin Wang, and Trevor Darrell. When do we not need larger vision models? arXiv preprint arXiv:2403. 13043, 2024.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
Jingqun Tang, Chunhui Lin, Zhen Zhao, Shu Wei, Binghong Wu, Qi Liu, Hao Feng, Yang Li, Siqi Wang, Lei Liao, et al. Textsquare: Scaling up text-centric visual instruction tuning. arXiv preprint arXiv:2404. 12803, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:230.113771, 2023.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing Jisionlanguaag model's perception of the world at any resolution. arXiv preprint arXiv:2409. /2191, 2024.
Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong. Ji Qi, Yan Wang, Junhui Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:231 1.0399,, 2023a.
Yonghui Wang, Wengang Zhou, Hao Feng, Keyi Zhou, and Houqiang Li. Towards improv- ing document understanding: An exploration on text-grounding via milmss arXiv preprint arXiv:2311.13194, 2023b.
Haoran Wei, Lingyu Kong, Jinyue Chen, Liang Zhao, Zheng Ge, En Yu, Jianjian Sun, Chunrui Han, and Xiangyu Zhang. Small language model meets with reinforced vision vocabulary. arXiv preprint arXiv:2401. /25003 2024.
X.a.. Grok-1.5 vision preview.
2024.
Ruyi Xu, Yuan Yao, Zonghao Guo, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, and Gao Huang. LLaVA-UHD: an Imm perceiving any aspect ratio and high-resolution images, arXiv preprint 2rXiv:2403.11703. 2024.
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang. Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, Qianyu Chen, Huarong Zhou, Zhensheng Zou, Haoye Zhang, Shengding Hu, Zhi Zheng, Jie Zhou, Jie Cai, Xu Han, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. Miniepm-v: A gpt-4v level milm on your phone. arXiv preprint arXiv:2408. 0/800, 2024.
Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, et al. Ureader: Universal ocr-free visually-situate language understanding with multimodal large language model. In The 2023 Conference on Empirical Methods Natural Language Processing, 2023a.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint 2304. 14/78, 2023b.
15

---

Published as a conference paper at ICLR 2025
Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou mplug-owl2: Revolutionizing
large language model with modality collaboration. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2024.
Ya-Qi Yu, Minghui Liao, Jihao Wu, Yongxin Liao, Xiaoyu Zheng, and Wei Zeng. Texthawk: Ex- ploring efficient fine-grained perception of multimodal large language models. arXiv preprint arXiv:2404.09224, 2024.
Zhengqing Yuan, Zhaoxu Li, and Lichao Sun. Tinygpt-v: Efficient multimodal large language model via small backbones. arXiv preprint arXiv:2312. 16862, 2023.
Jiaxin Zhang, Wentao Yang, Songxuan Lai, Zecheng Xie, and Lianwen Jin. Dockylin: A large mul- timodal model for visual document understanding with efficient visual slimming. arXiv preprint arXiv:2406. 191001,00,1991,, 2024.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01066 2022.
Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. Llavar: Enhanced visual instruction tuning for text-rich image understanding. arXiv preprint arXiv:2306. 17107, 2023.
Yichen Zhu, Minjie Zhu, Ning Liu, Zhicai Ou, Xiaofeng Mou, and Jian Tang. LLaVA-Phi: Efficient multi-modal assistant with small language model. arXiv preprint arXiv:2401003330, 2024.
A APPENDIX
A.I ABLATION STUDY OF THE MAXIMUM NUMBER OF TILES.
We conduct an experiment explore the effect of varying the number of tiles in CIP. The results are presented in Table I1 Our findings indicate that overall performance initially improves with an increase in the number of tiles but begins to decline after reaching a certain point. Optimal perfor- mance is achieved when the maximum number of tiles is set to 24. Therefore, we choose 24 as the default maximum number tiles. Additionally, we perform K-means clustering on the resolution ratios of images and use the clustering results as the predefined aspect ratios. We found that using the clustering results as predefined aspect ratios provides a slight improvement over manually preset them.
Table 11: Ablation study of the maximum number of tiles.
maximum number of tiles
TexiVQA
OCRBench
MMF
HallB
POPE
782
1837.2
39.0
36
75.7
784
1814.5
39.1
87.3
24
76.0
806
1884.2
38.8
88.0
12
75.5
796
1874.1
38.8
87.4
6
74.1
788
1879.2
37.9
87.2
K-means
76.2
806
1891.5
39.1
88.1
A.2 ABLATION STUDY OF DIFFERENT SETTINGS OF GROUPING OPERATION.
We conduct experiments to investigate different pre-defined aspect ratio settings for the CIP. All experiments are performed using 24 as the maximum number of tiles Nrae. The pre-defined aspect ratios are determined according to the following formula:
{g = (ng, x niw)) Nmin S Tip The S N Nmaz: Thp E N, nl E N].
where ny and represent the height and width of the grid g. respectively. the results are shown the Tab. 12 < represents the Nmi is set to _ * Nuite and the Nmax is set to 1 Nrae- According to the results of the experiment, we chose 1 < i < 1 for detailed group and 1/2 < < for adaptive group. In contrast, the global group employs fixed 1:1 aspect ratio.
16

---

Published as a conference paper at ICLR 2025
Table 12: Ablation study of the different settings of grouping operation.
Detailed Group
Adaptive Group
TexiVQa
OCRBench
MME
HallB
POPE
< <i<l
<i<
76.0
800
1886.7
38.7
87.7
<i<1
<<<<
76.1
804
1882.0
38.1
87.8
<i<1
<i<<
76.0
1884.2
38.8
88.0
<i<1
<< <
75.7
1873.7
38.0
87.9
<i<1
< <
75.6
1860.6
87.3
Table 13: Ablation study on the impact of different numbers of LLM layers in SCM.
The number of LLM layers
TeaVQA
OCRBench
MME
HallB
POPE
Flops (B)
Lasenc/Example
37.6
74.7
794
1886.0
38.7
86.1
424.4
0.92s
74.5
789
1878.2
38.3
86.0
408.6
0.89
kinds printing way meet your
Acveragge ORAN ACTUPS CORNIS
Ainb printin
244 scres privvo (iece (AveSA))
DRI DRA
VUNMILR
ORI
DRA
(a) Detail image:
Adaptive image:
(c) Global image: texts are blurry
texts have been severoly comrpptdd and are hard read.
texts can be recognized easily
Figure 4: Qualitative results of CIP. The green box indicates the text that needs to be recognized.
A.3 ABLATION STUDY OF THE DIFFERENT NUMBERS OF LLM LAYERS IN SCM.
To further investigate the effect of varying the number of layers in LLMs the compression of visual tokens, we conducted a series of experiments. All experiments are conducted in using 0.5 compression rate. The results are detailed Tab. [3] Our findings indicate that increasing the number of layers leads to enhanced model performance. Nevertheless, this improvement comes the cost of increased computational demands and higher latency. Balancing these factors, we decided to adopt two-layer L.M as our standard configuration, optimizing for both efficiency and performance.
A.4 EFFECTIVENESS OF ADAPTIVE GROUP
To more intuitively demonstrate the role of the adaptive group, we present visualization case, as shown in Fig. 4 We can find that the texts are severely corrupted and are hard to read in the detailed image. The global image is used to help retain some of the overall context. However, due to the low-resolution of global images, the texts are blurry. The adaptive group is capable of dynamically adjusting based on the needs of the detailed group and provides more fine- fine- grained feature representations. MILMS easily read the texts from the adaptive image.
17

---

Published as a conference paper at ICLR 2025
Calculate the
1:1
1:3
3:1
Pre-defined aspect ratio
Generate zspect ratto
2 Calculate
0.5
1.5
1:1
1:2
1:3
3:1
absolute differences
2:1
2:3
3:2
2:1
2:3
0.5
0.5
0.83
Obtain matched
Figure 5: The process of selecting the optimal ratio. First, we generate a pre-defined aspect ratios. Next, for a given input image, calculate its aspect ratio. Then, calculate the absolute differences between the image's s aspect ratio and each the pre-defined ratios. Lastly, we select the pre-defined aspect ratio that has the smallest difference as the optimal match.
A.5 THE PROCESS OF SELECTING THE OPTIMAL RATIO
Given an input image, we first compute the absolute differences between its aspect ratio and those within the detailed group: la bl. The aspect ratio with the smallest absolute difference that of the input image then selected as the matched ratio. The overall process is shown Fig.
A.6 EXPERIMENT ON VIDEO TASKS.
We also conduct experiments video tasks, using MMBench--idee. (Fang al. 2024) and Video- MME (Fu et al. 2024). The results are presented in Tab. [4]and Tab. [5] We found that CIP offers only marginal improvements on these two datasets. The primary reason for this is that these video datasets require relatively low-resolution inputs; they typically require small images without the need for resolution enhancement techniques, such as cropping strategies. Therefore, the enhance- ment brought by CIP on these datasets is limited.
Table 14: Comparison of different methods MMBench- Video.
Method
Overall Mean
CP
FP-S
FP-C
HI
Mean
LR
AR
RR
CSR
TR
Mean
LLaMA-VID
2024d
1.09
1.09
0.71
1.08
VidenSmeeminn (Qin mi 2024
1.12
1.38
1.13
0.32
1.13
0.77
1.27
1.11
1.01
1.10
NLXTA-NeXT-Vaal Lr 2024a
1.14
1.35
1.15
0.97
0.58
1.14
0.64
1.38
1.30
1.27
1,03
1.13
(Raselihe)
Mini-Monkey-2B (Ours)
1,20
1.45
1.22
1.06
0.74
1.22
0.89
1.19
1.42
1.17
1.05
1.16
Table 15: Performance MILMs on Video-MME with short, medium, and long durations, under the setting of "without subtitles".
Models
LLM Params
Without Subtitles (%)
Overall (%)
Short
Medium
Long
Open-source Video MLLMs
Video-LLaVe LVA (Lin glal. 7023g
7B
45.3
38.0
36.2
39.9
Liu
Video Misrr Greall
2024
7B
48.3
37.0
33.2
39.5
Chag-UniVi-V1.
7B
45.7
40.3
35.8
40.6
Open & Closed -source Image MILLMs
Owen-VL-Chal Wang
Owen-YL-Max Wine craie 40.2
55.8
49.2
48.9
51.3
InterVI- Chat. VT.3 1o0
2024
20B
60.2
46.4
45.6
50.7
InterV1.22 Chen
Mini-Monkey
2B
40.1
35.1
43.3
18

---

Published as a conference paper at ICLR 2025
Quantitative accuracy (%) comparison of model with existing multimodal large lan- guage models (MLLMs) on several benchmarks. Following TextMonkey Liu et al. (2024f). we use the accuracy metries to evaluate our method.
Table 16:
Scene Text Centric VQA
Document Oriented VQA
KIE SROIE
OCRRench
STVQA
TexVQA
DoeVQA
InfoVQA
CharQA
FUNSD
POIE
MiniCCMM-V-2.6-8B MiniCPM-V-22. 8B
63.8
73.4
53.7
42.9
62.1
80.4
852
65.3
73.1
81.7
55.3
70.6
43.4
62.5
81.1
858
A.7 MORE QUANTITATIVE RESULTS
We present several visualization results from Mini-Monkey, illustrated in Figure In Figure 6 a), we evaluate the model's performance on general multimodal comprehension. When asked about the characters depicted in the image, Mini-Monkey demonstrated its capability by accurately identifying multiple characters from the Avengers.
In Figure b)), tested the model's understanding of contextual scenarios. By posing questions related to the swimming pool setting, Mini-Monke not only correctly identified the activities taking place but also provided an insightful analysis of potential hazards associated with the environment, showcasing its ability to infer beyond the visible elements.
Figure Ge) highlights Mini-Monkey's proficiency in extracting structured information from images. We tasked the model with converting the visual data into a JSON format, and it successfully pro- duced detailed and accurate representation, indicating its strong capacity for data organization and structure.
Finally, in Figure d), we assessed the model's s ability to process and analyze menu-related infor- mation. Mini-Monkey was not only able to precisely recognize and read the text within the image but also effectively understood the context of the questions posed and performed the required math- ematical calculations, thereby demonstrating its comprchensive skill set in combining visual and textual analysis.
A.8 THE ACCURACY METRIC.
In this section, we detail the metric described in (Liu al. 2023c), which establishes a uni- form and comprehensible standard for assessing text-focused benchmarks. We utilize benchmarks such as CharQA (Masry et al., 2022). DocVQA (Mathew et al., (02T), InfoVQA (Mathew al. 2022), TextVQA (Singh et al. 2019). STVQA (Biten et al. (201)). FUNSD (Jaume et al.) 2019). SROIE (Huang et al., 2019), and POIE (Kuang et al. 2023). This metric evaluates outcomes by confirming the presence of the ground truth (GT) within the model's output while excluding any responses that are less than four characters in length. To further validate the effectiveness of our pro- posed CIP, we have also conducted experiments using this metric with the state-of-the-ee method, MiniCPM-V-2 2-26-V2.66-8- (Yao et al. 2024). As presented in Tab. 16 our findings reveal that even when applied to leading multimodal large language model like MMmCCPM-V-2.6--2 can enhance performance. These results further indicate the effectiveness of the proposed CIP.
the proposed CIP
A.9 LIMITATIONS.
To ensure the seamless application of our method across various architectures, we adopt an image- centric approach to construct image pyramid without introducing additional parameters. In future work, we will explore the use of trainable Feature Pyramid Network (FPN) for MLLMs, aiming to more efficiently leverage multi-scale features.
19

---

Published as a conference paper at ICLR 2025
h
What kind of sport can I play here?
You can play swimming and diving in this pool.
What preparations are needed for diving? Please provide concise introduction.
How many charactors are there in the picture?
To prepare for diving in this pool. you should first check the diving board's height and easure that it is within your skill lovel. Next, make sure you are wearing appropriale sw imwear and have a lowel nearby to dry Finally, practice your diving technique to ensure safety and efficiency in the water.
There are six characters in the picture.
ho are they?
W hal is the danger
sport?
The chamcters the picture are from the Avengers. From left to right, they are: Thor; 2. Hulk; 3. Iron Man; 4. Caplain A merica; Black Widow; Hawkeye
Tee danger diving in this pool includes the risk of injury from hilling the water colliding with other swimmers. A dditionally, there risk of drowning if you are not a strung swimmer or if you do not know how to perforn a proper dive. it important to always follow safety guidelines and use proper diving tochniques minimize these risks.
Appelignn
hat products are the hottom right comee of the image?
Shapl
The products in bottom right, comer of the image are desserts.
hat are the specific categories dessects?
Carabos
Dossiop
The specific categortes of desserts in the bollom right
Convert json format.
comer of the image are:
{"ülle:: "Plustic Can Take 500 Years To Blo-Degrade In The Ocean", "subtide": "Estimated number of years for selected itoms to bio dio-degr in marine environment", "Items":
Cheesecake; 7.7 Layer Cake; 3.
Apple Strudle: 4. Chocolate
Deuerto
Gol
Brownie; Rice Pudding: 6.
Coffee Cake; 7, Raspberry Joy
Bar; 8. Halvah Bar; 9, Chocolate
[( ('name": "Cigarette Butt", "estimate": 10), ("bane": "Plastic G Grocary Bag",
Covered; 10. Imond Halvah Bar
"estimate"; 20), ("name": "Slyrofoam
How much does it cost to buy two Layar Cakes? Please provide the calculation process.
Cup", "estimmat": 50), ("name":
"A "Aluminum Cans", "estimate" : 200),
(" I'name":" " Plastic Beverage Holder", "estimate": 400), ("name": "Disposable Diaper', "estimate": 450),
To calculate the cost of buying two Layer Cakes, need to look at the price listed for single 7 Layer Cake in the "Desserts" section of menu. The price for 7 Layer Cake $3.99.
(" I'name": Plassic Bolilo", "ostimate":
450), ("name": "Fishing Line".
"estimate" 600)), "note": Exxact time varies by product type and marine conditions. Cigurette butts and grocery hags upper estimate." "
find the cost of two Layer Cakes, we multiply the price Layer Cake by 2: $3.99 2 = $7.98
Therefore, it would cost $7.98 to buy two Layer Cakes.
Figure 6: Qualitative results of Mini-Monkey. Figures (a) and (b) pertain to general multimodal understanding, Figures (c) and (d) relate to document understanding.
20


Reviews:
[{"review_id": "HJM3K8IsaL", "reviewer": "Reviewer_cDq9", "summary": "The paper introduces the \"semantic sawtooth effect\" caused by common cropping strategies in high-resolution image scaling for MLLMs. To tackle this issue, they propose a Complementary Image Pyramid (CIP), a flexible and easy-to-integrate approach aimed at reducing semantic discontinuity by providing rich semantic information across different scales. Alongside CIP, they also introduce a Scale Compression Mechanism (SCM) to minimize computational overhead by compressing unnecessary visual tokens. These enhancements improve performance across various MLLM architectures and capacities, leading to the development of a lightweight model called Mini-Monkey, which shows notable improvements in multimodal and document understanding tasks.", "questions": "1. There is a lack of explanation for the setting of the Adaptive Group. While both detail and global are easy to understand, the paper does not explicitly state the benefits of detail group or provide experimental evidence to support it.\n2. The experiments lack ablation studies, such as removing the detail, adaptive, or global components. What would be the impact if one of these were removed? Which component is most critical? Secondly, what impact does the number of tiles in CIP have? How does the performance of CIP change with the number of tiles in the detail component? Also, what aspect ratios are set by default for CIP?\n3. The motivation behind SCM does not align well with the experiments. The paper mentions that \"certain scenarios may restrict the level of computational resources available,\" but the experimental part does not provide experiments on how different compression rates of SCM affect model acceleration and computational cost.\n\nIf the authors could supplement their experiments, I would be willing to raise the score to above borderline.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed CIP is logically clear and reasonable. Experiments show that CIP outperforms other cropping methods.\n2. The experiments are comprehensive, showing significant improvements across various model families and sizes, as well as multiple datasets, which demonstrates the effectiveness of the proposed CIP.\n3. The paper is well-written and clear.", "weaknesses": "1. The insight and inspiration of proposed Adaptive Group is not clear.\n\n2. Lack ablation studies on proposed CIP and SCM\n\nSee below for details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SGbdrgGaQO", "reviewer": "Reviewer_bHqc", "summary": "This paper proposes Mini-Monkey, a lightweight multimodal large language model that effectively mitigates the semantic sawtooth effect in high-resolution image processing through a Complementary Image Pyramid (CIP) and a Scale Compression Mechanism(SCM), achieving superior performance across various benchmarks.", "questions": "1、I would like to inquire why the paper does not mention the maximum resolution of the images that the model supports, as well as the corresponding comparative experiments.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1、The paper describes CIP for dynamic segmentation of images and SCM for compression of visual tokens to address the semantic sawtooth effect in MLLM high-resolution image processing, demonstrating innovations in addressing specific challenges.\n2、In the CIP module, the model focuses on the feature interactions of different sub-images\n3、In the SCM module, the model selectively compresses visual tokens. The interaction information of different types of visual tokens is also considered.", "weaknesses": "1、In Figure 2a, the pixel shuffle operation appears, but the paper does not reflect the transformation of the image features before and after this operation.\n2、According to the formula in line 241, the aspect ratios of the adaptive and detailed groups are not integer multiples. But for Figure 2b, the final selection of Ah is 1 and Dh is 3, which seems to be a contradiction.\n3、In the CIP module, the paper does not present a clear picture of how the predefined slice ratios appropriate to the size of the image are selected, i.e., what principle is it based on.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4QaAA00wPN", "reviewer": "Reviewer_HRBu", "summary": "Existing multimodal large language models (MLLMs) often use cropping when processing high-resolution images (divides the high-res image into multiple lower-resolution as the input). However, Non-Overlapping Cropping can lead to semantic discontinuity and semantic damage, referred to by the authors as the \"semantic sawtooth effect.\" On the other hand, Overlapping Cropping results in redundant visual information.\n\nTo address this, the paper proposes a complementary image pyramid, which aims to alleviate the semantic sawtooth effect in the context of Non-Overlapping Cropping. To mitigate the additional computational burden introduced by this module, the authors propose a Scale Compression Mechanism. This mechanism leverages the attention weights of the LLM and the proposed multi-scale image semantics in a training-free and parameter-free manner to compress redundant tokens.\n\nThe proposed approach achieves promising results on 8 general multimodal understanding benchmarks and 9 document understanding benchmarks.", "questions": "Please refer to the Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The use of the complementary image pyramid (CIP) to replace high-resolution input images with sub-images of various scales is an excellent idea. Compared to existing methods that introduce multi-scale visual signals through models, this approach is simpler and more effective, without requiring additional parameters or training.\n\n2. The Scale Compression Mechanism (SCM) reasonably and interestingly reduces the extra computational load brought by multi-scale input sub-images by compressing visual tokens.\n\n3. Both the proposed CIP and SCM do not require the introduction of additional parameters or training, making them applicable to different MLLMs.\n\n4. The method proposed in this paper achieves promising results in various general multimodal understanding and document understanding benchmarks.", "weaknesses": "1. The authors claim in lines 265-266 that \"a well-trained LLM from MLLM can effectively select the necessary visual features based on the input question,\" which seems to differ from the conclusions of existing MLLM works [1,2]. This discrepancy makes me question the effectiveness of the proposed method. If MLLMs cannot truly understand images, how can their attention weights be used here to compress visual tokens?\n\n2. The rationale for selecting only the first and second layers of the LLM to choose visual tokens is not sufficiently explained, and no ablation studies have been conducted. How would the results differ if more LLM layers were selected, only one LLM layer was chosen, or the selection was done randomly without using LLM attention priors? In conclusion, ablation studies have only been conducted on the Resolution Strategy, lacking ablation experiments on the compression of visual tokens.\n\n3. For the complementary image pyramid, the authors need to manually preset a set of predefined aspect ratios, which seems somewhat tricky. How these aspect ratios are set and why these specific values are chosen remains unclear. A better solution might be to perform K-means clustering on the resolution ratios of images and use the clustering results as the predefined aspect ratios.\n\n4. When comparing with other methods and conducting ablation studies, only the number of parameters and performance are shown, lacking comparisons on FLOPs. Although your proposed multi-scale input does not introduce new parameters, it does increase actual computational load. Therefore, in the ablation studies of Table 4, the actual computational load and inference overhead should also be compared.\n\nReference \n* [1] Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs, NeurIPS 2024\n* [2] Are We on the Right Way for Evaluating Large Vision-Language Models?, NeurIPS 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k9GqjLFo4X", "reviewer": "Reviewer_q5HV", "summary": "The paper focuses on the issue of semantic discontinuity in MLLM when scaling images to high resolution, particularly through a sliding-window cropping strategy that can misidentify small or irregularly shaped objects. To tackle this problem, the paper proposes the Complementary Image Pyramid (CIP), which dynamically constructs an image pyramid to enhance semantic information. Besides, the authors introduce a Scale Compression Mechanism (SCM) to minimize computational overhead by compressing redundant visual tokens. Experimental results show the proposed method achieves the best performance across diverse benchmarks.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Mini-Monkey tackles an important problem in MLLM: scaling images to high resolution. The Complementary Image Pyramid (CIP) introduces the pyramid structure, which is an interesting idea. Besides, CIP and SCM are plug-and-play, which can be easily integrated into different MLLMs.\n2. The experiments are sufficient.", "weaknesses": "1. Implications of this research. Whether the semantic sawtooth effect mentioned in the paper is a necessary issue to investigate, common Crop-based methods (such as LLaVA-UHD [1] and InternVL [2]) put all cropped regions into a sequence, which does not affect semantic continuity.\n2. The work is incremental. The core crop strategy has been widely used in other approaches. CIP is an incremental improvement and doesn't mean much to the community.\n3. The architecture is highly sophisticated. The global group is enough to solve the loss of fine-grained features caused by the detailed group. This brings the question of whether the proposed adaptive group is necessary for CIP.\n4. The writing needs further improvement. Authors are suggested to improve the readability of the paper. For example, it is hard to understand \"For the detailed group, we calculate the aspect ratio of the input image and then compare it with the aspect ratios within the detailed group by calculating the absolute differences.\" in L231-L233. How to compare? Another example: L270 \"We reuse the layer of the LLM as this LLM's Layer\". What's the difference between two LLMs?\n5. Incomplete experimental analysis. Experimental analysis should include analysis of reasons and not just a list of indicators.\n\n[1] Xu R, Yao Y, Guo Z, et al. Llava-uhd: an lmm perceiving any aspect ratio and high-resolution images[J]. arXiv preprint arXiv:2403.11703, 2024.\n\n[2] Chen Z, Wu J, Wang W, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 24185-24198.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)