Efficient Multi-modal Long Context Learning for Training-free Adaptation
Authors: Zehong Ma
Shiliang Zhang
Longhui Wei
Qi Tian
Venue: ICML 2025 poster (2025)
Keywords: Multi-modal Long Context Learning
Training-free Adaptation
Adaptive Pruning
Primary area: applications->computer_vision

Abstract: Traditional approaches to adapting multi-modal large language models (MLLMs) to new tasks have relied heavily on fine-tuning. This paper introduces Efficient Multi-Modal Long Context Learning (EMLoC), a novel training-free alternative that embeds demonstration examples directly into the model input. EMLoC offers a more efficient, flexible, and scalable solution for task adaptation. Because extremely lengthy inputs introduce prohibitive computational and memory overhead, EMLoC contributes a chunk-wise compression mechanism combined with layer-wise adaptive pruning. It condenses long-context multimodal inputs into compact, task-specific memory representations. By adaptively pruning tokens at each layer under a Jensen-Shannon divergence constraint, our method achieves a dramatic reduction in inference complexity without sacrificing performance. This approach is the first to seamlessly integrate compression and pruning techniques for multi-modal long-context learning, offering a scalable and efficient solution for real-world applications. Extensive experiments on diverse vision-language benchmarks demonstrate that EMLoC achieves performance on par with or superior to naive long-context approaches. Our results highlight the potential of EMLoC as a groundbreaking framework for efficient and flexible adaptation of multi-modal models in resource-constrained environments. Codes are publicly available at https://github.com/Zehong-Ma/EMLoC.

Efficient Multi-modal Long Context Learning for Training-free Adaptation
Zehong Ma Shiliang Zhang
1
Longhui Wei
Qi Tian
Abstract
Multi-modal Long Context Learning on ImageNet100
65
2.6k context
Traditional approaches to adapting multi-modal large language models (MILMs) to new tasks have relied heavily on fine-tuning. This paper introduces Efficient Multi-Modal Long Context Learning (EMLoC), novel training-free alter- native that embeds demonstration examples di- rectly into the model input. EMLoC offers more efficient, flexible, and scalable solution for task adaptation. Because extremely lengthy inputs in- troduce prohibitive computational and memory overhead, EMLoC contributes a chunk-war wise com- pression mechanism combined with layer-wise adaptive pruning. It condenses long-context mul- timodal inputs into compact, task-specific mem- ory representations. By adaptively pruning to- diver-
60
Compression
4.4X
55-
18K context
Accuracy
50
45
EMLoC(QwenZ-VL) MLoC(Qwen2-VV)
context
35
30
0
50
100
150
200
250
300
Number of Demonstration Examples
Figure I. The comparison between EMLoC MLoC on Ima- geNet100 with varying numbers of demonstration examples. With 200 examples, EMLoC achioves 4, 4,4x context compression over vanilla MLoC without performance loss. It significantly outper- forms MLoC with 50 examples using a similar context length.
kens at each layer under gence constraint, our method achieves a dramatic reduction in inference complexity without sac- rificing performance. This approach the first seamlessly integrate compression and pruning techniques for multi-modal long-com long-context learn- ing, offering scalable and efficient solution for real-world applications. Extensive experiments on diverse vision-language benchmarks demon- strate that EMLoC achieves performance on par with or superior to naive long-context approaches. Our results highlight the potential of EMLoC groundbreaking framework for efficient and flexible adaptation of multi-modal models in eesource-constriigg environments. Codes are publicly available at https:/githuu.coo//Zenngg Ma/EMLoC.
1. Introduction
In recent years, multi-modal large models (MLLMs) (Liu et al., 2024; Zhu et al., 2024; Bai et al., 2023) have achieved significant advancements, demonstrating remarkable suc- cess across a wide range of multi-modal tasks. Tradition- ally, transferring these models to downstream tasks relies supervised fine-tuning, including full fine-tuning and parameter-efficient methods (Hu al., 2022). However, these techniques require updating model parameters and of- ten incur substantial computational costs. Recently, MLLMs have evolved from processing single-image inputs (Liu et al., 2024) to handling multi-image and video data (Wang et al., 2024b; al., 2024; Chen et al., 2024c), while also sup- porting increasingly long context lengths.
'tttt Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University Peng Cheng Lab- oratory, Shenzhen, China 'Haawei Inc. "Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ). Correspondence to: Shiliang Zhang <slzhang.jjd jdl @ pku.cdu.cn.>.
A key observation, illustrated in Figure 1, is that provid- ing task-specific demonstration examples during inference significantly enhances model performance. We refer to this approach as Multi-modal Long Context learning (MLoC). However, directly feeding multiple multi-modal examples into the model often results in excessively long contexts,
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Efficient Multi-modal Long Context Learning for Training-free Adaptation
leading to prohibitively high inference costs.
We briefty review recent advances in these areas and discuss our contributions and differences with them.
To address these challenges, we introduce Efficient Multi- modal Long Context Learning for training-free free adaptation (EMLoC), a training-free adaptation method. EMLoC lever- ages the benefits of multi-modal long context learning while mitigating its computational drawbacks through a chunk- wise compression mechanism combined with layer-wise adaptive pruning. This approach compresses extensive multi-modal contexts into compact, task-specific memory representations, significantly reducing computational over- head without compromising task performance.
2.1. Multi-modal Large Language Models
The progress in LLMs has propelled the advancement of MLLMs. Flamingo(Alayrad et al., 2022) pioncered the integration of pre-trained visual encoder with the Chinchilla 70B(Hoffmann et al., 2022) LLM, demonstrat- ing strong zero-shot and few-shot performance on vision- language tasks. Since then, numerous open-source models have emerged, including Kosmos- (Huuass MiniGPT-4(Zhu al., 2023), and LLaVA (Liu et al., 2024). Subsequent research has expanded MILMS' functional ca- pabilities and improved their visual perception. Models like Kosmos-2(Peng et al., 2023), CogVLM (Wang et al., 2023), Shikra(Chen et al., 2023). Pink(Xuan et al., 2024), and LocLLLM et al., 2024a) incorporate localization through the pix2seq paradigm or connections with detection and segmentation models. Others, such as Qwen- VLaaai et al., 2023), Yi-VL(Young al., 2024), DeepSeek-VI.(L.. et al., 2024), Intern VLCChen et al., 2024c), and Intern- XComposer(Dong et al., 2024), enhance capabilities with high-resolution inputs and larger training datasets. Recently, models such as Miter-XCoompose--2... (Zhang et al., 2024a), Intern VL-2 (Chen et al., 2024c), MiniCPM-V-2.6 (Hu et al., 2024), and Qwen2-VI. (Wang et al., 2024b) have advanced to support multi-image understanding, video comprehen- sion, and multi-modal in-context learning, These recent studies have made it possible to explore MLLMs' multi- modal long-conted learning.
(Huang et al., 2024b),
The chunk-wise compression mechanism partitions a long context into smaller, manageable chunks, enabling a divide- and-conquer approach. This strategy reduces reliance on large-memory devices and significantly lowers computa- tional overhead. Within each chunk, we employ layer- wise adaptive pruning strategy, progressing from the top layer to the bottom layer. Using a greedy search algorithm, we determine the minimum number of tokens required per layer to ensure the Jensen-Shannon (JS) divergence be- tween the original and pruned outputs remains below a predefined threshold. This approach avoids noticeable per- formance degradation while maximizing token reduction. Unlike static methods (Xiao et al., 2024; Li et al., 2024; Cai et al., 2024), which reserve fixed number of tokens per layer, our adaptive strategy dynamically prunes tokens based on layer-specific importance, ensuring optimal efficiency without sacrificing accuracy.
The EMLoC framework operates without modifying model weights and requires only few forward passes to gen- erate task-specific compressed memory. This makes it lightweight, plug-and-plas solution that maintains task per- formance while drastically reducing inference costs. The contributions of this work are threefold:
2.2. Long Context Learning
In-context learning (ICL) is good adaptation method to improve the performance of downstream tasks. We focus on multi-modal long context learning for training-free adap- tation with many in-context examples.
Efficient Multi-modal Long Context Learning: We in- troduce chunk-wise compression and layer-wird wise adap- tive pruning to reduce computational costs and memory usage in long-context scenarios.
Scaling ICL. The foundational study by (Brown et al., 2020) demonstrates that increasing the number of in-context ex- amples improves the performance of LLMs. More recent works begin to explore the effects of scaling in-context ex- amples beyond 1000, with (Li et al., 2023), (Agarwal et al., 2024), and (Bertsch et al., 2024) showing consistent perfor- mance improvements in multiple text-based tasks. However, these studies are limited to text-only benchmarks and lack exploration in multi-modal tasks.
Adaptive Layer Importance Analysis: We dynamically analyze the importance of different layers in a layer- wise manner, offering new insights into token pruning strategies.
We establish a linear upper bound for the information loss and demonstrate the effectiveness of our method across diverse tasks with extensive experiments.
Multi-modal ICL. Despite significant advancements in MLLMs, research on multi-modal in in-context learning (ICL) remains in its early stages. Flamingo (Alayrac et al., 2022) pioneered the exploration of in-context capabilities in MLLMs, demonstrating few-shot learning across various langon-language tasks. Recent studies have evaluated the generalization abilities of models like GPT-4V and Gemini,
2. Related Work
This work is related to multi modal large language models, long context learning, and training- frree context compression.
2

---

Efficient Multi-modal Long Context Learning for Training-free Adaptation
highlighting that in-context learning enhances performance on out-of-domain and out-of-distribution tasks (Zhang et al., 2024b; Han et al., 2023b). However, these works have not fully explored the potential of leveraging extended context windows to incorporate more demonstration examples and long-context inputs. A recent study by (Jiang et al., 2024) in- vestigated many-shot in-context learning in MLLMs, reveal- ing that open-source MLLMs struggle with complex long contexts, while closed-source models perform significantly better. This paper focuses addressing the challenges of multi-modal long-context learning in open-source MILMMs, presenting an initial exploration of this domain. Our work aims to bridge the gap by enabling efficient and scalable long-context adaptation for open-source models.
Representing each example as (1t, Qa, Ar) consisting of an image 7;, a question Qi and corresponding answer Ag, the generation of C can be denoted as,
N C = ① Q1, Q.) -=1
(1)
Conditioning on the multi-modal long context C and multi- modal test queries X, MLLM can generate more accurate answers Y as follows:
Y = MLLM(C, X).
(2)
Increasing the number of examples N provides more infor- mative contexts and enhances performance, but significantly increases the inference cost and memory usage.
2.3. Training-free Context Compression
We thus propose reduce the long context C into compact task-specific memory M through chunk-wise compression with layer-wise adaptive pruning. Using M to denote the key-value (KV) cache of context C, we aim to extract more compact memory M from M.
Training-free context compression techniques have ad- vanced significantly to address the challenges handling extended sequences in LLMs. StreamingLLMM (Xiao al., 2024) retains Key-Value (KV) pairs from initial and recent tokens, leveraging attention concentration reduce mem- ory usage. SnapKV / (Li et al., 2024) dynamically identifies critical KV pairs based on attention patterns, minimizing cache size while preserving performance. H2O (Zhang et al., 2023) prioritizes recent and high-attention tokens with a dy- namic cache strategy, optimizing memory efficiency. Pyra- MidKV (Cai et al., 2024) adjusts KV cache size hierarchi- cally, allocating more memory to lower layers for efficient compression. Recent studies, such as Volta (Chen et al., 2024a) and FastV (Chen et al., 2024b), have focused on compressing visual contexts to enhance the inference accu- racy of mudti-modal large language models. Despite these advancements in text-only and image-only context compres- sion, multi-modal context compression remains relatively underexplored. In this work, we aim adaptively compress multi-modal long contexts into an efficient compact memory without sacrificing performance.
This can be achieved by dividing the C into multiple chunks and adaptively spotting and reserving important tokens across chunks and layers minimize the change of out- put probabilities distribution. The target of M extraction from chunk can be conceptually formulated
M = arg min JM), MCM
s.t. (P( M, X(Y M,X))
(3)
where P-s(-) denotes the -hasnnn divergence. which measures the difference between two probability dis- tributions. A is configurable upper bound regulating the trade-off between information loss and the length of M.
The overall framework of EMLoC is illustrated in Fig- ure 2(a). EMLoC implements the objective in Eq. (3) in two stages. It first partitions C into smaller chunks en- able processing on resource-constrained devices. As shown in Figure 2(b), we partition N demonstration examples into K chunks, each containing K chunk as C₁ i.c.,
3. Method
N examples. We denote each
This section introduces our EMLoC for training- free adapta- tion. We begin with an overview of the approach, followed by the introduction of chunk-wise compression and layer- wise adaptive pruning. Finally, we present theoretical analysis that establishes a linear upper bound on the infor- mation loss introduced by our compression strategy.
(C.)
(4)
Let Dr denote the individual demonstrations in chunk C₁ before concatenation. We iteratively compress each chunk to form the compressed memory:
3.1. Overview
Mr LAP Ck, Ds),
(5)
Given downstream multi modal task and its demonstration examples, our goal is to adapt a pre-trained MILMM with- out any training or parameter fine-tuning. To achieve this, we construct a long context as the input of MLLM by concatenating N task task-specific demonstration examples D.
where LAP(-) is our layer-wise adaptive pruning, and Ma-1 is the compressed memory from the previous chunks. It- crating this process from k = 1 to k = K yields a final compressed memory MR, i.e., M.
3

---

Efficient Multi-modal Long Context Learning for Training-free Adaptation
A nswer
JS Divergence Check
Pon
Pur
Purr
MLLM
Lay aayyr
Layer L
Lay
M spec-specific Memory
Layer
Layer 1
Layer
M Mi
H'
Ma-1
Mi
H
Layer
Fail Check
Pass Check
Chunk-wiss Compression
M
M
D
Forward Demonstrations
Pruning- Checking Steps
Layer- wise
Layer-wii
A ddpptive Pruning
daptive Pruning
Q Question
Cx D
Ck
D
Chunk k-
Chunk k
Task spec-scccific Demonstrations
Test Query
Chunk Partition
Task-specific + Demonstrations
(a) Efficient Multi-modal Long Context Learning
(b) Chunk-wise Compression
Figure 2. (a) The overall framewrk of efficient multi-modal long-context learning. (b) Chunk-wise compression with layer-adaptive pruning, where pruning steps iteratively update output probabilities are validated using JS divergence check. Gray squares indicate pruned tokens, with red and green arrows representing failed and successful pruning steps. respectively.
EMLoC proceeds to prune the KV cache of each chunk in layer-wise manner under constraint, retaining only the most relevant tokens for the downstream task. It is difficult to set a global upper bound A. An effective implementation to is achieved by setting a local constraint The details of layer-wise compression and implementation are presented in following parts.
the concatenated memory M M, and the individual demonstration examples D& R xT in chunk C. The out- put includes the original output probabilities pon of answer tokens, attention weights E R4x xT)xS and hidden states H E R # XT;
(IS) divergence
(Pori a, H) MLLM(M4-1 + Mg,
(7)
where L is the number layers, T denotes the length of each demonstration example, and S represents the sequence length of the current chunk. Then, for each layer, we cal- culate the importance score of each token in M,, using the accumulated attention weights from answer tokens. The answer tokens serve as an observation window to estimate the importance of each token in chunk C. The importance score of j-th token in /-th layer can be formulated
3.2. Layer-wise Adaptive Pruning
Different tasks require varying levels knowledge, result- ing in different degrees of token redundancy. Recent stud- ies (Xiao et al., 2024; Cai et al., 2024) show that earlier layers in large language models are more crucial than later ones. Instead of a fixed pruning strategy (Li et al., 2024; Cai et al., 2024) that retains a constant number of tokens per layer, we propose a layer-wise adaptive pruning strategy that dynamically adjusts the token number per layer while preserving performance. The optimal number of tokens per layer is the minimum required keep the divergence between the probability distributions conditioned on the original and compressed memory below predefined threshold ò. Detailed pseudocode is provided in Appendix A. The procedure proceeds as follows:
8
Σ
8
iEans-index
where ans jns_edexx is the index of answer tokens.
Pruning-Checking Step: Tokens with higher importance are retained, while less important ones are pruned iteratively from the top layer down. This top-down approach improves efficiency by avoiding forward passes through unprumed layers. At each step, pruning is applied to single layer using a candidate ratio r. greedily selected in ascending order from the retention ratio set R. Once the top layers above the I-th layer are pruned, the pruning of the (-th layer is formulated as selecting the top r x S tokens based on importance scores: Mr (TTpk( (x, ())))).
Forward Demonstrations: Given the context C₁ of k-th chunk, the KV cache M4, of this chunk is firstly extracted, conditioned on the previously compressed memory M
MI ExtractK (MM Ck).
forward pass is performed through the MILLM with
(9)

---

Efficient Multi-modal Long Context Learning for Training-free Adaptation
As illustrated in Figure 2 (b), can get the output proba- bilities Pier of all answer tokens in one pruning step:
where Das(;;) +) denotes JS distance, the arithmetic square root of JS divergence.
Bounding the Global JS Distance: To extend the local constraint to the global upper bound, we use the triangle inequality for the JS distance:
Pieer MLLM(ME @ M H'),
(10)
where M denotes the compressed memory from layer I onward for the previous k-1 chunks, M denotes the compressed memory from layer onward for the current k-th chunk, hidden states H' the input embeddings 1-th layer. Leveraging hidden states H' extracted in Equation (7) can reduce redundant computation in pruning steps.
Das (...) SDas (Pt. PMM (BMs) )
DJs (PD) MM- Pasee
(13)
The memory Ma-1 is a subset of Ma-1 @ Mr. which, in turn, is a subset of M. Thus, M Mlg more closely resembles the original full memory M. Consequently, the probability distribution Pi 10M, conditioned on M M,, is more similar to P than to M
After each pruning step, we perform the JS divergence check between the output distributions Pori and Pier to measure the information loss. If the divergence is below the threshold 6, the pruning process for the current layer is successful and continues to prune the next bottom layer otherwise select a larger from rentention ratio set to prune this layer again. As shown in Figure 2(b). a step marked by red arrows fails the check, while subsequent step with green arrows retains more tokens and successfully passes the check. This pruning-checking step ensures that the compressed memory retains the essential information for task performance while minimizing inference costs.
D1s (P22, Ph Me--1 Das (PR PME...
(14)
In other words, adding more chunks from the full memory reduces the divergence with Pgb Moreover, because the demonstrations D₁ are most closely related to the context in the k-th chunk itself, we empirically assume:
After successfully pruning all layer for ke-th chunk, we can get the compressed memory Mg of k chunks by concatenat- ing M and M.g.
PP. DJs (MM PM.. S Das (Prt P.D
(15)
3.3. Theoretical Analysis of Information Loss
Theoretical Upper Bound:
Combining the Equations (12)
to (15), we can derive the following global upper bound of the information loss:
is difficult set global upper bound A for JS diver- gence when sequentially processing multiple chunks. An effective implementation to A is achieved by setting local constraint 5 at each pruning-checking step. In this part, we theoretically analyze how the local constraint ó contributes to the global JS divergence between the full memory M and its compressed counterpart M.
Dis (PAR PMPx)
(16)
where the e is the JS distance between the probability distri- bution of full context C and that of the first uncompressed chunk C, Hence, by applying the local constraint at each counk-compression step, the global JS distance VĀ be- tween the compressed memory and the full memory con- trolled by 5 and chunk number K. More proof and analysis are depicted in Appendix D.
We assume that the N demonstration examples D follow the same data distribution the test set. Thus, during inference with the memory M, the probability distribution PM of test set equals to that of demonstration examples D:
4. Experiments
PM = P(AMM
=
= PM:
(11)
4.1. Experimental Setting
where æ denotes extracting the images and questions as multi-modal questions and a denotes getting the corre- sponding answers in the demonstration set.
Evaluation Dataset. We evaluate our EMLoC on six chal- lenging benchmarks: ImageNet100, a subset of ImageNet- Ik (Deng al., 2009) with the first 100 classes for recogni- tion, ScreenSpot for cross cross-platform GUI grounding, MME- RW for real-world multimodal tasks, Illusion VQA for illu- sion understanding, OK-VQA for knowledge-based QA, and YouCook2 for video understanding. For datasets without predefined validation splits, we randomly sample 100 test examples for evaluation.
Local JS Distance Constraint: With the individual demon- strations Dr of the k-th chunk, the JS distance between the probability distributions under Ma-1 @ Ml, and M₁ is bounded by a divergence distance threshold v8:
DJs (PDr (MM--1M," PM.i) S Vo,
(12)
5

---

Efficient Multi-modal Long Context Learning for Training-free Adaptation
Table Results of efficient multi-modal long context learning (FML.oC) in various downstream tasks. context length. i denotes using 50 examples. I represents utilizing 200 examples.
The value in the gray cell is the
Example Number
Method
ImageNet100
ScreenSpot
MME-RW
PlussinVQQ
OK-VQA
YouCook2
Llava 1.5 (7B)
12.3
9.7
28.2
24.1
53.6
IntervVL2 (8B)
12.5
2.7
33.9
28.0
47.1
88.0
Llama3,2-V (IIB)
47.6
8.1
14.6
33,0
MiniCPM-V2.6 (8B)
31.0
37.2
34.6
48.3
Qwen2-VL.77B)
28.0
14.2
36.6
35.3
52.1
25.4
43.2 f
14.7
39.4
38.8
58.4
86.9
5
MLoC
4109
1996
1924
1826
1401
5907
(Qwen2-VL)
62.6
41.1
58.6
108.8
20
16264
7905
7393
7271
5730
23464
EMLoC
63.6 І
18.3
42.2
40.9
58.7
102.0
20
(Qwen2-VI.)
3643
1415
1510
1878
934
6218
Implementation Details. Experiments are conducted NVIDIA L20 GPUs with 48GB of memory. Inference time measured with batch size of on one GPU. The default IS divergence threshold å is set to 0.005, and the chunk size 1.6k. The retention ratio set R [0.1, 0.2, 0.5, 1.0]. We Qwen2-VL as the base model due to its support for 32k multi-modal long contexts in any format and vision encoder that processes images any resolution without tiling. For the ImageNet100 dataset, the image resolution is 224x224, while other benchmarks use 448x448. The evalu- ation of other MLLMs uses their default image resolution.
Table 2. Multi-modal long context learning with varying numbers of examples on ImageNet 100. The value the gray cell is the context length.
Number of Examples
Method
0
25
50
200
300
28.0
39.9
43.2
62.6
62.9
MLoC
0
2053
4109
16264
24468
28.0
39.4
46.2
63.7
62.6
EMLoC
0
565
946
3643
5365
Results with varying numbers of examples. Table 2 shows how the performance of MLoC and EMLoC changes as the number of examples increases on the ImageNet100 dataset. Both MLoC and EMLoC show steady improvement with more examples, demonstrating strong long-context learning capabilities. However, this improvement comes with a sharp rise in context length, which inflates computa- tional overhead. In contrast, EMLoC compresses the context by nearly a quarter while maintaining or even surpassing MLoC's accuracy (e.g., 63.7 vs. 62.6 with 200 examples), This significant reduction in context length substantially decreases inference time while preserving performance.
4.2. Experimental Results
Performance of EMLoC on multiple tasks. As shown in Table I, multi-modal long-context learning significantly enhances Qwen2-VL's performance across all downstream tasks, whether using 20 demonstration examples. In tasks like ImageNet100 and YouCook2, leveraging an tremely long context yields substantial performance gains.
These results underscore the importance of efficient multi- modal long context learning for practical applications, par- ticularly ImageNet100 and YouCook2. When utilizing 20 examples, our EMLoC dramatically reduces the aver- age context length from 11338 2600, remarkable 77% reduction, without sacrificing performance. Notably, our EMLoC outperforms MI.oC with 5 demonstration examples and 3530 average context length by wide margin, even with a shorter context length. Furthermore, when utilizing 20 examples, EMLoC surpasses MLoC with full memory in most benchmarks. This improvement is likely due to the removal of irrelevant background noise or distractions during compression. As result, the compressed memory is more precise, leading to better performance.
Comparison with other multi-modal ICL methods. We have compared EMLoC with two other multi-modal in- context learning methods, RICES (Alayrac et al., 2022) in Flamingo and MTV (Huang et al., 2024a), on ImageNet100, MME-RW, and OK- VQA in Table 4.2. RICES retrieves the top 25% most relevant in-context samples from all samples. MTV extracts the mean activation of in-context examples as task vectors and finds the optimal replacement position of these task vectors. During inference, MTV replaces these task vectors the optimal position of the test sample, which fails to facilitate these tasks. Our EMLoC. achieves better av- erage performance across the three benchmarks. It's worth noting that RICES is an online retrieval- -agieeete- method, so it needs to forward the retrieved long context during each inference step. RICES takes 5 hours inference time and
Meanwhile, inference costs have been significantly reduced. On ImageNet100 00 with 200 examples, EMLoC reduces infer- ence FLOPs from 1.76T to 1.35T and total inference time from 1866s 1107s compared MLoC.
6

---

Efficient Multi-modal Long Context Learning for Training-free Adaptation
Table Comparison with other multi-modal in-context learning
Table 5. Comparison with other pruming strategies under the differ- ent compression ratios on ImageNet100 with 200 demonstrations and MME-RW with 20 demonstrations
(ICL) methods.
Method
ImageNet100
MME-RW
OK-VQA
MTV
32.7
27.8
ImageNet100
MME-RW
Method
RICES
64.5
40.5
58.5
Ratio
Acc
Ratio
Acc
EMLoC
63.7
42.2
58.7
MLCL
100%
62.6
100%
41.1
SnapKV/H20 V/H20
22.4%
47.6
20.4%
39.6
PyramidKV V
22.4%
49.3
20.4%
39.8
(Muro((Our))
22.4%
63.7
20.4%
42,2
Table Comparison with fine-tuning methods ImageNet100 with 200 examples, MME-RW with 20 examples, and OK-VQA
SnapKV/H20 V/H20
15.7%
47.3
14.1%
40.2
Py
15.7%
48.2
14.1%
39.4
with 20 examples.
EMI (Our)((Our))
15.7%
$7.7
14.1%
40.6
Method
ImageNet100 MME-RW OK-VQA Average
65.2
31.6
15.6
37.5
Analysis of
VPT
43.6
38.7
54.5
45.6
ЕРPТ
48.6
39.0
55.8
47.8
65
Full Fine-tuning
64.7
42.7
49.7
52.4
6000
LoRA 61.1
42.1
60.9
54.7
EMLoC 63.7
42.2
58.7
54.9
Context Length
4000
Accuracy
43G memory cost on ImageNet100, while our EMLoC re- quires only 18 minutes with 18G memory, showing clear advantages in efficiency.
80
3000
45-
Accuracy
Context Length
Comparison with fine-tuning methods. EMLoC is a training-free adaptation method that does not update any parameters the MLLM. With less adaptation time (144s), EMLoC is comparable to LoRA (234s) (Hu al., 2022) and full fine-tuning (820s). Our EMLoC also surpasses other PEFT methods, such 2 (Wang al., 2024c), VPT (Jia et al., 2022), and E (Han et al., 2023a). The training details of these PEFT methods and full fine-tuning are depicted in Appendix 1.
40
0.002
0.005
0.01
0.02
0.05
8
Figure 3. Performance and context length trends of EMLoC on ImageNet100 with 200 examples across different å values
Appendix C.I.
4.3. Ablation Studies
Results of different 8. Figure 3 presents the effect of vary- ing the Sensen-Shannor divergence threshold. A higher threshold results in greater compression but also leads to increased information loss. As the threshold increases, both accuracy and context length decrease. It demonstrates that 8 can effectively control the global upper bound A. The default threshold of 0.005 is selected balance between compression efficiency and performance preservation.
Comparison with other pruning strategies.
The layer-adaptive pruning strategy employs IS divergence analysis to dynamically evaluate token importance across layers. Unlike fixed pruning approaches, EMLoC maintains superior performance through selective token preservation. As evidenced in Table 5, at 22.4% compression Ima- geNet100, EMLoC sustains 63.7 while competitors suffer sigmificant degradation. This demonstrates our method's ef- fectiveness identifying and retaining critical tokens. The performance boundary emerges at extreme compression (15.7%, 6=0.02), where EMLoC's accuracy drops to 57.7, indicating essential token removal, while conventional meth- ods show paradoxical stability (47.3-48.2), This suggests competing approaches fail properly distinguish crucial tokens even at moderate compression (22.4%), as their per- formance collapses before reaching critical pruning thresh- olds. more comprehensive comparison with additional pruning strategies across various metrics can be found in
Results in various chunk lengths. Table 6 cxamines chunk length impacts on computational efficiency and compres- sion effectiveness. Longer chunks linearly increase mem- ory demands and compression ratios while maintaining sta- ble adaptation times (150s) and accuracy (62.4-63.7). Our chunk- wise compression enables memory-efficient process- ing without performance degradation notably supporting consumer GPUs (NVIDIA 4090) at 0.8k chunks with merely 1.3 accuracy drop. The default 1.6k configuration balances memory usage (38G) and compression effectiveness (22.4 ratio), while extreme 3.2k lengths cause OOM errors.

---

Efficient Multi-modal Long Context Learning for Training-free Adaptation
Table Results in various chunk lengths and chunk numbers
Table 8. Abalation studies observation window. indicates pre-
ImageNer100 with 200 demonstrations.
serving all answer tokens in the memory.
Chunk Length
Chunk Number
Adaptation Time
Peak Memory
Observation
Context Length
Acc
Ratio
Ratio
Acc
Window
0.8k
20
150s
62.4
24G
20.4
Answer Question+ Answer
3643
22.4%
61.9
1.6k
10
144s
63.7
38G
22.4
5304
32.6%
58.3
2.4k
7
149s
63.4
44G
24.7
8796
54.0%
55.4
2.8k
6
142s
62.5
48G
26.3
Answer
3643
22.4%
63.7
3.2k
00M
ImageNet100
Remain Token Number
13300
10000-
PyranidVv
EMLoC
0:304
PyramidKV
FM
Table 7. Results under different retention ratios.
14390
Svergence
0225
Retention Ratios
Context Length
13999
00000
Ratio
Acc
0015
2000
00100
0.2, 0.5, 1.0 0.1, 0.2,
3643
22.4%
63.7
4330-
2000
38 0000
gFD C.
0.2, 0.5, 1.0
4952
30.4%
61.6
000
0.05, 0.1, 0.2, 0.5, 1.0
3421
21.0%
58.6
[a] Layer
(b) Layer
0.1, 0.2, 0.5, 0.8, 1.0
3728
22.9%
62.2
MME-RW
Remain Token Number
2000
EMLOC
0/00
EMLOC
8000
PynamidKy
JS Divergence
PyremidiV
2000
c- 0..006
3300
Results with different retention ratios. The retention ratio r is selected greedily from an ascending predefined reten- tion ratios set R. progressing to larger ratios until satisfying the JS divergence constraint reaching 1.0. Two distinct patterns emerge based on the minimum ratio setting: When significantly below optimal compression levels (e.g.. 0.05 vs 22.4% optimal), 5 becomes the dominant control factor. As shown in Table 6 and Figure 3, this configuration en- ables adaptive balancing excessive layer compression is offset by others retaining more tokens to meet JS constraints, maintaining comparable overall ratios (21.0% vs 22.4%) Near-optimal minimum ratios (e.g., 0.2) directly determine final compression outcomes (30.4%). Based on empirical observations, we recommend setting the minimum ratio to approximately half of the optimal compression ratio. If the minimum ratio is too small, the overall compression ratio may not be sufficiently reduced, while simultancously dis- carding both important and redundant tokens at some layers. The final retention ratios R are set [0.1, 0.2, 0.5, 1.0].
1000
0.00
80
26
20
15
10
26
70
15
[0]
Layer Index
(d) Layer Index
Figure 4. Remaining token number of EMLoC and PyramidKV in ImageNet100 with 200 demonstrations and MME-RY RW with 20 demonstrations. The corresponding JS divergence after pruning is also illustrated to demonstrate the advantage of EMLoC.
4.4. Visualization of Compression
Remaining token number across layers Previous stud- ies (Xiao al., 2024; Cai et al., 2024) suggest that car- lier layers in large language models (LLMs) are more im- portant than later ones, advocating for pyramid-shape pruning strategy. However, our layer-wise adaptive prun- ing approach challenges this assumption, arguing that layer importance should be determined dynamically based on task-specific demonstrations rather than fixed heuristic.
Abalation studies observation window. For each token in the multi-modal long context, we use the answer tokens as the observation window to compute its importance score. The answer tokens are the most critical elements the senee--uustioon-nsweee pair, as they encapsulate the key semantic information of the example. In Table 8, we com- pare difffernt observation windows. The results show that answer tokens serve as an effective observation window, while the question tokens and image tokens often include significant amount of irrelevant or noisy data. Furthermore, as highlighted in the last row of Table 8, retaining answer tokens yields better performance, further confirming that answer tokens are crucial in multi-modal contexts.
As shown in Figure 4, experiments on ImageNet100 and MME-RW reveal that layer importance does not strictly follow a pyramid structure. Instead, certain early layers are particularly critical, as indicated by the spikes in Fig- ure 4(a) and (c). Pruning tokens these key layers causes significant shifts in the model's output distribution. For in- stance, Figure 4(b) and (d), applying PyramidKV pruning to these layers leads to sharp increase in JS divergence, significantly degrading performance. The corresponding performance metrics are provided in Table 5. Our findings highlight that layers 4, 8, and 14 of Qwen2-Ve VL are crucial in both ImageNet100 and MME-RW, as they retain signifi- cantly more tokens than adjacent layers.
8

---

Efficient Multi-modal Long Context Learning for Training-free Adaptation
References
ImageNet100
26%
Image
mage Quession
Agarwal, R., Singh, A., Zhang, L. M., Bohnet, B., Chan, S., Anand, A., Abbas, Z., Nova, A., Co-Reyes, J. D., Chu, E., et al. Many-shot in-context learning, arXiv preprint arXiv:2404. 1018, 2024.
3029
34%
1230
1903
44%
G0RD
22%
Prunod Tokens
(b) Reserved Tokens
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. Flamingo: visual language model for few- shot learning. NeurIPS, 35:23716- 2022.
4%
Image
Quicioon
Queekin
509
37%
4118
091
Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J. Qwen-VL: A frontier large vision- language model with versatile abilities. arXiv preprint 2tti::2008.19666 2023.
(c) Pruned Tokens
(d) Reserved Tokene
Figure 5. Distrihution of pruned and reserved tokens.
Bertsch, A., Ivgi, M., Alon, U., Berant, J., Gormley, M. R., and Neubig. G. In-context learning with long- context models: An in-depth exploration. arXiv preprint MXXi::405.002000 2024.
Distribution of Pruned and Reserved Tokens. This study serves an initial exploration of multi-modal context com- pression. As shown in Figure 5(a) and (c), image tokens make up the majority of pruned tokens, suggesting that the visual modality contains more redundancy than the textual modality. Besides, Figure 5(b) and (d) indicate that fewer image tokens are retained compared to text tokens, demon- strating effective pruning of redundant visual information. Additionally, variations in tasks may lead to differences in their compression rates.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu,
J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners, 2020.
5. Conclusion
Cai, Z., Zhang, Y., Gao, B., Liu, Y., Liu, T., Lu, K., Xiong, W., Dong, Y., Chang, B., Hu, J., et al. Pyramidkv: Dy- namic kv cache compression based on pyramidal informa- tion funneling. arXiv preprint arXiv:2406.02000 2024.
This paper presents EMLoC, a training-free method com- bining chunk-wise compression with layer-wise adaptive pruming build compact, task-specific memory for down- stream tasks. Experiment results show EMLoC reduces inference overhead while preserving strong performance in multiple visionllagguage tasks, providing a scalable solution for efficient multi-modal long context learning.
Chen, J., Ye, I.., He, J., Wang, Z.-Y., Khashabi, D., and Yuille, A. Llavolta: Efficient multi-modal models via stage-wise visual context compression. arXiv preprint arXiv:2406.20092 2024a.
Acknowledgements
Chen, K., Zhang, Z., Zeng, W., Zhang, R., Zhu, F., and Zhao, R. Shikra: Unleashing multimodal LLM's referential dialogue magic. arXiv preprint 2xXiv:2306.15995 2023,
This work is supported in part by Grant No. 2023-JCIQ- LA-001-088, in part by the Natural Science Foundation of China under Grant No. U20B2052, 61936011, 62236006, in part by the Okawa Foundation Research Award, in part by the Ant Group Research Fund, and in part by the Kun- Asne& Center Excellence, Peking University.
Chen, L., Zhao, H., Liu, T., Bai, S., Lin, J., Zhou, C., and Chang, B. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision- language models. In European Conference on Computer Vision, pp. 19-35. Springer, 2024b.
Impact Statement
Chen, Z., Wang, W., Tian, H., Ye, S., Gao, Z., Cui, E., Tong, W., Hu, K., Luo, J., Ma, Z., et al. How far are we to GPT4V? closing the gap to commercial multi- modal models with spen-source suites. arXiv preprint arXiv:2404, /682/, 2024c.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.

---

Efficient Multi-modal Long Context Learning for Training-free Adaptation
Cheng, K., Sun, Q. Chu, Y., Xu, F., Li, Y., Zhang, J., and Wu, Z. Seeclick: Harnessing gui grounding for advanced visual gui agents, arXiv preprint 200.::3.0...0335, 2024.
Huang, S., Dong, L., Wang. W., Hao, Y., Singhal, S., Ma, S., Lv, T., Cui, I.., Mohammed, O. K., Patra, B., al. Language is not all you need: Aligning perception with language models. NeurIPS, 36, 2024b.
Deng, J., Dong, W., Socher, R., Li, L.-J,, Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Iece, 2009.
Jia, M., Tang, l.., Chen, B.-C., Cardie, C., Belongie, S., Hariharan, B., and Lim, S.-N. Visual prompt tuning. In European conference on computer vision, pp. 709-727. Springer, 2022.
Dong, X., Zhang, P., Zang. Y., Cao, Y., Wang, B., Ouyang, L., Zhang. S., Duan, H., Zhang, W., Li, Y., et al. Internim- xcomposer2-4khd:: A pioneering large vision-language model handling resolutions from 336 pixels to 4k hd. arXiv preprint wrXiv:2404.065122 2024.
Jiang, Y. Irvin, J. A., Wang, J. H., Chaudhry, M. A., Chen, H., and Ng, A. Y. Many-shot in-context learn- ing in multimodal foundation models. In ICML 2024 Workshop on In- In-Context Leariing, 2024. URL https: / openreview.... net. forum? (aarrrmmmmdjj2:KWWee
Ge, S., Zhang, Y., Liu, L., Zhang, M., Han, J., and Gao, J. Model tells you what to discard: Adaptive kv cache compression for llms. arXiv preprint arXiv:2310.01800, 2023.
Li, M., Gong. S., Feng, J., Xu, Y., Zhang, J., Wu, Z., and Kong, L. In-context learning with many demonstration examples, arXiv preprint arXiv:2302.049. 2023.
Han, C., Wang, Q., Cui, Y., Cao, Z., Wang. W., Qi, S., and Liu, D. E2vpt: An effective and efficient approach for visual prompt tuning. In 2023 IEEE/CVF Interna- tional Conference Computer Vision (ICCV), pp. 17445- 17456. IEEE Computer Society, 2023a.
Li, Y., Huang, Y., Yang, B., Venkitesh, B., Locatelli, A., Ye, H., Cai, T., Lewis. P., and Chen, D. Snapkv: Llm knows what you are looking for before generation. arXiv preprint arXiv:2404. 14469, 2024.
Liu, H., Li, C., Wu, Q. and Lee, Y. J. Visual instruction tuning. NeurlPS, 36, 2024.
Han, C., Wang, Q., Cui, Y., Wang. W., Huang, L., Qi, S., and Liu, Facing the elephant in the room: Vi- sual prompt tuning or full finetuning? arXiv preprint arXiv:2401.1 12902, 2024.
Lu, H., Liu, W., Zhang, B., Wang, B., Dong, K., Liu, B., Sun, J., Ren, T., Li, Z., Sun, Y., et al. DeepSeek-VLL VL: Towards real-world iision-languag understanding, arXiv preprint 24Xhv:2403.0533 2024.
Han, Z., Zhou, G., He, R., Wang. J., Xie, X., Wu, T., Yin, Y., Khan, S., Yao, L., Liu, T., et al. How well does gpt-4v (ision) adapt to distribution shifts? preliminary investigation. arXiv preprint arXiv:2312.07424, 2023b.
Lu, W., Luu, R. K., and Buchler, M. J. Fine-tuning large language models for domain adaptation: Exploration of training strategies, scaling, model merging and synergis- tic capabilities. npj Computational Materials, 1 11(1):84, 2025.
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E.,, Casas, D. d. L.. Hendricks, L. A., Welbl, J., Clark, A., et al. Training compute-optimal large language models. arXiv preprint arXiv:2203. 2022.
Marino, K., Rastegari, M., Farhadi, A., and Mottaghi, R. Ok-vqa: A visual question answering benchmark requir- ing external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pp. 3195-3204, 2019.
Hu. E. J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. LoRA: Low-rank adap- tation of large language models. In International Confer- ence on Learning Representations, 2022. URL https: //openreview net / wiw.mmmiiddnnneeeeeeeeee
Peng, Z., Wang. W., Dong, L., Hao, Y., Huang. S., Ma, S., and Wei, F. Kosmos-2: Grounding multimodal large language models the world. arXiv preprint arXiv:2306. 2023.
Hu, S., Tu, Y., Han, X., He, C., Cui, G., Long, X., Zheng, Z., Fang, Y., Huang, Y., Zhao, W., al. MiniCPM: Unveiling the potential of small language models with scalable train- ing strategies. arXiv preprint ar arXiv:2404.06330 06395,, 2024.
Shahgir, H. S., Sayeed, K. S., Bhattacharjee, A., Ahmad, W. U. Dong. Y., and Shahriyar, R. IllusionVQA: A challenging optical illusion dataset for vision language models. In First Conference on Language Modeling, 2024. URL https: http://pppeevveeeeeee net/ forum? id=7ysaJGs7xx
Huang, B., Mitra, C., Karlinsky, L., Arbelle, A., Darrell, T., and Herzig, R. Multimodal task vectors enable many-shot multimodal in-context learning. Advances in Neural In- formation Processing Systems, 77:22124-22155 2024a.
10

---

Efficient Multi-modal Long Context Learning for Training-fre Adaptation
Wang, D., Xuan, S., and Zhang, S. Locllm: Exploiting gen- eralizable human keypoint localization via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024a.
Zhang, Y.-E, Zhang, H., Tian, H., Fu, C., Zhang, S., Wu, J., Li, F., Wang. K., Wen, Q. Zhang, Z., al. Mme- realworld: Could your multimodal IIm challenge high- resolution real-world scenarios that are difficult for hu- mans? arXiv preprint ar arXiv:2408. 13257, 2024c
Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.1219/ 2024b.
Zhang. Z.., Sheng, Y., Zhou, T., Chen, T., Zheng, L.., Cai, R., Song, Z., Tian, Y., Ré, C., Barrett, C., et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36:34661-34710 2023.
Wang. T., Liu, Y., Liang, J., Zhao, J., Cui, Y., Mao, Y., Nie, S., Liu, J., Feng, F., Xu, Z., et al. M2pt: Multimodal prompt tuning for zero-shot instruction learning. In Pro- ceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 3723-3740, 2024c.
Zhou, L.,, Xu, C., and Corso. J. Towards automatic learning of procedures from web instructional videos. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. MiniGPT-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
Wang, W., Lv, Q., Yu, W., Hong. W., Qi, J., Wang, Y., Ji, J., Yang, Z., Zhao, L.., Song, X., et al. CogVLM: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03079 2023.
Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. MiniGPT-4: Enhancing vision-language understanding with advanced large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https: /ppnreviewwweeeeeeee net forum? id=1t2bg88f27.
Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Ef- ficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations, 2024. URL Wtt..httss://ppereeeeeeee /pwnnevie...eeeeeee net/ forum? sdt.:misi111111111111 s5517v555122
Xuan, S., Guo, Q.. Yang, M., and Zhang, S. Pink: Unveiling the power of referential comprehension for multi-modal 1lms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp- 13838-13848, June 2024.
Yang, D., Han, X., Gao, Y., Hu, Y., Zhang, S., and Zhao, H. Pyramidinfer: Pyramid kv cache compression for high- throughput IIm inference. In Findings of the Association for Computational Linguistics ACL 2024, pp. 3258-3270, 2024.
Young, A., Chen, B., Li, C., Huang, C., Zhang, G., Zhang, G., Li, H., Zhu, J., Chen, J., Chang, J. et al. Yi: Open Foundation Models by 01.AI. arXiv preprint MxXiv:2403.06552.. 2024.
Zhang, P., Dong. X., Zang, Y., Cao, Y., Qian, R., Chen, L., Guo, Q. Duan, H., Wang, B., Ouyang, L., et al. :.eemllm-coomposrr-2.5:: A versatile large vision lan- guage model supporting long-contextual input and output. arXiv preprint AXXv:2407.033200 2024a.
Zhang, X., Li, J.. Chu, W. Hai, J., Xu, R., Yang, Y., Guan, S., Xu, J., and Cui, P. On the dfist--iiitiittiion gener- alization of multimodal large language models. arXiv preprint arXiv:2402.00 06599, 2024b.

---

Efficient Multi-modal Long Context Learning for Training-fre Adaptation
A. Pseudo Code of Layer-wise Adaptive Pruning
Algorithm 1: Laier-wise Adaptive Pruning
Input: C E R concatenated context of k-th chunk, where $ = x T;
Dy R x demonstration examples of k-th chunk in a batch;
M KV eache of previous k chunks;
Output: M2: Compressed memory of chunks.
MR ExtractKV Cx);
Pori H +- MLLM(ME-1 @ MM, D&);
MIg -- M.::
for to do
8 +- {Ei index (11))
for r E R:
// R is retention ratio set in ascending order
do
My - My (Topk(B'.)) r 5));
// S is chunk length
Pieer MILM(ME e ME' H
L t (poris Pier);
l S then
break;
// 8: JS divergence threshold
M₁ MR-1 1 ( M.
B. Prompt Template
In 1m0geNet100, 200 multi-modal examples evenly divided into 10 chunks. Each image (224x224) is encoded into approximately 64 tokens (may vary slightly due to dynamic aspect ratios), and the corresponding question-answer pair adds around 20 tokens, resulting in about 80 tokens per example. Each chunk contains 20 examples (roughly 1.6k tokens). The system prompt appears only at the start of the first chunk. Below an example structure:
Start of 1st chunk < <limsstart>> system)n You are a helpful assistant. < imeend ## sample
<minstart/>> userln </visionssarrt/> <Imagel, jpg> <|vision.end What category does the image belong to? < im end|>
< </im_start/> assistant)n <class 1>. 

Start of 2nd chunk
## sample 21
<lim.start/> userin <lvision.start >> ZImage21.jpo <lvision.endl> What category <
does the image belong to? < 

/ assistant), <class 11>. <limendd|>
For other image benchmarks, each image is encoded into 256 tokens 4488 resolution). Each chunk has 4 examples, resulting in chunk size of 1.1k-1.6k tokens. For the YouCook2 video benchmark, each video with 8 frames encoded into 1024 tokens, with videos per chunk, yielding 4.7k chunk size. If sample lengths vary significantly, algorithm to progressively fill each chunk up to maximum size.
greedy
12

---

Efficient Multi-modal Long Context Learning for Training-free Adaptation
Table Comparison with other context compression methods on ImageNet100. EMI,o increase the chunk number from 10 to 20 and utilize a group-wise strategy to save adaptation memory and time.
Method
Retention Ratio
Adapt Time
Adapt Memory
Infer Time
Infer Memory
Acc
MLoC
100%
28s
62G
31m
19G
62.6
PyramidKV
22.4%
54s
34G
19m
17G
49.3
FastGen
36.0%
45s
38G
37m
21G
49.3
PyramidInfer
24.6%
41s
42G
21m
17G
55.6
EMLoC
22.4%
144s
38G
18m
17G
63.7
EMLoC*
27.6%
85s
24G
19m
17G
60.9
Table 10. Comparison with LongVA and ML on VideoMME w/o subtitles with 384 frames.
Context
LLM
LLM
Peak
Overall
Method
Length
FLOPs
Time
Memory
ACC
Lo
55.5k
1715.5T
22h
41G
51.8
MLoC
27.9k
554.8T
7h
38G
60.3
EMLoC
2.3k
272.0T
5h
24G
60.1
C. More Experiments
C.1. Comparison with other Pruning Strategies
In Section 4.3 and Table of the original paper, we compared our adaptive EMLoC with two static KV-cache algorithms. Table 9 extends this comparison (Table 5) by including PyramidInfer (Yang al., 2024) and FastGen (Ge al., 2023). Most KV-cache methods focus uni-modal text compression, but fail to maintain original performance with high compression ratio. EMLoC retains only 22.4% of tokens while achieving 63.7% accuracy, outperforming FastGen (49.3% accuracy with 36% tokens) and PyramidInfer (55.6% accuracy with 24.6% tokens). Unlike existing KV-cache methods, EMLoC effectively maintains the full-context performance while significantly reducing the context length, thus improving efficiency.
To optimize the trade-off between adaptation cost and inference performance, we explore increasing the chunk number from 10 20 and a group-wise strategy (every two layers share the same retention ratio). This variant, EMLoC** reduces adaptation time from 144s to 85s and memory from 38G to 24G, the cost of slight accuracy degradationn63.77 (ottrioon(6.7-6660.0 and higher retention ratio (22.4% + 27.6). This allows for flexible implementation in computation constrained scenarios. The adaptation cost is significantly smaller compared with its gains in inference efficiency.
C.2. Experiment Long-Vides Benchmark.
In LongVA, each frame consists of 144 tokens, whereas in Qwen2-VL, 144 tokens represent every two frames through temporal pooling. Compared to our baseline MLoC, EMLoC significantly reduces computational overhead while maintaining nearly the same accuracy. Specifically, EMLoC reduces the average context length from 27.9k to just 2.3k tokens, LLM FLOPs from 554.8T to 272.0T, inference time from 7 hours to 5 hours, and peak GPU memory from 38G to 24G, while preserving a consistent accuracy (60.1 vs. 60.3).
To achieve this efficiency, we set 8 0.04 and configure the retention ratio to: [0.02, 0.1, 0.5, 1.0]. Instead of optimizing the retention ratio for each layer individually (layer-wise), adopt group-wise strategy, where every 14 layers are treated as single group and share the same retention ratio. This allows for more stable and efficient selection process during online inference. Under an identical setup (384 frames at the same resolution), both MI.CC and EML.oC outperform LongVA while requiring significantly fewer computations. EMLoC also enables real-time long-video understanding on consumer-grade GPUs such as the NVIDIA 3090, making it a more practical solution for real-world applications.
C.3. Robustness of Hyper-parameters
Those parameters have clear meanings and are easy to adjust. For high compression ratio, we can set a smaller retention ratio and a higher JS threshold 8, and the optimal pruning strategy will be identified heuristically. Our method avoids manually adjusting numerous parameters like FastGen (Ge et al., 2023) or PyramidInfer (Yang et al., 2024). Our experiments also show that the default hyperparameters are stable across different tasks, as depicted Table 11 and Table 12.
13

---

Efficient Multi-modal Long Context Learning for Training-free Adaptation
Table 11. Robustness Threshold show performance under different á values across tasks
ImageNet100
MME-RW
OK-VQA
0.002
63.7
42.2
58.6
0.005
63.7
42.2
$8.7
0.02
57.7
41.0
57.0
Table 12. Robustness of retention ratios across different tasks. demonstrate the robustness.
We provide the performance under different retention ratios across tasks to
Retention Ratios
ImageNet100
MME-RW
OK-VQA
[0.05, 0.1, 0.2, 0.5, 1] [0.1, 0.2, 0.5, [0.2. 0.5, 1]
58.6
41.6
58.3
63.7
42.2
58.7
61.6
41.7
58.6
C.4. Impact of task disparity on EMLoC
The impact measurement follows the method in (Han et al., 2024), with some modifications. Given that Qwen2V((MLLM) is pretrained on millions of image-text pairs covering lots of vision-language tasks, we use zero-shot accuracy as an indicator of the task disparity between downstream datasets and the pretraining datasets.
MLLMs perform well on OK-VQA (5,.1%), suggesting that the data and task of OK- OK-VQA are highly similar to those seen during pretraining. Meanwhile, ImageNet100 achieves 28% accuracy, indicating moderate similarity. In contrast, MedXper[QA only reaches 21.5% accuracy near random chance in five-choice QA-indicatigg significant dissimilarity.
Based on Table 13 and Table 14, the impact of task disparity can be summarized as follows:
Low task disparity (OK-VQA, ImageNer100); When task disparity low(highly moderately similar pretrained data), our EMLoC adapts well to the downstream tasks. It outperforms full fine-tuning OK-VQA when data is limited, and achieves average accuracy 48.2%, which is comparable to LoRA's 47.8%.
High task disparity & scarce data (MedXpertQA): All methods struggle. Adapting truly novel tasks typically demands extensive continued pretraining or finetuning (Lu al., 2025).
Larger downstream datasets(Imagee (mageeel1 with 200 examples): Full fine-tuning slightly outperforms both LoRA and EMLoC, echoing (Han et al., 2024)'s 's finding that "full fine-tuning gradually closes the performance gap as dataset size grows." "
Other tasks with low disparity(see Table 1): EMLoC also facilitates on tasks with small task disparity, such as MME-RV (OCR, remote sensing, driving), Illusion VQA (optical illusions), and YouCook2 (video captioning/activity recognition).
Table 13. Task disparity between downstream datasets and pretrained datasets.
Zero-Shot Accuracy
Similarity to Pretrained Dataset
Task
Dataset
Task
Disparity
OK-VQA
52.1
common-sense QA
Highly similar
Low
ImageNet100
28.0
image classification
Moderately similar
Low
MedXperQAA
21.5 (near random)
medical QA (medical image)
Dissimilar
High
D. Analysis of A
Our theoretical analysis establishes linear relationship between the global JS distance A and the chunk count K under worst-case assumptions (Equation (16)). However, empirical observations reveal more nuanced behavior. As shown in Figure 6(a), exhibits a strong positive correlation with the local constraint 8, aligning with our theoretical prediction that V x Vo. This confirms that effectively governs the global information loss, enabling practitioners to reliably control compression quality through this single parameter.
14

---

Efficient Multi-modal Long Context Learning for Training-fre Adaptation
Table 14. Impact of task disparity different adaptation methods.
Method
OK-VQA
ImageNet100
MedXperIQA
Average
Baseline (Qwen2-VL)
52.1
28.0
21.5
33.9
LoRA
60.9
61.1
21.5
47.8
Full Fine-tuning
49.7
64.7
22.0
45.5
EMLoC
58.7
63.7
22.2
48.2
Relation between 8
Relation between and K with 8=0.005
0.045
0.0100
0.040
0.0085
0.035
JS Divergence
0.030
Divergence
0.0090
0.025
0.0085
0.0080
0.015
0.010
0.0075
0.005
0.0070
0.000
0.002
0.005
0.01
0.02
0.05
6
(a)
(b) K
Figure 6. Trend of A on various and
Notably, Figure 6(b) demonstrates that A remains stable within a narrow range across varying K values when d is fixed. This diverges from the theoretical upper bound's linear dependence on K, suggesting our worst-case analysis accommodates challenging scenarios where chunk dependencies might compound errors. In practice, however, the weak inter-chunk dependencies in real-world datasets prevent accumulation across compression steps. Consequently, the effective upper bound simplifies to A S 76, where is small constant (typically 7 S 2 in our experiments), rather than scaling with K.
This phenomenon has important practical implications:
Hyperparameter tuning focuses primarily 8, substantially reducing configuration complexity
Users can freely increase to minimize memory usage without compromising information integrity
Our method thus achieves optimal balance between theoretical rigor and practical usability while the theoretical bound guarantees robustness, the empirical independence between chunks enables memory-efficient compression through large K values. This dual advantage makes our approach particularly suitable for long-context applications where GPU memory constraints are critical.
E. More Implementation Details
E.1. Training Details
In Table 4, compare EMLoC with some fine-tuning methods, such LoRA (Hu et al., 2022), fine-tuning, M (Wang et al., 2024c), VPT (Jia et al., 2022), and E*PT (Han et al., 2023a).
In LoRA adaptation, we apply LoRA adapters to all linear modules of the LLM, including qkv proj, out proj. up proj, and down down_proj. while keeping the vision encoder and multi-modal projector frozen. The rank and alpha are set to 16 and 32, respectively. full fine-tuning, only the LLM is fine-tuned with DeepSpeed ZeRO-3, leaving other parameters frozen. Other unspecified settings follow the default configurations in L.aMoFactory. The detailed hyperparameters are reported in
15

---

Efficient Multi-modal Long Context Learning for Training-free Adaptation
Table 15 and Table 16.
Table 15. Hyperparameters LoRA training
Hyperparameter
Value
Optimizer learning rate batch size
AdamW
3e-5
warmup ratio
0.1
epochs
clip norm
1.0
Table 16. Hyperparameters for full fine-tuning
Hyperparameter
Value
Optimizer learning rate batch size
AdamW
le-5
warmup ratio epochs
0.1
clip norm
1.0
For other PEFT methods, following the default configuration in MPPT (Wang et al., 2024c), the number of visual prompts and textual prompts is 20 and 10, respectively. The learning rate is set to 7e-4. The number of KV prompt token is 5, For Imagenet100, we optimize epochs with 125 steps. For MME-RW and OK-VQA, we just fine-tune 25 steps.
E.2. Dataset Details
We evaluate our method on ImageNet100, subset of ImageNet- (Deng et al., 2009) with the first 100 classes for inference efficiency. Demonstration examples are uniformly sampled from the training set, ensuring even distribution per class. For instance, in the 200-example setting, each class contributes two examples. Evaluation is conducted on the full validation set with 5000 images. Additionally, we evaluate on several other benchmarks: ScreenSpoo(Ceeng et al., 2024) for GUI grounding across diverse platforms, MZE-nW(Zhang et al., 2024c) for real-world tasks such as OCR, remote sensing, and autonomous driving, I et al., 2024) for evaluating optical-illusion understanding. OK- VX-rrionar et al., 2019) for open-ended question answering using external knowledge, and YouCook2 (Zhou et al., 2018) for cooking-video-relan tasks, e.g., captioning activity recognition.
16


Reviews:
[{"review_id": "jtHccHTjBU", "reviewer": "Reviewer_Pa96", "summary": "This paper introduces Efficient Multi-Modal Long Context Learning (EMLoC), a training-free approach that embeds many demonstration examples into large multi-modal inputs, then uses chunk-wise compression and layer-wise adaptive pruning to reduce the resulting key-value cache. By enforcing a Jensen–Shannon divergence threshold at each layer, EMLoC selectively retains important tokens without re-training the underlying model. Experiments on multiple vision-language tasks show that EMLoC preserves or improves performance. Experiments on ImageNet even show the training-free adapted model is comparable to the fine-tuned models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** Comparison with other multi-modal in-context learning methods in Table 1.\n\n**Table R1.1: Comparison with multi-modal in-context learning methods**\n\n| Method | ImageNet100 | MME-RW   | OK-VQA   |\n| ------ | ----------- | -------- | -------- |\n| MTV    | 32.7        | 27.8     | -        |\n| RICES  | **64.5**    | 40.5     | 58.5     |\n| EMLoC  | 63.7        | **42.2** | **58.7** |\n\n**A1:** Thanks for the suggestion! We have compared EMLoC with two other open-sourced multi-modal in-context learning methods, RICES[1] and MTV[2], on ImageNet100, MME-RW, and OK-VQA in Table R1.1. RICES retrieves the top 1/4 most relevant in-context samples from all samples. MTV extracts the mean activation of in-context examples as task vectors and finds the optimal replacement position of these task vectors. During inference, MTV replaces these task vectors at the optimal position of the test sample, which fails to facilitate these tasks. Our **EMLoC achieves better average performance** across the three benchmarks. It's worth noting that RICES is an online retrieval-augmented method, so it needs to forward the retrieved long context during each inference step. **RICES takes 5 hours inference time and 43G memory cost on ImageNet100, while our EMLoC requires only 18 minutes with 17G memory**, showing clear advantages in efficiency.\n\n[1] Flamingo: a visual language model for few-shot learning (NeurIPS 22)\n\n[2] Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning (NeurIPS 24)\n\n\n\n**Q2:** Comparison with LongVA and experiment on long-video benchmarks.\n\n**Table R1.2: Comparison with LongVA and MLoC on VideoMME w/o subtitles with 384 frames**\n\n| Method | Context Length | LLM FLOPs  | LLM Time | Peak Memory | Overall ACC |\n| ------ | -------------- | ---------- | -------- | ----------- | ----------- |\n| LongVA | 55.5k          | 1715.5T    | 22h      | 41G         | 51.8        |\n| MLoC   | 27.9k          | 554.8T     | 7h       | 38G         | **60.3**    |\n| EMLoC  | 2.3k           | **272.0T** | **5h**   | **24G**     | 60.1        |\n\n**A2:** Thank you for the insightful suggestion. As required by the reviewer, we have conducted experiments on the long-video benchmark VideoMME without subtitles, using 384 frames per video. In LongVA, each frame consists of 144 tokens, whereas in Qwen2-VL, 144 tokens represent every two frames through temporal pooling. Compared to our baseline MLoC, EMLoC significantly **reduces computational overhead while maintaining nearly the same accuracy**. Specifically, EMLoC reduces the average context length from 27.9k to just 2.3k tokens, LLM FLOPs from 554.8T to 272.0T, inference time from 7 hours to 5 hours, and peak GPU memory from 38G to 24G, while preserving a consistent accuracy (60.1 vs. 60.3).\n\nTo achieve this efficiency, we set $\\delta$ =0.04 and configured the retention ratio to \\[0.02, 0.1, 0.5, 1.0]. Instead of optimizing the retention ratio for each layer individually (layer-wise), we adopt a **group-wise strategy**, where every 14 layers are treated as a single group and share the same retention ratio. This allows for a more stable and efficient selection process during online inference. Under identical setup(384 frames at the same resolution), both MLoC and EMLoC outperform LongVA while requiring significantly fewer computations. EMLoC also enables the real-time long-video understanding on consumer-grade GPUs such as the NVIDIA 3090, making it a more practical solution for real-world applications.\n\n\n\n**Q3:** Expand evaluation to more diverse multi-modal benchmarks.\n\n**Table R1.3: Comparison with fine-tuning methods on more multi-modal benchmarks**\n\n| Method           | ImageNet100 | MME-RW | OK-VQA | Average  |\n| ---------------- | ----------- | ------ | ------ | -------- |\n| LoRA             | 61.1        | 42.1   | 60.9   | 54.7     |\n| Full Fine-tuning | 64.7        | 42.7   | 49.7   | 52.4     |\n| EMLoC            | 63.7        | 42.2   | 58.7   | **54.9** |\n\n**A3:** We appreciate the reviewer’s suggestion for further benchmarking. To this end, we have compared EMLoC with LoRA and full fine-tuning on additional multi-modal benchmarks(MME-RW and OK-VQA) as seen in Table R1.3. Our EMLoC shows comparable performance to LoRA and full fine-tuning in a training-free manner. This flexibility is essential in real-world applications, where fine-tuning may not always be feasible. The optimization steps and hyperparameters are the same as depicted in the Appendix C.1.", "claims_and_evidence": "1. EMLoC outperforms existing long-context models in efficiency and effectiveness. Not compared against other multi-modal in-context learning methods in Table 1.\n\n2. EMLoC generalizes well to different multi-modal tasks. No evaluation on long video understanding or effective retrieval within video context. Test on long video benchmarks to confirm seamless adaptation.\n\n3. EMLoC significantly reduces computational overhead while maintaining high performance. Supported by Line 325-328 experiments (FLOP and inference time reduction).\n\n4. EMLoC outperforms LoRA and achieves performance comparable to full fine-tuning. Only tested on ImageNet100, making the claim too strong. Expand evaluation to more diverse multi-modal benchmarks.", "methods_and_evaluation_criteria": "1. Relevance of chunk-wise compression and pruning: The methodology directly addresses the challenge of handling very long multi-modal inputs within limited computational resources. This aligns well with the stated goal of “efficient multi-modal long-context learning.”\n\n2. Choice of benchmarks: While the selected benchmarks do test multi-modal capabilities, they do not fully cover long video tasks or extensive multi-image retrieval scenarios. This partially demonstrates EMLoC’s potential but leaves out broader, real-world applications requiring extended temporal context.", "theoretical_claims": "The mathematical proofs in Sec. 3.3 appear generally sound.", "experimental_designs_or_analyses": "1. EMLoC is compared to fully fine-tuned and LoRA-based tuned models. However, it is only tested on ImageNet100. Please expand the experiments and evaluation to make your claim stronger.\n\n2. In terms of training-free long-context multi-modal learning, please compare against LongVA and experiment on some long-video benchmarks. \n\nZhang, Peiyuan, et al. \"Long context transfer from language to vision.\" arXiv preprint arXiv:2406.16852 (2024).", "relation_to_broader_scientific_literature": "1. Training-free adaptation resonates with in-context learning trends. Whereas concurrent works fine-tune or add adapters for new tasks, EMLoC echoes the literature advocating prompt-only or retrieval-style adaptation, with the unique twist of compressing multi-modal exemplars directly in the model’s KV cache.\n\n2. Bridging multi-modal ICL. Flamingo and other models have shown the possibility of in-context learning for multi-modal tasks but do not specifically address extremely long input contexts. EMLoC advances this line of work by integrating compression/pruning for more scalable “many-shot” demonstration examples.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Please see my suggestions in the above sections related to methods and claims, and experimental designs.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B7fRm8jBLU", "reviewer": "Reviewer_6UND", "summary": "Following the improvements brought by in-context examples in multi-modal LLMs, the context length compression has becomed a hot topic to make the technique more scalable. \nThis paper tackles the challenge by introducing layer-wise adaptive pruning, it also provides theoretical justification by doing this layer by layer through Jenseon-Shannon divergence constraint. The proposed method - EMLoC shows onpar or better performance against naive long-context approaches on vairous vision-language benchmarks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** Each chunk contains several examples as shown in experiment details. How does the author ensure each example has the same length? \n\n**A1:** Thank you for your comment. In ImageNet100, 200 multi-modal examples are evenly divided into 10 chunks. Each image (224×224) is encoded into approximately 64 tokens (may vary slightly due to dynamic aspect ratios), and the corresponding question-answer pair adds around 20 tokens, resulting in about 80 tokens per example. Each chunk contains 20 examples (roughly 1.6k tokens). The system prompt appears only at the start of the first chunk. Below is an example structure:\n\n```python\n# Start of 1st chunk\n
 system\\n You are a helpful assistant.
\n## sample 1\n
 user\\n 
 <Image1.jpg> 
 What category does the image belong to? 
\n
 assistant\\n <class 1>. 
   ...\n# Start of 2nd chunk\n## sample 21\n
 user\\n 
 <Image21.jpg> 
 What category does the image belong to? 
\n
 assistant\\n <class 11>. 
 ...\n```\n\nFor other image benchmarks, each image is encoded into 256 tokens(448×448 resolution). Each chunk has 4 examples, resulting in a chunk size of 1.1k–1.6k. For the YouCook2 video benchmark , each video with 8 frames is encoded into 1024 tokens, with 4 videos per chunk, yielding a 4.7k chunk size. If sample lengths vary significantly, we use a greedy algorithm to progressively fill each chunk up to a maximum size.\n\n**Q2:** The key contribution is the context compression technique while the author didn't discuss its relation/advantage over KV-cache algorithms such as PyramidInfer and FastGen. How about the performance/efficiency improvements with other KV cache methods?\n\n**Table R2: Comparison with other context compression methods on ImageNet100**\n\n| Method       | Retention Ratio | Adapt Time | Adapt Memory | Infer Time | Infer Memory | Acc      |\n| ------------ | --------------- | ---------- | ------------ | ---------- | ------------ | -------- |\n| MLoC         | 100%            | 28s        | 62G          | 31m        | 19G          | 62.6     |\n| PyramidKV    | 22.4%           | 54s        | 34G          | 19m        | 17G          | 49.3     |\n| FastGen      | 36.0%           | 45s        | 38G          | 37m        | 21G          | 49.3     |\n| PyramidInfer | 24.6%           | 41s        | 42G          | 21m        | 17G          | 55.6     |\n| **EMLoC**    | **22.4%**       | 144s       | 38G          | **18m**    | **17G**      | **63.7** |\n| **EMLoC***   | 27.6%           | 85s        | **24G**      | 19m        | **17G**      | 60.9     |\n\n**A2:** Thanks for the comments. In Section 4.3 and Table 4 of the original paper, we compared our adaptive EMLoC with two static KV-cache algorithms. Table R2 extends this comparison (Table 4) by including PyramidInfer and FastGen. Most KV-cache methods focus on uni-modal text compression, but fail to maintain original performance with a high compression ratio. **EMLoC retains only 22.4% of tokens while achieving 63.7% accuracy**, outperforming FastGen (49.3% accuracy with 36% tokens) and PyramidInfer (55.6% accuracy with 24.6% tokens). Unlike existing KV-cache methods, EMLoC effectively maintains the full-context performance while significantly reducing the context length, thus improving efficiency.\n\nTo optimize the trade-off between adaptation cost and inference performance, we explore **increasing the chunk count (10 → 20)** and a **group-wise strategy** (every two layers share the same retention ratio). This variant, **EMLoC\\***, reduces adaptation time from 144s to 85s and memory from 38G to 24G, at the cost of a slight accuracy degradation(63.7 → 60.9) and a higher retention ratio (22.4% → 27.6%). This allows for a flexibility implementation on computation constrained scenarios. The adapation cost is significantly smaller compared with its gains in inference efficiency. More discussion can be seen in the response to Q2 from Reviewer Niy5.\n\n**Q3:** Can this method be applied alongside model parallelization?\n\n**A3:** Yes, the proposed method is compatible with model parallelization. EMLoC only compresses the key-value cache of the context at each layer, and it does not alter the model weights or architecture. Modern model parallel frameworks, using pipeline or tensor parallelism, can distribute the KV cache across devices, allowing for parallelization techniques during EMLoC's adaptation or inference. Varying KV-cache lengths across layers may cause the computational or communication imbalance. To mitigate this, we can manually split the model based on the pruned KV-cache lengths, or automatically share free GPU resources with other models using Multi-Process Service (MPS)[1] or Transparent GPU Sharing (TGS)[2]. \n\n[1] NVIDIA MPS: https://docs.nvidia.com/deploy/mps/index.html\n[2] Transparent GPU sharing in container clouds for deep learning workloads. (NSDI 23)", "claims_and_evidence": "1. I think one of the underlying assumption of chunk-wise compression is each chunk contains several examples as shown in experiment details. How does the author makes sure each example has the same length?", "methods_and_evaluation_criteria": "I am not very familiar with vision-language benchmarks but the experiment setups makes sense to me.", "theoretical_claims": "I scan the JS constraint proof and finds it is correct.", "experimental_designs_or_analyses": "I have examined the experimental set up for various number of examples used in vision-language benchmarks.", "relation_to_broader_scientific_literature": "It is related to KV cache optimization in the LLM research.", "essential_references_not_discussed": "The key contribution is context compression technique while the author didn't discuss its relation/advantage over KV-cache algorithms such as PyramidInfer[1] and FastGen[2]\n\n[1] PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference\n[2] Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs", "other_strengths_and_weaknesses": "The method seems cannot be applied together with model parallelization techniques because of the way layer pruning is implemented.", "other_comments_or_suggestions": "N/A.", "questions_for_authors": "1. Can you provide examples, and chunks examples to help me understand how chunk-wise segmentation is possible in vision-language tasks? \n\n2. How about the performance/efficiency improvements with other KV cache techniques proposed for transformers?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NvT4oOxAnA", "reviewer": "Reviewer_Niy5", "summary": "This paper introduces EMLoC(Efficient Multimodal Long Context Learning), a training-free method to embed examples directly into the model input. It is implemented via layer-wise adaptive pruning. The authors first separate the context into chunks to prune tokens by importance measured with Jenson-Shannon divergence. The authors also show that their strategy can retain information within a certain upper bound of information loss. Experiments on diverse dataset and ablations are executed showing efficiency of their work.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** There are too many hyperparameters (retention ratio, JS threshold) that need to be found heuristically.\n\n**A1:** Thanks for the comments. Those parameters have clear meanings and are easy to adjust. For a high compression ratio, we can set a smaller retention ratio and a higher JS threshold $\\delta$, and the optimal pruning strategy will be identified heuristically. Our method avoids manually adjusting numerous parameters like FastGen or PyramidInfer. Our experiments also show that the default hyperparameters are stable across different tasks, as seen in following two tables.\n\n**Table R3.1: $\\delta$ across tasks**\n\n| $\\delta$  | ImageNet100 | MME-RW   | OK-VQA   |\n| --------- | ----------- | -------- | -------- |\n| 0.002     | 63.7        | 42.2     | 58.6     |\n| **0.005** | **63.7**    | **42.2** | **58.7** |\n| 0.02      | 57.7        | 41.0     | 57.0     |\n\n**Table R3.2: retention ratios across tasks**\n\n| Retention Ratios         | ImageNet100 | MME-RW   | OK-VQA   |\n| ------------------------ | ----------- | -------- | -------- |\n| [0.05, 0.1, 0.2, 0.5, 1] | 58.6        | 41.6     | 58.3     |\n| **[0.1, 0.2, 0.5, 1]**   | **63.7**    | **42.2** | **58.7** |\n| [0.2, 0.5, 1]            | 61.6        | 41.7     | 58.6     |\n\n**Q2:** The iterative algorithm can save memory usage, but I do not think this is computationally efficient.\n\n**A2:** While the iterative algorithm in EMLoC introduces a modest adaptation time(~144 s), this cost is **significantly smaller compared with its gains in inference efficiency and performance**. As depicted in L326-328 and Table R2, on ImageNet100, **EMLoC reduces inference time from 31 minutes of MLoC to 18 minutes**, and memory usage from 19G to 17G without accuracy degradation. Compared to the in-context learning method RICES in Table R1.1, EMLoC reduces inference time from 5 hours to 18 minutes and memory from 43G to 17G. Besides, PyramidKV sacrifice 16.1% accuracy to achieve faster adaptation(55s vs 144s). \n\nTo further address the computational concerns, EMLoC offers flexibility:\n\n1. **Group-wise pruning**: By grouping adjacent layers (e.g., 2 layers per group), adaptation time drops to **85 seconds** (a 40% reduction), with a slight accuracy drop.\n2. **Decoupled adaptation/inference**: Adaptation needs a **one-time cost** per task, while the pruned KV cache may be repetitively used for **thousands of times** during inference.\n\nMeanwhile, as shown in Table R1.2, EMLoC is also an efficient method in online long-video understanding, reducing the total LLM time from 7 hours to 5 hours, and peak GPU memory from 38G to 24G.\n\n**Q3:** Why compare EMLoC with LoRA and full fine-tuning in Table 3. The comparison with LoRA and full fine-tuning for adaptation time seems less convincing. Comparison with other context compression methods and MLoC for time and memory consumption.\n\n**A3:** Results in Table 3 and Table R1.3 demonstrate that EMLoC is comparable to fine-tuning methods on multiple multi-modal tasks. Those comparison demonstrate that our training-free EMLoC can achieve comparable performance with fine-tuning which needs extra training iterations. Therefore, EMLoC is a promising adaptation method with better efficiency. \n\nOur EMLoC is an efficient adaptation method that makes long-context learning practical even on consumer GPUs, with minimal inference cost. We further compare the time and memory consumption of various compression methods in Table R2 (Reviewer 6UND), where our method shows clear advantages in both inference efficiency and accuracy. For additional analysis, please refer to A2 of Reviewer 6UND and A2 of Reviewer Niy5.\n\n**Q4:** How did you train the model parameters for the context compression? Is it just fine-tuning the model for ImageNet?\n\n**A4:** EMLoC is a training-free adaptation method that adaptively searches for the optimal pruning strategy under a JS divergence constraint, without fine-tuning the pretrained MLLM.  For each task, a distinct pruned KV cache is generated and loaded into memory during inference, **eliminating the need for model retraining or redeployment**.\n\n**Q5:** Figure 1 seems less intuitive. Rather than using demonstration examples on the X-axis, why not use context length?\n\n**A5:** Thanks for the suggestion! We have revised Figure 1 by replacing the X-axis with context length, making it more intuitive in illustrating our advantages in both performance and efficiency.\n\n**Q6:** The details of LoRA and full fine-tuning should be more elaborated in the Appendix than in the current manuscript.\n\n**A6:** In LoRA adaptation, we apply LoRA adapters to all linear modules of the LLM, including qkv_proj, out_proj, up_proj, and down_proj, while keeping the vision encoder and multi-modal projector frozen.  The rank and alpha are set to 16 and 32, respectively. In full fine-tuning, only the LLM is fine-tuned with  DeepSpeed ZeRO-3, leaving other parameters frozen. Other unspecified settings follow the default configurations in LLaMAFactory.", "claims_and_evidence": "This paper claims that the layer-wise adaptive pruning strategy under Jenson-Shannon divergence contributes to the chunk-wise compression mechanism, further improving the accuracy and efficiency in the long context problems in adapting multi-modal large language models. There are no visible problems in the claims.", "methods_and_evaluation_criteria": "Overall idea sounds. However, there are too many hyper-parameters (retention ratio, divergence distance threshold) that need to be found heuristically. In addition, such iterative algorithm can save memory usage, I do not think this computationally efficient. For the evaluation, I have no idea why they compare their method with LoRA and Full fine-tuning in Table 3.", "theoretical_claims": "They claim that their layerwise token pruning method satisfies the upper bound of information loss related to the divergence distance threshold and the number of chunks.", "experimental_designs_or_analyses": "How did you train the model parameters for the context compression? Is it just fine-tuning the model for the imagenet classification? The comparison with LoRA and Full-fine-tuning for the adaptation time seems less unconvincing. If you want to claim your method is efficient at least it should be compare with other context compression methods or vanilla MLoC for the time consumption and memory consumption.", "relation_to_broader_scientific_literature": "This paper contributes to chunk-wise compression of long contexts in multimodal large language models. This method might helpful for the efficient long context generation methods (however, I cannot see the fair explanation or the experiment for the efficiency).", "essential_references_not_discussed": "All essential related works are well cited and discussed in this paper.", "other_strengths_and_weaknesses": "Refer to above sections.", "other_comments_or_suggestions": "Figure 1 seems less intuitive. Rather than using demonstration example in the X-axis why don't you use context length?", "questions_for_authors": "Refer to above sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VjflifCKX8", "reviewer": "Reviewer_iyMg", "summary": "While current popular adaptation for MLLMs heavily replies on fine-tuning, the paper proposes a novel training-free alternative that can embed demonstration examples directly into the model input. Due to the lengthy inputs might bring computational and memory overhead, the proposed method contributes a chunk-wise compression with layer-wise adaptive pruning. \n\nThe proposed method is able to reach a dramatic reduction in inference complexity while retaining the performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** Disparity between the pre-trained MLLM dataset and adaptation tasks. In [1], the paper demonstrates that task disparity might have a huge impact on vision task adaptation; would this be the case for in-context learning of MLLM?\n\n**A1:** We appreciate the reviewer’s insightful question on task disparity in MLLM adaptation. The great work [1] highlights the impact of task disparity on vision adaptation, which inspires us to explore influence of task disparity on in-context learning of MLLMs. To evaluate this, we conducted experiments on ImageNet100, OK-VQA, and MedXpertQA.\n\n**Table R4.1: Impact of task disparity on EMLoC**\n\n| Method             | ImageNet100 | OK-VQA | MedXpertQA |\n| :----------------- | :---------- | :----- | :--------- |\n| Baseline(Qwen2-VL) | 43.2        | 52.1   | 21.5       |\n| LoRA               | 61.1        | 60.9   | 21.5       |\n| Full Fine-tuning   | 64.7        | 49.7   | 22.0       |\n| EMLoC              | 63.7        | 58.7   | 22.2       |\n\nFor ImageNet100 and OK-VQA, where the MLLM has seen similar data and tasks during pre-training, EMLoC effectively adapts with limited data (200 and 20 examples, respectively). Full fine-tuning struggles on OK-VQA, probably due to the overfitting with only 20 examples. In contrast, all methods fail on MedXpertQA, a complex multi-choice medical dataset, where the baseline model shows poor performance, indicating no prior knowledge in this domain. With only 20 examples, adaptation remains ineffective. These results suggest that EMLoC could  efficiently adapt pre-trained knowledge with minimal data, but struggles when entirely new capabilities are required. This finding aligns with the conclusion in [1]. When there is a significant discrepancy between the pre-trained and downstream tasks, adaptation can be considerably more difficult. \n\n**Q2:** More PEFT fine-tuning methods should be included for completeness.\n\n**A2:** We appreciate the suggestion. In response, we compare **EMLoC** with **VPT** [1], **E²PT** [2], and **M²PT** [5] across three multi-modal tasks.\n\nOn ImageNet100 with 200 training examples, M²PT achieves the best performance due to its strong adaptation capacity, leveraging three different adapters(textual/visual prompts and multi-modal projector). However, when the number of training samples is reduced to 20 (as in MME-RW and OK-VQA), M²PT tends to overfit due to its increased number of optimized parameters. VPT, which optimizes only the visual prompts of the visual encoder, has fewer trainable parameters and a lower optimization capacity, leading to a weaker performance across all three tasks. However, its limited parameter tuning helps to mitigate overfitting risks to some extent. With visual prompt tokens and the powerful shared KV prompt tokens,  E$^2$PT achieves better fine-tuning performance than VPT. Following M²PT, the number of visual prompts and textual prompts are 20 and 10, respectively. The learning rate is set to 7e-4. The number of KV prompt token is 5. For Imagenet100, we optimize 5 epochs with 125 steps. For MME-RW and OK-VQA, we just fine-tune 25 steps.\n\n**As a training-free method, EMLoC achieves competitive performance across various tasks** and lowers the risk of overfitting. This makes it effective in low-data regimes, where traditional fine-tuning methods may struggle. These methods will be compared in our revised manuscript. We will also **release the code** of these PEFT methods in **LLaMAFactory** for the research community.\n\n**Table R4.2: Comparison with other PEFT methods on multi-modal benchmarks**\n\n| Method | ImageNet100 | MME-RW   | OK-VQA   |\n| ------ | ----------- | -------- | -------- |\n| VPT    | 43.6        | 38.7     | 54.5     |\n| E²PT   | 48.6        | 39.0     | 55.8     |\n| M²PT   | **65.2**    | 31.6     | 15.6     |\n| EMLoC  | 63.7        | **42.2** | **58.7** |\n\n**Q3:** The advantages of our EMLoC.\n\n**A3:** EMLoC is a training-free adaptation method designed for multi-modal long-context learning, effectively leveraging the strong capabilities of pre-trained MLLMs and offering a reduced risk of overfitting. This aligns with the growing trend in the test-time scaling era. \n\nTo the best of our knowledge, EMLoC is the first multi-modal KV-cache pruning method to achieve performance comparable to that of fine-tuning methods, while maintaining a high compression ratio. As shown in Table R2, EMLoC surpass the second best method PyramidInfer by 5.3% in accuracy with a less retention ratio. **In addition, EMLoC can also facilitate the online long-video understanding**. In Table R1.2, EMLoC reduces context length from 27.9k to just 2.3k, LLM FLOPs from 554.8T to 272.0T, and inference time from 7 hours to 5 hours, while preserving a consistent accuracy (60.1 vs. 60.3).\n\nOur adaptive pruning strategy provides valuable insights into multi-modal pruning and the importance of different layers in MLLMs in Fig.4 and Fig.5, which may inspire further exploration in multi-modal pruning techniques.", "claims_and_evidence": "The angle of this paper is interesting, while I have several questions:\n\n1. In Sec. 3.1, in order to adapt a pre-trained MLLM without any training or parameter fine-tuning, the proposed approach construct a long context by concatenating task-specific demonstration examples. This raise my first question on the disparity between the pre-trained MLLM dataset and adaptation tasks. In [1], the paper demonstrates that task disparity might be a huge impact on vision task adaptation, would this be the case for in-context learning of MLLM? The authors need to specifically discuss the datasets and possible disparities. \n\n[1] Facing the Elephant in the Room: Visual Prompt Tuning or Full Finetuning? ICLR 2024.", "methods_and_evaluation_criteria": "The main table needs further improvements. The current version only includes baselines such as MLoC with given number of examples. The proposed EMLoC, however, does not have a systematic comparison to fine-tuning methods (I have acknowledged that in Table 3 the authors included LoRA, and Full fine-tuning for comparison). However, more PEFT fine-tuning methods [1-5] should be included for completeness. \n\nRight now, I could not see the huge advantage that EMLoC can bring. This is very critical, as the authors claim that the proposed method is a good alternative to current fine-tuning approaches (including full fine-tuning and other fine-tuning approaches). \n\n[1] Visual Prompt Tuning. ECCV 2022.\n\n[2] E2VPT: An Effective and Efficient Approach for Visual Prompt Tuning. ICCV 2023.\n\n[3] Adaptformer: Adapting vision transformers for scalable visual recognition. NeurIPS 2022.\n\n[4] Learning expressive prompting with residuals for vision transformers. CVPR 2023.\n\n[5] M2PT: Multimodal Prompt Tuning for Zero-shot Instruction Learning. EMNLP 2024.", "theoretical_claims": "The theoretical claims of this paper is easy to follow, and I do not see any issues during review.", "experimental_designs_or_analyses": "The experimental designs are current insufficient, please see \"Methods And Evaluation Criteria\". More parameter-efficient fine-tuning approaches should be discussed, no simply having LoRA and full fine-tuning.", "relation_to_broader_scientific_literature": "The method proposed in this paper is interesting, instead of exhaustively digging from the area of finetuning, the method leverages the power of in-context learning for new task adaptation.", "essential_references_not_discussed": "Multimodal parameter finetuning papers, such as M2PT: Multimodal Prompt Tuning for Zero-shot Instruction Learning. EMNLP 2024 is not discussed. Other PEFT methods, though not common under the multimodal settings, should be covered/discussed as well.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "The paper is interesting, paving a promising way for leveraging the in-context learning for MLLM new task adaptation. However, there are two fundamental problems in current paper:\n\n1. See \"Claims And Evidence\";\n\n2. See \"Methods And Evaluation Criteria\".", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces EMLoC, a training-free method for efficient long-context multimodal learning via adaptive KV cache compression. While initial reviews raised concerns about computational efficiency (Niy5) and comparisons (6UND), the authors' rebuttal convincingly addressed these. The reviewers acknowledged the improvements, with iyMg upgrading to weak accept. The work's ​​clear technical contribution​​ and ​​strong empirical validation​​ justify acceptance. The AC recommends acceptance of the paper, and suggest the authors to further answer reviewer Niy5's question on explaining in more detail why those hyperparameters perform well across different datasets. Congratulations.
Decision: Accept (poster)