When MLLMs Meet Compression Distortion: A Coding Paradigm Tailored to MLLMs
Authors: Jinming Liu
Zhaoyang Jia
Jiahao Li
Bin Li
Xin Jin
Wenjun Zeng
Yan Lu
Venue: ICLR 2026 Poster (2026)
Keywords: Image Coding
Image Compression
Multimodal Large Language Models
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: The increasing deployment of powerful Multimodal Large Language Models (MLLMs), typically hosted on cloud platforms, urgently requires effective compression techniques to efficiently transmit signal inputs (e.g., images, videos) from edge devices with minimal bandwidth usage. However, conventional image codecs are optimized for fidelity to serve the Human Visual System (HVS) and ill-suited for MLLMs, in which diverse downstream tasks are jointly considered. In this paper, we first systematically analyze the impact of compression artifacts on several mainstream MLLMs. We find that: Compression distortion unevenly impacts different-level image features, leading to varying effects on MLLMs' downstream tasks depending on their feature-level reliance. Motivated by this discovery, we propose an image Codec TAilored to MLLMs (CoTAM) designed to adaptively protect multi-level features and suit different demands of downstream tasks. The encoder leverages CLIP's shallow-layer attention to generate an importance map for bit allocation, preserving critical semantic regions. Concurrently, the decoder integrates a lightweight adapter with a multi-level loss function to ensure the faithful reconstruction both of low-level details and high-level semantic context for robust synthesis of cross-level features. Extensive experiments validate that our method achieves up to 35.99\% bitrate saving while maintaining the same performance on the MLLM tasks, outperforming previous SOTA neural codecs. The code is released at https://github.com/jmliu206/CoTAM.

Published as a conference paper at ICLR 2026
WHEN MLLMS MEET COMPRESSION DISTORTION: A CODING PARADIGM TAILORED TO MLLMS
Jinming Liu Liu Zhaoyang Jiahao Li Bin Li Xin Jin Wenjun Zeng", Yan Lu Shanghai Jiao Tong University 3 Microsoft Research Asia im1 iull6622 jtu. edu. jinxin@eitech. edu.
Eastern Institute of Technology, Ningbo, China
tjja, 1i. jiahao, libin, yanlujemicrosoft.com com
ABSTRACT
The increasing deployment of powerful Multimodal Large Language Models (MLLMs), typically hosted on cloud platforms, urgently requires effective com- pression techniques to efficiently transmit signal inputs (e.g., images, videos) from edge devices with minimal bandwidth usage. However, conventional image codecs are optimized for fidelity to serve the Human Visual System (HVS) and ill- suited for MLLMs, in which diverse downstream tasks are jointly considered. In this paper, we first systematically analyze the impact of compression artifacts on several mainstream MILMss We find that: Compression distortion unevenly im- pacts different-level image features, leading to varying effects on MI.MMs down- stream tasks depending on their feature-level reliance. Motivated by this discov- ery, propose image Codec TAilored to MLLMs (CoTAM) designed to adap- tively protect multi-level features and suit different demands of downstream tasks. The encoder leverages CLIP's shallow- layer attention to generate importance map for bit allocation, preserving critical semantic regions. Concurrently, the de- coder integrates lightweight adapter with a multi-level loss function to ensure the faithful reconstruction both of low-level details and high-level semantic con- text for robust synthesis of cross-level features. Extensive experiments validate that our method achieves up to 35.99% bitrate saving while maintaining the same performance on the MLLM tasks, outperforming previous SOTA neural codecs. The code is released at http::::///. // com 10160m//ooO00
INTRODUCTION
The proliferation of MLLMs, such as GPT-4o Hurst al. (0244). Gemini Team et al. (0233). and LLaVA Liu et al.) (202a)) has marked paradigm shift in artificial intelligence, revolution- izing suman-machine interaction, content understanding Li al. (2023a), and automation Yin et al. (2024). These models possess insatiable appetite for high-quality visual data Zhu al. (2025) to fuel their powerful capabilities. As MLLM applications become ubiquitous from real-time vi- sual question answering mobile devices to complex scene analysis in cloud-based services- the demand for transmitting and storing image and video data is growing at an explosive rate. This surge creates critical bottleneck: the conflict between the need for high-fidelity visual input and the constraints of limited communication bandwidth and storage resources. Consequently, developing highly efficient compression techniques tailored for this new era is not notjuust beneficial but imperative.
However, existing compression techniques are ill-suited for the versatile, open-world nature of MLLMs. Conventional codecs engineered for the HVS Wallace (199T); He al. (2022) Li
et al. (024c). while Image Coding for Machine (ICM) methods target specific. narrow computer vision tasks Feng et al. (2022): Chamain et al. (2021). This misalignment leads to inconsistent per- formance across the diverse capabilities of MILMs. As illustrated in Fig. [T])bb), both methods exhibit erratic performance, excelling in some tasks while failing in others He et al.) (2022): Kao al. (2025). Fundamentally, these approaches do not address the crucial question of how MLLMs holistically perceive and are affected by compression artifacts.
"Jinming Liu and Zhaoyang Jia visiting students Microsoft Research Asia.

---

Published as a conference paper at ICLR 2026
To address this gap, our work begins with systematic investigation into this question. Our analysis reveals a crucial insight: Compression distortion unevenly impacts different-level image features, leading varying effects on MLLMs' downstream tasks depending their feature-level reliance. Specifically, as shown in Fig. (d),), our analysis reveals that: tasks that rely on either low-level structural features (e.g., large-font OCR) global high-level semantic features (e.g., overall scene understanding) both demonstrate relatively robust to compressed distortion. In contrast, tasks requir- ing a synthesis of cross-level features (e.g., counting objects) are highly susceptible, as compression artifacts disrupt the crucial integration of low-level information into a coherent high-level semantic. Motivated by this finding, we introduce an image Codec TAilored to MLLMS (CoTAM). At the en- coder, our codec leverages priors from the shallow layers of a pre-trained CLIP model Radford et al.) (202T) to guide rate allocation. At the decoder, lightweight adapter with the reconstruction prior and a multi-level objective function ensures that both low-level fidelity and high-level perception are faithfully restored. This mechanism resolves the conflicting demands of different task types, ensuring the reconstructed output is faithful to the MLLMs' needs. The main contributions of this work are summarized as follows:
We first provide a systematic analysis of MLLM performance under compression, revealing how MLLMs are affected by compression distortion.
We propose CoTAM whose encoder uses lightweight CLIP-based semantic priors for rate allocation while the decoder uses a multi-level loss and adapter with reconstruction priors to preserve multi-level information.
Our approach achieve significant bitrate savings while delivering consistently high perfor- mance across wide spectrum of MLLM tasks, and also shows compatibility with high- resolution and video-based MLLM scenarios.
Ees Leve
recognition companson
19%
Image quality
social relation
(G) Tisk Performance drop (fomm BLIC) un MME Denchmarks
OCA
Image style
61%
SA
31%
25%
RaCare retation
reasoeing
1.%
attibate recogrition
localiatioo
compentton
Aole
Ours (CoTAM)
ELIC (Human Centrici
Bridge
[Machine Cennrrrccc
(2) MME Beachnark
(b) MMDench Canegory-
(e)
Task Performance drop (frum ELIC) un MMBBnch Categury-1
Figure 1: (a)(b) Performance comparison of compression methods on MME Fu et al. (0023) and
MMBench Liu et al.) (2024b) under similar bitrates. For MMBench, we report the 10 most affected tasks (largest score drops). Human-centric codec ELIC al. (2022) excels on low-level structural tasks (e.g.. Large-font OCR) and the ICM method Bridge-d1 Kao et al. (2025) excels on high-level tasks (e.g., landmark identification), while our method consistently outperforms both. (c)(d) Com- pression distortion (from ELIC) affects tasks differently: Tasks relying on either low-level structural features or coarse high-level semantics (c.g., OCR and scene understanding) tend to be relatively robust, whereas those depending on cross-level features (e.g., counting) suffer more, reflecting a synthesis that fails when corrupted low-level information can longer be coherently structured by the high-level context. Seeing more benchmarks' sub-tasks and images in Appendix.
2 IMPACT ANALYSIS OF IMAGE DISTORTION ON MLLMS
2.1 PRELIMINARIES: THE MLLM PIPELINE
The architecture of mainstream MILMS Li et al. 2024a); Zhu et al. 2025) comprises three key
parts: vision encoder Radford et al. (022 Zhai al. (2023). ) a projector, and a LLM Bai et al. Touvron et al. 2023). The vision encoder, often a Vision Transformer (ViT) Han et al. input image into sequence of vision tokens. These vision tokens are then passed through the projector (e.g., some MLP layers), a
2023
2022 serves as the model's 's "eye", responsible for transforming
lightweight network that maps them into the L.M''s feature space. Finally, the LLM backbone (e.g..
2

---

Published as a conference paper at ICLR 2026
Question
Answwr of Raw Imaga
Answer of Compressed Image
Question
Answer of Raw Image
Answer of Compressed Image
What thu
Stroaberry
AI
AI
Hos many Strowberiss RS
it lnithed
Figure 2: How compression affects VQA tasks: while the MLLM's robust low-level structural and coarse-graine granse-g high-level semantic abilities enable it to identify the "strawberry" and "Al", it fails on tasks demanding fine-grained cross-level information, such providing accurate count.
[cas]
Jess)
Juca 7- Outlen
Kugghow Carter
Stage
Preliminary Screening
Stage2: Local Information Extraction
Stage
Integration
Inflow: CLS-Suided Outflow: Dispersed
Inflow: (L5--uided, Outflew Localized
Inflow CL5 Summary Taken, Outflow: Re-dispersed
Information flow across different layers of the vision encoder. According the flows, we can divide the information processing to three stages as Table [ In stage 3, a token with high attention longer represents its own local visual content, but instead transforms into high-level Li et al. (2023c) responsible for integrating global information.
Figure 3:
'summary token' Liu et al. (2025):
LLama Touvron et al. (2023). Qwen Team (222)) processes these projected vision tokens alongside a text prompt to perform cross modal reasoning and generate the final output.
This work investigates the effects of the compression distortion the Vision Encoder. Because it serves as the sole gateway for visual information into the MLLM, the quality of its output tokens directly dictates the upper bound the entire model's 's downstream performance. To this end, in this work, we isolate our analysis from the LLM backbone, whose behavior is conditioned on specific textual prompts, in order to derive general conclusions about the visual processing pipeline itself.
2.2 EXPLORING THE IMPACT OF IMAGE COMPRESSION DISTORTION TO MLLMS
To design a codec tailored to MLLMs, we must first understand what visual information MLLMs require and how this information acquisition process is affected by compression artifacts.
2.2.1
How DOES VISUAL INFORMATION FLOW IN MLLMS?
Prior work shows that weak high-level semantic capability in MLLMs induces hallucinations Fu
et al.
2023 whereas supplying richer, clearer image details substantially improves performance
al. 2024a). Together, these findings imply that strong MLLMs must exploit both low-level
and high-level semantics. A scan of mainstream benchmarks (MME Fu et al. (2023), MMBench Liu al. (2024b), SEED-Bench Li et al. (2023a)) confirms this breadth: tasks span object recognition and counting, spatial reasoning, OCR, compositional inference, and the interpretation of abstract concepts like emotion and intent. The diversity of these tasks also indicates that MLLMs rely on visual information across multiple levels of granularity from low-level pixel details to high-level semantic abstractions. For example, Fig. [ answering "What is the word?" requires low-level structural OCR capabilities, determining "W hat is this fruit?" demands high-level global semantic reasoning, while the response to "How many strawberries are in this picture?" needs both structural information and global semantics. This raises pivotal question: how does the vision encoder transform raw pixels into a feature representation that balances both low-level details and high- level semantics? To investigate this, we analyze the information flow within the vision encoder (CLIP Radford et al. (202I)). inspired by the inflow/outflow methodology of Tong et al.] (2025). Specifically, for self-attention map A NxN in given layer, where A₁; denotes the attention
3

---

Published as a conference paper at ICLR 2026
Table 1: The Three-Stage Pattern of Visual Information Processing in the Vision Encoder.
Information Flow (
[CLS] Attention (Pig (exbi))
PCA- PCA-Vieatized Fratures
Stage It Prefiinnur Screeiing
(Shallow Luyers)
Outllow: Seationod, perforn a beaad inital scnansing
buflo: Reseives globel guidance from (CLS).
Broad with higher mitestt on key srea.
Resembble rae testures and edges. no signtfecnt aggnogation.
Stage 2:
Information Extraction
Inflov: Remains snchonal to (CL token Concemtates neighooring putches.
Comerges ceraiis odges local regions.
Fatracs Involavel frature strccures.
Layers)
Globel Semontic Imagration Layen)
Outfoow: Disperses
Diveriides, from: (CLS) und
refized
Comerrges few toleess.
Extact level level srrctura decalls discor ded.
Acteetion Hap
Early Stage
Later Stage
Stage
Early Stage
Stage
Stage
PCA FCttiii
044
Raw Image
(b) Compressed Image
(c) Cosine Similarity
Figure 4: Three stages' CLS attention maps and PCA features Table 1 (layer 0, 5, 15, 22 in
vision encoder) for (a) the raw image and (b) the compressed image. (c) The visualization of cosine similarity between raw tokens and distorted tokens. Simiarity is lowest at Early Stage 3, indicating significant impact on cross-level features.
from source token i to target token j, we define two metries Information Inflow & Outflow to trace the primary information pathways. Inflow = argmax Aıj. and Outflow(k) argmax Aik.
The visualization of this information flow (Fig. 3) reveals a distinct three-stage feature processing pattern, which is detailed in Table and corroborated by the PCA Maćkiewicz & Ratajezak (1993) visualization and [CLS] attention maps in Fig. Ma). The process begins with Stage I: Preliminary Screening, where shallow layers perform broad, initial scan of the image, with attention scattered to capture raw textures and edges. This is followed by Stage 2: Local Information Extraction, where middle layers consolidate these findings: the Outflow becomes shorter, with attention con- verging on neighboring patches to analyze local features with clear structures. Finally, the deep layers execute Stage Global Semantic Integration. In this phase, the model integrates refined local features into a holistic, semantic representation, with attention converging on a few key "sum- mary tokens." Liu et al. 20255: Li al. (2023c)
To quantitatively validate our three-stage finding, we measure two layer-wise attention distance metries Dosovitskiy et al. (2020) on 1,000 images from the CC3M dataset Chaiseeinhhnnpinoooo al. (202T): the Average Attention Distance (Dagg) and the Average Max Attention Distance (Dropt).
N
N A., d(p..
Davg
Dip1
N Σ d(p:, Pargmax,
N
1=1
Here, A is the N. attention map from self-attention layer, where A.j is the attention weight from token j to token i. The term p; denotes the 2D spatial position of the i-th token in the input image. Consequently, d(pt.Ps) PU) represents the Euclidean distance between the positions of tokens i and j. As plotted in Fig. (b)),)), both metrics exhibit clear U-shaped trend. The average distance is high during Stage decreases for Stage 2, and increases again during Stage This quantitative trend strongly corroborates our findings.
2.2.2 How DOES COMPRESSION DISTORTION AFFECT MLLMS?
Having established the three-stage information flow model, we analyze its vulnerability to compres- sion distortion. By measuring the cosine similarity of feature tokens between original and com- pressed images each layer, clear pattern emerges, shown in Fig. c(c). While the low-level features in Stage and 2 prove relatively robust to compression, linearly and slowly decrease in similarity layer by layer, we observe sharp drop in similarity in the early phase of Stage which marks a critical failure in the formation of cross-level features. These features are uniquely vulner- able because their creation requires delicate synthesis of high-fidelity low-level details from Stage
4

---

Published as a conference paper at ICLR 2026
Sirenvyy
Extrection
Siskal Semeeble Intepeelint
Prafeisury Stresho
Lacal Informablor Setrwbon
Integrebice
Steeshy
information Excruciion
Stahel Senantic Ietegredion
High-leve!
Lowe
high-leeal
P. 1333
(c) Cosine Similarity between raw token
(a) Average Attention Distance
(b) Avenag Max Attention Distance
distorted cach layer
Figure 5: (a)(b) Attention distance and (c) the impact of distortion on internal tokens in the vision encoder. Low-level (Stage 2) and coarse high-level features (the later phase of Stage 3) are rel- atively robust to compression artifacts, while cross-level features (the early phase of Stage 3) are significantly affected because they require both high-fidelity low-level details and emerging high- level semantic context. Blue, green, and red indicate stages 1, 2, and 3, respectively.
and emerging high-level semantic context from Stage Consequently, even the subtle corruption of the low-level details by compression leads to disproportionately large failure in this synthesis process. In contrast, the similarity recovers in the later part of Stage 3, demonstrating that coarse, high-level semantics are more resilient. This finding is further corroborated by the attention maps, PCA features and cosine similarity in Fig (c).. While these visualizations show little change in PCA features and high cosine similarity in stages and 2 between original and compressed images, the token similarity in the early phase of stage 3 is significantly decreased. The later part of stage 3 is aimed at generating coarse high-level semantic information. Therefore, the impact of distorted details is diminished, resulting in higher overall cosine similarity. As shown in Fig. 2 compres- sion distortion only minimally affects questions of high-level semantics (e.g., "What is the fruit?") or low-level structure (e.g., "What is the word?"). Its impact is much greater, however, on tasks like counting or texture analysis, which demand both local details and global context.
Task-level validation confirms this hypothesis. As shown in Fig. (c)d), tasks requiring the synthe- sis of both detailed and semantic information (e.g., "count") degrade severely under compression. Conversely, tasks reliant on either robust low-level structures (OCR) or coarse high-level semantics (positional reasoning) remain resilient. This leads to a key insight: the critical failure point of compression is not uniform loss of different feature types, but disproportionate collapse of the cross-level representations that bridge lovelevel and high-level information.
The takeaways of the above analysis are the following:
Takeaways:
1. MLLMs require visual information at different levels perform diverse tasks.
2. The vision encoder in MLLMs operates in three stages: shallow layers handle initial filtering, middle layers extruct low-level features via local unalysis, and deep layers perform global semantic integration, sequentially assembling these features into cross-level and then high-level semantic representations.
information loss increases linearly in early layers, indicating that low-level features suffer only modest degradation. However, this compromises cross-level features, which rely on integrating low-level information with high-level context to preserve fine grained semantics. In contrast, coarse high-level features are moderately affected, as they depend more on abstract representations.
Compression-indle
These expose fundamental paradox in current ICM approaches/Kar et al. (2055 Li et al. (2024b); Chamain et al. (202T). They only try to preserve the high-level information but ignore the lowelevel information, which is important for MLLMs to generate cross-level features. Our work is thus built upon a new cornerstone: An effective codec must simultaneously preserve proper both low-level fidelity and high-level semantic information.
5

---

Published as a conference paper at ICLR 2026
Codecc
Mccerraction
ALLM
Cedeec
MLLMM
Enc.
Das
LLM
Fine
CLP
lowll
Cuart
Maraga
Auallary Branch (truinig-oony)
Shaller C.IP Guidence
(e) Training Stage.
[b) inference Stage.
(c)
Extension to High-resolution Image.
Figure 6: The framework of our method.
3 COTAM: CODEC TAILORED TO MLLMS
Our analysis reveals core principle for codec tailored to MILMs: it must preserve multi-level visual information. Based on this principle, we introduce CoTaM, codec designed with a dual- strategy approach, as depicted in Fig. ob)). First, drawing upon the insight from Takeaway 2-that the initial layers of a vision encoder perform preliminary information filtering- encoder uses shallow CLIP attention guide bitrate allocation, prioritizing important regions for MLLMs. Sec- ond, inspired by Takeaways and 3, our decoder uses the decompressed image as a reconstruction prior to retain robust low-level details and avoid domain shift. A latent feature adapter then injects semantic enhancements, and the entire model is optimized with a multi-level loss that supervises fidelity multi-level features. Furthermore, for high-resolution inputs, CoTaM incorporates Hier- archical Guidance mechanism to fuse multi-scale semantic information, making it compatible with the patch-based processing Liu al. (2024a) common in MLLMs for both images and videos.
3.1 BASE CODEC
Our base codec enables variable bitrates by adapting the multi-quantizer methodology from Jia al. (2025): Cui et al. (2021). We equip its internal layers with multiple sets learned quantization vectors for each bitrate to adaptively allocate bits for each spatial location. This allows the semantic importance map to select a specific vector for each region, thereby assigning more bits to critical
areas and fewer the rest areas. Further architectural details are provided in the Appendix.
3.2 SHALLOW CLIP-GUIDED ENCODER
Our Shallow CLIP- CLIP-guided encoder is born from the Takeaways 2 of prior analysis: the shallow layers of an MILM's vision encoder perform a preliminary screening to identify regions of potential importance. To leverage this early-stage intelligence, we average the [CLS] attention scores from the first three layers of frozen CLIP model Radford et al. (2021) chosen for their high atten- tion distance (Fig. - create small downsampled spatial map (e.g.. 8x8), which quantifies the semantic richness of each region.
This continuous map subsequently converted into discrete, three-level mask via statistics- based quantization method + ko. The three integer levels in this mask directly correspond to rate allocation instructions: decrease bitrate, maintain base bitrate, or increase bitrate. Crucially, due to the small size of this map and its quantization into only three values, the bitrate overhead for this map is negligible (128 bits for 336x336 input). This final mask then directly modulates the patch-wise basis, ensuring that
quantization parameters of our learned compression backbone semantically critical regions for MLLMs are allocated more bitrate and with higher fidelity.
3.3 MULTI-LEVEL FIDELITY DECODER
Our analysis revealed critical flaw in existing ICM methods: their pursuit of high-level semantic fidelity, they often degrade the low-level structured information, and also in turn lead to significant loss of cross-level features. To resolve this problem, our decoder is designed to preserve fidelity across the entire feature hierarchy. It achieves this through two key components:
First, our design leverages the decoded image as a reconstruction prior. This approach serves two critical functions. On the one hand, shown in Fig. [ and takeaway 3, since standard compression already effective at preserving robust low-level structures, using the decoded image ensures this foundational information is retained. On the other hand, it mitigates a potential domain shift, as

---

Published as a conference paper at ICLR 2026
MLLM vision encoders are pre-trained natural RGB images; providing the decoded image as a prior grounds the input in the expected domain. Upon this prior, a lightweight Latent Feature Adapter, composed of a single transformer block, operates directly on the decoded latent code from the bitstream. It generates a semantic enhancement feature that is fused (via element-wise addition) with the patch embeddings extracted from the decoded image. This strategy injects high- level guidance directly into the feature domain without disrupting the crucial low-level information.
Second, as illustrated in Fig. b(a), the entire framework trained end-to-end using multi-level fidelity loss, Liotal, to supervise the fidelity at both ends of the feature spectrum. This loss is weighted sum of two components:
Liial Lrow + Atigh
The first component, the low-level fidelity loss (Luw), is designed preserve fine-grained details often damaged by existing methods. Guided by our finding in Takeaway 3. it imposes critical constraints on the shallow layers by minimizing the Mean Squared Error (MSE) between the patch embedding features of the original and decoded images. Simultaneously, the high-level perceptual loss nLigh) ensures global semantic coherence by minimizing the MSE between the final-layer token representations of the original and our processed output.
3.4 EXTENSION TO HIGH-RESOLUTE
AND VIDEO INPUTS
Handling high-resolution images is a critical capability for MLLMs, making it imperative for codecs support them efficiently. This presents core dilemma. On one hand, guidance from single, fixed-size downsampled image is too coarse; as shown in G the background attention is rel- atively coarse, failing to focus on important information. A direct strategy adapt this, inspired by mainstream MLLM processing pipelines, is to employ a patch-based method where local guid- ance is applied each patch independently. The fundamental limitation of this approach, however, is its lack of global perception; it cannot determine which local information is crucial for building coherent semanties across different patches. For instance, in Fig. c(c), lacks sufficient attention on the person's 's head. Therefore, to resolve this conflict between local detail preservation and global se- mantic integrity, we propose our Hierarchical Guidance to fuse (via addition) both global and local maps, creating comprchensive guidance signal that is both locally precise and globally aware. On the other hand, we resize the decoded high-resolution features to get a global feature before they are processed by the adapter. This is done to match the expected input of the high-resolution MILM, which is composed of multiple high-resolution patches and a downsampled global patch.
Our method is also compatible with video MLLMs. Current mainstream approaches typically pro- cess videos by sampling a sequence of individual frames, a strategy analogous to the patch-based processing of high-resolution images. Consequently, our semantic guidance mechanism can be ap- plied on frame-by-frand basis to guide the compression of videos.
4 EXPERIMENT
4.1 EXPERIMENTAL SETTINGS
Codec Setting. Our framework is built upon two learned image compression models, ELIC He ct al. (2022) and DCAE Lu et al. (2025) to demonstrate the versatility of our approach in being integrated with different codecs. For model training, we utilized a dataset comprising one million images randomly sampled from the CC3M dataset Changpinyo al. (2021). The training protocol spans a total of five epochs, with the first epoch dedicated to an initialization phase using only the low-level fidelity loss (Ltow). This pre-training step ensures stable optimization trajectory by al- lowing the network to first grasp the reconstruction of basic structural features. For hyperparameters k. Noow and Anigh. we empirically set them to 0.75, 0. and 1, respectively.
MLLM Setting. For MLLM evaluation, our primary experiments were conducted on LLaVA- 1.5 Liu et al. (2024a) 7B and 13B variants with a CLIP encoder et al. (202T) to assess performance and scalability. To further substantiate the generalization capabilities of our method, we also performed tests on LLaVA-Onewe Onevision- 7B Li al. (2024a) (with a SigLIP encoderZhai et al.
2023) and Intern VI.2-BB Chen et al. (2024) (with Intern ViT encoder Giao et al. (202)))

---

Published as a conference paper at ICLR 2026
2014
941
ECA
A5:00
RVC.
130
940
324
Figure 7: Performance comparison on LL5VA-1.5-7B.
POPE
SEECBench
MMF
POPE
SeedBendh
MME
1120
Enieg 31
BLC
6111
031
27
6.0
-280
-112
611
Figure 8: Left three: Performance comparison on LLaVA-1.5-138 Right three: Performance com- parison with methods that fine-tuned the codec encoder.
Testing Benchmark Our evaluation protocol is twofold, assessing both MLLM tasks performance and image reconstruction quality. For image benchmark, we evaluated on MME Fu et al. (202)), TextVQA (Singh et al. (2019), POPE Li et al. (2023b), SeedBench Li et al. (2023a), VQAv2 Goyal al. (2017), MMMU Yue al. 2024). and MMBench Liu et al. (2024b). For video benchmark:
we used Video-MME Fu et al.
2025 For reconstruction metric, we report PSNR.
Compared Methods To position our work within the current landscape, we compared the codec against comprehensive set of baselines. For human-centric image compression methods, we se- lected ELIC He et al. (2022), and DACE Lu et al. (2025). For coding for machine methods, we compared against Bridge-d1 (fixing encoder), Bridge-d3 (finetuning encoder) Kao al. (2025) and ICMH-adapt al. (2024b). Since ICMH-adapt Li et al. (2024b) only supports the ResNet archi- tecture, we reimplemented this method and trained with our multi-level loss.
4.2
PERFORMANCE COMPARISON
4.2.1
LOW-RESOLUTION IMAGE BENCHMARK
Our primary validation, presented in Fig. 7 is conducted on the LLaVA-v1.5-7B model with a 336x336 input resolution. Using ELIC as base codec, our method consistently outperforms pre- vious approaches across six diverse benchmarks. As shown in Table under the same performance level, achieves 35.9 35.99% bitrate saving. To demonstrate its generalizability, we integrated our method with another SOTA codec, DCAE et al. 2025), and achieved similar performance gains. The scalability of our approach is further validated in Fig. g where we also show improvements on the larger LAV5-115--333 model, proving its effectiveness across different model scales.
Finetuning Codec. While our main approach freezes the codec sidestep the perfor- mance-reconstruction trade-off, we also test fine-tuning variant by adding a rate loss Minnen et al. (0018) to the objective (Eq. Tp. The results, presented in Fig. g show that even in this comparison with another fine-tuning method Kao et al. (2025), our approach demonstrates superior performance. Furthermore, both methods sigmficantly outperform the original, non-fine-tuned base codec.
TextVQA
MME
VideoMME
TorIVQA
VidooMME
200
11:0
Figure 9: Performance comparison on High-resolution and Video MLLM. Left three: LLaVA. Onevision-78 7BB. Right three: Intern VL2-8B.
8

---

Published as a conference paper at ICLR 2026
4.2.2 NIGH-RESOLUTION IMAGE AND VIDEO BENCHMARK
Addressing the significant overhead of high-resolution data, we extend our method to this domain. To the best of our knowledge, our work is the first to pioneer coding framework for high-resolution image and video MLLMs. We validate this on two mainstream models, LLaVA-One Vision- and VntemVL2-8B with results presented in Fig. 9 For high-resolution images, approach consis- tently outperforms the base codec. Because the current mainstream Video LLM usually extracts video frames into fixed frame images (such 16, 32 frames), our method also be directly applied to video MILM. The codec also achieves superior performance on Video-MME.
4.3 ABLATION STUDY
Framework. To assess each component's contribution, we perform an ablation study. As shown in Fig. [Taa) the removal of the Adapter module induces a catastrophic degradation in perfor- mance across all three benchmarks. This consistent and vast performance underscores the Adapter's role as an essential bridge between the compressed features and the downstream MLLM: its function in aligning feature spaces is both indispensable and universally critical.
Conversely, ablating the image reconstruction module (blue curve) also impacts performance, but with varying severity across benchmarks, reflecting different dependencies on visual fidelity. For TextVQA (Fig. [T]aa) and SeedBench (Fig. [[]c)), "Ours (w/o Rec.)" drops sharply relative to the full model, highlighting the value of reconstruction-indeee indducedd prior knowledge. In contrast, the impact on MME (Fig. (O))) is much milder.
Lastly, removing the clip guidance module (brown curve) consistently reduces performance across benchmarks, indicating it as an effective general optimization.
(a) Muttslesel Loss Analysic
(c)
Attention used for Guidance
Textiqee
MME
SeedBench
0um
-430 Bolccre
(b) Mierarchical Ouidance Anetyals
4
Readio
Liver:
Sungen
Loca
Diyer
Ne))
318
CGI
(b)
(c)
Figure 11: Ablation study modules. We test BD-score using ELIC as the anchor on MME.
Figure 10: Ablation study on framework.
Training Loss. We validate the necessity of our multi-level loss design. As shown in Fig. I1 relying solely on the high-level loss fails capture essential low-level details, while using only the low-level loss produces detailed yet semantically inconsistent results. Optimal performance is achieved by integrating both.
Hierarchical Guidance. For high-resolution images, our proposed Hierarchical Guidance im- proves the importance map by fusing local and global attention. The results in Fig. [T[b) demon- strate that it yields a clear performance improvement over a purely global guidance strategy.
Attention Maps. Our use of averaged attention maps from CLIP's first three layers is validated Fig. IT()), achieving optimal performance as shallow layers are better for holistic screening. In contrast, deeper layers emphasize global aggregation and thus degrade performance, consistent with our three-stage information flow model.
4.4 COMPLEXITY ANYLSIS
We analyze the computational complexity of our method in Table 2 Since approach only uti- lizes the first three shallow layers of the CLIP encoder, the increase in encoding time is marginal compared to the base codee. Furthermore, and the CLIP guidance only reallocates bit rates, the overall PSNR in Fig. [2 shows only a minor degradation compared to the base codec.
framework does not require tnn-ttnnig the codec

---

Published as a conference paper at ICLR 2026
Method
Encoding
Decoding
Total
BD-Rate
PSNK 1885
0,269
800
Ours (ELIC)
0.178 (+2.9%) 0.101 (+5.2%)
0.279 (+3.77%)
+3.99%%
DCAE
0.162
OA
Ours (DCAE)
0.080 (+3,9%% 0.091 (+7.1%)
0.171 (+5.66%)
-3.05%
Bix Per heal Dop!
Table 2: Comparison of times Kodak dataset (resized
Figure 12: PSNR comparison on Kodak dataset.
336x336), and average BD-rate on six MLLM benchmarks, which represents the bitrate saved to achieve the same score.
5 CONCLUSION
We conduct comprehensive analysis of how compression artifacts affect MLLMs, revealing that fine-grained semantic features cross-level features are highly vulnerable to subtle low-level dis- tortions. Based on this insight, we propose a codec tailored to MI..Ms, featuring CLIP-guided bit allocation and a multi-level fidelity preserved decoder. Our method consistently achieves significant bitrate savings while preserving MILMM performance across diverse tasks. This work underscores the importance of compression strategies aligned with the feature hierarchy MLLMs.
ACKNOWLEDGMENTS
This work was supported by Grants NSFC 62302246, ZINSFC LO23F000220,,0000,, Ningbo 2023Z237 & 20247284 & 2024Z289 & 2023CX050011 & 2025Z038 & 2025Z059, and supported by High Performance Computing Center at Eastern Institute of Technology and Ningbo Institute of Digital
Twin.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
Benjamin Bross, Kui Wang, Yan Ye, Shan Liu, Jianle Chen, Gary J Sullivan, and Jens-Rainer Ohm. Overview of the versatile video coding (vvc) standard and its applications. IEEE Transac- tions on Circuits and Systems for Video Technology, 31(10)3373-3374, 2021.
Labiru D Chamain, Fabien Racapé, Jean Bégaint, Akshay Pushparaja, and Simon Feltman. End-to- end optimized image compression for machines, a study. In 202/ Data Compression Conference (DCC), pp. 172.172.10 IEEE, 2021.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training recognize long-tail visual concepts. In Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, pp. 3558-3568, 2021.
Yi-Hsin Chen, Ying-Chieh Weng, Chia-Hao Kao, Cheng Chien, Wei-Chen Chiu, and Wen-Hsiao Peng. Transtic: Transferring transformer-bssed image compression from human perception to machine perception. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 23297-23307, 2023.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, al. Expanding performance boundaries of open- multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024.
Ze Cui, Jing Wang, Shangyin Gao, Tiansheng Guo, Yihui Feng, and Bo Bai. Asymmetric gained deep image compression with continuous rate adaptation. In Proceedings of the IEEE/CVF Con- ference Computer Vision and Putlern Recognition, pp. 10532-10541, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 11929, 2020.
Ruoyu Feng, Xin Jin, Zongyu Guo, Runsen Feng, Yixin Gao, Tianyu He, Zhizheng Zhang, Simeng Sun, and Zhibo Chen. Image coding for machines with omnipotent feature learning. In European Conference on Computer Vision, pp. 510-528. Springer, 2022.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. Mme: comprehensive evaluation benchmark for multimodal large language models. ArXiv, abs/2306. 13394, 2023, URL btttps:/aapi semant wwweettiscchhass
wwgg/Corpussiii : 259243928
Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evalu- ation benchmark of multi-modal 11ms video analysis. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 24 20108 2088-21188 2025.
Zhangwei Gao, Zhe Chen, Erfei Cui, Yiming Ren, Weiyun Wang, Jinguo Zhu, Hao Tian, Shenglong Ye, Junjun He, Xizhou Zhu, et al. Mini-intemvl: mexible-trannser pocket multi-modal model with 5% parameters and 90% performance. Visual Intelligence, 2(1):32, 2024.
Yash Goyal. Tejas Khot, Douglas Summers- Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and puttern recognition, pp. 6904-6913, 2017.
Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al. A survey on vision transformer. IEEE transactions on pattern analysis and machine intelligence, 45(1): (1700881 2022.

---

Published as a conference paper at ICLR 2026
Dailan He, Ziming Yang, Weikun Peng, Rui Ma, Hongwei Qin, and Yan Wang. Elie: Efficient learned image compression with unevenly grouped space-channel contextual adaptive coding. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 5718- 5727, 2022.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Zhaoyang Jia, Bin Li, Jiahao Li, Wenxuan Xie, Linfeng Qi. Houqiang Li, and Yan Lu. Towards practical real-time neural video compression. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 1254--12522, 2025.
Chia-Har Kao, Cheng Chien, Yu-Jen Tseng, Yi-Hsin Chen, Alessandro Gnutti, Shao- Yuan Lo, Wen-Hsiao Peng, and Riccardo Leonardi. Bridging compressed image latents and multimodal large language models. In The Thirteenth International Conference on Learning Representations. 2025.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.0m 03326....... 2024a.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Bench- marking multimodal Iims with generative comprehension. arXiv preprint arXiv:2307.16122 2023a.
Han Li, Shaohui Li, Shuangrui Ding, Wenrui Dai, Maida Cao, Chenglin Li, Junni Zou, and Hongkai Xiong. Image compression for machine and human vision with spatial-freuuency adaptation. In European Conference on Computer Vision, pp. 382-399. Springer, 2024b.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint 200:23355.03555 2023b.
Yiran Li, Junpeng Wang. Xin Dai, Liang Wang, Chin-Chia Michael Yeh, Yan Zheng, Wei Zhang, and Kwan-Liu Ma. How does attention work in vision transformers? visual analyties attempt. IEEE transactions on visualization and computer graphics, (000)2888-29000 2023c.
Zhiyuan Li, Yanhui Zhou, Hao Wei, Chenyang Ge, and Jingwen Jiang. Towards extreme image compression with latent feature guidance and diffusion prior. IEEE Transactions on Circuits and Systems for Video Technology, 2024c.
Zhuoyuan Li. Junqi Liao. Chuanbo Tang, Haotian Zhang, Yuqi Li, Yifan Bian, Xihua Sheng, Xinmin Feng, Yao Li, Changsheng Gao, et al. USTC-TD: A test dataset and benchmark for image and video coding in 2020s. IEEE Transactions on Multimedia, 2025.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lec. Visual instruction tuning. Advances in neural information processing systems, (0633482-334166, 2023a.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 26296-26306, 2024a.
Jinming Liu, Heming Sun, and Jiro Katto. Learned image compression with mixed transformer- cnn architectures. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 14388-14397, 2023b.
Yajie Liu, Guodong Wang, Jinjin Zhang, Qingjie Liu, and Di Huang. Unveiling the knowledge of clip for raining-free open-vocabulary semantic segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 5649-5657, 2025.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference computer vision, pp. 216-233. Springer, 2024b.
12

---

Published as a conference paper at ICLR 2026
Jingbo Lu, Leheng Zhang, Xingyu Zhou, Mu Li, Wen Li, and Shuhang Gu. Learned image compres- sion with dictionary-baic based entropy model. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 12850-12859, 2025.
Andrzej Maćkiewicz and Waldemar Ratajczak. Principal components analysis (pea). Computers & Geosciences, 19(3):303-342, 1993.
David Minnen, Johannes Ballé, and George D Toderici. Joint autoregressive and hierarchical priors for learned image compression. Advances neural information processing systems, 31, 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 7748-8766 PmLR, 2021.
Amanprect Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8317-8326, 2019.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Man--aat Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.
Qwen Team. Qwen2 technical report. arXiv preprint urXiv:2407.16671, 2024.
Jintao Tong, Wenwei Jin, Pengda Qin, Anqi Li, Yixiong Zou, Yuhong Li, Yuhua Li, and Ruixuan Li. Flowcut: Rethinking redundancy via information flow for efficient vision-language models. arXiv preprint 2.tiv:2505519536 2025.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 200. 2023.
Gregory K Wallace. The jpeg still picture compression standard. Communications of the ACM, 34 (4):3044, 1991.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. National Science Review, (1ree,,,1111111)) 11(12):wwae403 2024.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multi- modal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9556-9567, 2024.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training In Proceedings of the IEEE/CVF international conference on computer vision, pp. 11975-11986. 2023.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint 20Xiv:2504.1079, 2025.
13

---

Published as a conference paper at ICLR 2026
A
APPENDIX
A.1
Related Works
14
A.2
Benchmark Examples
14
A.3
Codec Training Strategy
15
A
4
Visualization of Bit-rate Allocation
16
A.5
Preliminary Experiments on Finetuning Strategies
16
A.6
Discussion on the Quantized CLIP Guidance Map
17
A.7
Discussion on Video MLLM
18
A.8
Attention Distance of Different Vision Encoders
18
A.9
ICM Method Task-wise Performance Drop Leaded by Compression Distortion
19
A. 10 Attention Map and PCA Features of Different Layers
21
A. Information Flow of Different Layer
A.12 Analysis on Other Datasets
22
A.13 LLM Usage
22
A.1 RELATED WORKS
A.1.1 MULTIMODAL LARGE LANGUAGE MODELS (MLLMS)
Multimodal Large Language Models (MLLMs), such as LLaVA Liu al. (2023a), Gemini Team et al. (2023), and GPT-4o Hurst et al. (2024). have demonstrated remarkable capabilities by aug- menting Large Language Models (LLMs) with visual perception. These models typically use vision encoder (e.g., Clip Radford et al. (02T)) SigLip Zhai et al. (2023) to process images and an LLM backbone (e.g., LLama Touvron et (2023), Qwen|Bar et al. (2023)) perform cross-modal reasoning. However, the prevailing cloud-edge deployment of MLLMs hosting powerful models on servers while capturing data at the edge- present a significant communication bottleneck. This challenge motivates our work to develop compression solution optimized not for human viewing, but for the unique perceptual needs of MLLMs.
A.1.2 IMAGE COMPRESSION
The fundamental goal of image compression to minimize the bits required represent an im- age-thereby reducing storage and transmission costs-while maintaining sufficient fidelity for its intended application. Conventional image compression, encompassing both traditional standards like JPEG and VVC Wallace (1991): Bross et al. (2021). and modern learned methods Liu et al. 2202bb Lu et al. (2025). is fundamentally optimized for the Human Visual System (HVS) et al. 2025 often by discarding information that is imperceptible to humans but potentially vital for machine analysis.
To bridge the gap created by this human-centric paradigm, the field of Image Coding for Machine (ICM) emerged. However, the predominant ICM approach |eng et al. (2022); Chen et al.] (2023); ei (2024b) involves tailoring codecs for narrow, specific tasks like object detection segmen- lation. However, this task-specificity is fundamentally at odds with the general- pprrose nature of MLLMs. Thus, a critical research gap remains for compression solution that preserves the full visual features required by these models.
A.2 BENCHMARK EXAMPLES
To illustrate the diversity of tasks that Multimodal Large Language Models are expected to per- form, provide representative examples from two key benchmarks used in evaluation. Fig. [nsoosseee selected guestion-answe answer pairs from the MME et al. (2023) and SEED-Bende
al." (202aa) benchmarks, respectively. These tasks range from object recognition and counting
14

---

Published as a conference paper at ICLR 2026
Optical Character Recognition (OCR). Notably, the OCR examples involve large-font text where understanding the overall structure and positional relationships is crucial for correct interpretation. Furthermore, the examples highlight the varied question formats MLLMs must address, encompass- ing both binary (Yes/No) judgments and multiple-choice selections. Collectively, these examples underscore the necessity for a compression codec to preserve wide spectrum of visual informa- tion-from fine-grained details and high-level semantics to the essential structural and positional cues required by these diverse tasks. This challenge is core motivation for our work.
OCR the word the logo l'ceest cheesel"?
Commonsense Reasoning
C'ust CHEESE
A- loanee
want to carry thing with rainy day. Is the thing
Answer the question using a single word or phrase.
the image appropriate choice?
Answer question using
single word phrase.
WIP
MARIGAN
Posters
Scene
"Is this movie titled perfect world (1993)? Answer the question using single word or phrase."
Does this image describe place of chalet?
Answer the question using single word or phrase.
Figure 13: The QA pair examples in MME Benchmark Fu (2023).
Scene Understanding
What is dominant color the image?
A. White
B. Brown
C. Black
Grey
Answer with the option's letter from the given choices directly.
Visual Reasoning
What is the most likely reasoning behind the man's shocked expression?
A. He got bad grade on a test
B. He suddenly remembered forgotten appointment
He's witnessing a fight
He didn't expect the amount of poker chips on the table Answer with the option's letter from the given choices directly.
Figure 14: The QA pair examples in SeedBench Benchmark
Li et al. 2023a
A.3 CODEC TRAINING STRATEGY
Following Jia al. (2025): ; Cui et al. (021I) codec is trained operate at multiple bitrates within single, unified model architecture, as shown in Fig. [5] The core of this variable-rate capability lies in the integration of learnable vectors at multiple intermediate layers of the encoder. These vectors perform a scaling of the feature maps to dynamically control the information flow and, consequently, the final rate-distortion trade-off.
Let the feature map at the output of the 1-th encoder layer be denoted as f. For a discrete set of N target bitrates R T1, T2,... - I.N, introduce N corresponding sets of learnable vectors. For
15

---

Published as a conference paper at ICLR 2026
Residual Blocks
Residual Blocks
P22 Muce
Residual Blocks
v22 Srmne
x
222
Leamed Vector
Context Model & Hyper- Prior
Quality Parameter
decore x2n
deconn x22
Dostmo x2
Residual Blocks
Residual Blocks
Residual Blocks
8
Figure 15: The variable-bitrates compression frameworks.
Question
Raw Image
Ground Truth
w/o Guidance
w/ Guidance
Quantized Map
Are there four dags appear in chis imnge?
Ansures
AnswIeT
Is is a Mosque?
Answerra
Answer:
Figure 16: The visualization results under similar total bitrates.
a given target rate R, a specific vector gl. is applied the feature map fi at each modulated layer This operation is formulated as:
= fi ① gt.r
(2)
where @ represents element-wise multiplication, and fj is the scaled feature map that serves as the input to the subsequent layer
During the training process, quality index v is randomly sampled in each iteration. This determines both the of gain vectors gt, be used in the forward pass and the corresponding trade-off' parameter A; for the loss function. The entire network, including all N sets of gain vectors, is optimized end-to-end using the rate-distortion loss:
- D + XR
where D is the distortion loss and R is the estimated bit rate. By using different X; for each quality level (where a larger A; encourages a lower bitrate). In our implementation, we empir- ically define N 10 quality levels, with the corresponding set of tradeoff parameters being As (0.00002, 0.00005, 0.0001, 0.0002, 0.0004, 0.0008, 0.0016, 0.0032, 0.0064, 0.0128).
A.4 VISUALIZATION OF BIT-RATE ALLOCATION
Fig. 16 visualizes the results our guidance map's 's bit-rate reallocation. It clearly shows that more bits are allocated to semantically important regions, leading higher fidelity for objects like dogs in the example.
A.5 PRELIMINARY EXPERIMENTS ON FINETUNING STRATEGIES
straightforward strategy to optimize a codec for MLLM is to directly finetune either the codec or the MILM on downstream cnttucttoo-followinng task. To evaluate the efficacy of these approaches, we conducted a set of preliminary experiments. We employed the L.aVA-Innstru dataset Liu et al.) 2023a) for finetuning, using standard cross-entropy loss in MIL.M the optimization objective.
16

---

Published as a conference paper at ICLR 2026
MME
1400
1200
800
ELIC
ELIC-CFT
ELIC-MFT
600
Ours/ELIC)
0.04
0.08
0.12
0.16
BPP
Figure 17: Preliminary experiments on finetuning strategies. ELIC-MFT indicates that the codec parameters frozen and the MLLM is finetuned. For ELIC-CFT, only the codec parameters finetuned, Our method only need to finetune the adapter.
k:00.55
k-0.5
k-0.55
k-0
k:055
we Guidance (o-1)
-12
-10
-30
-8
80 rate 081
8D rote 181
(a)
(b)
Figure 18: (a) Effect of the statistics-based guidance quantization parameter in + ko on BD- rate, All tested values improve over the no-guidance baseline, with performance remaining stable for & E [0.45, 0.85). (b) Impact of the number of quantization levels in the Guidance Quantized Map on BD-rate.
As illustrated in Figure 17 findings reveal the limitations of direct finetuning. When the codec parameters are frozen and the MI.LM is finetuned (ELIC-MFT), we observe only marginal per- formance gains. More strikingly, when freeze the MILMM and attempt finetune the codec (ELIC-CFT), the training process collapses, leading to catastrophic failure where the model loses its fundamental comprehension abilities. In stark contrast, our proposed method, which only re- quires finetuning lightweight adapter, yields substantial performance improvements. These results underscore the inadequacy of direct finetuning and motivate our approach.
A.6 DISCUSSION ON THE QUANTIZED CLIP GUIDANCE MAP
We explored multiple settings of the Quantized CLIP Guidance Map. First, we examined the statistics-based quantization method μa + ko with different values of k. As shown in Fig. a), all A values yield improvements, and the performance varies only slightly within the range [0.45, 0.85]. Although we adopt & 0.75 in the paper, other values within this suitable range produce similar results. Furthermore, as shown in Fig. 18(b), we investigated different numbers of quantization levels. While using multiple levels generally improves performance, the gain dimin- ishes when the spacing between levels becomes large (c.g., the five-level setting), possibly because the wider span assigns overly low quality some regions, thereby reducing overall performance.
17

---

Published as a conference paper at ICLR 2026
Figure 19: The input examples of video MLLM.
A.7 DISCUSSION ON VIDEO MLLM
Our method is also directly applicable video MILMs. We note that current mainstream video MLLMs, such as Li et al. (2024a); Chen et al. (2024); Zhu et al. (2025); Hurst et al. (2024), typically operate not on dense video streams, but on sparsely sampled sequence of keyframes (e.g., 16 or 32 frames extracted from the entire video), as shown in Fig. 19 This sparse sampling strategy inherently reduces the temporal redundancy between adjacent processed frames. Therefore, applying our image codec on frame-by-frame basis is practical and well-aligned strategy for this specific application. Consequently, our semantic guidance mechanism can be effectively applied to each sampled frame to guide the compression. While developing more advanced video codec that explicitly models the remaining long-range temporal correlations presents a valuable direction for future work, our current intra- frame approach offers strong and pragmatic baseline for compressing visual inputs for today's 's video MILMss
A.8 ATTENTION DISTANCE OF DIFFERENT VISION ENCODERS
To validate that our stree information flow model is general principle rather than an artifact of specific architecture, we extend our analysis to other prominent vision encoders, namely Intern-
VIT Chen et al. (2024) in Intern and SigLIP Zhai et al. 2023) in LLaVA VA-Onviision Li et al.) (202a)). As shown in Fig. 20 the average attention distance per layer for both encoders exhibits clear U-shaped trend, mirroring the pattern observed with the CLIP encoder in our main analysis. finding that vision encoders broadly follow a three-stage process: an initial broad screening (Stage 1). followed by localized feature extraction (Stage 2), and concluding with global semantic integration (Stage 3). Furthermore, Fig. reveals that the feature similarity un- der compression shows sharp drop during the early phase of Stage 3, followed by a recovery in the final layers. This behavior is consistent with our three-stage theory: the initial drop highlights the vulnerability of cross-level features during the synthesis of local details and emerging global context, while the subsequent rebound indicates the formation of a more stable, abstract semantic representation. Notably, SigLIP exhibits two sharp drops in similarity. We hypothesize this is due to SigLIP's architecture, which lacks a dedicated class token. Consequently. its deeper layers may need retain some local information for the final pooling process, leading to lower overall feature similarity. Nevertheless, the feature similarity in SigLIP's final layer still rebounds, which remains consistent with the global semantic integration phase of the three-stage process.
This corroborates
18

---

Published as a conference paper at ICLR 2026
Average Attention Distance per Layer (InterVVIT)
Average Attention Distance per Layer (SigLIP)
Average Attention Distance
Average Attention Distance
Transformer Layer Index
Transformer Layer Index
Figure 20: SigLIP).
The average attention distance per layer of different vision encoder (Intern and
ELIC
DCAE
Glhoertte Aecoge
DOSE
OGAS
Mutian
CLIP
ISCA
SigLIP
InternViT
Pump Danns
Figure 21: The impact (token similarity) of distortion on internal tokens the vision encoder.
A.9 ICM METHOD TASK-WISE PERFORMANCE DROP LEADED BY COMPRESSION DISTORTION
To provide more granular view of the inconsistent performance of existing codecs, we present a detailed task-wise breakdown of the performance degradation caused by compression. Fig. 22 shows the impact of ELIC, a codec optimized for human perception, on various sub-tasks within the MMBench benchmark. Fig. [3 further illustrates the performance drop on both MME and MM- Bench when using Bridge-d1, an Image Coding for Machine (ICM) method. These figures highlight that both numan-centric and machine-centre codecs exhibit erratic performance. excelling in some task categories while failing significantly in others. This inconsistency reinforces the argument that a new paradigm is needed one that is holistically tailored to the multi-level feature requirements of general-purpose MLLMs.
19

---

Published as a conference paper at ICLR 2026
High tres (-300))
Wed Copp (1--77))
42%
Pefommacc
38%
32%
29%
25%
21%
16%
15%
14%
10%
9%
compariso noonon
attribute
Image style
social
relation
celebritt
localization
property reasoning
objid
spatial relationstiip
function
reasoning
mage quality
attribute
Image, emotion
future, prediction
structuralized hove
nature
relation
actionn recognition
image. scene
image tople
identity reasoning
physical, relation
OCR
Figure 22: The task impact of compression distortion (from ELIC) on MMBench.
47%
H0P 9mp (4390)
31%
29%
2ng
1.5%
14%
9%
3%
prostion
Porformance drop on Benchmark
Hann teea (i-0m))
Crop (15-370)
50%
50
43%
42%
41%
39%
37%
27%
25%
17%
13%
13%
10%
6%
6%
image
Quality
attribute
companison
localization
structuralized. text
object
social relation
image style
nature
relation
property. reassoning
CCR
spatial relationship
celetrity
attribute
funetion
massniing
physical relation
future prediztion
image Topic
image emotion
identity. reassning
acttion recognition
Image, sciene
(b) Task Performence drop on MMBench Category-1
Figure 23: The task-wise impact of compression distortion (from ICM method Bridge-d1) on MME and MMBench.
20

---

Published as a conference paper at ICLR 2026
Layer1
Layer
Layer
Layor
Lay
Lays7
.ayor
Layor
Layor
Low
Layo
Laye
Loya
Layor
Figure 24: The attention maps of the class token in different layers.
A.10 ATTENTION MAP AND PCA FEATURES OF DIFFERENT LAYERS
Fig. 24 provides visual walkthrough of the [CLS] token's 's attention maps different layers of the vision encoder, substantiating our three-stage model. In the shallow layers (Stage I, c.g., layer 0), the attention is broad and scattered, performing preliminary screening of the entire image. As we move to the middle layers (Stage 2, e.g.. layer 7). attention becomes more focused, converging on local regions and edges to extract structured features. Finally, the deep layers (Stage 3, e.g., layer 22), the attention disperses again the model integrates globally aggregated information, with focus shifting to a few "summary tokens" that encapsulate high-level semantic concepts.
Complementing this, the Principal Component Analysis (PCA) visualizations in Fig. 25 reveal the cvolution of the features themselves. Features in the shallow layers resemble raw textures and edges. In the middle layers, these evolve into clearly structured local features. By the time reach the deep layers, the structural details are largely discarded in favor of abstract, high-level semantic representations. Together, the attention patterns and the feature visualizations provide strong, complementary evidence for the distinct information processing stages within the vision encoder.
A.11 INFORMATION FLOW OF DIFFERENT LAYER
To further dissect the information processing dynamics. we analyze the inflow and outflow patterns for tokens across different layers, as illustrated in Fig. 26 This analysis reveals clear three-stage progression. Initially, in Stage 1, tokens exhibit wide-ranging inflow and outflow without clear focus, a pattern characteristic of broad initial screening of the image. Subsequently. Stage 2 is marked by an asymmetric information flow: inflow remains anchored to the global [CLS] token for guidance, while outflow becomes highly localized to neighboring patches, reflecting focus on structured local feature extraction. Finally, in Stage 3, the dynamics shift again as most tokens re- ceive targeted inflow from a few "summary" tokens, which in turn broadcast their globally-integran semantic knowledge via global outflow. This dynamic confirms the final phase semantic synthesis and integration.
21

---

Published as a conference paper at ICLR 2026
Layer1
Layer
LayorG
Lays"
.ayar
Layo
Layo
Layt
Figure 25: The PCA features in different layers.
A.12 ANALYSIS ON OTHER DATASETS
In Fig. 27 present the results obtained from analyzing different datasets. It can be observed that the general curve trends are consistent with those in Fig. 5 indicating that different datasets do not affect conclusion.
A.13 LLM USAGE
We acknowledge the use of a large language model (LLM) to assist in the preparation of this manuscript. The LLM's role was strictly limited to improving grammar and refining language. It did not contribute to any of the core research components, such as the initial ideas, experimental design, data analysis, or interpretation the results.
22

---

Published as a conference paper at ICLR 2026
Lxper cuttow
Laper Youtfen
Jaye Ficker
Liper
Layer 3 Cuttow
Hour Befoww
Lowr 10 Gurlow
Loge 10 bdoon
Layer 12 Outbow
Layar 12 Iniow
Laper 12 Duttow
Lowr Ocrlow
bilinn
Outhow
Layer tritow
Durtow
Lager 21
Laper
Lawe
Lager Bili
Layar Outhow
Layer 23 tnfoww
Figure 26: The information flows of different layers.
23

---

Published as a conference paper at ICLR 2026
Antertion Dietanee
Ratrog Alten on Chaunce
Tarofemmr 1nder
Taralomae Index
(a) Kodak
(b) COCO
Figure 27: The attention distance on different datasets.
24


Reviews:
[{"review_id": "bYMgZl3Ojy", "reviewer": "Reviewer_iGt4", "summary": "This paper examines the impact of image compression on MLLMs, showing compression distortion harms cross-level features with low-level details with high-level semantics. Motivated by this insight, the authors propose CoTAM, a codec tailored for MLLMs that adaptively allocates bits based on shallow CLIP attention and reconstructs multi-level features via a lightweight adapter and multi-level loss. The encoder leverages semantic priors to prioritize important regions, while the decoder balances low-level fidelity and high-level semantics. CoTAM is compatible with both image and video MLLMs and supports high-resolution inputs through hierarchical guidance. Experiments show up to 36% bitrate savings without performance loss, outperforming both human-centric and machine-centric codecs.", "questions": "In general, I'm satified with the paper, but small concern remains. Please see the weaknesses section for details.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I think the paper has following strengths:\n\n1. Interesting finding and motivations: The paper provides analysis showing that compression artifacts disrupt cross-level features (merging low-level details with high-level semantics) rather than uniformly harming all visual cues. This finding is well-supported by attention-distance metrics and cosine-similarity experiments across multiple vision encoders. \n\n2. Good feasibility: The method wraps existing neural codecs with only a lightweight adapter (1 transformer block) plus a negligible 128-bit metadata overhead, making it easy to integrate into production pipelines without retraining.", "weaknesses": "However, I have some small concerns about the paper:\n\n1. Task-Specific Bias in Evaluation: Most of the benchmark tasks are on high-level semantic understanding tasks, what if the author benchmarks some low-level vision-related tasks besides OCR? I'm curious about that.\n\n2. I'm doubtful about the scalability part. CoTAM’s CLIP-guided encoder relies on shallow-layer attention maps (8×8 grid) for bit allocation. While this works for 336×336 inputs, its scalability to higher resolutions (e.g., 4K) or longer video sequences is unclear. The hierarchical guidance extension (Section 3.4) is only briefly tested, with no ablation on how local/global attention fusion scales computationally. A 128-bit overhead for 336×336 images may balloon for larger inputs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3ikgPKdUVB", "reviewer": "Reviewer_yANt", "summary": "This paper designs an image compression codec specifically for Multimodal Large Language Models (MLLMs). The authors argue that existing codecs, optimized for either the Human Visual System (HVS) or narrow machine-vision tasks, are ill-suited for the diverse, general-purpose nature of MLLMs. The paper's contribution is twofold: 1. It first presents an analysis of how compression affects MLLMs, modeling the vision encoder's information flow as a \"three-stage\" process (Screening, Local Extraction, Global Integration). The key finding is that compression disproportionately degrades \"cross-level features\"—the synthesis of low-level details and high-level semantics—while robust low-level and coarse high-level features are less affected. 2. Proposed Method (CoTAM): Based on this analysis, the authors propose a codec, CoTAM, which features (a) a \"Shallow CLIP Guidance\" mechanism at the encoder to allocate more bits to semantically important regions (identified by shallow CLIP-layer attention) and (b) a \"Multi-level Fidelity Decoder\" that uses a lightweight adapter and a multi-level loss function (supervising both shallow and deep features) to protect the vulnerable cross-level representations. Experiments across a comprehensive suite of MLLM benchmarks (MME, TextVQA, POPE, etc.) show that CoTAM achieves significant bitrate savings (up to 36% BD-rate) over human-centric (ELIC) and machine-centric (Bridge-d1) codecs, all while maintaining MLLM task performance and incurring minimal computational overhead.", "questions": "Please see the weakness part. Overall, I find the paper satisfying and interesting. The paper would be further improved if the authors addressed the weaknesses above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper tackles a problem of high practical and academic importance. As MLLMs become central to many applications, developing codecs that are \"aware\" of their specific perceptual needs is a critical research direction.\n\n2. The core methodology is well-motivated. The idea of first analyzing the internal failure point of MLLMs under compression (the vulnerability of cross-level features) and then designing a codec (multi-level loss, semantic guidance) to specifically protect that failure point is logical and elegant.\n\n3. The experimental validation is a major strength. The authors evaluate on a wide and diverse set of MLLM benchmarks, which is essential for \"general-purpose\" MLLM compression. The reported BD-rate savings of ~36% (Table 2) are substantial and clearly demonstrate the method's effectiveness over strong, relevant baselines.", "weaknesses": "Despite its strengths, the paper has several weaknesses that challenge its claims.\n\n1. The \"three-stage information flow\" (Sec 2.2.1) is presented as a novel discovery. However, the U-shaped attention distance curve (Fig 5a/b) and the general progression from local, low-level features in shallow layers to global, semantic features in deep layers are well-established properties of Vision Transformers. This part of the analysis feels more like a re-characterization of known ViT properties in the context of compression, rather than a fundamental new insight into MLLMs.\n\n2. The 'Shallow CLIP Guidance' (Sec 3.2) is implementation-specific, creating a dependency that the paper doesn't fully resolve. While the authors evaluate on MLLMs with SigLIP and InternViT encoders , they only demonstrate that CLIP guidance is better than no guidance on a simple ablation. The appendix (A.8) merely shows that these encoders have similar U-shaped attention curves; it does not validate that CLIP's shallow attention is an optimal, or even a valid, importance map for these architecturally different encoders. The study would be significantly strengthened by testing a 'matched-encoder' guidance, such as using a SigLIP-guided map for the SigLIP-based MLLM.\"\n\n3. The guidance map is an 8x8 grid quantized to only three levels ($\\mu \\pm k\\sigma$). For a 336x336 input, each guidance block covers a 42x42 pixel patch. This seems incredibly coarse. It is difficult to believe this crude map can effectively guide bit allocation for tasks requiring fine-grained detail or small object recognition. The paper needs to provide much stronger justification or analysis for this coarse-grained approach. Maybe add some fine-grained results.\n\n4. I am confused by some experimental results. For example, Figures 7 and 8 show that Bridge d1 is even worse than  ELIC. Can the author explain how the method is tested and which encoder is used?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "swhmV762Am", "reviewer": "Reviewer_xBDN", "summary": "The authors address the problem of image compression for multimodal large language models (MLLMs) in a multi-task setting. They conduct a detailed analysis of how the image compression process impacts features at different levels and draw several interesting and meaningful conclusions. Building on these analyses, they design a new image compression architecture—based on existing work—that improves MLLM task performance.", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "Overall, the idea is interesting, the topic is timely, and the paper is generally well written and well structured.", "weaknesses": "I have several concerns regarding the depth and clarity of the analysis section:\n- The proposed three-stage theory is intriguing, but the justification appears primarily driven by the observation in Figure 5(b). Although both Figures 5(a) and 5(b) exhibit a U-shaped curve, only Figure 5(b) shows a small plateau at the beginning that supports the notion of three distinct stages. a) Please clarify whether this plateau is the main empirical evidence for defining the three stages. b) The caption of Figure 5 refers to “Average Max Attention Distance,” whereas the text mentions “Distance to the Most-Attended Token.” Are these the same metric, or do they differ? If they differ, please explain the relationship between them. c) Is the precise separation between stages (e.g., between Stage 1 and Stage 2) dependent on the model architecture or the dataset used? Providing a more rigorous or generalizable explanation would strengthen the argument.\n\n- More details should be given regarding the compression methods used. For example, for most compression algorithms—traditional or learning-based—using a fixed quality factor across different images typically results in different bitrates. a) In generating Figure 4, did the authors fix the quality parameter across images, or did they fix the bitrate to ensure comparability? b) The same question applies to subsequent experiments in the experiment section: were all methods evaluated under consistent bitrate conditions? c) Additionally, please clarify whether the observed phenomenon holds across different quality levels or if it appears only under specific compression strengths.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YZh8swHR7J", "reviewer": "Reviewer_g6n1", "summary": "This paper notes that cloud-hosted Multimodal Large Language Models (MLLMs) urgently need efficient compression for edge-device image/video transmission with low bandwidth. Conventional codecs, optimized for Human Visual System (HVS) fidelity, are ill-suited for MLLMs’ diverse downstream tasks. Through analysis, the authors find compression distortion unevenly affects image features, impacting MLLMs based on their feature reliance. They propose CoTAM, an MLLM-tailored codec: its encoder uses CLIP’s shallow attention for bit-allocation (preserving key semantics), while its decoder employs a lightweight adapter and multi-level loss to reconstruct low-level details and high-level context. Experiments show CoTAM cuts bitrate by up to 35.99% while maintaining MLLM performance, outperforming prior SOTA neural codecs.", "questions": "1. The clarification of $[CLS]$ attention scores may improve the score of the paper.\n\n2. The variety of MLLMs and Codecs may result in different analysis.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Clear and concise description of the proposed method.\n\n2. It depicts the influence of compression distortion on the performance of MLLM, and analyzes the change between original and compressed image as shown in Figure 5.\n\n3. It can achieve the better results within TextVQA, MME and VideoMME tasks.\n\n4. \"a disproportionate collapse of the cross-level representations that bridge low-level and high-level information\", a interesting observation and may be the key to further improve the performance of MLLMs.", "weaknesses": "1. LLM backbones maybe matter, but this manuscript doesn't compare different LLM backbones.\n\n2. The Equation about $D_{avg}$ and $D_{top1}$ lack the detailed explanation for $A, d and p_i$.\n\n3. The average of $[CLS]$ attention scores lacks the motivation, and the detailed calculation of average score is not provided. Besides the other tokens in CLIP may work well even better.\n\n4. The global attention of high-resolution images in Figure 6(c) is not appropriate since the ratio of downsample is not clear, and how to downsample the image is not to provided.\n\n5. Traditional Codec methods are not compared with the proposed method though the $Adapter$ in Figure 6(b) is not in these traditional methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)