Bridging Compressed Image Latents and Multimodal Large Language Models
Authors: Chia-Hao Kao
Cheng Chien
Yu-Jen Tseng
Yi-Hsin Chen
Alessandro Gnutti
Shao-Yuan Lo
Wen-Hsiao Peng
Riccardo Leonardi
Venue: ICLR 2025 Poster (2025)
Keywords: Multimodal Large Language Models
Neural Image Compression
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: This paper presents the first-ever study of adapting compressed image latents to suit the needs of downstream vision tasks that adopt Multimodal Large Language Models (MLLMs). MLLMs have extended the success of large language models to modalities (e.g. images) beyond text, but their billion scale hinders deployment on resource-constrained end devices. While cloud-hosted MLLMs could be available, transmitting raw, uncompressed images captured by end devices to the cloud requires an efficient image compression system. To address this, we focus on emerging neural image compression and propose a novel framework with a lightweight transform-neck and a surrogate loss to adapt compressed image latents for MLLM-based vision tasks. 
Given the huge scale of MLLMs, our framework excludes the entire downstream MLLM except part of its visual encoder from training our system. This stands out from most existing coding for machine approaches that involve downstream networks in training and thus could be impractical when the networks are MLLMs. The proposed framework is general in that it is applicable to various MLLMs, neural image codecs, and multiple application scenarios, where the neural image codec can be (1) pre-trained for human perception without updating, (2) fully updated for joint human and machine perception, or (3) fully updated for only machine perception. 
Extensive experiments on different neural image codecs and various MLLMs show that our method achieves great rate-accuracy performance with much less complexity.

Published as a conference paper at ICLR 2025
BRIDGING COMPRESSED IMAGE LATENTS AND MULTI- MODAL LARGE LANGUAGE MODELS
Chia-Hao Kao'
Cheng Chien Yu-Jen Tseng²
Yi-Hsin Chen? Alessandro Gnutti!
Shao-Yuse Lo
Wen-Hsiao Peng² Riccardo Leonardi' 'University of Brescia, Italy NNational Yang Ming Chiao Tung University, Taiwan
'Honda Research Institute USA
ABSTRACT
This paper presents the first-ever study of adapting compressed image latents to suit the needs of downstream vision tasks that adopt Multimodal Large Language Models (MLLMs). MLLMs have extended the success of large language models modalities (e.g. images) beyond text, but their billion scale hinders deploy- ment on resource-constrainge end devices. While cloud-hosted MLLMs could be available, transmitting raw, uncompressed images captured by end devices to the cloud requires an efficient image compression system. To address this, we focus on emerging neural image compression and propose novel framework with a lightweight transform-neck and surrogate loss to adapt compressed image latents for MLLM-based vision tasks. Given the huge scale of MLLMs, our framework ex- cludes the entire downstream MLLM except part of its visual encoder from training our system. This stands out from most existing coding for machine approaches that involve downstream networks in training and thus could be impractical when the networks are MILMss. The proposed framework is general in that it is applicable to various MLLMs, neural image codecs, and multiple application scenarios, where the neural image codec can be (1) pre-trained for human perception without updat- ing, (2) fully updated for joint human and machine perception, (3) fully updated for only machine perception. Extensive experiments on different neural image codecs and various MLLMs show that our method achieves great rate-aceuracy performance with much less complexity.
INTRODUCTION
Large Language Models (LLMs) Touvron et al. 2023b a) have demonstrated impressive abilities in various Natural Language Processing (NLP) tasks. Building upon their success, the recent surge of research on Multimodal Large Language Models (MLLMs) extends LLM's abilities to modalities beyond languages, particularly images, opening up promising opportunities in various applications (Achiam et al. Cha al. 22024 Lin et al. 2024 2024 Zhang et al. (2024)). MLLMs have shown surprising capability for many vision tasks such classification (Zhu et al. 2024b), image captioning (Zhang et al., 2024a). Visual Question Answering (VQA) (Cha et al. [2224] Lin et al. 2024). and meme interpretation (Achiam al., 2023). These models excel in unseen tasks through instruction following or in-context learning, which is impossible for traditional vision networks.
However, the massive scale of MLLMs, often comprising billions of parameters, poses significant challenges for deployment on snoource-constraind end devices. While computation can be offloaded the cloud, transmitting images dloud-hosted MLLMs becomes necessary. In this case, efficient image compression techniques are crucial to reducing the required transmission bit-rate. Without compression, transmitting raw images incurs significant costs, particularly scale with numerous users. Our study shows that simply feeding the decoded image, generated by a fixed image codec trained for human perception, into an MLLM (Figure (a)) substantially degrades task performance, particularly when the image is coded at low rates. This highlights the critical need for efficient image compression that considers the requirements of downstream MLLM-based vision tasks.
To the best of our knowledge, there have been no attempts tackle image compression specifically for MLLMs. While many prior works address image compression for machine vision, commonly referred to coding for machines (Le et al. 2021b Chamain al. 2021 Matsubara al. 2022

---

Published as a conference paper at ICLR 2025
(1) Reconstruction for human perception-
(d) Ours: Transform-neek & surrogate loss Task performance
Tisk performance O
Training fasibility
Training feasibility C
(dl) Human perception codec
MLLM
Encoder
Decoder
MLLM
Commectrr
Encoder
Trnnsoonttteck
Vissaal
LLM
.ixe...iseed.....
Reconstructed image
(b) Coding machines: Image cuding
Transforned Intents
Task performance
Training feasibility A
(d2) Multi-task codec
MLLM
Enooder
Decoder
MLLM MAA
Encoder
Transform-coe nock
Empockr
Connectur
LAM
Reconstrucied
Decoder
(c) Coding for machines: Feature coding
Task performance
Training feasibility
(d3) Machine perception codec
Reconstmucted image
MLLM
MLLM (1/2)
Encoder
Decoder
MLLM (22)
Eneeder
Transform neck
Emaage
Visual
Connecter
LIM
Reconstructed features
Transformed
ppoppppoooonation puth
Bitsrream
Figure I: On the left is inadequate frameworks for image compression for MLLMs, where the image codec is trained for (a) human perception, (b) the downstream task network, or (c) compressing the intermediate features of the task network. On the right is the proposed transform-neck and surrogate loss under three distinct scenarios, with the image codec (dl) pre-trained for human perception, (d2) updated for joint human and machine perception, or (d3) updated for machine perception.
Liu et al. 2022b), these approaches cannot be directly applied to MLLMs. Two common approaches to coding for machines are image coding and feature coding. The image coding approaches (Le et al.
2021bba) optimize the image codec for specific downstream tasks and/or networks (Figure (b)), while the feature coding approaches (Ding et al., 2024) divide the task network into two parts and focus on compressing the intermediate features (Figure (c)). However, both approaches face the same issue: the training process becomes challenging when one needs to back pack-propagate a training objective through a massive MLLM to train the neural image codec. In practice, the billion-scale parameters of MLLMs make the existing coding for machine methods inapplicable.
In this paper, we propose the first neural image compression system for ML.Le vision tasks that enables compressed latents to suit the needs of downstream MLLMs. Notably, it is not our objective develop new image codec specifically for MLLM-based tasks. Instead, our method involves a lightweight transform-nec, and a novel surrogate loss. The transform-neck adapts the compressed image latents of an off-the-sheil neural image codec to match the intermediate features of the visual encoder-i.e. a component of MLLMs-bypassing the needs for full image reconstruction and reducing computational complexity. Our proposed surrogate loss function, which combines the cross-entuee cross-entropy and distillation terms, enables our system to be trained by paok-propggg solely through the visual encoder, thus eliminating the need for back pack-propagation through the entire MLLM.
The proposed method is general in that it is applicable to different neural image codecs under various application scenarios. First, if the downstream applications prioritize the image reconstruction quality for human interaction, our method can work with an perception (Figure (d)).. Without any modification or re- re-training of the codec, our method adapts the compressed image latents while maintaining the same image reconstruction quality. Second, when allowing the image codec to be updated, we propose a multi-task training strategy that optimizes the codec for both human and machine perception (Figure D (d2)). This significantly improves MLLM performance at the cost of marginal drop in the image's 's reconstruction quality. Finally, we consider an extreme setting in which the applications prioritize machine perception over image reconstruction. In this case, the encoder and the transform- neck are jointly optimized for the MILM systems exclusively (Figure (d3)). The main contributions of this work are summarized as follows:
image codec trained for human
It marks the first exploration into the field of neural image coding for MLLMss
The proposed transform-neds neck adapts the compressed image latents to downstream MLLMs, avoid- ing the need for image reconstruction and thus saving computational complexity.
The proposed surrogate loss leverages the visual encoder to update the system, avoiding back- propagating the task loss through the heavy MLLM.
The proposed framework is broadly applicable to a wide range of neural image codecs and MLLMs, regardless of their architectures.
2

---

Published as a conference paper at ICLR 2025
It is able to accommodate various application scenarios that involve human perception, machine perception, or both.
Last but not least, the transform-ness trained with our surrogate loss exhibits degree of universality, since it is readily applicable to multiple MILMs that share the same visual encoder, without the need for retraining. Our method achieves (1) up to 60-80% bit-rate reductions under the same recognition accuracy over existing image codecs (e.g. ELIC (He et al., 2022) and VVC intra coding (Bross al. 202I)) (Sections 4.2 and A.2) and (2) nearly 95% reduction in decoding kMAC/pixel as compared to performing Tull image reconstruction followed by enhancing the reconstructed image for MLLM-based tasks (Section 4.4). Our system can be successfully trained under various application scenarios on one RTX 4090 with 24GB of memory. This is not possible when the entire MLLM is
involved the training process.
RELATED WORKS
2.1 MULTIMODAL LARGE LANGUAGE MODELS
In recent years, there has been surge of interest in MLLMs following the impressive demonstration of LLM's ability in the NLP field (Touvron et al.) 2023b at Diang al., (202)). Many have sought to
extend the success of these models from text to other modalities, particularly images, and several works have shown their effectiveness on various tasks, such as image captioning (Li et al. 2023b]
Lin et al. 2024 Liu et al. 2023a), VQA (Cha et al. 2024 Zhang ct al. 2024a). Referring Expres- sion Comprehension (REC) (Chen et al.
2023a: Peng et al. 2024: Zhang et al. 2024b). few-shot classification Yuu et al. 2024 Zhu et al. 2024b),
action anticipation (Mittal et al. 2024).
Most existing MLLM approaches use a visual encoder to process the input image data, and then introduce a connector to bridge the image features to the tokens understandable by the LLM. Earlier works adopt simpler connector designs, such as linear projectors (Chen et al., 2223a Liu et al. 2023a), while subsequent works (Li et al. 2023b Cha et al. 2024 Zhang al. (024a) have refined upon the design for both performance and complexity. Furthermore, the entire MLLM can be further
fine-tuned enhance its capabilities through instruction tuning (Liu al. 2023a Zhu al. 2024a).
A notable aspect of the MLLMs is their reliance on existing pre-trained visual encoders in their systems, with CLIP (Radford et al. 2021) visual encoder being a very common choice for large number of methods (Cha et al. 2024 Chen et al. 2023a Zhang et al. 2024a Zhu et al. 2024b Lin al. 2024: Li et al., 2003bb. Trained on large image-text pair data, the CLIP visual encoder offers the feature space that combines language and image modalities in a sense, making it a desirable feature for MI.LMs. Notably, all the existing works on MI.LMss do not consider the scenarios where image compression is present, which is significant departure from
work. We note that some approaches (Shi et al. 2024 Li et al. 2024) perform token reduction to minimize the inference cost of the downstream MLLMs. These techniques are orthogonal to and can be combined with our method (see Section A.6 for more discussions).
2.2 IMAGE CODING FOR MACHINES
Neural image compression systems have made significant progress in the past few years. As a matter of fact, several works (He et al. 2022: Liu et al. (002bb) have even outperformed the traditional codecs such as intra coding in Vrossss et al. 2021). However, these methods primarily focus on the quality of reconstructed images for human perception. Coding for machines, in contrast, targets downstream machine vision over human perception, and it has attracted increasing attention recently.
A common approach simply involves training the compression system for a predefined target down- stream computer vision task (Le et al.] 2021bba Wang et al. 2022), enabling the reconstructed image to be suitable for machine vision, albeit potentially sacrificing perceptual quality. Conversely, Chamain et al. (022I) tune the task network to better process the compressed images, while Chen al. (202bb) leverage prompt-tuning method on Transformer-basel codecs to boost performance on multiple tasks. Also, with the trend of the new JPEG AI learning- based image coding standard (As censo al. 2023), some methods (Liu et al.) 2022a 2021 Mei et al.; [221 (Singh et al. (0200) utilize the compressed image latents instead of the reconstructed image for recogmition through bridging the latents to task network. On the other hand, (Ding al. 2024) directly compress the intermediate
3

---

Published as a conference paper at ICLR 2025
Device
Image Transmission
Server Cloud
Input Imagge
Neural Image Codee
Recunstructed Image
& Human
Doosder
Prompt
Transform-ncck
Multimodal Large Language
Prompt
Vieaa Inceeer
Connecto
LLM
Assistani #1
Cokeful parno
this image.
Litess Literrooistie Proistion
Layor Noeaaitation
Sat
Fooadd
Vieua Lnceder
Connector
LLMI
perched on brunch.
Prompt
Assistant #2
Whst the beanch?? but box hind]
Connecta:
Answer: C
Prompt a3
=>
LIM
Assistami #3
Can you provide coondinates the parot??
[0.33%, 0.677, 0.940)
Text Transmission
Viaua Eacader
Figure 2: Overall architecture of the proposed method.
features of recognition networks, while (Feng et al. (2022) learn the omnipotent features suitable for various tasks in a supervvse manner and fine- fine-tune each task network tail on such features.
It is crucial to note that none of the coding for machine methods considers MLLMs at the receiver side. All the above-mentioned methods leverage back proppapaaagtioon through recognition models to update the system or even re- re-train the recognition network itself, both of which are prohibitively expensive for MLLMs due to their huge scale. Therefore, the direct application of the same methods on MLLMs is almost infeasible. In addition, mainstream image coding for machines methods (e.g. Chen et al. (2023b); Ascenso et al. (203)) remain mostly task-specific. They typically adopt a task-based loss, which restricts the resulting models to be optimized for single task and recognition model, thus requiring re-training for each new task and incurring additional costs. We aim to be the first to propose neural image compression system designed for MILLMs, achieved through a lightweight transform-neck and surrogate loss, which bypasses the necessity of involving the entire billion-scale MLLM in the training process. Moreover, our surrogate loss incorporates cross-entropy loss to bridge visual features with the text domain for MLLMs, complementing the ceauue-consttaaning distillation loss. This combination further differentiates our approach from existing methods.
PROPOSED METHOD
3.1 PRELIMINARIES: NEURAL IMAGE CODECS
The high-level architecture of a neural image codec is depicted in the top central green box in Figure In a typical hyperprior based neural image codec (Ballé et al. 2018), the key components include the main encoder 9a. the main decoder gs. as well the hyperprior encoder h2 and decoder h Given an RGB image x E IR H>W where H and W represent the height and width of the image, respectively, Ia performs the analysis transform of x: and generates the image latent representation E R # first uniformly quantized
with N' indicating the channel size. To transmit y more efficiently, into ý and then entropy coded considering learned prior distribution p()). This learned distribution content dependent, thanks to the hyperprior encoder it, and decoder hg. In particular, ht, takes y as input and produces the side information 2 E N x # x क,, that is used to generate the learned distribution for entropy coding, where N₁. is the channel size of the side information. The quantized version of z, denoted as é, is transmitted into the bitstream, in order to recover ÿ. Lastly, á undergoes the synthesis transform with ga. which reconstructs the image RaxHxW
3.2 OVERALL FRAMEWORK
In this work, we focus on the scenario where MLLMs are hosted on the server side, while users on end devices need to perform inference on the remote model using both text and images as inputs. propose
Given the necessity of incorporating image compression to ensure efficient transmission, the first compression framework with the consideration of MLLMs as downstream application networks, aiming to mitigate the potential task performance drop caused by image compression.
4

---

Published as a conference paper at ICLR 2025
Figure 2 iillustrates our overall framework, which includes three major components: the neural image codec, our proposed transform-neck and the MLLM. The depicted MLLM system adheres to a typical structure, consisting of visual encoder, LLM, and connector component facilitating the transformation of features from the visual encoder to the LLM. Note that all the MILMMs are adopted off-the-shelf and without any update.
During inference, the input image at the end device is passed through an encoder Ia to generate the quantized latents for transmission. Next, ÿ is directly passed through a lightweight transform-neck T for transformation into middle layer of the visual encoder of an MLLM. We opt to adapt the image latents rather than the reconstructed images because the image latents inherently contain the information needed for reconstructing the image, and potentially the semantic information for the downstream tasks (when the image encoder is guided properly). By skipping the image decoding process, our method offers reduced computational complexity while maintaining the task performance. The rest of the MILM system operates without any changes to generate the desired output response.
In training, to address the challenge of back-propaggee propagating the task loss through the entire MLLM, we propose novel surrogate loss that updates the system by back proppaarpgg solely from the visual encoder (which is not re-trained), bypassing the iillion-paramete LLM. In our work, we examine three distinct settings denoted as (dl), (d2) and (d3), as illustrated in Figure [T Firstly, in (dI), consider the practical scenario where fixed off-the-shelf image codec pre-trained for human perception is directly used alongside our transform-neck. In this setting, our framework offers the option for users decode the image latents ÿ for reconstruction by using the decoder 9, instead of the transform-nnek. In this way, the quality of the decoded image is not affected, as the image codec is not updated in the present case. Then, we extend the analysis to scenarios (d2) and (d3) to examine the impact of jointly training the image codec and transform-nnck. In (d2), the entire image codec undergoes re-training to accommodate both human and machine perception, while (d3), the encoder is re-trained specifically for machine perception.
3.3 TRANSFORM-NECK
Our transform-nen is designed to be a lightweight module, consisting only of a linear projection, a self-attention mechanism, feed-forward layer, and two layer norms, shown in the central red box in Figure 2 Its purpose is to adapt the compressed image latents ý into an efficient representation for consumption by the downstream MILMMs. In fact, rather than reconstructing the image and using it as input to the MILM, we propose leveraging the latent representation directly.
Since the image encoder 9a already functions as a feature extractor, similar to the early layers of the visual encoder C, we bridge the output of our transform-neck directly into the intermediate features of C. effectively integrating the image codec with the MLLM system. The decision on which initial layers to bypass depends on the specific type of visual encoder used in the MLLM. For instance, when using the CLIP visual encoder as C, we found that connecting the transform-neck to the third Transformer layer, bypassing the first two, yields optimal results (see Section 4.5 for ablation experiments justifying this design). Note that skipping the initial layers of further reduces computational complexity of our framework. We denote the partial visual encoder C².
3.4 SURROGATE LOSS
To avoid involving huge MILMs in the training process, and thus bypassing back-propan proppppppanttoon through their entire structure, propose surrogate loss Ls which is back-propagated through only the partial visual encoder C". Specifically, we design our surrogate loss to consist of two terms: distillation loss Ldiat and cross-entropy loss LCg.
To retain the downstream MLLM performance, the resulting features C'(T(i)) when using our transformed latents should resemble closely those obtained when inputting the uncompressed image into C, that is C(r). To this end, we introduce the following distillation loss Ldist. aiming to minimize the Mean Squared Error (MSE) between the two output features:
Caist MSE(C" (T()), C(z)).
In addition to the distillation loss, we propose incorporating a second term, which is the cross-entropy loss LoE calculated using the following procedure. We take classification dataset consisting of
5

---

Published as a conference paper at ICLR 2025
Table I: Application scenarios for our method with corresponding training objective.
Update Image Codec
Human Viewing
Phase Loss Function
Phase Loss Function
Application Scenario
(dl) Human perception
X
Ls
(d2) Multi-task
Ls
R X(rddmccon (x, 8) + &Laist)
(d3) Machine perception
X
Ls
R + XLdiat
images and text labels {1,, 12..... Im which correspond to distinct classes. Each image belongs to one of these m classes, Next, we compute: (I) the n image embedding C' (ū)) and (2) the text embeddings Ce2), where C₃ is text encoder applied to each text label 1j. In particular, we use the CLIP text encoder, independently of the visual encoder integrated into the MLLM under consideration. For each image, embedding C" (T())) and each of the m. text embeddings C,(1,) ) for 1,, ...., m. This produces vector of similarity scores between the given image and all the m classes, that is:
then calculate the cosine similarity C:(:,) between its image
= CCS(CC (T(C) C.(4),, .... CS(CC (T()), (T()))
The resulting similarity vector is transformed into probability distribution over the m classes using softmax function. Finally, the cross-entropy loss is computed with respect the corresponding one-hot encoded ground truth label vector t, thus:
LcB = (Sooftmaxx)))
This approach aims to bridge the visual features to the text domain specifically for MLLM-based vision tasks, distinguishing our method from existing works in the field of coding for machines. We validate the effectiveness of our loss function design in the ablation study (Section 4.5
3.5 TRAINING PROCEDURE
To explore the capabilities of our method under the application scenarios introduced in Section 3.2 employ a two-phase training procedure that adapts different scenarios. The first phase is shared for all scenarios and focuses on training the transform-neck exclusively. The second phase, applicable only to scenarios where the codec can be updated, involves joint optimization of both the transform-neck and the image codec. Table I provides a summary of the training procedure.
Phase 1: Transform-neck Training In the first phase, we train only the transform-neck using progressive training strategy with the surrogate loss Ls and an off-the- image codec. This phase divided into three stages:
LCE.
epoch < Ei. E, S epoch < E2, epoch 2 Es,
Ls = uLcr + BLiists
Caisrr
where the weighting factors a and are set with ratio of 1:100 for the two loss terms, and E,, Es are empirically set to 20 and 40, respectively, in our experiments. This initial phase ensures that the transform-neck learns the transformation align with the target latent space.
Phase 2: Joint Optimization For (d2) and (d3), where the image codec is allowed to be re-trained produce latent representation more suitable for machine perception, the second phase is introduced with rrackform-neck and image codec jointly updated after phase converges.
The scenario (d2), referred as multi maski aims to accommodate both human and machine perception. As result, it is trained jointly with the transform-neck on both the distillation loss and traditional rate-distortion loss, i.e. L22 R + X(7drreonn (a., .) + &Ldist), where R = log p () log p(.) 8) the estimated rate of 9 and é, and drecon is the reconstruction loss calculated the MSE between the uncompressed image æ and the reconstructed image ž. The papre-parreeter controls the rate-distortion trade-off. while y and 5 weight the two losses.
In (d3), where the downstream applications do not require image reconstruction, the encoder and transform-neck jointly optimized to minimize the trade-off cost between the rate R and the distillation loss Ldists thus disregarding the reconstruction quality. The training objective is thus Las R + MLtist-

---

Published as a conference paper at ICLR 2025
Captioning
VQA
REC
Few-shot classification
120
64
0.86
96
63-
95
0.82
CDD
Seare
Accurccy
Accuracy
144
61
100
0.78
60
92
90
59
0.74
91
90
80
0.2
0.1
0.2
0.1
0.2
0.1
Bil-rate (bpp)
Bii-rate (bpp)
Bil-ratc (hpp)
Bit-rate (bpp)
Uncomppessed
Reconstraction
Post-processing
Ours (dl)
Our (d2)
(d3)
Figure 3: Rate-aceuracy comparison using various MLLMs on several tasks.
Table 2: Evaluated tasks with corresponding dataset and MILM.
Task
Dataset
MLLM
COCO Karpathy Test
LLaMA AaeiAA-dappa dapter
Captioning
Ous (dl)
Karpathy & Fei-Fei
2015
Zhang et al.
2024a
Oan (d2)
BenD-
Honeybee
Oars (d5)
VQA
Li al.
2023a
(Cha et al.
2024
RefCOCO-val
Shikra
REC
Kazemzadeh et al.
2014)
(Chen et al.
2023a
Fhw-shot classification
ImageNet
Tokenzeer Zhu et al.
Figure 4: Reconstruction perfor-
(bpp)
Deng et al.
2009
2024b
mance comparison on Kodak
EXPERIMENTAL RESULTS
4.1
EXPERIMENTAL SETTING
Training Details and Datasets. We utilize ELIC (He et al. 2022 as our image codec, which outputs image and hyperprior latents with N 320 and N2, = 192, respectively. ELIC is trained for human perception and adheres to the training strategy outlined in He al. (2022), using 8,000 images of the highest spatial resolution selected from ImageNet dataset. Four models are trained for four different rate points, corresponding À [0.004, 0.008, 0.016, 0.032] in (He al. 2022). For each of our scenarios (d1), (d2) and (d3), separate transform-need are trained on ImageNet dataset (Deng al., 2009) for individual values. For the scenario (d2) specifically, the ratio y ā = 60 leads to a good trade-off between human and machine perception. Given that most MLLMs adopt the pre-trained visual encoder of CLIP ViT-L/14 (Radford al. 2021) for image modality, as discussed in Section 2.1 we use the CLIP visual encoder as C' for training and conduct our primary experiments on MLLMs that incorporate
find empirically that fixing
It is worth noting that, since we consider MLLMs sharing the same visual encoder, we do not need to train separate systems for the different MILMS or tasks. Furthermore, we provide additional experiments in Section 4.6 with MLLMs that use visual encoders other than CLIP ViT to demonstrate the generalizability of our approach.
Targeted MLLM-based Vision Tasks. To validate the generalization ability of our proposed method, evaluate its performance four different MLLM systems for four different tasks. The tasks, datasets, corresponding MILMs, and metrics are listed in Table These configurations follow the settings outlined their original papers and the accompanying code, except for the few-shot classification task due to the inaccessibility of the code. We thus design a 5-way I-shot classification scenario evaluate the performance with in-context learning; the detailed setting described in supplementary material. All MILMS are used off-the-shelf without any fine-tuning.
Baselines. We introduce two baseline methods for comparison. The first one, denoted as Re- construction, involves inputting the reconstructed image generated by ELIC the MILLM system.
7

---

Published as a conference paper at ICLR 2025
Uncouupressed
Reconstrneted
Post-proeessed
LLaMA A-MA--daaptr
Captioning
BPP: 0.0958
Reconstruction: A man is walking an elephant down path. Post-processing A man feeding an elephant with his hand.
Ours (dl): A man is petting an clephant on the head.
I'm trying locate man with mask in <img>, Can you determine its coordinates for me?
Uncompressed
Reconstracted
Post-pracessed
BPP:0.061
REC Shikra
Cur
Figure Visualization examples of our proposed method in (d1), Reconstruction, and Post Post-procsssing on image captioning with LLaMA-Adapter and REC with Shikra.
The second one, denoted as Post-processing, adapts the reconstructed image to MLLMs through U-Net (Ronneberger et al. 2015) post-processing network, which is trained using the same surrogate loss as that adopted by our method. We remark that these image-domain baselines incur higher complexity than our lightweight transform- -ncck, as they involve decoding the image and potentially processing it further with the post-processing network.
4.2 PERFORMANCE COMPARISON
Figure 3 illustrates the performance of the baseline methods and our proposed scheme for the three examined scenarios with regards to two aspects: coding bit-rate, calculated as bits per pixel (bpp), and task performance. When comparing the baselines and our method scenario (d1), where the original ELIC is trained solely for human perception, we make the following observations. (1) Straightforwardly using the reconstructed images generated by codec trained for human perception leads to significant performance drop across all the tasks (Reconstruction). Such performance decline expected because the MLLMs are not trained with compressed images, thus hindering their recognition performance. This highlights the necessity of adapting image compression and/or image latents to MLLMs. (2) In contrast, our transform-neet method successfully boosts the performance using the same latent representations for reconstructing the image in Reconstruction, confirming the effectiveness of the proposed latent transformation without the decoding process. (3) Post-processing able reach comparable performance our (dl), offering another viable solution to the problem. However, it is worth noting that Post-processing requires relatively higher computational complexity with respect to our transform-neck method, rendering our approach preferable (see Section 4.4).
Next, we evaluate the effects of allowing the image codec to be re-trained. First, we observe that (d2) outperforms both (dI) and Post- procesrc.sssng. This indicates that fine-tuning the encoder indeed results in more suitable latent representation that can be better adapted to MLLMs. When examining the extreme setting (d3), we see significant further improvement in the task performance, approaching the performance upper bound with uncompressed images. This improvement comes at the cost of the image reconstruction quality, which, however, not relevant (d3). Figure 4 illustrates the rate-visual quality curves associated with the three scenarios. Interestingly, (d2) exhibits only marginal PSNR drop compared to (dl), while (d3) significantly compromises the quality of the decoded image. We stress that framework (i.e. the surrogate loss and transform (eekk) is able to accommodate different application scenarios, allowing for a variable trade-off between the task performance and the image reconstruction quality.
8

---

Published as a conference paper at ICLR 2025
Table Comparison of the KMACs/pixel and model size. The table omits the shared components of the two methods, i.e the image encoder, the partial CLIP visual encoder, the connector, and the LLM.
Method
Component
Params (M)
kMAC/pixel
Ours (d1, d2, or d3)
Transform-neck
13.19
52.795
Decoder
7.34
112.00
64.
1017.96 (+1828% %)
Post-processing
Post-processing network
31.04
835.72
(+386% %)
First 2 layers of CLIP visual encoder
25.78
70.24
110
110
CDD
CIDE
Utsompress
Ulscomppres
100
Recomeracioon
Recomitaation
Uncompeess
90
Oves (d1) fremove
Remove
40
CrossFintopy loss only
ELIC,
ELIC, Reconstraction
loss only
Rumove
TLC, Reconscuction
Ressove
20
50
TIC: (dl)
0.15
0.20
0.25
0.10
0.20
0.20
0.25
Die (bpp)
Bib mittraateeee
(a) Partial CLIP visual encoder
(b) Training objectives
(c) Different image codees
Figure 6: Rate-aceuracy comparison for three ablation studies evaluated with image captioning task.
4.3 VISUAIIZATION OF THE RESULTS
We present the visualization outcomes with downstream MLLM-based vision tasks Figure Our method (d1) is compared with the two baseline methods, Reconstruction and Post-processine with particular focus on how these models work at low bitrates to reflect bandwidth-ling limited scenario. In the second and third columns, we visualize the reconstructed and post-processed images from the two baselines, respectively, which exhibit drastically different characteristies. The former (Reconstruction) produces blurry and smooth images, while patterns into the post-pree posteproceseed images. Compared with the baselines. our method yields better MILMM results. More visualization are presented in Section A.3 of the supplementary material.
the latter (Post-processing) introduces some artificial
4.4 COMPLEXITY ANALYSIS
Table 3 compares the computational complexity between our proposed method and Post-processing baseline terms of model size and the kilo-multiply aaaliipl-accumuuatt--opprrrttt
per pixel (kMACs/pixel). Note that our method in Table 3 refers to any of (d1), (d2), and (d3), since they share the same computational complexity characteristics inference time. Our method offers lightweight solution with only 13 million parameters, as opposed to 64 million parameters with the post-processing approach. Moreover, in terms of KMAC/pixel, the difference stands out even more, considering that the post-processing network operates at the full image resolution while our method operates in the latent domain, where the image latents have much smaller spatial resolution.
4.5 ABLATION STUDIES
The following ablation experiments are performed based on (dl) justify our design choices.
Training Objective. Figure 6 (b) presents the performance of our method when trained exclusively with the cross-entropy loss distillation loss. observed that training with only the cross-entropy loss results in a significant performance drop. Although providing good initial update direction, this strategy is unable to learn an effective transformation for MLLMs. Instead, training solely with the distillation loss fails to update the transform-nech properly and leads to far inferior performance. This is potentially due to the stringent requirement of fitting the exact feature representations. Our proposed method, not simple application of distillation, achieves the highest performance.
To further support this finding, we present the following analysis in Figure 7 we first calculate the mean squared error (MSE) between CLIP visual encoder output tokens derived from uncompressed

---

Published as a conference paper at ICLR 2025
Image
Cruss-cnrrppy
Distillation
Ours
Image
Cross-enropy
Distillation
Ours
055
Figure 7: Visualization of MSE reduction on CLIP visual encoder output tokens before and after training using different loss functions. Darker red colors indicate greater MSE reduction.
mPPlug-Ow12 Owwl MMbench
Ospeey POPE (popular)
0.85
Uncompressed
Accuracy
Reconstruction
Score
64
0
0.80
(dl)
(d2)
0.75
Our (d3)
0.025
0.050
0.075
100)
0.125
0.05
0.10
0.15
0.20
0.25
(bpp)
Bit-rate (bpp)
Figure 8: Rate-accuracy comparison on two MILMMs not utilizing CLIP ViT visual encoder: mPI.UG-
Ow12 Ye et al. 2024) MMBBnch and Osprey Yuan et al. 2024) POPE (popular setting).
images and from our transform-nest neck, both before and after the transform- has been trained. Then, we compute the difference between these MSEs measure the improvement achieved by the specific training objectives all with equal training steps. Figure shows that the cross-entropy loss reduces feature matching errors primarily in foreground object regions, while the distillation loss reduces global matching errors. Our proposed progressive training strategy integrates these two losses, leading a greater reduction MSE and thus improved rate-accuracy performance.
Partial CLIP Visual Encoder. This experiment investigates the proper number of Transformer layers to remove from the CLIP visual encoder in order to strike good balance between complexity and performance. As shown in Figure O performance, whereas removing four or eight layers results in a noticeable performance drop. We thus remove the first two layers.
(a), removing the first one two layers achieves similar
Different Image Codecs. Figure (c) presents the performance comparison between our method and Reconstruction when they are tested with ELIC and TIC (Lu et al. 2022a b). TIC is a Transformer- based codec, whereas ELIC is convolutional neural network-basse codec. We see that our transform- neck still outperforms Reconstruction by significant margin when the image codec is changed from ELIC to TIC. This indicates that our method still effective on a different type of image codec.
4.6 GENERALIZATION
While we utilize the CLIP ViT visual encoder in our main experiments due to its wide popularity, our proposed method is applicable various downstream MILMMs regardless of the visual encoder they adopt. As illustrative examples, Figure presents the rate-accuracy performance of our re-trained scheme applied two MLLMs that do use the pre-trained CLIP VIT visual encoder: (1) mPlug- Owl2 (Ye et al. 2024) with a mustom-trainee ViT visual encoder, and (2) Osprey (Yuan et al. 2024) with CLIP ConvNeXt-basse visual encoder. Our method under all three settings clearly outperforms the Reconstruction baseline, confirming the generalizability of the proposed framework.
CONCLUSION
This paper proposes the first image compression system tailored to Multimodal Large Language Models (MLLMs). It introduces a transform-neck that bridges the compressed image latents and the intermediate layer of the visual encoder. By using our proposed surrogate loss, we avoid involving the entire MILM in the training process and ensure downstream task performance. With lower computational complexity, our method has demonstrated effecttivneess across a wide variety of tasks, MLLMs, and neural image codecs, outperforming other baselines in extensive experiments. One consideration that this paper focuses solely on the image compression aspect, leaving the exploration of MLL.M-based video audio coding for future work.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work is supported by National Science and Technology Council, Taiwan under the Grant NSTC 23--263--F-A4-007- MediaTek, and National Center for High-ppeformance Computing, Taiwan.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Joao Ascenso, Elena Alshina, and Touradj Ebrahimi. The jpeg ai standard: Providing efficient human and machine visual data consumption. Ieee Multimedia,
2023.
Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. Variational 2018.
image compression with scale hyperprior. arXiv preprint
Benjamin Bross, Yui Wang, Yan Ye, Shan Liu, Jianle Chen, Gary J Sullivan, and Jens- lens-Rainer Ohm. Overview of the versatile video coding (vvc) standard and its applications. IEEE Transactions on Circuits and Systems for Video Technology, 31 (110:33736-3766 2021.
Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. Honeybee: Locality-enhancer projector for multimodal 1lm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2024.
Lahiru D Chamain, Fabien Racapé, Jean Bégaint, Akshay Pushparaja, and Simon Feltman. End-to- end optimized image compression for machines, a study, In 2021 Data Compression Conference (DCC), pp. 163-172. IEEE, 2021.
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal I1m's 's referential dialogue magic. arXiv preprint arXiv:2306. 15195, 2023a.
Yi-Hsin Chen, Ying-Chieh Weng, Chia-Hao Kao, Cheng Chien, Wei-Chen Chiu, and Wen-Hsiao Peng. Transtic: Transferring transformer- based image compression from human perception to machine perception. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 23297-23307, 2023b.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna Imsys. org (accessed 14 April 2023), 2(3);6,22 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255 225. Icee, 2009.
Ding Ding, Zhenzhong Chen, Zizheng Liu, Xiaozhong Xu, and Shan Liu. Hierarchical image feature compression for machines via feature sparsity learning. IEEE Signal Processing Letters, 2024.
Ruoyu Feng, Xin Jin, Zongyu Guo, Runsen Feng, Yixin Gao, Tianyu He, Zhizheng Zhang, Simeng Sun, and Zhibo Chen. Image coding for machines with omnipotent feature learning. In Computer Aision-ECCY 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings. Part XXXVII, pp. 510-528. Springer, 2022.
Dailan He, Ziming Yang, Weikun Peng, Rui Ma, Hongwei Qin, and Yan Wang. Elie: Efficient learned image compression with unevenly grouped space-channel contextual adaptive coding. In Proceedings of the IEEE/CVF Conference Computer Vision and Putlern Recognition, pp.
5718-5722 2022.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.0682. 2023.

---

Published as a conference paper at ICLR 2025
Andrej Karpathy and Li Fei-Fei. Deep visual-semante alignments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2015.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 787-798, 2014.
Eastman Kodak. Kodak lossless true color image suite (PhotoCD PCD0992). http::/:://00
us/ grappiccs/ooaa
Nam Le, Honglei Zhang. Francesco Cricri, Ramin YouuavviY,.ualari,, and Esa Rahtu. Image coding for machines: end-to-end learned approach. In ICASSP 2-2/- IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1590-1594. IEEE, 2021a.
Nam Le, Honglei Zhang, Francesco Cricri, Ramin Ghaznavi-Youvallan, Hamed Rezazadegan Tavakoli, and Esa Rahtu. Learned image coding for machines: A content-adaptive approach. In 2021 IEEE International Conference on Multimedia and Expo (ICME), pp. 1-6. IEFF, 2021b.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Bench- marking multimodal Ilms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023b.
Wentong Li, Yuqian Yuan, Jian Liu, Dongqi Tang, Song Wang, Jie Qin, Jianke Zhu, and Lei Zhang, Tokenpacker: Efficient visual projector for multimodal 1lm. arXiv preprint arXiv:2407.02:3 02392, 2024.
Yifan Li, Yifan Du Kun Zhou, Jinpeng Wang. Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023c. URL htt.http//// /owpr/vveeeeeeeeeeeeeeee www.wwiwwwww.................... net forum? ? did=
xozJwOkZXFF
Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings. Part V 13, pp. 740-755. Springer, 2014.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Conference on Neural Information Processing Systems (NeurIPS), 2023a.
Jinming Liu, Heming Sun, and Jiro Katto. Learning in compressed domain for faster machine vision tasks. In 2021 International Conference on Visual Communications and Image Processing (VCIP), pp. 01-05 2021. doi: (0.1009/CI152222221..996765
Jinming Liu, Heming Sun, and Jiro Katto. Improving multiple machine vision tasks in the compressed domain. In 2022 26th International Conference on Pattern Recognition (ICPR), pp. 331-337. IEEE, 2022a.
Jinming Liu, Heming Sun, and Jiro Katto. Improving multiple machine vision tasks in the compressed domain. In 2022 26th International Conference on Pattern Recognition (ICPR), pp. 331-337. IEEE, 2022b.
Jinming Liu, Heming Sun, and Jiro Katto. Learned image compression with mixed transformer-cma architectures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18388-14397 2023b.
12

---

Published as a conference paper at ICLR 2025
Ming Lu, Fangdong Chen, Shiliang Pu, and Zhan Ma. High-efficiency lossy image coding through adaptive neighborhood information aggregation. arXiv preprint arXiv:2204.11445 2022a.
Ming Lu, Peiyao Guo, Huiqing Shi, Chuntong Cao, and Zhan Ma. Transformer-based image compression. In Data Compression Conference, 2022b.
Yoshitomo Matsubara, Ruihan Yang, Marco Levorato, and Stephan Mandt. Supervised compression for desource-constrainee edge computing systems. In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, pp. 2685 (695-2695,, 2022.
Yixin Mei, Fan Li, Li Li, and Zhu Li. Learn compression for objection detection vac with bridge. In 202/ International Conference on Visual Communications and Image Processing (VCIP), pp. 2101:22422221..6277.
1-5, 2021. doi: 10.1
Himangi Mittal, Nakul Agarwal, Shao-Yurr Lo, and Kwonjoon Lee. Can't make an omelette without breaking some eggs: Plausible action. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuminng Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. In International Conference on Learning Representations (ICLR), 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. 8748-872 8763. PMLR, 2021.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted Citeventtion-MICCA 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pp. 234-241, Springer, 2015.
Dachuan Shi, Chaofan Tao, Anyi Rao, Zhendong Yang, Chun Yuan, and Jiaqi Wang. Crossget: Cross-guided ensemble of tokens for accelerating vision-language transformers. In International Conference on Machine Learning (ICML), 2024.
Saurabh Singh, Sami Abu-El-Haija, Nick Johnston, Johannes Ballé, Abhinav Shrivastava, and George Toderici. End-to-end learning of compressible features. In 2020 IEEE International Conference on Image Processing (ICIP), pp. 3349-3353. IEEE, 2020.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprini arXiv:2302. /3971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Ilama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. 2023b.
Shurun Wang, Zhao Wang, Shiqi Wang, and Yan Ye. Deep image compression towards machine vision: A unified optimization framework. IEEE Transactions Circuits and Systems for Video Technology, 2022.
Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. mplug-ow12: Revolutionizing multi-modal large language model with modality collaboration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13040-13051, 2024.
Lijun Yu, Yong Cheng, Zhiruo Wang, Vivek Kumar, Wolfgang Macherey, Yanping Huang. David Ross, Irfan Essa, Yonatan Bisk, Ming-Hsuan multimodal generation with frozen lmms. Advances in Neural Information Processing Systems, 36, 2024.
Hsuan Yang, et al. Spae: Semantic pyramid autoencoder for
13

---

Published as a conference paper at ICLR 2025
Yugian Yuan, Wentong Li, Jian Liu, Dongqi Tang, Xinjie Luo, Chi Qin, Lei Zhang, and Jianke Zhu. Osprey: Pixel understanding with visual instruction tuning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 28202-28211, 2024.
Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. In International Conference on Learning Representations (ICLR), 2024a.
Yichi Zhang, Ziqiao Ma, Xiaofeng Gao, Suhaila Shakiah, Qiaozi Gao, and Joyce Chai. Groundhog: Grounding large language models to holistic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2024b.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. In International Conference on Learning Representations (ICLR), 2024a.
Lei Zhu, Fangyun Wei, and Yanye Lu. Beyond text: Frozen large language models in visual signal comprehension. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2024b.
14

---

Published as a conference paper at ICLR 2025
A SUPPLEMENTARY MATERIAL
A.1 IMPLEMENTATION DETAILS
Training. We use the Adam optimizer, configured with 3i at 0.9, 92 at 0.999, at 10-*, Weight decay is disabled. The transform-nece neck for each rate point undergoes training on RTX 4090 for approximately three days during the training stage.
Evaluation. For few-shot classification with V2L- V2k-Tokenizer (Zhu et al. 2024b). we design a 5-way 1-shot classification evaluation scenario. In particular, we generate 5000 groups of images from ImageNet dataset, where each group consists of five randomly sampled images from different classes, serving as the sample images, and one new image from one of the classes as the query image.
Different MILM is utilized for the evaluation of our proposed method on each task. In Table 4 we provide the detailed specifications the MLLM used our evaluation.
Table 4: The specifications of the MLLM used in our tasks.
Task
Model
LLM
Captioning VQA REC
LLaMA-Adapter vl (Zhang et al 2024a)
LLaMA-7B
(Tourron
2023a
Beneeybee-C7BBMM. (Cha el al.
2024
Vicuna-7B
CChang el al.
2023
Shikra-7B (Chen et al. 2023a
LLaMA-7B
[Tovvron al.
2023a
Few-shot classification
V2L-Tokenizer (Zhu 2024b)
LLaMA2-7B
Touvron al.
2023b
A.2 COMPARISON WITH VVC
Figure compares Reconstruction and our method in (d1) using ELIC, with the state-of state-of-the-ar -he- traditional codec VVC (VTM 17.0 intra coding). We set the QPs to (37, 40, 43, 46, 49) for VVC. It observed that VVC performs worse than Reconstruction across the tasks, which is potentially due to (1) the small spatial resolution (256x256) of input images that is not optimal for VVC, (2) its inferior rate-distortion performance compared to ELIC as reported in (He et 2022)
Captioning
VQA
65
120
100
CDDR
55
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.10
0.05
0.10
0.15
0,20
0,25
0.30
Bil-rate (bpp)
Bit-rate (bpp)
RPC
Fow-shot classification
95
Accuncyy
0.7
85
0.6
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.05
0.10
0.15
0.30
0.25
0.30
0.35
Bit-rate (bpp)
(bpp)
Uncompressed
Resomstracion
Ours (c1)
VTM
Figure 9: Rate-accuracy comparison using VTM several tasks.
15

---

Published as a conference paper at ICLR 2025
A.3 MORE VISUALIZATION
We present additional visualization results on four different evaluation tasks, including image caption- ing (Figure 12,, visual question answering (VQA) (Figure 13 referring expression comprehension
(REC) (Figure 14). and few-shot classification (Figure 15
A.4 LICENSE OF ASSETS USED
Table summarizes the used assets in our work along with their license terms.
Table 5: List of assets used in the paper with their corresponding license.
Assets
Licenses
ImageNet Deng ct al.
2009
Custom license. Available at http://mmgeenneeorrddddvaaaaaaa
COCO et al. (2014)
CC BY 4.0
SEED-Benchi et al.
2023a)
Apache 2.0
LdaMeeAdapter Zhane el al.
2024a)
GPL-3.0
Source code: Apache 2.0
Honeyhee Cha al. (2024)
Pretrained weights: CC BY-NC
Shikra Chen al. 2023a)
CC BY-NC 4.0
No license provided.
V2L-Tokenizan Zhu et al.
2024b
Code available at http://hiihbbbccom/////444440000
A.5
ADDITIONAL PERFORMANCE COMPARISON
Figure 10 presents an additional performance comparison of our proposed method on the same task (POPE benchmark (Li al. 2023c)) with two different MILMMs, Honeybee (Cha al. 2024) and
Shikra (Chen et al. 2023a)
Honrybee
Shikra
0.84
0.82
0.82
0.80
Accurayy
Accurccy
0.80
0.78
0.76
0.74
0.72
Bit-rate (bpp)
Bit-rate (hpp)
Unsompressed
(dl)
(d2)
(d3)
Figure 10: Performance comparison on POPE benchmark
Li al. 2023c) with different MLLMs.
A.6 COMPARISON WITH TOKEN REDUCTION METHODS
Token reduction methods (Shi et al. 2024 Li et al. 2024) aim to reduce the number of visual tokens for lowering the inference computational cost MLL.Ms. These methods differ fundamentally from method as they do not consider the transmission of visual tokens in compressed form. In contrast, our work encodes and transmits the image in compressed form, where the compressed image latents adapted to suit the downstream MILM and image reconstruction tasks. The focus of our work is at maintaining downstream performance while reducing transmission cost and decoding inference cost. Figure [] illustrates the different costs associated with each component in the system.
Ideally, one might propose generating visual tokens on the end device and using token reduction techniques compression method to reduce transmission bandwidth. However, this approach
16

---

Published as a conference paper at ICLR 2025
End Device
Transmission
Cloud
Partial Visual Encoder
Image
Encoder
Transform- neck
Connector & LLM
MLLM Response
Decoding Inference Cost
Transmission Cost
MLLM Inference Cost
Our Focus
Token Reduction
Figure 11: High-level architecture of our proposed method and different cost associated with each component.
would impose significant computational demands on the end device, making it impractical for coding for machines scenarios, where the primary goal is to offload heavy feature extraction computations to the cloud.
Notably, method and the token reduction technique could potentially complement each other to develop a more efficient system. For instance, our approach allows to save transmission bandwidth and reduce cloud-side complexity by eliminating the need for image decoding, then token reduction techniques can further optimize complexity on the cloud.
17

---

Published as a conference paper at ICLR 2025
Task: Captioning
Model: LLaMA- Aadpter
Uncompressed
Reconstructed
Post prol-prceesedd
BPP: 0.0725
Reconstruction: A microwave and a computer sitting on desk. Post-procsssing: microwave and a refrigerator sitting on top of table. Ours (dl): A microwave and toaster oven on a counter.
Uncompreseed
Reconstructed
Post-processod
BPP: 00,9288
Reconstruction: Two cats are standing on the ground near a bench. Post-processing: A dog and a cat are standing on sidewalk. Ours (dI): Two dogs are standing near bicycle on sidewalk.
Uncompressed
Reconstructed
Post-processod
BPP: 0.0910
Reconstruction: A blurry picture of a blender with knife. Post-processing: A close up of blurry image of bug.
Ours (dl): A close up of knife cutting into pizza.
Uncompressed
Reconstructed
Post-processed
BPP: 0.0899
Reconstruction: A young boy in red shirt and tie posing for picture. Post-processing: A young boy standing in front of wall with clock.
Ours (d1): A young boy in tie and white shirt.
Visualization examples of our proposed method in (d1), Reconstruction, and Post- processing image captioning with LLaMA-Adapter.
Figure 12:
18

---

Published as a conference paper at ICLR 2025
Task: Visual question answering Model: Honeybee
What the dog doing in the image?
A. Standing still
B. Chasing after something C. Lying down D. Jumping in the air
Uncompreseed
Reconstructed
Post-processod
BPP:0.082
Reconstruction:
B Post-processing: B
Ours (dl): A GT:
How many people are on the field in this image? A. Four B. Nine C. Twelve D. Eleven
Uncompreseed
Reconstructed
Post-pracessod
0P970.097
Reconstruction:
D
Post-processing: B Ours (dI): A GT: A
What is the person in the blue jacket holding? A. phone
B. Nothing C. A wallet D. A clipboard
Uncompressee
Reconstructed
Post-pracessed
BPP:0. 160
Reconstruction:
D Post-processing:
Ours (dl): GT: B
How many people lighting candles in this image?
A. Two B. One C. Three D. Four
Uneemapressed
Reconstracted
Post-pracessed
BPP:0.066
Reconstruction: A
Post-processing: Ours (dl): GT: C
Figure 13: Visualization examples of our proposed method in (d1), Reconstruction, and Post- processing VQA with Honeybee.
19

---

Published as a conference paper at ICLR 2025
Task: Referring expression comprehension
Model: Shikra
Guide me to the location of brown bear within the image <img> by providing its coordinates.
Uncompressed
Post-proesssei
BPP:0.040
(drl(dl)
Point me to the location of wine glass fär left in the picture <img> by providing its coordinates.
Uncompressed
Reconstructed
Post-prrncesaod
BPP:O. 1110.115
115
Can you assist in locating right female cop in <img>, and then provide its coordinates?
Uncompressed
Reconstructed
Post-princcssed
BPP:0.098
Qursi
Plac
Post
In the photograph <img>, could you pinpoint the location person holding a snowboard and tell me coordinates?
Uncompressed
Post-prrucessed
BPP:0.088 088
Recon
Figure Visualization examples of our proposed method in (d1), Reconstruction, and Post- processing on REC with Shikra.
20

---

Published as a conference paper at ICLR 2025
Task: Few-shot classification Model: V2L-tokenizer
Examples
T L
walking plunnigan
miller.
screw
Unsonpressed
Revonstructed
Post-processed
BPP: 0.0786
Query
Reconstruction: ptarmigan Post-processing ptarmigan
Ours (dl): walking stick
GT: walking stick
Examples
etterhound
bassinet
wall clock
whiskey jug
hippopotanuus
Uncomppessed
Reconstructdd
Post-processed
BPP: 0.1369
Query
Reconstruction: hippopotamus
Post-processing: wall clock
Ours (dl): otterhound
GT: otterhound
Hanel
Examples
dung beetle
2od
hombill
look
horned viper
Uncompressed
Reconstruterd
Post Pool-pracessed
BPP: 0.2329
Query
Reconstruction: horned viper Post-processing: homeed viper Ours (dl): hornbill GT: hombill
Figure 15: Visualization examples of our proposed method in (d1), Reconstruction, and Post- processing on few-shot classification with V2L-tokenizer.
21


Reviews:
[{"review_id": "CihjUQVDhm", "reviewer": "Reviewer_ESiA", "summary": "This paper suggests using compressed image latents for more efficient multimodal large language models (MLLMs). Specifically, a lightweight transform-neck and a surrogate loss are proposed to align compressed image latents with backbone LLMs for multimodal tasks. Experiments are conducted on some mainstream multimodal tasks, such as image captioning and VQA, and the proposed method demonstrates that it outperforms the reconstruction baseline.", "questions": "Please refer to Weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* This paper is easy to follow\n* The topic of reducing the cost of visual input in MLLMs is of great practical value\n* The idea of adapting compressed image latent to MLLMs makes sense to me", "weaknesses": "This paper can be further strengthened by:\n\n* One question that remains unclear about the motivation is why we need the proposed method of compressing image latent instead of existing token pruning/merging work on MLLMs such as crossget [1]. These methods can also reduce the costs of MLLMs, and I’d suggest the authors discuss the difference between existing acceleration methods for MLLMs and the proposed method, and highlight their unique contributions to efficient MLLMs.\n\n* The proposed method shows inconsistent performance degradation on different benchmarks. For example, it shows close performance to models using uncompressed image latent on image captioning while suffering from a significant performance degradation on the POPE benchmark. It would be better to have some insight into it.\n\nReference\n\n[1] Crossget: Cross-guided ensemble of tokens for accelerating vision-language transformers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ca0IbwJRd0", "reviewer": "Reviewer_wKjV", "summary": "This paper investigates adapting compressed image latents to meet the requirements of downstream vision tasks in Multimodal Large Language Models (MLLMs). The authors introduce a framework that employs a \"transform-neck\" structure and a surrogate loss function to adjust compressed image representations, without involving MLLMs in the training process. This design choice differs from conventional approaches that incorporate downstream models during training, making it potentially more feasible for resource-limited devices. The results show that this method achieves a balance between compression rate and accuracy across various neural image codecs and MLLMs, demonstrating its potential for efficient performance.", "questions": "1. Why are different MLLMs chosen for each task instead of evaluating multiple MLLMs on each task? A consistent comparison across tasks using various MLLMs would provide a clearer understanding of the method’s generalization capability.\n2. During Phase 1: Transform-neck Training, the motivation for using different loss functions at various stages across all epochs is not clearly explained, and there is a lack of corresponding ablation studies to support this design choice.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper focuses on an interesting scenario: how to achieve image compression in the context of MLLMs.\n2. The writing in this paper is well-organized, making it easy to follow and understand the authors' intentions.", "weaknesses": "1.  The proposed method appears to be a general image compression approach, lacking unique design elements specifically tailored for MLLMs. While the authors emphasize that this approach can achieve image compression in a resource-efficient manner to support MLLMs, they do not provide specific comparisons regarding the reduction in training costs—such as training time or computational resources—compared to methods that incorporate MLLMs directly into the training process.\n2. The proposed  SURROGATE LOSS is not novel, easily combination of the Mean Squared Error loss and the cross-entropy loss.\n3. The proposed method has limited applicability, as it requires alignment with specific vision encoders. The authors only align with CLIP, whereas many advanced MLLMs now use other, more powerful vision encoders, such as Intern-ViT or SGLIP. This method may need further adaptation to support these alternative encoders effectively.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XewU0zyIAK", "reviewer": "Reviewer_kXYn", "summary": "The paper introduces a novel framework designed to bridge compressed image latents with Multimodal Large Language Models (MLLMs) for downstream vision tasks. It addresses the challenge of deploying MLLMs on resource-constrained devices by proposing a lightweight transform-neck and a surrogate loss to adapt compressed images for MLLM-based vision tasks without the need for decoding the images. This approach reduces computational complexity and is generic, applicable to various neural image codecs and MLLMs sharing the same visual encoder.", "questions": "1. Will you plan to release the code ?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a lightweight transform-neck and a surrogate loss function, which together reduce computational complexity and avoid the need to back-propagate through the massive MLLMs, making the training process more efficient.\n2. The proposed framework is generic and can accommodate various neural image codecs and MLLMs that share the same visual encoder, enhancing its versatility and applicability across different models and tasks.\n3. The paper demonstrates the effectiveness of the proposed method through extensive experiments on different neural image codecs and MLLM-based vision tasks, showing improved rate-accuracy performance and less complexity compared to existing methods.", "weaknesses": "1. The paper's approach relies on the assumption that the downstream MLLMs will use the same pre-trained CLIP visual encoder, which may limit the applicability of the proposed method to MLLMs that employ custom or different visual encoders.\n2. The paper does not provide a comprehensive comparison with existing image compression methods beyond the context of MLLMs, which could be important for understanding the method's performance in a broader range of applications", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MzgnVNaP9O", "reviewer": "Reviewer_HmZL", "summary": "This paper explores methods to compress and decode images such that the decoded images can be successfully used with Large MLLMs, addressing the substantial overhead required for transmitting images to cloud-hosted MLLMs. Existing image encoding works for machine vision face a common problem: MLLM models are too large to train the neural image codec. To address the computational overhead of image compression and encoding in the context of MLLMs, this paper proposes a lightweight transform-neck and a novel surrogate loss. The lightweight transform-neck aims to align the features of the compressed images with the outputs of the visual encoder, avoiding the need for fully image decoding. The proposed surrogate loss allows the neural image codec to be trained using only the vision encoder, avoiding the need for backpropagating through the entire MLLM. The proposed method can be applied to different MLLMs and various neural image codecs in various application scenarios.", "questions": "Please refer to the Weaknesses", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper claims to be the first work specifically targeting neural image coding for MLLMs.\n2. It introduces a lightweight transform-neck that aligns the encoded image latents with the MLLM visual encoder outputs, thereby avoiding the decode operation in neural image coding and reducing computational complexity.\n3. The proposed surrogate loss allows updating the neural image coding model via the Visual Encoder without backpropagating through the entire MLLM, hence reducing the computation required.\n4. The proposed method is applicable to different downstream MLLMs, applications, and neural image codecs.", "weaknesses": "1. The motivation of this paper—studying image compression under MLLM to avoid transmission overhead—seems to lack practical significance. The primary goal of image compression is to reduce image resolution, which might be meaningless for MLLMs:\n* First, most MLLMs [1] use CLIP Image Encoder, which limits image resolution to 224x224. This resolution does not incur significant transmission overhead in current network communications and is negligible compared to the inference time of MLLMs.\n* Second, while some works [2] explore using high-resolution images as input, they usually crop high-resolution images into several low-resolution sub-images, often achieving better performance compared to compressed images. Therefore, researching how to compress images as MLLM input seems unnecessary.\n* Lastly, the authors do not explain the advantages of using compressed images as MLLM input over exploring more compact Image Encoders [3].\n\n2. The proposed method requires training different transform-necks and codecs for different downstream tasks, further leading me to question the necessity and rationale of this work.\n\nReference\n* [1] Visual Instruction Tuning, NeurIPS 2023\n* [2] About When do we not need larger vision models?, ECCV 2024\n* [3] TokenPacker: Efficient Visual Projector for Multimodal LLM, arXiv 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)