Bridging Compressed Image Latents and Multimodal Large Language Models

## Abstract

Abstract This paper presents the first-ever study of adapting compressed image latents to suit the needs of downstream vision tasks that adopt Multimodal Large Language Models (MLLMs). MLLMs have extended the success of large language models to modalities (e.g. images) beyond text, but their billion scale hinders deployment on resource-constrained end devices. While cloud-hosted MLLMs could be available, transmitting raw, uncompressed images captured by end devices to the cloud requires an efficient image compression system. To address this, we focus on emerging neural image compression and propose a novel framework with a lightweight transform-neck and a surrogate loss to adapt compressed image latents for MLLM-based vision tasks. The proposed framework is generic and applicable to multiple application scenarios, where the neural image codec can be (1) pre-trained for human perception without updating, (2) fully updated for joint human and machine perception, or (3) fully updated for only machine perception. The transform-neck trained with the surrogate loss is universal, for it can serve various downstream vision tasks enabled by a variety of MLLMs that share the same visual encoder. Our framework has the striking feature of excluding the downstream MLLMs from training the transform-neck, and potentially the neural image codec as well. This stands out from most existing coding for machine approaches that involve downstream networks in training and thus could be impractical when the networks are MLLMs. Extensive experiments on different neural image codecs and various MLLM-based vision tasks show that our method achieves great rate-accuracy performance with much less complexity, demonstrating its effectiveness.

## 1 Introduction

Large Language Models (LLMs) [, , ] have demonstrated impressive abilities in various Natural Language Processing (NLP) tasks. The recent surge of research on Multimodal Large Language Models (MLLMs) extends LLM’s abilities to data beyond languages [, , , , , ], particularly images, opening up promising opportunities in various applications. MLLMs have shown surprising capability for many vision tasks such as classification [], image captioning [, ], Visual Question Answering (VQA) [, ], and meme interpretation []. These models excel in unseen tasks through instruction following or in-context learning, which is impossible for traditional vision networks.

However, MLLM’s billion-scale size hinders deployment on resource-constrained end devices. While computation can be offloaded to the cloud, transmitting images to cloud-hosted MLLMs becomes necessary. Our study shows that directly feeding the decoded image, generated by a fixed neural image codec trained for human perception, into an MLLM (Figure 1 (a)) significantly degrades task performance, particularly when the image is coded at low rates. This highlights the need for efficient image compression that considers the requirements of downstream MLLM-based vision tasks.

Many prior works address image compression for machine vision, commonly referred to as coding for machines [, , , , ]. Two common approaches to coding for machines are image coding and feature coding. The image coding approaches [, ] optimize the image codec for specific downstream tasks and/or networks (Figure 1 (b)), while the feature coding approaches [] divide the task network into two parts and focus on compressing the intermediate features (Figure 1 (c)). However, both approaches face the same issue: they limit the trained system to be only suitable for one specific model or task, thus requiring separate parameters or models for different tasks. Additionally, while they may potentially yield high rate-accuracy performance, the training process becomes challenging when one needs to back-propagate a training objective through a massive MLLM to train the neural image codec. In practice, the billion-scale parameters of MLLMs make the existing coding for machine methods inapplicable. To the best of our knowledge, there have been no attempts to tackle compression for MLLMs.

Figure: Figure 1: On the left is inadequate frameworks for image compression for MLLMs, where the image codec is trained for (a) human perception, (b) the downstream task network, or (c) compressing the intermediate features of the task network. On the right is the proposed transform-neck and surrogate loss under three distinct scenarios, with the image codec (d1) pre-trained for human perception, (d2) updated for joint human and machine perception, or (d3) updated for machine perception.
Refer to caption: extracted/5759811/Figures/architecture/teaser_v5.png

In this paper, we propose the first neural image compression system for MLLM-based vision tasks that enables compressed image latents to suit the needs of downstream MLLMs. The proposed method involves a lightweight transform-neck and a surrogate loss. The transform-neck adapts compressed image latents to match the intermediate features of the CLIP visual encoder [], a common component in many popular MLLMs. This approach avoids the image decoding process and reduces computational complexity. To address the massive size of MLLMs, the surrogate loss updates our system with the CLIP visual encoder, refraining from back-propagating the task loss through the heavy MLLM. The transform-neck trained with the surrogate loss is universal and readily applicable to various MLLMs that share the same CLIP visual encoder, without requiring re-training.

The proposed method is generic and applicable to different neural image codecs under various application scenarios. First, if thedownstream applications prioritize image reconstruction quality for human interaction, our method can work with an off-the-shelf image codec trained for human perception (Figure 1 (d1)). Without any modification or re-training of the codec, our method adapts the compressed image latents while maintaining the same image reconstruction quality. Second, when allowing image codecs to be updated, we propose a multi-task training strategy that optimizes the codec for both human and machine perception (Figure 1 (d2)). This significantly improves MLLM performance at the cost of a marginal drop in reconstruction quality. Finally, we consider an extreme setting in which the applications prioritize machine perception over image reconstruction. In this case, the encoder and the transform-neck are jointly optimized for the MLLM systems exclusively (Figure 1 (d3)). On top of that, our transform-neck is agnostic to the architecture of neural image codecs, being able to work with various models.

The contributions of this work are summarized as follows:

- •
It marks the first exploration into the field of neural image coding for MLLMs.
- •
The proposed transform-neck adapts the compressed image latents to downstream MLLMs, avoiding the need for image decoding and saving computational complexity.
- •
The proposed surrogate loss leverages the CLIP visual encoder to update the system, avoiding back-propagating the task loss through the heavy MLLM.
- •
Our method is agnostic to the downstream MLLMs or tasks; without re-training, the resulting system is readily applicable to a wide variety of MLLMs sharing the same visual encoder.
- •
Our framework is able to accommodate various application scenarios that involve human perception, machine perception, or both.

## 2 Related Works

### 2.1 Multimodal Large Language Models

In recent years, there has been a surge of interest in MLLMs following the impressive demonstration of LLM’s ability in the NLP field [, , ]. Many have sought to extend the success of these models from text to other modalities, particularly images, and several works have shown their effectiveness on various tasks, such as image captioning [, , , , ], VQA [, , ], Referring Expression Comprehension (REC) [, , , , ], few-shot classification [, ], action anticipation [], meme interpretation [, ], biomedical reasoning [], OCR-free math reasoning [].

Most existing MLLM approaches use a visual encoder to process the input image data, and then introduce a connector to bridge the image features to the tokens understandable by the LLM. Earlier works adopt simpler connector designs, such as linear projectors [, ], while subsequent works [, , ] have refined upon the design for both performance and complexity. Furthermore, the entire MLLM can be further fine-tuned to enhance its capabilities through instruction tuning [, ].

A notable aspect of the MLLMs is their reliance on existing pre-trained visual encoders in their systems, with CLIP [] visual encoder being a very common choice for a large number of methods [, , , , , , , , , , ]. Trained on large image-text pair data, the CLIP visual encoder offers the feature space that combines language and image modalities in a sense, making it a desirable feature for MLLMs. Sharing the same visual encoder also gives us the opportunity to design a universal method for MLLMs. Notably, all the existing works on MLLMs do not consider the scenarios where image compression is present, which is a significant departure from our work.

### 2.2 Image Coding for Machines

Neural image compression systems have made significant progress in the past few years. As a matter of fact, several works [, , , ] have even outperformed the traditional codecs such as intra coding in VVC []. However, these methods primarily focus on the quality of reconstructed images for human perception. Coding for machines, in contrast, targets downstream machine vision instead of human perception, and it has attracted increasing attention recently.

A common approach simply involves training the compression system for a predefined target downstream computer vision task [, , ], enabling the reconstructed image to be suitable for machine vision, albeit potentially sacrificing perceptual quality. Conversely, Chamain et al. [] tune the task network to better process the compressed images, while Chen et al. [] leverage prompt-tuning method on Transformer-based codecs to boost performance on multiple tasks. Also, with the trend of the new JPEG AI learning-based image coding standard [], some methods [, , , ] utilize the compressed image latents instead of the reconstructed image for recognition through bridging the latents to task network. On the other hand, Ding et al. [] directly compress the intermediate features of recognition networks, while Feng et al. [] learn the omnipotent features suitable for various tasks in a self-supervised manner and fine-tune each task network tail on such features.

It is crucial to note that none of the coding for machine methods considers MLLMs at the receiver side. All the above-mentioned methods leverage back-propagation through recognition models to update the system, which is prohibitively expensive for MLLMs due to their huge scale. Therefore, the direct application of the same methods on MLLMs is almost infeasible. In addition, the use of a specific task loss restricts the resulting models to be optimized for a single task and recognition model, thus requiring re-training for each new task and incurring additional costs. We aim to be the first to propose a neural image compression system designed for MLLMs, achieved through a universal transform-neck and the adoption of a surrogate loss, which allows to bypass the necessity of involving the entire billion-scale MLLM in the training process.

## 3 Proposed Method

### 3.1 Preliminaries: Neural Image Codecs

The high-level architecture of a neural image codec is depicted in the top central green box in Figure 2. In a typical hyperprior-based neural image compression system [], the key components include the main encoder $g_{a}$, the main decoder $g_{s}$, as well as the hyperprior encoder $h_{a}$ and decoder $h_{s}$. Given an RGB image $x\in\mathbb{R}^{3\times H\times W}$, where $H$ and $W$ represent the height and width of the image, respectively, $g_{a}$ performs the analysis transform of $x$ and generates the image latent representation $y\in\mathbb{R}^{N\times\frac{H}{16}\times\frac{W}{16}}$, with $N$ indicating the channel size. To transmit $y$ more efficiently, it is first uniformly quantized into $\hat{y}$ and then entropy coded considering a learned prior distribution $p(\hat{y})$. This learned distribution is content dependent, thanks to the hyperprior encoder $h_{a}$ and decoder $h_{s}$. In particular, $h_{a}$ takes $y$ as input and produces the side information $z\in\mathbb{R}^{N_{h}\times\frac{H}{64}\times\frac{W}{64}}$, that is used to generate the learned distribution for entropy coding, where $N_{h}$ is the channel size of the side information. The quantized version of $z$, denoted as $\hat{z}$, is transmitted into the bitstream, in order to recover $\hat{y}$. Lastly, $\hat{y}$ undergoes the synthesis transform with $g_{s}$, which reconstructs the image $\hat{x}\in\mathbb{R}^{3\times H\times W}$.

Figure: Figure 2: Overall architecture of the proposed method.
Refer to caption: extracted/5759811/Figures/architecture/arch_v8.png

### 3.2 Overall Framework

In this work, we focus on the scenario where MLLMs are hosted on the server side, while users on end devices need to perform inference on the remote model using both text and images as inputs. Given the necessity of incorporating image compression to ensure efficient transmission, we propose a compression framework with the consideration of MLLMs as downstream application networks, aiming to mitigate the potential task performance drop caused by image compression. Figure 2 illustrates our overall framework, which includes three major components: the neural image codec, our proposed transform-neck, and the MLLM. The depicted MLLM system adheres to a typical structure, consisting of a visual encoder, an LLM, and a connector component facilitating the transformation of features from the visual encoder to the LLM.

During inference, an input image at the end device is passed through an encoder $g_{a}$ to generate the quantized latents $\hat{y}$ for transmission. In our proposed method, $\hat{y}$ is directly passed through a lightweight transform-neck $T$ for transformation into a middle layer of the visual encoder of an MLLM. We opt to adapt the image latents rather than the reconstructed images because the image latents inherently contain the information needed for reconstructing the image, and potentially the semantic information for the downstream tasks (when the image encoder is guided properly). By skipping the image decoding process, our method offers reduced computational complexity while maintaining the task performance. The rest of the MLLM system operates without any changes to generate the desired output response.

In our experiments, we examine three distinct settings denoted as (d1), (d2) and (d3), as illustrated in Figure 1. Firstly, in (d1), we consider the practical scenario where a fixed off-the-shelf image codec pre-trained for human perception is directly used alongside our transform-neck. In this setting, our framework offers the option for users to decode the image latents $\hat{y}$ for reconstruction by using the decoder $g_{s}$ instead of the transform-neck. The quality of the decoded image is not affected by the introduction of our transform-neck, as the image codec is not updated in the present case. Then, we extend the analysis to scenarios (d2) and (d3) to examine the impact of jointly training the image codec and transform-neck. In (d2), the entire image codec undergoes re-training to accommodate both human and machine perception, while in (d3), the encoder is re-trained specifically for machine perception. Regardless of the context examined, we circumvent the difficulties of back-propagating the task loss through MLLMs by introducing a surrogate loss. We remark that the resulting system is readily applicable to a wide variety of MLLMs and tasks. It needs no re-training of the system when these MLLMs adopt the same visual encoder.

### 3.3 Transform-neck

Our transform-neck is designed to be a lightweight module, consisting only of a linear projection, a self-attention mechanism, a feed-forward layer, and two layer norms, as shown in the central red box in Figure 2. Its purpose is to adapt the compressed image latents $\hat{y}$ to a form suitable for consumption by the downstream MLLMs.

Adapting the compressed image latents to existing MLLM systems is a non-trivial task, especially when aiming for a universal approach compatible with multiple MLLMs and tasks. To address this challenge, we observe that a large number of existing MLLM systems share the same pre-trained visual encoder, i.e. the CLIP visual encoder, as discussed in Section 2.1. Inspired by this observation, we propose to leverage the CLIP visual encoder, denoted by $C$, for guiding the latent transformation. Note that this approach makes our framework compatible with a wide variety of MLLMs, as long as they share the same visual encoder $C$.

Since the image encoder $g_{a}$ serves as a feature extractor akin to the initial layers of the visual encoder $C$, we propose to turn the output of our transform-neck directly into the intermediate features of $C$, effectively connecting the image codec and MLLM system. Specifically, the transform-neck processes $\hat{y}$ to align the third Transformer layer of $C$, skipping the first two Transformer layers and thus further reducing computational complexity. An ablation experiment in Section 4.5 justifies our design choice. We denote this partial CLIP visual encoder as $C^{\prime}$.

### 3.4 Surrogate Loss

To avoid involving huge MLLMs in the training process, thus bypassing back-propagation through them, we propose a surrogate loss, which is back-propagated through only the partial CLIP encoder $C^{\prime}$.
To retain the downstream MLLM performance, the output features $C^{\prime}(T(\hat{y}))$ of $C^{\prime}$ when using our transformed latents should resemble closely those obtained when inputting the uncompressed image into $C$, that is $C(x)$. To this end, we introduce the following distillation loss for training, aiming to minimize the Mean Squared Error (MSE) between the two output features:

$$ $\displaystyle\mathcal{L}_{dist}=\text{MSE}(C^{\prime}(T(\hat{y})),C(x)).$ (1) $$

The surrogate loss enables the resulting transform-neck to be applicable to various MLLMs sharing the same visual encoder $C$ without re-training.

However, we notice that applying the distillation loss alone during the early training phase can make the training challenging and unstable, potentially due to the strict requirement of fitting the exact representation. To address this, we adopt a progressive training strategy, by including an additional cross-entropy loss at first, which provides a better update direction in the early training phase. Thus, using a classification dataset, we first compute the cosine similarity between the image and text embeddings produced from the fixed CLIP visual and text encoder, respectively, when provided with transformed image latents and ground truth class labels. Then, the probability distribution over different recognition classes is calculated by applying the softmax operation to the resulting cosine similarities, and the cross-entropy loss is evaluated with respect to the ground truth labels.

### 3.5 Training Objective Under Different Settings

**Table 1: Application scenarios for our method with corresponding training objective.**
| Application scenario | Update image codec | Human viewing | Training objective |
| --- | --- | --- | --- |
| (d1) Human perception | ✗ | ✓ | $\mathcal{L}_{dist}$ |
| (d2) Multi-task | ✓ | ✓ | $R+\lambda(\gamma d_{recon}(x,\hat{x})+\beta\mathcal{L}_{dist})$ |
| (d3) Machine perception | ✓ | ✗ | $R+\lambda\mathcal{L}_{dist}$ |

To explore the capabilities of our method under the settings introduced in Section 3.2, we implement different training objectives, as summarized in Table 1.
In (d1), since we consider a fixed off-the-shelf codec for retaining high quality reconstructed images for human perception, we train our transform-neck simply using distillation loss as the sole loss function, i.e. $\mathcal{L}_{d1}=\mathcal{L}_{dist}$ .

On the other hand, in (d2), referred to as multi-task, the image codec is allowed to be re-trained to accommodate both human and machine perception. As a result, it is trained jointly with the transform-neck on both the distillation loss and traditional rate-distortion loss, i.e. $\mathcal{L}_{d2}=R+\lambda(\gamma d_{recon}(x,\hat{x})+\beta\mathcal{L}_{dist})$, where $R=-\log p(\hat{z})-\log p(\hat{y}|\hat{z})$ is the estimated rate of $\hat{y}$ and $\hat{z}$, and $d_{recon}$ is the reconstruction loss calculated as the MSE between the uncompressed image $x$ and the reconstructed image $\hat{x}$. The hyper-parameter $\lambda$ controls the rate-distortion trade-off, while $\gamma$ and $\beta$ weight the two losses.

In (d3), where the downstream applications do not require image reconstruction,
the encoder and transform-neck are jointly optimized to minimize the trade-off cost between the rate $R$ and the distillation loss $\mathcal{L}_{dist}$, thus disregarding the reconstruction quality. The training objective is thus $\mathcal{L}_{d3}=R+\lambda\mathcal{L}_{dist}$.

To facilitate the transform-neck in better learning the transformation with the image latents, the training process for (d2) and (d3) is conducted in three stages.
Initially, only $\mathcal{L}_{dist}$ is used as loss function without updating the image codec. Then, we jointly train the image codec and transform-neck using either $\mathcal{L}_{d2}$ or $\mathcal{L}_{d3}$. Finally, we further fine-tune the transform-neck using once again only $\mathcal{L}_{dist}$, while keeping the updated image codec frozen.

## 4 Experimental Results

### 4.1 Experimental Setting

#### Training Details and Datasets.

We utilize ELIC [] as our image codec, which outputs image and hyperprior latents with $N=320$ and $N_{h}=192$, respectively. ELIC is trained for human perception and adheres to the training strategy outlined in [], using 8,000 images of the highest spatial resolution selected from ImageNet dataset. Four models are trained for four different rate points, corresponding to $\lambda=[0.004,0.008,0.016,0.032]$ in [].
For each of our scenarios (d1), (d2) and (d3), separate transform-necks are trained on ImageNet dataset [] for individual $\lambda$ values.
We use the pre-trained visual encoder from CLIP ViT-L/14 [] as the reference for ground truth. For the scenario (d2), we find empirically that fixing the ratio $\gamma:\beta=60:1$ leads to a good trade-off between human and machine perception.

Figure: Figure 3: Rate-accuracy comparison using various MLLMs on several tasks.
Refer to caption: extracted/5759811/Figures/RD-curve/paper_main_RD.png

Figure: Table 2: Evaluation tasks with corresponding dataset, MLLM, and metric.
Refer to caption: extracted/5759811/Figures/RD-curve/paper_recon_kodak.png

#### Targeted MLLM-based Vision Tasks.

To validate the generalization ability of our proposed method, we evaluate its performance on four different MLLM systems for four different tasks.
Note that our method is independent of the downstream tasks, and thus the same set of transform-necks is used for all the evaluations. Additionally, all the MLLMs are employed off-the-shelf without any fine-tuning. The tasks, datasets, corresponding MLLMs, and metrics are listed in Table 4.1.
These configurations follow the settings outlined in their original papers and the accompanying code, except for the few-shot classification task due to the inaccessibility of the code.
We thus design a 5-way 1-shot classification scenario to evaluate the performance with in-context learning; the detailed setting is described in supplementary material.

#### Baselines.

We introduce two baseline methods for comparison. The first one, denoted as Reconstruction, involves inputting the reconstructed image generated by ELIC to the MLLM system. The second one, denoted as Post-processing, adapts the reconstructed image to MLLMs through a U-Net [] post-processing network, which is trained using the same surrogate loss as that adopted by our method. We remark that these image-domain baselines incur higher complexity than our lightweight transform-neck, as they involve decoding the image and potentially processing it further with the post-processing network. This aspect is discussed in detail in Section 4.4.

### 4.2 Performance Comparison

Figure 3 illustrates the performance of the baseline methods and our proposed scheme for the three examined scenarios with regards to two aspects: coding bit-rate, calculated as bits per pixel (bpp), and task performance. When comparing the baselines and our method in scenario (d1), where the original ELIC is trained solely for human perception, we make the following observations. (1) Straightforwardly using the reconstructed images generated by a codec trained for human perception leads to a significant performance drop across all the tasks (Reconstruction). Such performance decline is expected because the MLLMs are not trained with compressed images, thus hindering their recognition performance. This highlights the necessity of adapting image compression and/or image latents to MLLMs. (2) In contrast, our transform-neck method successfully boosts the performance using the same latent representations for reconstructing the image in Reconstruction, confirming the effectiveness of the proposed latent transformation without the decoding process. (3) Post-processing is able to reach comparable performance to our (d1), offering another viable solution to the problem. However, it is worth noting that Post-processing requires relatively higher computational complexity with respect to our transform-neck method, rendering our approach preferable (see Section 4.4) .

**Figure 5: Visualization examples of our proposed method in (d1), Reconstruction, and Post-processing on image captioning with LLaMA-Adapter and REC with Shikra.**
| Captioning<br>LLaMA-Adapter | Reconstruction: A man is walking an elephant down a path.<br>Post-processing: A man feeding an elephant with his hand.<br>Ours (d1): A man is petting an elephant on the head. | BPP: 0.0958 |
| --- | --- | --- |
| REC<br>Shikra | I’m trying to locate man with mask on in <img>. Can you determine its coordinates for me? | BPP:0.061 |

Next, we evaluate the effects of allowing the image codec to be re-trained. First, we observe that (d2) outperforms both (d1) and Post-processing. This indicates that fine-tuning the encoder indeed results in a more suitable latent representation that can be better adapted to MLLMs.
When examining the extreme setting (d3), we see significant further improvement in the task performance, approaching the performance upper bound with uncompressed images.
This improvement comes at the cost of the image reconstruction quality, which, however, is not relevant in (d3).
Figure 4 illustrates the rate-visual quality curves associated with the three scenarios. Interestingly, (d2) exhibits only a marginal PSNR drop compared to (d1), while (d3) significantly compromises the quality of the decoded image.
We stress that our framework (i.e. the surrogate loss and transform-neck) is able to accommodate different application scenarios, allowing for a variable trade-off between the task performance and the image reconstruction quality.

### 4.3 Visualization of the Results

We present the visualization of outcomes with downstream MLLM-based vision tasks in Figure 5. Our method (d1) is compared with the two baseline methods, Reconstruction and Post-processing, with particular focus on how these models work at low bitrates to reflect a bandwidth-limited scenario. In the second and third columns (from left to right), we visualize the reconstructed and post-processed images from the two baselines, respectively, which exhibit drastically different characteristics. The former (Reconstruction) produces blurry and smooth images, while the latter (Post-processing) introduces some artificial patterns into the post-processed images. Compared with the baselines, our method yields MLLM results closer to the ground truth across all the tasks.
Due to the space constraint, the results of VQA and few-shot classification are visualized in the supplementary material.

**Table 3: Comparison of the kMACs/pixel and model size. The table omits the shared components of the two methods, i.e. the image encoder, the partial CLIP visual encoder, the connector, and the LLM.**
| Method | Component | Params (M) | kMAC/pixel |  |  |
| --- | --- | --- | --- | --- | --- |
| Ours (d1, d2, or d3) | Transform-neck | 13.19 | 52.795 |  |  |
| Post-processing | Decoder | 7.34 | 64.16<br>(+386%) | 112.00 | 1017.96<br>(+1828%) |
| Post-processing network | 31.04 | 835.72 |  |  |  |
| First 2 layers of CLIP visual encoder | 25.78 | 70.24 |  |  |  |

### 4.4 Complexity Analysis

Table 3 compares the computational complexity between our proposed method and Post-processing baseline in terms of model size and the kilo-multiply-accumulate-operations per pixel (kMACs/pixel). Note that our method in Table 3 refers to any of (d1), (d2), and (d3), since they share the same computational complexity characteristics at inference time. Our method offers a lightweight solution with only 13 million parameters, as opposed to 64 million parameters with the post-processing approach. Moreover, in terms of kMAC/pixel, the difference stands out even more, considering that the post-processing network operates at the full image resolution while our method operates in the latent domain, where the image latents have a much smaller spatial resolution.

Figure: (a) Partial CLIP visual encoder
Refer to caption: extracted/5759811/Figures/RD-curve/paper_ablation_layer.png

### 4.5 Ablation Studies

The following ablation experiments are performed based on (d1) to justify our design choices.

#### Partial CLIP Visual Encoder.

This experiment investigates the proper number of Transformer layers to remove from the CLIP visual encoder in order to strike a good balance between complexity and performance. As shown in Figure 6 (a), removing the first one or two layers achieves similar performance, whereas removing four or eight layers results in a noticeable performance drop. We thus remove the first two layers.

#### Training Objective.

Figure 6 (b) presents the performance of our method when trained exclusively with the cross-entropy loss or distillation loss. It is observed that training with only the cross-entropy loss results in a significant performance drop. Although providing a good initial update direction, this strategy is unable to learn an effective transformation for MLLMs. Instead, training solely with the distillation loss fails to update the transform-neck properly and leads to far inferior performance. This is potentially due to the more stringent requirement of fitting the exact feature representations.

#### Different Image Codecs.

Figure 6 (c) presents the performance comparison between our method and Reconstruction when they are tested with ELIC and TIC [, ].
TIC is a Transformer-based codec, whereas ELIC is a convolutional neural network-based codec.
We see that our transform-neck still outperforms Reconstruction by a significant margin when the image codec is changed from ELIC to TIC. This indicates that our method is still effective on a different type of image codec.

## 5 Conclusion

This paper proposes the first image compression system tailored to Multimodal Large Language Models (MLLMs). It introduces a transform-neck that bridges the compressed image latents and the intermediate layer of the CLIP visual encoder, a common component in MLLMs. By using our proposed surrogate loss, we avoid involving the MLLM in the training process, making our transform-neck universally applicable. With lower computational complexity, our method has demonstrated effectiveness across a wide variety of tasks, MLLMs, and neual image codecs, outperforming other baselines in extensive experiments.
One consideration is that it requires the same pre-trained CLIP visual encoder to leverage the universal transform-neck, which may limit compatibility with MLLMs that use custom visual encoders. Furthermore, this paper focuses solely on the image compression aspect of MLLMs, leaving the exploration of video or audio coding for future work.

## Appendix A Supplementary Material

### A.1 Implementation Details

#### Training.

We adopt a progressive training strategy incorporating both the cross-entropy and distillation losses, divided into three stages. (1) We train our system using only the cross-entropy loss with a learning rate of $10^{-4}$ for 20 epochs. (2) We introduce the distillation loss with a ratio of cross-entropy loss to distillation loss set to 1:100, training at a learning rate of $10^{-5}$ for 20 epochs. (3) Lastly, we train our system using only the distillation loss for 20 epochs.

We use the Adam optimizer, configured with $\beta_{1}$ at 0.9, $\beta_{2}$ at 0.999, $\epsilon$ at $10^{-8}$. Weight decay is disabled. The transform-neck for each rate point undergoes training on an RTX 4090 for approximately three days during the training stage.

#### Evaluation.

For few-shot classification with V2L-Tokenizer [], we design a 5-way 1-shot classification evaluation scenario. In particular, we generate 5000 groups of images from ImageNet dataset, where each group consists of five randomly sampled images from different classes, serving as the sample images, and one new image from one of the classes as the query image.

Different MLLM is utilized for the evaluation of our proposed method on each task. In Table 4, we provide the detailed specifications of the MLLM used in our evaluation.

**Table 4: The specifications of the MLLM used in our tasks.**
| Task | Model | LLM |
| --- | --- | --- |
| Captioning | LLaMA-Adapter v1 [] | LLaMA-7B [] |
| VQA | Honeybee-C-7B-M144 [] | Vicuna-7B [] |
| REC | Shikra-7B [] | LLaMA-7B [] |
| Few-shot classification | V2L-Tokenizer [] | LLaMA2-7B [] |

### A.2 Comparison with VVC

Figure 7 compares Reconstruction and our method in (d1) using ELIC, with the state-of-the-art traditional codec VVC (VTM 17.0 intra coding). We set the QPs to $[37,40,43,46,49]$ for VVC. It is observed that VVC performs worse than Reconstruction across all the tasks, which is potentially due to (1) the small spatial resolution (256x256) of input images that is not optimal for VVC, (2) its inferior rate-distortion performance compared to ELIC as reported in [].

Figure: Figure 7: Rate-accuracy comparison using VTM on several tasks.
Refer to caption: extracted/5759811/Figures/RD-curve/paper_sup_vvc_v2.png

### A.3 More Visualization

We present additional visualization results on four different evaluation tasks, including image captioning (Figure 8), visual question answering (VQA) (Figure 9), referring expression comprehension (REC) (Figure 10), and few-shot classification (Figure 11).

### A.4 License of Assets Used

Table 5 summarizes the used assets in our work along with their license terms.

**Table 5: List of assets used in the paper with their corresponding license.**
| Assets | Licenses |
| --- | --- |
| ImageNet [] | Custom license. Available at https://image-net.org/download.php |
| COCO [] | CC BY 4.0 |
| SEED-Bench [] | Apache 2.0 |
| LLaMA-Adapter [] | GPL-3.0 |
| Honeybee [] | Source code: Apache 2.0<br>Pretrained weights: CC BY-NC 4.0 |
| Shikra [] | CC BY-NC 4.0 |
| V2L-Tokenizer [] | No license provided.<br>Code available at https://github.com/zh460045050/V2L-Tokenizer |

**Figure 8: Visualization examples of our proposed method in (d1), Reconstruction, and Post-processing on image captioning with LLaMA-Adapter.**
| Task: Captioning<br>Model: LLaMA-Adapter |  |
| --- | --- |
| Reconstruction: A microwave and a computer sitting on a desk.<br>Post-processing: A microwave and a refrigerator sitting on top of a table.<br>Ours (d1): A microwave and a toaster oven on a counter. | BPP: 0.0725 |
| Reconstruction: Two cats are standing on the ground near a bench.<br>Post-processing: A dog and a cat are standing on a sidewalk.<br>Ours (d1): Two dogs are standing near a bicycle on a sidewalk. | BPP: 0.0928 |
| Reconstruction: A blurry picture of a blender with a knife.<br>Post-processing: A close up of a blurry image of a bug.<br>Ours (d1): A close up of a knife cutting into a pizza. | BPP: 0.0910 |
| Reconstruction: A young boy in a red shirt and tie posing for a picture.<br>Post-processing: A young boy standing in front of a wall with a clock.<br>Ours (d1): A young boy in a tie and a white shirt. | BPP: 0.0899 |

**Figure 9: Visualization examples of our proposed method in (d1), Reconstruction, and Post-processing on VQA with Honeybee.**
| Task: Visual question answering<br>Model: Honeybee |  |
| --- | --- |
| What is the dog doing in the image?<br>A. Standing still  B. Chasing after something  C. Lying down  D. Jumping in the air<br>Reconstruction: B  Post-processing: B  Ours (d1): A  GT: A | BPP:0.082 |
| How many people are on the field in this image?<br>A. Four  B. Nine  C. Twelve  D. Eleven<br>Reconstruction: D  Post-processing: B  Ours (d1): A  GT: A | BPP:0.097 |
| What is the person in the blue jacket holding?<br>A. A phone  B. Nothing  C. A wallet  D. A clipboard<br>Reconstruction: D  Post-processing: D  Ours (d1): B  GT: B | BPP:0.160 |
| How many people are lighting candles in this image?<br>A. Two B. One C. Three D. Four<br>Reconstruction: A  Post-processing: A  Ours (d1): C  GT: C | BPP:0.066 |

**Figure 10: Visualization examples of our proposed method in (d1), Reconstruction, and Post-processing on REC with Shikra.**
| Task: Referring expression comprehension<br>Model: Shikra |  |
| --- | --- |
| Guide me to the location of brown bear within the image <img> by providing its coordinates. | BPP:0.040 |
| Point me to the location of wine glass far left in the picture <img> by providing its coordinates. | BPP:0.115 |
| Can you assist me in locating right female cop in <img>, and then provide its coordinates? | BPP:0.098 |
| In the photograph <img>, could you pinpoint the location of<br>person holding a snowboard and tell me its coordinates? | BPP:0.088 |

**Figure 11: Visualization examples of our proposed method in (d1), Reconstruction, and Post-processing on few-shot classification with V2L-tokenizer.**
| Task: Few-shot classification<br>Model: V2L-tokenizer |  |  |
| --- | --- | --- |
| Query                Examples | Reconstruction: ptarmigan  Post-processing: ptarmigan  Ours (d1): walking stick  GT: walking stick | BPP: 0.0786 |
| Query                Examples | Reconstruction: hippopotamus  Post-processing: wall clock  Ours (d1): otterhound  GT: otterhound | BPP: 0.1369 |
| Query                Examples | Reconstruction: horned viper  Post-processing: horned viper  Ours (d1): hornbill  GT: hornbill | BPP: 0.2329 |

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: Please refer to Weaknesses
soundness: 3
strengths: * This paper is easy to follow
* The topic of reducing the cost of visual input in MLLMs is of great practical value
* The idea of adapting compressed image latent to MLLMs makes sense to me
summary: This paper suggests using compressed image latents for more efficient multimodal large language models (MLLMs). Specifically, a lightweight transform-neck and a surrogate loss are proposed to align compressed image latents with backbone LLMs for multimodal tasks. Experiments are conducted on some mainstream multimodal tasks, such as image captioning and VQA, and the proposed method demonstrates that it outperforms the reconstruction baseline.
weaknesses: This paper can be further strengthened by:

* One question that remains unclear about the motivation is why we need the proposed method of compressing image latent instead of existing token pruning/merging work on MLLMs such as crossget [1]. These methods can also reduce the costs of MLLMs, and I’d suggest the authors discuss the difference between existing acceleration methods for MLLMs and the proposed method, and highlight their unique contributions to efficient MLLMs.

* The proposed method shows inconsistent performance degradation on different benchmarks. For example, it shows close performance to models using uncompressed image latent on image captioning while suffering from a significant performance degradation on the POPE benchmark. It would be better to have some insight into it.

Reference

[1] Crossget: Cross-guided ensemble of tokens for accelerating vision-language transformers.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Why are different MLLMs chosen for each task instead of evaluating multiple MLLMs on each task? A consistent comparison across tasks using various MLLMs would provide a clearer understanding of the method’s generalization capability.
2. During Phase 1: Transform-neck Training, the motivation for using different loss functions at various stages across all epochs is not clearly explained, and there is a lack of corresponding ablation studies to support this design choice.
soundness: 2
strengths: 1. This paper focuses on an interesting scenario: how to achieve image compression in the context of MLLMs.
2. The writing in this paper is well-organized, making it easy to follow and understand the authors' intentions.
summary: This paper investigates adapting compressed image latents to meet the requirements of downstream vision tasks in Multimodal Large Language Models (MLLMs). The authors introduce a framework that employs a "transform-neck" structure and a surrogate loss function to adjust compressed image representations, without involving MLLMs in the training process. This design choice differs from conventional approaches that incorporate downstream models during training, making it potentially more feasible for resource-limited devices. The results show that this method achieves a balance between compression rate and accuracy across various neural image codecs and MLLMs, demonstrating its potential for efficient performance.
weaknesses: 1.  The proposed method appears to be a general image compression approach, lacking unique design elements specifically tailored for MLLMs. While the authors emphasize that this approach can achieve image compression in a resource-efficient manner to support MLLMs, they do not provide specific comparisons regarding the reduction in training costs—such as training time or computational resources—compared to methods that incorporate MLLMs directly into the training process.
2. The proposed  SURROGATE LOSS is not novel, easily combination of the Mean Squared Error loss and the cross-entropy loss.
3. The proposed method has limited applicability, as it requires alignment with specific vision encoders. The authors only align with CLIP, whereas many advanced MLLMs now use other, more powerful vision encoders, such as Intern-ViT or SGLIP. This method may need further adaptation to support these alternative encoders effectively.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Will you plan to release the code ?
soundness: 3
strengths: 1. The paper introduces a lightweight transform-neck and a surrogate loss function, which together reduce computational complexity and avoid the need to back-propagate through the massive MLLMs, making the training process more efficient.
2. The proposed framework is generic and can accommodate various neural image codecs and MLLMs that share the same visual encoder, enhancing its versatility and applicability across different models and tasks.
3. The paper demonstrates the effectiveness of the proposed method through extensive experiments on different neural image codecs and MLLM-based vision tasks, showing improved rate-accuracy performance and less complexity compared to existing methods.
summary: The paper introduces a novel framework designed to bridge compressed image latents with Multimodal Large Language Models (MLLMs) for downstream vision tasks. It addresses the challenge of deploying MLLMs on resource-constrained devices by proposing a lightweight transform-neck and a surrogate loss to adapt compressed images for MLLM-based vision tasks without the need for decoding the images. This approach reduces computational complexity and is generic, applicable to various neural image codecs and MLLMs sharing the same visual encoder.
weaknesses: 1. The paper's approach relies on the assumption that the downstream MLLMs will use the same pre-trained CLIP visual encoder, which may limit the applicability of the proposed method to MLLMs that employ custom or different visual encoders.
2. The paper does not provide a comprehensive comparison with existing image compression methods beyond the context of MLLMs, which could be important for understanding the method's performance in a broader range of applications

---

confidence: 2
contribution: 2
initial_rating: 3
presentation: 2
questions: Please refer to the Weaknesses
soundness: 2
strengths: 1. This paper claims to be the first work specifically targeting neural image coding for MLLMs.
2. It introduces a lightweight transform-neck that aligns the encoded image latents with the MLLM visual encoder outputs, thereby avoiding the decode operation in neural image coding and reducing computational complexity.
3. The proposed surrogate loss allows updating the neural image coding model via the Visual Encoder without backpropagating through the entire MLLM, hence reducing the computation required.
4. The proposed method is applicable to different downstream MLLMs, applications, and neural image codecs.
summary: This paper explores methods to compress and decode images such that the decoded images can be successfully used with Large MLLMs, addressing the substantial overhead required for transmitting images to cloud-hosted MLLMs. Existing image encoding works for machine vision face a common problem: MLLM models are too large to train the neural image codec. To address the computational overhead of image compression and encoding in the context of MLLMs, this paper proposes a lightweight transform-neck and a novel surrogate loss. The lightweight transform-neck aims to align the features of the compressed images with the outputs of the visual encoder, avoiding the need for fully image decoding. The proposed surrogate loss allows the neural image codec to be trained using only the vision encoder, avoiding the need for backpropagating through the entire MLLM. The proposed method can be applied to different MLLMs and various neural image codecs in various application scenarios.
weaknesses: 1. The motivation of this paper—studying image compression under MLLM to avoid transmission overhead—seems to lack practical significance. The primary goal of image compression is to reduce image resolution, which might be meaningless for MLLMs:
* First, most MLLMs [1] use CLIP Image Encoder, which limits image resolution to 224x224. This resolution does not incur significant transmission overhead in current network communications and is negligible compared to the inference time of MLLMs.
* Second, while some works [2] explore using high-resolution images as input, they usually crop high-resolution images into several low-resolution sub-images, often achieving better performance compared to compressed images. Therefore, researching how to compress images as MLLM input seems unnecessary.
* Lastly, the authors do not explain the advantages of using compressed images as MLLM input over exploring more compact Image Encoders [3].

2. The proposed method requires training different transform-necks and codecs for different downstream tasks, further leading me to question the necessity and rationale of this work.

Reference
* [1] Visual Instruction Tuning, NeurIPS 2023
* [2] About When do we not need larger vision models?, ECCV 2024
* [3] TokenPacker: Efficient Visual Projector for Multimodal LLM, arXiv 2024

Meta-review: This paper studies using neural image compression to reduce the transmission rate between devices and servers for Multimodal Large Language Models. The proposed method involves a lightweight transform-neck, which is trained with surrogate loss to adapt image latents for MLLMs. The motivation of the paper is clearly explained and the paper is of good writing quality. The main weakness is the degradation on some benchmarks, which is also reported in previous literature. Given the detailed discussions of the methods in the paper and in the rebuttal, I would recommend accepting the paper.

Decision: Accept (Poster)