CMC-Bench: Towards a New Paradigm of Visual Signal Compression

## Abstract

Abstract Ultra-low bitrate image compression is a challenging and demanding topic. With the development of Large Multimodal Models (LMMs), a Cross Modality Compression (CMC) paradigm of Image-Text-Image has emerged. Compared with traditional codecs, this semantic-level compression can reduce image data size to 0.1% or even lower, which has strong potential applications. However, CMC has certain defects in consistency with the original image and perceptual quality. To address this problem, we introduce CMC-Bench, a benchmark of the cooperative performance of Image-to-Text (I2T) and Text-to-Image (T2I) models for image compression. This benchmark covers 18,000 and 40,000 images respectively to verify 6 mainstream I2T and 12 T2I models, including 160,000 subjective preference scores annotated by human experts. At ultra-low bitrates, this paper proves that the combination of some I2T and T2I models has surpassed the most advanced visual signal codecs; meanwhile, it highlights where LMMs can be further optimized toward the compression task. We encourage LMM developers to participate in this test to promote the evolution of visual signal codec protocols.

## 1 Introduction

Visual signal compression aims to minimize image data, playing a crucial role in delivering high-quality image/video services with limited network resources and storage capacity. Since the MPEG-1 [] standard was introduced, compression rates for visual signals have doubled  [, , , ] every decade. In recent years, traditional image codecs have achieved a 500 times compression rate while ensuring a decent visual experience for humans. However, traditional codecs are approaching the Shannon limit of 1,000 times Compression Rate (CR) in the upcoming next-generation protocols. Fortunately, the rapid development of Large Multimodal Models (LMMs) has opened up possibilities for such Ultra-low-Bitrate (ULB) compression.

Why use LMMs for compression? LMMs support the conversion between multiple modalities, where text consumes much less space than image modalities. By cascading Image-to-Text (I2T) and Text-to-Image (T2I) models, images can be compressed and reconstructed from semantic information. This Cross-Modality Compression (CMC) paradigm operates at the semantic level, which outperforms traditional codecs at the pixel level. It enables easy attainment of ULB, and even the Extreme-low Bitrate (ELB) for CR about 10,000 times.

However, at such low bitrates, CMC presents two significant issues that cannot be overlooked. (1) Consistency: The reconstruction process heavily relies on intermediate textual information. Any omission by the I2T model (encoder) or misunderstanding by the T2I model (decoder) can result in severe distortion. Unlike minor changes in brightness and color, this can lead to a semantic-level inversion of the entire image. (2) Perception: Textual encoding provides a coarse representation of the image, necessitating the T2I model to add details. Insufficient details degrade perceptual quality, while excessive ones compromise faithfulness to the original image. Unfortunately, as the bitrate decreases, the conflict between them  [, ] becomes more pronounced. As the consistency and perception failure cases in Figure 1, these issues jointly limit the application of CMC.

For LMMs, there is a lack of effective evaluation criteria both in terms of consistency and perceptual aspects. Although numerous benchmarks have recently emerged for LMMs, they are primarily designed to assess the performance of either I2T (Image-to-Text) or T2I (Text-to-Image) models working alone, such as captioning/visual question answering for I2T  [, ], or generation quality/realism for T2I  [, ].
Consequently, we introduce the first joint benchmark called CMC-Bench, aimed at testing the collaborative capabilities of I2T and T2I models. Our contributions include:

- •
A large-scale dataset consists of 58,000 images using the CMC paradigm. 4,000 images among them have 160,000 expert annotations, covering both consistency and perception issues, paving the way for information loss modeling in the I2T and T2I processes.
- •
A comprehensive evaluation standards, consisting of four compression modes under different requirements, along with the two dimensions mentioned above. We validate mainstream models (including 6 I2T and 12 T2I) to explore optimal combinations.
- •
A throughout comparison with traditional codecs. We compared the benchmark winner with existing image codecs, revealing the significant advantages of the CMC image compression paradigm and some remaining drawbacks. We encourage LMM developers (both I2T and T2I) to participate in CMC-Bench to further expand the application of CMC.

## 2 Related Works

Cross-Modality Compression. The earliest CMC method [] emerged in 2021, achieving a compression ratio of almost 10,000 times through text modality. However, as a simple combination of I2T and T2I models, their results often exhibit noticeable differences from the original images. Subsequently, Text+Sketch [] employed edge operators and ControlNet [] to refine CMC, but its consistency remained inferior to traditional codecs. The most advanced CMC methods, like M-CMC and MISC  [, , , , , ], have surpassed advanced codecs like VVC [] in both consistency and perception, indicating the promising future of this paradigm. Nevertheless, there is still room for improvement in these two aspects. All existing CMC methods are from one specific I2T and one T2I model, and the models used are relatively outdated. Considering the rapid development of Generative-AI, how to combine the latest models towards a better CMC becomes an unrevealed question.

Benchmark for LMM Evaluation. Existing benchmarks are mainly designed for T2I and I2T models. For I2T, they usually take a specific image sequence as input, compare the text output by LMM with the ground truth, and use the relevance of the two as a performance indicator. The annotation content includes common sense  [, ] or specific expert fields  [, , , ]. For T2I, the input is a carefully designed text prompt  [, ] (e.g. different themes, adjectives, and spatial relationship). They use specific visual encoders to process the output image of LMM and determine its alignment with the text as the generative performance  [, ]. However, as the current CMC paradigm is still immature, there is no pipeline for the joint evaluation of I2T+T2I model.

Table: Table 1: Comparison of image compression related dataset with subjective labeling.

Benchmark for Image Compression. Given the significance of visual information compression, several related competitions [, ] have been held in recent years. However, these competitions often limit their scope to Natural Scene Images (NSIs). Screen Content Images (SCIs)  [, ], which are prevalent on the internet, and the emerging AI-Generated Images (AIGIs)  [, , ] have received some attention with new datasets, but no existing dataset comprehensively considers them together. Moreover, the performance evaluation of compression algorithms can be challenging, often requiring subjective quality assessments from human viewers to train Image Quality Assessment (IQA)  [, , , , , , , ] models, which provide objective metrics for compression algorithms. In the context of ULB image compression, both the consistency between the distorted and reference images, as well as the inherent appeal of the distorted image in human perception, need to be annotated. Existing IQA datasets typically annotate only one aspect, while often in a coarse-grained manner through Single Stimulus (SS) or Double Stimulus (DS) comparisons. In contrast, Mean Opinion Score (MOS) derived from multiple subjects offers a more detailed and objective evaluation as shown in Table 1.

## 3 Dataset Construction

### 3.1 Ground Truth Selection

To provide a comprehensive and high-quality resource for various applications on the Internet, we carefully curated 1,000 high-quality images without compression distortion as the ground truth of CMC-Bench. Among them, NSIs are the most mainstream content, so we selected 400 images. At the same time, considering that SCIs are more common on screens and AIGIs are increasing on the Internet in the upcoming LMM era, we selected 300 images from each of these two categories. The specific content is as shown in Figure 2.

Figure: Figure 2: Source data illustration of CMC-Bench from three content types.
Refer to caption: x2.png

NSI: A collection of 200 high-quality Professional Generated Content (PGC) released by TV stations and photographers, specifically sampled from the CLIC database []; and 200 User Generated Content (UGC) by average users, selected from MS-COCO []. To ensure image quality, we employed Q-Align [] to filter out low-quality UGC that might be overexposed.

SCI: Consisting 100 computer graphics from CGIQA-6K [] in animated movies; 100 game renders from CCT and CGIQA-6K  [, ]; and 100 webpages with both images and text from CCT, SCID, and Webpage Saliency datasets  [, , ]. To maintain frame clarity, we also applied Q-Align [] to address factors like motion blur that affect visual quality.

AIGI: Comprises 50 images each, generated by 6 latest models: DALLE3, MidJourney, PG v25, PixArt $\alpha$, SDXL, and SSD-1B  [, , , , , ]. They have demonstrated exceptional preference in previous subjective ratings  [, , , , ], representing the pinnacle of AIGI capabilities.

### 3.2 Compression Mode

Drawing on previous work in CMC, we categorize CMC into four working modes, as shown in Figure 3. Each type employs distinct configurations and is suitable for different scenarios:

Text: The I2T model converts images to text and is directly restored by the T2I model. Due to its reliance on the text modality only, this approach achieves a CR of 10,000, ideal for ELB situations.

Pixel: Each $64\times 64$ blocks from ground truth are merged and quantized into one pixel. Beyond the Text mode, these pixels initialize the T2I process. The pixel representation is relatively compact, offering a CR of around 5,000, suitable for less rigorous ELB but higher demands on consistency.

Image: Traditional codecs are employed to compress the image, which serves as input for the T2I model for enhancement. Unlike the previous two, it omits the time-consuming I2T process by leaving the text input of the T2I model empty. This approach can achieve a CR of 1,000, suitable for ULB bandwidth but with high real-time requirements.

Full: Extending the Image mode, the T2I is guided by text content, encompassing the full pipeline of I2T, traditional codec, and T2I. It also has a CR of approximately 1,000, suitable for the most demanding performance scenarios.

### 3.3 Benchmark Candidates

We employ 6 I2T and 12 T2I models across four compression modes. Due to the absence of text, the I2T model is not used in the Image mode; while for T2I, among the 4 Image Reconstruction (IR) models requiring an initial image and are not compatible with Text and Pixel modes. The remaining 8 T2I generative models support all modes. We use one certain T2I, and validate all possible I2T models to verify their performance separately (vice versa for T2I validation).
For a fair comparison,
We fixed RealVis [] to minimize the T2I process distortion, which ensures the performance fluctuation mainly comes from the I2T.
Similarly, we fix I2T as GPT-4o [] when validating T2I models.
Each I2T model produces 3,000 images, while restorative and generative models for T2I have 2,000 and 4,000, respectively. A total of 18,000 + 40,000 = 58,000 images are generated.

I2T model: GPT-4o [], LLAVA-1.5 [], MPlugOwl-2 [], Qwen [], ShareGPT [], and InstructBLIP []. Except for one model [] for image captioning with default token length, we modify the output length of others to 10$\sim$20 words for a balance between bitrate and performance.

T2I model: Animate [], Dreamlike [], PG20 [], PG25 [], RealVis [], SD15 [], SDXL [], and SSD-1B [] as generative model; DiffBIR [], InstructPix [], PASD [], and StableSR [] as IR model. A higher denoising strength indicates a more obvious modification on the starting point. To balance the consistency and perception indicators, we set the strength of Full and Image modes as 0.5, the Pixel mode as 0.8, and the Text mode as the default 1.

Traditional codec: For Full and Image mode, we utilize the most advanced traditional codec VVC [] to provide a reference image.
Towards 1,000 times compression, we take its nearest bitrate that meets the ULB requirement, where the Quantizer Parameter (QP) is 53.

Figure: Figure 3: Illustration of 4 working modes of CMC. Text mode roughly reconstructs the semantic information, Pixel mode slightly improves low-level consistency, Image mode provides a similar structure towards ground truth but a different character, and Full mode has the best performance.
Refer to caption: x3.png

### 3.4 Human Preference Annotation

Referring from previous large-scale subjective annotation [] methods, we do not perform coarse-grained labeling on the entire dataset considering the limitation on annotator numbers. Instead, we fine-grain the annotations on 4,000 images to ensure multiple ratings for each image. Note that, as the benchmark indicator should be adjusted on subjective data, we did not directly select subsets from the 58,000 test images. Instead, we generated new images to prevent prior exposure to the content being evaluated.
Given the greater impact of T2I models on CMC tasks than I2T models, we follow the T2I paradigm described in Section 3.3. The I2T model is fixed as GPT-4o [] and combined with 12 different T2I models, compressing 100 ground truth into 4,000 distorted images. To ensure quality diversity, we randomly assigned strength from 0.2 to 0.9 rather than a fixed value. Each distorted image is paired with its corresponding ground truth and shown to 20 experienced participants who provided ratings on consistency and perception dimensions.
Each image is then summarized into two overall scores from 0 to 5, combining all participants’ feedback. For a detailed description of the experimental setup and data process, please refer to the supplementary.

## 4 Experiment

### 4.1 Evaluation Indicator Settings

All 6 I2T and 12 T2I LMMs are verified and tested by different parameters and fixed them towards an optimal situation according to Section 3.3, while the internal model weight remains zero-shot to ensure fairness in ranking. The specific parameters verification is provided in the supplementary. Taking the reference and distorted image pairs as input, We use TOPIQ [], the most advanced IQA metric in Full-Reference (FR) and No-Reference (NR) configuration to characterize consistency and perception. The average score of 1,000 ground truth images is reported as the final performance. Combining these two issues towards 4 working modes, the models are evaluated by 8 indicators for generative T2I, 6 indicators (exclude Image mode) for I2T, and 4 indicators (exclude Pixel and Text mode) for T2I restorative models. A weighted average of $2\times$ FR indicators and $1\times$ NR indicators is given as the overall score for ranking since the TOPIQ-FR has a smaller floating range than TOPIQ-NR. As restorative models only support ULB compression in Full and Image mode, but not ELB compression in Pixel and Text mode, the overall score of the T2I model is ranked under ULB and ELB respectively.

Table: Table 2: Correlation between objective IQA evaluation and subjective human preference.

In addition to TOPIQ, we also used four cutting-edge FR-IQA (AHIQ [], DISTS [], LPIPS [], PieAPP []) and NR-IQA (CLIPIQA [], CNNIQA [], DBCNN [], HyperIQA []) algorithms to objectively score the distorted images in terms of consistency and perception. The higher the Spearman ($\sigma$) and Kendall Rank-order Correlation Coefficient ($\kappa$), the better correlation between the objective and subjective scores. All models are trained on 80% of the distorted images in Section 3.4 and tested on the remaining 20%. Experiments in Table 2 show that the correlation between the fine-tuned TOPIQ [] and the subjective score is outstanding with $\sigma$ beyond 0.9 in both dimensions, making it appropriate performance indicators reflecting human preference for compressed images.

Figure: Figure 4: A radar map illustrates the collaboration of mainstream I2T (left) and T2I (right) LMMs. The model are tested as {6 different I2Ts + RealVis []} and {GPT-4o [] + 12 different T2Is}.
Refer to caption: x4.png

### 4.2 Benchmark Result and Discussion

Figure 4 shows the performance of 6 I2T models as encoders and 12 I2T models as decoders in image compression. For I2T, considering the different lengths of intermediate text, we show the bit-per-pixel (bpp) of each model together with the performance index, where ULB and ELB correspond to 0.024 and 0.0024 bpps respectively.

Table: Table 3: Benchmark result in Full and Image modes for 8 T2I generative and 4 restorative models at ultra-low bitrate (1,000 times compression). NSI/SCI/AIGI stands for the compressed image types. FR and NR indicate consistency and perception scores. [Key: Best; Second Best]

Table: Table 4: Benchmark result in Pixel and Text mode for 8 T2I generative models at extremely-low bitrate (10,000 times compression). [Key: Best; Second Best]

Table: Table 5: Benchmark result in Full and Pixel mode for 6 I2T models. [Key: Best; Second Best].

For 6 indicators in I2T LMMs, while GPT-4o [] does not perform well on Text-FR, it significantly outperforms on Full-FR. This suggests that although its generated text carries limited information, it has a strong orthogonal relationship with the low-level details of the image. This semantic information effectively compensates for the information loss after image compression. In addition, the given text facilitates the subsequent T2I model in decoding high-quality images, and its performance across various NR indicators is also commendable. In comparison, MPlugOwl-2 [] and InstructBLIP [] can effectively encode images into text, but their results are still inferior to GPT-4o. The only viable competitor is ShareGPT [], but it has a bpp of around 0.008, which is significantly larger than the other 5 models. This data size exceeds ELB and occupies one-third of the available ULB space. Considering multiple factors, GPT-4o remains the most suitable I2T model as the CMC encoder.

For 8 indicators in T2I LMMs, 2 restorative models  [, ] exhibit overwhelming consistency in Full and Image modes with acceptable perception results, enabling faithful image reconstruction close to the ground truth. However, its applicability is limited for the other 2 modes, particularly under the strict ELB conditions. The performance of the remaining models falls into two distinct extremes, where RealVis [] shows high consistency but PG25 [] demonstrates high perception. Given that it is feasible to enhance a compressed low-quality image with high fidelity to the original, while correcting a completely different high-quality image with low fidelity remains challenging, we opt to prioritize consistency by assigning it a higher weight. Consequently, considering the strong performance and wide versatility of RealVis, it is relatively more suitable than the CMC decoder.

To delve into the compression capability of I2T and T2I LMMs with different content on various modes, we present the T2I leaderboard under ULB and ELB conditions in Table 3 and Table 4, respectively, and showcase the performance of I2T models on Full and Pixel modes (Text mode attached in supplementary) in Table 5, with a discussion of content-specific analysis. A horizontal comparison among different modes in Tables 3 and 4 reveals that the Full mode has a clear advantage over the Image mode in terms of consistency and perception, indicating the significance of the text provided by the I2T model for T2I decoding. This text guidance not only enhances consistency but provides a clear target for the T2I process, thus also boosting perception. In contrast, the Pixel mode sacrifices perception for consistency compared to the Text mode. This is because the more control conditions added, the less room for creative freedom the model has, leading to a decrease in image aesthetics. However, for models that already have high perception scores [, ] in the Text mode, the trade-off of improving overall performance is acceptable.

Among NSI, SCI, and AIGI, different LMMs excel at different content. For instance, as shown in Table 3 and Table 4, PG25 [], trained on internet data, performs better in AIGI tasks; conversely, RealVis aims at image naturalness, manifesting its superior reconstruction capability in NSI. Regardless of the model employed, we observe that NSI generally yields higher consistency scores, while AIGI has higher perception scores. However, SCI stands out from the others, with the compression results of the same model lagging behind in both perception and consistency. This deficiency is relevant to certain words [] (even long paragraphs) within SCI, making I2T models unable to re-encode them into text, while the text generation capabilities of recent T2I models are still limited.
Besides, although the performance disparities among I2T models are not as significant as those in T2I models, Table 5 also clearly illustrates the limitation in SCI, indicating room for further optimization.

Figure: (a) Full and Image modes for 8 T2I generative and 4 restorative models
Refer to caption: x6.png

### 4.3 Subjective Data Analysis

Figure 5 presents the subjective preference for images decoded by 12 T2I models under ULB and 8 models under ELB. For ULB, the 3 restorative models  [, , ] exhibit slightly higher consistency compared to generative models, where PG25 achieves the highest perception score against all others. It is worth noting that the restorative models are more robust. The upper and lower bounds of the scores in each dimension seldom surpass 1.0, whereas the randomness of the generative models notably deteriorates. As the bitrate further decreases to ELB, consistency scores of all models decline, while perception scores have slight improvement. In summary, apart from Animate [] specifically for cartoon styles, and InstructPix [] that significantly alters images, all other models demonstrate potential applications in CMC. Additionally, by averaging all scores, we find that the models ranking based on subjective scores aligns closely with the objective ones shown in Table 3 and Table 4. This finding validates the reasonability of our previous experiments and highlights that, compared to perception, humans tend to focus more on consistency when viewing compressed images.

### 4.4 Compare with Traditional Codecs

Figure: Figure 6: Comparison of CMC-Bench winners against existing image codecs, evaluated by 4 consistency and 4 perception metrics indicated by marked and plain background. The combination of I2T and T2I models generally exceeds the existing codecs under the same bitrate.
Refer to caption: x8.png

To validate the practicality of the CMC paradigm, we select 2 outstanding combinations of I2T and T2I models from CMC-Bench, and compare them with 3 mainstream codecs: AVC [], HEVC [], and VVC [] at I-Frame mode, and the latest semantic codec pipeline CDC []. Given the superiority of GPT-4o [] as the encoder, we initially pair it with the top-ranked decoder DiffBIR []. Considering applications on different modes, excluding the reconstructive model, we also assess its performance with the third-tanked decoder RealVis []. These two approaches with four bitrates correspond to Text, Pixel, Image, Full modes are shown in Figure 6. To comprehensively compare the two paradigms across different dimensions, we add 3 Consistency metrics: CLIPSIM [], LPIPS [], and SSIM []; and 3 Perception metrics: CLIPIQA [], LIQE [], and FID []. Models ranked higher prioritize semantic information, while those lower focus on pixel-level consistency.

Both CMC paradigms demonstrate an advance in terms of most metrics. Given that SSIM is purely pixel-based, the performance drop due to generative compression is expected. The lead in perception is particularly notable, as it surpasses traditional codecs at extremely low bitrates. However, the advantage in consistency is relatively smaller, achieving a reduction of around 30% in bitrate compared to traditional methods at 0.02 bpp. The DiffBIR decoder generally shows better performance, while RealVis fits A wider range of bitrates.
In summary, based on the above analyses, we believe that CMC holds a certain advantage over traditional encoding. However, for implementing LMMs into the next generation of visual signal codecs, further optimization is required in the following aspects:

Enhanced T2I models: Both encoders and decoders are crucial in CMC, but decoders are more decisive. Future T2I models should possess more sophisticated control mechanisms, ensuring high-quality generation while maintaining consistency with reference images and text.

Better adaption to SCI: the compression performance of SCI is inferior to NSI and AIGI, necessitating LMMs with specialized understanding and generating mechanisms to handle SCI.

Wider bitrate range: Although leading in ULB and ELB, the margin of consistency improvement is not as pronounced as perception. Future efforts should focus on CMC at higher bitrates, incorporating more control information to aid in reconstructing the original image, ultimately achieving superiority across all bitrates and dimensions as compared to traditional codecs.

## 5 Conclusion

We construct CMC-Bench, a benchmark for assessing the collaborative functioning of I2T and T2I models in image compression. Anticipating the bitrate requirements for codecs in the next decade, we proposed four collaboration modes among LMMs, along with two indicators of consistency and perception. By employing 6 mainstream I2T and 12 T2I models, we collected 58,000 distorted images through CMC with 160,000 human subjective annotations to train objective metrics for comprehensive evaluation. Our assessment demonstrates that even without dedicated training for compression tasks, combinations of several advanced I2T and T2I models have already surpassed traditional codecs in multiple aspects. However, there is still a long way to go before LMMs can directly become the future codecs paradigm. We sincerely hope that CMC-Bench will inspire future LMMs to perform better compression towards the evolution of visual signal codecs.

## Appendix A Appendix

In this section, we briefly describe the content of the checklist requirements. Considering that our experiments tried a variety of parameter configurations, the conclusions under different configurations are also stated here, including specific ablation data.

### A.1 Limitations and Social Impact

Limitation 1: Although we have considered most of the mainstream I2T and T2I models in CMC-Bench (till March 2024), the number of models is still insufficient to fully characterize the performance of all current LMMs on CMC. Taking the open-source T2 model as an example, more than 20,000 models have been released on huggingface (till May 2024). Although we cannot run all models, the capabilities of some relatively unpopular or more advanced LMMs in the future need to be further updated on CMC-Bench.

Limitation 2: CMC-Bench is currently designed for the performance verification of image compression, not video compression. Considering that the temporal information of videos is relatively complex, the current LMMs are only applicable to image compression, which makes it difficult to ensure consistency with the reference when generating videos. However, as LMMs gradually apply to video compression in the future, CMC-Bench will also be evaluated at the video level.

Social Impact: Through the CMC paradigm, the size of the image can be compressed by 1,000 times, and even 10,000 times in extreme cases. This will effectively promote image communication between a large number of terminals under limited bandwidth, thereby realizing multi-device collaboration in the Internet of Things and semantic communication. Considering that traditional codecs have encountered bottlenecks after three decades of development and the compression rate is gradually approaching the Shannon limit, we believe that LMM will effectively achieve semantic-level compression and thus become the future evolution direction of visual information codec protocol.

### A.2 Subjective Annotation Settings

Compliant with the ITU-R BT.500-13 [] standard, we invited 20 viewers (11 male, 9 female) in this subjective experiment with normal lighting levels. Images are presented on the iMac display together with the ground truth in random order on the screen, with a resolution of up to 4096 $\times$ 2304. Both ground truth and distorted images are accessible for subjective. Considering the consistency between the reference and distorted image, and the perceptual quality of the only distorted image, subjects were asked to give two scores within the range of [0, 5], where each one-point interval stands for poor, bad, fair, good, or excellent quality. The user interface is shown in Figure A1.

Figure: Figure A1: Subjective annotation interface presenting with the distorted (left) and reference (right) image. Each viewer is asked to provide (1) a Consistency score between two images from 0 to 5; (2) a Perception score of the distorted image; (3) an NSFW flag when they feel offended.
Refer to caption: extracted/5662848/fig-supp/interface.png

Each user, in accordance with the Helsinki Declaration, provides informed consent for their data to be used in experiments. To prevent NSFW content, we implement three preventive measures: (1) Conduct a thorough manual screening of the ground truth; (2) Utilize the SD safety checker [] during decoding; (3) Incorporate an ‘offensive’ flag in the annotation process, allowing viewers to report NSFW content if encountered. The data confirms that the ground truth is safe, with approximately 0.2% of distorted images receiving reports, which is generally acceptable.

In case of visual fatigue, we split the database into $g\in[0,10]$ groups including $M=400$ images each, while limiting the experiment time to an hour. After collecting every viewer’s quality ratings, we compute the Spearman Rank-order Correlation Coefficient (SRoCC) between them and the global average and remove the outliers with SRoCC lower than 0.6. Then we normalize the average score $s$ for between each session to avoid inter-session scoring differences as:

$$ $s_{ij}(g)=r_{ij}(g)-\frac{1}{M}\sum_{i=0}^{g\cdot M-1}r_{ij}+2.5,$ (1) $$

where $(i,j)$ represent the index of the image and viewer and $r$ stands for raw score. We observed a fairly even distribution of subjective scores on both dimensions and bar graphs for each score range are provided in the Supplementary. Then subjective scores are converted to Z-scores $z_{ij}$ by:

$$ $z_{ij}=\frac{s_{ij}-\mu_{j}}{\sigma_{j}},$ (2) $$

where $\mu_{j}=\frac{1}{N}\sum_{i=0}^{N-1}s_{ij}$, $\sigma_{j}=\sqrt{\frac{1}{N-1}\sum_{i=0}^{N-1}(s_{ij}-\mu_{i})^{2}}$ and $N=10$ is the number of subjects, which is finally reported as MOS, namely golden user annotations. The distribution of Consistency and the Perception MOS is shown in Figure A2, which proves that extremely low and high scores are rare, and most scores are between 1 and 4. The Perception score is concentrated in the medium-low area, while the Consistency dimension tends to be moderately high.

Figure: Figure A2: Mean Opinion Score distribution of the Consistency and the Perception dimension.
Refer to caption: x17.png

### A.3 Experimental Platform

For 4,000 labeled image pairs, we trained five FR and five NR quality indicators for 50 epochs using Adam optimizer on a local NVIDIA GeForce RTX 4090. Among which 80/20 for training/testing. We take MSE loss with a learning rate at $2\times 10^{-5}$. The TOPIQ-FR and TOPIQ-NR are set as objective indicators for Consistency/Perception. Noted these 4,000 training data images are not included in the source data for objective evaluation for a fair comparison.

The LMMs are validated on a server with four NVIDIA RTX A6000, using I2T in different output lengths, and T2I in different strengths, combining through Text/Pixel/Image/Full modes.

### A.4 Image-to-Text Model Configuration

Towards different output lengths, we applied different prompts as the input for I2T models. The prompt follows previous CMC templates [], with length [5,10,20,50]. For example, to describe Figure A3 in different lengths, the input prompt format and output text from GPT-4o [] are:

Figure: Figure A3: Example ground truth.
Refer to caption: extracted/5662848/fig-supp/NSI_pgc_002.png

Q(5): ‘Generate an informative paragraph in 5 words based on the image.’
A(5): ‘Traditional attire on serene river.’

Q(10): ‘Generate an informative paragraph in 10 words based on the image. Considering Light, Color.’
A(10): ‘Two individuals navigate a wooden boat in traditional attire, serene.’

Q(20): ‘Generate an informative paragraph in 20 words based on the image. Considering Light, Color, Clarity.’
A(20): ‘A woman in traditional attire and a conical hat, seated on a wooden boat, with another person rowing on water.’

Q(50): ‘Generate an informative paragraph in 50 words based on the image. Considering Light, Color, Clarity, Dense Caption, and Region Semantic.’
A(50): ‘A woman in a purple dress sits near the front of a boat, her hand adjusting a straw hat. Behind her, a person in a beige top and floral pants stands near the boat’s wooden oars. The setting is a calm, expansive body of water under a cloudy sky.’

Figure: Figure A4: Setting I2T model with different output lengths for CMC task in Full mode, evaluated by 4 consistency and 4 perception metrics indicated by marked and plain background. A moderate output length can realize a satisfying performance while saving bitrate.
Refer to caption: x18.png

To explore how much information the above four output lengths can represent. We use different I2T models, and combine them with the most effective T2I model (RealVis []) under the above four output lengths, and use four Consistency and four Perception indicators for analysis, as shown in Figure A4, where the four datapoints of each curve represent four output lengths. The experimental results show that most I2T models can dynamically adjust the output length, except that InstructBLIP for image annotation cannot input prompt, and ShareGPT is not sensitive to the specified output length. Overall, when inputting Q(5), the reconstruction effect is relatively poor because of the short output; when inputting Q(50), the overly long paragraph from the I2T model cannot be understood by the T2I model, so the performance is not significantly improved while wasting bitrate. By observing the trend of all curves, we find that when the bpp of the text is between 0.002-0.003, the balance between performance and bitrate can be achieved. Therefore, for each model, we choose the output length closest to this bitrate, that is, Q(20) for GPT-4o [], MPlugOwl-2 []; Q(10) for LLAVA [], Qwen [], ShareGPT []; and the default length for InstructBLIP [].

### A.5 Text-to-Image Model Configuration

In Text mode, since there is no reference image as a starting point, the denoising strength is undisputedly 1. In the other three modes, we adjusted different intensities with a granularity of 0.1. For Full and Image modes that provide a reference image, a high denoising strength will waste the information of this reference, so we verified the performance under strength from 0.2 to 0.8; for Pixel mode, since the pixel provides less information than the compressed image, we increased the strength and range from 0.4 to 0.99 (as strength=1 will ignore the reference).

The verification of Full/Image/Pixel results are shown in Figure A5/A6/A7 respectively, using same 4 Consistency and 4 Perception indicators. In general, as the strength increases, the Consistency index increases first and then decreases, while the Perception index continues to rise. This is because the greater the strength, the more details the T2I model adds to the image, thereby improving the Perception score. However, for Consistency, the added details at low strength can indeed make up for the unclear areas in the reference image, thereby performing restoration; but when the strength increases, the added details are inconsistent with the original image, and instead bring negative optimization to the reference image. Thus, a good strength requires a trade-off between Consistency and Perception. Taking both dimensions into consideration, we set the strength of Full and Image mode to 0.5 and the Pixel mode to 0.8.

Figure: Figure A5: Setting T2I model with different strength for CMC task in Full mode, evaluated by 4 consistency and 4 perception metrics indicated by marked and plain background. A strength of 0.5 can reach a balance between consistency and perception.
Refer to caption: x27.png

Figure: Figure A6: Setting T2I model with different strength for CMC task in Image mode, evaluated by 4 consistency and 4 perception metrics indicated by marked and plain background. A strength of 0.5 can reach a balance between consistency and perception.
Refer to caption: x36.png

Figure: Figure A7: Setting T2I model with different strength for CMC task in Pixel mode, evaluated by 4 consistency and 4 perception metrics indicated by marked and plain background. A strength of 0.8 can reach a balance between consistency and perception.
Refer to caption: x44.png

### A.6 Applicability on Different Reference Image

In the main text, the VVC provides the reference image with QP=53. In Table A1 we further use VVC with QP=51,48,45 as the reference image for T2I model denoising to perform CMC. At higher bitrates, CMC still has an overwhelming advantage over VVC in the Perception metric. Except for SSIM, CMC achieves comprehensive optimization of all other indicators compared to traditional codecs, but the optimization range gradually decreases with the increase of bitrate. Moreover, once the QP is lower than 45, it will fall behind in the Consistency indicators. In summary, compared with traditional codecs, CMC can achieve an overall improvement in Perception and Consistency at low bitrates. However, when bpp increases to 0.1 or above, the improvement in Perception comes at the cost of Consistency. This indicates that ideal performance at higher bitrates is an important factor when using LMMs for image compression.

Table: Table A1: Changing reference image as different VVC QP level in Full mode before CMC decoding, evaluated by 4 consistency and 4 perception metrics indicated by marked and plain background.

### A.7 Example Result Visualization

The CMC result visualization is shown from Figure A8 to A13, all result use GPT-4o [] as encoder, and Animate[]/ Dreamlike[]/PG20[]/PG25[]/RealVis[] as decoder (from left to right). Four working modes Full/Image/Pixel/Text are all included (from top to bottom). For different modes, the compression results from LMMs show that as the bitrate decreases, the decoded image is more different from the ground truth. Among them, the Full mode can obtain results generally similar to the ground truth; the Image mode will lose some semantic details while introducing artifacts; the Pixel mode loses more details but ensures the consistency of the overall composition; and the result generated by Text is significantly different from the ground truth.

For the performance of the CMC on different contents, Figure A8/A9 reveals it performs most satisfactorily on AIGIs; Figure A10/A11 indicates it can also obtain results consistent with ground truth on NSIs, but it is easy to lose details such as human faces and vehicle signs; Figure A12/A13 implies it is the least ideal on SCIs, as it misunderstands the relationship between characters in the movie or games, and cannot draw formed letters on webpages. In conclusion, CMC is a promising visual signal compression method, but to become a universal codec standard in the future, the robustness to all content types needs to be improved.

Figure: Figure A8: Visualization of an AIGI (Human) on the left after CMC. Row: Full/Image/Pixel/Text mode. Column: Animate[]/ Dreamlike[]/PG20[]/PG25[]/RealVis[] as decoder.
Refer to caption: extracted/5662848/fig-supp/exp-aigi-0.png

Figure: Figure A9: Visualization of an AIGI (Object) on the left after CMC. Row: Full/Image/Pixel/Text mode. Column: Animate[]/ Dreamlike[]/PG20[]/PG25[]/RealVis[] as decoder.
Refer to caption: extracted/5662848/fig-supp/exp-aigi-1.png

Figure: Figure A10: Visualization of an NSI (PGC) on the left after CMC. Row: Full/Image/Pixel/Text mode. Column: Animate[]/ Dreamlike[]/PG20[]/PG25[]/RealVis[] as decoder.
Refer to caption: extracted/5662848/fig-supp/exp-nsi-0.png

Figure: Figure A11: Visualization of an NGI (UGC) on the left after CMC. Row: Full/Image/Pixel/Text mode. Column: Animate[]/ Dreamlike[]/PG20[]/PG25[]/RealVis[] as decoder.
Refer to caption: extracted/5662848/fig-supp/exp-nsi-1.png

Figure: Figure A12: Visualization of an SCI (Movie) on the left after CMC. Row: Full/Image/Pixel/Text mode. Column: Animate[]/ Dreamlike[]/PG20[]/PG25[]/RealVis[] as decoder.
Refer to caption: extracted/5662848/fig-supp/exp-sci-0.png

Figure: Figure A13: Visualization of an SCI (Webpage) on the left after CMC. Row: Full/Image/Pixel/Text mode. Column: Animate[]/ Dreamlike[]/PG20[]/PG25[]/RealVis[] as decoder.
Refer to caption: extracted/5662848/fig-supp/exp-sci-1.png

### A.8 Data Statement

The CMC-Bench dataset is released under the CC BY 4.0 license. This includes all ground truth, distorted images, subjective annotations, and the weight of the Consistency/Perception evaluation model. All LMM developers can test their performance through our public scripts, and all image compression researchers can obtain the public I2T+T2I LMM pipeline. We believe these resources can inspire the next generation of visual signal codec protocols.

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 5
presentation: 4
questions: 1. The method for obtaining the compression rate (CR) values in Section 3.2 is unclear. Different codecs and rate points are expected to yield significantly varied results.

2. The reported -FR and -NR values are the weighted average of multiple metrics, but it is unclear how the proposed weighting ensures a reasonable assessment.

3. In Fig. 6, the gap between the upper and lower bounds of most variation bars exceeds 1, with some even approaching 2. Given that the MOS scale ranges only from 1 to 5, is such a large gap reasonable? Can reliable conclusions be drawn from this? It is also unclear how the rankings in Fig. 6 were determined given the substantial variation in MOS scores.

4. It is unclear how this paper deal with the randomness in CMC-based codecs.

5. According to the visualization results shown in Figs. 8-13, although the decoded images appear perceptually good, they differ significantly from the input images. Moreover, the RD curves suggest that CMC-based codecs are effective only at extremely low rates and cannot be extended to higher rates. Even if CMC-based codecs outperform traditional codecs in some objective evaluations, do they truly have practical applications in real-world compression scenarios? Even if CMC-based codecs outperform traditional codecs in some objective evaluations at extremely low rates, do they really have practical applications in real-world compression scenarios?

6. It appears that all the data used for training and evaluating the FR/NR quality indicators are generated using GPT-4 (OpenAI, 2023) and 12 different T2I models. However, this does not include images generated by other methods, such as traditional codecs or any learning-based codecs. Wouldn't the indicators trained on this data be biased?
soundness: 3
strengths: 1. The idea of conducting a benchmark to evaluate CMC-based codecs is valuable.
2. This paper provides a comprehensive set of experiments studying various combinations of I2T and T2I models.
summary: This paper introduces CMC-Bench, a benchmark designed to evaluate ultra-low bitrate image compression using a Cross Modality Compression (CMC) approach with Image-to-Text (I2T) and Text-to-Image (T2I) models. To assess the coding performance of CMC-based codecs, four collaboration modes are presented along with two indicators: consistency and perception. Experimental results show that some combinations of existing I2T and T2I models outperform traditional codecs in terms of some quality metrics at ultra-low bitrates.
weaknesses: 1. This paper does not propose a new model but instead (1) creates a mixed dataset by collecting images from several existing datasets and (2) combines existing I2T and T2I models to evaluate their coding performance. The technical novelty of this paper is therefore questionable.
2. The reported -FR and -NR values are the weighted average of multiple metrics, but it is unclear how the proposed weighting ensures a reasonable assessment.
3. In Fig. 6, the gap between the upper and lower bounds of most variation bars exceeds 1, with some even approaching 2. Given that the MOS scale ranges only from 1 to 5, is such a large gap reasonable? Can reliable conclusions be drawn from this?
4. Although the authors claim that some combinations of existing I2T and T2I models outperform traditional codecs at ultra-low bitrates, this is only demonstrated for specific objective functions. A MOS comparison and subjective evaluation between traditional codecs and CMC-based codecs are not provided.
5. PSNR result is not reported. 
6. The RD curve of GPT-4o + RealVis in Fig.7 looks very weird.
7. It appears that all the data used for training and evaluating the FR/NR quality indicators are generated using GPT-4 (OpenAI, 2023) and 12 different T2I models. However, this does not include images generated by other methods, such as traditional codecs or any learning-based codecs. Wouldn't the indicators trained on this data be biased?

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: As mentioned above, I am wondering is there any specific example that proves the proposed benchmark dataset can help resolve the inconsistency issue of cross-modality compression?

Except the issues I mentioned in the Weakness section, I also wonder whether there is any bias during the human annotation. Some experimental details should be added to ensure the subjective scores are reliable and unbiased.
soundness: 3
strengths: I appreciate the effort that the authors put in annotating subjective scores and building such a relatively large dataset for image compression.
summary: This paper builds a benchmark dataset for Cross Modality Compression (CMC), which aims at achieving much better compression at extremely low bitrates. This benchmark covers tens thousands of image samples and evaluates on 6 Image2Text models and 12 Text2Image models for CMC task. Human annotations are given to evaluate the subjective preference of compression results. There are four modes tested for cross modality compression in this paper, including text-conditioned compression (Text-mode) pixel-conditioned compression (Pixel-mode) traditional codec-based generative compression (Image-mode), and a combination of them all (Full-mode). After analyzing the collected dataset, this paper concludes that without dedicated training for compression, combinations of several advanced I2T and T2I models have already surpassed traditional codecs in multiple aspects.
weaknesses: 1. Throughout the paper, I did not find any non-trivial insight from the proposed benchmark for the whole community. The main conclusion of the proposed benchmark assessment is that: in the task of ultra-low rate compression, codecs based on generative models (e.g., T2I models) could perform better than traditional codecs in many aspects. However, actually it is almost a common sense in the compression community, I would suggest the authors to think deeper so that it can get some valuable conclusions, such as “which type of T2I architecture would perform better for CMC task, auto-regressive or denoising diffusion?”

2. Some sentences in this paper are overclaimed. For example, in abstract, it is written as 
“CMC has certain defects in consistency with the original image consistency with the original image and perceptual quality. To address this problem, we introduced CMC-bench…”
How could the benchmark address the issue of inconsistency in CMC. There is no technical algorithm proposed in this paper to address this issue, neither any direct insight in the proposed benchmark dataset that guides the community to address the inconsistency issue.

3. I have some doubts on the reliability of the proposed benchmark dataset. For example, in figure7, the performance of GPT-4o +RealVis fluctuates in FID-bpp. Sometimes FID increases and sometimes FID decreases as bitrates increases.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1.	For each column in Table 1, it is recommended to provide the full name and corresponding unit (e.g., "dis") in the main text or caption.

2.	According to the description in the text, the radar chart on the right side of Figure 5 represents "{GPT-4o + 12 different T2I models}," but it appears there are only six methods shown on the chart. The description is somewhat confusing.

3.	In Section 4.2, the first sentence of the first paragraph should be changed to “Figure 5 shows the performance of 6 I2T models as encoders and 12 T2I models as decoders in image compression.”

4.	The three colors in the histogram in Figure A3 are not all explained within the figure.

5.	I am curious about how the authors balanced consistency and perceptual quality when establishing the evaluation metrics, or if any experiments or considerations were conducted regarding this. Also, did the authors perform any ablation studies to determine the final weights of each evaluation metric?

6.	It seems the authors could further adjust the organization of the main text and appendix to prioritize more important information in the main body.

7.	The authors have indeed undertaken extensive integration and experiments related to previous data and evaluation methods in establishing the CMC-Bench standardized evaluation platform for I2T-T2I. However, this work seems more like an integration of previous research efforts, including the sources of the new dataset and the evaluation metrics, which resemble normalized methods from prior I2T-T2I tasks. Are the authors concerned that this paper’s originality might be questioned?
soundness: 3
strengths: This paper introduces a new benchmark platform for Cross Modality Compression (CMC) called CMC-Bench, offering a fresh approach and guiding direction for future ultra-low bitrate image compression. The paper includes 58,000 images and 160,000 human annotations, offering a large-scale and comprehensive evaluation. Furthermore, authors presents rigorous comparative experiments with existing traditional codecs, demonstrating the advantages of CMC, which is highly persuasive. Although it does not propose a complete solution, the dataset and evaluation metrics provided in this work are likely to inspire further research.
summary: The paper introduces CMC-Bench, a benchmark for Cross Modality Compression (CMC) in image compression. Utilizing advancements in Large Multimodal Models (LMMs), the authors explore an Image-to-Text-to-Image (I2T-T2I) approach, where visual signals are first transformed into text and then regenerated into images, achieving compression rates as high as 10,000 times compared to traditional codecs. However, the CMC paradigm faces challenges in consistency and perception fidelity at ultra-low bitrates. CMC-Bench evaluates 6 I2T and 12 T2I models across 58,000 images, covering consistency and perception dimensions with 160,000 subjective human annotations, providing a comprehensive dataset for future codec development. Experimental results indicate that CMC has the potential to outperform traditional codecs but requires further refinement in handling Screen Content Images (SCI) and balancing consistency and perception.
weaknesses: Some details require further discussion, such as the balance between consistency and perceptual quality. Additionally, certain expressions and the structure of the paper would benefit from further refinement and polishing.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: Improvement:

(1) I hope the author can add the std of each T2I model. Currently, I only see the mean result on the leaderboard. As you know, the performance of T2I is unstable. Adding std can make the benchmark more complete.

(2) I suggest the author open source all data. Including raw data, test code, and subjective quality labels. As a benchmark, LLM developers need an end-to-end testing pipeline.
soundness: 3
strengths: (1) Sufficient and comprehensive experiments, namely benchmarks for most current T2I and I2T models. The author has extensively considered mainstream LLMs and explored their optimal combination.

(2) Comparison with traditional codecs such as VVC. Experiments show that CMC has advantages over 7 of the 8 fidelity/quality indicators. This proves the significance of CMC.

(3) Exquisite visualization. Figures 1/2 are both relatively intuitive.
summary: This pioneering work explores the application of LLMs in the context of codec, namely cross-modality compression (CMC). The overall paradigm is innovative, as a combination of I2T+T2I pipeline. Experimental results indicate a satisfying fidelity and extremely low bitrate, which has significant advantages over traditional compression paradigms.
weaknesses: (1) Ultra-low bitrate needs a better definition. Why 1,000 times? Thousand-fold compression is indeed the highlight of this article, but has there been work in the past that can achieve this bpp? This needs to be further explained, as this is the core novelty of this article.

(2) The author experimented with 12 T2Is but only 6 I2Ts. One or two more I2Ts should be added to the benchmark list.

Meta-review: This paper attempts to evaluate several potential approaches to using large multimodal models (LMMs) for image compression. It is a compelling idea to represent images simply using a textual description – it could potentially yield ultra-low bit rate image compression methods.

The authors collect an impressive amount of data using various multimodal models, such as image-to-text (I2T) and text-to-image (T2I) models, and evaluate the size of the textual representation (and/or a severely downsampled version of the image) against the reconstructed image using subjective (human) experiments as well as an adapted version of TOPIQ, an image quality assessment method.

After reading the paper and digesting the comments of the reviewers as well as the authors' rebuttal, I have decided to reject this paper, for the following reasons:

1. The authors focus on ranking various models used for encoding and decoding against each other. In my opinion, this is "putting the cart before the horse".

   Many in the image compression community have justifiable doubts around the robustness of the proposed methods, as the authors admit in the introduction. I.e.: it is clear that there are individual images that might be extremely well represented using a short text, but would this method reliably give compression gains for arbitrary images? The results shown in the paper do seem to point towards a "no" to this question (for example: the picture of the woman in Fig. 1 representing a different person, relatively large error bars in Fig. 6, erratic rate–quality plots in Figs. 7 and A4), but the authors neglect discussing this.

   I tend to agree with Reviewer CJXY: The paper's focus should be on identifying what robustness issues there are, what properties of I2T and T2I models give rise to them, and generate insights that last beyond any individual LMM. This would enable the community to learn something, and make progress, independent of the fast-paced development of LMMs. Once we are in a regime of more incremental research, it makes sense to focus on a more quantitative evaluation as the authors do here.

2. I further share methodical concerns with Reviewer kYAX: For example, I am not convinced that fine-tuning TOPIQ yields an unbiased evaluation method. When we consider conceptually novel compression methods such as here, subjective evaluation is more reliable, since objective IQA methods are often only validated with respect to specific classes of compression/generation methods and may not generalize well enough. It would be more appropriate to focus on subjective evaluation, or spend more time validating and justifying specific metrics for the use in this context.

3. The language of the paper is muddled and imprecise, which makes details difficult to understand. For example, the abstract talks about using I2T and T2I models, but in fact, authors also use downsampled image representations for compression. As another example, I find the section on page 6 starting in line 298 almost impossible to parse – after several attempts, it is still unclear to me what the authors exactly did to come up with an evaluation method. There are more examples.

With this said, I don't believe all is lost – for example, for a future paper, the authors could use the data they gathered with human experiments, and mine it to generate more qualitative insights.

Decision: Reject