CMC-Bench: Towards a New Paradigm of Visual Signal Compression

Authors: Chunyi Li, Xiele Wu, Haoning Wu, Donghui Feng, Zicheng Zhang, Guo Lu, Xiaohong Liu, Xiongkuo Min, Guangtao Zhai, Weisi Lin
Primary area: datasets and benchmarks
Keywords: Image Compression, Large Multimodal Model, Image Quality Assessment

Abstract:
Ultra-low bitrate image compression is a challenging and demanding topic. With the development of Large Multimodal Models (LMMs), a Cross Modality Compression (CMC) paradigm of Image-Text-Image has emerged. Compared with traditional codecs, this semantic-level compression can **reduce image data size to 0.1% or even lower**, which has strong potential applications. However, CMC has certain defects in consistency with the original image and perceptual quality. To address this problem, we introduce CMC-Bench, a benchmark of the **cooperative performance of Image-to-Text (I2T) and Text-to-Image (T2I) models for image compression**. This benchmark covers 18,000 and 40,000 images respectively to verify 6 mainstream I2T and 12 T2I models, including 160,000 subjective preference scores annotated by human experts. At ultra-low bitrates, this paper proves that the combination of some I2T and T2I models has surpassed the most advanced visual signal codecs; meanwhile, it highlights where LMMs can be further optimized toward the compression task. We encourage LMM developers to participate in this test to promote the evolution of visual signal codec protocols.

Decision: reject
Reviewer scores: 5, 5, 6, 8

Reviews and discussion:
<thread>
creation date:
1730430644632

thread_type:
official_review

<thread_object>
creator: reviewer_hkwn

<content>
summary:
This pioneering work explores the application of LLMs in the context of codec, namely cross-modality compression (CMC). The overall paradigm is innovative, as a combination of I2T+T2I pipeline. Experimental results indicate a satisfying fidelity and extremely low bitrate, which has significant advantages over traditional compression paradigms.

strengths:
(1) Sufficient and comprehensive experiments, namely benchmarks for most current T2I and I2T models. The author has extensively considered mainstream LLMs and explored their optimal combination.

(2) Comparison with traditional codecs such as VVC. Experiments show that CMC has advantages over 7 of the 8 fidelity/quality indicators. This proves the significance of CMC.

(3) Exquisite visualization. Figures 1/2 are both relatively intuitive.

weaknesses:
(1) Ultra-low bitrate needs a better definition. Why 1,000 times? Thousand-fold compression is indeed the highlight of this article, but has there been work in the past that can achieve this bpp? This needs to be further explained, as this is the core novelty of this article.

(2) The author experimented with 12 T2Is but only 6 I2Ts. One or two more I2Ts should be added to the benchmark list.

questions:
Improvement:

(1) I hope the author can add the std of each T2I model. Currently, I only see the mean result on the leaderboard. As you know, the performance of T2I is unstable. Adding std can make the benchmark more complete.

(2) I suggest the author open source all data. Including raw data, test code, and subjective quality labels. As a benchmark, LLM developers need an end-to-end testing pipeline.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730476129682

thread_type:
official_review

<thread_object>
creator: reviewer_1puj

<content>
summary:
The paper introduces CMC-Bench, a benchmark for Cross Modality Compression (CMC) in image compression. Utilizing advancements in Large Multimodal Models (LMMs), the authors explore an Image-to-Text-to-Image (I2T-T2I) approach, where visual signals are first transformed into text and then regenerated into images, achieving compression rates as high as 10,000 times compared to traditional codecs. However, the CMC paradigm faces challenges in consistency and perception fidelity at ultra-low bitrates. CMC-Bench evaluates 6 I2T and 12 T2I models across 58,000 images, covering consistency and perception dimensions with 160,000 subjective human annotations, providing a comprehensive dataset for future codec development. Experimental results indicate that CMC has the potential to outperform traditional codecs but requires further refinement in handling Screen Content Images (SCI) and balancing consistency and perception.

strengths:
This paper introduces a new benchmark platform for Cross Modality Compression (CMC) called CMC-Bench, offering a fresh approach and guiding direction for future ultra-low bitrate image compression. The paper includes 58,000 images and 160,000 human annotations, offering a large-scale and comprehensive evaluation. Furthermore, authors presents rigorous comparative experiments with existing traditional codecs, demonstrating the advantages of CMC, which is highly persuasive. Although it does not propose a complete solution, the dataset and evaluation metrics provided in this work are likely to inspire further research.

weaknesses:
Some details require further discussion, such as the balance between consistency and perceptual quality. Additionally, certain expressions and the structure of the paper would benefit from further refinement and polishing.

questions:
1.	For each column in Table 1, it is recommended to provide the full name and corresponding unit (e.g., "dis") in the main text or caption.

2.	According to the description in the text, the radar chart on the right side of Figure 5 represents "{GPT-4o + 12 different T2I models}," but it appears there are only six methods shown on the chart. The description is somewhat confusing.

3.	In Section 4.2, the first sentence of the first paragraph should be changed to “Figure 5 shows the performance of 6 I2T models as encoders and 12 T2I models as decoders in image compression.”

4.	The three colors in the histogram in Figure A3 are not all explained within the figure.

5.	I am curious about how the authors balanced consistency and perceptual quality when establishing the evaluation metrics, or if any experiments or considerations were conducted regarding this. Also, did the authors perform any ablation studies to determine the final weights of each evaluation metric?

6.	It seems the authors could further adjust the organization of the main text and appendix to prioritize more important information in the main body.

7.	The authors have indeed undertaken extensive integration and experiments related to previous data and evaluation methods in establishing the CMC-Bench standardized evaluation platform for I2T-T2I. However, this work seems more like an integration of previous research efforts, including the sources of the new dataset and the evaluation metrics, which resemble normalized methods from prior I2T-T2I tasks. Are the authors concerned that this paper’s originality might be questioned?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730638930289

thread_type:
official_review

<thread_object>
creator: reviewer_cjxy

<content>
summary:
This paper builds a benchmark dataset for Cross Modality Compression (CMC), which aims at achieving much better compression at extremely low bitrates. This benchmark covers tens thousands of image samples and evaluates on 6 Image2Text models and 12 Text2Image models for CMC task. Human annotations are given to evaluate the subjective preference of compression results. There are four modes tested for cross modality compression in this paper, including text-conditioned compression (Text-mode) pixel-conditioned compression (Pixel-mode) traditional codec-based generative compression (Image-mode), and a combination of them all (Full-mode). After analyzing the collected dataset, this paper concludes that without dedicated training for compression, combinations of several advanced I2T and T2I models have already surpassed traditional codecs in multiple aspects.

strengths:
I appreciate the effort that the authors put in annotating subjective scores and building such a relatively large dataset for image compression.

weaknesses:
1. Throughout the paper, I did not find any non-trivial insight from the proposed benchmark for the whole community. The main conclusion of the proposed benchmark assessment is that: in the task of ultra-low rate compression, codecs based on generative models (e.g., T2I models) could perform better than traditional codecs in many aspects. However, actually it is almost a common sense in the compression community, I would suggest the authors to think deeper so that it can get some valuable conclusions, such as “which type of T2I architecture would perform better for CMC task, auto-regressive or denoising diffusion?”

2. Some sentences in this paper are overclaimed. For example, in abstract, it is written as 
“CMC has certain defects in consistency with the original image consistency with the original image and perceptual quality. To address this problem, we introduced CMC-bench…”
How could the benchmark address the issue of inconsistency in CMC. There is no technical algorithm proposed in this paper to address this issue, neither any direct insight in the proposed benchmark dataset that guides the community to address the inconsistency issue.

3. I have some doubts on the reliability of the proposed benchmark dataset. For example, in figure7, the performance of GPT-4o +RealVis fluctuates in FID-bpp. Sometimes FID increases and sometimes FID decreases as bitrates increases.

questions:
As mentioned above, I am wondering is there any specific example that proves the proposed benchmark dataset can help resolve the inconsistency issue of cross-modality compression?

Except the issues I mentioned in the Weakness section, I also wonder whether there is any bias during the human annotation. Some experimental details should be added to ensure the subjective scores are reliable and unbiased.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730915591379

thread_type:
official_review

<thread_object>
creator: reviewer_kyax

<content>
summary:
This paper introduces CMC-Bench, a benchmark designed to evaluate ultra-low bitrate image compression using a Cross Modality Compression (CMC) approach with Image-to-Text (I2T) and Text-to-Image (T2I) models. To assess the coding performance of CMC-based codecs, four collaboration modes are presented along with two indicators: consistency and perception. Experimental results show that some combinations of existing I2T and T2I models outperform traditional codecs in terms of some quality metrics at ultra-low bitrates.

strengths:
1. The idea of conducting a benchmark to evaluate CMC-based codecs is valuable.
2. This paper provides a comprehensive set of experiments studying various combinations of I2T and T2I models.

weaknesses:
1. This paper does not propose a new model but instead (1) creates a mixed dataset by collecting images from several existing datasets and (2) combines existing I2T and T2I models to evaluate their coding performance. The technical novelty of this paper is therefore questionable.
2. The reported -FR and -NR values are the weighted average of multiple metrics, but it is unclear how the proposed weighting ensures a reasonable assessment.
3. In Fig. 6, the gap between the upper and lower bounds of most variation bars exceeds 1, with some even approaching 2. Given that the MOS scale ranges only from 1 to 5, is such a large gap reasonable? Can reliable conclusions be drawn from this?
4. Although the authors claim that some combinations of existing I2T and T2I models outperform traditional codecs at ultra-low bitrates, this is only demonstrated for specific objective functions. A MOS comparison and subjective evaluation between traditional codecs and CMC-based codecs are not provided.
5. PSNR result is not reported. 
6. The RD curve of GPT-4o + RealVis in Fig.7 looks very weird.
7. It appears that all the data used for training and evaluating the FR/NR quality indicators are generated using GPT-4 (OpenAI, 2023) and 12 different T2I models. However, this does not include images generated by other methods, such as traditional codecs or any learning-based codecs. Wouldn't the indicators trained on this data be biased?

questions:
1. The method for obtaining the compression rate (CR) values in Section 3.2 is unclear. Different codecs and rate points are expected to yield significantly varied results.

2. The reported -FR and -NR values are the weighted average of multiple metrics, but it is unclear how the proposed weighting ensures a reasonable assessment.

3. In Fig. 6, the gap between the upper and lower bounds of most variation bars exceeds 1, with some even approaching 2. Given that the MOS scale ranges only from 1 to 5, is such a large gap reasonable? Can reliable conclusions be drawn from this? It is also unclear how the rankings in Fig. 6 were determined given the substantial variation in MOS scores.

4. It is unclear how this paper deal with the randomness in CMC-based codecs.

5. According to the visualization results shown in Figs. 8-13, although the decoded images appear perceptually good, they differ significantly from the input images. Moreover, the RD curves suggest that CMC-based codecs are effective only at extremely low rates and cannot be extended to higher rates. Even if CMC-based codecs outperform traditional codecs in some objective evaluations, do they truly have practical applications in real-world compression scenarios? Even if CMC-based codecs outperform traditional codecs in some objective evaluations at extremely low rates, do they really have practical applications in real-world compression scenarios?

6. It appears that all the data used for training and evaluating the FR/NR quality indicators are generated using GPT-4 (OpenAI, 2023) and 12 different T2I models. However, this does not include images generated by other methods, such as traditional codecs or any learning-based codecs. Wouldn't the indicators trained on this data be biased?
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper attempts to evaluate several potential approaches to using large multimodal models (LMMs) for image compression. It is a compelling idea to represent images simply using a textual description – it could potentially yield ultra-low bit rate image compression methods.

The authors collect an impressive amount of data using various multimodal models, such as image-to-text (I2T) and text-to-image (T2I) models, and evaluate the size of the textual representation (and/or a severely downsampled version of the image) against the reconstructed image using subjective (human) experiments as well as an adapted version of TOPIQ, an image quality assessment method.

After reading the paper and digesting the comments of the reviewers as well as the authors' rebuttal, I have decided to reject this paper, for the following reasons:

1. The authors focus on ranking various models used for encoding and decoding against each other. In my opinion, this is "putting the cart before the horse".

   Many in the image compression community have justifiable doubts around the robustness of the proposed methods, as the authors admit in the introduction. I.e.: it is clear that there are individual images that might be extremely well represented using a short text, but would this method reliably give compression gains for arbitrary images? The results shown in the paper do seem to point towards a "no" to this question (for example: the picture of the woman in Fig. 1 representing a different person, relatively large error bars in Fig. 6, erratic rate–quality plots in Figs. 7 and A4), but the authors neglect discussing this.

   I tend to agree with Reviewer CJXY: The paper's focus should be on identifying what robustness issues there are, what properties of I2T and T2I models give rise to them, and generate insights that last beyond any individual LMM. This would enable the community to learn something, and make progress, independent of the fast-paced development of LMMs. Once we are in a regime of more incremental research, it makes sense to focus on a more quantitative evaluation as the authors do here.

2. I further share methodical concerns with Reviewer kYAX: For example, I am not convinced that fine-tuning TOPIQ yields an unbiased evaluation method. When we consider conceptually novel compression methods such as here, subjective evaluation is more reliable, since objective IQA methods are often only validated with respect to specific classes of compression/generation methods and may not generalize well enough. It would be more appropriate to focus on subjective evaluation, or spend more time validating and justifying specific metrics for the use in this context.

3. The language of the paper is muddled and imprecise, which makes details difficult to understand. For example, the abstract talks about using I2T and T2I models, but in fact, authors also use downsampled image representations for compression. As another example, I find the section on page 6 starting in line 298 almost impossible to parse – after several attempts, it is still unclear to me what the authors exactly did to come up with an evaluation method. There are more examples.

With this said, I don't believe all is lost – for example, for a future paper, the authors could use the data they gathered with human experiments, and mine it to generate more qualitative insights.