DiffPC: Diffusion-based High Perceptual Fidelity Image Compression with Semantic Refinement

Authors: Yichong Xia, Yimin Zhou, Jinpeng Wang, Baoyi An, Haoqian Wang, Yaowei Wang, Bin Chen
Primary area: generative models
Keywords: lossy image compression, diffusion model

Abstract:
Reconstructing high-quality images under low bitrates conditions presents a challenge, and previous methods have made this task feasible by leveraging the priors of diffusion models.  However, the effective exploration of pre-trained latent diffusion models and semantic information integration in image compression tasks still needs further study. To address this issue, we introduce Diffusion-based High Perceptual Fidelity Image Compression with Semantic Refinement (DiffPC), a two-stage image compression framework based on stable diffusion. DiffPC efficiently encodes low-level image information, enabling the highly realistic reconstruction of the original image by leveraging high-level semantic features and the prior knowledge inherent in diffusion models. Specifically, DiffPC utilizes a multi-feature compressor to represent crucial low-level information with minimal bitrates and employs pre-embedding to acquire more robust hybrid semantics, thereby providing additional context for the decoding end. Furthermore, we have devised a control module tailored for image compression tasks, ensuring structural and textural consistency in reconstruction even at low bitrates and preventing decoding collapses induced by condition leakage. Extensive experiments demonstrate that our method achieves state-of-the-art perceptual fidelity and surpasses previous perceptual image compression methods by a significant margin in statistical fidelity.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 5

Reviews and discussion:
<thread>
creation date:
1730129930187

thread_type:
official_review

<thread_object>
creator: reviewer_surn

<content>
summary:
DiffPC is a novel two-stage image compression framework leveraging stable diffusion models to enhance perceptual fidelity at low bitrates. The model first compresses low-level image details and then integrates semantic features to reconstruct high-quality images. Using modules like a multi-feature compressor, IC-ControlNet, and a semantic pre-embedding module, DiffPC manages to maintain structural and textural consistency.

strengths:
1. State-of-the-Art performance. Demonstrates superior perceptual and statistical fidelity at low bitrates compared to other image compression methods.
2. Efficient semantic integration. The pre-embedding module effectively combines textual and visual semantics, validating that optimizing semantic information has the potential to improve image reconstruction quality. This is a fascinating discovery.

weaknesses:
1. It is recommended that the authors improve the clarity and accuracy of the equations to facilitate reader comprehension. In the "related work" section, some formulas are not precise. For instance, in Eq. 2 and Eq. 3, the variances for the forward and reverse processes are both represented by ${\beta}t$, which is incorrect. Notation such as $DN{\theta}$ is typically represented as $\epsilon_{\theta}$ in existing literature. If the authors are referencing prior work, it is advisable to adhere to standard conventions to improve readability. Additionally, $\textbf{c}$ generally denotes conditional controls, i.e., “text prompts”, which corresponds to $\textbf{s}$ for "textual descriptions" in Eq. 5.

2. For lines 78-87, using `\citep` allows multiple references to be embedded smoothly within a sentence without disrupting the structure. This improves readability, especially when citing multiple sources.

3. Similar works that fine-tune pre-trained diffusion models for low-quality image enhancement include:

   - [1] Lin, X., He, J., Chen, Z., Lyu, Z., Dai, B., Yu, F., ... & Dong, C. (2023) in "DiffBIR: Towards Blind Image Restoration with Generative Diffusion Prior".
   - [2] Wang, Y., Yu, Y., Yang, W., Guo, L., Chau, L. P., Kot, A. C., & Wen, B. (2023) in "ExposureDiffusion: Learning to Expose for Low-Light Image Enhancement" (ICCV).
   - [3] Ma, J., Zhu, Y., You, C., & Wang, B. (2023) in "Pre-trained Diffusion Models for Plug-and-Play Medical Image Enhancement" (MICCAI).

Could the authors further clarify how this manuscript differs from these related works?

questions:
1. In Line 45, are *realism* and *perceptual fidelity* truly equivalent concepts, or could their distinctions be further explored?

2. The experimental section demonstrates that PSNR and SSIM metrics for generative models are not optimal. Could some insightful explanations be provided for why this is the case?

3. Lines 48-49: Are human perception and image semantic consistency necessarily aligned? Not all visual tasks require precise details; for instance, in certain scene understanding tasks, a lower bitrate may be sufficient to convey overall semantics without losing information essential to human perception.

4. The mentions of decoding collapse and condition leakage in the abstract are insightful explanations. Is there corresponding data analysis in the experimental section to support these claims?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730130935144

thread_type:
official_review

<thread_object>
creator: reviewer_1rum

<content>
summary:
(1).This paper proposes a diffusion-based high-perceptual fidelity image compression with semantic refinement. 
(2).The proposed method uses the low-level image information to ensure fidelity reconstruction,  and uses high-level texture semantics and robust prior in pre-trained latent stable diffusion to ensure realism reconstruction. 
(3).Extensive experiments demonstrate its effectiveness in reconstructing images with high perceptual quality texture at extremely low bitrates

strengths:
Originality: This paper explores the pre-trained latent diffusion model using semantic control signals for extreme image compression. This method clearly explains that two types of semantics, namely low-level image information (visual semantics) and high-level semantics (textual semantics), can support image compression.

Quality: The method is well designed and integrates the pre-trained latent diffusion model and semantic information for extreme image compression.

Clarity: The paper is generally well written and organized.

Significance: The significance of this work is profound as it addresses the extreme image compression using pre-trained latent diffusion models and semantic information.

weaknesses:
(1). The authors do not describe some representative perceptual image compression methods corresponding to their work in the related work.  
[1]. Multi-modality deep network for extreme learned image compression. AAAI 2023.
[2]. Extreme Image Compression using fine-tuned VQGANs. Data Compression Conference 2024. 
[3]. Towards Extreme Image Compression with Latent Feature Guidance and Diffusion Prior. IEEE TCSVT 2024.
[4]. Consistency Guided Diffusion Model with Neural Syntax for Perceptual Image Compression. ACMMM 2024.
[5]. HybridFlow: Infusing Continuity into Masked Codebook for Extreme Low-Bitrate Image Compression. ACMMM 2024.
[6]. Neural Image Compression with Text-guided Encoding for both Pixel-level and Perceptual Fidelity. ICML 2024.

(2). In Figure 2, it is confusing that you use the same color of block to represent cross-attention layers. The pre-embedding module is not shown in Figure 2. The ICCN (i.e., IC-ControlNet) is not explained in Figure 2 or your paper.

(3). In Eq.(7), the c ̂_x is a degraded image instead of features, which is the input of an image encoder, as shown in Figure 2. You need to add the image encoder to your equation.

(4). In the part ``IC-ControlNet and Time-aware Decoupling” of section 3.2, the conditions are integrated with noise and processed through residual blocks before entering the main network. I cannot find any residual blocks as you say in Figure 9.

(5). In Section 3.3, the symbol 〖text〗_x should be shown in Figure 2. The semantic pre-embedding module should be marked in Figure 2 (the same as (2)). The blended semantic s_x should be added to Figure 2 for clarity.

(6). Please compare your method with VQGAN-based compression (DCC2024), TACO (ICML2024), and DiffEIC (IEEE TCSVT2024) and add them to Figures 4 and 5. VQGAN-based IC performs extreme image compression using codebook prior. TACO uses the text prior and DiffEIC uses the diffusion prior, which are very relevant to your work.

Reference papers:
[1]. Extreme Image Compression using fine-tuned VQGANs. Data Compression Conference 2024. 
[2]. Towards Extreme Image Compression with Latent Feature Guidance and Diffusion Prior. IEEE TCSVT 2024.
[3].Neural Image Compression with Text-guided Encoding for both Pixel-level and Perceptual Fidelity. ICML 2024.

(7). What is TAC in your ablation study?  I think it is the TAD module in your procedure.

(8). The text can assist image compression. I think you need to further analyze the text description for image compression. For example, you can use different caption methods to generate captions, which are used for your compression framework (test the text robustness of your method).

(9). The author should compare the proposed method with state-of-the-art methods in terms of model complexity.

questions:
(1). Why do you need intermediate features f1 and f2 for a multi-feature compressor? Maybe using a simple compressor without multi-scale features is simple but effective. Please add an ablation study to validate its effectiveness for image compression.

(2). Could you give a visual comparison of using importance-weighted MSE and standard MSE? It ensures proper bit allocation as you describe in your paper. You can visualize the bit allocation map to explain this.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730339353668

thread_type:
official_review

<thread_object>
creator: reviewer_gpep

<content>
summary:
The paper proposes a diffusion-based image compression framework, DiffPC, aimed to achieve high perceptual fidelity at low bit rates. This model leverages both low-level degraded image representation and high-level hybrid semantic representation to control pre-trained Stable Diffusion. Experimental results show that the proposed method surpasses existing methods across various benchmarks, delivering superior perceptual quality and statistical fidelity.

strengths:
1. The usage of high-level hybrid semantic representation in diffusion-based image compression methods is novel. I appreciate the idea of extracting semantic information from $\hat{c}_x$.

2. The paper effectively communicates the purpose and rationale behind each module in the DiffPC framework, providing clear justifications for their functionality.

3. The paper includes thorough experiments with quantitative and qualitative metrics across multiple datasets, validating the effectiveness of DiffPC.

weaknesses:
1. The paper lacks a thorough comparison and analysis of computational complexity. As diffusion models inherently introduce high decoding complexity, it is essential to evaluate and compare this aspect against previous SOTA methods.

2. The paper has some issues related to writing clarity and accuracy of details. Please refer to questions 1-9 for specifics.

3. There are issues with citation formatting in certain parts of the paper. For instance, on page 2, line 079, the sentence ``… generative adversarial frameworks Mentzer et al. (2020); Muckley et al. (2023)`` should be formatted as ``… generative adversarial frameworks (Mentzer et al., 2020; Muckley et al., 2023).`` 

4. Other existing image compression methods [1, 2] could be discussed and compared.

[1] Zhiyuan Li, Yanhui Zhou, Hao Wei, Chenyang Ge, and Jingwen Jiang. Towards extreme image compression with latent feature guidance and diffusion prior. IEEE Transactions on Circuits and Systems for Video Technology, 2024.

[2] Hagyeong Lee, Minkyu Kim, Jun-Hyuk Kim, Seungeon Kim, Dokwan Oh, and Jaeho Lee. Neural image compression with text-guided encoding for both pixel-level and perceptual fidelity. In International Conference on Machine Learning, 2024

questions:
1. On page 2, line 097, ``Careil et al. (2023) ...`` PerCo was published in ICLR2024.

2. On page 3, line 138, in the sentence ``He et al. (2022), based on the orthogonality of features in channel and spatial dimensions, devised an asymmetric autoregressive entropy encoder..`` there are two periods at the end.

3. On page 4, line 183, there is a grammatical error in the sentence: ``Here, c and s respectively refer to low-level image controls (e.g., image contours, image degradation), with s denoting high-level semantic controls (e.g., textual descriptions of images, category label).``

4. On page 10, line 520, in the sentence ``(2) w/o TAC: Removing the TAC ...``, should `TAC` actually be `TAD`?

5. On page 11, line 545, ``En d-to-end optimized image compression.`` There is an extra space that should be removed.

6. Is Eq. (3) accurate? $\mathcal{DN}_{\theta}(z_t, t)$ is the predicting noise rather than the mean.

7. In Eq. (13), what does $\eta$ denote in $TAD_{\eta}$?

8. In Figure 10(a), it appears that an arrow from the conv3×3 layer to the Spatial Transformer is missing. In Figure 10(b), there are two instances of $\varepsilon$. Should they be represented by different symbols to avoid confusion?

9. On page 4, line 178, the introduction to Stable Diffusion is incomplete, as it does not explain how $\hat{z}_0$ is obtained within Stable Diffusion.

10. The authors mention that ``the importance-weighted loss assigns more bits to texture details, enabling a more precise reconstruction of these features.`` Could the authors provide a visualization result of the bit allocation to support this conclusion?

11. What impact do multi-scale features have on performance? In Eq. (12), the loss $L_{imp}$ is calculated based on the difference between $z_0$ and $c$, where multi-scale features do not appear to be necessary. Additionally, the authors have not demonstrated the role or necessity of multi-scale features in the ablation study. Clarifying these points would strengthen the paper’s argument for including multi-scale features.

12. In Eq. (13), how is the initial value of $w$, specifically $\sigma_{z_0}^2$, determined in practice? Additionally, how do you ensure that the trainable hyperparameter $w$ correctly learns to achieve precise bit allocation? 

13. This module adopts a pre-embedding approach that combines textual semantics with visual semantics. Could the authors clarify the individual roles of textual semantics and visual semantics? Additionally, what impact would using only textual semantics or only visual semantics have on performance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730384142676

thread_type:
official_review

<thread_object>
creator: reviewer_hiqs

<content>
summary:
This paper presents DiffPC, a two-stage image compression framework leveraging stable diffusion models. In the first stage, a multi-feature compressor encodes essential low-level details using minimal bitrates, while a pre-embedding module captures robust hybrid semantics by combining textual and visual information. In the second stage, DiffPC employs IC-ControlNet to ensure structural and textural consistency during image reconstruction, mitigating issues such as condition leakage. Extensive experiments demonstrate that DiffPC achieves state-of-the-art performance in both perceptual and statistical fidelity across various benchmark datasets, outperforming existing neural compression methods.

strengths:
DiffPC introduces an framework by integrating diffusion models with semantic refinement for image compression  and the paper conducts thorough quantitative and qualitative evaluations across multiple datasets and compares DiffPC with several state-of-the-art methods .The results consistently show superior performance in both perceptual metrics and statistical fidelity, particularly at ultra-low bitrates.

weaknesses:
Multi-step inference combined with heavy ControlNet and pre-trained base models pose significant computational challenges for the decoding side. While pre-trained T2I diffusion models can enhance the quality of compressed images, their high computational cost makes them an unacceptable solution for image compression.

questions:
[1] Although latent diffusion can enhance semantic details, it can also compromise some structural details, such as small faces and small text, which are not demonstrated in the paper.

[2] I would like to understand the performance comparison between directly using VAE compression and DiffPC.

I will consider raising the score once all concerns are addressed.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732195734175

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Joint response  about model complexity

comment:
We concur with the reviewer's suggestion to augment the experiments concerning the intricacy of encoding and decoding. We believe that supplementing this experiment would be advantageous to our manuscript.
- Apart from Perco, all our other baselines were tested using the same Nvidia 3080ti GPU. Due to the additional VRAM requirements of Perco, we conducted the tests using the Nvidia A6000 GPU, which possesses higher GFLOPS. Hence, in the table, we have utilized the symbol ">" to signify that the expected encoding and decoding time of Perco on the 3080ti GPU is greater than the values we have reported.
- For all the baselines, we calculate the runtime for all the Kodak images at a resolution of 512x768 and present the average time in seconds. We include these experiments in the revised Sec A.5

**Table 1.** Encoding and decoding delays on the gpu. Test latency on Kodak.

 BD-rate is calculated on CLIC2020 dataset, with FID as the metric.

|      Model      | Encoding (s) | Decoding(s) | BD-rate (%) $\downarrow$ |
| :-------------: | :----------: | :---------: | :--------: |
|     Cheng20     |    3.081     |    6.678    |  >10^(3)   |
|      HiFiC      |    0.013     |  3*10^(-5)  |   289.26   |
|     MS-ILLM     |    0.069     |    0.068    |   79.83    |
|  VQ-GAN based   |    0.011     |    0.011    |     -      |
|      TACO       |    0.121     |    0.144    |   62.96    |
|       CDC       |    0.007     |    3.080    |   143.54   |
|     DiffEIC     |    0.430     |    6.619    |   10.45    |
|      Perco      |    >0.767    |   >15.261   |     -      |
| Ours (steps=50) |    0.089     |    7.325    |     0      |
| Ours (steps=20) |    0.089     |    3.378    |    6.25    |
| Ours (steps=5)  |    0.089     |    0.886    |   14.59    |

- The term "steps" denotes the sampling steps required for decoding, with our default sampling step set to 50 in the paper. It is evident that by encoding only the latent code on the encoding side, DiffPC exhibits satisfactory encoding speed: surpassing most diffusion baselines and showcasing similar encoding latency to Gan-based methods. On the decoding side, we can actually reduce the sampling steps to decrease decoding latency. Even when we reduce the sampling steps to 5, DiffPC maintains superior performance and demonstrates considerable decoding speed. With a sampling step of 50, DiffEIC's decoding speed surpasses that of Perco, a diffusion-based model, and is comparable to the decoding speed of Cheng20, a neural compressor based on sequential autoregressive entropy models.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732447963050

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Modifications to the revision

comment:
We have revised the initial draft by incorporating the baseline as requested by the reviewers and by including more detailed ablation experiments between modules. Furthermore, we have rectified some figures and statements to enhance the clarity of the paper, thus averting misunderstandings. Specifically, our revisions are as follows:

**Experiment Revisions**:

- In Section 4.2, we introduced a new baseline[1] [2] [3],  along with additional experimental analyses. The introduction of this new baseline does not compromise the superiority of our proposed approach.
- In Section 4.3, we augmented the ablation experiments concerning text semantics and the multi-scale compressor module.
- In Appendix A.5, we incorporated experiments and analyses regarding encoding and decoding complexities.
- In Appendix A.6, we included visualizations and qualitative analyses for special images (including faces and text).
-  In Appendix A.7.1, we included ablation experiments on sampling steps, showcasing the decoding performance differences of DiffPC when using varying sampling steps.
- In Appendix A.7.2, we added an analysis on text robustness.
- In Appendix A.7.3, we included visual results on compressor bit rate allocation to further elucidate the contribution of multi-scale structures in optimizing bit rate allocation.

**Figure and Description Revisions**:


- We have revised typo errors and formatting mistakes in accordance with the reviewers' feedback.
- Figure 2 has been altered to include more detailed module annotations and symbol explanations.
- Figure 9 has been revised to incorporate the previously omitted residual module and complete the schematic diagram for this module.
- Figure 10 has been modified to rectify symbol descriptions and incorrect details.
- Errors in the spelling of the module TAD in Figure 8 have been corrected.
- The description of equation (3) has been adjusted to align more closely with a standard definition.
- Equation (7) has been modified to better reflect the description of the formula in accordance with our algorithmic process.

---

[1] Zhiyuan Li, Yanhui Zhou, Hao Wei, Chenyang Ge, and Jingwen Jiang. Towards extreme image compression with latent feature guidance and diffusion prior. IEEE Transactions on Circuits and Systems for Video Technology, 2024.

[2] Hagyeong Lee, Minkyu Kim, Jun-Hyuk Kim, Seungeon Kim, Dokwan Oh, and Jaeho Lee. Neural image compression with text-guided encoding for both pixel-level and perceptual fidelity. In International Conference on Machine
Learning, 2024

[3] Mao, Qi, et al. "Extreme image compression using fine-tuned vqgans." 2024 Data Compression Conference (DCC). IEEE, 2024.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper receives a mixed rating of (5, 6, 6, 6). The reviewers generally acknowledge the strong performance of the proposed DiffPC in the task of image compression, while the novelty of the proposed method receives mixed comments. After carefully reviewing the paper, review, and rebuttal, the AC agrees that the proposed method achieves decent performance, and the proposed modules have values to the research community. Therefore, the AC recommends an acceptance, with the suggestion that grammatical errors and minor writing flaws should be resolved in the camera ready version.