DiffPC: Diffusion-based High Perceptual Fidelity Image Compression with Semantic Refinement
Authors: Yichong Xia
Yimin Zhou
Jinpeng Wang
Baoyi An
Haoqian Wang
Yaowei Wang
Bin Chen
Venue: ICLR 2025 Poster (2025)
Keywords: lossy image compression
diffusion model
Primary area: generative models

Abstract: Reconstructing high-quality images under low bitrates conditions presents a challenge, and previous methods have made this task feasible by leveraging the priors of diffusion models.  However, the effective exploration of pre-trained latent diffusion models and semantic information integration in image compression tasks still needs further study. To address this issue, we introduce Diffusion-based High Perceptual Fidelity Image Compression with Semantic Refinement (DiffPC), a two-stage image compression framework based on stable diffusion. DiffPC efficiently encodes low-level image information, enabling the highly realistic reconstruction of the original image by leveraging high-level semantic features and the prior knowledge inherent in diffusion models. Specifically, DiffPC utilizes a multi-feature compressor to represent crucial low-level information with minimal bitrates and employs pre-embedding to acquire more robust hybrid semantics, thereby providing additional context for the decoding end. Furthermore, we have devised a control module tailored for image compression tasks, ensuring structural and textural consistency in reconstruction even at low bitrates and preventing decoding collapses induced by condition leakage. Extensive experiments demonstrate that our method achieves state-of-the-art perceptual fidelity and surpasses previous perceptual image compression methods by a significant margin in statistical fidelity.

Published as a conference paper at ICLR 2025
DIFFPC: DIFFUSION-BASED HIGH PERCEPTUAL FI- DELITY IMAGE COMPRESSION WITH SEMANTIC RE- FINEMENT
Yichong Xial.31, Yimin Zhou Jinpenn Wang', Baoyi An', Haoqian Wang', Bin Chen "Tsinghua Shenzhen International Graduate School, "Harbin Institute of Technology. Shenzhen *Png Cheng Laboratory, 'Huawei Technologies Company Ltd.
ABSTRACT
Reconstructing high-quality images under low bitrates conditions presents a chal- lenge, and previous methods have made this task feasible by leveraging the priors of diffusion models. However, the effective exploration of pre-trained latent dif- fusion models and semantic information integration in image compression tasks still needs further study. To address this issue, we introduce Diffusion-based High Perceptual Fidelity Image Compression with Semantic Refinement (DiffPC), a two-stage image compression framework based on stable diffusion. DifIPC ef- ficiently encodes low-level image information, enabling the highly realistic re- construction of the original image by leveraging high-level semantic features and the prior knowledge inherent in diffusion models. Specifically, DiffPC utilizes multi-feature compressor represent crucial low-level information with mini- mal bitrates and employs pre-embedding to acquire more robust hybrid semantics, thereby providing additional context for the decoding end. Furthermore, we have devised control module tailored for image compression tasks, ensuring struc- tural and textural consistency in reconstruction even low bitrates and preventing decoding collapses induced by condition leakage. Extensive experiments demon- strate that our method achieves state-of-the f-tth-eart perceptual fidelity and surpasses previous perceptual image compression methods by significant margin in statis- tical fidelity.
INTRODUCTION
In this epoch of rapid multimedia advancement, the constraints of limited network bandwidth and costly storage hinder the transmission and preservation of large-scale high-definition raw images, rendering image compression algorithms with high compression rates increasingly crucial. Tra- ditional compression standards (Bellard) WWllace 1992) employ manually crafted transformations to seek compression representations, yet these methods may exhibit severe block artifacts or even chromatic aberrations at low bitrates (s 0.2 bpp). Neural image compression algorithms (Ballé jt al. 2016 Cheng et al.) 2020) He et al. (202),, based on end-to-end optimization schemes for rate- distortion have demonstrated superior performance compared to traditional compression standards. Nonetheless, these distortion-driven approaches may still yield displeasing blurriness in scenarios with constrained bandwidth. At low bitrates, the relevance of pixel-level metrics such as Mean Squared error (MSE) as an evaluation criterion drops, directly resulting in the loss of substantial texture details and realism (also known as perceptual fidelity in such compression schemes. (Blau & Michaeli 2019 Agustsson et al. (023) characterize this phenomenon as triple trade-off be- tween bit rate, distortion, and realism. This signifies that seeking compact representations low bit rates in the pixel domain inevitably leads to decline in human perception and a lack of hu- man observer's 's image semantic consistency. These semantic deficiencies diminish the practicality of compression algorithms low bitrates, thus making exploring the trade-offs between bit rate and perceptual fidelity an imperative subject of study.
*Corresponding author(s). Equal contribution. 'ee will utilize this pair of synonyms simultancously. both of which can be measured using perceptual metrics such as I.PIPS and FID.

---

Published as a conference paper at ICLR 2025
Ours 0.053bpp
Original
CDC
ELIC 0.059bpp
0.057bpp
Ours
HIFIC 0.05abpp
Original
Original
MS-ILLM 0.05 0.059ppp
Ours
0.049bpp
0.049bpp
Figure 1: Qualitative comparison between HiFiC Mentzer et al. 2020), MS-ILLM (Muckley et al, 2023 ELIC (He et al.] (022), CDC (Yang & Mandt (022), and our proposed approach. DiffPC
(Ours) capable of reconstructing complex textures with realism, even at extremely low bit rates. In contrast, both HiFiC and MS-ILLM exhibit significant artifacts and texture loss. Best viewed when zoomed in.
The encoder-decode architecture enhances the realism decoded images by introducing perceptual loss and generative adversarial frameworks (Mentzer et al. 2020 Muckley et al. 2023 These ef- forts partially compensate for the semantic information and texture lost in distortion-driver schemes. Yet, excessive reliance handerafted losses leads inevitable artifacts in low bitrate scenarios, diminishing the model's statistical fidelity. Compared to GAN, diffusion models have proven to generate high-realism images with enhanced statistical fidelity over the years (Dhariwal & Nichol, 2021 Rombach et al.) 2022). This suggests that diffusion models are better suited for low bitrate im- age compression scenarios. Yang & Mandt (2024) realizes this concept by employing the DDPM framework (Ho et al.) 2020) in the image domain and achieves superior result. However, these endeavors entail substantial time investment for retraining diffusion components, and the model's generalization is highly constrained by training data and computational resources.
Recently, the text-to-image Latent Diffusion Model (LDM) (Rombach et al. [022] has further ex- plored the potential of diffusion models in generative tasks. These data-driven foundational diffu- sion models have proven to offer strong priors for various visual tasks such as image segmentation (Tia al. 2024) and image super-resolution (Lin et al. 2023) and possess the capability to fuse multimodal semantic information. (Lei et al.] 2023) have attempted to control pre-trained LDMs by encoding sketches and textual semantics, sampling image recovery during the decoding process within the diffusion framework. Given the challenges jointly optimizing semantic embeddings and compressing representations of target transmissions, they have invested significant resources in iterative semantic embedding and alignment. Furthermore, while encoding solely semantic infor- mation significantly reduces the bitrate, the trade-off dramatically decreases fidelity. (Careil al. 2024) have employed fully trained conditional LDM and encoded conditional images using train- able codebooks. On one hand, this requires meticulous model optimization on datasets consisting of millions of images; on the other hand, VQ-based compressors that exclusively accept latent in- puts cannot recover information lost during the encoding process in the L.DM encoder, resulting in suboptimal bitrate allocation during compression.
To address the challenges above, we have proposed the Diffusion-based High Perceptual Fidelity Image Compression with Semantic Refinement (DinIPC). DifiPC leverages image-level control flow and lementic-level control flow to faithfully reconstruct highly realistic decoded images. At the encoding end, employ multi-featurr compressor that reconstruct high-qualin, image-level control flows even at extremely low bit rates. Furthermore, we have devised IC- Ce-CCnooolleet to enable pixel-level precision control for these low-level image controls. At the decoding end, to compensate
2

---

Published as a conference paper at ICLR 2025
for semantic deficiencies leading edge distortions and texture losses, we have devised pre- embedding module to efficiently generate robust hybrid semantics: textual semantic signals of the images are pre- me-dlodllated with low-level image signals and injected into the diffusion model through a linear mapping layer. With these characteristics, DiffPC can achieve superior perceptual and statistical fidelity even low bit rates, effectively recovering texture details that align more closely with human perception, as illustrated in Figure
In sum, our contributions are as follows:
We propose two-stage lossy image compression framework. In the first stage, DiffPC can attain superior bitrate allocation and reconstruct a more precise low-level image con- trol branch guided by importance-went weighted losses. In the second stage, DiffPC efficiently integrates a high-level semantic control branch to exert more precise control.
We devise IC-ControlNet (ICCN) to utilize a low-level image control branch and effec- tively integrate the robust prior of pre-trained LDM. ICCN enables precise control over the sampling process, ensuring the denoising model faithfully recovers the pre-compressed image.
We introduce a hybrid semantic refinement module generate a high-level semantic con- trol branch. This module adopts a pre-embedding approach that combines textual semantics with visual semantics, producing hybrid semantic representation that easily injectable into the denoising model.
RELATED WORK
Learned lossy Image Compression The advancement of deep neural networks has spurred the emergence of deep compression algorithms, showcasing performance surpassing traditional image compression standards like JPEG (Wallace 1992). (Ballé et al.] 2016)pioneered an end-to-end model based on autoencoders. (Ballé et al. 2018 introduced a hyperprior entropy model, while
(Minnen et al., 2018)) elevated performance significantly by introducing prior through autoregres- sive context modeling at the expense of decoding complexity. (Cheng et al., 2020) employed a discrete mixture model more precisely model the latent distribution. (He et al. 2022), based on the orthogonality of features in channel and spatial dimensions, devised an asymmetric autoregres- sive entropy encoder. Based on Shannon's 's rate-distortion theory (Shannon 1948). the optimization objectives of these approaches can be formalized as follows:
L = R(y)= AD 2))
In the above, ğ represents the distorted latent representation of the input image x. and R(ÿ) es- timates the entropy of ÿ to provide the required number of bits for encoding. D (ar, 4)) measures the distortion of the reconstructed image, which is often defined as pixel-wise loss (i.e., MSE). As (Blau& & Michaeli, 2019) has indicated that solely optimizing the MSF-bittraee function can severely compromise the statistical fidelity of compressed data, (Mentzer et al. 2020 proposed the use of generative adversarial network structures combined with adversarial and perceptual losses, effee- tively enhancing the realism of decoded images. Subsequently, (Muckley et al. 2023) introduced discriminator based local image representations, greatly improving statistical fidelity. Sub- sequent work has explored image compression at ultra-low bitrates within generative architectures Jiang al. 2023 Mao et al., 2024 Lu al., 2024). (Lee al. 2024) enhances perceptual fidelity by incorporating additional textual semantics into the GAN structure. Influenced by the develop- ment of diffusion models, (Hoogeboom et al. 2023) adopted the ddpm architecture, using a neural autoencoder as a baseline compressor. Building upon this, (Yang & Mandi 2024| Kuang et al., 2024] replaced the compressor with a VAE structure with hyperprior and jointly trained de- noising network and compressor, none of them consider the use of additional bits for transmitting semantic information. (Lei et al. 2023) utilized a text-to-image latent diffusion model, engaging in time-consuming image captioning through prompt inversion iterations and adding image sketch assistance in decoding, enabling image compression at extremely low bit rates. (Careil et al. 2024) employed fully trained LDM, achieving better performance gains a high training cost. (Li al. 2024b) leveraged a pre-trained diffusion model prior but overlooked high-level semantics and bitrate allocation at the encoding end, resulting in suboptimal performance.
3

---

Published as a conference paper at ICLR 2025
Diffusion Model The diffusion model utilizes priors from non-equilibrium statistical physics to transform the data distribution 2o to a known distribution ør (typically a Gaussian distribution) through Markov chain. The forward process of this Markov chain q (z+ zイ-1) is defined as gradually adding manually designed noise and then fitting the reverse sampling process p (z)-1 z() through a neural network Mo(.). Specificccl, the forward and reverse processes are defined as:
q (2 2 po (zt-1 z1) =N (= Mo t). Eo (z).
=N (ze v1 1,
(3)
(Ho et al. 2020) improved upon the original diffusion probabilistic model and demonstrated that optimizing the Evidence Lower Bound (ELBO) of the data distribution is equivalent to optimizing the following objective:
Dan = E ||c- DNo (Vāizo + vI dec, t)]l²
(4)
where (-) stands for denoising network. (Ho & Salimans) 2022) devised a framework for con- ditional diffusion models, outperforming GANs in image generation. By employing large-scale text encoder networks, conditional diffusion models can generate high-realism images from natural lan-
guage prompts (Rombach al., 2022). The Latent Diffusion Model (LDM) utilizes autoencoders to confine the denoising process within the low-dimensional embeddings of the data, significantly re- ducing complexity. Building upon the LDM framework, Stable Diffusion (Rombach et al., (2022) es- tablished large-scale text-to-image to-inage latent diffusion model capable of receiving multimodal control inputs and consistently producing high-fidelity images. Stable Diffusion (SD) employs a pre-trained encoder & to encode an image into latent variable 2n E(x). Subsequently, SD) performs noise addition and denoising process similar to 3 in the latent space. Similar to the optimization objective for Stable Diffusion is:
Cad E, [ [Je-DMG DNo (Vāizo + v1 āie, c, 8, 112
(5)
Here, e and respectively refer to low-level image controls (e.g., image contours, image degrada- tion), and high-level semantic controls (e.g.. textual deseriptions of images, category label). Our work is built upon the framework of Stable Diffusion, leveraging the robust priors of large-scale LDM on natural images and their semantic fusion capabilities.
METHODOLOGY
3.1
OVERALL FRAMEWORK
Our DifPPC framework is illustrated as Figure and the pseudocode for the algorithm can be found in Appendix A.2 During training, the model can be delincated into two stages: the initial stage involves training the compressor and incorporating image branch controls, while the subsequent stage entails training the semantic pre-embedding module and integrating mixed semantic branch controls. During inference, the process can be dichotomized into image compression and image reconstruction. Within the image compression process, the input image x is initially passed through the pre-trained encoder & to obtain zn E(x). Simultaneously. intermediate features fi and f2 extracted. These outputs are then fed into the multi-feature compressor Ma(-), yielding distorted image-level control
y M5 f1, 9 - Qw)), é= ())
Mg and Må represent the encoder and decoder of the compressor Mg. respectively, with Q denot- ing the quantization operation. Simultaneously, the textual description text, derived from image- captioning for æ undergoes lossless encoding and is transmitted alongside to the decoding end. Notably, the transmission of texrt,, occupies minimal number of bits (S 0.0001
During the image reconstruction phase, é initially decoded by the entropy decoder and then input into the Stable- Saable-Diffusion'ss 's decoder D to obtain a degraded representation éN of æ. Subsequently, é and text, are jointly fed into the Q-Former (QF(. (,, -)) to yield mixed semantic output s,:
é D(e), 82 QF(Einn (ex) terlz);
4

---

Published as a conference paper at ICLR 2025
X
Tire-sory Deccupitng
X
Denwsing
CHTA
Dirqense
Ieporarne Wegyvd (e
(a) First-stage: Learning image-level compressed representation
X
Time
X
Derwiing
D
penguin roery
text.
Image Enc
Seraaiic
image corto Danch
(b) Second-stage: Learning semantic-level hybrid condition
Figure 2: Illustration the proposed Diffusion-based High Perceptual Fidelity Image Compression with Semantic Refinement (DiffPC). (a) In the first stage, DiffPC employs variational compressor that receives multi-scale features to generate distorted image control branches. These low-level conditions used govern the pre-trained latent diffusion model through IC-ControNet (ICCN) (Section [.2). (b) In the second stage, DiffPC utilizes pre-embedding module to efficiently inject a blend of textual and visual semantics (Section 3.3)
where (-ng(() stands for pre-trained image encoder. The semantic control flow 8æ and the image control flow è will individually exert control over the conditional diffusion model through cross- attention layers and the IC-ControlNet. Subsequently, after sampling and decoding, the reconstruc- tion image & is obtained:
fg = Sampler (z,, DN (-, è, sz. TT, a = D(2o).
(8)
Here, 7 represents the number of sampling steps.
3.2 STAGE I: LEARNING COMPRESSOR AND LOW-LEVEL CONDITIONS
Multi-feature Compressor To simultaneously reduce the redundancy of latent Zg and prevent infor- mation loss caused by downsampling in the encoder &, have devised variational multi-featun compressor M.).). as depicted in the Figure 3(a). This compressor leverages residual extraction modules and cross-attention to fuse multi-scale features fi and fz. We have adopted the SCCTX from paper (He et al.) 2022) as the foundational entropy model. Diverging from neural compressors in the image domain, M.(-) reconstructs the control terms of the conditional diffusion model. To ensure that the results of conditional generation align with expectations, we aim to minimize the following Kullback- Leibler divergence:
DKL. (p(zolx), p(zo(()) = DKL. (2(2)))(),))))))))))))())(())))) ((z S1, [2)))-
The encoder of the pre-trained VAE used in Stable Diffusion explicitly models p(zo [x) How-
ever, obtaining p(zole (é) during training comes at a high cost, requiring numerous samples to be taken during the diffusion process. Fortunately, we have demonstrated that optimizing the above objective does not necessitate frequent sampling during retraining:
Theorem 3.1. Given the imput image x, the VAE-based encoder E, the VAE-based compressor M where zn {x), è Ma(Za). We have:
(pzop(z) x), p(zzz))) <
(10)
Proof. See Appendix A.I
"Here, za refers the population of the sample vector zo, and forth.
5

---

Published as a conference paper at ICLR 2025
Outpue
ResS
Denshiing
Oity
ICCN
Groand
Feet
Gullt
to untions
ogg
Condition
Cimerity
GOCTI
Image
Proposed
[es)
(a) Architecture multi-feature compressor
(b) Illustration of Condition Leakage and Time-Aware Decoupling
Figure 3: Illustration of each module in the first stage. (a) The multi-feature compressor we propose takes multiple imputs Zo- f1. and f2. s-Attn and c-Attn denote self-attention and cross-attention, respectively. (b) During the baseline denoising process, Conditional Image Leakage (CIL) occurred, causing the output image (CII. Output) to mimic distortions and artifacts from conditional images. Our proposed solution circumvents this phenomenon.
Furthermore, assuming both distributions are parameterized normal distributions with equal vari- ance, we have p(zo x) = (2; Pla, Σ) and py (c zo) A (ê; pe, E), where the covariance matrix I. In this scenario, the optimization objective transforms into a sariance-weighted MSE:
Σ = 2g
mig
arg min DKL (p(oo))))))) (C 2o)) arg min
[1 mell
(11)
20
In fact, Zz models the importance of the image latent. Significant high-frequency regions (such as textures and edges) are modeled by E with lower variances, while less crucial flat regions (like large areas of uniform color) are modeled with higher variances. Following this intuitive observation, we set the variance trainable hyperparameter initialized to o
-mmo (Hia me))
Cimp (Min pe)
(12)
w
We refer to this optimization objective as Importance-Weighted MSE. The importance-weighhed loss assigns more bits texture details, enabling a more precise reconstruction of these features. We showcase visualizations of the bit rate allocation in Appendix/A.7.3] to substantiate this point. In practice, the mean of Zo, are often reduced their samples 2o,
IC-ControlNet and Time-aware Decoupling In compression tasks, ControlNet provides robust non-semantic control for conditional diffusion models; however, falls short in precision control, resulting in significant decline in fidelity, as illustrated in Figure 8 To address this challenge, we propose the IC-ControlNet framework. During the input stage, we utilize convolutional modulation layers to effectively accommodate distorted conditions è. These conditions are then integrated with noise 2, and processed through residual blocks before entering the main network. IC-ControlNet enhances the control intensity of é, reducing uncertainties during the generation process.
Nevertheless, an excessively stringent control at low bitrates harbors drawbacks: aggressive quan- tization may induce pathological degradation in é, escalating the challenge of predicting zo for the denoising model. This tendency may incline the denoising model to forsake forecasting the true data distribution and instead produce samples akin to the conditioned è; this phenomenon is termed as condition leakage (Zhao et al.) 2024). Within the compression framework, this phenomenon mani- fests the denoising model forfeiting its denoising efficacy, consequently reconstructing distorted images, as illustrated in Figure (1)) To address this issue, we propose the Time-Aware Decou- pling (TAD) module. The details of TAD can be found in the Appendix A.44 TAD endeavors to disentangle the control factor è from IC-ControlNet, thereby transforming the noise prediction of the diffusion model into residual noise prediction:
é = DNa (Vozzo + /1 cgee 2,1) + Tê, l).
(13)
7 represents the optimizable parameter of TAD. The incorporation of TAD and residual structures ensures the stability and reliability of the denoising process, illustrated in Figure 3 (2).)). For

---

Published as a conference paper at ICLR 2025
further experimental results, please refer to Section 4.3 In essence, the initial phase is supervised by:
Lttage =
Cimp (zo, 6) + X2 + LCSD,
(14)
LCSD = C,,,c, |||- c DN (Vázzo + v1 a,c, e,t) (C, 11)
(15)
Here, Lrate denotes the learned quantized latent representation by the compressor along with the hyperprior compression rate. The parameters Å and X2 are utilized achieve a trade-off between rate and distortion.
3.3 STAGE II: LEARNING SEMANTIC EMBEDDING AND HIGH-LEVEL. CONDITIONS
Our experiments demonstrate that relying solely on low-level image control branches fails to achieve satisfactory realism. Additionally, the direct integration of high-level textual semantics through the stable diffusion interface proves ineffective. To overcome this challenge and avoid the costly iterative learning of semantic representations highlighted in (Lei et al. 2023)) we have developed semantic eeeeeddddding module. As depicted in Figure the decoding process begins by acquiring degraded representation éy.. Although é₂ lacks significant texture and detail compared to æ, retains essential low-frequency information, such as object shapes and colors-elemente that are often absent in textual semantics. The visual semantics from C₂ and the textual semantics textz derived from image captioning are then jointly integrated into a pre-trained Q-Former, resulting in blended semantic output 8g-
During this phase, we keep the parameters of the compressor Mg frozen. To enable the frozen conditional denoising network to adapt to the new semantic input, we have introduced linear pro- jection layer modulate the blended semantic and unfreeze all cross-attention layers integrating semantic fusion within the denoising diffusion network for fine-tuning. Experimental results demon- strate a notable enhancement in the perceptual quality of decoded images through the fine-tuning of attention layers and semantic pre-embedding.
The loss in the second phase is solely diffusion loss:
Catage 2
dx,, |||
DN (Vāzzo + VI āie, 8, (G.
(16)
To address the color shift issue caused by the diffusion model (Choi et al. 2022). we were inspired by Wang et al., 2024) to perform color correction the decoded image æ. Interestingly, observed that the colors in degraded representation é₂ retain accuracy even as lose a substantial amount of normalize the color of the decoded image å to align its mean and variance with é.- Further, we have found that color correction can be achieved through a learnable decoder that enhances certain perceptual metrics. For more details, please refer the Appendix A.4
high-frequency components. Consequently,
4
EXPERIMENTS
4.1
EXPERIMENTAL SETUP
Metrics. Our statistical metrics can be categorized into the following three groups. (1) Reference- based distortion-bass metrics: PSNR and MS-SSIM (Wang et al. (003). These metrics offer pixel- level distortions but have been shown to be ineffective at providing valid measure of perceptual realism at low bit rates. (2) Reference-based perceptual metrics: LPIPS (Zhang et al. 2018). DISTS (Ding et al.) 2020). These metrics are widely utilized for assessing the perceptual quality of images and have been demonstrated to be more correlated with human judgment. (3) No-reference percep- nual metrics: FID (Heusel et al. 2017). KID (Binkowski al. 2018). CLIP-IQA (Wang et al. 2023). Among these, FID) and KID measure the distribution difference between compressed and original images, capturing the statistical fidelity the compression scheme. CLIP-IQA is an image quality assessment metric that utilizes cross-modal model to score the realism and perceptual quality of images. Baseline. We compared DiffPC with several state-of-the-
neural compression schemes. This includes ELIC (He et al. 2022), model based on VAE and -ite-distortion optimization, HiFiC (Menterr et al.) (000)) based on GAN architecture and perceptual loss, and its enhanced version. MS- ILLMM (Muckley et al. 2023). Furthermore, we also compared approach with VQGAN based

---

Published as a conference paper at ICLR 2025
BLIC
HIFIC
MS-LIM
CDC
DIREIC
TACO
VOGAN
per sicel
BR per poce
B1G pet pine)
2.00
0.20
0.26
0335
0.08
0.22
2332
0372
5202
330
3.90
330
3:100
0:20
032
5353
$2.0
0.000
0.000
Figure 4: Comparisons of methods across various metrics for the CLIC 2020 test set.
BPG
BLIC
HIFIC
MS-LIM
EDC
DiNEiC
TACO
VOGAN
Proposed
POWR
0:425 3340 3015 1,05 ath
310 2319 5220
352x
3357
132
E:201
5509
1:34
0.56
0.. 5203
B1s per pke!
perplecl
B15
145-55
2.4
0335
C248
0:300
0.20
0.35
Bit pora)
Bits per plool
815 per
Figure 5: Comparisons of methods across various metrics for the DIV2K validation set.
(Mo al. 2024) methods and the solution TACO (Lee et al. 2024) that incorporates textual seman- tic priors. For the diffusion model-based compression baseline. we conducted our main experiments comparing against CDC (Yang & Mandt 2024) and DiffEIC Li et al. (2024b). To ensure a fair comparison, we retraced and retrained these baselines the LSDIR dataset using their open-source code and default settings, resulting in training outcomes closely aligned with the original results. worth noting that for VQGAN (Mao al. 2024). we utilized the official checkpoint trained on the ImageNet (Deng et al. 2009) dataset, which is 14 times larger than the LSDIR dataset.
In the case of HiFiC and CDC, we not only utilized their reported lowest bit rates but also ex- tended the comparison to even lower bit rates. Additionally, we contrasted these approaches with the traditional compression format BPG (Bellard), which is the image compression component of HEVC. It is noteworthy that due to some diffusion-based baselines (Hoogeboom et al., 2023 Carcil al. 2024) lacking sufficient validation results and experimental details, we only present DifiPC's
supenoo statistical fidelity in the Appendix A.8
Datasets. For validation, we referenced (Muckley et al. 2023) and employed three widely recog- nized image compression benchmark datasets: CLIC2000 (George Toderici 2020). DIV2K (Timo- fte et al. 2017). and Kodak (Company). CLIC2020 comprises 428 high-definition images, DIV2K includes 100 2K resolution high-definition images, and Kodak consists of 24 natural images with resolution of 768 x 512. Due to computational constraints, except for the Kodak dataset, we center- cropped the original images to 1024 x 1024 resolution. As Kodak lacks extensive validation, present the experimental results in the Appendix A.8)
Furthermore, following the approaches of (Hoogeboom et al. 2023 Careil et al. 2024). we validated the model's statistical fidelity using COCO30K (Lin et al. 2014) and present the results in the Appendix A.8
8

---

Published as a conference paper at ICLR 2025
Diggell
EUR
HPC
MSHILM
CDC
Pregeied
BM LUMPSR/SGIT
506008441900 JOR4
0.9PAN SERGAO EX08
DONAEOLPROO 1014
AUSSRAN C864
Figure 6: Qualitative illustrations of various methods on Kodak and CLIC2020 datasets. DiffPC reconstructs images without any artifacts and delicately restores intricate and complex textures.
4.2 MAIN RESULTS
Quantitative comparisons.
Figures 4 and shows the performance across different bit rates. ELIC, based on rate-distortion optimization, exhibits sig- nificant shortcomings across all percep- tual metrics, failing to outperform even the hand-craft image compression stan- dard BPG. HiFiC maintains excellent per- formance at higher bit rates; however, it shows poorer performance lower bit rates (S 0.1 bpp). MS-ILLM demon- strates satisfactory performance im- provement over HiFiC lower bit rates, approaching CDC's performance. Never- theless, our proposed DifiPC consistently outperforms all baseline solutions in perceptual quality across all bit rates. Additionally, DifiPC achieves significantly better performance the FID and KID) metries compared other baselines, indicating that approach maintains excellent statisti- cal fidelity even at low bitrates. Compared to TACO, which also utilizes textual semantics, DiffPC achieved significantly higher perceptual fidelity and statistical fidelity in low bitrate scenarios. In contrast to DiffEIC, which employs pre-trained diffusion model prior, DifIPC attained superior statistical fidelity through the fusion of mixed semantics. Additionally, due to improved bitrate allocation, our approach exhibits lower distortion and enhanced perceptual fidelity compared to Dif- fEIC. Further, we observe that the generalization performance of VQ-GAN terms of bitrates is concerning: it exhibits an anomalous situation where the rate-distortion curve shows a monotonous increase at boundary bitrates. contrast, DiffPC demonstrates stronger bitrate generalization capa- bilities and the ability to adapt to various fidelity requirements. Besides, compared pixel-level distortions, although they are proven to have limited reference value at lower bitrates (Careil et al. 2024 DiffPC demonstrates better fidelity in distortion metries compared to the diffusion-barr based CDC and competes with HiFiC in performance.
CLIP IOA (loft)
0.0
0.2
0.4
0.6
0.8
1.0
BPG
GUICEE
Method
MS-ILLM
CDC
0 3UP-00
2000 Neem
0.20
0.18
0.16
0.14
0.12
0.10
0.08
0.05
0.04
(right)
Figure 7: Comparing visual quality using CLIP-IQA.
Qualitative comparisons. We compared the visual quality of image reconstruction between DiffPC and baseline methods at ultra-low bit rates (S 0.08 bpp) using CLIP-IQA, as shown in Figure 7 Across three datasets, DiffPC reconstructed images at the lowest bit rate that significantly surpassed

---

Published as a conference paper at ICLR 2025
Orighna
Proposed
w/o
w/o Att-fretun
w/o Pre-enieeddling
w/o ICN
UPP5-0.289
PP
LFI5-0.30
LSPS-6.322
LPPS-0.351
(a) Qualitative ablation experiments on the Kodak dutaset
pe-arbesiing
senantoo
Prowerr
(b) Quantitative ablation experiments
Figure 8: Ablation study of the modules in DifiPC.
other neural codecs in realism. Figures and we presented visualizations of the reconstructed images. At ultra-low bit rates, while ELIC's reconstructed images exhibit minimal distortion, their visual quality is dismal. HiFiC and MS-ILLM's optimization relies on a single perceptual metric, leading to varying degrees of artifacts and noise low bit rates. CDC, constrained by the DDPM structure, exhibits certain color shifts and detail distortions during image decoding. Conversely, DifiPC's generated results at low bit rates demonstrate remarkably superior visual quality and alism. Whether capturing intricate textures (such as hair in Figure (6 and foliage in Figure I or finer details (like eyelashes in Figure 6). DiffPC excels in capturing and reliably reconstructing with minimal bitrates.
4.3 ABLATION STUDY
In this section, conducted ablation studies on various modules of DiffPC, and the quantitative and qualitative results are depicted in Figure X
(1) W/O importance-Witghted MSE: By substituting Limp with a standard MSE, as shown in Fig- ure 8 b), the model fails to achieve the perceptual quality of the original design even with increased bit rates. (2) w/o TAD: Removing the TAD module significantly impairs the model's performance at extremely low bit rates and alters the object morphology in the generated images, as illustrated in Figure (b). (3) w/o ICCN: Directly replacing IC-ControlNet with ControlNet, seen in Figure a(a,, results in almost ineffective control within the diffusion framework. (4) w/o Pre- embedding: By omitting Q-Former and solely relying on textual descriptions for semantic con- trol, there is a notable decrease in visual quality. Injecting simple text semantics in a conventional manner leads to similar outcome as injecting noise. (5) w/o Attn-finetune: By not fine-tuning the cross-attention layers in the second training stage, the difficulty of integrating semantic control branches into the denoising network increases, directly causing performance degradation. (6) w/o multi-feature: The removal of the multiscale feature structure from the compressor resulted in the loss of eighfrreuencc information. This leads to a significant performance drop at ultra-low bit rates. (7) w/o text semantics: We substituted the text prompts in the pre-embedding module with empty strings. Due to the lack textual cues aiding global semantics and stability, performance shows noticeable decline at ultra-low bit rates.
CONCLUSION
We introduce a novel neural compression framework, DifIPC, which leverages the priors from pre-trained latent diffusion model to reconstruct images with high realism and visual quality low bitrate scenarios. Unlike other generative model-based compression schemes, DiffPC achieves pre- cise bit-rate allocation through multi-feature compressor and incorporates a gre-emmedding module for efficient semantic information injection into the conditional diffusion model. Extensive experi- ments demonstrate that our proposed approach faithfully reconstructs images even at extremely low bit rates while preserving high perceptual quality textures.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work is supported in part by the National Natural Science Foundation of China un- der grant 62301189, 624B2088, the PCNL KEY project (PCL2203AAS6-1), and Shenzhen Sci- ence and Technology Program under Grant KIZZZ20240003100000000 (1077702000441110022000111010000 RCBS202210080031 GXWD202208 1172933001...
REFERENCES
Eirikur Agustsson, David Minnen, George Toderici, and Fabian Mentzer. Multi-realism image com- pression with conditional generator. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22324-22333, 2023.
Johannes Ballé, Valero Laparra, and Eero Simoncelli. End-to-end optimized image compression. arXiv preprint arXiv: 10/704, 2016.
Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. Variational image compression with scale hyperprior. arXiv preprint arXiv: arXiv:/802.00 1802.0/436, 2018.
Bellard. Bpg image format. https: ftttps://belard.orgggg ord/bgggg/pg/
Mikolaj Binkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. arXiv preprint
2018.
Yochai Blau and Tomer Michaeli. Rethinking lossy compression: The rate-distortion-prreee pprrretton- tradcoff. In International Conference on Machine Learning, pp. 675-685. PMLR, 2019.
Marlene Careil, Matthew J. Muckley, Jakob Verbeck, and Stéphane Lathuilière. Towards image compression with perfect realism at ultra-low bitrates. In The Twelfth International Confer- Learning Representations, 2024. URL https: tettpp://openeee http//ppeeevvieeeeeeeeeeeeeeeeee eww net. formmoormmmmmmm
kt.EEU9JJcq
Zhengxue Cheng, Heming Sun, Masaru Takeuchi, and Jiro Katto. Learned image compression with discretized gaussian mixture likelihoods and attention modules. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 7939-7948. 2020.
Jooyoung Choi, Jungbeom Lee, Chachun Shin, Sungwon Kim. Hyunwoo Kim. and Sungroh Yoon. Perception prioritized training of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11472-11488 1481, 2022.
Eastman Kodak Company. graphics/kodake
Kodak lossless color image suite.
URL http://rOk.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-254-2 255, 2009. doi: 10.1100/0VP. 5206848..
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780-8794 2021.
Keyan Ding, Kede Ma. Shiqi Wang, and Eero P Simoncelli. Image quality assessment: Unifying structure and texture similarity. IEEE transactions on pattern analysis and machine intelligence, 44(5):2567-2581, 2020.
Wenzhe Shi George Toderici. Workshop and challenge on learned image compression (clic2020). 2020. URL www.compression.co
Dailan He. Ziming Yang, Weikun Peng, Rui Ma, Hongwei Qin, and Yan Wang. Elic: Efficient learned image compression with unevenly grouped space-channel contextual adaptive coding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 5718-5727, 2022.
11

---

Published as a conference paper at ICLR 2025
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho and Tim Salimans. Classifier-free arXiv:2207. 12598, 2022.
diffusion guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-668 6856, 2020.
Emiel Hoogeboom, Eirikur Agustsson. Fabian Mentzer, Luca Versari, George Toderici, and Lucas Theis. High-fidelity image compression with score-based generative models. arXiv preprint arXiv:2305. 18231, 2023.
Xuhao Jiang, Weimin Tan, Tian Tan, Bo Yan, and Liquan Shen. Multi-modality deep network for extreme learned image compression. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 1033-1041, 2023.
Diederik P Kingma. Auto-encoding variational bayes. arXiv preprint
2013.
Haowei Kuang, Yiyang Ma, Wenhan Yang. Zongming Guo, and Jiaying Liu. Consistency guided diffusion model with neural syntax for perceptual image compression. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 1622-1631, 2024.
Hagyeong Lee, Minkyu Kim, Jun-Hyuk Kim, Seungeon Kim, Dokwan Oh, and Jaeho Lee. Neural image compression with text-guided encoding for both pixel-level and perceptual fidelity. In International Conference on Machine Learning, 2024.
Eric Lei, Yigit Berkay Uslu, Hamed Hassani, and Shirin Sacedi Bidokhti. Text+ sketch: Image compression at ultra low rates. arXiv preprint arXiv:2307.0194; 2023.
Dongxu Li, Junnan Li, and Steven Hoi. Blip-diffusion: Pre-trained subject representation for con- trollable text-to-image generation and editing. Advances in Neural Information Processing Sys- tems, 36, 2024a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. International conference machine learning, pp. 19730-19742. PMLR, 2023a.
Yawei Li, Kai Zhang, Jingyun Liang, Jiezhang Cao, Ce Liu, Rui Gong, Yulun Zhang, Hao Tang, Yun Liu, Denis Demandolx, al. Lsdir: large scale dataset for image restoration. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 1775-1787, 2023b.
Zhiyuan Li, Yanhui Zhou, Hao Wei, Chenyang Ge, and Jingwen Jiang. Towards extreme image compression with latent feature guidance and diffusion prior. IEEE Transactions on Circuits and Systems for Video Technology, 2024b. doi: 10. 109/1CSVVT.222444555555
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part 13, pp. 740-755. Springer, 2014.
Xinqi Lin, Jingwen He. Ziyan Chen, Zhaoyang Lyu, Bo Dai, Fanghua Yu, Wanli Ouyang, Yu Qiao, and Chao Dong. Diffbir: Towards blind image restoration with generative diffusion prior. arXiv preprint arXiv:2308. 15770, 2023.
I Loshchilov. Decoupled weight decay regularization. arXiv preprint urXiv: 2017.
Lei Lu, Yanyue Xie, Wei Jiang. Wei Wang, Xue Lin, and Yanzhi Wang. Hybridflow: Infusing continuity into masked codebook for extreme low-bitrate image compression. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 3010-3018. 2024.
Qi Mao, Tinghan Yang, Yinuo Zhang, Zijian Wang, Meng Wang, Shiqi Wang, Libiao Jin, and Siwei Ma. Extreme image compression using fine-tuned vggans. In 2024 Data Compression Conference (DCC), pp. 203-212. IEEE, 2024.
12

---

Published as a conference paper at ICLR 2025
Fabian Mentzer, George D Toderici, Michael Tschannen, and Eirikur Agustsson. High-fidelity gen- erative image compression. Advances in Neural Information Processing Systems, 33:11913- 11924, 2020.
David Minnen, Johannes Ballé, and George Toderici. Joint autoregressive and hierarchical priors for learned image compression. Advances in neural information processing systems, 31, 2018.
Matthew Muckley, Alaaeldin El-Nouby, Karen Ullrich, Hervé Jégou, and Jakob Verbeek. Im- proving statistical fidelity for neural image compression with implicit local likelihood models. In International Conference on Machine Learning, pp. 15426-25444 PMLR, 2023.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning, pp. 8162-8171. PMLR, 2021.
Byeongjun Park, Sangmin Woo, Hyojun Go, Jin- Young Kim, and Changick Kim. Denoising task routing for diffusion models. arXiv preprini arXi::23 2023.
Mr Murahari Reddy, Mr Sk Masthan Basha, Mr M Chinnaiahgari Hari, and Mr N Penchalaiah. Dall-e: Creating images from text. UGC Care Group I Journal, 8(14):71-75, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings the IEEE/CVF confer- ence on computer vision and pultern recognition, pp. 10684-10695, 2022.
Claude Elwood Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379 423, 1948.
Junjiao Tian, Lavisha Aggarwal, Andrea Colaco, Zsolt Kira, and Mar Gonvalez-Franco. Diffuse attend and segment: Unsupervised zero-shot segmentation using stable diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3554-3563, 2024.
Radu Timofte, Eirikur Agustsson, Luc Van Gool, Ming-Hsuan Yang, Lei Zhang, Bee Lim, et al. Ntire 2017 challenge on single image super-resolution: Methods and results. In The IEEE Con- ference Computer Vision and Pattern Recognition (CVPR) Workshops, July 2017.
Gregory K Wallace. The jpeg still picture compression standard. IEEE transactions consumer
electronics, 38())xxviixxxxix 1992.
Jianyi Wang, Kelvin CK Chan, and Chen Change Loy. Exploring clip for assessing the look and feel of images. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pp. 2023.
Jianyi Wang, Zongsheng Yue, Shangchen Zhou, Kelvin CK Chan, and Chen Change Loy. Exploiting diffusion prior for real-world image super-resolution. International Journal of Computer Vision, pp. 1-21, 2024.
Peng Wang, An Yang. Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-tor seweennc-t--sqqeeeee to-sequence learning framework. CoRR, abs/2202.03052, 2022.
Zhou Wang, Eero P Simoncelli, and Alan C Bovik. Multiscale structural similarity for image quality assessment. In The -eeittySeventh Asilomar Conference on Signals, Systems Computers, 2003, volume 2, pp. 1398-1402. Ieee, 2003.
Ruihan Yang and Stephan Mandt. Lossy image compression with conditional diffusion models. Advances in Neural Information Processing Systems, 36. 2024.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 586-595, 2018.
Min Zhao, Hongzhou Zhu, Chendong Xiang, Kaiwen Zheng, Chongxuan Li, and Jun Zhu. Identi- fying and solving conditional image leakage in image-to-vided diffusion model. arXiv preprint arXiv:2406. 15735, 2024.
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 PROOFS
Theorem A.1. Given the input image x, the VAE-based encoder &, the VAE-based compressor Mo. and the compressor's ' encoder 2 the compressor's 's decoder M Quantization operation Q, where the
2g {xx), y (2(2)), y Q(y), é (g(g). We have:
S (x
(17)
Proof. According the definition of KL. divergence, have:
/ p(zolx) log p(zalx)p(() dzo
DRI(p(zo (x..)p(200x).p2zll)))
(18)
py (e(ezo))(22
(z)) log p(c)
DKL
dzn-
(19)
p(zo)
Since Ma is VAE-based and è Ma(zo). According (Kingma 2013). is generated by (y)
Due to the distortion caused by quantization Q, Mg actually estimates the distribution p(zolý). Therefore, we have:
p(zolý) p(zo)
= Px + p(zo|x) log
dzo
(20)
p(zu vy
= (x) + p(zo)x) log
dzo
(21)
p(zz 18 )d
< px)
(22)
When Q is the identity function (meaning there is no quantization distortion), in this case. M)
(y) fits the input distribution p(zo), have:
p(c)
(KL(p(zo)x), px) + p(zolx) p(zo)
dzg = (x
(23)
Above all:
DKL (nn(ppoo)).. S (x
(24)
A.2
MORE DETAILS OF ALGORITHM PROCEDURE
In this section, supplement the explanation of the encoding and decoding processes of DiffPC through pseudocode. The detailed encoding and decoding processes are shown below.
A.3 FURTHER EXPERIMENTAL DETAILS
A.3.1 DETAILS OF THE MODEL TRAINING
Our model was trained on the LSDIR dataset (Li et al. 2023b), which comprises 84,991 high- definition natural images. During training, these images were randomly cropped resolution of 512 x 512. Our foundational conditional diffusion model leverages Stable Diffusion 2. 2b Throughout all training stages, we employed AdamW (Loshchilov 2017] as the optimizer, with learning rates set at 1 x 10 for the initial phase and x 10 for the subsequent phase. The batch size was consistently maintained
In the initial training phase, we employed an entropy estimator SCCTX (He at. al. 2022) with group number To achieve compression different bit rates, we set the parameter A2 Section 3.2 to 0.2 and then adjusted ê E (4, 16, 64, 128). At this stage, we will train with 80000 steps.
hhtn:://nggggggaaacooaaaiiiiiiin 2-b-ssse
14

---

Published as a conference paper at ICLR 2025
Algorithm Encoding Process
1: Given input data compressor encoder Mg()). stable diffusion's encoder E-), quantization
operation Q
2: zo. f1. f2 (xx)
3: y M f2)
4: y Qạy)
5: teztz Image captioning(a)
6: Encode Ù. text, binary file
7: Output encoded data
Algorithm 2 Decoding Process
1: Given compressor decoder (.),), stable diffusion's decoder D(-), pre-embedding module QF(-), Denosing network DNa, diffusion steps
2: Decode y. leatz from binary file
3: ê = ()
4: é₂ D(c)
5: SN
(F(Cz.tertz)
6: 2T ~ I(0.I)
7: for E [T...., do
8: 21-1 = Sampler (DN, (ze, é, 8n, ().)
9: end for
10: & D(z))
11: Brec = Color correction(;, 2.)
12: Output Frec
In the second training phase, the parameters of the compressor were frozen. We utilized BLIP-2 Li al. 2023a) for extracting textual semantic descriptions of images and employed the pre-trained Q-Former and image-encoder from BLIP-diffusion (Li et al. 2024a). Finally, for color correction in decoded output, we applied wavelet-color correction (Wang al. 2024). this stage, we will train with 60000 steps.
We did not apply warm-up in the first stage but utilized LambbaLLinarScheduleer with parameters warm_up wtwpu_pppttsseeeee 10000 and fsttart=ee-6 in the second stage. For sampling, we utilized IDDPM Nichol & Dhariwal 2021) as the sampler with a uniform setting of 50 sampling steps, as reported to be optimal in the original paper. Additionally, all experiments were conducted an Nvidia A6000 GPU. Code is released at http:://gttub.coo//aaees///////m
A.3.2 DETAILS OF THE MODEL TESTING
Implementation of the baseline approach All the baseline approaches utilized employed their respective official open-source codes 4516 To ensure result alignment, these baselines were retrained on LSDIR. The results we reproduced closely match the original papers, with the retrained HiFiC even outperforming the performance of the official checkpoint during testing.
Evaluation metrics for testing For LPIPS, mented using DIST'S pyttorhh FID and KID metrics were calculated using functions provided by torchmetrics. image, with feature size of 2048. Following the approach in (Mentzer et al. 2020). during FID and KID testing, images were partitioned. Specifically, from each // x W image, we initially extracted HH/J fl [W/f] nve-overlg f x crops. Subsequently. the extraction origin was shifted by f/2 in both dimensions to obtain another ([H/f] 1) (LW/f) 1) patches. A fixed value of f 256 was used for all evaluations.
utilized the lpips library, while DISTS imple-
*CDC:
compression
SHiFiC: htSS:///wwm////mmoooo//oooetette
MMS-ILLMM:
15

---

Published as a conference paper at ICLR 2025
IC-ControlNet
Norm
Restlock
Trainable copy 2
Trainable copy
Trainable capy
Treinable copyy
Trainable oxpy
SiLu
SiLu
Zaro cote
Z60 Sime
cone
Zan cate
Resblock
Norm
Encoderr Block
Encoderr Block
Encoder Block 3
Decoder Block 2
Decoder Block 1
SiLu
Encoder Block
Middle Brook
Decoder Blook
Z
Decoder
Zero Conv
Reablock
Figure Architecture of IC-ControlNet.
Conv3xs
Convloution
Milak Maac
Time-swar
Rea Block
SILU
D
Milck
Camvaxa
Spatial
Zero Convloution
RRDB
(a) Trainable
correction
Architecture decoupling
Figure 10: Architecture of different modules.
A.4 ARCIITECTURES OF MODULES
We intricately illustrated the structure of the control module IC-ControlNet and the architecture of the Time-aware decoupling designed in the first stage, depicted in Figure 9 and Figure 10(b). respectively. The concept behind ICCN's design was inspired by (Lin et al.) (023), while the time- aware task router in the Time-aware decoupling was adapted from (Park al. [002)). We treated each time step as an independent task, generating tasks to integrate temporal information.
As mentioned in Section 3.3 possesses comprehensive color information, prompting a trainable fusion decoder for integration, as illustrated in Figure T0 This module comprises feature fusion module and trainable SD decoder D. Since training this module occurs after the second-stai training, we term it the 3rd-stage training. During the training of this decoder, all other parameters of the backbone network are frozen, and the L.PPS between the decoded image and the original image is employed as the loss for optimization. Our experiments indicate that this boosts perceptual metrics further during testing; however, conversely, statistical fidelity may marginally de- crease, shown in the Figure 12 Furthermore, akin to HiliC and ILLM, we observed that optimiz- ing with LPIPS inevitably introduces subtle grid-like artifacts. These artifacts primarily concentrate in intricate texture regions, depieted in Figure II] While this approach shows enhancements in some metrics, we believe it compromises the realism of the images, so it is presented solely as an optional additional component.
devise
16

---

Published as a conference paper at ICLR 2025
Proposed
LPIPS=0.33 DISTS=0.10
Proposed with 3rd- stage
LPIPS=0.29 29
DISTS=0.09
Figure 11: Qualitative ablation study of trainable fusion decoder.
Proposed w/ 3rd stage
Proposed
0.325
0.900
0.275
0.250
F1D
0.175
0.150
Bits pixel
Bits pixe!
Figure 12: Quantitative ablation study of trainable fusion decoder.
A.5 COMPLEXITY COMPARISONS
We compared the encoding and decoding latency of our models with the baseline in a GPU environ- ment and included various performance comparisons the rightmost side of the table, shown in the Table T In the table, 'steps' represent the number of sampling steps required for decoding, with default of 50 steps in our work. Our BD-rate is calculated with DifIPC the baseline, where a higher value indicates a larger performance gap compared to DifiPC. It is important to note that, except for Perco, all tests were conducted usage during inference, Perco was tested on an A6000 GPU, which has superior GFLOPS. In the table, we use the symbol ">" to indicate that in the same GPU environment. Perco's encoding and decoding speeds exceed the values we provide.
the same Nvidia 3080ti GPU. Due high VRAM
can be observed that by encoding only the latent code the encoding end, DiffPC achieves satisfactory encoding speed: it outperforms most diffusion baselines in encoding speed and has a similar encoding latency to GAN-based methods Mentzer et al. (2020); Muckley et al. (2023). At the decoding end, we actually reduce the number of sampling steps to decrease decoding latency. Even when reducing the sampling steps to 5, DiffPC maintains superior performance with considerable decoding speed. Even with 50 sampling steps, DiffEIC's decoding speed surpasses Perco, which is also based on a diffusion structure, and is comparable to the decoding speed of C0eng20, neural compressor based on sequence autoregressive entropy model.
17

---

Published as a conference paper at ICLR 2025
Model
Encoding (8)
Decoding (s)
BD-rate(%)
ELIC He et al. 2022
0.009
0.008
4254.39
Cheng20 Cheng al.
(2020
3.081
6.678
HiFiC Mentzer et al. 2020
0.013
3 10
289.26
MS-ILLM M et al. 20033
0.069
0.068
79.83
CDC CDC
Yang & Mandt 2024
0.007
3,080
143.54
VQGAN based Mao et al. 2024
0.011
0.011
TACO et al.
2024
0.120
0.144
62.96
DiffEIC Li al. (2024b)
0.430
6.619
10.45
Perco Careil (2024
20.767
25.2261
DitPC (steps=50)
0.089
7.325
0.00
DiffPC (steps=20)
0.089
3.378
6.25
DifIPC (steps=5)
0.089
0.886
14.59
Table I: Encoding and decoding time (seconds) on Kodak dataset. BD-rate (%) is calculated on CLIC2020 dataset, with FID the metric.
A.6 SPECIAL SCENARIO DISCUSSION
In this section, we conducted a qualitative visual analysis of images containing 'text' and images containing faccs'. Initially, categorized 'text" and 'faces' into three classes based their pro- portions within the entire frame large, medium, and small and discussed them separately, as shown in Figures and 14
In instances of small text (Figure 13 it is evident that even the most cutting-edye neural compres- ELIC based traditional rate-distortion optimization cannot restore discernible text. Similarly, Gan-based methods also fall short. Given the premise semantic loss in decoding, our proposed DiffPC decoding yields sharper edges and enhanced perceptual quality in the images. In the medium text category, all generative compression schemes exhibit some degree of text distortion. However, notably, compared to GAN-based methods which show more pronounced artifacts and structural deficiencies, DifIPC maintains the semantic consistency of text to the maximum extent while pre- serving details and textures in other parts. In the case of large text, all approaches can reconstruct high-fidelit textual information.
Regarding images containing faces (Figure 14. the large and medium categories, DiffPC main- tains remarkable structural consistency without distortions. Conversely, Gan-based solutions exhibit structural distortions, while ELIC shows severe blurring, making facial recognition challenging. In the small face category, both Gan-based methods and ELIC display significant distortions and struc- tural chaos. contrast, DifIPC sacrifices a certain level of semantic consistency to enhance overall structural coherence and texture details.
conclusion, we believe that the proposed DifIPC achieves an optimal triple balance between real- ism, distortion, and bit rate at low encoding rates. Nonetheless, approach still has shortcomings in maintaining semantic consistency for extremely small faces and text, which we aim to improve in future endeavors.
A.7 FURTHER ABLATION EXPERIMENT
A.7.1 NUMBER OF TIMESTEPS
In Figure 15 show how DISTS, LPIPS, MS-SSIM and FID vary when change the number of denoising steps. We do the evaluation for four different bitrates, ranging from 0.0043 to 0.1811 bpp. It is evident that higher bit rates, reducing the number of samples has minimal impact performance degradation. This implies that in high bit rate scenarios, we can decrease sampling to accelerate decoding speed. At lower bit rates, perceptual metrics exhibit more noticeable de-
18

---

Published as a conference paper at ICLR 2025
8PP:0.04
8PP:0.05 053
8PP:0,005 005
Small Text size
BPP-0.036
PP.00
8 PP:00,048
Medium Text size
ICLR
ICLR
ICLR
ICLR
ICLR
Large Text
Workshop and Challenge on Learned Image Compression
Workshop and Challenge on Learned Image Compression
Workshop and Challenge on Learned Image Compression
Workshop and Challenge on Learned Image Compression
Workshop and Challenge on Learned Image Compression
(a) Original
(b) ELIC
(c) HIFIC
(d)MS-ILLM
(e) Ours
Figure 13: An image containing text. The image is from the CLIC2020 dataset, and the upper left corner of the image is the bit rate (BPP) of the image.
BPP-0.031
BBP-0.037
BPP-0.037
BPP-0.038
Large Face size
BPP
BPP PPP-00c 051
0.053
BPP-6.055 o5s
Medium
Face size
BPP:0.046
Small
Face size
(a) Original
(b) EUCC
HIFIC
IdIMS-ILLM
Figure 14: An image containing human face. The image is from the CLIC2020 dataset, and the upper left corner of the image is the bit rate (BPP) of the image.
cline with reduction in the number of samples. However, overall, the statistical fidelity is not significantly affected by the number of samples.
A.7.2 TEXT ROBUSTNESS
We employed four different approaches for textual descriptions: (1) BLIP2 (5 <word count < 10):
Forcing the word count output of BLIP2 be within 10 words. (2) OFA-tiny: Using OFA (Wang et al. 2022) as an image-captioning model to obtain textual prompts for the images. (3) Defauls text: All images uniformly utilized the default text description: "high quality, extreme detail." (4)
19

---

Published as a conference paper at ICLR 2025
1po-2.0423
apa-2.000
226-
502-1 1001
LETTE
per pied
Bo piid
Bla
Figure 15: Quantitative comparisons of different number of denoising steps. Evaluation DIV2K datasets.
Text type
LPIPS
DISTS
PSNR
BLIP2 (55 words counts 10)
1.686
1.579
0.891
OFA-tiny
1.742
-0.311
0.320
Default text
4.502
5,455
0.447
Random string
19.577
35.816
82.989
BLIIP2 w10swords count=30)
Table Text robustness evaluation. Tested on Kodak with BD-rate (%) based on DiftPC (Ours). The bold sections represent the textual prompt acquisition approach used by DifiPC.
Random string: Employing random strings of equal length (20 characters). The result is shown in (A.7.2 It is worth noting that the approach utilized by DifIPC falls under BLIP2 (10$word 20) which serves the baseline for our BD-rate calculations. It can be observed that as long as the textual description aligns with the semantic content of the image, it generally does not compromise the model's reconstruction performance. Even when default text prompts are used, the model can still derive benefits from the text.
A.7.3 VISUALIZATION OF BIT ALLOCATION MAP
As mentioned in the Section 3.2 we employed importance-wecighted loss to achieve improved bit
rate allocation. We visualized partial latent representation bit rate allocation diagrams on the Kodak dataset, as depicted in Figure 16
is quite evident that the importance-weightnn loss we employed effectively prevents the excessive allocation of bits in flat regions (such the sky in the image) and instead allocates more bits to areas with intricate textures, which aligns with our expectations.
A.8 FURTHER EXPERIMENTAL RESULTS
Further Quantitative Results: We present the results of all bascline methods along with DiffPC on the Kodak dataset, as shown Figure 17 Due the limited size of the Kodak dataset, which contains only 24 im- ages, reliable FID and KID scores could not be calculated. Therefore, we only dis- play the remaining metrics. can be ob-
HHE
Perco
FID
served that DiffPC maintains outstanding per- formance on the Kodak dataset as well. Fur- thermore, for more comprehensive valida- tion, we evaluated the statistical fidelity of various methods on the COCO 30K dataset following the settings in (Hoogeboom et al. 2023), shown in Figure [8] Although the COCO 30K dataset is not a commonly accepted benchmark for image compression tasks, its ca- pability assess the statistical fidelity of models has been validated by numerous works, such as
0.20
0.30
Bits per pixel
Figure 18: FID tested on COCO 30K.
20

---

Published as a conference paper at ICLR 2025
380
522
400
[a] Original
W/o Importa wee-weghted MSE
[c] Proposed
[6] Comparison
Figure 16: The bit rate allocation visualization, where darker colors represent fewer allocation of bits. To enhance clarity, we interpolated the lower-level features to match the original image dimensions.
BPG
ELIC
HIFIC
MS-ILLM
CDC
Proposed
0.30
0.25
LPIPS
0.4
DISTS
0.15
0.3
0.10
0.2
0.05
0.025
0.075
0.150
0.200
0.025
0.100
0.125
0.150
0.175
Bits per pixel
Bits per pixel
35
10
30
0.9
MS-SSIM
PSNR
20
0.7
15
0.6
0.025
0.050
0.075
0.100
0.125
0.150
0.175
0.200
0.025
0.050
0.075
0.100
0.125
0.150
0.175
0.200
Bits per pixel
Bits per pixel
Figure 17: Comparisons of methods across various metrics for the Kodak dataset.
(Reddy et al. 2021). Following the experimental setup in (Hoogeboom et al. 2023 Careil et al,, 2024), partitioned the images into 256 256 patches. Note that is challenging to extract mean- ingful textual descriptions from such small patches, so we used Null Character as a placeholder for textual semantics. Despite this, our model still demonstrated superior statistical fidelity, indicating that DifiPCis not highly dependent on textual semantic information in images.
We showcase additional visual results of DifiPC, HiFiC, and ILLM on three baseline datasets, as illustrated from Figure 19 to Figure 30 Our approach reconstructs details with superior realism using the least number bits and avoids artifacts that typically arise when reconstructing fine textures.
Further Qualitative Results:
21

---

Published as a conference paper at ICLR 2025
Figure 19: Orignal image (Kodak).
Figure 20: DifrPC(OUrs) 0.065 bpp, 0.252 LPIPS
22

---

Published as a conference paper at ICLR 2025
Figure 21: MS-ILLM 0.085 bpp. 0.271 LPIPS
Figure 22: HiFFc 0.080 bpp, 0.353 LPIPS
23

---

Published as a conference paper at ICLR 2025
Figure 23: Orignal image (CLIC2020).
Figure 24: DiflPC(OUrs) 0.039 bpp, 0.249 LPIPS
24

---

Published as a conference paper at ICLR 2025
Figure 25: MS-ILLM 0.037 bpp. 0.263 LPIPS
Figure 26: HiFic 0.037 bpp. 0.363 LPIPS
25

---

Published as a conference paper at ICLR 2025
Figure 27: Orignal image (DIV2K).
Figure 28: DiffPC(Ours) 0.079 bpp, 0.368 LPIPS
26

---

Published as a conference paper at ICLR 2025
Figure 29: MS-ILLM 0.104 bpp. 0.369 LPIPS
Figure 30: HiFic 101 bpp. 0.410 LPIPS
27


Reviews:
[{"review_id": "Syfj5qW9Dq", "reviewer": "Reviewer_HiqS", "summary": "This paper presents DiffPC, a two-stage image compression framework leveraging stable diffusion models. In the first stage, a multi-feature compressor encodes essential low-level details using minimal bitrates, while a pre-embedding module captures robust hybrid semantics by combining textual and visual information. In the second stage, DiffPC employs IC-ControlNet to ensure structural and textural consistency during image reconstruction, mitigating issues such as condition leakage. Extensive experiments demonstrate that DiffPC achieves state-of-the-art performance in both perceptual and statistical fidelity across various benchmark datasets, outperforming existing neural compression methods.", "questions": "[1] Although latent diffusion can enhance semantic details, it can also compromise some structural details, such as small faces and small text, which are not demonstrated in the paper.\n\n[2] I would like to understand the performance comparison between directly using VAE compression and DiffPC.\n\nI will consider raising the score once all concerns are addressed.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "DiffPC introduces an framework by integrating diffusion models with semantic refinement for image compression  and the paper conducts thorough quantitative and qualitative evaluations across multiple datasets and compares DiffPC with several state-of-the-art methods .The results consistently show superior performance in both perceptual metrics and statistical fidelity, particularly at ultra-low bitrates.", "weaknesses": "Multi-step inference combined with heavy ControlNet and pre-trained base models pose significant computational challenges for the decoding side. While pre-trained T2I diffusion models can enhance the quality of compressed images, their high computational cost makes them an unacceptable solution for image compression.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U6JBeP6Liq", "reviewer": "Reviewer_GPEP", "summary": "The paper proposes a diffusion-based image compression framework, DiffPC, aimed to achieve high perceptual fidelity at low bit rates. This model leverages both low-level degraded image representation and high-level hybrid semantic representation to control pre-trained Stable Diffusion. Experimental results show that the proposed method surpasses existing methods across various benchmarks, delivering superior perceptual quality and statistical fidelity.", "questions": "1. On page 2, line 097, ``Careil et al. (2023) ...`` PerCo was published in ICLR2024.\n\n2. On page 3, line 138, in the sentence ``He et al. (2022), based on the orthogonality of features in channel and spatial dimensions, devised an asymmetric autoregressive entropy encoder..`` there are two periods at the end.\n\n3. On page 4, line 183, there is a grammatical error in the sentence: ``Here, c and s respectively refer to low-level image controls (e.g., image contours, image degradation), with s denoting high-level semantic controls (e.g., textual descriptions of images, category label).``\n\n4. On page 10, line 520, in the sentence ``(2) w/o TAC: Removing the TAC ...``, should `TAC` actually be `TAD`?\n\n5. On page 11, line 545, ``En d-to-end optimized image compression.`` There is an extra space that should be removed.\n\n6. Is Eq. (3) accurate? $\\mathcal{DN}_{\\theta}(z_t, t)$ is the predicting noise rather than the mean.\n\n7. In Eq. (13), what does $\\eta$ denote in $TAD_{\\eta}$?\n\n8. In Figure 10(a), it appears that an arrow from the conv3×3 layer to the Spatial Transformer is missing. In Figure 10(b), there are two instances of $\\varepsilon$. Should they be represented by different symbols to avoid confusion?\n\n9. On page 4, line 178, the introduction to Stable Diffusion is incomplete, as it does not explain how $\\hat{z}_0$ is obtained within Stable Diffusion.\n\n10. The authors mention that ``the importance-weighted loss assigns more bits to texture details, enabling a more precise reconstruction of these features.`` Could the authors provide a visualization result of the bit allocation to support this conclusion?\n\n11. What impact do multi-scale features have on performance? In Eq. (12), the loss $L_{imp}$ is calculated based on the difference between $z_0$ and $c$, where multi-scale features do not appear to be necessary. Additionally, the authors have not demonstrated the role or necessity of multi-scale features in the ablation study. Clarifying these points would strengthen the paper’s argument for including multi-scale features.\n\n12. In Eq. (13), how is the initial value of $w$, specifically $\\sigma_{z_0}^2$, determined in practice? Additionally, how do you ensure that the trainable hyperparameter $w$ correctly learns to achieve precise bit allocation? \n\n13. This module adopts a pre-embedding approach that combines textual semantics with visual semantics. Could the authors clarify the individual roles of textual semantics and visual semantics? Additionally, what impact would using only textual semantics or only visual semantics have on performance?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The usage of high-level hybrid semantic representation in diffusion-based image compression methods is novel. I appreciate the idea of extracting semantic information from $\\hat{c}_x$.\n\n2. The paper effectively communicates the purpose and rationale behind each module in the DiffPC framework, providing clear justifications for their functionality.\n\n3. The paper includes thorough experiments with quantitative and qualitative metrics across multiple datasets, validating the effectiveness of DiffPC.", "weaknesses": "1. The paper lacks a thorough comparison and analysis of computational complexity. As diffusion models inherently introduce high decoding complexity, it is essential to evaluate and compare this aspect against previous SOTA methods.\n\n2. The paper has some issues related to writing clarity and accuracy of details. Please refer to questions 1-9 for specifics.\n\n3. There are issues with citation formatting in certain parts of the paper. For instance, on page 2, line 079, the sentence ``… generative adversarial frameworks Mentzer et al. (2020); Muckley et al. (2023)`` should be formatted as ``… generative adversarial frameworks (Mentzer et al., 2020; Muckley et al., 2023).`` \n\n4. Other existing image compression methods [1, 2] could be discussed and compared.\n\n[1] Zhiyuan Li, Yanhui Zhou, Hao Wei, Chenyang Ge, and Jingwen Jiang. Towards extreme image compression with latent feature guidance and diffusion prior. IEEE Transactions on Circuits and Systems for Video Technology, 2024.\n\n[2] Hagyeong Lee, Minkyu Kim, Jun-Hyuk Kim, Seungeon Kim, Dokwan Oh, and Jaeho Lee. Neural image compression with text-guided encoding for both pixel-level and perceptual fidelity. In International Conference on Machine Learning, 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1xFWaj8CcK", "reviewer": "Reviewer_1ruM", "summary": "(1).This paper proposes a diffusion-based high-perceptual fidelity image compression with semantic refinement. \n(2).The proposed method uses the low-level image information to ensure fidelity reconstruction,  and uses high-level texture semantics and robust prior in pre-trained latent stable diffusion to ensure realism reconstruction. \n(3).Extensive experiments demonstrate its effectiveness in reconstructing images with high perceptual quality texture at extremely low bitrates", "questions": "(1). Why do you need intermediate features f1 and f2 for a multi-feature compressor? Maybe using a simple compressor without multi-scale features is simple but effective. Please add an ablation study to validate its effectiveness for image compression.\n\n(2). Could you give a visual comparison of using importance-weighted MSE and standard MSE? It ensures proper bit allocation as you describe in your paper. You can visualize the bit allocation map to explain this.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Originality: This paper explores the pre-trained latent diffusion model using semantic control signals for extreme image compression. This method clearly explains that two types of semantics, namely low-level image information (visual semantics) and high-level semantics (textual semantics), can support image compression.\n\nQuality: The method is well designed and integrates the pre-trained latent diffusion model and semantic information for extreme image compression.\n\nClarity: The paper is generally well written and organized.\n\nSignificance: The significance of this work is profound as it addresses the extreme image compression using pre-trained latent diffusion models and semantic information.", "weaknesses": "(1). The authors do not describe some representative perceptual image compression methods corresponding to their work in the related work.  \n[1]. Multi-modality deep network for extreme learned image compression. AAAI 2023.\n[2]. Extreme Image Compression using fine-tuned VQGANs. Data Compression Conference 2024. \n[3]. Towards Extreme Image Compression with Latent Feature Guidance and Diffusion Prior. IEEE TCSVT 2024.\n[4]. Consistency Guided Diffusion Model with Neural Syntax for Perceptual Image Compression. ACMMM 2024.\n[5]. HybridFlow: Infusing Continuity into Masked Codebook for Extreme Low-Bitrate Image Compression. ACMMM 2024.\n[6]. Neural Image Compression with Text-guided Encoding for both Pixel-level and Perceptual Fidelity. ICML 2024.\n\n(2). In Figure 2, it is confusing that you use the same color of block to represent cross-attention layers. The pre-embedding module is not shown in Figure 2. The ICCN (i.e., IC-ControlNet) is not explained in Figure 2 or your paper.\n\n(3). In Eq.(7), the c ̂_x is a degraded image instead of features, which is the input of an image encoder, as shown in Figure 2. You need to add the image encoder to your equation.\n\n(4). In the part ``IC-ControlNet and Time-aware Decoupling” of section 3.2, the conditions are integrated with noise and processed through residual blocks before entering the main network. I cannot find any residual blocks as you say in Figure 9.\n\n(5). In Section 3.3, the symbol 〖text〗_x should be shown in Figure 2. The semantic pre-embedding module should be marked in Figure 2 (the same as (2)). The blended semantic s_x should be added to Figure 2 for clarity.\n\n(6). Please compare your method with VQGAN-based compression (DCC2024), TACO (ICML2024), and DiffEIC (IEEE TCSVT2024) and add them to Figures 4 and 5. VQGAN-based IC performs extreme image compression using codebook prior. TACO uses the text prior and DiffEIC uses the diffusion prior, which are very relevant to your work.\n\nReference papers:\n[1]. Extreme Image Compression using fine-tuned VQGANs. Data Compression Conference 2024. \n[2]. Towards Extreme Image Compression with Latent Feature Guidance and Diffusion Prior. IEEE TCSVT 2024.\n[3].Neural Image Compression with Text-guided Encoding for both Pixel-level and Perceptual Fidelity. ICML 2024.\n\n(7). What is TAC in your ablation study?  I think it is the TAD module in your procedure.\n\n(8). The text can assist image compression. I think you need to further analyze the text description for image compression. For example, you can use different caption methods to generate captions, which are used for your compression framework (test the text robustness of your method).\n\n(9). The author should compare the proposed method with state-of-the-art methods in terms of model complexity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rwA0EhC3iW", "reviewer": "Reviewer_suRn", "summary": "DiffPC is a novel two-stage image compression framework leveraging stable diffusion models to enhance perceptual fidelity at low bitrates. The model first compresses low-level image details and then integrates semantic features to reconstruct high-quality images. Using modules like a multi-feature compressor, IC-ControlNet, and a semantic pre-embedding module, DiffPC manages to maintain structural and textural consistency.", "questions": "1. In Line 45, are *realism* and *perceptual fidelity* truly equivalent concepts, or could their distinctions be further explored?\n\n2. The experimental section demonstrates that PSNR and SSIM metrics for generative models are not optimal. Could some insightful explanations be provided for why this is the case?\n\n3. Lines 48-49: Are human perception and image semantic consistency necessarily aligned? Not all visual tasks require precise details; for instance, in certain scene understanding tasks, a lower bitrate may be sufficient to convey overall semantics without losing information essential to human perception.\n\n4. The mentions of decoding collapse and condition leakage in the abstract are insightful explanations. Is there corresponding data analysis in the experimental section to support these claims?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. State-of-the-Art performance. Demonstrates superior perceptual and statistical fidelity at low bitrates compared to other image compression methods.\n2. Efficient semantic integration. The pre-embedding module effectively combines textual and visual semantics, validating that optimizing semantic information has the potential to improve image reconstruction quality. This is a fascinating discovery.", "weaknesses": "1. It is recommended that the authors improve the clarity and accuracy of the equations to facilitate reader comprehension. In the \"related work\" section, some formulas are not precise. For instance, in Eq. 2 and Eq. 3, the variances for the forward and reverse processes are both represented by ${\\beta}t$, which is incorrect. Notation such as $DN{\\theta}$ is typically represented as $\\epsilon_{\\theta}$ in existing literature. If the authors are referencing prior work, it is advisable to adhere to standard conventions to improve readability. Additionally, $\\textbf{c}$ generally denotes conditional controls, i.e., “text prompts”, which corresponds to $\\textbf{s}$ for \"textual descriptions\" in Eq. 5.\n\n2. For lines 78-87, using `\\citep` allows multiple references to be embedded smoothly within a sentence without disrupting the structure. This improves readability, especially when citing multiple sources.\n\n3. Similar works that fine-tune pre-trained diffusion models for low-quality image enhancement include:\n\n   - [1] Lin, X., He, J., Chen, Z., Lyu, Z., Dai, B., Yu, F., ... & Dong, C. (2023) in \"DiffBIR: Towards Blind Image Restoration with Generative Diffusion Prior\".\n   - [2] Wang, Y., Yu, Y., Yang, W., Guo, L., Chau, L. P., Kot, A. C., & Wen, B. (2023) in \"ExposureDiffusion: Learning to Expose for Low-Light Image Enhancement\" (ICCV).\n   - [3] Ma, J., Zhu, Y., You, C., & Wang, B. (2023) in \"Pre-trained Diffusion Models for Plug-and-Play Medical Image Enhancement\" (MICCAI).\n\nCould the authors further clarify how this manuscript differs from these related works?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)