Ultra Lowrate Image Compression with Semantic Residual Coding and Compression-aware Diffusion
Authors: Anle Ke
Xu Zhang
Tong Chen
Ming Lu
Chao Zhou
Jiawen Gu
Zhan Ma
Venue: ICML 2025 poster (2025)
Keywords: image compression
diffusion models
large language model
Primary area: applications->computer_vision
TL;DR: This paper propose a residual-guided framework that uses semantic residual coding and compression-aware diffusion model to achieve high-fidelity, ultra-low bit-rate image compression.

Abstract: Existing multimodal large model-based image compression frameworks often rely on a fragmented integration of semantic retrieval, latent compression, and generative models, resulting in suboptimal performance in both reconstruction fidelity and coding efficiency. To address these challenges, we propose a residual-guided ultra lowrate image compression named ResULIC, which incorporates residual signals into both semantic retrieval and the diffusion-based generation process. Specifically, we introduce Semantic Residual Coding (SRC) to capture the semantic disparity between the original image and its compressed latent representation. A perceptual fidelity optimizer is further applied for superior reconstruction quality. Additionally, we present the Compression-aware Diffusion Model (CDM), which establishes an optimal alignment between bitrates and diffusion time steps, improving compression-reconstruction synergy. Extensive experiments demonstrate the effectiveness of ResULIC, achieving superior objective and subjective performance compared to state-of-the-art diffusion-based methods with -80.7\%, -66.3\% BD-rate saving in terms of LPIPS and FID.

Ultra Lowrate Image Compression with Semantic Residual Coding and Compression-ware Diffusion
Anle Ke
Xu Zhang'
Tong Chen
Ming Lu'
Chao Zhou
Jiawen Gu
Zhan Ma
Abstract
besig
Existing multimodal large model-based image compression frameworks often rely on frag- mented integration of semantic retrieval, latent compression, and generative models, resulting in suboptimal performance in both reconstruc- tion fidelity and coding efficiency. To address these challenges, we propose a residual-guided ul- lowrate image compression named ResULIC, which incorporates residual signals into both se- mantic retrieval and the diffusion-based gener- ation process. Specifically, we introduce Se- mantic Residual Coding (SRC) to capture the semantic disparity between the original image and its compressed latent representation. A per- ceptual fidelity optimizer is further applied for superior reconstruction quality. Additionally, present the Compression-aware Diffuusio Model (CDM), which establishes optimal alignment between bitrates and diffusion time steps, improving compression-reconstuttii syn- ergy. Extensive experiments demonstrate the ef- fectiveness of ResULIC, achieving superior ob- jective and subjective performance compared state-of-the-art diffusson--ased methods with 80.7%, -66.3% BD-rate saving in terms of LPIPS and FID. Project page is available at https: / //njuvision http:/iiiiiinngguuiie...o github. wtc.h...ooo.... Lo/Res/IIC/....
Our frrmmeekk
AN79
RABULIC a PerCo
ORTIC
Teut-Statch
Percoptual Reatio
Realtere
Percepruall rideiiyy
Distvttion
Figure I. (a) The separate design for existing frameworks. The MLLM indicates the Multimodal Large Vision-Langune Model. The latent coding represents the compression features within the latent space. (b) Our pipeline with proposed Semantic Residual Coding and Compression-awan Diffusion Model. (c) Comparison with existing diffusion-baat based ultra lowrate image compression methods on CLIC2020 datasct.
in both objective metrics and subjective evaluations. How- ever, at lower bitrates, these methods often struggle with excessively smooth textures or the loss of fine details and structural information.
In response to these challenges, extensive research has been conducted on optimizing perceptual quality. Among them, Generative Adversarial Network (Goodfellow al., 2014) (GAN)-based approaches (Mentzer et al., 2020; Muckley et al., 2023) demonstrated competitive performance with visually-pleasing reconstruction. Some works have further attempted to enhance the visual reconstruction quality by in- corporating text guidance. Within the conventional encoder- decoder framework, they have integrated semantic infor- mation into either the encoding process (Lee et al., 2024) both the encoding and decoding processes (Jiang et al., 2023) to improve the perceptual quality of reconstruction.
1. Introduction
In recent years, dearning-based image compression tech- niques (Ballé et al., 2018; Minnen et al., 2018; Chen et al., 2021; Lu et al., 2022; Duan et al., 2023) have gained consid- erable attention and shown superior performance compared traditional codecs, such JPEG2000 (Taubman al., 2002) and VVC Intra Profile (ITU-T & ISO/IEC, 2020),
More recently, the advent of diffusion models (Ho et al., 2020; Rombach et al., 2022) has provided a turning point for this predicament. Existing diffusion model-based im- age compression methods (Careil et al., 2024; Lei al., 2023a; Li et al., 2024a) have shown more impressive results than GAN-based methods, achieving high visual quality reconstruction at extremely low bitrates (< 0.01 bits per pixel (bpp)). However, the reconstruction reliability (indi- cating consistency and fidelity) remains unsatisfactory, with significant differences from the original inputs.
'chool of Electronic Science and Engineering, Nanjing Uni- versity "K Kuaishou Technology. Correspondence to: Tong Chen <chentong@@ @ www..cducnn....
Proceedings of the 4 International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Ultra Lowrate Image Compression with Semantic Residual Coding and Compressio--wwaee Diffussonn
Ultra low bit-rate (<0.005bpp)
Low bit-rate (<0.05bpp)
FISH & CHIPS CHIPS
FISHE CHIPS
21SHA CHIPS
PIEHA AHEHAGHIPS
FISHA CHIPS
Original
[b) VTM 0.006ibpp
(c) Shanth
(d) Perco 0L0037bpp
(e) Ours 0.00344ppp
(6) Original
(b) MS DIS--LLM
(6) Perco
(d) DIMEIC 0.019tppp
(c) Ours 0.013bpp
0,021bpp
0L047bpp
0.0ESkpp
Figure 2. Visual comparison at extremely low bitrates. The bitrate is averaged over samples for each tested method. The major structure on the left (< 0.005 bpp) and the details of the text, hand, and clock on the right (< 0.05 bpp) better preserved.
From a compression standpoint, the objective is to extract compact and accurate information with minimal bitrate con- sumption, while ensuring the reconstructed images with balanced perceptual realism and fidelity. However, two key challenges hinder the effective integration of generative models into image compression tasks:
Compression-aware Diffussin Process, which effectively en- hances reconstruction fidelity while significantly improving decoding efficiency.
Experimental results show that our model achieves both objectively and subjectively pleasing results compared to current state-of-the-art approaches, with significantly re- duced decoding latency. The contributions of our work can be summarized as follows:
Multimodal Semantic Integration: The effective inte- gration of multimodal semantic information, minimiz- ing redundancy while ensuring high perceptual fidelity at extremely low bitrates.
We propose the Semantic Residual Coding (SRC), im- plementing multimodal large models as the residual extractor to remove the redundant semantic informa- tion between the original image and the compressed latent features, achieving joint bitrate reduction with efficient token index coding. A differential prompt op- timization method can be further applied to efficiently search for text prompts with improved fidelity.
Compression Generratinnnnn Alignment: Modeling the compression ratio with the noise scale in the difsusion process, enabling efficient and consistent reconstruc- tions across varying compression levels.
For challenge 1, as shown in Figure (a)), existing Multi- modal Large Language Models (MLLM)-based methods primarily focus on simply integrating information from both texts and other content (such as sketches, color maps, or structures) to reconstruct images, overlooking the semantic information already embedded in both sources, leading to semantic redundancy. To address this problem, we propose to implement& semantic residual coding module in Fig- ure 1(b) into our multimodal image compression framework, aiming to achieve overall minimal bitrate consumption. Be- sides, to optimize the perceptual fidelity, we propose dif- ferential prompt optimization strategy to find the optimal text prompts for improving the reconstruction consistency.
We propose the Compression-aware Diffusion Model (CDM), which modeling the relationship between dif- fusion time steps and the bitrate of compressed im- ages, sigmificantly enhancing the reconstruction fidelity while reducing decoding latency.
Based on the above key modules, a high-fidelity Residual-guided Ultra Lowrate Image Compression framework named ResULIC is proposed, achieving impressive visual quality at ultra-low bitrates, outper- forming existing SOTA method PerCo by -80.7% and -66.3% BD-rate saving in terms of LPIPS and FID.
For challenge 2, the degradation introduced by compression and the diffusion noising process share a common character- istic: as noise increases (or the compression ratio becomes higher), less information is preserved in the degraded image. Consequently, the compression ratio aligns inherently with the diffusion time steps. In this context, we aim to model this correlation. As illustrated in Figure 1(b), we incorporate the latent residual into the diffusion process and propose a
2. Background
Recent advancements in learned image compression, such as Ballé et al. (2017) and subsequent works (Ballé et al., 2018; Minnen et al., 2018: Chen et al., 2021; He et al., 2022: Lu et al., 2022: Duan et al., 2023), have showcased the potential of neural networks and advanced features like hyperpriors
2

---

Ultra Lowrate Image Compression with Semantic Residual Coding and Compresion-aware Diffusson
X
X
Semantic Residual Coding
Compression-aware Diffusion Model
CLIP
Semantic residual retrivaal
Perceptual fideitty optimier
Festure Compressor
CDM
X
nolen
ZE
Fived
Trahnable
Bis
Token indox
Arthmett Encoding
Arthmelic Decoding
Figure 3. ResULIC Overview: (1) The feature compressor transforms the original image 2 into the compressed latent feature &c. (2) The Semantic residual retrieval (Srr) generates optimized captions by analyzing both the decoded image x' and the original x. with the plugin play module Perceptual fidelity optimizer (Pfo) to further improve reconstruction quality. (3) Text tokens are embedded into c and combined with Sn as conditions for the Conpresiion-wware Diffusion Model (CDM) to generate the final image 2,..
and context models to significantly improve distortioooonn performance. Meanwhile, GAN-based approaches (Mentzer et al., 2020; Pan et al., 2022; Muckley et al., 2023) have focused on optimizing the reconstruction fidelity, especially for low-bitrate scenarios. Most Recently, compression algo- rithms leveraging targe-scale pre-trained generative models, such as Diffusion, have emerged, demonstrating highly com- petitive performance.
low-quality initialization during inference, resulting in decoupled and suboptimal process. Methods like ResShift and RDDM introduce custom noise schedulers, which re- quire full retraining and are incompatible with off-the-shelf diffusion models. Taking a step further, we propose a bitrate- aware residual diffusion scheme specifically designed for image compression, retaining the original noise scheduler for compatibility. Additionally, our evaluation demonstrates that dynamically adjusting diffusion time steps based on compression ratios enables a more efficient framework.
diffusion-based Generative Models. Diffusion mod- els, inspired by non-equilibrium statistical physics (Sohl- Dickstein et al., 2015), have achieved great success in visual tasks through advancements like DDPM (Ho et al., 2020) and LDM (Rombach et al., 2022), while strategies such as adding trainable networks (Zhang et al., 2023; Mou al., 2023; Zavadski al., 2024) enable efficient fine-tuning of pretrained models to reduce resource requirements.
Diffusion- Compressors. Several diffusion- based image codecs have been proposed recently (Theis et al., 2022: Ghouse et al., 2023; Hoogeboom al., 2023; Yang Mandt, 2023; Ma et al.. 2024: Xu al., 2024). These meth- ods utilize diffusion models achieve good performance at relatively high bitrates (bpp > 0.1). With the emergence of latent LDMs (Rombach et al., 2022), Lei et al. (2023b); Li et al. (2024a;b) employed it for low-bitrate compres- sion, further enhancing reconstruction quality. Similarly, Careil et al. (2024) integrated vector-quand vector-quantized image fea- tures and captions generated by a feed-forward model (Li et al., 2022) to improve compression performance. Overall, current iiffuusion-based codecs have begun to show outstand- ing performance, but the fidelity gap between Al-generated and original content persists, and the coding efficiency of these frameworks has not been fully explored at low-bitrate scenarios.
While diffusion models and their fine-tuning strategies have shown significant promise, an additional challenge lies in ensuring the reliability of generated content, particularly for tasks like compression. Previous works, such as textual inversion (Gal al., 2023) and Dreambooth (Ruiz et al., 2023), use soft prompts (continuous, learnable vectors op- timized for specific tasks) optimized for visual similarity, but these are not suitable for compression due to their high- dimensional nature. Other approaches, like PEZ (Wen et al., 2023), utilize discrete prompt optimization by projecting learnable continuous embeddings into the space of discrete embedding vectors to perform prompt optimization. The final optimized hard prompts remain in textual form and are capable of producing highly consistent images.
3. Framework Overview
The framework is illustrated in Figure 3. It can be divided into three major parts: the feature compressor, the seman- tic residual coding (Sec. 4) and the bitrate-aware diffusion model (Sec. 5).
In recent years, there has been a notable increase in the use of diffusion models for image restoration. Prominent works such as ResShift (Yue et al., 2024) and RDDM (Liu et al., 2024b) enhance the diffusion process by incorporat- ing the residuals between the original and degraded data. This approach significantly enhances reconstruction quality. Additionally, some works like PASD (Yang et al., 2024), StableSR (Wang et al., 2024a), start the diffusion process from low-quality images instead of pure noise to enhance efficiency and accelerate sampling. PASD relies solely on
The overall compression target consists of two interrelated parts: the latent features, which contain information such as structures, textures, and contours, and the extracted texts, which capture the remaining semantic information in the im- age. Two compressors are proposed to handle these compo- nents in a coordinated framework. The Feature Compressor first map the image x to latent space, obtaining the latent
3

---

Ultra Lowrate Image Compression with Semantic Residual Coding and Compresion-awwre Diffuusion
feature 2o Ex)). A neural compression network with similar architecture like (He et al., 2022) with encoder Es and decoder Dg is then implemented to compress the latent features.
common practice, we also get the caption from the decoded image directly from the decoded compressed latent feature. Subsequently, we input both captions into an LLM, which outputs semantic information present in the original image but missing in the compressed image as
(1) The reconstructed &e will serve guiding condition for Visual Adapter to control the diffusion to produce the final high-quality reconstruction. Detailed description can be found in the Appendix C.I.
Ze = Co = (z)(())))
Cres = fmilm (1)) Omlim Eiitton (x');
(5)
where @milm indicates the residual retrieval process real- ized by L.L.. This process enables the extraction of pre- cise and compact textual conditions, allowing for adaptive optimization of bitrate allocation between text and latent representations. A detailed demonstration is provided in Sec- tion 6.2 in Figure 10. As the latent bitrate R., increases, x' + I and Rg + O, our framework approaches pure postprocessing. Conversely, as Rs, decreases, x' + 0 and Ro... - Re- requiring more information to be encoded within the semantic representation.
z' D(ze), c= fmutim
R R. + ReZ!
3r = D(fam
Re, (c, N)).
Here It represents the overall bitrates combining texts and latents. The texts e is retrieved through MLLM model fmim- x' represents the intermediate reconstruction by the latent decoder D. Final reconstructed image x, is obtained by a diffusion model fam and the latent decoder.
4.2. Perceptual fidelity optimization (Pfo)
4. Semantic Residual Coding (SRC)
franted donst with black and orago
deaginutcioo cosuiiiiieee duaited proda
desgimatt occulsed grease bots sipaets
As aforementioned, existing methods usually rely on MILM to extract full-text descriptions of the original image as conditional guidance for compression or post-processing. Take postprocessing-bute baste frameworks as an example: no bits are needed to transmit the text, as it is reconstructed on the decoder side. However, lost semanties cannot be accurately restored.
0.0337app
50088pp
(a) Original
(b) Ours, w/o Pfo
[c] Ours, w/ PEZ
(d) Ours, w| Ple
Figure 5. Demonstration Pfo optimized prompts and correspond- ing reconstruction. The prompts after optimization are human- readable with mixed real words and gibherish (non-word token sequences).
C c= Emtino (I'), Re 0.
(3)
Meanwhile, many existing attempts based on diffusion sep- arately transmit two bitstreams for image and text, the re- trieved texts can be redundant since much of the information is already contained in 2', resulting in unnecessary bitrate overhead.
A Although captions generated by MLLM assist reconstruc- tion, they often fail to capture detailed textures and struc- tures, creating a consistency gap between the reconstructed and original images. This limitation hinders fidelity. To address this, we propose differential optimization process tailored for the diffusion model. Our goal is to find the prompt that is most suitable for the entire ResULIC with the best perceptual fidelity.
c c= fmimee)))) (1), Re + O.
(4)
It would become an obvious drawback for compression tasks especially when the total bitrate is low. Two modules are proposed to address this issue.
Formally, input captions are first converted into token in- dices using a tokenizer, referencing a predefined vocabulary glVixd used in CLIP (Radford et al., 2021), where [V] is the vocabulary size of the model and d the dimension of the embeddings. Instead of random initialization in PEZ. here we initialize the learnable continuous embedding P P= =ei,,.... [ei,..., EM] by previously obtained Cres, where A is the number of tokens worth of vectors to optimize. The sub- sequent optimization process can be seen in A Algorithm 1.
4.1. Semantic residual retrieval (Srr)
(2) bam sarrundddd infoctedin poss"
Cres barn oetected pand
Nismatc Redundancy
samounded
Figure 4. The process of Semantic Residual Retrieval (Srr) in- volves implementing the MLLM to remove redundant text and correct inconsistencies.
The optimization is driven by the loss function:
Cpro >YEE [llc- co(2n, n, Ze, Eciipe (P') (P))
In Figure 4, we redesign the caption retrieval pipeline. In addition to using the original image to obtain full caption as
+ Xo (P', (),,,),
(6)

---

Ultra Lowrate Image Compression with Semantic Residual Coding and Compresion-aware Difffusion
Algorithm 1 Pfo: Perceptual fidelity optimization
Diffusion
-100
-200
-300
-400
-600
N
Input:
Diffusion model: 0, CLIP model: S2, Target image:
2, Initialed embedding: P. Added Timestpps: N,, De-
noising steps: Na. Selected Timesteps: n; Learning
0
Zu
10,2 + VI RyCy
rate: X, Optimization steps: i
1: for I,..., do
2: # Forward Projection:
Compression
-0.1
-0.06
-0.03
-0.01
-0.005
Bpp
3: P' <- PP)
Figure 6. The intrinsic relationship hetween diffusion steps and compression ratios (bitrates). Larger diffusion steps correspond to lower bitrates.
4: # Select time step nt:
5: n + random! ( (Na/N,, 2N.,N.,...,
6: # Calculate the gradient w.r.t the projected embed-
In image compression, efficiency and reliability take prece- dence over the diversity of reconstructed images. Starting from standard Gaussian noise introduces unnecessary un- certainty. Therefore, we aim to avoid scenarios where the endpoint of noise addition (i.e., the starting point of denois- ing) is purely random noise. To address this, we define the following formulation:
ding:
7: g 4 Vp (O,ro I, n, x))
8:
# +Update the embedding:
9: P-P-
10: end for
11: return P Projg[P)
NN = JYN, Fe + V ÜN, N <T
(8)
where the first term represents the denoising loss for pre- dicting the noise at timestep n, stabilize the optimiza- tion process, we also incorporate Equation Lamx (P', = 1- 1-S(Edip (P")) (x)) as an auxiliary loss function with a small weighting factor do. Eelip- and Ediipv denote the text and visual encoders of the CLIP model, S is the cosine similarity between two embedding vectors. During optimization, the embedding P' is projected into the dis- crete space using the Projg function, which finds the nearest embedding P' Proje (P) in the CLIP embedding space. Euclidean distance is used for this projection, ensuring that the learned embeddings stay aligned with the vocabulary space of the CLIP model. The final optimized texts can be aP after several obtained from the updated P P Y dheo
Existing works (Yang et al., 2024; Wang al., 2024a) em- ploy similar strategies for tasks like deblurring and super- resolution typically, assuming fixed degradation levels (e.g.. upsampling ratios). In contrast, we jointly explore the cor- relation among bitrate, distortion, and diffusion steps, as shown in Figure 7. A significant challenge lies in under- standing how varying levels of compression noise, quanti- fied by the compressed latent residual prex Ze 20, impact the diffusion process.
5.1. Modeling Diffusion Steps with Compression Levels
DDPM (Ho al., 2020) already provided a very prelimi- nary prototype on modeling the bitrates and the reconstruc- tion quality under the scenario of lossy image compression. In diffusion models, the information entropy Ry at each timestep N can be approximately quantified using the KL divergence between the true posterior q= zo) and the model posterior po (2NN-1/2N) reflects the information entropy of zN
iterations with learning rate y.
Index Coding. In addition, we propose an efficient text encoding method that replaces zlib-based character encod- ing by encoding text embedding indices, with the decoding side using CLIP's text encoder to map these indices back to embeddings, significantly reducing bitrate consumption. See Appendix A.3 for details.
Rx ~ (g(En(2-11 | 2N, 20) || po(xN-1 2N)).
(9)
5. Compression-awarr Diffusion Model (CDM)
This brutal correlation mapping between Rate and Diffusion steps provides the intuition that: At larger N (near the end of the diffusion process with high noise), zy carries less information about 2p- Thus, the KL divergence and Rx are smaller, enabling higher compression rates but lower reconstruction quality. In contrast, smaller N (near the start of the diffusion process with low noise), 2N retains more information about 2o, leading to larger KL divergence and Rx, resulting in lower compression rates but higher reconstruction quality.
During the forward process for typical diffusion models, Gaussian noise is gradually added to the clean latent feature 2e. The intensity of the noise added at each step is controlled by the noise schedule 3r. This process can be written as:
= Vārzo + VI - āze, E (1, ,T,, (7)
where E ~ I), is a sample from a standard Gaussian distribution. Here, a, = 3, and a, = II'm1 a; As t increases, the corrupted representation 2n gradually ap- proaches Gaussian distribution.
Figure 6 visually illustrates this principle. To address these variations, we need to dynamically adjust N based on the
5

---

Ultra Lowrate Image Compression with Semantic Residual Coding and Compresion-aware Diffussion
En-1- Consequently, conditioning on žn- provides addi-
tional information about Zn. implying q(2n En 1 20, ze)
g(2n Iz ze). Hence, In 1 2n 20, Re.
Based on Theorem 5.2, the conditional probability
Tims Stop (t))
DPP
BAP
(56 C.35 LIDE E.IS CDS
N (yon- + v1 -n-1-nn- Pres: (1-0a-1) d))
simplifies to q( q(2n-1/2n; 20, zo) = q( (22n-1 20: ze)
Figure 7. Correlation between Bitrates and Diffusson Steps. (a) Reconstruction quality reaches the peak different diffusion steps for different bitrates. (b) Peak Projection Curve where optimal N, decreases hpp increases. (c) The adaptive strategy in (b) performs notably better (red curve) than fixed diffusion steps.
Assumption 5.3. Assume that the conditional probability can also be expressed as:
g(2n-1 2n. 20, 2o) ~ N (Zn-1; bnān + Zn o
(12)
Substituting Equation (11) into Equation (12), we obtain:
compression ratio. In Figure 7, we present three-factor analysis of rate (bpp), NN,, and quality (LPIPS). As observed, reconstruction quality varies with bitrate. Fixing the denois- ing steps, as shown in Figure 7(c), results in suboptimal performance. Guided by the observation in Figure 7, we propose to adapt the fixed N into compressed bitrate- adaptive N,. Specificallly, for different compression ratios, our method selects varying addingg endpoints while retaining the original noise schedule of the diffusion model. CDM overcomes the limitation of modifying noise sched- ules in the pre-trained models, achieving visually pleasing reconstruction without the need for retraining. Details of the diffusion process provided the following parts. Rel- evant proofs and derivations are provided in the subsequent sections and the Appendix B.
bn v an-1
(13)
v1 On
Sn Von-1 - Van In
(14)
vāN,
7a = 7n V1 GN,
(15)
Here, set o, to the same as in DDIM (Song et al., 2021),
on " 1-8, -- V1- ān ter. By setting 7 = = = 1, we correspond to dif- ferent sampling strategies, i.e., deterministic sampling or stochastic sampling. In the following experiments, 1) is set to unless stated otherwise. Comprehensive deriva- tions and experiments are available in Appendix B.5 and B.6. During directional sampling, samples are drawn from the model-predicted distribution (o 22,, So, ze). i.e., 2n In + Čnăo + one, as illustrated in Algorithm 2 in the Appendix B.6.
ān-i where 1) is a hyperparame-
5.2. Noise Adding Process
Existing methods that utilize residual diffusion, such as ResShift (Yue et al., 2024) and RDDM (Liu al., 2024b), typically employ new noise schedules and design Marko- vian forward process. Here, to implement Equation (8), we adopt the noise schedule of Stable Diffusion (Rombach et al., 2022) and design non-Markovian forward process that does not rely on q (2)). Definition 5.1 (Noise Addition Mechanism).
5.4. Training Objective
While keeping the diffusion model fixed, we need to train the encoder and decoder in the latent space as well as the visual adapter. We derive the following simplified loss function for training, detailed proofs are provided in Appendix B.7:
Lvss = W EEgg,, ||e cg( C, Ze, t)l
(16)
(0= ~ N
NN (Vonz + v1 On (1 Gn
SnVI-an where w Vii -I-VV and the condition
(10)
from texts. During training, we omitting the w parameter to stabilize training. Furthermore, we followed Equation (36) in the Appendix to estimate the original data do decode it into the pixel domain 2o. and performed further perceptual optimization using a weighted combination of LPIPS and MSF Then we can rewrite Equation (16) as:
where in can be sampled using the following equation:
Zn = Vanzo + VI On (TinPres + en), n E (I,...,,,))
(11)
5.3. Reverse Sampling Process
Lvi = Ezo. ||e ca(zn, C, Fn, 1)
Theorem 5.2 (Conditional Independence of on and 1n-1). Given the distributions defined in Equation (11), we have on 1 -n 2g: Ze:
Aa MSB 20) + A, LLIPS in)
(17)
Combining the Lvis with feature compressor, we define the total loss function as:
Proof. Under the given model, 2n is determined solely by (20, =e) and the noise term ens which is independent of
= Lvis + LD + AnLn
(18)
6

---

Ultra Lowrate Image Compression with Semantic Residual Coding and Compresion-awwre Diffussinn
original
Tatts (A)
tuit-spirad church
hnssooiic buildinggs boalt four people
partly
POW
cloudy sunlighturt g ever gin
0.0000kpp
X
(X)
(ip))
Sypp)
church metsrmment
The seu tiee we Historic four people
0.124hpp
Bity pileite sunlight.
0.122app
O 0.13lapp
DISTS
Mecuni spirrd
four poopll
Structure leys durt
parly cleusly
buildings Therk
rttrata (spp)
Brate lbsp!
Di-rate topp)
0.021hpp
bost tweeer
0.027app
0.comapp
Figure& 8, Quantitative comparision with statc-of-the methods on CLIC-2020 dataset.
ragt
repooii la 1al
Seln-ppked
buiiding
satron four
Thordde nag raperta pan
cluutyy aby.
LD= (z 2c) here is the distortion between 2g and ze- CR = R(2e) indicates the rate of the compressed latent. We adjust An to control the trade-off between rate, distortion, and the diffusion prediction loss.
Figure 9. Demonstration of the Srr at different bitrates. As the bitrates get smaller, more residual information would be retrieved. Keywords are highlighted with colors.
Appendix D.
6. Experiments
6.1. Experimental Setup
6.2. Main Results
Stable diffusion v2.1 (Rombach et al., 2022) is used as the backbone diffusion model. And the MLLM GPT4o (Ope- nAI, 2024) is applied to capture the image caption and their corresponding residual information. See Appendix C.4 for training details.
Figure 8 presents a quantitative comparison across multiple metrics the CLIC-2020 dataset. The results demonstrate that our method, ResULIC, consistently outperforms state- of-the-art art approaches on perceptual realism metrics such as FID and KID. Moreover, for perceptual fidelity metrics like DISTS, ResULIC achieves superior performance, while for LPIPS, it excels at relatively low bitrates, highlighting its effectiveness in improving fidelity. Additional comparisons datasets including Kodak, DIV2K, Tecnick, and MS- COCO are provided in Appendix A.1.
For evaluation, we tested several widely used datasets in- cluding CLIC-2020 (Toderici et al., 2020) in the main pa- per, and Kodak (Kodak, 1993), DIV2K (Agustsson & Tim- ofte, 2017), Tecnick (Asuni & Giachetti, 2014) and MS- COCO (Caesar et al., 2018) in Appendix A. Multiple met- ries are evaluated including PSNR, MS ASS-SSI (Wang et al., 2003), LPIPS (Zhang et al., 2018), DISTS (Ding et al., 2020) FID (Heusel et al., 2017) and KID (Binkowski et al., 2018). Among them, FID and KID are used to evaluate perceptual realism by matching the feature distributions between the original and reconstructed image sets. In con- trast, LPIPS and DISTS balance realism and fidelity, with relatively greater emphasis on the latter.
In addition, we performed BD-Rate (Bjontegaard, 2001) comparison with existing methods using MS-ILLM as the anchor, as shown in Table 1. Our method achieves the best performance in terms of perceptual realism and perceptual fidelity, while outperforming existing diffusion- based ultra- low bitrate compression methods on distortion metrics.
Table I. BD-Rate (%) comparison with state-of- thee-rrrr methods on CLIC-2020 datasets. ResULIC w/o Pfo is faster variant with the Pfo module disabled, while still maintaining good performance.
We compare our method to state-of-the-ar codecs, includ- ing the traditional codec VTM (Bross et al., 2021), GAN- based neural compressors HiFiC (Mentzer et al., 2020), MS-ILLM (Muckley et al., 2023), as well as diffusion- based compressors Text-sketch (Lei et al., 2023a), Dif- MFIC (Li et al., 2024a). CDC (Yang & Mandt, 2022) and PerCo (Careil et al., 2024). PerCo evaluated here is repro- duced version'. as the official implementation is not publicly available. The diffusion model in this version has been fully refined for the compression task. Additionally, we provide comparisons with the original paper's reported results in
Type
Malheh
Penaptual Realon
Panspiaal Filelty
Dituction
FID
KID
DISTS
LFIFS
PSNR
MS SSIM
GAN lased
MS ELLM
HIFI
65.0
154.7
39.0
75.7
0
122.3
DINEIC
.284
-2.9
Dilfusion based
RodTLIC Pio
ParCo
-549
-6.7
25.7
-577
113.4
-18.
120.4
ResULR:
42.9
48.
-51A
26.9
1303
31.3
Complexity. The average encoding time for 768x512 Kodak image is O. 10s for the latent compressor, 5.77s for semantic residual coding using Sir (+180s if Pfo is enabled). Decoding averages 0.60s/0.43s for 4/3 denoising steps. More detailed complexity analysis is in Appendix C.3.
7

---

Ultra Lowrate Image Compression with Semantic Residual Coding and Compresion-awwre Diffusson
Table Ablation experiments comparing each module on the CLIC-2020 dataset using PerCo the anchor.
Table 3. Performance comparison of MLLM models Kodak.
BD-Rate (8)
Method
M.J.M
Metiod
CDM
Tndex Coding
Sr
Pto
wf
LPWS
DISTS
HD
Kin
1SNR
MS-SSIM
LPIPS
DISTS
PSNR
MS-SSIM
8
0
0
GPT-4a
0
Pertin
-50
46
-3
-3.1
-6.1
466
Llama-3.2-11B
1.5
0.1
1.5
-0.2
-159
-14.
-157
-14.1
-5.9
-20
ResULIC w/
Owen-VL-MAX
-1.6
3.2
0.5
0.52
ResULIC-v
28
-10.5
-96
.360
-888
SenseChat-Visioon
0.42
-1.1
0.3
0.8
-156
-58.1
-76.2
GPT-40
-6.1
-3,8
0.8
3.1
ResULIC
00.7
.7.0
Llama-3.2-11B
-5.8
-4.2
1.2
3.5
-90.7
71.
-3.3
46.3
-6.3
96.3
-312
ReUULI
Qwen-VL-MAX
-6.0
-3.9
0.9
3.1
SenseChat- Vision
-5.7
-4.0
1.1
3.4
6.3. Evaluation
Instruct (Dubey et al., 2024), and three online models gpt-4o (OpenAI, 2024), SenseChat-Visiee and Qwen- Qwen-VL-MMr VL-MAX (Wang et al., 2024b). Table demonstrates the flexibility of our method across different MLLMs. The tested MLLMs achieve stable and comparable performance, especially after applying the Pfo module to optimize semantic retrieval.
Table 2 presents a comprehensive ablation study on the performance impact of each proposed module.
Case 1: Detailed comparison with PerCo. To more clearly demonstrate the advantages of our method over PerCo, we used PerCo an anchor for comparison. (1) The results in Table 2 demonstrate that the Pfo module and Index Coning integrate seamlessly into the PerCo framework, consistently improving reconstruction quality. (2) To solely evaluate the impact of the latent compressor, we provide a special ver- sion, ResULIC-VQ. which replaces our latent compressor with the same VQ compressor used in PerCo and disables all newly introduced modules. These comparisons highlight that the stable gains originate from our proposed modules.
Case 3: Effectiveness of Compression-aware Diffusion.
By integrating Definition 5.1, ResULIC leverages the exist- ing noise schedule of Stable Diffusion. This approach elimi- nates the computational overhead associated with retraining new latent diffusion model, as required by methods like ResShift and RDDM when modifying the noise schedule. It also significantly reduces denoising time redundancy while achieving substantial improvements across all metrics.
Case 2: Effectiveness of Semantic Residual Coding.
Table 4 uses the Adjustable Noise Schedule (ANS) proposed in PASD (Yang al., 2024) as the baseline for comparison. ANS affects only the inference stage by introducing signal information from the low-quality input image. Compared to w/o CDM (i.e., without using our Definition 5.1 for training, and initializing from random noise during testing). ANS shows some improvements, but it remains inadequate. In contrast, our proposed CDM bridges training and inference, effectively reducing the train-test discrepancy and achieving superior performance.
To quantify the gains of SRC, we conducted comprehen- sive experiments on the CLIC-2020 dataset. The results in Table demonstrate that the SIT and Pfo module brings consistent improvements in realism and fidelity to ResULIIC.
Adaptiveness of Srr to different bitrates. In Figure 9, as the bitrate decreases, semantic information in x' diminishes, leading to vague or incorrect descriptions, yet reconstruction quality remains stable. Figure 10 further shows that as total hpp decreases, the text bitrate proportion increases significantly, highlighting SIr's adaptivity varying bitrates. Additional visual examples are provided in the Appendix D.
Table 4. Bitrate-aware Diffusion enables both accelerated infer- ence speed (with fewer steps) and improved BD-rate performance.
Opp Ratio
0.06
Method
CDM
BD-Rate(96) 1.
1199
(Denoising Steps)
LPIPS
DISTS
PSNR
MS-SSIM
19%
0002
ANS
x(50/20)
0
0
0
0
0.060
3.00/9
ResULIC
x(50/20)
4.6
5.3
6.8
47.1
(4/3)
-54.4
-65.8
-49.7
-23.9
0/BL
al.atent Seature
0120006 4/700000 RASOONA BACONAD 2,N4000
Tixe [ree]
Tixe (srved)
wio Ser ww/ Str
7. Conclusion
Figure 10. The relative text/latent bitrate ratio under different total bitrates. The Srr effcctively removes more redundancy at lower bitrates.
In this paper, we proposed ResULIC, a residual-guided ultra lowrate image compression with Semantic Residual Coding and Biterate-aware Diffusion. Extensive experiments show both promising perceptual quality and reliability under ul- tra low bitrates. The proposed optimization strategy also demonstrates flexibility in being quickly applied to existing frameworks.
The impact of different MLLMs. MLLM models have been evolving rapidly. Existing methods may use differ- ent MLLMs for text retrieval. Here, we evaluate several models, including one offline model Llama-3.2-1 -ision-
8

---

Ultra Lowrate Image Compression with Semantic Residual Coding and Compresion-awwre Diffusson
Acknowledgement
Careil, M., Muckley, M. J., Verbeek, J., and Lathuilière, Towards image compression with perfect realism at ultra- low bitrates. In The Twelfth International Conference on Learning Representations, 2024. URL https:// openreviee ew net forum? d=kt.dETJJJ..
This work was supported in part by the Key Project of Jiangsu Science and Technology Department under Grant BK20243038, and part by the Key Project of the National Natural Science Foundation of China under Grant 62431011. The authors would like to express their sincere gratitude to the Interdisciplinary Research Center for Future Intelligent Chips (Chip-X) and Yachen Foundation for their invalu- able support. This project was also funded by Kuaishou Technology.
Chen, T., Liu, H., Ma, Z., Shen, Q. Cao, X., and Wang, Y. End-to-end learnt image compression via non-local attention optimization and improved context modeling. IEEE Transactions Image Processing, 30:3179-3199 -191, 2021.
Ding, K., Ma, K., Wang, S., and Simoncelli, E. Image quality assessment: Unifying structure and texture simi- larity. IEEE transactions on pattern analysis and machine intelligence, 44(5):2577-258) 2020.
Impact Statement
This work aims to advance the field of machine learning with applications in image compression. It holds potential value for research on image compression tasks under extreme bandwidth constraints and, to the best our knowledge, raises no ethical concerns.
Duan, Z., Lu, M., Ma, J., Huang, Y., Ma, Z., and Zhu, F. Qarv: Quantization-awan resnet vae for lossy image compression. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
References
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.21783 2024. URL https:// arxiv. awxiv.org//ao 2 21783.
Agustsson, and Timofte, R. Ntir 2017 challenge on single image super-resolution: Dataset and study. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition Workshops, pp. 126-135, 2017.
Asuni, N. and Giachetti, A. Testimages: a large-scale archive for testing visual devices and basic image pro- cessing algorithms. In STAG, pp. 63-70, 2014.
Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A. H., Chechik, G., and Cohen-or, D. An image is worth word: Personalizing text-to-image generation using textual inversion. In ICLR, 2023.
Ballé, J.. Laparra, V., and Simoncelli, E. P. End-to-end opti- mized image compression. In International Conference on Learning Representations, 2017.
Ghouse, N. F., Petersen, J., Wiggers, A., Xu, T., and Sautière, residual diffusion model for high per- ceptual quality codee augmentation. arXiv preprint urXiv:2301.0548) 2023.
Ballé, J., Minnen, D., Singh, S., Hwang, S. J., and Johnston, N. Variational image compression with scale hyperprior. In International Conference on Learning Represenaaions, 2018.
Goodfellow, I., Pouget-Abadie, J., Mirza, M. Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. Advances in neural informa- tion processing systems, 27, 2014.
Binkkoski,, M., Sutherland, D. J., Arbel, M., and Gret- ton, A. Demystifying mmd gans. arXiv preprint 1801.01 2018.
He, D., Yang, Z., Peng, W., Ma. R., Qin, H., and Wang, Y. Elie: Efficient learned image compression with unevenly grouped space-channel contextual adaptive coding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5718-5727, 2022.
Bjontegaard, G. Calculation of average psnr differences between rd-curves. ITU SGI6 Doc. VCEG-M33, 2001.
Bross, B., Wang, Y.-K., Ye, Y., Liu, S., Chen. J., Sullivan, G. J., and Ohm, J.-R. Overview of the versatile video coding (vvc) standard and its applications. IEEE Trans- actions on Circuits and Systems for Video Technology. (10):3736-7764, 2021.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Caesar, H., Uijlings, J., and Ferrari, V. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1209-1218, 2018.
Ho, J., Jain, A., and Abbeel, Denoising diffusion proba- bilistic models. In Advances in neural information pro- cessing systems, volume 33, pp. 6840-6851, 2020.

---

Ultra Lowrate Image Compression with Semantic Residual Coding and Compresion-aware Diffussinn
Hoogeboom, E., Agustsson, E., Mentzer, F,, Versari, L., Toderici, G., and Theis, L.. High-fidelity image compres- sion with score-based generative models. arXiv preprint arXiv:2305.182311 2023.
Liu, H., Li, C., Wu, Q. and Lee, Y.J. Visual instruction
ing. Advances in neural information processing systems,
36, 2024a.
Liu, J., Lu, G., Hu, Z., and Xu, D. A unified end-to-end framework for efficient deep image compression. arXiv preprint arXiv:2002.03370 2020.
ITU-T and ISO/IEC Versatile video coding. ITU-T Rec. H.266 and ISO/IEC 23090-3, 2020.
Jia, Z., Li, J., Li, B., Li, H., and Lu, Y. Generative la- tent coding for ultra-low bitrate image compression. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 26088-26098 2024.
Liu, J., Wang. Q. Fan, H., Wang, Y., Tang, Y., and Qu, L. Residual denoising diffusion models. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2773-2783, 2024b.
Jiang, X., Tan, W., Tan, T., Yan, B., and Shen, L. Multi- modality deep network for extreme learned image com- pression. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 1033-1041, 2023.
Loshchilov, I. and Hutter, F. Decoupled weight decay regu- larization. arXiv preprint arXiv:
2017.
Lu, M., Guo, P. Shi, H., Cao, C., and Ma, Z. Transformer- based image compression. In 2022 Data Compression Conference (DCC), pp. 469 469. IEEE, 2022.
Kodak, E. Kodak lossless true color image suite (photocd ped0992), 1993. URL tattt::/rr00 us / graphicss kodak/.
Ma, Y., Yang, W. and Liu, Correcting iiffusion-based perceptual image compression with privileged end-to-ent decoder. arXiv preprint arXiv:2404.04916. 2024.
Lee, H., Kim, M., Kim, J.-H., Kim, S., Oh, D., and Lee, Neural image compression with text-guided encoding for both pixel-level and perceptual fidelity. In International Conference on Machine Learning, 2024.
Mentzer, F., Toderici, G. D., Tschannen, M., and Agustsson, E. High-fidelity generative image compression. Advances in Neural Information Processing Systems, 33:1 1913- 1 11924, 2020.
Lei, E., Uslu, Y. B., Hassani, H., and Bidokhti, S. Text+ sketch: Image compression at ultra low rates. In ICML. 2023 Workshop Neural Compression: From Information Theory to Applications, 2023a.
Minnen, D., Ballé, J., and Toderici, G. D. Joint autore- gressive and hierarchical priors for learned image com- pression. In Advances in Neural Information Processing Systems, 2018.
Lei, E., Uslu, Y. B., Hassani, H., and Bidokhti, S. Text+ sketch: Image compression ultra low rates. arXiv preprint ar arXiv:2307.01944, 2023b.
Mou, C., Wang, X., Xie, L., Zhang, J., Qi, Z., Shan, Y., and Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprint arXiv:23302.08. 08453,, 2023.
Qie, X. T2i-adapter:
Li, J., Li, D., Xiong, C., and Hoi, S. Blip: Bootstrapping language- gaeguagg--mageeaae pre-training for unified rision-languaee understanding and generation. In International confer- ence on machine learning, pp. 12888-12900. PMLR, 2022.
Muckley, M. J., El-Nouby, A., Ullrich, K., Jégou, H., and Verbeek, J. Improving statistical fidelity for neural im- age compression with implicit local likelihood models. In International Conference on Machine Learning, pp. 25426-25443. PMLR, 2023.
Li, Y., Zhang, K., Liang, J., Cao, J., Liu, C., Gong, R., Zhang, Y., Tang, H., Liu, Y., Demandolx, D., et al. Lsdir: A large scale dataset for image restoration. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1775-1787, 2023.
OpenAL. Gpt-4o: An optimized language model. https: / www. openai. com/ gpt 4o, 2024.
Li, Z.,, Zhou, Y., Wei, H., Ge, C., and Jiang, J. Towards extreme image compression with latent feature guidance and diffusion prior. arXiv preprint arXiv:2404.18820 2024a.
Pan, Z., Zhou, X., and Tian, H. Extreme generative image compression by learning text embedding from diffusion models. arXiv preprint arXiv:22 2022.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 7748-8763. PMLR, 2021.
Li, Z., Zhou, Y., Wei, H., Ge, C., and Mian, A. Diffusion- based extreme image compression with compressed fea- ture initialization. arXiv preprint arXiv:2410.02640, 2024b.
10

---

Ultra Lowrate Image Compression with Semantic Residual Coding and Compresion-aware Diffusson
Rombach, R., Blattmann, A., Lorenz, D., Esser, and Ommer, B. High-resolution image synthesis with latent difffusion models. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Xu, T., Zhu, Z., He, D., Li, Y., Guo, L.,, Wang, Y., Wang, Z., Qin, H., Wang, Y., Liu, J., et al. Idempo- tence and perceptual image compression. arXiv preprint arXiv:2401.088 08920, 2024.
Yang. R. and Mandt, S. Lossy image compression with conditional diffusion models. arXiv preprint arXiv:2209.0692 2022.
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., and Aberman, K. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 22500-22510, 2023.
Yang, R. and Mandt, Lossy image compression with conditional diffusion models. Advances in Neural Infor- mation Processing Systems, 36, 2023.
Sohl- Sick-Dickstein, Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequi- librium thermodynamics. In International conference machine learning, pp. 2256-2265. PMLR, 2015.
Yang, T., Wu, R., Ren, P., Xie, X., and Zhang, L. Pixel- aware stable diffusion for realistic image super-resolution and personalized stylization. In European Conference on Computer Vision, pp. 74-91. Springer, 2024.
Song. J., Meng, C., and Ermon, S. Denoising diffusion implicit models. ICLR, 2021.
Yue, Z., Wang, J., and Loy, C. C. Resshift:
Efficient diffu- sion model for image super-resolution by residual shifting, Advances in Neural Information Processing Systems, 36, 2024.
Taubman, D. S., Marcellin, M. W. and Rabbani, M. Jpeg2000: Image compression fundamentals, standards and practice. Journal of Electronic Imaging, 11(2)2886 287, 2002.
Zavadski, D., Feiden, J.-F,, and Rother, C. Controlnet-xs: Rethinking the control of text-to-image diffusion models as feedback-control systems. In European Conference on Computer Vision, pp. 343-362. Springer, 2024.
Theis, L., Salimans, T., Hoffman, M. D., and Mentzer, F. Lossy compression with gaussian diffusion. arXiv preprint arXiv:2206. XxXv:22006008889 08889, 2022.
Zhang, L.,, Rao, A., and Agrawala, M. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3836-3847, 2023.
Toderici, G., Theis, L., Johnston, N., Agustsson, E., Mentzer, F., Ballé, J., Shi, W., and Timofte, Clic 2020: Challenge learned image compression, 2020.
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., and Wang, O. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 586- 595, 2018.
Wang, J., Yue, Z., Zhou, S., Chan, K. C., and Loy, C. C. Exploiting diffusion prior for real-world image super- resolution. International Journal of Computer Vision, pp. 1-21, 2024a.
Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., Fan, Y., Dang. K., Du, M., Ren, X., Men, R., Liu, D., Zhou, C., Zhou, J., and Lin, J. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.1211 12191, 2024b.
Wang, Z., Simoncelli, E. P., and Bovik, A. C. Multiscale structural similarity for image quality assessment. In The Shritt-Seventh Asilomar Conference on Signals, Sys- tems & Computers, 2003, volume 2, pp. 1398-1402. Icee, 2003.
Wen, Y., Jain, N., Kirchenbauer, J., Goldblum, M., Geiping, J., and Goldstein, T. Hard prompts made easy: Gradient- based discrete optimization for prompt tuning and dis- covery. In Advances in Neural Information Processing Systems, volume 36, 2023.

---

Ultra Lowrate Image Compression with Semantic Residual Coding and Compresion-aware Diffuusion
A. Additional Evaluation
Better
Perceptual Reallor (FID) 1
DFFICC
Tos slanch
Perceptual Fidelity (1 LPIPS) 1
Figure 11. Perceptual realism vs. perceptual fidelity at 0.02 bpp on the CLIC2020 dataset.
A.1. Comparison on other dataset
In addition to the experiments on the CLIC2020 (Toderici et al., 2020) dataset, conducted more detailed comparisons on MS-COCO (Caesar et al., 2018), DIV2K (Agustsson & Timofte, 2017), Tecnick (Asuni & Giachetti, 2014), and Kodak (Kodak, 1993). As shown in Figure 12, 13, 14, our method consistently outperforms state-of-the-art approaches on perceptual realism metrics such as FID and KID, and achieves superior performance in perceptual fidelity metrics like DISTS, while excelling in LPIPS at relatively low bitrates, demonstrating effectiveness in improving fidelity. Since ConDiff (Ma et al., 2024) and GLC (Jia et al., 2024) are not open-souree and tested differently on other datasets, we selected two bitrate points from their Kodak experiments for comparison, illustrated in Figure 15.
Evaluation details: For evaluation on the CLIC2020, DIV2K, and Tecnick datasets, we followed the approach of CDC (Yang & Mandt, 2022) by resizing images to a short side of 768 and then center-cropping them to 768x768. For MSCOCO-3K, we randomly selected 3,000 images from the MSCOCO dataset and, following PerCo, resized them to 512x512 for testing. The FID and KID metrics for all datasets were calculated on 256x256 patches, as described in HiFiC (Mentzer et al., 2020).
ResULIC w/o Pfo
ResULIC
MS-LLMM
PerCo
DINEEIC
CDC
0.0
MS 5..5
0.3
PSNR
22
0.7
0.2
0.5
1824
164
1024
10
Bit-rate tbpo!
Bitrrat (bp9)
Bitrrate (bpp)
0.025
14
24
DISTS
0.30
0.015
0.15
0.05
10
Bit-rate tppp
Bitrate (bps)
Bittrte (bpp)
Figure 12. Quantitative comparision with sfate-of-the-ar methods MSCOCO-3K 3K datasets.
12

---

Ultra Lowrate Image Compression with Semantic Residual Coding and Compresion-aware Difffusion
RecULIC wio Pfc
RestLIC
MS-ILLM
ParCo
DinExC
Text-SSeetch
CDC
65
MS 15-5555M
0.3
PSNR
0.7
0.3
0.5
1240
10c
10a
10
118-rxte (hop)
Dibrate (bpa)
Bibrrte (hop)
0.30
0:323
0.25
0.020
0:015
0.20
0:080
0:55
rat
Bit (Oop)
Figure 13. Quantitative comparision with slate-of-the--eee methods on DIV2K datasets.
ResULIC w/s Plc
ResULIC
MS ILLM
PerCo
DITTEIC
HIFIC
Text Sketch
CDC
033
0.2
30
Bitrate (bpp)
Bit-rate (boo)
B1: ritirate (opi))
03
330
0.22
0.2
0//2/
53
033
030
0.5
15
212
15
Blbste [hrp!
Dibeate (hop)
Bionnte (aps)
Figure 14, Quantitative comparison with state-of-the-ar methods on Tecnick datasets.
A.2. Additional Experiments of Pfo
We also compared our method with existing approaches, such as the hard prompt method PEZ. Both visual and quantitative results confirm that Pfo significantly enhances perceptual quality and objective metric performance, particularly under low-bitrate conditions. For example, as shown in Table 5, the optimized textual information produced by Pfo improves the fidelity of frameworks like PerCo and Text+Sketch, which employ entirely different architectures for compressing low-level content. Notably, this enhancement is achieved without requiring additional retraining, as the Pfo module leverages pretrained MLLM models, making it flexible and efficient plug-in tool.
A.3. The Impact of Index Coding
Our proposed index coding is simple yet effective method for text encoding. As illustrated in Figure 16, the text encoding logic of CLIP first tokenizes the prompt into several words, matches these words to their corresponding indices in a predefined vocabulary, and then maps these indices to their respective text embedding vectors. In contrast to directly encoding text strings, our method encodes the indices corresponding to CLIP's text embedding vectors. From Figure 16, we can see that our method can save a lot of text bit rate consumption compared to zlib.
13

---

Ultra Lowrate Image Compression with Semantic Residual Coding and Compresion-awwre Diffusson
RestALIC win Pta ResuLIC
MS-ILLM PerCo
DITEIC MIFIC
TRat- Ratt-Skethh
GLC
CDC
Cerbiif
2.
LPIPS
0.3
30-1
32-4
30nd
10-
Bit-rate (bpp)
Bit-rate (bpp)
1.23
1.20
0.7
DISTS
0.13
1677 Bit-rate (bpp)
Bit-rate (bpp)
Figure 15. Quantitative comparison with state-of-the-art methods Kodak datasets.
Table 5. Evaluation of Pfo with different low-level content compressors. PerCO uses "codebook "codebbok++hypepppiie +hyperprior" model for compression. Text+Sketch compresses binary contour maps has already applied PEZ framework.
Compressor (content) No optimizer
bpp/LPIPS
w/ PEZ
w/ Pfo
PerCo Text+Sketch ResULIC
0.0033/0.546
0.0035/0.532
0.0033/0.518
0.0280/0.586
0.0280/0.565
0.0028/0.49
0,0028/0.49
0.0028/0.46
A.4. The impact of sampling method
As shown in Appendix B.5, when 17 = and 7 1, the sampling process corresponds to deterministic sampling and stochastic sampling, respectively. We conducted experiments the CLIC2020 dataset, with the results presented in the figure below. These results reveal that adding noise during sampling compromises the consistency of reconstruction. Therefore, deterministic sampling (17 = 0) is the better choice.
14

---

Ultra Lowrate Image Compression with Semantic Residual Coding and Compresion-aware Diffusson
440 llts
Index Coding
Someone hoiding watching
144 bits
Coding
Someone while watching
red barn with sioo sits farm. bemn is reflected pond of weterThe sky cloudy and bam surroudedd by
824 bits
Someone holding phone while wanching video on It
bilts
Coding
Get Index
12100, 533, 5050, 320, 1951, 1519, 2113, 1455, 525, 585)
The image captures picturesque coastal scene with white lighthouse with black
wib
2000
red mofed perched lighthouse, outrrop. The structures
nestled nexe surrounded by lush greenery. The foreground is dominated by rugged rocks and the ocean, where waves crash against the shore. The texture of the rocks rough and jagged, while the water appears smooth and reflective.
house,
Arithmetic Coding
1336 bits
Index Coding
Figure 16. Visualization of Index Coding.
Determinstic Sa Sampligin==0) 0)
Stochastic Sampling(q=11
04
LPIP5
PSNR
MS-SSIM
2
TFP
10
10
122
Bit-rate (bpp)
Bit-rate (bop)
Bit-rate (bpp)
4.10
0.09
DISTS
0.22
0.000
1.000
0.15
0.04
1F.
10
0.000
10
122
Bit-rate (bpp)
Bit-rate (bpp)
Bit-rate (bpp)
Figure 17. Quantitative comparision of deterministic sampling(n 0) and stochastic sampling(1) on CLIC2020 datasets.
B. Mathematical Framework
In this section, we formalize the noise addition mechanism integral to our model and derive the essential parameter relationships. We begin by defining the distributions of the latent variables 2, and -, establish their independence, and subsequently derive the conditional probabilities. Finally, we present the parameter settings for different sampling methods, specifically DDIM and DDPM, and discuss the formulation of the training loss.
B.1. Noise Addition Mechanism
Definition B.1 (Noise Addition Mechanism). For each timestep n, the latent variables žn and 2n-1 are defined by the following Gaussian distributions:
Zn N (V (Vonzo v1 InPrrss (1 tn.)
(19)
nn ~ N Van-120 + vI an- 1n- (1 - an))))
(20)
Here, ān denotes the cumulative product of an up to timestep n, %o is a scaling factor, Pres represents a residual term, and I the identity matrix.
15

---

Ultra Lowrate Image Compression with Semantic Residual Coding and Compresion-awwre Diffusion
B.2. Independence of Consecutive Latent Variables
Theorem B.2 (Conditional Independence of 2n and 2p-1). Given the distributions defined in Equations (1I), we have
2n 1 2 20: Re:
Proof. Under the given model, 2n is determined solely by (zo: and the noise term en which is independent of én-1- Consequently, conditioning 2n-1 provides additional information about Zn, implying g(zn En 20; *c)
((2n (0, ze). Hence, 2n - 2n-1 20, Ze-
B.3. Conditional Probability Reduction
Based on Theorem B.2, the conditional probability simplifies to:
( Zn, 20, ze) q(=n-1 20, ze) ~N (va (vn- + V1 1h--17--1Pee (1 an-1)l)
(21)
B.4. Derivation Parameter Relationships
Assume that the conditional probability can also be expressed as:
q(zn-1 Zn, 20, ze) N (zn-1i Ln Snz0, o
(22)
Theorem B.3 (Parameter Relationships). For Equations (21) and (22) to be equivalent, the following system of equations must be satisfied:
In an
an + Sn = Vūn-1:
2 (1 σn) oh On-1,
(23)
vI 1n = in vI
an.
Proof. Equate the expressions for zn-1 from Equations (21) and (22):
2n-1 2n + Sn FnE,
(24)
= Vān-120 + vI ān-1 (7n-1Pres + en-1)
(25)
Substitute the expression for 2n from Equation (19) into the first equation:
En = in 20 + /1 On TrPres + V1 Cintr) + Sazo 20 + one
= (In On + Sm) + v On YoPres
(ane + v anfn)
(26)
Comparing coefficients with the second expression for 2n-1, we obtain the system of equations (23). The noise terms and en are mutually independent and follow standard normal distributions, leading the condition on the variances.
B.5. Sampling Method Parameter Settings
The parameter o, varies depending on the chosen sample method. We present the parameter settings for two prominent sampling methods: Deterministic Sampling and Stochastic Sampling.
B.5.1. DETERMINISTIC SAMPLING
When employing the Deterministic Sampling, the noise parameter is set to zero:
(27)
Substituting into the system of equations (23), we derive the specific parameter configurations for the sampler:
vI an-1
bn
(28)
V1 - On
II
an-1
Gn Vān-1
Van
(29)
/1 an
7/ 7n-1-
(30)
16

---

Ultra Lowrate Image Compression with Semantic Residual Coding and Compresion-aware Diffusion
To ensure consistency with Equation (19),
VaN, 7n v1 ON.
B.5.2. STOCHASTIC SAMPLING
For the Stochastic Sampling, the noise parameter is defined as:
1 On-1
V1
an ax
On
(31)
an
Substituting this on into the system of equations (23), obtain the parameter settings for the sampler:
1 0n-1
Vān
bn
(32)
ān
Vdn-1
1 an-1
Vān
5n vān-1
Vān
(33)
1 On
Vān-1
7n = (vT-an
On
On
Vnn--
(34)
Van
Vann -
To maintain consistency with Equation (19), we set:
VI an
ÖN,
Yn
Van
B.6. Sampling
Algorithm 2 Compression-aware Diffusion (Sampling)
Input:
Diffusion model 0, compressed feature &e Compute 2N, VaN,2. 2. + /I CN,
2: for n = .y....,, do
3: Compute &o based
4: n = N, then
5: Eo + Ze
6: else
Vanin vI-on(9necttn) ău +-
7:
Va.. -V.-vII-a VI-an
8: end if
9: Compute lns Sn. on by in van
V1-aa
Sn an Van-1"
On 'B
1-ū,-1
1-an
V
an-1"
10: Compute 2n-1 Inōn Eneo Gne
11: end for
12:
return 2g
B.7. Training Objective
For training, based on Equation (19), we express nn as:
Zn Vanzo + v1 an (DmPrrs + kn)
Vonzo + v1 anatates + v1 Gnhn
Vonzo + v1 - an7n 20) v1 nn
(Van V1 -an7)) 20 + v1 an 7n + V1 an En.
(35)
17

---

Ultra Lowrate Image Compression with Semantic Residual Coding and Compresion-aware Diffussonn
From Equation (35), derive the expression for 2g as follows:
(zn On (7 + Ga)), if # N,
20
Vān
v1 On 7m
(36)
Ze+
if n N.
The optimization for visual adapter is achieved by minimizing the following negative ELBO, i.e.,
Σ DKL [9(2n-1
Anz Ro. xe) po(zn-1 Ano ze)1
(37)
By combining this with Equation (22), we derive the visual loss function as follows:
Viss exotee 1n + Zn (In + Gn
Thus, for n # N, the training loss Lvis can be reformulated as:
Lvis- = 2. Ezp.cre,tteel20 - 2oll
22 E Ez
(zn - vI - ān . + En))
Vān VI- On "nn
(zn VI- on (7/ + co C, Zos t)))
Van - V1 - -On Yn
Sn an - Ezn.n,Aeite ||e EØ(2n, C2 Ros 1)1
(38)
Vān - VI On 7/
( On
Sn ān
When = N, the training loss Lvis is set to zero. We define the coefficient w training stability, this coefficient wn, is omitted during training.
To enhance
VI
On 7n
B.8. Conclusion
By formalizing the noise addition mechanism and deriving the requisite parameter relationships, we establish robust mathematical foundation for our model. The distinct parameter settings for deterministic and stochastic samplers facilitate flexibility in sampling strategies while maintaining consistency with the underlying probabilistic framework. Furthermore, the training objective Lvss is meticulously formulated to minimize the discrepancy between the true noise and the predicted noise ea, thereby ensuring effective model training.
C. Experiment Details
C.1. Visual Adapter
We create trainable copy of the pretrained UNet encoder and middle block, denoted as Ucopy. While ControlNet employs an additional convolutional neural network to map control images (e.g., Canny edges or depth maps) from the pixel domain to the latent space, our method eliminates the need for such an extra network due to the prior handling of latent features. Additionally, following ControlNet-XS (Zavadski et al., 2024), we design the copied encoder and middle block in accordance with the CosSrolNet-XS Type B architecture, illustrated in Fig. 18.
C.2. Perceptual Fidelity Optimizer
The Pfo follows the completion of semantic residual retrieval. During optimization, we use the AdamW opti- mizer (Loshchilov Hutter, 2017) with a learning rate set 0.3. We balance the time cost and reconstruction quality by using 500 steps for optimization, which takes approximately 175s for Kodak 768x512 image. During the optimization process, the time steps n are not randomly selected from all O to 1000 as in the forward process of the diffusion model. Instead, they are chosen based on our denoising steps. For instance, if we use 4-step DDIM sampling for denoising, our optimization process specifically targets the noise levels at these 4 steps. For more visual results, see Figure 22. And, we perform inference every 50 optimization steps and select the result with the lowest LPIPS (Zhang et al., 2018) to ensure the fidelity.
18

---

Ultra Lowrate Image Compression with Semantic Residual Coding and Compresion-aware Diffusion
Table Detailed complexity of each component.
Semantic Residual retrieval
Diffusion (4steps/3steps)
Latent
Total
Srr (+Pfo)
Index Coding
Enc (s)
0.10
5.77 (+175.29)
0.0015 0.0004
5.87 (+175.29)
Dec (s)
0.06
0.54/0.37
0.60/0.43
C.3. Complexity Evaluation
Table 6, we provide the complexity results each part. The Pfo module takes most of the encoding time for its iterative updating process. The total decoding speed remains competitive with existing methods. As shown in Table 7, complexity comparison with existing methods listed. These data were all tested the Kodak dataset using an RTX 4090 GPU,
Visual adapter
Z
Concat
Z
Figure 18. The architecture Visual Adapter.
Table Complexity Comparison of different methods based denoising steps, encoding speed, decoding speed. "w/o SRC' indicates removing both MILM and Pfo. indicates the extra time required for retrieving the full image caption.
Type
Method
Denoising steps
Encoding Speed (s)
Decoding Speed (s)
Cheng2020 ELIC
2.86
6.69
VAE-based method
0.057
0.079
MS-ILLM
0.038
0.059
GAN-based method
HiFiC
0,036
0.061
DiffEI
50 20
0.128
4.57/1.96
Diffusion-based method
PerCo
20/5
0.08 0.32)
2.13/0.64
Text-Sketch
25
62.045
12.028
Ours w/o SRC
4/3
0.10 (+ 3,24)
0.60/0.43
Ours
4/3
181.16
0.60/0.43
C.4. Other Implementation Details
1) Details of Baseline Model: To ensure a fair comparison, we evaluated all methods as follows: For open-sourced approaches (e.g., DiffEIC, PerCo, CDC, MS-ILLM, Text-Sketch), we used their publicly available pretrained models. For non-open-sourcee methods, we relied on the official results reported in their respective publications (c.g GLC, CorDDiff). Additionally, since the open-sourced HiFiC model operates higher bitrate, our data was sourced from the reproduced
19

---

Ultra Lowrate Image Compression with Semantic Residual Coding and Compressio--wwre Diffussinn
version by the DiffEIC's author. For PerCo, the performance of the open-sourced version is slightly different with the offical results, so we provided comparison with both version.
2) Training: The Feature Compressor and the Visual Adapter models which is based on the stable diffusion v2.1 are trained on the LSDIR (Li et al., 2023) and Flicker2w (Liu et al., 2020) dataset. We first preprocess this dataset using LLa (Liu et al., 2024a) to obtain a caption corresponding to each image. Note that this training process is divided into two stages. In the first stage, we set X and Xp to O, and RR to {24, 14, 4, 2, 1), training for 150K iterations. In the second stage, we set Ad to 1, Ap to {0.4, 0.6, 0.8, 1, 1), and AR to [36, 16, 6, 3, 1.5), training for 100K iterations. During training, we center-crop images to dimension of 512 512 and randomly set 30% of the captions empty strings enhance the model's generative capabilities and sensitivity to prompts.
As shown in Eq. (18), the term Ln R(ze) is estimated by probability estimation model with Pg during training, which is formulated as R = E[- (p4)]. Here, ý represents the output obtained by passing through the feature encoder followed by quantization.
3) Semantic Residual Retrieval, After the training of the Latent Compressor finished. The decoded image x' and its corresponding original image a are used extract the semantic residual.
The prompt for GPT4o to extract the raw captions is:
"Please describe this picture detail with 40 words. Do not provide any description about feelings.
Then we use the captured fmii(()) (Ir) and Emimm (a') to further capture the residual information Cres- The prompt used for GPT-4o
is:
"Original Image: Fmllm (x)'; Compressed Image: Fmllm (x')². Provide information that in the original image but not included in or mismatch with the compressed image. Don't 't include information that is already in the compressed image. Please use most compact words. Do not include the description for the compressed image. For example: if input is Original Image: A red barn surrounded by trees, reflected in pond. Compressed Image: red house surrounded by trees. Residual caption is A barn reflected in pond. Please refer to this to output. Do not appear words like 'compressed image', 'original image' and 'The semantic residual is'. If you think that the two descriptions mean almost the same thing, please output an empty string. "
D. More Results
(a)Original
(b)PerCo (official), 0.011bpp
(c)PerCo (SD2), 0.033bpp
(d)Ours, 0.011bpp
Figure 19, Visual comparison with the official PerCo version.
In Figure 19, the reproduced version of PerCO (marked as PerCo(SD2)) provides better visual quality than the original one. However, in terms of objective metrics in Table 8, the official PerCo version performs better. Therefore, we provide comparisons with both versions to demonstrate our method's efficiency.
In Figure 23, we also provide more examples of semantic residual by MLLM. In Figure ??, While each metric improves with its respective optimization, optimizd results tend to produce smoother textures, such as in water and trees, whereas LPIPS and AS-SSSI optimizations are better at preserving details.
20

---

Ultra Lowrate Image Compression with Semantic Residual Coding and Compresion-awwre Diffusion
Table Comparison with results in PerCo's original paper of BD-Rate. Note that the PerCo (official) is also LPPPS optimized.
Our method still shows significantly better performance.
BD-Rate (%) Kodak
Methods
LPIPS
PSNR
MS-SSIM
PerCo(official) PerCo(SD2)
21.7
88.3
15.4
Ours w/o Pfo
-34.5
-54.7
-33.4
Ours
-41.5
-52.0
-32.7
In Figure 20, 21, and 24, we present visualizations for different hpp values. It can be observed that our method achieves significantly impressive subjective quality.
21

---

Ultra Lowrate Image Compression with Semantic Residual Coding and Compresion-aware Diffussonn
RIDRID
Is IT FRID
Is IT FRID
Is IT FRID
Is IT FRID
YET?
YET?
YET?
YET?
(a) Original
(b) MS-ILLM, 0.046bpp (c) PerCo, 0.033bpp
(d) Ours 0.023 bpp
Figure 20, Visual comparison at low bpp.
22

---

Ultra Lowrate Image Compression with Semantic Residual Coding and Compresion-awwre Diffusion
(a) Original
(b) MS-ILLM, 0.153bpp
(c) PerCo, 0.125 bpp
(d) Ours 0.129 bpp
Figure 21. Visual comparison at high bpp.
muscular character with spiky, golden- yellow
followers lovehim icon avetar forums leaderboard gay humilitournanee urnaments
dbz commando circuits
judajudaromanticoom keeppints networking pics
0031bpp
O. 0.0031bpp
0.0031bpp
relocation shirehour greenpeace constituency top funfactfriday attractions germany pagoda japan
stunt wallpapers
traditional multi-tiered pagoda with curved roofs, against sky and surrounded by trees
villanova latest
wilksstgggernng attractions
pagoda temples transmission crate
staggering church
0.0038bppt
0.0038bpp
0.0038bpp
(a) Original
(b) Ours, w/o Pfo
(c) Ours, w/ PEZ
(d) Ours, w/ Pfo
Figure 22, Effectiveness of Pio optimization and corresponding texts
23

---

Ultra Lowrate Image Compression with Semantic Residual Coding and Compresion-aware Diffusion
Original
milmm (x)
Five colorful hats labeled Bahamas, in neon yellow, orange, green, pink, and blue,
arranged on wooden panel.
0.0011bpp
X
Lyr (X')
Cres
+Cres
mmtim (X)
0.138
Five colorful caps i yellow, orange, green, pink, and blue, diplayed rowon wooden surface. Bahamas wittan on each
0.133
0.133
0 bpp
Five colorful yellow, orange, green, pink, and blue, lndd roran wooden surface
0.038
0.040
Caps labeled aahamas 0. 0.0002bpp
0.0022
0.002
0.0042
yellow, orange, green, pink, and are arrnngd togethe
Cats aahamas arranged on a wooden panel 0.0005hpp
f mittm (X)
Original
An arched window with closed blue shutters set in wall. In front, two hibiscus flowers-one pink and one red.
0.0013bpp
x
Fytttmn
Cres
+Crss
mitm (X)
0,135
piikk and red hibiiccus fower foroground
0,135
0:137
arc window with dased blue thutters the background
bpp
0.043
0.044
arched window with blue shutters, red hibiscus fowess and green follage against vill
hibiscus flowers
0.0004ppp
0.0022
0.0031
0.0042
A pink, hibiscus flowers and arched window with blue shutters
painting wall with bluned ound
0.0008bpp
Figure 23, More visualization examples of MLLM MLLM- Semantic Residual Retrieval.
24

---

Ultra Lowrate Image Compression with Semantic Residual Coding and Compresion-awwre Diffusion
(b) PerCo 0.0033 Bpp
(c) Ours w/o Pfo 0.0030 Bpp
(d) Ours
(a) original
0.0030 Bpp
Figure 24. More subjective visual comparisons
25


Reviews:
[{"review_id": "wRRTkEZ2u6", "reviewer": "Reviewer_jfbj", "summary": "This paper proposes a diffusion model-based image compression method. First, a pretrained codec is used to obtain a latent representation, which guides the generation process of the diffusion model. Second, semantic information is introduced as additional guidance. To reduce the overhead of transmitting semantic information, only the residual between the ground truth and the decoded image’s semantics is transmitted. Experimental results show that on CLIC20, the proposed method achieves a lower FID compared to other approaches. For this paper, I cannot provide an evaluation of the novelty of the diffusion model part, as I am not very familiar with diffusion models. My main concern lies in the model's complexity and its comparison with GLC.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback. Regarding your inquiry, we would like to provide the following clarification:\n\nQ1: **Comparing with GAN-based GLC**\n\nA1: We highlight our advantage over GLC as below:\n1. **Ultra-low Rate Support**: GLC is a very competitive and representative GAN-based compression. However, similar to other GAN-based approaches (e.g., MS-ILLM, HiFiC), it faces challenges at ultra-low bitrates. Although GLC achieved bitrates below 0.01 bpp on datasets like CelebAHQ, such performance requires task-specific retraining for facial images, limiting its generalizability. For general natura images (e.g. Kodak, CLIC), GLC's minimum achievable bitrate is approximately 0.02 bpp, whereas our method remains effective even at an ultra-low bitrate of 0.0026 bpp. **As the bitrate decreases further, the effectiveness of feature-only optimization gradually diminishes.**\n2. **Simplified and Stable Optimization**: GLC employs GAN and a complex 3-stage training for refining compressed features. In contrast, our approach achieves superior visual quality and competitive objective results currently with simple ***MSE-optimized latent compressor***, and mainly focusing on how diffusion models can effectively enhance these features. From the perspective of feature compression, GAN based GLC, diffusion based PerCO, and our method share similar principle—using a pretrained VQGAN/VAE to perform image compression the latent features. However, the optimization strategies diverge. \n\nIn essence, GAN-based and Diffusion-based compression frameworks are two competitve pathes. These two approaches are highly complementary and can mutually reinforce each other.  Combining GLC's feature refinement with diffusion recovery could further advance ultra low-bitrate image compression performance.\n\nQ2: **Full resolution Memory Cosumption**\n\nA2: We fully understand this concern and would like to provide the following clarifications:\n1. **Test Setting**: As mentioned by the reviewer, our current test setting is to align with existing diffusion-based methods such as CDC and DiffEIC for fair comparison.\n2. **Memory Benchmark**: On full-resolution CLIC images (2048×1365):\n  - while GLC's closed-source nature prevented direct evaluation, its backbone VQGAN's peak memory usage is 11.2GB (w/o compression module of GLC) \n  - Our method's peak usage is 17.8GB (at the Diffusion's VAE decoder).\n\nThis marginal increase in memory usage represents a reasonable tradeoff. Considering our method's unique capability of achieving ultra-low bitrate compression and the potential of diffusion-based frameworks, we believe they hold promise for substantial advancements in both performance and efficiency.\n\nQ3: **Full resolution results and Comparing with GLC about FID**\n\nA3: As mentioned, our method demonstrates more promising performance at ultra low rates (<0.01bpp). GLC’s paper evaluates only a narrow bitrate range (0.02-0.03bpp), we matched this range for fair comparison on CLIC and DIV2K (full resolution) and MS-COCO 30K (256×256, following GLC’s setup). The results can be seen at https://anonymous.4open.science/api/repo/public-5C7B/file/comparison.png?v=f3d457fd. As can be seen from the figure, our method still provides better or competitive results over GLC for full resolution tests.\n\nQ4: **Question about Table 4: Fewer steps but better performance**\n\nA4: In Table 4, we provided two baselines for comparison. One is a strategy ANS used in previous works, the other is our ResULIC without the CDM strategy. The results showed that with **CDM** and our **Predicted-$x_0$ training strategy**, better performance can be achieved with fewer steps.  \n\nMore explanation of these two modules:\n\n**CDM**: CDM accelerates denoising by initiating diffusion from a distribution conditioned on compressed feature means $z_c$ rather than pure noise (Eq.8). This intermediate starting point reduces sampling steps while enhancing fidelity. \n\n**Predicted-$x_0$ Strategy**: Our sampling process (Algorithm 2 in the Appendix) suffers from error accumulation when deriving $z_0$ from noise predictions. The training strategy predicts $x_0$ to reduce error accumulation from noise prediction, improving accuracy and enabling high-fidelity reconstruction with fewer steps.\n\nAdditional experiments are provided in the table to demonstrate reduced decoding delay and improved performance（In \"Denoising Step\", the numbers represents sampling steps (50，20，4) for <0.05bpp and  (20，10，3) for ≥0.05bpp）:\n\n||||||BD-Rate(%)↓|||\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Method|w/predicted-x_0 training|w/CDM|Denoising Steps|LPIPS|DISTS|PSNR|MS-SSIM|\n|ANS|×|×|50/20|0|0|0|0|\n||×|×|50/20|4.6|5.3|6.8|47.1|\n||×|√|20/10|-29.3|-35.4|-19.5|-10.6|\n|**ResULIC**|√|×|20/10|-25.8|-29.5|-25.3|-16.7|\n||√|√|20/10|-58.4|-66.8|-48.7|-22.9|\n||√|√|4/3|-54.4|-65.8|-49.7|-23.9|\n\nWe appreciate your insights and hope this clarifies the trade-offs and unique contributions of our work.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "As a perceptual quality-optimized image compression method, it is reasonable for the authors to use FID, LPIPS, and DISTS as evaluation metrics.", "theoretical_claims": "I am not very familiar with diffusion models, but I reviewed the proof as thoroughly as possible.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "Researchers working on integrating diffusion models into image compression may find this paper interesting.", "essential_references_not_discussed": "I believe this paper provides a sufficient discussion of related work but lacks coverage of distortion-oriented image compression methods.", "other_strengths_and_weaknesses": "This paper achieves strong performance in terms of FID. Compared to other diffusion-based methods, it requires fewer iteration steps, resulting in a faster decoding speed. However, I believe its decoding speed is still slower than GAN-based methods and comes with higher computational complexity. According to Figure 15, its LPIPS performance is not as good as the GAN-based GLC, yet it comes with higher complexity. \n\nFor evaluation on the CLIC2020, DIV2K, and Tecnick datasets, the authors followed the approach of CDC (Yang & Mandt, 2022) by resizing images to a short side of 768 and then center-cropping them to 768×768. I wonder if the authors could provide results for full-resolution inference, as I am curious about the memory overhead of the diffusion model during full-resolution inference.\n\nIn the GLC paper, the FID evaluation method is consistent with that of HiFiC. The authors should compare their method with GLC.\n\nI am generally leaning towards borderline.", "other_comments_or_suggestions": "line 1033:  which takes approximately **175s minutes** for a Kodak", "questions_for_authors": "How to understand the result in Table 4 where fewer steps lead to better performance?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ve2IK6kyq0", "reviewer": "Reviewer_XQ42", "summary": "The present paper describes a perception-oriented image compression framework that utilizes the residual multi-modal semantics (w.r.t what’s could be recovered by the latent decoder) as guidance conditioning for the diffusion denoising process towards improved fidelity at lower rate costs. The insight that fidelity improves with increased number of denoising steps, for a given bitrate, the paper also describe a variable compression level method with improved flexibility. The method differs from prior MLLM-based image compression methods by accounting for the semantic redundancy between the image latent representations and multi-modal embeddings, thus yielding improved rate-perception trade-off.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: **What is the difference between conditioning the diffusion model directly with visual cues instead of explicit texts**\n\nA1: In our method, explicit texts primarily compensate for semantic gaps in the visual condition, especially at ultra-low bitrates. For example:\n- **Low-bitrate regime**: In extreme low-bitrate scenarios where compressed features may retain only basic structural information (e.g., object locations and edges), relying solely on these limited visual cues can lead to vital semantic mismatch (e.g cat to dog).  Our method adds text descriptions to fill in missing details, bridging the semantic gap from heavy compression. The subjective results can be viewed at: https://anonymous.4open.science/api/repo/public-2D3D/file/test2.jpg?v=1ef61402\n- **High-bitrate regime**: As the bitrate increases, the visual condition retains sufficient information, and the reliance on text gradually diminishes. This is empirically validated in Figures 9,10 of our paper, where the residual text length drops to zero at higher bitrates.\nThus, visual conditions and texts are synergistic: the former anchors structural fidelity, while the latter contains compact high-level semantics (visual cues contain richer and preciser information, while texts are more compact). Our hybrid solution can utilize both sources to achieve overall improved performance.\n\n|w/o text||w/ text||\n|:-:|:-:|:-:|:-:|\n|Bpp|LPIPS(↓)|Bpp|LPIPS(↓)|\n|0.002163|**0.496696**|0.002681|**0.468738**|\n|0.004901|**0.403217**|0.005481|**0.386965**|\n|0.021926|**0.205847**|0.022849|**0.232723**|\n|0.062561|**0.135126**|0.063262|**0.123806**|\n\nQ2: **Would there be information loss during semantic residual retrieval**\n\nA2: During the process of semantic residual retrieval, what we extract is not directly the semantic content of residual features, but rather, as illustrated in Figure 6, we use MLLM to extract the residual information between the original image $x$'s caption and caption of the compressed image $x'$: $caption_{res}=MLLM(x)-MLLM(x')$, instead of $MLLM(x-x')$ . The table below shows the results of semantic residual retrieval for different models (GPT-4o, Llama-3.2-11B) in Kodak.\n\nTherefore, the semantic residual in the form the text is usually conveyable as long as the text description matches with the input image. In terms of text-level information loss, it mainly depends on the capability of the underlying MLLM models as shown here in the table.\n\n|w/o Srr||W/ Srr(GPT-4o)||W/ Srr(Llama-3.2-11B)||\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|Bpp|LPIPS(↓)|Bpp|LPIPS(↓)|Bpp|LPIPS(↓)|\n|0.005189|**0.476899**|0.00372|**0.468738**|0.003737|**0.493859**|\n|0.008308|**0.386744**|0.006777|**0.386965**|0.006969|**0.390045**|\n|0.033089|**0.233531**|0.031589|**0.232723**|0.031614|**0.233985**|\n\nFrom the table, we can observe that since GPT-4o possesses stronger capabilities, the LPIPS between reconstructions after SRR and those from the original captions remains nearly identical. In contrast, Llama-3.2-11B shows some degradation, suggesting potential loss of certain keywords in the process.\n\nHowever, as you have rightly considered, the semantic residual represented in texts may not perfectly align with the pixel-level residual signal. Hence, we proposed the PFO module to compensate for the feature information that natural language cannot describe. Please correct us if we misunderstand the question you raised. \n\nQ3: **Why would such “gibberish” prompts be beneficial to guiding the denoising process**\n\nA3: Thank you for raising this insightful question. Regarding this issue, we provide the following clarification of the PFO pipeline: \nA word is first converted into an embedded token space, and the PFO performs search in the token space through gradient backpropagation guided by output image reconstruction quality. The token vector is iterally updated and finally converted back as a new word. Semantic control in diffusion models operates through CLIP's text embedding space rather than surface-level language. \n\nPFO performs gradient-based search in this latent token space, optimized solely for image reconstruction quality without explicit linguistic constraints. Therefore, the PFO may break the grammar and readability of the word. Consequently, the generated captions may not always exhibit high readability (though still retaining interpretable tokens such as \"DBZ (Dragon Ball Z),\" \"pagoda,\" and \"temples,\" as shown in Figure 23). \n\nHowever, they can achieve higher semantic alignment with the original image in CLIP's embedding space. The high-dimension token space encapsulates information beyond the expressive capacity of human-readable language, thereby enabling more effective guidance.\n\nQ4: **Figure 5 is not properly referred to in the main text**\n\nA4: Thank you for your reminder. The principle of Figure 5 is indeed consistent with our response to **Q3**, and we will subsequently add a detailed explanation in the main text.", "claims_and_evidence": "The empirical evidence and ablative analysis are comprehensive and could adequately support the claims. It makes sense to steer the denoising diffusion process with only semantic residuals and the experimental results look promising.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria, for the most part, make sense for the problem at hand. The only issue is, I wonder what is the difference between conditioning the diffusion model directly with visual cues instead of explicit texts? The semantic residual is expected to be of high-frequencies whose encapsulated information may not always be `conveyable' via textual descriptions. Would there be information loss during semantic residual retrieval? Also, it could be seen from Figure 5 in the main text and, for instance, Figure 23 in the supplementary materials, that the textual prompts could contain gibberish that could not be comprehended by human. Why would such prompts be beneficial to guiding the denoising process?", "theoretical_claims": "I have checked the correctness of the proofs. They look sensible to me.", "experimental_designs_or_analyses": "I have checked the soundness of experimental designs and analyses in the main text as well as the supplementary materials. Please refer to  **Methods and Evaluation Criteria** section for my questions.", "relation_to_broader_scientific_literature": "They are related to foundational model-based generation, in particular conditional diffusion models and multi-modal vision language models.", "essential_references_not_discussed": "No essential references missing to the best of my knowledge.", "other_strengths_and_weaknesses": "N/A.", "other_comments_or_suggestions": "= I think Figure 5 is not properly referred to in the main text.", "questions_for_authors": "Please refer to  **Methods and Evaluation Criteria** section for my questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ta0kw9LHty", "reviewer": "Reviewer_5kJu", "summary": "This paper introduces ResULIC, a novel framework for ultra-low-bitrate image compression that integrates semantic residual coding (SRC) and a compression-aware diffusion model (CDM). SRC is proposed to capture the semantic disparity between the original image and its compressed latent representation. CDM is used to establish a relationship between diffusion time steps and compression ratios, improving reconstruction fidelity and efficiency. Compared to the state-of-the-art diffusion-based compression method PerCo, ResULIC achieves superior rate-distortion performance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer’s insightful comments and recognition of our work’s potential. The questions raised are highly valuable for improving the robustness and impact of our research.\n\nQ1: **Implementation details about compared methods**\n\nA1: Thank you for your reminder. To ensure a fair comparison, we evaluated all methods as follows: \nFor open-sourced approaches (e.g., DiffEIC, PerCo, CDC, MS-ILLM, Text-Sketch), we used their publicly available pretrained models. For non-open-sourced methods, we relied on the official results reported in their respective publications (e.g GLC, CorrDiff).\nAdditionally, since the open-sourced HiFiC model operates at a higher bitrate, our data was sourced from the reproduced version by the DiffEIC's author. For PerCo, the performance of the open-sourced version is slightly different with the offical results, so we provided comparison with both version.\n\nQ2: **Choice of Stable Diffusion v2.1 as Backbone**\n\nA2: Thank you for raising this insightful question. It is possible and in line with expectations that  larger and better diffusion models can have improved performance. The choice of Stable Diffusion v2.1 as the backbone was primarily driven by two key considerations:\n- Fair Comparison with Existing Works: Current state-of-the-art methods (e.g., PerCo, DiffEIC) adopt SD v2.1 as their base model. To ensure a direct and equitable evaluation of our method’s performance improvements, we maintained consistency in the backbone architecture.\n- Complexity-Performance Balance: While advanced models like FLUX could potentially enhance compression performance, their large parameter size (12B) and high GPU memory usage make them impractical for real-world applications for image compression. Therefore, we opted to use smaller models to validate the reliability of our method.\n\nWe fully acknowledge that leveraging newer diffusion models is a promising direction for further improvements. In fact, we are actively working to secure additional computational resources to explore such extensions in future research. At the same time, we are also investigating how to better utilize the quantized version of FLUX to enhance our performance. Your suggestion aligns perfectly with our roadmap for advancing this work.\n\nQ3: **User Studies and Perceptual Validation**\n\nA3: To fully demonstrate the comparison of subjective quality, we have included some example images in the main text and appendix. However, we fully acknowledge that user studies are indispensable for comprehensive validation.  \n\nAfter concentrated testing efforts, we're pleased to share some initial subjective user study results(https://anonymous.4open.science/api/repo/public-E45F/file/user_study.png?v=0cb25072). Due to time constraints, we conducted a preliminary evaluation at two bitrate points with 20 participants. Each participant assessed randomly selected images from the DIV2K test set (30 images per bitrate point). The final results were calculated as the average of all votes. Subjective tests confirm our method's clear visual advantage over baselines. We will keep working on providng a more comprehensive user studies. \n\nThank you again for the constructive feedback—we will incorporate these suggestions to strengthen our research further.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "Limited real-world testing or user studies. While objective metrics like LPIPS and FID are useful, human perceptual studies (e.g., Mean Opinion Score) would provide additional validation of realism. This is not a major issue, but including user evaluations would further strengthen the claims.", "relation_to_broader_scientific_literature": "1. ResULIC builds directly on recent research in learned image compression (Balle et al. (2017, 2018), HiFiC (Mentzer et al., 2020)) , multimodal generative models (PerCo (Careil et al., 2024)), and diffusion-based restoration (ResShift (Yue et al., 2024), RDDM (Liu et al., 2024)).\n2. It differentiates itself by combining residual-based multimodal compression with diffusion-aware bitrate modeling, which has not been explicitly addressed before. \n3. Its evaluation approach is aligned with state-of-the-art perceptual fidelity methods, reinforcing its relevance in the field.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n1. The idea of using MLLM-based retrieval to encode only missing semantic details rather than full textual descriptions is an innovative and practical refinement.\n\n2. CDM dynamically adjusts diffusion steps based on compression levels, improving efficiency without needing extensive modifications to pre-trained models.\n\n3. Strong experimental results and relevance to ultra-low bitrate applications.\n\n4. Comprehensive experiments and ablation studies.\n\n\nWeaknesses:\n\n1. Why was Stable Diffusion v2.1 specifically chosen as the backbone? If the latest SOTA diffusion model FLUX is used, is possible to further improve the compression performance.\n\n2. Limited real-world user studies: While ResULIC shows strong performance in perceptual metrics (FID, LPIPS), there are no human perceptual studies. Real-world testing on actual user preference rankings could validate if the perceived quality aligns with objective metrics.", "other_comments_or_suggestions": "No.", "questions_for_authors": "No.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SCkjgjDRtO", "reviewer": "Reviewer_PLiJ", "summary": "The paper introduces an ultra-low rate image compression method combining Semantic Residual Coding and a Compression-aware Diffusion Model. SRC efficiently encodes semantic differences between original and compressed latent images, minimizing redundancy, while CDM aligns diffusion steps with compression levels to improve reconstruction fidelity. Experiments demonstrate that ResULIC surpasses existing diffusion and GAN-based methods, achieving up to 80.7% and 66.3% BD-rate savings in LPIPS and FID metrics, respectively, particularly excelling at ultra-low bitrates.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your kind recognition and these insightful questions—they are indeed crucial for understanding our work.\n\nQ1: **Regarding the computational cost of PFO and practical use cases**\n\nA1: For PFO, our goal is to demonstrate the potential of prompt optimization in further improving the overall fidelity, even though it is currently computationally intensive.  \nAdditionally, **the PFO introduces no additional latency during decoding**, making it acceptable for performance-critical scenarios, such as cloud-based restoration services where computational resources are readily available and the encoding latency is not strictly constrained.\nHowever, even without PFO, our method still achieves superior performance compared to existing approaches (as demonstrated in Table 1). This underscores the effectiveness of our core framework. Furthermore, we are actively working on optimizing PFO's efficiency to facilitate its wider practical deployment in future applications. \n\nQ2: **Reproduction vs. official PerCo implementation**\n\nA2: Since the official PerCo code is not publicly available, we mainly compared with a publicly avaiable version. This reproduced version provides some drop in metrics like LPIPS/MS-SSIM, but better at FID/KID and visual quality.\nNotbaly, to ensure a fair comparison with both versions: We have also benchmarked against PerCo’s official  objective results (in Table 9) and subjective comparisons (in Fig. 20) in Appendix D. We also list the table below. Our method also consistently outperforms PerCo (offical), even when comparing with its strongest metrics like PSNR/LPIPS/MS-SSIM.\n\n|||BD-Rate(%)↓|||\n|:-:|:-:|:-:|:-:|:-:|\n|Method|LPIPS|PSNR|MS-SSIM|\n|PerCo(offical)|0|0|0|\n|PerCo(SD2.1)|21.7|88.3|15.4|\n|**ResULIC W/o Pfo**|-34.5|-54.7|-33.4|\n|**ResULIC**|-41.5|-52.0|-32.7|", "claims_and_evidence": "Most claims are supported by experimental evidence regarding Semantic Residual Coding and the Compression-aware Diffusion Model, both validated by quantitative and qualitative analyses. However, the Pfo module increases encoding time (~175s/image), raising practical concerns not fully addressed.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are appropriately designed for ultra-low bitrate image compression. However, the practicality of the computationally expensive Perceptual Fidelity Optimization module (~175s/image) warrants further clarification.", "theoretical_claims": "The theoretical claims and proofs look correct and are clearly presented. However, I did not look into the details very much.", "experimental_designs_or_analyses": "Yes, the experimental design and analyses are rigorous, with metrics and ablations. Minor issues include reliance on unofficial baseline implementations (e.g., PerCo) and insufficient clarity on differences from originals.", "relation_to_broader_scientific_literature": "The paper combines diffusion-based generative models (PerCo, DiffEIC), multimodal semantic retrieval (MLLMs), and residual coding into a unified compression framework.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n- combination of existing ideas\n- clearly presented and significant\n\nWeaknesses:\n- high computational cost", "other_comments_or_suggestions": "N/A", "questions_for_authors": "- Given the high computational cost of PFO, can you clarify practical use cases?\n- Could differences between your reproduction and the official PerCo implementation affect your comparisons?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper receives one weak reject, two weak accept and one accept. The authors address most of the reviewers' concerns. I encourage the authors to revise the paper based on the comments from the reviewers.
Decision: Accept (poster)