Lossy Image Compression with Conditional Diffusion Models
Authors: Ruihan Yang
Stephan Mandt
Venue: NeurIPS 2023 poster (2023)
Keywords: generative model
diffusion model
image compression
computer vision

Abstract: This paper outlines an end-to-end optimized lossy image compression framework using diffusion generative models. The approach relies on the transform coding paradigm, where an image is mapped into a latent space for entropy coding and, from there, mapped back to the data space for reconstruction. In contrast to VAE-based neural compression, where the (mean) decoder is a deterministic neural network, our decoder is a conditional diffusion model. Our approach thus introduces an additional "content" latent variable on which the reverse diffusion process is conditioned and uses this variable to store information about the image. The remaining ``texture'' variables characterizing the diffusion process are synthesized at decoding time. We show that the model's performance can be tuned toward perceptual metrics of interest. Our extensive experiments involving multiple datasets and image quality assessment metrics show that our approach yields stronger reported FID scores than the GAN-based model, while also yielding competitive performance with VAE-based models in several distortion metrics. Furthermore, training the diffusion with  $\mathcal{X}$-parameterization enables high-quality reconstructions in only a handful of decoding steps, greatly affecting the model's practicality. Our code is available at: https://github.com/buggyyang/CDC_compression

Lossy Image Compression with Conditional Diffusion Models
Ruihan Yang
Stephan Mandt
Department of Computer Science University of California Irvine ruihan, yang@uci. edu
Department of Computer Science University of California Irvine mandt@uci. Seuci.edu edu
Abstract
This paper outlines an end-to-end optimized lossy image compression framework using diffusion generative models. The approach relies on the transform coding paradigm, where an image is mapped into latent space for entropy coding and, from there, mapped back to the data space for reconstruction. In contrast to VAE- based neural compression, where the (mean) decoder deterministic neural network, our decoder is conditional diffusion model. Our approach thus intro- duces an additional "content" latent variable on which the reverse diffusion process is conditioned and uses this variable store information about the image. The remaining "texture" variables characterizing the diffusion process are synthesized decoding time. We show that the model's performance can tuned toward per- ceptual metrics of interest. Our extensive experiments involving multiple datasets and image quality assessment metrics show that our approach yields stronger ported FID scores than the GAN-based model, while also yielding competitive performance with VAE-based models in several distortion metrics. Furthermore, training the diffusion with X' --arameterizztionn enables high-qualit reconstructions in only handful of decoding steps. greatly affecting the model's practicality. Our code is available at: https: / //gi thub. com/buggyyang CDC CDDD_compressio
Introduction
With visual media vastly dominating consumer internet traffic, developing new efficient codecs for images and videos has become evermore crucial (Ciscol 2017). The past few years have shown considerable progress in deep learning- based image codees that have outperformed clas- sical codecs in terms of the inherent tradeoff' between (expected file size) and distortion (quality loss) (Ballé et al. 2018 Minnen et al.
CaRasiAn 3.haaan
XX-1
2to) Dirup
R1
R
Descding
Figure 1: Overview of proposed compression architecture. A discrete "content" latent variable contains information about the image. Upon de- coding, this variable is used for conditioning de- noising diffusion process. The involved "texture" variables X1:N are synthesized on the fly.
2018 Minnen Minnen & Singh [2200] Zhu al. [202]]
Yang et 2000 et al. |0200 Yang et al.
2023). Recent research promises even more compression gains upon optimizing for percep- tual quality, i.e., increasing the tolerance for imperceivable distortion for the benefit of lower
rates (Blau Michaeli 2019). For example, adding adversarial losses (Agustsson et al. 2019 Mentzer et al. 2020) showed good perceptual quality at low bitrates.
Most state-of-the-art learned codecs currently rely on transform coding and involve hierarchical "compressive" variational autoencoders (Ballé et al. 2018 Minnen et al. 2018 Cheng et al. 2020).
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

These models simultaneously transform the data into a lower dimensional latent space and use learned prior model for entropy-coding the latent representations into short bit strings. Using either Gaussian or Laplacian decoders, these models directly optimize for low MSF/MAE distortion performance. Given the increasing focus on perceptual performance over distortion, and the fact that VAEs suffer from mode averaging behavior inducing blurriness (Zhao et al. 2017) suggest expected performance gains when replacing the Gaussian decoder with a more expressive conditional generative model.
This paper proposes to relax the typical requirement of Gaussian (or Laplacian) decoders in compres- sion setups and presents a more expressive generative model instead: a conditional diffusion model. Diffusion models have achieved remarkable results on high-quality image generation tasks (Ho et al.) 2020; Song et al. (021b]a) a). By hybridizing hierarchical compressive VAEs (Ballé et al. (018) with conditional diffusion models, we create novel deep generative model with promising properties for perceptual image compression. This approach is related to but distinct from the recently proposed Diff-AEs (Preechakul al,, 2022), which are neither variational (as needed for entropy coding) nor tailored to the demands of image compression.
We evaluate our new compression model on four datasets and investigate total of 16 different metrics, ranging from distortion metrics, perceptual reference metrics, and no-reference perceptual metrics. We find that the approach yields the best reported performance in FID and otherwise comparable with the best available compression models while showing more consistent behavior across the different tasks. We also show that making the decoder more stochastic vs. deterministic offers a new possibility to steer the tradeoff between distortion and perceptual quality (Blau & Michaeli, 2019). Crucially, we find that a certain parameterization-2 --prediction (Salimans & Ho 2022) can yield high-quality reconstructions in only handfull of diffusion steps.
In sum, our contributions are as follows:
We propose novel transform-coding-eee models. The approach uses an encoder to map images onto contextual latent variable; this latent variable is then fed as context into diffusion model for reconstructing the data. The approach can be modified to enhance several perceptual metrics of interest.
lossy compression scheme using diffusion
We derive our model's 's loss function from a variational upper bound to the diffusion model's implicit dit-oiitootion function. The resulting distortion term is distinct from traditional VABS in capturing a richer decoding distribution. Moreover, it achieves high-quality recon- structions in only a handful of denoising steps.
We provide substantial empirical evidence that variant of our approach is, in many cases, better than the GAN-based models in terms of perceptual quality, such as FID. Our base model also shows comparable rate-distortion performance with MSE-optimized baselines. To this end, we considered four test sets, three baseline models (Wang et al.) 2022 Mentzer et al.
2020 Cheng et al. 2020), and up to sixteen image quality assessment metrics.
2 Related Work
Tradsform-coding Lossy Image Compression
The sridely-established classical codecs such as
JPEG (Wallace 1991), BPG (Bellard, 2018). WEBP (Google, 2022) have recently been challenged
by end-to-end learned codecs (Ballé et al. [2018] Minnen al. 2018 Minnen Singh (2200) Yang et al. 2020) Cheng et al., 2020 Zhu et al. 2021 Wang et al. 2022 He et al. (022a). These methods typically draw on the non-linear transform coding paradigm as realized by hierarchical VAEs. Usually, neural codecs are optimized to simultancously minimize rate and distortion metrics, such as mean squared error or structural similarity.
In contrast to neural compression approaches targeting traditional metrics, some recent works have explored compression models enhance realism (Agustsson et al., 2019; Mentzer et al.. 2020) T'schannen et al. 2018 Agustsson et al. 2022] He et al.) 2022b). A theoretical background for these approaches was provided by Blau & Michaeli (2019); Zhang et al. (202T); ; in particular Blau & Michaeli (2019) discussed the existence of a fundamental tradeoff between distortion and realism, i.e., both goals can not be achieved at the same time in given architecture. To realize this tradeoff
in compression, Mentzer al. (2020): Agustsson al. (2022) optimized the nutoencoder-basee

---

compression model with an additional adversarial loss, while GAN training in this context can be unstable and requires variety of design choices.
Diffussio Models Probabilistic diffusion models showed impressive performance on image genera-
tion tasks, with perceptual qualities comparable to those of highly-tuned GANs while maintaining
stable training (Song & Ermon, 2019; Ho et al. 2020; Song et al. 2021b
Song & Ermon 2019
Kingma et al., 2021] Yang et al. 2022 Ho et al. 2022rSaharia Saharia et al. 2022 Preechakul et al. 2022).
Popular recent diffusion models include Dall-E2 (Ramesh et al. 2022) and Stable-Diffusieg (Rom
bach et al. 2022). Some works also proposed diffusion models for compression. Hoogeboom et al.
(2021) evaluated an autoregressive diffusion model (ADM) on a lossless compression task. Besides the difference between lossy and lossless compression, the model is only tested on low-resolution CIFAR-10 (Krizhevsky et al. 2009) dataset. Theis et al. (2022) used generic unconditional diffusion model to lossily communicate Gaussian samples. Their method leverages "reverse channel coding", which is orthogonal to the usual transform-codin
paradigm. While it's under active research, there is currently no practical method that can reduce its extensive computational cost without restrictive Theis & Ahmed 2022).
limitation (Li & El Gamal 2018; Flamich et al., 2020 2022
3 Method
3.1 Background
Denoising diffusion models are hierarchical latent variable models that generate data by sequence of iterative stochastic denoising steps (Sohl-Dickstein et al. 2015) Ho et al. 2020; Song et al.) 202 Song & Ermon 2019). These models describe joint distribution over data Xy and latent variables
X1:N such that po(xx) (xo) = po (Xu:N )dx1:N. While a diffusion process (denoted by q) incrementally
destroys structure, its reverse process pe generates structure. Both processes involve Markovian
dynamics between a sequence of transitional steps (denoted by n)), where
[(x,/xnn11) = (xx,, 1
x,, 1, 3,II);
(1)
Po p0(xn-1(xx,) (x Mo Mo n), I).
The variance schedule 3, E 1) can be either fixed or learned; besides it, the diffusion process is parameter-frre. The denoising process predicts the posterior mean from the diffusion process and is parameterized by neural network Mx 12).
Denoising Diffusion Probabilistic Model (DDPM) (Ho et al. 2020) showed tractable objective function for training the reverse process. A simplified version of their objective resulted in the following noise parameterization where one seeks to predict the noise used to perturb particular image Xy from the noisy image X, at noise level nt:
L(O, xp) EE,elle Eo (x, (X)),
Above, n ~ Unif(1, [J. e N At test time, data can be generated by ancestral sampling using Langevin dynamics. Alternatively, Song et al. (202Ia) proposed the Denoising Diffusion Implicit Model (DDIM) that follows a de- terministic generation procedure after an initial stochastic draw from the prior. Our paper uses the DDIM scheme at test time, see Section 3.2 for details.
x,, (X0) ) = v Xo + vI E, and a, = -I 3.).
Neural image compression seeks outperform traditional image codecs by machine-learned models. Our approach draws on the transform cooenn--oiig--- based neural image compression approach (Theis et al, 2017] Ballé et al. 2018 Minnen al. 2018), where the data are non-linearly transformed into a latent space, and subsequently discretized and entropy-coded under a learned "prior". The approach shows a strong formal resemblance to VAES and shall be reviewed in this terminology.
Let z be a continuous latent variable and à = |z] the corresponding rounded, integer vector. The VAE-based compression approach consists of stochastic encoder e(z]x), prior p(z), and a decoder p(x|z). The model is trained using the negative modified ELBO,
(() x) D XR Ex-e(z)
|- log ( X p())).
(3)
The first term measures the average reconstruction of the data (distortion D, usually measured by MSE), while the second term measures the costs of entropy coding the latent variables under the prior
3

---

Algorithm Training the model (left); Encoding/Decoddig data Xo (right). X' X-prediction model.
Sample Xq dataset
Given Nust
repeat
2 Ency (xo)]
U(0, 2, Nirai)
2 (())
binary file (entropy code using P(2))
(0.11)
x = (or XN ~ 721) for stochastic
Xn X0nx00 + vI One
decoding)
2 Enca(Xo) (Xo) ( (00,5 0.5)
for n= Nist jest to do
X Xe Xø(Xn, n n/Nurain 2)
x, (xx,,
LD = I-a. X0= Xo
--a.,
L (1 p) LD + pdl &o, xo) X logg P())
Xy Xo n/N 2)
x, v von-1oo + v1 Onn
(0, Φ) (0, Φ) - 8 Vo (learning rate: 5)
end for return X0
until converge
(bitrate R). The encoder e(z x) 21 (Enns (x) - 1. Enc, (x) 1) is boxed-shaped distribution that simulates rounding at training time through noise injection due to the reparameterization trick. Note its differential entropy equals zero.
Once the VAE is trained, we en/decode data using the deterministic components as 2 = |Enc(x)
and & Dec(ž). We convert the continuous p(z) into discrete P(ž) using P(2) CDF
0.5) CDF 0.5), where CDF, is the cumulative distribution function of p(z) (z): ; see (Yang et al.
2023
[Sec. 2.1.6] for details. The discrete prior is used for entropy coding & (Ballé al. 20018).
While VAE-based approaches have used simplistic (e.g., Gaussian) decoders, we show that can get significantly better results when defining the decoder p(x|z) as a conditional diffusion model.
3.2
Conditional Diffusion Model for Compression
The basis of our compression approach is a new latent variable model: the diffusion variational autoencoder. This model has "semantic" latent variable for encoding the image content, and set of "texture" variables XI:N describing residual information,
p(Xo:N: p(x /z)30)./)))(()))) zz)(z))
As detailed below, the decoder will follow a denoising process conditioned on z. Drawing methods described in Section we use a neural encoder c(z)x,) encode the image. The prior p(z) is two-level hierarchical prior (commonly used in learned image compression) and is used for entropy coding z after quantization (Ballé al. 2018). Next, we discuss the novel decoder model.
Decoder and training objective We construct the conditional denoising diffusion model in similar way to the nor-variational diffusion autoencoder of PPrechhakll et al. (2022): WWang al.] (2023). In analogy to Eq. [. we introduce a conditional denoising diffusion process for decoding the latent 2,
P8 = p(xx) II Po(xx-11Xx Xn., Sx.21 z) = p(xx) II ((xx-1lMo(aa) Z, n), P,,I).
(5)
Since the texture latent variables XI.N are compressed but synthesized decoding time, the optimal encoder and prior should be learned jointly with the decoder's 's marginal likelihood p(x, Iz) = [((Xo.N [z)xxI.N while targeting certain tradeoff between rate and distortion specified by Lagrange parameter A. We can upper-bound this rate-distortion (R-D) objective by invoking Jensen's 's inequality,
E E log p(xolz) - pog) S (x))))))x) [Tupper (Xolz) (z) Alogp(z))
where we introduced Lupper (xx) (z) - Exx.N- xo) log g(x) x Xo)
p(xo. (x)
as the variational upper
bound to the diffusion model's negative data log likelihood (Ho et al. 2020). We realize that
Lupper (xpper(Xolz)) corresponds to novel image distortion metric induced by the conditional diffusion
model (in analogy to how a Gaussian decoder induces the MSE distortion). This term measures the model's s ability to reconstruct the image based on z. In contrast, log p(x) measures the number of bits needed compress under the prior. As in most other works neural image compression (Ballé

---

et al. 2018
Minnen et al. 2018 Yang et al. 2023), we use box-shaped stochastic encoder e( e(z)X))
that simulates rounding by noise injection at training time.
In analogy to Eq. 2 we simplify the training objective by using the denoising score matching loss,
Lupper (xo z)) Exp,n.ellc (o(xxx 2, Nuain
111 Exg,n. 1 Otp
Onn
||x0
Xo (xxx Z,
I? (6)
Nuain
The noise level n and on are defined in Eq. 2 Instead of conditioning on n, we condition the
model the pseudo-continuous variable Nmm which offers additional flexibility in choosing the number of denoising steps for decoding (e.g..
use a Nust smaller than Nuain). The right- hand-side equation describes an alternative form of the loss function, where Xg directly learns to reconstruct Xn instead of (Salimans & Ho 2022). We easily derive the equivalence with
co (X,, 2, +))
Decoding process Once the model is trained, we entropy-decode using the prior p(z) and conditionally decode the image Xo using ancestral sampling. We consider two decoding schemes: a stochastic one with XN ~ NN 7² (where 7 >> 0) and deterministic version with x. 0 (or 7 = 0), both following the DDIM sampling method:
(oo,, 2, N) + aI-
(7)
Since the variables X1:N are not stored but generated at test time, these "texture" variables can result in variable reconstructions upon stochastic decoding (see Figure for decoding with different 7). Algorithm summarizes training and encoding/decoding.
Fast decoding using X' X'-prediction In most applications of diffusion models, the iterative generative process is major roadblock towards fast generation. Although various methods have been proposed to reduce the number of iterations, they often require additional post-processing methods, such as progress distillation (Salimans & Ho 2022) and parallel denoising (Zheng et al. 2022).
Surprisingly. in our use case of diffusion models, we found that the X' X'prrddctionn model with only a handfull of decoding steps achieves comparable compression performance to the e-model with hundreds of steps, without the need of any post-processing. This can be explained by closely inspecting --prediction objective from Eq. (6) that almost looks like an autoencoder loss, with the modification that n and X,, are given as additional inputs. When is large, x,, looks like pure noise and doesn't contain much information about Xp: this case, will ignore this input and reconstruct the data based on the content latent variable In contrast, if is small, X, will closely resemble xo and hence carry more information than z to reconstruct the image. This is to say that our diffusion objective inherits the properties of an autoencoder to reconstruct the data in single iteration; however, successive decoding allows the model to refine this estimate and arrive at a reconstruction closer to the data manifold, with beneficial effects for perceptual properties.
Optional Perceptual Loss While Eq. already describes viable loss function for our conditional diffusion compression model, we can influence the perceptual quality of the compressed images by introducing additional loss functions similar to (Mentzer et al.) 2020).
First, we note that the decoded data point can be understood as a function of the low-level latent
X,, the latent code z, and the iteration n, such that Xo (x,; 2, n//) or Xx aa- (x/
Va.
When minimizing a perceptual metric d(-s)). :), we can therefore add a new term to the loss:
X
Lp E, (z [d( (xo, Xa) and Le Ex=e(z//xo) Lupper (Xulz) \ 1-p logp(z))
(8)
L == pLp + Le-
(9)
This loss term is weighted by an additional Lagrange multiplier p E 00, 1), resulting in a three-way tradeoff between rate, distortion, and perceptual quality (Yang et al. 2023).
We stress that different variations of perceptual losses for compression have been proposed Yang et al. 2023). While this paper uses the widely-adopter LPIPS loss (Zhang et al.) 2018). other approaches add an adversarial term that seek to make the reconstructions indistinguishable from reconstructed images. In this setup, Blau & Michaeli (2019) have proven mathematically that it is impossible to simultaneously obtain abitrarily good perceptual qualities and low distortions. this paper, we observe similar fundamental tradeoff between perception and distortion.
5

---

HFR M5 M55-hpprr
DGML
NSC
CDC 03 (Oun)
BPG
CDC
HDD
LPIPS.
H-IMT
MS-SSIMF
4..5
200
30-
0.07
D:51
0.00
0:91
012
0.0
1.5
0.2
644
1/8
022
1.4
00.
01.5
0.2
11.
0.6
0.5
PIFAPR]
DISTS]
SSIMT
PSNR1
11:90
1.15
0,00
1.00
0.75
0.00
33
0.50
0.00
0.4
0.5
152
015
0.55
0.8
Figure 2: Tradeoffs between bitrate (X-axes, in bpp) and different metrics (y-axes) for various models tested on DIV2K. We consider both perceptual (red frames) and distortion metrics (blue framess. Arrows in the plot titles indicate whether high (?) low (1) values indicate better score. CDC (proposed) its basic version (deterministic, without finetuning to LPIPS) compares favorably in distortion metrics, while CDC with stochastic decoding and added LPIPS losses performs favorably on perceptual metrics.
Experiments
We conducted large-scale compression evaluation involving multiple image quality metrics and test datasets, Besides metrics measuring the traditional distortion scores, also consider metrics that can reflect perceptual quality. While some of these metrics are fixed, others are learned from data. We will refer to our approach "Conditional Diffusion Compression" (CDC).
Metries We selected sixteen metrics for image quality evaluations, of which we present eight most widely-used ones in the main paper and the remaining eight in the appendix. Specifically, several more recently proposed learned metrics (Heusel er al. 2017 Zhang et al., 2018 Prashnani al. 2018 Ding al. 2020) capture perceptual properties/realism better than other non-learned methods; we denote these metrics as perceptual metrics and the others as distortion metrics. It is important to note that when calculating FID, we follow Mentzer al. (2020) by segmenting images into non-overlapping patches of 256 x 256 resolution. A brief introduction about the metrics is also available in Appendix
Test Data To support compression quality assessment, consider the following datasets with necessary preprocessing: 1. Kodak (Franzen] 2013): The dataset consists of 24 high-quality
images at 768 x 512 resolution. 2. Teenick (Asuni & Giachetti 2014): We use 100 natural images with 600 x 600 resolutions and then downsample these images to 512 x 512 resolution. 3. DIV2K (Agustsson & Timofte 2017): The validation of this dataset contains 100 high-quality images. We resize the images with the shorter dimension being equal to 768px. Then, each image is center-cropped 768 768 squared shape. 4. COC02017 (Lin al. 2014]: For this dataset, extract all test images with resolutions higher than 512 x 512 and resize them to 384 x 384 resolution remove compression artifacts. The resulting dataset consists of 2695 images.
Model Training Our model was trained using the well-established Vimeo-90k dataset (Xue et al. 2019), which includes 90,000 video clips, each containing 7-frame sequences with a resolution of 448 x 256 pixels, curated from vimeo.com. For each iteration, we randomly selected one frame from every video clip, which was then subject to random cropping achieve uniform resolution of 256 x 256 pixels.

---

(a) Ground Truth
(b) CDC(p 0.9) (bpp=0.205)
(c) HiFiC (bpp= 207)
(d) Ground Truth
(e) CDC(P 0.9) (bpp=0.398)
(1) HiFiC (bpp=0.456)
Figure 3: Reconstructed Kodak images (cropped images, see full images in Appendix F). 1st row: compared to HiFiC under similar bitrate, our model retains more details around the eyes of the parrot. 2nd row: our model still gets slightly better visual reconstruction than HiFiC while using less bitrate.
The training procedure initiated with a warm-up phase, setting 10 and running the model for approximately 700,000 steps. Subsequently, we increased À to align with the desired bitrates and continued training for an additional 1, 1,000,000 steps until the model reached convergence.
For the c-prediction model, our training utilized diffusion process comprising Nurai = 20, 000 steps. Conversely, the number of diffusion steps for the X-prediction model is Nusin 8, 193. We implemented a linear variance schedule to optimize the e-prediction model, while cosine schedule selected for the X'prediction model optimization. Throughout the training regime, we maintained a batch size of 4. The Adam optimizer (Kingma & Bal (014)) was employed to facilitate efficient convergence. We commenced with an initial learning rate of Ir 5 10-5 which was reduced by 20% after every 100,000 steps, ultimately clipped to a learning rate of Ir = 2 x 10-5.
4.1 Baseline Comparisons
Baselines and Model Variants We showed two variants of our X'-rredictoon CDC model. Our first proposed model optimized in the presence of an additive perceptual reconstruction term at = 0.9, which is the largest p-value chose. For this variant, we used xx ~ (O, >²1) with 0.8 to reconstruct the images. The other proposed version is the base model, trained without the additional perceptual term (p 0) and using a deterministic decoding with xx = 0. As discussed below, this base version performs better in terms of distortion metrics, while the stochastic and LPIPS-informed version performs better in perceptual metrics.
We compare our method with several neural compression methods. The best reported perceptual results were obtained by HiFiC (Mentzer et al., 2020). This model is optimized by adversarial network and employs additional perceptual and traditional distortion losses (LPIPS and MSE). In terms of rate-distortion performance, two VAE models are selected: DGML (Cheng et al. (020) and NSC (Wang et al. 2022). Both are the improvements over the MSE-trained Mcan-Scale Hyperprior (MS-Hyper) architecture (Minnen et al. 2018). For a fair comparison, did not the content- adaptive encoding for NSC model. For comparisons with classical codecs, we choose BPG as a reference.
Figure 2 illustrates the tradeoff between bitrate and image quality using the DIV2K dataset. We only employ 17 steps to decode the images with X' d'-prediction model, which is much more efficient than e-prediction model that requires hundreds of steps to achieve comparable performance (see Appendix H for results on other datasets and comparison with e-prediction). In the figure, dashed lines represent the baseline models, while solid lines depict our proposed CDC models. The eight shown plots present two different types of metrics, distinguished by their respective frame colors.
7

---

1.stpp
5 Rays
17 a0ys
staps
65 3:35
F10L
LPIPS.
FSIM
SSIM?
11:30
0.14
0L050
0.080
0.
0.080
0.915
0.5
3.3
0.4
0.5
0.3
0.4
0.3
0.6
SSIM1
11.5
0.94
32.
11.00
0.50
115
011
0.50
Figure 4: Compression performance with different numbers of decoding step. We use 7
(deterministic decoding) to plot distortion curves and for perceptual quality curves.
Perceptual Metrics (red). The red subplots depict perceptual metrics that assess compression for realism performance. Our findings reveal that our proposed CDC model (p 0.9) achieves the best performance in three out of four metrics. The closest competitor HiFiC baseline. Notably, HiFiC demonstrates the highest score in LPIPS but exhibits suboptimal performance in all other metrics.
Distortion Metrics (blue). The blue subfigures present distortion-basee metrics. We note that the CDC model with 0 produces relatively favorable results in distortion metrics, excluding PSNR. shows on-par scores with the best baselines FSIM, SSIM, and MS-SSIM scores, despite none of the shown models being specifically optimized for these three metrics. In contrast, "classical" neural compression models (Minnen et al. 2018 Wang et al. 2022, Cheng et al. 2020) directly target MSE distortion by minimizing an ELBO objective with Gaussian decoders, resulting in better PSNR scores.
Our proposed versions CDC (p=0) and CDC (p=0.9) show qualitative differences in perceptual and distortion metrics. Setting p 0 only optimizes a trade-off between bitrate and the diffusion loss; compared to p = 0.9, this results in better performance in model-based distortion metrics (i.c., except PSNR). Fig. qualitativvey shows that the resulting decoded images show fewer over-smoothing artifacts than VAE-based codecs (Cheng et al., (020). In contrast, CDC (p=0.9) performs the best in perceptual metrics. These often based extracted neural network features, such Inception or VGG (Szegedy et al.] 2016 (Simonyan & Zisserman 2014), and are more susceptible to image realism. By varying p. we can hence control a three-way trade-off among distortion, perceptual quality, and bitrate (See Appendix for results with other values).
Distortion vs. Perception Our experiments revealed the aforementioned distortion--perceptionn tradcoff in learned compression [Blau& & Michaeli 2019). In contrast perceptual metrics, distortions such as PSNR are very sensitive to imperceptible image translations (Wang et al. (205)). The benefit of distortions is that they carry out direct comparison between the reconstructed and original image, albeit using a debatable metric (Dosovitskiy & Brox 2016). The question of whether distortion or perceptual quality is more relevant may ultimately not easily solvable; yet plausible that most compression gains can be expected when targeting combination of perception/realism and distortion, rather than distortion alone (Mentzer et al.) 2020 Yang al. 2023). Especially, our method's s strong performance in terms of FID, one of the most widely-adopted perceptual evaluation schemes (Ho
al. 2020 Song & Ermon 2019 Mentzer et 2020 Brock al. 2019 Song al. 202 b).
seems promising in this regard.

---

(a) Ground Truth (b) y = MS MS-SSIM-0,977 LPIPS=0.064
Y =
(d) Y = 0.8
( MS MSSSSIM==,964 LPIPS=0.058
MS-SSIM-0.979 L.PIP==0.078
MS- MSS-S91M0 L.PIP=00.054
Figure 5: Qualitative comparison of deterministic and stochastic decoding methods. Deterministic decoding typically results in smoother image reconstruction. By increasing the noise used upon decoding the images, we observe more and more detail and rugged texture on the face of the sculpture. (7 0.8) show the best agreement with the ground truth image. All the images share the same bpp.
4.2 Ablation Studies
Influence decoding steps In the previous section, we demonstrated that the CDC X' X-prediction model can achieve decent performance with a small number of decoding steps. In Figure 4 we further investigate the compression performance of the X' X-prediction(p decoding steps. Our findings reveal that when employing stochastic decoding, the model consistently produces better perceptual results as the number of decoding steps increases. However, in the case of deterministic decoding, more decoding steps do not lead to a substantial improvement in distortion.
0) model using different
Our findings show that the X'pprediction model can behave similarly to a Gaussian VAE decoder. In this scenario, the latent code becomes the primary determinant of the decoding outcome, enabling the model reconstruct the original image X9 with single decoding step. However, even a single decoding step has a tendency to reconstruct data closer to the data mode, only guaranteeing an acceptable distortion score. To effectively improve perceptual quality, it is crucial to incorporate more iterative decoding steps, particularly when utilizing stochastic decoding. Thus, we further explore the impact stochastic decoding through the following ablation experiment.
Stochastic Decoding By adjusting the noise level parameter, denoted as 7, during the image decoding process, we can achieve different decoding outcomes. In order to investigate the impact of the noise on the decoding results, we present Figure 5 which provides both quantitative and qualitative evidence for 4 candidates y values. Our findings indicate that larger values lead to improved perceptual quality and higher distortion, as evidenced by lower LPIPS and lower MS- SSIM. Values of greater than 0.8 not only increase distortion but also diminish perceptual quality. In terms of finding the optimal balance, a ? value of 0.8 offers the lowest LPIPS and the best qualitative outcomes shown in Figure 5 From qualitative standpoint, introduces plausible high-frequency textures. Although these textures may not perfectly match the uncompressed ones (which impossible), they are visually appealing when appropriate ? is chosen. For additional insights into our decoding process, we provide visualizations of the decoding steps in Appendix G These visualizations showcase how "texture" variables evolve during decoding. We observe that this phenomenon becomes particularly pronounced when employing the e-prediction, as X' pprediction models exhibit a stronger inclination towards reconstruction.
notice that the noise
5 Conclusion & Discussion
We proposed transform- cereffrrm-ooiing.... based neural image compression approach using diffusion models. use denoising decoder to iteratively reconstruct a compressed image encoded by an ordinary neural encoder. Our loss term is derived from first principles and combines rate-distortion variational autoencoders with denoising diffusion models. We conduct quantitative and qualitative experiments to compare our method against several GAN and VAE based neural codecs. Our approach achieves

---

promising results in terms of the rate- rate-perception tradeoff, outperforming GAN baselines in three out of four metrics, including FID. In terms of classical distortion, our approach still performs comparable to highly-competitive baselines.
Limitations & Societal Impacts In our quest to enhance compression performance, further im- provement can be achieved by integrating advanced techniques such as autoregressive entropy models or iterative encoding. We leave such studies for future research.
One critical societal concern associated with neural compression, particularly when prioritizing perceptual quality, is the risk of misrepresenting data in the low bitrate regime. Within this context, there is a possibility that the model may generate hallucinated lower-level details. For example, compressing human face may lead to a misrepresentation of the person's identity, which raises important considerations regarding fairness and trustworthiness in learned compression.
6
Acknowledgement
The authors acknowledge support by the IARPA WRIVA program, the National Science Foundation (NSF) under the NSF CAREER Award 2047418; NSF Grants 2003237 and 2007719, the Department of Energy, Office Science under grant DE-SC0022331, as well as gifts from Intel, Disney, and Qualcomm.
References
Agustsson, E. and Timofte, R. Ntire 2017 challenge single image super-resolution: Dataset and study. The IEEE Conference on Computer Vision and Puttern Recognition (CVPR) Workshops, July 2017.
Agustsson, E., Tschannen, M., Mentzer, F., Timofte, R., and Gool, L. V. Generative adversarial networks for extreme learned image compression. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 221-231, 2019.
Agustsson, E., Minnen, D., Toderici, G., and Mentzer, F. Multi-realism image compression with conditional generator. arXiv preprint 2rXiv:22/2.13824 2022.
Asuni, N. and Giachetti, A. Testimages: large-scale archive for testing visual devices and basic image processing algorithms, stag smart tools & apps for graphics conference, 2014., 2014.
Ballé, J., Minnen, D., Singh, S., Hwang, S. J., and Johnston, N. Variational image compression with scale hyperprior. International Conference Learning Representations, 2018.
Bégaint, J., Racapé, F., Feltman, S., and Pushparaja, Compressai: a pytorch library and evaluation platform for end-to-end compression research. arXiv preprint arXiv:2011.03002 2020.
Bellard, F. Bpg image format, 2018. UtL/tttps://elllardddo org/bp//
Blau, Y. and Michaeli, T. Rethinking lossy compression: The aate-distortion-peneppti tradeoff. In International Conference on Machine Learning, pp. 675-685. PMLR, 2019.
Brock, A., Donahue, J.. and Simonyan, K. Large scale gan training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019.
Cheng, Z. Sun, H., Takeuchi, M., and Katto, Learned image compression with discretized gaussian mixture likelihoods and attention modules. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1939-7948, 2020.
Cisco. Visual network index cisco. forecast and methodology. White Paper, 2017.
Ding, K., Ma. K., Wang. S., and Simoncelli, E. P. Image quality assessment: Unifying structure and texture similarity. IEEE transactions on pattern analysis and machine intelligence, 2020.
Dosovitskiy, A. and Brox, Generating images with perceptual similarity metries based on deep networks. Advances in neural information processing systems, 29, 2016.
10

---

Flamich, G., Havasi, M., and Heránndez-obaad J. M. Compressing images by encoding their latent representations with relative entropy coding. Advances in Neural Information Processing Systems, 33:16131-16141, 2020.
Flamich, G., Markou, S., and Hernández-Lobato, M. Fast relative entropy coding with a coding. arXiv preprint arXiv:2201.1287) 2022.
Franzen, R. W. True color kodak images, 2013. URL htt.:tpp//zokk.sssgaaaacc//////p
Google, An image format for the web; webp; google developers, 2022. URL/ltttps:/ddeeelppps
google. cte.gll...mmeeeeee caomm/pped//ee /tepp////eb/////
He, D., Yang, Z., Peng. W., Ma, R., Qin, H., and Wang. Y. Elic:
Efficient learned image compression with unevenly grouped phanelchannel contextual adaptive coding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5718-5727, 2022a.
He, D., Yang, Z., Yu, H., Xu, T., Luo, J., Chen, Y., Gao, C., Shi, X., Qin, H., and Wang, Y. Po-elic: perieption-orienneed efficient learned image coding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1764-1769, 2022b.
He, K., Zhang, X., Ren, S., and Sun, Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Ho, J., Jain, A., and Abbeel, Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Ho, J., Salimans, T., Gritsenko, A., Chan, W. Norouzi, M., and Fleet, D. J. Video diffusion models. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances Neural Information Processing Systems, volume 35, pp. 8633-8646. Curran As- neurips cc/paper_pppc.. ff Latles/pppe//222.. sociates, Inc., 2022. /ttps Paper ComfperrrCooeeeeeee... pdf
Hoogeboom, E., Gritsenko, A. A., Bastings, J., Poole, B., van den Berg. R., and Salimans, T. Autoregressive diffusion models. In International Conference on Learning Representations, 2021.
Kingma, D., Salimans, T., Poole, B., and Ho, J. Variational diffusion models. Advances in neural information processing systems, 34:21696-21700 2021.
Kingma, D. P. and Ba, J. arXiv:1412.69880 2014.
Adam: method for stochastic optimization. arXiv preprint
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images, 2009.
Kuznetsova, A., Rom, H., Alldrin, N., Uijlings, J., Krasin, I., Pont- Pont-Tuset, J., Kamali, S., Popov, S., Malloci, M., Kolesnikov, A., Duerig, T., and Ferrari, V. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. I/CV, 2020.
Li, C. T. and Gamal, A. Strong functional representation lemma and applications to coding theorems. IEEE Transactions on Information Theory, 64(11):6967-69) 2018
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L.. Microsoft coco: Common objects context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Mentzer, F., Toderici, G. D., Tschannen, M., and Agustsson, E. High-fidelity generative image compression. Advances in Neural Information Processing Systems, 33:1 1913-119244 2020.
Minnen, D. and Singh, S. Channel-wise autoregressive entropy models for learned image compression. In 2020 IEEE International Conference Image Processing (ICIP), pp. 3339-3343. IEEE, 2020.

---

Minnen, D., Ballé, J., and Toderici, G. D. Joint autoregressive and hierarchical priors for learned image compression. Advances in neural information processing systems, 31, 2018.
Prashnani, E., Cai, H., Mostofi, Y., and Sen, P. Pieapp: Perceptual image-error assessment through pairwise preference. In Proceedings of the IEEE Conference on Computer Vision and Putlern Recognition, pp. 1808- 1817, 2018.
Preechakul, K., Chatthee, N., Wizadwongsa, S., and Suwajanakor,, S. Diffusion autoencoders: Toward meaningful and decodable representation. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2022.
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06/25, 2022.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 10684-10695, 2022.
Saharia, C., Ho, J., Chan, W., Salimans, T., Fleet, D. J., and Norouzi, M. Image super-resolution via iterative refinement, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Salimans, T. and Ho, J. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations, 2022.
Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409.15660, 2014.
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pp. 2256-2265 PMLR, 2015.
Song. J., Meng, C., and Ermon, S. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021a.
Song, Y. and Ermon, S. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
Song. Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. International Conference on Learning Representations, 2021b.
Szegedy, C., Vanhoucke, V., loffe, S., Shlens, J., and Wojna, Z. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-284 28262 2016.
Theis, L. and Ahmed, N. Y. Algorithms for the communication of samples. International Conference Machine Learning, pp. 21308-21328, PMLR, 2022.
Theis, L., Shi, W., Cunningham, A., and Huszár, F. Lossy image compression with compressive autoencoders. International Conference on Learning Representations, 2017.
Theis, L., Salimans. T., Hoffman, M. D., and Mentzer, F. Lossy compression with gaussian diffusion. arXiv preprint arXiv:2206.0888) 2022.
Tschannen, M., Agustsson, E., and Lucic, M. Deep generative models for distributtonn-prserving
lossy compression. Advances in neural information processing systems, 31, 2018.
Wallace, G. K. The jpeg still picture compression standard. Communications of the ACM, 34(4): 30-44, 1991.
Wang. D., Yang. W., Hu, Y., and Liu, J. Neural data-dependent transform for learned image compression. Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 13379-17388, 2022.
12

---

Wang. Y. Schiff, Y., Gokaslan, A., Pan, W., Wang, F., De Sa, C.. and Kuleshov, V. Infodiffu- sion: Representation learning using information maximizing diffusion models. arXiv preprint arXiv:2306.08837, 2023.
Wang, Z., Bovik, A. C.,, and Simoncelli, E. P. Structural approaches to image quality assessment. Handbook of image and video processing, 7(18), 2005.
Xue, T., Chen, B., Wu, J., Wei, D., and Freeman, W. T. Video enhancement with task-oriented flow. International Journal of Computer Vision (ICV),, 127(8))1106- 1125, 2019.
Yang, R., Srivastava, P., and Mandt, S. Diffusion probabillistic modeling for video generation. arXiv preprint arXiv:2203. 09481, 2022.
Yang, Y., Bamler, R., and Mandt, S. Improving inference for neural image compression. Advances in Neural Information Processing Systems, 33:573-584, 2020.
Yang, Y., Mandt, S., and Theis, L. An introduction to neural data compression. Foundations and Trends in Computer Graphics and Vision, 2023.
Zhang, G., Qian, J., Chen, J., and Khisti, A. Universal www.-iitootion-pereeeteeee representations for lossy compression. Advances in Neural Information Processing Systems, 34: 34:11517-11529, 2021.
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., and Wang, O. The unreasonable effectiveness of deep features as a perceptual metric. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
Zhao, S., Song, J., and Ermon, S. Towards deeper understanding of variational autoencoding models. arXiv preprint arXiv: /702.08658 2017.
Zheng. H., Nie, W., Vahdat, A., Azizzadenesheli, K., and Anandkumar, Fast sampling of diffusion models via operator learning. arXiv preprint arXiv:2211.13449 2022.
Zhu, Y., Yang, Y., and Cohen, T. Transformer-based transform coding. In International Conference on Learning Representations, 2021.
13

---

A Pretrained Baselines
We refer to Bégaint (2020) for pretrained MS-Hyper and DGML models. For HiFiC model, we use the pretrained models implemented in the publicly available repositories" Both models were sufficiently trained natural image datasets (Xue al. 2019) Kuznetsova al. 2020 For NSC (Wang et al. 2022) baseline, we use the official codebase and DIV2K training dataset to train the model.
B Architectures
%
Liver
Unier
Liine
Loew
Lnier
Line
Liven
Liver
Lioe
Looe
Loww
Lower
EMD
EMD
EMD
EMD
ENC
ENG
ENG
ENC
3x TransCorw 12
p(y'
3 Conw 12
Hypor Prior iBen MB Hypu (Mrres zine modet
ResBlock
ResBlock
ResBlock
ResBlodk
UncarAttertion
Uneantttenton
TransCorw
Embedder (EMC)
Einooder (ENC)
Downssmpling Unit
Upsampinng Unt (UU)
(DU)
Figure 6: Visualization of our model architecture
The denoising module's s design adopts a U-Net architecture similar to that of the DDIM (Song et al., 202 and DDPM (Ho et al. 2020) projects. Each U-Net unit is composed two ResNet blocks (He et al. 2016), an attention block, and a convolutional up/downsampling block. We employ six U-Net units for both the downsampling and upsampling processes. the downsampling units, the channel dimension is 64 j, where represents the index of the layer ranging from to 6. The upsampling units follow the reverse order. Each encoder module comprises a ResNet block and a convolutional downsampling block. For embedding conditioning, we employ ResNet blocks and transposed convolutions to upscale z to match the spatial dimension of the inputs of the initial four U-Net downsampling units. This allows the embedder with the input of the corresponding U-Net unit.
perform conditioning by concatenating the output of
Figure 6 also describes our design choice of the model. We list the additional detailed specifications that we did not clarify in the main paper as follow:
The hyper prior structure shares the same design Minnen et al. 20018 The channel
number of the hyper latent y is set as 256.
http:://ggihah.oo//uutttttnnne www///stin bttps///gttuub..om/Deeeee wwwpe///wwwooooooaa//eeaaaaaaaa Wang/Neural Syntax-Cottxxaaa Code
14

---

We use 3x3 convolution for most of the convolutional layers. The only exceptions are the 1st conv-layer of the first DU component and the 1st layer of the 1st ENC component, where we use 7x7 convolution for wider receptive field.
IN is embedded by a linear layer, which expand the 1-dimensional scalar to the same channel size as the corresponding DU/UU units. We then add the expanded tensor to the intermediate ResBlock of each DU/UU unit.
C Model Efficiency
We provide information on the model parameter size of the proposed model and baselines, and the corresponding time cost of running a full forward pass in T We run benchmarking on server with a RTX A6000. We decode 24 images from Kodak dataset and calculate the average neural decoding time, which does not include entropy-coding process, all the model share similar entropy model.
CDC (1 step)
CDC (17 steps)
HiFiC
MS-hyper 17.5M
DGML
Number of Parameters Decoding Time (Seconds)
53.8M
53.8M
181.47M
26.5M
0.015
1.04
0.0051
0.0011
0.0025
Table 1: Model and decoding time.
Our model exhibits superior memory efficiency compared to HiFiC. However, diffusion models suffer from slow decoding speed owing to their iterative denoising process. In the benchmark model utilized in our main paper, the decoding of an image takes approximately second. Although this is slower than the baselines, it remains within an acceptable time range. Further optimization of the neural network module, such as the removal of the attention module, holds the potential to enhance efficiency even further.
D Additional explanation on experiment metrics
FID, as the most popular metric for evaluating the realism of images, measures the divergence (Fréchet Distance) between the statistical distributions of compressed image latent features and ground truth ones. The model extracts features from Inception network and calculates the latent features' corresponding mean and covariance. LPIPS measures the 12 distance between two latent embeddings from VGG-Net/AlexNet.. Likewise, PieAPP provides different measurement of perceptual score based model that is trained with the pairwise probabilities. DISTS measures the structural and textural similarities based on multiple layers of network feature maps and an algorithm inspired by SSIM. CKDN leverages distillation method that extract knowledge distribution from reference images, which can help calculate the likelihood of the restored image under such distribution. Both MUSIQ and DBCNN non-reference metrics, they both use deep network models (transformer and CNN, respectively) that are pre-trained on labeled image data with Mean Opinion Score. For non-learned metrics (model-based methods), FSIM uses the phase congruency and the color gradient magnitude of two images to calculate the similarity. The (CWMMS-JSSIM family uses insights about human perception contrasts to construct similarity metric to mimic human perception better. GMSD evaluates the distance between image color gradient magnitudes. NLPD means normalized Laplacian pyramid distance, which derives from a simple model of the human visual system and is also sensitive to the contrast of the images. VSI reflects quantitative measure visual saliency that is widely studied by psychologists and neurobiologists. MAD implements a multi-stage algorithm also inspired by the human visual system for distortion score calculation.
15

---

E Supplemental Ablation Study
By manipulating the trade-off parameter p. our model can be trained prioritize either perceptual quality or traditional distortion performance. In Figure we present the rate-distortion curves for the COCO dataset. Throughout our study, we examine four distinct values of p (0, 0.32, 0.64, 0.9). The outcomes illustrate that higher values of p result in improved perceptual quality but at the cost of worsened distortions in most scenarios. viable, as perceptual quality ceases to exhibit noticeable improvements.
worth noting that values exceeding p > 0.9 are not
CDC (ors)
(our)
(cun)
(can)
CKDN1
GMSD]
0.50
1601
10
0.85
PIEAPP
MUSIQT
MS-SSIM"
NLPDJ
0.00
1123
0.33
0.4
0.33
6/5
0.6
0.6
LPIPS]
DBCNNT
CW-S5IIM
VS1
1.001
11:20
0.993
0.985
0.4
0.5
0.4
0.6
0.4
0.6
0.6
0.8
FSIM]
PSNR
330
0.15
0.00
83
0.07
0.000
Figure 7: rate-distortion curves with different values
F Additional visualization of the compressed images and decoding variability visualization
16

---

Figure 8: Ground Truth
Figure 9: CDC Xe(p = 0.9), bpp=0.205
17

---

Figure 10: HiFiC bpp=0.207
ESIIT
Figure 11: Ground Truth
18

---

CR
Figure 12: CDC Xs(p = 0.9), bpp=0.398
Figure 13: HiFiC bpp=0.456
19

---

Figure 14: We stochastically decode the same latent variable z and y = 0.8 but different random seed for XN ~N (ID. Different random seeds may yield low-level textural distinction.
20

---

G Visualizations of the Decoding Process
Rm
mm
A deterministic decoding scheme
mm
A stochastic decoding scheme
Figure 15: Comparing two decoding schemes for decoding the same image, this visualiza- tion illustrates the evolution of the texture variable x, at five different time steps (n (0%N, 30,N, 60%N, 90%N, 100%NN) using total of N decoding steps. In the deterministic decoding scheme, the model transforms grayscale image into the actual reconstruction. Conversely, in the stochastic decoding scheme, noise is dynamically "denoised" during the process, resulting in reconstruction that incorporates high-frequue frequency details.
21

---

H Additional Rate-Distortion(Perepiiin) Results
HFC
DGML
NSC
CDC a 688 (cun)
CDC & p ((un)
MS
BPG
CDC
CDC
RID)
GMSOO
0.00
30-
005
11224
D00
002
0.30
622
0.4
3.50
1.85
0.23
0.75
0.00
0.13
0.2
0.4
06
0.2
0.6
0.2
0.4
0.6
0.5
0.2
0.4
0.6
LPIPS]
DBCKN
CW
1.80
0.995
0.25
0.993
0:985
0.000
0.30
D.083
01973
0.3
0.1
0/6
0.5
0.7
0.1
0.6
0.5
0.5
0.4
0.6
0.5
0.5
0.4
0.6
0.0
FSIM1
PSMR
MAD.
121
0.00
34
100
0.20
300
0100
0.3
0.1
105
1.5
012
11.
11.5
11.5
0122
01.
11.5
11.5
0.2
0.5
0.5
Figure 16: Rate-Distortion (eproetton(eerreptttoonn for COCO dataset. We use 500 decoding steps for eo model.
22

---

(ears
ME-Hyper
o [oum]
CDC p- (owns
[ID]
CKDN
SSIMI
GMSDL
D155
11121
D154
D:07
PIEAPP
MUSIQT
MS-SSIM-
NLPDJ
1.25
0.50
1.00
0.77
0.15
0.50
0.00
0.55
LPIPS]
DBCNNT
CW-SSIMM
VSH
0.25
01.20
0.201
0.080
0.00
0.5
0/6
DISTS,
FSIM
PSNR
MAD.
1.500
103
0.56
1100
0.5
0.5
0.5
Figure 17: Sale-DistortionnPerrpiiii for Tecnick dataset. We use 500 decoding steps for eg model.
23

---

DGML
(ears
ME-Hyper
o [oum)
CDC p- (owns
[1D]
CKDN
SSIMI
GMSD]
700
0.645
150
11:00
0.103
0073
003)
34-
0.500
00921
0.000
PIEAPP
MUSIQT
MS-SSIM-
NLPDJ
3.50
1133
0.75
1.00
0.00
0.50
LPIPS]
DBCNNT
CWW-SSIMT
VSI
0.991
0.3
60
0:99)
0.5
0.95
0.985
0.34
04
0.8
0.4
DISTS,
FSIM
PSNR
MAD.
122
0.15
27
103
0..%
23
0.1
0.1
0.5
0.1
0.0
Figure 18: Rate-Disooriion(eereptoon for Kodak dataset. We use 500 decoding steps for eg model.
24

---

DGML BPG
p (curs) CDC (ouns)
As p (suss)
Ms Hyper
CDC to - [oun]
F10
CKDNI
SSIMI
GMSDL
0.13
200
D/54
005
60
0.577
25
0.570
0.90
0.5
0.2
0.2
0/6
0.2
PIEAPP
MUSIQI
MS-S5IM"
NUPOJ
0.33
1.55
02.05
0.21
1.00
0150
0.30
LPIPS]
DBCNN1
CW S5IM1
VSh
C1/93
0122
02105
0L093
0.3
0.085
35
DISTSL
FSIM
PSNR
MAD.
11.20
110
103
142
01.00
0.10
0.07
0.05
Figure 19: Cate-Distortion(Peereptoon for DIV2K dataset. We use 500 decoding steps for eg model.. The complete 16 metrics.
25


Reviews:
[{"review_id": "pI6oDjbuYd", "reviewer": "Reviewer_kiiX", "summary": "This paper proposes an end-to-end learned image compression method based on diffusion model, where it consists of a VAE for latent feature encoding and a conditional diffusion decoder. Concerning theoretical parts, the authors also derive the variational upper bound to the diffusion model’s negative data log likelihood. Apart from that, by training the diffusion model with X-parameterization, the model can enable high-quality reconstructions in only a handful of decoding steps. Experimental results on 4 test datasets with 16 different metrics show that the proposed method can achieve comparable results compared to existing methods.", "questions": "1. I am confused about the theoretical analysis or rationale behind why using the X-parameterization rather than the $\\sigma$ parameterization can significantly reduce the number of decoding steps. I think this should be addressed in the rebuttal and also in the paper.\n2. I am also wonder if the authors are planing to release their code?", "limitations": "NA", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well and clearly written. The method part also includes some theoretical derivations.\n2. The method achieve sgood performance with extensive and complete evaluations on 4 different datasets.\n3. The introduction of X-parameterization significantly reduces the long decoding time, which makes the proposed diffusion-based method promising for applications.", "weaknesses": "1. The method itself is reasonable but without a clear motivation. It would be nice if the authors could spend some words on discussing some key advantages of using diffusion model for generative-based LIC? E.g., it would be nice if there is a significant improvement in terms of perceptual quality compared to VAE-based generative LIC, or that diffusion-based LIC can have some other benefits that VAE-based cannot have.\n2. Although with X-parameterization, the runtime is improved to be within a handful number of decoding steps, still as shown in the supplementary materials, the decoding time is still significantly longer than existing LIC methods.\n3. It is also welcome to have as many as evaluation (e.g., 16 metrics) is good, however, without explanations or analysis, it can be hard for readers who are not familiar with some of the metrics to qualitatively interpret the results. Compared to more metrics, personally, i prefer reporting comparison results with more publicly available methods on only 2-3 representative or well-known evolution metrics in the main paper while putting the others in the supplementary materials.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The method itself is reasonable but without a clear motivation.. What’s the key advantages\n\nPlease see our general response 1 above. \n\n> Although with X-parameterization..decoding time is still significantly longer than existing LIC methods.\n\nOur aim is to investigate the potential of diffusion models for compression purposes, for now focusing on fundamental rate-distortion and rate-perception tradeoffs. This is in line with the general trend in the neural compression community, where all existing approaches are currently 1-2 orders of magnitude slower than conventional image codecs (a recent exception is [Yang et al. 2023](arxiv.org/pdf/2304.06244.pdf)). We thus stress that our achieved decoding time performance is not completely unreasonable and in line with existing work. We remain optimistic that, through the development of enhanced architectures and continued research, runtime competitiveness can be substantially improved in the future. We believe that the current limitations in speed should not serve as a basis for dismissing early-stage research in neural compression.\n\n> I prefer reporting comparison results [...] on only 2-3 representative [...] metrics\n\nWe appreciate the suggestion of reducing the evaluation metrics in the main paper, but since the most important perceptual metrics are still subject to an ongoing debate, we prefer to keep eight (out of 16) metrics in the main paper. All 16 metrics were reported in Appendix H. We also provide a brief discussion of the metrics we used in Appendix D. \n\n> I am confused about the theoretical analysis or rationale behind why using the X-parameterization..\n\nPlease see our general response 3.\n\n> Code release\n\nPlease see our general response 4.\n"}, {"review_id": "IiYe072aHn", "reviewer": "Reviewer_QC9o", "summary": "In this paper, a conditioned diffusion model is proposed to improve the perceptual quality of learned image compression. Different from widely used VAE-like nonlinear transform coding frameworks, the proposed model adopts a diffusion process to synthesize most of the texture, conditioned on a learned hyperprior variable z. To further improve the perceptual quality, a mixed objective is proposed, combining the ELBO target of the diffusion model with a perceptual loss term. The proposed approaches show reasonable performance on most perceptual metrics, surpassing baselines like HiFiC.", "questions": "See above.", "limitations": "The authors have addressed the limitation.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This is overall an interesting paper. The authors intend to borrow the very success of diffusion models to the LIC approach, to promote the perceptual quality due to the improved generation ability. The proposed approach of using DDPM/DDIM as a conditioned texture synthesizer is intuitive and easy to follow. The presented results are overall impressive and encouraging. They present a new diagram for this line of research, which tend to inspire many future studies. ", "weaknesses": "1. The approach seems somewhat empirical. We can explain many of L1-LPIPS-GAN joint losses i.e. the one adopted by HiFiC as it corresponds to an implicit likelihood distribution (discussed in Ball'e 2018 hyperprior paper, typically an energy-based model) or some divergence measure (see f-gan). Let I cannot connect a similar theoretical analysis with the objective described in eq.8 & eq.9. Though this perceptual loss term is intuitive, I suggest the authors provide some more theoretical analysis to convince.\n2. I wonder why the LPIPS performance of CDC is worse than HiFiC, especially since the model is directly supervised under an LPIPS perceptual loss. A better FID (generation quality) with worse LPIPS (perceptual fidelity) is somewhat suspicious. Can we explain this because the diffusion models generate more 'hallucination' texture seems true but with lower fidelity to the ground truth? This may be true according to Figure 3(b), some artifact black-white textures near the eye not exiting in GT are generated in CDC, from my perspective.\n3. It may be unfair to compare the proposed model directly with HiFiC, as the network architecture of HiFiC is relatively lighter with less parameter volume than the UNet. The authors should report the model size and discuss the influence of improved model capacity. It may be better to discuss some larger LIC baselines eg PO-ELIC or evaluate a smaller UNet synthesizer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The approach seems somewhat empirical... provide some more theoretical analysis to convince.\n\nNeural image compression is largely an empirical field, so our paper does not stand out in this regard. Yet, we made sure our optimization objective follows from a rigorous lower bound on a latent variable model’s intractable marginal likelihood. This is outlined in the text and equations following line 158. We also contrast the capabilities of our approach to HiFiC, which optimizes a GAN-like loss instead.  \n\n> I wonder why the LPIPS performance of CDC is worse than HiFiC… \n\nWe note that both our method and HiFiC employ LPIPS as an auxiliary loss term, suggesting that HiFiC might also exhibit strong performance in LPIPS measurements. However, our approach differs from GANs, which learn a more rigid mapping from the latent variable $z$ to the decoded images. In contrast, our diffusion models incorporate a noise term $x_N$, which introduces perturbations. This noise-induced variability may result in suboptimal LPIPS outcomes but yields better FID scores, highlighting a key distinction between our method and GAN-based approaches. Thus, our diffusion models offer unique benefits in terms of image quality and generative capabilities.\n\n> Can we explain this (follow the question above) because the diffusion models generate more 'hallucination' texture that seems true but with lower fidelity to the ground truth?\n\nFollow our answer above, yes. While the noise perturbation reduce distortion metrics including the “perceptual distortion” LPIPS, it yields better FID scores. And it indeed generates hallucination textures as you can see. We also discussed this point in the limitation section.\n\n> It may be unfair to compare the proposed model directly with HiFiC, as the network architecture of HiFiC is relatively lighter with less parameter volume than the UNet\n\nWe have included essential information regarding the architecture and model size comparison between our model and HiFiC in Appendix A&C. Note that our approach utilizes a lightweight UNet, with the largest conv-layer containing only 384 channels. This results in 53.8 million parameters, versus HiFiC having 181.5 million parameters. The primary challenge for diffusion models lies in the iterative decoding process, which may result in slower decoding times. However, as we pointed out above, we think that the exploration of diffusion-based approaches should not be prematurely discarded on this basis as long as the decoding times are not completely off-the-charts. We consider one order of magnitude slower decoding times than traditional methods as an accomplishment and a reasonable basis for further development. We will defer the exploration of even faster decoding methods to future studies.\n"}, {"review_id": "FWguMlXbm3", "reviewer": "Reviewer_wzSD", "summary": "This paper proposes a diffusion-based image codec. It has two major components: a VAE-based encoder and a diffusion-based decoder. The VAE-based encoder generates the image latents, which serve as the conditioning signal for the diffusion-based image reconstruction. Both components are trained end-to-end with an objective function that involves a rate and a distortion estimate. The rate estimate is from the hyerprior model while the distortion estimate is a weighted sum of the perceptual loss and the mean squared error between the denoised image and the original image. ", "questions": "Please see my points in the Weaknesses box. ", "limitations": "The authors may like to comment on the reproducibility of the proposed method. For example, can experts in this field re-produce the results easily by following the training procedure given in this paper? Or the training may involve lots of \"know-hows\". ", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "(1) The idea of combining a VAE-based encoder and a diffusion-based decoder for learned image compression in an end-to-end trainable framework is novel and new. \n\n(2) Experimental results look interesting and promising.   ", "weaknesses": "(1) It is rarely seen that a diffusion model needs to be trained with a perceptual loss in order to achieve better subjective quality. From Fig. 2, without the perceptual loss (i.e. \\rho=0), the proposed method does not really stand out among the competing methods in terms of perceptual metrics. More insights need to be provided. \n\n(2) It is unclear whether one could use more decoding steps without the perceptual loss to achieve the same perceptual quality as with the perceptual loss with fewer decoding steps. This may justify the use of the perceptual loss from the perspective of the decoding complexity. \n\n(3) When examining carefully the training objective, I notice that L_upper is weighted additionally by the noise schedule coefficient \\alpha_n whereas the perceptual loss is not. Does this suggest that the perceptual loss may dominate the distortion estimation? I wonder what would happen if \\rho is increased beyond 0.9. \n\n(4) Surprisingly, in this work, the x-prediction (17 steps) is able to achieve better performance than the \\epsilon prediction (500 steps) with fewer decoding steps. This is a bit counterintuitive and is different from how the diffusion model is typically trained. Is there any explanation behind this observation? A more thorough comparison between these two design choices should be provided to justify the use of the x-prediction.\n\n(5) In Figure 4, the distortion curves are visualized with r=0 whereas the perceptual curves are with r=1. How about the other way around, r=1 for distortion and r=0 for perceptual?\n\n(6) In Appendix B, is there any reason for adding the z embedding to only the first four downsampling blocks instead of all the blocks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> It is rarely seen that a diffusion model needs to be trained with a perceptual loss.. More insights need to be provided\n\nPlease see our general response 2\n\n> It is unclear whether one could use more decoding steps without the perceptual loss to achieve the same perceptual quality as with the perceptual loss with fewer decoding steps\n\nMore denoising steps can improve the perceptual quality. As we showed in Figure 4, by increasing the number of denoising steps exponentially, the perceptual quality improves but saturates. Unfortunately, this improvement still cannot fill in the gap between models with and without perceptual loss.\n\n>  Does this suggest that the perceptual loss may dominate the distortion estimation?.. I wonder what would happen if $\\rho$ is increased beyond 0.9\n\nThis is a good question. We add a new plot in our general response Figure 1, which suggests that this does not seem to be the case. If the perceptual loss were to significantly dominate the loss function and overshadow the stochastic score matching loss, a single x-prediction step should achieve similar reconstruction quality as multiple reconstruction steps, as $\\mathcal{X_\\theta}(x_n, z, n)$ may approximate $\\mathcal{X_\\theta}(z)$. However, our approach's performance markedly improves with multiple diffusion steps (1 vs 17). This evidence indicates that both LPIPS and diffusion losses play crucial roles in our methodology.\n\nMoreover, it is important to note that the weighting term $\\frac{\\alpha_n}{1-\\alpha_n}$ dynamically changes according to the subsample index n, potentially becoming very large when $\\alpha_n$ approaches 1, or very small as $\\alpha_n$ approaches 0. We also discussed the impact of the $\\rho$ value in Appendix E. While larger $\\rho$ values do result in higher perceptual quality, it will also saturate. $\\rho>0.9$ will not be able to provide prominent perceptual quality improvements.\n\n>  the x-prediction (17 steps) is able to achieve better performance than the \\epsilon prediction (500 steps) with fewer decoding steps… counterintuitive\n\nPlease see our general response 3.\n\n> the distortion curves are visualized with r=0 whereas the perceptual curves are with r=1. How about the other way around\n\nWe actually did these experiments as well and decided at the last minute to simplify the figure and not show them for simplicity. We provide the remaining figures for completeness in general rebuttal Figure 2.\n\n> is there any reason for adding the z embedding to only the first four downsampling blocks\n\nThis is due to the U-Net architecture our diffusion model uses. Our neural encoder (ENC) consists of only four downsampling layers; therefore, only the first four U-Net layers correspond to the spatial dimensions of the outputs from the four upsampling EMD layers. This is the rationale behind concatenating only four layers instead of six.\n\n> Reproducibility\n\nSee our general response 4"}, {"review_id": "g43Fw4p8zJ", "reviewer": "Reviewer_ys1i", "summary": "This paper presents a method of using conditional diffusion as a stochastic decoder for neural image compression. The approach learns a discrete latent representation of an image and conditions the diffusion process on this representation. Compression is achieved by learning a distribution over the latent representation and running an entropy coder/decoder as is commonly done in other neural image compression models.\n\nThe main benefit of this approach is that diffusion (empirically) does a better job that perceptual metrics (like LPIPS) or adversarial losses in boosting the \"realism\" or \"perceptual quality\" of reconstructed images. The paper shows that with multiple metrics (Fig. 2) and images (Fig. 3).\n\nThe paper also shows that relatively few diffusion steps are required to generate high-quality reconstructions (Fig. 4) due to the conditioning and the use of X-prediction instead of predicting epsilon (the noise for step t). I didn't see runtime numbers in the paper, but the reduction of diffusion steps should lead to \"moderate\" decode times (contrast with early diffusion methods that required hundreds or thousands of steps).", "questions": "If I understand the formulation properly, rho controls the weight on LPIPS loss used during training (Eq. 9). In Fig. 2, I'm surprised there's such a large gap between the \"CDC \\rho=0\" (blue) and \"CDC \\rho=09\" (orange) curves for FID. It makes sense that there's a large gap for LPIPS and DISTS since high values of rho mean you're using the test metric (or close to it in the case of DISTS) as part of the training process (to be clear: training with LPIPS is common so I'm not implying this is a problem). But I would have expected diffusion alone to improve FID scores.\n\nSo is LPIPS doing the heavy lifting here, at least relative to the metrics? Is there a better metric for measuring the benefit of using diffusion vs. a GAN (plus LPIPS) loss as in HiFiC?\n", "limitations": "adequately addressed", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The main strength of the paper is the effective use of a diffusion-based decoder for neural image compression that leads to high subjective/perceptual visual quality according to both quantitative metrics and the example reconstructions in the paper. In particular, the results appear better than those from similar, GAN-based models (like HiFiC), and the ability to adjust the stochasticity at inference time (by adjusting gamma, which controls the amount of noise used by the reverse-diffusion process) is quite interesting and potentially helpful reducing concerns about hallucinated details.\n\nThe paper is also very well-written and easy to follow, and diffusion-based stochastic decoders are of high interest to the neural image/video compression community.", "weaknesses": "The main weakness of the paper is low novelty since diffusion for image generation is well-known at this point so its use in a codec is a fairly obvious application (though it's not technically easy to achieve good results).\n\nThe authors should cite this paper on the same topic, though as far as I know it has not been published in a peer-reviewed, archival venue so it doesn't block publication of submission 3900:\n\nHoogeboom 2023. High-Fidelity Image Compression with Score-based Generative Models\nhttps://arxiv.org/abs/2305.18231\n\nThe authors should also compare, if possible, against more recent GAN-based models that outperform HiFiC:\n\nPO-ELIC: Perception-Oriented Efficient Learned Image Coding\nHe 2022. https://arxiv.org/abs/2205.14501\n\nMulti-Realism Image Compression with a Conditional Generator\nAgustsson 2023. https://arxiv.org/abs/2212.13824\n\n\nThe inclusion of runtime numbers and a rater study would also strengthen the paper, but I don't think a rater study should be required for publication for every compression paper.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> not novel enough\n\nPlease see general response 1\n\n> The authors should cite this paper on the same topic\n\nThank you for pointing this out; we also noticed these important related work and will cite and discuss it in the final paper version. Note that some works are concurrent and appeared after this submission.\n\n> The inclusion of runtime numbers and a rater study\n\nWe reported runtime in Appendix C. We agree that a rater study could be an interesting option for image quality evaluation. While user studies are clearly very nice and valuable, we need to consider our available resources of funds to include such a study in the camera-ready version. \n\n> I would have expected diffusion alone to improve FID scores. LPIPS is doing a heavy lift.\n\nWe agree that this is a surprising outcome of our study, but was rigorously tested across 16 metrics. Please refer to general response 2 for details. Since the most important perceptual metrics are still subject to an ongoing debate, it's hard to say if there is a better metric."}, {"review_id": "fTN2rv0O5y", "reviewer": "Reviewer_9a7p", "summary": "In this paper, the authors propose a novel architecture of lossy image compression using conditional diffusion models. They use an encoder to map images into contextual latent variables and train a conditional diffusion model under the guidance of these contextual latent variables as the decoder.  Moreover, they add perceptual loss into the loss of diffusion model, so this approach can steer the tradeoff between distortion, rate and perceptual quality by changing the parameter of its loss and diffusion model. Finally, they conduct sufficient experiments to explore how the parameters influence the final performance and to compare with other approaches.", "questions": "1. In section 4, I wonder why the images of the Kodak dataset are not resized while the others are resized, and how the diffusion model trained on an image size of 256×256 can be applied to images of different sizes when testing.", "limitations": "Yes, the authors have considered the limitations and societal impact of neural compression, particularly when prioritizing perceptual quality, is the risk of misrepresenting data in the low bitrate regime.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The architecture based on the conditional diffusion model is interesting and creative, and as is shown from the experiments in 3.2, this architecture can also finish the reversed progress of the conditional diffusion model in a handful of steps.\n2. Sufficient experiments have been conducted, including comparison with other methods in sixteen metrics, and exploration of how parameters like ρ, γ, and decoding steps influence the final performance.\n2. Overall the results prove that this method can steer the tradeoff between distortion, rate and perceptual quality, and the final performance with specific parameters in the paper is also quite competitive in different metrics. ", "weaknesses": "1. Some implementation details are not illustrated clearly, including how to get the \"17 steps\" of the reversed progress in 4.1 which may be helpful to understand the schedule of the reversed progress, the cost of encoding and decoding time, and I think a picture including the image of every single step in 17 steps when predicting x0, may help to prove that the architecture can reconstruct the \"texture\" structure of the image in a handful of steps.\n\n2. More qualitative and quantitative results may be helpful to prove the performance and practicability of the method. including images with different resolutions and different styles of images.\n\n2. I think, when testing, the parameters of the diffusion model should be given more specifically, including the total number of steps, the true number of steps after resampling, and the type of schedule(cosine or linear), because these parameters can influence the decoding steps metioned in the paper.\n\n3. In formula (7), I think it can only represent the situation of γ=0, that is the DDIM, but when γ>0, I think there should be an extra random noise according to the formula(predicting noise is changed to predicting x0)\n\n   $$\n   \\boldsymbol{x}_{t-1}=\\sqrt{\\alpha_{t-1}} \\underbrace{\\left(\\frac{\\boldsymbol{x}_t-\\sqrt{1-\\alpha_t} \\epsilon_\\theta^{(t)}\\left(\\boldsymbol{x}_t\\right)}{\\sqrt{\\alpha_t}}\\right)}_{\\text {\"predicted } \\boldsymbol{x}_0 \"}+\\underbrace{\\sqrt{1-\\alpha_{t-1}-\\sigma_t^2} \\cdot \\epsilon_\\theta^{(t)}\\left(\\boldsymbol{x}_t\\right)}_{\\text {\"direction pointing to } \\boldsymbol{x}_t \\text { \" }}+\\underbrace{\\sigma_t \\epsilon_t}_{\\text {random noise }}\n   $$", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Some implementation details are not illustrated clearly\n\nWe have supplied the decoding time of the model and architecture details in Appendices A and C. Additionally, Appendix G contained visualizations of image transitions during the decoding process. We chose 17 as the number of steps to strike a balance between achieving satisfactory perceptual quality and maintaining a reasonable decoding time.\n\n> More qualitative and quantitative results.\n\nWe had more visualizations in Appendix F. The rate-(distortion/perception) curves for all of our datasets were available in Appendix H. Regarding your valuable suggestion of plotting image x_n over iterations: note we already included exactly this in Appendix G Figure 15.\n\n>  the parameters of the diffusion model should be given more specifically\n\nIn Appendix B, we have included comprehensive images and descriptions of our model's architecture. We employ a linear noise schedule for the diffusion process, and we apologize for not mentioning this in the paper and we also found that there are not big performance gaps between cosine schedules and the linear ones. We are not sure what you mean by “true number of steps after resampling”, could you elaborate on this? As stated in our experiment section, we used 20,000 steps for training the epsilon-prediction model and 8,000 steps for the x-prediction model. The 17 steps refer to the number of decoding steps utilized for decoding during the main evaluation. We will refer to the appendix in the main paper more explicitly. \n\n> In formula (7), I think it can only represent the situation of γ=0\n\nThere may be a misunderstanding of the source of stochasticity in DDIM. DDIM converts an original stochastic differential equation (DDPM) into a deterministic ODE; the only source of noise is in the initial state of $x_N$. All our algorithms use this DDIM scheme, which was shown to provide better sample qualities than following the stochastic differential equation especially under small sampling steps (see [Song et al. 2020](https://arxiv.org/pdf/2010.02502.pdf)). Note that when we say deterministic/stochastic in our model, we only refer to whether $x_N$ is fixed to 0 or stochastically sampled from $N(0, \\gamma^2I)$. The $\\gamma$ term solely controls the variance of the initial noise $x_N$. The noise term in your formula is weighted by $\\sigma_n$, which is fixed to zero for DDIM.\n\n>  I wonder why the images of the Kodak dataset are not resized while the others are resized. how the diffusion model trained on an image size of 256×256 can be applied to images of different sizes when testing.\n\nThe Kodak dataset is a widely-used image compression benchmark dataset containing high-quality images. For the other datasets, we resize the images to eliminate compression artifacts introduced by their original codecs, such as JPEG. This is common in the literature since otherwise JPEG compression would not incur additional compression losses and have an unrealistic advantage. It is important to note that we employed a maximum of 6 downsample and 6 upsample layers with a scale factor of 2, which implies that the width/height of images would ideally be a factor of 64. Lastly, our model is fully convolutional, enabling it to handle various image sizes."}]

Author rebuttal:
# General Response:

We thank all reviewers for their valuable feedback. Before we address them individually, we would like to address common questions in their reviews. 

> 1. Motivation, novelty, and advantages of our work

Our diffusion model provides a new approach towards “perceptual compression”, a field of growing importance (See Yang et al. 2022 https://arxiv.org/pdf/2202.06533.pdf). All current SOTA VAE-based neural image compression methods involve a probabilistic decoder p(x|z) that models the distribution over possible image reconstructions x given a global latent image representation z. While VAE-based decoders typically reconstruct the *mean* of p(x|z), this can lead to blurry artifacts since the true p(x|z) is multi-modal and hence the mean can be far away from the data manifold. In contrast, our approach *samples* from p(x|z) using a conditional diffusion model. This avoids mode averaging, leading to better perceptual qualities. Thus, our approach is a first alternative to an important class of GAN-based approaches (such as HiFiC) that bias the data reconstruction deterministically towards the mode of p(x|z). Such models of perceptual compression are currently at the forefront of learned image compression due to their potential for extreme compression gains. 

We show empirically that our approach outperforms GAN-based approaches on various perceptual metrics as well as traditional distortion metrics. We also show that diffusion models allow various decoding strategies with one trained model, which provide more decoding flexibility than GAN-based models.

Our work exhibits novelty in several aspects. It is the first successful application of transform coding with diffusion models. Other diffusion approaches either focus on lossless compression (which is fundamentally different) or rely on more computationally expensive relative entropy coding schemes, as we have already discussed in our paper.

Additionally, our approach involves several crucial design decisions that contribute to its strong performance:
X-prediction: This technique enhances the reconstruction quality in our model, enabling stochastic image reconstructions in only a handful (as opposed to hundreds) of iterations.
Conditioning mechanism: Our unique method for incorporating a global image representation ensures competitive diffusion model performance.
An additional optional LPIPS loss can boost the perceptual quality of the diffusion model.

> 2. Why our diffusion-based approach uses an LPIPS loss

We stress that the LPIPS term is an optional addition in both HiFiC (an important perceptual compression baseline) and in our approach. HiFiC is essentially a VAE-GAN hybrid model, where a GAN discriminator biases the VAE reconstructions towards a solution closer to the data manifold. In contrast, our diffusion model can be seen as a VAE-Diffusion hybrid model with a stochastic, diffusion-based decoder. From this perspective, it is not surprising that addition an additional loss such as LPIPS can improve the performance in *both* approaches, especially when the focus is on perceptual metrics. A model with three losses is just more expressive than a model with two. We also note that the LPIPS term is one way of modeling an additional data likelihood $p(x_0|x_n)$ in diffusion models. This likelihood term was previously shown to improve sample quality in other tasks with diffusion models, e.g., [Austin et al. 2021](https://arxiv.org/pdf/2107.03006.pdf) equation 5.

> 3. The rationale behind x-prediction

X-prediction was initially introduced in the diffusion model by [Sohl-Dickstein et al. 2015](http://proceedings.mlr.press/v37/sohl-dickstein15.pdf), who directly employed the variational lower bound (with x-prediction) as their training objective. Subsequently, the DDPM paper by [Ho et al. 2020](https://arxiv.org/pdf/2006.11239.pdf) further established the connection between the variational lower bound and the $\epsilon$-prediction based score-matching loss. As demonstrated in the paper [Saliman et al. 2022](https://arxiv.org/pdf/2202.00512.pdf), empirical evidence has shown that x-parameterization can outperform epsilon-prediction, particularly when the number of sampling steps is limited (with progressive distillation) in unconditional generation.

Inspired by the above works, while we originally experimented with epsilon-parameterization, we found the surprising result that x-parameterization speeds up sampling by more than an order of magnitude. A possible explanation is that we utilize an encoder to generate a content latent variable, which is adept at providing a robust signal for input reconstruction even within a single denoising step (as illustrated in Fig 4), and that our loss resembles a VAE in this scenario. We note that our Eq.6 also described the mathematical equivalence between epsilon and x-prediction.

> 4. Appendix and Code availability

We note that we provided an appendix in the supplemental material. We apologize that we didn’t attach the necessary links or zip files regarding the code in our paper; this was originally intended by us but we lost track of it. According to the NeurIPS guidelines, we can not share a link to a repository at this stage, but we send the code to the Area Chair who can be contacted by you to obtain the code. 

Meta-review: Author rebuttal addressed most of the reviewer concerns and all reviewers are leaning accept after final discussion.
Decision: Accept (poster)