Lossy Image Compression with Conditional Diffusion Models
Authors: Ruihan Yang
Stephan Mandt
Venue: Submitted to ICLR 2023 (2023)
Keywords: neural image compression
denoising diffusion models

Abstract: Denoising diffusion models have recently marked a milestone in high-quality image generation. One may thus wonder if they are suitable for neural image compression. This paper outlines an end-to-end optimized image compression framework based on a conditional diffusion model, drawing on the transform-coding paradigm. Besides the latent variables inherent to the diffusion process, this paper introduces an additional discrete "content" latent variable to condition the denoising process on. This variable is equipped with a hierarchical prior for entropy coding. The remaining "texture" latent variables characterizing the diffusion process are synthesized (either stochastically or deterministically) at decoding time. We furthermore show that the performance can be tuned toward perceptual metrics of interest. Our extensive experiments involving five datasets and 16 image perceptual quality assessment metrics show that our approach not only compares favorably in terms of rate and perceptual distortion tradeoffs but also shows robust performance under all metrics while other baselines show less consistent behavior. 

Under review as a conference paper at ICLR 2023
LOSSY IMAGE COMPRESSION WITH CONDITIONAL DIFFUSION MODELS
Anonymous authors
Paper under double-blind review
ABSTRACT
Denoising diffusion models have recently marked milestone in high-quality im- age generation. One may thus wonder if they are suitable for neural image com- pression. This paper outlines an end-to-end optimized image compression frame- work based on a conditional diffusion model, drawing on the transform-codin paradigm. Besides the latent variables inherent to the diffusion process, this paper introduces an additional discrete "content" latent variable to condition the denois- ing process. This variable is equipped with hierarchical prior for entropy cod- ing. The remaining "texture" latent variables characterizing the diffusion process are synthesized (either stochastically or deterministically) at decoding time. We furthermore show that the performance can be tuned toward perceptual metrics interest. Our extensive experiments involving five datasets and sixteen image quality assessment metrics show that approach not only compares favorably in rate-perceptual quality but also shows close distortion performance with state- of-the-art models.
INTRODUCTION
With visual media vastly dominating consumer internet traffic, developing new efficient codecs for images and videos has become evermore crucial (Cisco 2017). The past few years have shown considerable progress in deep learning- based image codecs that have outperformed classical codecs
in terms of the inherent tradeoff' between rate (expected file size) and distortion (quality loss) (Ballè
et al. 2018 Minnen et al. 2018 Minnen & Singh 2020 Zhu et al. 2021 Yang et al.) 2020 Cheng jet al. 2020: Yang et al.] 2022b). Recent research promises even more compression gains upon
optimizing for perceptual quality, i.c.. increasing the tolerance for imperceivable distortion for the benefit of lower rates (Blau & Michaeli] 2019). For example, recent works involving adversarial
losses (Agustsson al. 2019: Mentzer et al.) (2220) show good perceptual quality at low bitrates.
Most state-of-the-arr art learned codecs currently rely on the transform coding paradigm and involve hierarchical "compressive" variational autoencoders (Ballé et al., (0188) Minnen al.] [2088] Min- nen & Singh 2020). These models simultaneously transform the data into a lower dimensional latent space and
learned prior model for entropy-coding the latent representations into short bit strings. Using either Gaussian or Laplacian decoders, these models directly optimize for low MSE/MAE distortion performance. Given the inereasing focus on perceptual performance over dis- tortion, and VAES suffer from mode averaging behavior inducing blurriness (Zhao et al., (2017), one may wonder if better perceptual results can expected by replacing the Gaussian decoder with a more expressive conditional generative model.
This paper proposes to relax the typical requirement of Gaussian (or Laplacian) decoders com- pression setups and presents a more expressive generative model instead: a conditional diffu- sion model. Diffusion models have achieved remarkable results on high-quality image generation tasks (Ho et al. 2020: Song al. 202 a). By hybridizing hierarchical compressive VAEs (Balle) ctt al. 2018) with conditional diffusion models, we create a novel deep generative model with promis- ing properties for perceptual image compression. This approach is related but distinct from the recently proposed Diff-AEs (Preechakul et al. 2022), which are neither variational (as needed for entropy coding) nor tailored to the demands of image compression.
We evaluate our new compression model on five datasets and investigate a total of 16 different metrics, ranging from distortion metrics, perceptual reference metrics, and no-reference perceptual

---

Under review as a conference paper at ICLR 2023
metrics. We find that the approach is comparable with the best available compression models while showing more consistent behavior across the different tasks. We also show that making the decoder more stochastic vs. deterministic will decrease over-smoothing while degrading distortion, showing once more that perceptual quality is distinct from good reconstruction (Blau & Micchalli, 2019).
In sum, our contributions
follows:
We propose the first transform- cewfform-ooding-baae lossy compression scheme using diffusion models. The approach uses a VAE-style encoder map images onto a contextual latent variable; this latent variable is then fed as context into a diffusion model for reconstructing the data. The approach can be modified to enhance several perceptual metrics of interest.
We derive our model's loss function systematically from variational lower bound to the data bog-likeihood. The resulting distortion term is distinct from traditional VAEs and is better suited for modeling the residual noise than a conditional Gaussian distribution.
We provide substantial empirical evidence that variant of our approach is, in many cases, better than the state-of-the-art in term of perceptual quality. Our base model also shows on-par rate-distortion performance with two MSE-optimized baselines. To this end, we considered five test sets, three baseline models (Wang et al.] [022] Mentzer et al. 2020) Cheng et al. p020), and 16 image quality assessment metries (classical and neural).
RELATED WORK
We discuss related works on Lossy Compression, Compression For Realism and Diffussion Models.
Lossy Image Compression The widely-established classical codecs such JPEG WWallace 1991]. BPG (Bellard 2018). WEBP (Google 2022) have recently been challenged by end-to-end learned codecs (Ballé al. 2018, Minnen et al. 2018 Minnen & Singh 2020 Yang et al. 2020 Cheng et al.] 2020 Zhu et al.J (002)). These methods typically draw on the non-linear transform coding paradigm realized by hierarchical VAEs. Usually, neural codecs are optimized to simulta- neously minimize rate and distortion metrics, such as mean squared error or structural similarity.
Compression For Realism In contrast to neural compression approaches targeting traditional metries, some recent works have explored compression models to enhance realism (Agustsson et al. 2019 Mentzer et af., 2020, Tschannen et al. 2018 A theoretical background for these approaches was provided by Blau & Michaeli (2019): ZZhang et al. (2021), who considered opti- mizing the autoencoder-
compression model with additional distortion terms based on neural metrics (e.g. LPIPS (Zhang et al. (2) adversarial losses (Goodfellow et al. 2014 Rippel & Bourdev 2017). Since GAN training introduces a variety of instabilities, successful deployment of these methods requires variety of design choices.
Diffusion Models Probabilistic diffusion models showed impressive performance on image gen- eration tasks, with perceptual qualities comparable those of highly-tuned GANs while maintain- ing stable training (Song & Ermon 2019 Ho et al. 2020; Song et al. 2021b] Song & Ermon Kingma al.,2022 Yang et al. 2022a Ho et al.J 2022b; Saharia al. 2022; Preechakul et al. 2022p. Popular recent diffusion models include Dall-E2 (Ramesh et al. 2022) and Stable- Diffusion (Rombach et al.) 2022). Some works also proposed diffusion models for compression. Hoogeboom et al. (2021) evaluated an autoregressive diffusion model (ADM) on a lossless com- pression task. Besides the difference between lossy and lossless compression, the model only tested on low-resolution CIFAR-I0 (Krizhevsky et al. 2009) dataset. In concurrent work, (This et al. (2022) proposed a diffusion model for lossy compression, using a generic unconditional dif- fusion model that can communicate Gaussian samples, but there is currently no practical method that can reduce its extensive computational cost without restrictive assumption or additional coding
2019
costs (Li & El Gamal 2018 Flamich et al. 2020 2022
Theis & Ahmed 2022).
METHOD
We review diffusion models and neural compression methods and then discuss model design.
2

---

Under review as a conference paper at ICLR 2023
Gaussian Diffusion
Xo
XN-1
X
P(a) Entropy Coding
K
10101...
Encoding
or
Conditioning
XN 2
%0
X
XN-1
X
Decoding
Figure 1: Overview of our proposed compression architecture. A discrete "content" latent variable contains information about the image. Upon decoding, this variable us used for conditioning denoising diffusion process. The involved "texture" latent variables XI.N are synthesized on the fly.
3.1 BACKGROUND
Denoising diffusion models are hierarchical latent variable models that generate data by a se- quence of iterative stochastic denoising steps (Sohl-Dickstein et al. 2015 Ho et al. 2020 Song et al. 2021a Song & Ermon 2019p. These models describe a joint distribution over data X and la- tent variables X1:N such that pe (xo) ( )dx While diffusion process (denoted by q) incrementally dextroys structure, its reverse process generates structure. Both processes involve Markovian dynamics between a sequence of transitional steps (denoted by n), where
q(x,/ -1) (x v B, B,I);; (x xx.) A NX Ma(Xn, B,I). (1)
The variance schedule 3n (0, 1) can be either fixed learned; besides it, the diffusion process is parameter-free. The denoising process predicts the posterior mean from the diffusion process and is parameterized by neural network ()) 12).
Denoising Diffusion Probabilistic Model (DDPM) (Ho et al. 2020) showed tractable objective function for training the reverse process. A simplified version of their objective resulted in the following noise parameterization, where one seeks to predict the noise used to perturb particular image X from the noisy image Xn at noise level vi:
L(O, xy) En,elle Eo (Ii, N), On -I_ At test time, data can be generated by ancestral sampling using Langevin dy- namics. Alternatively, Sonng et al. (202 proposed the Denoising Diffusion Implicit Model (DDIM) that follows a deterministic generation procedure after initial stochastic draw from the prior. Our
Above.
IN(,I),
xx,x)
Va + vI- a,c, and
paper uses the DDIM scheme at test time, see Section 3.2 for details.
Neural image compression seeks outperform traditional image codecs by machine-learned models. Our approach draws on the transform-coding-baee neural image compression approach (Theis et al. 2017 Ballé et al., 2018 Minnen et al.) 2018), where the data non-linearly trans- formed into a latent space, and subsequently discretized and entropy-coded under learned "prior". The approach shows a strong formal resemblance VAEs and shall be reviewed in this terminology.
Let be a continuous latent variable and à = [z] the corresponding rounded, integer vector. The VAE-based compression approach consists of stochastic encoder e(z]x), prior p(z), and de- coder p(x|z). The model is trained using the negative modified ELBO,
C(), x) Ez logllop((pp// px||z) - > logg(())]
(3)
The first term measures the average reconstruction of the data (distortion), while the second term measures the costs of entropy coding the latent variables under the prior (bitrate). The encoder e(zix) ([Enc((x)
Enc + 1) is boxed-shaped distribution that simulates rounding training time through noise injection due to the reparameterization trick. Note has zero entropy.
Once the VAE is trained, en/decode data using the deterministic components and & = Dec(z). We convert the continuous p(z) into a discrete P(z) over the integer lattice as described in YYang al. (002bb [Sec. 2.1.6] and use for entropy coding å (Ballé al.) 2018).
Enc(x)]
3

---

Under review as a conference paper at ICLR 2023
While VAE-based approaches have used simplistic (e.g.. Gaussian) decoders, show that get significantly better results when defining the decoder p(x[z) as a conditional diffusion model.
3.2 CONDITIONAL DIFFUSION MODEL FOR COMPRESSION
The basis of our compression approach is new latent variable model: the diffusion variational autoencoder. This model has a "semantic" latent variable z for encoding the image content, and a set of "texture" latent variables X1:N describing residual information,
(4) As detailed below, the decoder will follow denoising process conditioned on z. Drawing meth- ods described in Section 3.1 we use a neural encoder e(2]xo) to encode the image. The prior p(z) is two-level hierarchical prior (commonly used in learned image compression) and is used for entropy coding z after quantization (Ballé et al., 2018). Next, we discuss the novel decoder model.
p(Xo.N: z) = x(xo.N /z)p(()).
Decoder and training objective We construct the conditional denoising diffusion model in a sim- ilar way to the non- vaniaariatooall diffusion autoencoder Preechakul al. (0022) In analogy to Eq. we introduce a conditional denoising diffusion process for decoding the latent variables z,
(x 2) p(xN) II [o 2) p(xx) II [x 2, (3,
(5)
Since the texture latent variables X1.N are not compressed but synthesized at decoding time, the optimal encoder and prior should be learned jointly with the decoder's marginal likelihood p(xolz) f p(xo:N] [z)dx while targeting a certain tradeoff between rate and distortion spec- ified by Lagrange parameter X. We can upper-bound this rate-distortion (R-D) objective as
Exxe(x)xo) log D(x z)
p(z)]
[- log
p p(Xo:N|Z)) Alogp(z)
Ex (xLNNo(x
(x xoo
o(x1,x xx)] as the variational lower bound to the diffusion model's conditional data likelihood (inducing upper bound on the RD ob- jective). We realize that - log Plower (Xp) z) corresponds to novel image distortion metric induced by the conditional diffusion model (in analogy to how a Gaussian decoder induces the MSE dis- tortion). This term measures the model's ability to reconstruct the image based on z. In contrast, log p(z) measures the number of bits needed to compress under the prior. As in most other works on neural image compression (Ballé et al.] (0188: Minnen et al.) 2018) Yang et al.) 20226), we use a box-shaped stochastic encoder e(z) Xo) that simulates rounding by noise injection at training time.
For brevity, we define log Plower l2) Rx x(x)) loog p(Xo
Following Ho et al. (2020) and in analogy to Eq. 2 we simplify the training objective by using the denoising score matching loss
- Plower (Xol2) R Ex <0 (x, (Xo), n/Nerninaaal - 2
The noise level n and an are defined in Eq. [ Instead of conditioning on n, we condition the model on the continuuuss variable n/Nerain which offers additional flexibility in choosing the number of denoising steps for decoding (e.g., we can use a Niest smaller than N.rai)). This pseudo-continuous scheme has a related continuous version (Kingma al. 2021).
Decoding process Once the model is trained, we entropy-decode z using the prior p(z) and con- ditionally decode the image Xo using ancestral sampling. We consider two decoding schemes: stochastic one with xN ~ (0, (where 0 < < 1) and deterministic version with XN (or = 0), both following the DDIM denoising method (Song et al. 2021a):
(8) In both cases, the denoising network Eg is defined as in Eq. 7 Since the latent variables X1:N not stored but generated at test time, these "texture" latent variables can result in variable reconstructions upon stochastic decoding (see Figure 4 same image with different random seeds). We find that the DDPM scheme by (Ho et al. 2020) leads to worse results at decoding time (see Section 4.2 for details).
X an-1
X,, (o
an (x,,, n, 2).
van
for for decoding with different y and Figure 14 for decoding the
summarizes training and encoding/decoding. We find that the E, loss leads to better perceptual qualities and fewer color artifacts E2, see also (Saharia al. 2022).
Algorithm [
4

---

Under review as a conference paper at ICLR 2023
Algorithm 1: Training (Left); Encoding and Decoding (Right). We note that the entropy coding
part largely follows the method proposed by [Balé al. (2018); the discrete latent variable simulated by adding uniform noise during training and quantized to the nearest integer (2- [z]
during testing. We calculate discrete likelihood as P = CDF, - 0.5) CDF 0.5)
for continuous distribution p(z). Both À and p are manually selected for training.
while not converged do
Given Ntest;
Sample Xg dataset;
2 =
(xo)(x
n 1(00, 1, 2., Nerain);
2 2
binary file;
c (O,T)) I):
XN = 0 or XN ~ (0,77
X, van + vI anG
for л= Neett to do
2 (xE) 2/2 Excς(xo) + +2);
cØ (n, NN 2):
Ny = 8 Vl
Xo- 8, Tn,
van
Van
LD = I - = V 2)1:
xn-1 v Van XX0 + v1 1681
= (1 - -p)LD + pd(Xo, xo) Alog P(z) end
() = ()) V
Xn return X0
Xo:
end
Optional Perceptual Distortion While Eq. 7 already describes a viable loss function for our conditional diffusion compression model, we can influence the perceptual quality of the compressed images by introducing additional loss functions similar (Mentzer et al. 2020).
First, we note that the decoded data point can be understood as function of the higher-level latent When
Xn, the latent code z, and the iteration n, such that Xy (x z, n)
To yan
minimizing a perceptual metric d(.:-) -) in image space, we can therefore add a new term to the loss:
Tpperceptual= Ex xo) [d [ (x,, 2, n); xo)].
A
L = plppereopcual (1 - p) (x [log Plower (xol (z) +
log p(z)]-
(10)
1-
This loss term weighted by an additional Lagrange multiplier E [0. 1). resulting in a three- way tradeoff between rate, distortion, and perceptual quality (Yang et al., 20.2b, Blau & Michaeli] 2019) In this paper, we choose the widely-adopted LPIPS loss (Zhang et al. 2018a) for Lperceptual- (Note that Blau & Michaeli (2019) define the "perception" as divergence of distributions without reference to the ground truth image, which is different from LPIPS.)
Architecture The design of the denoising module follows similar U-Net architecture used in DDIM (Song et al., 202 la) and DDPM (Ho al.] 2020) projects. Each U-Net unit includes two ResNet blocks (He et al. 2016), one attention block and a convolutional up/downsampling block. We use six U-Net units for both downsampling and upsampling process. The channel dimension for each downsampling unit is 64 x j. where j is the index of the layer range from I to 6; the upsampling units follow the reverse order. Each encoder module consists of one ResNet block and one convolutional downsampling block. For conditioning with embedding, and transposed convolution to upscale z to the same spatial dimension as the inputs of the beginning four U-Net downsampling units, so that we can perform conditioning by concatenating the the output of the embedder and the input of the corresponding U-Net unit. See Appendix B and Figure 5 for more details.
use ResNet blocks
4 EXPERIMENTS
We conducted large-scale compression evaluation involving 16 image quality metrics and 5 test datasets. Besides metrics measuring differences between compressed and raw images ('full ref- erence metrics"), we also considered "no-reference metrics" that evaluate image quality without reference to the raw image. While some of these metrics are fixed, others are learned from data. We will refer approach "Conditional Diffusion Compression" (CDC) in the following.
5

---

Under review as a conference paper at ICLR 2023
PIEAPPPPsshnani et al
2018
Full Reference
Learned
Perceptual Metric
LPIPS Zhang et all
201
Full Reference
Learned
Perceptual Metric
Ding et
Full Reference
Learned
Perceptual Metric
CKDNZZheng
2021
Full Reference
Learned
Perceptual Metric
MUSIO Ke et al.
2021
No- RefReeereecee
Learned
Perceptual Metric
DBCNN(Zling el al.
(2018b)
No- Reierence
Learned
Perceptual Meiric
(Hcuscl
(2017)
No-Refcrence
Learned
Perceptual Metrie
FSIM Zhang
2011
Full Reference
Not Learned
Distortion Metric
SSIMI Wang el al.
|
2004
Full Reference
Nat Learned
Distortion Metric
MS.SSMM Wang et
21000
Fulll Reference
Tearned
Distnrtion Metric
CW-SSIM Sampal et (2009)
Full Relerence
Not Learned
Distortion Metric
PSNR
Full Reference
Not Learned
Distortion Metrie
GMSD et 2013
Full Reference
Learned
Distortion Metric
NCPDiLapara 6
Full Reference
Learned
Distortion Metric
VSI Zhang al. 2014
Full Reference
Not Learned
Distortion Metric
MaDiLa Chandler 2010
Full Relerence
Not Learned
Distortion Metrie
Table 1: A list of the used evaluation metrics
Metrics We selected 16 metrics from multiple categories: full-reference metrics, no-reference metrics, learned metrics, and not-leamed metrics. We list these metrics and their corresponding categories in Table [ Some more recently proposed learned metrics (Zhang al. 20088|] Prashnani et al. 2018 Ding et al. 2020 Zheng et al. 2021] are believed to capture perceptual similarity better than other non-learned methods, we denote these metrics as perceplual metrics and the others as distortion metrics. Note that full reference metrics measure the differences between the compressed images and their corresponding ground truths, whereas no-reference metries (except FID) calculate the scores without specific ground truth. FID (Heusel et al.) 2017) is a special no-reference metric since the scores are calculated by some divergence between compressed images and ground truth distributions. Similar to (Mentzer et al., (2020), we calculate FID by segmenting images into non- overlapping 256 x 256 resolution patches for small test sets (S 100 images).
Test Data To support compression quality assessment, we consider following datasets with necessary preprocessing: 1. Kodak (Franzen 2013): The data consists of 24 high-quality images at 768x512 (512x768) resolution. We do not evaluate the FID score of the dataset as 24 images only yield 144 image patches. 2. Tecnick (Asuni & Giachetti] 2014):: We use 100 natural images with 600x600 resolutions and we downsample these images 512x5 resolution. 3. DIV2K (Agusts- son & Timofie 2017): The validation set of this dataset contains 100 high-quality images. We resize the images with the shorter dimension being equal to 768px. Then, each image is center-cropped to 768x768 squared shape. 4. COC02017 (Lin et al. 2014): For this dataset, we extract all test images with resolutions higher than 512x512 and resize them to 384x384 resolution to remove compression artifacts. The resulting dataset consists of 2695 images. 5. ArtBench (Liao et al.) (022): We use this dataset to conduct an out-of-distribution test, as it comprises 60000 images of artwork from 10 different artistic styles. We randomly select 1800 256x256 images from the surrealism style.
Model Training We use the Vimeo- (Xue et al., 2019) dataset to train our model, consisting of 90,000 clips of 7-frame sequences 448x256 resolution collected from vimeo.com This dataset is widely used for video compression research. We randomly select one frame from each clip and crop the frame randomly to 256x256 resolution in each epoch. At the beginning of training, we warm-up the model by setting À = 10-4 and keep it running for around 500,000 steps. Then, we increase to (0.01 28, 0.0256, 0.0512), respectively, and keep the model running for another 1 1,000,000 steps until the model converges. For the models with & O, we fine-tune the pretrained model with 0 for another 500,000 steps. We use Nuin 20000, batch size=4 and the Adam (Kingma & Ba 2014] optimizer in all cases. The learning rate is initialized as Ir = 5 x 10-5 and then declines by 20% every 100,000 steps until Ir 2 10-5
4.1 BASELINE COMPARISONS
Baselines and Model Variants We showed two variants of our CDC model. Our first proposed model is optimized in the presence of an additive perceptual reconstruction term at p = 0.9. Here, used DDIM stochastic decoding following XN ~ NN II)) with y 0.8 to reconstruct the images. The other proposed version is the base model, trained without the additional perceptual term (p- 0) and using deterministic decoding with xx O. We used 500 iteration steps for

---

Under review as a conference paper at ICLR 2023
HEC (fleeta 3330)
- MS-5MM (Cheeg 2000)
MS (Wingg 2009
CDC (ilonettadid (punt
(Minnc
BPC
FIDL
CKDN
SSIM
GMSDL
100
75-
0.381
6.93)
50-
0L573
35
0.070
0.90
64921
0.5
0.
0.4
MUSIQI
955-51M1-
1923
0.15
[PIPS]
DRCNN
CWV-55IMT
VSH
0122
0,500
0.855
0.2
012
0.4
0.2
0.
DI
+SIMT
PSNK
MAD.
0.22
0.00
12.5
0.30
33.0
0.5
0.5
0.2
0.4
0.6
0.2
0.0
0.2 0.5 0.5
Figure 2: Tradeoffs between bitrate (x-axes, in bpp) and different metries (y-axes) for various mod- els tested on DIV2k (see Supplement G for other data sets). We consider both perceptual (red frames) and distortion metrics (green frames). Arrows in the plot titles indicate whether high (f) low (L) values indicate a better score. As an overall tendency, we find that CDC (proposed) in its basic version (deterministic, without finetuning LPIPS) compares favorably in distortion metrics, while CDC with stochastic decoding and added LPIPS losses performs favorably when focusing on perceptual metrics.
decoding. As discussed below, we find that this base version performs better in terms of distortion metrics, while the stochastic and LPIPS-infomme version performs better in perceptual metrics.
We compare our method with several recent neural compression methods that are considered state of the art. The best reported perceptual results obtained by HiFiC (Mentzer al. 2020). This model is optimized by an adversarial network and employs additional perceptual and traditional distortion losses (LPIPS and MSE). terms of rate-distortion performance. two current state-of-ther art models include MS-GMM (Cheng et al. 2020) and NSC (Wang et (202)). Both are improve over the MSE-traine Mean-Scale Hyperprior (MS-Hyper) architecture (Minnen al. 2018). For comparisons with classical codecs, we also compare against the HEVC-based BPG codec.
Figure shows the tradeoff between bitrates and image quality metrics on DIV2k dataset (see Sup- plement G for other data sets). Baseline models have dashed lines, and proposed ones (CDC) have solid lines. We will discuss subfigures according to their metric types, indicated by the frame color.
Perceptual Metrics (red). The red subplots contain perceptual metrics Table Cheng et al. (2020) and Wang et al. (2022) show high scores in MUSIQ, but show subleading performance in other metrics. HiFIC is a much stronger baseline, but CDC(p = 0.9) still outperforms all models in 4 out of 7 metrics. The fact that HiFiC performs better than CDC in LPIPS may be explained by the fact that CDC has access to both ground truth and reconstructed images at training time, while CDC only has access to stochastic estimates of the latter.

---

Under review as a conference paper at ICLR 2023
SILP SHOOTER
SHROTER
(a) Ground Truth
(b) CDC( = 0.9) 0opp=0.20S)
(c) HiRAC (bpp=0.211)
(d) Ground Truth
(e) CDC(p 0) (bpp=0.228)
(i) MS-GMM (bpp=0.294)
Reconstructed Kodak images at comparable bitrates. 1s row: compared to HiFiC, our model more accurately retains the blue colors in the English letters and preserves more textures around the letters. 2nd row: compared to MS-GMM, the diffusion model preserves more texture details (the curtain inside the window).
Figure 3:
Distortion Metries (green). Green subfigures include metrics listed as distortion-based in Ta- ble "Cassical" neural compression models (Minnen et al. 2018 Wang et al., 2022 Cheng et al. 2020) directly target MSE distortion by minimizing an ELBO-objective with Gaussian de- coders. In contrast, our approach is not optimized for this task. Surprisingly, our base model CDC(p = 0) still gives relatively good results: it has an on-par distortion performance with the best baselines in 7 out of 9 metrics and the best score in CW-SSIM. As we expected, our model underperforms in PSNR score. Overall, we find that the deterministic version of our algorithm outperforms its stochastic variant in terms distortion metrics.
is widely acknowledged that current distortion-baset metrics are not perfectly suited for assessing compression performance. For example, translating all pixels in an image by single pixel leads to an unperceivable distortion, but large MSE (Yang et al. 2022b). We therefore consider first group of perception-basee metrics more meaningful. In this aspect, stochastic diffusion- baseed decoding scheme achieves very promising performance. Especially its strong performance in terms of FID, of the most widely-adopted perceptual evaluation schemes (Ho al., 2020 [ong &
Ermon 2019 Mentzer et al. 20200 Brock et al. 2019 Song et al. 2021a seems promising.
While the diffusion model's tendency to hallucinate high-frequency details may be visually appeal- ing, it harms MSE distortion which encourages over-smoothing. This over-smoothing behavior is
shown in Figure where MS-GMM shows less details in the reconstruction compared CDC.
4.2 STOCHASTIC DECODING
Our model allows both stochastic and deterministic decoding by varying the noise level in the image decoding process. Since stochastic decoding is unintuitive and typically not desirable, one can make the decoding process still reproducible by using fixed random seed. Additionally, we also consider DDPM (Ho et al., 2020) schemes that adds noise perturbation every decoding step.
Figure 4 compares the compression performance of different decoding schemes in 4 perceptual and traditional metrics (We note that the metrics are sufficient to reflect the overall trend, so we omit the others due to the page limit). The plots show that larger improves perceptual quality over the deterministic variants but bring down distortion scores when y S 0.8. also appears that the value 0.8 yields the best perceptual scores and visualization quality. larger only leads to worse distortion but also perceptual quality. We also observe that DDPM cannot do better than the original DDIM scheme when using the same
8

---

Under review as a conference paper at ICLR 2023
Catariatss Decddig
sucurs dwoding (DOPM) 64
Swerant Caooiing (DOPN
Sodlwi Cueckig (0CTM
tocurs Owoding (900)
Dacoding
RIDL
PIEAPP!
LPIPS]
DISTS,
0.5
0.16
50
0.7
0.12
0.5
0.00
30
0.5
D403
0.4
MSIM
55IM
SSIM
0.5
SSIM
4.5
0..55
0.00
0191
0.50
0191
0.94
(a) Ground Truth
(b) Deterministic
(c) DDIM 0.6 (d) DDIM 0.8
(c) DDIM
Figure 4: Quantitative (top figure) and qualitative (bottom figure) comparison of deterministic and stochastic decoding methods. Deterministic decoding typically results in smoother image recon- struction. By increasing the noise y used upon decoding the images, we observe more and more detail and rugged texture the face of the sculpture. Qualitatively, DDIM (7 0.8) seems show the best agreement with the ground truth image.
5 CONCLUSION & DISCUSSION
We proposed transform coddin---oonn----o based neural image compression approach using diffusion models. We use denoising decoder to iteratively reconstruct compressed image encoded by an ordinary neural encoder. Our loss term is derived from first principles and combines rate-distortion variational autoencoders with demoising diffusion models. We conduct quantitative and qualitative experiments to compare our method against several state-of-the-ant neural and classical codecs. Our approach achieves promising results in terms of the rate-perception tradeoff, outperforming several state-of- the-art baselines in four out of seven metrics, including FID. In terms of classical rate-distortion performance, our approach still performs comparable most highly-competitive baselines.
Future Research It's a known issue that iterative decoding can be slow compared to a normal decoder. Whether for transmitting data. However, we can also trade-off decoding speed against image quality by varying the decoding steps or using recent ideas for accelerating diffusion models (Salimans & Ho 2022). We also note that our work paves the path for future research on transform -oooornm--oonggggg based compression with diffusion models. Specifically, the compression performance can be further improved by incorporating the ideas from VAE-based coding methods such as more powerful prior distributions (Minnen et al. 2018 Minnen & Singh| 2020 Cheng et al., 2020), transformers (Zhu) et al. 2021). iterative inference (Yang et 2020) and attention modules (Cheng al. 20200) etc. Similarly, one can also improve the model from the diffusion model perspective, such as using different parameterizations (Ho al. 2022a Salimans & Ho 2022) discrete autoregressive
this is relevant depends the application, such the available bandwidth
diffusion distributions (Hoogeboom et al. 2021 Austin et al. 2021 Lee et al. 202T) and latent
diffusion (Rombach et al. 2022).

---

Under review as a conference paper at ICLR 2023
REFERENCES
Eirikur Agustsson and Radu Timofte. Ntire 2017 challenge on single image super-resolution: Dataset and study. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, July 2017.
Eirikur Agustsson, Michael Tschannen, Fabian Mentzer, Radu Timofte, and Luc Van Gool. Gen- erative adversarial networks for extreme learned image compression. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 221-231, 2019. 2019.
N. Asuni and A Giachetti. Testimages: large-scale archive for testing visual devices and basic image processing algorithms, stag smart tools & apps for graphics conference, 2014. 2014.
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 14:11791-17999 2021.
Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. Variational im- age compression with a scale hyperprior. International Conference on Learning Representations, 2018.
Jean Bégaint, Fabien Racapé, Simon Feltman, and Akshay Pushparaja. Compressai: a py- torch library and evaluation platform for end-to-end compression research. arXiv preprint arXiv:2011.03029, 2020.
Fabrice Bellard. Bpg image format. 2018. URL [RRLttts:/bbellard.ooog
Yochai Blau and Tomer Michaeli. Rethinking lossy compression: The weddiitorttoo-ppereppoo tradeoff. In International Conference on Machine Learning, pp. 675-685. PMLR, 2019.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. International Conference on Learning Representations, 2019.
Zhengxue Cheng, Heming Sun, Masaru Takeuchi. and Jiro Katto. Learned image compression with discretized gaussian mixture likelihoods and attention modules. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7939-7948, 2020.
Cisco. Visual network index cisco. forecast and methodology. White Paper, 2017.
Keyan Ding, Kede Ma, Shiqi Wang, and Eero P Simoncelli. Image quality assessment: Unifying structure and texture similarity. IEEE transactions on pattern analysis and machine intelligence, 2020.
Gergely Flamich, Marton Havasi, and José Miguel Hermández-Lobato Compressing images by en- coding their latent representations with relative entropy coding. Advances in Neural Information Processing Systems, 13:16111-1114,, 2020.
Gergely Flamich, Stratis Markou, and José Miguel Hemández-Lobato. Fast relative entropy coding with a coding, arXiv preprint arXiv:2201. 12857, 2022.
Richard W. Franzen. True color kodak images. 2013. URL http::/z0k.us//rappiinn
kodak/]
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sher- jil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In
Z. Ghahramani, M. Welling, Cortes, N. Lawrence, and K.Q Weinberger (eds.), Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014. URL https:://ppooeeddssss.s neuripss cc cc/papper/211///11e (aa2100122211111353ee//aaooooe12
Google. An image format for the web; webp; google developers, 2022. URL https:// developers. le. stoooglle.cmm/pppe///eee./.
10

---

Under review as a conference paper at ICLR 2023
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840 -108440-651,,,, 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:22/0.02303, 2022a.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. arXiv preprint arXiv:2204.03458 2022h.
Emiel Hoogeboom, Alexey A Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans. Autoregressive diffusion models. In International Conference on Learning Rep- resentations, 2021.
Junjie Ke, Qifei Wang, Yilin Wang. Peyman Milanfar, and Feng Yang. Musiq: Multi-scale image quality transformer. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 5148-5157, 2021.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Ad- vances in neural information processing systems, 34:21696-21777, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1412.6988, 2014.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Mattco Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. IJC,, 2020.
Valero Laparra, Johannes Ballé, Alexander Berardino, and Eero P Simoncelli. Perceptual image quality assessment using normalized laplacian pyramid. Electronic Imaging, 2016(16):1-6. 2016.
Eric Cooper Larson and Damon Michael Chandler. Most apparent distortion: full-reference image quality assessment and the role of strategy. Journal of electronic imaging, 19(1):01 1006, 2010.
Sang-gil Lee, Heeseung Kim, Chachun Shin, Xu Tan, Chang Liu, Qi Meng, Tao Qin, Wei Chen, Sungroh Yoon, and Tie- Yan Liu. Priorgrad: Improving conditional denoising diffusion models with data-driven adaptive prior. arXiv preprint arXiv:2106.0 0644000666666.... 2021.
Cheuk Ting Li and Abbas El Gamal. Strong functional representation lemma and applications to coding theorems. IEEE Transactions on Information Theory, 64(1 (6(11)):967-6778 2018.
Peiyuan Liao, Xiuyu Li, Xihui Liu, and Kurt Keutzer. The artbench dataset: Benchmarking genera- tive models with artworks. arXiv preprint arXiv:2206.1 1404, 2022.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Fabian Mentzer, George D Toderici, Michael Tschannen, and Eirikur Agustsson. High-fidelity gen- erative image compression. Advances in Neural Information Processing Systems, 33:1 1913- 11924, 2020.
11

---

Under review as a conference paper at ICLR 2023
David Minnen and Saurabh Singh. Channel-wite autoregressive entropy models for learned image compression. In 2020 IEEE International Conference on Image Processing (ICIP), pp. 3339- 3343. IEEE, 2020.
David Minnen, Johannes Ballé, and George Toderici. Joint autoregressive and hierarchical priors for learned image compression. Advances in neural information processing systems, 31, 2018.
Ekta Prashnani, Hong Cai, Yasamin Mostofi, and Pradeep Sen. Pieapp: Perceptual image-error assessment through pairwise preference. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 1808-1817, 2018.
Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasorn Suwajanakom. Diffu- sion autoencoders: Toward meaningful and decodable representation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen, Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204006125, 2022.
Oren Rippel and Lubomir Bourdev. Real-time adaptive image compression. In International Con- ference on Machine Learning, pp. 2922-2930. PMLR, 2017.
Robin Rombach, Andreas Blattmann, Dominík Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2022.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint (.Xix:2222.005122 2022.
Mehul P Sampat, Zhou Wang, Shalini Gupta, Alan Conrad Bovik, and Mia K Markey. Complex wavelet structural similarity: new image similarity index. JEEE transactions on image process- ing. 18(11)233855-22000 2009.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrimm thermodynamics. In International Conference on Machine Learn- ing, pp. 2256-2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffussin implicit models. In Interna- tional Conference on Learning Representations, 2021a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representaations, 2021b.
Lucas Theis and Noureldin Y Ahmed. Algorithms for the communication of samples. In Interna- lional Conference on Machine Learning, pp. 21308-21328. PMLR, 2022.
Lucas Theis, Wenzhe Shi, Andrew Cunningham, and Ferenc Huszár. Lossy image compression with compressive autoencoders. International Conference Learning Representations, 2017.
Lucas Theis, Tim Salimans, Matthew D Hoffman,, and Fabian Mentzer. Lossy compression with
gaussian diffusion. arXiv preprint arXiv:2206.0888 2022.
Michael Tschannen, Firikur Agustsson, and Mario Lucic. Deep generative models for distribution- preserving lossy compression. Advances in neural information processing systems, 31, 2018
Gregory K Wallace. The jpeg still picture compression standard. Communications of the ACM, 34 (4):0-444, 1991.
12

---

Under review as a conference paper at ICLR 2023
Dezhao Wang, Wenhan Yang, Yueyu Hu, and Jiaying Liu. Neural data data-dependent transform for learned image compression. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 17379-17388, 2022.
Zhou Wang, Eero Simoncelli, and Alan C Bovik. Multiscale structural similarity for image quality assessment. In The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003, volume 2. pp. 1398-1402. Ieee, 2003.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE transactions image processing, 13(4):600- 612, 2004.
Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, and William T Freeman. Video enhancement with task-oriented flow. International Journal of Computer Vision (ICV), 186 1125, 2019.
Wufeng Xue, Lei Zhang, Xuanqin Mou, and Alan C Bovik. Gradient magnitude similarity deviation: A highly efficient perceptual image quality index. IEEE transactions on image processing. 23(2): 684-695 695, 2013.
Ruihan Yang, Prakhar Srivastava, and Stephan Mandt. Diffusson probabilistic modeling for video generation. arXiv preprint arXiv:2203. 00448991, 2022a.
Yibo Yang, Robert Bamler, and Stephan Mandt. Improving inference for neural image compression. Advances in Neural Information Processing Systems, 33:573-584, 2020.
Yibo Yang, Stephan Mandt, and Lucas Theis. An introduction to neural data compression. arXiv preprint 2ttiv:220220653333 2022b.
George Zhang, Jingjing Qian, Jun Chen, and Ashish Khisti. Universal pee-ddisoortoo-perrppppe representations for lossy compression. Advances in Neural Information Processing Systems, 34: 11517-11529, 2021.
Lin Zhang, Lei Zhang, Xuanqin Mou, and David Zhang. Fsim: A feature similarity index for image quality assessment. IEEE transactions on Image Processing, 20(8):2378-2886, 2011.
Lin Zhang, Ying Shen, and Hongyu Li. Vsi: A visual saliency-induced index for perceptual image quality assessment. IEEE Transactions on Image Processing, 23(10))4270-42)) 4281, 2014. doi: 10. 11110/99122014423400000
Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595. 2a.
Weixia Zhang, Kede Ma, Jia Yan, Dexiang Deng, and Zhou Wang. Blind image quality assessment using deep bilinear convolutional neural network. IEEE Transactions on Circuits and Systems for Video Technology. 30(1):36-47, 2018b.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Towards deeper understanding of variational autoencoding models. arXiv preprint /702::/702.0080 1702.08022 2017.
Heliang Zheng, Huan Yang, Jianlong Fu, Zheng- Jun Zha, and Jiebo Luo. Learning conditional knowledge distillation for Segraded-reference image quality assessment. Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 10242-10251, 2021.
Yinhao Zhu, Yang Yang, and Taco Cohen. Transformer-based transform coding. In International Conference on Learning Representations, 2021.
13

---

Under review as a conference paper at ICLR 2023
A PRETRAINED BASELINES
We refer to Bégaint et al. 2020) for pretrained MS- MS-Hyper and MS- models. For HiFiC model, we use the model implementer by a 3rd party researchers" Both models were sufficiently trained on natural image datasets (Xue et al. 2019 Kuznetsova et al. (020). For NSC (Wang et al. 2022) baseline, we use the official codebase and DIV2K training dataset train the model.
B ARCHITECTURES
Skip concatenation
X
INP
X
p(nl)
3 TransComv 12
3 Com 12
ResBlock
AesRlock
ResBlock
ResSBlok
ReeBlock
Liwen-ltentio
LinaaAttantion
TranaCorw 12
Conv ]2
Conv ]2
TransComv 12
Embedder (EMD)
Downsampling Unk
Upsamping Unit
Figure 5: Visualization of our model architecture
Figure Sigure5dessen describes our design choice of the model. We list the additional detailed specifications that we did not clarify in the main paper as follow:
The hyper prior structure shares the same design Minnen al. (2088) The channel number of the hyper latent y is set as 256.
We 3x3 convolution for most of the convolutional layers. The only exceptions the 1st conv-layer of the first DU component and the 1st layer of the 1st ENC component, where 7x7 convolution for wider receptive field.
i is embedded by a linear layer, which expand the I-dimensional scalar to the same chan- nel size as the corresponding DU/UU units. We then add the expanded the inter- mediate ResBlock of each DU/UU unit.
14

---

Under review as a conference paper at ICLR 2023
C COMPUTE
We provide information on the model parameter size of the proposed model and baselines, and the corresponding time cost of running full forward pass in T We run benchmarking on a with a RTX Titan GPU. We execute each code snippet 24 times to encode and decode 24 images from Kodak dataset and calculate the average running time. We do not include entropy-coding time here as these models can share the same entropy- corroppy-cddngggggo method.
CDC (100 steps)
CDC (500 steps)
HiFiC
MS-GMM
NSC
Model Size (Mega Bytes)
214.5
214.5
725.9
106.4
211.9
Inference time (Seconds)
10.91
54.92
1.01
0.52
0.86
Table 2: The memory usage and time consumption of a forward pass.
Our model also shows significantly better memory efficiency than HiFiC, but is inevitable that our diffusion models have slow decoding speed due to the iterative denoising process. We note that the decoding speed is proportional to the number of decoding steps. As shown Figure [6] decoding the image with 100 steps do not have observable qualitative difference to 500-step decoding, but can be times faster.
Figure 6: Qualitative comparison of models with different number of decoding steps. Left: 100
steps; Right: 500 steps
15

---

Under review as a conference paper at ICLR 2023
D SUPPLEMENTAL ABLATION STUDY
By varying the trade-off term we can train a model either prefer perceptual quality or traditional distortion performance. Figure shows the rate-distortion curves for COCO dataset with determin- istic decoding scheme. We consider four values in the study (0, 0.32, 0.64, 0.9). The result shows that larger p leads better perceptual quality but worse distortions in most cases. We also note that 0.9 is not available as perceptual quality can not be perceivably improved and there is also risk that the training may fail.
COC (oas))
CDC - 0.22 (ozs)
CDC 3/14 (cuns
CDC C9 (cars)
FID)
CKDN
SSIM
GMSD)
30-
0.015
000
0.000
0.00
002
112
11.
04
0.6
0.2
11.
115
16.
0.2
0.4
016
11.5
062
0-4
0.6
0.8
PIEAPP]
MUSIQU
NLPDJ
113
60
0.31
0.6
0.00
0.15
06-
0.00
LPIPS1
DBCNNT
55IM5
VS1
0.993
24
0.00
0.385
0/
0.6
0.5
0.6
DI5T5.
FSIM
PSNR"
MAD.
103
63-
0.07
0.00
1.1
0.5
0.2
11.
115
0.2
11:5
11.5
16.
0.1
0.5
0.50
Figure 7: rate-distortion curves with different p values
16

---

Under review as a conference paper at ICLR 2023
E ADDITIONAL VISUALIZATION OF THE COMPRESSED IMAGES
Figure 8: CDC(p = 0.9), bpp=0.35. Left: deterministic; Right: stochastic
Figure 9: CDC(p 0.9), bpp=0.205, Left: deterministic; Right: stochastic
Figure 10: CDC(p = 0.9), bpp=0. 145.
145. Left: deterministic; Right: stochastic
17

---

Under review as a conference paper at ICLR 2023
Figure 11: CDC(p = 0), bpp=1.20. Left: deterministic; Right: stochastic
Figure 12: CDC(p 0), bpp=0.636. Left: deterministic; Right: stochastic
Figure 13: CDC(p 0), bpp=0.228. Left: deterministie; Right: stochastic
18

---

Under review as a conference paper at ICLR 2023
F DECODING VARIABILITY
Figure 14: We stochastically decode the same latent variable with the same y = 0.8 but different random seed for Xx ~ ~N(0, (I). The decoded images only have small distinctions but we can still find there are slight detailed textural differences.
19

---

Under review as a conference paper at ICLR 2023
HECC (flerte 3333)
- MS-5MM (Cheeg 2000)
MS (Kingg 2009
CDC (ilonettrtie (punt)
(Minnce
BPC
FID)
CKDN
SSIM
GMSDL
0.
0.01
6033
0.013
0.80
0.0
96
MUSIQ
1.5
0.54
[IISS]
DRCKN
CW--5IMT
VSH
0.2
0.93
0.22
0.101
0.9
0.4
0.2
02
0.4
3.0
0.5
DI
FSIMT
PSNK
MAD.
127
11:20
11.50
35.01
100
11.5
33.11
53
17.5
0.2
01.6
0.2
0.0
0.5
0.2
0.4
0.0
Figure 15: Rate-Distortion oooo-sttorttin(eeepptioonn for COCO dataset
G ADDITIONAL RATEDDDSTORRTOOOPPEEEPTTOO RESULTS
20

---

Under review as a conference paper at ICLR 2023
(Cheeg 2808)
(Mtng 2009
(((ptectodd) (parg
Myee (Mianc 2018)
BPC (HEVC]
CDC (chuemnnisst) [ouss
HDL
CKDN1
SSIMT
GMSD)
0.073
72
0.393
0.00
0.50
06
PIEAPP]
MUSIQ
MS SSIM
NLPDJ
70
0.75
0.00
00
0.2
0.1
0.6
0.5
012
11.1
1.5
0.5
0.2
01.1
0.5
0.5
0.3
0.5
0.6
0.5
[PPSS]
CXU-55IM|
VSH
0.220
0.50
0.995
0.:2
20
0.985
0100
0.903
0.5
DISTS)
FSIM1
PSNK"
MAD.
0.00
103
37.5
73.
0.04
Figure 16: Rate- Disttootiion(eercptiinnn for Tecnick dataset
21

---

Under review as a conference paper at ICLR 2023
(Cheeg 2090)
(Mtng 2009
(((pttetadd) (parg
My.H (Minnne 2018)
BPC (HEVC]
CDC (chuemnnisst) [ouss
HDL
CKDN1
SSIMT
GMSD)
200
0,001
[..3
56-
0.70
0:4
0.8
0.4
PIEAPP]
MUSIQI
MS SSIM
NLPDJ
0.90
1.0
0.90
0.23
0.5
0.2
0.1
0.5
0.5
0.3
0.1
016
0.5
01
0.1
0..6
0.6
0.2
0.0
3.5
15.
|PIPS
CXN-55IM
VS1
1.81
0.3
0.1
20
0334
0095
01.6
0.6
DISTS)
FSIM1
PSNR"
MAD.
33.11
0832
32.5
103
0.10
Figure 17: Rate- Booo--ttottionPPreeeeee for Kodak dataset
22

---

Under review as a conference paper at ICLR 2023
(Cheeg 2808)
(Mtng 2009
(((otcttdd)) (parg
MyeH (Mixne 2018)
BPC (HEVC)
CDC (chhemnnistt)) [ouss
HDL
CKDN1
SSIMI
GMSDD
01103
0.000
1.05
20
0.081
0.2
3.2
0.2
0.
PICAPP]
MUSIQI
MS SSIM
NLPDJ
055
0.00
0.15
0.5
0.50
0.2
1/11
056
108
3.2
0111
0.6
0.5
0.3
01:
0.5
0.55
012
0.4
00.0
0.0
PIPSS]
DRCNNT
CWN-55IM1
VSH
0.:3
0.00
0.97
0.95
0.00
3.2
DISTS)
FSIM1
PSNR"
MAD.
14.20
0.00
12.5
11
0.30
0.855
23.11
0.8
Figure 18: Satee-DistrtionnPeeeeppe for ArtBench(surrealimm) dataset.
23


Reviews:
[{"review_id": "rsrtbUgM-e", "reviewer": "Reviewer_NPK1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this work, the authors propose a novel model architecture for lossy image compression. Specifically, they propose combining a VAE encoder architecture with a conditional diffusion decoder to compress image data. The authors derive an Evidence Lower Bound (ELBO) for their model, introducing a standard weighting coefficient for allowing different rate-distortion tradeoffs. They train their proposed model on the Vimeo-90k image dataset, and test it on five different image datasets, using a variety of sixteen different assessment metrics, showing promising results when compared with state-of-the art approaches.", "summary_of_the_review": "In summary, I find the architecture proposed in this work an interesting and useful one. This architecture enables entropy coding to be used in conjunction with diffusion decoders, which is a good contribution to the literature, even though it does not exhibit the best performance across all the metrics used by the authors. The extensive evaluation done in the paper is also welcome.\n\nHowever, I also found that the architecture was not particularly well explained, and the paper would benefit considerably from an effort to clarify how it operates, especially since this is the main contribution of this work. Currently, I would mark the paper as marginally below the acceptance threshold, giving it a score of 5, but encourage the authors to make an effort to improve their exposition, in which case I would be happy to consider improving this score. I also expect that the authors will adjust their claims regarding competitive performance across all evaluation metrics, which does not appear to be exactly the case.", "strength_and_weaknesses": "## Strengths\n\n**Novel image compression architecture:** The authors propose combining a VAE encoder together with a conditional diffusion model for performing image compression. This is a novel architecture which allows diffusion models to be used in conjunction with entropy coding. Specifically, the latent variable of the VAE can be coded using entropy coding, and the diffusion model can be conditioned on this latent variable to decode the image. This is an interesting idea since it enables high-performance diffusion models to be used with entropy coding, which is both fast and efficient (in terms of the code lengths it produces), rather than relying on relative entropy coding, which can be either prohibitively slow, restrictive in terms of its assumptions, or inefficient in terms of code length.\n\n**Competitive performance and extensive evaluation:** The authors evaluate their architecture on several different test datasets, using a variety of evaluation metrics. On some of these metrics, their architecture outperforms the most competitive models which they evaluate, while on others it performs in par or markedly worse than many of the other models. While the proposed method does not outperform all competitive models across all metrics, the extensive evaluation which the authors conduct is very welcome. While they could have picked the subset of these metrics which is favourable towards their model, they have included a variety of metrics making the comparison more extensive and transparent.\n\n## Weaknesses\n\n**Significance of the contribution:** While the authors' architecture does exhibit performance competitive to that of several strong methods from the literature, it also performs markedly worse across several of the metrics used. As this is the sole contribution of the paper, it makes one wonder whether this is yet another compression architectures to the long list of existing methods. That being said, their architecture does enable entropy coding with a diffusion model as the decoder, and with further improvements this architecture could yield more promising results. However, I think that this work would benefit by addressing the following points:\n\n1. The authors are advised to modify their assessment of how their model stacks up against alternatives. For example, in the conclusion they say \"Our approach yields competitive rate-distortion(perception) performance against all baseline models\", a claim which is not true since their model actually performs worse than state-of-the-art methods across several of the evaluation metrics. A more measured and nuanced position here (as well as throughout the paper) is necessary.\n2. The authors could outline their ideas on further improving their architecture, to improve its compression performance. If, for example, the authors do not foresee any improvements on the performance that could be extracted from their architecture, then their contribution is of limited interest since it consists solely of this architecture and, presently, it is not entirely obvious that this architecture is clearly preferable than the alternatives they evaluate.\n3. A more detailed discussion on the various evaluation metrics and their relationships to specific applications, would be welcome. For example, the authors could highlight the types of applications in which the evaluation metrics correlate with overall performance, and the kinds of settings in which their architecture might be more or less favourable.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "rX_n1fOInk", "reviewer": "Reviewer_Tnww", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a lossy image compression algorithm. The framework is similar to previous work of transform coding built on VAEs, and the proposed method replaced the the decoder with a conditional diffusion model. The paper provided comprehensive evaluations on various datasets and a number of metrics, and demonstrated that the proposed approach is compatible, and sometimes better than the state of the art.", "summary_of_the_review": "The paper is well written, and provides valuable empirical comparison between the proposed and existing approaches on neural image compression. However, the novelty of this work is limited, and its improvement over baselines is marginal.", "strength_and_weaknesses": "Strength\n\n* Combining diffusion models and image compression is both interesting and expected. While previous works have explored this basic idea to a certain extent (as reviewed in this paper), this paper explored it most thoroughly to my knowledge. The comprehensive experiments provided valuable comparison between algorithms.\n\n* The proposed perceptual loss is well placed in the diffusion model, which might find its use beyond the proposed approach. It is formally simple, and demonstrated to be able to substantially improve perceptual quality.\n\nWeakness\n\n* The paper rightly claimed their \"competitive\" performance against baseline models. However, the at best marginal improvement is hard to justify the significantly increased computational cost (e.g., 500 diffusion steps in decoding).\n\n* The perceptual distortion loss could be better explored. E.g., It would be interesting to see how the rate-distortion trade-off is affected by different weights of the perceptual loss, but the paper only presented results from $\\rho = 0$ and $\\rho = 0.9$.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "TxeVqbL3Xb3", "reviewer": "Reviewer_9ej8", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper examines the use of conditional diffusion models (at the decoder side) for end-to-end optimized lossy image compression.", "summary_of_the_review": "With a much clearer algorithm description, improved rate-distortion performance, and computational complexity analysis, this paper has a chance to get in.", "strength_and_weaknesses": "Strengths:\n\n1. The use of conditional diffusion models in the context of neural image compression is somewhat novel.\n\n2. The experimental results, especially using more than 15 image quality metrics, are comprehensive.\n\nWeaknesses:\n\n1. The proposed method is not clearly described. \n\na) Why the authors choose to start the derivation from the negative data log-likelihood (the inequality above Eq. (7)), which is more relevant to lossless image compression? \n\nb) When and how do the index $n$ and $\\mathbf{x}_n$ come into play in Eq. (9), provided that there are no such terms on the LHS? \n\nc) Why is $\\log p(\\mathbf{z})$ instead of $\\log P(\\mathbf{\\hat{z}})$ in the rate-distortion objective? Are we supposed to compute the discrete entropy as a proxy of the expected code length?\n\nd) How to learn the quantization centers and how to model $\\log P(\\mathbf{\\hat{z}})$ are not mentioned.\n\ne) How to set the trade-off parameter $\\lambda$ is not given. At lower bits, do we also need to reduce the dimensionality of the latent representation to encourage bit savings?\n\n2. The rate-distortion performance is not outstanding compared to existing methods, e.g., HiFiC and MS-GMM, under standard and widely used metrics - PSNR and SSIM. Of particular interest, the proposed method jointly optimized for LPIPS (with $\\rho=0.9$) under-performs HiFiC under the same metric - LPIPS. The authors may want to give some explanations.\n\n3. The computational complexity in terms of the number of model parameters, and the encoding and decoding time should be provided to gauge whether the improved rate-distortion performance is worthwhile.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CBqYJKM5ji", "reviewer": "Reviewer_ocNf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new image compression model based on the paradigm of neural image compression. The main novelty is to introduce a diffusion based decoder, instead of the traditional Gaussian or Laplacian decoders. Some experiments are conducted to verify the effectiveness.", "summary_of_the_review": "In summary, the motivation of this work is clear, but it lacks some important analysis on the proposed method. Additionally, the experiments are insufficient. ", "strength_and_weaknesses": "Strengths:\n1. The idea is very intuitive and impressive. To enhance the capability of the decoder in image compression, it replace the Gaussian and Laplacian decoder with a powerful diffusion model.\n2. The paper adopt as many as possible metrics (16) to evaluate the performance, which comprehensively reflects the advantages and disadvantages of the proposed method. \n\nWeaknesses:\n1. In this paper, $z$ is called semantic latent variable while $x_{1:N}$  texture latent variable. Why does $z$ and $x_{1:N}$ control the semantic and texture information respectively? If you could provide some visual proofs, it would be better.\n2. The hyper-parameter $\\rho$ makes a trade-off between the diffusion lower bound and the perceptual loss. More quantitative and qualitative comparisons under different $\\rho$ values should be added to analyse its influence.\n3. More visual results under different bitrates should be also provided.\n4. The experimental comparisons are not sufficient enough, some recent works are not compared. E.g,\n    Neural data-dependent transform for learned image compression, CVPR 2022.\n    Enhanced invertible encoding for learned image compression, MM, 2021.\n    Transformer-based transform coding, ICLR, 2021.\n  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "1SzG5NBCkN", "reviewer": "Reviewer_aJ8A", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a lossy image compression method using the condition diffusion model. This is an end-to-end framework based on a condition diffusion model. This is a new attempt to use the diffusion model for lossy image compression. Extensive experiments are conducted to show the advantages and robustness of the proposed method. ", "summary_of_the_review": "The paper has some significant novelty flaws and the technique and experiment are not convincing enough, which are the most important factors in my rating. ", "strength_and_weaknesses": "Strength\n\n+New attempt\nThis work tries to use the diffusion model for lossy image compression. As claimed, this is the first of this kind.\n\n+Extensive experiment\nTo show the effectiveness of the proposed method, 5 datasets, and 16 IQA metrics are used. The proposed method achieves promising performance. \n\nWeakness\n\n-Unclear motivation\nThis is not clear why the use of the diffusion model is necessary. What is the advantage of the diffusion model for this task? To my best knowledge, the diffusion model is hard to preserve fidelity while fidelity is crucial for image compression. Besides, the diffusion model is slow, which also challenges the requirement of image compression.\n\n-Limited novelty\nIt is intuitive to compress the image into the latent space. The diffusion model is a choice.  However, the novelty of how to use the diffusion model is limited in the work. All these ideas are intuitive or they have been used in previous related works. \n\n-Insufficient analysis and comparison\n1) The efficiency of different methods should be discussed. \n2) Under different IQA metrics, the proposed with different parameter settings obtains better performance. How to choose the parameter when using the proposed method in practical applications？\n3) More methods published after the 2020 year should be included for comparison.\n\n-Other issues\n1) It seems that the proposed method can only process the image with a size of 64X. \n2) The ground truth image shown in the paper is not high-quality. It would be good if high-quality images can be used to show the advantages of the proposed method. For current visual comparison, it is hard to distinguish the performance of different methods as the ground truth images are still low-quality. \n3) The paper is not well-written. Some content is hard to understand. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject