Neural Image Compression with a Diffusion-based Decoder
Authors: Noor Fathima Khanum Mohamed Ghouse
Jens Petersen
Auke J. Wiggers
Tianlin Xu
Guillaume Sautiere
Venue: Submitted to ICLR 2023 (2023)
Keywords: neural
image
lossy
compression
diffusion
gan
perceptual

Abstract: Diffusion probabilistic models have recently achieved remarkable success in generating high quality image and video data. In this work, we build on this class of generative models and introduce a method for lossy compression of high resolution images. The resulting codec, which we call \emph{DIffuson-based Residual Augmentation Codec (DIRAC)}, is the first neural codec to allow smooth traversal of the rate-distortion-perception tradeoff at test time, while obtaining competitive performance with GAN-based methods in perceptual quality. Furthermore, while sampling from diffusion probabilistic models is notoriously expensive, we show that in the compression setting the number of steps can be drastically reduced.

Under review as a conference paper at ICLR 2023
NEURAL IMAGE COMPRESSION WITH A DIFFUSION- BASED DECODER
Anonymous authors Paper under double-blind review
ABSTRACT
Diffusion probabilistic models have recently achieved remarkable success in gen- erating high quality image and video data. In this work, we build on this class of generative models and introduce a method for lossy compression of high resolu- tion images. The resulting codec, which we call Diffuson-basel based Residual Aug- mentation Codec (DIRAC), the first neural codec allow smooth traversal of the rate- diiee-istottiin--o cistortion-perception tradeoff at test time, while obtaining competitive performance with GAN-based methods in perceptual quality. Furthermore, while sampling from diffusion prohabilistic models is notoriously expensive, we show that in the compression setting the number of steps can be drastically reduced.
INTRODUCTION
As new set of generative approaches, denoising diffusion probabilistic models (DDPMs) (Sohl- Dickstein et al. 2013) have recently shown incredible performance in the generation of high- resolution images with high perceptual quality. For example, they have powered large text-to-imaae models such as DALL--E 2 (Ramesh et al.] (022) and Imagen (Saharia et al. 2022b), which are ca- pable of producing realistic high-resolution images based on arbitrary text prompts. Likewise, diffu- sion models have demonstrated impressive results on image-to-image tasks such as super-resolution (Saharia et al. 2022c Ho et al. 2022a). deblurring (Whang et al. 2022) or inpainting (Saharia al., 2022a). in many cases outperforming generative adversarial networks (GANs) (Dhariwal & Nichol, 2021 Our goal in this work is leverage these capabilities in the context of learned compression.
Neural codecs, which learn to compress from example data, are typically trained to minimize dis- tortion between an input and a reconstruction, as well as the bitrate used to transmit the data (Theis) et al. 2017). However, optimizing for distortion may result in blurry reconstructions, and recent work has identified the importance of perceptual quality. In particular, Mentzer et al. (2020) demon- strated that GAN-based image codec is able to output reconstructions that look more realistic to human observer than a distortion-optimize tradeoff between perceptual quality and fidelity is fundamental one (Blau & Michaeli 2019), and finding a good operating point is not trivial and likely application-dependent
codec, at the cost of some fidelity to the original. This
Ideally, one selects this operating point test time. Although adaptive rate control is commonly used, only the work lwai et al. (202T) allows to balance distortion and perceptual quality dynam- ically. To the best of our knowledge, our work proposes the first neural codec that can navigate the full aute-distoriion-perepptioo
tradeoff at test time. An earlier diffusion-based codec has been proposed by Theis al. (2022), but their encoding scheme prohibitively expensive. Concurrent work by Yang & Mandt (2022) demonstrated a more practical diffusion- based codec, but we show that different design choices allow us outperform their model by a large margin.
Our approach, called Diffusion Residual Augmentation Codec (DIRAC), uses a base codec to pro- duce an initial reconstruction with minimal distortion, then enhances its perceptual quality using a denoising diffusion probabilistic model. By varying the number of sampling steps, the DDPM can smoothly interpolate between a high fidelity output and one with high perceptual quality, Combined with multi-rate base codec, this enables a user to navigate the axes of rate-distortion and distortion- perception independently using separate control parameters. Moreover, by making a high fidelity initial prediction first, DDPM sampling can be stopped at any point, allowing the codec to operate more efficiently. We show that we can alter the sampling schedule at test time, and that we obtain strong performance with 20 sampling steps or less.

---

Under review as a conference paper at ICLR 2023
In summary, this paper makes the following contributions:
We demonstrate one the first practical iiffusion-based codecs for high-resolution image compression, with competitive performance in both distortion and perceptual quality.
2. We show that this codec enables fine dynamic control over all independent axes of the rate- distortion-perceptionn tradeoff test time. To the best of our knowledge, this is the first neural codec with this ability.
3. Although DDPMs are notoriously expensive to sample from, we show that our design choices enable substantial speedup, allowing sampling in 20 steps or fewer.
2 METHOD
2.1 DENOISING DIFFUSION PROBABILISTIC MODELS
Denoising diffusion probabilistic models (DDPMs) (Sohl-Dickstein al. 2013 Ho al. 2020
latent variable models in which the latent variables X....., are defined as T step Markov chain with Gaussian transitions. Through the Markov chain, the forward process of DDPMs gradually corrupts the original data Xo:
T (x1TT Xo) = II ((xx(xx-1) g(xe(xx-1) = NX(x: V 11xx-1,
(1)
Here, (xx, 1x- is Gaussian distribution that models step x, conditioned the previous step Xt-1. is also possible to further condition these transitions on Xo (Song et al. (0020). The vari-
ances 3, are typically chosen empirically and referred the noise schedule, although they be learned as well (Dhariwal & Nichol 2021).
The key idea of DDPMs is that, if the corrupted data step T follows a distribution which permits efficient sampling, we can construct a generative model by reversing the forward pro- cess. For example, the series q(xe xt-1) in Eq. ( converges to standard normal distribution for appropriately chosen St. This can be easily seen in the closed-fome expression for Eq. ! X Vo v1 - age. Here E N(e;; O, I), and we define == - B, and ã₁ = II Os- DDPMs are therefore generative models that learn to recover the original data by modeling the reverse transitions:
(X( p(xT) I (x 1 po (x(x-11x4 N(x t), Eo (x.,
Here pe ( xt). parametrized by weights 0, models a Gaussian distribution that reverses state X back to X and the base distribution is defined as the standard normal p(xr) == -=>(xx;;)) O.T). The variance Eg usually chosen be I and can be learned or chosen empirically. Predictions from the DDPM can then be generated by iteratively applying the model and following the transitions (x) We give more details on the sampling procedure in Section 2.3 and Appendix A.5
The training objective can then be constructed in the context of variational inference, by approximat- ing the posterior distribution of the latent variables pe (x1.T (xo) by the forward process 1(x1.T xx))- Using Bayes' rule to obtain (xxt_1/Xe,30 Xo), maximizing the evidence lower bound (ELBO) pe(Xo) then equivalent to minimizing the sum of T Kullback-eiiler (KL) divergences. It can be shown that this loss can then be rewritten simple minimization between true data and denoising prediction (more details in Appendix[A.1]
CDMPM
wil|xo (x(xx,t)]
1.x0.e
Here go (x):))) f) is a model that directly predicts Xo from xt, and w, is a weighting factor. Note that this integrates into the expectation; while the full loss should sum over all f, it is common
2

---

Under review as a conference paper at ICLR 2023
CLCC lest fu lrnss
37
TNO
better
H0u कt TH
36
DIRAC-1 (110AP)
35
hpp
Hypertprior
Tiu a Th
TN
33
x
DIRAC-80 \ 1100b50
Reforence 550
Hi at aR TNI
DIRAC 100 MIMAAAAE
1u at TH -H--JOO
aua RRY- JoO
HIFICH11240
Figure I: Top left: distortion- distortion-perceetion curves for DIRAC, hyperprior baseline, a HiFiC baseline, and the standard codec VTM, showing that our codec is able navigate a wide range of distortion- perception tradeoffs dynamically at test time (increasing marker size indicates number of diffusion steps). Bottom and right: Example reconstructions for various operating points, as well the HiFiC baseline (cross). HiFiC is excellent at synthesizing textures, which often matches the contents of image well (e.g. red/leftmost crop), but in some cases the textures appear unnatural (e.g. text in green/center crop). Image contents may be more perceptually pleasing when slightly blurrier (e.g. green neon text in blue/rightmost crop), so that a prediction with high fidelity (DIRAC-80, circle) also has higher perceptual quality. Our codee can adjust this tradeoff dynamically, e.g. match user preferences. Rate matching of models done using cubic spline interpolation.
practice to instead sample and perform Monte-Carlo integration over time, Ho et al. 2020) found that reweighting the individual loss terms improve perceptual quality. The theoretically derived terms for We become very for large small (see Appendix A. and Appendix A.3). We instead choose w, to balance all terms evenly.
2.2 NEURAL DATA COMPRESSION
Neural network based codecs are systems that learn compress data from examples. Most mod- neural codecs variations of compressive autoencoders (Theis et al.] 2017). which transmit data using an autoencoder-like architecture consisting three components. With a slight abuse of notation, let x tized latent variable y. Given this latent variable, a neural decoder Do produces a reconstruction & Typically, a neural prior Po models the distribution of latent variables p(y), in order to losslessly compress latents using an entropy coding algorithm to - logp(y) bits expectation. Neural codecs typically trained using rate-distortion objective consisting of two terms:
be the datapoint compress. A neural encoder Eg takes and outputs quan-
CRD = E [Ldistontonn x(y)) + Arate (y)) x~p(xx
(4)
3

---

Under review as a conference paper at ICLR 2023
The distortion loss Lissootionn is a distance between the reconstruction & and the ground truth x, usually measured by mean squared error, Ldistontion Xx x) = X x|² The rate loss corresponds to the number of bits needed to transmit the quantized latent variable y under the neural prior, equal to Lraae( (y) = - log (y). The tradeoff parameter Arate determines the expected compression ratio: for high Arate few bits should be used, and vice versa. As it is impractical to train one codec per bitrate, a common choice is to condition the codec on the tradeoff parameter, and learn to operate at multiple bitrates by varying Aeate at training time (Song et al. 2021 Rippel al. 2021
2.3 DIFFUSION-ASEE NEURAL CODEC
A DDPM-based decoder While the previous two sections were meant as introduction to the necessary background, this section describes contributions made in this work. A DDPM can be constructed conditional model that exploits useful side-information, for example target class or desirable latent features. In the data compression setting, one could build a DDPM conditioned on a compressed (quantized) latent representation, y of the input x, as is done in concurrent work (Yang & Mandi 2022). .. In this work we choose to condition our diffusion model on an initial reconstruction from a decoder Dÿ. which leads to learn to improve perceptual quality, as we outline below.
Assuming a perfect encoder/decoder pair, the reconstruction & will have minimal distortion for the given bitrate. The DDPM D has access to the same information as DjJ meaning that in expectation it cannot improve fidelity further. As the latent y is compressed discrete representation, there are multiple images that result in the same latent, the preimage of y: p(xly) = p(x[ An optimal DDPM will model the distribution p(x X) exactly for any given X. Although the perceptual metric that best corresponds to human perception is unknown, Blau & Michaeli| (2019) formalize perceptual quality a distance between the image distribution and the distribution of reconstructions, Perfectly modeling p(x ()) will trivially result in zero distance, and should thus lead to perfect perceptual quality under this definition. Note that this definition is also the basis of common perceptual metrics such as the Frechet Inception Distance (FID) (Heusel et al. (017])
Furthermore, Blau& & Michaeli 2019) show that rate, distortion and perception are in triple trade- off. As our initial reconstruction X has the highest fidelity in expectation, and the bitrate is fixed, expect an increase in perceptual quality result in decrease in fidelity. Intuitively this makes sense: we know that & minimizes the distortion to all x ~ p(x[x), i.c., it is the distribution mean when distortion is measured in L2. Samples from p(x(x) will have a non-zero expected distance from the mean, and thus reduced fidelity. As a result we have, in expectation, an initial reconstruc- tion with optimal fidelity, and a DDPM-ennanced reconstruction with optimal perceptual quality.
Although in practice our encoder, decoder and DDPM are not optimal, we show that the distortion- perception tradeoff occurs empirically, and that intermediate samples from the DDPM will lead to a smooth traversal from high fidelity image to one with high perceptual quality. We choose only model residuals r = x x with the diffusion model, which means the DDPM output is added to the initial reconstruction. This approach is visualized in Fig. 2 In theory this should be equivalent to predicting full images, but residuals follow an approximately Gaussian distribution, which we believe may be easier to model. More details on this choice are given in Appendix A.4
Optimization Our model is optimized using a dediittttioo-eeeeppttii loss, a combination of the rate-distortion loss of Eq. and the DDPM loss term of Eq.
(x)
LRIP
E x=p((x)
Ldistortion (x,, x) + dmneeCune + Aperseppton Coneepppiiononooopeep CDDPM (x, x)
(5)
|||x Dy(x))1= - log Po(y) + Apenception| l|ro (r,r
(6)
x,e,
where we define the residual ro --=--- x. and the DDPM learns to model p(ro| x).
We find that training end-to-end with this objective does not work well. While end-to-end training of diffusion models with additional encoders has been demonstrated (Preechakul et al. 2022 Yang & Mandi (2022), it is a common choice to perform training in separate stages (Rombach et al. 2022 Pandey et al. 2022). We train our codec in two stages, training only the base codec first using
LRD.
4

---

Under review as a conference paper at ICLR 2023
meun-scale hyperprior
Eg
y
D3
Pg
p())
input image
initial reconstruction
T7-1
Dg
D
Gaussian noise
diffusion probabilistic model
final residual
final reconstruction
Figure 2: Overview of DIRAC, our proposed codec. A mean-scale hyperprior (optimized for dis- tortion) transmits the input image x via quantized latent y. resulting in an initial reconstruction x. This initial reconstruction is used as conditioning for U-Net based DDPM (optimized for percep- tual quality) to generate a residual ro. The final reconstruction is then & = X ro.
then freezing its parameters and training the DDPM using LDDPM Final finetuning does not yield any improvements, we omit it. Details on optimization parameters are given in Section 3
Fast sampling The computational cost of generation using diffusion models is often much higher than that of alternative models such GANs (Ho et al. 2020), T' model forward passes are required to sample one datapoint. Many efforts in this field were devoted to improving the sampling speed, see Section for an overview. In this work, we use the deterministic sampling algorithm introduced by Denoising Diffusion Implicit Models (DDIM) (Song et al. 2020), as is more robust reducing the number of steps than DDPM sampling. We provide more context on this scheme in Appendix A.5 While we use T = 1000 during training, the number of sampling steps can be reduced to 100 at test time at negligible cost performance, by redistributing the timesteps based on the scheme proposed by Nichol & Dhariwal (022I).
As mentioned earlier, the DDPM in DIRAC aims to improve perceptual quality at the cost of fidelity. We demonstrate in this work that the distortion-percepiie tradeoff can be navigated smoothly with the iterative DDIM sampling procedure: in practice, predicted residuals early in sampling have a low norm and are close zero, and their norm increases gradually with each step. This means that the sampling procedure can be stopped at any point, e.g., when the desired perceptual quality is achieved, when a compute budget reached. To indicate how many sampling steps have been performed, we refer to our model as DIRAC-n, going from DIRAC-1 to
While early stopping navigates the perception-distortion tradeoff, find that it is also possible to skip initial sampling steps, similar to observations by Dhariwal & Nichol 2022 ); Lyu et al. (2022). For the chosen noise schedule, early latents X are close Gaussian noise. By inserting appropri- ately scaled noise as "latent" at timestep t, we show in Section 4 that up to 80% of the sampling procedure can be skipped with negligible decrease in performance. The result is that our codec vastly more practical than other diffusion-based approaches in the compression context. Addition- ally, suggests that better noise schedules are possible in this setting.
EXPERIMENTS
Baselines Our codec able dynamically adjust the perception-fidelie tradeoff at test time. We therefore compare it to the state-of-the-art methods in both peteppercn and rate-distortion performance, One of the strongest perceptual codecs HiFiC [Mentzer et al. 2020). Our reimple- mentation obtains slightly better performance than their reported results. While neural codecs have obtained strong rate-distortin performance, the standard codec VTM (Bross et al. (021) is still one of the best performing codecs in the low bitrate regime. We therefore include VTM as rate- distortion baseline. Note that while our hyperprior base model (Ballé et al. 2018 Minnen et al.] 2018) is not state-of-the-art, it is well-established method for which high quality open source
5

---

Under review as a conference paper at ICLR 2023
Refercang AREREAA
BERANA PAXO
DIBOE
185
LBOKET
Figure 3: Selected CLIC2020 set crops showcasing three cases where different distortion- perception operating points are preferred. Based on the authors' judgement, green dots indicate best subjective quality. Left shows how DIRAC-100 (percep. quality) fares better than HiFiC on text input, center shows how DIRAC-1 (fidelity) is beneficial for small text and faces, right shows HiFiC's superiority for fine texture generation. Best viewed electronically.
implementations available. We also include the older JPEG codec (Wallace 1991). Standard baselines are evaluated using the VTM reference software, Compress framework (Bégaint et al. 2020) and libjpeg in Pillow. We provide details how reproduce baselines in Appendix B.5
Datasets For training, we use the training split of the high-resolution CLIC2020 dataset (Toderici al. 2020) (1633 images of varying resolutions). We follow the three-step preprocessing pipeline of
HiFiC (Menteer et al.) 2020): for each image, randomly resize according to scale factor uniformly sampled from the range [0.3. 1.0], then take random 256 x 256 crop. then perform a horizontal flip with probability 0.5. For validation and model selection, we use the CLIC 2020 validation set (102 images). We test on two datasets: CLIC2020 test set (428 images) and the Kodak dataset (Kodak) 1991] (24 images), both common benchmarks for image compression. We evaluate on full resolution images: we reflect-pad the network input for sides to be multiple of 64 64ethe total downsampling factor within the encoder and hyper-encoder- and crop the output back to the original resolution.
Metrics We evaluate our method using both distortion and objective perceptual quality measures. For distortion, we use the common PSNR and MS-SSIM (Wang et al. 2003) distortion measures. As perceptual quality measures, we use two different metrics: the full-reference LPIPS (Zhang et al. 2018) metric, and the Frechet Inception Distance (FID) (Heusel et al., 2017) metric, which measures the distance between the target distribution p x) and the distribution of reconstructions p(&). We also show Inception Score (Salimans al. 2016). KID Binkkowsii
al. [002] and NIQE (Mittal et al. [203)] in Appendix C.3 The latter requires a large dataset of images; following HiFiC evaluation, we take half-overlapping 256 256 patches from each image in a test dataset instead of resizing full-resolution images. All metrics are computed on images in the RGB color space. To report bitrates, we perform entropy coding and take the filesize. This leads to no more than 0.5% overhead compared to the theoretical bitrate.
Training details We provide full details on the implementation in Appendix B and report com- putational cost in Appendix B.4 We only repeat the most important details here. As mentioned earlier, we separate training into stages, which allows to ignore the Aperception factor in Eq. ():
(1a) pre-train single-rate mean-scale hyperprior (Minnen et al., 2018) for 2M iterations using the ERD loss, (Ib) finetune multi-rate hyperprior for 1M iterations using the resulting weights, and (2) train the diffusion model for 650k steps using LPDPM. while keeping the hyperprior frozen. Training is performed using the Adam optimizer (Kingma & Ba 2015) with learning rate of 10-4 and no learning rate decay for the diffusion model, while learning rate is decayed by factor of 0.5 for the last 500k steps for the hyperprior model.
Our mean-scale hyperprior architecture follows (Balé et al., 2018). It uses 4 downsam- pling/upsampling layers the encoder/decoder, and 2 downsampiinguusampiigg layers the

---

Under review as a conference paper at ICLR 2023
byper-encoderrllype--deoooo The model has 21.4 million parameters in total. For context, the HiFiC baseline has 181.5 million.
We base our implementation of DDPMs on the official open source implementation of Dhariwal & Nichol (2021 and base most of our default architecture settings on the 256 x 256 DDPM from Preechakul al. 2022), which uses U-Net architecture (Ronneberger al. 2015). This model has 108,4 million parameters. Conditioning on X is achieved by concatenating x and the DDPM latent Xt. Early experiments revealed that certain architectural choices resulted in improved performance. To validate these, we perform an ablation study in Appendix [.I] using a smaller version of our model, and smaller crops of the validation set. In summary, we find that using the training objective of Eq. (3) (instead of the commonly used e objective of Ho et al. (2020), see Appendix training on larger crops with increased model capacity results in large improvements in perceptual quality. Yet, the study also suggests our final model configuration may have room for further improvement.
4 RESULTS
Distortion- petirpoooonneptioonn tradeoff In Fig. [ we show the distortionn-percetiion tradeoff for our model compared to the hyperprior base model and HiFiC. The sampling steps of the diffusion model grad- ually increase perceptual quality (lower FID), at the expense of fidelity. The figure visualizes this trajectory for an example image from the CLIC2020 test set. We find that a single sampling step from our model improves PSNR over the hyperprior base codec, likely due larger model capacity and longer training time.
HiFiC performs best in FID, but we emphasize that depending on the setting, PSNR may be more im- portant. Different content and applications require different iettoriion-perception operating points. To support this argument, we provide additional qualitative examples in Fig. 3 For each case, a different method is preferred between HiFiC, DIRAC-1 (highest fidelity) and DIRAC-100 (highest perceptual quality). While it is clear from the rightmost example that HiFiC is excellent at syn- thesizing texture, both left and middle example show that HiFiC hallucinations lead to poor results on information rich high frequency content like small text. DIRAC's dynamic test-time control of tradeoff allows mitigating the issue, it spans close the entire distortion- perception tradeoff, which makes it amenable to more use-cases. Additional qualitative results, as well as other approaches' reconstruction of the same examples, can be found in Appendix C.4
distortion-percepe
Comparison to state of the art While the distortion-perceetioon perception tradcoff is shown in Fig. I visualize the individual rate-distortion and rate- perceepecepionn tradeoffs in Fig. 4 showing both FID and LPIPS as measures of perceptual quality. Note that FID computation on Kodak is not reliable due to the low number of available samples, and we only show it for completeness. We show our model in two configurations: DIRAC-10K 100 (100 sampling steps) has maximal perceptual quality, and DIRAC- I (single sampling step) has minimal distortion. HiFiC demonstrates the best perceptual quality, but has poor performance in PSNR. In terms of PSNR, DIRAC-1 is close VTM in the low-rate regime and beats it in the high-rate regime. We also show the concurrent work by Yang & Mandi (2022) for LPIPS on Kodak (scores extracted from their Fig. 2c, scores for other panels not available). Our model outperforms theirs by large margin in the low-rate regime.
Efficient sampling In Fig. 5 we show the difference between sampling the full 100 steps (dotted line), and skipping the first 80 steps (solid line). on the CLIC2020 test set and for two different bitrates. The left and center panels show the change of PSNR and FID, respectively, as more sam- pling steps are performed. As seen in Fig. I both PSNR and FII decrease over time, traversing the distortion-peree perception tradeoff. Sampling can be stopped at any point to achieve the desired balance. For the lower rate example the change is more gradual, while the high-rate case shows little change during large parts of the sampling procedure. We further find that there is little difference in FID when we skip the first 80 sampling steps, while PSNR actually improves for the low-rate example. Note that the ability to start sampling from step 80 also means that all tradeoff points before that can be reached with a single one-shot prediction by our model, i.c., by scaling noise to the appropriate variance and performing single diffusion step. These findings translate to the iistortion--erceppti tradeoff in the rightmost panel, where we only show the low-rate example. DIRAC, using only the
The official implementation of Dhariwal & Nichol 2021 http:///hhhiicccoooommm/////alaa
7

---

Under review as a conference paper at ICLR 2023
PSRR (00,
PI
L
35
0.35
30
0.30
0.25
34
0.20
15
0.35
32
0.30
30
28
0.00
120
0.40
DIRAC-100 (perceo. queltyi
DIRAC-1 (nde
0.30
Hyparorie (Minnan 2018)
80
0.25
HIS [Mentzer
30
0.30
20
0.05
0.6
Figure 4: Rate-distortion (left) and perceepetttion (middle & right) curves for the CLIC2020 test set (top) and Kodak dataset (bottom). Although FID on Kodak is based on a low number of crops, we report here for completeness.
Pul 300
33.5
PSNR 1.
P932 (th.
33.0
15-
32.5
10
20
40
50
182
100
20
40 00
800
120
10
15
20
30
35
Step
Step
FD
Figure 5: Analysis of the effect of early stopping and late starting on the CLIC2020 test set. We show tradeoffs for rate-distortion (left), rate-perception (center) and centortion-pereeptiin (right). The default sampling schedule of 100 steps is shown dotted line (invisible in case of perfect overlap), the solid line shows performance when skipping the first 80 steps.
last 20 sampling steps, actually performs better than the full sampling procedure in terms PSNR, and the final FID is only negligibly worse. This behaviour is rate-dependent: at high bitrates results are virtually identical, at lower rates (shown in Fig. we see an increase PSNR.
5 RELATED WORK
Neural data compression Neural codecs have seen major advances in both the image (Minnen
al.) 2017: Rippel & Bourdev 2017 Ballé et al. 2018 Minnen et al. 2018) and video domain
(Wu et al. 20188 Lu et 2019 Rippel et al. 2019 Habibian et al..[001g | et al.) 2020
Hu et al. 2021 2022). By conditioning on Arate variable corresponding to a desired bitrate, some codecs have been extended to operate multiple bitrates (Wu et al. 2020 [Song et al. 2021 Rippel incepption-focusee compression methods are mainly led by GAN-based approaches Agusssson et al. 2019 Mentzer et al. 2020 Yang al. 202 Mentzer et al. (021)) Recent GAN- based works allow trading off fidelity and perceptual quality using control parameters and latent masking (Wu et al.) 2020; Twai et al. 202 Unlike existing work that masks the latent reduce
ctt al. (021).
the bitrate and determine which areas of the input are high fidelity, our codec does not entangle rate-distortion and perception-distortion tradeoffs.
8

---

Under review as a conference paper at ICLR 2023
Diffusion probabilistic models
DDPMs have proved to be successful in various application areas,
including image and video generation (Ho et al.
2022a Yang et al. 2022), representation learning
(Preechakul et al., 2022 Pandey et al. 2022). language-to to-maae generation (Ramesh et al. 2022
Saharia et al. 2022b), and image-to-image modelling such as image deblurring (Whang et al. 2022
and restoration (Kawar et al. 20222 Saharia al. 2022a).
In the context of data compression, Hoogeboom et al. (2021) show that DDPMs can be used in the lossless compression setting. In the lossy compression setting, Ho et al. (2020) showed that if the continuous latents can be transmitted, a DDPM enables progressive coding. Theis et al. (2022) make this approach feasible in practice by using reverse channel coding. Although their approach is powerful and elegant, it is currently unlikely to be of practical use due to its high computational
cost. In concurrent work. Yang & Mandt (2022) propose codec where a conditional DDPM takes the role of the decoder, directly producing a reconstruction from a latent variable. In contrast, our codec first creates an initial reconstruction and then models the residuals between and the ground truth, which enables direct control over perception-distoriae and results in improved performance.
The sampling process for diffusion models is typically inefficient as it requires a high number of sampling steps to achieve the best performance. Although it is not our main focus, we show in this work that the compression setting lends itself well to speeding up sampling. Other works have shown that the diffusion process can be started at late timesteps (Lyu al. 2022). ). and that this could mean that better-suited noise schedules exist (Nichol & Dhariwal (021), ). Various other methods were proposed to improve the sampling speed, either through model distillation (Luhman & Luhman 2021 Salimans & Ho 2021), by altering the sampling procedure (Song et al. 2020), or by working owddimensional latent space (Rombach et al., 2022). In this work, we use DDIM (Song al., 2020) as it provides a training-free sampling scheme based on a generalization of the forward process to Markovvan diffusion processes. This choice of forward process leads to "shorter" Markov Chains, which increases sample efficiency.
6 DISCUSSION AND LIMITATIONS
Our generative codec navigates the rate-distortion disttottiioneeercption tradcoff dynamically at test time, meaning the user can determine the importance of bitrate, fidelity and perceptual quality themselves. Although this improves user control over the amount of hallucinated content, we currently do not control where such hallucinations occur. Similar to GAN-based codecs, we observe that increasing perception sometimes harms fidelity in small regions with semantically important content, such as faces and text. Addressing this limitation is an important next step for generative codees.
Additionally, although we are able to drastically reduce the number of sampling steps, it is still fairly expensive use a DDPM on the receiver side. The fact that we see an increase PSNR after single sampling step means that the base hyperprior codec is far from perfect, and that improving the base model or adopting a standard codee like VTM would increase the fidelity of the initial reconstruction. This likely means that the DDPM can be made smaller to reduce computational complexity. We also find that many sampling steps can be skipped without loss in performance. This suggests that improved noise schedules, or conditioning the DDPM on the noise so that different schedules can be used at test time (W Whang et al. 2022). may further improve performance.
7 CONCLUSION
In this work, we propose a new neural image compression method called Diffusion- based Residual Augmentation Codec (DIRAC). This codec uses a variable bitrate base model to transmit an initial reconstruction with high fidelity to the original input, and then uses diffusion probabilistic model to improve its perceptual quality. We show that this design choice enables fine control over the rate- distortion- ppertiion-eeeettion tradeoff at test time, which for example enables users to choose if an image is decoded with high fidelity high perceptual quality. Furthermore, although diffusion probabilistic models are notoriously expensive to sample from, we show that in the compression setting, the number of sampling steps be reduced dramatically. Our work shows that diffusion- codecs have compelling practical advantages that make them promising candidate for high perceptual quality data compression.

---

Under review as a conference paper at ICLR 2023
Ethics statement Neural codecs may be biased towards the data they have seen during training, for example leading to worse reconstructions for data with little support. Additionally, generative codecs may generate false but plausible details, which can be harmful for cases where fidelity important. We observe this especially when examining reconstructions containing text and faces. Deploying generative codecs will therefore require careful investigation of the desired operating points and their failure cases. By giving the user control over the distortion-percepiion tradcoff, our codec facilitates selection of this operating point at test-time.
Reproducibility statement All datasets used in this work are publicly available, and we show our three-step data augmentation pipeline Section 3 The proposed codec consists two components: a base image codec, and DDPM that performs enhancement of the initial reconstruction.
The base image codec is mean-scale hyperprior (Ballé et al. 2018). High quality implementations of this neural codec (and similar image codecs) are available via CompressAl, see for example this mean-scale hyperprior implementation link on GitHub) This library also provides entropy coding functionality. Relevant hyperparameters for hyperprior training listed in Appendix [B.1
The DDPM component was trained using the open source implementation of (Dhariwal & Nichol 202T) The main change in implementation is that our DDPM is conditioned on an initial recon- struction from a mean-scale hyperprior, which is achieved by concatenating it with the DDPM latent (these two tensors have the same spatial dimensions). We list all hyperparameter settings in Table T as well as the name of command line arguments corresponding to those parameters. We provide details on sampling in Appendix A.5 Appendix B.3
Lastly, we provide information about computational complexity and training compute in Ap- pendix B.4 Instructions for reproducing baselines are provided in Appendix B.5
REFERENCES
Eirikur Agustsson and Radu Timofte. Ntire 2017 challenge on single image super-resolution: Dataset and study. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR) Workshops, July 2017.
Eirikur Agustsson, Michael Tschannen, Fabian Mentzer, Radu Timofte, and Luc Van Gool. Gener- ative adversarial networks for extreme learned image compression. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2019.
Eirikur Agustsson, David Minnen, Nick Johnston, Johannes Balle, Sung Jin Hwang, and George Toderici. Scale-space flow for end-to-end optimized video compression. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2020.
Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. Variational image compression with a scale hyperprior. In International Conference on Learning Represen- tations, 2018
Jean Bégaint, Fabien Racapé, Simon Feltman, and Akshay Pushparaja. CompressAI: Py- Torch library and evaluation platform for end-to-end compression research. arXiv preprint arXiv:201 arXiv:2011.03029 1.03029, 2020.
Mikotaj Binkowski, Danica J. Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. In International Conference on Learning Representations, Feb 2022.
Yochai Blau and Tomer Michaeli. Rethinking lossy compression: The rate-distortion-eerepee
tradeoff. In International Conference on Machine Learning, pp. 675-685. PMLR, 2019.
Benjamin Bross, Yu- Wang, Yan Ye, Shan Liu, Jianle Chen, Gary J Sullivan, and Jens-Rainer Ohm. Overview of the versatile video coding (VVC) standard and its applications. IEEE Trans- actions Circuits and Systems for Video Technology, 3 31100)3736-3764 2021.
Kamil Deja, Anna Kuzina, Tomasz Trzciñssi,, and Jakub M Tomezak. On amalyzing generative and denoising capabilities of diffusion- deep generative models. Neural Information Processing Systems, 2022.
10

---

Under review as a conference paper at ICLR 2023
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs image synthesis. Ad- vances Neural Information Processing Systems, 34:8780-8794 8794, 2021.
Zongyu Guo, Zhizheng Zhang, Runsen Feng, and Zhibo Chen. Soft then hard: Rethinking the quantization in neural image compression. In International Conference on Machine Learning, pp. 001399929992 PMLR, 2021.
Amirhossein Habibian, Ties van Rozendaal, Jakub M Tomczak, and Taco S Cohen. Video compres- sion with rate-distortion autoencoders. In IEEE International Conference on Computer Vision, 2019.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Sali- mans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research. 23:47-1, 2022a.
Jonathan Ho, Tim Salimans, Alexey A Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. In ICLR Workshop on Deep Generative Models for Highly Struc- tured Data, 2022b.
Emiel Hoogeboom, Alexey Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans. Autoregressive diffusion models. In International Conference on Learning Rep- resentations, 2021.
Zhihao Hu, Guo Lu, and Dong Xu. FVC: A new framework towards deep video compression in feature space. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1502-1511, 2021.
Zhihao Hu, Guo Lu, Jinyang Guo, Shan Liu, Wei Jiang, and Dong Xu. Coarse-to fine deep video coding with hyperprior-guider mode prediction. In Proceedings of the IEEE conference on Com- puter Vision and Pattern Recognition, 2022.
Shoma Iwai, Tomo Miyazaki, Yoshihiro Sugaya, and Shinichiro Omachi. Fidelity ex- treme image compression with generative adversarial networks. In 2020 25th International Con- ference Puttern Recognition (ICPR), pp. 8235-8242. IEEE, 2021.
Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. In ICLR Workshop Deep Generative Models for Highly Structured Data, 2022.
Younhee Kim, Seunghyun Cho, Jooyoung Lee, Se-Yoon Jeong, Jin Soo Choi, and Jihoon Do. To- wards the perceptual quality enhancement of low bit-rate compressed images. In 2020 IEEE/CVF Conference on Computer Vision and Puttern Recognition Workshops (CVPRW), pp. 565-569, Jun 2020.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In International Conference on Learning Representations, 2015.
Photoed dataset Kodak. Kodak photoed dataset, 1991. URL http::://kk.ss/graaapaapa
kodak/]
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zimicck. Microsoft coco: Common objects in context. In Proceedings of the European Conference on Computer Vision, pp. 740-755, 2014.
Guo Lu, Wanli Ouyang, Dong Xu, Xiaoyun Zhang, Chunlei Cai, and Zhiyong Gao. DVC: An end- to-end deep video compression framework. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2019.
11

---

Under review as a conference paper at ICLR 2023
Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint Stti::2101.023300 2021.
Zhaoyang Lyu, Xudong Xu, Ceyuan Yang, Dahua Lin, and Bo Dai. Accelerating diffusion models via early stop of the diffusion process. arXiv preprint arXiv: 12255, 2022.
Chao Ma, Chih- Yuan Yang, Xiaokang Yang. and Ming- Haung-ssann Yang. Learning a no-reference quality metric for singl-image super- resert-esolution. Computer vision and image understanding: CVIU, 158: 1-16, May 2017.
Fabian Mentzer, George Toderici, Michael Tschannen, and Eirikur Agustsson. High-fidelity gener- ative image compression. In Neural Information Processing Systems, 2020.
Fabian Mentzer, Eirikur Agustsson, Johannes Ballé, David Minnen, Nick Johnston, and George Toderici. Neural Video Compression using GANs for Detail Synthesis and Propagation. arXiv preprint arXiv:2107.1200 2021.
David Minnen, George Toderici, Michele Covell, Troy Chinen, Nick Johnston, Joel Shor, Sung Jin Hwang, Damien Vincent, and Saurabh Singh. Spatially adaptive image compression using a tiled deep network. In 2017 IEEE International Conference Image Processing (ICIP), pp. 2796- 2800. IEEE, 2017.
David Minnen, Johannes Ballé, and George Toderici. Joint autoregressive and hierarchical priors for learned image compression. Neural Information Processing Systems, 2018.
Anish Mittal, Rajiv Soundararajan, and Alan C. Bovik. Making a "completely blind" image quality analyzer. IEEE Signal Processing Letters, 20(3):209-112, Mar 2013.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8171. PMLR, 2021.
Kushagra Pandey, Avideep Mukherjee. Piyush Rai, and Abhishek Kumar. DiffuseVAE: Effi- cient, controllable and high-fidelity generation from low-dimensional latents. arXiv preprint 2rXiv:2201.00308. 2022.
Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasom Suwajanakom. Dif- fusion autoencoders: Toward meaningful and decodable representation. In Proceedings of the JEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 10619-10629, 2022.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents, 2022.
Oren Rippel and Lubomir Bourdev. Real-Time adaptive image compression. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 2922-2930. PMLR, 2017.
Oren Rippel, Sanjay Nair, Carissa Lew, Steve Branson, Alexander G. Anderson, and Lubomir Bour- dev. Learned video compression. In IEEE International Conference on Computer Vision, October 2019.
Oren Rippel, Alexander Anderson, Kedar Tatwawadi, Sanjay Nair, Craig Lytle, and Lubomir Bourdev. ELF-VC: Efficient Learned Flexible- Rate Video Coding. Neural Information Process- ing Systems, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Patlern Recognition, pp. 10684-10695. 2022.
O. Ronneberger, P.Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention (MICCAI), volume 9351 of LNCS, pp. 234-241. Springer, 2015. URL httt:://in informatik. (available on arXiv: 1505:055
uni att.ee/bbbrr..ee/ulaaaaaaaaaaaaa http:////e.........ooo////////// [cs.CV]).
12

---

Under review as a conference paper at ICLR 2023
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-ing mnage-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings, pp. 1-10, 2022a.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Sal- imans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic texi-to-image diffu- sion models with deep language understanding, 2022b.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022c.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations, 2021.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, and Vicki Cheung. Improved techniques for training gans. In Neural Information Processing Systems, 2016.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using non-equilibrium thermodynamics. In International Conference on Machine Learn- ing. pp. 2256-2265, PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations, 2020.
Myungseo Song, Jinyoung Choi, and Bohyung Han. Variable-rate deep image compression through spatially-adaptive feature transform. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2380-2389, 2021.
Lucas Theis, Wenzhe Shi, Andrew Cunningham, and Ferenc Huszár. Lossy image compression with compressive autoencoders. International Conference on Learning Representations, 2017.
Lucas Theis, Tim Salimans, Matthew D Hoffman, and Fabian Mentzer. Lossy Compression with Gaussian Diffusion. arXiv preprint arXiv:2206 08889, 2022.
George Toderici, Wenzhe Shi, Radu Timofte, Lucas Theis, Johannes Balle, Eirikur Agustsson, Nick Johnston, and Fabian Mentzer. Workshop and challenge on learned image compression (clic2020). In Proceedings of the IEEE conference Computer Vision and Puttern Recognition. CVPR, 2020. URL http:
www compression.cc.
Gregory K. Wallace. The JPEG Still Picture Compression Standard. Commun, ACM, 1991. ISSN 0001-0782. doi: 10. 10.114551 141//00008.000089 URL 103089
Huairui Wang, Guangjie Ren, Tong Ouyang, Junxi Zhang, Wenwei Han, Zizheng Liu, and Zhen- zhong Chen. Perceptual in-loop filter for image and video compression. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (Cciopps(CCPE (CVPRW), pp. 1769-1772, Jun 2022.
Zhou Wang, Eero Simoncelli, and Alan C Bovik. Multiscale structural similarity for image quality assessment. In The Serith- Asilomar Conference on Signals, Systems & Computers, 2003, volume 2. pp. 1398-11 1402. Ieee, 2003.
Jay Whang, Mauricio Delbracio, Hossein Talebi, Chitwan Saharia, Alexandros G Dimakis, and Pey- man Milanfar. Deblurring via stochastic refinement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16293-16303, 2022.
Chao-Yura Wu, Nayan Singhal, and Philipp Krähenbühl. Video compression through image inter- polation. Proceedings of the European Conference Computer Vision, 2018.
Lirong Wu, Kejie Huang, and Haibin Shen. A GAN- tunable image compression system. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2334- 2342, 2020.
13

---

Under review as a conference paper at ICLR 2023
Ren Yang, Luc Van Gool, and Radu Timofte. Perceptual learned video compression with recurrent conditional GAN. arXiv preprint arXiv:2109.03002, September 2021.
Ruihan Yang and Stephan Mandt. Lossy image compression with conditional diffusion models.
arXiv preprint arXiv:2209.0668 06950, 2022.
Ruihan Yang, Prakhar Srivastava, and Stephan Mandt. Diffusion probabilistic modeling for video generation. arXiv preprint arXiv:2203. 09481, Mar 2022.
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018. doi: 10. 10. 20..200088..0.....
Yinhao Zhu, Yang Yang, and Taco Cohen. Transformer-baset transform coding. In International Conference on Learning Representations, Mar 2022.
14

---

Under review as a conference paper at ICLR 2023
A METHOD
A.1 DERIVATION OF DDPM LOSS FOR LEARNING XD
Following Sohl-Dickstein al. 2015), the ELBO on the log likelihood of pe(xo) can be written as T T-step Kullback-eiblle (KL) divergences: K (x [x,, (x))lpo(((_1 fort -,'., Under the assumption that the two distributions of interest are both Gaussian, we further assume Eg (x t) = of to be time dependent constants. This reduces the KL divergence at step to comparison from the model mean to the posterior mean of the forward process, i,c
Bg 2
(Xt, xo) px e)1
+ +C,
where C is a constant, and jr denotes the posterior mean of the forward process conditioned on the data input Xo, i..., (xx Xo).
Note that, using the Bayes' rule and the Markovian property of the forward process, we can rewrite
9(xx-1 Xt
g(xt-1 xo)
g(xe| xo)
g(xt-1 1 ) Xt-1)
(8)
q(xx Xo)
where the distribution (xxe = (xx; vā (1-ā,) I) with a := 1- Br and ā₁ == II._ as.
With the known distribution of g(xt) (xo), we can sample xt at an arbitrary step by
Xx (Xo, E) = V vI vl-ā,e
ā,e,
where e~,(O,
Now each of the components on the RHS of Eq, (8) is defined as a Gaussian distribution with known parameters. As result, we can find the posterior mean fú the following explicit form:
Xo) = 1a xo + 1 Cz
Vau ā)-1) Xt 11X00 + Exxt
(10)
at
By plugging Eq. 10 into Eq. 77. and choosing a parametrization that matches psg jı, at each 1x,,(xe77 + & (where ge is a function that directly
diffusion step t, for example μg (Xt, t) predicts Xo from xx), we arrive at an objective function diffusion step t:
min E
willxo
()( for == 1, 1
(11)
L,xn.c
where u. = n// 22 and is absorbed into the expectation instead of being summed over.
A.2 PARAMETRIZATION FOR LEARNING THE NOISE
For completeness, we use this section to restate the approach of Ho et al. (020), who parametrize models to predict the noise instead of the data Xo.
Ho et al. 2020 reparameterized Eq. 99 to obtain a function of xo: (xt: E)
Va, (xt +
vI a,E) where ~ .N I). To minimize the KL divergence (7) at time t, the authors chose formulation that learns the noise c, which leads to the following desired equality:
po (xe, t) (xe.
1(,, (x, + vI -,EE))
(12)
Va
Expanding the posterior mean according to Eq. mizes prediction errors in the e-space:
110 we can derive an objective function that mini-
Le Exo.c velle colVa,xo v1 t1e,
(13)
where the weight coefficients 14 = B3/((2far(1 -ār)).
15

---

Under review as a conference paper at ICLR 2023
Image Space
Residual Space
0.0075
density
0.0050
0.05
0.0025
0,0000
0.00
0.006
density
0.004
0.002
0.05
0.000
0.00
100
200
density
0.006
0.10
0.004
0.002
0.000
0.00
100
150
200
250
-40
-20
0 value
40
value
Figure 6: Distributions of pixel values in 1000 random 256x256 crops of the CLIC train dataset, left for the target image x, right for the residual X where the initial reconstruction is from single-rate mean-scale hyperprior trained with Arae 0.03. Rows corresponds to the red, green and blue channels respectively.
A.3 NOISE SCHEDULE AND LOSS WEIGIITS
We choose the noise schedule 3 such that the ã; follow cosine-shaped curve. This is of the most common choices for the noise schedule. We also reweight the individual loss terms from Eq. to w. 1, inspired by Ho et al. (2020) who do the same for loss formulation given in Eq. [3) (meaning they set e = 1). The theoretically derived w/ 772 /?/() results extremely large weights for small t. We find that w, works better, likely because of the more evenly distributed loss scales, but it is possible that better weights exist.
A.4 MOTIVATION FOR DDPM MODELING IN RESIDUAL SPACE
DIRAC outputs an initial reconstruction &, then models the distribution of residuals p(r[x) using DDPM, where r = X. In theory, one could model the image distribution p(x|X) directly. From an entropy perspective, the two approaches are equal: the entropy of p(r(X) is equal to the entropy of p(x[X) since, given the data x, the residual r deterministic function of the initial reconstruction. Yet, while their entropy might be similar, the shape of these distributions is different.
In Fig. 6 we show the histogram of pixel values in the target image (left) and the residuals (right) for each RGB channel. We observe that residual values approximately follow normal distribution, whereas the pixel values image space show a more intricate distribution. Given fact that DDPMs (in their typical formulation) map Gaussian noise the target distribution, conjecture that it desirable that the target distribution is close to a normal distribution, and that helps reduce the number of sampling steps required to obtain satisfactory perceptual quality.
A.5 SAMPLING DESCRIPTION
We 100 DDIM sampling steps by default (Song al. 2020). We describe modified sampling schemes, namely the early stopping and late start procedures, in more detail. Both schemes speed up sampling by skipping steps entirely.
16

---

Under review as a conference paper at ICLR 2023
Early stopping The DDPM enhancement model of DIRAC predicts the (often sparse) residual We find that we are able to stop sampling at any point before we reach the final step, by
r ==
x. x-x simply using the intermediate prediction for the residual (r,) D2 (Te. t, X) as the final sample. Specifically, we observed that during sampling, early intermediate predictions are close to zero, and that they become sharper stopping late results in high perceptual quality. Intuitively, this makes sense as well: we know that in the limit of perfect models, X has the highest possible expected fidelity, and the DDPM can only increase quality by decreasing fidelity.
time. Stopping early then typically means higher PSNR, whereas
Stopping early in this manner is somewhat similar to the scheme proposed by Ho et al. (2020), who use this intermediate prediction to describe a progressive coding scheme. However, we have already transmitted the initial reconstruction &, and all DDPM sampling happens on the receiver side, so the settings are quite different. It is the receiver-side generation of residuals that allows to navigate the the distortion- -prrrppttnnn tradeoff (Blau & Michaeli 2019), and that enables early stopping to achieve desired tradeoff or to reduce compute requirements. Optimizing sampling trajectories of diffusion models is an active field, and further research on the sampling in different (conditional) settings may lead to actionable insights (Deja et al. 2022).
Late start Similar to findings of Nichol & Dhariwal (2021 and Lyu et al. (2022), we observe that it is possible to skip several sampling steps. In particular, take initial sample NN and scale it to match the expected standard deviation at timestep given by the forward process, then plug this "latent" into the reverse process. [Nichol & Dhariwal (2021) mainly start observation to motivate the use of different noise schedule. Based on the similarity of our observations to theirs, we believe there may be noise schedules that are better suited for the image compression setting, but we leave that investigation for future work. The key finding in our work is that we can skip a large part (up 80%) of the initial steps without performance degradation.
the late-
B IMPLEMENTATION DETAILS
B.1 ARCHITECTURE AND TRAINING
Base codec We use the "mean-scale hyperprior" architecture from Minnen et al. (2018). This is an hierarchical VAE with quantized latent variables. The first level encoder and decoder produce the quantized latent variable and decode it to a reconstruction. The second level, which is the prior model, uses hyper-encoder to produce a so-called quantized hyper-latent z, then maps that to parameters My; Ty using a hyper-decoder. The hyper-latent distribution is typically modeled using an unconditional prior p(z). The probability of the quantized latent under the prior is then equal to Puy. ).).
p(y z) = (ylpyy.
The encoder consist 4 stride convolutions with kernel size 5, and similar transpose convolution
layers for the decoder. The hyper-encoder consists of stride convolution with kernel size 3, followed by stride 2 convolutions with kernel size 5. We use 320 channels for every layer in the encoder/decooerr, and 192 channels for the snper-encoderllyper-eeedddd
As the codec uses + 2 stride convolutions, it has a so-called downsampling factor of 2 64x To enable transmission of images that have spatial dimensions not divisible by this factor, we use 'reflect' padding to pad the image, transmit the padded image, then crop to the original resolution on the receiver side. We transmit the original spatial dimensions as 16 bit integers. This bit cost is negligible compared to the cost of transmitting the content.
Latent quantization is performed using "mixed" strategy: both decoders see the quantized latents during training, and a straight-through estimator is used to make sure gradients pass through the hard quantization operation; the prior computes the loss based on latents quantized using additive uniform quantization noise 11 ~ U(-0.5, 0.5) (Guo et al. 2021). }. For more details and architecture visualizations, refer the reader to Ballé et al. (2018) Minnen et al. (018).
Multi-rate training Multi-rate capabilities are added using a scheme similar to that of (ong et al. 2021) Rippel et al. (202T), by conditioning on Armee during training and evaluation.
17

---

Under review as a conference paper at ICLR 2023
During training, the multi-rate codee must see range of tradeoff parameters in order to learn the desired effect of Arate- For each batch, we sample a value X' ~ [' 1], then skew this by a factor y to obtain X" X". We set 7 3.0 to favor sampling of high bitrate batches, we found that this improves fidelity for low bitrate settings as well. The final sampled tradeoff parameter Arate is then obtained via linear interpolation in log space:
log2 Araee (l0g((Amx))
X" llgg2 (0min))
(14)
where Amax 0.0128 and Amin 0.0001. Arate used for the loss, while X" provided to the base codec. is embedded in a soft one-hot vector dimensions X, following (Rippel et al. 2021 For example, this means the vector [1, 0, corresponds to the value X" 1, and [0, 0.5, 0.5, 0] corresponds to X" 0.5.
Each layer in the codec (sender and receiver side) is conditioned the tradcoff parameter using output activation scale-and-shife following (Song et al.) 202I). This works as follows: for each activation in the codec hace. which has Cast channels, a linear layer projects the 4-element tensor onto tensor with 2- 2- elements. The first Cact elements correspond the scale Sact, the remainder corresponds to the shift dyct The conditioned output activation is then equal to haat
Sact + dac--
At test time, the user picks the rate-distortion operating point by selecting Ayate value. Conditioning is then performed in the same way as during training. Of course, we need to transmit Arate to the receiver end as well. To do so, we quantize X" 16 bit integer, and transmit it at negligible bitrate cost.
DDPM architecture Many of our U-Net architecture choices were adopted from the open-source implementation of Dhariwal & Nichol (202T), and refer the reader to those works for detailed explanation of the meaning of all hyperparameters. Table provides an overview of the hyperpa- rameter settings for the full DIRAC model, well as the DiRAC-small, which is the base model for our ablation study. Horizontal lines separate U-Net parameters, diffusion process parameters, and training hyperparameters.
The channel multiplier corresponds to the increase in width with respect to the base number of channels in each layer. Following [Preechakul et al. 2022), we train our largest models using a U-Net with 6 separate blocks, increasing the multiplier every 2 blocks. Self-attention layers are removed from all layers except the bottleneck of the U-Net. We use the cosine noise schedule of (Nichol & Dhariwal 2021). as we saw better performance with this schedule in early experiments. However, an ablation study Appendix B.2 shows that even better hyperparameters may exist.
Training DIRAC We separate training into two steps: 1) train our base codec for a rate distortion objective Eq. (4) and 2) then train the DDPM using the perceptual objective of Eq. (3).
Step 1 split into two steps as well. First, we train a single rate hyperprior for million iterations using Arate 0.001. Then, we add Arate conditioning layers, and finetune the multi-rate hyperprior for 1M additional iterations, as described in the paragraph "Multi-rate training". Although rate- distortion performance is important, the choice of base codec likely is not critical, and any multi-rate image codec may suffice.
In step 2, we use the base codec to produce initial reconstructions X, and train a X-conditional DDPM to perform enhancement. In order to support the multi-rate hyperprior, the DDPM needs to see reconstructions for many different Arate at training time. In practice, this can be achieved by using the sampling schedule used for multi-rate hyperprior training. The DDPM is trained for 650,000 iterations, the hyperparameter settings specified in Table
B.2 ARCHITECTURE ABLATION STUDY
We adopt most architecture settings for the DDPM from Dhariwal & Nichol (2021), but observed that certain architectural choices provided a significant performance improvement in early exper- iments low-resolution data. To validate this. ablate some architecture choices in Figure 7 Ablation studies were performed on a smaller reference model called DIRAC-small (details in Ta- ble to enable training single V100 GPU, whereas DIRAC is trained on 2 A100 GPUs. For
18

---

Under review as a conference paper at ICLR 2023
Parameter
Command line
DIRAC
DIRAC-small
Channel multiplier
channel mult channels
1,1,2,2,4,4
1,2.3
Base num. channels
128
32
Learn oz
learn_sigma
false
false
Group normalization
group_norm
true
true
Attention resolutions
attention.resoouttion
none
none
Num. attention heads
num_heads
Objective Eq. (3)
predict.xstare
true
true
Num. ResBlocks
num cum.res.block
Scale shift conditioning
use_scale_shift.nomm
false
false
Diffusion steps
diffusion steps noise schedule
1000
1000
Noise schedule
cosine
cosine
Batch size
batch.size
128
32
Learning rate
1r
10-4
10-4
Training steps
iterations
650k
600k
Input resolution
image.size
256x256
256x256
Use DDIM sampling Resampling timesteps
use_ddim
true
true
t timestep_respacing
ddim 100
ddim100
Table 1: Hyperparameters for DIRAC and the smaller ablation model during training and sampling.
We refer the reader the official open source implementation of Dhariwal Nichol (2021) for more details on these parameters.
cuivcal 512v512 center
CUIC 512x512
Group normalization: OFF
140
Noise schedule: linear
40
Training size: 64x64
DIRAC-100 (percep. quelity)
DIRAC-small (ablation reference)
BDL+1
25
40
20
20
0.1
0.2
0.3
0.5
0.6
0.1
0.2
0.3
0.4
0.5
0.6
bpp[i]
bpp
Figure 7: Architecture ablation study for DDPM model. Ablation reference (DIRAC-small) model
defined in Table
all ablations, we sample using DDIM, reducing sampling 100 timesteps, similar manner to Nichol & Dhariwal (202I). We evaluate PSNR and FID metrics 512x512 center crops of the CLIC validation dataset.
We make the following observations. Training smaller crops, using the objective of Ho et al. 2020) instead of the To objective of Eq. (3) results in much worse performance. However, disabling group normalization and using linear noise schedule both improve performance. Our final model does not use these settings, meaning further improvements may be possible for DIRAC. The full DIRAC model performs better than DIRAC-small, especially in FID lower bitrates, meaning that
19

---

Under review as a conference paper at ICLR 2023
increased model capacity likely contributes to performance. Nevertheless, using smaller DDPM is possible if compute cost more important than FID.
B.3 SAMPLING
We DDIM sampling with 100 timesteps for all experiments, following Song al. 2020)), unless stated otherwise. Sampling for 100 timesteps instead of the 1000 used during training requires "timestep respacing" of the noise schedule, which we achieve using the scheme proposed in Nichol & Dhariwal (202T).
To enable sampling for images with shapes not divisible by the downsampling factor of the U-Net, we pad the conditioning using 'reflect' padding, sample an image of the resulting size, then crop to the original spatial dimensions. This does not affect the bitrate, as all information already lives on the receiver end.
As mentioned earlier, for all early stopping and late start experiments, we use the intermediate timestep & prediction (r,) D x) the best-so-far prediction. This is the only sensible choice: the latent r, is still close to random noise at carlier iterations.
B.4 COMPUTE
We provide information on the computational complexity of model components, and the cost of sampling. in Table 2 We run benchmarking CUDA driver version 455.32.00, CUDA 11.1. We use 1,000 forward passes on square inputs of size 256 x 256 and 1024 x 1024, use GPU warmup, and record inference time using CUDA events. We do include time entropy code here.
desktop workstation with Nvidia 3080 Ti card,
Parameter count
Runtime 256 (ms)
Runtime 1024 (ms)
DIRAC hyperprior DIRAC U-Net
21.4M
13.9 + 1.0
82.7 + 1.81
108.4M
39.3 + 1.2
485.6 + 1.17
HiFiC
181.5M
18.4 + 1.1
129.2 0.64
Table 2: Computational complexity for each model. Runtime mean and standard deviation were obtained using 1,000 forward passes. Note that none the models here were explicitly optimized for inference speed, numbers indications only.
All our (base codec) multi-rate hyperpriors were trained on a single Nvidia V100 card. Final DIRAC models were trained on Nvidia A 100 cards using data paralellism. Ablation model (DIRAC-small) settings were chosen that training could be performed one Nvidia V V100 card.
fair comparison between methods would not only require equalization bitrate, but ideally also equalization of training and test-time compute. For example, HiFiC decoding complexity is sub- stantially lower than that of DIRAC if many sampling steps are used, which will give DIRAC-100 an unfair advantage. DDPMs also more datapoints than HiFiC during training larger batch size is used. this work, we mainly focused on fcasibility of the approach, and have therefore not equalized compute.
20

---

Under review as a conference paper at ICLR 2023
B.5 BASELINES
Standard codee baselines We run VTM-17.0 https://vcgit. www.niifraunofeer.co
and use CompressAl https:// to prepare the encoding command. Com- pressAl converts given input RGB images to YUV444 before coding them "all intra" mode, then converts the reconstructed YUV444 images back RGB. These conversions are lossless. We use the default all intra configuration, and use QPs {22, 27, 32, 37, 40}, where higher QPs corresponds low bitrate and vice versa.
github. com/ ctw.//nneeDigggaaaaaaaa......aa.
Specifically, the command used to gather VTM-17.0 evaluations is:
pythen -m compressai... utsils.a.... bench vtm fpath-to-immge-follder -C <path-to-VIM17 httpa//oooooaeeeeete/eeoooeeeeee
-b Meth--o-VIMM. 0ww.11....s0sssee -0tssottaaaerebbin q [22, 27, 32, 37, 40]
For JPEG evaluations, we use JPEG functionality in Pillow. We again use CompressAl to prepare the encoding command.
python3 -m compressai.. sutissiiiiiiisss. utiis bench jpeg <path-to-imag--felen qq [5, 10, 15, 20, 25, 30, 35, 40, 45, 55]
Hyperprior baselines We train mean-scale hyperpriors for various rate- dittoiitootion tradeoff points by setting Arate to one of {0.006, 0.005, 0.004, 0. 0.003, 0.002, 0.001, 0.0003., where 0.006 corre- sponds to the lowest bitrate model. Baseline models were trained for up to 2 million iterations, where we select the best model based on Kodak PSNR. Selecting baseline models based on their test dataset performance may be unfair to other methods, but ensures the baseline is strong.
HiFIC baselines We train HiFiC models, as described by (Mentzer et al. 2020). for 5 bitrate targets: {0.15, 0.225, 0.3, 0.375, 0.45). Each model is trained in two stages. In the first stage, we train a mean-scale hyperprior with a rate-distortion objective for 1 million iterations, where the distortion objective combines mean squared error with the perceptual distortion metric LPIPS (Zhang et al. 2018). In the second stage, a conditional discriminator is added and the model is finetuned for an additional million iterations using a rate-distortion-Gan learning rate is decayed from 10 to 10-5 after 500,000 iterations. We use the models at the final timestep million iterations) as bascline models.
loss. In both stages,
We follow the training procedure described in the paper, with two important deviations. First, we use Arate warmup for 100K iterations, instead of the 50K iterations used in the paper. Second, we enable this warmup schedule for GAN finetuming as well. We find that these changes result in slightly improved performance with respect to the scores reported by Mentzer et al. (2020), especially at low bitrate.
C ADDITIONAL RESULTS
C.1 ROBUSTNESS TO DIFFERENT BASE CODECS
In Fig. x we apply DIRAC-100 (maximum perceptual quality), which was trained with hyperprior base codec, directly on top of other base codecs, without finetuning. Specifically, we use JPEG and VTM as examples of standard codees, and SwinT (Zhu al. 2022 as state-of-the-art neural codec. We find that our model still improves perceptual quality, indicating that it is robust to the choice of base codec. It is reasonable to assume that training on these base codees would improve performance further. On VTM and SwinT, the gain in perceptual quality is still large, close to what we achieve for the hyperprior base codec. Using JPEG, the increase is only moderate. We believe this due to the fact that JPEG exhibits strong artifacts at low rates, specifically blocking, Reconstructions from VTM and SwinT, on the other hand, are pretty similar to those from the hyperprior, in that low-rate reconstructions tend be blurry.
21

---

Under review as a conference paper at ICLR 2023
JPEG
JPEG DIRAC-100
38
50
MAS Hyperprior
M&S Hyperprior DIRAC-100
VTM-17.1
VTM-17.1 DIRAC-100
SwinT DIRAC-100 100
RID
PSAR
32
20
30
10
bpp [4]
bpp 4
Figure 8: We apply DIRAC-100, trained with mean-and-scale hyperprior base codec, to recon- structions from other base codecs, namely JPEG and VTM-17.1, on the CLIC2020 test set. Without finetuning, our model can still tradeoff fidelity and perceptual quality effectively.
C.2 COMPARISON TO IMAGE ENHANCEMENT APPROACHES
As do not train model end-to-end, it also be viewed as image enhancement method. We therefore compare it to two recent image enhancement works that focus on perceptual quality. While it is difficult achieve perfect configuration parity, both works enhance version of VTM and report scores on CLIC dataset. We take DIRAC-10 trained on hyperprior base codec and apply top of VTM-17.1 without finetuning. Our scores calculated on the same dataset that each method reports on
Kim et al. 2020 train a model on the CLIC2020 train dataset combined with MS COCO Lin
enhance VTM-7.3. They report scores the CLIC2020 test set, and use the Percep- Iual Index (P)) as perceptual metric, which is defined as:
al. 2014
PI (10 Ma) NIQE
(15)
Ma a no-reference quality metric defined in (best quality).
Ma ct 2017), going from 0 (poorest quality) to 10
Wang et al. 2022 base their approach on the Enhancement Compression Model (ECM which turn built on VTM. They train model to enhance its output an in-loop fashion, meaning it's not post-processing approach. As recent enhancement approach focused on perceptual quality, we still decided to include it. They train on DIV2K [Agustsson & Timofte 2017) and report scores on the CLIC2022 validation set, using LPIPS perceptual metric.
Even though we apply our model on VTM without finetuning, we achieve performance comparable to WWan al. (2022) with similar perceptual quality improvement, at the cost of more PSNR. We assume that with additional finetuning, our approach would outperform theirs.
C.3
PERCEPTUAL METRICS
In Fig. 10
we show additional metrics, namely the Inception Score (IS) (Salimans et al. 2016 Kernel Inception Distance (KID) (inkkowkki al. 2022 and NIQE MMittal et al. 2013 IS and KID are distribution metrics and thus not very reliable on Kodak, which only contains 24 images. As
22

---

Under review as a conference paper at ICLR 2023
5.50
34
Type3 "Spee
CUC2000 tosk o00
33
4.50
12
4.5
4.00
3.75
33
22
CLX2302 va Iaaionn
p3i c4h
0.25
020
29
28
Figure We compare DIRAC-100 with a VTM-17.1 base codec (trained on a hyperprior base codec, without finetuning) with two different methods, namely the work by Kim et al. (2020) (top row) and the ArabicaPerceptuul method from Wang al. (202)) (bottom row). The former reports PI (see Eq. 15 as perceptual metric CLIC2020 test set, while the latter reports LPIPS on the
CLIC2022 validation dataset. Note that computation very expensive, and results for our model will be added as soon available.
result, we observe negative values for HifiC and our model, because we use an unbiased estimator. While overall the relative performance is similar that seen for FID (see Fig we were surprised to see unstable scores for HiFiC in NIQE. All other methods, including ours, show a smooth decay with rate increase.
C.4 QUALITATIVE RESULTS
We include more qualitative results below, including additional baselines JPEG and VTM. Dis- tributed over Fig. Fig. 12 Fig. 13 we show 9 images from the CLIC 2020 test set, with selected crops zoom salient regions.
23

---

Under review as a conference paper at ICLR 2023
Inception
15
13
0.025
0.520
Arr
0.015
0.020
20-
Inception Score
KID|43
NOE
0.07
8.0
7.0
0.05
5.0
DIRAC-106 (percep- quality
0.03
Hypororior (Mince 2018)
HisE (Menkoer 20201
0.61
JREG
45
0.00
0.4
1.0
Figure 10: Additional perceptual metrics for CLIC2020 (top) and Kodak dataset (bottom). Note that distribution metrics (Inception Score and KID) are not reliable for small datasets like Kodak. Solid black lines show original data scores for no-reference metric NIQE.
24

---

Under review as a conference paper at ICLR 2023 3
Figure 11: Qualitative examples (part 1). Left panel shows reference, then within each panel each row shows zooms for JPEG, VTM, DIRAC at sampling step I. 80 and 100, and finally HIFIC, All methods are roughly the same file size, Best viewed electronically,
MESS DEON
MID--3 E50
The MONATEE
25

---

Under review as a conference paper at ICLR 2023 3
Figure 12: Qualitative examples (part 2). Left panel shows reference, then within each panel each row shows zooms for JPEG, VTM, DIRAC at sampling step I. 80 and 100, and finally HIFIC. All methods are roughly the same file size, Best viewed electronically,
SO & THAO - J6l
THE THA T&
J
Sill THA -
TO -
26

---

Under teview as a conference paper at ICLR 2023 3
Figure 13: Qualitative examples (part 3). Left panel shows reference, then within each panel each row shows zooms for JPEG, VTM, DIRAC at sampling step I. 80 and 100, and finally HIFIC. All methods are roughly the same file size, Best viewed electronically
9TN-13 17 CLE
BIRN-- TE CATTA
BIEES I SAA
BIRAC LO
BIRAC
ORO FIE
The BRANO
27


Reviews:
[{"review_id": "_yPR3k0FJq", "reviewer": "Reviewer_gzCy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper uses a residual diffusion-based model to augment a learned image compression model. The combined model has two main advantages. First, the diffusion post-processing improves perceptual quality (at the expense of increased distortion, i.e. FID improves while PSNR falls). Second, the approach leads to a single model that can target different different distortion-perception trade-offs at inference time.\n\nThe model is also multirate meaning that a single model can target different bit rates. This capability alone is not novel, but the end result here is a single compression model that can be adjusted at runtime to target different points in the 3D rate-distortion-perception space. As far as I know (and as the authors claim) this is the first model to have this flexibility / capability.\n\nThe final contribution of the paper is showing that the model can skip a large percent of the diffusion steps with only a minor reduction in perceptual quality (as measured by FID). It's not entirely clear *why* this works, but the empirical demonstration is convincing, and the authors argue that it implies that the noise schedule is suboptimal.\n\n\n\n", "summary_of_the_review": "I have a \"marginal accept\" score because a flexible model in terms of rate, distortion, and perceptual is very interesting, and I haven't seen that before. Using diffusion for image compression makes sense but isn't particularly novel on its own. As mentioned above, further comparison to GAN-based methods and via a user study would strengthen the paper as would outperforming HiFiC, the generative image compression baseline.\n", "strength_and_weaknesses": "The primary strength of the paper is a the creation of a single model that can flexibly target different points in rate-distortion-perception space. The use of a residual diffusion model allows the model to target perceptual quality. Diffusion models can be slow (especially in pixel-space) but the authors mitigate this downside to some extent by showing that a large number of diffusion steps can be skipped with minimal degradation to perceptual quality (specifically, they show that up to 80 steps can be skipped when evaluating up to 100 diffusion steps).\n\nThere are two primary weaknesses of the paper. First, as mentioned above, the decoder needs to run a diffusion model (up to 20 steps), which means it's quite a bit slower than a typical neural image compression model (and these are already an order of magnitude slower than standard codecs). Second, even after 100 diffusion steps, the perceptual quality does not match/exceed the performance of HiFiC, a GAN-based image compression model that targets perceptual quality.\n\nTo some extent, the same argument can be made about distortion, i.e. DIRAC-1, which minimizes distortion (max PSNR) is not SOTA. However, since the approach uses two separate models, it seems fair to say that any codec could be plugged in to the first stage (which generates a low-distortion image) and thus non-SOTA PSNR is not a deficiency of the approach per se.\n\nFinally, a user study is always appreciated when perceptual quality is a goal. So that would strengthen the paper, but I don't think publication should require it so long as reasonable perceptual metrics (like FID and LPIPS which are reported) are used.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "V7cZLr03UjB", "reviewer": "Reviewer_G974", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a post-processing method for neural image compression using a diffusion probabilistic model to improve the perceptual quality of the reconstruction results . With a variable-rate model that has been fully trained, no more bits need to be coded. The fidelity and perceptual quality are determined by adding residuals to different degrees with controllable sampling steps.", "summary_of_the_review": "The paper proposes a good choice for compressed image enhancement. Further studies should be conducted to improve the novelty.", "strength_and_weaknesses": "Strengths\n1. This work enables users to choose if an image is decoded with high fidelity or high perceptual quality by changing the sampling steps of the diffusion-based residual augmentation module.\n2. This work shows that in the compression setting, the number of sampling steps can be reduced dramatically, although typical diffusion probabilistic models are notoriously expensive to sample from.\n\nWeaknesses\n1. This work belongs to the post-processing category, and the correlation with compression itself is not convincing and attractive to some extent. That is to say, the diffusion method, which has been illustrated as effective ways in many other low-level tasks (e.g., restoration, super-resolution) can be easily migrated in this compression task. As a comparison, the work HiFiC is an end-to-end trained method using generative loss to improve perceptual quality. The novelty of this work, especially for the specific compression task is quite limited. \n2. The curve diagrams that are appeared in the paper are not very intuitive for understanding, especially Fig.5. The author should modify them.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "00mEcxEoVX", "reviewer": "Reviewer_Yvxp", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors applied the diffusion model to the decoded image of the deep image compression model (Minnen 2018) learned with MSE, aiming to improve perceptual quality. Experimental results show that post-processing with the diffusion model improve LPIPS and FID than the original model (Minnen 2018).\nThe authors also show that the degree of post-processing can be adjusted at the test phase by early stopping the diffusion sampling.", "summary_of_the_review": "Although the direction of using the diffusion model to improve the visual quality of compressed images is interesting, but i think the contents of this paper need to be reconsidered because of the difficulty in seeing the important contributions of the proposed method, the lack of sufficient evaluation, and the insufficient comparison with related studies.", "strength_and_weaknesses": "Strength\n- It may be new that the diffusion process was utilized as a quality enhancement post-processing network for the compressed image, allowing the degree of post-processing can be adjusted at the test phase by early stopping the diffusion sampling.\n\nWeakness\n- The compression part used the conventional Minnen 2018, and the architecture of the diffusion process used Dhariwal & Nichol 2021 and Preechakul 2022, it seems no particular innovations. Although the authors combined these with the loss function but they did not seem to success to learning the effective entire end-to-end process, so the research contents is considered to be more like an image enhancement task with the diffusion model. (In this sense, it may be interesting to consider and verify to what extent the diffusion model is effective in reducing compression artifacts for other codecs, including traditional ones.)\n- Perceptual quality was only evaluated by LPIPS and FID, and no subjective image quality evaluation was performed.\n- Since this paper is considered to be close to the task of image enhancement, i think it should be quantitatively compared with other approaches of enhancement networks that aim to improve the perceptual quality of compressed images (e.g. Y. Kim et al., Towards the Perceptual Quality Enhancement of Low Bit-rate Compressed Images, CVPRW 2020).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject