Lossy Compression with Gaussian Diffusion
Authors: Lucas Theis
Tim Salimans
Matthew Douglas Hoffman
Fabian Mentzer
Venue: Submitted to ICLR 2023 (2023)
Keywords: diffusion
compression
information theory

Abstract: We consider a novel lossy compression approach based on unconditional diffusion generative models, which we call DiffC. Unlike modern compression schemes which rely on transform coding and quantization to restrict the transmitted information, DiffC relies on the efficient communication of pixels corrupted by Gaussian noise. We implement a proof of concept and find that it works surprisingly well despite the lack of an encoder transform, outperforming the state-of-the-art generative compression method HiFiC on ImageNet 64x64. DiffC only uses a single model to encode and denoise corrupted pixels at arbitrary bitrates. The approach further provides support for progressive coding, that is, decoding from partial bit streams. We perform a rate-distortion analysis to gain a deeper understanding of its performance, providing analytical results for multivariate Gaussian data as well as theoretic bounds for general distributions. Furthermore, we prove that a flow-based reconstruction achieves a 3 dB gain over ancestral sampling at high bitrates.

Under review as a conference paper at ICLR 2023
LOSSY COMPRESSION WITH GAUSSIAN DIFFUSION
Anonymous authors Paper under double-blind review
ABSTRACT
We consider novel lossy compression approach based unconditional diffusion generative models, which we call DiffC. Unlike modern compression schemes which rely transform coding and quantization to restrict the transmitted informa- tion, DifIC relies on the efficient communication of pixels corrupted by Gaussian noise. We implement proof of concept and find that it works surprisingly well despite the lack of an encoder transform, outperforming the state-of-the-ar gen- erative compression method HiFiC on ImageNet 64x64. DiffC only uses single model to encode and denoise corrupted pixels at arbitrary bitrates. The approach further provides support for progressive coding, that is, decoding from partial bit streams. We perform a rate-distortion analysis to gain a deeper understanding of its performance, providing analytical results for multivariate Gaussian data as well as theoretic bounds for general distributions. Furthermore, we prove that flow-based reconstruction achieves a 3 dB gain over ancestral sampling at high bitrates.
INTRODUCTION
We are interested in the problem of lossy compression with perfect realism. As in typical lossy compression applications, goal is to communicate data using few bits possible while simultancously introducing as little distortion as possible. However, we additionally require that reconstructions X have (approximately) the same marginal distribution as the data, X ~ X. When this constraint is met, reconstructions are indistinguishable from real data or, in other words, appear perfectly realistic. Lossy compression with realism constraints is receiving increasing attention as more powerful generative models bring solution ever closer within reach. Theoretical arguments
(Blau and Michaeli| 2018 2019 Theis and Agustsson 202T Theis and Wagner 202T) and empirical
results (T schannen et al., 2018 Agustsson et al. 2019) Mentzer et al. (020) suggest that generative
compression approaches have the potential to achieve significantly lower bitrates at similar perceived quality than approaches targeting distortions alone.
The basic idea behind existing generative compression approaches is to replace the decoder with a conditional generative model and to sample reconstructions. Diffusion models (Sohl Dickstein et al.
2015;
Ho et al.] 2020) also known as score-based generative models (Song et al.] 202 Dockhorn class of generative models which have recently received lot of attention for
allso
al. 2022
their ability to generate realistic images (e.g.. Dhariwal and Nichol 2021 Nichol et al.) 2021 Ho
al. 2022 Kim and Ye 2022 Ramesh et al. (002)). While generative compression work has mostly
relied on generative adversarial networks (Goodfellow et al. 2014 Tschannen et al. 2018 Agustsson
al. 2019 Mentzer et al. 2020 Gao et | 2021).011) Saharia al. (222)) provided evidence that this
approach may also work well with diffusion models by using conditional diffusion models for JPEG
artefact removal.
In Section 3 we describe a novel lossy compression approach based on diffusion models. Unlike typical generative compression approaches, our approach relies on an unconditionally trained genera- tive model. Modern lossy compression schemes comprise at least an encoder transform, decoder transform, and an entropy model (Ballé et al. 2021 Yang et al. 2022) where our approach only uses single model. Surprisingly, we find that this simple approach can work well despite lacking an encoder transform- cransform-inseedd, we add isotropic Gaussian noise directly to the pixels (Section 5). By using varying degrees of Gaussian noise, the same model can further be used to communicate data at arbitrary bitrates. The approach is naturally progressive, that is, reconstructions can be generated from an incomplete bitstream.

---

Under review as a conference paper at ICLR 2023
To better understand why the approach works well, we perform rate- ante-distottion analysis in Section 4 We find that isotropic Gaussian noise is generally not optimal even for the case of Gaussian distributed data and mean-square error (MSE) distortion. However, also observe that isotropic noise is close to optimal. We further prove that a reconstruction based on the probability flow ODE (Song al. 2 cuts the distortion in half at high bit-rates when compared ancestral sampling from the diffusion model.
We will use capital letters such as X denote random variables, lower-case letters such x to denote corresponding instances and non-bold letters such as xi for scalars. We reserve log for the logarithm to base and will use In for the natural logarithm.
2 RELATED WORK
Many previous papers observed connections between variational autoencoders (VAEs; Kingma and Welling 20144) Rezende al. 2014) and rate-distortion optimization (e.g.. Theis al. 2017 Ballè
et al. [0117] 1017jAlemi et al. 2018 Brekelmans et al. 2019 Agustsson and Theis 2020). More closely related to our approach, Agustsson and Theis (2020) turned VAE into a practical lossy compression
scheme by using dithered quantization to communicate uniform samples. Similarly, our scheme relies on random coding to communicate Gaussian samples and uses diffusion models, which can be viewed as hierarchical VAEs with fixed encoder.
Ho et al. (2020) considered the rate-distortion performance of an idealized but closely related compression scheme based on diffusion models. In contrast to Ho et al. (2020), we are considering distortion under a perfect realism constraint and provide the first theoretical and empirical results demonstrating that the approach works well. Importantly, random coding is known to provide little benefit and can even hurt performance when only targeting a rate-distortion trade-off (Agustsson and Theis 2020 Theis and Agustsson 2021). On the other hand, random codes can perform significantly better than deterministic codes when realism constraints are considered Thhii and Agustsson (202)) Ho al. (2020) contemplated the use of minimal random coding (MRC: Havasi et al.) (219)) to encode Gaussian samples. However, MRC only communicates an approximate sample. In contrast, we consider schemes which communicate an exact sample, allowing us to avoid issues such as error propagation. Finally, we use an upper bound instead of a lower bound as proxy for the coding cost, which guarantees that our estimated rates are achievable.
While modern lossy compression schemes rely on transform coding, very early work by Roberts (9622) experimented with dithered quantization applied directly to grayscale pixels. Roberts (1962) found that dither was perceptually more pleasing than the banding artefacts caused by quantization. Similarly, we apply Gaussian noise directly to pixels but additionally use a powerful generative model for entropy coding and denoising.
Another line of work in compression explored anisotropic diffusion to denoise and inpaint missing pixels (Galié et al.] 2008). This use of diffusion is fundamentally different from ours. Anisotropic diffusion has the effect of smoothing an individual image whereas the diffusion processes considered in this paper are increasing high spatial frequency content of individual images but have smoothing effect on the distribution over images.
Yan et al. 2021) claimed that under a perfect realism constraint, the best achievable rate is R(D/2), where R is the rate-distortion function (Eq. 7 It was further claimed that optimal performance can be achieved by optimizing an encoder for distortion alone while ignoring the realism constraint and using ancestral sampling at the decoder. Contrary to these claims, we show that our approach can exceed this performance and achieve up dB better signal-to-noise ratio at the same rate (Figure The discrepancy can be explained by Yan et al. (202T) only considering deterministic codes whereas we allow random codes with access to shared randomness. In random codes, the communicated bits not only depend on the data but are function of the data and an additional source of randomness shared between the encoder and the decoder (typically implemented by a essuddorandoo number generator). Our results are in line with the findings of Theis and Agustsson (2021) who showed on a toy example that shared randomness can lead to significantly better performance in the one-shot setting, and those of Zhang et al. (2021) and Wagner (2022) who studied the rate- nate-distortion perortiionneeeepttion function (Blau and Michaeli 2019) of normal distributions. In this paper, we provide additional
results for the multivariate Gaussian case (Section 4.2
2

---

Under review as a conference paper at ICLR 2023
DiffC-F/A HiFiC
BPG
JPEG
A
B
0.2015 0,2062
0,2734
0.2793
0,2695
27.5d8
25./dB
23.4dB
23.GdB
18.5d8
D.2785
0:2719
0:2754
0.2852
gZs k,+1, x)
p(zz (a+t1)
25
22.
1dB
19.8d3
0.1661
0,1659
0.2245
0.2402
0.2363
ZT
Zr-1
Z+t1
Ze
Z
Z1
X
28.0d8
25. .d8
24.5dB
23.1dB
19.1d
Figure I: A: A visualization of lossy compression with unconditional diffusion models. B: Bitrates (bits per pixel; black) and PSNR scores (red) of various approaches including JPEG (4:2:0, headerless) applied to images from the validation set of ImageNet 64x64. For more examples see Appendix !|
increasing number of neural compression approaches targeting realism (e.g.. Tschannen et al. 2018 Agustsson et al. 2019: Mentzer et al. 2020 Gao et al. 2021 Lytchier al. 2021 Zeghidour ct al. 2022). However, virtually all of these approaches rely on transform coding combined with distortions based on VGG (Simonyan and Zisserman 2015) and adversarial losses (Goodfellow et al. 2014). contrast, we use single unconditionally trained diffusion model (Sohl-Dickstein et al. 2015). Unconditional diffusion models have been used for lossless compression with the help of bits- back coding (Kingma et al. 2021) but bits-back coding by itself is unsuitable for lossy compression. We show that sigmificant bitrate savings can be achieved compared lossless compression even by allowing imperceptible distortions (Fig.
3 LOSSY COMPRESSION WITH DIFFUSION
The basic idea behind our compression approach is to efficiently communicate corrupted version of the data,
Z, = 1
where ~~ (0,1).
(1)
from the sender the receiver, and then use diffusion generative model to generate a recon- struction. Z, can be viewed as the solution to a Gaussian diffusion process given by the stochastic differential equation (SDE)
1 B,Ze dt + BBdW,, Zo X, where of = dz. -
B,
dr
and W, Brownian motion. Diffusion generative models try invert this process by learning the conditional distributions p(z, z,)) for < (Song et al. (202]). If s and f are sufficiently close, then this conditional distribution approximately Gaussian. We refer SSohi-Dickstein al. (2015). Ho et al. 2020), and (Song et al. (202T) for further background on diffusion models.
Noise has negative effect the performance of typical compression schemes (Al-Shaykh and Mersereau 1998). However, [Bennett and Shor (2002) proved that it is possible to communicate an instance of Z, using not much more than X, bits. Note that this mutual information decreases as the level of noise increases. Li and El Gamal (2018) described more concrete random coding approach for communicating an exact sample of Z, (Appendix A An upper bound was provided for its coding cost, namely
|X Z,] + Log [[ Z,] + +
bits. Notice that the second and third term become negligible when the mutual information is sufficiently large. If the sender and receiver do not have access the marginal of Z, but instead
3

---

Under review as a conference paper at ICLR 2023
assume the marginal distribution to be pt, the upper bound on the coding cost becomes Theis and Yosri 2022)
C2 + log(Ct +
where C. = Ex [
X)
|| Pe (2+)]l
and q the distribution of Ze given X, which in our case is Gaussian. In practice, the coding cost
can be significantly closer to C, than the upper bound (Theis and Yosri 2022 Flamich et al. 2022).
We refer to Theis and Yosri 2022) for introduction to the problem of efficient sample commu- nication-all also known as reverse channel coding-as well as discussion of practical implementations of the approach of Li and El Gamal (2018). To follow the results of this paper, the reader only needs to know that an exact sample of a distribution can be communicated with a number of bits which is at most the bound given in Eq. 4 and that this is possible even when q continuous. The bound above is analogous to the well- result that the cost of entropy coding can be bounded in terms of H 1, (e.g..Coveer Cover and Thomas 2006). However, to provide some intuition for reverse channel coding, we briefly describe the high-level idea. Candidates Z Z7 Z..... are generated by drawing samples from pt. The encoder then selects one of the candidates with index N manner similar to rejection sampling such that Z ~ q. Since the candidates are independent of the data, they can be generated by both the sender and receiver (for example, using a pseudo-random number generator with the same random seed) and only the selected candidate's 's index N needs to be communicated. The entropy of N* is bounded by Eq. 4 Further details and pseudocode are provided in Appendix
where H is a cross-entropy (e.g..
Unfortunately, Gaussian diffusion models do not provide us with tractable marginal distributions Pt. Instead, they give us access to conditional distributions p(z, Zu+1) and assume pr is isotropic Gaussian. This suggests scheme where first transmit an instance of Zr and then successively refine the information received by the sender by transmitting an instance of Z, given Z++1 until Z, is reached. This approach incurs an overhead for the coding cost of each conditional sample (which we consider in Fig. [0] of Appendix []. Alternatively, we can communicate a Gaussian sample from the joint distribution ( X) directly while assuming marginal distribution p(zT:)). This achieves a coding cost upper bounded by Eq. where
C E (DKL X) Pr (ZT)]] E [DKL]g(z, Ie41,
p(zz Zs+1)]l- (5)
Reverse channel coding still poses several unsolved challenges in practice. In particular, the scheme proposed by Li and El Gamal (2018) is computationally expensive though progress on more efficient schemes is being made (Agustsson and Theis, 2020 Theis and Yosri 20222 Flamich al. 2022). In the following we will mostly ignore issues of computational complexity and instead focus on the question of whether the approach described above is worth considering all. After all, it is not immediately clear that adding isotropic Gaussian noise directly to the data would limit information in useful way.
We will consider two alternatives for reconstructing data given Z,. First, we will consider ancestral sampling, X ~ p(x Z.), which corresponds to simulating the SDE in Eq. 2ii reverse (Song et al. 202T). Second, we will consider a deterministic reconstruction which instead tries to reverse the ODE
dz; = (-2 Bizi -
-2" [.VVInp((z)) dt.
(6)
(202 showed that this "probability flow" ODE produces the same trajectory of marginal distributions pt as the Gaussian diffusion process in Eq. 2 and that it can be simulated using the same model of V In (z.)). We will refer to these alternatives as DiffC-A when ancestral sampling is used and DiffC-F when the flow-based reconstruction is used
Maoutsa et al. (2020) and [ong et al.
4 A RATE-DISTORTION ANALYSIS
In this section we try to understand the performance of DiffC from rate-distortion perspective. This will be achieved by considering the Gaussian case where optimal rate-distortion trade-offs can be computed analytically and by providing bounds on the performance in the general case. Throughout this paper, we measure distortion in terms of squared error. For our theoretical analysis we will further assume that the diffusion model has learned the data distribution perfectly.
4

---

Under review as a conference paper at ICLR 2023
The (information) rate-distortion function is given by R(D) = infx [[,,X]
subject to
B|| - {| D.
It measures the smallest achievable bitrate for given level of distortion and decreases as D increases
The rate as defined above does not make any assumptions on the marginal distribution of the reconstructions. However, here we demand perfect realism, that is. X ~ X. To achieve this constraint, a deterministic encoder requires a higher bitrate of D/2) (Blau and Michaeli 2019 Theis and Agustsson 2021). As we will see below, lower bitrates can be achieved using random codes as in our diffusion approach. Nevertheless, R(D/2) serves as an interesting benchmark as most existing codecs deterministic codes, that is, the bits received by the decoder are solely determined by the data.
For an M-dimensional Gaussian data source whose covariance has eigenvalues X the rate-distortion function is known to be (Cover and Thomas. 2006)
(D) E. log(A./ ( D:) where D; min(A,, 8)
for some threshold 9 chosen such that D Σ, D, For sufficiently small distortion D and assuming positive eigenvalues, we have constant D, = D/M.
4.1 STANDARD NORMAL DISTRIBUTION
As simple first example, consider standard normal distribution X (N(0,)) 1). Using ancestral sampling, the reconstruction becomes
x= v-v1-=2 -2 oV U, V ~ NO, 1) and we have dropped the dependence to reduce clutter. The distortion and rate this case are easily calculated to be
where Z -=VI-02 o2x oU,
1 2 - (D(22). 2 D
D = - 20
7[x, 2| = -og
log
(10)
This matches the performance of an optimal deterministic code. However, Z already has the desired standard normal distribution and adding further noise it did nothing to increase the realism or reduce the distortion of the reconstruction. The flow-based reconstruction instead yields dZ, = 0 and X Z (by inserting the standard normal for p: in Eq. ( resulting the smaller distortion
D= = = X)
E[( 2)²1= = 2
(11)
4.2 MULTIVARIATE GAUSSIAN
Next, let us consider X ~ NO, Σ) and Z -1- + UU where U ~ (O I). Assume X; are the eigenvalues of Σ. Since both the squared reconstruction eгтоr and the mutual information between X and Z are invariant under rotations of X, we can assume the covariance to be diagonal. Otherwise we just rotate X diagonalize the covariance matrix without affecting the results of our analysis. If X ~ P(X Z), we get the distortion and rate (Appendix C
(12) That is, the performance is generally worse than the perfor- mance achieved by the best deterministic encoder. We can modify the diffusion process improve the rate-distortin performance of ancestral sampling. Namely, let V, ~ A/ 1).
D = | X - x
I Z = 1 Σ, log(A,/D.) (D/2).
where D. = (x =
Z: V 72 A.U.
X.
V 2 + 74 A.V.
(13)
where 7f min 8/x;) for some 0. This amounts to using different noise schedule along different principal directions instead of adding the same amount of noise in all directions. For natural images,
'The bitrate given by an information rate- dit-diitootion may only be achievable asymptotically by encoding many data points jointly. To keep our discussion focused, we ignore any potential overhead incurred by one-shot coding and use mutual information as proxy for the rate achieved in practice.
5

---

Under review as a conference paper at ICLR 2023
A
25
B
25
DiffC-A
20
20
DiffC-F
SNR [dB]
SNR [dB]
DiffC-A
15
DifiC-F
10
10
P-A
5
5
P-F
0
0.5
1.5 2
2.5
0
64
128
192
256
Rate, /[X, Z] [bpd]
Component
Figure 2: A: Rate-distortion curves for a Gaussian source fitted to 16x 16 image patches extracted from ImageNet 64x64. Isotropic noise performs nearly as well the optimal noise (dashed). As an additional point comparison, we include pink noise (P) matching the covariance of the data distribution. The curve of DifIC-A" corresponds to R (D/2). A flow-based reconstruction yields up to dB better signal-to- monal-to-noise ratio (SNR). B: SNR broken down by principal component. The level of noise here is fixed to yield rate of approximately 0.391 bits per dimension for each type of noise. Note that the SNR of DifíC-A" is zero for over half of the components.
the modified schedule destroys information in figuffrequeccy components more quickly (Fig. 2B) and for Gaussian data sources again matches the performance of the best deterministic code.
D = D And 2E,
T| = E, log 7 E, log(A, R2 (14)
where D; = Ax min(A;, 0). Still better performance can be achieved via flow-based reconstruc- tion. Here, isotropic noise again suboptimal and the optimal noise a flow-based reconstruction is given by (Appendix
02 Z, = a V 1- af where a; = -V /Ai
(15)
for some Z already has the desired distribution and we can set X Z.
We will refer to the two approaches using optimized noise as DifIC-A" and DiffC-F", respectively, though strictly speaking these types of noise may longer correspond to diffusion processes. Figure ZA shows the rate-distortion performance of the various noise schedules and reconstructions on the example of 256-dimensional Gaussian fitted 16x16 grayscale image patches extracted from 64x64 downsampled ImageNet images (van den Oord et al. 2016). Here, SNR = 10 10 10 1o [ I))..
|||||| (12
4.3 GENERAL DATA DISTRIBUTIONS
Considering more general source distributions, first result bounds the rate of DifiC-A".
Theorem 1. Let X Ω + R be a random variable with finite differential entropy. zero mean and covariance diag (AI, .MM) Let U ~ (N( and define
Zi v 1- /
X~ Z).
(16)
where 7/ min(1, (/)x))ffeeeo for some 0. Further, let X* be Gaussian random variable with the same first and second-ordi moments as and let Z be defined analogously to but in terms of X*. Then if R is the rate-distortion function of X and R is the rate-distortion function of X*,
X Z) < R (D/2) DxL. [Pz Pz- S R(D/2) + [Px || Px-] DRa. Pz-] (17)
where = |||| X X |1.
Proof. See Appendix

---

Under review as a conference paper at ICLR 2023
9.1719
0.5554
0.2421
0.1916
0.1297
0.0538
0.0256
10.4572
0.6486
0.2554
1974
0.1240
C.0429
0.0198
32.4dB
25./dB
25.4dB
23.3dB
18.6dB
1d.rdB
31./dB
25.8dB
24.
22.4dB
19.3dB
16.3dB
14.1717
1.3065
0.4438
0.3257
0.1877
0.0611
0.0243
9.7106
0.7521
0.2842
0.2191
0.1383
0.0946
0.0265
27 9dB
22 8dB
21 0dB
19.4dB
17.2dB
12 9d
30 8dB 25. 1dB
3dB
22 3dB
17.7dB
16.5dB
Figure 3: Top images visualize messages communicated at the estimated bitrate (bits per pixel) shown in black. The bottom row shows reconstructions produced by DiffC-F and corresponding PSNR values are shown in red.
In line with expectations, this result implies that when X is approximately Gaussian, the rate of DiffC-A" not far from the rate of the best deterministic encoder, R(D/2). further implies that the rate is close to R( D/2) in the high bitrate regime if the differential entropy of X is finite. This can be seen by noting that the second KL divergence will approach the first KL divergence as the rate increases, since Pz- Px- and the distribution of Z will be increasingly similar to X.
Our next result compares the error of DifTC-F with DifiC-A's 's at the same bitrate. For simplicity. we assume that X has smooth density and further consider the following measure of smoothness,
G = E V [x)](
(18)
Among distributions with a continuously differentiable density and unit variance, the standard normal distribution minimizes and achieves G 1. For comparison, the Laplace distribution has G (Alternatively, imagine sequence of smooth approximations converging to the Laplace density.) For discrete data such as RGB images, we may instead consider the distribution of pixels with an imperceptible amount of Gaussian noise added to it (see also Fig. 5 Appendix F).
Let Ze be defined as in Eq. | XA ~ P(X Zi) and let R = Zo be the solution to Eq. with Ze as initial condition. Then
Theorem 2. Let X Ω + RM have a smooth density with finite G (Eq. 18
X
X|
lim a.
E XA |
=
2
(19)
Proof. See Appendix
This result implies that in the limit of high bitrates, the error of flow-based reconstruction is only half that of the the reconstruction obtained with ancestral sampling from perfect model. This is consistent with Fig. 2 where we can observe an advantage of roughly 3 dB of DiffC-F over DiffC-A. Finally, we provide conditions under which a flow-based reconstruction is provably the best reconstruction from input corrupted by Gaussian noise.
Theorem 3. Ler = QS where Q is orthogonal matrix and S Ω > RM is random vector
with smooth density and S, JL. S, for all i j. Define Z, as in Eq. 1f X Zo is the solution to the ODE in Eq. given Zi as initial condition, then
E|J[X= - X|| < E - -X
(20)
for any & with X Ze which achieves perfect realism, X X.
Proof. See Appendix G
7

---

Under review as a conference paper at ICLR 2023
60
40
BPG
50
35
HiFiC
PSNR [dB]
40
30
HiFiC (pretrained)
FID
30
25
DiffC-F
DiffC-A
20
20
10
15
10
0.5
1.5
2.5
0.5
1
1.5
2.5
Bits per pixel
Bits per pixel
Figure 4: A comparison DiffC with BPG and the GAN-based neural compression method HiFiC in terms of FID and PSNR on ImageNet 64x64.
EXPERIMENTS
As a proof of concept, we implemented DiffC based on VDM Kingma et al. 2021). VDM is diffusion model which was optimized for log-likelihood (i.e., lossless compression) but not for perceptual quality. This suggests VDM should work well in the high bitrate regime but not necessarily lower bitrates. Nevertheless,
find that we achieve surprisingly good performance across wide range of bitrates. We used exactly the same network architecture and training setup as Kingma et al. (202T) except with a smaller batch size of 64 images and training our model for only 1.34M updates (instead of 2M updates with batch size of 512) due to resource considerations. We used 1000 diffusion steps.
5.1 DATASET, METRICS, AND BASELINES
We used the downsampled version of the ImageNet dataset (Deng et al. 2009) (64x64 pixels) first used by van den Oord et al. (2016). The test set of ImageNet is known contain many duplicates and to overlap with the training set (Kolesnikov et al.] 2019). For a more meaningful evaluation (especially when comparing non-neural baselines), set as well as 744 images also occuring in the training set (based on SHA-256 hashes of the images). On this subset, we measured negative ELBO of 3.48 bits per dimension for our model.
removed 4952 duplicates from the validation
We report FID (Huseel et al. 2017) and PSNR scores to quantify the performance of the different approaches. As is common the compression literature, in this section calculate a PSNR score for each image before averaging. For easier comparison with our theoretical results, we also offer PSNR scores calculated from the average MSE (Appendix[] although the numbers do not change markedly, When comparing bitrates between models, we used estimates of the upper bound given by Eq. 4 for DiffC.
We compare against BPG (Bellard 2018), strong non-neural image codec based on the HEVC video codee which known for achieving good rate-distortion results. We also compare against HiFiC (Mentzer al.) 2020), which is the state-of-the-a generative image compression model in terms of visual quality on high-resolution images. The approach is optimized for a combination of LPIPS (Zhang al. 2018), MSE, and an adversarial loss (Goodfellow et al. 2014). The architecture of HiFiC is optimized for larger images and uses significant downscaling. We found that adapting the architecture of HiFiC slightly by making the last/first layer of the encoder/decoder have stride instead of stride 2 improves FID on ImageNet 64x64 compared to the publicly available model. In addition to training the model from scratch. we also tried initializing the non- od-addpted filters from the public model and found that this improved results slightly. We trained HiFiC models targeting 5 different bitrates.
8

---

Under review as a conference paper at ICLR 2023
5.2 RESULTS
We find that DiffC-F gives perceptually pleasing results even at extremely low bitrates of around 0.2 bits per pixel (Fig. 3) Reconstructions are also still perceptually pleasing when the PSNR is relatively low at around 22 dB (c.g.. compare to BPG in Fig. [B). We further find that at very low bitrates. HiFiC produces artefacts typical for GANs while we did not observe similar artefacts with DiffC. Similar conclusions can be drawn from our quantitative comparison, with DifiC-F significantly outperforming HiFiC in terms of FID. FID scores of DifíC-A were only slightly worse (Fig. 4T).
At high bitrates, DiffC-F achieves PSNR roughly 2.4 dB higher than DiffC-A. This is line with our theoretical predictions (3 dB) considering that the diffusion model only approximates the true distribution. PSNR values of DiffC-F and DifíC-A both exceed those of HiFiC and BPG, suggesting that Gaussian diffusion works well a rate-distortion sense even for highly non-Gaussian distributions (Fig. 4B). Additional results are provided in Appendix [ including results for progressive coding and HiFiC trained for MSE only.
DISCUSSION
We presented and analyzed new lossy compression approach based on diffusion models. This approach has the potential to greatly simplify lossy compression with realism constraints. Where typical generative approaches use an encoder, decoder, an entropy model, an adversarial model and another model as part of a perceptual distortion loss, and train multiple sets of models targeting different bitrates, DifIC only uses single unconditionally trained diffusion model. The fact that adding Gaussian noise to pixels achieves great rate-distortion performance raises interesting questions about the role of the encoder transform in lossy compression. Nevertheless, we expect further improvements are possible in terms of perceptual quality by applying DiffC in a latent space.
Applying DiffC in dimer-dimensionall transform space would also help to reduce its computational cost (Vahdat et al., 2021 Rombach et al., 2021 Gu et al.) 2021 Pandey et al.) 2022). The high computational cost of DifCC makes it impractical in its current form. Generating single image
with VDM requires many diffusion steps, each involving the application of a deep neural network.
However, speeding up diffusion models is highly active area of research (e.g.. Watson al. 2021
Vahdat et al.] 2021 Jolicoeur-Martinaau Martineau et al. [202]] Kong and Ping 2021; Salimans and Ho 2022
Zhang and Chen 2022). For example. [Salimans and Ho (2022) were able to reduce the number of diffusion steps from 1000 to around 4 at comparable sample quality. The computational cost of communicating sample using the approach of Li and Hl Gamal (2088) grows exponentially with the coding cost. However, reverse channel coding is another active area of research (e.g.. [avaaii ctt al., 2019 Agustsson and Theis 2020 Flamich et al. 2020) and much faster methods already exist for Low-dimensional Gaussian distributions (Theis and Yosri 2022 Flamich et al. 2022). Our work offers strong motivation for further research into more efficient reverse channel coding schemes.
As mentioned in Section 3 reverse channel coding may be applied after each diffusion step send sample of [Zz Ze+1 X). or alternatively to the joint distribution g(Z X). The former approach
has the advantage of lower computational cost due to the exponential growth with the coding cost. Furthermore, the model's 's score function only needs to be evaluated once per diffusion step to compute conditional mean while the latter approach requires many more evaluations (one for each candidate considered by the reverse channel coding scheme). Fig. I0 shows that this approach-whin already much more sractical-still significantly outperforms HiFiC., Another interesting avenue to consider is replacing Gaussian g(z, Z,+1. X) with a uniform distribution, which can be simulated very efficiently (e.g.. Zamir and Feder 1996 Agustsson and Theis 2020).
is
We provided an initial theoretical analysis of DiffC. In particular, we analyzed the Gaussian case and proved that DifiC-A" performs well when either the data distribution close to Gaussian or when the bitrate is high. In particular, the rate of DiffC-A" approaches R(D/2) at high birates. We further proved that DifIC-F can achieve 3 dB better SNR at high bitrates compared to DifIC-A. Taken together, these results suggest that R(D) may be achievable at high bitrates where current approaches based on nonlinear transform coding can only achieve RD/22). However, many theoretical questions have been left for future research. For instance, how does the performance of DiffC-A differ from DiffC-A" A And can we extend Theorem to prove optimality of a blowd reconstruction from noisy data for broader class of distributions?
9

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
Our appendix provides extensive proofs of the claims made in our paper. Every effort has been made make explicit the assumptions in our claims. The models used in our empirical results (VDM, HiFiC) are based on open source code. We further provide code to reproduce the results of Fig. 2
ETHICS STATEMENT
Lossy compression achieving perfect realism produces reconstructions which are indistinguishable from real data and which hide loss of information. This property desirable in applications such as video streaming for entertainment purposes but can be problematic in applications such as surveillance, medical imaging, document compression where the image content influences critical decisions. The availability of better generative compression methods increases the risk of misuse in these areas.
REFERENCES
E. Agustsson and L. Theis. Universally Quantized Neural Compression. In Advances in Neural Information Processing Systems 33, 2020.
E. Agustsson, M. Tschannen, F. Mentzer, R. Timofte, and L. V. Gool. Generative adversarial networks for extreme learned image compression. Proceedings of the IEEE International Conference on Computer Vision, pages 221-231, 2019.
O. Al-Shaykh and R. Mersereau. Lossy compression of noisy images. IEEE Transactions on Image Processing, 7(1 7(12):1641-163) 1998. doi: N0.1109/83.730377
A. Alemi, B. Poole, I. Fischer, J. Dillon, R. A. Saurous, and K. Murphy. Fixing broken elbo. In International Conference on Machine Learning, pages 159-168. PMLR, 2018.
Ballé, Laparra, and Simoncelli. End-to-end Optimized Image Compression, International Conference Learning Representations, 2017.
J. Ballé, P. A. Chou, D. Minnen, S. Singh, N. Johnston, E. Agustsson, S. J. Hwang, and G. Toderici. Nonlinear transform coding. IEEE Journal of Selected Topics in Signal Processing, 15(2):339-353, 2021. doi: 10.1 N0.1109/ISSSPP22000333355
Bellard. BPG Image format. httpp::/bbllarrd.orrgg
2018.
C. H. Bennett and P. W. Shor. Entanglementt-Asisted Capacity of Quantum Channel and the Reverse Shannon Theorem. IEEE Trans. Info. Theory, 48(10), 2002.
Y. Blau and T. Michaeli. The perceptiion-distorioonn tradeoff. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6228-6237, 2018.
Y. Blau and Michacli. Rethinking lossy compression: The rate-distortion perecpttiion-neepetiionoo tradeoff. In International Conference Machine Learning, 2019.
R. Brekelmans, D. Moyer, A. Galstyan, and G. Ver Steeg. Exact rate-distortion in autoencoders via echo noise. In Advances in Neural Information Processing Systems, volume 32, 2019.
T. M. Cover and J. A. Thomas. Elements of Information Theory 2nd Edition (Wiley Series in Telecommunications and Signal Processing). Wiley, 2006.
Deng, K. Li, M. Do, H. Su, and L. Fei-Fei. Construction and Analysis of a Large Scale Image Ontology. Vision Sciences Society, 2009.
P. Dhariwal and A. Nichol. Diffusion models beat GANs image synthesis. Advances in Neural Information Processing Systems, 34, 2021.
Dockhom, A. Vahdat, and K. Kreis. Score-based generative modeling with critically-damped langevin diffusion. In International Conference Learning Representations (ICLR), 2022.
10

---

Under review as a conference paper at ICLR 2023
G. Flamich, M. Havasi, and J. M. Hernández-Lobato Compressing Images by Encoding Their Latent Representations with Relative Entropy Coding, 2020. Advances in Neural Information Processing Systems 34.
G. Flamich, S. Markou, and J. M. Hemánneez-Lobto. Fast relative entropy coding with a" coding, 2022.
I. Galić, J. Weickert, M. Welk, A. Bruhn, A. Belyaev, and H.-P. Seidel. Image compression with anisotropic diffusion. Journal of Mathematical Imaging and Vision, 31(2):255 269, 2008. doi: 10. 1007/$10851-008-0008--
URL btttps://doo www.oi.orgg//// 1007
S. Gao, Y. Shi, T. Guo, Z. Qiu, Y. Ge. Z. Cui, Y. Feng. Wang, and B. Bai. Perceptual learned image compression with continuous rate adaptation. In 4th Challenge on Learned Image Compression, Jun 2021.
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672-2680, 2014.
S. Gu, D. Chen, J. Bao, F. Wen, B. Zhang, D. Chen, L. Yuan, and B. Guo. Vector quantized diffusion model for text-to-image synthesis. arXiv preprint
2021.
M. Havasi, R. Peharz, and J. M. Hernndeez-Lobato. Minimal Random Code Leaminng Getting Bits Back from Compressed Model Parameters. In International Conference on Learning Representa- tions, 2019.
M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems, pages 6626-6637, 2017.
J. Ho, A. Jain, and Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, volume 33, pages 6840-6851. Curran Associates, Inc., 2020.
J. Ho, C. Saharia, W. Chan, D. J. Fleet, M. Norouzi, and T. Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23(47):1-33, 2022.
A. Jolicoeur- Martineau, K. Li, R. Piché-Taillefer, T. Kachman, and I. Mitliagkas. Gotta Go Fast When Generating Data with Score-Base Based Models. arXiv preprint arXiv:2 105. /4080, 2021.
G. Kim and C. Ye. DiffusionCLIP: Text-guided image manipulation using diffusion models. In IEEF/CVF Conference Computer Vision and Pattern Recognition, 2022.
D. Kingma and M. Welling. Auto-encoding variational Bayes. In International Conference Learning Representations, 2014.
D. P. Kingma, T. Salimans, B. Poole, and J. Ho. On density estimation with diffusion models. In
A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
A. Kolesnikov, L. Beyer, X. Zhai, J. Puigcerver, J. Yung. S. Gelly, and N. Houlsby. Big Transfer (BiT): General Visual Representation Learning, 2019.
S. Kolouri, K. Nadjahi, U. Simsekli, R. Badeau, and G. Rohde. Generalized sliced wasserstein distances. In Advances in Neural Information Processing Systems, volume 32, Curran Associates, Inc., 2019.
Z. Kong and W. Ping. On fast sampling of diffusion probabilistic models. arXiv preprint 1rX.v:2106.00132. 2021.
C. Li and A. El Gamal. Strong Functional Representation Lemma and Applications to Coding Theorems. IEEE Transactions Information Theory, 64(11):6967-6978, 2018. doi: 10.1 109/TTT. 2018.2865570.
A. Lytchier, J. Xu, C. Finlay, C. Cursio, V. Koshkina, C. Besenbruch, and A. Zafar. Perceptually- guided lossy image compression. In 4th Challenge Learned Image Compression, Jun 2021.

---

Under review as a conference paper at ICLR 2023
C. J. Maddison. A Poisson process model for Monte Carlo. In Perturbation, Optimization, and Statistics. MIT Press, 2016.
D. Maoutsa, S. Reich, and M. Opper. Interacting particle solutions of fokker-planck equations through moddient-og-ddesitty estimation. Entroyy, 22(8), 2020. ISSN 1099-4300. doi: 10.3390/622008802.
URL http:: //ww: www mdpi. com./ 1099 4300 20000999330//////82222222
Mentzer, G. D. Toderici, M. Tschannen, and E. Agustsson. High-fidelity generative image
compression. Advances in Neural Information Processing Systems, 33, 2020.
A. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen. GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint urXiv:21 10741, 2021.
K. Pandey, A. Mukherjee, P. Rai, and A. Kumar. Diffusevae: Efficient, controllable and fidelity generation from oow-iimenio latents. arXiv preprint arXiv:2201.00300 2022.
A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
D. J. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International Conference Machine Learning. 2014.
H. Robbins. An empirical Bayes approach to statistics. In Proceedings of the Third Berkeley Symposium Mathematical Statisties and Probability, volume 1, pages 157-163, 1956.
L. G. Roberts. Picture Coding Using Pseudo-Random Noise. IRE Transactions on Information Theory, 1962.
R. Rombach, A. Blattmann, Lorenz, P. Esser, and Ommer. High-resolutinn image synthesis with latent diffusion models. arXiv preprint arXiv:21 2021.
C. Saharia, W. Chan, H. Chang, C. Lee, Ho, T. Salimans, D. Fleet, and M. Norouzi. Palette: Image-to-Image Diffusion Models. CoRR, abs/21 0///2111..500 1.05826, 2021.
T. Salimans and J. Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations, 2022.
C. Shannon. Communication in the presence of noise. Proceedings the IRE, 37(1):00-21, 1949. doi: 10.1109/JRPRCC.199 1949..2390.199992299999999
K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition.
In International Conference on Learning Representations, 2015.
J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. International Conference on Machine Learning, pages 2256-2265. PMLR, 2015.
Y. Song, J. Sohh-Diccstein, P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based generative modeling through stochastic differential equations. In International Conference Learning Representations, 2021.
L. Theis and E. Agustsson. On the advantages of stochastic encoders. In Neural Compression Workshop at ICLR, 2021.
L. Theis and A. B. Wagner. A coding theorem for the aate-disttrtion-perrqppe function. In Neural Compression Workshop at ICLR, 2021.
L. Theis and N. Yosri. Algorithms for the Communication of Samples. In Proceedings of the 39th International Conference on Machine Learning, 2022.
L. Theis, W. Shi, A. Cunningham, and F. Huszár. Lossy image compression with compressive autoencoders. In International Conference on Learning Representations, 2017.
12

---

Under review as a conference paper at ICLR 2023
M. Tschannen, E. Agustsson, and M. Lucic. Deep generative models for distribution Histribution-preserving lossy compression. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
A. Vahdat, K. Kreis, and J. Kautz. Score-based generative modeling in latent space, 2021.
A. den Oord, N. Kalchbrenner, and K. Kavukcuoglu. Pixel Recurrent Neural Networks. In Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1747-1756. PMLR, 2016.
A. B. Wagner. The wew---oototon-Prrrppeee Tradeoff: The Role of Common Randomness, 2022. arXiv:2202.04147.
D. Watson, J. Ho, M. Norouzi, and W. Chan. Learning to efficiently sample from diffusion proba- bilistic models. CoRR, abs/2 00.003 2021.
Y. Wu. ECE598: Information-theoreet methods high-dimensional statistics, 2016.
Yan, Wen, R. Ying, C. Ma, and Liu. On perceptual lossy compression: The cost of perceptual reconstruction and optimal training framework. In Proceedings of the International Conference on Machine Learning (ICML), 2021.
Y. Yang, S. Mandt, and L. Theis. An introduction to neural data compression, 2022.
R. Zamir and M. Feder. Information rates of pre/post-filtes filtterd- dithered quantizers. IEEE Transactions Information Theory, (35):1340-1353, 1996. doi: 10. 1.09/18.532876
N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi. Soundstream: An end-to-end neural audio codec. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 30: 495-507, 2022. doi: 10.1 10..1009/AASLP2222.3333939
G. Zhang, J. Qian, J. Chen, and A. Khisti. Universal lossy compression. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
representations for
Q Zhang and Chen. Fast sampling of diffusion models with exponential integrator, 2022.
R. Zhang, P. Isola, A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as a perceptual metric. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018
A REVERSE CHANNEL CODING
Algorithm Encoding (Li and Gamal 2018 Theis and Yosri 2022)
Require: g, Wmin
n,n,s s" 0, 1,
2: repeat
3:
+- simulate(n.p)
4: +- 1 + exponeetiaal))
5:
<- l p(=)/q(z)
6:
i < then
7:
8" n + 8, n
8:
end if
9:
n
10: until
s" - t- Wmin
11:
return n²
13

---

Under review as a conference paper at ICLR 2023
Algorithm 2 Decoding
Require: np,P I: return s simulate(n",5)
For completeness, we here reproduce pseudocode by Theis and Yosri (2022) of the sampling scheme first considered by Maddison (2016) and later by Li and Gamal (2018) for the purpose of reverse
channel coding. Similar to rejection sampling, the encoding process accepts a candidate generating distribution p, a target distribution y, and bound on the density ratio
Wmin inn p(z) q(z))
(21)
The encoding process returns index N (random due the exponential noise) such that Zn-
follows the distribution (Algorithm []. Importantly, the algorithm produces an exact sample in
finite number of steps (Maddison 2016). Furthermore, the coding cost of N* is bounded by (Li and
El Gamal 2018)
H[N 1|x z + (X,, Z + +
+
(22)
and this bound can be achieved by entropy encoding N with Zipf distribution px(n) n- which has a single parameter
x=1+
(23)
[ Z te loge+1
In practice, the coding cost for Gaussians may be significantly lower than this bound.
In the encoding process, the function s simulate(n, p) returns the nth candidate Zn (in practice, this would be achieved with a pseudo-random number generator though could also imagine large list of previously generated and shared samples). The function exponential(1 produces a random sample from an exponential distribution with rate
Unlike encoding, decoding is fast as it only amounts to selecting the right candidate once N has been received (Algorithm)
B NORMAL DISTRIBUTION WITH NON-UNIT VARIANCE
The case of I-dimensional Gaussian with varying variance is essentially the same as for a standard normal. In both cases, Gaussian source is communicated through Gaussian channel. Let X ~0 A) and
V - x2x + oU
(24)
where before U ~ 1). Let & P(X Z). We define
82
(25)
π + X 2a
Then
1
1|1, - - - X
log - 12-
log o² = =-log-
(26)
2
and
D D= E - x = XE -
2X0²
(27)
due to Eq. 10 which tells us the squared error the standard normal X - as a function of the information contained in Z. Taken together, we again have
1 log
1[x, 2|
= Rd (D/2).
(28)
2
D/2
14

---

Under review as a conference paper at ICLR 2023
C MULTIVARIATE GAUSSIAN
Let X N(O,E) and let
Z V1 x2x oU
(29)
where U ~ NNO, I). Note that both the mutual information and the squared error are invariant under rotations of X. We can therefore assume that the covariance diagonal.
Σ = diag ..,.... Am).
(30)
Defining
0² 0 0² + Xi X,
(31)
as in Appendix B have
D == E - 81 = Σ B((X, - x2,9 =2 Σ AOo
(32)
and
1 2j = Σ 1[x,, 2,] = - 2 Σ
log o?
(33)
Let D, 'oo, then write
2 Σ log X D:
D D= Σ D.,
1|, 2
(34)
For fixed distortion D, the rate in Eq. 34 as function of D, is known be minimized by the so-called reverse water-filling solution given in Eq. 8 (Shannon 1949 Cover and Thomas 2006).
that is, D; min(A,, 0) where 0 must be chosen so that D 2 E, Dj- Hence, D, defined above is generally suboptimal and we must have
log A Dj
πx Z
R*(D/2).
(35)
D OPTIMAL NOISE SCHEDULE FOR FLOW-BASED RECONSTRUCTION
Lemma 1. Let X ~ (O, Σ) with diagonal covariance matrix
Σ = diag (h1, ;
(36)
and A > 0. Further let
= a,, + V1 ViVi.
x= Z.
(37)
where U ~ NN(0 I) and
a-- (<++ + (V==VV+++=-))) 00 /xa.
(38)
Then & achieves the minimal rate at distortion level D = BiJ[X - { among all reconstructions
satisfying the realism constraint X X.
Proof. The lowest rate achievable by a code (with access to source of shared randomness) is (This
and Wagner 2021
infx 1X, X We can rewrite the rate as
subject to E
X
|||||
D and X~ X.
(39)
inf /|x, X =
inf h
X
h
X
h
X
X
2h X
supx A|X, X].
(40)
15

---

Under review as a conference paper at ICLR 2023
That is, need to maximize the differential entropy of (X,X) subject to constraints. For the distortion constraint, we have
B|| - X | = E| X + E
X
2E X 2 - 2 S D.
(41)
We will first relax the realism constraint to the weaker constraints below and then show that the solution also satisfies the stronger realism constraint:
E E = O. E[X A
(42)
Consider relaxing the problem even further and jointly optimize over both X and X with constraints on the first and second moments of both random variables. The joint maximum entropy distribution then takes the form
p(x, 8) exp EBer Beas
En
Ne + Emar Σ Ug Σ EGmadd
(43)
that is, it is Gaussian with precision matrix which has zeros everywhere except the diagonal and the off-diagonals corresponding to the interactions between X; and X; In other words, the joint precision matrix S consists of four blocks where each block a diagonal matrix. is not difficult to
see by blockwise inversion that then the covariance matrix C = S must have the same structure. Let Cx& be the diagonal matrix corresponding to the covariance between X and X. We need to maximize
h X X + const x In |C|
(44)
In Cxxxxx - Cxx
(45)
Imn(s - E[X,X,13)
(46)
subject to
Σ E Σ A D
(47)
Let c = X and form the Lagrangian
((c,n.n)= = = -
-2) +1
D
+ Σ P4G.
(48)
where the last term is due to the constraint & 2 0. The KKT conditions are
G4
+
+ 14 = 0,
(49)
dc,
X2
of
D
Σ
Gi Σ
\ +
0,
P4c = 0,
(50)
2
8
Σ 4
Σ
A +
2 0, 2
42 0
(51)
#4 2 0,
17 7 0,
(52)
yielding e or
C =
X2
(53)
41²
2n
If = 0 for some i, then μ4 77 (Eq. 49 and therefore μr 17 0 by Eq. 52 But then G;
by
Eq. 49 for all i. By Eq. 5T] we must then have
D D X
(54)
16

---

Under review as a conference paper at ICLR 2023
This implies that for sufficiently large distortion, must have c for all i. Defining 0 (2n) gives that for to be optimal, we must have
E[X,X,] = A3 + 02
(55)
for some 0 determined by D. Summarizing what we have so far, we have shown that (under relaxed realism constraints) the rate is minimized by a random variable X jointly Gaussian with X and covariance matrix whose entries zero except those specified by Eqs. 42 and Since the marginal distribution of is Gaussian with the desired mean and covariance, it also satisfies the stronger realism constraint X ~ X.
On the other hand, X ~ X
X. In particular,
E E[x,X:] = 2,
(56)
1,,,,XX; + 11 (V
(57)
=,,
(58)
=ai
(59)
= / X² + - 0.
(60)
has the desired property. Thus, & minimizes the rate at any given level of distortion.
E PROOF OF THEOREM
Lemma 2. Let X : Ω RM be random variable with finite differential entropy and let X*
be a Gaussian random variable with matching first and second-order moments. Let R(D) be the rate-distortion function of X and R (D) be the rate-distortion function of X'. Then
(D) S R(D) + Dxx[Px || Px-].
(61)
Proof. Zamir and Feder (196) proved the result for M 1. We here extend the proof M >
First, observe that
DRI|Px Px-1
E[ log Px- (X) x[X] EE log px- X )] h X
(62)
(63)
[[x*) X
(64)
since log px is quadratic form and X and X have matching moments. By the Shannon lower bound Wu 2016).
M
D
R(D) 2 M[X
log
2Tee M
(65)
2
Let Mi... AM be the eigenvalues of the covariance of X. Then by Eqs. 64
and
65
we have
M
D
R(D) M D[Px, Px.
log
2
(66)
M
Σ log
1
AE D[Px, Px-]
(67)
2
D/M
log
DIPx, Px-1
(68)
2
Di
R (D) - D P.
(69)
where D; = min(0, >x) and 0 is such that D Σ D. The inequality follows from the optimality of the water- water-filling filling solution given by the D; (Shannon 1949 Cover and Thomas 2006). Bringing the KL divergence to the other side of the equation gives the desired result.
17

---

Under review as a conference paper at ICLR 2023
Theorem 1. Let X: : Ω + RM be a random variable with finite differential entropy, zero mean and AM)). Let U ~ ((,1)) and define
covariance diag
2 V 1 + A.U.
X ~ P<P 2).
(70)
where 7/ min(1, 0f X2) for some O. Further, let X be Gaussian random variable with the same first and second-ordld order moments as X and let Z be defined analogously to but in terms of X". Then if R. is the rate-distortion function of X and R is the rate-distortion function of X*,
1X, 2 - R (D/2) - [xL[P2 |1 Pz-1
(71)
S R DKL. Px || Px-] DKL. PPz Pz-
(72)
where D- = -B[|X |.
Proof. We have
D. = EI(XX: - x.)
(73)
= E[(X,; - E[X, 7 B Z - 11
(74)
[ E[X, Z (E|X, 7 X,) 2(X, B|X,
[E
Z X.)]
(75)
= E[(X, - E[X, Z) + E 2 X1
(76)
- -2Ez [ x -X. -XX- 21
3 E|X Z
& Z]]
(77)
-2E[(X, B[X, 2
(78)
s 2 - / 1 2,
(79)
2B[(1 - (1 22))x; V 1- 7- v A.U.)9
(80)
-2(Var/>/ + Varly 78- (-
(81)
= =2(7/>: + (1 - 7,)
(82)
= -27/>;.
(83)
where in Eq. 77 we used that X T X
Z and in Eq. 78 we used that (x, Z) (X. Z). Eq. 79
follows because the conditional expectation minimizes the squared error among all estimators of X, For the overall distortion, we therefore have
D B - E =D
(84)
Note that D is the distortion we would have gotten if X were Gaussian (Eq. 14
Define
V1 = (1-??) / and
Y. = (1 (1-2?) 7 = +
(85)
and let Y be the Gaussian random variable defined analogously to Y except in terms of z" instead of Z. To obtain the rate, first observe that
1X", z-] 1[X", Y"]
(86)
= [ X* + V]
(87)
h(X' + V] X +V X"]
(88)
= hXX' + - V]-M[V]
(89)
h/X + VJ- h[V]+ 4|X" + V] - -| VI
(90)
= h + VI - /|X V X + h + V - h|X + V]
991)
X[X,X + V - (ellogy-(YY")) Elloggy
(92)
= =T[X, Y - Ellogyy (Y')) + + ((oogyY(Y))
(93)
= =[[X, Y - E[log PY (Y) + (Y)))
(94)
= = Y + DKL Px Py-]
(95)
= Z + DRL. Pzz P.
(96)
18

---

Under review as a conference paper at ICLR 2023
400
350
300
250
G
200
150
100
50
0
0.2
0.4
0.6
0.8
dt
Figure While Go is undefined for images with discretized pixels, we may instead consider the distribution of pixels with imperceptible Gaussian noise added to it. We estimate the corresponding G. using the diffusion model, the results of which are shown this plot. G, converges M or approaches 1.
where the first step and last step follow from the invariance of mutual information and KL divergence under invertible transformations. Eq. (4 follows because log py- a quadratic form and Y and Y have matching moments. We therefore have
[ Z
_[ z-1 DKL [Pz || Pz-1
(97)
RD (2) D D ||
Pz-1
(98)
R (D/2) - DKL. Pz Pz.]
(99)
R(D/2) [xL[Px Px.] DKL[Pz || Pz-1,
(100)
where the second equality is due Eq. 14 the first inequality is due D D* and the second inequality follows from the Shannon lower bound and Lemma 2
F PROOF OF THEOREM 2
Theorem compares the reconstruction error of XA ~ P(X Z,) with & 2n where 2n the solution of
dz: (- B Bizr 3. dt.
(101)
given Z, = V1+ + U as initial condition. To derive the following results it is often more convenient to work with the "variance exploding" diffusion process
Y, = (1 of) - - Z X + of?) - too = + U
(102)
instead of the "variance preserving" process Z₁ (Song et al. 2021). We fr for the marginal density of Y, and reserve Pt for the density Z,
We further define the following quantity for continuously differentiable densities, which can be viewed as a measure of smoothness of a density,
G
E In (x)
(103)
It can be shown that when ||| = 1, we have G 2 M with equality when X is isotropic Gaussian. That is, the isotropic Gaussian is the smoothest distribution (with continuously differentiable density) as measured by G. We further define
G
|I 2 log (Z:) and
G = E Vy log p. (Y.)
(104)
which are linked by the chain rule, G, = (1 G, Using a trained diffusion model, we can obtain estimates of G, for ImageNet 64x64, which shown in Fig. 5
19

---

Under review as a conference paper at ICLR 2023
Lemma 3. Diffusion increases the smoothness of a distribution, G. S Go-
Proof. We have
du = / p(u V. In (y)
Vy (yre
(105)
/ p(u yt) Vy In
(e())p( ye) du
(106)
p(u ye)
S p(u V Inp())ppp u) / p(u V Imp(u yz) du
(107)
S p(u y.) Inp(y: [) - 0 f p(u y.) Vy Io nuu) dx
(108)
(109)
and therefore
G. B In Pr Y.)
(110)
E V y In po Y, - 77c U) Y.II
(111)
S |E In po (Y, - m/ U) Y.]] E|| V 1n po [3]
(112)
(113)
Go
(114)
due to Jensen's 's inequality.
We will also need the following known useful identity which is special case of Tweedie's formula (Robbins 1956). We include a derivation for completeness.
Lemma 4. Let X have a density and let Y., Pt, and qh be defined as above. Then
E yel = Yt + o V In (e(.).
(115)
Proof.
Vy (og)) = p(x yt Vy log (y)e) dx
(116)
f p(x y.) Vy log Pv( yc) dx
(117)
p(x yt)
f p(x yy) V log (p(y: xx)p(x)) dx
(118)
/ p(x y. v p(x y:) dx
(119)
/ p(x yxt) V logp(y; x) dx
(120)
[ p(x V.) log.N (yrix, dx
(121)
f p(x yt) 1 (x - ye) dx
(122)
(E|x
yr] y.)
(123)
L
The following three lemmas relate the reconstruction errors of X, and & to the smoothness of the source distribution measured by Go and Gy,,
20

---

Under review as a conference paper at ICLR 2023
Lemma 5. Let X have a density and let XA. ne- and G2 be defined above. Then
E|| X| 2n M 2n/ 2n M - 21/2 (1 o
(124)
(125)
Proof.
XA - X = | Y Y, - X
(126)
X Ye + Y: - X - -Y) X)]
(127)
Y.Il + ||| Y. - x1
(128)
2 Y, Y. E[ X Yi]
(129)
FI||X Y, + E, Y. x
(130)
+ + - Y, Y, E,Y, - X Y&]l
(131)
- - - - X Y.JI
(132)
-. - - Y
(133)
-2 ln
(134)
-2 2 V (Z,)
(135)
2n M -1=))G
(136)
Lemma 6. Let X have a density and let Rp, The and Go be defined as above. Then
E[| E[|| & - Y. - 7h Go-
(137)
Proof. We have
E
X Y. = (Y.) - - [X)
(138)
where F2 is the invertible function which maps x to yr according to the ODE dy: - V In (ye) dt, yo =,
(139)
where de relates to 77 as follows,
1i 22 / at dr.
(140)
Different schedules are equivalent up to reparametrizzatoon of the time parameter (Kingma et al. 2021). For now, assume the parametrization a = 1 (or n = 2t)). Integrating the above ODE then yields
y: F((x) = x Vn Vinp-(y-)dr
(141)
Consider the following Riemann sum approximation of F.,
N-1 L Vinp:, (yt..)
(x) x Σ
(142)
where I,, nt/N and yt., = F,,, (x). Since the gradient of the log-density is continuous and the integral over a compact interval, the partial derivatives are bounded inside the interval and the Riemann sum converges to
F((x)= = lim (x)
(143)
N-20
21

---

Under review as a conference paper at ICLR 2023
Thus,
B|||X - F(X)]]]== = ||||
lim
FL,N (X)11
(144)
N -0000
N 1
EI
lim
v Im (Y
(145)
N
n=
N 1
E
lim
In Pto (1
(146)
N-000
N
0
N-1
< E
lim
|V In pt. (Y )1
(147)
N
lim
E
V Pr. (Y 111)
(148)
1 N-1
lim
G..
(149)
N
12 N-1
lim
Σ
Go
(150)
Noc
N
n=0
Goo
(151)
1
T/ Go.
(152)
4
Eq. 147 again uses Jensen's 's inequality. Eq. 148 (swapping limit and expectation) follows from the dominated convergence theorem since each element of the sequence is bounded by Goo (Lemma 3
Lemma 7. Let X have a smooth density and let & and Go be defined as above. Then
1 nh + 2n (1 oZ)Gr.
B[|| X x < n M +
(153)
2
Proof.
8
X = E[[ & F E| Y, + HXX Y,J X
(154)
E|| X F X Y Y. B| Y.J - +
(155)
< E X X B -
(156)
= E
Xx - Y, + V (n YY) + -Ell U
(157)
1
{ Y.) + n1) v (V.) ||/
= E
+ +7iM M
(158)
2
S 2E|
Y, 2 Vnn
(159)
2 1 2n (1- -; + 7/M
(160)
< / Gr + (1-a:) M
(161)
where the first inequality is due to the conditional expectation minimizing squared error, the second inequality is due to Jensen's 's inequality and the last inequality is due to Lemma 6
We are finally in a position to prove Theorem
Theorem 2. Let X Ω + T have a smooth density with finite
G
(162)
22

---

Under review as a conference paper at ICLR 2023
Let Z, - U with U I.. Let XA P(X Zi) and ler Xx Zo be the solution to Eq. 6 with Zμ as initial condition. Then
B1J 8 X
2
lim OL
(163)
XA
X
Proof. The limit is to be understood the one-sided limit from above. We have
B
X
Mf + 1
2 27, + 2ny (1 of
lim
lim a1-10
(164)
x 1
21/ M 21//' (1 oz
M/M + 1 Go + 21/ Go
lim o +0
(165)
2m M Go
Go
2n, M + 20 Go Sij Go
lim
(166)
n-
4mp M 811 Go
2M + Go + 24 Go
lim
(167)
7--00
4M 24 Go
2M
(168)
4M
1
(169)
2
where the first inequality follows from Lemmas 5 and 7 and we applicd L' L'Hôpital's rule twice.
the second inequality is due to Lemma 3
G PROOF OF THEOREM
Theorem 3. LetX QS where Q is orthogonal matrix and S Q -> RM is random vector with smooth density and S, 11 Sj for all i + j. Define
Z. V xfx U where U~ U~N(01,)
(170)
& = Zog is the solution to the ODE in Eq. 6 Z, as initial condition, then
E|J| F 1 I BIJl'' - X 17
(171)
for any X with X' 11 X Z. which achieves perfect realism, X' ~ X.
Proof. Define the variance exploding diffusion process as
dYY = VGWW, with (1 (1- ( dd
(172)
so that
(173)
Further define F, the function which maps the solution of the ODE in Eq Eq with starting condition Zg x. Then F, is invertible (Song et al. (222)) and we can write XF F; (Z,). Further, let Fe be the corresponding function for the variance exploding process such that
(y) = FT (VI-o3y)
F:=
(174)
For arbitrary X with & 11 X Y,, we have
X
| = E|| E[X| Y,J + E Y, - X|
(175)
= E|| E[X Y, + E Y, - x
(176)
+ E[( - E|X
(E Y, X)]
(177)
E|||* E[X Y,
Y,] x
(178)
+ 1E,, [E [X E|X Y.)
Y, Ex [EX - Y.
(179)
=E|JI* E|X Y
Y.) x
(180)
23

---

Under review as a conference paper at ICLR 2023
Define XMSE (Y) E[X Y.).
Assume M = 1 so that = S. We first show that then t/e a monotone function of yr:
(ve) = dy
B| yrl
(181)
Y1 + n dy
ln (v))
(182)
dy
0
1 + 1l dy
In (1(e)
(183)
02 pe(y))p(x) Me) dx
=1+ n
p( v1)
In
(184)
oy²
p(æ: ve)
0²
0²
= +1+ nh
In
Pr 1x) dr 1h pla
ve) Dy²
In p(x 3e) dx
(185)
Oy
= 1 2nk Th / p((x vt) yy²
02
(ye - =))² dx + n ((ve)
(186)
1 f p( vi) Dy
① (ye 2) dx + m/ (((c)
(187)
= 1 / p(20 1 dr + nh. (ve)
(188)
= vi (Vee
(189)
In p(x vi) 2 dy
ni p(x: ve)
dx
(190)
0
(191)
where (y.) is the Fisher information of yt- Assume (f(())) = 0 for some yı. Then
In Ix ve) 0
(192)
dy
almost surely (Eq. 190p. Then also
p(ye x))(
In p(X vr)
In
In Inp(y. 1 X)
In (xn) =
(193)
ay
Oy
(v(
δy
or
θ By In p(y 1x)
2 Oy
n (y: X)
In (x)e
(194)
almost surely. This implies X is almost surely constant, that is, p(x: yr) is degenerate distribution. But this contradicts our assumption that p(x) is smooth. Since p(y: x) is Gaussian with mean x and therefore smooth as function of p(z yt) p(z)p(V, x) must also be smooth. Hence, must have v/ (yx) > everywhere. Since (vt)) is strictly monotone is also invertible.
Consider the squared Wasserstein metric.
W...x, P inf
E( {MSS)) X:X-x
(195)
where the infimum is over all random variables with the same marginal distribution as X and which may depend on &MSE (or equivalently may depend on Y,). The solution to this problem known from transportation theory to be X = - (& (c.g., Kolouri et al. 22019). where &o the CDF of X, & is the CDF of XMSE, and it assumed that the measure of X is absolutely continuous
24

---

Under review as a conference paper at ICLR 2023
with respect to the Lebesgue measure. We have
fr = P(X <x)
(196)
P(PT Sx)
(197)
P(Y S F((z))
(198)
S P.()))
(199)
=P(XMSE I (F(()))))
(200)
-(((()(((()))
(201)
implying
X 0 (0 - (x (8MSE))
(202)
(203)
(V))
(204)
-XF
(205)
and therefore that Xx is optimal.
Now let M > 1. Since E|| P - x is invariant under the choice of Q, we can assume Q =
without changing the results of our analysis so that X S and (X,, Yu) 11 (1)) for + j. Since then
Inpr(ze) Σ In Pr())))
(206)
the ODE (Eq. 6 can be decomposed into M separate problems
1 1 2 In pro(zes) di
dmn
2 Ozu
(207)
2
for which we already know the solution is of the form zu (1 (1- of) with
Yls Fee(z,)) أ5. (0 (Poo(z)))),
(208)
where T is the CDF of
X MSE = E Yısl = E[X, Y. = XMSE,
(209)
On the other hand,
inf x
|| &MSIel 11
inf Σ E((R, - 8)
(210)
X:X~x
inf
E((R, &MSSE )
(211)
8:X~X
Σ inf
B[(X; X )2
(212)
8xXixX:
Σ inf Xalii
B((,; {
(213)
Σ E(( 1 (Bes (XAMSE)) X
(214)
Σ B((AFF 8MSRA)]
(215)
E| & - -MMSF111.
(216)
That is, & minimizes the squared error among all reconstructions achieving perfect realism. The second inequality follows due to the weaker constraint on the right-hand side; X ~ X implies X; X; but not vice versa. Eqs. 214 and
and 215 follow from our proof of the case M = 1.
25

---

Under review as a conference paper at ICLR 2023
H COMPUTE RESOURCES
Training VDM took about 13 days using 32 TPUv3 cores (https://cloud. google. com/ tpp)) No hyperparameter searches were performed tune VDM for this paper. Training one HiFiC model took about 4 days using 2 V100 GPUs and we trained 10 models targeting 5 different bitrates (with and without pretrained weights). few additional training runs were performed for HiFiC tume the architecture (reducing the stride) while targeting a single bitrate.
26

---

Under review as a conference paper at ICLR 2023
1 ADDITIONAL FIGURES
7.0614
0.4737
0980
0.1608
0.1090
0.0504
0.0250
9.5251
0.7373
0.2474
0,1814
0.1106
0.0427
0.0219
32
27. 5dB
26. 3dB
24, 3dB
19 OdB
15.3dB
30 6dB
25. 1dB
24.0dB
22, 2dB
18.6dB
15. 9dB
11.3938
0,7859
0.2792
0.2096
0,1285
0.0465
0.0230
11.2516
0.9364
0.3099
0.2269
0.1345
0.0496
0.0222
30.6dB
25. 2dB
3dB
22 2d
19.
16.7dB
29.7cB
24.0dB
23.2dB
21.7dB
18.3dB
16.2dB
13.8554
1.0940
0.2784
0.1873
0.0941
0.0241
0.0087
7.6759
0.3733
0.1551
0.1181
0.0719
0.0201
0.0077
28.1dB
23.5dB
23.0dB
21.9dB
20.4dB
19.4dB
34.2dB
2r. 2r.9dB
25.6dB
23.1dB
20. 1dB
18
9./528
0./452
0.2926
0.2241
0.1368
0.0469
0.0192
10.8943
0./239
0.2/48
0.2041
0.12//
0.045/
0.0188
31.0dB
25 2dB
23 2dB
20. .d
17.8dB IdB
31.0dB
4dB
24.1dB
21B
18. 1BB
15B2dB
6.6616
0.6604
0.3295
0.2675
0.39
0.0867
0.0424
5.6494
0.4354
0.1626
0.1270
0.0858
0.0385
0.0210
32. 32.4dB
26. 4dB
24. 8dB
22 0dB
16. 1dB
13.9dB
32.9cB
28.0dB
26.5dB
24.8dB
20.4dB
19. OcB
9.0989
0.9002
0.3991
0.3213
0.2077
0.0762
0.3332
11.9855
1.3185
0.4928
0.3656
0.2213
0.0716
0.0284
31.4B
28B3dB
20B3dB
15.7dB
13.8dB
28.9dB
22.8dB
21.5dB
19.8dB
16.3dB
15.0dB
11.4337
1.0215
0 4202
0. 0.3283
0.2181
0.0932
0.0421
9.7464
0.8392
0.3434
0.2/24
0.1786
0.0/85
0.0389
29.9dB
23.9dB 2dB
20.7d8 15.5dB
14.4dB
30.7dB
25.2dB
23.1dB
20.9dB
17.7d8
14.0dB
Figure 6: Top images visualize messages communicated at the estimated bitrate shown in black. The left-most bitrate corresponds to lossless compression with our VDM model. The bottom row shows reconstructions produced by DiffC-F and corresponding PSNR values red.
27

---

Under review as a conference paper at ICLR 2023
DiffC-F/A
HiFIC
BPG
JPEG
DifiC-F/A
HiFIC
BPG
JPEG
0.3294
0.3294
0.3555
0.3438
0.3320
C,4241
0.4213
0.4238
0.4414
0.4043
B1.CoB
19.7dB
19. 3B
19. 99
17.5dB
23.9dB
22.1cB
19.4cB
19.5dB
16. 16.
0.2829
0.2622
0.2598
0.2520
0.2773
C,2590
0.2572
0.2910
0.2461
0.3008
25.2cB
23.3dB
21. 5d
21.6dB
19.6dB
25.8dB
24.2cB
21.8cB
22.3dB
19.4dB
0.1908
0.1908
0.1758
0.1582
0.1758
C.2879
0.2895
0.2988
0.2969
0.2871
23.CoB
21.6clBB
21.5dBB
23.0dBB
21.5d
25.1dB
23.5cB
22.4cB
21 21.8
18.2dB
0.2458
0.2445
0.2632
0.2656
0.2871
0.2510
0.2502
0.2578
0.2441
0.2520
26.7cB
25.2dB
22. 2d
22.0dB
19.7dB
25.1dB
23.7cB
22.3eB
22.0dB
18.7dB
0.2963
0.2968
0.2773
0.2451
0.2852
C.2305
0.2305
0.2520
0.7480
0.2422
25.4cB
23.0dlB
2dB
21.7dB
19.9dB
23.2dB
21.6cB
21 6
21. 4d
19. 3dB
0.3332
0.3286
0.416C
0.4756
0.4102
C.1586
0.1563
0.2051
0.1953
0.1855
199 10om
B
23. 3dB
20. 6dB
20. 20
17.6dBB
27.9dB
26.1dB
24.9cB
26.3dB 0.3203
23 6dB
0.4029
0.4100
0.3828
0.4219
0.3633
C.3694
0.3694
0.3789
0.3828
24.3dB
22.5dBB
19.8dB
19.7dB
16.9dB
21.5dB
19.4cB
19.7cB
19.9dB
18.9dB
Figure 7: Additional reconstructions generated by different compression methods. The left-most column shows the uncompressed image. Bitrates are shown in black and PSNR values in red.
28

---

Under review as a conference paper at ICLR 2023
40
BPG
35
HiFiC
HiFIC (pretrained)
30
PSNR
DiffC-F
25
DiffC-A
20
15
10
0
0.5
1
1.5
2
Bits per pixel
Figure 8: PSNR values in Section 5 were computed by calculating PSNR score for each image and averaging. In contrast, this plot shows PSNR values corresponding to average MSE.
60
40
DiffC-F
50
35
DiffC-F (100)
DiffC-F (40)
40
30
PSNR
DiffC-F (20)
30
DiffC-F (10)
HiFiC (pretrained)
20
10
15
10
0
0.5
1.5
2
0
0.5
1
1.5
2
Bits per pixel
Bits per pixel
Figure 9: Performance relative to upper bound on the coding cost when progressively communicat- ing information chunks of bits using the approach oi and El Gamal (2018). The coding cost estimated using (B log + 1) 5), where C, the total amount of information sent (Eq, 5). At 10 bits the PSNR is comparable to our strongest baseline but the FID remains significantly lower.
90
40
80
BPG
35
70
HiFiC
PSNR [dB]
60
30
HiFiC (pretrained)
FID
50
25
HiFiC (MSE)
40
DiffC-F
30
20
DiffC-A
20
15
10
10
0.5
1
1.5
2
2.5
0.5
1
1.5
2.5
Bits per pixel
Bits per pixel
Figure 10: This figure contains additional results for HiFiC trained from scratch for MSE only. We only targeted single bit-rate. The PSNR improves slightly while the FID score gets significantly worse.
29


Reviews:
[{"review_id": "hG7_ov3afGb", "reviewer": "Reviewer_sBei", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new lossy image compression paradigm based on unconditional diffusion models. The noise corrupted image is compressed and sent to the decoder via reverse channel coding, and the reconstruction of the input image is done through the denoising process applied to the received noise corrupted image. Effort was made to understand the theoretical RD behavior of such a new coding paradigm. The authors also try to compare it with BPG and some other learned codecs on ImageNet 64x64.\n", "summary_of_the_review": "This is a good theoretical paper. However, the experimental results on ImageNet64x64 are inconclusive and less convincing. BPG is not optimized for coding small-resolution images. This may also be true for HiFiC. As a result, it becomes unclear whether the proposed method is really worth further investigation. Complexity aspects are mentioned briefly but not addressed fully. It is expected that both the denoise process and reverse channel coding are complex. Their practicality is in question. Overall, this is an early attempt to use \"unconditional\" diffusion models for lossy image compression. It is to be noted that another late publication approaches lossy compression using \"conditional\" diffusion models and provides more empirical results. ", "strength_and_weaknesses": "Strengths:\n\n(1) The idea of using unconditional diffusion models for lossy image compression is novel. It has the striking feature that the encoding of an image can be done easily by corrupting the image with a (preferably non-isotropic) Gaussian noise. \n\n(2) The theoretical analysis presented in this paper is thorough and rigorous. \n\n(3) The authors made an early attempt to compare the proposed method with BPG and some learned codecs on ImageNet 64x64. \n\n(4) The conclusion section of this paper is insightful, and provides a good summary of the pros and cons of the proposed method. \n\nWeaknesses:\n\n(1) The experimental results on ImageNet64x64 are inconclusive and less convincing. Apparently, BPG is not optimized for coding small-resolution images. This may also be true for HiFiC. \n\n(2) While this is a good theoretical paper, the authors are alerted to a relevant work (Lossy Image Compression with Conditional Diffusion Models) that uses “conditional” diffusion models for lossy image compression. The proposed method is a direct contrast to this one.\nhttps://arxiv.org/abs/2209.06950\n\n(3) Complexity aspects are not fully addressed, although mentioned briefly. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "h5O294xTqf0", "reviewer": "Reviewer_JDSJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a diffusion based lossy image compression approach. In the evaluation, the proposed method outperforms a baseline. ", "summary_of_the_review": "As the major technical contribution of this paper is not clear, I recommend a reject. I would raise the score if the authors provide a good clarification.", "strength_and_weaknesses": "Strength:\n1. The topic of using diffusion for lossy compression is interesting. \n\nWeaknesses:\n1. It’s hard to justify the originality of the proposed main methodology. The contribution is marginal. As far as I understand, all technologies of this paper is introduced from other paper, like diffusion and reserve channel coding.\n2. Although the author claim they have some improvement, but they only compared with HIFIC.  As far as I know, there are plenty of lossy compression works every year [1, 2, 3]. The author should compare with these works, too.\n3. The author claim in abstract, that this method is efficient. However, there’s no following experiment or number that can prove this claim.\n4. The lossy compression techniques are also widely applied on video coding area, too. The author should add experiment on video compression.\n\n[1] Cui Z, Wang J, Gao S, et al. Asymmetric gained deep image compression with continuous rate adaptation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021: 10532-10541.\n[2] Xie Y, Cheng K L, Chen Q. Enhanced invertible encoding for learned image compression[C]//Proceedings of the 29th ACM International Conference on Multimedia. 2021: 162-170.\n[3] Deng X, Yang W, Yang R, et al. Deep homography for efficient stereo image compression[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021: 1492-1501.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7CjAfeEr1i", "reviewer": "Reviewer_22JQ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "[NOTE: the citations I make below can all be found in the bibliography of the paper under review.]\n\nThis paper studies methods for doing lossy compression using a diffusion model, under the constraint of 'perfect realism'. 'Perfect realism' is defined in the introduction to the paper as the requirement that the marginal distribution of the reconstructions approximately matches the marginal distribution of the data (perhaps 'approximate realism' might be a better term for this property?).\n\nA fairly detailed theoretical analysis was presented, as well as experiments which confirmed the analysis and showed that the method could perform well, assuming that the compression rate of a method in the paper of Li & El Gamal (2018) could actually be achieved. Unfortunately, the computational complexity of this method is exponential in the dimensionality of the latent space, and the authors did not attempt to implement actual compression.", "summary_of_the_review": "This was a very well written paper and enjoyable to read, but I have serious concerns about its substance, particularly regarding the (im)practicality of the method presented, the fact that the authors were unable to implement the method, and the similarity to existing work, particularly Ho et al. 2020, therefore I judge it to be marginally below the acceptance threshold for ICLR.", "strength_and_weaknesses": "### Strengths\n - The paper is very well written, and was enjoyable to read.\n - The paper was relatively honest about the limitations of the method.\n - I think the mathematical results were rigorous.\n### Weaknesses\n - I'm concerned about the practicality of the methods presented in the paper, and particularly the use of methods like that of Li and El Gamal, which have computational complexity that scales exponentially with dimension (there are a number of other papers with which I have the same issue, such as Ho et al. 2020; Flamich et al. 2020; and Havasi et al. 2019). Normally we would refer to such methods as intractible, rather than merely 'computationally expensive' (pg. 4, fourth paragraph). The paper under review does explicitly mention the exponential scaling, but only in the 'Discussion' section at the end of the paper. Given that we are unlikely to get a perfect model for natural images (at least not this year), approximate realism is the best we will be able to achieve anyway, and therefore I want to know why we can't try a method which we can actually run, such as straightforward deterministic quantization of the latents, at least as a baseline.\n - Related to the above point, I was very disappointed that the authors didn't manage to implement their compression method, even for small-scale data. The comparisons to other methods, both quantitative and qualitative, seem unfair given that (as far as I know) all of those methods have actually been implemented, whereas the methods in the paper under review have not. Particularly the claims of 'simplicity' of the method. If it's really so simple then why not implement it?\n - I also felt that the novelty of diffusion model lossy compression with 'perfect realism' was limited. In particular I am comparing the paper to Ho et al. 2020, which presented rate distortion results (also not implemented, and IMHO problematic for similar reasons to the paper under review) for a diffusion model, the only difference being the 'perfect realism' constraint. IIUC Ho et al. mentioned that one could do this (though they don't use the term 'perfect realism') just below eqn. 15 in their paper. This is not a criticism of the mathematical analysis, which I did not have enough time to thoroughly comprehend, I hope other reviewers are able to comment on the novelty and significance of this.\n\n### Minor points, typos and questions\n - In the first sentence of section 5, the authors say \"we implemented DiffC based on VDM\", perhaps this could be rephrased as \"we measured the performance of DiffC based on VDM\".\n - Typo in the second sentence of the second paragraph of section 5.2: \"This is line\" should be \" This is in line\".", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject