SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ruihan Yang
ruihan.yang@uci.edu
Stephan Mandt
mandt@uci.edu
This paper outlines an end-to-end optimized lossy image compression framework using diffusion generative models. The approach relies on the transform coding paradigm, where an image is mapped into a latent space for entropy coding and, from there, mapped back to the data space for reconstruction. In contrast to VAE-based neural compression, where the (mean) decoder is a deterministic neural network, our decoder is a conditional diffusion model. Our approach thus introduces an additional "content" latent variable on which the reverse diffusion process is conditioned and uses this variable to store information about the image. The remaining "texture" variables characterizing the diffusion process are synthesized at decoding time. We show that the model's performance can be tuned toward perceptual metrics of interest. Our extensive experiments involving multiple datasets and image quality assessment metrics show that our approach yields stronger reported FID scores than the GAN-based model, while also yielding competitive performance with VAE-based models in several distortion metrics. Furthermore, training the diffusion with  $\mathcal{X}$ -parameterization enables high-quality reconstructions in only a handful of decoding steps, greatly affecting the model's practicality. Our code is available at: https://github.com/buggyyang/CDC_compression
With visual media vastly dominating consumer internet traffic, developing new efficient codecs for images and videos has become evermore crucial (Cisco, 2017). The past few years have shown considerable progress in deep learning-based image codecs that have outperformed classical codecs in terms of the inherent tradeoff between rate (expected file size) and distortion (quality loss) (Ballé et al., 2018; Minnen et al., 2018; Minnen & Singh, 2020; Zhu et al., 2021; Yang et al., 2020; Cheng et al., 2020; Yang et al., 2023). Recent research promises even more compression gains upon optimizing for perceptual quality, i.e., increasing the tolerance for imperceivable distortion for the benefit of lower
rates (Blau & Michaeli, 2019). For example, adding adversarial losses (Agustsson et al., 2019; Mentzer et al., 2020) showed good perceptual quality at low bitrates.
Figure 1: Overview of our proposed compression architecture. A discrete "content" latent variable  $\hat{\mathbf{z}}$  contains information about the image. Upon decoding, this variable is used for conditioning a denoising diffusion process. The involved "texture" variables  $\bar{\mathbf{x}}_{1:N}$  are synthesized on the fly.
Most state-of-the-art learned codescels currently rely on transform coding and involve hierarchical "compressive" variational autoencoders (Ballé et al., 2018; Minnen et al., 2018; Cheng et al., 2020).
These models simultaneously transform the data into a lower dimensional latent space and use a learned prior model for entropy-coding the latent representations into short bit strings. Using either Gaussian or Laplacian decoders, these models directly optimize for low MSE/MAE distortion performance. Given the increasing focus on perceptual performance over distortion, and the fact that VAEs suffer from mode averaging behavior inducing blurriness (Zhao et al., 2017) suggest expected performance gains when replacing the Gaussian decoder with a more expressive conditional generative model.
This paper proposes to relax the typical requirement of Gaussian (or Laplacian) decoders in compression setups and presents a more expressive generative model instead: a conditional diffusion model. Diffusion models have achieved remarkable results on high-quality image generation tasks (Ho et al., 2020; Song et al., 2021b,a). By hybridizing hierarchical compressive VAEs (Balle et al., 2018) with conditional diffusion models, we create a novel deep generative model with promising properties for perceptual image compression. This approach is related to but distinct from the recently proposed Diff-AEs (Preechakul et al., 2022), which are neither variational (as needed for entropy coding) nor tailored to the demands of image compression.
We evaluate our new compression model on four datasets and investigate a total of 16 different metrics, ranging from distortion metrics, perceptual reference metrics, and no-reference perceptual metrics. We find that the approach yields the best reported performance in FID and is otherwise comparable with the best available compression models while showing more consistent behavior across the different tasks. We also show that making the decoder more stochastic vs. deterministic offers a new possibility to steer the tradeoff between distortion and perceptual quality (Blau & Michaeli, 2019). Crucially, we find that a certain parameterization- $\mathcal{X}$ -prediction (Salimans & Ho, 2022)-can yield high-quality reconstructions in only a handfull of diffusion steps.
In sum, our contributions are as follows:
- We propose a novel transform-coding-based lossy compression scheme using diffusion models. The approach uses an encoder to map images onto a contextual latent variable; this latent variable is then fed as context into a diffusion model for reconstructing the data. The approach can be modified to enhance several perceptual metrics of interest.
- We derive our model's loss function from a variational upper bound to the diffusion model's implicit rate-distortion function. The resulting distortion term is distinct from traditional VAEs in capturing a richer decoding distribution. Moreover, it achieves high-quality reconstructions in only a handful of denoising steps.
- We provide substantial empirical evidence that a variant of our approach is, in many cases, better than the GAN-based models in terms of perceptual quality, such as FID. Our base model also shows comparable rate-distortion performance with MSE-optimized baselines. To this end, we considered four test sets, three baseline models (Wang et al., 2022; Mentzer et al., 2020; Cheng et al., 2020), and up to sixteen image quality assessment metrics.
Transform-coding Lossy Image Compression The widely-established classical CODECs such as JPEG (Wallace, 1991), BPG (Bellard, 2018), WEBP (Google, 2022) have recently been challenged by end-to-end learned CODEcs (Ballé et al., 2018; Minnen et al., 2018; Minnen & Singh, 2020; Yang et al., 2020; Cheng et al., 2020; Zhu et al., 2021; Wang et al., 2022; He et al., 2022a). These methods typically draw on the non-linear transform coding paradigm as realized by hierarchical VAEs. Usually, neural CODEcs are optimized to simultaneously minimize rate and distortion metrics, such as mean squared error or structural similarity.
In contrast to neural compression approaches targeting traditional metrics, some recent works have explored compression models to enhance realism (Agustsson et al., 2019; Mentzer et al., 2020; Tschannen et al., 2018; Agustsson et al., 2022; He et al., 2022b). A theoretical background for these approaches was provided by Blau & Michaeli (2019); Zhang et al. (2021); in particular Blau & Michaeli (2019) discussed the existence of a fundamental tradeoff between distortion and realism, i.e., both goals can not be achieved at the same time in a given architecture. To realize this tradeoff in compression, Mentzer et al. (2020); Agustsson et al. (2022) optimized the autoencoder-based
compression model with an additional adversarial loss, while GAN training in this context can be unstable and requires a variety of design choices.
Diffusion Models Probabilistic diffusion models showed impressive performance on image generation tasks, with perceptual qualities comparable to those of highly-tuned GANs while maintaining stable training (Song & Ermon, 2019; Ho et al., 2020; Song et al., 2021b; Song & Ermon, 2019; Kingma et al., 2021; Yang et al., 2022; Ho et al., 2022; Saharia et al., 2022; Preechakul et al., 2022). Popular recent diffusion models include Dall-E2 (Ramesh et al., 2022) and Stable-Diffusion (Rombach et al., 2022). Some works also proposed diffusion models for compression. Hoogeboom et al. (2021) evaluated an autoregressive diffusion model (ADM) on a lossless compression task. Besides the difference between lossy and lossless compression, the model is only tested on low-resolution CIFAR-10 (Krizhevsky et al., 2009) dataset. Theis et al. (2022) used a generic unconditional diffusion model to lossily communicate Gaussian samples. Their method leverages "reverse channel coding", which is orthogonal to the usual transform-coding paradigm. While it's under active research, there is currently no practical method that can reduce its extensive computational cost without restrictive limitation (Li & El Gamal, 2018; Flamich et al., 2020, 2022; Theis & Ahmed, 2022).
Denoising diffusion models are hierarchical latent variable models that generate data by a sequence of iterative stochastic denoising steps (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2021a; Song & Ermon, 2019). These models describe a joint distribution over data  $\mathbf{x}_0$  and latent variables  $\mathbf{x}_{1:N}$  such that  $p_{\theta}(\mathbf{x}_0) = \int p_{\theta}(\mathbf{x}_{0:N}) d\mathbf{x}_{1:N}$ . While a diffusion process (denoted by  $q$ ) incrementally destroys structure, its reverse process  $p_{\theta}$  generates structure. Both processes involve Markovian dynamics between a sequence of transitional steps (denoted by  $n$ ), where
$$
q \left(\mathbf {x} _ {n} \mid \mathbf {x} _ {n - 1}\right) = \mathcal {N} \left(\mathbf {x} _ {n} \mid \sqrt {1 - \beta_ {n}} \mathbf {x} _ {n - 1}, \beta_ {n} \mathbf {I}\right); \tag {1}
$$
$$
p _ {\theta} (\mathbf {x} _ {n - 1} | \mathbf {x} _ {n}) = \mathcal {N} (\mathbf {x} _ {n - 1} | M _ {\theta} (\mathbf {x} _ {n}, n), \beta_ {n} \mathbf {I}).
$$
The variance schedule  $\beta_{n}\in (0,1)$  can be either fixed or learned; besides it, the diffusion process is parameter-free. The denoising process predicts the posterior mean from the diffusion process and is parameterized by a neural network  $M_{\theta}(\mathbf{x}_n,n)$ .
Denoising Diffusion Probabilistic Model (DDPM) (Ho et al., 2020) showed a tractable objective function for training the reverse process. A simplified version of their objective resulted in the following noise parameterization, where one seeks to predict the noise  $\epsilon$  used to perturb a particular image  $\mathbf{x}_0$  from the noisy image  $\mathbf{x}_n$  at noise level  $n$ :
$$
L \left(\theta , \mathbf {x} _ {0}\right) = \mathbb {E} _ {n, \epsilon} \| \epsilon - \epsilon_ {\theta} \left(\mathbf {x} _ {n} \left(\mathbf {x} _ {0}\right), n\right) \| ^ {2}. \tag {2}
$$
Above,  $n \sim \mathrm{Unif}\{1,\dots,N\}, \epsilon \sim \mathcal{N}(\mathbf{0},\mathbf{I}), \mathbf{x}_n(\mathbf{x}_0) = \sqrt{\alpha_n}\mathbf{x}_0 + \sqrt{1 - \alpha_n}\epsilon$ , and  $\alpha_{n} = \prod_{i = 1}^{n}(1 - \beta_{i})$ . At test time, data can be generated by ancestral sampling using Langevin dynamics. Alternatively, Song et al. (2021a) proposed the Denoising Diffusion Implicit Model (DDIM) that follows a deterministic generation procedure after an initial stochastic draw from the prior. Our paper uses the DDIM scheme at test time, see Section 3.2 for details.
Neural image compression seeks to outperform traditional image codecs by machine-learned models. Our approach draws on the transform-coding-based neural image compression approach (Theis et al., 2017; Ballé et al., 2018; Minnen et al., 2018), where the data are non-linearly transformed into a latent space, and subsequently discretized and entropy-coded under a learned "prior". The approach shows a strong formal resemblance to VAEs and shall be reviewed in this terminology.
Let  $\mathbf{z}$  be a continuous latent variable and  $\hat{\mathbf{z}} = \lfloor \mathbf{z} \rfloor$  the corresponding rounded, integer vector. The VAE-based compression approach consists of a stochastic encoder  $e(\mathbf{z}|\mathbf{x})$ , a prior  $p(\mathbf{z})$ , and a decoder  $p(\mathbf{x}|\mathbf{z})$ . The model is trained using the negative modified ELBO,
$$
\mathcal {L} (\lambda , \mathbf {x}) = \mathcal {D} + \lambda \mathcal {R} = \mathbb {E} _ {\mathbf {z} \sim e (\mathbf {z} | \mathbf {x})} [ - \log p (\mathbf {x} | \mathbf {z}) - \lambda \log p (\mathbf {z}) ]. \tag {3}
$$
The first term measures the average reconstruction of the data (distortion  $\mathcal{D}$ , usually measured by MSE), while the second term measures the costs of entropy coding the latent variables under the prior
Algorithm 1 Training the model (left); Encoding/Decoding data  $\mathbf{x}_0$  (right).  $\mathcal{X}$  -prediction model.
Sample x0 ~ dataset	Given Ntest
repeat	ˆz = [Encφ(x0)]
n ~ U(0,1,2,...,Ntrain)	ˆz←P(ˆz) binary file (entropy code using P(ˆz))
ε ~ N(0,I)
ˆx̅n = √αn x0 + √1-αn ε	ˆx̅N = 0 (or xN ~ N(0,γ2I) for stochastic decoding)
ˆz = Encφ(x0) + U(-0.5,0.5)
ˆx̅0 = Xθ(ˆx̅n, n/Ntrain,ˆz)	for n=NTest to 1 do
L_D = α_n/1-α_n | x_0 - x̅0|^2	εθ = x_n - √α_n x_θ(x_n(x_0), z, n/N) / √1-α_n
L = (1 - ρ)L_D + ρd(ˆx̅0, x_0) - λ log_2 P(ˆz)	ˆx̅0 = Xθ(ˆx̅n, n/Ntest,ˆz)
(θ,φ) = (θ,φ) - ε∇θ,φL (learning rate: ε)	ˆx̅n-1 = √α_n-1ˆx̅0 + √1 - α_n-1εθ
until converge	end for return x̅0
(bitrate  $\mathcal{R}$ ). The encoder  $e(\mathbf{z}|\mathbf{x}) = \mathcal{U}(\mathrm{Enc}_{\phi}(\mathbf{x}) - \frac{1}{2},\mathrm{Enc}_{\phi}(\mathbf{x}) + \frac{1}{2})$  is a boxed-shaped distribution that simulates rounding at training time through noise injection due to the reparameterization trick. Note its differential entropy equals zero.
Once the VAE is trained, we en/decode data using the deterministic components as  $\hat{\mathbf{z}} = \lfloor \mathrm{Enc}(\mathbf{x})\rfloor$  and  $\hat{\mathbf{x}} = \mathrm{Dec}(\hat{\mathbf{z}})$ . We convert the continuous  $p(\mathbf{z})$  into a discrete  $P(\hat{\mathbf{z}})$  using  $P(\hat{\mathbf{z}}) = \mathrm{CDF}_p(\hat{\mathbf{z}} + 0.5) - \mathrm{CDF}_p(\hat{\mathbf{z}} - 0.5)$ , where  $\mathrm{CDF}_p$  is the cumulative distribution function of  $p(\mathbf{z})$ ; see (Yang et al., 2023) [Sec. 2.1.6] for details. The discrete prior is used for entropy coding  $\hat{\mathbf{z}}$  (Balle et al., 2018).
While VAE-based approaches have used simplistic (e.g., Gaussian) decoders, we show that can get significantly better results when defining the decoder  $p(\mathbf{x}|\mathbf{z})$  as a conditional diffusion model.
3.2 Conditional Diffusion Model for Compression
The basis of our compression approach is a new latent variable model: the diffusion variational autoencoder. This model has a "semantic" latent variable  $\mathbf{z}$  for encoding the image content, and a set of "texture" variables  $\mathbf{x}_{1:N}$  describing residual information,
$$
p \left(\mathbf {x} _ {0: N}, \mathbf {z}\right) = p \left(\mathbf {x} _ {0: N} \mid \mathbf {z}\right) p (\mathbf {z}). \tag {4}
$$
As detailed below, the decoder will follow a denoising process conditioned on  $\mathbf{z}$ . Drawing on methods described in Section 3.1, we use a neural encoder  $e(\mathbf{z}|\mathbf{x}_0)$  to encode the image. The prior  $p(\mathbf{z})$  is a two-level hierarchical prior (commonly used in learned image compression) and is used for entropy coding  $\mathbf{z}$  after quantization (Ballé et al., 2018). Next, we discuss the novel decoder model.
Decoder and training objective We construct the conditional denoising diffusion model in a similar way to the non-variational diffusion autoencoder of Preechakul et al. (2022); Wang et al. (2023). In analogy to Eq. 1, we introduce a conditional denoising diffusion process for decoding the latent  $\mathbf{z}$ ,
$$
p _ {\theta} \left(\mathbf {x} _ {0: T} \mid \mathbf {z}\right) = p \left(\mathbf {x} _ {N}\right) \prod p _ {\theta} \left(\mathbf {x} _ {n - 1} \mid \mathbf {x} _ {n}, \mathbf {z}\right) = p \left(\mathbf {x} _ {N}\right) \prod \mathcal {N} \left(\mathbf {x} _ {n - 1} \mid M _ {\theta} \left(\mathbf {x} _ {n}, \mathbf {z}, n\right), \beta_ {n} \mathbf {I}\right). \tag {5}
$$
Since the texture latent variables  $\mathbf{x}_{1:N}$  are not compressed but synthesized at decoding time, the optimal encoder and prior should be learned jointly with the decoder's marginal likelihood  $p(\mathbf{x}_0|\mathbf{z}) = \int p(\mathbf{x}_{0:N}|\mathbf{z})d\mathbf{x}_{1:N}$  while targeting a certain tradeoff between rate and distortion specified by a Lagrange parameter  $\lambda$ . We can upper-bound this rate-distortion (R-D) objective by invoking Jensen's inequality,
$$
\mathbb {E} _ {\mathbf {z} \sim e (\mathbf {z} | \mathbf {x} _ {0})} [ - \log p (\mathbf {x} _ {0} | \mathbf {z}) - \lambda \log p (\mathbf {z}) ] \leq \mathbb {E} _ {\mathbf {z} \sim e (\mathbf {z} | \mathbf {x} _ {0})} \left[ L _ {\text {u p p e r}} (\mathbf {x} _ {0} | \mathbf {z}) - \lambda \log p (\mathbf {z}) \right],
$$
where we introduced  $L_{\mathrm{upper}}(\mathbf{x}_0|\mathbf{z}) = -\mathbb{E}_{\mathbf{x}_{1:N}\sim q(\mathbf{x}_{1:N}|\mathbf{x}_0)}\left[\log \frac{p(\mathbf{x}_{0:N}|\mathbf{z})}{q(\mathbf{x}_{1:N}|\mathbf{x}_0)}\right]$  as the variational upper bound to the diffusion model's negative data log likelihood (Ho et al., 2020). We realize that  $L_{\mathrm{upper}}(\mathbf{x}_0|\mathbf{z})$  corresponds to a novel image distortion metric induced by the conditional diffusion model (in analogy to how a Gaussian decoder induces the MSE distortion). This term measures the model's ability to reconstruct the image based on  $\mathbf{z}$ . In contrast,  $-\log p(\mathbf{z})$  measures the number of bits needed to compress  $\mathbf{z}$  under the prior. As in most other works on neural image compression (Balle
et al., 2018; Minnen et al., 2018; Yang et al., 2023), we use a box-shaped stochastic encoder  $e(\mathbf{z}|\mathbf{x}_0)$  that simulates rounding by noise injection at training time.
In analogy to Eq. 2, we simplify the training objective by using the denoising score matching loss,
$$
L _ {\text {u p p e r}} (\mathbf {x} _ {0} | \mathbf {z}) \approx \mathbb {E} _ {\mathbf {x} _ {0}, n, \epsilon} \| \epsilon - \epsilon_ {\theta} (\mathbf {x} _ {n}, \mathbf {z}, \frac {n}{N _ {\text {t r a i n}}}) \| ^ {2} = \mathbb {E} _ {\mathbf {x} _ {0}, n, \epsilon} \frac {\alpha_ {n}}{1 - \alpha_ {n}} \| \mathbf {x} _ {0} - \mathcal {X} _ {\theta} (\mathbf {x} _ {n}, \mathbf {z}, \frac {n}{N _ {\text {t r a i n}}}) \| ^ {2} \tag {6}
$$
The noise level  $n$  and  $\alpha_{n}$  are defined in Eq. 2. Instead of conditioning on  $n$ , we condition the model on the pseudo-continuous variable  $\frac{n}{N_{\mathrm{train}}}$  which offers additional flexibility in choosing the number of denoising steps for decoding (e.g., we can use a  $N_{\mathrm{test}}$  smaller than  $N_{\mathrm{train}}$ ). The right-hand-side equation describes an alternative form of the loss function, where  $\mathcal{X}_{\theta}$  directly learns to reconstruct  $\mathbf{x}_0$  instead of  $\epsilon$  (Salimans & Ho, 2022). We can easily derive the equivalence with
$$
\epsilon_ {\theta} \big (\mathbf {x} _ {n}, \mathbf {z}, \frac {n}{N} \big) = \frac {\mathbf {x} _ {n} - \sqrt {\alpha_ {n}} \chi_ {\theta} \big (\mathbf {x} _ {n} , \mathbf {z} , \frac {n}{N} \big)}{\sqrt {1 - \alpha_ {n}}}.
$$
Decoding process Once the model is trained, we entropy-decode  $\mathbf{z}$  using the prior  $p(\mathbf{z})$  and conditionally decode the image  $\mathbf{x}_0$  using ancestral sampling. We consider two decoding schemes: a stochastic one with  $\mathbf{x}_N\sim \mathcal{N}(0,\gamma^2 I)$  (where  $\gamma >0$ ) and a deterministic version with  $\mathbf{x}_N = \mathbf{0}$  (or  $\gamma = 0$ ), both following the DDIM sampling method:
$$
\mathbf {x} _ {n - 1} = \sqrt {\alpha_ {n - 1}} \mathcal {X} _ {\theta} \left(\mathbf {x} _ {n}, \mathbf {z}, \frac {n}{N}\right) + \sqrt {1 - \alpha_ {n - 1}} \epsilon_ {\theta} \left(\mathbf {x} _ {n}, \mathbf {z}, \frac {n}{N}\right) \tag {7}
$$
Since the variables  $\mathbf{x}_{1:N}$  are not stored but generated at test time, these "texture" variables can result in variable reconstructions upon stochastic decoding (see Figure 5 for decoding with different  $\gamma$ ). Algorithm 1 summarizes training and encoding/decoding.
Fast decoding using  $\mathcal{X}$ -prediction In most applications of diffusion models, the iterative generative process is a major roadblock towards fast generation. Although various methods have been proposed to reduce the number of iterations, they often require additional post-processing methods, such as progress distillation (Salimans & Ho, 2022) and parallel denoising (Zheng et al., 2022).
Surprisingly, in our use case of diffusion models, we found that the  $\mathcal{X}$ -prediction model with only a handful of decoding steps achieves comparable compression performance to the  $\epsilon$ -model with hundreds of steps, without the need of any post-processing. This can be explained by closely inspecting  $\mathcal{X}$ -prediction objective from Eq. 6 that almost looks like an autoencoder loss, with the modification that  $n$  and  $\mathbf{x}_n$  are given as additional inputs. When  $n$  is large,  $\mathbf{x}_n$  looks like pure noise and doesn't contain much information about  $\mathbf{x}_0$ ; in this case,  $\mathcal{X}$  will ignore this input and reconstruct the data based on the content latent variable  $\mathbf{z}$ . In contrast, if  $n$  is small,  $\mathbf{x}_n$  will closely resemble  $\mathbf{x}_0$  and hence carry more information than  $\mathbf{z}$  to reconstruct the image. This is to say that our diffusion objective inherits the properties of an autoencoder to reconstruct the data in a single iteration; however, successive decoding allows the model to refine this estimate and arrive at a reconstruction closer to the data manifold, with beneficial effects for perceptual properties.
Optional Perceptual Loss While Eq. 6 already describes a viable loss function for our conditional diffusion compression model, we can influence the perceptual quality of the compressed images by introducing additional loss functions similar to (Mentzer et al., 2020).
First, we note that the decoded data point can be understood as a function of the low-level latent  $\mathbf{x}_n$ , the latent code  $\mathbf{z}$ , and the iteration  $n$ , such that  $\bar{\mathbf{x}}_0 = \mathcal{X}_{\theta}(\mathbf{x}_n,\mathbf{z},n / N)$  or  $\frac{\mathbf{x}_n - \sqrt{1 - \alpha_n}\epsilon_\theta(\mathbf{x}_n,\mathbf{z},n / N)}{\sqrt{\alpha_n}}$ . When minimizing a perceptual metric  $d(\cdot ,\cdot)$ , we can therefore add a new term to the loss:
$$
L _ {\mathrm {p}} = \mathbb {E} _ {\epsilon , n, \mathbf {z} \sim e (\mathbf {z} | \mathbf {x} _ {0})} [ d (\bar {\mathbf {x}} _ {0}, \mathbf {x} _ {0}) ] \text {a n d} L _ {\mathrm {c}} = \mathbb {E} _ {\mathbf {z} \sim e (\mathbf {z} | \mathbf {x} _ {0})} [ L _ {\text {u p p e r}} (\mathbf {x} _ {0} | \mathbf {z}) - \frac {\lambda}{1 - \rho} \log p (\mathbf {z}) ] \tag {8}
$$
$$
L = \rho L _ {\mathrm {p}} + (1 - \rho) L _ {\mathrm {c}}. \tag {9}
$$
This loss term is weighted by an additional Lagrange multiplier  $\rho \in [0,1)$ , resulting in a three-way tradeoff between rate, distortion, and perceptual quality (Yang et al., 2023).
We stress that different variations of perceptual losses for compression have been proposed (Yang et al., 2023). While this paper uses the widely-adopted LPIPS loss (Zhang et al., 2018), other approaches add an adversarial term that seek to make the reconstructions indistinguishable from reconstructed images. In this setup, Blau & Michaeli (2019) have proven mathematically that it is impossible to simultaneously obtain abitrarily good perceptual qualities and low distortions. In this paper, we observe a similar fundamental tradeoff between perception and distortion.
Figure 2: Tradeoffs between bitrate (x-axes, in bpp) and different metrics (y-axes) for various models tested on DIV2K. We consider both perceptual (red frames) and distortion metrics (blue frames). Arrows in the plot titles indicate whether high  $(\uparrow)$  or low  $(\downarrow)$  values indicate a better score. CDC (proposed) in its basic version (deterministic, without finetuning to LPIPS) compares favorably in distortion metrics, while CDC with stochastic decoding and added LPIPS losses performs favorably on perceptual metrics.