Towards image compression with perfect realism at ultra-low bitrates
Authors: Marlene Careil
Matthew J. Muckley
Jakob Verbeek
Stéphane Lathuilière
Venue: ICLR 2024 poster (2024)
Keywords: image compression
diffusion models
generative image models
Primary area: generative models
TL;DR: Image compression at 0.003 bits-per-pixel with a diffusion decoder

Abstract: Image  codecs are typically optimized to trade-off bitrate vs. distortion metrics. At low bitrates, this leads to  compression artefacts which are easily perceptible, even when training with perceptual or adversarial losses. To improve image quality and remove dependency on the bitrate we propose to decode with iterative diffusion models. We condition the decoding process on a vector-quantized image representation, as well as a global image description to provide additional context. We dub our model `PerCo'' for ``perceptual compression'', and compare it to state-of-the-art codecs at rates from 0.1 down to 0.003 bits per pixel. The latter rate  is more than an order of magnitude smaller than those considered in most prior work, compressing a 512x768 Kodak image with less than 153 bytes. Despite this ultra-low bitrate, our approach maintains the ability to reconstruct realistic images. We find that our model leads to reconstructions with state-of-the-art visual quality as measured by FID and KID. As predicted by rate-distortion-perception theory, visual quality is less dependent on the bitrate than previous methods.

Published as a conference paper at ICLR 2024
TOWARDS IMAGE COMPRESSION WITH PERFECT REAL- ISM AT ULTRA-LOW BITRATES
Marlène Careil Matthew J. paackley',Jakon Verbeek', Stéphane Lathuilière
Meta Al
2LTCI, Télécom Paris, IP Paris
mmuckley, Lmn jieebeekk @meta.com com paris. fr
stephane
ABSTRACT
Image codecs are typically optimized to trade-off bitrate vs. distortion metrics. At low bitrates, this leads to compression artefacts which are easily perceptible, even when training with perceptual or adversarial losses. To improve image quality and remove dependency on the bitrate we propose decode with iterative diffusion models. We condition the decoding process on a vector-quantized image repre- sentation, as well global image description to provide additional context. We dub our model "PerCo" for "perceptual compression", and compare it to state- of-the-an codecs at rates from 0.1 down 0.003 bits per pixel. The latter rate is more than an order of magnitude smaller than those considered in most prior work, compressing a 512 x 768 Kodak image with less than 153 bytes. Despite this ultra-low bitrate,
approach maintains the ability to reconstruct realistic im- ages. We find that our model leads to reconstructions with state-of- offte-o-thhe-ar visual quality measured by FID and KID. As predicted by rate- ditt-distortion- pptrrpttooo theory, visual quality is less dependent on the bitrate than previous methods.
INTRODUCTION
Traditional image and video codecs are optimized for the rate-distortion function (Shannon, 1948), which minimizes the expected size of the data under a distortion constraint such as mean-squared error. Recent research has developed neural image compression methods that surpass handcrafted image compression codecs in terms of rate-distortion performance (Ballé et al., 2017; Ballé et al., 2018; Cheng et al., 2020; He et al., 2022a). However, optimization for the rate-distortion function comes at a cost of "realism", where realism is mathematically related to the statistical fidelity or f-divergence between the compressed image distribution and the true image distribution (Blau & Michaeli, 2018; 2019). The typical qualitative manifestation of unrealistic images is blurring.
Generative modeling compensates for artefacts such as blurring by introducing a divergence term (Agustsson al., 2019; Mentzer al., 2020; Muckley al., 2023), typically in the form of an adversarial discriminator loss, which improves human-perceptuull performance (Mentzer et al., 2020). Such codees are called "generative compression" codecs, and are evaluated terms of the rate-distortion-realis tradeoff (Blau & Michaeli, 2019). A further result of rate-distrrtion-realiim theory is the theoretical possibility of perfect realism codec, i.e., codec with no f-divergence, and zero FID across all rates, with no more than twofold increase in mean-squared error from the rate-distortion optimal codec. This result motivates research on perfect realism codecs, where orig- inal and reconstruction are different from each other, but it is not possible to tell which is which. Such codecs are particularly interesting for extremely low bitrate settings, where existing codecs introduce severe artefacts that are casily perceptible, see Fig. 1. So far, the primary effort towards building such a codec was the work of Theis et al. (2022), who demonstrated lower FID scores than HiFiC (Mentzer al., 2020) via combination of pretrained variational diffusion model and verse channel coding. However, computational constraints prohibited application to images larger than 64 64 pixels.
In this paper we make further progress towards perfect-realism codecs. Similar to Theis et al. (2022), leverage a pretrained text-to-image diffusion model; however, rather than using the diffusion

---

Published as a conference paper at ICLR 2024
Original
Text- Sketch, 0.0281 hpp
VTM, 0,0242 hpp
MS-ILLM, 0.0065 bpp
PerCo (ours), 0,0032 bp
Kodak images compressed with the diffusion-basee Text-Sketch approach PICS (Lei et al., 2023), the hand-crafted codec VTM (ytm), MS-ILLM (Muckley al., 2023) which leverages an adversarial loss, and with PerCo (ours). Taking the lowest available bitrate for each method.
Figure I:
model for compression directly, we use it as a decoder in a VQ-VAE-like autoencoder. The (ap- proximate) log- log-likelihood loss of diffusion models offers an alternative to the use of distortions such as MSE or LPIPS used to train most neural compression models that make strong assumptions on the conditonal distribution on images given the latents which are inappropriate at low bitrates, and lead to compression artefacts. For decoding, allows us to obtain a set of reconstructions that reflect the uncertainty about the original source im- age. We augment the quantized image representation with global image description that captures the high-level semantic information of the image. To this end, we use an automatically generated textual image description, e.g. using BLIP (Li et al., 2023). Alternatively. this take the form of a global image feature extracted from a pre-trained image backbone. We demonstrate that this high-level information is helpful for compression performance at extremely low rates. Our work is closely related to that of Lei et al. (2023), who explored text-conditioned diffusion decoders for image compression, and the only prior work that we are aware of that considers the same ultra-low bitrates as we do. In contrast, their work uses (automatically generated) binary contour maps spatial conditioning signal which carry little detail on colors and textures, and leads reconstruc- tions that are not faithful to the original. Experimentally we observe significantly improved FID and KID scores compared to competing methods. Moreover, we find that FID and KID are much more stable across bitrates than other methods, which aligns with our goal of image compression with perfect realism.
sample from the conditional diffusion model, which
In sum, our contributions are as follows:
We develop novel diffusion model called PerCo for image compression that conditioned on both a vector-quantized latent image representation and a textual image description.
We obtain realistic reconstructions bitrates as low as 0.003 bits per pixel, significantly improving over previous work, see Figure 1.
We obtain state-of-the-ard FID and KID performance on the MS-COCO 30k dataset; and observe no significant degradation of FID when reducing the bitrate.
2 RELATED WORK
Neural image compression codecs. Ballé et al. (2017) demonstrated that end-to-end neural image compression codec could outperform the classical JPEG codec in terms of rate-distortion per- formance. Ballé et al. (2018) enhance this approach by conditioning the latent on a "hyperprior" that encodes high-level information in the image, greatly improving performance. Follow-up works im- proved the latent conditioning mechanism (Minnen al., 2018; Minnen & Singh, 2020; Cheng et al., 2020: He et al., 2022a). Optimization for rate-distortion alone leads to distributional mismatch be- tween the compressed image distribution and the natural image distribution (Blau & Michaeli, 2018;
2

---

Published as a conference paper at ICLR 2024
2019). Agustsson et al. (2019) and Mentzer et al. (2020) added adversarial loss term and demon- strated its usefulness for human perception. Similarly He et al. (2022b) builds upon ELIC (He et al., 2022a) by adding perceptual losses including adversarial ones. uollow-up works have improved the discriminator architecture (Muckley et al., 2023), decode-time realism guidance (Agustsson et al., 2023), or investigated alternative autoencoders with discrete entropy models (EI-Nouby et al., 2023). Xu et al. (2023) introduce the notion of conditional perceptual quality, where perceptual quality is dependent on side information. Our work is related to this, as the global textual or visual represen- tation on which we condition the decoder can be seen as form of side information.
Diffusion models. Ho et al. (2020) improved the original diffusion probabilistic model (Sohl- Dickstein et al., 2015) that gave it impressive performance on image generation benchmarks. Dhari- wal & Nichol (2021) demonstrated superior performance to GANs for image generation with clas- sifier guidance, while Kingma et al. (2021) improved diffusion model likelihood estimation. Condi- tioning diffusion models on large text-encoder networks enables these models to generate realistic images from natural language prompts (Nichol et al., 2022; Saharia et al., 2022), and defining the diffusion process in an autoencoder latent space reduces the computational cost of the diffusion pro- cess. In work we build upon large-scale pretrained text-conditioned diffusion models to benefit from the image generation capabilities embedded in them.
Diffusion generative compressors. Ho et al. (2020) considered compression with diffusion models and reverse channel coding and the corresponding rate-distortion performance, but reported only experiments on 32 x 32 CIFAR-10 images and MSE distortion. Theis et al. (2022) demonstrated that the reverse channel coding approach gave FID numbers superior to HiFiC for 64x64 ImageNet images. Saharia al. (2022) introduced super-resolution diffusion model that removed artefacts from JPEG images. Ghouse et al. (2023) developed a complete version of a residual diffusion codec with images of high quality for suite of codecs. Hoogeboom et al. (2023) showed similar results using a neural autoencoder baseline codec, and using a separate diffusion model that does not require any additional bits to refine the reconstruction. Yang Mandt (2023) developed an alternative approach, where a hyperprior-bee based neural encoder was trained jointly with a diffusion decoder, also showing improved performance to HiFiC. In Pan et al. (2022), they optimize textual embedding on top of pretrained text-to-image diffusion model. They also design a compression guidance method used at each denoising step to better reconstruct images. Lei et al. (2023) leverage text-conditioned diffusion models for image compression at very low bitrates, using prompt inversion encode the image into text, and adding spatial detail via compressed binary contour maps. Our work is similar, but replace computationally costly prompt inversion (Wen al., 2023) with fast feed-forward image captioning (Li et al., 2023), and replace binary contour maps which carry little appearance information with end-to-end learned vector-quantized image features.
3
PERCEPTUAL COMPRESSION WITH A DIFFUSION DECODER
3.1 OVERALL FRAMEWORK
The foundation of most lossy compression lies in rate-distortion theory (Shannon, 1948), which quantifies the balance between the bitrate needed to transmit signal vs. the distortion of the signal. Let us assume an input signal æ, with its corresponding quantized representation and reconstruc- tion f(z). Neural compression is commonly achieved via the minimization of a training objective formulated as a linear combination of a rate and a distortion term:
LRD = Epp ( + XLD()(z),
(1)
where Pa is the data distribution and P=| is the posterior distribution of the quantized codes. The rate term (z) estimates the bitrate either by means of scalar quantization and continuous entropy model (Ballé et al., 2017), or using pector-quantization in combination with discrete entropy mod- els (El-Nouby et al., 2023). For simplicity, in PerCo, we employ quaat--quat combined with a uniform entropy model leading to a constant rate controlled through two hyper-parameters. the number of elements that are quantized, and the codebook size. Therefore, the rate LR can be consid- ered fixed when training our models and can be dropped from the optimization formulation. Next, we introduce a new formulation of the distortion term, LD. enabling the integration of a pre-trained diffusion model, thereby achieving higher level of perceptual quality. Our aim with this formu- lation is to leverage a large pre-trained diffusion model as a robust image prior, enabling realistic image reconstruction even at extremely low bitrates.
3

---

Published as a conference paper at ICLR 2024
Image Captioning
Text Encoder
Arithmetic coding
01100010101..
Codebook
LDM Encoder
Hyper Encoder
Quantization
Conditional Dilfusion Model
LDM Decodle
10101110100.
Dittusion loop (*T)
Overview of PerCo. The LDM encoder maps an RGB image into the latent space of the diffusion model. The "hyper encoder" then maps the image to hyper-latent with smaller spatial resolution, which then vector-quantizer and represented as a bitstream using uniform coding- The image captioning model generates textual description of the input image, which is losslessly compressed, and processed by a text encoder to condition the diffusion model. The diffusion model reconstructs the input image in its latent space conditioned on the output of the text encoder and the hyper encoder. Finally, the LDM decoder maps the latent reconstruction back to RGB pixel space.
Figure 2:
Diffusion models "Sohl-Dicksteii al., 2015; Ho et al., 2020) approximate data distribution by gradually denoising a unit Gaussian random variable. This achieved by training a denoising function reverse a diffusion process that takes initial sample, denoted æn, from the data distribution and adds small Gaussian noise over a large number of steps. In practice, a noise estima- tor eœ (2t 2) implemented as a neural network is trained to denoise input data points at timesteps t e [0. Ti, where I Va + /1 ayc is a corrupted version of the original data point zo with Gaussian noise c. The scalar a fixes the noise strength, varying from arp 2 for no noise to or ~ for pure Gaussian noise. Finally, is an optional condition, such as a text-prompt in text-to-image models. In our case, z the quantized representation of the input image. Once the noise estimator is trained, the model can generate samples by drawing TT No0,, 7), and applying iterative denoising for f = T........ 1. To reduce the computational cost, we use a latent diffusion model (LDM) where the variables æ, are defined in the latent space of autoencoder.
We formulate our distortion loss within the probabilistic diffusion framework. For every the diffu- sion step we compute estimation 1;-1 of x;-1 from z and æ; and minimize its error:
C Ep E. ||,,-1 c. (0 2)]2
Up a multiplicative constant, this loss can be rewritten
Dib Erg Br....
E c
|| co 2, 1)1
We obtain the final loss by taking the expectation of Eq. (3) w.r.t. the steps i, and use prediction objective of Salimans & Ho (2022), which is more stable during training. The loss can be augmented with any image-level loss such as LPIPS (Zhang et al., 2018) to improve perceptual image similarity.
3.2 ENCODING LOCAL AND GLOBAL CONTEXT
We encode the input image as z = (zl, 2g), where 2 and 2g carry complementary local and global context, respectively. The overall structure of our compression scheme is illustrated in Figure 2.
Local spatial encoding. We encode the input image into a quantized tensor proceeding three stages. First, we employ the LDM encoder B(-) to achieve a first dimension reduction. Assuming an imput image of resolution 512 x 512, we obtain a feature map of dimension 4 x 64 x 64. Second, add lightweight "hyper encoder" H composed of several convolutional layers to project the LDM features to a lower spatial resolution tensor H The spatial resolution, hxuc, of H, is chosen according the target bitrate. Third, proceed to quantization of H, to obtain 2] via vector- quantization. As in VQ-VAE (van den Oord et al., 2017; Razavi et al., 2019), each of the h x w vectors he in H, mapped an element in codebook learned using the vector quantization loss:
Lve E,, [1)sg(hs) 2z [|g((2,) h.111
(4)
4

---

Published as a conference paper at ICLR 2024
where s.()) is the stop-gradde ttop-gradient operation, and 2g is the mapping of h2 to its closest codebook entry. For better codebook usage, we follow Yu et al. (2022) and set the output of the hyper encoder to be relatively low dimensional: 32 in our case. We also found it beneficial to replace the codebook loss by exponential moving average on the codehooks and thus the first term in Eq. (4) is no longer used. The quantized encoder output is then upsampled to 64 x 64 resolution, to be channel-wise concatenated to the input of the noise estimation network of the diffusion model. The hyper-network trained end-to-end while finetuning the diffusion model. For simplicity, we opt for uniform coding to map the quantized latents to a bitstream. Using log2 V bits to encode each element in the h x w sized hyper latent encoding, where V the codebook size, results total of hw log, V bits.
Global encoding with image captioning, While z can accurately encode local information, our ex- periments show that relying on 2 alone leads to unsatisfying realism. Therefore, we add a global en- coding of the image 2g which provides additional context. For this purpose, we use an off-the-shelf state-of-the-art image captioning model, such as BLIP-2 (Li et al., 2023) or IDEFICS (Laurançon et al., 2023), which we keep fixed when training model. Note that user-generated captions can also be used, as explored in our experiments. Similar to Lei et al. (2023), we losslessly compress the caption using Lempel-Ziv coding as implemented in the zlib library (zli) to obtain zz. Alternatively, we also explore the use of global image features extracted using an image backbone network.
Decoding with diffusion model. The quantized local image features in ø are fed to the denoising U-Net of the diffusion model via concatenation with the latent features Tn of the current time step. To adapt to this change of dimension, we extend the kernel of the first convolutional layer of the U-Net by adding number of channels that corresponds to the channel dimension of ž which is randomly initialized. The global encoding 2g is losslessly decoded and passed to the diffusion model through cross-attention layers of the pre-trained diffusion model.
Furthermore, we found it beneficial to use classifier-free guidance (Ho & Salimans, 2021), which we apply at inference time for the text conditioning zg. i.e. we contrast conditioning on local featues 2j alone vs. conditioning on both ,n and zg. This leads to the noise estimate:
Eg = Ex (Z), 0), f) - A (#0( (Z), zg), f) e (z), t)),
(5)
where Ay is the guidance scale. We empirically found X, = 3 to work well. To enable classifier-free guidance, we train PerCo by dropping the text-conditioning in 10% of the training iterations. When dropping the text-conditioning we use a constant learned text-embedding instead.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Implementation details. We base our model off text-conditioned latent diffusion model, with latent space provided by a convolutional autoencoder and diffusion model with an architecture sim- ilar to GLIDE (Nichol et al., 2022). The model is pre-trained on proprietary dataset consisting of around 300M image-caption pairs. We also experiment with an image-conditioned model. We use BLIP-2 (Li et al., 2023) obtain image captions for conditioning the decoder, and limit the maxi- mum caption length to 32 tokens. The hyper encoder consists of nine residual convolutional blocks, and includes a number of downsampling layers depending on the spatial map resolution selected for the target bitrate. In total, the hyper encoder contains between 4M and 8M parameters. During all our experiments, the autoencoder weights are frozen. We train the hyper encoder and finetune the diffusion model on OpenImages (Kuznetsova et al., 2020), similar to Lei et al. (2023) and Muckley et al. (2023). We use random 512 512 crops, and instead of finetuning the full U-Net we found beneficial to only finetune the linear layers present of the diffusion model, representing around 15% of all weights.
Datasets. For evaluation, COCO we evaluate at resolution 256 x 256 by selecting the same images from the 2014 validation al. (2023). We evaluate at
the Kodak dataset (Franzen, 1999) as well MS-COCO 30k. On
set (Lin et al., 2014) as Hoogeboom et al. (2023) and Agustsson resolution 512x512 on the 2017 training set (Caesar et al., 2018), which is the same resolution used for evaluation by Lei et al. (2023), and use captions and label maps for some metrics.
Metrics. To quantify image quality we use FID (Heusel et al., 2017) and KID (Binkowski et al., 2018), which match feature distributions between sets of original images and their reconstructions.
5

---

Published as a conference paper at ICLR 2024
PerCp VS ILLM
PerCo-irage
VTH
HFC
Shutch
DRAC
KS PaRCHGAN
Kodal
MS-COCO 30k
MS-COCO 30k
MS-COCO 30k
30
50
0.7
3.55
0.00
2.0
20-
0.2
0.2
15:0
2011
0.1
1011
2012
10%0
12t4
10"05
20"
per
0.7
50
0.00
0.5
0.4
2.44
0.04
0.3
20
0.01
0.2
Sts per PMC
B1G pct
511 pcl plod
sct
Figure 3: Evaluation of PerCo and other image compression codecs on Kodak and MS-COCO 30k.
We measure distortion using MS-SSIM (Wang et al., 2003) and LPIPS (Zhang et al., 2018). In App. B. we also report PSNR, but find these low-level distortion metrics to be less meaningful for low rates, see Fig. 14. Therefore we include other more semantic-level metrics. We compute the CLIP score to measure global alignment of reconstructed samples with ground truth captions (Hes- sel al., 2021). Unless otherwise specified, we condition our model on captions generated by BLIP2, which are different from the ground-truth captions used to compute the CLIP score. Second, we adopt the mean Intersection over Union (mIoU) metric measure the local semantic preserva- tion, e.g. Sushko et al. (2022). In this case, pass reconstructed images through pre- pre-trained semantic segmentation model, and compare the predicted semantic maps to the ground truth ones. We take the pretrained ViT-Adapter segmentation network (Chen al., 2023), which trained on the COCO train split. For Kodak we only report LPIPS and MS-SSIM, as the dataset is too small (24 images) reliably compute FID and KID, and does come with semantic annotations.
Baselines. We include representative codecs from several families. VTM is state-of-the-arr hand- crafted codec used in the VVC video codec (vtm). Among neural compressors using adversarial losses, we include MS-ILLM (Muckley et al., 2023) which improves over HiFiC (Mentzer et al., 2020) using an adversarial loss based multi-class classification. For HiFiC we include their model with the lowest available bitrate, and add MS-PatchGAN, which is a close reproduction of HiFiC by Muckley et al. (2023). We compare to multi-realism approach of Agustsson et (2023), at their lowest bitrate with the best realism. Text-Sketch (Lei et al., 2023) is the only prior work we are aware of that evaluates image compression at bitrates below 0.01, based on text-conditioned diffusion models. We include two other recent diffusion-based approaches: DIRAC (Ghouse et al., 2023) and HFD/DDPM (Hoogeboom et al., 2023).
4.2 MAIN RESULTS
Comparison to state-of-the-art methods. In Fig. 3 we compare our results to state-of-the-art codecs. We observe that PerCo (ours) yields significantly lower (better) FID and KID compared to other approaches at lower bitrates ( (<0.04 bpp), and that our FID and KID curves are much flatter, indicating a decoupling of realism and bitrate that is not seen for other methods. For the semantics-related CLIP and mloU metrics we also obtain consistent improvements over all other methods, in particular at low rates. For LPIPS, PerCo is better than Text-Sketch and VTM, but somewhat worse than other approaches for rates >0.01 bpp. Similarly, for MS-SSIM improve methods. However, as we illustrate Figs. 10 and 14 in App. B.2, such similarity metrics are not necessarily meaningful at low bitrates, as they do not capture realism.
Gure
HEDIDDeM
Mult-Reelsmm
0.22 0.5
016
Figure 4: Evaluation on COCO at 256 256 res.
Text-Sketch, but are worse than other

---

Published as a conference paper at ICLR 2024
Original
MS- ILLMM 0.0065 hpp
Text Sketch, PIC. 0.0025 hpp
PerCo (ours), 0,0032 hpp
Original
MS-ILLM. M. 0.013 bpp
Text-Skerch. PICS, 0.028 bpp
PorCo (ours) 0.01 bpp
Figure 5: Comparing PerCo on images from the Kodak dataset to MS-ILLM (Muckley et al., 2023) which leverages an adversarial loss, and the diffusion-based Text-sketch approach (Lei et al., 2023) conditioned on a text only (top, PIC, 0.0025 bpp) and text sketch (bottom, PICS, 0.028 bpp).
dpq 00O'O ddq 110'0
Figure 6: Reconstruction sampled a Kodak image at 0.003 and 0.011 bpp. Best viewed zoomed.
In Fig. 4 we compare to HFD/DDPM (using their lowest bitrates: 0. 107 and 0.175 bpp) on FID on MS-COCO 30k at 256 256 resolution, as used in their paper. We also include MS-ILLM and the multi-realism approach of Agustsson et al. (2023). We find that PerCo yields best FID for both rates.
Qualitative comparisons. We provide qualitative comparisons of PerCo baselines in Fig. and Fig. 5. PerCo yields reconstructions of higher quality with less artefacts, more faithful colors, and realistic details. VTM and MS-ILLM reconstructions are blurry and/or blocky, while Text-Sketch strongly deviates from the originals in color. When using text-only conditioning for Text-Sketch (PIC, 0.0025 bpp), it is not able to achieve spatial alignment, by lack of local image encoding.
4.3 ABLATIONS
Diversity in reconstructions. With our model, reconstructions are sampled from a conditional diffusion model. Ideally, the conditional distribution concentrates all mass perfectly on the original input image. practice, this does not happen due limited model capacity, training data, and bitrate. In particular, expect that lower bitrates wwhree there is less information about the original sample- the conditional distribution would have more variance, and we expect to observe more diversity among the samples. The reconstruction two bitrates in Fig. 6 confirm this.
Conditioning modalities. Fig. 7, we evaluate PerCo using each conditioning modality separately (textual spatial), to analyze the contribution of each. We compare BLIP-2 (Li et al., 2023) to generate captions (default) to using IDEFICS (Laurançon et al., 2023) which produces more detailed descriptions, and ground-truth (GT) captions. When using descriptive captions, consider

---

Published as a conference paper at ICLR 2024
coptione Eneading
captions onl Ehcading-
only Spatial Encoding
ony Encading
25
30-
45
23-
25
0.6
20
20.5
277
50.4
230
25-
25
0.2
15
10
per slae
perpted
per skael
per steel
Figure 7: Ablation of conditioning on local features and/or different captions, See text for detail.
Ablations of classifier-free guidance forms (left), and impact of conditional diffusion model and quantization on reconstruction abilities (right). All models at 020112 bpp. except for LDM autoencoder and PerCo w/o quantization which use single precision for non-quantized latents.
Table 1:
Guidance type
LFID
LLPIPS 1SSIM fmloU CCLIP
Method
LFID LLPIPS 1SSIM fmloU CCLP
None
7.89
0.40
0.58
30.84
28.36
LDM autoencoder 0.67
0.07
0.92
51.04
30.19
Text and spatial
13.84
0.41
0.58
32.26
28.87
PerCo w/o quant.
0.68
0.07
0.92
51.03
30.18
PerCo: Text only
4.42
0.39
0.58
46.64
29.05
PerCo (0.012 bpp)
4.42
0.39
0.58
46.64
29.05
conditioning on a learned textual embedding that is constant across images. We also tried using an empty text fixed generic text -"A high quality photograph." "-prraphhh--a for all images, and observed similar or slightly worse performance as the learned text embedding setting.
The model using the constant text embedding (PerCo w/ only Spatial Encoding) performs signif- icantly worse than the other models in terms of FID and CLIP score. For the LPIPS and mloU metries, which rely on the alignment of the original and reconstructed images, the results are com- parable to other models. This can be attributed to the fact that accurate alignment is primarily achieved through local feature conditioning, which remains unchanged in this model. Consistently, we also notice that when using textual condition only (blue, orange, and red square), the perfor- mance in terms of LPIPS and mloU is worse than for the models that use spatial conditioning, while the CLIP scores are in line with the models using both textual and spatial conditioning. Both types of conditioning contribute to improving FID, and the text-only conditioned models perform in between the models with only spatial conditioning and those with both types of conditioning.
Between models using captions from BLIP (blue curves) and IDEFICS (orange curves) in combi- nation with spatial conditioning, we observe similar FID scores and somewhat better CLIP scores for the more detailed IDEFICS captions. In terms of LPIPS and mloU we find that the models with BLIP captions perform better, which is largely due bitrate required to encode the longer captions: leaving out the textual encoding cost, both models perform very similarly. Overall, using ground- truth captions (GT, red curves) yields similar results using BLIP, while BLIP and IDEFICS yield slightly better mIoU and CLIP scores. As COCO contains five GT captions per image, we ensure that when conditioning on GT caption, a different one is used to compute the CLIP score.
Classifier-free guidance. In Tab. 1 (left), we compare three options regarding classifier-free guid- ance (CFG) in our model (i) not using CFG, (ii) applying CFG on the text and spatial feature maps, i.e. contrasting full conditioning to no conditioning (we use an constant learned text embedding, and fixed all-zeros feature maps for local conditioning), and (iii) applying CFG to text-condiioonin only, i.e. always conditioning on the spatial feature map. We find that best results are obtained when CFG applied to textual conditioning only, which yields significantly better FID and mloU metrics.
Impact of autoencoder and quantization. To assess to which extent performance is bounded by the LDM autoencoder or the use of limited bitrate, we compare PerCo to the LDM encoder/decoder reconstruction and to PerCo trained without quantization in Tab. I (right). We observe that the LDM autoencoder significantly improves all metrics, and that PerCo trained without quantization obtains roughly the same performance as the autoencoder. This shows that PerCo is a strong conditional model, and that its performance is mainly bounded by the quantization module which throttles the bitrate of the image encoding, but could also lead to suboptimal training through noisy gradients.
8

---

Published as a conference paper at ICLR 2024
0.0032 hpp
0.0071 hpp
0.0111 hpp
0.0323
0.0526 hpp
0.0950 hpp
0.1263 hgp
3.305
0.5
0.902
3.223
3.200
0.2
6.5
3285-
0.4
3.882
staps
stops
difiatinn St650
Figure 8: Evolution of FID. LPIPS, MS-SSIM and CLIP scores depending on the number of denois- ing steps for different bitrates. In all cases the model has been trained using 50 denoising steps.
Number of timesteps as a tradeoff between perception and distortion. In Figure 8, we show how FID, LPIPS, MS-SSIM and CLIP scores vary when we change the number of denoising steps. We do the evaluation for seven different bitrates, ranging from 0.0032 to 0.1263 hpp. First, we notice that for higher bitrates, the metrics are quite constant across the number of timesteps, which means that the model has less uncertainty about the image to reconstruct and obtains optimal performance after only few denoising steps. Secondly, bitrates. Indeed, when the number timesteps is small, FID is worse, but MS-SSIM is optimal, while this phenomenon reverses when increasing the number of denoising steps.
observe tradcoff between distortion and realism for lower
Conditioning on global image features. Besides the text-conditioned model described above and used in most experiments (PerCo), we also experimented with model conditioned on image embed- dings extracted from an image backhone network (PerCo-Image). While we use lossless Lempel-Ziv coding to compress the captions for the text-conditioned model, we use product quantization (Jégou et al., 2011) for lossy compression of the image embeddings. We divide the embeddings into M subvectors and quantize each subvector using a separate codebook with V elements. We empiri- cally found that we achieve similar performance compared to training with non-quantized image embeddings with M 16 and V 1024. For 512 x 512 images, the bitrate obtained for the image embeddings is 0.00061 bpp, which is 3.5x lower than the bitrate of 0.00219 hpp for BLIP image captions. From the results Fig. 3. we observe that the model with global image embeddings obtains competitive results, but has slightly worse pairwise image reconstruction metrics at high bitrates in terms of LPIPS and MS-SSIM on Kodak and MS-COCO 30k.
CONCLUSION
We proposed PerCo, an image compression model that combines a VQ- VAE-like encoder with a diffusion-based decoder, and includes second conditioning stream based textual image de- scriptions. The iterative diffusion decoder allows for more realistic reconstructions, in particular at very low bitrates, as compared to feed-forward decoders used in previous work, even when trained with perceptual and adversarial losses. With our work we make step towards perfect realism codecs: observe realistic reconstructions bitrates as low as 0.003 bits per pixel, a bitrate regime that to our knowledge is explored in only one prior work (Lei et al., 2023). We find that semantics-driven metrics such as CLIP score and mloU are improved overall and in particular for low rates, and FID and KID are dramatically improved at low bitrates and much more stable across different bitrates.
Limitations. In this study we focused on medium sized images up to 768 x 512 resolution, similar to prior work on diffusion-based methods (Theis et al., 2022; Lei et al., 2023) that evaluated on 64x64 and 512x512 images, respectively. Extension to higher resolutions can possibly be achieved using a patch-based approach, see e.g. Hoogeboom et al. (2023). PerCo exhibits somewhat poorer reconstruction performance in terms MSS-SSIM. PSNR and LPIPS (the latter for rates >0.01), than existing approaches. This seems at least in part to be due to limitations of the LDM autoencoder, see App. B.2. is also probably explained by the tradeoff existing between distortion and percep- tion (Blau& & Michaeli, 2019). We leave detailed study of these two points future work.

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
The use of machine learning models, in general and for image compression in particular, can lead to biases in the model related the training data, and potentially work better data similar those used to train the models. Therefore the models as described in this paper should be used for research purposes only, and not deployed in practice without extensive analysis of their potential biases.
ACKNOWLEDGEMENTS
The photo in Figure is by Emily Allen, licensed CC BY 2.0, and taken from https: // www. flickr. com/ hwm/photos// 9945 99453901 @NO0 /220269972223 We would like to thank Oron ppottss/9455300
Ashual, Uriel Singer, Adam Polyak and Shelly Sheynin for sharing the lext-to-image model.
REFERENCES
VTM reference software for VVC. URL https: fraunhffer /e/ jvet/ VWCSoftware_ VTM 2,
zlib - compression compatible with gzip. URL hol.htsp://docc. http:///.....pthhooor///////////
z zlib. 5
Eirikur Agustsson, Michael Tschannen, Fabian Mentzer, Radu Timofte, and Luc Van Gool. Gener- ative adversarial networks for extreme learned image compression. In ICCV, 2019. 1,3
Eirikur Agustsson, David Minnen, George Toderici, and Fabian Mentzer. Multi-realism image com-
pression with conditional generator. CVPR, 2023. 3, 5, 7,
Johannes Ballé, Valero Laparra, and Ecro P Simoncelli. End-to-end optimized image compression. In ICLR, 2017. 1, 2.
Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. Variational image compression with a scale hyperprior. In ICLR, 2018. 1,2
Mikolaj Binkowski, Danica Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD
GANs. In ICLR, 2018. 5
Yochai Blau and Tomer Michaeli. The perception-distore tradeoff. In CVPR, 2018. 1,2
Yochai Blau and Tomer Michaeli. Rethinking lossy compression: The rate-distortion-percepiio tradeoff. In ICML, 2019. 3, 9
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. COCO-Stuff: Thing and stuff classes in context. In CVPR, 2018. 5
Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. In ICLR, 2023, 6
Zhengxue Cheng, Heming Sun, Masaru Takeuchi, and Jiro Katto. Learned image compression with discretized Gaussian mixture likelihoods and attention modules. In CVPR, 2020. 1,2
Prafulla Dhariwal and Alex Nichol. Diffusion models beat GANs on image synthesis. In NeurIPS, 2021. 3
Alaaeldin El-Nouby, Matthew Muckley, Karen Ullrich, Ivan Laptev, Jakob Verbeek, and Hervé Jégou. Image compression with product quantized masked image modeling. Trans. Mach. Learn. Res., 2023. 3
Rich Franzen. Kodak PhotoCD dataset, 1999. URL http:://t/rt
http://ww....ggappppssso/ooooo 5
Noor Fathima Ghouse, Jens Petersen, Auke Wiggers, Tianlin Xu, and Guillaume Sautière. A residual diffusion model for high perceptual quality codec augmentation. arXiv preprint, 2301.05 3301.05489, 2023. 3, 6, 13
10

---

Published as a conference paper at ICLR 2024
Dailan He, Ziming Yang, Weikun Peng. Rui Ma, Hongwei Qin, and Yan Wang. ELIC: Efficient learned image compression with unevenly grouped space- ppacnelhannel contextual adaptive coding. In CVPR, June 2022a. 1, 3,
Dailan He, Ziming Yang. Hongjuu Yu, Tongda Xu, Jixiang Luo, Yuan Chen, Chenjian Gao, Xinjie Shi, Hongwei Qin, and Yan Wang. PO-ELIC: Perception-oriented efficient learned image coding. In CVPR, 2022b.
Jack Hessel, Ani Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: A reference-free evaluation metric for image captioning. In EMNLP. 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by two time-scale update rule converge a local Nash equilibrium. NeurIPS, 2017. 5
J. Ho and T. Salimans. Classifier-free diffusion guidance. In NeurIPS Workshop Deep Generative Models and Downstream Applications, 2021. 5
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS,
volume 33, 2020.
Emiel Hoogeboom, Eirikur Agustsson, Fabian Mentzer, Luca Versari, George Toderici, and Lueas Theis. High-fidelity image compression with score-based generative models. arXiv preprint, 2305.18231, 2023. 5, 6, 9, 13
Herve Jégou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search. IEEE TPAMI, 33(1):117-128, 2011.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. In NeurlPS, 2021.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Fusee, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, al. The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale. IJV,, 128(7):1956-199 1981, 2020.
Hugo Laurançon, Daniel van Strien, Stas Bekman, Leo Tronchon, Lucile Saulnier, Thomas Wang, Siddharth Karamcheti, Amanpreet Singh, Giada Pistilli, Yacine Jernite, and Victor Sanh. Intro- ducing IDEFICS; An open reproduction of state-of-the-art visual language model, 2023. URL wt/bco///////..occc.cccc. hto/blogg/deffcs..... http//ifffffss...mm 5.7
Eric Lei, Yigit Berkay Uslu, Hamed Hassani, and Shirin Saeedi Bidokht. Text sketch: Image compression at ultra low rates. In ICML Neural Compression Workshop, 2023. 2, 3, 6, 7, 13, 17
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. In ICML, 2023. 2, 3, 5,
T.-Y. Lin, M. Maire, S. Belongie, L. Bourdev, R. Girshick, J. Hays, P. Perona, D. Ramanan, P. Dollár, and Zitnick. Microsoft COCO: common objects context. In ECCV, 2014.
Fabian Mentzer, George Toderici, Michael Tschannen, and Eirikur Agustsson. High-fidelity gen- erative image compression. In NeurlPS, 2020. 1, 6, 13
David Minnen and Saurabh Singh. Channel-wise autoregressive entropy models for learned image compression. In ICIP, 2020.
David Minnen, Johannes Ballé, and George D Toderici. Joint autoregressive and hierarchical priors for learned image compression. In NeurIPS, 2018. 2
Matthew J. Muckley, Alaacldin El-Nouby, Karen Ulirich, Hervé Jégou, and Jakob Verbeek. Im- proving statistical fidelity for neural image compression with implicit local likelihood models. In ICML., 2023. 1, 3, 5, 13
11

---

Published as a conference paper at ICLR 2024
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen, GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. In ICML, 2022. 3.5
Zhihong Pan, Xin Zhou, and Hao Tian. Extreme generative image compression by learning text embedding from diffusion models. arXiv preprint, 2211.07793, 2022.
Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with VAE-2... NewrlPS, 2019.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE TPAMI, 45(4)):713-4726, 2022.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In ICLR, 2022,
C. Shannon. A mathematical theory communication. Bell Syst. Tech. J., 27(3):379-423, 1948.
1,3
Jascha Sohl-Dickstee Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, 2015. 3, 4
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021. 13
Vadim Sushko, Edgar Schönfeld, Dan Zhang. Juergen Gall, Bernt Schiele, and Anna Khoreva. OA- SIS: Only adversarial supervision for semantic image synthesis. IJCV, 130(12):2903-25)) 29233, 2022.
Lucas Theis, Tim Salimans, Matthew D Hoffman, and Fabian Mentzer. Lossy compression with
Gaussian diffusion. arXiv preprint, 2206.08889, 2022. 1, 9
Aaron den Oord, Oriol Vinyals, and koray Kavukcuoglu. Neural discrete representation learn- ing. In NewrIPS, 2017. 4
Z. Wang, E.P. Simoncelli, and A.C. Bovik. Multiscale structural similarity for image quality assess- ment. ACSSC, volume 2. pp. 1398-1402, 2003.
Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery. arXiv preprint, 2302.03660 2023.
Tongda Xu, Qian Zhang, Yanghao Li, Dailan He, Zhe Wang, Yuanyuan Wang, Hongwei Qin, Yan Wang, Jingjing Liu, and Ya-Qin Zhang. Conditional perceptual quality preserving image com- pression. arxiv preprint, 2308 08444, 2023. 3
Ruihan Yang and Stephan Mandt. Lossy image compression with conditional diffusion models. In ICML Neural Compression Workshop, 2023. 3
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, and Yonghui Wu Jason Baldridge. Vector-quantized image modeling with improved VQGAN. In ICLR, 2022. 5
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018. 4. 6
12

---

Published as a conference paper at ICLR 2024
A EXPERIMENTAL DETAILS
Evaluation of third-party models. Where possible we reported evaluation metrics reported in the original papers. For Text-Sketch Lei et al. (2023), we used the publicly released checkpoints, and used them for evaluation with the provided code. Note that Text-Sketch resized all images at resolution 512 for evaluation on the DIV2K and CLIC 2021 datasets, which makes the compar- ison with other baselines not immediate. We prefer to evaluate on original images instead, and for that purpose used the Kodak dataset which is commonly used to evaluate image compression models, and contains images of 512 x 768 pixels. We also evaluate on the MS-COCO 2014 valida- tion set, which is less commonly used, but which offers ground truth semantic segmentation maps and descriptive captions which enable evaluation with the semantics oriented mIoU and CLIP score metrics. For MS-ILLM and MS-PatchGAN (Muckley et al., 2023), checkpoints for models trained additional low bitrates were obtained from the authors through personal communication. For HFD/DDPM (Hoogeboom et al., 2023), DIRAC (Ghouse et al., 2023) and Multi-Realise (Agusts- son et al., 2023), we obtained the numbers shown their paper through personal communication, as there are no available github. To evaluate VTM, we followed same the procedure as described by Muckley et al. (2023) in appendix B of their paper. For HifiC (Mentzer et al., 2020), we took the checkpoints available from their github.
Additional implementation details. We take pretrained latent diffusion model doing text-to- image generation that contains 1.4B parameters. We fix the weights of the LDM autoencoder and finetune 15% of the diffusion model weights. We train our hyper-encoder for 5 epochs with AdamW optimizer at a peak learning rate of le-4 and a weight decay of 0.01, and a batch size of 160. We apply linear warmup for the first 10k training iterations. For bitrates higher that 0.05, we found it beneficial to add a LPIPS loss in image space to reconstruct more faithfully images. As this loss requires to backpropagate gradients through the LDM decoder (even though the LDM decoder is frozen), it is more memory intensive and we had to decrease the batch size to 40. For the vector quantization module, we used the vector-quantize-pe pouunniie-pptorch library.
To finetune our latent diffusion model we used a grid of 50 timesteps. At inference time, we use 20 denoising steps obtain good performance for the lowest bitrates. For bitrates of 0.05 and higher, we use five denoising steps as this is enough to obtain optimal performance, see Fig. 8 from the main paper. Besides, we sample images with DDIM deterministic sampling schedule, i.e. by fixing o, = 0 for all t E j0, T), where T is the number of denoising steps, see Song et al. (2021).
When testing different textual conditioning modalities (either BLIP or IDEFICS) in our ablation study in Fig. 7, the conditioning used at inference time is also used when training the models. When using human-specifen captions, we use PerCo trained on IL.IP captions, as there are no ground-truth captions for the OpenImagesv6 dataset we used to train our model.
We use the clean-fid library to compute FID and KID. 3, For CLIP score, we compute image and text embeddings with the CLIP backbone ViT-B/32. LPIPS is calculated using the original github", and MS-SSIM from the FAIR Neural Compression library.
Details on the bitrates used to evaluate our models. To achieve a specific target bitrate, we take into account the average bitrates required to encode the textual conditioning, and then set the spatial grid size and vocabulary size of the local conditioning accordingly. More specifically, for COCO at resolution 512x 512, 2, the average textual bpp for BLIP is 0.00219, while it is 0.00622 for IDEFICS, and 0.00271 for ground truth captions. For Kodak at resolution 768x512, it is 0.00141 with II.IP and 0.00415 bpp with IDEFICS. In the experiments, we select the spatial size and codebook dimension of the quantization module to adjust the spatial encoding bitrate depending on the targeted total bitrate. We found that codebooks with less than 64 elements were detrimental to the training, so we only used larger codebooks. Table 3 lists the hpp for the spatial coding for the configurations we used to proeuce the results in our experiments. For the ablation study in Fig. 7 we used other
"httpss//giittbb.cc//afaeeee// Neura 1omprrest
13

---

Published as a conference paper at ICLR 2024
combinations of spatial grid size and vocabulary size for the tested variants to approximately match the bitrates of our default PerCo model.
Inference Speed. We run benchmarking A100 gpus using 20 denoising steps, and 5 denoising steps for bitrates higher than 0.05 bits per pixel. We compute the runtime for all the Kodak images resolution 512x768 and provide the average time seconds with its standard deviation. The encoding complexity of PerCo is quite similar to adversarial- based compression methods such as MS-ILLM and HifiC (which has the same architecture as MS-ILLM). Encoding with Text-Sketch (PIC/PICS) is orders of magnitude slower because it is optimizing text embedding for each image. The decoding complexity of PerCo using 20 denoising steps (hitrates below 0.05 hpp) comparable to that of Text-Sketch (PIC) and about 8 times faster than Text-Sketch (PICS), because PICS is conditioning on sketches which takes time to decompress. When using only five denoising steps, the decoding complexity drops rouhgly by factor four, showing that the decoding is dominated by evaluations of the denoising network. Compared to HiFiC and MS-ILLM, the PerCo decoder using 5 denoising steps is about eight times slower, and when using 20 steps this becomes factor 32. Please note that the running time of VTM is computed on CPU, so it may explain the longer runtime. About the diffusion-based models DIRAC and HFD/DDPM, we couldn't 't evaluate their models as there is no available codebase, but we found some evaluations decoding time. They report decoding time which is about four times slower than what they report for HifiC, which also shows that diffusion- based methods are slower than seiversarial-basee ones.
DIRAC paper about
Table 2: Encoding and Decoding Speed (in seconds).
Model VTM
Encoding Speed (in sec.)
Decoding Speed (in sec.)
16.892 = 7.574
0.135 + 0.002
MS-ILLM
0.084 = 0,021
0.080 + 0,007
Text-Sketch (PIC)
163.070 + 0.380
2.725 + 0.012
Text-Sketch (PICS)
190.231 = 2.476
19.288 + 0.251
PerCo 5 denoising steps
0.080 = 0.018
0.665 + 0.009
PerCo 20 denoising steps
0.080 = 0.018
2.551 + 0.018
The average decoding times for the 24 images of Kodak dataset are 0.67 secs. and 2.54 secs. when decoding with 5 and 20 steps, respectively. Timings performed when decoding one image at time.
Table 3: Summary of the different combinations of spatial grid size and codebook size used for spatial encoding of 512 x 512 images.
Spatial size
Codebook size
Spatial hpp
256
0.1250
64x64
64
0.0937
8196
0.0507
32x
256
0.0313
16 16x16 16
1024
0.0098
1024
0.0024
8
256
0.0019
B ADDITIONAL EXPERIMENTAL RESULTS
B. QUANTITATIVE RESULTS
In Figure 9, complete Figure 3 from the main paper with the PSNR metric computed Kodak and COCO. We observe that PerCo yields consistently better (higher) PSNR than Text-Sil -ketthh but also consistently worse (lower) than MS-II.LM. To understand the reason behind this relatively low
14

---

Published as a conference paper at ICLR 2024
Ours MS-
Text-Sketch
DIRAC HIFCC
LDM AutoEncoder Reconstruction MS-PatchGAN
Mutti-Reallsm
VTM
Kodak
MS-COCO 30k
27.5
25.0
25
22.5
PSW
20.0
PSNR
20
17.5
15.0
15
12.5
10
10.0
10
10
10-2
10-1
bpps
Bits per pixel
Figure 9: PSNR on Kodak and MS-COCO30k
PSNR, we also include the PSNR of our LDM autoencoder (black horizontal line), which does not involve quantization. Interestingly, we observe that the LDM autoencoder achieves a PSNR perfor- mance comparable to MS-ILLM at a bitrate of 0.08 and VTM at bitrate of 0.04. The PSNR of the LDM autoencoder presumably explains the lower PSNR performance we observe with PerCo across different bitrates. To gain further insights into this behavior, visually inspect the reconstruction for a Kodak image in Figure 10. From the example, the LDM autoencoder seems to preserve more relevant detail and texture than the MS-ILLM reconstruction, despite the lower PSNR.
In Figure 11, we provide an ablation on the addition of the LPIPS loss, which shows that it leads to small improvements for both LPIPS and FID higher bitrates.
In Figure 16, we provide evaluation on an additional global image similarity metric that consists in computing CLIP scores with cosine similarity between CLIP image embeddings of original and reconstructed images, instead of cosine similarity between text captions and reconstructed images. This is motivated by the fact that noticed that CLIP text score was saturating at high bitrates, as it reaches the upper bound of 30.4 when computed on real images instead of reconstructed ones, while CLIP image score threshold is when evaluated on real images which higher than what obtained at high bitrates for all the baselines.
B.2 QUALITATIVE RESULTS
Ablations on text. In this ablation study, we conduct a detailed analysis comprehend the effects of employing text as global embedding. In Figure 12, we present qualitative examples across two bitrates, where we manipulate the conditioning text. At lower bitrates for the spatial conditioning (first and second rows), we observe that textual conditioning has significant influence on the recon- structed image. can even alter the semantics of the scene, such as transforming lighthouse into castle (second row, third column). Furthermore, as image quality substantially deteriorates when text is not used (first row, second column). Con- versely, when the bitrate of the local encoding is higher (last row), notice that text has minimal influence on the reconstructed image.
note the positive impact of text conditioning,
Reconstruction at different bitrates. In Figure 13, we show reconstructions of two MS-COCO 30k images using PerCo trained for different targeted spatial rates, ranging from 0.0046 0. 0.1279 bpps. The image quality is quite stable across bitrates, while the similarity with the original images increases with the bitrate.
Qualitative analysis of LDM autoencoder reconstruction When computing the averaged simi- larity scores Kodak dataset for the LDM autoencoder, we obtain 0.0775 LPIPS, 0.9131 MS-SSIM and 25.6003 PSNR which is lower than MS-ILLM at 0.1535 bpp for all the scores and lower than MS-ILLM 0.0806 hpp PSNR. In Figure 10 we show Kodak image illustrating that. Even
15

---

Published as a conference paper at ICLR 2024
ZENTIM
Original
NENTI
MS-ILLAN 0.0806 hpp. 24.44 PSNR, 0.8547 MS SSIM. 0.1095 LPIPS
ZENTI
MS-ILLM, 0,1335 bpp. 25.97 PSNR. 0.0555 MS-SSIM. 0,071 LIISS
RENTI
LDM autoenooder, 23.17 PSNR, 0.8674 MS-SSIM, 0.0880 LPIPS
Figure 10: Qualitative comparison of reconstructions using MS-ILLM and our LDM autoencoder comparable PSNR of a Kodak image (left) along with two zooms (middle, and right).
16

---

Published as a conference paper at ICLR 2024
Ours mn Uhis Ours win IPIPS
0.5
0.4
4-
LPIPS
0.3
0.2
2
0.1
10-2
10
10
10
Bits per pixel
Bits per pixel
Figure 11: Ablation of adding the LPIPS loss to train the diffusion decoder.
if the LDM autoencoder produces highly detailed images, we observe that reconstruction fidelity somewhat worse than MS-ILLM higher bitrates. Indeed, when looking at image details such as the waves, notice that MS-ILLM exhibits some blurring at 0.0806 bpp while LDM produces sharp details for a similar PSNR. When comparing with MS-ILLM the higher bitrate of O. 1535 bpp, notice that the letters on the boat less faithful for LDM Autoencoder than MS-ILLM with respect to the original image.
Qualitative analysis of pixelwise similarity metries We observed that pixelwise distortion met- rics such as PSNR, MS-SSIM and LPIPS are not very meaningful at very low bitrates, as already pointed out by Lei et al. (2023). worse in PSNR, 0.1766 points worse in MS-SSIM and 0.085 points worse in LPIPS than MS-ILLM 0.0065 bpp. However, at these very low bit-rates PerCo yields a much sharper and realistic recon- struction than MS-ILI.M, and preserves most of the semantics of the scene. These examples provide some indication that these metries do not align well with human preference for image quality at very low bitrates.
the example in Figure 14, PerCo at 0.0031 bpp is 3.33 points
Additional comparisons to baselines In Figure 15, we compare PerCo at O. 1264 bpp. the highest bitrate trained on, VTM, HiFiC and MS-ILLM similar bitrates. At this bitrate, PerCo faithfully reconstructs the original images with similarly detailed textures as HiFiC and MS-ILLM, and fewer artifacts than VTM.
17

---

Published as a conference paper at ICLR 2024
Original
lighthouse
Spatial byy: 0,0019,
Textual bpp:
Sputial 0.0019, Textual bpy: 0.0016
"A lighthouse
A house and red salety ring picket Tighthouse" Spatial hpp: hpp: 0.0020
"An old casilee Spotial bpe: 0.0019, bpp:
Spatial hpp: Textual hpp: 0.0010
No test
"A lighthouse with white picker tence" (BL
An mld.dddccii casii"
Spatial hpp: 0.1250, Textual hpp: O.
Spiaiaal hyp: 0512 Textual hpp: 0.0016
Spotial hpp: 0.1250, Textual hpp: 0.0010
Figure 12: Reconstructions of Kodak image when varying the global textual conditioning with spatial hpp of 0.0019 (first two rows) and O. 150 hpp (last row). Samples with the same spatial bpp are reconstructed from the same initial Gaussian noise from the same model.
18

---

Published as a conference paper at ICLR 2024
Original
0.0048 hpp
0.0087 bpp
0.0127 bpp
0.0536 bpp
0.1279 bpp
Original
0.0046 bpp
0085 bpp
STOP
STOP
STOP
0.0125 hpp
0.0534 hpp
0.77 bpp
P
STOP
STOP
STOP
Figure 13: Reconstructions of two MS-COCO 30k images by PerCo at different bitrates. Textual conditionings are "Bedroom scene with a bookcase, blue comforter and window" and "A stop sign mounted down on it's post", respectively.
19

---

Published as a conference paper at ICLR 2024
Onianall
MS ILMM 0.0665 bpp. 20.54 PSNR, 0.5672 MS 0.5372 LPIPS
PerCo 0.0070 hpp. 18.49 PSNR. 0.4700 MS SSIM. 0.5239 LPIPS
PerCo 0.0031 bpp. 17.21 PSNR, 0.390 MS-SSIM, 0.6222
Figure 14: Qualitative comparison of reconstructions using MS-ILLM and PerCo at low bitrates of a Kodak image (left) along with two zooms (middle, and right). MS-ILLM yields a very blurry recontruction at this operating point with PerCo yields a more realistic reconstruction, yet PerCo obtains a substantially lower PSNR, MS-SSIM and LPIPS.
20

---

Published as a conference paper at ICLR 2024
Original
VTM (0.1552 bpp)
HifiC (0.18 bpp)
MS-ILLM (0. 15347 bpp)
Ours (0.1 1265 bpp)
Original
VTM (0.1552 bpp
HifiC (0.18 hpp)
MS-IIILM (0. 15347 bpp
Ours (0. 1264 bpp)
Figure 15: Reconstructions of Kodak images at the highest bitrate range we evaluated models on.
PerCo MS-ILLM
PerCo-Image Text-Sketch
VTM DIRAC
HiFIC
MS-PatchGAN
1.00
0.95
CLIP image
0.90
0.85
0.80
0.75
0.70
10 2
10
Bits per pixel
Figure 16: Evaluation in terms of the CLIP image metric.
21


Reviews:
[{"review_id": "T4l1DZBVzl", "reviewer": "Reviewer_6W37", "summary": "This paper proposes a new image compression method, which decodes compressed latent features with a iterative diffusion model, instead of a single-pass feed-forward decoder. Specifically, the introduced diffusion compressor conditions on two kinds of features:  (1) a global textual image description obtained from a image captioning model; (2) a vector-quantized image representation in latent space.  According to the experimental results, the proposed method beats the state-of-the-art methods in visual quality at ultra-low bitrate.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper introduces a new image compression architecture. It proposes to combine a global textual image description and a vector-quantized latent image representation as the conditions of the latent diffusion model to decode the images.\n- The reconstruction results are much better than the exisiting works at ultra-low bitrates.", "weaknesses": "- The idea of applying diffusion model on image compression is not new (as reviewed in the paper). But at least for me, combining it with textual information is interesting and motivating for image compression task.\n- The proposed method only optimizes the distortion/perceptual term but drops the rate term. It is not justified why the rate term can be dropped.\nWhat impact will have if include the rate term into optimization? \n- One limitation of diffusion model is the inference speed. This paper do not provide the running time of the proposed method, especifically the decoding time. A fair comparsion of running time (both encoding and decoding ) with the competing methods (both learned and traditional) should be provided and discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DVkMiGC1iN", "reviewer": "Reviewer_FDrq", "summary": "This paper proposes an image generative compression framework called PerCo. This model applies the diffusion model to the field of image compression and achieves visually good performance at extremely low bit rate compression. Compared to previous methods, this work leads to better visual quality as measured by subjective metrics like FID and KID.", "questions": "A minor question: on page 4, in “Local spatial encoding” paragraph line 6, “Third, we proceed to quantization of $H_s$ to obtain $z_g$ via vector quantization”.  From my understanding, $z_g$ should be $z_l$ ? (ps, a typo here: “quatization” to “quantization” )", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed method combines text and image description for the diffusion model to improve compression performance. Compared to previous works, it achieves higher quality and more realistic image representation at very low bit rates. The model proposed is valuable for future exploration of applying diffusion models to the field of image compression.\nMoreover, the semantic logic and argumentation of this article are clear. The extensive comparative analysis also makes the article more convincing.", "weaknesses": "1）The training and inference complexity of such diffusion-based methods is usually large. The authors should provide some preliminary results like running time or memory consumption.\n\n2）Another existing diffusion-based image compression framework [1] combining text and image information can be discussed.\n\n[1] Pan, Zhihong, Xin Zhou, and Hao Tian. \"Extreme Generative Image Compression by Learning Text Embedding from Diffusion Models.\" arXiv preprint arXiv:2211.07793 (2022).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4mHh0evD8o", "reviewer": "Reviewer_p6Yn", "summary": "## Summary\n* The authors present a technique to compress images into 0.003 bbp by text-to-image latent diffusion models. They show that in very low bitrate, pretrained caption becomes a very efficient latent representation for image compression.", "questions": "## Questions\n* In MS-ILLM, the VQ-VAE is forzen once after pre-trained. While in this paper the VQ-VAE participate the end-to-end training. Then a natural question to ask is whether include the VQ-VAE of MS-ILLM in training improves the result?\n* A recent preprint [Conditional Perceptual Quality Preserving Image Compression] also justifies the adoptation of side information in image compression. The authors can probably connect their work to rate-distortion-perception trade-off with the approach similar the in preprint.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "## Strength\n* There has been several diffusion based image compression works. A major problem of diffusion based compression is that what special advantage can diffusion brings with the cost of decoding time. This work find a specific scenario (ultra low bitrate) that is hard to be achieved by GAN models, which justifies the adoptation of diffusion over GAN, and I like it a lot.\n* The visual results look strikingly good. No zoom-up is needed to see the obvious advantage over other methods.\n* It is also interesting to see that text is a efficient representation for image compression.", "weaknesses": "## Weakness\n* I am not really sure what is the point of evaluating LPIPS in Fig.3. Obviously MS-ILLM and HiFiC are trained with LPIPS, and this brings unfair advantage over the proposed approach. And as the authors are optimizing their approach towards divergence based perceptual quality [Blau 2018], using FID and KID should be enough. Reporting MS-SSIM and mIoU is also weird. \n* The authors think achieveing a low MS-SSIM, PSNR and LPIPS is a weakness, which I can not agree. By RDP theory [Blau 2019], it is absolutely normal for perceptual codec being outperformed on image-wise distortion metrics.\n* The authors are enouraged to present results in their original aspect ratio. Currently some results in Fig. 1 looks weird as they are squeezed horizontally.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7jNjDn2h0o", "reviewer": "Reviewer_Z1Pi", "summary": "This paper proposes to use diffusion models as the decoder.  \nThe diffusion decoder is conditioned on both a vector-quantized latent image representation and a textual image description (losslessly coded separately).  \nThe proposed methods achieve realistic reconstructions at bitrates as low as 0.003 bits per pixel, significantly outperforming previous works.", "questions": "Overall I think this is a good work. Please refer to the weakness part for the discussion phase", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The technical contribution is novel to me. I like the idea of conditioning the generative codec on a vector-quantized image representation along with a global textual image description to provide additional context.  \n2. The paper is well written and easy to follow. Enough background information is provided to general readers. It is a great work bridging learnt image compression and AIGC.  \n3. The results are significantly better than previous works.   \n4. Ablation studies are thorough and convincing.", "weaknesses": "1. Though the perceptual quality is SOTA, the decoding complexity is not compared. It is better to show the decoding complexity since diffusion models tend to be slower. Comparing the decoding complexity (i.e. latency and flops) with previous perceptual image compresion methods can provide more helpful information to the community.   \n2. Missing citations. Following Hoogeboom et al 2023, previous SOTA PO-ELIC should be compared or at least discussed if it is hard to make direct comparison: He, Dailan, et al. \"PO-ELIC: Perception-Oriented Efficient Learned Image Coding.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022. \n3. Other Suggestions. The idea of conditioning the generative codec on image representation along with an extra descriptor (which is losslessly coded) is very interesting. I noticed that the following work provides a theoretical perspective on this conditional coding setting and also explains that the perceptual quality measured by FID is not affected by bitrate, please consider discussing this to provide more information to readers.\nXu, Tongda, et al. \"Conditional Perceptual Quality Preserving Image Compression.\" arXiv preprint arXiv:2308.08154 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)