<figure><img src="image_1.png" /></figure>

HAL
open science

# Linearly transformed color guide for low-bitrate diffusion based image compression

Tom Bordin, Thomas Maugey

<figure><img src="image_2.png" /></figure>

## To cite this version:

Tom Bordin, Thomas Maugey. Linearly transformed color guide for low-bitrate diffusion based image compression. IEEE Transactions on Image Processing, In press, pp.15. hal-04882103

HAL Id: hal-04882103

https://inria.hal.science/hal-04882103v1

Submitted on 13 Jan 2025

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Linearly transformed color guide for low-bitrate
diffusion based image compression

Tom Bordin
INRIA
Rennes, France

Thomas Maugey
INRIA
Rennes France

***Abstract***—This study addresses the challenge of controlling the global color aspect of images generated by a diffusion model without training or fine-tuning. We rewrite the guidance equations to ensure that the outputs are closer to a known color map, without compromising the quality of the generation. Our method results in new guidance equations. In the context of color guidance, we show that the scaling of the guidance should not decrease but rather increase throughout the diffusion process. In a second contribution, our guidance is applied in a compression framework, where we combine both semantic and general color information of the image to decode at very low cost. We show that our method is effective in improving the fidelity and realism of compressed images at extremely low bit rates (10⁻²bpp), performing better on these criteria when compared to other classical or more semantically oriented approaches. The implementation of our method is available on gitlab¹.

# I. INTRODUCTION

Image generation has made significant progress, mainly due to the emergence of diffusion models [1]. These generative methods rely heavily on computation and data to produce the highest quality images. Therefore, when using the latest networks for other applications, it may be interesting to avoid retraining or fine-tuning them, but rather to adapt the models. The currently popular and widely used text or image conditioning is often not sufficient for all applications that require finer control. Getting control of the outputs without retraining becomes a challenge. One of the difficulties in controlling image generation lies in the fact that most recent models are trained to work in the latent space of a fixed VAE, as proposed in [2], these models are called Latent Diffusion Models (LDMs). By reducing the dimension considered, the model can be shrunk and the generation process accelerated. However, VAE coding in LDMs loses the structure of the signal during denoising, so assumptions we can make about an image cannot easily be applied to the decoded latent. This is particularly the case for frequency decomposition, where the organization of the signal in latent space is unknown. Thus, most methods in the literature that add control to diffusion models have been designed for pixel space. The solutions for controlling outputs in pixel space are not always relevant for LDMs, and unfortunately most current state-of-the-art models are trained in this configuration.

<figure><img src="image_4.png" /><figcaption>Fig. 1: Two images generated with a Diffusion Model, conditioned by the same semantics (*rocky mountain* and *cloudy sky*). Contrary to the left image, the right image is obtained by guiding the diffusion model towards a general color aspect (top right).</figcaption></figure>

In this paper, we focus on a specific type of control on LDMs, we aim to generate an image that is coherent with a given general color aspect which is given by a color map, we illustrate the concept in Fig. 1, adding control to an already semantic conditional generation. Control through color maps can be a useful tool for image generation, especially in the context of compression, where, a really coarse color map can be used as a basis for generating details. There are already some ways to control color for diffusion in the pixel space, such as generating only the high frequencies using a decomposition of the image proposed by the authors of [3], or using a specific scheduler for the diffusion model [4]. However, to the best of our knowledge, a good control of the color in LDM without training, has not been presented. Some works propose training methods to add control to diffusion models, by learning the directions in the latent space associated with semantic meaning [5] or by adding a condition as input [6]. However, we do not want to rely on training or fine-tuning a new model for the specific task of color map conditioning, as this would not take advantage of the latest models that could be trained in the future. We want to focus on a training-free control that is adaptable to any diffusion model, and in particular to LDMs. In SdEdit [7], the authors propose a method that provides some control over the color of the output using cartoon-like color maps, but this control comes at the cost of a trade-off between

¹https://gitlab.inria.fr/tbordin/color-guidance
the quality of the generation and the fidelity to the color condition. Exploiting classifier guidance [8] which requires training, in [9], the authors propose a universal guidance, with a training-free method to control any diffusion model for any condition. In the specific case of color guidance, we noticed that among the existing methods, only a part of the information used for control is preserved when guiding the generation.

We propose a new formulation of the guidance specific to color map control. We rewrite the equations, inspired by universal-guidance [9], and show that, for color maps, contrary to the general case, the scaling of the guidance term should not decrease during diffusion but rather increase. We call our method *fine color guidance*, and we show that we exploit most of the information provided by the color map, without compromising the quality of the generated images.

The tradeoff between distortion and perception, highlighted by Blau.et al [10], shows that optimizing an MSE criterion comes at the expense of the quality in terms of the realism of the image. Moreover, in [11] they further detail that this tradeoff is even more present as the rate decreases. Thus, at extremely low bitrates (< 10⁻² bpp), optimizing the MSE criterion leads to an important loss of image quality which can also lead to a loss of semantics. In terms of perception, the pixel error becomes less relevant, or almost meaningless, as the bitrate decreases. But by discarding the distortion as a criterion, the compression must rely on other aspects of the image. Compression approaches that move away from the MSE as a reconstruction criterion, instead focus more on semantic descriptors [12], [13]. One way of representing an image is through a high-level color description that gives only a global aspect of what it contains. Controlling the color map during image generation becomes interesting in this case. Inspired by the CoClicO framework, which is based on a CLIP and color semantic proposed in [14], we propose to apply our fine-color guidance to improve on image compression. We compare the compression framework with PICS, another recent generative approach [15], a classical encoder VVC [16] and methods in between such as MS-ILLM [17] or HiFiC [18] at similar bitrates.

After a brief reminder of how diffusion models work, we present the formulation of our problem in section II. We then compare existing color control solutions in section III. We present our proposed method for fine-guidance in section IV and its application to image compression in an extremely low bitrate framework in section V. Finally, in section VI, we compare our fine color guidance with other existing methods and their impact on compression.

## II. CONTROLLING COLORS IN DIFFUSION MODELS

Our goal is to add control, without requiring training, to any image diffusion model by using color information. Since the latest models work in the latent space of a VAE, control should also be available in this context. First, let us recall how diffusion models work.

### A. Diffusion models

Diffusion models are denoisers that start from a random Gaussian noise and iteratively converge to a *clean* signal. With the popularity of diffusion models, the need for fast generation and fast training has encouraged the emergence of latent diffusion models [2], which diffuse in the latent space of a fixed VAE. In the following, to be more concise, we consider pixel space diffusion to be a special case of latent diffusion where the VAE is the identity, see Fig. 2.

<figure><img src="image_5.png" /><figcaption>Fig. 2: Illustration of the latent diffusion models. Diffusion in the domain of pixels becomes a particular case where there is no encoder or decoders $E_{ldm} = D_{ldm} = I$</figcaption></figure>

1) **Noise model:** Diffusion models learn to invert an iterative noise process on the signal. The model denoises on a series of timesteps [0, T], with a scaling noise controlled by the parameters {$\alpha_t$}$_{t=0}^T$. For a given signal $z_0$, at a given timestep $t$, the added noise follows the schedule:

$$z_t = \sqrt{\alpha_t z_0 + \sqrt{1 - \alpha_t \epsilon}} \quad (1)$$
where $\epsilon \sim \mathcal{N}(0, I)$

The mean of $z_t$ moves progressively towards 0 while the amount of noise increases with $t$. Going from the value of the signal $z_0$ at $t = 0$ to a normal distribution at $t = T$. The diffusion model reverses this noise by estimating the value of $\epsilon$ at each step.

2) **Standard diffusion models:** The diffusion model learns to predict the noise $\epsilon$ added to $z_t$ in (1) from the observation of the pair $(z_t, t)$. The training loss for the timestep $t$ can be written as:

$$\mathcal{L}_t(\theta) = \mathbb{E}_{q(x_0); \epsilon \sim \mathcal{N}(0, I)} \left[ \|\epsilon - \epsilon_\theta(x_t(x_0, \epsilon), t)\|^2 \right] \quad (2)$$

When generating an image with a diffusion model, the noising process is reversed. Starting from a random gaussian noise as $z_T$, we iteratively estimate the next sample $z_{t-1}$ until reaching $t = 0$. Different methods – called schedulers – can be used to estimate $z_{t-1}$, for instance in denoising diffusion implicit model (DDIM) [19]:

$$z_{t-1} = \sqrt{\alpha_{t-1}} \underbrace{\left( \frac{z_t - \sqrt{1 - \alpha_t \epsilon_\theta}(z_t, t)}{\sqrt{\alpha_t}} \right)}_{\text{"predicted } z_0 : \hat{z}_0(z_t, t)} + \underbrace{\sqrt{1 - \alpha_{t-1} \epsilon_\theta}(z_t, t)}_{\text{"direction pointing to } z_t} \quad (3)$$
We are able to estimate a predicted signal $\hat{z}_0(z_t, t)$ at each step of the generation process, and refine the prediction as the process is repeated. When the timestep reaches 0, we have a clean signal $z_0$, which can be decoded by the VAE to obtain a generated image $x_0$.

## B. Formulation

Using an already trained and fixed diffusion model $\epsilon_\theta$, that can generate images $x_0(\epsilon_\theta)$. We want to use a color map $c$, to control the outputs of the model, so that the generated image has the global appearance of the color map. We also want to ensure that this control is not achieved at the expense of the generative power of the model, thereby compromising the realism or quality of the images. The control, should be achieved without retraining, keeping the parameters $\theta$ of the model intact, but modifying the noise estimation, noted as $\tilde{\epsilon}_\theta(c) = \tilde{\epsilon}_\theta(\epsilon_\theta, c)$ to relax the notations. The problem can be written as:

$$ \begin{gathered} \min_{\tilde{\epsilon}_{\theta}} d(c, x_0(\tilde{\epsilon}_{\theta}(c))) \text{ s.t.} \quad (4) \\ \Psi(x_0(\epsilon_{\theta})) = \Psi(x_0(\tilde{\epsilon}_{\theta}(c))) \end{gathered} $$

where $d$ measures the consistency between a color map and an image, and $\Psi$ measures realism. In other words, the control function should minimize the error of color map while maintaining the generative power. Since the control is done during the diffusion process, the estimation of the correction has to be assessed by using noisy observation of the latent image $z_t$, this additional level of abstraction makes it difficult to have an explicit function.

## III. EXISTING SOLUTIONS

Several solutions already address the previously formulated problem, but none of them satisfies all the conditions: no training, working in LDMs and maximizing the accuracy towards the color guide. We present the pros and cons of several methods and motivate the need for a specific method for color guidance, especially in the case of latent diffusion models. We summarize the comparison of the different methods and present the corresponding key equations in Tab. I. We denote by control accuracy the ability of the method to generate images close to the guide. This is illustrated further in Tab. II.

### A. Conditional diffusion models

A conditional diffusion model follows the same principle as a standard diffusion model, the main difference being that the prediction of the noise is done over $(z_t, t, c)$. The condition, $c$ is then given as input of the diffusion model at each timestep, with the parameters $\theta$ trained to take $c$ into account. Conditional models have become widely popular for text-to-image generation or image-to-image generation applications, however, they require either retraining a whole network or fine-tuning an already trained network for the desired condition. This is something we want to avoid in our framework.

<figure><img src="image_6.png" /><figcaption>Fig. 3: Illustration of different methods to control the diffusion process. The region of possible sample is illustrated in blue. The black arrow shows a potential diffusion schedule. The first figure shows a standard diffusion process with no control. Our goal is to focus the output region as close as possible to $z_0|c$.</figcaption></figure>

### B. Enforcing colors in diffusion model

Given a diffusion model $\epsilon_\theta$, it is sometimes possible to enforce a condition on the output of the model, ensuring that the output *exactly* follows our condition. For color map conditioning, this is the case, but only if the diffusion process is done in the pixel space, denoising over images rather than their latents. Then, for a given image, $z_0$ can be uniquely separated into lower and higher frequencies $z_0 = l(z_0)+h(z_0)$ for specified functions $l$ and $h$. For example, the authors in [3] use a wavelet decomposition. It is then possible to enforce the value of the low (or high frequencies) of the generated image by replacing at each step the generated low frequencies by our target $c$. With $\hat{z}_0(z_t, t)$ the predicted $z_0$ in (3) is replaced by its value at each step by:

$$ \hat{z}_0(z_t, t, c) = c + h(\hat{z}_0(z_t, t)) \quad (5) $$

At each step, the low frequencies of the sample are enforced, and only the higher frequencies are generated. Enforced diffusion is illustrated in Fig. 3b, where the vertical movement in the arrow indicates the correction of the low frequencies. This method is similar to the inpainting process described in [20], where a mask is used to do the partitioning of $z_0$ rather than a frequency decomposition. This method can be used when the constraint on $c$ needs to be strong and when the diffusion is done in pixel space. However, with the popularity of diffusion models, the speed of their training and sampling time has become a major concern, and most models are now processed in the latent space of the VAE to reduce the dimension. This method does not extend to latents where the decomposition of the signal is not defined.

### C. Initialized DM

Another method to control the generation of the diffusion model is initialization. First introduced in [7], the diffusion
process can be initialized with information on the signal instead of using a random noise. With this method, we influence the diffusion process by giving partial information on the signal we desire to get. In practice, it is done by starting the generation at a later timestep $0 < \tau < T$ with information on the color degraded at the corresponding scaling of noise:

$$z_{\tau} = \sqrt{\alpha_{\tau}} E_{ldm}(c) + \sqrt{1 - \alpha_{\tau}} \epsilon \quad (6)$$

Depending on the amount of information we want to preserve, we can adjust the start timestep to add more or less noise. This method is quite simple, requires no training, and even shortens the generation time. However, the generation process is not controlled toward the output and can still drift away from initialization, as shown in the diagram in Fig. 3c. Also, the authors in [7] show the existence of a trade-off between the fidelity to the c condition and the quality of the generated signal. This tradeoff depends on the starting timestep $\tau$. Indeed, the higher the timestep, the more noise we add and the further away from c we get.

### D. Classifier guidance

Instead of initializing as in III-C, or replacing as in III-B, authors in [8] propose to guide the diffusion at each iteration towards an image that fits a class condition. Moreover, in [9] authors propose a method they call universal to guide the output towards a condition without any training.

First introduced as classifier guidance, to condition on a set of classes, guidance can be extended to any type of condition. Conditional models (III-A) learn to guide through classifier-free guidance [21]. On the other hand, classifier guidance uses a trained classifier to do so. A guided diffusion model process is illustrated in Fig. 3d, the guidance correction is applied after each timestep, iteratively getting closer to the condition.

The guidance term comes from a parallel between diffusion models and energy based models. Indeed, if $f_\theta$ is an energy based model trained to predict the log likelihood $\log p(x)$, the corresponding training loss is formulated as:

$$\mathcal{L}_{\sigma}(\theta) = \mathbb{E}_{x \sim q(x) \epsilon \mathcal{N}(0, I)} \left[ \left\| \frac{\epsilon}{\sigma} + \nabla_x f_\theta(x + \sigma\epsilon) \right\|^2 \right] \quad (7)$$

Putting in parallel, the training loss of $\epsilon_\theta$ (2) and $f_\theta$ (7) we get:

$$\epsilon_{\theta}(z_{t}, t) = -\sqrt{1 - \alpha_{t}} \nabla_{z_{t}} \log p_{t}(z_{t}) \quad (8)$$

And using the Bayes formula we can then get the classifier guidance formula [22] to control the generation:

$$\tilde{c}_{\theta}(z_t, t, c) = \epsilon_{\theta}(z_t, t) - \sqrt{1 - \alpha_t} \nabla_{z_t} \log p_t(c|z_t) \quad (9)$$

The impact of the guidance is often scaled by a scaling factor $s$, $s \cdot \sqrt{1 - \alpha_t \nabla_{z_t} \log p_t(c|z_t)}$. Increasing the scaling factor then focuses the outputs more towards fidelity to the condition, at the cost of the diversity of generation. To simplify and shorten notations, we will note $G(\cdot, \cdot, \cdot)$ the guidance correction operator:

$$G(z_t, t, c) = -\sqrt{1 - \alpha_t} \nabla_{z_t} \log p_t(c|z_t) \quad (10)$$

The only requirement to guide a model becomes to estimate the value of $\nabla_{z_t} \log p_t(c|z_t)$, the gradient of the log likelihood of the condition c given a noisy observation $z_t$. Originally, guidance was first applied to image classification conditions, using the gradients of trained image classifiers that already estimated the probability density. Guidance becomes more difficult to apply when we do not have access to a trained model giving the probability density. The guidance term then must be estimated over the generated images or generated latents during inference.

### E. Untrained guidance

The authors in [5] propose to measure the log likelihood using the predicted signal $\hat{z}_0(z_t, t)$ (3) rather than the noisy signal $z_t$. In [9] the authors propose to go further and to approximate the log likelihood $\log p_t(c|\hat{z}_0(z_t, t))$ by a distance between the estimated condition on the predicted signal $\hat{c}_t = c(\hat{z}_0(z_t, t))$, and the given condition. In the case of color guidance, with their formulation, the correction becomes:

$$G(z_t, t, c) = s\sqrt{1 - \alpha_t} \nabla_{z_t} \|c - \hat{c}_t\|^2 \quad (11)$$

where the scale $s$ is specific to a color map size and must be determined empirically. Their approximation comes from the intuition that the estimated condition should be close to the given condition if and only if the probability density is high. However, this formulation varies greatly with the type of condition we give, and getting the right value for $s$ is detrimental to guiding the model and is not always sufficient. Furthermore, this estimation of the probability density does not vary with the amount of noise present in the signal.

## IV. PROPOSED FINE COLOR GUIDANCE FOR DIFFUSION MODELS

Inspired by the formulation of universal guidance(11), we propose to study how the estimation of guidance could be formulated in the specific context of color conditioning. We first model the color map before unrolling the formula for guidance.

### A. Modeling the color map

The color map $c$ represents coarse information about the color of an image $x$. For our problem, we define the color map as the low frequencies of the image. Using the 2d-DCT, $x = VDV^T$, we extract the $m$ lowest frequencies in the DCT. In other words:

$$c = Ax \quad (12)$$

is a $m$ dimension vector, and $A$ is a threshold operator on the frequencies in the DCT basis, which can be written as:

$$A = V_{[m,n]}^{g} I_{m} V_{[m,n]}^{T} \quad (13)$$

where $V_{[m,n]}$ stands for the sub-matrix containing the $m$ first rows and $n$ first columns of $V$, and where $V_{[m,n]}^g$ is the generalized inverse of $V_{[m,n]}$. In the following, we consider that $c \in \mathbb{R}^m$ is a color map of size $m$.
<table><caption>TABLE I: Comparison of control methods in pixel space</caption><thead><tr><th>Method</th><th>Key equation</th><th>c =</th><th>Training</th><th>Control accuracy</th><th>Extends to LDM</th></tr></thead><tbody><tr><td>Trained conditioning [1]</td><td>ε<sub>θ</sub>(z<sub>t</sub>, t, c)</td><td>Anything</td><td>Yes</td><td><b>Good</b></td><td><b>Yes</b> (same equation)</td></tr><tr><td>Enforced conditioning [20]</td><td>0</td><td>Separable<sup>1</sup></td><td><b>No</b></td><td><b>Good</b></td><td>No</td></tr><tr><td>Initialized diffusion [7]</td><td>z<sub>t</sub> = √α<sub>t</sub>c√1 - α<sub>t</sub>ε</td><td>Image/Color</td><td><b>No</b></td><td>Average</td><td><b>Yes</b> (same equation)</td></tr><tr><td>Classifier guidance [8]</td><td>-√1 - α<sub>t</sub>∇z<sub>t</sub> log p<sub>t</sub>(c|z<sub>t</sub>)</td><td>Class value</td><td>Yes</td><td><b>Good</b></td><td>No</td></tr><tr><td>Universal guidance [9]</td><td>s√1 - α<sub>t</sub>∇z<sub>t</sub> ||c - ˆc<sub>t</sub>||<sup>2</sup><sub>2</sub></td><td>Anything</td><td><b>No</b></td><td>Average</td><td><b>Yes</b> (same equation)</td></tr><tr><td>Ours</td><td>α<sub>t</sub> / (2λ<sub>t</sub>√1 - α<sub>t</sub>) ∇z<sub>t</sub> ||c - ˆc<sub>t</sub>||<sup>2</sup><sub>2</sub> A x (i.e color)</td><td>Ax</td><td><b>No</b></td><td><b>Good</b></td><td><b>Yes</b> (Eq. (28))</td></tr></tbody></table>

## B. Formulation

The formulation, previously defined in section II in Eq.(4), can now be adapted and simplified. With the notation of guidance defined in (9) and (10), the formulation becomes:

$$ \begin{gathered} \min_G \|c - A\mathbf{x}_0(\tilde{\epsilon}_\theta(c))\|_2^2 \quad \text{s.t.} \tag{14} \\ \Psi(\mathbf{x}_0(\epsilon_\theta)) = \Psi(\mathbf{x}_0(\tilde{\epsilon}_\theta(c))) \end{gathered} $$

we measure the distance between the conditioning color map and the color map of the output with a simple L norm. We add a guidance term G to the diffusion model to modify the noise estimation $\tilde{\epsilon}_\theta$. The problem lies in writing the explicit formulation of the guidance function G optimally.

## C. Unrolling classifier guidance equations

The guidance of a diffusion model relies on the classifier guidance expression in (9). As proposed in [5], we use the predicted signal for the estimation in (3) rather than the noisy signal.

### 1) The predicted signal: $\hat{z}_0(z_t, t)$

We know from (1) that for a given sample $z_0$, the noisy samples $z_t$ at timestep $t$ follow a multivariate normal distribution:

$$ z_t \sim \mathcal{N}(\sqrt{\alpha_t}z_0, (1-\alpha_t)\mathbf{I}) \tag{15} $$

Moreover, at any timestep $t$, we can build a prediction $\hat{z}_0$ of the value of $z_0$. Using the formulation from DDIMs (3) the estimated image at timestep $t$ is:

$$ \hat{z}_0(z_t, t) = \frac{z_t - \sqrt{1 - \alpha_t} \epsilon_\theta(z_t, t)}{\sqrt{\alpha_t}} \tag{16} $$

We then measure the difference between the estimated image and the real image. Introducing $z_0$ through (1), we see that $\tilde{z}_0(z_t, t)$ is gaussian and centered on the real image $z_0$:

$$ \begin{aligned} \hat{z}_0(z_t, t) &= \frac{\sqrt{\alpha_t}z_0 - \sqrt{1 - \alpha_t}(\epsilon_\theta(z_t, t) - \epsilon)}{\sqrt{\alpha_t}} \\ &= z_0 - \frac{\sqrt{1 - \alpha_t}}{\sqrt{\alpha_t}} \Delta\epsilon_t \end{aligned} \tag{17} $$

Where $\Delta\epsilon_t$ is the noise prediction error of the diffusion model at timestep $t$. We measured the distribution of the error for a given diffusion model, and we observed that depending on the value of $t$:

$$ \Delta\epsilon_t \sim \mathcal{N}(0, \bar{\lambda}_t^2 \mathbf{I}) \tag{18} $$

with $\bar{\lambda}_t$ measured empirically over a set of images. For each timestep $t$, the standard deviation $\lambda_t$ is estimated. To do so, we compute the standard deviation of $\epsilon_\theta(z_t, t) - \epsilon$ for several sampled noise values and over images of a dataset independent of the one used for testing. The values of $\lambda_t$ are then used at each step during the inference as an estimation of the distribution of the error.

We can then conclude that the predicted sample $\hat{z}_0(z_t, t)$ follows a gaussian distribution centered on the real value $z_0$:

$$ \hat{z}_0(z_t, t) \sim \mathcal{N} \left( z_0, \bar{\lambda}_t^2 \frac{1 - \alpha_t}{\alpha_t} \mathbf{I} \right) \tag{19} $$

We now have the distribution of the predicted signal given the noisy observation. Let us now move on to the estimation of the density probability: $p_t(c, \hat{z}_0(z_t, t))$.

## D. Guidance for pixel space diffusion

In the pixel space (i.e $z = E_{ldm}(x) = x$), at the timestep $t$, we can evaluate $\hat{c}_t$ the estimated color map over the predicted image $\hat{x}_0(x_t, t)$, in other words, the predicted color map:

$$ \begin{aligned} \hat{c}_t(x_t, t) &= A\hat{x}_0(x_t, t) \\ &= c + \bar{\lambda}_t \frac{\sqrt{1 - \alpha_t}}{\sqrt{\alpha_t}} A\epsilon \end{aligned} \tag{20} $$

With our definition of color map, $AA^T = I_m$, we detail in appendix IX the impact on the formulation when guiding with another linear transform. The distribution of the estimated color map is then also a gaussian centered on the targeted color map $c$:

$$ c|x_t \sim \mathcal{N} \left( \hat{c}_t, \bar{\lambda}_t^2 \frac{1 - \alpha_t}{\alpha_t} I_m \right) \tag{21} $$

We now have an explicit formula of the distribution of $c|x_t$. We also get the value of $p_t(c|x_t)$, the probability density of a multivariate distribution, that we were looking for:

$$ p_t(c|x_t) = \frac{1}{\sqrt{(2\pi)^m}} \exp \left( -\frac{\alpha_t}{2\bar{\lambda}_t^2(1-\alpha_t)} \|c - \hat{c}_t\|_2^2 \right) \tag{22} $$

We can then use (22) in the guidance term in (10). The guidance correction term for a color map in a pixel space diffusion model then becomes:

$$ G(x_t, t, c) = \frac{\alpha_t}{2\bar{\lambda}_t^2 \sqrt{1 - \alpha_t}} \nabla_{x_t} \|c - \hat{c}_t\|_2^2 \tag{23} $$

We can see that the correction term to guide the color in the diffusion model is similar to the one proposed in universal

<footnote n="1">¹The condition can be separated from the image (masking, decomposition, etc.)</footnote>
guidance (11), since both use an L² norm. However, there are still significant differences. First, the coefficient in front of the norm scales differently with t. In fact, in (11), $\sqrt{1 - \alpha_t}$ decreases, guiding less as the diffusion progresses, while in ours the term increases along the process. Moreover, once $\bar{\lambda}_t$ is determined, there is no need to tune any parameter in the loss. Our guidance in pixel space gives fewer guarantees than an enforced approach such as [3]. However, since it takes less advantage of the image in pixel space, it can be adapted to latent diffusion models and be applied to the state-of-the-art models.

## E. Guidance for latent diffusion

While in the pixel space we had access to an estimated image, here, we only have an estimated latent vector. However, we can still approximate an image as $\hat{x}_0 = D_{ldm}(\hat{z}_0)$, the decoded predicted latent. Similarly to what we had in the pixel space, we would like to know the distribution of the predicted images depending on the timestep. To do so, since the distribution of $\hat{z}_0(z_t, t)$ at each timestep is a multivariate normal, we look at the response of the decoder $D_{ldm}$ when gaussian noise is added to the latent vector. In other words, we observe the behavior of the decoder to $z_\lambda \sim \mathcal{N}(z, \lambda I)$ for a wide range of $\lambda$ and images $z$. We found that for small increase in noise, the decoded image is not far from the original one, while still recovering gaussian noise at the output. Moreover, we model the amount of noise after the decoding as a linear function of $\lambda$, estimating the parameters $\bar{a}, \bar{b}$ such as:

$$D_{ldm}(z_\lambda) \sim \mathcal{N} \left( D_{ldm}(z) + \bar{a}\lambda, \bar{b}^2 \lambda^2 \mathbf{I}_m \right) \quad (24)$$

Adding noise in the latent space thus provokes a shift of the mean in the decoded image. The decoded image distribution becomes:

$$\hat{x}_0(x_t, t) \sim \mathcal{N} \left( x_0 + \bar{a}\bar{\lambda}_t \frac{\sqrt{1 - \alpha_t}}{\sqrt{\alpha_t}}, \bar{b}^2\bar{\lambda}_t^2 \frac{1 - \alpha_t}{\alpha_t} \mathbf{I}_m \right) \quad (25)$$

From here, the computations are similar to what we had in the pixel space, just taking into account the shift of the mean, the predicted color map becomes:

$$\begin{aligned} \hat{c}_t(x_t, t) &= A \hat{x}_0(x_t, t) \\ &= c + \bar{a}\bar{\lambda}_t \frac{\sqrt{1 - \alpha_t}}{\sqrt{\alpha_t}} A \mathbf{1}_{hw} + \bar{b}\bar{\lambda}_t \frac{\sqrt{1 - \alpha_t}}{\sqrt{\alpha_t}} A \epsilon \end{aligned} \quad (26)$$

and the distribution of the estimated color map becomes:

$$\hat{c}_t(x_t, t) \sim \mathcal{N} \left( c + \bar{a}\bar{\lambda}_t \frac{\sqrt{1 - \alpha_t}}{\sqrt{\alpha_t}} A \mathbf{1}_{hw}, \bar{b}^2\bar{\lambda}_t^2 \frac{1 - \alpha_t}{\alpha_t} \mathbf{I}_m \right) h \quad (27)$$

The guidance correction term using a color map in a LDM then is:

$$G(z_t, t, c) = \frac{\alpha_t}{2b^2\bar{\lambda}_t^2 \sqrt{1 - \alpha_t}} \nabla_{z_t} \|c - \hat{c}_t - \bar{a} \frac{\bar{\lambda}_t \sqrt{1 - \alpha_t}}{\sqrt{\alpha_t}} A \mathbf{1}_{hw} \|^2 \quad (28)$$

The guidance correction in the latent is similar to the pixel space (23). The scaling is slightly different, and we take into

<figure><img src="image_7.png" /><figcaption>Fig. 4: Scaling of the gradient in guidance. In universal-guidance, the scaling decreases to 0, we observed that theoretically, guidance for color should follow another guidance schedule.</figcaption></figure>

account the shifting of the mean produced by the decoder. For an ideal pair of encoder and decoder, we then have $(\bar{a}, \bar{b}) = (0, 1)$, which results in the pixel space guidance.

We confirmed the intuition of universal guidance, with the gradient of the L2 norm appearing also in our formulation. However, in universal guidance, the estimation of the probability does not vary with the amount of noise in the signal. Whereas, if the amount of noise is low, the probability should be higher. This can be observed in our formulation, the importance of guidance increases, the less noise there is in the signal. This is a crucial difference, as changing the scaling linearly and independently of the timestep in universal guidance is not sufficient to reach good performances. We present, in Fig.4, the comparison between the two guidance schedule. We note that this observation is specific to our formulation of the color maps, but could also be generalized to other conditions computed as a linear transformation of the image. The scaling of our method changes quadratically with the size of the color map $m$ because of the term $\|c - \hat{c}_t\|_2^2$. In our case, multiplying the guidance by a scaling factor then means interpreting the color map as of higher resolution, thus resulting in poorer image quality. Decreasing it, means interpreting it as lower resolution, thus resulting in lower color fidelity and not using all the information sent.

## V. APPLICATION TO COMPRESSION

The latest diffusion models can now generate a high variety of images with impressive quality [23]. When applied to generative compression, they can be used either at high bitrates to generate small details and avoid artifacts [24], or at lower bitrates, they can be used as decoder using semantic information for conditional generation [13], [25], [26]. Our definition of color map IV-A, particularly aligns with image compression purposes. Here, we propose to follow the pipeline of CoCliCo [14] and we represent an image as a combination of semantic and color information. However, in order to make
<figure><img src="image_8.png" /><figcaption>Fig. 5: Encoding-decoding pipeline of our framework. Our guidance term $G$ is added over the CoCliCo framework based on CLIP and color semantic. We provide a color correction during generation.</figcaption></figure>

the best use of the given color information, we propose to rely on our formulation of guidance to condition the generation.

### A. Compression framework

The image is encoded into a vector pair $\sigma = (\sigma_s, \sigma_c)$ of semantic and color information, it is decoded using a generative decoder conditioned on the semantic, and guided to correspond to the color. The goal of the compression becomes to generate images with high realism while preserving the maximum semantic information contained in $\sigma$ at a given, extremely low, bitrate.

We call $\Phi$ the semantic projection of an image. The range of the function $\Phi$ then defines a semantic space. We measure the loss of semantic as a distance $d$ in this semantic space. Using this definition, noting $\mathcal{E}, \mathcal{D}$ the encoder and decoder $\mathbf{x}_{\text{out}} = \mathcal{D}(\mathcal{E}(\mathbf{x}_{\text{in}}))$, we can then formulate the problem of image compression as:

$$ \min_{\mathcal{D}} d(\Phi(\mathbf{x}_{\text{in}}), \Phi(\mathbf{x}_{\text{out}})) \text{ s.t.} \quad (29) \\ \Psi(\mathbf{x}_{\text{out}}) < \Psi_{\text{min}} \text{ and } R(\mathcal{E}(\mathbf{x}_{\text{in}})) < R_{\text{min}} $$

where $R$ is the function of the rate and $\Psi$ measures the realism of an image. Similarly to the CoCliCo [14] framework, we want to minimize the loss of semantic while preserving high realism at a given rate. In our context, the decoder is a fixed function given by the diffusion model that can be influenced by the guide $G$. The rate is measured on the encoded image, i.e, on the color map and the semantic. Seeing the general color aspect as semantic information, the distance $d$ defined here in (29) encompasses the color metric defined in section II in (4).

### B. Representation of the semantic

The image is encoded in two parts: the semantic $\sigma_s$ and the color $\sigma_c$. We already described the choice and motivation for the color map, but we have not discussed the semantic yet. As proposed in the CoCliCo framework [14], we choose to use the foundation model CLIP [27] to encode the semantic of an image. This choice is motivated firstly by the capacity of abstraction of this model but also by its popularity. Indeed,

since CLIP is used to encode images and text in the same projection space, this model has been an advantageous choice for text representation in recent text-to-image models, resulting in a large amount of available trained models.

In Fig. 5, we illustrate the proposed encoding-decoding pipeline. On the one hand, $\sigma_s$ is extracted using CLIP as a semantic encoder $\mathcal{E}_s$, and on the other hand, the color map $\sigma_c$ is computed using frequency decomposition. Both the CLIP vector and the color map are quantified as illustrated in Fig. 5 to reduce the rate. The color map is encoded as an $m \times m$ image in YUV 4:2:0 format with a uniform color palette over $b_c$ bits per channel. The CLIP latent vector is clamped, then uniformly quantized on $b_s$ bits per dimension. The bitstream produced by the uniform quantization of the color map and of the clip vector are encoded losslessly and separately. We use an asymmetric numeral system(ANS) [28] entropy coder to do so. This coder combines advantages of both the arithmetic coding and Huffman coding. The rate of an image can then be expressed as:

$$ \begin{aligned} R(\sigma^q) &= R(\sigma_s^q) + R(\sigma_c^q) \\ &= r_s * 768 + 1.5 * r_c * m^2 \end{aligned} $$

where, $r_s$ and $r_c$ are respectively the rate for coding a symbol for the CLIP vector and pixel channel, after the entropy coding. The 1.5 coefficient comes from the YUV 4:2:0 format. This compact representation allows us to reach extremely low bitrates.

### C. Decoding the semantic with a guided diffusion model

The diffusion model used for decoding thus takes two conditions as inputs: a learned semantic conditioning on $\sigma_s$, and a guided color conditioning on $\sigma_c$. The conditional diffusion model $\epsilon_{\theta}$ predicts the noise $\epsilon$ from the triplet $(z_t, t, \sigma_s)$ as described in section III-A. Our contribution lies in the correction term using the color map guidance term (23) or (28) respectively, for image and latent diffusion, during the
<figure><img src="image_9.png" /><figcaption>Fig. 6: Color guidance applied to pixel space diffusion models.</figcaption></figure>

generative decoding of images. We then have the following noise estimation function, based on a CLIP diffusion model:

$$ \tilde{\epsilon}_{\theta}(z_t, t, \sigma_s, \sigma_c) = \epsilon_{\theta}(z_t, t, \sigma_s) + G(z_t, t, \sigma_c) \quad (30) $$

## VI. EVALUATION

In this section, we compare several color control methods described previously. We show that our guidance can be applied to pixel-space diffusion, but more importantly, it also performs well when applied to latent diffusion models, outperforming currently available methods on the latter. We then apply our formulation of guidance in the context of compression in the CoCliCo framework, and show how it compares to other extremely low bit-rate and semantically focused methods.

### A. Performance of color control

1) Guidance in the pixel space: We present the guidance for pixel space diffusion. We use an unconditional diffusion model presented in [8] and pretrained to generate 512 × 512 images. This model does not use semantic information about the image and does not always produce realistic images, but it is sufficient to assess the impact of color guidance. We show in Fig. 6 some images generated using our guidance terms presented in Eq. (23) and universal guidance (11). We can see that the generated images follow the given color maps, being closer to the original color map than when using other methods.

2) Guidance in the latent space: For latent diffusion guidance, we use another model to compare the different methods to control colors. The model we use is an LDM conditioned on CLIP latent vectors as inputs for text-to-image synthesis. We repurposed it for image-to-image synthesis using the CLIP image encoder instead of the text encoder which refers to $E_{clip}$ in Fig. 5. The weights of the model are available on diffusers [29]. We use the scheduler DPMSolver [30], for

<table><caption>TABLE II: Evaluation of color control</caption><thead><tr><th rowspan="2"></th><th rowspan="2">Color fidelity ↓<br>Color map MSE</th><th colspan="2">Image realism ↑</th></tr><tr><th>CLIPIQA [33]</th><th>DBCNN [34]</th></tr></thead><tbody><tr><th>Initialized [7]</th><td>0.062</td><td>0.49</td><td>0.48</td></tr><tr><th>Universal [9]</th><td>0.032</td><td>0.69</td><td>0.66</td></tr><tr><th>Ours</th><td>0.015</td><td>0.66</td><td>0.63</td></tr></tbody></table>

faster generation, to step from $z_t$ to $z_{t-1}$, with 50 timesteps for generation. We use a dataset composed of images from the Landscape dataset [31] and the Kodak dataset [32], the images have been cropped to fit in 768 × 768, but our method can easily be adapted to any image resolution. The estimation of $\lambda_t$ is an important part of the scaling schedule, as it depends on the performance of the diffusion model. We have found that this parameter can be estimated quite accurately over a small dataset (100 images). Increasing the precision by using more images did not provide convincing improvements in performance.

We compare our fine color guidance with universal guidance applied to color and initialized diffusion from [7]. We use a color guidance scale of 1 for our method, using only the determined parameters. In the case of universal guidance, we compute the ideal scaling $s$ in its formula (11), using the scaling of our guidance formulation at the first timestep. We have experimentally verified that this scaling factor works best for their method. For initialized diffusion, we start the generation at 55% of the total timesteps to maximize the information on the given color map without compromising the quality of the generation too much, as explained in [7]. We show the performance of the different methods in Fig. 7. The images generated by the fine control systematically follow the given color map very closely. On the other hand, the other two methods drift away from the original image, losing some information provided by the color map. The universal guidance is slightly better than the initialized diffusion. We confirm these results with a quantitative evaluation, presented in Tab. II. Using the MSE between the color maps of the original and the generated image, we notice that our method preserves most of the information. We also confirm using image quality assessment metric (IQA) that, while initialized diffusion comes at the cost of the quality of images, using universal guidance or fine guidance does not degrade the image. The quality score of the generated images is similar to natural clean images of the dataset.

As an ablation study on the guidance, we also compare the methods to a simple baseline, with constant scaling factor, guiding at the same strength, independently of the timestep. We present the results of two different scaling factors, one using the strength of the first timestep in universal guidance, see Fig. 4, and one much higher, closer to the scaling of the last steps in our method. We show that setting an arbitrary scaling schedule does not work as well as following our theoretical schedule, see Fig. 8.

Fine guidance can be applied to different color map resolu-
<figure><img src="image_10.png" /><figcaption>Fig. 7: Visual comparison between the different methods to control colors.</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>Fig. 8: Comparison of the universal and our guidance scaling factor with constant scaling factors.</figcaption></figure>

tions. We show in Fig. 9 a comparison with universal guidance, on 16 × 16 and 8 × 8 color maps. With less information, the guidance constraints are relaxed, and the difference between methods is less noticeable. We can see that the universal guidance does not use the additional information provided by a higher resolution color maps, indeed, the images generated using 8 × 8 color maps are quite similar to the ones generated using higher resolutions, this affirmation is later confirmed in the next section. We can also see, from the lower resolution generated images, that the semantic information becomes crucial when the color maps are not precise enough, such as

Eiffel Tower color map in 8 × 8. We can no longer see the semantic information from the color map only, and semantic is required to get an image close to the original.

3) *Discussion*: Initializing the diffusion is a simple method that does not require a computation of gradients, however, this is a method that loses both in realism of generation and fidelity to the color map because of the presence of a tradeoff between both. Our formulation of guidance is similar to the formulation of universal guidance, as both coincidentally requires computing similar gradients. The main difference between the two methods lies in the scaling of the loss. While in universal, the importance of the guide decreases after each step, we found that to optimally guide the scaling should not decrease in $\sqrt{1 - \alpha_t}$ but remain high instead because of the ratio $\sqrt{\alpha_t}/\bar{\lambda}_t$, the generated results confirm this observation.

## B. Fine guidance for compression

We apply fine color guidance in the compression framework previously described in Section V. The color map and the CLIP latent are quantized to reach lower bitrate. The clip latent is quantized using 1 bit per dimension, and the color map palette is quantized using 5 bits per channel, as in the CoCliCo pipeline. We use the fine color guidance to preserve most of the information sent within the color map. We compare our
<figure><img src="image_12.png" /><figcaption>Fig. 9: Color map guidance at different resolutions.</figcaption></figure>

method with state-of-the-art compression methods, especially VVC. But we also compare it to other generative approaches that aim at achieving semantic compression rather than pixel-value fidelity, such as Text+sketch [15]. We consider the generation of images in patch of 768 × 768. The experiments were run on an NVIDIA A40 GPU on the CLIC2020 dataset. We measured an average encoding time of 0.43 seconds. The decoding time is much longer, with 69.13 seconds on average for 50 timesteps. The non-training approach comes at the cost of the decoding time, the estimation of the gradient is done during inference and thus has an impact on decoding rather than training. For a large scale usage, it might be preferable to invest the time needed in training the model. But for a research application, where the number of generated images is lower, allowing to add conditioning on any diffusion model. Moreover, when comparing with universal guidance [9], the closest available method to guide on color, to improve their results, they introduced *forward* and *backward* guidance. The gradients are then estimated several times per timestep (from 2 to 10 depending on the condition), multiplying the decoding time by the same amount.

We evaluate our method on two configurations. One using

only a 16 × 16 color map resolution, targeting a bitrate similar to the lowest VVC quantization parameter. And another one, keeping more information on the colors, using a 25 × 25 color map, targeting a bitrate similar to text+sketch. For the 25 × 25 configuration, we compare our method to Text+Sketch, their choice of representation uses similar in bitrate. For VVC, we use the intra coder(v1.6) with Vvenc implementation [16], and we target similar bitrate. Images in Fig. 10 showcase the different compression framework, because the focus of VVC is an optimization of PSNR, the quality of the reconstructed image is lacking. Images reconstructed using a text and sketch semantic representation have a distinctly different aspect because of the incoherence of color, sometimes mistaking the sky for something else in the reconstruction. We show that furthermore in Fig. 11 that images generated using our framework can have a high quality of texture on top of following the color map. This offers clear advantages over classical methods like VVC.

With the other configuration in 16 × 16, showed in Fig. 12, at those rates, the classical approach cannot always preserve information of the color as seen in the first and fifth row. The reconstructed images are still faithful in terms of semantic,
<figure><img src="image_13.png" /><figcaption>Fig. 10: Compression with 25 × 25 resolution color maps (0.0041 bpp).</figcaption></figure>

rather losing fidelity to the input in terms of color and thus PSNR.

A quantitative evaluation of the compression framework is presented in Tab. III. We evaluate on the two criteria presented in our formulation in (29) comparing, on lower or equal bitrates. The first criterion, Ψ, is the preservation of the given semantic. We measure the preservation of the general color aspect using the MSE, and preservation of a more textual aspect of the semantic using CLIP sine similarity. Over the color preservation, we place right behind VVC on both configurations. Note that if the MSE is used as a color preservation metric here, the objective of VVC and ours do not align, indeed, we do not aim at pixel fidelity only. Over the semantic aspect, our method scores higher than the others. The second criterion is the realism, Φ, used for the evaluation of

the guidance, we also use IQA metrics, and while text+sketch scores higher on this criterion, this is mainly due to the use of a different model rather than a performance issue caused by our guidance. Indeed, we reach a realism score which is close to the images from the dataset. IQA metrics do not necessarily reflect perfectly the notion of realism, which is hard to grasp, however, we noted that it makes a good distinction between images with a lot of artifacts, such as VVC, and clean images. The realism of generated images can also be appreciated in Fig. 11. On both configurations, our method manages to preserve the semantic and color information while keeping low bitrates. Our fine guidance also allows us to use higher resolution color maps without wasting the bitrate invested in it. Indeed, as observed in Fig. 9, while our method shows an important difference in terms of MSE when increasing the
<figure><img src="image_14.png" /><figcaption>Fig. 11: Images decoded using 25x25 resolution color maps and quantized CLIP latent for encoding (0.0041 bpp).</figcaption></figure>

<table><caption>TABLE III: Compression at extremely low bitrates, *proposed approach, bpp measured on 768 × 768 images</caption><thead><tr><th rowspan="2"></th><th colspan="2">Φ ↓</th><th colspan="3">Ψ</th><th colspan="2">R ↓</th></tr><tr><th>NMSE</th><th>CLIP [27]</th><th>CLIPIQA [33]↑</th><th>DBCNN [34]↑</th><th>FID [35]↓</th><th>bits</th><th>bpp(10<sup>-3</sup>)</th></tr></thead><tbody><tr><td>Original images</td><td>-</td><td>0.00</td><td>0.70</td><td>0.63</td><td>0.</td><td>-</td><td>-</td></tr><tr><td>VVC (qp 63)</td><td><strong>0.011</strong></td><td>0.41</td><td>0.20</td><td>0.20</td><td>22.2</td><td>3220</td><td>5.4</td></tr><tr><td>CLIP+Initialized (16 × 16)</td><td>0.091</td><td>0.21</td><td>0.56</td><td>0.50</td><td>10.5</td><td>1114</td><td>1.9</td></tr><tr><td>CLIP+Universal (16 × 16)</td><td>0.034</td><td><strong>0.18</strong></td><td><strong>0.64</strong></td><td><strong>0.60</strong></td><td><strong>3.8</strong></td><td>1114</td><td>1.9</td></tr><tr><td>CLIP+Fine guidance* (16 × 16)</td><td><strong>0.021</strong></td><td><strong>0.18</strong></td><td><strong>0.62</strong></td><td><strong>0.58</strong></td><td><strong>4.0</strong></td><td>1114</td><td>1.9</td></tr><tr><td>VVC (qp 55)</td><td><strong>0.007</strong></td><td>0.30</td><td>0.19</td><td>0.26</td><td>18.0</td><td>6640</td><td>11.3</td></tr><tr><td>Text+Sketch [15]</td><td>0.150</td><td><strong>0.19</strong></td><td><strong>0.73</strong></td><td><strong>0.61</strong></td><td>26.7</td><td>6638</td><td>11.3</td></tr><tr><td>CLIP+Initialized (25 × 25)</td><td>0.035</td><td>0.22</td><td>0.55</td><td>0.50</td><td>11.3</td><td>2437</td><td>4.1</td></tr><tr><td>CLIP+Universal (25 × 25)</td><td>0.033</td><td>0.20</td><td>0.64</td><td>0.60</td><td><strong>3.2</strong></td><td>2437</td><td>4.1</td></tr><tr><td>CLIP+Fine guidance* (25 × 25)</td><td><strong>0.016</strong></td><td><strong>0.16</strong></td><td><strong>0.64</strong></td><td><strong>0.60</strong></td><td><strong>3.2</strong></td><td>2437</td><td>4.1</td></tr></tbody></table>

resolution of the color map, the score barely changes for other guidance methods. We show in the figure and table that the resolution of the color maps is an interesting parameter to vary. We present in the table the rate per image and bits per pixel, but our description of an image does not depend on its size.

We also compared our method to MS-ILLM [17], PICS [15], VVC [16] and HiFiC [18](lowest bitrate) on the CLIC2020 dataset. We notice from the comparison on more pixel based metrics, Fig. 14a, that our method, although slightly worse in terms of PSNR, can be seen as a continuation of MS-ILLM or HiFiC at lower bitrates. When it comes to perception, Fig. 14b, the evaluation of our methods shows on both IQA metrics and FID that the realism of the generated images is on par with HiFiC independently of the rate, improving on this point when compared to other methods. Moreover, we can see from the last part of the Fig. 14c that the CLIP semantic is preserved, also independently of the rate, while classical methods tend to lose the semantic when the bitrate drops too low. We also retain information about the color of the image that was lost in the semantic proposed by

PICS.

In PerCo [36], authors propose an image coder trained on conditional coded latents. The code for PerCo was not available, and we could not obtain a quantitative comparison. We show in figure 13 on a decoded image from their article the comparison in performances at similar and half the bitrate with our method. We show that our approach enables a rendering that is of the same visual quality and semantic fidelity towards the input, without requiring a specific retraining (as PerCo does).

The MSE is here used as a metric of color fidelity and not as an objective to optimize at all cost. The goal of guidance is to get closer to the information brought by the color map. The use of our fine color guidance in compression brings fidelity to the color map, it can be applied on the latest diffusion model, even when trained in the latent space of a VAE.

## VII. CONCLUSION

We proposed a method for color guidance in diffusion models, more specifically, for latent diffusion models that
<figure><img src="image_15.png" /><figcaption>Fig. 12: Compression with 16 × 16 resolution color maps (0.0019 bpp).</figcaption></figure>

<figure><img src="image_16.png" /><figcaption>Fig. 13: Comparison of our method with PerCo at a similar and half the bitrate.</figcaption></figure>

the use of our method in a context of compression at extremely low bitrates, showing that a low resolution color map can be a good descriptor to add to the semantic. We show, in the context of the CoCliCo coding pipeline, that the information given by the color map is better preserved using fine color guidance, avoiding the waste of bitrate.

VIII. ACKNOWLEDGEMENTS

does not require any training. This method outperforms other training-free methods and can be applied to any latent or pixel space diffusion model. We can then advantageously use it on the latest diffusion models to control the output. We showcased

This work was funded by the French National Research Agency (MADARE, Project-ANR-21-CE48-0002). A special thanks to Rémi Piau for the discussion around entropy coders.
<figure><img src="image_17.png" /><figcaption>(a) Comparison with classical compression metrics</figcaption></figure>

<figure><img src="image_18.png" /><figcaption>(b) Evaluation of the perception (Ψ)</figcaption></figure>

<figure><img src="image_19.png" /><figcaption>(c) Preservation of the semantic (Φ)</figcaption></figure>

Fig. 14: PSNR, LPIPS, FID, CLIP, and MSE between 25 × 25 color maps measured on HiFiC [18], MS-ILLM [17], VVC [16], PICS [15], and our method in function of the rate on the CLIC2020 dataset.

REFERENCES

[1] Y. Song and S. Ermon, "Generative modeling by estimating gradients of the data distribution," *Advances in neural information processing systems*, vol. 32, 2019.

[2] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, "High-Resolution Image Synthesis with Latent Diffusion Models," *Apr. 2022, arXiv:2112.10752 [cs]*. [Online]. Available: http://arxiv.org/abs/2112.10752

[3] D. You, A. Floros, and P. L. Dragotti, "Indigo: An inn-guided probabilistic diffusion algorithm for inverse problems," *arXiv preprint arXiv:2306.02949*, 2023.

[4] B. Kawar, M. Elad, S. Ermon, and J. Song, "Denoising diffusion restoration models," *Advances in Neural Information Processing Systems*, vol. 35, pp. 23593–23606, 2022.

[5] M. Kwon, J. Jeong, and Y. Uh, "Diffusion models already have a semantic latent space," *arXiv preprint arXiv:2210.10960*, 2022.

[6] L. Zhang, A. Rao, and M. Agrawala, "Adding conditional control to text-to-image diffusion models," in *Proceedings of the IEEE/CVF International Conference on Computer Vision*, 2023, pp. 3836–3847.

[7] C. Meng, Y. He, Y. Song, J. Song, J. Wu, J.-Y. Zhu, and S. Ermon, "Sdedit: Guided image synthesis and editing with stochastic differential equations," *arXiv preprint arXiv:2108.01073*, 2021.

[8] P. Dhariwal and A. Nichol, "Diffusion Models Beat GANs on Image Synthesis," in *Advances in Neural Information Processing Systems*, vol. 34. Curran Associates, Inc., 2021, pp. 8780–8794. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2021/hash/49ad23d1ec9fa4bd8d77d02681df5cfa-Abstract.html

[9] A. Bansal, H.-M. Chu, A. Schwarzschild, S. Sengupta, M. Goldblum, J. Geiping, and T. Goldstein, "Universal guidance for diffusion models," in *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 2023, pp. 843–852.

[10] Y. Blau and T. Michaeli, "The perception-distortion tradeoff," in *Proceedings of the IEEE conference on computer vision and pattern recognition*, 2018, pp. 6228–6237.

[11] —, "Rethinking lossy compression: The rate-distortion-perception tradeoff," in *International Conference on Machine Learning*. PMLR, 2019, pp. 675–685.

[12] F. Pezone, O. Musa, G. Caire, and S. Barbarossa, "Semantic-preserving image coding based on conditional diffusion models," *arXiv preprint arXiv:2310.15737*, 2023.

[13] T. Bordin and T. Maugey, "Semantic based generative compression of images for extremely low bitrates," in *2023 IEEE 25th International Conference on Computer Vision*, 2023, pp. 3836–3847.
Workshop on Multimedia Signal Processing (MMSP). IEEE, 2023, pp. 1–6.

[14] T. Bordin, T. Bachard, and T. Maugey, “Coclico: Extremely low bitrate image compression based on clip semantic and tiny color map,” in 2024 IEEE 37th International Workshop of Picture Coding Symposium (PCS). IEEE, 2024.

[15] E. Lei, Y. B. Uslu, H. Hassani, and S. S. Bidokhti, “Text+ sketch: Image compression at ultra low rates,” arXiv preprint arXiv:2307.01944, 2023.

[16] A. Wieckowski, J. Brandenburg, T. Hinz, C. Bartnik, V. George, G. Hege, C. Helmrich, A. Henkel, C. Lehmann, C. Stoffers, I. Zupancic, B. Bross, and D. Marpe, “Vvenc: An open and optimized vvc encoder implementation,” in Proc. IEEE International Conference on Multimedia Expo Workshops (ICMEW), pp. 1–2.

[17] M. J. Muckley, A. El-Nouby, K. Ullrich, H. Jégou, and J. Verbeek, “Improving statistical fidelity for neural image compression with implicit local likelihood models,” in International Conference on Machine Learning. PMLR, 2023, pp. 25 426–25 443.

[18] F. Mentzer, G. D. Toderici, M. Tschannen, and E. Agustsson, “High-fidelity generative image compression,” Advances in Neural Information Processing Systems, vol. 33, pp. 11 913–11 924, 2020.

[19] J. Song, C. Meng, and S. Ermon, “Denoising diffusion implicit models,” arXiv preprint arXiv:2010.02502, 2020.

[20] A. Lugmayr, M. Danelljan, A. Romero, F. Yu, R. Timofte, and L. Van Gool, “Repaint: Inpainting using denoising diffusion probabilistic models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 11 461–11 471.

[21] J. Ho and T. Salimans, “Classifier-free diffusion guidance,” arXiv preprint arXiv:2207.12598, 2022.

[22] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, “Score-based generative modeling through stochastic differential equations,” arXiv preprint arXiv:2011.13456, 2020.

[23] J. Betker, G. Goh, L. Jing, T. Brooks, J. Wang, L. Li, L. Ouyang, J. Zhuang, J. Lee, Y. Guo et al., “Improving image generation with better captions,” Computer Science. https://cdn.openai.com/papers/dall-e-3.pdf, vol. 2, no. 3, p. 8, 2023.

[24] E. Agustsson, D. Minnen, G. Toderici, and F. Mentzer, “Multi-realism image compression with a conditional generator,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 22 324–22 333.

[25] Z. Pan, X. Zhou, and H. Tian, “Extreme generative image compression by learning text embedding from diffusion models,” arXiv preprint arXiv:2211.07793, 2022.

[26] E. Grassucci, S. Barbarossa, and D. Comminiello, “Generative semantic communication: Diffusion models beyond bit recovery,” arXiv preprint arXiv:2306.04321, 2023.

[27] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning. PMLR, 2021, pp. 8748–8763.

[28] J. Duda, “Asymmetric numeral systems: entropy coding combining speed of huffman coding with compression rate of arithmetic coding,” arXiv preprint arXiv:1311.2540, 2013.

[29] P. von Platen, S. Patil, A. Lozhkov, P. Cuenca, N. Lambert, K. Rasul, M. Davaadorj, and T. Wolf, “Diffusers: State-of-the-art diffusion models,” https://github.com/huggingface/diffusers, 2022.

[30] C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, and J. Zhu, “Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models,” arXiv preprint arXiv:2211.01095, 2022.

[31] P. Esser, R. Rombach, and B. Ommer, “Taming transformers for high-resolution image synthesis,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 12 873–12 883.

[32] “Kodak: The kodak color image dataset,” http://r0k.us/graphics/kodak/.

[33] J. Wang, K. C. Chan, and C. C. Loy, “Exploring clip for assessing the look and feel of images,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 2, 2023, pp. 2555–2563.

[34] W. Zhang, K. Ma, J. Yan, D. Deng, and Z. Wang, “Blind image quality assessment using a deep bilinear convolutional neural network,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 30, no. 1, pp. 36–47, 2018.

[35] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “Gans trained by a two time-scale update rule converge to a local nash equilibrium,” Advances in neural information processing systems, vol. 30, 2017.

[36] M. Careil, M. J. Muckley, J. Verbeek, and S. Lathuilière, “Towards image compression with perfect realism at ultra-low bitrates,” in The Twelfth International Conference on Learning Representations, 2023.

## IX. APPENDIX

We showed the equations of guidance when the given condition is a color map. We formulated the color map as a linear transform of the data, a decimation over the DCT of the image.

This guidance can be generalized to any linear transform condition. Indeed, if $c = Az$, the equations of IV from (15) to (20) remain the same. In the case of color guidance, for our definition of the color map, we had $AA^T = I_m$ where $m$ was the number of frequencies kept from the DCT. More generally, the Eq. (22) can be written for any matrix $A$ as:

$$c|x_t \sim N\left(\hat{c}_t, \bar{\lambda}_t^2 \frac{1 - \alpha_t}{\alpha_t} AA^T\right) \quad (31)$$

We can then conclude on the guidance of a linearly filtered condition:

$$G(x_t, t, c) = \frac{\sqrt{\alpha_t}}{2\lambda_t} \nabla_{z_t} (c - \hat{c}_t)^T (AA^T)^{-1} (c - \hat{c}_t) \quad (32)$$

Using the pseudo inverse of $AA^T$ could be a solution when the matrix is not invertible. We did not explore more complex transformation than the linear. However, other work [4] present similar approaches when the diffusion is done in the pixel space.

We show another application of our guidance, this time, guiding only over higher frequencies of the image. Using the same formulation as for colors, we adapt the condition to be on other factors of the DCT and guide on this new $c$. We can thus guide on the outline of the objects rather than on the general color aspect. We show some results and the versatility of our formulation in Fig. 15.

<figure><img src="image_20.png" /><figcaption>Fig. 15: Guiding on the outline of the objects using our formulation on higher frequencies.</figcaption></figure>