# Variational Bayes Image Restoration With Compressive Autoencoders

Maud Biquard, Marie Chabert, Florence Genin, Christophe Latry, Thomas Oberlin

► To cite this version:

Maud Biquard, Marie Chabert, Florence Genin, Christophe Latry, Thomas Oberlin. Variational Bayes Image Restoration With Compressive Autoencoders. IEEE Transactions on Image Processing, 2025, 34, pp.2896-2909. 10.1109/TIP.2025.3564829. hal-05450929

HAL Id: hal-05450929

https://hal.science/hal-05450929v1

Submitted on 14 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# Variational Bayes image restoration with compressive autoencoders

Maud Biquard

ISAE-Supaero / CNES

31400 Toulouse, France

maud.biquard@isae-supaero.fr

Marie Chabert

IRIT/INP-ENSEEIHT

31000 Toulouse, France

marie.chabert@toulouse-inp.fr

Florence Genin, Christophe Latry

CNES

31400 Toulouse, France

firstname.lastname@cnes.fr

Thomas Oberlin

ISAE-Supaero

31400 Toulouse, France

thomas.oberlin@isae-supaero.fr

## Abstract

Regularization of inverse problems is of paramount importance in computational imaging. The ability of neural networks to learn efficient image representations has been recently exploited to design powerful data-driven regularizers. While state-of-the-art plug-and-play (PnP) methods rely on an implicit regularization provided by neural denoisers, alternative Bayesian approaches consider Maximum A Posteriori (MAP) estimation in the latent space of a generative model, thus with an explicit regularization. However, state-of-the-art deep generative models require a huge amount of training data compared to denoisers. Besides, their complexity hampers the optimization involved in latent MAP derivation. In this work, we first propose to use compressive autoencoders instead. These networks, which can be seen as variational autoencoders with a flexible latent prior, are smaller and easier to train than state-of-the-art generative models. As a second contribution, we introduce the Variational Bayes Latent Estimation (VBLE) algorithm, which performs latent estimation within the framework of variational inference. Thanks to a simple yet efficient parameterization of the variational posterior, VBLE allows for fast and easy (approximate) posterior sampling. Experimental results on image datasets BSD and FFHQ demonstrate that VBLE reaches similar performance as state-of-the-art PnP methods, while being able to quantify uncertainties significantly faster than other existing posterior sampling techniques. The code associated to this paper is available in https://github.com/MaudBqrd/VBLE.

**Index terms**— Image restoration, Inverse problems, Deep regularization, Posterior sampling

## 1 Introduction

Image restoration tasks, such as deblurring, inpainting, or super-resolution, consist in recovering a clean image $x$ from its noisy measurement $y$, based on the forward model $y = Ax + w$, with $A$ representing the degradation operator and $w$ an additive noise term. Within the Bayesian framework, this ill-posed inverse problem is typically solved by finding the Maximum A Posteriori (MAP) estimate, solution to the problem

$$ \max_x p_{X|Y}(x|y) \Leftrightarrow \min_x -\log p_{Y|X}(y|x) - \log p_X(x) \quad (1) $$

where $\log p_{Y|X}(y|x)$ is the observation log-likelihood, which is quadratic for Gaussian noise, and $-\log p_X(x)$ acts as a regularization term, promoting solutions that are most compatible with the prior distribution $p_X(x)$. Classical regularizations include total variation [57], Tikhonov regularization [64], and sparsity-promoting penalties on well-chosen representations such as wavelet bases or dictionaries [63, 30, 46].

Deep learning has led to substantial performance gains in image restoration tasks. A first category of methods directly solves the inverse problem after a supervised end-to-end training on a dataset of original-degraded image pairs related by the forward model [26, 58]. Their performance is impressive but only for the specific inverse problem considered during training. For that reason, a second category of methods aims at learning the regularization only, leveraging the forward model to solve the inverse problem. Besides offering better interpretability, they allow to solve a wide range of inverse problems with the same neural network.

Among them, plug-and-play (PnP) image restoration methods [66] yield excellent performance on a wide variety of image restoration tasks. These methods classically use splitting algorithms, such as ADMM [66, 12] or HQS
[70, 38], that separately handle the data term and the regularization term in the optimization problem (1). Their key concept is to use Gaussian denoisers [40], and in particular deep denoisers [71] in place of the proximal operator of the regularization. More recently, the introduction of generative denoisers, such as denoising diffusion models [36, 51], within the PnP framework, has demonstrated remarkable performance in solving inverse problems [42, 17, 73, 11, 56, 32].

A more direct way to learn the regularization consists in estimating the data distribution within generative models. This line of work, referred to as latent optimization in the following, seeks the inverse problem solution in the latent space of a generative model. Specifically, given a generative model $G$ which has been trained on a dataset of ideal images, the seminal work [7] computes a MAP estimate in its latent space by solving the following optimization problem using gradient descent:

$$ \max_z p_{Z|Y}(z|y) \Leftrightarrow \min_z -\log p_{Y|Z}(y|z) - \log p_Z(z), \quad (2) $$

where $z$ is the latent variable to be optimized, $p_Z(z)$ is the generative latent prior, and $p_{Y|Z}(y|z) = p_{Y|X}(y|G(z))$, that is $G$ is considered as a deterministic transformation from the latent to the image space. The original approach introduced in [7] is efficient for solving severely ill-posed inverse problems on images lying on a low-dimensional manifold, such as centered close-ups on human faces. Unfortunately, the manifold constraint $x = G(z)$ too strongly restricts the solution space when dealing with images lying on a high-dimensional manifold, which is the case for highly diverse and/or non-structured images, or when solving less severely ill-posed inverse problems. To address this limitation, [23, 22] permit small deviations from the generative manifold while [34, 27, 28] propose to jointly optimize $x$ and $z$ using splitting algorithms. Solving eq. (2) using Normalizing Flows (NF) [25] is also an option as it ensures the accessibility of any element in the image space [1, 52]. Even so, the quality of the solution remains highly dependent on the generative model. Unfortunately, the use of very deep state-of-the-art generative models makes the optimization of eq. (2) problematic. Indeed, the gradient descent often gets trapped in poor local minima [20] and it can be regarded as too computationally demanding for restoring a single image. Moreover, the training of very deep generative models requires a lot of ideal images, which might not be accessible in some application contexts. To circumvent gradient descent, [34] uses the encoder of a variational autoencoder (VAE) as a stochastic approximate of the generative model posterior in an alternate optimization scheme on $(x, z)$, while [55] obtains promising performance by extending this approach to hierarchical VAEs [60], at the cost of additional image restoration hyperparameters. Alternatively, PULSE [48] uses gradient descent over a high probability

hyper-sphere in the latent space. This allows to reduce the blur induced by the quadratic latent regularization associated with the Gaussian latent prior.

All the above mentioned methods only provide a single point estimate of the inverse problem solution. However, many applications require an estimation of the confidence behind this solution. To this end, several Bayesian methods enable to sample the posterior distribution $p_{X|Y}(x|y)$ of the inverse problem solution. Some methods yield stochastic solutions to the inverse problem, relying, for instance, on the implicit prior provided by Gaussian denoisers [39] or exploiting diffusion model properties [73][42]. However, posterior sampling is then computationally expensive since the inverse problem must be solved for each sample generation. In this regard, Markov Chain Monte Carlo (MCMC) can lead to more effective techniques. In particular, [29] proposes to use Unadjusted Langevin Algorithm (ULA) to solve imaging inverse problems. Building on this, PnP-ULA [44] approximates the log-likelihood gradient in ULA by employing a Gaussian denoiser in a PnP framework. NF-ULA [10] directly approximates this gradient using normalizing flows while [37] uses a MCMC sampling scheme in the latent space of a generative model. Many of these methods offer theoretical convergence guarantees towards the actual posterior distribution. However it requires in practice many iterations which makes the restoration computationally heavy, despite recent attempts to speed up Markov chain convergence [18, 53]. Interestingly, variational inference (VI) [45] consists in explicitly approximating the posterior distribution by a parametric distribution. This allows for a fast sampling procedure. However, the existing VI deep-learning-based methods essentially learn, in a supervised manner, a neural network that fits the variational parameters to solve a given inverse problem [69, 6, 65].

Focusing on latent optimization methods, this paper addresses the issues arising when dealing with diverse or non-structured images. Moreover, taking benefit of the latent framework, the proposed approach offers efficient means for posterior sampling. Hence, our contribution is twofold. First, we propose compressive autoencoders (CAEs) [3] as an alternative to state-of-the-art generative models for latent optimization methods. We consider in particular CAEs with a hyperprior [4], which yield excellent results in compression and can be seen as VAEs with an adaptive latent prior. In addition, they are significantly smaller than state-of-the-art generative models. On one hand, their adaptability allows them to properly regularize inverse problems on non-structured images. On the other hand, their light structure makes the latent optimization using gradient descent [7] both scalable and effective. Second, we introduce the Variational Bayes Latent Estimation (VBLE) algorithm which estimates the latent posterior $p_{Z|Y}(z|y)$ by variational inference, leveraging CAE characteristics to design a simple yet
efficient approximation of $p_{Z|Y}(z|y)$. In this way, VBLE enables to estimate the posterior distribution with negligible additional computational cost.

We conduct a comprehensive set of experiments on FFHQ [41] and BSD [47] datasets, comprising three different inverse problems: deblurring, single image super-resolution (SISR), and inpainting. Additionally, we perform a thorough study to demonstrate the relevance of VBLE posterior distribution. The proposed approach yields competitive image restoration results compared to state-of-the-art methods. Furthermore, it outperforms other posterior sampling methods in terms of computation time and GPU load, while yielding consistent abilities for posterior sampling and uncertainty quantification compared to state-of-the-art baselines.

# 2 Background

## 2.1 Variational autoencoders

VAEs are composed of an encoder, known as the inference model as it approximates the unknown posterior distribution, and of a decoder used as a generative model [43]. Both are neural networks, whose weights, $\phi$ and $\theta$ respectively, are learned using variational inference [6]. Let the image $x$ depend on a latent variable $z$ according to the following generative model:

$$p_{\theta}(x, z) = p_{\theta}(x|z)p_{\theta}(z). \qquad (3)$$

$p_{\theta}(z)$ is typically a simple distribution, often chosen as $N(0, I)$, while $p_{\theta}(x|z)$ represents the distribution learned by the decoder. The posterior distribution, defined as

$$p_{\theta}(z|x) = \frac{p_{\theta}(x|z)p_{\theta}(z)}{\int_{z} p_{\theta}(x|z)p_{\theta}(z)} \qquad (4)$$

is often intractable and is approximated by $q_{\phi}(z|x)$, the inference model produced by the VAE encoder.

The weights $\theta$ and $\phi$ are learned during the training of the VAE by maximizing the Evidence Lower BOund (ELBO):

$$\mathcal{L}_{\theta,\phi}(x) = \log p_{\theta}(x) - KL(q_{\phi}(z|x) || p_{\theta}(z|x)) \qquad (5)$$

where KL denotes the Kullblack-Leibler divergence, measuring the distance between distributions. Hence, by maximizing the ELBO on a dataset $\mathcal{D}$ containing several images $x$, the log-likelihood of $\mathcal{D}$ is maximized, while the distance between the approximate and the true posterior distributions is minimized. However, this version of the ELBO is intractable and can be rewritten as follows:

$$\mathcal{L}_{\theta,\phi}(x) = \mathbb{E}_{q_{\phi}(z|x)}[\log p_{\theta}(x|z)] - KL(q_{\phi}(z|x) || p_{\theta}(z)). \qquad (6)$$

<figure><img src="image_2.png" /><figcaption>Figure 1. Structure of a compressive autoencoder with hyperprior as in [4]. All mentioned distributions are factorized, in particular $N(\mu^z, (\sigma^z)^2) * U(\pm 0.5) = \prod_k [N(\mu_k^z, (\sigma_k^z)^2) * U(-0.5, 0.5)]$.</figcaption></figure>

It consists of two terms. The first one is a data fidelity term which ensures a good reconstruction of the input $x$ by the autoencoder. The second one encourages the latent space to match the shape of the prior distribution $p_{\theta}(z)$. Leveraging the reparameterization trick [43], Stochastic Gradient Variational Bayes (SGVB) estimates can be derived from the ELBO to update $(\theta, \phi)$.

## 2.2 Variational compressive autoencoders

Machine learning and especially deep learning have raised a significant interest in the lossy image compression community. In the widespread transform coding framework, the image is transformed, the obtained image representation is quantized and subsequently compressed using a lossless entropy encoder [68]. The more decorrelated and sparse the transform is, the more effective the compression becomes. Several reference model-based methods use the wavelet transform [59] while state-of-the-art deep learning methods use data-dependent transforms provided by the encoder part of a compressive autoencoder (CAE) [3, 4, 50, 15]. CAEs are learned by minimizing a rate-distortion trade-off $\mathcal{L} = \text{Rate} + \alpha \times \text{Distortion}$ [3, 4]. On one hand, the quantization introduces distortion between the original and the decompressed image. This distortion is typically measured by the Mean Squared Error (MSE) between the network input and output. On the other hand, the entropy encoder design requires a prior distribution or so-called entropy model $p$ for the quantized latent representation, denoted by $z$ in the following. However, $z$ actual distribution, denoted by $q$, generally differs from $p$. The rate is approximated by the Shannon cross-entropy: $\text{Rate} = \mathbb{E}_{z \sim q}[-\log_2 p(z)]$. This value is minimized when $p = q$, that is when the rate corresponds to the information entropy.

A specificity of CAE training comes from the fact that quantization is non-differentiable. It is thus approximated
during training by the addition of a uniform noise, which classically models quantization noise [8]. This noise is akin to the noise introduced during the training of a VAE in the generative framework. In the following, we denote by $\bar{z}$ the representation before the quantization. The latent variable $z$ is thus a noisy version of $\bar{z}$. Furthermore, state-of-the-art CAEs incorporate a hyperprior [4, 50, 13]. This additional autoencoder takes $\bar{z}$ as input and estimates $z$'s mean $\mu^z$ and standard deviation $\sigma^z$, as illustrated in Fig. 1. The prior on $z$ is then defined as the $\bar{z}$-dependent factorized distribution $\prod_k \mathcal{N}(\mu_k^z, (\sigma_k^z)^2)$ convolved by a uniform distribution to align with quantization. The hyperprior introduces another latent variable $h$, of a much smaller dimension than $z$. $h$ is also quantized, entropy coded, and transmitted.

CAEs can be formulated as VAEs [3, 4], with particular generative and inference models. Consider the following generative model, corresponding to a CAE with a hyperprior:

$$p_{\theta}(x, z, h) = p_{\theta}(x|z)p_{\theta}(z|h)p_{\theta}(h) \quad (7)$$

$$\text{with } p_{\theta}(x|z) = \prod_{k} \mathcal{N}(x_k; D_{\theta}(z)_k, \frac{1}{2\alpha \log(2)})$$

$$p_{\theta}(z|h) = \prod_{k} \left[ \mathcal{N}(\mu_k^z, (\sigma_k^z)^2) * U(z_k; [-\frac{1}{2}, \frac{1}{2}]) \right]$$

$$p_{\theta}(h) = \prod_{k} p_{\psi}(h_k)$$

where $D_{\theta}$ represents the decoder part of the autoencoder, and $p_{\psi}$ denotes a factorized parametric prior with weights $\psi$ learned during training. Consider also the following inference model:

$$q_{\phi}(z, h|x) = q_{\phi}(z|x, h)q_{\phi}(h|x) \quad (8)$$

$$\text{with } q_{\phi}(z|x, h) = \prod_k \mathcal{U}(z_k; [\bar{z}_k - \frac{1}{2}, \bar{z}_k + \frac{1}{2}]),$$

$$q_{\phi}(h|x) = \prod_k \mathcal{U}(h_k; [\bar{h}_k - \frac{1}{2}, \bar{h}_k + \frac{1}{2}]).$$

Then, the ELBO from eq. (6), expressed for a hierarchical VAE with two latent variables, corresponds to the rate-distortion loss, up to a $\log(2)$ factor:

$$\mathcal{L}(x) = \mathbb{E}_{q_{\phi}(z, h|x)} \left[ \log q_{\phi}(z, h|x) - \log p_{\theta}(x|z, h) - \log p_{\theta}(z, h) \right] \quad (9)$$

$$\propto 0 + \log(2)(\alpha \text{Distortion}(x, z) + \text{Rate}(z, h)).$$

Note that $\alpha$ controls the rate-distortion tradeoff, which is similar to the KL-data fidelity tradeoff for VAE that is generally controlled by a $\gamma$ parameter assuming a Gaussian decoder of variance $\gamma^2$ [19].

Furthermore, state-of-the-art CAEs combine a second autoencoder with an autoregressive part to design an improved hyperprior [50, 15]. In this case, CAEs cannot be entirely seen as VAEs, although their formulation stays close.

Finally, CAEs are powerful neural networks, while remaining often scalable, as they are to be used in embedded systems [21]. In particular, these networks use a parametric activation function, known as Generalized Divisive Normalization [2] (GDN), which, compared to classical activation functions, affords an equivalent approximation capacity for natural images with far shallower networks.

# 3 Proposed method

## 3.1 Regularization with compressive autoencoders

We claim that CAEs, introduced in Sec. 2.2, are good candidates to be used when restoring images with latent optimization methods. First, they can be viewed as VAEs and may be employed similarly. Second, they remain relatively light as their structure has been optimized for embedded image processing, enabling the use of gradient descent to compute the MAP-z estimate of eq. (2). Last, the hyperprior offers a flexible z-adaptive latent prior that better models the true latent distribution than typical VAE priors. Therefore, under the formalism of Sec. 2.2 summarized in Fig. 1, the MAP-z estimate for CAEs can be formulated as:

$$z^* = \arg \min_z - \log p_{Y|Z}(y|z) + \lambda \text{Rate}_\theta(z, E_{h,\phi}(z)) \quad (10)$$

where $E_{h,\phi}(z)$ denotes the output of the hyperencoder, $\text{Rate}_\theta(z, E_{h,\phi}(z)) = -\log_2 p_{\theta}(z, E_{h,\phi}(z))$ and, for the inverse problem $y = Ax + w$ defined in the introduction, $-\log p_{Y|Z}(y|z) \propto \|AD_{\theta}(z) - y\|_2^2$. Note that, in a proper MAP framework, the optimization should be with respect to both $z$ and $h$, and the second term in (10) should rather be $\lambda \text{Rate}_\theta(z, h)$. As $h$ is low-dimensional and does not significantly impact the rate, we choose the approximation $h = E_{h,\phi}(z)$. We show, in the ablation study provided in Sec. 4.5, that this approximation does not impact the algorithm performance.

## 3.2 Variational Bayes Latent Estimation (VBLE)

We still assume that $p_{Y|Z}(y|z) = p_{Y|X}(y|D_{\theta}(z))$. Recall that Bora et al. [7] method to solve inverse problems yields a MAP estimate $\arg\max_z p_{Z|Y}(z|y)$ in the latent space. Unlike this deterministic approach resulting in a point estimate, we wish to approximate $p_{Z|Y}(z|y)$ through variational inference (VI), leveraging VAEs and CAEs latent structure to propose a simple posterior parameterization. Hence, we present a stochastic version of eq. (10), termed Variational Bayes Latent Estimation (VBLE), which is illustrated in Fig. 3. It consists of performing VI with the
<figure><img src="image_3.png" /><figcaption>Figure 2. Overview of the image restoration process. First, a compressive autoencoder (CAE) is trained on a dataset of ideal images. Then, to restore a degraded image $y$, the latent posterior $p_{Z|Y}(z|y)$ is approximated using variational inference, and finally, the approximated posterior $q_{\bar{z},a}(z)$ is sampled in order to compute an MMSE estimate of the solution as well as uncertainties.</figcaption></figure>

following parametric families:

$$ 
\begin{align} 
\mathcal{E}_{\bar{z},a} &= \{q_{\bar{z},a}(z) | \bar{z}, a \in \mathbb{R}^{C \times M \times N}, a > 0\} \tag{11} \\ 
\text{with } q_{\bar{z},a}(z) &= \prod_k \mathcal{U}(z_k; [\bar{z}_k - \frac{a_k}{2}, \bar{z}_k + \frac{a_k}{2}]) \quad \text{(CAE case),} \\ 
q_{\bar{z},a}(z) &= \prod_k \mathcal{N}(z_k; \bar{z}_k, a_k^2) \quad \text{(VAE case).} 
\end{align} 
$$

Note that we detail both CAE and VAE frameworks, as VBLE can be applied with both of them. These parametric families are based on the same distribution as the inference encoder distribution $q_{\phi}(z|x)$, i.e. uniform for CAEs and Gaussian for VAEs. Their parameters are the mean latent representation $\bar{z}$, and an additional $a$ parameter with the same dimensions as $\bar{z}$, which models the uncertainty related to each coefficient of $z$. A representation of the posterior $q_{\bar{z},a}(z)$ is given in Fig. 2. The choice of a posterior belonging to the same distribution family as the encoder posterior $q_{\phi}(z|x)$ is a simple yet relevant choice. Indeed the encoder is trained so that sampling $q_{\phi}(z|x)$ provides the most likely latent variables given an input image $x$. Here, we aim to find the most likely latent variables given a measure $y$. Hence, adopting a unimodal distribution similar to the encoder posterior, however with an optimized variance, is a natural choice. Experiments in Sec. 4.4 demonstrate that this choice is beneficial to VBLE posterior sampling ability.

Then, employing a similar framework as the VAE training procedure described in Sec. 2.1, we maximize the ELBO:

$$ \arg\max_{\bar{z},a} \mathcal{L}_{\bar{z},a} = \arg\max_{z,a} \mathbb{E}_{q_{\bar{z},a}(z)} \left[ \log p_{Y|Z}(y|z) + \log p_{\theta}(z) - \log q_{\bar{z},a}(z) \right] \quad (12) $$

with $\log p_{\theta}(z)$ being the log-likelihood of the latent prior, that is, in particular, $\log N(z; 0, I)$ for a traditional VAE

<figure><img src="image_4.png" /><figcaption>Figure 3. Variational Bayes Latent Estimation (VBLE) graph. VI stands for Variational Inference.</figcaption></figure>

as introduced in eq. (3), and $\log(2) \text{ Rate}_{\theta}(z, E_{h,\phi}(\bar{z}))$ for a CAE with a hyperprior, as introduced in eq. (10). Note that $\log q_{\bar{z},a}(z) = -\sum_k \log a_k$ up to a constant in both Gaussian and uniform cases. We can then use the following reparameterization, for $z \sim q_{\bar{z},a}(z)$,

$$ z = \begin{cases} \bar{z} + a \odot \epsilon \quad \text{with } \epsilon \sim \prod_k \mathcal{U}(\epsilon_k; [-\frac{1}{2}, \frac{1}{2}]) & \text{(CAE case)} \\ \bar{z} + a \odot \epsilon \quad \text{with } \epsilon \sim \mathcal{N}(0, I) & \text{(VAE case)} \end{cases} $$

where $\odot$ denotes the Hadamard, point-wise product between two tensors. Hence, the expectation in eq. (12) can be taken on $\epsilon$ using the reparametrization trick [43]. Therefore, an SGVB estimate of the ELBO can be derived, leading to algorithm 1.

Hence, given the optimal parameters $(\bar{z}^*, a^*)$, the distribution $q_{\bar{z}^*,a^*}(z)$ is supposed to approximate the posterior $p_{Z|Y}(z|y)$. Then, two point estimates can be derived for the restored image:

$$ x_{\text{MMSE-z}}^* = D_{\theta}(\bar{z}^*), \quad (13) $$

$$ x_{\text{MMSE-x}}^* = \frac{1}{L} \sum_{i=1}^{L} D_{\theta}(z_i) \text{ with } z_i \sim q_{\bar{z}^*,a^*}(z_i). \quad (14) $$

Both MMSE-x and MMSE-z estimates are relevant and can be used in practice. Note that this variational framework
**Algorithm 1** Variational Bayes Latent Estimation

Require: y a degraded image, $\bar{z}_0 \in \mathbb{R}^{C \times M \times N}$, $a_0 \in \mathbb{R}^{C \times M \times N} = (1)_{i,j,l}$, $k=0, \eta > 0$

while not convergence do

$$ 
\begin{aligned}
z & \sim q_{\bar{z}_k, a_k}(z) \\
\binom{\bar{z}_{k+1}}{a_{k+1}} & = \binom{\bar{z}_k}{a_k} - \eta \nabla_{\bar{z},a} \left[ -\log p_{Y|Z}(y|z) - \log p_{\theta}(z) + \log q_{\bar{z},a}(z) \right] \\
k & = k + 1
\end{aligned}
$$

end while

$$ \textbf{return} (\bar{z}^*, a^*) = (\bar{z}_k, a_k) $$

does not increase the restoration time compared to its deterministic counterpart MAP-z [7]. Indeed, after convergence of VBLE, sampling from the approximate posterior is fast since it only requires forward passes through the decoder, which can be parallelized within batches.

## 3.3 Discussion and position to related works

Regarding latent optimization methods, most of them consider point estimates [7, 34, 27]. Yet, several approaches have proposed posterior sampling or ensemble methods that enable uncertainty quantification. It may be through sampling the latent posterior using MCMC [37, 62] or, as in the case of PULSE [48], by running a spherical gradient descent multiple times with different random initializations. PULSE does not provide samples of the latent posterior, but several MAP estimates. On the contrary, [37, 62] sample from the latent posterior. In particular, in [37], the latent posterior is defined exactly as in VBLE, that is $p_{Z|Y}(z|y) \propto p(z)p_{Y|X}(y|D_{\theta}(z))$, thus the two methods are expected to produce similar posterior samples. Compared to those methods, VBLE is a Variational Inference (VI) method, that only approximates the posterior distribution, but with a single latent optimization of the variational parameters. This key difference leads to a significantly reduced computational load for VBLE compared to MCMC methods [37, 62] and PULSE [48], as drawing several samples requires several latent optimizations with PULSE, and typically more than $10^6$ iterations with MCMC algorithms. Furthermore, to transform the latent samples into images, we consider a deterministic decoder, that is $p_{\theta}(x|z) = \delta_{D_{\theta}(z)}(x)$ while [62] obtains image samples by sampling $p(x|z, y) \propto p(x|z)p(y|x)$. By doing so in VBLE, algorithm 1 would remain unchanged, but the decoding, in particular of Equations 13,14 would be modified: the images would be potentially a bit less blurry. However, our main concern is that the decoder distribution $p_{\theta}(x|z) = N(x; D_{\theta}(z), \gamma^2 I)$ classically used in VAEs and CAEs is not well adapted as the fixed variance is not informative enough. Alternative approaches to improve the decoder variance [28] could be used. As this is not straightforward

to apply, this interesting perspective is left for future work.

Concerning VI methods, they have been widely used to approximate posterior distributions of inverse problem solutions [6, 65]. However, the approaches which combine VI with deep generative modeling are essentially designed in a (semi-)supervised manner to solve a specific inverse problem [65, 31, 33], while VBLE is able to solve different inverse problems with the same network. In [14], denoising is performed through score prior-guided deep VI, VI being used to approximate the noise model parameters. This requires optimizing the weights of a convolutional neural network (CNN) that predicts the noise parameters for each input image y to be restored. In contrast, since VBLE assumes that the noise and the degradation are known, it only uses VI to approximate the latent posterior. For that purpose, the variational parameters are directly optimized using gradient descent. Hence, up to our knowledge, VBLE is the first latent optimization method which explicitly estimates the latent posterior with variational inference to solve general inverse problems. Moreover, the uniform posterior shape for VBLE with CAE, which mimics the encoder posterior, is novel and yields remarkable results for solving imaging inverse problems, as will be shown in the next section.

Regarding the use of CAEs as regularizers, the relationship between CAEs and VAEs has been established in the literature [3, 49]. But, as far as we know, this is the first time that CAEs are used as priors for image restoration tasks. Although CAEs are not as deep as state-of-the-art hierarchical VAEs and may not exhibit the same performance for image generation, we believe that they make appropriate priors for image restoration. In particular, the parameter $\alpha$, which tunes the rate-distortion trade-off, provides additional flexibility and can be adjusted for a given inverse problem.

## 4 Numerical experiments

In this section, we evaluate the performance of VBLE using CAEs. With these experiments, our purpose is twofold. First, we aim to show that VBLE, in particular with the use of CAEs, yields state-of-the-art results for image restoration tasks. Second, we wish to demonstrate the huge interest of the proposed method for posterior sampling, which can be crucial for many applications.

In that purpose, we first detail the experimental setup, and then present image restoration results on two datasets. The first one, FFHQ, is highly structured and thus well-suited for generative models, whereas the second one, BSD, contains natural images with more diversity. Then, we assess the quality of our proposed approximate posterior both quantitatively and qualitatively compared to state-of-the-art posterior sampling baselines. Finally, we conduct an ablation study about our method and its variants.
## 4.1 Experimental setup

**Inverse problems** All experiments in this section are performed on subsets of BSD500 [47] and FFHQ [41] test datasets, both subsets composed of 47 images of size 256 × 256. VBLE and the baselines are evaluated on three inverse problems: deblurring, single image super-resolution (SISR), and inpainting. For deblurring, two Gaussian kernels of standard deviation $\sigma_{blur} \in \{1, 3\}$ are tested as well as a motion blur kernel, for several Gaussian noise levels. We consider the noiseless case for SISR and inpainting with random masks, and a small amount of noise is added for inpainting with structured masks. SISR ×2 and ×4 is conducted with bicubic downsampling. We employ a 50% random mask for inpainting experiments with random masks. For inpainting with structured masks, custom masks are designed on 5 test images of each dataset.

**Metrics** We use four metrics to measure the discrepancy between the restored and ground truth images: Peak Signal-to-Noise Ratio (PSNR), Structural SIMilarity [67] (SSIM), Learned Image Patch Similarity [72] (LPIPS) and Fréchet Inception Distance (FID) [35]. The PSNR derives from the pixel-wise mean-squared error, while SSIM and LPIPS are, respectively, classical and deep learning-based perceptual metrics. The FID is a no-reference perceptual metric. For each inverse problem, we evaluate the FID after extracting 49 patches of size 64 × 64 from each restored image.

**Compressive autoencoder architectures** We employ pretrained networks from the compressAI [9] library at different bitrates. We choose two models, from Minnen et al. [50], denoted as mbt, and Cheng et al. [15], denoted as cheng. mbt is a CAE with a hyperprior combining a second autoencoder with an autoregressive model, but its encoder and decoder structures remain simple CNNs. cheng retains the same structure as mbt, but with more elaborate encoders and decoders as they possess self-attention layers, yielding state-of-the-art compression results. For mbt and cheng models, two structures, a light and a bigger one, exist [9], which are used respectively for low and high bitrates. For mbt, a high bitrate structure is used for $\alpha \ge 0.013$, with $\alpha$ the bitrate parameter, and a low bitrate structure for $\alpha \le 0.0067$. For cheng, we have chosen the high bitrate structure for high and low bitrates, as it provides the best image restoration results. The number of parameters for each network is given in Sec. 4.3. The networks from compressAI are pretrained on natural images. We finetune them for about 100k iterations on BSD and FFHQ train datasets.

**VBLE parameters** For VBLE using CAEs, the network with the most appropriate bitrate needs to be chosen for

each inverse problem. We have trained models at 7 different bitrates. The bitrate parameter $\alpha$ to choose, as well as the regularization parameter $\lambda$, are tuned by a grid search algorithm on a 3-image validation set for each dataset. The other parameters are fixed, in particular, we use Adam optimizer with a learning rate of 0.1 during 1000 iterations for all restoration tasks. Lastly, the MMSE-x estimate, computed by averaging 100 VBLE posterior samples, is used for all comparisons as it yielded the best results.

**Image restoration parameters** To tune image restoration hyperparameters of VBLE and of the baselines, a grid search algorithm has been used on separate 3 image sets of FFHQ and BSD test datasets. The values of all the VBLE parameters for the different problems are given in the appendix, as well as the parameters that have been tuned or fixed for each baseline.

## 4.2 FFHQ experiments

First, we demonstrate the relevance of VBLE using CAEs for restoring highly structured face images from FFHQ dataset. To assess the effectiveness of both the CAE architecture and VBLE algorithm, we provide restoration results with VBLE (denoted as VBLE wCAE in the experiments) et also with its deterministic counterpart MAP-z using the same CAE (denoted as MAP-z wCAE). Results are provided with cheng and mbt CAE structures.

**FFHQ Baselines** We compare our method to several latent optimization approaches. First, MAP-z wVAE, proposed by Bora et al. [7] is the deterministic counterpart of VBLE, which seems a natural baseline. We also compare to PULSE [48], which performs spherical gradient descent in the latent space. Then, we compare to the state-of-the-art diffusion-based methods DiffPIR [73] and DDRM [42]. We use 100 Neural Function Evaluations (NFEs) for both methods. DiffPIR's and DDRM's outputs are a sample from the posterior distribution $p(x|y)$. Hence, they can be considered as posterior sampling methods, even if sampling from the posterior requires running the restoration process multiple times. When evaluating image restoration performance, we choose to run DiffPIR and DDRM only once, as is classically done in the literature. For DDRM, we do not perform motion blur as it is not implemented in the source code. For these baselines, we use an available FFHQ pre-trained diffusion model [16]. For Bora et al. method [7], we train two vanilla VAE models on FFHQ, VAE-V1, and VAE-V2. With these models, our aim is to demonstrate the effectiveness of the two latent variable structure specific to CAEs. Hence, these models are vanilla VAEs, that is VAEs with a single latent variable. Their structure has been designed so that it has the same depth, same number of hidden channels and same latent dimension as mbt with (respectively
<table><caption>Table 1. FFHQ results on diverse inverse problems. For methods using CAEs, (mbt) means that mbt CAE [50] is used, otherwise cheng CAE [15] is used. σ is the noise level used for deblurring. σ$_{k}$ is the kernel standard deviation for Gaussian deblurring.</caption><thead><tr><th colspan="11">Deblur - σ = 7.65</th></tr><tr><th rowspan="2">Method</th><th colspan="4">Gaussian, σ<sub>k</sub> = 1</th><th colspan="4">Gaussian, σ<sub>k</sub> = 3</th><th colspan="4">Motion</th></tr><tr><th>PSNR ↑</th><th>SSIM ↑</th><th>LPIPS ↓</th><th>FID ↓</th><th>PSNR ↑</th><th>SSIM ↑</th><th>LPIPS ↓</th><th>FID ↓</th><th>PSNR ↑</th><th>SSIM ↑</th><th>LPIPS ↓</th><th>FID ↓</th></tr></thead><tbody><tr><td>VBLE wCAE</td><td><strong>34.48</strong></td><td><strong>0.9250</strong></td><td><u>0.1473</u></td><td>38.03</td><td><strong>29.57</strong></td><td><strong>0.8214</strong></td><td><u>0.2859</u></td><td>74.12</td><td><strong>32.22</strong></td><td><strong>0.8800</strong></td><td><u>0.1933</u></td><td><strong>40.70</strong></td></tr><tr><td>MAP-z wCAE</td><td>34.17</td><td>0.9152</td><td>0.1684</td><td>48.88</td><td><u>29.44</u></td><td><u>0.8176</u></td><td>0.2977</td><td>92.65</td><td><u>32.01</u></td><td>0.8717</td><td>0.2441</td><td>64.08</td></tr><tr><td>VBLE wCAE (mbt)</td><td>34.21</td><td>0.9177</td><td>0.1600</td><td>43.44</td><td>28.70</td><td>0.7686</td><td>0.3212</td><td>102.96</td><td>31.97</td><td>0.8741</td><td>0.2149</td><td>48.15</td></tr><tr><td>MAP-z wVAE V1</td><td>32.78</td><td>0.8868</td><td>0.1990</td><td>71.43</td><td>28.72</td><td>0.7961</td><td>0.3242</td><td>120.15</td><td>27.95</td><td>0.6582</td><td>0.4331</td><td>192.81</td></tr><tr><td>MAP-z wVAE V2</td><td>30.91</td><td>0.8486</td><td>0.2688</td><td>66.01</td><td>28.47</td><td>0.7849</td><td>0.3263</td><td><u>79.12</u></td><td>29.44</td><td>0.8112</td><td>0.3088</td><td>73.55</td></tr><tr><td>PULSE</td><td>29.99</td><td>0.8289</td><td>0.2848</td><td>64.42</td><td>27.83</td><td>0.7670</td><td>0.3429</td><td>81.00</td><td>28.36</td><td>0.7848</td><td>0.3266</td><td>74.85</td></tr><tr><td>DiffPIR</td><td>33.92</td><td>0.9049</td><td><strong>0.1368</strong></td><td><u>36.12</u></td><td>29.41</td><td>0.8102</td><td><strong>0.2691</strong></td><td>100.98</td><td>31.98</td><td>0.8677</td><td><strong>0.1793</strong></td><td><u>42.50</u></td></tr><tr><td>DDRM</td><td>32.22</td><td>0.8831</td><td>0.1785</td><td><strong>33.84</strong></td><td>24.74</td><td>0.7051</td><td>0.2931</td><td><strong>41.01</strong></td><td>x</td><td>x</td><td>x</td><td>x</td></tr></tbody></table><table><thead><tr><th colspan="11">SISR and Inp. Method</th></tr><tr><th rowspan="2">Method</th><th colspan="4">SISR ×2</th><th colspan="4">SISR ×4</th><th colspan="4">Inpainting (random)</th></tr><tr><th>PSNR ↑</th><th>SSIM ↑</th><th>LPIPS ↓</th><th>FID ↓</th><th>PSNR ↑</th><th>SSIM ↑</th><th>LPIPS ↓</th><th>FID ↓</th><th>PSNR ↑</th><th>SSIM ↑</th><th>LPIPS ↓</th><th>FID ↓</th></tr></thead><tbody><tr><td>VBLE wCAE</td><td><strong>36.16</strong></td><td><strong>0.9468</strong></td><td>0.1251</td><td>40.44</td><td><strong>31.18</strong></td><td><strong>0.8670</strong></td><td>0.2290</td><td><u>56.21</u></td><td><u>35.94</u></td><td><u>0.9584</u></td><td>0.0638</td><td>31.01</td></tr><tr><td>MAP-z wCAE</td><td><u>36.30</u></td><td><strong>0.9516</strong></td><td>0.0995</td><td>34.59</td><td><u>31.08</u></td><td><u>0.8681</u></td><td>0.2205</td><td>62.91</td><td><u>36.91</u></td><td><u>0.9618</u></td><td><u>0.0715</u></td><td><u>30.42</u></td></tr><tr><td>VBLE wCAE (mbt)</td><td>35.99</td><td>0.9514</td><td>0.0956</td><td>38.64</td><td>31.01</td><td>0.8680</td><td>0.2258</td><td>60.95</td><td><strong>37.04</strong></td><td><strong>0.9643</strong></td><td>0.0738</td><td>33.73</td></tr><tr><td>MAP-z wVAE V1</td><td>34.35</td><td>0.9311</td><td><strong>0.0893</strong></td><td>38.51</td><td>30.24</td><td>0.8499</td><td>0.2201</td><td>75.74</td><td>35.21</td><td>0.9443</td><td>0.0775</td><td>35.00</td></tr><tr><td>MAP-z wVAE V2</td><td>31.20</td><td>0.8534</td><td>0.2592</td><td>64.24</td><td>29.53</td><td>0.8226</td><td>0.2765</td><td>66.55</td><td>31.01</td><td>0.8513</td><td>0.2660</td><td>66.29</td></tr><tr><td>PULSE</td><td>29.79</td><td>0.8203</td><td>0.2989</td><td>66.85</td><td>27.11</td><td>0.7395</td><td>0.3927</td><td>85.81</td><td>30.06</td><td>0.8300</td><td>0.2858</td><td>64.37</td></tr><tr><td>DiffPIR</td><td>34.94</td><td>0.9269</td><td>0.1089</td><td><u>34.63</u></td><td>30.71</td><td>0.8492</td><td><u>0.2042</u></td><td>68.34</td><td>36.08</td><td>0.9470</td><td><strong>0.0653</strong></td><td>30.93</td></tr><tr><td>DDRM</td><td>35.81</td><td>0.9479</td><td><strong>0.0640</strong></td><td><strong>27.19</strong></td><td>30.80</td><td>0.8608</td><td><strong>0.1749</strong></td><td><strong>38.65</strong></td><td>34.43</td><td>0.9372</td><td>0.0814</td><td><strong>24.54</strong></td></tr></tbody></table>

<table><caption>Table 2. BSD results on diverse inverse problems. σ is the noise level used for deblurring. σ$_{k}$ is the kernel standard deviation for Gaussian deblurring.</caption><thead><tr><th colspan="11">BSD - σ = 7.65</th></tr><tr><th rowspan="2">Method</th><th colspan="4">Deblur (Gauss., σ<sub>k</sub> = 1)</th><th colspan="4">Deblur (Gauss., σ<sub>k</sub> = 3)</th><th colspan="4">Deblur (Motion)</th></tr><tr><th>PSNR ↑</th><th>SSIM ↑</th><th>LPIPS ↓</th><th>FID ↓</th><th>PSNR ↑</th><th>SSIM ↑</th><th>LPIPS ↓</th><th>FID ↓</th><th>PSNR ↑</th><th>SSIM ↑</th><th>LPIPS ↓</th><th>FID ↓</th></tr></thead><tbody><tr><td>VBLE wCAE</td><td><strong>29.62</strong></td><td><u>0.8598</u></td><td>0.1901</td><td>61.38</td><td>24.66</td><td>0.6433</td><td><u>0.3956</u></td><td>89.22</td><td>28.38</td><td><u>0.8158</u></td><td><u>0.2356</u></td><td>64.91</td></tr><tr><td>MAPz wCAE</td><td>29.28</td><td>0.8405</td><td>0.2215</td><td>68.03</td><td>24.48</td><td>0.6347</td><td>0.4193</td><td>119.67</td><td>28.07</td><td>0.7896</td><td>0.3111</td><td>101.16</td></tr><tr><td>PnP-ULA</td><td>27.71</td><td>0.7809</td><td>0.2675</td><td>67.50</td><td>23.75</td><td>0.5913</td><td>0.4592</td><td>114.23</td><td>27.17</td><td>0.7402</td><td>0.2643</td><td>71.56</td></tr><tr><td>DPIR</td><td><u>29.72</u></td><td><strong>0.8623</strong></td><td>0.2127</td><td>73.76</td><td>24.41</td><td><strong>0.6427</strong></td><td>0.4315</td><td>134.92</td><td><strong>28.69</strong></td><td><strong>0.8285</strong></td><td>0.2606</td><td>83.44</td></tr><tr><td>PnP-ADMM</td><td>28.96</td><td>0.8456</td><td>0.2155</td><td>67.93</td><td>24.24</td><td>0.6333</td><td>0.4350</td><td>131.58</td><td>28.16</td><td>0.8045</td><td>0.3349</td><td>127.81</td></tr><tr><td>DiffPIR</td><td>29.26</td><td>0.8420</td><td><strong>0.1832</strong></td><td><strong>55.34</strong></td><td><strong>24.58</strong></td><td>0.6405</td><td><strong>0.3605</strong></td><td>98.93</td><td>28.15</td><td>0.7955</td><td><strong>0.2230</strong></td><td><strong>61.19</strong></td></tr><tr><td>DDRM</td><td>27.03</td><td>0.7763</td><td>0.2238</td><td><u>56.12</u></td><td>20.32</td><td>0.4747</td><td>0.4376</td><td><strong>71.75</strong></td><td>x</td><td>x</td><td>x</td><td>x</td></tr></tbody></table><table><thead><tr><th colspan="11">BSD Method</th></tr><tr><th rowspan="2">Method</th><th colspan="4">SISR ×2</th><th colspan="4">SISR ×4</th><th colspan="4">Inpainting (Random)</th></tr><tr><th>PSNR ↑</th><th>SSIM ↑</th><th>LPIPS ↓</th><th>FID ↓</th><th>PSNR ↑</th><th>SSIM ↑</th><th>LPIPS ↓</th><th>FID ↓</th><th>PSNR ↑</th><th>SSIM ↑</th><th>LPIPS ↓</th><th>FID ↓</th></tr></thead><tbody><tr><td>VBLE wCAE</td><td><strong>29.55</strong></td><td>0.8748</td><td>0.1802</td><td>65.72</td><td><strong>25.53</strong></td><td><strong>0.7032</strong></td><td>0.3507</td><td>99.09</td><td>30.21</td><td>0.9070</td><td>0.1207</td><td>56.60</td></tr><tr><td>MAPz wCAE</td><td><strong>29.65</strong></td><td>0.8757</td><td>0.1675</td><td>62.53</td><td><u>25.38</u></td><td><u>0.6966</u></td><td>0.3462</td><td>100.87</td><td>30.45</td><td>0.9091</td><td>0.1247</td><td>56.25</td></tr><tr><td>PnP-ULA</td><td>28.42</td><td><u>0.8763</u></td><td>0.1659</td><td>57.40</td><td>24.82</td><td>0.6894</td><td>0.3730</td><td>108.86</td><td>26.29</td><td>0.8122</td><td>0.2497</td><td>81.46</td></tr><tr><td>DPIR</td><td>29.55</td><td><strong>0.8784</strong></td><td>0.1706</td><td>61.35</td><td>25.18</td><td>0.6937</td><td>0.3652</td><td>114.41</td><td><u>31.58</u></td><td><u>0.9316</u></td><td><strong>0.0659</strong></td><td>58.20</td></tr><tr><td>PnP-ADMM</td><td>29.31</td><td>0.8678</td><td>0.1907</td><td>66.94</td><td>24.77</td><td>0.6659</td><td>0.4227</td><td>139.96</td><td><strong>31.69</strong></td><td><strong>0.9322</strong></td><td><u>0.0675</u></td><td><u>53.68</u></td></tr><tr><td>DiffPIR</td><td>29.20</td><td>0.8539</td><td><strong>0.1645</strong></td><td>57.78</td><td>25.03</td><td>0.6771</td><td><u>0.3164</u></td><td>88.83</td><td>30.41</td><td>0.9018</td><td>0.0919</td><td>55.94</td></tr><tr><td>DDRM</td><td>28.73</td><td>0.8691</td><td><strong>0.1284</strong></td><td><strong>51.27</strong></td><td>25.12</td><td>0.6895</td><td><strong>0.2885</strong></td><td><strong>64.57</strong></td><td>29.48</td><td>0.8754</td><td>0.1268</td><td><strong>52.59</strong></td></tr></tbody></table>

without) hyperprior in the case of VAE-V1 (respectively VAE-V2). Hence, VAE-V1 encoder possesses 5 convolutional layers and VAE-V2 7 convolutional layers. The latent dimensions of VAE-V1 and V2 correspond to z (first latent variable) and h (second latent variable) dimensions of mbt CAE. Both models have been trained on FFHQ using a learning rate of $1e^{-4}$ during at least 300k iterations. We choose a Gaussian assumption for the decoder, that is $p_{\theta}(x|z) = N(D_{\theta}(z), \gamma^2I)$. The variance $\gamma^2$ is considered a global parameter and is learned with the other network parameters. VAE-V2 model is used for PULSE [48] baseline. Although more elaborate VAEs are likely to perform better as long as their size does not hamper the latent optimization, here we aim to assess the benefit of the CAE architecture, with its specific hyperprior. Hence we choose to compare

with VAEs having similar complexity.

**Results** Quantitative results on FFHQ are provided in Tab. 1. First, VBLE exhibits state-of-the-art results compared to the baselines. Compared to DiffPIR, VBLE exhibits better PSNR and SSIM, a similar FID, and slightly lower LPIPS scores, which indicates that these methods have a slightly different position in the perception-distortion tradeoff [5]. DDRM outperforms all methods in FID, but is below VBLE and DiffPIR for the other metrics. Then Bora's method MAP-z with VAEs and PULSE are outperformed by VBLE. Furthermore, VBLE with CAE yields better metrics than MAP-z with CAE, hence the use of VBLE algorithm instead of a deterministic gradient descent not only enables posterior sampling but also improves point
<figure><img src="image_5.png" /><figcaption>Figure 4. Visual results on FFHQ. From top to bottom: motion deblurring ($\sigma = 7.65/255$), SISR $\times 4$, inpainting with random masks.</figcaption></figure>

estimation results. Additionally, visual comparisons are provided in Fig. 4. VBLE, DDRM and DiffPIR perform very well on FFHQ. Compared to VBLE, the diffusion-based methods yield particularly good reconstruction on details such as eyes and teeth. This can be partly explained by the MMSE-x estimate chosen for VBLE. Indeed, it corresponds to the posterior mean, which tends to be smoother but more faithful to the measurement. VAE-V1, which imitates a CAE without hyperprior, seems to poorly restore structure while VAE-V2, which has the same depth as a CAE with hyperprior but has a single latent variable, lacks high frequencies. This demonstrates the interest of the CAE structure and its hyperprior: its two-latent variable structure achieves retrieving both semantic information and finer details.

## 4.3 BSD experiments

<table><caption>Table 3. Complexity analysis of our method and the baselines of Section IV.C-D. # means "number of". The times, the number of Neural Function Evaluations (NFEs), the number of backward steps, and maximum GPU memory are given for inpainting a BSD image on a Nvidia Quadro RTX 8000. Time and #NFEs are given for computing either 1 estimate or 100 posterior samples (for VBLE, optimization AND sampling included).</caption><thead><tr><th rowspan="2"></th><th colspan="2">1 solution</th><th colspan="2">100 post.samples</th><th rowspan="2">GPU mem. (MiB)</th><th rowspan="2">#NN</th><th rowspan="2">#Backward params</th></tr><tr><th>Time</th><th>#NFEs</th><th>Time</th><th>#NFEs</th></tr></thead><tbody><tr><td>VBLE wCAE</td><td>22.4s</td><td>1000</td><td><strong>24.3s</strong></td><td><strong>1 100</strong></td><td>400</td><td><strong>29M</strong></td><td>1000</td></tr><tr><td>PnP-ULA</td><td>x</td><td>x</td><td>1h23min</td><td>100 000</td><td>341</td><td>32M</td><td>0</td></tr><tr><td>DPIR</td><td><strong>1.9s</strong></td><td><strong>50</strong></td><td>x</td><td>x</td><td><strong>280</strong></td><td>32M</td><td>0</td></tr><tr><td>PnP-ADMM</td><td>4.1s</td><td>150</td><td>x</td><td>x</td><td>282</td><td>32M</td><td>0</td></tr><tr><td>DiffPIR</td><td>21s</td><td>100</td><td>35min</td><td>10 000</td><td>2690</td><td>552M</td><td>0</td></tr><tr><td>DDRM</td><td>44.5s</td><td>100</td><td>1h14min</td><td>10 000</td><td>36 270</td><td>552M</td><td>0</td></tr></tbody></table>

We now compare VBLE using CAEs to state-of-the-art PnP baselines on BSD dataset, which is composed of various

natural images. We consider four state-of-the-art methods: DPIR [70], PnP-ADMM [66], DiffPIR [73] and DDRM [42]. As for FFHQ, we do not perform motion blur with DDRM as it is not implemented in the source code. We also compare to the MCMC approach PnP-ULA [44], which is a state-of-the-art posterior sampling method. For DPIR, PnP-ADMM, and PnP-ULA, we use an available version of DRUNet [70] trained on BSD. For DiffPIR and DDRM, a pretrained model on ImageNet [24] is used, that we finetune on BSD dataset. For PnP-ULA, $10^5$ MCMC iterations are performed. Similarly to FFHQ results, we provide restoration results with VBLE and its deterministic counterpart using CAE (denoted by MAPz wCAE in the experiments).

Table 2 shows quantitative results on deblurring and SISR inverse problems, while Fig. 5 provides a visual comparison. VBLE exhibits very consistent performance as it reaches the first or second rank for a majority of inverse problems and metrics. Besides, VBLE clearly outperforms PnP-ULA. Visually, except for PnP-ULA, all methods perform very well. In particular, the diffusion-based results are particularly sharp, while VBLE shows very relevant results, staying faithful to the image.

Additionally, Tab. 3 provides insights about the computation time, maximum GPU memory, and other complexity-related information. VBLE time includes the variational inference optimization and the sampling for one restoration. In particular, for drawing 100 posterior samples, VBLE is much faster than DiffPIR, DDRM, and PnP-ULA, with a lower computational load than DiffPIR and DDRM. Note that algorithm 1 is performed during 1000 iterations, thus VBLE, even for computing a single estimate, requires 1000 NFEs.

Therefore, VBLE yields state-of-the-art image restoration results compared to very competitive diffusion and PnP
<figure><img src="image_6.png" /><figcaption>Figure 5. Qualitative image restoration results on BSD. From top to bottom: Motion deblurring (σ = 7.65/255), SISR ×4, SISR ×2 and two block inpainting experiments (σ = 2.55/255).</figcaption></figure>

baselines while outperforming the posterior sampling methods in terms of computation time and GPU load. This demonstrates the huge interest of VBLE with CAE for image restoration tasks, especially when uncertainty quantification is required as it enables posterior sampling with a far lower computational cost than MCMC or diffusion-based methods.

## 4.4 Posterior distribution quality assessment

Sections 4.2 and 4.3 show that VBLE using CAEs reaches state-of-the-art point estimation results for image restoration while outperforming the other posterior sampling methods. In this section, we analyze the quality of VBLE approximate posterior distribution, qualitatively and quantitatively compared to DiffPIR and PnP-ULA. First, Tab. 4 provides an in-depth comparison between VBLE, DiffPIR, and PnP-ULA. In this table, the metrics are computed from the posterior mean, estimated by the average of 100 posterior samples for each image. Thus, DiffPIR is run 100 times for each of these images. Finally, the different metrics are averaged on the whole test set. DiffPIR has very good PSNR and LPIPS but is far slower than VBLE. Furthermore, in this table, we evaluate the quality of the different posteriors using confidence intervals. Precisely, for

each pixel of an image, we compute a confidence interval of a given level using 100 posterior samples. Then, we evaluate the interval coverage probability (ICP), that is the proportion of ground truth pixels lying in the predicted interval. In Tab. 4, ICPs at level 95% are provided for several inverse problems. PnP-ULA has the best ICPs, while VBLE ICPs are slightly below and DiffPIR is outperformed by PnP-ULA and VBLE. Hence, VBLE offers an interesting trade-off as it remains competitive with DiffPIR, while having a relevant approximate posterior and above all, being significantly faster than both methods for posterior sampling.

Second, in Fig. 6, we provide, for the different methods, some posterior samples, as well as a comparison between the true pixelwise error and the 95% quantile of the predicted error. A coverage graph is also provided for each image. Precisely, for each image, a confidence interval of level $\beta$ is derived from the squared error for each pixel. Then, the empirical coverage is plotted as a function of the theoretical coverage $\beta$ varying from 0 to 100%. This evaluation process is the pixelwise counterpart of the one proposed in [54]. For both images of Fig. 6, VBLE exhibits on average better coverage graphs than DiffPIR and PnP-ULA. Visually, the predicted errors are relevant. In comparison, PnP-ULA pre-
<table><caption>Table 4. Comparison of posterior sampling methods. DiffPIR-100 corresponds DiffPIR, which is run 100 times for each image to get 100 posterior samples. The time is the computation time needed for drawing 100 posterior samples for an inpainting problem on a Nvidia Quadro RTX 8000 (for VBLE time, optimization AND sampling is included). ICP 95% is the confidence coverage probability of level 95% (the closer to 0.95, the better). σ = 7.65 for deblurring.</caption><thead><tr><th rowspan="2">FFHQ<br>Method</th><th colspan="4">Deblur (Gaus.) σ<sub>k</sub> = 1</th><th colspan="4">Deblur (Motion)</th><th colspan="4">SISR ×2</th><th colspan="4">SISR ×4</th></tr><tr><th>Time</th><th>PSNR ↑</th><th>LPIPS ↓</th><th>ICP 95%</th><th>PSNR ↑</th><th>LPIPS ↓</th><th>ICP 95%</th><th>PSNR ↑</th><th>LPIPS ↓</th><th>ICP 95%</th><th>PSNR ↑</th><th>LPIPS ↓</th><th>ICP 95%</th><th>PSNR ↑</th><th>LPIPS ↓</th><th>ICP 95%</th></tr></thead><tbody><tr><td>VBLE wCAE</td><td>27sec</td><td>34.48</td><td>0.1473</td><td>0.85</td><td>32.22</td><td>0.1933</td><td>0.87</td><td>36.16</td><td>0.1251</td><td>0.57</td><td>31.18</td><td>0.2290</td><td>0.73</td><td></td><td></td><td></td></tr><tr><td>DiffPIR-100</td><td>8min10sec</td><td><strong>34.78</strong></td><td><strong>0.1445</strong></td><td>0.76</td><td><strong>32.95</strong></td><td><strong>0.1869</strong></td><td>0.75</td><td><strong>35.78</strong></td><td><strong>0.1207</strong></td><td>0.79</td><td><strong>31.21</strong></td><td><strong>0.2252</strong></td><td>0.69</td><td></td><td></td><td></td></tr><tr><td>PnP-ULA</td><td>1h23min</td><td>32.51</td><td>0.2014</td><td><strong>0.99</strong></td><td>30.35</td><td>0.2367</td><td><strong>0.99</strong></td><td>30.82</td><td>0.1570</td><td><strong>0.92</strong></td><td>28.94</td><td>0.2495</td><td>0.68</td><td></td><td></td><td></td></tr></tbody><tbody><tr><th rowspan="2">BSD<br>Method</th><th colspan="4">Deblur (Gaus.), σ<sub>k</sub> = 1</th><th colspan="4">Deblur (Motion)</th><th colspan="4">SISR ×2</th><th colspan="4">SISR ×4</th></tr><tr><th>Time</th><th>PSNR ↑</th><th>LPIPS ↓</th><th>ICP 95%</th><th>PSNR ↑</th><th>LPIPS ↓</th><th>ICP 95%</th><th>PSNR ↑</th><th>LPIPS ↓</th><th>ICP 95%</th><th>PSNR ↑</th><th>LPIPS ↓</th><th>ICP 95%</th><th>PSNR ↑</th><th>LPIPS ↓</th><th>ICP 95%</th></tr><tr><td>VBLE wCAE</td><td>27sec</td><td>29.62</td><td>0.1901</td><td>0.81</td><td>28.38</td><td>0.2356</td><td>0.78</td><td>29.55</td><td>0.1802</td><td>0.44</td><td><strong>25.53</strong></td><td><strong>0.3507</strong></td><td>0.60</td><td></td><td></td><td></td></tr><tr><td>DiffPIR-100</td><td>35min00sec</td><td><strong>29.88</strong></td><td><strong>0.1847</strong></td><td>0.67</td><td><strong>29.09</strong></td><td><strong>0.2351</strong></td><td>0.70</td><td><strong>29.61</strong></td><td><strong>0.1789</strong></td><td>0.61</td><td><strong>25.36</strong></td><td><strong>0.3394</strong></td><td>0.69</td><td></td><td></td><td></td></tr><tr><td>PnP-ULA</td><td>1h23min</td><td>27.71</td><td>0.2675</td><td><strong>0.95</strong></td><td>27.17</td><td>0.2643</td><td><strong>0.98</strong></td><td>28.42</td><td><strong>0.1659</strong></td><td><strong>0.78</strong></td><td>24.82</td><td>0.3730</td><td>0.59</td><td></td><td></td><td></td></tr></tbody></table>

<figure><img src="image_7.png" /><figcaption>Figure 6. Visual comparison of posterior sampling ability of VBLE, DiffPIR and PnP-ULA. Top: SISR ×2 on FFHQ. Bottom: Motion deblurring on BSD. The MMSE estimate is the average of 100 samples for each method. The true error denotes the absolute error $|x - \hat{x}|$ between the ground truth and the MMSE. The predicted error (95%) denotes the 95% quantile of $\{|\hat{x} - x_i|\}_i$ with $x_i$ a posterior sample.</figcaption></figure>
<figure><img src="image_8.png" /><figcaption>Figure 7. Example of VBLE posterior sampling ability. Left: The MMSE-x, which is the estimate used in all quantitative comparison, corresponds to the average over samples in the image space. Marginal std corresponds to pixelwise marginal standard deviations computed with 100 posterior samples. Right: Illustration of the latent diversity on FFHQ for a SISR ×4 problem. The samples lie on a grid. The coordinates correspond to the latent coordinates of the samples, projected on two orthogonal directions with high variance in the latent space.</figcaption></figure>

dictions are over or underestimated depending on the prob-
lems, probably due to the difficulty of the MCMC algorithm
to explore efficiently the posterior distribution. DiffPIR pre-
dictions seem relevant but underestimated in the SISR prob-
lem.

Lastly, VBLE posterior sampling ability is shown in
more detail in Fig. 7. In the left part of the figure, posterior
samples are sharp and realistic, showing the interest of for-
mulating the posterior distribution in the latent space, using
a uniform distribution. The MMSE-x estimate is smoother
than the samples and, hence, tends to be more faithful to
the target image. Additionally, the right part of Fig. 7 pro-
vides an insight into the obtained latent diversity, using pos-
terior samples projected on two orthogonal directions in the
latent space with a high variance, that is lying in the set
{a⊙u|u ∈ {-0.5, 0.5}C×M×N} with a the optimized vari-
ational parameter from eq. (11). Following the first direc-
tion, the catchlight, that is the white light reflect in the eye,
goes from right to left. Following the second direction, its
size increases. This illustrates the uncertainty that can be
learnt within this latent uncertainty model.

Therefore, in addition to state-of-the-art point estimation
results, VBLE approximate posterior distribution is rele-

vant, as it provides accurate confidence intervals and realis-
tic posterior samples. This validates the choice of the para-
metric family chosen to estimate the posterior in the latent
space of CAEs. In particular, an unimodal latent distribu-
tion with a similar shape as the encoder posterior q<sub>ϕ</sub>(z|x)
shape provides a remarkably simple way to model the pos-
terior for an image restoration task.

4.5 Ablation Study

<table><caption>Table 5. Quantitative results for different variants of our method. MMSE-x and MMSE-z denote the two possible estimators defined in Sec. 3.2. For Gaussian deblur, the noise level is σ = 7.65/255 and kernel deviation is σ_k = 3.</caption>
  <thead>
    <tr>
      <th rowspan="2">Method</th>
      <th colspan="3">FFHQ - SISR ×4</th>
      <th colspan="3">BSD - Gaussian deblur</th>
    </tr>
    <tr>
      <th>PSNR ↑</th>
      <th>LPIPS ↓</th>
      <th>SSIM ↑</th>
      <th>PSNR ↑</th>
      <th>LPIPS ↓</th>
      <th>SSIM ↑</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>VBLE MMSE-x</td>
      <td><strong>31.18</strong></td>
      <td>0.2290</td>
      <td>0.8669</td>
      <td>24.66</td>
      <td>0.3956</td>
      <td>0.6433</td>
    </tr>
    <tr>
      <td>VBLE MMSE-z</td>
      <td>31.17</td>
      <td>0.2468</td>
      <td>0.8662</td>
      <td><strong>24.73</strong></td>
      <td>0.4278</td>
      <td><strong>0.6519</strong></td>
    </tr>
    <tr>
      <td>VBLE<sub>z,h</sub> MMSE-x</td>
      <td><strong>31.18</strong></td>
      <td><strong>0.2281</strong></td>
      <td><strong>0.8671</strong></td>
      <td>24.68</td>
      <td><strong>0.3952</strong></td>
      <td>0.6444</td>
    </tr>
  </tbody>
</table>

For the experiments presented in the paper, some choices
have been made. In particular, the MMSE-x estimate has
been preferred to the MMSE-z estimate (see Sec. 3.2), while
only *z* has been optimized as we approximated the second
latent variable $h = E_{h,\phi}(\bar{z})$ (see Sec. 3.1). In this section, in order to justify these choices, we perform quantitative and qualitative results with the MMSE-z estimate and with an optimization on $(z, h)$ (denoted by $\text{VBLE}_{z,h}$). All these compared variants of the proposed method share the same restoration parameters (i.e. same $\lambda$ and $\alpha$) and use the same CAE.

<figure><img src="image_9.png" /><figcaption>Figure 8. Visual results for different variants of our method. Top: SISR ×4. Bottom: Gaussian deblurring ($\sigma_k = 3$, $\sigma = 7.65/255$).</figcaption></figure>

Table 5 shows quantitative results for VBLE and its variant on two inverse problems. $\text{VBLE}_{z,h}$ and the reference version yield almost identical results. Thus, the optimization on $(z, h)$ does not perform better than the optimization on $z$ only. For that reason, we choose to optimize only on $z$ for the main experiments, as the formulation is much simpler. Furthermore, the MMSE-z estimate yields similar PSNR and SSIM results as the MMSE-x estimate but exhibits always a poorer LPIPS. Furthermore, visual results are given in Fig. 8. While $\text{VBLE}_{z,h}$ and the reference version are visually very similar, the MMSE-z estimate is less textured and detailed compared to the MMSE-x, especially on the deblurring problem. This, combined with slightly lower metrics, justifies the use of the MMSE-x estimate for the main experiments.

<figure><img src="image_10.png" /><figcaption>Figure 9. Influence of hyperparameters $\lambda$ (regularization parameter) and $\alpha$ (bitrate parameter) on VBLE algorithm. To the left, on a deblurring problem on a BSD image, to the right, on a SISR problem on a FFHQ image.</figcaption></figure>

Additionally, a study on VBLE hyperparameters $\alpha$ (bitrate parameter) and $\lambda$ (regularization parameter) is provided in Fig. 9. Parameter $\lambda$ seems a more sensitive parameter than $\alpha$, hence a CAE with a single bitrate could solve a wide range of inverse problems. Furthermore, the prior on FFHQ seems to be more informative compared to the one on BSD, which makes sense as BSD dataset structure is much harder to learn.

# 5 Conclusion

In this paper, we have proposed the use of variational compressive autoencoders for regularizing imaging inverse problems. Additionally, we have introduced the VBLE algorithm which performs variational inference in the latent space of CAEs or VAEs. VBLE enables efficient approximation of the posterior distribution associated with image restoration tasks. In our experiments, compressive autoencoders, combined with VBLE, achieve state-of-the-art results and provide compelling posterior sampling abilities, outperforming other posterior sampling methods in terms of computational time and GPU load. These demonstrate the interest of CAEs for image restoration as well as the possibility of estimating simply and efficiently the posterior distribution in the latent space of a generative model.

Further work will be dedicated to enhancing the performance of VBLE, by considering more expressive variational distribution families or by a joint estimation in the latent and image space [34, 27]. Another key perspective is to consider modulated CAEs [61] which allow for compression at multiple bit-rates, thus enabling the regularization of inverse problems of various difficulties with a single neural network.

# Acknowledgments

This work was partly supported by CNES under project name DEEPREG, and ANITI under grant agreement ANR-19-PI3A-0004.

# References

[1] M. Asim, M. Daniels, O. Leong, A. Ahmed, and P. Hand. Invertible generative models for inverse problems: mitigating representation error and dataset bias. In *International Conference on Machine Learning (ICML)*, pages 399–409. PMLR, 2020.

[2] J. Ballé, V. Laparra, and E. P. Simoncelli. Density modeling of images using a generalized normalization transformation. In *International Conference on Learning Representations (ICLR)*, 2016.

[3] J. Ballé, V. Laparra, and E. P. Simoncelli. End-to-end optimized image compression. In *International Conference of Learning Representations (ICLR)*, 2017.
[4] J. Ballé, D. Minnen, S. Singh, S. J. Hwang, and N. Johnston. Variational image compression with a scale hyperprior. In *International Conference of Learning Representations (ICLR)*, 2018.

[5] Y. Blau and T. Michaeli. The perception-distortion trade-off. In *IEEE/CVF conference on computer vision and pattern recognition (CVPR)*, pages 6228–6237, 2018.

[6] D. M. Blei, A. Kucukelbir, and J. D. McAuliffe. Variational inference: A review for statisticians. *Journal of the American statistical Association*, 112(518):859–877, 2017.

[7] A. Bora, A. Jalal, E. Price, and A. G. Dimakis. Compressed sensing using generative models. In *International Conference on Machine Learning (ICML)*, 2017.

[8] A. C. Bovik. *Handbook of image and video processing*. Academic press, 2010.

[9] J. Bégaint, F. Racapé, S. Feltman, and A. Pushparaja. CompressAI: a PyTorch library and evaluation platform for end-to-end compression research. *arXiv preprint arXiv:2011.03029*, 2020.

[10] Z. Cai, J. Tang, S. Mukherjee, J. Li, C. B. Schönlieb, and X. Zhang. NF-ULA: Langevin Monte Carlo with Normalizing Flow Prior for Imaging Inverse Problems. *arXiv preprint arXiv:2304.08342*, 2023.

[11] J. Cao, Y. Shi, K. Zhang, Y. Zhang, R. Timofte, and L. Van Gool. Deep equilibrium diffusion restoration with parallel sampling. In *EEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 2824–2834, 2024.

[12] S. H. Chan, X. Wang, and O. A. Elgendy. Plug-and-play ADMM for image restoration: Fixed-point convergence and applications. *IEEE Transactions on Computational Imaging*, 3(1):84–98, 2016.

[13] Z. Chen, W. Guo, Y. Feng, Y. Li, C. Zhao, Y. Ren, and L. Shao. Deep-learned regularization and proximal operator for image compressive sensing. *IEEE Transactions on Image Processing (TIP)*, 30:7112–7126, 2021.

[14] J. Cheng, T. Liu, and S. Tan. Score priors guided deep variational inference for unsupervised real-world single image denoising. In *IEEE/CVF International Conference on Computer Vision (ICCV)*, pages 12937–12948, 2023.

[15] Z. Cheng, H. Sun, M. Takeuchi, and J. Katto. Learned image compression with discretized gaussian mixture likelihoods and attention modules. In *IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, 2020.

[16] J. Choi, S. Kim, Y. Jeong, Y. Gwon, and S. Yoon. ILVR: Conditioning method for denoising diffusion probabilistic models. In *IEEE/CVF International Conference on Computer Vision (ICCV)*, pages 14347–14356, 2021.

[17] H. Chung, J. Kim, M. T. Mccann, M. L. Klasky, and J. C. Ye. Diffusion posterior sampling for general noisy inverse problems. In *International Conference on Learning Representations (ICLR)*, 2023.

[18] F. Coeurdoux, N. Dobigeon, and P. Chainais. Plug-and-play split gibbs sampler: embedding deep generative priors in bayesian inference. *IEEE Transactions on Image Processing (TIP)*, 33:3496–3507, 2024.

[19] B. Dai and D. Wipf. Diagnosing and enhancing vae models. In *International Conference of Learning Representations*, 2019.

[20] G. Daras, A. Odena, H. Zhang, and A. G. Dimakis. Your local GAN: Designing two dimensional local attention mechanisms for generative models. In *IEEE/CVF conference on computer vision and pattern recognition (CVPR)*, pages 14531–14539, 2020.

[21] V. A. de Oliveira, M. Chabert, T. Oberlin, C. Poulliat, M. Bruno, C. Latry, M. Carlavan, S. Henrot, F. Falzon, and R. Camarero. Satellite image compression and denoising with neural networks. *IEEE Geoscience and Remote Sensing Letters*, 19:1–5, 2022.

[22] J. Dean, G. Daras, and A. Dimakis. Intermediate layer optimization for inverse problems using deep generative models. In *NeurIPS 2020 Workshop on Deep Learning and Inverse Problems*, 2020.

[23] M. Dhar, A. Grover, and S. Ermon. Modeling sparse deviations for compressed sensing using generative models. In *International Conference on Machine Learning (ICLR)*, pages 1214–1223. PMLR, 2018.

[24] P. Dhariwal and A. Nichol. Diffusion models beat GANs on image synthesis. *Advances in Neural Information Processing Systems (NeurIPS)*, 34:8780–8794, 2021.

[25] L. Dinh, D. Krueger, and Y. Bengio. NICE: Non-linear independent components estimation. In *International Conference of Learning Representations (ICLR)*, 2015.

[26] C. Dong, C. C. Loy, K. He, and X. Tang. Learning a deep convolutional network for image super-resolution. *European Conference on Computer Vision (ECCV)*, pages 184–199, 2014.

[27] M. Duff, N. D. F. Campbell, and M. J. Ehrhardt. Regularising inverse problems with generative machine learning models. *Journal of Mathematical Imaging and Vision*, 2023.

[28] M. A. Duff, I. J. Simpson, M. J. Ehrhardt, and N. D. Campbell. VAEs with structured image covariance applied to compressed sensing MRI. *Physics in Medicine & Biology*, 68(16):165008, 2023.

[29] A. Durmus, E. Moulines, and M. Pereyra. Efficient bayesian computation by proximal Markov chain Monte Carlo: when Langevin meets Moreau. *SIAM Journal on Imaging Sciences*, 11(1):473–506, 2018.

[30] M. Elad. *Sparse and Redundant Representations: From Theory to Applications in Signal and Image Processing*, volume 2. Springer, 2010.

[31] S. Gao and X. Zhuang. Bayesian image super-resolution with deep modeling of image statistics. *IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)*, 45(2):1405–1423, 2022.
[32] T. Garber and T. Tirer. Image restoration by denoising diffusion models with iteratively preconditioned guidance. In *IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 25245–25254, 2024.

[33] H. Goh, S. Sheriffdeen, J. Wittmer, and T. Bui-Thanh. Solving bayesian inverse problems via variational autoencoders. In *J. Bruna, J. Hesthaven, and L. Zdeborova, editors, Proceedings of the 2nd Mathematical and Scientific Machine Learning Conference*, volume 145 of *Proceedings of Machine Learning Research*, pages 386–425. PMLR, 16–19 Aug 2022.

[34] M. Gonzàlez, A. Almansa, and P. Tan. Solving inverse problems by joint posterior maximization with autoencoding prior. *SIAM Journal on Imaging Sciences*, 15(2):822–859, 2022.

[35] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. *Advances in neural information processing systems (NeurIPS)*, 30, 2017.

[36] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. *Advances in Neural Information Processing Systems (NeurIPS)*, 2020.

[37] M. Holden, M. Pereyra, and K. C. Zygalakis. Bayesian imaging with data-driven priors encoded by neural networks. *SIAM Journal on Imaging Sciences*, 15(2):892–924, 2022.

[38] S. Hurault, A. Leclaire, and N. Papadakis. Gradient step denoiser for convergent plug-and-play. In *International Conference on Learning Representations (ICLR)*, 2022.

[39] Z. Kadkhodaie and E. Simoncelli. Stochastic solutions for linear inverse problems using the prior implicit in a denoiser. *Neural Information Processing Systems (NeurIPS)*, 34:13242–13254, 2021.

[40] U. S. Kamilov, H. Mansour, and B. Wohlberg. A plug-and-play priors approach for solving nonlinear imaging inverse problems. *IEEE Signal Processing Letters*, 24(12):1872–1876, 2017.

[41] T. Karras, S. Laine, and T. Aila. A style-based generator architecture for generative adversarial networks. In *IEEE/CVF conference on computer vision and pattern recognition (CVPR)*, pages 4401–4410, 2019.

[42] B. Kawar, M. Elad, S. Ermon, and J. Song. Denoising diffusion restoration models. *Advances in Neural Information Processing Systems (NeurIPS)*, 35:23593–23606, 2022.

[43] D. P. Kingma and M. Welling. Auto-encoding variational bayes. In *International Conference on Learning Representations (ICLR)*, 2014.

[44] R. Laumont, V. D. Bortoli, A. Almansa, J. Delon, A. Durmus, and M. Pereyra. Bayesian Imaging using plug & play priors: when Langevin meets Tweedie. *SIAM Journal on Imaging Sciences*, 15(2):701–737, 2022.

[45] A. C. Likas and N. P. Galatsanos. A variational approach for bayesian blind image deconvolution. *IEEE transactions on signal processing*, 52(8):2222–2233, 2004.

[46] S. Mallat. *Wavelet tour of signal processing: The sparse way*. Elsevier, 1999.

[47] D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In *Proceedings eighth IEEE international conference on computer vision. ICCV 2001*, volume 2, pages 416–423. IEEE, 2001.

[48] S. Menon, A. Damian, S. Hu, N. Ravi, and C. Rudin. Pulse: Self-supervised photo upsampling via latent space exploration of generative models. In *IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 2437–2445, 2020.

[49] F. Mentzer, G. D. Toderici, M. Tschannen, and E. Agustsson. High-fidelity generative image compression. In *H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems (NeurIPS)*, volume 33, pages 11913–11924. Curran Associates, Inc., 2020.

[50] D. Minnen, J. Ballé, and G. D. Toderici. Joint autoregressive and hierarchical priors for learned image compression. In *Advances in neural information processing systems (NeurIPS)*, volume 31, 2018.

[51] A. Q. Nichol and P. Dhariwal. Improved denoising diffusion probabilistic models. In *International Conference on Machine Learning (ICML)*, pages 8162–8171. PMLR, 2021.

[52] T. Oberlin and M. Verm. Regularization via deep generative models: an analysis point of view. In *International Conference on Image Processing (ICIP)*, pages 404–408. IEEE, 2021.

[53] M. Pereyra, L. V. Mieles, and K. C. Zygalakis. Accelerating Proximal Markov Chain Monte Carlo by Using an Explicit Stabilized Method. *SIAM Journal on Imaging Sciences*, 13(2):905–935, 2020.

[54] M. Pereyra and J. Tachella. Equivariant bootstrapping for uncertainty quantification in imaging inverse problems. In *International Conference on Artificial Intelligence and Statistics*, pages 4141–4149. PMLR, 2024.

[55] J. Prost, A. Houdard, A. Almansa, and N. Papadakis. Inverse problem regularization with hierarchical variational autoencoders. In *IEEE International Conference on Computer Vision (ICCV)*, 2023.

[56] L. Rout, Y. Chen, A. Kumar, C. Caramanis, S. Shakkottai, and W.-S. Chu. Beyond first-order tweedie: Solving inverse problems using latent diffusion. In *IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 9472–9481, 2024.

[57] L. I. Rudin, S. Osher, and E. Fatemi. Nonlinear total variation based noise removal algorithms. *Physica D: Nonlinear Phenomena*, 1992.

[58] C. Saharia, J. Ho, W. Chan, T. Salimans, D. J. Fleet, and M. Norouzi. Image super-resolution via iterative refinement. *IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)*, 45(4):4713–4726, 2022.
[59] A. Skodras, C. Christopoulos, and T. Ebrahimi. The jpeg 2000 still image compression standard. *IEEE Signal processing magazine*, 18(5):36–58, 2001.

[60] C. K. Soenderby, T. Raiko, L. Maalœ, S. K. Soenderby, and O. Winther. Ladder variational autoencoders. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, *Advances in Neural Information Processing Systems (NeurIPS)*, volume 29. Curran Associates, Inc., 2016.

[61] M. Song, J. Choi, and B. Han. Variable-rate deep image compression through spatially-adaptive feature transform. In *IEEE/CVF International Conference on Computer Vision (ICCV)*, pages 2380–2389, 2021.

[62] K. C. Tezcan, N. Karani, C. F. Baumgartner, and E. Konukoglu. Sampling possible reconstructions of undersampled acquisitions in mr imaging with a deep learned prior. *IEEE Transactions on Medical Imaging*, 41(7):1885–1896, 2022.

[63] R. Tibshirani. Regression shrinkage and selection via the lasso. *Journal of the Royal Statistical Society: Series B (Methodological)*, 1996.

[64] A. Tikhonov. Solution of incorrectly formulated problems and the regularization method. *Soviet Math.*, 1963.

[65] F. Tonolini, B. S. Jensen, and R. Murray-Smith. Variational sparse coding. In *Uncertainty in Artificial Intelligence*, pages 690–700. PMLR, 2020.

[66] S. V. Venkatakrishnan, C. A. Bouman, and B. Wohlberg. Plug-and-play priors for model based reconstruction. *IEEE Global Conference on Signal and Information Processing (GlobalSIP)*, 2013.

[67] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image quality assessment: from error visibility to structural similarity. *IEEE Transactions on Image Processing (TIP)*, 13(4):600–612, 2004.

[68] P. A. Wintz. Transform picture coding. *Proceedings of the IEEE*, 60(7):809–820, 1972.

[69] Z. Yue, H. Yong, Q. Zhao, L. Zhang, D. Meng, and K.-Y. K. Wong. Deep variational network toward blind image restoration. *IEEE Transactions on Pattern Analysis and Machine Intelligence*, 2024.

[70] K. Zhang, Y. Li, W. Zuo, L. Zhang, L. Van Gool, and R. Timofte. Plug-and-play image restoration with deep denoiser prior. *IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)*, 44(10):6360–6376, 2021.

[71] K. Zhang, W. Zuo, S. Gu, and L. Zhang. Learning deep CNN denoiser prior for image restoration. In *IEEE conference on computer vision and pattern recognition (CVPR)*, pages 3929–3938, 2017.

[72] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as a perceptual metric. In *IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 586–595, 2018.

[73] Y. Zhu, K. Zhang, J. Liang, J. Cao, B. Wen, R. Timofte, and L. Van Gool. Denoising diffusion models for plug-and-play image restoration. In *IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 1219–1229, 2023.
The following Tab. 6 and Tab. 7 list all the parameters used in the experiments.

<table><caption>Table 6. Detailed experimental settings for VBLE and the baselines.</caption><thead><tr><th>Method</th><th>Optimized parameters</th><th>Fixed parameters</th></tr></thead><tbody><tr><td>VBLE wCAE</td><td>- λ (regularization)<br/>- α (bitrate)</td><td>- 1000 iterations</td></tr><tr><td>MAPz wCAE</td><td>- λ (regularization)<br/>- α (bitrate)</td><td>- 1000 iterations</td></tr><tr><td>DPIR</td><td>- λ (regularization)<br/>- number of iterations</td><td>x</td></tr><tr><td>PnP-ADMM</td><td>- λ (regularization)<br/>- number of iterations<br/>- denoiser noise</td><td>x</td></tr><tr><td>PnP-ULA</td><td>- λ (regularization)<br/>- denoiser noise<br/>- step size of the Markov Chain</td><td>- 10<sup>5</sup> iterations<br/>- Thinning 200<br/>- Burn-in 0.8</td></tr><tr><td>DiffPIR</td><td>- λ (regularization)<br/>- ζ</td><td>- 100 NFEs</td></tr><tr><td>DDRM</td><td>x</td><td>- η = 0.85<br/>- η<sub>b</sub> = 1<br/>- 100 NFEs</td></tr><tr><td>PULSE</td><td>x</td><td>- 300 iterations</td></tr><tr><td>MAPz wVAE<br/>(Bora et al.)</td><td>- λ (regularization)</td><td>- 500 iterations</td></tr></tbody></table>

<table><caption>Table 7. Detailed parameter setting for VBLE using cheng and mbt compressive networks, for all experiments in the paper. λ is the regularization parameter and α the bitrate parameter. Specifically, each CAE is trained with the loss $\mathcal{L} = 255^2 \alpha \text{Distortion} + \text{Rate}$. For mbt and cheng models, two structures, a light and a bigger one, exist [9], which are used respectively for low and high bitrates. For mbt, high bitrate structure is used for $\alpha \ge 0.013$, and low bitrate structure for $\alpha \le 0.0067$. For cheng, we have chosen the high bitrate structure for high and low bitrates, as it provides the best image restoration results.</caption><thead><tr><th rowspan="2">BSD</th><th rowspan="2"></th><th colspan="3">Deblur σ = 2.55/255</th><th colspan="3">Deblur σ = 7.65/255</th><th colspan="2">SISR</th><th rowspan="2">Inpainting<br/>50%</th></tr><tr><th>σ<sub>k</sub> = 1</th><th>σ<sub>k</sub> = 3</th><th>Motion</th><th>σ<sub>k</sub> = 1</th><th>σ<sub>k</sub> = 3</th><th>Motion</th><th>×2</th><th>×4</th></tr></thead><tbody><tr><td rowspan="2">cheng</td><td>α</td><td>0.0932</td><td>0.0067</td><td>0.0483</td><td>0.0067</td><td>0.0035</td><td>0.0067</td><td>0.025</td><td>0.0035</td><td>0.1800</td></tr><tr><td>λ</td><td>0.7</td><td>1.4</td><td>0.7</td><td>0.7</td><td>0.7</td><td>0.7</td><td>3</td><td>2.5</td><td>10</td></tr><tr><td rowspan="2">mbt</td><td>α</td><td>0.0483</td><td>0.0067</td><td>0.025</td><td>0.0067</td><td>0.0035</td><td>0.0067</td><td>0.025</td><td>0.0067</td><td>0.0932</td></tr><tr><td>λ</td><td>0.7</td><td>0.8</td><td>0.6</td><td>0.7</td><td>0.6</td><td>0.6</td><td>2.3</td><td>3.4</td><td>4.5</td></tr></tbody></table><table><thead><tr><th rowspan="2">FFHQ</th><th rowspan="2"></th><th colspan="3">Deblur σ = 2.55/255</th><th colspan="3">Deblur σ = 7.65/255</th><th colspan="2">SISR</th><th rowspan="2">Inpainting<br/>50%</th></tr><tr><th>σ<sub>k</sub> = 1</th><th>σ<sub>k</sub> = 3</th><th>Motion</th><th>σ<sub>k</sub> = 1</th><th>σ<sub>k</sub> = 3</th><th>Motion</th><th>×2</th><th>×4</th></tr></thead><tbody><tr><td rowspan="2">cheng</td><td>α</td><td>0.1800</td><td>0.0067</td><td>0.0932</td><td>0.025</td><td>0.0035</td><td>0.025</td><td>0.025</td><td>0.0013</td><td>0.1800</td></tr><tr><td>λ</td><td>0.8</td><td>2.7</td><td>1.0</td><td>1.0</td><td>1.8</td><td>0.9</td><td>4.0</td><td>4.6</td><td>2.0</td></tr><tr><td rowspan="2">mbt</td><td>α</td><td>0.0932</td><td>0.0067</td><td>0.0483</td><td>0.013</td><td>0.0035</td><td>0.0067</td><td>0.0483</td><td>0.013</td><td>0.0932</td></tr><tr><td>λ</td><td>1.0</td><td>0.6</td><td>0.9</td><td>1.0</td><td>0.9</td><td>0.8</td><td>3.2</td><td>1.8</td><td>2.0</td></tr></tbody></table>