From Posterior Sampling to Meaningful Diversity in Image Restoration
Authors: Noa Cohen
Hila Manor
Yuval Bahat
Tomer Michaeli
Venue: ICLR 2024 poster (2024)
Keywords: Image restoration
Diverse image restoration
Inverse problems
Posterior sampling
Meaningful diversity
Primary area: generative models
TL;DR: We highlight the inherent conceptual problem in posterior sampling for image restoration. We examine the source of the problem and propose different ways to obtain more meaningfully diverse restorations, as confirmed by an extensive user study.

Abstract: Image restoration problems are typically ill-posed in the sense that each degraded image can be restored in infinitely many valid ways. To accommodate this, many works generate a diverse set of outputs by attempting to randomly sample from the posterior distribution of natural images given the degraded input. Here we argue that this strategy is commonly of limited practical value because of the heavy tail of the posterior distribution. Consider for example inpainting a missing region of the sky in an image. Since there is a high probability that the missing region contains no object but clouds, any set of samples from the posterior would be entirely dominated by (practically identical) completions of sky. However, arguably, presenting users with only one clear sky completion, along with several alternative solutions such as airships, birds, and balloons, would better outline the set of possibilities. In this paper, we initiate the study of **meaningfully diverse** image restoration. We explore several post-processing approaches that can be combined with any diverse image restoration method to yield semantically meaningful diversity. Moreover, we propose a practical approach for allowing diffusion based image restoration methods to generate meaningfully diverse outputs, while incurring only negligent computational overhead. We conduct extensive user studies to analyze the proposed techniques, and find the strategy of reducing similarity between outputs to be significantly favorable over posterior sampling. Code and examples are available on the [project's webpage](https://noa-cohen.github.io/MeaningfulDiversityInIR/).

Published as a conference paper at ICLR 2024
FROM POSTERIOR SAMPLING TO MEANINGFUL DIVERSITY IN IMAGE RESTORATION
Noa Cohen
Hila Manor
Technion - Israel Institute of Technology noa . cwhen@ campus technion ac.
Technion Israel Institute of Technology hi lla manor @campus techni on. ac.
Yuval Bahat
Tomer Michaeli
Princeton University
Technion Israel Institute of Technology tomer m@ee. technion. ac ac.il
yuval baall.baattee @araaat@mmai... com
ABSTRACT
Image restoration problems are typically ill-posed in the sense that each degraded image can be restored in infinitely many valid ways. To accommodate this, many works generate a diverse set of outputs by attempting to randomly sample from the posterior distribution of natural images given the degraded input. Here ar- gue that this strategy is commonly of limited practical value because of the heavy tail of the posterior distribution. Consider for example inpainting missing region of the sky in an image. Since there is a high probability that the missing region contains no object but clouds, any tirely dominated by (practically identical) completions of sky. However, arguably, presenting users with only one clear sky completion, along with several alterna- tive solutions such as airships. birds, and balloons, would better outline the set of possibilities. In this paper, we initiate the study of meaningfully diverse image restoration. We explore several post-processing approaches that can be combined with any diverse image restoration method to yield semantically meaningful di- versity. Moreover, we propose practical approach for allowing diffusion based image restoration methods to generate meaningfully diverse outputs, while incur- ring only negligent computational overhead. We conduct extensive user studies to analyze the proposed techniques, and find the strategy of reducing similarity between outputs be significantly favorable over posterior sampling. Code and examples are available on the project's 's webpage.
samples from the posterior would be en-
INTRODUCTION
Image restoration a collective name for tasks in which a corrupted or low resolution image is restored into better quality one. Example tasks include image inpainting, super-resolution, com- pression artifact reduction and denoising. Common to most image restoration problems is their ill-posed nature, which causes each degraded image have infinitely many valid restoration solu- tions. Depending on the severity of the degradation, these solutions may differ significantly, and often correspond diverse semantic meanings (Bahat & Michaeli, 2020).
In the past, image restoration methods were commonly designed to output a single solution for each degraded input (Haris al., 2018: Kupyn et al., 2019; Liang al., 2021; Pathak et al., 2016; Wang et al., 2018; Zhang et al., 2017). In recent years, however, growing research effort is devoted to methods that can produce a range of different valid solutions for every degraded input, including in super-resolution (Bahat & Michaeli, 2020; Kawar et al., 2022; Lugmayr et al., 2022b), inpaint- ing (Hong et al., 2019; Liu et al., 2021; Song et al., 2023), colorization (Saharia et al., 2022: Wang et al., 2022; Wu et al., 2021), and denoising (Kawar al., 202 lb; 2022; Ohayon al., 2021). Broadly speaking, these methods strive to generate samples from the posterior distribution Pxiy of high-quality images X given the degraded input image Y. Diverse restoration can then be achieved by repeatedly sampling from this posterior distribution. To allow this, significant research effort is devoted into approximating the posterior distribution, e.g., using Generative Adversarial Networks

---

Published as a conference paper at ICLR 2024
Degraded1 Image
Vanilla Posterior Sampling
Alternative, Meaningfully Diverse Sampling
Figure I: Approximate posterior sampling vs. meaningfully diverse sampling in image restora- tion. Restoration generative models aiming to sample from the posterior tend to generate images that highly resemble one another semantically (left). In contrast, the meaningful plausible solutions on the right convey a broader range of restoration possibilities. Such sets of restorations achieved using the FPS approach explored in Sec. 4.
(GANs) (Hong et al., 2019; Ohayon al., 2021), auto-regressive models (Li et al., 2022; Wan al., 2021), invertible models (Lugmayr et al., 2020), energy-based models (Kawar et al., 2021a; Nijkamp et al., 2019), or more recently, denoising diffusion models (Kawar al., 2022; Wang et al., 2022).
In this work, we question whether sampling from the posterior distribution is the optimal strategy for achieving meaningful solution diversity in image restoration. Consider, for example, the task of inpainting patch in the sky like the one depicted in the third row of Fig. 1. In this case, the posterior distribution would be entirely dominated by patches partly cloudy sky. Repeatedly sampling patches from this distribution would, with very high probability, yield interchangeable results that appear as reproductions for any human observer. Locating a notably different result would therefore involve exhausting, oftentimes prohibitívely long, re-sampling sequence. In contrast, argue that presenting a set of alternative completions depicting an airship, balloons, or even a parachutist would better convey the actual possible diversity to a user.
Here we initiate the study of meaningfully diverse image restoration, which aims at reflecting to a user the perceptual range of plausible solutions rather than adhering to their likelihood. We start by analyzing the nature of the posterior distribution, as estimated by existing diverse image restoration models, in the tasks of inpainting and super-resolution. We show both qualitatively and quantita- tively that this posterior is quite often heavy tailed. As we illustrate, this implies that if the number of images presented to a user is restricted to e.g., 5, then with very high probability this set is not going to be representative. Namely, it will typically exhibit low diversity and will not span the range of possible semantics. We then move on to explore several baseline techniques for sub-sampling a large set of solutions produced by a posterior sampler, as to present users with small diverse set of plausible restorations. Finally, we propose a practical approach that can endow diffusion based image restoration models with the ability produce small diverse sets. The findings of our analysis (both qualitatively and via a user study) suggest that techniques that explicitly seek to max- imize distances between the presented images, whether by modifying the generation process in post-processing, are significantly advantageous over random sampling from the posterior.
2 RELATED WORK
Approximate posterior sampling. Recent years have seen shift from the one-to-one restoration paradigm to diverse image restoration. Methods that generate diverse solutions are based on various approaches, including VAES (Peng et al., 2021: 1; Prakash et al., 2021; Zheng et al., 2019), GANs (Cai & Wei, 2020; Liu et al., 2021: Zhao et al., 2020; 2021), normalizing flows (Helminger et al., 2021; Lugmayr et al., 2020) and diffusion models (Kawar et al., 2022; Lugmayr et al., 2022a; Wang et al., 2022). Common to these methods is that they aim for sampling from the posterior distribution of natural images given the degraded input. While this generates some diversity, in many cases the vast majority of samples produced this way have the same semantic meanings.
2

---

Given an input image y that is a degraded version of some high-quality image x, our goal is to com- pose set of N outputs X' {z .... N) such that each x' constitutes a plausible reconstruction of x, while X' as a whole reflects the diversity of possible reconstructions in a meaningful manner. Published as a conference paper at ICLR 2024
Enhancing perceptual coverage. Several works increase sample diversity in unconditional gen- eration, e.g., by pushing towards higher coverage of low density regions (Sehwag et al., 2022; Yu et al., 2020). For conditional generation, previous works attempled to battle the effect of the heavy- tailed nature of visual data (Sehwag et al., 2022) by encouraging exploration of the sample space during training (Mao et al., 2019). As show, the approximated posterior of restoration models exhibits a similar heavy-tailed nature. For linear inverse problems, diversity can be increased, e,g., by using geometric-based methods to traverse the latent space (Montanaro al., 2022). However, these works do not improve on the redundancy when simultaneously sampling a batch from the heavy-tailed distribution (see e.g., Fig. ((d) in Sehwag et al. (2022), which depicts two pairs of very similar images within a set of 12 unconditional image samples). Our work is the first to explore ways to produce a representative set of meaningfully diverse solutions.
Interactive exploration of solutions. Another approach for conveying the range of plausible restorations is to hand over the reins to the user, by developing controllable methods. These methods allow the user to explore the space of possible restoration by various means, including graphical user interface tools (Bahat & Michaeli, 2020; 2021: Weber et al., 2020), editing of semantic maps (Buh- ler et al., 2020), manipulation some latent space (Lugmayr et al., 2020; Wang al., 2019), and via textual prompts describing a desired output (Bai et al., 2023; Chen et al., 2018; Ma et al., 2022; Zhang et al., 2020). These approaches are mainly suitable for editing applications, where the user has some end-goal in mind, and are also time consuming and require skill to obtain a desired result.
Uncertainty quantification. Rather than generating a diverse set of solutions, several methods present to the user single prediction along with some visualization of the uncertainty around that prediction. These visualizations include heatmaps depicting per-pixel confidence-levele (An- gelopoulos al., 2022; Lee Chung, 2019), as well as upper and lower bounds (Horwitz & Hoshen, 2022: Sankaranarayanan et al., 2022) that span the set of possibilities with high probability, either in pixel space or semantically along latent directions. However, per-pixel maps tend to convey lit- tle information about semantics, and latent space analyses require a generative model in which all attributes of interest are perfectly disentangled (a property rarely satisfied in practice).
LIMITATIONS OF POSTERIOR SAMPLING
When sampling multiple times from diverse restoration models, the samples tend repeat them- selves, exhibiting only minor semantic variability. This is illustrated in Fig. 2, which depicts two masked images with corresponding 10 random samples each, obtained from RePaint (Lugmayr et al., 2022a), a diverse inpainting method. As can be seen, none of the 10 completions corresponding to the eye region depict glasses, and none of the 10 samples corresponding to the mouth region depict a closed mouth. Yet, when examining 100 samples from the model, it is evident that such completions are possible; they are simply rare (2 out of 100 samples). This behavior also seen in Figs. and 5.
We argue that this phenomenon stems from the fact that the posterior distribution is often heavy- tailed along semantically interesting directions. Heavy-tailed distributions assign a non-negligible probability to distinct "outliers". In the context of image restoration, these outliers often correspond to different semantic meanings. This effect can be seen on the right pane of Fig. 2, which depicts the histogram of the projections of the 100 posterior samples onto their first principal component.
A quantitative measure of the tailedness of a distribution Px with mean μ and variance a², is its kur- tosis, E Ex-rx [((X - /1)/a) ).) The normal distribution family has kurtosis of 3, and distributions with kurtosis larger than 3 are heavy tailed. As can be seen, both posterior distributions in Fig. have very high kurtosis values. As we show in Fig. 3, cases in which the posterior heavy tailed are not rare. For roughly 12% of the inspected masked face images, the estimated kurtosis value of the restorations obtained with RePaint was greater than 5, while only about 0.12% of Gaussian distributions over space with the same dimension are likely to reach this value.
4 WHAT MAKES A SET OF RECONSTRUCTIONS MEANINGFULI DIVERSE?
By "meaningful" we mean that rather than adhering to the posterior distribution of x given y, we
3

---

Published as a conference paper at ICLR 2024
Kurtosis=25.43
Kurtosis=10.36
Figure 2: Histograms of the projections of features from two collections of posterior samples onto their first principal component. Each collection contains 100 reconstructions of an inpainted image. In the upper example PCA was applied on pixel space, and in the lower example features of an attribute predictor. The high distribution kurtosis marked on the graphs are due to rare, yet non negligible points distant from the mean. We fit a mixture of 2 Gaussians to each distribution and plot the dominant Gaussian, to allow visual comparison of the tail.
deep
Pixel space
Feature space
Images
Images
Gaussians
Gaussians
10"
10
10a
10
10
Kurtosis
Kurtosis
Figure 3: Statisties of kurtoses in posterior distributions. We calculate kurtoses values for projec- tions of features from 54 collections of posterior samples, 100 samples each, onto their first principal component (orange). Left pane shows restored pixel values features, while the right pane shows feature activations extracted from an attribute predictor. For comparison, we also show statistics of kurtoses of 800 multivariate Gaussians with the same dimensions (blue), each estimated from 100 samples. The non-negligible occurrence of very high kurtoses in images (compared with their Gaus- sian equivalents) indicates their heavy tailed distributions. Whiskers mark the (5, 95] percentiles.
want A' cover the perceptual range of plausible reconstructions of to the maximal extent pos- sible (depending on N). In practical applications, we would want N to be small (e.g., 5) to avoid the need of tedious scrolling through many restorations. Our goal in this section is examine what mathematically characterizes meaningfully diverse set of solutions. We do not attempt to devise a practical method yet, task which defer Sec. but rather only to understand the principles that should guide one in the pursuit of such method. To do so, we explore three approaches for choosing the samples include the representative set X' from a larger set of solutions X' {i ... .5 Ñ > N, generated by some diverse image restoration method. We illustrate the approaches quali- tatively and measure their effectiveness user studies. We note that a set of samples can either be presented to user all at once or via hierarchical structure (see App. G).
Given a degraded input image y, we start by generating a large set of solutions X using a diverse image restoration method. We then extract perceptually meaningful features for all N images in X and use the distances between these features as proxy to the perceptual dissimilarity between the images. In each of the three approaches we consider, order to sub-sample X into X', exploring different concepts of diversity. As a running example, illustrate the approaches on the 2D distribution shown in Fig. 4, and inpainting and super- resolution, as shown in Fig. (see details in Sec. 4.1). Note that a small random sample from the distribution of Fig. 4 (second pane) is likely to include only points from the dominant mode, and thus does not convey to a viewer the existence of other modes. We consider the following approaches.
these distances in a different way in
Cluster representatives A straightforward way to represent the different semantic modes in A via clustering. Specifically, we apply the K-means algorithm over the feature representations of all images in X, setting the number of clusters K the desired number of solutions, N. We then
4

---

Published as a conference paper at ICLR 2024
Large Sample
Small Representative Set
K-Means Uniformization
Farthest Point Strategy
37-
Figure 4: Methods for choosing a small representative set. We compare three baseline approaches for meaningfully representing set X' of Ñ 1000 red points drawn from an imbalanced mixture of 10 Gaussians (left), by using a subset X' of only N = 20 points. Note how the presented approaches differ in their abilities cover sparse and dense regions of the original set X. In this example, X' is dominated by the central Gaussian which contains 95% of the probability mass.
construct X' by choosing for each of the N clusters the image in X closest to its center in feature space. As seen in Figs. 4 and 5, this approach leads to a more diverse set than random sampling. However, the set can be redundant, multiple points may originate from the dominant mode.
Uniform coverage of the posterior's effective support In theory, one could go about our goal of covering the perceptual range plausible reconstructions by sampling uniformly from the effective support of the posterior distribution Px|y over semantic feature space. This technique boils down to increasing the relative probability of sampling less likely solutions at the expense of decreasing the chances of repeatedly sampling the most likely ones, and we therefore refer to it as Uniformization. This can be done by assigning to each member of probability mass that is inversely proportional to the density of the posterior at that point, and populating X' by sampling from X' without repetition according to these probabilities. Please refer to App. A for detailed description of this approach. As seen Figs. and 5, inherent limitation of this approach is that may under-represent high- probability modes their effective support is small. For example in Fig. 4, although Uniformization leads to a diverse set, this set does not contain a single representative from the dominant mode.
Distant representatives The third approach explore aims sample far as possible from one another in feature space, and relies on the Farthest Point Strategy (FPS), originally proposed for progressive image sampling (Eldar al., 1997). The first image in this approach is sampled randomly from X', With high probability, we can expect it to come from a dense area in feature space and thus to represents the most prevalent semantics the set X. The remaining N-1 images are then added in an iterative manner, each time choosing the image in A' that is farthest away from the set constructed thus far. Note that here we do not aim obtain a uniform coverage, but rather to sample subset that maximizes the pairwise distances in some semantically meaningful feature space. This approach thus explicitly pushes towards semantic variability. Contrary to the previous approaches, the distribution of the samples obtained from FPS highly depends on the size of the set from which we sample. The larger is, the greater the probability that the set contains extremely solutions. In FPS, these very rare solutions are likely to be chosen first. To control the probability of choosing improbable samples, FPS can be applied to a random subset of L S N images from X. As can be seen in Figs. 4 and 5, FPS chooses a diverse hand covers all modes of the distribution (contrary to Uniformization) and on the other hand is not redundant (in contrast to K -emans).. Here we used Ñ (please see the effect of L in App. B).
of images that are as
of samples that one
4.1 QUALITATIVE ASSESSMENT AND USER STUDIES
To assess the ability of each of the approaches discussed above to achieve meaningful diversity, we perform a qualitative evaluation and conduct a comprehensive user study. We experiment with two image restoration tasks: inpainting and noisy 16x super-resolution with a bicubic down- sampling kernel and a noise level of 0.05. We analyze them in two domains: face images from the CelebAMask-HHl dataset (Lee et al., 2020) and natural images from the PartImagenet dataset (He et al., 2022). We use RePaint (Lugmayr et al., 2022a) and DDRM (Kawar et al., 2022) as our base diverse restoration models for inpainting and super-resolution. respectively. For faces, features of the AnyCost attribute predictor (Lin et al., 2021), which was trained to identify a range of facial features such smile, hair color and use of lipstick, well as accessories such glasses.
deep
5

---

Published as a conference paper at ICLR 2024
Degraded Image
M-Means K-Means/Posterior Posterior
UniÉ
FPS
Figure 5: Diversely sampling image restorations. Using five images to represent sets of 100 restorations corresponding degraded images (shown above), images from CelebAMask-HO (left) and PartImagenet (Right). The posterior subset (first row) is comprised of randomly drawn restoration solutions, while subsequent rows are constructed using the explored baselines.
Super Resolution
Inpainting
Diversity
Coverage
Diversity
Coverage
67%
60%
55%
58%
55%
63%
61%
68%
65%
33%
50%
39%
63%
59%
...pp..
Pos.... .......
K-means
K-means
Posterior
K-means
I
K-means
Postetior
FPS
Unit.
FPS
Unif.
FPS
Unif.
FPP
Unif.
Figure 6: Human perceived diversity and coverage of likely solutions. For each domain,
report the percentage of users perceiving higher diversity in the explored sampling approaches com- pared sampling from the approximate posterior, and the percentage of users perceiving sufficient coverage by any of the sampling approaches (including vanilla sampling from the posterior). We use bootstrapping for calculating confidence intervals.
We reduce the dimensions of those features to 25 using PCA, and use L as the distance metric. For PartImagenet,
deep features from VGG-16 (Simonyan & Zisserman, 2015) directly and via the LPIPS metric (Zhang et al., 2018). For face inpainting we define four varied possible masks, construct masks using PartImagenet segments. In all experiments, we use
and for PartImagenet an initial set X' of N = 100 images generated from the model, and compose a set X' of N = representatives (see App. for more details).
Figures 1 and 5 (as well as the additional figures in App. I) show several qualitative results. In all those cases, the semantic diversity in random of images sampled from the posterior very low. This is while the FPS and Uniformization approaches manage to compose more meaningfully diverse sets that better cover the range possible solutions, e.g., by inpainting different objects or portraying diverse face expressions (Fig, 5). These approaches automatically pick such restorations, which exist among the 100 samples in X, despite being rare.
We conducted user studies through Amazon Mechanical Turk (AMT) on both the inpainting and the supertiesolution tasks, using 50 randomly selected face images from the CelebA.Mask-HG HQQ dataset per task. AMT users asked answer sequence of 50 questions, after completing short tutorial comprising two practice questions with feedback. To evaluate whether subset constitutes meaningfully diverse representation of the possible restorations for degraded image, study

---

Published as a conference paper at ICLR 2024
Degraded Image
Vanilla Generation
Diversity- Guided Generation
DDNN
RePaint
DDRM
DES
Figure 7: Generating diverse image restorations. Qualitative comparison of 5 restorations corre- sponding degraded images (center) generated by the models specified the left, without (left) and with (right) diversity guidance.
comprised two types of tests (for both restoration tasks). The first is paired diversity test, in which users were shown a set of five images sampled randomly from the approximate posterior against five images sampled using one of the explored approaches, and were asked to pick the more diverse set. The second is an unpaired coverage test, in which we generated an additional (101") solution used as a target image, and showed users set of five images sampled using one of the four approaches. The users had to answer whether it includes at least one image very-similar to the target.
The results for both tests are reported in Fig. 6. As can be seen on the left pane, the diversity of preferred significantly less times than the diversity of any of
approximate posterior sampling the other proposed approaches. Among the three studied approaches, FPS was considered the most diverse. The results on the right pane suggest that all approaches, with the exception of Uniformiza- tion in inpainting, yield similar coverage for likely solutions, with a perceived similar image in approximately 60% of the times. This means that the ability of the other two approaches (especially that FPS) to yield meaningful diversity does not come at the expense of covering the likely solu- tions, compared with approximate posterior sampling (which by definition tends to present the more likely restoration solutions). In contrast, coverage by the Uniformization approach is found to be low, which aligns with the qualitative observation from Fig. 4.
Overall, the results from the two human perception tests confirm that, for the purpose of composing meaningfully diverse subset of restorations, the FPS approach has a clear advantage over the K-means and Uniformization alternatives, and an even clearer ad advantage over randomly sampling from the posterior. While introducing small drop in covering of the peak of the heavy- distribution, it shows significant advantage in terms of presenting additional semantically diverse plausible restorations. Please refer to App. E for more details on the user studies.
A PRACTICAL METHOD FOR GENERATING MEANINGFUL DIVERSITY
Equipped with the insights from Sec. 4, we now turn to propose a practical method for generating a set of meaningfully diverse image restorations. We focus on restoration techniques that are based on diffusion models, as they achieve state-of-the-art results. Diffusion models generate samples by attempting to reverse a diffusion process defined over timesteps / E (0..... 7}. Specifically, their sampling process starts timestep = 7 and gradually progresses until reaching = 0. in which the final sample is obtained. Since these models involve long iterative process. using them to sample N reconstructions in order to eventually keep only N < N images, commonly impractical. However, we that good sampling strategy is one that strives reduce similarity between samples. diffusion models, such an effect can be achieved using guidance mechanisms.
Specifically, we run the diffusion process to simultancously generate N images, all conditioned on the same input y but driven by different noise samples. In each timestep t within the generation process, diffusion models produce estimate of the clean image. Let X, -.. (i))) be the
7

---

Published as a conference paper at ICLR 2024
Diversity
Quality
Super Resolution
70%
H
Super Resolation
63%
Inpainting
60%
+
Inpainting 46%
Figure 8: Human perceived diversity and quality. We report on the left and right panes the per- centage of users perceiving higher diversity and quality, respectively, in our diversity- genera- tion process compared to the vanilla process. We calculate confidence intervals using bootstrapping.
set of N predictions of clean images at time step t, one for each image in the batch. We aim for No to be the final restorations presented to the user (equivalent to of Sec. 4), and therefore aim to reduce the similarities between the images in X, at every timestep t. To achieve this, we follow the approach of Dhariwal & Nichol (2021), ), and add to each clean image prediction i E Xr the gradient of loss function that captures the dissimilarity between Loje and its nearest neighbor within the set, golt arg minl,e (* d( d( 2), where d(-,-) is dissimilarity measure. In particular, modify each prediction as
.NNNN
E Vel ( 2ONN)
+b k 4/12 + "
@ooee
where 77 is a step size that controls the guidance strength and the gradient is with respect to the first argument of d((,,) -). The factor t/T reduces the guidance strength throughout the diffusion process.
In practice, we found the squared L distance in pixel space to work quite well as a dissimilarity measure. However, to avoid pushing samples away from each other when they are already far apart, we clamp the distance to some upper bound D. Specifically, let S be the number of unknowns in our inverse problem (i.e., the number of elements in the image x: for super-resolution and the number of masked elements in inpainting). Then, we take our dissimilarity metric to be d(u, v) = 1 Illu w/l || vll S SD and d(u, v) = 1/2 S D2 if ||u - vll SD. The parameter D controls the minimal distance from which we do not apply a guidance step (i.e., the distance from which the predictions are considered dissimilar enough). Substituting this distance metric into (1), leads to our update step
t + S11 + "T (sh): =).NN)
|
< <SD).
where If-} is the indicator function.
6 EXPERIMENTS
We now evaluate the effectiveness of our guidance approach in enhancing meaningful diversity. We focus on four diffusion- based restoration methods that attempt to draw samples from the pos- terior: RePaint (Lugmayr et al., 2022a), DDRM (Kawar et al., 2022), DDNM (Wang et al., 2022), and DPS (Chung al., 2023). For each of them, we compare the restorations generated by the vanilla method to those obtained with our guidance, using the same noise samples for a fair compar- ison. We experiment with inpainting and super-resolution as our restoration tasks, using the same datasets, images, and masks (where applicable) as in Sec. 4.1 (for results on image colorization see App. I.1). In addition to the task of noisy 16x super-resolution on CelebAMask-HHi 4x super-resolution on PartImagenet, as well as noiseless super-resolution on both datasets. We also experimented with the methods of (Choi et al., 2021; Wei et al., 2022) for noiseless super-resolution, but found their consistency to be poor (LR-PSNK < 45dB) and thus discarded them (see App. C). In all our experiments, we use N 5 representatives to compose the set X. We conduct quantitative comparisons, well user studies. Qualitative results are shown in Fig. 7 and in App. I.
we add noisy
Human perception tests. As in Sec. 4.1, we conducted paired diversity test to compare between the vanilla generation and the diiersity-guiders generation. However, in contrast to Sec. 4.1, which the restorations in A' were chosen among model outputs, here we intervene in the generation process. We therefore also examined whether this causes decrease in image quality, by conducting a paired image quality test in which users were asked to choose which one of two images has higher quality: a vanilla restoration or a guided one. The studies share the base configuration used in Sec. 4.1, using RePaint for inpainting and DDRM for super-resolution both images from the CelebAMask -HQ dataset (see App. E for additional details). As seen in Fig. 8, the guided restorations were chosen as more diverse significantly more often, while their quality was perceived as at least comparable.
8

---

Published as a conference paper at ICLR 2024
Table I: Quantitative results on CelebAMask-HO in noisy (left) and noiseless (right) 16x super
resolution. For each method we report results of both vanilla sampling and sampling with guidance.
0.05
Model
LPIPS Div. (1)
NIQE (1)
LR-PSNR
LPIPS Div. (t)
NIQE (1)
LR-PSNR (1)
DDRM
0.19
8.47
31.59
0.18
8.30
54.69
Guidance
0.25
7.85
31.00
0.24
7.54
53.82
DDNM
N/A
N/A
N/A
0.18
7.40
81.24
Guidance
N/A
N/A
N/A
0.26
6.92
75.04
DPS
0.29
5.72
30.00
0.25
5.41
$2.05
Guidance
0.34
5.16
28.98
0.28
5.05
53.45
Table 2: Quantitative results on PartImageNee in noisy (left) and noiseless (right) 4x super resolu- tion. For each method we report results of both vanilla sampling and sampling with guidance.
0.05
Model
LPIPS Div. (1)
NIQE (L)
LR-PSNR
LPIPS Div. (t)
NIQE (L)
LR-PSNR (1)
DDRM
0.15
10.16
32.86
0.09
8.93
55.36
Guidance
0.18
9,06
32.80
0.12
8.18
55.12
DDNM
N/A
N/A
N/A
0.10
9.63
71.40
Guidance
N/A
N/A
N/A
0.16
8.52
69.54
DPS
0.31
7.27
28.
0.33
15.27
46.92
Guidance
0.33
7.62
28.06
0.40
20.48
47.56
Quantitative analysis. Tables 1, 2 and 3 report quantitative comparisons between vanilla and guided restoration. As common in diverse restoration works (Alkobi al., 2023; Saharia et al., 2022; Zhao et al., 2021), we use the average LPIPS distance (Zhang al., 2018) between all pairs within the set as measure for semantic diversity. We further report the NIQE image qual- ity score (Mittal et al., 2012), and the LR-PSNR metric which quantifies con- sistency with the low-resolution input image in the case of super-resolution We use N/A to denote configurations that missing in the source codes of the base methods. For comparison, also report the results of the GAN based inpainting method MAT (Li al., 2022), which achieves lower di- versity than all diffusion-based meth- ods. As seen all tables, our guid- ance method improves the LPIPS di- versity while maintaining similar NIQE and LRRPSSR levels. The only excep- tion is noiscless inpainting with DPS in Tab. 2, where the NIQE increases but is poor to begin with.
Table 3: Quantitative results CelebAMask- (left)
and PartImageNet (right) image inpainting.
CelebAMask--H
PartImageNet
Model
LPIPS Div. (1)
LPIPS
NIQE (J)
NIQE (L)
Div. (1)
MAT
0.03
4.69
N/A
N/A
DDNM + Guidance
0.06
5.35
0.07
5.82
0.09
5.31
0.08
5.81
RePaint
0.08
5.07
0.09
5.41
+ Guuddacee
0.09
5.05
0.10
5.34
DPS Guidance
0.07
4.97
0.10
5,35
0.09
4.91
0.11
5.37
7 CONCLUSION
We showed that posterior sampling, strategy that has gained popularity in image restoration, is limited its ability to summarize the range of semantically different solutions with a small number of samples. We thus proposed to break-away from posterior sampling and rather aim for composing small but meaningfully diverse sets of solutions. We started by a thorough exploration of what makes set of reconstructions meaningfully diverse, and then harnessed the conclusions for developing diffusion- based restoration methods. We demonstrated quantitatively and via user studies that our methods outperform vanilla posterior sampling. Directions for future work outlined in App. H.

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
As the field of deep learning advances, image restoration models find increasing use in the everyday lives of many around the globe. The ill-posed nature of image restoration tasks, namely, the lack of a unique solution, contribute to uncertainty in the results of image restoration. This especially erucial when the use is for scientific imaging, medical imaging, and other safety critical domains, where presenting restorations that are all drawn from the dominant modes may lead to misjudge- ments regarding the true, yet unknown, information in the original image. It is thus important to outline and visualize this uncertainty when proposing restoration methods, and to convey to the user the abundance of possible solutions. We therefor believe that the discussed concept of meaning- fully diverse sampling could benefit the field of image restoration, commencing with the proposed approach.
REPRODUCIBILITY STATEMENT
We refer cohen.github.. www.ggittu..ooMaainnge.o MnnaannnnuulDiversttyiiii running all of the proposed baseline approaches, well code that includes guidance for all four image restoration methods compared in the paper.
code repository from project's webpage at https://noa-
The repository includes the required scripts for
ACKNOWLEDGFMENTS
The research TM was partially supported by the Israel Science Foundation (grant no. 2318/22), by the Ollendorff Miverva Center, ECE faculty, Technion, and by gift from KLA. YB has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Aktodowska-Curte grant agreement 945422. The Miriam and Aaron Gutwirth Memorial Fellowship supported the rescarch HM.
REFERENCES
Noa Alkobi, Tamar Rott Shaham, and Tomer Michaeli. Internal diverse image completion. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 648- 658, 2023.
Anastasios N Angelopoulos, Amit Pal Kohli, Stephen Bates, Michael Jordan, Jitendra Malik, Thayer Alshaabi, Srigokul Upadhyayula, and Yaniv Romano. Image-to-image regression with distribution- free uncertainty quantification and applications in imaging. In International Confer- ence on Machine Learning, pp. 717-730. PMLR, 2022. 3
Yuval Bahat and Tomer Michaeli. Explorable super resolution. In Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition, pp. 2716-2725, 2020. 1.3
Yuval Bahat and Tomer Michaeli. What's in the image? explorable decoding of compressed images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2908 2917, 2021.
Yunpeng Bai, Cairong Wang, Shuzhao Xie, Chao Dong, Chun Yuan, and Zhi Wang. TextIR: A simple framework for text-based editable image restoration. arXiv preprint arXiv:2302.14736 2023. 3
Marcel C Buhler, Andrés Romero, and Radu Timofte. DeepSEE: Deep disentangled semantic ex- plorative extreme super-resolution. In Proceedings of the Asian Conference on Computer Vision, 2020. 3
Weiwei Cai and Zhanguo Wei. PiiGAN: generative adversarial networks for pluralistic image in- painting, IEEE Access, 8:48451-48463. 2020.
Jianbo Chen, Yelong Shen, Jianfeng Gao, Jingjing Liu, and Xiaodong Liu. Language-bast based image editing with recurrent attentive models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8721-8729, 2018.
10

---

Published as a conference paper at ICLR 2024
Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. ILVR: Con- ditioning method for denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 14367-14376, October 2021. 8, 19
Hyungjin Chung, Jeongsol Kim, Michael Thompson Mccann, Mare Louis Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. In The Eleventh Interna- tional Conference on Learning Representations, 2023. 8, 19,
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recogmition. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 4690-4699, 2019. 24
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8799 2021. 8, 19
Yuval Eldar, Michael Lindenbaum, Moshe Porat, and Yehoshua Y Zeevi. The farthest point strategy for progressive image sampling. IEEE Transactions Image Processing, 609):1305-1315, 1997.
Muhammad Haris, Gregory Shakhnarovich, and Norimichi Ukita. Deep back-projection networks for super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1673, 2018.
Ju Hc, Shuo Yang, Shaokang Yang, Adam Kortylewski, Xiaoding Yuan, Jie-Neng Chen, Shuai Liu, Cheng Yang. Qihang Yu, and Alan Yuille. PartlmageNet: A large. high-quality dataset of parts. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part VIII. pp. 128 145, 2022. 5
Leonhard Helminger, Michael Bernasconi, Abdelaziz Djelouah, Markus Gross, and Christopher Schroers. Generic image restoration with flow based priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 234-343,20 2021.
Seunghoon Hong, Dingdong Yang, Yunseok Jang, Tianchen Zhao, and Honglak Lee. Diversity- sensitive conditional generative adversarial networks. In 7th International Conference on Learn- ing Representations, ICLR 2019. International Conference Learning Representations, ICLR, 2019. 1,2
Eliahu Horwitz and Yedid Hoshen. Conffusion: Confidence intervals for diffusion models. arXiv preprint arXiv:2211.0977 2022. 3
Bahjat Kawar, Gregory Vaksman, and Michael Elad. Snips: Solving noisy inverse problems stochas- tically. Advances in Neural Information Processing Systems, 34:21757-21777 2021a.
Bahjat Kawar, Gregory Vaksman, and Michael Elad. Stochastic image denoising by sampling from the posterior distribution. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1866-1875 2021b.
Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. 1, 2,2,5 5, 8, 19, 20,20,35 35
O Kupyn, T Martyniuk, J Wu, and Z Wang. DeblurGAN-v2: Deblurring faster and better. In Proceedings of the IEEE International Conference on Computer Vision, pp- 8877-8886, 2019. 1
magnitude)
Changwoo Lee and Ki-Seok Chung. GRAM: Gradient rescaling attention model for data uncertainty estimation in single image super resolution. In 2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA), pp. 8-13. IEEF, 2019.
Cheng-Har Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. MaskGAN: Towards diverse and interactive facial image manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 5549-555555 2020. 24

---

Published as a conference paper at ICLR 2024
Wenbo Li, Zhe Lin, Kun Zhou, Lu Qi. Yi Wang, and Jiaya Jia. MAT: Mask-aware transformer for large hole image inpainting. In Proceedings of the IEEE/CVE conference on computer vision and patter recognition, pp. 10758-10768, 2022. 2, 9. 20
Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. SwinIR: Image restoration using swin transformer. In Proceedings of the IEEE/CVF international confer-
ence on computer vision, pp. 1833-1844, 2021.
Lin, Richard Zhang, Frieder Ganz, Song Han, and Jun-Yan Zhu. Anycost GANs for interactive image synthesis and editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14986-14996, 2021. 5, 24
Hongyu Liu, Ziyu Wan, Wei Huang. Yibing Song, Xintong Han, and Jing Liao. PD-GAN: Prob- abilistic diverse GAN for image inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9371-9381, 2021. 1,2
Andreas Lugmayr, Martin Danelljan, Luc Van Gool, and Radu Timofte. SRFlow: Learning the super-resolution space with normalizing flow. Computer Vision-ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part V, volume 12350, pp. 715- 732. Springer, 2020. 2,
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pp- 11461-11471, 2022a. 2022a. 2. 3, 5, 8, 19, 20, 29, 36, 37
Andreas Lugmayr, Martin Danelljan, Radu Timofte, Kang-wook Kim, Younggeun Kim, Jae-young Lee, Zechao Li, Jinshan Pan, Dongseok Shim, Ki-Ung Song, et al. NTIRE 2022 challenge on learning the super-resolution space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 786-797, 2022b. 1, 19
Chenxi Ma, Bo Yan, Qing Lin, Weimin Tan, and Siming Chen. Rethinking super-resolution as text-guided details generation. In Proceedings of the 30th ACM International Conference on Multimedia, pp. 3461-3469, 2022. 3
Qi Mao, Hsin- Ying Lee, Hung- Yu Tseng, Siwei Ma, and Ming- Hssan Yang. Mode seeking genera- tive adversarial networks for diverse image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1429-1437, 2019.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. Interna- tional Conference on Learning Representations, 2022. 19
Anish Mittal, Rajiv Soundararajan, and Alan C Bovik. Making a "completely blind" image quality analyzer. IEEE Signal processing letters, 20(3):209-21,, 2012.
Antonio Montanaro, Diego Valsesia, and Enrico Magli. Exploring the solution space of linear in- verse problems with GAN latent geometry. In 2022 IEEE International Conference on Image Processing (ICIP), pp. 1381-1385. IEEE, 2022. 3
Erik Nijkamp, Mitch Hill, Song-Chun Zhu, and Ying Nian Wu. Learning non-convergent non- persistent short-run meme toward energy-based model. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pp. 5232-5242, 2019. 2
Guy Ohayon, Theo Adrai, Gregory Vaksman, Michael Elad, and Peyman Milanfar. High perceptual quality image denoising with posterior sampling CGAN. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1805-1813, 2021. 1,2
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue. Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2536-2544, 2016.
12

---

Published as a conference paper at ICLR 2024
Jialun Peng, Dong Liu, Songcen Xu, and Houqiang Li. Generating diverse structure for image inpainting with hierarchical VQ- VAE. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10775-10784, 2021. 2
Mangal Prakash, Alexander Krull, and Florian Jug. Fully unsupervised diversity denoising with convolutional variational autoencoders. In International Conference on Learning Representa- lions, 2021. 2
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lec, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: mnage-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings, pp. 1-10, 2022. 1, 9
Swami Sankaranarayanan, Anastasios Nikolas Angelopoulos, Stephen Bates, Yaniv Romano, and Phillip Isola. Semantic uncertainty intervals for disentangled latent spaces, In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. 3
Vikash Sehwag. Caner Hazirbas, Albert Gordo, Firat Ozgenel, and Cristian Canton. Generating high fidelity data from low-density regions using diffusion models. In Proceedings the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 11492-1 1501, 2022. 3
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015. 6. 20
Jiaming Song, Arash Vahdat, Morteza Mardani, and Jan Kautz. Pseudoinverse-guided diffusion models for inverse problems. In International Conference Learning Representations, 2023.
Ziyu Wan, Jingho Zhang, Dongdong Chen, and Jing Liao. High-fidelity pluralistic image comple- tion with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4692-4701, 2021. 2
Wei Wang, Ruiming Guo, Yapeng Tian, and Wenming Yang. CFSNet: Toward controllable fea- ture space for image restoration. In Proceedings of the IEEE/CVF international conference computer vision, pp. 4140-4149, 2019. 3
Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. ESRGAN: Enhanced super-resolution generative adversarial networks. In Pro- ceedings of the European conference on computer vision (ECCV) workshops, pp. 0-0, 2018.
Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion 2022. 1, 2, 8, 19, 20, 30, 33, 34
null-space model. arXiv preprint arXiv:2212.000
Thomas Weber, Heinrich Hußmann, Zhiwei Han, Stefan Matthes, and Yuanting Liu. Draw with me: Human-in-the-loop for image restoration. In Proceedings of the 25th International Conference Intelligent User Interfaces, pp. 243-253, 2020. 3
Tianyi Wei, Dongdong Chen, Wenbo Zhou, Jing Liao, Weiming Zhang, Lu Yuan, Gang Hua, and Nenghai Yu. E2style: Improve the efficiency and effectiveness of stylegan inversion. IEEE Transactions on Image Processing, 31:e:..133000 31:3267-3280 2022. 8, 19
Yanze Wu, Xintao Wang, Yu Li, Honglun Zhang, Xun Zhao, and Ying Shan. Towards vivid and diverse image colorization with generative color prior. In Proceedings of the IEEE/CVF Intera- tional Conference on Computer Vision, 2021. I
Ning Yu, Ke Li, Peng Zhou, Jitendra Malik, Larry S Davis, and Mario Fritz. Inclusive gan: Im- proving data and minority coverage in generative models. In Proceedings of the 16th European Conference on Computer Vision, ECCV 2020, 2020.
Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond gaussian denoiser: Residual learning of deep enn for image denoising. IEEE Transactions on Image Processing. 26 (7):3142-313) 2017.
13

---

Published as a conference paper at ICLR 2024
Lisai Zhang, Qingeai Chen, Baotian Hu, and Shuoran Jiang. Text-guided neural image inpainting. In Proceedings of the 28th ACM International Conference on Multimedia, pp. 1302-1310, 2020. 3
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595 2018. 6,
Lei Zhao, Qihang Mo, Sihuan Lin, Zhizhong Wang. Zhiwen Zuo, Haibo Chen, Wei Xing, and Dongming Lu. UCTGAN: Diverse image inpainting based unsupervised cross-space transla- tion. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 5741-5750, 2020. 2
Puning Zhao and Lifeng Lai. Analysis of KNN density estimation. IEEE Transactions Informa- tion Theory,
2022. 15
Shengyu Zhao, Jonathan Cui, Yilun Sheng, Yue Dong, Xiao Liang, Eric I-Chao Chang, and Yan Xu. Large scale image completion via co-modulated generative adversarial networks. In International Conference on Learning Representations, 2021. 2,
Chuanxia Zheng, Tat-Jen Cham, and Jianfei Cai. Pluralistic image completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1438-1447, 2019.
14

---

Published as a conference paper at ICLR 2024
SUPPLEMENTARY MATERIAL
A DETAILS OF THE UNIFORMIZATION APPROACH
Let fz; denote the probability density function of the posterior' at point 2;, and assume for now that it known and has compact support. In the Uniformization method assign to each member of X' a probability mass that is inversely proportional to its density,
T
W(:))
(3)
7(5))
We then populate X' by sampling from X without repetition according to the probabilities W(::). In practice, the probability density fx) is not known. We therefore estimate it from the samples in L' using the k-Nearest Neighbor (KNN) density estimator (Zhao & Lai, 2022),
k h-
f(x) M MBV(B(((
(4)
Here, Pa (z) is the distance between x and its htth nearest neighbor & and (BB(r) is the volume of ball of radius r, which in d-dimensional Euclidean space is given by
n 4/2
V(B(r))
(5)
r ( +
where I' is the gamma function. Using equation 5 in equation 4 and substituting the result into equation 3, we finally obtain the sampling probabilities for Da E &:
W(:)) Px(x:))
(6)
E PM (x)))d
Note that in many cases, the support of Px|y may be very large, or even unbounded. This implies that the larger our initial set X is, the larger the chances that it includes highly unlikely and peculiar solutions. Although these are in principle valid restorations, below some degree of likelihood, they are not representative of the effective support of the posterior. Hence, omit the percent of the least probable restorations.
more inherent limitation of the Uniformization method is that it may under-represent high- probability modes if their effective support is small. This can be seen in Fig. 4, where although Uniformization leads to diverse set, this set does not contain single representative from the dom- inant mode. Thus in this case, 95% of the samples in X' do not have single representative in X.
Note that estimating distributions in high dimensions is fundamentally difficult task, which requires the number of samples N be exponential in the dimension d to guarantee reasonable accuracy. This problem can be partially resolved by reducing the dimensionality of the features (e.g.. using PCA) prior to invoking the KNN estimator. We follow this practice in our feature based image restoration experiments.
We used - = 100%, k = 10 for the estimator in the toy example, and k otherwise.
For notational convenience omit the dependence on y-
15

---

Published as a conference paper at ICLR 2024
B THE EFFECT OF DISCARDING POINTS FROM X' IN THE BASELINE APPROACHES
With all subsampling approaches, we exploit distances calculated over a semantic feature space as means for locating interesting representative images. Specifically, consider image that is far from all other images in our feature space as being semantically dissimilar to them, and the general logic discussed thus far is that such image constitutes interesting restoration solution and thus a good candidate to serve as representative sample. However, beyond some distance, dissim- ilarity all other images may rather indicate that this restoration is unnatural and too improbable, and thus would better be disregarded.
Two of our analyzed subsampling approaches address this aspect using additional hyper- parameter. In FPS, we denote by L the number of solutions that are randomly sampled from X' before initiating the FPS algorithm. Due to the heavy tail nature of the posterior, setting a smaller value for increases the chances of discarding such unnatural restorations even before running FPS, which in turn results in subset X' leaning towards the more likely restorations. Similarly, in the Uniformization approach, we can use only a certain percent 7 of the solutions. However, while the solutions that we keep in the FPS case are chosen randomly, here we take into account the estimated probability of each solution in order intentionally use only the percent of the most prohable restorations. Figures 9 and 10 illustrate the effects of those parameters in the toy Gaussian mixture problem discussed in the main text, while Figures 11-13 illustrate their effect image restoration experiments.
Large Sample Random
Farchest Point Strategy
L/ 12.5%
L/N 25%
L/Ñ 50%
L/N 100%
Figure 9: Effect of L. on FPS sampling. A toy example comparing the represented set sampled from an imbalanced mixture of 10 Gaussians (left), using subset X' of only N 20 points, for different values of L. Note how the samples spread as L approaches N.
Large Sample
Uniformization
Random
98%
100%
Figure 10: Effect of 7 on Uniformization sampling. toy example comparing the represented set sampled from imbalanced mixture of 10 Gaussians (left), using subset X of only N = 20 points, for different values of r. Note how the central Gaussian which contains 95% the probability mass contains samples for 100%.
16

---

Published as a conference paper at ICLR 2024
Degraded Image
Posterior
Farthest Point Strategy
Uniformization
25%
90%
75%
95%
100%
100%
Figure 11: Effect discarding points before subsampling in super-resolution on CelebA HQ. Note the lack in representation of open-moute smiles when applying Uniformization (right) with r 100% despite the fact that smiles dominate the approximated posterior distribution. This
aligns with the behaviour of the toy example.
Degraded
Posterior
Image
Farthest Point Strategy
Uniformization
25%
90%
75%
95%
100%
101%
Figure 12: Effect of discarding points before subsampling in image inpainting on CelebAMask- HQ. Note how the sunglasses inpainting option is among the first to be omitted in both subsampling methods. This demonstrates the effect of syper-parametert and on the maximal degree of presented peculiarity in the FPS and Uniformization approaches, respectively.
17

---

Published as a conference paper at ICLR 2024
Degraded Image
Posterior
Farthest Point Strategy
Uniformization
25%
90%
75%
95%
100%
100%
Figure 13: Effect of discarding points before subsampling in image inpainting on PartIma- genet.
18

---

Published as a conference paper at ICLR 2024
C EXPERIMENTAL DETAILS
Pre-processing. In all experiments, we crop the images into square and resize them to 256 256, to satisfy the input dimensions expected by models. For all super-resolution experiments, bicubic downsampling applied to the original images create their degraded version, and random noise was added for noisy super-resolution (according to the denoted noise level).
the landmark-aligned face images in the CelebAMask-HO dataset and define four masks: Large and small masks covering roughly the area of the eyes, and large and small masks covering the mouth and chin. For each face sample one of the four possible masks. For inpainting of PartImagenet images, we combine the masks all parts of the object and the minimal bounding box that contains them all.
Masks. For face impainting,
Pretrained models. For PartImageNet we use the same checkpoint of Dhariwal & Nichol (2021) across all models. For HelcbAMask-HO,, we use the checkpoint of Meng al. (2022) in DDRM, DDNM and DPS, and in RePaint we use the checkpoint used in their source code.
Guidance parameters. The varied diffusion methods used in all guidance experiments (Chung al., 2023; Kawar et al., 2022; Lugmayr et al., 2022a; Wang et al., 2022) display noise spaces with different statistics during their sampling process. This raises the need for differently tuned guidance hyper- parammetrssssss for each method, and sometimes for different domains. Tabs. and 5 lists the guidance parameters used in all figures and tables presented in the paper.
Table 4: Values of the guidance step-size hyper-parameter 7) used in our experiments.
CelebAMask-HO
PartImageNet
Model
Super Resolution
Super Resolution
Image Inpainting
Image Inpainting
0.05
0.05
DDRM
0.8
0.8
N/A
0.8
0.8
N/A
DDNM
N/A
0.9
N/A
0.9
DPS
0.5
0.3
0.5
0.5
0.5
0.5
RePaint
N/A
N/A
0.3
N/A
N/A
0.3
Table 5: Values of D, i.e., the upper distance bound for applying guidance, used in our experiments.
CelebAMMssk-HO
PartImageNet
Model
Super Resolution
Super Resolution
Image Inpainting
Image Inpainting
0.05
0
0.05
0
DDRM
0.0004
0.0004
N/A
0.0004
0.0004
N/A
DDNM DPS
N/A
0.0005
0.0015
N/A
0.0003
0.0008
0.06
0.001
0.009
0.0005
0.0005
0.009
RePaint
N/A
N/A
0.00028
N/A
N/A
0.00028
Consistency constraints. As explained in Sec. 6, in our experiments we consider only consistent models. We regard model as consistent based on the mean PSNR of its reconstructions computed between the degraded input image y and degraded version of the reconstruction, e.g., LR-PSNR the task super-resolution. For noiseless inverse problems, such as noiseless super-resolution or inpainting, we follow Lugmayr et al. (2022b) and use 45dB the minimal required value to be considered consistent. This allows for a deviation of bit more than one gray-scale level. We additionally experimented with ILVR (Choi al., 2021) I) and E2Style (Wei et al., 2022), which were both found to be inconsistent (even when trying to tune ILVR's range hyper-parameter, E2Style does not have parameter to tune).
19

---

Published as a conference paper at ICLR 2024
DPS Vanilla Generations
Degraded Image
Restored Image
Degraded Image
Restored
Degraded Image
Restored Image
Degraded Image
Restored Image
Image
Inpainting
Super Resolution
Figure 14:
Examples of artifacts in the generations of DPS. We show here results generated by DPS with G = 2 for inpainting, and Çi 10 for noiseless super-resolution.
Tuning the hyper-parameter & of DPS. While RePaint (Lugmayr et al., 2022a), MAT (Li et al., 2022), DDRM (Kawar al., 2022) and DDNM (Wang al., 2022) are inherently consistent, DPS (Chung et al., 2023) is not, and its consistency is controlled by G. hyper-parameter. To allow fair comparison,
searched for the minimal & value per experiment, that would yield consistent results without introducing saturation artifacts (which we found to increase with Ç)). In particular, for impainting on CelebAMask- consistent and plausible (artifact-free results for inpainting on PartImageNet, we resorted to using the setting of S adopted from the the CelebAMass HQ configuration, when reporting the re- sults in the bottom right cell of Tab. 3. However, note that the restorations there are inconsistent with their corresponding inputs. For noiseless super-resolution on CelebAMask-HH HQ we use G = 10. For noiseless super-resolution on PartImageNet we use Ç = 3. While this is the minimal value that we found to yield consistent results, these contained saturation artifacts, as evident by their corre- sponding high NIQE value in Tab. 2. We nevertheless provide the results for completeness. For noisy super-resolution we follow a rule of thumb of having the samples' LR-PSNR around 26dB, which aligns with the expectation that the low-resolution restoration should deviate from the low- resolution noisy input by approximately the noise level o, = 0.05. Following this rule of thumb use & 1 for noisy super-resolution on both domains. Examples of DPS saturation artifacts (resulting from high C values) can be seen in Fig. 14.
we use
Since we were unable find Ç. values yielding
Measured diversity and image quality. In Tabs. 1, and 3 we report I. LPIPS diversity and NIQB?. In all experiments, the LPIPS diversity was computed by measuring the average LPIPS distance over all possible pairs X, with VGG-16 (Simonyan & Zisserman, 2015) as the neural features architecture.
We use PyTorch Toolbox for Image Quality Assessment available at http://gitthh.com//bnoonngg///- PyTorch for computing NIQE and the I.PIPS distance.
20

---

Published as a conference paper at ICLR 2024
D THE EFFECTS OF THE GUIDANCE HYPERPARAMETERS
Here we discuss the effects of the guidance hyperparameters 7), which is the step size that controls the guidance strength, and D, which controls the minimal distance from which we do not apply guidance step.
We provide qualitative and quantitative results CelebA.Mask-HOQ image inpainting in Figs. 15 and 16 and Tabs. 6 and 7, respectively. As can be seen, increasing yields higher diversity in the sampled Too large 77 values can cause saturation effects, but those can be least partially mitigated by adjusting 1) accordingly. Intuitively speaking, increasing 1) allows for larger changes to take effect via guidance. This means that the diversity increases as increases. The effect of using a minimal distance D at all is very noticeable in the last row of Fig. 16. In this row, no minimal distance was set, and therefore the full guidance strength of 7j is visible. Setting D allows truncating the effect for some samples, while still using large that will help push similar samples away from one another. Both nyperparameters work in conjunction, as setting one parameter too-small will yield to lower diversity.
Table 6: Effect of y) on results for HQ in image inpainting. Here D is fixed 0.0003.
LPIPS Div. (f)
NIQE (1)
(Posterior)
0.090
5.637
0.07
0.105
5,495
0.1
0.109
5.506
0.3
0.113
5,519
Table 7: Effect of D on results for CelebAMash-HQ in image inpainting. Here is fixed at 0.09.
D
LPIPS Div. (f)
NIQE (4)
0
0,090
5.637
0.0002
0.094
5.552
0.0003
0.108
5.472
0.0004
0.126
5.554
0O
0.141
5,450
21

---

Published as a conference paper at ICLR 2024
Degraded Image
n
|
Posterior
0.07
0.1
0.3
Figure 15: The effeet of using different step sizes on the diversity of the results. Here, we fix
D to 0.003.
22

---

Published as a conference paper at ICLR 2024
Degraded Image
D
Posterior
0.0002
0.0003
0.0004
8
Figure 16: The effect of using different upper bound distances D on the diversity of the results. In this example, not setting upper bound (last row) results in some mild artifacts, e.g. overly bright regions as well as less-realistic earring appearances. Here, we fix to 0.09.
23

---

Published as a conference paper at ICLR 2024
E
USER STUDIES
Beyond reporting the results in Fig. 6 in the main text, we further visualize the data collected in the user studies discussed in 4.1 on a 2/ plane depicting the trade off between the two characteristies of each sampling approach: (i) the diversity perceived by users compared with the diversity of random samples from the approximated posterior, and (ii) the coverage of more likely solutions by the sub- sampled set X' All sub-sampling approaches achieve greater diversity compared to random samples from the approximate posterior in both super-resolution and impainting tasks, performed on images from the their corresponding different positions on the diversity-coverage coverage plane. In both tasks, sampling according to K K-means achieves the highest coverage of likely solutions, at the expense of relatively low diversity values. Sub-sampling using FPS achieves the highest diversity.
dataset Lee et al. (2020). However, the visualization in Fig. 17 indicates
Inpainting
Super Resolution
67.5
67.5
Posterior
K-moons Uniioomitation
63.0
FPS
62.5
62.5
Divessty [X]
00.0
00.0
57.5
57.5
55.0
52.5
50.0
50.0
55
40
45
60
Coverage [50]
Coverage [%]]
plane. A representative set needs to trade-off covering the possible solution set, and seeking diversity in the subset of images presented. Diversity of the three explored approaches was measured relative to approximated posterior samples, hence the value determined for the posterior sampling is in theory 50%.
Figure 17:
In our exploration of what mathematically characterizes a meaningfully diverse set of solutions in Sec. 4 build upon semantic deep features. The choice of which semantic deep features to use in the sub-sampling procedure impacts the diversity as perceived by users, and should therefore be tuned according to the type of diversity aimed for. In the user studies of these baseline approaches, discussed in 4.1, we did not guide the users what type of diversity to seek (e.g., diverse facial expressions diverse identities). However, for all our sub-sampling approaches, we used deep features from the AnyCost attribute predictor Lin et al. (2021). We now validate our choice to use L distance over such features as proxy for human perceptual dissimilarity by comparing it with other feature domains and metrics in Fig. 18. Each plot depicts the distances from the target images presented in each question to their nearest neighbor amongst the set of images X presented to the user, against the percentage of users who perceived at least one of the presented images as similar to the target. We utilize different feature domain to calculate the distances in each sub-plot, and report the corresponding Pearson and Spearman correlations in the sub-titles (lower is better, as we compare distance against similarity). All plots correspond to the image- inpainting task. Note that the best correlation is measured for the case of using the deep features of the attribute predictor, compared to using cosine distance between deep features of ArcFace Deng al. (2019), using the pixels of the inpainted patches, or using the logits of the attribute predictor. The significant dimension reduction (e.g.. from 32768 to 25 dimensions in the case of the Any attribute predictor) only slightly degrades correlation values when using distance over deep features. Finally, in Figs. 19-21 we include screenshots of the instructions and random questions presented to the users in all three types of user studies conducted in this work.
24

---

Published as a conference paper at ICLR 2024
All Features
Reduced Dimensions
Logits
Peanson cotrelation 0.66
Peacson cotrelation -0.64
Pearson correlation -0.50
Spenzman cottelation
Spearman correlation
Spearman correlation: -0.48
Attribute Predictor
Dust from
Pemtonnd Simbarity (6
Pervioed Simiarit XX]
Precioee Stallirity
Peanson correlation 0.55
Pranson corrolation
0.52
Spenrmau correlation
0.58
Spearman correlation
00.6
60
ArcFace
20
20
10
25
30
70
300
20
310
Perceeee Stularity
Pextied Samilrity X0]
Peanson corvolition -0.08
Prazson correlation
Spearman correlation -0.07
Spearman cottelation
-0.19
12-
12
Patch Pixels
Dust Stm NW
1.0
1.0
0.8
0.4
0.4
0.2
0.2
25
300
70
100
20
100
Percoesed Siuiloity (%)
Perieved
Semuloity X0]
Figure 18: Correlation between semantic distances and similarity as perceived by users. Post processing of the data collected in the user study, evaluating the semantic distance used (top left).
25

---

Published as a conference paper at ICLR 2024
About this HIT:
should
This user study quancilies the effectlveness of new image ristoration software. queny, you'll image missing (biod) region appcoring screen.
inage you
imgges
images
gf sour variag varialy.
You complete short praciiee (lees than minute) before starting.
Start the practicel
(a) Instructions presented to the user.
Image 1
(b) Example for question set of inpainting restorations.
Image
(c) Example for a question on a set of super resolution restorations.
Figure 19: Paired diversity test. After reading instructions (upper) participant had to answer which of the lines shows images with greater variety.
26

---

Published as a conference paper at ICLR 2024
About this HIT:
Roshould talee about menooo
This user study cuand fix the offactivren mese of new imaçe restoration sofhwae.
In each quary-oou lass aniginal imaqu that la ofllow revciation sppeernng the top lop and notorsion
the top signt
inay
aimiarr mecortruttd
iugel
sosien images
creclette (lese min
Start the practice!
(a) Instructions presented to the user.
Image 1,50
(b) Example for question of super resolution restorations.
Image
(c) Example for question on a set of inpainting restorations.
Figure 20: Unpaired coverage test. After reading instructions (upper) participant had to answer whether any the shown images is very similar to the target image.
27

---

Published as a conference paper at ICLR 2024
About this HIT:
It should take about minutes.
This user study quantifies the effectiveness new image restoration software:
query, you'll low-resolution image appearing top
the
Below image, you will restored resolution versions that image. Select the image which has higher quality.
Start experiment!
(a) Instructions presented to the user.
Image 1/50
Original Image
This image has higher quality!
This image has higher quality!
(b) Example for question on set of super resolution restorations.
Image 1/50
Original Image
This image has higher quality!
This image has higher quality!
(c) Example for question on set of inpainting restorations.
Figure 21: Paired image quality test. After reading instructions (upper) participant had to answer which image is perceived with higher quality.
28

---

Published as a conference paper at ICLR 2024
F AN ALTERNATIVE GUIDANCE STRATEGY
In Sec. 5 we proposed to increase the set's diversity by adding to each clean image prediction Zofe Xr the gradient of the dissimilarity between coje and its nearest neighbor within the set,
4 .N.N Ofr
arg
min 2
x)
An alternative could be to use the dissimilarity between each image Zo and the average of all N images the set,
AVG
Σ P
ofe
N
JE(I.....N)
We opted for the simpler alternative that utilizes the nearest neighbor Zuje since we found the two alternatives to yield visually similar results. We illustrate differences between the approaches in Figs. 22, 23, 24 and 25, where the values for in the inpainting task are 0.24 and 0.3 for the average and nearest neighbor cases respectively, and in the super resolution task are 0.64 and 0.8 for the average and nearest neighbor cases respectively.
Degraded Image
Posterior
Average NN
Figure 22: Comparing the alternative guidance strategies for inpainting on CelebAMask-HO..
We compare posterior sampling against both guidance using dissimilarity calculated relative the nearest neighbor (NN) and against guidance using dissimilarity calculated relative to the average image the set (Average). Restorations generated by RePaint (Lugmayr al., 2022a).
Degraded Image
Average NN Posterior
Figure 23: Comparing the alternative guidance strategies for inpainting on PartImageNet.
We compare posterior sampling against both guidance using dissimilarity calculated relative to the nearest neighbor (NN) and against guidance using dissimilarity calculated relative to the average image of the set (Average). Restorations generated by RePaint (Lugmayr et al., 2022a).
29

---

Published as a conference paper at ICLR 2024
Degraded Image
Posterior
Average NN
E
Figure 24: Comparing the alternative guidance strategies for noiseless super-resolution on CelebAMask-HOO, We compare posterior sampling against both guidance using dissimilarity cal- culated relative to the nearest neighbor (NN) and against guidance using dissimilarity calculated relative to the average image of the set (Average). Restorations generated by DDNM (Wang et al.,
2022).
Degraded Image
Average NN Posterior
Figure 25: Comparing the alternative guidance strategies for noiseless super--esolltion on Par-
tlmageNet. We compare posterior sampling against both guidance using dissimilarity calculated relative to the nearest neighbor (NN) and against guidance using dissimilarity calculated relative to the average image of the (Average). Restorations generated by DDNM (Wang et al., 2022).
30

---

Published as a conference paper at ICLR 2024
G HIERARCHICAL EXPLORATION
In some cases, a subset of restorations from 2' may not suffice for outlining the complete range of possibilities. naive solution in such cases is simply increase N. However, as N grows, presenting user with all images at once becomes ineffective and even impractical. We propose an alternative scheme to facilitate user exploration by introducing a hierarchical structure that allows users explore the realm of possibilities in X' in an intuitive manner, by viewing only up to N images at time. This is achieved by organizing the restorations in X' in tree-like structure with progressively fine-grand ine-grained distinctions. To this end, we exploit the semantic significance of the distances feature space twice; once for sampling the representative set each hierarchy level (e.g., using FPS), and a second time for determining the descendants for each shown sample, by using its perceptual' nearest neighbor. This tree structure allows convenient interactive exploration of the set X, where, each stage of the exploration all images of the current hierarchy are presented to the user. Then, to further explore possibilities that are semantically similar to one of the shown images, the user can choose that image and move on to examine its children.
Constructing the tree consists of two stages that are repeated in recursive manner: choosing rep- resentative samples from within a set of possible solutions (which is initialized as the whole set X'), and associating with each representative image a subset of the set of possible solutions which will form its own set of possible solutions down the recursion. Specifically, for each set of images not explored (initially all image restorations in X'), sampling method is invoked to sample up to N images and present them to the user. Any sampling method can be used (e.g.. FPS, Uniformization, etc.), which aims for meaningful representation. All remaining images more the sampled images based on similarity. This forms (up to) sets, each associated with one representative image. Now, for each of these sets, the process is repeated recursively until the associated set is smaller than N. This induces tree structure on X, demonstrated in Fig. 26, where the association was done by partitioning according to nearest neighbors by the similarity distance.
associated with
Algorithm Hierarchical exploration
X: set of restored images N: number of images to display at each time-step
SM: sampling method
PM: partition method
1: function MAIN
2: MainRoot + empty node
3:
EXPLOREIMAGES(A? MainRoot)
4: function EXPLOREIMACESS(maAS, root)
5:
if number of images < NN then root.children = images
6:
7:
else
8:
children + SM(mmages) for i (1..... N) do
9:
10:
Descendants < PM(children, i, images)
sub_tree + MAAPLOOREIMAGES(Dssceeeae
children(i))
12:
mottchildreen.pppenddee tree)
13:
return root
Ensure: Tree data structure under MainRoot with all images as its vertices.
31

---

Published as a conference paper at ICLR 2024
RTG
(a)
(c)
Figure 26: Visualization of the hierarchical exploration. The implied trees when setting N 4, using the FPS sampling method, from total of N 25 restorations. The degraded image is marked in blue at the top right. Different attributes from the attribute predictor are expressed each example, depending on the variety in the restorations. Note the variations in makeup and smile the top pane, eye-wear and eyebrow expression on the middle pane and general appearance in the bottom pane.
32

---

Published as a conference paper at ICLR 2024
H DIRECTIONS FOR FUTURE WORK
We investigated general meaningful diversity, which focuses on exploring different kinds of diversity once. For example, in the context of restoration of face images, we aimed for our representative set to cover diverse face structures, glasses, makeup, etc. However, for certain applications it can be desirable reflect the diversity for specific property, e-g.. covering multiple types of facial hair and accessories while keeping the identity fixed, or covering multiple identities while keeping the facial expression fixed. The ability achieve diversity in only specific attributes can potentially be important in e. the medical domain, for example to allow radiologist to view range of plausible pathological interpretations for specific tumor in CT scan, or to present a forensic investigator with representative subset of headwear that are consistent with low quality surveillance camera footage. Additionally, We believe future work may focus on developing similar approaches for enhancing meaningful diversity in other restoration tasks.
I ADDITIONAL RESULTS
Throughout experiments in the main paper explored meaningful diversity in the context of inpainting and image super-resolution (with and without additive noise). To demonstrate the wide applicability of method, now present results the task of image colorization, well as additional comparisons on image inpainting and super-resolution.
1.1 IMAGE COLORIZATION
Figures and 28 present comparisons between our diversity guided generation and vanilla genera- tion for the task of colorization. These results were generated using DDNM Wang al. (2022), where the guidance parameters used colorization on CelebAMask-HQ are 7) = 0.08, D 0.0005, and the guidance parameters for colorization PartImageNet are 0.08, D 0.0003. As can be seen, our method significantly increases the diversity the restorations, revealing varia- tions in background and hair color, well in face skin tones. This is achieved while remaining consistent with the grayscale input images. Indeed, the average PSNR between the grayscale in- put image and the grayscale version of our reconstructions is 56.3dB for the face colorizations and 58.0dD for the colorizations of the PartImageNet images.
Degraded Image
Vanilla Generation
EiversityGuuide Generation
Figure 27: Comparisons of colorization CelebAMask-HO, with and without diversity- guidance. Restorations generated by DDNM (Wang et al., 2022).
33

---

Published as a conference paper at ICLR 2024
Degraded Image
Vanilla Generation
Diversity- Guideed Generation
Figure 28: Comparisons of colorization on PartImageNet, with and without diversity-guidann Restorations generated by DDNM (Wang et al., 2022).
34

---

Published as a conference paper at ICLR 2024
1.2 ADDITIONAL COMPARISONS ON IMAGE INPAINTING AND SUPER RESOLUTION
Degraded
Approx. Postcrior Sampling
Explored Sampling Methods
Image
Uniformization
KeManns
25
Figure 29: Additional comparisons of noisy 16x super resolution with
0.05 on
CelebAMask-HH with sub-sampling approaches vs. using the approximate posterior. Restora- tions created using DDRM Kawar al. (2022).
35

---

Published as a conference paper at ICLR 2024
Degraded Image
Approx. Posterior Sampling
Explored Sampling Methods
FPS
Uniformization
K-MMans
Figure 30: Additional comparisons of inpainting on CelebAMask HQ with sub-sampling ap- proaches using the approximate posterior. Restorations created using RePaint Lugmayr al. (2022a).
36

---

Published as a conference paper at ICLR 2024
Degraded Image
Approx. Posterior Sampling
Explored Sampling Methods
TPS
Uniformization
K- M-Maans
Figure 31: Additional comparisons of inpainting on PartImagenet with sub-sampling ap-
proaches using the approximate posterior. Restorations created using RePaint Lugmayr al. (2022a).
Degraded Image
Vanilla Generation
Diversity- Guided Generation
DDNM DDNM
PePuin
RePaint
Figure 32: Additional comparisons of inpainting CelebAMaskkHOO, with and without
diversity-guidanee Restoration method marked on the left.
37

---

Published as a conference paper at ICLR 2024
Degraded Image
Vanilla Generation
Diversity- Guided Generation
DDNM
DDNM
RePaint
Re Paint
Figure 33: Additional comparisons of inpainting on PartImageNeet, with and without diversity- guidance. Restoration method marked on the left.
Degraded
Vanilla Generation
Diversity- Guided Generation
Image
DDRM
DDNM
DDRM
DPS
Figure 34: Additional comparisons noisy and noiseless super-resolution on CelebAMask-
HQ, with and without diversity-guidand Restoration method and noise level marked on the left.
Degraded Image
Vanilla Generation
Diversity- Guided Generation
DDRM
DDNM
DDRM
0.05
DPS
Figure 35: Additional comparisons of noisy and noiseless 4x super-resolution on PartIma-
geNet, with and without diversity-guidanc Restoration method and noise level marked on the left.
38


Reviews:
[{"review_id": "Wy4F2CQF3g", "reviewer": "Reviewer_ppuX", "summary": "This paper discusses the problem of generating a diverse set of reconstructed samples when solving an imaging inverse problem using a pretrained diffusion model. The traditional setup is to cast the inverse problem as one of posterior distribution sampling. However, as the authors discussed this may lead to a concentrated set of samples where most of them look very similar to each other. The paper discusses different ways of measuring and modeling diversity in image reconstruction and later introduces a method for generating a given number of samples  conditioned on a low-resolution / low-quality image, maximizing diversity and using a pretrained diffusion model. The method is evaluated on popular benchmarks on two restoration tasks: super-resolution (4/16x, noisy and noiseless) and inpainting.", "questions": "The paper is in general well-written, addresses an interesting problem, and present an interesting solution. So I'm in favour of accepting this paper as it is now. However, if the authors would like to make the paper better I list a few questions/comments that could be helpful.\n\n1.  **Analysis of the proposed method**. The method seems to do the job, but I wonder if the authors could provide more insight on their proposed solution. For example, a few alternative designs could be:\n    * Instead of increasing distance with respect to NN, just maximize the sample variance (or average pairwise distance as in the reported LPIPS metric)\n    * What happens with the sample average. In the traditional formulation, the average of the generated samples should be close to the posterior mean (which is also another possible estimator that minimizes distortion). Have you tried to compute and compare the sample means w/wo the diversified guidance ? I guess that the mean wouldn't be as good as in the traditional case, but maybe a more robust mean estimator could lead to a very similar result (an estimator that minimizes distortion) \n\n2. **Other Experiments**. In particular regarding Colorization (e.g., one very relevant practical problem is face colorization, can we generate a meaningful diverse set of reconstructed images when starting from a gray photo of a face?)\n\n3. **Connections to other work on GANs** (and mode collapse which seems related).\n\n    * Mao, Q., Lee, H.Y., Tseng, H.Y., Ma, S. and Yang, M.H., 2019. Mode seeking generative adversarial networks for diverse image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 1429-1437).\n\n   * Yu, N., Li, K., Zhou, P., Malik, J., Davis, L. and Fritz, M., 2020. Inclusive gan: Improving data and minority coverage in generative models. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXII 16 (pp. 377-393). Springer International Publishing.\n  \n4. **Training**. This paper is not about training models that allow to generate a diverse set of reconstructions. But, I wonder if the authors could give a comment on wether we could incorporate the idea of diverse sample generation also during training. In particular I wonder if the authors could comment on wether this is an interesting thing to consider, for example in the context of conditional generation ([SR3](https://iterative-refinement.github.io/), [Palette](https://iterative-refinement.github.io/palette/)) or current Bridge restoration models ([InDI](https://openreview.net/forum?id=VmyFF5lL3F), [I2SB](https://i2sb.github.io/))", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "* The paper is generally well-written, address an interesting and relevant problem with a clear narrative and presentation.\n* The technical contribution of the paper is good: analizes a problem and propose a practical solution. This could lead to more research on this problem (\"meaningfully diverse image restoration\")", "weaknesses": "Not weakness per se, more like observations:\n* Experimental results are interesting but limited. For example, image colorization is one of the main problems where diversity reconstruction could have a significant impact.  \n* Presentation could be a little more balanced. In particular, the experimental section (Sec 6) could be a little longer and maybe show more results, discuss a little more some of the algorithmic decisions (presented in Sec 5). The authors opted to put more content on Sec 3 and 4, which seems also a valid decision.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ep7gWQB1Ad", "reviewer": "Reviewer_iJud", "summary": "This paper is concerned with  generating  diverse solutions to ill-posed image restoration problems. Usual random sampling from the heavy tailed posterior distribution will have samples from high density regions with a much higher probability than from low density regions, making it an impractical approach to generate diverse solutions from limited samples. Given a large number of samples from posterior, the paper explores three approaches to extract limited number of samples representing meaningful diversity. Further, the paper proposes a simple guidance mechanism to improve diversity of solutions. This involves running the diffusion process to simultaneously generate multiple images starting from different noise samples, and encouraging dissimilarity between estimates of clean images at every step. This guidance improves diversity of solutions of recent diffusion based restoration methods while  maintaining reconstruction quality.", "questions": "Could the authors clarify the following issues:\n\nIt is not clarified what kind of down-sampling is used to obtain degraded images for super-resolution.\n \nFig. 14, degraded input for super-resolution looks wierd, the results of DPS look like down-sampled images.\n\nFigs 28 , 29 super-resolution or inpainting. degraded image looks like a low resolution image, caption says inpainting.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well motivated and well-written.\n\nThe proposed technique is simple and straightforward to implement and the work is reproducible. \n\nThe proposed method improves the diversity of solutions.", "weaknesses": "The observation that random sampling leads to limited diversity for long tailed distributions was also noted in [a] in the context of image generation.  [a] also proposes to modify the sampling process in diffusion models to generate samples from low density regions of a long tailed distribution. Though the focus in [a] is on generation, and not restoration, it is still a relevant work, and could be cited by the authors.\n\nThe authors could also cite {b] which attempts to generate diverse solutions to linear inverse problems using pretrained gans.\n\nWhile the proposed guidance method improves diversity and is useful, the technical contribution  seems rather limited. This is the reason for my rating.\n\n[a] Sehwag et al. \"Generating high fidelity data from low-density regions using diffusion models.\" In CVPR 2022\n\n[b]  Montanaro etal. \"Exploring the solution space of linear inverse problems with gan latent geometry.\" In  IEEE International Conference on Image Processing  (ICIP) 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "i2UivWONai", "reviewer": "Reviewer_Fph6", "summary": "The current trend in image restoration methods involves sampling multiple outputs from the posterior distribution rather than generating a single one. This paper has highlighted a fundamental limitation of posterior sampling, specifically in generating semantically distinct possibilities using a reasonably small number of samples. This limitation arises from the heavy-tailed nature of the posterior distribution along semantically relevant directions. As a solution, the paper suggests creating compositions of small but meaningfully diverse outputs. The study also delves into a comprehensive examination of what constitutes a set of reconstructions that are meaningfully diverse. To achieve this, the paper explores various post-processing approaches, including Uniformization, K-Means Centers, and Farthest Point Strategy, to obtain semantically meaningful and diverse outputs using existing image restoration methods. Furthermore, the paper introduces a practical approach for enabling diffusion-based image restoration methods to produce outputs that are meaningfully diverse. The effectiveness of these methods is validated through quantitative measures and user studies, demonstrating their superiority over conventional posterior sampling techniques.", "questions": "I have some concerns while reviewing the main paper, but they have been addressed in the supplementary material.\n\nA suggestion for the authors: The discussion on future work could be expanded, e.g., what specific desired variations can be for potential research in the field? This would provide more clarity and guidance for researchers interested in pursuing further work in this area.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-  identify an inherent conceptual issue with posterior sampling and elucidate the root cause of this problem\n- conduct an analysis to determine the criteria for meaningful diversity. This analysis involves an exploration of three fundamental strategies for diverse sub-sampling.\n-  introduce a novel diffusion-based generation strategy, which offers a practical means to achieve restoration results that are meaningfully diverse.", "weaknesses": "None", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)