Inversion by Direct Iteration: An Alternative to Denoising Diffusion for Image Restoration
Authors: Mauricio Delbracio
Peyman Milanfar
Venue: Accepted by TMLR ()

Abstract: Inversion by Direct Iteration (InDI) is a new formulation for supervised image restoration that avoids the so-called ``regression to the mean'' effect and produces more realistic and detailed images than existing regression-based methods. It does this by gradually improving image quality in small steps, similar to generative denoising diffusion models.

Image restoration is an ill-posed problem where multiple high-quality images are plausible reconstructions of a given low-quality input. Therefore, the outcome of a single step regression model is typically  an aggregate of all possible explanations, therefore lacking details and realism. The main advantage of InDI is that it does not try to predict the clean target image in a single step but instead gradually improves the image in small steps, resulting in better perceptual quality.

While generative denoising diffusion models also work in small steps, our formulation is distinct in that it does not require knowledge of any analytic form of the degradation process. Instead, we directly learn an iterative restoration process from low-quality and high-quality paired examples. InDI can be applied to virtually any image degradation, given paired training data. In conditional denoising diffusion image restoration the denoising network generates the restored image by repeatedly denoising an initial image of pure noise, conditioned on the degraded input. Contrary to conditional denoising formulations, InDI directly proceeds by iteratively restoring the input low-quality image, producing high-quality results on a variety of image restoration tasks, including motion and out-of-focus deblurring, super-resolution, compression artifact removal, and denoising.

Published in Transactions on Machine Learning Research (06/2023)
Inversion by Direct Iteration: An Alternative to Denoising Diffusion for Image Restoration
Mauricio Delbracio
mdelbra Groogle.goo
Google Research
Peyman Milanfar Google Research
malanfar&@google..com
Reviewed on OpenReview: https: // openreview. net/ met/ forum? id- VmyFF51L33
Abstract
Inversion by Direct Iteration (InDI) is new formulation for supervised image restoration that avoids the so-called "regression to the mean" effect and produces more realistic and detailed images than existing regression- aggressoon-based methods. It does this by gradually improving image quality in small steps, similar to generative denoising diffusion models.
Image restoration an iol-posed problem where multiple high-quality images are plausible reconstructions of a given low-quality input. Therefore, the outcome of single step regres- sion model is typically and realism. The main advantage of InDI is that it does not try to predict the clean target image in a single step but instead gradually improves the image in small steps, resulting in better perceptual quality.
aggregate of all possible explanations, therefore lacking details
While generative denoising diffusion models also work in small steps, our formulation is dis- tinet in that it does not require knowledge of any analytic form of the degradation process. Instead, directly learn iterative restoration process from low-quality and high-quality paired examples. InDI can be applied to virtually any image degradation, given paired training data. In conditional denoising diffusion image restoration the denoising network generates the restored image by repeatedly denoising an initial image of pure noisc, condi- lioned on the degraded input. Contrary to conditional denoising formulations, InDI directly proceeds by iteratively restoring the input low-quality image, producing high-quality re- sults on a variety of image restoration tasks, including motion and out-of-focus deblurring, super-resolution, compression artifact removal, and denoising.
1 Introduction
Recovering a high-quality image from a low-quality observation is a fundamental problem in computer vision and computational imaging. Single image restoration a highly ill-posed inverse problem where multiple plansible sharp and clean images could lead to the very same degraded observation. The typical supervisod approach to formulate image restoration as a problem of inferring the underlying image given low-quality version of it, by training model with paired examples of the relevant degradation (Ongie et al. 2020 One of the most common approaches is to directly minimize pixel reconstruction error using the L1 or Le loss; approach that correlates well with the popular PSNR (peak signal-to-noiso-ratio metric. However, it has been observed often in recent literature that measures such as PSNR (and in general point-distortion metrics) do not correlate well to human perception Blau Michaeli 2018 Delbracio et al. 2021b Freirich et al., )021). Despite these shortcomings, much of the recent rescarch work has boen focused on improving deep architectures and optimizing variety of point-loss formulations, resulting in general models that give aggregate improved image in step of inference.

---

Published in Transactions on Machine Learning Research (06/2023)
To see the issues more concretely, let's assume that we are given image pairs (æ.y) p(z. y) where x represents a target high-quality image, and y represents the respective degraded observation. For instance, a may be pristine images degraded by a combination of /lur/comppe ccompression and noise to yield A typical regression approach would predict x directly from y using a trained model f(y) = Fo(y) z æ, by minimizing the expected pixel error in some (e.g. Ip)) metric as follows:
min Ex,y Foy) ællp min
Σ [1Fo(yy) x x='1p.
In the case 2, the minimum mean-squared error (MMSE) optimal solution is the conditional expectation: #MMSE(3) (y) = E æ y = x y)
This evidently results in an image that the (weighted) average of all plausible reconstruction: resulting image will not have a natural appearance as the details have been wiped out due to the effect aggregation (i.e. "regression to the mean") effect. The problem is compounded for more ill-posed problems. That is, the more ill-posed the inverse problem, the larger the set of plausible reconstructions and therefore the more severe the effect of the aggregation implicd by the expectation of the posterior. To mitigate this problem, recent. works have introduced additional loss terms (Galys et al.) 2016 Mechrez et al. 2018b aj Delbracio et al. 2021b Kupyn et al., 2018) that seek a balance in the formulation so the final image has improved perceptual quality (more on this in the next section).
T This
In this work, we explicitly address this problem by avoiding single-step prediction of the clean image, and instead iterating series of inferences, where at each step solve an 'casier' (i.e., less ill-posed) inverse problem than the original. Specifically, we generate sequence of intermediate restorations where at each step the goal is to reconstruct only slightly less corrupted image. The core observation underlying this approach is this: a small-step restoration largely awvids the perresion-to-the-maee plausible 'slightly-less-bolk images is relatively small. The core technical component enabling our approach is still a single deep model, but one that is trained to predict a better image given one with an intermediate level of degradation in the previous step, as summarized by Algorithm
effect because the set of
2 Background
Recently, much work on imaging inverse problems has been focused on using generative formulations (Bora ct al.]
2017
Kawar et al. 2022). Generative adversarial formulations train restoration networks with an
adversarial loss that forces the restored image be on the distribution of high-quality signals (Kupyn)
2018 2019 Asim al. 2020). GANs are in general hard to train and also hard to control image et al.
hallucinations since the two terms play antagonic role (Lugmayr et al. 2021
Image priors, including generative ones, can be used to solve inverse problems in an unsupervised fashion where the degradation operator is only know at inference time Rudin & Osher 1994 Venkatakrishnan et al.
2013 Delbracio et al. 2021a Romano et al.) 2017 Ongie et al.) 2020
DDPMs have been recently adapted
for unsupervised model-based image restoration (Kawar et al. 2021a
2022 Kadkhodaic & Simoncelli 2021
Jalal et al. 2021a Laumont et. al. 2022
Chung al.
2022
Kawar et al. 2022).
How this approach compares to Denoising Diffusion. Denoising Diffusion Probabilistic Models
(DDPMs) (Soh--Dickttein et al., 2015 Ho et al.] 2020 Song et al.) 2021a) and Score-based models (Song
& Ermon 2019 2020 Song et al. 2021bb have emerged as two powerful classes of generative models that produce high-quality samples by inverting a known diffusion (degradation) process. The standard Gaussian
denoising formulation has been extended to more general corruption process (Bansal et al. 2022 Hooge-
boom & Salimans, 2022 Daras al. 2023
Deasy al. 2021 Hoogeboom et al. 2022a b Nachmani et al.
2021 Johnson et al. 2021 Lee et al.
2022
Ye et al.
2022). The main common idea is to analytically define
known degradation process that is reversed to generate new samples starting from fully degraded image
(e.g., pure noise). The inference procedure makes use of the known analytical degradation at every step.
A similar statement is true for other p p 2 which case the mean replaced by another aggregation operator (e.g. median
for I.))
2

---

Published in Transactions on Machine Learning Research (06/2023)
By contrast, in our formulation we do not require knowledge of any analytic form of the degradation process, we directly learn an iterative restoration process from low-quality/ /hihh-uuaitty paired examples. This implies that we can apply our iterative procedure to virtually any degradation as long as we are given image pairs. Additionally, our formulation is motivated only from the idea of splitting the original inverse problem into multiple smaller ones. We do not require any knowledge of the underlying probability distributions, or the conditional distributions, at any step. Our inference procodure is solely based on the idea of restoring the signal a little bit at each step. This approach, with minimal assumptions, gives a unified formulation for any supervised image restoration problem under the same framework.
The natural extension of DDPMs to image restoration tasks is through the use of a Conditional DDPM models (cDDPM) LLi et al. 2021 Saharia et al.) 2021 2022 Whang et al. 2022). The goal of a cDDPM is to generate plausible reconstructions given the low-quality imput (e.g., by generating samples from the posterior distribution). The idea is to train a supervised denoising diffusion model using paired examples that is conditioned on the low-quality input. The denoising network learns to generate a valid restored image (sample) by repeatedly denoising an initial image of pure noise. Our formulation has some similarily to conditional diffusion models, but contrary to the denoising formulation, we directly proceed by iteratively restoring the input image.
Overall, our method is forward evaluate the formulation on four different restoration tasks using different perceptual quality metrics. As shown, our method produces samples of higher quality than the state-of-the-at regression formulations while maintaining sigh-fidelity with respect to the original sample.
forward to implement and train, and produces high-quality results. We
3 Related Work
The goal of image restoration is to generate a high-quality image from its degraded low-quality measurement
(e.g., low-resolution, compressed, noisy, blurry). Since the seminal nuper-resolution work of Dong et al. (0015)
many recent. image restoration methods adopt an end-to-end supervised formulation where a deep neura
network is trained to directly produce a point estimate (Zhao et al. 2016 Lim et al. 2017 Tao et al. 2018
Chen et al.j 2018) These methods rely on low-quality high-quality image pairs to train a regression model.
Most of the work has been focused on developing better and more powerful network architectures (Zamir et al.) 2022 Chen et al., 2022 Tu et al.) 2022, Zamir et al. 2021) so we can achieve better pixel-level roconstruction. While this formulation lcads to state-of-the-art PSNR, the image generated is at best an average of all plausible solutions (regression to the mean). In the limit case where the low-quality image is completely obfuscated the best prediction in terms of PSNR is the average of the distribution.
Arjovsky et al. 2017), and adversarial formula- 2018 2019) have been introduced to push the generated image towards the manifold of natural images. GANs suffer from unstable training (Arora et all, 2017 Salimans et al. 2016 Arjovsky et al., 2017], while being prone to introduce significant image hallu- cinations. This is direct consequence of a non-reference formulation that directly tries to minimize the distance of the range of the generator to the manifold of natural images (Cohen et al. 2018).
Generative adversarial networks (Goodfellow et al. 2014 tions (eedig el. al.] 2017 Isola el al. 2017 Kupyn et al.
Blau & Michaeli (2018) proved that there is a trade-off between image perceptual quality and distortion. It is not possible to minimize both distortion and perceptual quality simultaneously. In fact, minimizing the average point distortion (e.g., PSNR) can be only done in detriment of the perceptual quality (Blau &
Michaeli 2018 Freirich et al. 2021).
A powerful way of avoiding the regression to the mean is to formulate the problem
of sampling
from the posterior distribution (Kawar et al. 202Tb 2022 Ohayon et al. 2021 Kadkhodaie & Simoncelli,
2021
W Whang et al. 2022). An additional benefit of this formulation is to be able to generate multiple
different plausible solutions that can be used for uncertainty quantification [Whang et al. 2021 or improving
fairness (Jalal et al. 2021b
Variational auto-encoders (Prakash et al.) 2020), Normalizing flows
Lugmayr et al. 2020
2021 and Dif-
fusion probabilistic models (DPMs) (Saharia et al. 2021 Li et al. 2021 Whang et, al.
2022 have been
3

---

Published in Transactions on Machine Learning Research (06/2023)
successfully applied to different image restoration tasks, where diverse set of candidates can be generated from the learned posterior (Prakash et al. 2020).
Denoising Diffussio Probabilistic Models (DDPMs)
(Soh-DDicsttiin et al.
2015 Ho et al.
2020
Song et al.
2021a), Score- based models (Song & Ermon 2019
2020 Song et al. 2021b
and their
recent generaliza-
tions (Bansal et al., [022 Hoogeboom & Salimans, 2022
Daras et al.
2023
Deasy et al.
2021
Hoogeboom
et al. 2022a bJ Nachmani et al. [ et al. 2021 Lee et al.
2022 Ye et al. 2022) generate high-
quality samples by inverting known degradation process. The main strategy is to analytically define a
known degradation process that is reversed to generate new samples starting from fully degraded image (o.g., (urg noise). [ansal et al. (002)) introduced Cold Diffusion a generative framework to generate images by reverting arbitrary (known) degradations. They show promising results even with non-stochastic degra- dations such as blur, masking or pixelization. The strategy is to define intermediate analytical degradations (diffusion) and then revert them step by step. Daras et al. (2023) presented Soft Diffusion generalization of diffusion models to linear degradations. The authors argue that noise is a fundamental component that is needed to provably learn the score.
In this work, we adopt similar strategy and propose to decompose the image restoration problem into sequence of intermediate steps each of them being much casier problem to solve (i.e., less ill-posed). This path of intermediate reconstructions takes us from a low-quality input to a high-quality reconstruction through a series of slightly less corrupted signals. Different than in traditional generative diffission formula- tions, the degradation is only implicilly given through a series of pair images (low-quality and high-quality). In our formulation, we define the intermediate steps as a convex combination of the parge//inpus signals. This induces simple linear propagation from the high-quality sample to the low-quality one.
An allernative formulation of the supervised image restoration problem is to use a conditional denoising diffusion models to generate samples from the posterior distribution (Li et al 2021 Saharia et al.) 2021 2022 Whang et al. 2022h. The overall idea is to train a denoising diffusion model that is conditioned on the low-quality input. The denoising network learns to generate a valid restored image by repeatedly denoising an initial image of pure noise. Our formulation has some similarity to conditional diffusion models, but contrary to the denoising formulation, we directly proceed by iteratively restoring the input image.
The very recent work by Luo et al. (2023a) and Welker et al. 2022). and the concurrent work by Luo et al. (003bb); Song et al. (2023) introduce related image restoration techniques based on ODE/SDE diffusion formulations. A major difference with our work is that InD1 is completely motivated and formulated from elementary principles by splitting the restoration tasks into multiple smaller ones. There is also significant amount of recent related work that analyzes the connection of diffusion image generat ion with Bridge and Flow matching, Optimal Transport and Schrodinger bridges (Albergo et al. 2023 Shi et al., 2023 Liu et al., 2023). Finally, the concurrent work of Heitz et al. (2023) adopts a linear diffusion scheme for image generation similar to, but less general than, the one in InDI.
4 InDI: Our Proposed Formulation
Given (Ir, y) ~ p(xx, y); we define a continuous forward degradation process by
at (1 f)a + ty, with & [0,
(1)
The idea of this forward process is that it starts from a clean sharp image at time t = 0, and then degrades it to the blurry/noisy observation at time t = 1. Here, o, indexed by t, represents an intermediate degraded image between the low-quality input y (i.e., L = 1) and the high-quality sharp target æ (i.e., L = 0). Following the common notation in diffusion models, we will refer to the index as the time-step.
Our recovery method starts with the input degraded image (time generates the best possible reconstruction at time L 1-a 5. This can be done, for example, by the short-time conditional mean h. E/ ax-s 2x) to the estimate. As we will show, by repeating this process we can thus invert the full degradation little by little. The following proposition provides the cornerstone of our approach.
1), and then at a given timo-step
4

---

Published in Transactions on Machine Learning Research (06/2023)
15
35
05
regrassion
455
05
-15
(a)
(b)
Figure 1: 2D Toy Example. Estimation of conditional mean and iterated estimation for points from multimodal (4 modes) distribution under: (a) Denoising strong Gaussian noise (H I); and (b) missing information recovery, i.c., H = [1, 0, under moderate noise. Blue points represent observed samples, while red ones are the regression prediction. The black (hollow) circles represent the final point in our iterative procedure, always reaching a valid point in the data manifold (orange points). The small green circles indicate the iterative restoration path.
Proposition 4.1. Let Bs xt be given from equation 7 where S l. Then,
x' xc (1 (1-
8
E æo xx + 8 7
The proof is a direct consequence change of variables and is given Appendix A
According this proposition, the posterior mean (e.g. MMSE estimate) at time the estimate at time by first estimating the clean image (æ 2o), and then doing convex combination with the estimate 4, at time s. We can then apply the following scheme to move from to = =t
be deduced from
8 tt (1- dc.
④._5 = E Dt- 2x
E[
(2)
The process starts from à and the step < 1 controls the "speed" of the reverse process (e.g., at constant "speed", t where N controls the total number of steps).
Remark 4.2. For the iteration procedure in equation to be well defined we require that El Zo it is well defined. Thus, we require Pz, (f:) 0.
An intuitive motivation for the requirement in Remark 4.2 that we need to move through a path of plansible samples & at every step One simple way guarantce this is by adding a small amount of noise y. Then p(y); and therefore p(æ;) will be non-zero everywhere. More discussion about this presented at the end of this section, but first, present toy example to motivate approach.
A Toy Example: Let us assume observe noisy samples H + n, where n ~ NOO IdId), drawn from a discrete multimodal distribution p(x) = Σi wrdz-c, where C; E R and w, and Σ w, 1. Let C; be the intermediate degraded samples according to equation I In this simple example, there is closed form expression for all posterior distributions and conditional means; namely,
p(x,(a)) G (
H2
Hic, 0+
and p(x.) Σ w,G
(3)
a+
1=1
where H, = - 1)T + IH and G(æz) is a Gaussian kernel with identity covariance and la. Then,
p(x)(=))p(x) p(x:)
xdx 1 C,w,G
Σd
m. H.C.
E Ex-p(m|mn)
Σd WyG @: H.C..c

---

Published in Transactions on Machine Learning Research (06/2023)
The iteration from equation 2
becomes:
L D C,w;G
( 8 t de.
oe
(1-
d H.HHi +
E W.G
Figure | shows the results of applying the iterative regression scheme given by the above equation in two dif- ferent examples. The ilerative regression converges one of the lour possible modes (shown in orange), while the regression to the mean is always weighted average of all possible modes (i.e., blurry reconstruction, shown in rod).
Training and Inference: InDI ideal iterative scheme given by equation 2 requires to compute an estimate of the clean image at every step t (i.e., E| æo xe D). To that extent, we train a family regressors F= each of them specialized in reconstructing ay from æt at a given That is,
min E Eawppl(a Be Fo t) 2 22lp-
(4)
where p(t) is a predefined distribution for & (e-g,, uniform). The model Fg allows us to do incremental reconstruction where from time step t we predict the slightly less corrupted signal at time t as given in equation 2 Thus, the iterative scheme becomes:
c - Polér, + 71- Er.
(5)
where 0 8 S 1. Although 8 could be a function of time, practice we use a constant time step, 5 N where N is the number of steps.
leads to ordinary differential InDI Residual Flow ODE: In the limit, as 8 + 0, equation 5 equation (ODE); namely
dx: lim x1-6
2t (x.,,)
(6)
8-50
where in the ideal case t) Ej xu wz). The ODE can be interpreted the right-hand side is the (normalized) residual of the inversion process at time t. We are interested in the solution of this equation at t = 0, starting from the initial condition #1 = y at t = 1. The residual flow formulation can be used to develop other mumerical procedures using standard ODE solvers. Exploring this is left to future work.
"residual flow" because
Another use of the continuous formulation is understand the behavior of the proposed iterative procedure in terms of concrete examples. In Appendix B we show how the residual flow can be used to analyze the specific case where the prior is Gaussian and the restoration task is denoising.
Connection to Denoising Score-Matching and Probabilistic ODE: An interesting connection emerges when the degradation is Gaussian noise (standard deviation o²). In this case, InDI's ODE in equa- tion boils down to the score-matching probabilistic ODE of Song et al. (2021b). More specifically, let C; + tn, so the noise level in I4 is of too?. The probabilistic flow ODE (Eq(13) Song al.
(002bb) is given by
dx; 1 dd V lo t log);(x;).
dt 2 dt
According to the denoising score-matching (DSM) approximation (Vincent 2011
-V log (xx)) s St Fo(a..
Tt. Fo(as,)) t)
of
1202
so we end up recovering preciscly the same ODE as in InDI. Furthermore, when Fo(x,, t) = E[ x x; (i.c.. MMSE estimator) the DSM approximation is exact and the relation is given by Tweedie's formula (Robbins) 1956
Elron 2011

---

Published in Transactions on Machine Learning Research (06/2023)
Algorithm Iterative Image Restoration (Inference)
Require: (,(,,))),, n ~ I0,
1 = y + ein
for to 0 with stop - do
~~ I0,
2 ie + (a 1) (1 (1- iz. (1- 0)
8 v)
E7
Update rule from equation 10
end for return L0
Stochastic Perturbation: To make sure we have the regularity requirements for the iterative procedure from equation 2 to be well defined (Remark 4.2), we add a small amount of white noise to the low-quality input. As shown in Section 5 this leads to a significant improvement in image quality in certain tasks (in particular those that are restorations from deterministic degradations).
Our model with this noise perturbation becomes:
3 = - 1) + ly' = (1 - 1) + by + len with E 1],
where y' = y en, is a small constant (e.g., 0.01, where image values are in [-1, 1)), and n NO, Id). A slightly more general formulation incorporates the perturbation general Brownian motion, where we can explicitly control the level of noise at each step. That is,
x; = (1 - t)x + ty + Vien with t E [0,
where e, is non-negative function, and 77, is the standard Brownian motion having mean and covariance tI at index t.
In this more general setting, the base training objective becomes
min Eg wv Fn- ((1 1)a: ly te,n;)) arllp
(9)
And as result, the general inference procedure in equation 2 bocomes:
8 5 - F) + (1 (1- 0 &, (t- (8) &CC
(10)
where the reconstruction process starts from t = 1, 21 = y + en and n ~ (d, At each step, a new Ç ~~ (0, Id) is sampled and noise is added to the current state. The added Gaussian noise such that the noise at time has variance f2G² as required by equation g. To be well defined 6 needs to be non-negative non-increasing function of t. In the limit case where &, e we in the simplified case given by equation 7 the noise perturbation is pure Brownian motion.
while if e, Vi
Our full iterative restoration inference scheme is given Algorithm
5 Experiments
We train and evaluate our framework four widely popular image restoration tasks: motion deblurring, defocus deblurring, compression artifacts removal and single image super-resolution. Our formulation is generative-based, and we show that can be used for image generation even if this is not the main focus of the present work. To evaluate the quality of the proposed method we compute several distortion based and perceptual metrics: PSNR, I.PIPS (Zhang et al. 2018). FID (Fréchet Inception Distance) (Heusel et. al. 2017), and KID (Kernel Inception Distance) (Binkowski et al. 2018
Perception- Distortion tradeoff (Blau & Michaeli, 2018): To illustrate the potential of the method, we present results when using different number of steps for the reconstruction. This has direct impact

---

Published in Transactions on Machine Learning Research (06/2023)
the perception ciccppiiondditortioon distortion tradcoff. In general, single step reconstruction with our model, will lead to estimate that minimizes the average point distortion (e.g., PSNR) but this can be only done to the detriment of the perceptual quality.
Model Architecture and Training: We adopt a U-Net-like architecture similar to the ones diffusion strategies (Saharia al., 2021 Whang al. 2022). Following Whhang et al. (2022) we removed attention layers and group normalization to have a fully-convolutional architecture. The size of the model varies for each evaluated task (in general chose model size proportional to the size of the dataset to avoid significant overfitting). The model is trained image crops using ADAM optimizer. Learning rates and other hyper-parameters are given in Appendix C For each experiment, we train a single model Fo(:;)) t) that is conditioned on the parameter The model is trained using the loss function of equation with = 1. We found that the distribution of t, p(t) plays an important role. Section 6.3 analysis of impact.
we present an empirical
5.1 Motion Deblurring
Motion deblurring a very challenging restoration task. Motion intrinsically random in the sense that a priori don't have known degradation model. The current best end-to-end deep learning solution is to train regression models using paired data sharp, blurry frames. One of the most adopted training datasets is the GoPro motion deblurring dataset (Nah et al. 2017) containing 3214 pairs of clean and blurry 1280 720 images (1111 are reserved for evaluation). The blurry frames are generated by recording high-frame rate video clips and then averaging consecutive frames to simulate blurs caused due to longer exposure. We follow the standard setup (Nah et al.) 2017 Kupyn et al. 2019 Chen et al.) 2021a, Cho et al.) 2021 Suin et al. 2020 Zhang et al. 2019) and perform training data augmentation with random horizontal/ /verticall flips and 90/180/ 180/270 rotations. We did not introduce additional noise to the blurry inputs (c 0 in equation
Input
Restormer
Maxim
DvSR
Ours
Reference
Figure Examples of deblurred images from GoPro dataset. Our iterative reconstruction leads achieves better reconstruction of detailed textures than regression based models (Restormer, Maxim) and similar quality than conditional DPMs (DvSR). More results provided in Appendix.
Figure D shows visual comparison our itcrative image restoration and current statc-of-tho-art deblurring models. The iterative scheme produces images with much more details than regression based solutions (Restormer (Zamir al. 2022 MAXIM (Tu et. al. 2022 Our results similar to the ones generated by
8

---

Published in Transactions on Machine Learning Research (06/2023)
Table 1: Image motion deblurring on the GoPro Nah et al. 2017 dataset.
Best values and
second-best values for each metric are color-coded. KID values are scaled by a factor of 1000 for read- ability.
Perceptual
Distortion
LPIPS4 NIQEJ FIDA KIDJ PSNR1 SSIM1
Ground Truth
0.0
3.21
0.0
xc
1.000
HINet Chen et al. 2021a
0.088
4.01
17.91
8.15
32.77
0.960
NeR Zamirr al. 2021
0.089
4.09
20.18
9.10
32.66
0.959
MIMO-UNeL- +
Cho el al. 2021
0.091
4.03
18.05
8.17
32.45
0.957
SAPHNet (Suin al. 2020
0.101
3.99
19.06
8.48
31.89
0.953
DeblurGA Nv2 (Kupyn et al.
2019
0.117
3.68
13.40
4.41
29.08
0.918
DvSR (Whang et al.
2022)
0.059
3.39
4.04
0.98
31.66
0.948
DvSR-SA Whang
al. 2022
0.078
4.07
17.46
8.03
33.23
0.963
Restormer
Zamir el al 2022
0.084
4.11
19.33
8.78
32.92
0.961
MAXIM
Tu et al. 2022
0.087
3.94
22.76
10.06
32.86
0.962
NAFNet
Chen al. 2022
0.078
4.07
17.87
8.27
33.71
0.967
Ours (10 steps)
0.058
3.32
3.55
0.56
31.49
0.946
Quantitative results on the GoPro dataset are presented in Table [ The proposed iterative reconstruction procedure achieves a new state-of-the-ar performance across perceptual metrics while maintaining competitive PSNR to existing methods.
current conditional diffusion models (DvSR (Whang et al. 2022
18
16
14
10
ctdps
Seterence
49
31.50 32.25 32.50 32.75 33.00 PSNR
Figure 3: Number of steps. The total number of steps in the iterative regression has direct impact on the quality. The number of steps seems to control the Perception-distortion tradeoff. One step leads to the best possible MSE reconstruction (minimum distortion) but large perceptual discrepancy.
Number of steps. Figure shows the impact of the mumber inference steps on the Perception Perception-Disoortion Distortion trade-off (Blu Michaeli 2018). While doing a reconstruction on a single step (e.g., direct regression) produces the best PSNR, the perceptual metrics are significantly improved when the number of steps is larger than one. Both metrics can't be optimized simultaneously (Blau & Michaeli, 2018
5.2 Single-Image Super-resolution
We evaluated the iterative restoration methodology on single-image 4x super-resolution on the div2k dataset (Agustsson & Timofte 2017). This dataset contains 1000 2Krrsolution images (800 for train- ing, 100 images for validation, 100 testing). We compare to other state-of the art models that span from regression models having powerful architectures (Wang et al.] 2018 Chen et al.] (021b) Liang et al. 2022 and/or or generative formulations: GAN based, i... LDL (Liang et al. 2022), ESRGAN (Wang 2020
2018
BSRGAN (Zhang et al. 2021 and also based Normalixing Flows, SRFLOW (Lugmayr et al.
9

---

Published in Transactions on Machine Learning Research (06/2023)
Input
RRDB
SRFIow
LDL
Ours
Reference
BRSC AM
: 9
M
H =
P BJIB SS0BTIS
Figure 4: Examples image 4x upscaling. Our iterative reconstruction leads achieves better reconstruction of detailed textures than RRDB regression model (Wang el. al. 2018), less high-frequency art artifacts than SRFlow (Lugmayr al. 2020) generative normalizing flow, and comparable visual quality than LDL (Liang et al. 2022 state-of-theearl customized generative adversarial model. More results are given in Appendix.
Figure b() summarizes the quantitative results on 4x SR div2k validation dataset, Figure shows a selection of results. Our proposod framework leads to upscaled images with more defined structure than regression based formulations produeing larger PSNR, e.g., RRDB (Wang et. al. 2018). The recently introduced adversarial formulation LDL [Liang et al.) 2022 produces slightly better fine grain details. This could indicate that in the situation where there is limited training data, careful adversarial formulation may be more data efficient.
The importance of adding noise deterministic super-resolution. In our formulation of super- resolution, the degradation deterministic linear (blurring plus subsampling) operator. Figure 5(a) shows the importance of adding small amount of noise to the input image. Directly applying the original iterative procedure (without adding noise to the input) leads to blurry reconstruction (high PSNR but low FID score, 0.0 in Figure 5 (a)). Adding small amount of noise (e > 0 in Figure 5(a) leads to significant better results in terms of perceptual quality (e.g., FID score).
5.3 Defocus deblurring
Defocus deblurring the task of reducing the blur due to limited depth-of-field or misfocus. For such purposes used the Canon dual-pixel (DP) defocus dataset. (DDPD) provided by Abuolaim & Brown
10

---

Published in Transactions on Machine Learning Research (06/2023)
60
Type PSNRT LPIPS)
FIDI
KIDI
==0.0
= 0.005
2
50
== 0.015
(Dw(SwinIR)
Liang et
2022
GAN 26.64
0.102
10.80
0.400
= 0.030
(DL(RRDB)
Liang
2022
GAN 26.43
0.109
11.36
0.582
ESRGAN (Wang et al.
2013 8
GAN 25.54
0.125
12.77
0.732
SRFLOW Lugmayr
2020
NF 26.10
0.127
20.21
4.444
BSRGAN Zhang al.
2021
GAN 24.35
0,241
31.43
5.229
RRDB (Wang et al. 2018
REG 28.37
0.264
50.98
19.000
500200100
LIIF Chen al. 2021b]
REG 28.15
0.269
53.16
19.958
BSRNET (Zhang al. 2021
REG 25.90
0.348
84.11
36.901
$9.55
Ours (100 steps, = 0.015)
26.45
0.136
15.39
1.761
PSNR
(a)
(b)
Figure 5: 4x Super-resolution on div2k dataset second-best values for each metric are color-coded
Agustsson
& Timofte 2017
Best values
and
2 and train defocus deblurring model only using single image input (i.e., we don't use the dual-pixel images given in the dataset). The DDPD dataset contains 1000 pairs sharp and blurry 6720 x 4480 images, of which 30% are reserved for validation and testing. The blurry/shan frames are generated by capturing two consecutive snapshots by changing the camera parameters (lens aperture). high frame rate video clips and then averaging consecutive frames to simulate blurs caused due to longer exposure.
Input
step
steps
steps
Reference
Figure 6: Examples of restored images from defocus DDPD dataset. The iterative reconstruction leads to images with more texture compared to the direct regression (1 step). More results are provided in Appendix.
Figure D shows a visual comparison of our iterative image restoration when different number of inference steps is used. Increasing the number of steps has a direct impact on the quality of the result. Quantitative results the DDPD dataset sunmarized in Table 3 in Appendix. As in the other experiments the best
11

---

Published in Transactions on Machine Learning Research (06/2023)
PSNR is obtained with single step (direct regression), while the best perceptual metries obtained when the restoration is done in multiple steps. We did not introduce additional noise to the blurry inputs (e 0 in equation 7
5.4 Compression artifact removal
JPEG compression introduces blocking artifacts and lack high-frequency details. We evaluated the pro- posed method on the task of removing strong JPEG compression artifacts (quality factor 15). To generate the training data
the 1000 div2k high-quality images Agustsson & Timofte, 2017 We evaluated
the model div2k validation set.
JPEG Q1S
step
20 steps
100 steps
Reference
Figure 7: JPEG Compression artifacts removal (quality factor 15). As more inference steps are used more details are generated.
Figure 7 show some visual results of restored images with the model applying a different number of steps. As more inference steps are used the restored images have more details. More results on JPEG compression removal are discussed in the next section.
6 Discussion
6.1 A generative framework
A natural question to ask is whether the proposed approach is also generative in the spirit of diffusion formulations. Namely, if we take our formulation to the limit where the low-quality image is fully degraded, then could we potentially generate new samples from scratch?
To test the idea we trained a restoration model that starts from pure Gaussian noise paired to a 64 64 cclebA image and then proceed as described above. Figure K shows some generated samples with this formulation. The generated samples have FID=9919, which is not faaeeoff-hheardf but illustrate the point. In this specific case, our proposed methodology leads to similar denoising training loss the one in DDPM (Ho) et a.. 2020 Despite this similarity, the two methods come from different motivatioons/formllaiio therefore have different inference strategies. We didn't fine-tune architecture or hyper-parameters to boost
and
2Tt. competitive with her methods from couple years ago
12

---

Published in Transactions on Machine Learning Research (06/2023)
Figure 8: Examples of CelebA 64 x 64 generated samples (FID=9.19, 150 steps). Our iterative reconstruction can be potentially used to generated samples from fully-degraded image (noise).
the performance since our goal was to present the idea and show that the formulation, at its core, can be generative as well.
6.2 Comparison of Inference Algorithms
In what follows we discuss different alternatives for recovering the clean sample with the trained models.
Naive Procedure and Relevance Cold Diffusion: Given equation ! one may be tempted to directly replace the clean image by the current estimate. This would lead to at (1 Fo 1) ty, and the inference iterative rule would become
i (1 1 ( F)
8)y.
(11)
Cold Diffusion (Bansal et al. D(x, s), where s controls the strength. Our formulation more general in the sense that we don't 't require an explicit knowledge of D. To apply Cold Diffusion sampling our context, we define D 8) = (1 155-s). s) + sy
2022 proposes to generate images by inverting an arbitrary known degradation
(given by equation D This leads to Cold Diffusion's 's naive sampling (Algorithm in Bansal et al. (2022) )
- D(Fo (āt,, f),
1 + Fo (t 8)y-
(12)
Note that this sampling scheme is the same as the one in Eq. 11
Cold diffusion improved sampling (Algorithm
2 in Bansal et al. (2022) is given by,
2,58 = ã. ( t), D( t)
(13)
è; - (1 1--t) F) (b,, ty + (1 (1-t+ Fa t)+ + (t -5y
(14)
c. + 5(Fo y).
(15)
In Figure 9 we compare our inference algorithm (equation 5), the naive inference algorithm (equation and our adaptation of the Cold Diffusion sampler to formulation (equation 15). In general, the naive sampler produces good results with very few steps (N=2,3) but then diverges. Our adaptation of Cold Diffusion sampler produces competitive results, while leading to slightly worse FID scores for the same distortion level than our proposed algorithm. In the limit, as the number of steps becomes very large, Cold Diffusion sampler seems to converge to a stable point, while ours after a certain large number of steps, deteriorates.
Figure g shows the FID score of CelebA 64x64 generated images when using the three different variants of the inference algorithm (sampler).
13

---

Published in Transactions on Machine Learning Research (06/2023)
20.0
60
naive cold diffusion proposed
naive cold diffusion proposed
17.5
50
40
10.0
30
10
7.5
5.0
100
20
20
100
50
10
1000
100
10
%
30.5
31.0
31.5
32.0
32.5
33.0
25.0
27.0
27.5
28.0
PSNR
PSNR
(a) Motion Deblurring (GoPro dataset)
(b) Upscaling 4x (div2k dataset)
140
60
naive cold diffusion
130
naive cold diffusion
proposed
proposed
110
50
100
90
2
1000
30
60
50
1000
2015
20
10
100
10
27.5
28.5
29.0
29.5
30.0
30.5
2
8
16
32
64
128
256
512
PSNR
(c) De-Jpeg Q15 (div2k dataset)
(d) Generative imaging CelebA 64
Figure 9: Impact of the Inference Algorithm for some of the different tested tasks. The naive inference algorithm (equation 11 produces a good bascline when used with very few steps (i.c. 2-5) but then di- verges. The proposed updated rule (equation 5) produces better results than the one adapted from Cold Diffusion (Bansal al.] (2022) given in (equation 15). At very large number of steps, Cold Diffusion seems to be the most stable of the three tested samplers. Each of the points in cach shown curve represents the results with different nunber of steps (similar to what is shown in Figure 3 Points that leads to values that are out-of-the shown region are left out for improving visualization.
6.3 Impact of distribution p(t)
The impact of the distribution of used during training has a clear impact on performance. We evaluated several different options that are summarized in Figure 10 (a). Figure I0 (b) shows the results when different distributions are adopted. The best results are obtained when the model is trained with a bias towards (more degradation). Intuitively, this could imply that the iterative procedure needs to be more certain of the direction to move at the very early steps of the procedure. Nonet the best distribution can depend a combination of model capacity and restoration task so we are not drawing general conclusions.
6.4 Impact of adding noise inverting deterministic degradations
JPEG compression is a non-linear, but deterministic, degradation. We empirically verified that adding a small amount of noise helps to improve the results as shown in Figure II We tested the variant of the inference algorithm that adds noise each step (so the noise bocomes Brownian motion, e.g., G == c/v)). and adding constant noise level at the initial step (a c). We did not observe any practical difference the two approaches.
14

---

Published in Transactions on Machine Learning Research (06/2023)
key
p(t)
description
'linear_o 0'
r- 100 1]
Uniform distribution
'linear_a
t~ (1 - a] + ab1, where <
Uniform distribution w/bias to
'bias_th tt
t= g(s), ~00 [ 1] and g(s) = sin (sTT22)
Sine based distribution (bias to t 1)
'bias_to t
== g(s), - 1] and g(s) sin ((+ 1)x (2)
Sine based distribution (bias to L 0)
'bias_to tt_ t1
t= = g(s), with 20,0 1] and g(5) = sin (sT/2) 2
Sine based distribution (bias to L 1)
(a) Different evaluated p(() training distributions.
0.0
linear 0.0
blas 11 blas, 10_
linear 0.5
blas 10
llnear
60
linear_1.
0.0
linear_ 1.0
linear: 2.0
linear_2.0
08
40
30
50
20
0.2
20
27.5
28.5
29.5
30.5
0.2
0.4
(b) PSNR FID plot for different p()).
(c) Visualization of p(t) cumulative distributions.
Figure 10: Impact of the distribution of p(t) during training. Results for JPEG Compression removal (Q-15). The best results are obtained when the distribution of & is biased towards 1== 1.
0.22
60
cld
eif)
0.005
0.030
0.015
a(t) - 0.015KF
0.030NZ
0.20
400
200 0.000
0.0015
0.0
0.030
ett)=
k(t) 0.030/VE
60t) 0.015V
50
0.18
10
15
40
15
LPPS
0.16
50
200
100
0.14
0.12
20
27.5
28.0
28.5
29.0
29.5
30.0
0.187.5
28.0
28.5
29.0
29.5
30,0
PSNR
(b)
Figure 11: Impact of adding noise to the low-quality input in JPEG compression removal.
6.5 Comparison to Conditional Denoising Diffusion Model
We compare InDI to a vanilla conditional DDPM (Ho et al. 2020 We trained a vanilla conditional DDPM, using the (continuous) noise level an additional input, similarly as done in Saharia et al. (2021); Whang et al. 2022). The model architecture is the same as in InDI but the auxiliary noise image (needed in any DDPM) is concatenated with the low-quality input at each step. Figure 12 shows a comparison between InDI and the conditional DDPM. To generate the DDPM plot we merged several possible noise schedules using different nunber of steps that span the perceplion-distortion tradeofT. InDI produces comparable results using significant less number steps than the vanilla DDPM.
7 Conclusions and Limitations
We presented novel formulation of image restoration that circumvents the regression-to-thl mean problem. This allows to get restored images with superior realism and perceptual quality, while still having low
15

---

Published in Transactions on Machine Learning Research (06/2023)
70
conditional DDPM
InDI
conditional DDPM
InDi
0.16
0 40
LPPS
0.14
100
500
500
2000
10
2000
500
50
2013
10
0.10
500
2025
27.5
28.0
28.5
29.0
29.5
30.0
305
PSNR
(b) PSNR VS FID.
(c) PSNR vs LPIPS.
Comparison of InDI to conditional Denoising Diffusion Probabilistic Model (DDPM). The DDPM requires a noise schedule for inference. The plot showed herein was built by fusioning six different noise schedules taking the best PSNR vs FID score at a given number of steps. Our proposed InDI algorithm produces comparable results in much less number of steps.
Figure 12:
distortion error. Our method is motivated by the observation that restoration from a small distortion is a better-conditioned problem. We therefore break a restoration task into many small ones each of them easier (and less ill-posed) than the larger problem we solve overall. This enables our iterative approach transforming the degraded image into a high-quality image, spirit similar to current generative diffusson models.
Limitations. The present formulation is supervised one, requiring paired training data. As such, for each type of degradation we need to train specialized model, in contrast to unsupervised formula- tions such as RED (Romano et al., 2017), PnP (Venkatakrishnan et al. 2013 Kamilov et al. 2022 or DDRM (Kawar et al. 2022). Additionally, given the dependence to paired training data, its performance for out-of-distribution samples is not guaranteed. This question requires more in-depth analysis. Finally, while the proposed itcrative inference algorithm produces high-quality restorations, in some tasks performance degrades after a certain number of steps. This likely due the accumulation of errors, and will likely require more robust inference scheme.
For future work, we would like to better characterize the limiting points of the proposed inference procedure. Other possible research avenues are developing robust. formulations that can successfully handle out-of- domain input.
Acknowledgments
The authors would like to thank our colleagues Jon Barron, Tim Salimans, Jascha Sohl-diccstein, Ben Poole, José Lezama, Sergey loffe, and Jason Baldridge for helpful discussions.
References
Abdullah Abuolaim and Michael S Brown. Defocus deblurring using dual-pixel data. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part X 16, pp. 111-126. Springer, 2020.
Eirikur Agustsson and Radu Timofte. Ntire 2017 challenge on single image super-resolution: Dataset and study. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, July 2017.
Michael Albergo, Nicholas M Boffi, and Eric Vanden- Stochastic interpolants: A unifying frame- work for flows and diffusions. Xio preprint arXim:2303.0077 2023.
16

---

Published in Transactions on Machine Learning Research (06/2023)
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pp. 214-223 223. PMTRR 2017.
Sanjeev Arora, Rong Ge, Yingyu Liang, Tengyu Ma, and Yi Zhang. Generalization and equilibrium in generative adversarial nets (GANs). In International Conference Machine Learning, pp. 224 2322. PMLR, 2017.
Muhammad Asim, Fahad Shamshad, and Ali Ahmod. Blind image deconvolution using deep generative priors. IEEE Transactions on Computational Imaging, 6:1493 1506, 2020.
Arpit Bansal, Eitan Borgnia, Mong-Min Chu, Jie Li, Hamid Kazemi, Furong Huang, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Cold Diffusion: Inverting arbitrary image transforms without noise. arXiv preprint Xiv:220. 09392,, 2022.
Mikolaj Binkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. In International Conference on Learning Representations, 2018.
Yochai Blau and Tomer Michaeli. The Perceptiion-Dittoriin TradeofT. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6228 6237, 2018.
Ashish Bora, Ajil Jalal, Eric Price, and Alexandros G Dimakis. Compressed sensing using generative models. In International Conference on Machine Learning (ICML), pp. 537 546. PMLR, 2017.
Chen Chen, Qifeng Chen, Jia Xu, and Vladlen Koltun. Learning to see in the dark. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 3291 3300, 2018.
Liangyu Chen, Xin Lu, Jie Zhang, Xiaojie Chu, and Chengpeng Chen. Hinet: Half instance normalization network for image restoration. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 192-192,400 June 2021a.
Liangyu Chen, Xiaojie Chu, Xiangyu Zhang, and Jian Sun. Simple baselines for image restoration. In Com- puter Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part VII, pp. 17-33. Springer, 2022.
Yinbo Chen, Sifei Liu, and Xiaolong Wang. Learning continuous image representation with local implicit image function. In Proceedings of the SEEE/CVE conference on computer vision and pattern recognition, pp. 8628 8638, 2021b.
Sung-Jin Cho, Sco- Won Ji, Jun-Pyo Hong, Scung- Won Jung, and Sung-Jca Ko. Rethinking coarsc-to- fine approach in single image deblurring. In Proceedings of the IEEE/CV International Conference on Computer Vision (ICCV), pp. 4641 4641-4650, October 2021.
Hyungjin Chung, Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye. Improving diffusion models for inverse problems using manifold constraints. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
Joseph Paul Cohen, Margaux Luck, and Sina Honari. Distribution matching losses can hallucinate features in medical image translation. In International conference on medical image computing and computer-assisted assisted intervention, pp. 529-536. Springer, 2018
Giannis Daras, Mauricio Delbracio, Hossein Talebi, Alex Dimakis, and Peyman Milanfar. Soft diffusion: Score matching with general corruptions. Transactions Machine Learning Research, 2023. ISSN 2835- 8856.
Jacob Deasy Nikola Simidjievski, and Pietro Liò. Heavy-tailed denoising score matching. arXio preprint arXiv:2112.09788, 2021.
Mauricio Delbracio, Ignacio Garcia-Dorado, Sungjoon Choi, Damien Kelly, and Peyman Milanfar. Polyblur: Removing mild blur by polynomial reblurring. IEEE Trensactions on Computational Imaging, 7:837-848, 2.21a.
17

---

Published in Transactions on Machine Learning Research (06/2023)
Mauricio Delbracio, Hossein Talcbei, and Peyman Milanfar. Projected distribution loss for image enhance- ment. In 2021 TEEE International Conference on Compulational Photograhyy (TCCP), pp- 1-12. TREE Computer Society, 2021b.
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Image super-resolution using deep convolu- tional networks. IEEE transactions on pattern analysis and machine intelligence, 38(2):295 307, 2015.
Bradley Efron. Tweedie's formula and selection bias. Journal of the American Statisticnl Association, 106 (496):1602 1614, 2011.
Dror Freirich, Tomer Michacli, and Ron Meir. theory of the Distortion--errcptionn tradcoff in Wasserstein space. Advances in Neural Information Processing Systems, 34:25661 25672, 2021.
Lcon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pallern recognition, pp. 2414- 2423, 2016.
Tan Goodfellow Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Eric Heitz, Laurent Belcour, and Thomas Chambon, Iterative a-(de) blending: minimalist deterministic diffusion model. arXiv preprint arXiv:2305.0344 2023.
Martin Heusel, Hubert Ramssuer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by two time-scale update rule converge local nash equilibrium. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Newral Information Processing Systems, volume 33, pp. 6840 6851, 2020.
Emiel Hoogeboom and Tim Salimans. Blurring diffuuson models. ar preprint 4rX:::2009.055 2022.
Emiel Hoogeboom, Alexcy A. Gritsenko, Jasmijn Bastings, Ben Poolc, Rianne van den Berg, and Tim Salimans. Autoregressive diffusion models. In International Conference on Learning Representations, 2022a.
Emiel Hoogeboom, Victor Garcia Satorras, Clément Vignac, and Max Welling, Equivariant diffusion for molecule generation in 3d. In International Conference on Machine Learning, pp. 8867-8887. PMLR, 2022b.
Phillip Isola, Jun- Yan Zhu, Tinghui Zhou, and Alexei A Efros. image-to-image translation with conditional adversarial networks. In IEEE conference on computer vision and pottern recognition, pp. 1125 1134, 2017.
Ajil Jalal, Marius Arvinte, Giannis Daras, Eric Price, Alexandros G Dimakis, and Jon Tamir. Robust compressed sensing mri with deep generative priors. Advonces in Newral Information Processing Systems, 34:14938-14954, 2021a.
Ajil Jalal, Sushrut Karmalkar, Jessica Hoffmann, Alex Dimakis, and Eric Price. Fairness for image generation with uncertain sensitive attributes. In International Conference on Machine Learning, pp. 4721-4732.
PMLR, 2021b.
Danicl D Johnson, Jacob Austin, Rianne den Berg, and Daniel Tarlow. Beyond in-place corruption: Insertion and deletion in denoising probabilistic models. arxio preprinl arXin:2107.07071, 2021.
Zahra Kadkhodaic and Ecro Simoncelli. Stochastic solutions for linear inverse problems using the prior implicit in denoiser. In Thirly- Conference on Neural Information Processing Systems, 2021.
18

---

Published in Transactions on Machine Learning Research (06/2023)
Clugbek S Kamilov, Charles A Bouman, Gregery T Buzzard, and Brendt Wohlberg. Plug-and-play methods for integrating physical and learned models in computational imaging. arXiv preprint orXiv:2203. 17061, 2022.
Bahjat Kawar, Gregory Vaksman, and Michael Elad. SNIPS: Solving noisy inverse problems stochastically. In Thirly-Fifh Fifth Conference on Neural Information Processing Systems, 2021a.
Bahjat Kawar, Gregory Vaksman, and Michael Elad. Stochastic image denoising by sampling from the poste- rior distribution. In Proceedings of the International Conference Computer Vision (ICCV) Workshops, 2021b.
Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. In Advances in Newral Information Processing Systems, 2022.
Orest Kupyn, Volodymyr Budzan, Mykola Mykhailych, Dmytro Mishkin, and Jiří Matas. Deblurgan: Blind motion deblurring using conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8183 -83-8192,41 2018.
Orest Kupyn, Tetiana Martyniuk, Junru Wu, and Zhangyang Wang, Deblurgan-v2: Deblurring (orders-of- magnitude) faster and better. In The IEEE International Conference on Compuler Vision (ICCV), Oct 2019.
Rémi Laumont, Valentin De Bortoli, Andrés Almansa, Julie Delon, Alain Durmus, and Marcelo Pereyra. Bayesian imaging using plug & play priors: when langevin meets tweedie. SIAM Journal on Imaging Sciences, 15(2):701-737, 2022.
Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, An- drew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic single image super- resolution using a generative adversarial network. In IEEE conference on computer vision and pattern recognition, pp. 4681 4690, 2017.
Sangyun Lee, Hyungjin Chung, Jachyeon Kim, and Jong Chul Ye. Progressive deblurring of diffusion models for coarso-to-fine image synthesis. arXiv preprint ar 2xxx:2077.11992 2022.
Haoying Li, Yifan Yang, Meng Chang, Huajun Feng, Zhihai Xu, Qi Li, and Yueting Chen. Srdiff: Single image super-resolution with diffusion probabilistic models. arXiv preprint rxXiv2214414455 2021.
Jie Liang Hui Zeng, and Lei Zhang. Details or artifacts: A locally discriminative learning approach to realistic image super-resolution. Proceedings of the IEEE/CVE Conference on Compuler Vision and Pattern Recognition, pp. 5657 5666, 2022.
Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lec. Enhanced deep residual networks for single image super-resolution. In Proceedings of the IEEE conference computer vision and pattern recognition workshops, pp. 136-144, 2017.
Guan-Horng Liu, Arash Vahdat, De-An Huang, Evangelos A Theodorou, Weili Nie, and Anima. Anandkumar. F'sb: Image-to ton--ttoimagee schrödinger bridge. arXiv preprint arXiv:2302.055: 05872, 2023.
Andreas Lugmayr, Martin Danelljan, Lue Van Gool, and Radu Timofte, Srflow: Learning the super- resolution space with normalizing flow. In European Conference on Computer Vision, pp. 715-732. Springer, 2020.
Andreas Lugmayr, Martin Danelljan, and Radu Timofte. Ntire 2021 learning the super-resolution space challenge. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pp. 596 612, 2021.
Ziwei Luo, Fredrik K Gustafsson, Zheng Zhao, Jens Sjölund, and Thomas B Schön. Image restoration with mean-reverting stochastic differential equations. Xrxiv preprint or Xiv:2300 11699, 2023a.
19

---

Published in Transactions on Machine Learning Research (06/2023)
Ziwei Luo, Fredrik K Gustafsson, Zheng Zhao, Jens Sjölund, and Thomas B Schön. Refusion: Enabling large-size realistic image restoration with latent-space diffusion models. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 1680 1691, 2023b.
Roey Mechrez, Itamar Talmi, Firas Shama, and Lihi Zehnik-Manor. Maintaining natural image statistics with the contextual loss. In Asian Conference Computer Vision, pp. 427-443. Springer, 2018a.
Roey Mechrez, Itamar Talmi, and Lihi Zelnik-Manon The contextual loss for image transformation with non-aligned data. In Europeon Conference on Computer Vision (ECCV), pp. 768-783, 2018b.
Eliya Nachmani, Robin San Roman, and Lior Wolf. Denoising diffusion gamma models. arXiv preprint arXiv:21 00.i:2110.00500 05948, 2021.
Scungjun Nah, Tae Hyun Kim, and Kyoung Mu Lec. Deep multi-scale convolutional neural network for dynamic scene deblurring. In The IEEE Conference on Computer Vision and Pollern Recognition (CVPR), July 2017.
Guy Ohayon, Theo Adrai, Gregory Vaksman, Michael Elad, and Peyman Milanfar. High perceptual qual- ity image denoising with posterior sampling cgan. In Proceedings of the International Conference Computer Vision (ICCV) Workshops, 2021.
Gregory Ongie, Ajil Jalal, Christopher Metzler, Richard G Baraniuk, Alexandros G Dimakis, and Rebecca Willett. Deep learning techniques for inverse problems in imaging. IEEE Journal on Selected Areas in Information Theory, 1(1):39-56 56, 2020.
Mangal Prakash, Alexander Krull, and Florian Jug. Fully unsupervised diversity denoising with convolutional variational autoencoders. In International Conference on Learning Representations, 2020.
Herbert Robbins. An empirical bayes approach to statistics. In Proc. 3rd Berkeley Symp. Math. Statist. Probab., 1956, volume 1, pp. 157-163, 1956.
Yaniv Romano, Michael Elad, and Peyman Milanfar. The little engine that could: Regularization by de- noising (RED). SIAM Journal on Imaging Sciences, 10(4):1804-1844, 2017.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. International Conference on Medicul image compuling and compuler-assiseed interven- tion, pp. 234 241. Springer, 2015.
Leonid I Rudin and Stanley Osher. Total variation based image restoration with free local constraints. In Proceedings of 1st international conference on image processing, volume 1, pp. 31-35. IEEE, 1994.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David Fleet, and Mohammad Norouzi. Image super-rosolution via itcrative refinement. arXiv preprint 2rXi::2144.001
2021.
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-inage diffusson models. In ACM SIGGRAPH 2022 Conference Proceedings, pp. 1-10, 2022.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training GANs. Advances neural information processing systems, 29, 2016.
Yuyang Shi, Valentin De Bortoli, Andrew Campbell, and Arnaud Doucet. Diffusion schr odinger bridge matching. arXiv preprint arXio:2303. 166522 2023.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference Machine Learning, pp. 2256 2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference Learning Represendations, 2021a.
20

---

Published in Transactions on Machine Learning Research (06/2023)
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Aduances in Newral Information Processing Systems, 32, 2019.
Yang Song and Stefano Ermon. Improved techniques for training score- based generative models. Advances in neural information processing systems, 33:12438-12448 2020.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Represenlations, 2021b.
Yang Song, Prafulla Dhariwal, Mark Chen, and Tlya Sutskever. Consistency models. arXiv preprint arXiv:2303.014 2023.
Maitreya Suin, Kuldeep Purohit, and A. N. Rajagopalan. Spatially-attentive patch-hicrarchical network for adaplive motion deblurring. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pattern Recognition (CVPR), June 2020.
Xin Tao, Hongyun Gao, Xiaoyong Shen, Jue Wang, and Jiaya Jia. Scale-recurrent network for deep image deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxim: Multi-axis mip for image processing. In Proceedings of the FEEE/CVE Conference on Computer Vision and Paltern Recognition, pp. 5769-5780, 2022.
Singanallur V Venkatakrisshan,, Charles A Bouman, and Brendt Wohlberg. Plug-and-plas priors for model based reconstruction. In 2013 TEEE Global Conference Signal and Information Processing, pp. 945-948. IEEE, 2013.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation, 23 (7):1661 1674, 2011.
Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. ESRGAN: enhanced super-resolution generative adversarial networks. In Proceedings of the European conference computer vision (ECCV) workshops, pp. 0-0, 2018.
Simon Welker, Henry N. Chapman, and Timo Gerkmann. Blind dristing: Diffusion models with a linear SDE drift term for blind image restoration tasks. In The Symbiosis of Deep Learning and Differential Equations 11, 2022. URL httppp///ppparrreee..eeeeessteea
Jay Whang, Erik Lindgren, and Alex Dimakis. Composing normalizing flows for inverse problems. In International Conference on Machine Learning, pp. 11158 11169. PMLR, 2021.
Jay Whang, Mauricio Delbracio, Hossein Talebi, Chitwan Saharia, Alexandros G Dimakis, and Peyman Milanfar. Deblurring via stochastic refinement. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp.
2022.
Mao Ye, Lemeng Wu, and Qiang Liu. First hitting diffusion models. arXiv preprint ar Xrx::22209.0111 2022.
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming- Hsuan Yang, and Ling Shao. Multi-stage progressive image restoration. In Proceedings of the IEEE/CVF conference on computer vision and pottern recognition, pp. 14821 148311 2021.
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming -suan Yang. Restormer: Efficient transformer for high-resolution image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognation, pp. 5728-5739. 2022.
Hongguang Zhang, Yuchao Dai, Hongdong Li, and Piotr Koniusz. Deep stacked hicrarchical multi-patch network for image deblurring. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition (CVPR), June 2019.
21

---

Published in Transactions on Machine Learning Research (06/2023)
Kai Zhang, Jingyun Liang, Luc Van Gool, and Radu Timofte. Designing practical degradation model for deep blind image super-resoltton. In Proceedings of the IEEE/CVP International Conference Computer Vision, pp. 4791 4791-4800, 4800, 2021.
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effective- ness of deep features perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Hang Zhao, Orazio Gallo, Iuri Frosio, and Jan Kautz. Loss functions for image restoration with neural networks. IEEE Transaclions on computalional imaging, 3(1):47 57, 2016.
A Proof of Proposition 4.1
Proof. We have x, = (1 s)x + sy, and xz = (1 - t) + ty, so by substituting y from one to the other we gel,
(1 - 1)x
(1
8) +
(16)
t
8
sx
+
2
x + sx
(17)
t
[
8
8
Z2.
(18)
Then,
E Cx xz
(19)
to. - 500
02x
dat
(20)
(1 5) + 800,
t Palm (x)) )
(21)
[1
:)
xp m, (x/x;))d + -0
(22)
E[ xc xt. + s
t z
(23)
where we have applied the fact that Pa, lws (xa (xx) Px a. (x) (x0)x)) and w 1m.-821
0
B Denoising with Gaussian Prior
Let's analyze InDI's behavior in the particular case where p(x) = N(c, o) and the restoration lask is denoising y = x + where is white Gaussian of fixed standard deviation ON. Then, p(y(x) = ox and Ex[ xx] d3x,4193 where we have used the fact that x, (1 l)x + ty= x + In. oftt'o
InDI's idcal ODE is given by dx: x -B///x] which in this case becomes:
di
dx; (x
dt
o o +o²
We are interested in solving this equation at f = 0, with boundary condition y at t= 1. This is separable ODE having general solution: x; = c (y - c)
The solution at t = 0, is where = a ov Haf
XInDI = c (y c)
oe o
22

---

Published in Transactions on Machine Learning Research (06/2023)
Note that E[XIaDI] = = E[x], and the covariance cov(XInDI) cov(y) o² (o. + o Given p(x) and p( ((IDD)) are Gaussian distributions with same mean and covariance, we have p(XinD)) p(x). That is, in the limit, InDI generates samples from the prior distribution p(x).
o cov(x), since cov(y)
It is worth noting that in this case the MMSE and MAP estimates coincide,
J.y + N
XMMSE
XMAP
022 + ON
and are in fact different from InDI's estimate.
C Model and Training Details
In all our restoration experiments we use U-Net-like architecture (Roneeerger et al 2015) similar to the one in SR3 (Saharia et al.) 2021 and DvSR (Whang et al. 2022). We followed the same adaptations as the ones introduced in Whang et al. (202)) to make it fully-convolutional (removed self-attention layers and group normalization). Our U-Net has an adaptive number of resolutions each of them having an arbitrary number of channels (given by a multiplication factor from a base set of channels).
Table summarizes the model definition for each of the tested applications.
Table 2: Model and training parameters for each restoration task.
channels multipliers
noise
p(t)
batch size learning rate
crop size
# params.
motion deblurring
64
1,2,3,4]
0
bias_ti t1
256
10
128 128
27.68M
defocus deblurring
64
1,2,4,4]
e 0
Linear_0.0. 0..5
1024
10-4
128 x 128
33.57M
JPEG restoration
64
[1,2,4,4
0.060
linear 1.0
1024
10-1
128
128
33.57M
4x super-resolution
96
[1,2,344
e 0.015
bias_ti
256
10
256 x 256
62.25M
All models are trained for 500K steps using 32 TPUv3 cores. We used the Adam optimizer with a fixed learning rate, and FMA decay rate of 0.9999. Models were trained using the respective indicated distribution for p(t). For the super-resolution model, low-resolution crops of size 64 64 are upscaled using bilinear interpolation to 256 256 before feeding them into the model.
D Additional Results
Table 3: Defocus Deblurring on the DDPD dataset Abuulaim Brown 2020 Best values and
socond-best values for each metric color-coded. KID values scaled by factor of 1000 for read- ability
Steps
PSNR
LPIPS
FID
KID
1
24.75
0.206
40.80
16.72
2
24.74
0.201
23.29
6.18
24.55
0.195
17.92
3.52
10
24.24
0.188
15.68
2.19
50
23.89
0.186
16.12
2.48
100
23.82
0.187
16.34
2.63
500
23.77
0.189
16.53
2.49
23

---

Published in Transactions on Machine Learning Research (06/2023)
Input
Reference
step
4 steps
steps
20 steps
Figure 13: Additional GoPro deblurring results. The proposed method (InDI) applied with different number of reconstruction steps. Best viewed electronically.
24

---

Published in Transactions on Machine Learning Research (06/2023)
Input
Reference
step
4 steps
steps
20 steps
Figure 14: Additional GoPro deblurring results. The proposed method (InDI) applied with different number of reconstruction steps. Best viewed electronically.
25

---

Published in Transactions on Machine Learning Research (06/2023)
Input
Reference
step
4 steps
10sssess
20 steps
Figure 15: Additional GoPro deblurring results. The proposed method (InDI) applied with different number of reconstruction steps. Best viewed electronically.
26

---

Published in Transactions on Machine Learning Research (06/2023)
Impat
Reference
step
steps
20 steps.
100 steps
Figure 16: 4x super-resolution results (diy2k dataset). The proposed method (InDI) applied with different number of reconstruction steps. Best viewed electronically.
27

---

Published in Transactions on Machine Learning Research (06/2023)
Imput
Reference
step
steps
20 steps
100 steps
Figure 17: 4x super-resolution results (diy2k dataset). The proposed method (InDI) applied with different number of reconstruction steps. Best viewed electronically.
28

---

Published in Transactions on Machine Learning Research (06/2023)
Input
Reference
step
steps
20 steps
100 steps
Figure 18: 4x super-resolution results (div2k dataset). The proposed method (InDI) applied with different number of reconstruction steps. Best viewed electronically.
29

---

Published in Transactions on Machine Learning Research (06/2023)
Imput
Lstop
stops
20 steps
TOO steps
Figure 19: 4x super-resolution results (div2k dataset). The proposed method (InDI) applied with different number of reconstruction steps. Best viewed electronically.
30

---

Published in Transactions on Machine Learning Research (06/2023)
R शाहा
غ शारा
STROSE
Refevence
s शारा
غ शााा
OF
steps
т शाराा
غ शारा
20 stepso
100 steps
Figure 20: JPEG compression artifact removal results (quality factor 15, div2k test dataset). The proposed method (InDI) applied with different number of reconstruction steps. Best viewed clectronically.
31

---

Published in Transactions on Machine Learning Research (06/2023)
Imput
Reference
step
4 steps
20 steps
100 steps
Figure 21: JPEG compression artifact removal results (quality factor 15, div2k test dataset). The proposed method (InDI) applied with different number of reconstruction steps. Best viewed electronically.
32

---

Published in Transactions on Machine Learning Research (06/2023)
Input
Reference
step
4 steps
20 steps
100 steps
Figure 22: JPEG compression artifact removal results (quality factor 15, div2k test dataset). The proposed
method (InDI) applied with different number of reconstruction steps. Best viewed electronically.
33

---

Published in Transactions on Machine Learning Research (06/2023)
Imput
Reference
Sheps
20 steps
100 steps
Figure 23: JPEG compression artifact removal results (quality factor 15, div2k test dataset). The proposed
method (InDI) applied with different number of reconstruction steps. Best viewed electronically.
34


Reviews:
[{"review_id": "65QgMBQrMT", "reviewer": "Reviewer_16FV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n-Well-motivated\n\n-Very clearly written\n\n-Interesting\n\n-Thorough ablation studies\n\n\nWeaknesses:\n\n-A few minor presentation issues\n\n-Not state-of-the-art at everything\n\n-Doesn't generalize (requires retraining per inverse problem)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a novel diffusion-based image restoration process (Inversion by Direct Inversion) designed to solve imaging inverse problems without explicit knowledge of the degradation forward process. The proposed technique assumes access only to paired clean and degrades images (x,y)~p(x,y); it does not need to know p(y|x). That is, unlike other diffusion techniques (e.g., cold diffusion) it does not need to know the degradation process. The proposed method then constructs a series of intermediates samples x_{t_1}, x_{t_2}, ... from convex combinations of x and y. It trains a restoration model to reconstruct x from each of these samples, effectively training a network to compute E[x|x_t]. This restoration model is repeatedly applied (with a principled weighting scheme) to reconstruct x. If the restoration model uses only a few large steps it has little distortion (high PSNR) while if it is applied many iterations with small steps it has higher perceptual quality. The authors apply the proposed method to motion deblurring, super-resolution, defocus deblurring, and JPEG artifact removal. It offers SOTA performance at motion deblurring and is competitive with existing techniques on the other tasks.\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I support accepting the paper, but addressing the following would strengthen the work:\n\nOverall, the paper provides a comprehensive survey of the literature and related work. The recent papers [A] and the follow-up [B] (concurrent) seem closely related and may be worth mentioning.\n\n[A] Luo, Ziwei, et al. \"Image Restoration with Mean-Reverting Stochastic Differential Equations.\" arXiv preprint arXiv:2301.11699 (2023).\n\n[B] Luo, Ziwei, et al. \"Refusion: Enabling Large-Size Realistic Image Restoration with Latent-Space Diffusion Models.\" arXiv preprint arXiv:2304.08291 (2023).\n\nHighlighting the best and second best results in all tables (as they were done in table 1) would make them easier to read and interpret.\n\n\n#### Typos and suggestions\n\nPg 10: \"Table 5(a)\"-->\"Figure 5(a)\".\n\nPg 10: I suggest making 5(b) Table 2\n\nPg 10: It's not obvious whether the \"ours\" in 5(b) had noise injection or not. The adjacent figure 5(a) suggests this is an important distinction to make.", "broader_impact_concerns": "I have no concerns about this work. Like other generative techniques, it could potentially be misused by bad actors.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "AX4wnB6ImA", "reviewer": "Reviewer_6LtM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1) The described algorithm is remarkably straightforward to implement and does not necessitate any knowledge of the degradation process's analytic form, distinguishing it from generative denoising diffusion models.\n \n2) The paper is well-written, with clear motivation and easy-to-understand proofs. The implementation process is presented in a user-friendly manner.\n\n3) The authors effectively showcase the performance of their approach across various image restoration tasks, comparing it to both cold diffusion and conditional denoising diffusion models.\n\n4) The authors also reveal the significant impact of bias during training time (with p(t) skewed towards t=1) on performance. This finding highlights the importance of the iterative procedure's increased certainty in determining the appropriate direction during the initial stages of the process.\n\nWeakness:\n1) Although the authors demonstrate the \"regression to the mean\" effect using a 2D toy model, they do not fully explain how INDI mitigates this effect. They claim that \"the set of plausible 'slightly-less-bad' images is relatively small,\" but the reasoning behind this statement remains unclear from the 2D example. Consider the following example for the MMSE case. Given the model $y = Hx + n$ and the specified prior distribution $p(x) = \\sum_{i=1}^d w_{i} \\delta_{x-c_{i}}$, the MMSE solution is $\\int x p(x|y) dx = c \\int x p(y|x)p(x)dx$, with $c = \\frac{1}{p(y)}$. The posterior distribution can be expressed as the product of the likelihood term and the prior. Under the Gaussian noise assumption, it becomes $c \\int x \\exp^{-\\frac{-||y - Hx||_{2}^2}{\\sigma^2}} p(x) dx$, where $p(x)$ is the discrete multimodal distribution.\n\nConsequently, the integral simplifies to the sum over the four modalities: $\\sum_{i=1}^4 w_{i} c_{i} \\exp^{-\\frac{-||y - Hc_{i}||_{2}^2}{\\sigma^2}}$.\n\nFor the first denoising model, where $H = I$, the regression to mean effect is apparent, as it is an average over the $c_{i}$'s weighted by the distance of $y$ to each $c_{i}$. In the second inpainting problem, the non-trivial null-space of $H$ causes the y-coordinate information to be erased, and the MMSE iterate converges to a point where the x-axis coordinate matches that of the true modalities, while the y-axis of the MMSE solution is simply the average of the y-axis of the modalities. The \"regression to the mean effect\" is thus a joint result of the multimodal distribution of $p(x)$ and the non-trivial null-space of $H$ (and possibly the condition number in certain cases).\n\nConsidering this observation, how does INDI's update equation (equation 2) with the closed-form solution for the 2D model help alleviate the regression to mean effect? Is it because $E[x_{0}|\\hat{x_{t}}]$ depends on $H_{t}$, which no longer has a null-space? A more detailed mathematical explanation of how INDI mitigates the regression to mean effect would be appreciated.\n\n2) The ODE in equation-6 should be revised to reflect the residual ODE flow. However, the motivation for the ODE flow in this case remains unclear. It would be helpful if the authors could elaborate on what additional insights the ODE flow provides in this context.\n\n3) It is surprising that the authors did not include any image inpainting experiments in their study. Given the current motivation and implementation of INDI, image inpainting appears to be a highly suitable task for the algorithm. The inclusion of such experiments would strengthen the paper and provide further evidence of INDI's applicability across different image restoration tasks.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work presents a novel formulation for supervised image restoration, which aims to circumvent the so-called \"regression to the mean\" effect. The proposed algorithm functions by iteratively restoring low-quality input images through a series of small steps. The underlying intuition is that taking smaller restoration steps can largely avoid the regression-to-the-mean effect, as the set of plausible minor changes is relatively small.\n\nThe authors justify their approach by examining a simple 2D model, illustrating that the minimum mean-squared error (MMSE) optimal solution is susceptible to the \"regression to the mean\" effect. In contrast, iterative restoration converges to one of the plausible modes. The authors further support their claims by applying the method to various image restoration tasks, demonstrating that the Indi algorithm produces high-quality results for image denoising tasks.\n\nOverall, this is an innovative and straightforward approach to image restoration, offering more strengths than weaknesses. Consequently, I recommend acceptance of the paper following minor revisions.\n\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I recommend the following major changes (1, 2) to address the weaknesses identified in the paper:\n\n1) Weakness-1: The authors should provide a mathematical argument, corresponding to the 2D toy problem, that demonstrates how INDI can alleviate the regression to the mean effect. This argument should be based on the closed-form solution for the 2D experiment and should explain whether INDI reduces the degeneracy of the operator H.\n\n2) Weakness-2: To provide a more comprehensive evaluation, the authors should include image inpainting experiments in the experiments section of the paper.\n\nAdditionally, I suggest a minor change:\n\n3) The authors should clarify the insights provided by the residual ODE flow and explain its relevance to the overall work.", "broader_impact_concerns": "No broader impact concerns are noted by the reviewer. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rImBK5p8Vx", "reviewer": "Reviewer_9cRu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strenghts:\n- The method is generally sound and an innovative technique to deal with the well-known distortion-perception tradeoff. \n- Controlling the number of steps seems to achieve different perception-distortion tradeoff which might be useful in certain applications\n- generally good experimental results\n\nWeaknesses:\n- as it is an iterative method, inference can be quite expensive\n- performance does not always seem to be state of the art, neither in minimizing distortion nor in maximizing visual quality (see e.g. Fig.5) \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors present a novel image restoration technique for the supervised learning setting which allows improved perceptual quality. The core of the method is to iteratively refine a solution in small steps. This allows to avoid the regression to the mean effect and get a solution closer to the image manifold.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think the paper could be strengthened by some revisions and the inclusion of additional material. \n\nI would like to see an extended set of comparisons with state-of-the-art methods where the extreme case of 1 iteration (which should maximize PSNR) is also shown. Additionally, it could be interesting to see results on the denoising problem. There are also questions about fairness of comparisons since the network architecture is different from other baselines. What would be the performance of the architecture when trained in a conventional manner?\n\nIt could also be interesting to discuss if the authors think there are connections between their work and the literature on plug and play methods (https://arxiv.org/abs/2203.17061).\n\nAs minor points: a few typos should be fixed throughout the paper; sections 2 and 3 are partially redundant and could be merged in a single section.", "broader_impact_concerns": "No concerns on broader impact.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a very well-written paper on a novel formulation for supervised image restoration. All reviewers appreciate the simple yet effective framework and solid evaluation on five different tasks (motion deblurring, super-resolution, defocus deblurring, jpeg artifact removal, and generative modeling), demonstrating the proposed method's applicability to various image restoration problems. 

The notable strengths by the reviewers are:

"remarkably straightforward to implement" - Reviewer 6LtM
"clear motivation and easy-to-understand proofs" - Reviewer 6LtM
"Thorough ablation studies" - Reviewer 16FV
"The method is generally sound and an innovative technique to deal with the well-known distortion-perception tradeoff" - Reviewer 9cRu

There were some missing references and discussions of connection with prior work raised by the reviewers. After the authors' responses, all reviewers strongly support accepting the paper. The AE appreciates the simplicity of the work and the general applicability of the proposed method on different problems. While, the current method does not outperform the state-of-the-art on each individual task, the AE agrees that this is not the main focus of the paper. The method's simplicity and promising performance may inspire many follow-up research. The AE thus recommends "accepting as is" with featured certification. 



Decision: Accept (poster)