A Modular Conditional Diffusion Framework for Image Reconstruction
Authors: Magauiya Zhussip
Iaroslav Sergeevich Koshelev
Stamatios Lefkimmiatis
Venue: NeurIPS 2024 poster (2024)
Keywords: diffusion model; image restoration; super-resolution; motion deblurring
Primary area: diffusion_based_models

Abstract: Diffusion Probabilistic Models (DPMs) have been recently utilized to deal with various blind image restoration (IR) tasks, where they have demonstrated outstanding performance in terms of perceptual quality. However, the task-specific nature of existing solutions and the excessive computational costs related to their training, make such models impractical and challenging to use for different IR tasks than those that were initially trained for. This hinders their wider adoption especially by those who lack access to powerful computational resources and vast amounts of training data. In this work we aim to address the above issues and enable the successful adoption of DPMs in practical IR-related applications. Towards this goal, we propose a modular diffusion probabilistic IR framework (DP-IR), which allows us to combine the performance benefits of existing pre-trained state-of-the-art IR networks and generative DPMs, while it requires only the additional training of a small module (0.7M params) related to the particular IR task of interest. Moreover, the architecture of our proposed framework allows us to employ a sampling strategy that leads to at least four times reduction of neural function evaluations without any performance loss, while it can also be combined with existing acceleration techniques (e.g. DDIM). We evaluate our model on four benchmarks for the tasks of burst JDD-SR, dynamic scene deblurring, and super-resolution. Our method outperforms existing approaches in terms of perceptual quality while retaining a competitive performance in relation to fidelity metrics.

A Modular Conditional Diffusion Framework for Image Reconstruction
Magauiya Zhussip"
Iaroslav Koshelev" AI Foundation and Algorithm Lab ys www.eeee@gaaall............mm..e com
Stamatis Lefkimmiatis' MTS AI
MTS AI
m. www.sssssstsss...mm
s. lefkimmiatis@tts
Abstract
Diffusion Probabilistic Models (DPMs) have been recently utilized to deal with various blind image restoration (IR) tasks, where they have demonstrated outstanding performance in terms of perceptual quality. However, the task-specific nature of existing solutions and the excessive computational costs related to their training, make such models impractical and challenging to use for different IR tasks than those that were initially trained for. This hinders their wider adoption, especially by those who lack access to powerful computational resources and vast amount of training data. this work we aim to address the above issues and enable the successful adoption of DPMs in practical IR-related applications. Towards this goal, we propose a modular diffusion probabilistic IR framework (DP-IR), which allows us combine the performance benefits of existing pre-trained state- of-the-art IR networks and generative DPMs, while it requires only the additional training of relatively small module (0.7M params) related to the particular IR task interest. Moreover, the architecture of the proposed framework allows for sampling strategy that leads to least four times reduction of neural function evaluations without suffering any performance loss, while can also be combined with existing acceleration techniques such DDIM. We evaluate model four benchmarks for the tasks burst JDD-SR, dynamic scene deblurring, and super- resolution. Our method outperforms existing approaches in terms of perceptual quality while retains a competitive performance with respect to fidelity metrics.
Introduction
With the advent of deep learning we have witnessed outstanding results in wide range of computer vision tasks [89). including many challenging blind image restoration (IR) problems [84] such as burst imaging [40], super-resolution (SR) [9) deconvolution [58]. etc. The standard approach for supervised learning in blind IR setting involves training feed-forward network that should estimate the latent image based on the available low-quality measurements. Such models are usually trained to maximize fidelity metrics like PSNR SSIM, but the visual quality of the resulting images is sub-optimal [6] The inclusion of perceptual losses 33 to the objective can improve the visual results, but fails to convincingly address the problem.
promising direction towards results high visual quality is to consider such problems within a generative framework. Several generative models have been recently proposed including Variational Autoencoders (VAEs) [33], Generative Adversarial Neural Networks (GANs) (22). Normalizing
Flows (NFs) [6] and Diffusion Probabilistic Models (DPMs) [65] Due to their impressive results in image generation, they have been further utilized to perform conditional sampling of high- quality images, with their low-quality distorted counterparts playing the role of the conditional
Equual contribution
Work performed while at Al Foundation and Algorithm Lab
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

t
72
J2
y
Image Restoration Network JR
E ( y
COIADD LReL
O
Fusion Network
2D
CaneD
Cuno
(y)
Denoising Network
Φ0p (x) 20
f
Sigmoid
W
o+
Φ0p (it,
Linear
LReLi
E o y .t
T0
Sinusoidal Positional Encoding
Repeating
Perceptron
Dense layer (T) features)
structure
( features)
Denoising Network Φ0p (me, ōx), an IR R 2D t). A small version of MIRNet [81] is used as the Network Pp:R (y) and Fusion Network Φb (x.
Figure I: The proposed architecture consists of three modules:
IR
Denoising Network, while pre trained SwinlR (42) or BSRT (50) or FFTFormer R4 is used as the IR Network, depending on the task. See section 3.3 for a detailed description.
input [21 38 46 39]
To date, DPMs appear be the most promising framework and lead to the best results among all existing generative approaches.
Nevertheless, there certain limitations that existing DPMs face, which hinder their wider adoption in IR tasks. In particular, inference of such models involves a sampling process that requires large number (in the order of hundreds) of neural function evaluations (NFEs), which can be computationally very expensive, especially when considering images of high resolution. Another important limitation that efficient conditioning proposed for DPMs in order to make them applicable to a wider range of blind IR problems. Indeed, all of the existing methods aim to learn the parameters of single input-conditioned network for specific blind IR task. As result, the trained model overfits on the distribution of the condition space, and the whole model has be retrained if we need to employ it to a different reconstruction task than the one that was initially trained for. Considering the huge amount of data and computational resources required for training single DPM (see appendix A such re-training becomes infeasible at least one of the previous requirements is not satisfied.
the image measurements has yet to be
In this work we aim address the above issues by proposing a novel conditional diffusion network coupled with an accelerated sampling process. Specifically, our network adopts an improved conditioning strategy and is built on the foundation of existing off sff-lf IR networks paired with denoising module, which is applicable to a variety of reconstruction problems without requiring any re-training. Additionally, we introduce an accelerated sampling procedure that is enabled by our proposed network architecture and allows the merging of a large number of sampling steps in a single one, computed with a single NFE. Our proposed acceleration can work tandem with accelerated sampling schemes such as DDIM [66]. To assess the performance of our network, we validate on three challenging blind IR tasks, namely, burst joint demosaicking, denoising and super-resolutio (JDD-SR), dynamic scene deblurring, and 4x single image super-resolution (SISR). In all of the tested scenarios, our approach demonstrates the best perception-distorie trade-off among the state-of-the-ar (SOTA) methods, while compared to other DPM-basee solutions it requires a smaller number of sampling steps.
2 Related Works
Burst Image Restoration. One of the pioneering works in multi-frame IR was introduced in 70, solution was proposed. Then, several MAP models with various regularization terms have been designed to cope with visual artifacts caused by operating in the Using the same MAP framework, JDD-SR method robust to noise
where a leqquenc-ddomii-beee
frequency domain [3
18
63
and outliers was developed in 19 Mcanwhile, the block matching alignment algorithm of (25) was extended by [79] to obtain robust motion model with the aid of an adaptive kernel interpolation method merging sparse pixel samples.
Advancements in deep learning have led to beghh-peffrming methods such as those in 17
37 5
49 41 The DBSR approach [4] aligns multiple input frames the feature space utilizing an optical
flow estimator (e.g. PWCNet (68) and employs an attention-bast based fusion mechanism to aggregate

---

features. In [37] differentiable image registration module has been introduced, which exploits the aliasing effects appearing in bursts of low-resolution (LR) images. In [4]] KBNet estimates blur kernels for burst sequence to incorporate them with LR features so to generate better super-resolved image, while [7 BIPNet attempts to fuse complementary information from the burst sequence with the help of generated pseudo-burst features. Another line of work effectively employs deformable convolutions for the inter-frame alignment task [49 results in various tasks, including burst SR.
and achieves SOTA
Single Image Restoration. Among the recent IR methods, the most successful ones are those that adopt an end-to-end supervised formulation, where deep neural network is trained to directly map low-quality and degraded image to a point estimate of the latent high-quality image [8] 45 69 83] Consequently, in the pursuit of further improving the reconstructions and achieving better pixel- level result, more advanced network architectures have been proposed [82 42 [0] 34), at the cost
of being more computationally heavy. While this formulation leads to SOTA fidelity (e.g.. PSNR, SSIM), the produced output is an average/median of all plausible predictions, which typically lacks high- freeurryyy information (e-g. texture).
Generative adversarial networks (GAN) 22 have been adopted by several IR methods such as SISR [38 73 75 and dynamic scene deblurring [35 36 85] produce more natural and perceptually pleasing results. Although this adversarial non-reference formulation aims to push the predictions towards the manifold of natural images, it is also prone to introducing unrealistic texture and hallucinations in the output [4]. Moreover, the adversarial training process requires extra supervision as it can easily fall into mode collapse or may diverge
62]
Likelihood based deep generative models such NFs 47
auto-regressive models 23 and
VAFs (57] have also been applied to IR tasks, where one can obtain diverse set of predictions from a learned posterior [57] Conditioned on LR inputs, flow-based methods attempt to map high-resolution (HR) images to the latent flow-space. Although such techniques circumvent the training instability met in GANs, strong architectural constraints (e.g. network invertibility) still remain an issue.
Recently another class of methods based on a stochastic diffusion process has been introduced and demonstrated outstanding performance on various tasks that range from unconditional image generation 226
55 60 to image-to-ing
translation/restoration 33 61
78
20
60
15
76 59]]
DvSR proposed in [8 employs a "predict-and-refine" conditional diffusion method specifically
tailored for the image deblurring task, while SRDiff (39) utilizes features of a pretrained SR model for conditional super-resolved image generation. Further, recent works in 776 [) have considered several IR tasks (e.g. inpainting, super-resolution, colorization, etc.). In conclusion, their ability to capture complex statisties of the visual world, makes DPMs a very attractive solution that is worth being further investigated.
3 Proposed Conditional Diffusion Model
3.1 Background
Denoising Diffusion Probabilistic Models (DDPMs) 26 65) Markovian Variational Autoencoders where the dimension of the latent variables matches the dimension of the data. Starting with sample ap E R. the encoding sequence (æ.) traverses the latent space with a diffiusion process defined by a Gaussian transition probability:
special cases of Hierarchical
q (x, x1-1) = N (z:: (1 /1 B,
(1)
The sequence 0 < $1, S2...., Br < 1 that appears in eq. ( defines the noise scheduling for the forward process in such way so that the latent variable at the final timestep T approximates the standard Gaussian: æ7 ~ N (ær;; 0, I.). Based on this diffusion process, it is possible to express the transition probability directly from æo to a; in closed form as:
q (x)) = N (xxi Vārep, (1 AN),
(2)
where 0a = and o₁ II' 0.
The reverse process is enabled by the posterior distribution which is represented in the form:
p ( za) - (x:-13 (#t, 2o) IN):
(3)
3

---

The above result implies that the conditional score function can be approximated by utilizing a trained where je (Wt; Vān-i Fo + yat(l-ā--1) iy and -- B, DDPMs aim to approximate 1-û;
its mean by the quantity μe (æt; t), which is learned from training data, and then utilize eq. 3 to perform sampling. There are different possible parameterizations of μr (æt Lo), which accordingly lead to different interpretations for the transition mean [48). In this work, we pursue the one based on the score function V logp (2)), which reads as:
Me (Ze, vo) Dt + (1 - a4) V (x.
Var
In this case, the reverse process defined in eq.
3 can be considered as sampling via Annealed
Langevin Dynamics, in which the score function is approximated by the quantity so (Iz, 1) learned via denoising score matching [28 72).
3.2 Conditional Score Matching
The diffusion models described above do not take into account the dependence of the sampled data on their degraded observations y R.M, when we are dealing with IR problems. Fortunately, the score-based models can be extended to accommodate conditional sampling by replacing the score function in eq. (4) with conditional score function Z.. log p (æx ly) For non-blind IR problems, popular approach is to decompose the conditional score function into a score function V log (ax)) and log-likelihood gradient term Va, log (y/ (z,) [54 677. This last term is directly dependent on the image formation model, which unfortunately is unknown for blind IR tasks, Therefore, most of the existing works [39 55 60] aim instead to learn the primal conditional score function Va, log p (ax, yy)
via ad-hoc conditional denoising score matching. In this work, we also utilize the primal conditional score function, but we rely on its explicit form as given in the following lemma, whose proof is provided in the appendix
Lemma 3.1. Let y E ERERN ~ - (aroly). and x. E CRN, Cr E R are defined as in eq. the conditional score function is computed as:
2 Then,
Vee log (ar,)))
ID y. x.) E
Zz
joint reconstruction and denoising model. Specifically, if the augmented variable [y x represents the union of the degraded data y and the noisy data Tt, then the conditional expected value E [ =r] corresponds to the reconstructed underlying image To from the measurements z,. A joint reconstruction model Φ,, (y. Te. t) can be trained by the minimization of the empirical expected pixel mean-squared error (MSE) across the samples from the training dataset D (Io. y, xt, t):
min "... Φe (y, xr, zu 1 min Ellde (y' ( t) 10))==(f1
(6)
The optimal solution is the conditional expectation M (y, xz, t) = EE [ b, ærI. and, thus, such trained model can be substituted in eq. (5). This amounts to approximating the conditional score
function Væ. log (xxly) with sy (y, xz, t) (y. (t.
3.3 Proposed Network Architecture
Our objective is to parameterize the function &, (y, xt. f) in form of a neural network (CNN) and design a specific architecture of this network. The absence of explicit knowledge about the formation model Zo + y requires the network to learn it implicitly from training data. Such an approach generally results in over-fitting meaning that the trained model can only employed for the task was originally trained for [39. [8] To overcome this problem, we initially build on the hypothesis that the conditional expectation E [aool y. æx] related to the conditional score function in eq. (5). can be approximated by function of two easier to compute conditional expectations, that is
E p y xx] ~ f (E [x yy] E [xo (c:)).
Based on such an approximation, it is now possible to learn a single unconditional generative denoising model that can be applied in different reconstruction problems. Further, we note that despite the absence of good approximation of the likelihood term, [ro]y]] various task-specific networks trained with fidelity objective are readily available in the literature. Indeed, using

---

similar reasoning as the one provided for eq. (6. such reconstruction networks can output a good approximation of the quantity E [ y). This finally motivates us to express the joint reconstruction and denoising network Φ, (V. Fit () into three components (see Figure 1. Specifically, our network can be described as Φ6 (00 (v) , 6b0 (t) ,t), where T² vä. NN' (æo, dN is the
noisy version of To with noise variance og = 1-ae according to eq. 2 and the sub-modules
Φ 08rn' & are defined next.
IR network &R (y), which is learned in supervised manner to predict E. [anoly) ly). Specifically, we employ the BSKT-Small [50] for burst JDD-SR, FFTformer [34] for dynamic scene deblurring, and SwinlR 42 for SISR. We do not train these networks but use their publicly available trained weights.
Denoising network bbp (æ,, ot), which is learned in a supervised manner to predict E [ |z,] by denoising #t. Specifically, we employ smaller version of MIRNet (8I), which call MIRNet-S. It is obtained by reducing the amount of RRG and MRB blocks from the original architecture to three and one, respectively. We refer the original paper [81] for the detailed description of the blocks structure, as we use them without any modifications. Once trained, this network is reused for all considered reconstruction problems. We note that we our motivation for utilizing a smaller version of MIRNet as Denoising module, is to approximately match the number of parameters and the computational complexity of the networks used in our framework with those of the alternative methods under study. This way we can ensure a fair evaluation and comparison among competing methods. Such strategy has allowed us to achieve direct performance comparisons under similar conditions.
Fusion network o5. (at ag .t), which predicts the conditional expectation E [aog] y, æt]. This module refines and combines the predictions of the previous two networks and is the only one that needs to be trained for each specific IR task. The fusion network accepts as inputs the image estimates z R _ D and timestep t. Its architecture consists of two branches. The first one involves a convolution layer with ny output channels followed by a single dense block [27] without batch normalization. Its purpose is to independently encode both input images into the corresponding features f1, fa with n channels each. The second branch encodes the timestep f into vector of weights w E (0, 1)" using the sinusoidal positional encoding 71I, followed by a two layer perceptron and a sigmoid function as the final activation. The features fi, f2 and the weights w are then passed to the Convex Combination Channel Attention (3CA) layer, which performs the per- channel aggregation of input features as a convex combination of the form: w @ fi + (1 w) @ f2. The output of this layer is decoded by two consequent dense blocks with n channels each, followed by a convolution layer which produces the final output E [xo y. xz]. This proposed architecture results in significantly smaller network size than those of the Denoising and IR modules. Thus we can train the fusion network fast and by using only a small amount of problem-spectee data. While we explored several basic fusion architectures, we did not delve into extensive research to ascertain the optimal design. Our proposed fusion module serves as a proof of concept, validating our framework and demonstrating its potential for performance enhancement. A comprehensive investigation into optimal fusion architectures remains a promising area for future research.
training
Such modular overall architecture allows to capitalize on the existing SOTA non-blind denoising and blind IR networks, while also allows us to easily replace any of these networks when better ones become available in the future. As we describe next, another important advantage of our proposed pipeline, is that it allows us to achieve a significant acceleration for the sampling process without incurring any loss of reconstruction quality.
3.4 Proposed Accelerated Sampling
According eq- B3, our conditional denoiser should be evaluated for all timesteps t. = T, which leads to total of I NFEs. We note that by construction, for the forward process it holds that ØT ~ N (0, IN). This means that in the beginning of sampling, the latent variable ær does not contain any information about pp. It is also reasonable to expect that a similar lack of information about xy exists for a number of steps prior to T. Specifically, for those steps we expect that the quantity E [zu y.. xy] is heavily influenced by E [zoo]]]. while the contribution of E [ x is not significant enough. A theoretical justification for this argument is provided in appendix C Based on the above reasoning, we select timestep such that for the first T reverse steps we use the following approximation: E. (Poly,a) Ri [moly]] = &R (y). This is achieved by disabling the lower branch of our proposed conditional score matching network, namely the Denoising &b, (æ., ō,) and Fusion o5, (ao xxx)) modules (Figure Ip. Our strategy can be further supported by the recent
5

---

q (20)
Table 1: Performance evaluation on the task of Burst JDD-SR. We highlight the overall best for each metric.
(at) (
I T
Methods
PSNR
SSIM
LPIPS
T
NFE. 4 Params.
p (00-11 10
20 me))
Target
0C.
N/A
N/A
p (a, (r,, (zo)
DeepRep
34.66
0.927
0.136
0.07
NIA
12.1M
EBSR
36.05
0.940
0.111
0.15
26.0M
Figure 2: Forward and reverse ditfusion process. Blue solid arrows: transitions at the forward pass with sampling distribution from eq. (T). Dashed arrow:
BIPNet
34,86
0.934
0.112
0.03
NIA
6.7M
SmallSmall
35.91
0.940
0.109
0.12
NIA
4.9M
BSRT
36.98
0.947
0.095
0.16
N/A
20.7M
cumulative transition probability from eq. ( Black
Ours
35.53
0.933
0.084
0.02
21.6M
solid arrows: transitions at the backward pass with the sampling distribution from eq. (3)- Red solid
arrow: closed-form cumulative transition probability from eq. representing our accelerated sampling.
where it has been demonstrated that the image sampling via DPMs could be divided into stages depending on the reverse process timesteps. In this spirit we activate the Denoising and Fusion modules at a timestep r that is selected experimentally for the particular IR task of interest. Our results clearly indicate that the reconstruction result is going be exactly the same whether utilize the multi-step reverse diffusion process or the proposed one-step process that is described in Lemma 3.2 Indeed, since the quantity E. (agg ly] is predicted by the IR network, which does not depend on the reverse diffusion parameters, 0orn
study
IR (1) needs to be evaluated only once and its output
can be re-used throughout the whole iterative sampling procedure. Expanding more this idea, we show in Lemma 3.2 that it is possible to omit entirely the first T reverse diffusion steps and instead perform single step directly from T with procedure very similar to the one obtained for the diffusion process in eq. 22) from eq. I We provide the derivation in appendix D
Lemma 3.2. The transition probability defined in eq. B for single reverse step, can be extended to k reverse steps starting from Iı as:
p( p( mo0 = N (at- Me, (I,, 2o), of
(8)
where
k-1
k-1
Hi,k (Ie, mo) =
pt-k+
To- + II- x, and of = Σ
1+
(-i-1
2
(9)
0m
1m0
In the above equations we make use of the following notation:
Vā--18 and I 7t- 1 C2.
V
1-0,-1 On 1-a; Thn=
for i 2j
(10)
for i {
Since for the first - steps 20g is approximated by the quantity E am y). which is independent of the timestep i, utilize eq. (8) directly sample x, as
2r p (wT, E (aro ()) = N (a.3 HT. (MT, E (xolv)) -)
(11)
This allows us to reduce the NFEs from T+1 1 to 1+,, meaning that the required evaluations of Denoising Fusion networks reduced from T to In both cases, we additionally count a single evaluation of the IR network. We depict our acceleration strategy with a red arrow in Figure 2. Finally, in practice we use 200
for burst JDD-SR and dynamic scene deblurring, and T - for SISR, effectively reducing the NFEs by two orders of magnitude and factor of four, respectively.
The proposed acceleration procedure can be also interpreted as starting the sampling from step r of the latent space using non-standard Gaussian distribution as defined in eq. (TT. instead of starting from step T and using standard Gaussian sample TT A (0. IN). We note, that similar idea was explored in [13 [2]. where it was proposed to start the sampling from an observation that has been passed through a predefined number of forward diffusion steps. our case the starting point for sampling is obtained via the approximated reverse process, which as a consequence of Lemma 3.2 does not alter the final reconstruction result. another words, if we approximate my with IE [mt y then the reconstruction result will be the same both for the multi-step reverse diffusion process and

---

the proposed one-step process. Moreover, it is easy to show that our strategy generalizes the one proposed in 13 52 as leads to the same starting point if we make the following specific choices: ar ~ A (Vor oo ly) (1 - aT) I) and of = - B, In practice oT 0, so the first condition holds almost exactly. The second condition represents the particular choice of the noise variance used in the reverse process, with several existing parametrizations [26 55 Our method is compatible with all of them and leads to different distributions for the starting point. In all our experiments we use the parametrization of = 3, from [26]. More detailed conceptual and technical differences along with experimental results are provided in appendix [.] Furthermore, our acceleration strategy is complimentary to other sampling acceleration techniques |66 [1]. To demonstrate this, utilize the DDIM [66] sampling to further reduce the NFEs by a factor of five for the SISR task. As result, the reverse diffusion process used for this problem requires T
20 + NFEs in total.
4 Experimental Results
We evaluate our method on four public datasets across a range of tasks, namely burst JDD-SR, dynamic scene deblurring. and SISR. Below we describe our specific architecture and design choices related to all utilized modules.
Training. Our training procedure consists of two stages. We first employ diverse, yet small DF2K (combination of DIV2K []] and Flickr2K (45)) dataset to train a Denoising Module for Gaussian denoising in the sRGB domain with input noise levels ranging in [0, 244.3). These noise levels corresponds to timesteps in the range of [0, 250] for the diffusion process with T = 1000. We use the original training procedure of MIRNet [8]] learn the parameters of our MIRNet-S architecture. At the second stage we train our Fusion Modules with ny = 64 for each IR task and the corresponding pre-trained off-the-shelf IR network. It is worth noting, that this stage the parameters of Denoising and IR modules are kept frozen and only the Fusion Module is trained. Specifically, we train it for 300k iterations with learning rate of 10-4 x 0.99 it/1000, batch size of 128, and crop size of 256 x 256. To train our Fusion networks we use datasets that are common among our main competitors, specifically the ZurichRAW2RGB [29] dataset for burst JDD-SR, GoPro [53] for dynamic scene deblurring and DIV2K I for 4x SISR. For burst JDD-SR the Fusion network is trained in the sRGB domain. All the networks are trained using the Ascend 910 AI accelerators 44]. To make our results reproducible, we provide a full description of the training procedure in appendix H
Inference. For each IR task we use the procedure described in section to obtain the reconstructed images with 7' = 1000. To demonstrate the effectiveness of our approach, for each problem of interest except for burst JDD-SR we need half of the NFEs compared to the diffusion- that uses the least number of sampling steps. Specifically, for dynamic scene deblurring we use T = 5, resulting in 200x acceleration achieved solely by our proposed sampling strategy. This amounts to 6 NFEs when counting the additional IR network evaluation performed to skip the first T -= = 995 steps using eq. (8). For SISR we select r = 250, which results in 4x acceleration using our sampling procedure. In order demonstrate how it can be complemented by other proposed acceleration strategies, for the final r = 250 steps we achieve 5x step reduction by employing DDIM sampling (66). The combination of both acceleration strategies results in 20x step reduction and 51 NFEs overall. Applying the DDIM acceleration technique on top of our proposed one-step strategy leads an insignificant quantitatiee/quaiiatvee difference (see appendix G compared to our original scheme. Since for the burst JDD-SR problem no diffusion- proposed, we use the same setting as for the dynamic scene deblurring problem, as it requires the smallest NFEs. In all our experiments we use the linear scheduling of the diffusion process variances 3, [2 x 10 - 10 defined in eq.
competitor
methods have yet been
Evaluation. For the burst JDD-SR evaluation we use the SyntheticBurst test set 44, consisting of 300 synthetically pre-generated raw burst sequences. Each sequence contains 14 noisy raw LR images with handshake motion, whose corresponding targets have a resolution of 320 x 320. Since networks trained sRGB images, the outputs of all methods are converted to the sRGB space prior to comparison. For dynamic scene deblurring we evaluate on the GoPro test [53] and HIDE [64] benchmarks, which contain 1111 and 2025 images of 720p resolution, respectively. For 4x SISR we use the DIV2K validation dataset [ consisting of 100 images of 2K resolution.
For the quantitative evaluation of the reconstruction quality we rely on the widely used fidelity metrics PSNR and SSIM (77). and the fiference-bbsed perceptual metric LPIPS (86). Morcover,
7

---

Table 2: Performance evaluation on the GoPro and HIDE test sets for dynamic scene deblurring. 1 indicates that public implementation is unavailable and the scores are copied from the authors' paper. We highlight the overall best for each metric, and the best among perceptual-orietten methods.
Methods
GoPro
HDDE
NFE + Params 4
PSNR
SSIM? LPIPS.. T
PSNRt
SSIM LPIPS, +
Target
0C
06
N/A
IINNet
32.77
0.960
0,088
0.033
30.33
0.932
0.120
0.044
N/A
88.6M
MPRNet
32,66
0.959
0.089
0.027
30.96
0.939
0.114
0.059
N/A
20.1M
MIMO-UNat+
32.44
0.957
0.091
0.034
29.99
0.930
0.124
0.028
N/A
16.1M
NAFNet
33.71
0.967
0.078
0.017
31.32
0.943
0.103
0.024
N/
67.9M
Restomeer
32,90
31,20
0.942
0.109
N/A
FFTFormer
34.21
0.969
0.071
0.012
31.62
0.946
0.096
0.006
NA
16.6M
Perceptual oriented Methods
DeblurGANZ2
0.025
27.51
0,885
0.159
N/
60.9M
DvSR
31,66
0.948
0.059
29.77
0.922
0.089
500
26.1M
icDPM
31.19
0,943
0,057
29.14
0.910
0.088
500
52.0M
InDiT
31.49
0.946
0.058
10
27.7M
Ours
33.72
0.963
0.053
0.011
31.32
0.937
0.087
0.002
33.2M
Input: 0.424
HINer: 0.127
Restormer: 136
DeblurGANv2: 0.171
InDI: 0.080
Target
NAFNet: 0.083
FFTFormrr: 0.059
DvSR: 0.072
Ours: 0.059
Visual comparisons the GoPro test set for the task of dynamic scene deblurring (best viewed by zooming in). Every output image accompanied by its LPIPS value.
Figure 3:
also utilize the non-reference image quality assessment (NR-IQA) metric TOPIQ 8 and report the absolute distance between the output and the target scores, which we indicate as TOPIQs.
4.1 Results
Burst JDD-SR. In Table I we compare our proposed pipeline with existing methods, namely DBSR 4. DeepRep 5 and the current SOTA methods, namely BIPNet [17. BSRT-Small [50), BSRT-Large [50], and EBSR [4)]. Our method demonstrates SOTA performance across the perceptual metries while maintaining competitive PSNR and SSIM scores compared to existing methods. Thus, our method reconstructs images that are closer to the target based on the human perception while maintaining a high level of fidelity. We refer to figures in the appendix M for a qualitative visual assessment. Furthermore, we notice an improvement in terms of visual quality and perceptual metrics compared to BSRT-Small, which we use as the IR module of choice in our framework. This indicates that approach preserves the fidelity of the IR model outputs, while enhancing their perceptual quality by running few reverse diffusion steps. It is worth noting that for this case, where a burst of raw images serves input, we use the exact same denoising network that and which we later deploy to all considered single-input IR tasks. This highlights the generalization ability of our approach not only different IR problems but also to different input formats. Note that AFCNet [51] LKR [77, and Burstormer 17 are not included our comparisons due to the absence of publicly available implementation (or trained network parameters). Finally, the comparison with SOTA EBSR and BSRT shows that our DP-IR reconstructions compares favorably in terms of visual quality, while not lacking significantly in terms of fidelity.
trained on sRGB images
Dynamic Scene Deblurring. Table show quantitative results on the GoPro 53 and HIDE [4
datasets, respectively. We compare our approach with the SOTA reconstruction-bast
methods:
NAFNet [59], FFTFormer 34 and diffusion-based methods: DvSR 78. InDI [15], and icDPM [59]. Our framework outperforms all competing methods across the perceptual metrics and demonstrates the best perception-diste distorion (P-D) trade-off among all perceptual based methods. Moreover, our DP-IR uses twice less number reverse steps (NFE=5) compared to the state-of-the-an InDI and still achieves better perceptual quality (e.g. L.PPS) and is more consistent with the ground-truth (+2.22dB
8

---

Input: 0.341
SwinIR: 0.123
HCFlow: 0.109
ESRGAN: 0.092
InDI: 0.078
Target
0.128
0.122
0.077
Visual comparisons the DIV2K validation for the task of 4x bicubic super-resolution (best viewed by zooming in). Every output image is accompanied by its LPIPS value.
Figure 4:
Table 3: Performance evaluation on the DIV2K Table 4: Ablation various denoiser and IR networks validation set for SISR. We highlight the overall on DIV2K validation for SISR.
best for each metric, and the best among perceptual- oriented methods.
Denoiser IR Network PSNRT SSIM
LPIPS T
Turget
inf
0
UDF RRDB
21.33
0.377
0.149
0.006
Methods
PSNRT SSIM LPIPS TOPIQ
NFE Paramns
MIRNet-S RRDB
28.12
0.795
0,150
0.014
Target
N/A
N/A
MIRNet SwinIR
28.12
0.793
0.140
0.002
SRResNet
29.07
0.824
0.266
0,046
N/A
1.5M
RRDB
29.48
0.834
0.254
0.038
N/A
16.7M
Table 5:
Ablation on various fusion networks on
SwinlR
29.63
0.837
0.248
0.030
N/A
11.9M
DIV2K validation for 1x SR task
LIIF
29.30
0.830
0.258
0.046
N/A
22.3M
Fusion Module
PSNRT SSIMT
LPIPS
Params
HAT
29.75
0.840
0.245
0.035
N/A
20.6M
TDWA
27.26
0.751
0.266
8385
Perceptul--oiiated Methods
L-DWT
29.27
0.233
ESRGAN
0.758
0.1
0,014
NIA
16.7M
U-Net
28.19
0.785
0,141
31.0M
HCFlow
27.02
0.766
0.124
0.021
N/A
23.2M
Proposed Fusion
27.93
0.777
0.149
0.7M
SwinIR-GAN
24.88
0.734
0,222
0.115
N/A
11.9M
Table 6: Perception-Distortionn trade-off DIV2K
LDM
0.697
0.218
100
validation for 4x SISR.
SRDif
27.14
0.773
0.129
0.008
100
23.6M
InDI
26.45
0.741
0.136
0.009
100
62.3M
Timestop.
100
150
200
250
350
IDM
27.35
0.782
0.147
0,008
2000
116.6M
PSNR 28.77
28 28.24
28.06
27.93 27.81
27.71
Ours
0.793
0.140
0.002
51
28.5M
LPIPS
0.161 0.155 0.152
0.146
in PSNR). Despite the fact that our fusion module is trained only on the GoPro dataset, the gains in perceptual quality do transfer over the HIDE test images, providing SOTA scores for the perceptual metrics. Also, among perceptual-oriented methods, DP-IR outperforms the closest competitor DvSR by 1.55dB in terms of PSNR. Visual comparisons of our method and the SOTA deblurring models: NAFNet, FFTFormer, DvSR, and InDI is depicted in Figure 3 From these results we observe that our model shows noticeable improvement in perceptual quality. Moreover, we have performed computational cost analysis for the diffusion-basee models and significantly outperformed existing methods from ~2 to 100 times (see appendix (F)
Super-Resolution. We compare our method with irconstruction-basee
38
73 42
10, GAN-
Table summarizes the quantitative results the DIV2K validation set. Our solution produces the best fidelity scores among all six perceptual-based methods and the best TOPIQ perceptual metric among all competing methods. The visual comparison in Figure reveells that framework produces super-resolved images that exhibit more refined structures and fine-grained details. Additional examples for all reported IR tasks provided in appendix M
based 73 42 NF-baaed 4 models, and DPMs (60,
39
15
20
Ablation Studies
Modular Approach. One of the main benefits of our framework is its ability to capitalize on the performance of existing restoration networks a relatively low additional computational cost. To showease this, we conduct an experiment on the task of 4x SISR using two different denoising architectures, namely UDP / and MIRNet-S [8I) and two IR architectures, namely RRDB (1 and SwinIR 42). UDP is trained with the same settings as MIRNet-S and the fusion module is retrained for each of the three cases. From Table 4 we observe that the same IR network combined with a better denoising module, leads to better fidelity (see PSNR, SSIM). A same trend, but for perceptual metrics is observed if one upgrades the IR module from RRDB to the SwinIR and keeps the same denoising module. This clearly indicates that one can achieve better results by employing either

---

more powerful denoiser or IR module, without the need to fully retrain the entire score estimator as is the common practice followed in most of the existing methods (e.g. LDM, SRDiff, DySR, etc. )
Fusion Strategies. In this study we use the UDP denoiser and the RRDB network from Table 4 and study several different fusion approaches, namely the Time-dependent Weighted Averaging (TDWA), Learnable Discrete Wavelet Transform (L-DWTT), our proposed Fusion network, and U-Net with time embeddings [56]. TDWA consists of sinusoidal positional encoding followed by a three-layer MLP, which predicts the weights for the timestep t. Those weights are then passed to the 3CA layer (section 3.3 together with the outputs from the denoiser and IR modules to perform the fusion in the image space. In contrast, L-DWT directly learns weights for each scale and channel of 3-level Haar DWT (2. L-DWT has only 30 trainable parameters, which needs significantly less training time and indicates that, as expected, more powerful fusion module leads to better perceptual and reconstruction quality. Overall, we see that the proposed Fusion network shows a better balance between the reconstruction, visual quality and the computational cost.
data. Table S
Perception-Distortion Trade-off. By varying the timestep т, when the denoiser and fusion modules are activated, one can favor the percepiual quality over the reconstruction fidelity (see Table (6. Here, we use the same UDP denoiser and RRDB as the IR module and experiment on the DIV2K [[] validation for the task of 4x SISR. Table shows that the denoiser can operate corresponds to a wider noise range than the one the denoiser initially trained for. Furthermore, we observe the same perception- -iiptoooonnnitooiiinnnnoo trade-off for dynamic scene deblurring task (see Appendix Table 13
> 250, which
Limitations Our empirical findings highlight that the optimal selection of is intrinsically linked to the nature of the reconstruction problem, particularly the output quality of the IR Network. While we have experimentally identified optimal parameters for each test dataset in this study, we posit that a more refined approach would involve tailoring the acceleration parameters on an individual sample basis. However, the absence of a dependable methodology for assessing the quality of the IR and Denoising Networks' outputs at specific diffusion process timesteps especially in the absence of ground truth data- constitutes considerable challenge. This underscores a compelling avenue for future inquiry into adaptive optimization of accelcration parameters.
Furthermore, notable constraint of our approach is its reliance on the efficacy of the employed Denoising and IR modules. As such, for novel image restoration tasks where a pre-trained IR network is unavailable, our framework might be inapplicable. Additionally, for imaging modalities (e.g. medical imaging) lacking a trained score-matching network (denoising module), it is imperative to either fine-tune existing module or undertake comprehensive re-training with appropriate image datasets.
Conclusion
We present modular conditional diffusion probabilistic framework for IR problems along with sampling acceleration strategy that achieves a significant speed-up during the inference stage. Our framework achieves SOTA results both quantitatively and visually on the tasks of burst JDD-SR, dynamic scene deblurring, and 4x SISR without the need for re- re-training on a large pool of data and significant computational cost. This is mainly accomplished by utilizing pretrained models and only training a relatively small fusion module. While in this work we have not exhaustively considered all blind problems, we hope that our results can serve as a positive indication that the perceptual quality of the reconstructed outputs can improve significantly at the cost of only several additional NFEs, making possible a wider adoption of DPMs for IR applications even when there are tight requirements on computational complexity. Our ablation studies indicate that variety of pretrained networks can be used with our method and further improvements on the results can be achieved by utilizing better denoising, IR, and fusion modules.
References
[1] Eirikur Agustsson and Radu Timofte. Ntire 2017 challenge on single image super-resolution: Dataset and study. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2017.
[2] Sanjeev Arora, Rong Ge, Yingyu Liang, Tengyu Ma, and Yi Zhang. Generalization and equilibrium in generative adversarial nets (gans). In International conference on machine learning, pages 224-232. PMLR, 2017.
10

---

[3] Benedicte Bascle, Andrew Blake, and Andrew Zisserman. Motion deblurring and super- resolution from an image sequence, In European conference on computer vision, pages 571-582. Springer, 1996.
[4] Goutam Bhat, Martin Danelljan, Luc Van Gool, and Radu Timofte. Deep burst super-resolution. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 9209-9218, 2021.
[5] Goutam Bhat, Martin Danelljan, Fisher Yu, Luc Van Gool, and Radu Timofte. Deep reparametrization of multi-frame super-resolution and denoising. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2460-2470, 2021.
[6] Yochai Blau and Tomer Michaeli. The perception-diisortion tradeoff. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6228-6237 2018.
[7] Tim Brooks, Ben Mildenhall, Tianfan Xue, Jiawen Chen, Dillon Sharlet, and Jonathan T Barron. Unprocessing images for learned raw denoising, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11036-11045 2019.
[8] Chaofeng Chen, Jiadi Mo, Jingwen Hou, Haoning Wu, Liang Liao, Wenxiu Sun, Qiong Yan, and Weisi Lin. Topiq: top-down approach from semantics to distortions for image quality assessment, 2023.
[9] Honggang Chen, Xiaohai He, Linbo Qing, Yuanyuan Wu, Chao Ren, Ray Sheriff, and Ce Zhu. Real-world single image super-resolution: A brief review. Information Fusion, 79:124-145 2022.
[10] Xiangyu Chen, Xintao Wang, Jiantao Zhou, Yu Qiao, and Chao Dong, Activating more pixels in image super-resolution transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2333772237, 2023.
[11] Yinbo Chen, Sifei Liu, and Xiaolong Wang. Learning continuous image representation with local implicit image function. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 8628-8638, 2021.
[12] Jooyoung Choi, Jungbeom Lee, Chaehun Shin, Sungwon Kim, Hyunwoo Kim, and Sungroh Yoon. Perception prioritized training diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 11472-1148 1481, 2022.
[13] Hyungjin Chung, Byeongsu Sim, and Jong Chul Ye. Come-closer-diin diffuse-fffustiiie Accelerating conditional diffusion models for inverse problems through stochastic contraction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12413-12422, 2022.
[14] Joseph Paul Cohen, Margaux Luck, and Sina Honari. Distribution matching losses can hallucinate features in medical image translation. In International conference on medical image computing and computer-assiste intervention, pages 529-536. Springer, 2018
[15) Mauricio Delbracio and Peyman Milanfar. Inversion by direct iteration: An alternative to denoising diffusion for image restoration. Transactions on Machine Learning Research, 2023. Featured Certification.
[16] Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv preprint arXiv: 141.455 16, 2014.
[17] Akshay Dudhane, Syed Waqas Zamir, Salman Khan, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Burst image restoration and enhancement, In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 5759-5768, 2022.
[18] Michael Elad and Arie Feuer. Restoration of a single superresolution image from several blurred, noisy, and undersampled measured images. IEEE transactions on image processing, 6(12): 1646-1658, 1997.

---

[19] Sina Farsiu, M Dirk Robinson, Michael Elad, and Peyman Milanfar. Fast and robust multiframe super resolution. IEEE transactions on image processing, 13(10):327-1344, 2004.
[20] Sicheng Gao, Xuhui Liu, Bohan Zeng, Sheng Xu, Yanjing Li, Xiaoyan Luo, Jianzhuang Liu, Xiantong Zhen, and Baochang Zhang. Implicit diffusion models for continuous super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 10021-10030, 2023.
[21] Ioannis Gatopoulos, Maarten Stol, and Jakub M. Tomezak. Super-resolution variational auto- encoders, 2020.
[22] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[23] Baisong Guo, Xiaoyun Zhang, Haoning Wu, Yu Wang. Ya Zhang, and Yan-Feng Wang. Lar-sr: A local autoregressive model for image super-resolution. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 1909-1918, 2022.
[24] Alfred Haar. Zur theorie der orthogonalen funktionenssysteme. Georg- Boong-uuuust-Unvvrsittaa Universitat, Gottingen., 1909.
[25] Samuel W Hasinoff, Dillon Sharlet, Ryan Geiss, Andrew Adams, Jonathan Barron, Florian Kainz, Jiawen Chen, and Marc Levoy. Burst photography for high dynamic range and low-light imaging on mobile cameras. ACM Transactions on Graphics (ToG), 35(6):1-12, 2016.
[26] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, pages 6840-6851. Curran Associates, Inc., 2020.
[27] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4700-4708, 2017.
[28] Aapo Hyvärinen. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(24):695-700 2005.
[29] Andrey Ignatov, Lue Van Gool, and Radu Timofte. Replacing mobile camera isp with a single deep learning model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 536-537, 2020.
[30] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part Il 14, pages 694-711. Springer, 2016.
[31] Tero Karras, Miik Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In Proc. NeurIPS, 2022.
[32] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 14Xiv:1412.6995 2014.
[33] Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. In 2nd International Conference Learning Representations, ICLR 2014, 2014.
[34] Lingshun Kong, Jiangxin Dong, Jianjun Ge, Mingqiang Li, and Jinshan Pan. Efficient frequency domain-based transformers for high-quality image deblurring. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5886-589 2023.
[35] Orest Kupyn, Volodymyr Budzan, Mykola Mykhailych, Dmytro Mishkin, and Jiř Matas. Deblurgan: Blind motion deblurring using conditional adversarial networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 8183-8192 2018.
[36] Orest Kupyn, Tetiana Martyniuk, Junru Wu, and Zhangyang Wang. Deblurgan-v2: Deblurring (frders-of-mannitude) faster and better. The IEEE International Conference on Computer Vision (ICCV), 2019.
12

---

[37] Bruno Lecouat, Jean Ponce, and Julien Mairal. Lucas-kanade reloaded: End-to-end super- resolution from raw image bursts. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2370-2379. 2021.
[38] Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zchan Wang, et al. Photo-realistie single image super-reson resouutooo using a generative adversarial network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4681-466 1-940, 2017.
[39] Haoying Li, Yifan Yang, Meng Chang, Huajun Feng, Zhihai Xu, Qi Li, and Yueting Chen. Srdiff: Single image super-resolution with diffusion probabilistic models. arXiv preprint arXiv:2104.1499 14951, 2021.
[40] Yawei Li, Yulun Zhang, Radu Timofte, Luc Van Gool, Lei Yu, Youwei Li, Xinpeng Li, Ting Jiang, Qi Wu, Mingyan Han, et al. Ntire 2023 challenge on efficient super-resolution. Methods and results. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1921-1959, 2023.
[41] Wenyi Lian and Shanglian Peng. Kemel-aware burst blind super-resolution. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 4892-4901 2023.
[42] Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1833-1844, 2021.
[43] Jingyun Liang, Andreas Lugmayr, Kai Zhang, Martin Danelljan, Luc Van Gool, and Radu Timofte. Hierarchical conditional flow: A unified framework for image super-resolution and image rescaling. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4076-4085, 2021.
[44] Heng Liao, Jiajin Tu, Jing Xia, Hu Liu, Xiping Zhou, Honghui Yuan, and Yuxing Hu. Ascend: a scalable and unified architecture for ubiquitous deep neural network computing. Industry track paper. In 2021 IEEE International Symposium on Perg--Pefforancc Computer Architecture (IIPCA), pages 789-801, 2021.
[45] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep residual networks for single image super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 136-144, 2017.
[46] Andreas Lugmayr, Martin Danelljan, Lue Van Gool, and Radu Timofte. Srflow: Learning the ruper-resolution space with normalizing flow. In European Conference on Computer Vision, pages 715-732. Springer, 2020.
[47] Andreas Lugmayr. Martin Danelljan, and Radu Timofte. Ntire 2021 learning the super-resolution space challenge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 596-612, 2021.
[48] Calvin Luo. Understanding diffusion models: A unified perspective, 2022.
[49] Ziwei Luo, Lei Yu, Xuan Mo, Youwei Li, Lanpeng Jia, Haoqiang Fan, Jian Sun, and Shuaicheng Liu. Ebsr: Feature enhanced burst super-resolution with deformable alignment. Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 471-478. 2021.
[50] Ziwei Luo, Youwei Li, Shen Cheng, Lei Yu, Qi Wu, Zhihong Wen, Haoqiang Fan, Jian Sun, and Shuaicheng Liu. Bsrt: Improving burst super-resolution with swin transformer and flow-guided deformable alignment. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 1908,008, 2022.
[51] Nancy Mehta, Akshay Dudhane, Subrahmanyam Murala, Syed Waqas Zamir, Salman Khan, and Fahad Shahbaz Khan. Adaptive feature consolidation network for burst super-resolution. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 1279-12880 2022.
13

---

[52] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2022.
[53] Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. Deep multi-scale convolutional neural network for dynamic scene deblurring. In The IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2017.
[54] A. Nguyen, J. Clune, Y. Bengio, A. Dosovitskiy, and J. Yosinski. Plug & play generative networks: Conditional iterative generation images in latent space. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1510-3520, Los Alamitos, CA, USA, 2017. IEEE Computer Society.
[55] Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. arXiv preprint aat:::2202097722 2021.
[56] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabílistic models. In Proceedings of the 38th International Conference on Machine Learning, pages 8162-8171. PMLR, 2021.
[57] Mangal Prakash, Alexander Krull, and Florian Jug. Fully unsupervised diversity denoising with convolutional variational autoencoders. In International Conference on Learning Representations, 2020.
[58] Dongwei Ren, Kai Zhang, Qilong Wang, Qinghua Hu, and Wangmeng Zuo. Neural blind deconvolution using deep priors. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 3341-3350, 2020.
[59] Mengwei Ren, Mauricio Delbracio, Hossein Talebi, Guido Gerig, and Peyman Milanfar. Multiscale structure guided diffusion for image deblurring. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10721-10733, 2023.
[60] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684-10695, 2022.
[61] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. arXiv preprint DrXiv:2104.07636 2021.
[62] Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
[63] Richard R Schultz and Robert I. Stevenson. Extraction of high-resolution frames from video sequences. IEEE transactions on image processing, 566);996-1011, 1996.
[64] Ziyi Shen, Wenguan Wang, Jianbing Shen, Haibin Ling, Tingfa Xu, and Ling Shao. Human- aware motion deblurring. In IEEE International Conference on Computer Vision, 2019.
[65] Jascha Sohl-Dickstein, Erie Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Proceedings of the 32nd International Conference on Machine Learning, pages 2256-2265. PMLR, 2015.
[66] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021.
[67] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
[68] Deqing Sun, Xiaodong Yang, Ming- Yu Liu, and Jan Kautz. Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8934-8944 2018.
14

---

[69] Fu-Jen Tsai, Yan-Tsung Peng, Yen- Yu Lin, Chung-Chi Chi Tsai, and Chia-Wen Lin. Stripformer: Strip transformer for fast image deblurring. In European Conference on Computer Vision, pages 146-162. Springer, 2022.
[70] R Tsai. Multiframe image restoration and registration. Advance Computer Visual and Image Processing, 1:317-339, 1984.
[71] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[72] Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Computation,
2011.
[73] Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong. Yu Qiao, and Chen Change Loy. Esrgan: Enhanced super-resolution generative adversarial networks. In Proceedings of the European conference on computer vision (ECCV) workshops, pages 0-0, 2018.
[74] Xintao Wang, Kelvin CK Chan, Ke Yu, Chao Dong, and Chen Change Loy. Edvr: Video restoration with enhanced deformable convolutional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0-0, 2019.
[75] Xintao Wang, Liangbin Xie, Chao Dong, and Ying Shan, Real-esrgan: Training real-world blind super-resolution with pure synthetic data. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1905-1914, 1914, 2021.
[76] Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. In The Eleventh International Conference on Learning Representations, 2023.
[77] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions image processing, 13(4): 600-612, 2004.
[78] Jay Whang, Mauricio Delbracio, Hossein Talebi, Chitwan Saharia, Alexandros G Dimakis, and Peyman Milanfar. Deblurring via stochastic refinement. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 16293-16303, 2022.
[79] Bartlomiej Wronski, Ignacio Dooaad, Manfred Ernst, Damien Kelly, Michael Krainin, Chia-Kai Liang, Marc Levoy, and Peyman Milanfar. Handheld multi-frame super-resolution. ACM Transactions on Graphics (TOG), 38(4):1-18, 2019.
[80] Bin Xia, Yulun Zhang, Shiyin Wang, Yitong Wang, Xinglong Wu, Yapeng Tian, Wenming Yang, and Lue Van Gool. Diffir: Efficient diffusion model for image restoration. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 13095-13108 2023.
[81] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao. Learning enriched features for real image restoration and enhancement. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXV 16, pages 492-511. Springer, 2020.
[82] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuae Yang, and Ling Shao. Multi-stage progressive image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14821-14831, 2021.
[83] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Restormer: Efficient transformer for high-resolution image restoration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5728-5739, 2022.
[84] Lujun Zhai, Yonghui Wang, Suxia Cui, and Yu Zhou. A comprehensive review of deep maaming-based real-world image restoration. IEEE Access, 11211049226667 2023.
15

---

[85] Kaihao Zhang, Wenhan Luo, Yiran Zhong, Lin Ma, Bjom Stenger. Wei Liu, and Hongdong Li. Deblurring by realistic blurring. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 2737-2746, 2020.
[86] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2018.
[87] Wenlong Zhang, Yihao Liu, Chao Dong, and Yu Qiao. Ranksrgan: Super resolution generative adversarial networks with learning to rank. IEEE Transactions on Pattern Analysis and Machine Intelligence, (4410))7149-71666 2021.
[88] Hang Zhao, Orazio Gallo, Iuri Frosio, and Jan Kautz. Loss functions for image restoration with neural networks. IEEE Transactions on computational imaging, 3(1))47-57, 2016.
[89] Zhengxia Zou, Keyan Chen, Zhenwei Shi, Yuhong Guo, and lieping Ye. Object detection in 20 years: A survey. Proceedings of the IEEE, 111(3):257-27,, 2023.
16

---

A Conditional DPM Training Requirements
To the best of our knowledge. all diffusion-basee approaches that have been proposed in the literature to deal with image restoration tasks, require the training of far larger conditional backbone networks (~100-00M params). This turns out to significantly more challenging both terms of necessary training data and computational resources. To showcase this, we provide an indicative example below. If we adopt the existing diffusion- based SISR baselines and train them for a completely different restoration problem, by following the original authors' training strategies it turns out that the computational and data requirements are significantly higher than those our method.
Table Comparison the proposed approach against existing DPM methods for SISR task terms training dataset requirements and training parameters.
Method
Params required
Data required
Ours
1x
1x
SRDiff
34x
LDM
~ 240x
1000x
InDI
89x
IDM
167x
1x
Based on these data, we can safely state that our strategy provides reasonable trade-off between the required training complexity and the competitive performance of our method to variety of blind inverse problems.
B Proof of Lemma 3.1
To derive the conditional score function, we first express the conditional probability p (xx, yy) as:
/ (xt, æolv) dxo = / (ox y 2o) (zoly) dæo / (x0)(0)) p (xo y dxo,
p (xxl)))
(12)
where we used the fact that æ, is conditionally independent of and according to eq. 2 it holds p (wely, mn) g (ar (zo). Differentiating both sides of eq. 11) with respect to (W.r.t)) æ. we get:
/ (xoly) Vn. (0)) dæs-
V Va.p(v
(13)
Based on the definition of a (2))aa)) in eq. -q (2)) æ vo Substituting this result back to eq.
2). also holds that: (a
13 we get:
p(welv)
you
/ 20g (aze)zro) p (xolly) dæy
[ (20e p (xooll) dæc- (14)
Ve,P(Zxl1)) 1 ā.
āi
Next, we divide both sides in eq. 13) with p (xx,1) and use that V.n. log (2x11) we get:
Va.p(a))))
p(a.ly)
q (are (are p (2oly) dxo xe (15)
VR, logp(1::l)) (z:ly) = (1 xx)
(Va
xo
pxx
We further express the integral in eq. (15) follows:
f xu q (ax (2p) (xoly) dxy = xo 9 (ate (20))0o) p (wo y) (1)
dx = f xu (x y)
(x)a (a0o y) (2t,, 4) doc
(wxly)
(10, 1)) p (3)
= / æco p (xt, No. y) (xt, v)
dx = zop (wo xx) dx
E [x0 #t)
Substituting this result in eq. 15) finally leads us to the result of the lemma: Va.p(x:11) /an R R(moly.m
17

---

C Theoretical Justification of the Conditional Expectation Approximation
By construction, for the forward process from eq. (1) holds that ær N (0. Ix) when T oc, which means that in the beginning of reverse sampling, the latent variable ær does not contain any information about zg- Below provide theoretical result that can serve that during the sampling process and up to some timestep т, we can use an approximation of ky, æt] E [xol given that the contribution of E (wo [2r,] is significant enough. This is achieved by disabling the lower branch of our proposed conditional score matching network, which includes the Denoising Φg, (Te, στ) and Fusion Φē (z xx)) modules (Figure Ib. A formal theoretical analysis is available under the following simplifications.
indication
The observation model is linear:
Awo +
Denoising Network is an identity transformation:
60, (it, σz) = Ct.
Image Restoration Network is back-projection: 00, (y) = A
Fusion Network is a convex combination in spatial domain: Φ5, (x.²...)) a.. wwx
(1 - w)=
Diffussin process approaches the continuous- time regime: T + 20.
Our theoretical result provided in the form of the following proposition.
Proposition C.1. Let E R be the measurements obtained according to the following observation Then, for any given y, E RNN, and any fixed E (0,1), there exists a timestep т, such that for all > the back-projected signal A y approximates By better than the convex combination A + (1 -w) in the following sense:
model: y A + n, where A E RMxN || S 1 and n N (0, JyJM) M S
Eyjan IDo A " 11 I Ey mm 1000 (wA") y + +1-
(16)
where 3e ~ N (x o [ o² 1-ae is defined as a noisy version of an within the diffusion process described by eqs. 1 and
or with T -> xc.
For our proof we use the following intermediate result.
Lemma C.2. Let o E RT, x p (x) N (0, diag (o²)), B Raxx, Then the following holds:
/
Ix Bx p (x) dx (diag B)
(17)
Proof. We first note that since Cov (a) = diag (o²), the random variables æ; are independent, p (x) II (x;), and distributed as a; (x,) = N of). We further note that it holds:
x Bæ
B
Σ D B
B
(18)
Using these results, the derivation of eq. 17 is straightforward:
f n Bæ p (x) do Σ
n
f a.p (mx) dx: f a, (or,) day II / p (mx) das) (19)
n1
B'
1--=1
kfi.j
+
B (ppp(x)) (mx) da: II
(mx) dry =
B = tr (diag (a²) B),
1=1
since for all i = I, it holds that f p (x:) dæ; = 1, xxp (æ:) dx; O. xip (x:) dx; = o.
We start our proof from computing the left part of the inequality in eq. (6). Since y Ar n.
where n ~ N (0, or M, we have that: p (y)mz) = N (Amo o3 TM). This allows us to write:
y Azo 11
E Eylar xD - A ATy
/ /| - ATylppyy dy / |1:00 - AT yyy
exp
20
dy IT.
M
18

---

We now apply change of variables from y to n y Azo, for which dn = dy. Using the fact
that
exp
|n]]2 203
= p (m)) = N (o, Aa)) we have that:
(v2ndy)
I1 = / |1:00 - Axo - ATT n2 2 (n) dn.
(20)
Next, we denote Awo 20 A Axr = -IIN A A) wn and expand the norm inside the integral to get:
I1 =1 Aao / p(n) dn -2 mp (n) dn + I AAAAAA np (n) dn.
Since p (n) N (0,00 is a zero-mean Gaussian probability distribution, holds that (n(n) dn I np (n) dn O, and we get:
I = Aut Amo + n T AA mp (n)))) dn = Anj + tr (o())AAT) = Aaj Amo + +
(21)
where we have used the result of lemma C.2 Substituting back the value of Azg, we end up with:
Ey ||:00 A y 2 1 (IN - A A) m + o A|
(22)
Next, we compute the right part of the inequality in eq. (16). We first note, that the quantities y Axo + n and at vā,zg + Et are independent when conditioned on Vp. as n N (MM) and E, N (O, (1 āe) IN) are independent random noise vectors. This allows us write: p 2x = p (yl (n) p (ara (an)). Using these results, the expectation Ey. (e0 |1:00 (wATy y + (1 - w) ))) l2 can be written as:
yy Im 120
(WAl y (1-w)):)1
(23)
-I2
||a00 -
(wATyy + (1 (1ww w) Ør
Vos 112 p ( (y) mo) (xx) dyda..
Similarly to eq.
(20). we apply the following change of integration variables: n = y Axo, dy = dn
and 6: Dt / dx: de,. We additionally note, that To (wAyy y + (1-w)
vār xe
2o wA1 T Azo (1 (1-wi w) xo - WA
ATn n L- Et, and that vo - wA Axo - (1 - w) 20 =
va. w IIN ATA) X0 = wAo. As result, the integral Ia takes the form: A)
Is / ||wARo- - WAn 1 w
Er (1)) (Er) drde:
(24)
Vau
Now we use the augmented variable
é= [n
e[] (p) (E:) di.g (Lo3ITs (1- p((e),
where with 1, we denote the vector of dimension i filled with ones. We also denote with W 1- IN as in this case, We WAT An Jou
WAT T
1- e.. With all these modifications, the integral
va.
In takes the form:
/ ||wAxo- Welāp((e) de w² AaAafo / p(e) de 2w w / /Eple) de (25)
Ig
+ / / <wTTWee(()) de = u Axj Auo + tr (diag (Lo2ITs (-- wTw).
Further, we define D = diag (1oi1Ta (1- (1- 1>1") and compute the respective trace in eq. (55
as:
tr (DW W) = tr (WDWT)& tr ((WD!?2 ( (WDW?)))
)) /WD
= (1 w)² 1 1 ai
IN 1P-- w20i/ Al2: +(1 w) 1 جة N.
02
19

---

Based on the above. the integral Zg takes the form:
12 = = Amo + w/ 2 + (1 - w)² =w) ōt
= = + =²]
δe
N.
(26)
āi
a2
As a result of our derivations, the inequality from Proposition C.1 takes the form:
AV y S w² ya aoloollaa al0 A y (1 w)² āt N.
y 1|:00
(27)
A The quantity E ylu æo
y11 has finite value, as it can be upper bounded as follows:
ylo mo 200
A y (IN A zo 1 + و A
IN A A 2 2D 1/ o min (M, |
S (111l22 + 3] N min (M, A 122 S 4N min (M,
In the above chain of incqualities we have used the submultiplicative property and triangle inequality for lg matrix norms together with the norms equivalence inequality for l and Frobenius matrix norms, where we also upper-bound the rank matrix with its minimal dimension. Additionally, we have used the assumptions of Proposition C.1 specifically dy 1 and | I 1.
As to the second term of the rhs of eq. diffusion process, it holds that lim,-7 following condition:
we note that since lim 02 = 0 by design of the This formally translates to the
lim I-ās
ENDENN: 1 Ot
VE E
2c
(28)
ā
Selection of = 1he (4 + min (M/N,1)) 2 Hee Bylm
agg Avlll
0 into eq. (88 translates it to
T-w
(1 - w²
Eylan a00 A y 1
Yw (0, (0, (w) E N: V
(29)
w) N
which concludes the proof given the equivalence of eq.
27 and eq. (16)
D Proof of Lemma 3.2
We prove the correctness of the transition kernel formula by induction on k.
Base case. First, we focus on the case k 1, for which eq. ( should match the transition probability from eq. Indeed, from eq. (TO). it holds:
1
M-1 De 1 1== =
(30)
1 Voa (1 -
(31)
1 1- Ctt
Σ (T)= 2 of (T/-1)² og = o?-
(32)
Substituting these results in eq. 9 and then in eq.
g leads us to eq.
3
Induction Step. Let the induction hypothesis from eq. 8 to valid for some k, such that 1- k (0, T). Then, need show that eq. (8) is also valid for 1.
We note, that from eq. (3) we have Pt-k-1 = p4- (aq,k,, Tn) + o5- where 61 N (0, IN). At the same time, since eq. (8) is valid for A by the induction hypothesis, we have that æt-k
20

---

Ht.k Dt, To) + J1,62, where ~ N (0, In). Combining these two equations with eqs.
3
8
and D. we get:
Lt 11-k-1 va--k-1 Pp-k
00-k (1 kt-k-1)
20 +
21-k + Jc-KE1
(33)
- Ōz-k
Vot-h (1 0
R-1
pl- pt- 7--
Vas-k-13t-k
(a-kk a4
-
1)
D. p
k+1
+ 2t
1 āi-k
1 ai-k
1=0
2 k-1
04 k (1 04- k-
o2_,62 0.1EE1 (1
(1 kx-
To move further, we first prove that Vi,j E N 2 J. r from eq. (10 can be decomposed as: = r This easy to show by direct substitution, that is:
ryr vaj 1 0j-1 1 āj
dj an II
0j-1
1 --aj1 IV
an 1 ā
On 1 ā,
a; 11 11
n 3+1
(34)
We additionally note that it holds:
/ aa- (1 0z-k-1) pl-k
(35)
1 - Ö4-k
1-k
14-k-11 13-k
7/- - -
(36)
of k (Ti-k (f(-k-1)
oē-K=
(37)
as Pi-k [-k-1 = 1, since - -
First, we compute the multiplier of æz in eq. derive the following result:
33
Since 2 t-t-k + Vk E N+ we can use eq.
34
O āt-k-) pe-k+1 k+l - r
(38)
1 1-ār-k
where we have additionally utilized the result of eq. (35) Next, we simplify the multiplier of æo from eq. 33 We divide the sum inside this multiplier into two parts:
k-2
2-2
r 7--1 Σ r Ta
7 It-k+1
k+1
Σ
rt- 1-1111 7 + N--+11
(39)
where separate the 7--A-1 term we have used the lower case of I from eq. 10 Here additionally note, that if 1: = 1, then the remaining sum E
1-i-1 rt-k+1 becomes zero. If > 1, then
for all the terms of this sum it holds t- & - > t- k 1. Combining this with the results of eqs. to (36). we can compute the multiplier of æo from eq. 33 as:
(34
04-k (1 Ōr-k-1
var-k- 18-k
1 -aakk
#-
7-1
Ōr-k
t-1-1
m + 70--14441
1=0
1=0
k-2
A
+r
7
Ti-i-
7- i=k-
[
pt- 1-1-1 Nt-s-
(40)
To simplify the remaining part of eq. 33
which involves the terms with the Gaussian noise vectors
E and E2, utilize the fact that Va, b
E1 N (O,IN) holds: vaei + Vbeg
21

---

N (0, (a + b) IN). Based on this, we can write:
0t-h à (1 Or ā
(T==+t)) o2_ 22_ 0 ~ N (0,02,
(41)
where, if additionally the results of eqs.
35 and (37 have that:
2,++ = (I
(I)
or + (Ti= 07
(42)
==0
Now, we use the same strategy as in eq.
39
and eq.
40 to simplify this expression:
Σ
(
of_ (T
of + (Ti-k-1) o
E of
1,
(43)
From eqs.
38
40
and 43
we get:
ht-k-1 ~ N (pe o2,
(44)
where
Pc,+1 (xt, vo) = Σ
pt 1-i- 1 + I
(45)
1=0
k (ri )) oi=
1,441
(46)
k=
By direct substitution of le k + 1 into eq. 99) it is casy to show that
j1,,+1 (v) M1 (0),
0 = 0i
which implies that the induction hypothesis from eq. ( is valid for le + 1. This completes the proof of the induction step and combined with with the base case it proves by induction the validity of eq. (8) for every feasible k:
E
Comparative Analysis: Proposed Accelerated Sampling and Prior work
E.1
Conceptual Difference
Using our notation, both 13 and 52 propose to start the reverse process from timestep and noisy version æ, of the initial estimate æg. which we denote by E [xoly]] The main conceptual difference of our approach is that in these cases is obtained using the forward diffusion process, while our case we end up in æ; using the reverse process. The initial motivation for our proposed approach is also different. In particular, while we motivate our procedure from a probabilistic viewpoint and propose to approximate the conditional score function as a composition of three functions, the authors in [13] base their strategy on the contrastive property of reverse SDEs, while the authors in [52) use the re-projection of unrealistic images to the manifold of natural images in the noisy latent space.
E.2 Technical Difference
Given that in our work we consider the standard DDPM realization of diffusion process (VP-SDE). we will explain the existing differences under this scenario. The authors of [13 and 52) propose parameterize x, as
a
E
by] + vI a, z N (0,1).
22

---

In contrast, in our case by using (q.(10) we adopt the following parametrization:
T-r-1 F =
T-T-1 2
+++1 --i- 1-
(zaly) + r *r xT +
Σ
(TT-i-1)
07
1=0
where 2, ~ (
Thus, parametrization is more general and it is possible show by induction, that under certain conditions leads to the exact same 2, as in 13 and [52].
Finally, to experimentally demonstrate that our approach exhibits certain benefits compared to the ones described in [13] and [52] we conducted additional comparisons for the SISR problem between the different sampling strategies (see Table [8. From these results it is clear that our proposed strategy works better practice and leads superior results both in terms of fidelity and perceptual quality.
Table 8: Comparison of the proposed acceleration scheme and prior works 13
52
for the SISR task.
Acceleration Strategy
PSNR
SSIM f
LPIPS +
TOPIQ
NFE
Ours
28.12
0.793
0.140
0.002
51
B and
32
28.05
0.783
0.142
0.016
51
F
Computational Cost Analysis
In this section, caleulate the computational cost for each diffusion-based method in terms of TFLOPs. The input image size for all competing approaches is 720p (1280 720).
Table 9: Computational cost of the proposed and existing diffusion-based methods the Dynamic Scene Deblurring task with 720p input resolution.
Method
TFLOP (equation)
TFLOP Total +
DvSR [78]
NFE + 4.8
604.8
icDPM [59]
4.8x 5.2
2405.2
InDI [55
NFE
48.0
Ours
4.3xNFE 1.9
23.4
The proper way interpret equations Table is as follows: TFLOP total = N + y. where is the TFLOP complexity for single backbone pass within the diffusion process, N is the total number of neural function evaluations (NFEs) per sampling process, and is the complexity of sub-modules that have to be run once image (e.g. Image Restoration network in our method, pre- pre-processing net for icDPM (59 and DvSR (78). Based on these results, we observe that the computational cost of our method is significantly lower compared to our diffusion-based competitors.
G Proposed one-step acceleration with/without DDIM
Utilizing our one-step acceleration, we bypass steps from to r, allowing us either straighfforwardyy execute the remaining f r steps apply any existing acceleration strategies. To demonstrate that our one-step acceleration complementary to existing accelerated sampling strategies, combined the DDIM [66) acceleration technique with our one-step acceleration for the single-image super-resolution (SISR) task (refer to Table [O]. We notice that no significant quantitative/quaitttvee difference after applying this acceleration technique has been observed.
Table 10: Results for the proposed one-step acceleration with/without DDIM acceleration tested on SISR task
Method
PSNR
SSIM
LPIPS 1
TOPIQ 4
NFE
Ours (with DDIM)
28.12
0.793
0.140
0.002
51
Ours w/o DDIM
28.16
0.794
0.139
0.007
251
23

---

H
Training Procedure
Denoising Module. We use two denoising architectures, namely MIRNet-S and UDP for our main experiments and ablation studies. MIRNet-S (15.95M parameters) is lighter version of the original MIRNet (31.78M parameters) [1] where MSRB is decreased from 2 1. In contrast, UDP (11 (11.78M parameters) 77] architecture is not modified and used as it is. Both models are trained for the Gaussian denoising task in the sRGB domain with input noise level E [0, 244.3). More specifically, for each element of batch the noise standard deviation is selected randomly using uniform sampling within this range. The batch size, training crop size, and initial learning rate are set to (in total), 192 192, and 2- 10-4, respectively. Overall, all denoisers are trained for 1M iterations 8 Ascend 910 AI accelerators using the Adam (32) optimizer with default parameters and a decaying learning rate scheduler: Ir, Ira * 7y*/20001, where y 0.999. We employ an MSE loss to train the denoising model and concatenate the noise level with the noisy image as an input to the denoiser same as
IR Module. As mentioned in the main manuscript, we utilize existing models with publicly available pretrained parameters. For the tasks of burst JDD-SR, dynamic scene deblurring, and SISR we have used BSRT-Small [50), FFTFormer [34,, and SwinIR [42), respectively. A description of each one of these IR networks, including their number of trainable parameters, is provided in Table II
Fusion Module. For each IR task under study, we follow the same protocol. We train the fusion module for 300K iterations with batch size of 128 (in total), and crop size of 256 256. The training takes place on 8 Ascend 910 Al accelerators, while the optimizer of choice is Adam 32 with default parameters and a learning rate scheduler lr.- = Irp * >ls/100u], where 7 = 0.99. For each one of the studied IR tasks we train our Fusion module on dedicated dataset. Specifically, we use the ZurichRaw2RGH (29] dataset for burst JDD-SR, GoPro [53] for dynamic scene deblurring, and DIV2K [1] for SISR. The selection of these specific datasets is motivated by the fact that they widely used by all the competing methods for network training related to the IR tasks of interest. The detailed description of training data we used for each problem is provided below.
JDD-SR. Following the same protocol as in 4 37 49 5
17 50 we generate 46K burst sets from the training set of ZurichRAW2RGB Each burst set, which contains 14 low-resolution images in the raw domain, is inferenced by BSRT-Small [50] and post-processed to produce an image in the sRGB domain. Then, these predictions are used as input to train only our Fusion module, while Denoising and IR networks are frozen.
Dynamic Scene Deblurring. We follow the standard protocol for this problem and use the GoPro dataset for training. Specifically, we use 3214 pairs of clean and blurry 1280 x 720 images, out of which we have excluded the 1111 pairs reserved for evaluation purposes. In order to provide a fair comparison, we follow exactly the same setup as in [35 85 83 34 and train only the fusion module using the provided GoPro training data.
SISR We employ the well-known DIV2K [ dataset for the SISR task. This dataset contains a set of 800 images of 2K resolution, which we used for training our Fusion module. Following the standard protocol, we use the additionally provided 100 2K images for evaluation.
I Fusion Module
The main goal of our proposed Fusion module is to predict the conditional expectation E [xo rolly, ær] given the estimates of EE [x,] and [zuly]], which are produced by the denoising and IR modules, respectively. To do so, our fusion network accepts as inputs the image estimates x x and a timestep t. Its exact architecture is depicted in Figure it consists of two branches. The upper branch operates on z 20 JR and produces the corresponding features f1, f2 using single dense block (27] without a normalization layer. The lower branch encodes the timestep / into a vector of weights uz E (0, 1)"r using the sinusoidal positional encoding [71), followed by a two layer MLP, and sigmoid function as the final activation. Then, we perform a weighted summation of fi, f2 in the feature space. Finally, two consequent dense blocks (27), with ny channels each, followed by
D
24

---

Fusion Network: &0r (x 2b m
Conyll
20
IRR
DeneBllak
I:
Denectbek
Dotectlle
Camr3D
EE
æ y. 2t,]
Linea
DanseBookk
Sinuwoidi Postioral Encoding
R..
network ustput
Denoioer corgut
Figure 5: Detailed visualization of the proposed Fusion module.
Table 11: Number of parameters and source code of methods used as IR module in our framework.
IR Modules
Parameters
Code & Weights
ESSRT-Smal 4.92M
link
FFTFormer
16.56M
Innk
SwinIR
11.85M
link
a convolution layer produce the final estimate of E [2o y. Øz]. Our proposed architecture has only 0.73M learnable parameters, which is significantly lower (7-22 times) compared the Denoising and IR modules. As result, the Fusion module requires less training time/resources and can be trained only small amount of problem-specific training data.
Fusion Module Robustness
In order to assess the robustness/generalzzaiie ability of our method, we have conducted additional experiments where we evaluate the reconstruction quality achieved when the Fusion network, which was originally trained on the MIRNet-S and SwinlR pair, is combined with the following pairs of denosing and IR networks:
The Fusion module is combined with the same pair of denoising and IR networks as those used during its training.
The Fusion module is combined with different IR network and the same denoising network as the one used during its training.
The Fusion module is combined with a different denoising network and the same IR network as the one used during its training.
The Fusion module combined with different denoising and IR networks than the ones used during its training.
Table 12: The performance Fusion module for different train/test pair scenarios for 4x SR task. Trained Pair
Tested Pair
PSNR1
SSIM
LPIPSJ
TOPIQ
Target
OC
0
1
MIRNet-S SwinIR
MIRNet-S + SwinIR
28.12
0.793
0.140
0.002
2
MIRNet-S + SwinIR
MIRNet-S + RRDB
28.20
0.795
0.144
0.023
3
MIRNet-S + SwinIR
UDP + SwinIR
28.47
0.808
0.177
0.017
4 MIRNet-S + SwinIR
UDP + RRDB
28.46
0.807
0.183
0.025
From these experiments as shown in the table above, we observe that changing the denoising network to less powerful one leads to noticeable drop in terms of perceptual quality ( (30% in LPIPS) and
25

---

slightly blurrier results, which corresponds to PSNR increase by 0.2dB. Conversely, altering the IR network has a minimal impact on both the reconstruction and perceptual metrics. In summary. under the particular IR task, the Fusion network evaluated with different pair of denoising and IR modules demonstrates good generalization ability and robust behavior.
K
Perception-Fidelity Trade-off
We have evaluated method on the GoPro dataset (motion deblurring) for different when the denoising and Fusion modules are activated. From Table [3 we observe that the perception-fdee trade-off follows similar trend to the for the SISR 4x task. The main difference is that in this particular case the necessary reverse diffusion steps are less than SISR.
Table 13: Perception-
Distortion Perception-Distortion
Trade-off on GoPro validation for dynamic seene deblurring task.
10
20
25
PSNR, dB LPIPS
34.21
34.02
33.72
33.52
33.23
33.14
0.071
0.057
0.053
0.052
0.052
0.052
L Information about Competing methods.
Below, we provide links to the code implementation and trained weights for baselines used for comparison.
Table 14: Code and model weights of the competing methods for burst JDD-SR task.
Method
Link to Code
Link to Weights
DBSR
link
link
DeepRep
link
Innk
EBSR
link
link
BIPNet
link
link
BSRT
Iink
link
Table 15: Code and model weights of the competing methods for dynamic scene deblurring task.
Method
Link Code
Link Weights
HINet
link
link
MPRNet
Iink
Imn
MIMO-UNet+
link
NAFNet
Inn
Restormer
Inn
Im
FFTFormer
link
link
DeblurGANv2
link
1nk
M
Additional Results
In the section, we provide additional visual comparison the proposed method with existing SOTA approaches for all IR tasks under study.
26

---

Table 16: Code and model weights of the competing methods for SISR task.
Method
Link to Code
Link to Weights
SRResNet
link
link
RRDB
Ink
link
SwinIR
Ink
Inn
LIIF
Innk
Iink
HAT
link
link
ESRGAN
HCFlow
Ink
Ink
SwinIR-GAN
link
link
LDM
link
link
SRDiff
Ink
lnn
IDM
Innk
link
DeepRep:
BSRT-Small:
BSRT-Large:
Target
BIPNet: 0.069
EBSR: 0.054
Ours: 0.036
Figure Visual comparison our approach against competing methods on the Burst JDD-SR task (best viewed by zooming in). Every output image is accompanied by its LPIPS value.
DBSR: 0.280
DeepRep: 0.270
BSRT-Small: 0.225
BSRT-Large: 0.258
Target
BIPNet: 0.258
EBSR: 0.252
Ours: 0.131
Figure 7: Visual comparison of approach against competing methods the Burst JDD-SR task (best viewed by zooming in). Every output image is accompanied by its LPIPS value.
DeepRep: 0.108
BSRT-Large:
Target
BIPNet: 0.089
FBSR: 0.083
Oure: 0.042
Figure Visual comparison our approach against competing methods on the Burst JDD-SR task (best viewed by zooming in). Every output image is accompanied by its LPIPS value.
27

---

DeepRep: 0.079
LaRgeL
Target
BIPNet: 0.043
EBSR: 0.035
Ours: 0.030
Figure Visual comparison our approach against competing methods on the Burst JDD-SR task (best viewed by zooming in). Every output image is accompanied by its LPIPS value.
Input: 0.477
HINet: 0.128
Restormer: 0.135
DehlurGANv2: 0.195
InDi: 0.111
Target
NAFNer: 0.1
FFTFormer: 0.094
Dv5R: 0.106
Ours: 0.092
Figure 10: Visual comparison of our approach against competing methods on the GoPro test set for the task of dynamic scene deblurring (best viewed by zooming in). Every output image is accompanied by its LPIPS value.
Input: 0.547
HINet: 0.171
Restormer: 0.137
DeblurGANy:: 0,284
ieDPM: 0.121
Target
NAFNet: 0.133
FFTFormr:: 0.122
DvSR: 0.127
Ours: 0.101
Figure 11: Visual comparison of approach against competing methods the GoPro set for the task of dynamic scene deblurring (best viewed by zooming in). Every output image is accompanied by LPIPS value.
Input: 0.533
HINet: 0.157
Restormer: 0.095
DeblurGANv2: 0.213
icDPM: 0.112
Target
NAFNet: 0.086
FFTFomer: 0.105
DvSR: 0.134
Ours: 0.097
Figure 12: Visual comparison of our approach against competing methods on the HIDE test set for the task of dynamic scene deblurring (best viewed by zooming in). Every output image is accompanied by its LPIPS value.
28

---

Input: 0.402
HINer: 0.242
Restoreer: 0.248
DeblurGANY2: 0.303
icDPM: 0.160
Target
NAFNet: 0,207
FFTFormrr: 0.182
DvSR: 0.192
Ours: 0.150
Figure 13: Visual comparison of our approach against competing methods on the HIDE test set for the task of dynamic scene deblurring (best viewed by zooming in). Every output image is accompanied by its LPIPS value.
25. R B8V 899
48 6
84 RAP A49
2 W8V 84
Input: 0.630
SwinIR: 0.056
HCFlow 0.208
ESRGAN: 0.194
InDI: 0.245
RV 84R
989
RS R
88
2B.
R
31 9
4
R0
R
11
R8
8
0.1
Target
HAT: 0.045
LDM: 0.273
SRDIE 0.212
Ours: 0.048
Figure 14: Visual comparison of our approach against competing methods the DIV2K validation set for the task of 4x SISR (best viewed by zooming in). Every output image accompanied its LPIPS value.
Input: 0.244
SwinIR: 0.102
HCFlow: 0.075
ESRGAN: 0.074
Target
HAT: 0.116
LDM: 0.190
SRDiFF: 0.090
Ours: 0.072
Figure 15: Visual comparison of our approach against competing methods the DIV2K validation the task of 4x SISR (best viewed by zooming in). Every output image accompanied by its LPIPS value.
Input: 0.394
SwintR: O. 114
HCFlow: 0.046
ESRGAN: 0.044
InDI: 0.057
Target
0.022
Figure 16: Visual comparison of our approach against competing methods on the DIV2K validation set for the task of 4x SISR (best viewed by zooming in). Every output image accompanied by its L.IPS value.
29

---

Input: 0.335
SwinIR: 0,230
HCFlow: 0.114
ESRGAN: 0.127
InDI: 0.194
Target
HAT: 0.219
LDM: 0.269
SRDHT: 0.119
Ours: 0.140
Figure Visual comparison of our approach against competing methods DIV2K validation task of 4x super-resolution (best viewed by zooming in). Every output image is accompanied by its LPIPS value.
30

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims made in the abstract and section I match the theoretical and experimental results provided in the next sections. We do not mention any aspirational claims in our work.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of our work are properly discussed in dedicated paragraph 5
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
31

---

Answer: [Yes]
Justification: For all the theoretical results mentioned in the paper, in section 3 and in appendix we provide detailed descriptions, full set of assumptions and complete proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: In section 4 and appendices H and we provide the complete and detailed description of our training and inference procedures, that enables the reproducibility of our results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on nature the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
32

---

5. Open access to data and code
Question:
Does the paper provide open access to the data and code, with sufficient instructions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: All the data we used in our work is publicly available. Upon the acceptance of the paper plan to release the inference code together with the trained models checkpoints.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (httpss:/niis.cc.
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/nips. wttpcc/pcccccccccccccooooo hutee/ggguudeeeoeeeuuuubiiiiiiii wwwssssssllieeeeeeeoo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: section 4 and appendices and we provide the detailed description of our training and inference procedures.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Following the common practices in the field of image restoration, do not report the error bars, the correct estimation of them is computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
33

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification; For all the experiments the information on the computer resources is provided in appendix H
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question:
Does the research conducted the paper conform, in every respect, with the http://wwwrrarrroooooo/occc/taaa
NeurlPS Code of Ethics
Answer: [Yes]
Justification:
Our research strictly follows the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there is a special consideration due laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Besides the fact that in our work we train the conditional generative models, their limited capacity does not allow to generate anything that could be considered as a negative social impact. The proposed techniques are used to increase the perceptual appearance of the low-quality images and hardly pose any positive/ncgative societal impact.
34

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymemt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA].
Justification: All the data used for training of our networks pre-trained networks that we have utilized in our work is publicly available and widely known be safe, so the paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: all the datasets used in the paper publicly available, and in section 4 properly cite the papers where such datasets were introduced.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
35

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [NA].
Justification: The submission does not release new assets. Upon acceptance we plan to release well documented inference code under the non-commercial usage licence.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA].
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
36

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
37


Reviews:
[{"review_id": "W5Ht1Lw9IG", "reviewer": "Reviewer_WWsb", "summary": "This paper proposes a modular diffusion probabilistic IR framework to combine the performance benefits of existing pre-trained state-of-the-art IR networks and generative DPMs with a light-weight fusion network. Experimental results on burst JDD-SR, dynamic scene deblurring, and super-resolution demonstrate its superior performance and better perceptual quality.", "questions": "1. Why utilize a smaller version of MIRNet as the denoiser, not other architecture used for diffusion?\n2. Why not utilize open-sourced diffusion model, like stable diffusion.\n3. In Eq. 10, the author gets x_{\\tau} through posterior sampling from x_{T} and E[x_{0}|y], why not directly noising E[x_{0}|y] to x_{\\tau}? The idea that “Lemma 3.2 makes the final reconstruction quality unaffected” is confusing, because E[x_{0}|y] itself is not the true x_{0}. I think an ablation study is needed to verify your idea.\n4. Although the ablation of different combination of image restoration network and denoising network is given, an ablation of only utilize image restoration network or only utilize denoising network is also needed.\n5. Minor error: a) “in section 3.4” in line 228 may be “in Table 1”; b) The Table 1 should be table 2?", "limitations": "yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed framework is based on the existing IR network and equipped with a denoising module, which is suitable for various reconstruction problems without retraining and reduces the burden of computing resources and training data.\n2. An accelerated sampling algorithm is proposed to further reduce the computational burden during inference.\n3. Experimental results on burst JDD-SR, dynamic scene deblurring, and super-resolution demonstrate its superior performance and better perceptual quality, highlighting DP-IR’s versatility.", "weaknesses": "1. The contribution of accelerated sampling seems trivial. As the author said, references [13],[52] already proposed a similar idea.\n2. Although the author criticizes the task-specific nature of existing solutions, DP-IR still needs to train the fusion network for each specific task, which makes it less versatile.\n3. Absence of necessary ablation experiment:\na) The effectiveness of Eq.10’s sampling strategy; \nb)Inference without image restoration network, i.e., input with y directly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1** The contribution of accelerated sampling seems trivial. As the author said, references [13],[52] already proposed a similar idea.\n\nWe agree with the reviewer that our acceleration strategy indeed bears certain similarities to the strategies presented in the referred papers. However, as we also highlight in lines 233-244 of the manuscript, our approach is more general than those proposed in [13],[52]. Below we discuss in more detail both the conceptual and technical differences, and show empirically that our approach leads to better reconstruction results.\n\n### Conceptual difference\nUsing our notation, both [13] and [52] propose to start the reverse process from a timestep $\\tau$ and a noisy version $\\textbf{x}_\\tau$ of the initial estimate of $\\textbf{x}_0$, which we denote by $\\mathbb{E}[\\textbf{x}_0|\\textbf{y}]$. The main conceptual difference of our approach is that in these cases $\\textbf{x}\\_\\tau$ is obtained using the forward diffusion process, while in our case we end up in $\\textbf{x}\\_\\tau$ using the reverse process. The initial motivation for our proposed approach is also different. In particular, while we motivate our procedure from a probabilistic viewpoint and propose to approximate the conditional score function as a composition of three functions, the authors in [13] base their strategy on the contrastive property of reverse SDEs, while the authors in [52] use the re-projection of unrealistic images to the manifold of natural images in the noisy latent space.\n\n### Technical difference\nGiven that in our work we consider the standard DDPM realization of diffusion process (VP-SDE), we will explain the existing differences under this scenario. The authors of [13] and [52] propose to parameterize $\\textbf{x}\\_\\tau$ as \n\n$\\textbf{x}\\_\\tau=\\sqrt{\\bar{\\alpha}\\_\\tau}\\mathbb{E}[\\textbf{x}\\_{0}|\\textbf{y}]+\\sqrt{1-\\bar{\\alpha}\\_\\tau} \\textbf{z}$, where $\\textbf{z}\\sim\\mathcal{N}(\\textbf{0},\\textbf{I}).$\n\nIn contrast, in our case by using Eq.(10) we adopt the following parameterization: \n\n$\\textbf{x}\\_\\tau=\\sum_{i=0}^{T-\\tau-1}\\Gamma\\_{T-i-1}^{\\tau+1}\\gamma\\_{T-i}\\mathbb{E}[\\textbf{x}\\_0|\\textbf{y}]+\\Gamma\\_T^{\\tau+1}\\textbf{x}\\_T+\\sqrt{\\sum_{i=0}^{T-\\tau-1}(\\Gamma\\_{T-i-1}^{\\tau+1})^2\\sigma\\_{T - i}^2}\\textbf{z}$, where $\\textbf{z},\\textbf{x}\\_T\\sim\\mathcal{N}(\\textbf{0},\\textbf{I})$.\n\nAs we have already highlighted in lines 233-244 of the manuscript, our parameterization is more general and it is possible to show by induction, that under certain conditions it leads to the exact same $\\textbf{x}_\\tau$ as in [13] and [52].\n\nFinally, to experimentally demonstrate that our approach exhibits certain benefits compared to the ones described in [13] and [52], we conducted additional comparisons for the SISR problem between the different sampling strategies. From these results it is clear that our proposed strategy works better in practice and leads to superior results both in terms of fidelity and perceptual quality.\n\n|Acceleration Strategy|PSNR $\\uparrow$|SSIM $\\uparrow$|LPIPS $\\downarrow$|TOPIQ$_{\\Delta}$ $\\downarrow$|NFE $\\downarrow$|\n|-|-|-|-|-|-|\n|Ours|$\\textbf{28.12}$|$\\textbf{0.793}$|$\\textbf{0.140}$|$\\textbf{0.002}$|51|\n|[13],[52]|28.05|0.783|0.142|0.016|51|\n\n> **Q2** Although the author criticizes the task-specific nature of existing solutions, DP-IR still needs to train the fusion network for each specific task, which makes it less versatile. \n\nPlease see section \"Versatility of proposed framework\" in our general response.\n> **Q3**  Absence of necessary ablation experiment: a) The effectiveness of Eq.10’s sampling strategy; b)Inference without image restoration network, i.e., input with y directly.\n\na) Regarding the effectiveness of our sampling strategy as described in Eq. (10), we refer the reviewer to our answer to his first comment **Q1**.\n\nb) It is important to note that the condition $\\textbf{y}$ can represent the measurement signal from different imaging modalities. As a result $\\textbf{y}$ typically lies in a different domain than the one of the target signal $\\textbf{x}\\_{0}$. Therefore, the direct fusion of the output of the denoising module $\\boldsymbol{\\phi}_{\\boldsymbol{\\theta}_D}^D\\left(\\tilde{\\boldsymbol{x}}_t, \\tilde{\\sigma}_t\\right)$ and the condition $\\textbf{y}$ is not always feasible, but requires the processing of $\\textbf{y}$ to ensure that both signals lie in a common domain. An illustrative example of such a case is the burst JDD-SR (Joint Demoisaicing, Denoising, and Super-Resolution) task, where the measurement signal $\\textbf{y}$ consists of several image frames with each one lying in the domain of raw (mosaicked) and low-resolution images. Had we chosen to not process $\\textbf{y}$ with an image restoration network, then we would have to deploy a fusion module which would be required to pre-process $\\textbf{y}$ before performing the actual fusion. In this case, for every different inverse problem we would have to carefully design a specific architecture for the fusion model. Such a strategy would be less versatile than our current approach and would not take advantage of existing pre-trained sota restoration networks.\n> **Q4** Why utilize a smaller version of MIRNet as the denoiser, not other architecture used for diffusion? Why not utilize open-sourced diffusion model, like stable diffusion?\n\nPlease see section \"Network architectures choice\" in our general response.\n> **Q5**  In Eq. 10, the author gets $\\textbf{x}\\_{\\tau}$ through posterior sampling from $\\textbf{x}\\_{T}$ and $E[\\textbf{x}\\_{0}|\\textbf{y}]$, why not directly noising  $E[\\textbf{x}\\_{0}|\\textbf{y}]$ to $\\textbf{x}\\_{\\tau}$?\n\nWe refer the reviewer to our answer to **Q1** where we motivate our choice for doing so.\n\n**For the rest of the responses please see our official comment below**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zWWZbnCnVa", "reviewer": "Reviewer_hKXQ", "summary": "This manuscript proposed a modular conditional diffusion model for image reconstruction, consisting of three components: a pre-trained image restoration network, a denoising network, and a fusion network.", "questions": "\tThe standard and rationale for selecting the baseline models have not been detailed. For example, why were specific image restoration networks, denoising networks, and fusion networks chosen? Were these choices based on certain performance metrics, relevant literature, or experimental results? Providing this explanation will help readers understand the rationale behind the model design.\n\tWere other acceleration techniques used during the sampling process? If so, please provide specific technical details and experimental results, comparing performance differences before and after applying these acceleration techniques.\n\tIt is suggested that the authors revisit the formula in line 115 to ensure that each part is clear and understandable. If there are any symbols or variables that are not clearly defined, please provide explicit definitions. Consider adding explanatory text before and after the formula to help readers understand its meaning and application.\n\tIt is advisable for the authors to verify if the PSNR values in Table 5 are consistent with those in other tables. If all tables should indicate the PSNR Target values as ∞, ensure consistency across all tables to avoid confusion for readers during their review.\n\tIt is recommended that the authors redesign the color scheme of Figure 1 to enhance its aesthetic appeal and improve clarity for better understanding.", "limitations": "The authors do not point out the limitations of the work and do not offer further perspectives. I hope it will be improved.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The model reduces computational load by minimizing the number of network modules that need to be trained and by utilizing lightweight networks. The fusion network is the only component that needs to be trained for each specific IR task. The model also achieves significant acceleration in the sampling process without any loss of reconstruction quality.", "weaknesses": "The analysis of trade-offs between model size, speed, and performance may be insufficient. There is insufficient explanation regarding the reasons for selecting specific modules.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q8** The standard and rationale for selecting the baseline models have not been detailed. For example, why were specific image restoration networks, denoising networks, and fusion networks chosen? Were these choices based on certain performance metrics, relevant literature, or experimental results? Providing this explanation will help readers understand the rationale behind the model design.\n\nPlease refer to the section \"Network architectures choice\" in general response. Thank you! \n\n> **Q9** Were other acceleration techniques used during the sampling process? If so, please provide specific technical details and experimental results, comparing performance differences before and after applying these acceleration techniques\n\nAs we mention in lines 244-246 and 270-273 of the manuscript, for the SISR task we further employ the DDIM acceleration technique during the sampling process. Following the reviewer's suggestion, we will include an additional section in the appendix with a short technical description of this technique and we will refer to it inside the main text of the manuscript. During our experiments, we didn't observe significant quantitative/qualitative difference after applying this acceleration technique. To be more specific, we provide below a comparison of the reconstruction performance with and without utilizing DDIM, which we will also include in the newly added section in the appendix. \n\n| Method           | PSNR $\\uparrow$ | SSIM $\\uparrow$ | LPIPS $\\downarrow$ | TOPIQ$_{\\Delta} $ $\\downarrow$ | NFE $\\downarrow$ |\n|------------------|-----------------|-----------------|--------------------|--------------------------------|------------------|\n| Ours (with DDIM) | 28.12           | 0.793           | 0.140              | $\\textbf{0.002}$                   | 51               |\n| Ours w/o DDIM    | $\\textbf{28.16}$  | $\\textbf{0.794}$  | $\\textbf{0.139}$     | 0.007                          | 251              |\n\n> **Q10** It is suggested that the authors revisit the formula in line 115 to ensure that each part is clear and understandable. If there are any symbols or variables that are not clearly defined, please provide explicit definitions. Consider adding explanatory text before and after the formula to help readers understand its meaning and application.\n\nWe thank the reviewer for his suggestion. Indeed, in the original version of the manuscript we have missed explaining the purpose of the $\\beta\\_{t}$ parameter. In the revised version we clarify that $\\beta\\_{t}$ is the noise scheduling parameter for the forward process. We will also re-write the formula to make it more compact and clear.\n\n> **Q11** It is advisable for the authors to verify if the PSNR values in Table 5 are consistent with those in other tables. If all tables should indicate the PSNR Target values as $\\inf$, ensure consistency across all tables to avoid confusion for readers during their review.}\n\nWe thank the reviewer for raising this point. Indeed, in  Table 5 there was a typo where we used the word \"inf\" instead of the \"inf\" sign, which has now been corrected.\n\n> **Q12** It is recommended that the authors redesign the color scheme of Figure 1 to enhance its aesthetic appeal and improve clarity for better understanding.\n\nFollowing the reviewer's suggestion, we have redesigned Figure 1 and change its horizontal dimension in order to match the text width. We believe that these changes will improve its aesthetic appeal. \n\n> **Q13** The authors do not point out the limitations of the work and do not offer further perspectives. I hope it will be improved.\n\nPlease see section \"Limitations and further perspectives\" in our general response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0P2cirklPZ", "reviewer": "Reviewer_PdT6", "summary": "This paper proposes a new approach to improving the efficiency and applicability of Diffusion Probabilistic Models (DPMs) for various image restoration (IR) tasks. A new modular diffusion probabilistic image restoration (DP-IR) framework combines pre-trained state-of-the-art IR networks with generative DPMs. This framework requires only a small additional module (0.7M parameters) to be trained for specific IR tasks, making it more practical and less computationally expensive. This framework is evaluated on four benchmarks, covering tasks such as burst JDD-SR, dynamic scene deblurring, and super-resolution.", "questions": "How sensitive is your framework to the choice of hyperparameters?", "limitations": "Elaborate on future work directions that could address these limitations. For instance, proposing specific research avenues to enhance the generalizability of the framework, or developing more adaptive hyperparameter tuning methods.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Originality: Modular framework and accelerated sampling strategy.\nQuality: Robust methodology and comprehensive experimental validation.\nClarity: Clear presentation.\nSignificance: By reducing the computational requirements and enhancing the generalizability of IR methods, the framework could facilitate the adoption of advanced image restoration techniques in real-world scenarios where computational resources are limited.", "weaknesses": "Computational cost analysis: A more detailed analysis of the computational cost, including memory usage and inference time, would be helpful.\n\nExperimental validation: While the paper demonstrates strong performance on specific benchmarks, the experiments could benefit from additional datasets to further validate the generalizability of the proposed framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q14** Computational cost analysis: A more detailed analysis of the computational cost, including memory usage and inference time, would be helpful.\n\nWhile we agree with the reviewer that the inclusion of such information has practical value, we would also like to highlight existing problems that prevents us from doing so:\n1. Both memory usage and inference time are highly dependent on the efficiency of implementation of the particular method and the hardware equipment used. Moreover, depending on the implementation, the memory usage can be sacrificed for faster inference and vice-versa. \n2. All our diffusion-based competitors in dynamic scene deblurring problem and one competitor in SISR (InDI) do not offer publicly available implementations, making such direct comparisons impossible.\n\nTo avoid any speculations on these subjects, we have refrained from performing such comparisons. For our computational cost analysis we rely solely on the Neural Function Evaluations (NFEs) to explicitly compare the inference compute, and the number of parameters to implicitly compare the training data and training compute. For the NFEs comparison we assume that all the competing diffusion-based methods have similar computational complexity in terms of FLOPs per single NFE. Under this assumption, to perform a comparison for the inference costs it is sufficient to just compare the amount of NFEs. \n\nNevertheless, we agree with the reviewer, that this is not the best approach for a computational cost analysis, and for this reason we additionally report below the exact FLOPs per 720p input for competing methods in dynamic scene deblurring problem.\n| Method | TFLOPs (equation)     | TFLOPs Total $\\downarrow$ |\n|--------|-----------------------|---------------------------|\n| DvSR   | 1.2$\\times$N + 4.8  | 604.8                     |\n| icDPM  | 4.8$\\times$N + 5.2  | 2405.2                    |\n| InDI   | 4.8$\\times$N| 48.0                      |\n| Ours   | 4.3$\\times$N + 1.9  | **23.4**             |\n\nThe proper way to interpret these equations is as follows: $\\textrm{TFLOPs Total} = x \\times N + y$, where $x$ is the TFLOPs complexity for a single backbone pass within the diffusion process, $N$ is the total number of neural function evaluations (NFEs) per sampling process, and $y$ is the complexity of sub-modules that have to be run once per image (e.g. Image Restoration network in our method, pre-processing net for icDPM and DvSR). Based on these results, we observe that the computational cost of our method is significantly lower compared to our diffusion-based competitors. \n\nWe have included this information regarding the FLOPs in the revised version of the manuscript and hope that it enhances the computational cost analysis.\n> **Q15** Experimental validation: While the paper demonstrates strong performance on specific benchmarks, the experiments could benefit from additional datasets to further validate the generalizability of the proposed framework.\n\nWe validated our method using widely recognized and publicly available datasets to ensure a fair and reproducible assessment of our approach. Utilizing these commonly used datasets allows for more straightforward benchmarking and comparison with other methods. However, for other less commonly used or proprietary datasets, it becomes difficult to make direct comparisons. This is primarily due to the fact that many state-of-the-art (SOTA) approaches do not release their implementations publicly. Consequently, without access to these implementations, reproducing their results or performing a head-to-head comparison on different datasets poses significant challenges. Therefore, our evaluation is limited to datasets for which there is widely available and accessible benchmark data.\n\n> **Q16** How sensitive is your framework to the choice of hyperparameters?\n\nThe main hyperparameter of our framework is the time threshold $\\tau$, and we refer the reviewer to Table 7 and Table 10 of our manuscript for the detailed ablation study of its influence on the reconstruction quality. All the other parameters of our framework, specifically the total number of diffusion steps $T$ and diffusion coefficients $\\\\{\\beta_t\\\\}_{t=1}^T$ were taken as-is from the prior literature on diffusion models. We should note, that no other works on image reconstruction via reverse diffusion perform the sensitivity study wrt the values of this sequence, so in our work we also refrained from doing that and leave this topic as a possible future research direction. \n\nHowever, we have investigated the sensitivity of our approach to the choice of the specific modules, and we refer the reviewer to Table 5 in the main text and section F in the Appendix for details of this study. \n\n> **Q17** Elaborate on future work directions that could address these limitations. For instance, proposing specific research avenues to enhance the generalizability of the framework, or developing more adaptive hyperparameter tuning methods.\n\nPlease find our comment in section \"Limitations and further perspectives\" of our general response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mvpFTG8yj8", "reviewer": "Reviewer_B8jE", "summary": "This paper proposes a modular conditional diffusion framework for image reconstruction. Specifically, a small module is trained and combined with pretrained IR networks and DPMs. Experiments show that this method is effective.", "questions": "Considering the overall contribution and the weaknesses, I choose \"Borderline reject\". I am willing to adjust my score for my potential misunderstanding.", "limitations": "The limitations are discussed.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. According to the quantitative and qualitative results, the method shows great performances on several tasks.\n  \n2. The method can be applied to tasks combining task-specific pre-trained models, making the method easy to be extended.\n  \n3. The module requiring additional training is small and the method can be trained efficiently.", "weaknesses": "- The figures in this paper are of poor quality. The authors should revise the figures to make the formulas clear and professional.\n  \n- Overall I think the modular framework is of limited insight. The design is clean but not so informative for image restoration and some other related domains.\n  \n- If I understand correctly, this method is agnostic to the specific tasks of image restoration, though the authors present several tasks in this paper, it would be good if more qualitative results on various tasks are given.\n  \n- I think the arguements in line 45-49 are not proper. As the authors discuss the conditional space, sometimes we may not need too much time for the DPM training especially we have the base model and can perform efficient fine-tuning. Also, the case that is discussed (1M images and 120GPU-days for LDM) does not precisely align with the conditional context. These values correspond to the original LDM but we today usually use SD which is trained on huger datasets and takes more time training. Here if we want to say a single condition space tailored DPM may require expensive resources, maybe it is better include cases on finetuning SD or other domain-specific DPM.\n  \n- For some baseline methos, it is unclear how they are implemented and compared. For example, LDM is selected as the baseline which gives qualitative and quantitative, but I am not sure how it is used and which checkpoint is selected. The authors can clearly indicate how baselines are implemented.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q18** The figures in this paper are of poor quality. The authors should revise the figures to make the formulas clear and professional.\n\nIn our submitted manuscript we made sure that both Figures 1 and 2 are vector images and that the formulas inside these figures follow the same LaTeX style as we use inside the text. Following the reviewer's comment, we have checked the visual appearance of them in different pdf viewers and found no mentioned issues. We include the pdf file with raster screenshots of both figures as an attachment to our **general response** and kindly ask the reviewer to compare their visual appearance with those in our manuscript. For the camera-ready version we plan to expand the horizontal dimension of Figure 1 to match the full page width, which we hope that will enhance its appearance. \n\n> **Q19** Overall I think the modular framework is of limited insight. The design is clean but not so informative for image restoration and some other related domains.\n\nWe respectfully disagree with this particular assessment of the reviewer. Previous DPM-based works [40, 60, 78] have attempted to directly estimate the conditional expectation $\\mathbb{E} \\left[\\bf{x_0} | \\bf{y}, \\bf{x_t}\\right]$ (or residual noise) and have achieved outstanding performance by sacrificing the generalizability to other tasks. In contrast, we demonstrate that $\\mathbb{E} \\left[\\bf{x_0} | \\bf{y}, \\bf{x_t}\\right]$ in Eq. (5) can be expressed as a fusion of two separate expectations: $\\mathbb{E} \\left[\\bf{x_0} | \\bf{y}\\right]$ and $\\mathbb{E} \\left[\\bf{x_0} | \\bf{x_t}\\right]$. This approach allows us to learn a single unconditional generative denoising function to estimate $\\mathbb{E} \\left[\\bf{x_0} | \\bf{x_t}\\right]$ and apply it to different image reconstruction problems with minimal computational overhead, by training our fusion module on a small dataset. Further, we have verified the validity of our approach on three different and very competitive blind image reconstruction tasks by performing comparisons with competitive sota methods on several public benchmarks.\n\n> **Q20** If I understand correctly, this method is agnostic to the specific tasks of image restoration, though the authors present several tasks in this paper, it would be good if more qualitative results on various tasks are given.\n\nOur method is not fully agnostic to the image restoration task at hand, since in order to apply it to a new unseen task the training of a small Fusion network (0.7M params) is required. Due to page limitations and in order to reserve enough space to explain adequately the key ideas of our work, similar to the common practice followed by several other existing works ([78],[59],[60],[40],[21]), we have decided to conduct experiments and perform comparisons on three challenging tasks from different modalities, while we plan to consider additional reconstruction tasks as a future work.\n\n> **Q21** I think the arguments in line 45-49 are not proper. As the authors discuss the conditional space, sometimes we may not need too much time for the DPM training especially we have the base model and can perform efficient fine-tuning. Also, the case that is discussed (1M images and 120GPU-days for LDM) does not precisely align with the conditional context. These values correspond to the original LDM but we today usually use SD which is trained on huger datasets and takes more time training. Here if we want to say a single condition space tailored DPM may require expensive resources, maybe it is better to include cases on finetuning SD or other domain-specific DPM.\n\nThis is a valid point raised by the reviewer and in the revised version we plan to include a discussion about existing methods that fine-tunes DPMs. One such example is the T2I-Adapter [A], which requires 12 GPU-days on an NVIDIA V100 32GB GPU and utilizes between 164K and 500K images, depending on the task. Another relevant example is ControlNet [B], which demands between 4 and 25 GPU-days on an NVIDIA A100 80GB GPU (approximately three times faster than the NVIDIA V100) and employs training datasets ranging from 25K to 3M images. Furthermore, the aforementioned adapters increase the training parameters at least by 77M and up to half of the parameter size of Stable Diffusion.\n\n[A] Mou, Chong, et al. \"T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 5. 2024.  \n[B] Zhang, Lvmin, Anyi Rao, and Maneesh Agrawala. \"Adding conditional control to text-to-image diffusion models.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n> **Q22** For some baseline methods, it is unclear how they are implemented and compared. For example, LDM is selected as the baseline which gives qualitative and quantitative, but I am not sure how it is used and which checkpoint is selected. The authors can clearly indicate how baselines are implemented.\n\nThe LDM and its checkpoint that we have used for comparison is the original model that was trained for the SISR task: https://ommer-lab.com/files/latent-diffusion/sr_bsr.zip.\n\nFollowing the reviewer's suggestion, we have included an additional section in the Appendix with the description and links to the code for all baselines that we report in the manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# General response 

We would like to thank all reviewers for the insightful questions and valuable suggestions. In this response we would like to address common questions that were asked by more than one reviewer.

## Limitations and further perspectives
In the manuscript (lines 356-363) we have briefly described one limitation of our framework related to the optimal choice of $\tau$, which is dependent of the particular reconstruction task at hand and needs to be experimentally selected. Another limitation that exists, and which we have added in the revised version of the manuscript, is that the performance of our method is bounded by the performance of the utilized backbone networks (Denoising and IR module). Therefore, for a novel image restoration task, where pre-trained IR network does not exist, our framework is not applicable. Finally, in the case of imaging modalities (e.g. medical imaging) where a score-matching network (denoising module) has not been trained for, it is important to either fine-tune an existing denoising module or completely re-train one using appropriate image data.

## Network architectures choice

**Denoising and IRNet modules.** The reason for utilizing a smaller size of MIRNet as a Denoising module is that we wanted to approximately match the number of parameters and the computational complexity of the networks used in our framework with those of the alternative methods under study, in order to ensure a fair evaluation and comparison. This strategy has allowed us to achieve direct performance comparisons under similar conditions. Regarding the utilization of stable diffusion, reviewer WWsb makes a valid point and according to our ablation studies (see Table 5) we observe that by utilizing more powerful modules we can expect improvements on the overall performance. The main reason for not using Stable Diffusion is that given the network size constraints that we had, as we explained above, we experimentally found that the selected network architecture provided the best results.

**Fusion Module.** We have experimented with several basic fusion architectures, but we did not conduct an exhaustive research about the most performing architecture. Our proposed fusion module serves as a proof of concept for the validity of our overall proposed framework and the performance improvements that it can achieve. A more in-depth investigation of appropriate fusion architectures can serve as a very interesting future research direction.

## Versatility of proposed framework
Our strategy cannot work out of the box for every inverse problem but requires tuning of a relatively small 0.7M parameter fusion network for each particular problem.
Nevertheless, we would like to emphasize that the main focus of our work is blind image restoration problems. To the best of our knowledge, all diffusion-based approaches that have been proposed in the literature to deal with such tasks, require the training of far larger conditional backbone networks (~10-100M params). This turns out to be significantly more challenging both in terms of necessary training data and computational resources. To showcase this, we provide an indicative example below. If we adopt the existing diffusion-based SISR baselines and train them for a completely different restoration problem, by following the original authors' training strategies it turns out that the computational and data requirements are significantly higher than those of our method.  

|Method|Params required|Data required|
|-|-|-|
|Ours|1x|1x|
|SRDiff|$\sim$34x|$\sim$4x|
|LDM|$\sim$240x|$\sim$1000x|
|InDI|$\sim$89x|$\sim$ 1x|
|IDM|$\sim$167x|$\sim$ 1x|

Based on these data, we can safely state that our strategy provides a reasonable trade-off between the required training complexity and the competitive performance of our method to a variety of blind inverse problems.

## Screenshot of Raster figures of our framework
Below, we have attached pdf of raster figures to compare the visual quality.

Meta-review: Authors propose a module diffusion model that can be used for image
restoration. The main focus is on making use of a diffusion model
feasible through reducing the computational cost. To this end, a
modular structure is used.

Initial reviewers main concern was the experimental validation,
ablation studies and design choices. During the rebuttal, authors
clearly explained their choices and through small additional
experiments could convince all the reviewers. The final ratings were
3BA and 1WA. While the novelty is not extremely high, I think this
would be a valuable contribution.
Decision: Accept (poster)