Resfusion: Denoising Diffusion Probabilistic Models for Image Restoration Based on Prior Residual Noise
Authors: Zhenning Shi
Haoshuai Zheng
Chen Xu
Changsheng Dong
Bin Pan
Xie xueshuo
Along He
Tao Li
Huazhu Fu
Venue: NeurIPS 2024 poster (2024)
Keywords: Diffusion based models; Image restoration;  Shadow removal; Low-light enhancement; Deraining
Primary area: diffusion_based_models
TL;DR: We propose Resfusion, a general framework that incorporates the residual term into the diffusion forward process, starting the reverse process directly from the noisy degraded images.

Abstract: Recently, research on denoising diffusion models has expanded its application to the field of image restoration. Traditional diffusion-based image restoration methods utilize degraded images as conditional input to effectively guide the reverse generation process, without modifying the original denoising diffusion process. However, since the degraded images already include low-frequency information, starting from Gaussian white noise will result in increased sampling steps. We propose Resfusion, a general framework that incorporates the residual term into the diffusion forward process, starting the reverse process directly from the noisy degraded images. The form of our inference process is consistent with the DDPM. We introduced a weighted residual noise, named resnoise, as the prediction target and explicitly provide the quantitative relationship between the residual term and the noise term in resnoise. By leveraging a smooth equivalence transformation, Resfusion determine the optimal acceleration step and maintains the integrity of existing noise schedules, unifying the training and inference processes. The experimental results demonstrate that Resfusion exhibits competitive performance on ISTD dataset, LOL dataset and Raindrop dataset with only five sampling steps. Furthermore, Resfusion can be easily applied to image generation and emerges with strong versatility. Our code and model are available at https://github.com/nkicsl/Resfusion.

Resfusion: Denoising Diffusion Probabilistic Models for Image Restoration Based on Prior Residual Noise
Zhenning Shi
Haoshuai Zheng'
Chen Xu!
Changsheng Dong' Huazhu Fu'
Bin Pan
Xueshuo Xie
Along He Tao Li
'College of Computer Science, Nankai University School of Statistics and Data Science, Nankai University Haihe Lab of ITAI
Institute of High Performance Computing, S
Abstract
Recently, research denoising diffusion models has expanded its application the field of image restoration. Traditional diffusion-based image restoration meth- ods utilize degraded images conditional input effectively guide the reverse generation process, without modifying the original denoising diffusion process. However, since the degraded images already include low-free wwwfrequency information, starting from Gaussian white noise will result in increased sampling steps. We propose Resfusion, general framework that incorporates the residual term into the diffusion forward process, starting the reverse process directly from the noisy de- graded images. The form of our inference process is consistent with the DDPM. We introduced a weighted residual noise, named resnoise, as the prediction target and explicitly provide the quantitative relationship between the residual noise term in resnoise. By leveraging a smooth equivalence transformation, Resfu- sion determine the optimal acceleration step and maintains the integrity of existing noise schedules, unifying the training and inference processes. The experimental re- sults demonstrate that Resfusion exhibits competitive performance on ISTD dataset, LOL dataset and Raindrop dataset with only five sampling steps. Furthermore, Res- fusion can be easily applied to image generation and emerges with strong versatility. Our code and model are available at https://giina.ccom///issssssssss
and the
Introduction
Denoising diffusion models [1, 2] have emerged powerful and effective conditional generative models, demonstrating remarkable success in synthesizing high-fidelity data for image generation. Saharia et al. [3] proved that these generative processes can be applied image restoration by feeding degraded images as conditional input into the score network. SNIPS [4] combines annealed Langevin dynamics and Newton's 's method arrive posterior sampling algorithm, exploring the generative diffusion processes to solve the general linear inverse problems. Based on these, many diffusion-
models were adapted for downstream image restoration tasks [5, 6, 7, 8, 10, 11]. For traditional diffusion-baat based models, the reverse process begins with Gaussian white noise, considering only the degraded images as the conditional input. This results in an increased number of sampling steps. Image restoration tasks often focus on restoring and editing specific high-frequency details while preserving crucial low-frequency information, such as the image structure. The degraded images used as conditional input inherently contain the low-frequency information. Therefore, initiating the reverse process from Gaussian white noise for image restoration tasks appears unnecessary and inefficient.
"Corresponingg authors: litao@ mankai.edu.co. hcalong2020 163.com
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

T anyminn L lv -2
qix Xx..
&
X
X
E 1x)
Figure The proposed Resfusion a general framework for image restoration and can easily expand image generation (setting Bo 0). We introduce the residual term (R 20 20) into the forward process, redefine q(z, x(-1) 1) to q [ R) (as shown by the orange arrow), and name this diffusion process resnoise diffusion. Through employing novel technique called "smooth equivalence transformation" we can directly use the degraded image sg to obtain 2/-" (as shown by the blue arrow). We bridge the gap between the input image and ground truth, unifying the training and inference processes.
Consequently, some works have proposed to generate clean images directly from degraded images or noisy degraded images. InDI [12] restores clean images through the reverse process of direct iteration to degraded images; DDRM [13] reformulate the image restoration tasks as inverse problems when the mapping between clean and degraded images is available; IR-SDE [14] directly models the image degradation process using mean-reverting SDE (Stochastic Differential Equations): IBSB [15] constructs a Schrödinger bridge between clean and degraded data distributions; Resshift [16] shifts the residual term from degraded low- resoouutttion images to high-resolution images, performing the recovery in the latent space. Liu et al. [17] introduced the Residual Denoising Diffusion Models (RDDM), generalizing the diffusion process of InDI and SS RDDM points out that co-learning the residual term and the noise term can effectively improve the model performance. However, RDDM has some limitations. Firstly, RDDM predicts the residual term and the noise term separately, without explicitly specifying their quantitative relationship. Secondly, due to its forward process adopting an accumulation strategy for the residual term and the noise term, its forward and reverse processes inconsistent with the DDPM [1]. which results in poor generalization and interpretability. Thirdly, RDDM requires the design of a complex noise schedule, as utilizing existing noise schedules would result in performance loss.
To solve the problems mentioned above, propose Resfusion, general framework for image restoration and can be easily expand to image generation. By introducing the residual term into the diffusion forward process, bridge the gap between the input image and the ground truth, starting the reverse process directly from the noisy degraded images. We calculate the quantitative relationship between the residual term and the noise term, naming their weighted sum as the resnoise. Through the smooth equivalence transformation, we determine the optimal acceleration step and unify the training and inference processes. As versatile methodology for image restoration, Resfusion does not require any physical prior knowledge. Resfusion allows for the direct use of existing noise schedules, and the image restoration process can be completed in just five sampling steps.
Our contributions can be summarized as follows:
First, by introducing the residual term into the diffusion forward process, our resnoise- diffusion process starts the reverse process directly from the noisy degraded images, closing the gap between the degraded input and the ground truth.
Second, we explicitly provide the guantitative relationship between the residual term and the noise term in the loss function, and name the weighted residual noise resnoise. Through transforming the learning of the noise term into the resmoise term, the form of our reverse inference process is consistent with the DDPM.
Third, through the smooth equivalence transformation in resnoise-diffusion process, we determine the optimal acceleration step and unify the training and inference processes. The optimal acceleration step is non-trivial where the posterior probability distribution is

---

equivalent to the prior probability distribution at this step. Moreover, we can directly use the existing noise schedule instead of redesigning the noise schedule.
2 Methodology
2.1 Learning the resnoise
First, we denote the input degraded image and the ground truth as fy and Zo- In order to extend the diffusion process of Denoising Diffusion Probabilistic Models (DDPM) [1] to image restoration, we define the residual term as Eq. (1).
R = Z0 T0
(I)
Then the forward process can be defined Eq. (2) and Eq. (3). We provide a detailed explanation for
why we introduce the residual term R to Eq. (3) in this way in Sec. 2.2 and Figure 2. Consistent with the definition in DDPM, we use the notation }3 1 and az = II.=1
T
q 2o, R) = II
(2)
t=1
q(11/2111, R) (m Vata--1 + Var) (1-04))
(3)
Then the redefined forward process can be formalized Eq. (4), where represents the Gaussian white noise.
vo + (1 Var VI- c c~ N(0,1)
(4)
According to Eq. (4), 24 can be reparameterized Eq. (5).
T₁ Va + (1 va.) /1 - Tte, E~ N(0
(5)
We easily incorporate this forward process into the vanilla DDPM. We introduce a residual noise, named as resnoise (symbolized as rese), to describe the gap between the current estimate 22 and the ground truth In, and the term to be minimized can be formulated as Eq. (6). Detailed proof can be found in Appendix A.I.
TESE = e (1 /os)v at R. Exg.e.:[]rrss TESEe 1)
(6)
3r
Through this process, we transform the learning of c0(22, 1) into reseg(Et, l). eg (ate, 1) represents the noise of the noisy ground truth, while rese (x I) represents the residual noise hetween the input degraded images and the ground truth. We name this process resnose-diffusion
2.2 Smooth equivalence transformation
According to Eq. (5) and Eq. (1), we can derive Eq. (7). It worth mentioning that ær is uncom- putable because the ground truth ag is unavailable in Eq. (7) during the reverse process, so we can not initialize zy directly.
Ft = (2V0r - 1)00 + (1 varj + v1 o4C, T)
Fortunately, the weighted coefficient of Fp. which is (2 1) in Eq. (7), can be very close to zero. Since the input degraded image åo is available, we can find a time step T" where Fr. is computable. When T is sufficiently large, the variation of Va,(t < T) with respect to time t is smooth. We call this technique smooth equivalence transformation. Therefore, we can derive T" as Eq. (8) and obtain æT in Eq. (9) with small bias. This bias can also be eliminated through the Truncated Schedule technique that we propose next.
T arg min 0v;
(8)
Fr (1 V / 20 vI o = Vorio + vI-vare arre
(9)
Thus we only need to minimize Eq. (6) when t < T" since Tre is available, as shown in Eq. (10).
T'
(x(x)) = Jxyrqpe
Panta (WT))
Po (2. [x)ddr Tr.
(10)

---

Gaussian noise
Xo
T
80
X
&
va JR-
X
X
R
resnoise- diffusion diffisi-difiisinn
R
Figure 2: The working principle of Resfusion, Zo represents the distribution of the ground truth, while žn represents the distribution of the degraded images. &g Ip represents the gap between them, defined as the residual term R in Eq. (1). Resfusion does not explicitly guide êg to Zo- Instead, it implicitly learns the distribution of R by doing -ifoose-diffusion reverse process from 2t to Zo. The resnoise-diffusion reverse process can be imagined as doing diffusion reverse process from R + to zo (as shown by the viole! arrow), guiding Ii gradually towards zo along this direction. Following the principles of similar triangles, the coefficient of It at step is computed as Va. At any step t during the training process, Iy can be calculated based on zi and R through Eq. (4).
The resnoise-diffusse reverse process can be formulated as Eq. (11) and Eq. (12). Consistent with the definition DDPM, the Eg is taken fixed as 3
1- 3..
1-6,
T II Po(x)-1la4)
Po(2o:T-11/TT) 1TT)
(11)
[=1
Po(2,-1/1)) = N1 t), (o t)), Po(zolz)) (zai (0 1))
(12)
The mean Ho (Z,,t) of resnoise-diffusion reverse process can be formalized as Eq. (13), which is demonstrated in Appendix A.1 from Eq. (19) Eq. (23).
Be
po( = Vat
(xt
press))
resco)
(13)
VT- Oz
Similar to previous work [5, 17], our method enhances the diffusion model by incorporating conditioning function. This function integrates latent representation from both the current estimate I, and the input degraded image δn. Then Eq. (6) can be modified to Eq. (14).
Ezo.e.ellresc. - resco 2o. [
(14)
Just like the vanilla DDPM, Resfusion gradually fit the current estimate x, to the ground truth 2o. implicitly reducing the residual term R between So and To with the resnoise. When define â as Eq. (15) with the same Gaussian noise e in Fz, f.r can be seen as an intermediate result in an implicit DDPM process with the input degraded image &o as the target distribution. We essentially quantitatively computed the accelerated step I", on which step TT is closed to âT. When T is large enough, the approximate equal sign will become an equal sign. The implicit DDPM reverse process (e to i.)) is deterministic because 20 is available during the inference. The determination of step I' corresponds the point where the posterior probability distribution tresnoise-diffusic reverse process) becomes consistent with the prior probability distribution (implicit DDPM reverse process).
ft Varãg + v1 - O4e, VeT R 1 V CCT
R
FT årr
(15)
2'
As shown in Fig. 2, the desnoise-diffusio reverse process (R 6 to Fo) intersects with implicit DDPM reverse process (e to 20), this intersection corresponds to step T". The intersection of two

---

diagonals of a parallelogram is the midpoint of them (corresponding to 0.5 in Eq. (II)), but due to the discrete nature of the diffusion process, the acceleration point actually falls on the point closest to the intersection, which is step T as Eq. (8). Meanwhile, since 2T" is available, resnoise-diffusion steps after step T" are not necessary according to Eq. (10). Therefore, Resfusion can directly start from step T" for both inference and training process. Because the a coefficient of esnoise-diffustie exactly the same as vanilla DDPM, Resfusion can directly use any existing noise schedule. In practical implementation,
utilized technique called Truncated Schedule to control the offset between IT. and TT. when small. Further details can found in Appendix A.7.
3 Experiments
Table 1:
Quantitative comparisons with other shadow removal methods. We report PSNR, SSIM [18]
and MAE in the shadow region (S), the non-shadow region (NS) and all image (ALL). The best and second-bes best results are highlighted in bold and underlined. "1" (resp. "{") means the larger (resp. smaller), the better. We the symbol "_" indicate models or results that are unavailable.
ISTD [1]
Shadow Region (S)
Non-Shadow Region (NS)
Image (ALL)
Method
Params
PSNR
SSIM 1 MAF
PSNR
SSIM
MAF |
PSNR
SSIM 1
MAE
Input Image
22.40
0.936
32.11
27.32
0.976
6.83
20.56
0.893
10.97
ST-CGAN [20]
31 8M
33.74
0.981
9.99
29.51
0.958
6.05
27.44
0.929
6.65
256 258
DSC [21]
22.3M
34.64
0,984
8.72
31.26
0.969
5.04
29.00
0.944
5.59
DHAN [22]
21 RM
35,53
0.988
7.49
31.05
0.971
5.30
29.11
0.954
5,66
NusionNet [23]]
34.71
0.975
0,880
27.19
UnlisldingNet [24]
10.1M
36.95
0.987
8.29
31.54
0.978
4.55
29.85
0.960
5.09
DMIN [25]
22.8M
RDDM (SM Res N) [17]
15.5M
36.74
0.988
6.67
33.18
0.979
4,27
30.91
0.962
4,67
Resfusion tours)
34.26
Input Image
22,34
0.935
33.23
26.45
0.947
7.25
20.33
0.874
1133
Original
ARGAN
DHAN [22]
21.8
34.79
0.983
8.13
29.54
0.941
5.94
27.88
0.921
6.29
[27]
Resfusion (ours)
7.7M
36.45
0.985
7.08
32.08
0.950
5.02
30.09
0.932
5.34
Input
(2019)
DHAN (2020)
DMTN (2023)
Resfusion (ours)
Ground Truth
Figure 3: Visual comparisons of the restored results by different shadow-remove methods on the ISTD dataset.
To verify the performance of Resfusion, we conducted experiments on three image restoration tasks, including shadow removal, low-light enhancement and deraining. For fair comparisons, we use an U-net [28] structure which is the same RDDM as the backbone. We simply concatenate 2; and 2o the channel dimension and feed them into the network. For all the tasks, we only employ one U-net to predict resnoise. Furthermore, we simply utilize truncated linear schedule [29] and perform only five sampling steps for all datasets. The experimental setting details are provided the Appendix A.4.
ISTD dataset [19] a dataset designed for shadow removal, comprising 1870 sets of image triplets consisting of shadow image, shadow mask, and shadow-fres image. It consists of 1330 image triplets for training and 540 image triplets for quantitative evaluations. We compare the proposed method with the popular shadow removal methods, i.c., ST-CGAN [20], DSC [21], ARGAN [26], DHAN [22],
5

---

Table 2: Quantitative comparisons with other low- light enhancement methods. We report PSNR, SSIM and LPIPS [38]. The best and second-best results are highlighted in bold and underlined. "1" (resp. "4") means the larger (resp. smaller), the better.
Table 3: Quantitative comparisons with other deraining methods. We report PSNR and SSIM. The best and second-best results are highlighted in bold and underlined. "T" means the larger, the better.
LOL|300
Raindrop [39]
Method
Params
PSNR
SSIM
LPIPS
Method
Params
PSNR
SSIM
YChCr space, 256 x 256
YCbCr space, 256 x 256
Input Image
Input Image
RDDM (SM-Res-N) [17]
15.5M
23.90
0.931
RDDM (SM-Res-N) [17]
15.5M
32.51
0.956
RDDM (SM-Res) [17]
2.7M
25.39
0.937
0.116
Resfusion (ours)
7.7M
34.40
0.975
Resfusion (ours)
7.7M
30,02
0.954
0.070
YChCr space, Original
RGB space, Original
Input Image
25.40
0.882
Input Image
7.77
0.191
0.560
pix2pix [40]
28.02
0.855
RetinexNet [30]
0.6M
16.77
0.560
AtientiveGAN [39]
6.2M
0.917
KinD [31]
8.0M
20.87
0.790
0.170
DuRN [41]
10.2M
31.24
0.926
KinD++ [32]
9.6M
21.30
RaindropAtt [42]
31.44
Zero-DCIB [33]
0.3M
14.86
0.562
0.335
All-in-One [43]
31.12
0.927
EnlightenGiAN [34]
8.6M
17.48
0.652
0.322
IDT [44]
16.4M
31.87
0.931
Restormer [35]
22.37
WeatherDiffes [5]
82.9M
30.71
0.931
LLFormer [19]
24.6M
23.65
0.816
0.169
RainDropDiff [5]
109.7M
32.43
0.933
Resfusion (ours)
7.7M
24.63
Resfusion (ours)
7.7M
Input
(2021)
Restormar (2022)
(2023)
(ours)
Input
AttentiveGAN (2018)
RaindropAttn (2019)
WeatherDiff (2023)
Resfusion (ours)
Ground Truth
Figure 4: Visual comparisons of the restored results by different image restoration methods on the LOL dataset and the Raindrop dataset.
FusionNet [23], CANet [27], UnfoldingNet [24], DMTN [25], and RDDM(SM-Res-N [17]. In order ensure fair comparison, we conducted experiments on two settings for the ISTD dataset, following the methods used in DMTN [25] and DHAN [22]: (1) The results evaluated resolution of 256 x 256 after being resized. (2) The original image resolutions (640 480) are maintained for evaluation.
LOL dataset [30] comprises 500 pairs of images, consisting of both low-light and normal- light versions, which are further divided into 485 training pairs and 15 evaluation pairs. The low-light images contain noise produced during the photo capture process. We compare the proposed method with the popular low-light enhancement methods, i.e., RetinexNet [30], KinD [31], KinD++ [32],
Zero-DCE [33], EnlightenGAN [34], Restormer [35], LLFormer [19], RDDM (SM-Res-N) [17], and RDDM (SM-Res)
[17]. Some existing methods [8, 36, 37] calculate metrics by adjusting the overall brightness based on reference images (called as using GT-mcan). However, this approach can introduce biases and potential unfairness. In accordance with LLFormer [19], we compute metrics without utilizing any reference information. To ensure a fair comparison, we conducted experiments on two settings for the LOL dataset, following the methods employed in RDDM [17] and LLFormer [19]: (1) The results are evaluated at a resolution of 256 x 256 after being resized. PSNR and SSIM are evaluated based in YCbCr color space. (2) The original image resolutions (600 400) are maintained for evaluation. PSNR and SSIM are evaluated RGB color space.
Raindrop dataset [39] is dataset designed for deraining, comprising 861 training image pairs for training, and image pairs dedicated for quantitative evaluations, denoted in [39] and [5] as Raindrop-A. We compare the proposed method with the popular deraining methods, i.c. pix2pix [40],

---

AttentiveGAN [39], DuRN [41], RaindropAttn [42], All-in-One [43], IDT [44], WeatherDiff [5], RainDropDiff [5], and (DDM(SM-Res-NiD))
We conduct experiments on two settings for the
Raindrop dataset for fairness, following the methods employed in RDDM [17] and WeatherDiff [5]: (1) The results are evaluated a resolution of 256 256 after being resized. (2) The original image resolutions are maintained for evaluation.
Quantitative comparison. As shown in the Tables 1, & 3, We provide the quantitative evaluation results on ISTD dataset, LOL dataset and Raindrop dataset. Our methods clearly outperform all competing methods by significant margins in terms of PSNR, SSIM, MAE and LPIPS across all three datasets. The current experimental results demonstrate that Resfusion achieves highly competitive results under these conditions: (1) utilizing only one U-net to predict resnoise. (2)
simply concatenating and 2o in the channel dimension. (3) employing a simple truncated linear schedule. (4) conducting only five sampling steps for all datasets. contrast, alternative methods often rely on intricate network architectures, including multi-stage [19, 23, 24], multi-scale [45], multi-branch [22] and prior knowledge of physics [9, 30, 33, 42], complex noise schedules [16, 17] and patch-overlapping strategy [5]. For the ISTD dataset and Raindrop dataset, we only employ one U-net to predict resnoise, outperforming RDDM with two U-nets to predict the residual term and the noise separately in terms of PSNR and SSIM. Resfusion use half the number of parameters of RDDM and achieved better quantitative evaluation metrics. For the LOL dataset, under the same parameters, Resfusion outperforms RDDM in terms of PSNR (+18%) and LPIPS (-40%) significantly. Furthermore, for all datasets, we employed a simple truncated linear schedule, while RDDM utilized complex custom noise schedule.
4
Ablation Study
4.1 The analysis of the residual term and the noise term
Input
Only Remove Noise
Only Remove Residual
Remove Resnolse
Ground Truth
Figure 5: The analysis of the residual term and the noise term on the LOL dataset. Only removing noise will reconstruct the details of the degraded image without causing any semantic shift. Only removing residual can only accomplish the semantic shift (from low-light to normal-light) without reconstructing the details. Removing resnoise can achieve both the semantic shift and the detail reconstruction.
Resfusion and DDPM share the consist form of the reverse inference process. The DDPM reverse process restores the original image distribution by gradually removing the noise (Eg) from Gaussian white noise, while resnoise-diffusio reverse process restores clean image by gradually removing the resnoise (reseg) from the noisy degraded images. Mathematically. the resnoise term (resc,) and
the noise term (Eg) only differ in the form weighted residual term (U-varyTTog Rg). other
D. words, Resfusion subtracts an extra weighted residual term while removing the noise term during each step of the reverse process.
According the Green's 's theorem [46], when the neural network is sufficiently robust, the components of the resnoise should be path independent. Based on this belief, we trained two separate neural networks on the LOL dataset. One network predicts only e and removes only the noise term during the resnoise-diffusion reverse process. The other network predicts only [I-VkrvVT Ro and removes only the weighted residual term during the resnoise-diffusion reverse process. As shown
7

---

in Fig 5, we qualitatively determine the functionalities of each component in the loss function of Resfusion. The weighted residual term are responsible for semantic shift, while the noise term handles detail reconstruction. Predicting resnoise resca achieves both semantic shifts and detail reconstruction, bridging the gap between the input degraded image and the ground truth, enabling effective image restoration.
4.2 Equivalent representations of the loss function
Input
X
R
Its
Ground Truth
Figure 6: Visual comparisons of the restored results generated by other equivalent loss functions on Raindrop dataset and LOL dataset. Using rese the loss function allows for better restoration of details while completing semantic shifting.
Based on Eq. (1), since én available, once we acquire either za or R, we can determine the other. Moreover, according to Eq. (18), since is obtainable and considering the equivalence between zo and R, we can also obtain the equivalent reverse process of resnoise-diffusion by predicting either Toe Rg. Similar to DDPM, the fundamental purpose of Resfusion is also predict zn (which is equivalent to predicting R). In contrast, DDPM reconstructs the distribution of the original image To by incrementally reducing noise from the Gaussian white noise, whereas Resfusion skips the generation of low-frequency information by utilizing &g for initialization. Therefore, in addition to the noise removal, Resfusion also involves removing weighted residual term to accomplish the semantic shifting. is worth mentioning that, unlike RDDM, predicting the noise term ep is not an equivalent loss function. Due the truncated schedule adopt, the starting point 2T" will only consist of weighted dn and c. As do is obtainable and serves as conditional input to the neural network, directly predicting the noise term would lead to the neural network learning simple pattern, resulting in training failure.
We compare the quantitative performance obtained by using different equivalent prediction targets as loss functions on ISTD dataset, LOL dataset and Raindrop dataset. We use the same backbone as described in Sec. 3 and employ the same truncated linear schedule, performing five sampling steps for all datasets. The original image resolutions are maintained for evaluation. As shown in Table 4. predicting rese, outperformed predicting Zoo and R, in terms of PSNR and SSIM on all three datasets. Predicting resca resulted better reconstruction of the fine details, as illustrated in Fig 6.
Table 4: Quantitative comparisons with other equivalent loss functions on ISTD dataset, LOL dataset and Raindrop dataset. We report PSNR, SSIM, MAE and LPIPS. The best and second-best results are highlighted in bold and underlined. "t" (resp. "L") means the larger (resp. smaller), the better.
ISTD [19]
LOL [30]
RainDrop [39]
Prediction Targets
PSNR
SSIM
MAB
PSNR
SSIM
LPIPS 1
PSNR
SSIM t
Z000
29.67
0.927
5.35
23.10
0.813
0.150
32.51
0.935
Rg
29.75
0.930
5.26
22.87
0.807
0.143
32.57
0.935
reség
30.09
0.932
5.34
24.63
0.860
0.107
32.61

---

DDPM
10 steps
20 steps
50 steps
100 steps
Resfusion
10 steps
20 steps
50 steps
100 steps
Figure 7: Visual comparisons between DDPM and Resfusion on the CIFARIO (32 x 32) dataset. do not cherry-pick any results. With the same sampling steps, Resfusion outperforms DDPM in semantic generation and detail reconstruction.
5 Discussion
Resfusion is not limited to image restoration. fact, It is a versatile framework which can be applied any general image generation domain. For image generation tasks, it is impossible to obtain any additional information, we redefine âg as a zero matrix. Thus we can obtain new definition of the residual term R =00 for image generation. Therefore, rese is redefined as resc = (1-var)vT-a zg- Applied with the redefined residual term and the resnoise, Resfusion's 's 8, forward and reverse process for image generation are completely consistent with the original resnoise- diffusion process for image retoration.
The redefined rese further explains why Resfusion can complete the reverse inference process in fewer sampling steps than DDPM, under the same noise schedule. Due to consistency between the reverse processes, both of Resfusion and DDPM require the removal of the noise term. However, compared to DDPM, Resfusion additionally add weighted zo instead of simply removing the noise. This is the key factor that allows Resfusion to diffuse faster.
We train DDPM, Resfusion, and DDIM [2] the CIFARIO (32 32) dataset [47] with the same backbone. The experimental details are provided in the Appendix A.4. A truncated linear schedule used for Resfusion, and linear schedule is used for DDPM and DDIM. We employ the Frechet Inception Distance (FID) [48] as the quantitative metric. As shown in Table 5, Resfusion significantly outperforms DDPM with the same sampling steps. At nearly half of the sampling steps, Resfusion achieves similar FID with DDPM. Interestingly. when using truncated linear schedule, the value of TT/T also closed 0.5, further validating Resfusion's ''s accelerated sampling property. Consistent with DDPM, Resfusion also performs stochastic steps during the reverse process. Due to its stochastic nature, similar to DDPM, Resfusion's 's performance is lower than the deterministic DDIM.
Table 5: Quantitative comparisons with DDPM and DDIM CIFAR1O (32 32) dataset. We report FID under different sampling steps. "4" means the smaller, the better.
CIFARIO (FID !)
DDPM
Resfusion(ours)
DDIM
10 steps
43.11
28.81
18.37
20 steps
24.88
15.46
10.93
50 steps
14.02
7.96
7.39
100 steps
9.79
6.31
6.21

---

Figure 8: Visualization of the five sampling steps, where the blue arrow represents the smooth equivalence transformation, and the red box represents the resnoise-diffusion reverse process. We select the LOL web Test dataset (which does not have ground truth) and the Raindrop-B test dataset (which is much more challenging than Raindrop-A) to showcase the effects of low-light enhancement and deraining. We directly use the pretrained models on LOL dataset and Raindrop dataset, demonstrating the strong robustness Resfusion.
Conclusion
We propose the Resfusion, general framework for image restoration. We explicitly provide the quantitative relationship between the residual term and the noise term, named as resnoise. Through employing the smooth equivalence transformation, Resfusion does not require any prior physical knowledge and can directly utilize existing noise schedules. Experimental results shows that Resfusion exhibits competitive performance for shadow removal, low-light enhancement, and deraining tasks, with only five sampling steps. It is important to that Resfusion not limited to image restoration and be applied to any image generation domain. The versatility of our framework lies in its ability to simultaneously model the residual term and the noise term. Our subsequent experiments have demonstrated that Resfusion can be easily applied to various image generation tasks and exhibits strong competitiveness.
unify the training and inference process.
7 Acknowledgements
This work is partially supported by the National Natural Science Foundation of China (62272248), Natural Science Foundation of Tianjin (23JCZDJC01010, 23JCNNIC0010) and the Key Program of Science and Technology Foundation Tianjin (24HHXCSS00004).
10

---

References
[1] Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 05:6840-6851,, 2020.
[2] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020.
[3] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE transactions pattern analysis and machine intelligence, 45(4):4713-4726 2022.
[4] Bahjat Kawar, Gregory Vaksman, and Michael Elad. Snips: Solving noisy inverse problems stochastically. Advances in Neural Information Processing Systems, 34:21757-2176,, 2021.
[5] Ozan Oxdenizci and Robert Legenstein. Restoring vision in adverse weather conditions with patch-hased denoising diffusion models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[6] Jay Whang, Mauricio Delbracio, Hossein Talebi, Chitwan Saharia, Alexandros G Dimakis, and Peyman Milanfar. Deblurring via stochastic refinement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition. pages 15293-16300 2022.
[7] Tao Wang, Kaihao Zhang, Ziqian Shao, Wenhan Luo, Bjorn Stenger, Tac- Tac-Kyun Kim, Wei Liu, and Hongdong Li. Lldiffusion: Learning degradation representations in diffusion models for low-light image enhancement. arXiv preprint ar30v 1467, 2023.
[8] Jinhui Hou, Zhiyu Zhu, Junhui Hou, Hui Liu, Huanqiang Zeng, and Hui Yuan. Global structure-aware diffusion process for low-light image enhancement. Advances Neural Information Processing Systems, 36, 2024.
[9] Langing Guo, Chong Wang, Wenhan Yang, Siyu Huang, Yufei Wang. Hanspeter Pfister, and Bihan Wen. Shadowdiffusion: When degradation prior meets diffusion model for shadow removal. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 14049-14058, 2023.
[10] Chunming He, Chengyu Fang, Yulun Zhang. Kai Li. Longxiang Tang. Chenyu You, Fengyang Xiao, Zhenhua Guo, and Xiu Li. Reti-diff: Illumination degradation image restoration with retinex-bass based latent diffusion model. arXiv preprint arXiv:2311.11638, 2023.
[11] Chunming He, Yuqi Shen, Chengyu Fang. Fengyang Xiao, Longxiang Tang. Yulun Zhang, Wangmeng Zuo, Zhenhua Guo, and Xiu Diffusion models in low-level vision: survey, arXiv preprint arXiv:2406.1113: 2024.
[12] Mauricio Delbracio and Peyman Milanfar. Inversion by direct iteration: An alternative to denoising diffusion for image restoration. arXiv preprint arXiv:2303. 10003//4333 /1335, 2023.
[13] Bahjat Kawar, Michael Flad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. Advances in Neural Information Processing Systems, 35:23593-2366 236066 2022.
[14] Ziwei Luo, Fredrik K Gustafsson, Zheng Zhao, Jens Sjölund, and Thomas B Schön. Image restoration with rean-ievgrting stochastic differential equations. arXiv preprint arXiv:2301.11699 2023.
[15] Guan-Horng Liu, Arash Vahdat, De-An Huang, Evangelos A Theodorou, Weili Nie. and Anima Anandku- mar. sb: Image-to-image schrödinger bridge. arXiv preprint wrXivv2302.05872. 2023.
[16] Zongsheng Yue, Jianyi Wang, and Chen Change Loy. Resshift: Efficient diffussin model for image super-resolution by residual shifting, Advances Neural Information Processing Systems, 36, 2024.
[17] Jiawei Liu, Qiang Wang, Huijie Fan, Yinong Wang. Yandong Tang, and Liangqiong Qu. Residual denoising diffusion models. arXiv preprint ar0iv:2308.1 13712, 2023.
[18] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. JEEE transactions on image processing, 13(4):600-61 -022 2004.
[19] Tao Wang, Kaihan Zhang, Tianrun Shen, Wenhan Luo, Bjorn Stenger, and Tong Lu. Uttr--iigh-defiinne low-light image enhancement: A benchmark and transformer-basee method. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 37, pages 2654-2662, 2023.
[20] Jifeng Wang, Xiang Li, and Jian Yang. Stacked conditional generative adversarial networks for jointly learning shadow detection and shadow removal. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages (788-1797 2018.

---

[21] Xiaowei Hu, Chi-Wing Fu, Lei Zhu, Jing Qin, and Pheng- Ann Heng. Direction aware spatial context features for shadow detection and removal. IEEE transactions on pattern analysis and machine intelligence, 42(1 1):2755-2888. 2019.
[22] Xiaodong Cun, Chi-Man Pun, and Cheng Shi. Towards ghost-free shadow removal via dual hierarchical aggregation network and shadow matting gan. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages
2020.
[23] Lan Fu, Changqing Zhou, Qing Guo, Felix Juefei-Xu, Hongkai Yu. Wei Feng, Yang Liu, and Song Wang. Auto-exposure fusion for single-image shadow removal. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10571-10580, 2021.
[24] Yurui Zhu, Zeyu Xiao, Yanchi Fang, Xueyang Fu, Zhiwei Xiong, and Zueng-Jun Zha. Efficient model- driven network for shadow removal. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 3635-3643 2022.
[25] Jiawei Liu, Qiang Wang, Huijie Fan, Wentao Li, Liangqiong Qu, and Yandong Tang. A decoupled multi-task network for shadow removal. IEEE Transactions Multimedia, 2023.
[26] Bin Ding, Chengjiang Long, Ling Zhang, and Chunxia Xiao. Argan: Attentive recurrent generative adversarial network for shadow detection and removal. Proceedings of the IEEF/CVF international conference on computer vision, pages 10222-1022,, 2019.
(27] Zipci Chen, Chengjiang Long, Ling Zhang, and Chunxia Xiao. Canet: A context-aware network for shadow removal. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4743-4752, 2021.
[28] Olaf Ronneherger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234-241. Springer, 2015.
[29] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pages 8162-8171. PMLR, 2021.
[30] Chen Wei, Wenjing Wang, Wenhan Yang, and Jiaying Liu. Deep retinex decomposition for low-light enhancement. arXiv preprint arXiv: 1008:/80..000 04560, 2018.
[31] Yonghua Zhang, Jiawan Zhang, and Xiaojie Guo. Kindling the darkness: A practical low-light image enhancer. In Proceedings of the 27th ACM international conference on multimedia, pages 1632-1640, 2019.
[32] Yonghua Zhang, Xiaojie Guo, Jiayi Ma, Wei Liu, and Jiawan Zhang. Beyond brightening low-light images. International Journal of Computer Vision, 129:1013-1037, 2021.
[33] Chunle Guo, Chongyi Li, Jichang Guo, Chen Change Loy, Junhui Hou, Sam Kwong, and Runmin Cong. Zero- Cero-reference deep curve estimation for low-light image enhancement. In Proceedings of the IEEE/CVF conference on computer vision and pattrrn recognition, pages 1780-1789, 2020.
[34] Yifan Jiang, Xinyu Gong, Ding Liu, Yu Cheng, Chen Fang, Xiaohui Shen, Jianchao Yang. Pan Zhou, and Zhangyang Wang. Enlightengan: Deep light enhancement without paired supervision. IEFE transactions on image processing, 30:2340-2349, 2021.
[35] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Restormer: Efficient transformer for high-resolution image restoration. In Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, pages 5728-572 5739, 2022.
[36] Yufei Wang, Renjie Wan, Wenhan Yang, Haoliang Li, Lap-Pui Chau, and Alex Kot. Low-light image enhancement with normalizing flow. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 2604-2612, 2022.
[37] Dewei Zhou, Zongxin Yang, and Yi Yang. Pyramid diffusion models for low-light image enhancement. arXiv preprint arXiv:2305. 10028, 2023.
[38] Richard Zhang, Phillip Isola, Alexii A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586 586-595, 2018.
12

---

[39] Rui Qian, Robby T Tan, Wenhan Yang, Jiajun Su, and Jiaying Liu. Attentive generative adversarial network for raindrop removal from single image. In Proceedings of the IEEE conference on computer vision and puttern recognition, pages 2482-2491, 2018.
[40] Phillip Isola, Jun- Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image to image translation with conditional adversarial networks. In Proceedings of the IEFF conference on computer vision and pattern recognition, pages 1125-1134, 2017.
[41] Xing Liu, Masanori Suganuma, Zhun Sun, and Takayuki Okatani. Dual residual networks leveraging the potential of paired operations for image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7007-7016, 2019.
[42] Yuhui Quan, Shijie Deng, Yixin Chen, and Hui Ji. Deep learning for seeing through window with raindrops. In Proceedings of the IEEE/CCV International Conference on Computer Vision, pages 2463-2471, 2019.
[43] Ruoteng Li, Robby T Tan, and Loong-Fah Cheong. All in one bad weather removal using architectural search. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3175-310 3185, 2020.
[44] Jie Xiao, Xueyang Fu, Aiping Liu, Feng Wu, and Zheng-Jun Zha. Image de-raining transformer. JEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[45] Langing Guo, Siyu Huang, Ding Liu, Hao Cheng, and Bihan Wen. Shadowformerr global context helps shadow removal. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 710-718, 2023.
[46] Bernhard Riemann. Grundlagen fur eine allgemeine Theorie der Functionen einer veründerlichen com- plexen Grösse. Adalbert Rente, 1867.
[47] Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
[48] Martin Heusel. Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[49] Calvin Luo. Understanding diffusion models: A unified perspective. arXiv preprint arXiv:2208.1 11970 2022.
[50] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprin! arXiv:2209.0300 2022.
[51] Arpit Bansal, Eitan Borgnia, Hong-Min Chu, Jie Li, Hamid Kazemi, Furong Huang, Micah Goldblum. Jonas Geiping, and Tom Goldstein. Cold diffusion: Inverting arbitrary image transforms without noise. Advances in Neural Information Processing Systems, 36, 2024.
[52] Zongsheng Yue, Jianyi Wang, and Chen Change Loy. Resshift: Efficient diffusion model for image super-resolution by residual shifting, arXiv preprint urXiv:2307.12344 2023.
[53] Patrick Essen, Robin Rombach, and Bjom Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873-12883, 2021.
[54] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv: 10196, 2017.
[55] Yunjey Choi. multiple domains. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 3188-8197, 2020.
Youngjung Uh. Jaejun Yoo. and Jung-Wor Woo Ha. Stargan v2: Diverse image synthesis for
[56] Ilya Loshchilov and Frank Hutter. 17Xiv:171.001001, 2017.
Decoupled
weight decay regularization. arxiv preprint
[57] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[58] Hieu Le and Dimitris Samaras. Shadow removal via shadow image decomposition. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 8578 8587, 2019.
13

---

[59] Yurui Zhu, Jie Huang, Xueyang Fu, Feng Zhao, Qibin Sun, and Zheng-Jun Zha. Bijective mapping network for shadow removal. In Proceedings the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 5627-5636, 2022.
[60] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of IEEE international conference on computer vision, pages 3730-3738. 2015.
[61] Wenhan Yang, Wenjing Wang, Haofeng Huang, Shiqi Wang, and Jiaying Liu. Sparse gradient regularized deep retinex network for robust low-light image enhancement. IEEE Transactions on Image Processing, 30:2072-20885 2021.
[62] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[63] Ziwei Luo, Fredrik K Gustafison, Zheng Zhao, Jens Sjölund, and Thomas B Schön. Refusion: Enabling large-size size realistic image restoration with latent-space diffusion models. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 1680-1691, 2023.
[64] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning, Advances in neural information processing systems, 30, 2017.
14

---

A Appendix Section
A.1 Detailed proof
According to Eq. (4) and Eq. (5), the forward process can be written as Eq. (16) and Eq. (17).
q( R) = Na var + (1 - (1
R1 (1-aa))
(16)
(x, Fo, R) = (zx va + (1- (1-
(17)
By simply performing a change of variable (my + Zg R, F2 - - R. 21-1 It-1 R). the
derivation of Eq. (18) is identical in form to (71) (84) in the reference [49]. where line 6 7 of Eq. (18) corresponds (73).
( To, R)
g Ro. R)g(x R))
(x R)
Naxi Vatte-1 +
(1 (1
ai, (1 a
Na (1 + (1 -1- vā,) -
x exp( It (Vaiu1-1 + (1 va() [x,-1 (Va, 20 + (1 v /a_ R)
2(1 a4)
2(1 (T--1)
[e (Varso + (1 va))
2(1 (++)
expl- -1 (xee - R) /a - -R)]2
[3x-11 - Rt) - 00 (00 R2)
2(1 at)
2(1 02-1)
[xx R) Var R))3
2(1 (ip)
<N R: aos (x R) + - - R) 3,11)
at
Vai(1 (1 Cq- 10) - R) + V vā-- a1) - R)
x <N(x-11:
R, B.)
1 at
(18)
Then we can derive Ta No. R) as Eq. (19).
/a, 1
(Io - R) + val - a(-1)
jaaa, Zo. R) = 1- Ti
1-as (x) - (R)-))+A R
(19)
We can derive Eq. (20) through Eq. (5).
Ft. - R = v TT. (Jn - 17) + - 6~ N(O,
(20)
Thus we can derive Eq. (21) through Eq. (19) and Eq. (20). By simply performing change of
variables (Zg + Fp R, Ft - It R), the derivation process becomes exactly identical in form to
the derivation of equations (115)- (124) in the reference [49], where Eq. (20) corresponds to (115)
and Eq. (19) corresponds to (116).
L C Esor,:
| 2o, R) (0022 e, RI), 1)1
20,
(21)
A 20?
1
(1o. R. R)
c) _ R)
B.
ua(x R),
Vau.
VI T.
According to Eq. (22), we can modify Eq. (21) as Eq, (23).
8.
B
(Ir - R
t)
+
R
(+t R + Va
V1 de
Var
v1 dk
8r
(x) (1 Voi R
6)
Voi
v1
T
(22)
/ 0
31
8
(xt
R
Va
g,
V1 0
V1
a
3r
(c+ (1 - Vai)
(a)vI
Fr
R)]
Vor
vI
15

---

1--1 - C
Pe (1
var) at R)] po(as (R) ()]1 (23)
EEzoeett 20; vaz
[2. c, R)
(e
vI-a. at
According to [1], the minimize term become Eq. (6).
A.2
Comparison with other methods
The main difference is in how the denoising diffusion, score, flow, or Schrödinger's bridge are adapted to image restoration. Different methods select various elements as prediction targets: the noise term (Shadow Diffusion [9], SR3 [3], WeatherDiffusion [5]). the residual term (DvSR [6], Rectified Flow [50]), the target image (ColdDiffusion [51], InDI [12D. or its linear transformation term SB (15]). Similar to RDDM [17], Resfusion simultaneously predicts both the residual term and the noise term, and provides the quantitive relationship between them.
Comparison with traditional diffusion- methods. Traditional diffusion-basee image restora- tion methods [5, 6, 7, 8, 9] adapt the diffusion model for image restoration tasks with degraded images as conditional input to implicitly guide the reverse generation process, without altering the original denoising diffusion process [1, 2]. Starting the reverse process from Gaussian white noise, traditional diffusion- brauuddo---beed models consider only the degraded images as conditional input, resulting in an increased number of sampling steps. Meanwhile, these models are often task sse-iseci,,,, requiring the design of different model structures based on different scenarios. By introducing the residual term into the diffusion forward process, Resfusion bridge the gap between the input degraded images and ground truth, starting the reverse process directly from the noisy degraded images. As a versatile methodology for image restoration, Resfusion does not require any physical prior knowledge, and the image restoration can be completed in just five sampling steps.
Comparison with RDDM [17]. RDDM can be seen as a diffusion process from the noisy input degraded image to the ground truth, while Resfusion represents a diffusion process from the noisy residual term to the ground truth. RDDM predicts the residual term and the noise term separately without specifying their weighted relationship, using a complex Automatic Objective Selection Algo- rithm (AOSA) to learn them. In contrast, Resfusion calculates the quantitative relationship between the residual term and the noise term, naming their weighted sum as resnoise. RDDM's ''s forward process accumulates the residual term and the noise term, making its forward and backward processes inconsistent with DDPM, leading to poor generalization and interpretability. By transforming the learning of the noise term into the resnoise term, Resfusion's 's reverse inference process becomes consistent with DDPM, unifying the training and inference processes. Lastly, RDDM requires a customized noise schedule, as using existing noise schedules results in performance loss. Through the smooth equivalence transformation in resnoise-diffusion process, Resfusion can directly use the existing noise schedule.
Comparison with Resshift [52]. Similar to RDDM, Resshift's forward process also adopts an accumulation strategy for the residual term and the noise term. Therefore. Resshift also requires the design of a complex noise schedule, which is formulated as equation (10) in Resshift [52]. Resfusion can directly use the existing noise schedule instead of redesigning the noise schedule. The reverse process of Resshift is inconsistent with DDPM. The form of Resfusion's 's reverse inference process consistent with the DDPM, leading to better generalization and interpretability. The prediction target of Resshift is zo. while the prediction target of Resfusion is rese. Given that the essence of rese is the noise term with an offset, and LDM models mainly predict the noise term, the loss function of Resfusion is extremely friendly to fine-tuning techniques such as Lora, which helps further scale up. Resshift diffuses in the latent space, utilizing the powerful encoding capability of models like VQ-GAN [53]. Resfusion, on the other hand, directly diffuses in the RGB space. Resshift only explores fixed degradations such as image super-resolution. Resfusion explores more complex scenarios, including shadow removal, low-light enhancement, and deraining.
Comparison with DvSR [6]. DvSR predicts clean images from input degraded images using a traditional (non-diffusion) network and calculates the residual term between the ground truth and the predicted clean images. DvSR employs denoising-based diffusion models to predict the noise term like DDPM, generating the residual term from Gaussian white noise. Unlike DvSR, Resfusion does not directly learn the residual term. Instead, it indirectly learns the distribution of the residual term through resnoisc-diiee diffoisidd process.
16

---

Comparison with ColdDiffusion [51]. ColdDiffusion aims to completely remove random noise from the diffusion model, replacing with other transformations like blurring and masking, In contrast, Resfusion still incorporates noise diffusion. As shown in our ablation study, Resfusion requires the noise term for detail recovery. Since ColdDiffusion discards random noise, needs additional degradation injection enhance generation diversity. To simulate degradation processes for various restoration tasks, ColdDiffusion uses Gaussian blur for deblurring, snowification transform for snow removal, etc. These specific explorations might lose generality. Resfusion employs the residual term for directed diffusion from the ground truth to the noisy residual term, eliminating the need for task-specific degradation operators. Additionally, Resfusion provides solid theoretical derivation, whereas ColdDiffusion lacks a theoretical basis.
A.3 Image translation
Resfusion can also be implemented in image-to-image distribution transformation. By redefining co as the translated image and zo the target image, we can easily transition Resfusion from image restoration to image translation. We train Resfusion with a truncated linear schedule on CelebA-HQ (64 64) dataset [54] and AFHQV2 (64 x 64) dataset [55] for image translation with 50 sampling steps. We selected the following image translation tasks: "Dog + Cat". "Male - Cat", "Male + Female", and "Female - Male". As shown Fig. 9, Resfusion can effectively model the shift between image domains, making it unified methodology for wider range of image generation tasks.
Dog Cat
Male Cat
Male + Female
Female + Male
Figure 9: Visual results for image translation the CelebA-HQ dataset and AFHQV2 dataset. The
images are presented in pairs, with the translated image on the left and the target image on the right. We showease the visual results of Resfusion for image translation tasks "Dog - Cat", "Male - Cat", "Male + Female", and "Female - Male".
17

---

Table 6: Experimental settings for Resfusion during the training stage.
Image Restoration
Image Generation
Image
Tasks
Shadow Removal
Low-light
Deraining
Translation
CelebA-HQ
Datasets
ISTD
LOL
Raindrop
CIFAR-10
AFHQ-V2
Batch size Image/patch size
32
32
32
128
128
256
256
256
32
64
fr
Inn
lin
lin
0
lin
Sampling steps
5
5
5
10-
50
Learning rate
1.1e-4
1.1c-4
1.1e-4
2c-4
2c-4
Training epochs
5k
3k
3k
A.4 Experimental setting details
We use the PyTorch Lightning framework to train all the models, utilizing the AdamW [56] optimizer with the default settings of PyTorch [57]. Following Liu et [17], we utilize the THOP to compute the number of parameters (Params) and multiply-accumula
operations (MACs). We only employ one U-net to predict resnoise and simply utilize a truncated Linear schedule across all tasks. For all the tasks, we implement regular MSE loss as the loss function. The detailed experimental settings are provided in Table 6. All experiments listed in Table be carried out with 8 NVIDIA RTX A6000 GPUs
Image restoration. We evaluate our method on several image restoration tasks, including shadow removal, low-light enhancement, and image deraining on 3 different datasets. For fair comparisons, the results of other image restoration methods are referenced from previously published papers [5, 7. 17, 19, 45] whenever possible. For all image restoration tasks, we used an identical U-net as the backbone, which the same as RDDM [17]. We take the shadow images and shadow masks together as the input condition (similar to [17, 58, 59]) for the ISTD dataset and only degraded images as the input condition for other datasets. We simply concatenate T; and Ộo (and shadow masks) together in the channel dimension and feed them into the network. For the LOL dataset, we do not use pre-processing and post-processing techniques like Histogram Equalization and GT-mmen. For the Raindrop dataset, we evaluate PSNR and SSIM based on the luminance channel Y of the YCbCr color space in accordance with previous work [5, 17, 39J. We employ a patch size of 256 x 256 for all the datasets during the training stage. We use the peak signal-to cennal-o--noise ratio (PSNR), structural similarity (SSIM), learned perceptual image patch similarity (LPIPS), and mean absolute error (MAE) quantitative metrics.
Image generation and image translation. For image generation on the CIFARIO (32 x 32) dataset, we utilize the same U-net structure as DDIM [2]. In contrast to DDIM which employs a linear schedule with T 1000 and a quadratic selection procedure to select sub-sampling steps, we use linear schedule with T 100 and a linear selection procedure select sub-sampling steps (for fair comparison with truncated linear schedule) while training DDPM and DDIM. We use the Frechet Inception Distance (FID) as the quantitative metric. For image translation tasks, employ U-net structure with the same configuration as used in DDIM for CelebA [60] as the backbone. To increase the diversity of the generated images, the translated images not fed into the network as conditional input.
A.5 Algorithm
Based on the derivations from the Sec. 2.1 and Sec. 2.2, the training and inference processes of Resfusion can represented as Algorithm 1 and Algorithm 2. We highlight the modifications in our training and inference algorithms compared to DDPM in red. Just like the vanilla Denoising Diffusion Probabilistic Models (DDPM), Resfusion gradually fit &; to Zo, implicitly reducing the residual term between áo and with the resnoise during the reverse inference process. Through transforming the learning of the noise term into the resnoise term, the form resnoise-diffusio reverse inference process is consist with DDPM, leading excellent interpretability.
18

---

Algorithm Training Algorithm for Resfusion Algorithm Inference Algorithm for Resfusion
Require:
total diffusion steps T. degraded Require: total diffusion steps T. degraded image
image and ground truth dataset ( =0)n
cg. pretrained Resfusion model reseg.
B. = 1-a; 1-ū--1 fe
T arg min, 1va,,
T arg min Ivā
repeat
Sample ~~ N (,,,1
Sample (+h, 20) D, N(O.T)
2T v arr + =V aTe
Sample ~~ Uni (1rm
for t= = T T - -I,..., do
R = -p
Sample 2 N(O,
Jt = T +1 v R C.C
21-1 vas
1 (2) VI-a;
(reseo(Te,,00. fo.
(1- L-vaa)/i-a R
B.
V B1z
take gradient step on Vellresc resco 8o. t)l
end for
until convergence
return TD
val
1 (21
VI-ai
resco (F1 $o. 1))
A.6 Resource efficiency
We compare the parameters, multppy--cccmmlatt operation (MACs) and inference time with other image restoration methods on ISTD [20] dataset, LOL [30] dataset and Raindrop [39] dataset by THOP, using 256 x 256 images the input. For ISTD dataset, PSNR and SSIM are evaluated at a resolution of 256 x 256 after being resized. For LOL dataset and Raindrop dataset, the original image resolutions are maintained for the evaluation of PSNR and SSIM. The experimental results are quoted from the results of previous papers as well as our implementation based on open source code.
As shown in Table 7, for the ISTD dataset, compared to Shadow Diffusion [9], Resfusion has 5 fewer parameters, 5 x fewer sampling steps, and 20x fewer MACs. For the LOL dataset, compared to LLDiffusion [7], Resfusion has 6x fewer sampling steps. For the Raindrop dataset, compared to RainDiff [5], Resfusion has 10x fewer parameters, 10x fewer sampling steps, and 50x fewer MACs. Experiments in shadow removal, low-light enhancement, and deraining demonstrate the effectiveness of Resfusion, enabling computationally constrained researchers to utilize our model for image restoration tasks.
Table Resource efficiency and performance analysis by THOP on ISTD dataset, LOL dataset and Raindrop dataset. "MAC" means multiply-accumule operation. The best and second-best results are highlighted in bold and underlined. "f" (resp. "1") means the larger (resp. smaller), the better. We use the symbol "." to indicate models or results that are unavailable.
Methods
PSNR
SSIM
Params
MACs(G) Steps
Inference Time (s)
ISTD Dataset
Shadow Diffusion [9]
32.33
0.969
55.5M
0.024 = 0.600
SR3 [3]
27.49
0.871
155.3M
1553xx100- = 15530.0
-X 100
Resfusion (ours)
31.81
0.965
7.7M
33.3x5 167.5
0.027 0.135
LOL Dataset
LLFormer [19]
23.65
0.816
24.5M
22.0x 22.0
0.092 0.092 0.092
LLDiffusion [7]
24.65
0.843
-x30 =
-X =-
Resfusion (ours)
24.63
0.860
7.7M
32.9x5 164.5
0.027 0.135
Raindrop Dataset
RainDiffes [5]
32.29
0.942
-x10=
-x10=-
RainDiff [5]
32.43
0.933
109.7M
248.4x50 = 12420.0
-x50= =-
WeatherDiffea [5]
30.71
0.931
82.9M
463.1x25 = 11577.5
0.328 8.20
WeatherDiffias [5]
29.66
0.923
85.6M
261.8x50
13090.0
0.439x50- = 21.95
Resfusion (ours)
32.61
0.938
7.7M
32.9x5 164.5
0.027 0.135
19

---

A.7
Truncated schedule
Ery Dodit unt wniood Thrnuttl Schotuls
Input
Truncated Schdule
Original Schoule
(b)
Figure 10: (a) Visualization of the relationship between the enror coefficient and T. Technically, can use the Truncated Schedule to eliminate this error when is small. (b) Visual comparisons between Truncated Schedule and Original Schedule under five sampling steps (T' /T = 5/12). In terms of visual perception, the absence of Truncated Schedule will lead to residual shadows.
We observed that in the actual diffusion forward process, the noise addition steps are uniformly spaced and discrete. The discontinuity of diffusion steps implies that when we approximate the acceleration point using Eq. (8), the offset of this approximate acceleration point relative to the ideal acceleration point is unavoidable, because ensuring the existence of intersection point with no offset requires that the gray arrow and the violet arrow in Fig. 2 must be continuous. This offset actually quantifies the confidence level of the approximate equivalence TT iT in Eq. (15). When T is small, the diffusion steps are divided sparsely, and the offset can be unacceptable. The absolute value of the offset can be derived Eq. (24).
(24) As shown in Figure 10 (a), the absolute offset ||(1 - 201-2vas a Rl exponentially decreases with the increase of T. When T is relatively small, this error is not negligible. However, this potential instability can be avoided in practical experiments, with a noise schedule named Truncated Schedule based on the existing noise schedules. In order to control the offset, we define an offset threshold h with default value of 0.01. When decreasing Vo,, the first element less than 0.5 is denoted as Va, the difference between 0.5 and va, greater than the offset threshold h, Va,, will be reassigned to 0.5 and the following elements will be truncated from here. Since the diffusion steps after the acceleration point is not involved in the actual diffusion process, direct truncation can avoid potential risks. Taking the truncated linear schedule [29] and ' = 25 as example, Fig. 11 demonstrates how to achieve the acceleration point T" = 10. As shown Figure 10 (b), Truncated Schedule can effectively eliminate "residual shadows".
| èT || = @1(2v/r' 1)x0 + (1-2var) Rol [1(1 2//TT"))
Truncated Unear Schedule
0.9
06
V0
0.4
02-
0.
25
Figure 11: The schematic diagram of our truncated linear schedule. Taking T = 25, the left figure shows the 10 diffusion steps obtained by truncated linear schedule; the right figure shows the comparison of the truncated linear schedule and the linear schedule [29].
20

---

A.8 LOL-v2-real dataset
Table 8: Quantitative comparisons with other low-light enhancement methods on LOL-v2-real dataset. We report PSNR, SSIM and LPIPS. The best and second-bess results are highlighted bold and underlined. "f" (resp. "{") means the larger (resp. smaller), the better.
LOL-v2-real [61]
Method
PSNR
SSIM t
LPIPS +
Restormer [35]
18.69
0.834
0.232
LLFormer [19]
20.06
0.792
0.211
Resfusion (ours)
22.06
0.839
0.175
The LOL-v2-real dataset [61] includes visual degradations such as decreased visibility, intensive noise, and biased color. It contains 689 image pairs of both low-light and normal-light versions for training and 100 image pairs for evaluation. All experimental settings are exactly the same as the LOL dataset. As shown Figure 12, compared to Histogram Equalization, Resfusion can significantly reduce noise, while also achieving better color offset, demonstrating strong denoising capabilities. provide results terms of PSNR, SSIM, and LPIPS on LOL-v2-real dataset Table 8.
kpst
Figure 12: Visual comparisons of the restored results by different image restoration methods on the LOL-v2-real dataset.
A.9 Limitations and Future work
Task specific. Our main effort has been directed towards creating general prototype model for image restoration and generation. This approach may lead to some performance limitations when compared task-specific state-of-the-art methods [9, 10]. To enhance performance for particular tasks, potential strategies include employing task-specific backbones, incorporating physical prior knowledge, and utilizing customized noise schedules.
Feature fusion. In the reverse process, we simply concatenate the noisy image 2; at time step with the conditional input 2o in the channel dimension. is worth exploring efficient feature fusion strategics, such as cross-attention multi-stage, multi-scale, and multi-branch.
Latent space. The diffusion process in Resfusion is conducted in the original pixel space. Some studies [52, 62, 63] have shown that conducting diffusion process in the latent space [53, 64] can significantly reduce computational complexity while ensuring the quality of generated images, which worth exploring in the future.
21

---

A.10 More results and failure cases
Input
DSC (2019)
DHAN (2020)
DMTN (2023)
Resfusion (ours)
Ground Truth
Figure 13: More visual comparisons of the restored results by different shadow-removal methods on the ISTD dataset.
Input
ELGAN (2021)
Restormer (2022)
LLformer (2023)
Resfusion (ours)
Ground Truth
Figure 14: More visual comparisons of the restored results by different low-light enhancement methods on the LOL dataset.
22

---

Input
(2018)
RaindropAttn (2019)
(2023)
(ours)
Figure 15: More visual comparisons of the restored results by different deraining methods on the Raindrop dataset.
Input
DSC (2019)
DHAN (2020)
DMTN (2023)
Resfusion (ours)
Ground Truth
Figure 16: Visual comparisons of the restored results by different shadow-remova methods on the ISTD dataset. (failure cases)
PURPOSE
PUBPORE
PURRUSE
POBPOSI
PURPOSE
Input
Raindrophttn
Figure 17: Visual comparisons the restored results by different deraining methods on the Raindrop dataset. (failure cases)
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The principal assertions in the abstract and introduction accurately represent
the paper's 's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of the work in Appendix A.9.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]

---

Justification: As shown in section and Appendix A.1, we provide the full set of assumptions and a complete (and correct) proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide detailed information about our experimental setting section and Appendix A.4. We provide the anonymous inference results link in our supplementary material's README.md.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
25

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: provide our code and anonymous inference results link in our supplemen- tary material. We used only open-source datasets for all our experiments.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / httppsscccppbciccccceeceeecc heabblic/giidee CodeeSbbm ssi aasionPoiicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: As shown in section 3 and Appendix A.4, we specify all the training and test details.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars reported because it would be computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
26

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: As shown in Appendix A.4 and Appendix A.6, we provide sufficient informa- tion the computer resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: httpp::/neet //tp:://neurp.peeeeee wtt.ipp...ppppuuee cccpublic//ette d ines?
Answer: [Yes]
Justification: The research is conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The tasks selected all existing public problems, and the datasets used are widely utilized and publicly available.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
27

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The creators original owners of assets used in the paper are properly credited, and the licenses and terms of use are explicitly mentioned and properly respected.
Justification:
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
28

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The code and inference results introduced in the paper are well documented.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
29

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "nn8QQ6NtYr", "reviewer": "Reviewer_vAm8", "summary": "This paper proposes a method that leverages generative diffusion for image restoration tasks. The authors suggest incorporating the residual term, defined as the difference between the corrupted and clean images, into the forward and reverse diffusion processes. The forward process is described by a Markov chain, where the probability of the image at each time step $t$ is conditioned on the previous step, $t - 1$, and the residual term. This results in a forward where at each time step $t$, the image obtained at the previous time step $t - 1$ is mixed with the residual term and an additional white Gaussian noise. Consequently, the image at each time step can be described as a weighted sum of three components: the clean image, the residual term, and an additive white Gaussian noise. \n\nSince the residual term is the difference between the corrupted and clean images, this weighted sum can be rewritten as a weighted sum of the clean image, corrupted image, and Gaussian noise. The forward process stops when the weight of the clean image becomes approximately zero. Thus, the forward process gradually transfers a clean image into a weighted sum of the corrupted image and white Gaussian noise. Correspondingly, the reverse process starts from this weighted sum and incrementally reduces the corruption and noise. Because the corrupted image is available, the initialization for the reverse process is easily obtained by mixing the corrupted image with white Gaussian noise. \n\nThe authors derive the expression for the mean of the reverse process and train a network to predict, at each step, the difference between the corrupted noisy image obtained at that step and the clean image. The proposed algorithm evaluated thorough experiments on three reconstruction tasks: shadow removal, low light enhancement, and deraining, comparing the performance with recent competing methods. In all experiments, the authors apply five diffusion steps. The results show that the proposed algorithm generally outperforms the competitors while consuming fewer computational resources (multiplication operations times the number of diffusion steps).", "questions": "To provide a more comprehensive background, it might be beneficial to cite [3] which introduced a general framework for leveraging generative diffusion for solving image restoration problems where the observed image is contaminated by linear degradation and additive white Gaussian noise.\n\n[3] Bahjat Kawar, Gregory Vaksman, and Michael Elad. SNIPS: Solving noisy inverse problems stochastically. Advances in neural information processing Systems 34:21757-21769, 2021.", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "I believe the idea of conditioning on the residual term is original and significant. It allows deriving a reverse process that starts from the corrupted image mixed with noise rather than from pure noise, reducing the number of diffusion steps required for image restoration.", "weaknesses": "- Several notations are used without being defined. For example, the definitions of $\\alpha_t$, $\\beta_t$, $\\bar{\\alpha}_t$ are missing. Although the paper generally follows the notations defined in [1], defining these notations in the current paper would make it easier for readers to follow the explanations.\n- Some mathematical derivations are not detailed enough. For example, the proof in Appendix A.1 is not sufficiently detailed. Another example is derivation of $\\mu_{\\theta}$ in Equation (13), which is missing. It is completely fine to rely on derivations in [1] or [2], but I believe including the relevant parts of the derivations in the appendix would greatly help readers follow the derivations.\n\n[1] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020.\n\n[2] Calvin Luo. Understanding diffusion models: A unified perspective. arXiv preprint arXiv:2208.11970, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review and thoughtful feedback. Below we address specific questions and comments.\n\n> Several notations are used without being defined. For example, the definitions of \n$\\alpha_t$, $\\beta_t$, $\\overline{\\alpha}_{t}$ are missing. \n\nFor all constant hyperparameters, our definitions are completely consistent with the reference [1].\n\nWe will certainly add a detailed definition of the constant hyperparameters in the revised paper.\n\nWe provide a detailed definition (which is also declared in Author Rebuttal Section 1):\n- $\\beta_t = 1 - \\alpha_t$\n- $\\overline{\\alpha}_{t} =  \\prod_{s=1}^{t}\\alpha _s $\n- the $\\Sigma_{\\theta}$ is taken fixed as $\\widetilde\\beta_{t} = \\frac{1-\\overline{\\alpha}_{t-1}}{1-\\overline{\\alpha}_{t}} \\beta_t$\n\n> Some mathematical derivations are not detailed enough. For example, the proof in Appendix A.1 is not sufficiently detailed. \n\nBy simply performing a change of variable ($x_0 \\rightarrow x_0-R$, $x_t \\rightarrow x_t-R$, $x_{t-1} \\rightarrow x_{t-1}-R$), the derivation of Eq. (18) is identical in form to (71)-(84) in the reference [2], where line 6-7 of Eq. (18) corresponds to (73).\n\nWe will certainly add a detailed derivation of Eq. (18) in the revised paper.\n\n>  Another example is derivation of $\\mu_{\\theta}$ in Equation (13), which is missing.\n\nThe derivation of $\\mu_{\\theta}$ corresponds to Eq. (19) - Eq. (23) in Appendix A.1. We provide a detailed explanation of the derivation process in Author Rebuttal Section 2.\n\nWe will certainly clarify the connection between Eq. (13) and Eq. (19) - Eq. (23) in the revised paper.\n\n> To provide a more comprehensive background, it might be beneficial to cite [3] which introduced a general framework for leveraging generative diffusion for solving image restoration problems where the observed image is contaminated by linear degradation and additive white Gaussian noise.\n\nSNIPS [3] combines annealed Langevin dynamics and Newton's method to arrive at a posterior sampling algorithm, recovering the clean images from the white Gaussian noise.\nAs a pioneer in exploring the generative diffusion processes to solve the general linear inverse problems ($y=Hx+z$), SNIPS has made remarkable contributions to image restoration.\nBy incorporating the residual term into the diffusion forward process, Resfusion recovers the clean images directly from the noisy degraded images.\nAlso, Resfusion explores more complex scenarios where the degradation $H$ and the noise level are unknown,\nsuch as shadow removal, low-light enhancement, and deraining, achieving competitive performance.\n\nWe will certainly introduce SNIPS [3] in the background of the revised paper.\n\n## References\n[1] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020.\n\n[2] Calvin Luo. Understanding diffusion models: A unified perspective. arXiv preprint arXiv:2208.11970, 2022.\n\n[3] Bahjat Kawar, Gregory Vaksman, and Michael Elad. SNIPS: Solving noisy inverse problems stochastically. Advances in neural information processing Systems 34:21757-21769, 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "agBLNfUTXS", "reviewer": "Reviewer_DA7x", "summary": "This paper proposes to start the reverse diffusion process from the noisy degraded images for image restoration. It introduces a weighted residual noise as the prediction target and leverage a smooth equivalence transformation to find the starting noise. The experiments shows competitive performance on shalow removal, low-light enhancement and deraining, with shortened sampling steps.", "questions": "See weakness.", "limitations": "See weakness.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1, It predicts the residual noise to allow for diffusion directly from the noisy degraded images\n\n2, It transforms the learning of the noise term into the renoise term and follows the same inference process as DDPM\n\n3, Shortened inference steps without redesigning the noise schedule; sota results", "weaknesses": "1, ResShift also shifts toward the residual term. What are the advantages and disadvantages compared with ResShift?\n\n2, Visualization of the five inference steps is lacking.\n\n3, What is the results with the complete inference steps (e.g., T=1000)?\n\n4, In discussion, what is the model size? Can a 7.7M model achieve such good results in Figure 7?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review and thoughtful feedback. Below we address specific questions and comments.\n\n> ResShift also shifts toward the residual term. What are the advantages and disadvantages compared with ResShift?\n\nWe provide the differences between Resfusion and Resshift:\n- Similar to RDDM [1], Resshift [2]'s forward process also adopts an accumulation strategy for the residual term and the noise term.\n  Therefore, Resshift also requires the design of a complex noise schedule, which is formulated as equation (10) in reference [2].\n  Resfusion can directly use the existing noise schedule instead of redesigning the noise schedule.\n- The reverse process of Resshift is inconsistent with DDPM [3]. \n  The form of Resfusion's reverse inference process is consistent with the DDPM, leading to better generalization and interpretability.\n- The prediction target of Resshift is $x_{0}$, while the prediction target of Resfusion is $res\\epsilon$. \n  Given that the essence of $res\\epsilon$ is noise with an offset, and LDM models mainly predict noise, \n  the loss function of Resfusion is extremely friendly to fine-tuning techniques such as Lora, which helps further scale up.\n- In terms of the forward process, Resshift only performs shifting on the residual term. \n  Resfusion not only shifts the residual term but also degrades the ground truth $x_{0}$, which ensures the image normalization.\n- Resshift diffuses in the latent space, utilizing the powerful encoding capability of models like VQ-GAN. \n  Resfusion, on the other hand, directly diffuses in the RGB space.\n- Resshift only explores fixed degradations such as image super-resolution. \n  Resfusion explores more complex degradations including shadow removal, low-light enhancement, and deraining.\n\nWe will certainly add the differences between Resfusion and Resshift into Appendix A.2 of the revised paper.\n\n> Visualization of the five inference steps is lacking.\n\nWe present the visualization results of the five sampling steps and use the pretrained model on the LOL dataset to directly infer images from the Internet (without ground truth). \nThe visual results are stunning, as detailed in Figure 1 in the Author Rebuttal PDF.\n\nWe will certainly add the visualization in the revised paper.\n\n> What is the results with the complete inference steps (e.g., T=1000)?\n\nWe try $T'/T=100/272$ on the Raindrop dataset. \nAll hyperparameters are consistent with the original paper. \nIt is worth noting that increasing T will lead to an increase in training time (as the model needs to learn more scales of resnoise) and a decrease in inference speed.\nThe results of PSNR, SSIM, and LPIPS are provided. \nIncreasing T will result in a slight decrease in PSNR and SSIM (this may be due to the training set and test set not being completely i.i.d.), \nbut it will yield better visual perception (LPIPS).\n\n| $T'/T$           | PSNR $\\uparrow$ | SSIM $\\uparrow$ | LPIPS $\\downarrow$ |\n|------------------|:---------------:|:---------------:|:------------------:|\n| Resfusion (ours) |                 |                 |                    |\n| 5/12             |      32.61      |      0.938      |       0.061        |\n| 100/272          |      32.04      |      0.923      |       0.050        |\n\nA branch out: Will reducing T in traditional diffusion-based models increase PSNR and SSIM?\n\nWe try WeatherDiff$_{64}$ [4] on the Raindrop dataset, the answer is definitely __*NO*__.\nThe results of PSNR, SSIM, and LPIPS are provided.\n\n| $T$                    | PSNR $\\uparrow$ | SSIM $\\uparrow$ | LPIPS $\\downarrow$ |\n|------------------------|:---------------:|:---------------:|:------------------:|\n| WeatherDiff$_{64}$ [4] |                 |                 |                    |\n| 5                      |      27.24      |      0.925      |       0.074        |\n| 25                     |      30.71      |      0.931      |       0.060        |\n\n> In discussion, what is the model size? Can a 7.7M model achieve such good results in Figure 7?\n\nAs mentioned in discussion and detailed in Appendix A.4, for image generation on the CIFAR10 ($32\\times32$) dataset, \nwe utilize the same U-net structure as DDIM [5]. The parameter size of the denoising backbone is 35.72M.\n\n## References\n[1] Liu, Jiawei, et al. Residual denoising diffusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.\n\n[2] Yue, Zongsheng, Jianyi Wang, and Chen Change Loy. Resshift: Efficient diffusion model for image super-resolution by residual shifting. Advances in Neural Information Processing Systems, 2024.\n\n[3] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020.\n\n[4] Özdenizci, Ozan, and Robert Legenstein. Restoring vision in adverse weather conditions with patch-based denoising diffusion models. IEEE Transactions on Pattern Analysis and Machine Intelligence 45.8 (2023): 10346-10357.\n\n[5] Song, Jiaming, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SBQogqNVNq", "reviewer": "Reviewer_4Jy9", "summary": "This paper presents a general diffusion framework for image restoration, named Resfusion. The main idea is to introduce a residual term to DDPM to directly generate clean images from degraded images. Moreover, the form of the inference process is consistent with the DDPM and allows very few sampling steps. The model is evaluated on ISTD, LOL, and Raindrop dataset. Several ablation experiments on residual terms and loss functions are also conducted. In discussion, the authors show that their method can be used for general image generation by set $\\hat{x}_0$ to 0.", "questions": "The proposed method seems to be sensitive to noise. Can you train the model for the denoising tasks?", "limitations": "Please see the Weaknesses.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The idea of adding a residual term to the diffusion process is interesting.\n- The proposed smooth equivalence transformation is promising.\n- The experimental results are good.", "weaknesses": "- How do you get/design Eq. (3)? I wonder if the term $(1-\\sqrt{\\alpha_t})R$ is manually designed or derived from a specific equation.\n- In Eq. (6), $\\beta_t$ is not defined in the context.\n- I can understand that Eq. (8) aims to ensure the coefficient of $x_0$ in Eq. (7) is close to 0. But how can you obtain Eq. (9) in which $\\hat{x}_0$ is non-zero?\n- The overall derivation in Section 2.2 is unclear and confusing. Please make the connections between Equations more smooth. In Eq. (12), how do you get the variance $\\Sigma_\\theta$? And how do you obtain Eq. (13)? Please explain why and add the reference papers.\n- In experiments, it would be better to unify the evaluation metrics. For example, using PSNR, SSIM, LPIPS for all tasks and datasets (FID or MAE can be also added).\n- The presentation (especially the derivation) should be improved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review and thoughtful feedback. Below we address specific questions and comments.\n\n> How do you get/design Eq. (3)? I wonder if the term $(1-\\sqrt{\\alpha_t})R$ is manually designed or derived from a specific equation.\n\nAs shown in Figure 2 in the original paper, \nthe resnoise-diffusion reverse process can be imagined as doing diffusion reverse process from $R+\\epsilon$ to $x_{0}$ (as shown by the violet arrow).\n\n$x_{t}$ can be represented as a weighted sum of $x_{0}$ and $R$ (in the forward process, we know the ground truth).\nAt timestep $t$, to maintain consistency with DDPM [1], the coefficient of $x_{0}$ is determined as $\\sqrt{\\overline\\alpha_t}$.\nFollowing the principles of similar triangles, the coefficient of $R$ at step $t$ is computed as $1-\\sqrt{\\overline{\\alpha}_{t}}$, and we can derive Eq. (5).\n\nThrough the reparameterization technique, we can derive Eq. (3) (or Eq. (4)) and Eq. (5) from each other. \n\n> In Eq. (6), $\\beta_t$ is not defined in the context.\n\n>  In Eq. (12), how do you get the variance $\\Sigma_{\\theta}$ ?\n\nFor all constant hyperparameters, our definitions are completely consistent with the DDPM [1].\n\nWe will certainly add a detailed definition of the constant hyperparameters in the revised paper.\n\nWe provide a detailed definition (which is also declared in Author Rebuttal Section 1):\n- $\\beta_t = 1 - \\alpha_t$\n- $\\overline{\\alpha}_{t} =  \\prod_{s=1}^{t}\\alpha _s $\n- the $\\Sigma_{\\theta}$ is taken fixed as $\\widetilde\\beta_{t} = \\frac{1-\\overline{\\alpha}_{t-1}}{1-\\overline{\\alpha}_{t}} \\beta_t$\n\n> I can understand that Eq. (8) aims to ensure the coefficient of $x_{0}$ in Eq. (7) is close to 0. But how can you obtain Eq. (9) in which $\\hat{x}_{0}$ is non-zero?\n\nAs defined in the first line of Section 2.1 in the original paper, $\\hat{x}_{0}$ represents the input degraded image,\nwhich is obtainable during the resnoise-diffusion reverse process.\n\n> The overall derivation in Section 2.2 is unclear and confusing. Please make the connections between Equations more smooth.\n\nWe provide a detailed logical relationship for the equations in Section 2.2:\n- To obtain a computable starting point, we introduce the smooth equivalence transformation technique, corresponding to Eq. (7) - Eq. (9).\n- Because both the forward and backward processes only involve $t \\le T'$, \n  we unify the training and inference processes, corresponding to Eq. (10) - Eq. (14).\n- We explained the working principle of Resfusion from the perspective of vector intersection, corresponding to Eq. (15).\n\nWe will certainly provide a more detailed logical relationship for the equations in Section 2.2 in the revised paper.\n\n> And how do you obtain Eq. (13)? \n\nThe derivation of $\\mu_{\\theta}$ corresponds to Eq. (19) - Eq. (23) in Appendix A.1. We provide a detailed explanation of the derivation process in Author Rebuttal Section 2.\n\nWe will certainly clarify the connection between Eq. (13) and Eq. (19) - Eq. (23) in the revised paper.\n\n> In experiments, it would be better to unify the evaluation metrics. For example, using PSNR, SSIM, LPIPS for all tasks and datasets.\n\nWe provide results in terms of PSNR, SSIM, and LPIPS for all tasks:\n\n| Methods                | PSNR $\\uparrow$ | SSIM $\\uparrow$ | LPIPS $\\downarrow$ |\n|------------------------|:---------------:|:---------------:|:------------------:|\n| ISTD Dataset           |                 |                 |                    |\n| DMTN [2]               |      30.42      |      0.965      |       0.037        |\n| RDDM (SM-Res-N) [3]    |      30.91      |      0.962      |       0.031        |\n| Resfusion (ours)       |      31.81      |      0.965      |       0.030        |\n| LOL Dataset            |                 |                 |                    |\n| Restormer [4]          |      22.37      |      0.816      |       0.141        |\n| LLFormer [5]           |      23.65      |      0.816      |       0.169        |\n| Resfusion (ours)       |      24.63      |      0.860      |       0.107        |\n| Raindrop Dataset       |                 |                 |                    |\n| IDT [6]                |      31.87      |      0.931      |       0.058        |\n| WeatherDiff$_{64}$ [7] |      30.71      |      0.931      |       0.060        |\n| Resfusion (ours)       |      32.61      |      0.938      |       0.061        |\n\n> The proposed method seems to be sensitive to noise. Can you train the model for the denoising tasks?\n\nThe LOL-v2-real [8] dataset includes visual degradations such as decreased visibility, intensive noise, and biased color.\nAs shown in Figure 2 of the Author Rebuttal PDF, Compared to Histogram Equalization, \nResfusion can significantly reduce noise, while also achieving a better color offset, demonstrating strong denoising capabilities.\nWe provide results in terms of PSNR, SSIM, and LPIPS on LOL-v2-real dataset:\n\n| Methods             | PSNR $\\uparrow$ | SSIM $\\uparrow$ | LPIPS $\\downarrow$ |\n|---------------------|:---------------:|:---------------:|:------------------:|\n| LOL-v2-real Dataset |                 |                 |                    |\n| Restormer [4]       |      18.69      |      0.834      |       0.232        |\n| LLFormer [5]        |      20.06      |      0.792      |       0.211        |\n| Resfusion (ours)    |      22.06      |      0.839      |       0.175        |\n\nWe will certainly add experiments on the LOL-v2-real dataset into the revised paper.\n\n## References\n[1] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020.\n\n[2] - [7] briefed due to word limit restrictions.\n\n[8] Yang, Wenhan, et al. Sparse gradient regularized deep retinex network for robust low-light image enhancement. IEEE Transactions on Image Processing 30 (2021): 2072-2086.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0yKVDNU4M3", "reviewer": "Reviewer_KnJY", "summary": "The paper presents Resfusion that leverages prior residual noise to improve restoration performance. It introduces a smooth equivalence transformation for learning residual noise and demonstrates the efficacy of Resfusion through extensive experiments and ablation studies.", "questions": "See the weakness.", "limitations": "The authors have discussed the limitations clearly.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written and the idea is clearly illustrated. The technique of finding T' to simplify Eq 6 is clever. The experiments are detailed and show non-trivial improvements.", "weaknesses": "1. Since T' is smaller than the usually used T=1000, the sampling process of the proposed method seems to be faster than previous methods. But there are no comparisons on inference speed.\n\n2. What is the scheduling of \\alpahs? The authors are suggested to add more experimental analysis about the error introduced by ignoring terms in Eq 6.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review and thoughtful feedback. Below we address specific questions and comments.\n\n> Since T' is smaller than the usually used T=1000, the sampling process of the proposed method seems to be faster than previous methods. But there are no comparisons on inference speed.\n\nWe demonstrate the comparison of different methods' inference speed on the ISTD dataset, LOL dataset and Raindrop dataset.\nWhen testing inference speed, all images are resized to $256\\times256$, using single NVIDIA RTX A6000,  with all configurations being identical to Appendix A.6:\n\n| Methods                 | PSNR$\\uparrow$ | SSIM$\\uparrow$ | Inference Time (s)$\\downarrow$ |\n|-------------------------|:--------------:|:--------------:|:------------------------------:|\n| ISTD Dataset            |                |                |                                |\n| Shadow Diffusion [1]    |     32.33      |     0.969      |    0.024$\\times$25 = 0.600     |\n| Resfusion (ours)        |     31.81      |     0.965      |     0.027$\\times$5 = 0.135     |\n| LOL Dataset             |                |                |                                |\n| LLFormer [2]            |     23.65      |     0.816      |     0.092$\\times$1 = 0.092     |\n| Resfusion (ours)        |     24.63      |     0.860      |     0.027$\\times$5 = 0.135     |\n| Raindrop Dataset        |                |                |                                |\n| WeatherDiff$_{64}$ [3]  |     30.71      |     0.931      |     0.328$\\times$25 = 8.20     |\n| WeatherDiff$_{128}$ [3] |     29.66      |     0.923      |    0.439$\\times$50 = 21.95     |\n| Resfusion (ours)        |     32.61      |     0.938      |     0.027$\\times$5 = 0.135     |\n\nWe will certainly add the comparison of inference speed into Appendix A.6 of the revised paper.\n\n> What is the scheduling of \\alpahs?\n\nFor all experiments,\nwe used the truncated version of the Linear Schedule from reference [4] as the noise schedule for $\\beta_t$ (or $\\alpha_t$),\nwhich we refer to as the Truncated Linear Schedule.\nWe provide a detailed implementation of the Truncated Linear Schedule in Appendix A.7.\n\n> The authors are suggested to add more experimental analysis about the error introduced by ignoring terms in Eq 6.\n\nThe ground truth of $x_{T'}$ is formulated as Eq. (7):\n\n${x}_{T'}= (2\\sqrt{\\overline\\alpha_{T'}}-1) x_{0}+(1-\\sqrt{\\overline\\alpha_{T'}})\\hat{x}_{0}+\\sqrt{1-\\overline\\alpha_{T'}}\\epsilon$\n\nAnd the estimated acceleration point $\\hat{x}_{T'}$ is formulated as Eq. (9) or Eq. (15):\n\n$\\hat{x}_{T'}= \\sqrt{\\overline\\alpha_{T'}}\\hat{x}_{0}+\\sqrt{1-\\overline\\alpha_{T'}}\\epsilon$\n\nSince,\n\n$R = \\hat{x}_{0} - x_{0}$\n\nThe absolute value of the error can be derived as:\n\n$||x_{T'} - \\hat{x}_{T'}||$\n$=||(2\\sqrt{\\overline\\alpha_{T'}}-1) x_{0} + (1-2\\sqrt{\\overline\\alpha_{T'}})\\hat{x}_{0}||$\n$=||(1-2\\sqrt{\\overline\\alpha_{T'}})R||$\n\nAs shown in Figure 3 (a) of the Author Rebuttal PDF, the error $||(1-2\\sqrt{\\overline\\alpha_{T'}})R||$ exponentially decreases with the increase of T.\nWhen T is relatively small, this error is not negligible. Fortunately, we can eliminate this error through the technique of Truncated Schedule when T is small (when T is large, Truncated Schedule is actually consistent with Original Schedule).\n\nThe core idea is that the diffusion steps after the acceleration point are not involved in the actual diffusion process.\nWe provide a detailed implementation of the Truncated Linear Schedule in Appendix A.7.\nWe provide a detailed code implementation of the Truncated Linear Schedule in Author Rebuttal Section 3.\n\nAs shown in Figure 3 (b) of the Author Rebuttal PDF, when $T'/T=5/12$, Truncated Schedule can effectively eliminate \"residual shadows\".\nThe results of PSNR, SSIM, and LPIPS between Truncated Schedule and Original Schedule on ISTD dataset are provided:\n\n\n| Methods            | PSNR$\\uparrow$ | SSIM$\\uparrow$ | LPIPS$\\downarrow$ |\n|--------------------|:--------------:|:--------------:|:-----------------:|\n| ISTD Dataset       |                |                |                   |\n| Truncated Schedule |     31.81      |     0.965      |       0.030       |\n| Original Schedule  |     29.41      |     0.964      |       0.036       |\n\nWe will certainly add the analysis of the error into the revised paper.\n\n## References\n[1] Guo, Lanqing, et al. Shadowdiffusion: When degradation prior meets diffusion model for shadow removal. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.\n\n[2] Wang, Tao, et al. Ultra-high-definition low-light image enhancement: A benchmark and transformer-based method. Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 3. 2023.\n\n[3] Özdenizci, Ozan, and Robert Legenstein. Restoring vision in adverse weather conditions with patch-based denoising diffusion models. IEEE Transactions on Pattern Analysis and Machine Intelligence 45.8 (2023): 10346-10357.\n\n[4] Nichol, Alexander Quinn, and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. International conference on machine learning. PMLR, 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank the reviewers for their valuable and encouraging comments. Below we address specific questions and comments.

## Section 1: Definition of constant hyperparameters.

For all constant hyperparameters, our definitions are completely consistent with the reference [1].
We provide a detailed definition:
- $\beta_t = 1 - \alpha_t$
- $\overline{\alpha}_{t} =  \prod_{s=1}^{t}\alpha _s $
- the $\Sigma_{\theta}$ is taken fixed as $\widetilde\beta_{t} = \frac{1-\overline{\alpha}_{t-1}}{1-\overline{\alpha}_{t}} \beta_t$


## Section 2: Explanation of the derivation of $\mu_{\theta}$.

The derivation of $\mu_{\theta}$ corresponds to Eq. (19) - Eq. (23) in Appendix A.1. 
We provide a detailed explanation of the derivation process:
- The ground truth mean $\widetilde\mu$ for the resnoise-diffusion reverse process is derived from Eq. (18) and formalized by Eq. (19).
  We aim to approximate the unattainable $\widetilde\mu$ by learning a $\mu_{\theta}$.
- According to Eq. (20), we can simplify $\widetilde\mu$ in Eq. (19) to the form in Eq. (21). 
  By simply performing a change of variables ($x_0 \rightarrow x_0-R$, $x_t \rightarrow x_t-R$), 
  the derivation process becomes exactly identical in form to the derivation of equations (115)-(124) in the reference [2],
  where Eq. (20) corresponds to (115) and Eq. (19) corresponds to (116).
- According to Eq. (22), we can modify Eq. (21) as Eq. (23). 
  Since $x_t$ is obtainable, we only need to learn $res\epsilon$, and $\mu_{\theta}$ is formulated as Eq. (13).


## Section 3: Truncated noise schedule

For all experiments, 
we used the truncated version of the Linear Schedule from reference [3] as the noise schedule for $\beta_t$ (or $\alpha_t$), 
which we refer to as the Truncated Linear Schedule. 
We provide a detailed implementation of the Truncated Linear Schedule in Appendix A.7.

It is worth mentioning that in the Supplementary Material, 
we provide the relationship between $\sqrt{\overline{\alpha}_{t}}$, $T$, and $T'$ under the Truncated Linear Schedule in the ./assets/acc_T_change_table.xlsx file. 
We provide a detailed code implementation for the Truncated strategy (self._alpha_hat represents $\overline{\alpha}$):
```
self._sqrt_alpha_hat = torch.sqrt(self._alpha_hat)
idx = find_closest_index(self._sqrt_alpha_hat, 0.5)
if 0.5 - self._sqrt_alpha_hat[idx].item() > 0.01:
    # The value after the acceleration point is useless and can be discarded
    self._sqrt_alpha_hat = torch.cat((self._sqrt_alpha_hat[:idx], torch.tensor([0.5])))
    self._alpha_hat = torch.cat((self._alpha_hat[:idx], torch.tensor([0.5 ** 2])))
    self._alpha = torch.cat((self._alpha[:idx], torch.tensor([0.5 ** 2]) / self._alpha_hat[idx-1]))
    self._beta = 1.0 - self._alpha
```


## Section 4: Visualization

We present the visualization results of the five sampling steps and use the pretrained model on the LOL dataset to directly infer images from the Internet (without ground truth). 
The visual results are stunning, as detailed in Figure 1 in the rebuttal PDF. The PDF is about 48M, please be patient.

## References
[1] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020.

[2] Calvin Luo. Understanding diffusion models: A unified perspective. arXiv preprint arXiv:2208.11970, 2022.

[3] Nichol, Alexander Quinn, and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. International conference on machine learning. PMLR, 2021.

Meta-review: The paper introduces Resfusion, a novel framework for image restoration using denoising diffusion probabilistic models (DDPM). The key innovation is incorporating a residual term into the diffusion forward process, enabling the reverse process to start directly from noisy degraded images, thereby reducing the number of diffusion steps. The method shows competitive performance across multiple image restoration tasks, including shadow removal, low-light enhancement, and deraining.
We are glad the accept this paper because of its all positive ratings. The paper presents a solid contribution to the field of image restoration with innovative ideas. 
We encourage the authors to include the suggestions from the reviewers into their camera-ready version.
Decision: Accept (poster)